[Edge Computing Series #4] Implementing a Micro Data Lake
Sungjin Kim, CEO of Machbase
If this is your first article in the series, please read the following installments first.
Introduction
This article explains a working micro data lake implemented by Machbase and the components of its architecture. Ideally, we would describe a customer deployment, but security and information-disclosure restrictions prevent us from doing so. Instead, a later installment will examine a working deployment with a similar structure. Before looking at that example, we will explore the technologies selected to build the data lake and the reasoning behind the implementation.
The Core Data Technology at the Edge: Machbase Neo, a Time-Series Database
We have reached the central question. As discussed earlier in the series, an edge device must process large volumes of real-time data despite limited hardware resources. We suggested that the difficulty of meeting this requirement helped make centralized data management in the cloud or on servers the dominant model.
Machbase selected its own time-series database engine as the core data-processing software because it offers sufficient performance and functionality for a data lake despite these constraints. How well does it actually perform on edge devices? We tested several devices and measured the results below. More powerful edge devices and servers may deliver several times the performance shown here.
Performance Across Edge Hardware
The following table summarizes specifications and data-processing performance for devices ranging from very modest hardware to more capable edge systems.
INSERT measures maximum ingestion throughput while fully utilizing CPU and I/O resources. SELECT measures the time to retrieve 80,000 sensor records within a particular time range after the entire dataset has been loaded.
Production edge servers are often industrial PCs, or IPCs, designed for durability and long-term operation under demanding field conditions. The devices below are not all industrial PCs, but they provide a useful indication of edge-processing performance.
Device | CPU / OS | Memory | Insert performance (records/s) | Total records (storage) | Select: 80,000 records (seconds) |
TURCK IM18-CCM60 (eMMC) | 32-bit ARM, 1 core (Linux) | 1GB | 41,582 | 10 million (44MB) | 0.06 |
Raspberry Pi 4 (USB SSD) | 64-bit ARM, 4 cores (Linux) | 8GB | 199,547 | 100 million (500MB) | 0.04 |
vRaptor PEC (SSD) | 64-bit ARM, 24 cores (Linux) | 16GB | 492,219 | 1 billion (4.5GB) | 0.048 |
ODROID-M1 (SSD) | 64-bit ARM, 4 cores (Linux) | 8GB | 809,644 | 313.3 billion (1.8TB) | 0.021 |
ODROID-H3 (SSD) | 64-bit Intel, 4 cores (Linux) | 8GB | 2,317,972 | 1 trillion (3.5TB) | 0.015 |
Storage was provided by eMMC or SSD. On the Raspberry Pi, we used an external SSD connected over USB 3.0 because direct installation was less convenient. This limited ingestion performance. A directly connected PCIe SSD could potentially improve performance by two to three times or more.
Data schema
The schema contains three basic columns for sensor data.
CREATE TAG TABLE tagtbl (name VARCHAR(20) PRIMARY KEY, time DATETIME BASETIME, value DOUBLE SUMMARIZED)Extraction query
For a particular sensor, the query extracts 80,000 records covering 30 minutes from the full dataset, which may contain 10 million, 100 million, or even 1 trillion records.
select count(*) from (select * from tag where name = 'EQ0^TAG1' and time between to_date('2019-01-01 00:00:00') and to_date('2019-01-1 00:30:00'));About the Edge Devices
Having summarized the results, we will briefly introduce the hardware used in the tests.
Developed by the German company Turck, this compact device is installed inside equipment cabinets to collect and monitor data. It was not designed with a DBMS in mind: it has one modest ARM CPU core, only 1GB of memory, a customized 32-bit Linux system, and approximately 8GB of total storage. Nevertheless, we installed Machbase and used it as an edge device. It collected more than 40,000 sensor records per second while allowing real-time queries. It may be too small to call a full micro data lake, but it is a useful example of performance on highly constrained hardware.
The Raspberry Pi is a familiar example of a small edge device. It is widely used for both testing and field deployments, with abundant documentation and many potential applications. Machbase installed its database on the device and ran various tests. Linux distributions such as Ubuntu and a rich supporting ecosystem make it possible to build a capable micro data lake at low cost, provided storage and enclosure requirements are addressed.
One drawback is the default microSD storage. Under I/O-intensive database workloads, reliability can be inadequate; the author notes failures within a year. An additional hardware arrangement is therefore needed to connect a suitable SSD. Modules for this purpose are available.
V-Raptor is a family of compact ARM servers developed by the Korean company XSLAB. The tested model has 24 cores and offers an attractive balance of price and performance. The company also offers the compact V-Raptor SQ Nano family. Although there are many cores, their clock speed is around 1GHz, so workloads requiring fast processing in a single session may need patience. Fanless operation, low power consumption, and suitability for sustained use make these ARM servers an interesting alternative to Intel systems.
The ODROID-M1 is a small board computer from the Korean company Hardkernel, in a similar category to the Raspberry Pi. Its specifications resemble those of the Raspberry Pi 4, while offering a low-cost platform with plenty of supporting information. Installing an SSD is straightforward; we ran our tests directly from a Samsung SSD. Its ARM CPU is relatively modest, reflecting the choice of a Rockchip processor intended for long-term availability. It may be less suitable for extracting very large real-time vibration datasets, but it delivers excellent performance and stability for ordinary sensor-data collection and use.
The same manufacturer offers the ODROID-H3 with an Intel Celeron processor. Its power consumption is higher than that of the ARM systems, but overall benchmark performance is also substantially higher. It includes a cooling fan.
With a Samsung SSD connected through PCIe, ingestion exceeded two million records per second. Even with one trillion records accumulated over a long period, queries returned results in less than 0.1 seconds.
Actual field deployments often use finished industrial systems such as Lenovo ThinkEdge or Advantech IPC products. The author expects such systems to provide at least the performance level of the ODROID-H3.
Is a Time-Series Database Enough to Serve as the Core Repository?
As discussed earlier, a micro data lake has a specific purpose. If its business scope is the processing of large sensor datasets, a time-series database already has the essential qualifications to act as the micro data lake itself.
Real-time compression allows approximately 100 billion sensor records to fit in just 512GB of storage. With 8GB of memory and four to eight CPU cores, it can ingest hundreds of thousands of records per second.
Providing comparable storage and processing through a cloud service can incur substantial storage-management costs. Adding the API costs of subsequent real-time extraction and transformation highlights the potential economic advantage of this architecture.
To put 100 billion records in perspective, it is roughly the volume produced by continuously storing 1,000 records per second for three years. That much data can fit in 512GB.
What About the Other Functions of a Micro Data Lake?
A time-series database is an excellent repository, but a data lake requires more than storage. How should the remaining capabilities and deployment constraints be addressed?
Examples include: sending selected edge data to a server in real time; protecting communication between the two tiers with encryption; and remotely operating, managing, and updating dozens of edge devices. Requiring users to implement and validate all of this themselves would be complex. These functions must be part of the overall micro data lake service.
For this reason, Machbase developed CEMS, the Cloud-Edge Management Solution, as an edge-computing data-infrastructure platform for micro data lake deployments. Its detailed specifications are outside this article’s scope. Here, we focus on how the data lake operates on the edge device.
Mapping Requirements to Implementation
We will now assess how Machbase addresses the requirements discussed throughout the series. Assume the specific goal is to build a data lake for large-scale sensor-data processing on edge devices.
Detailed Analysis
The following table outlines the technology used to meet the general requirements.
Requirement | Implementation |
Hardware requirements | Hardware supported by Machbase Neo: ARM or Intel, limited-resource devices, and multiple operating systems |
Structured sensor data | Tag tables |
Sensor ingestion: storage throughput | Approximately 40,000 to 2 million records per second |
Sensor ingestion: indexing throughput | Real-time indexing at up to 2 million records per second |
Sensor ingestion: compression | Up to 100:1 real-time compression; typically 4.5–7 bytes per sensor data point |
Data extraction and search, with 100 billion stored records | Retrieve 10,000 records for a sensor and time range in less than 0.1 seconds |
Statistical processing, with 100 billion stored records | Rollups support monthly or yearly charts for a sensor in less than 0.1 seconds |
Data security: storage encryption | Use operating-system encryption to protect data if storage is physically removed |
Data security: communication encryption | End-to-end encrypted communication using X.509 certificates |
Data security: physical distribution | An inherent characteristic of micro data lakes |
Scalability | Scale-out is not required for this scope; scale-up is possible |
High availability | Requires future support or customization |
Integration with cloud data | Available through CEMS |
These capabilities show that Machbase Neo can provide the core of an edge-based micro data lake. It also offers the following additional features for edge deployments.
Additional Features
Machbase Neo feature | Details |
Built-in MQTT server | MQTT clients can send data directly to the DBMS without a separate server |
Built-in web server | Use REST APIs over HTTP and build services without installing a separate web server |
Built-in SSH server | Access and operate the edge database over secure SSH connections |
X.509-based encrypted connections | Strengthen data security |
Multiple CPUs and operating systems | Windows on Intel; Linux and macOS on ARM or Intel |
Built-in dashboards | Fast data visualization |
Integration with external tools | Support for Grafana and Telegraf |
Integration with external databases | Connect to SQLite, SQL Server, MySQL, PostgreSQL, and others to insert or retrieve data |
TQL data-transformation language | Build and serve backend APIs without separate application programming |
Built-in analytical tools | Tag Analyzer supports short- and long-term visualization and analysis, including 2D and 3D FFT charts. See the FFT article. |
Fully web-based DBMS management | Perform database operations through the web interface |
The following video provides an overview of Machbase Neo’s features. Please watch it for more detail.
All of these features run on hardware ranging from palm-sized devices to large servers. This makes Machbase Neo a data-infrastructure platform capable of supporting data lakes in many different environments.
Conclusion
In this fourth installment, we examined why Machbase Neo was chosen to implement a micro data lake and how it meets the overall requirements. To keep the discussion focused, we only briefly introduced CEMS, the additional software layer used in a complete deployment. The final installment will examine a working data lake and consider future directions.
At the time of writing, much of the market is either unaware of these possibilities or considers them technically out of reach—perhaps three years away. I hope these innovations are adopted sooner and help Korea become a country that leads the way into that future.
