[Edge Computing Series #3] The Architecture of Micro Data Lakes
Andrew Sungjin Kim, CEO of Machbase
The Architecture at a Glance
The diagram below shows the architecture Machbase is working toward. Let us examine its components and the ideas behind them.
Transparent Data Access for Users
If I had to choose one phrase to summarize this series, it would be “transparent data access.” That is the central idea behind everything discussed so far.
What does it mean in an edge computing environment?
A. Abstracting the Physical Location of Data
The first requirement is to abstract away the physical location of data. In simpler terms, users should not need to distinguish whether the data they access is in the cloud or at the edge. Administrators can, of course, still see its location.
Earlier in the series, I described the conventional cloud-centered approach: move data generated at the edge into cloud storage or a central data lake, and build services around that central repository.
For users, having everything in one place is reassuring. They can analyze and use the data without worrying about where to find it—although the monthly costs may be less visible in that view.
A micro data lake should provide a similar experience at the interface level. Users access logical datasets such as “alarm data,” “raw data from equipment A,” or “trend data for the past year,” without needing to know the physical storage location.
The key is that these names contain no location information. If most data remains in micro data lakes, the cloud storage and data access charges associated with that data can be greatly reduced. The next section examines this point.
B. Reducing Integration and Management Costs
When most data is stored in micro data lakes, less cloud storage is needed.
Another benefit is that an office PC can access edge data through the cloud while most processing remains on the edge device. In cloud arrangements where incoming traffic is not charged, transferring intermediate results from the edge to the cloud can also avoid an ingress fee. Delivery from the cloud to the user’s PC may still be billable. The aim is to avoid moving and processing all of the raw data in a metered cloud service.
The savings come with a performance tradeoff: results travel from a physically remote edge device, through the cloud, to the user’s machine. In our experiments, the location of the data made little perceptible difference except when downloading large raw datasets through the cloud. Modern edge and cloud hardware helps make this practical.
Commercial cloud data lakes and database-as-a-service offerings commonly charge for both storage and data access or query processing. When large datasets must be analyzed centrally, those charges can constrain application design and use. Unexpectedly large bills are one consequence of insufficiently controlled resource consumption.
C. Flexibility and Freedom in Data Management
Part 2 described a company’s data processing difficulties. Some use cases cannot centralize edge data effectively even with substantial time and money; in others, the data loses much of its usefulness during transfer. Without an edge data lake, large volumes may be processed briefly and discarded, or stored indefinitely as text files that are difficult to analyze.
How does having a micro data lake change those choices?
For small data volumes, the difference may be limited. Above a certain scale, however, organizations can compare costs and benefits and choose whether to retain data at the edge or send it to the cloud. That is a significant increase in flexibility.
When the edge generates too much data to transfer, the data cannot be discarded, and timely analysis is required, a micro data lake becomes a practical way to meet those constraints.
What Belongs in the Cloud?
What should be stored centrally? The micro data lake concept is still being developed, so the answer is not fixed. A useful principle is to retain the key information needed to understand the overall service and data environment.
Rather than every individual edge record, centralize the data most useful for service operations and cross-system analysis. Examples include:
Alarms and events
Selected statistical summaries
Key trends and consolidated analytical results
Critical archival records
These are natural candidates for central storage.
Users can monitor and analyze this information in the cloud, then access the detailed data at the edge when a particular issue needs investigation.
As an illustrative 20:80 model, the cloud might hold about 20% of the data needed by the business while serving roughly 80% of data access requests. This is a conceptual assumption, not a measured universal ratio.
What Belongs at the Edge?
Micro data lakes at the edge are suited to:
Raw, historical, and detailed data
Related metadata retained for defined periods
For raw and historical data, the idea is to retain nearly all relevant measurements produced at the edge. They provide evidence for investigating the cause of an alarm or event at a particular point in the past.
Using the same illustrative 20:80 model, micro data lakes might retain more than 80% of the business’s data while accounting for about 20% of access activity.
Benefits and Tradeoffs
Benefits
The advantages can be summarized as follows.
Lower storage costs
Distribute storage and obtain more capacity for a given budget.
Lower data access and management costs
Avoid cloud-service query charges for data processed within the micro data lake.
Greater flexibility
Choose where data is stored and in what form, according to the architecture.
Support for high-frequency data
Store, analyze, and process vibration, current, and pressure readings arriving at thousands or tens of thousands of records per second.
Greater control over data security
Keep raw data on edge servers inside the organization or production site, reducing the need to send it outside.
More responsive services
Analyze and respond quickly because the raw data is available locally at the edge.
Protection against data loss during network outages
Continue storing data in the micro data lake when the edge-to-cloud network is unavailable.
Tradeoffs
No technology offers only advantages, and micro data lakes are no exception.
Potentially slower access to raw data
A direct cloud-data access path is conceptually two-tier, while a micro data lake introduces a third tier. Network and deployment conditions can therefore affect raw-data access performance.
More edge equipment to manage
Distributing data across multiple edge devices increases the work required to manage their hardware, software, and availability.
Maturity and operational reliability
At the time of this proposal, practical deployments of the micro data lake concept were still limited. Its technical constraints and operational behavior therefore required careful evaluation.
Potential Applications
The following areas could benefit from distributing data between edge storage and central services.
Machine tools
Steelmaking and rolling processes
Mobility data, including autonomous driving
Continuous processes, including battery manufacturing
Electric vehicle chargers and related equipment
Smart buildings, power generation, cities, and grids
Energy distribution, transmission, and power systems
Other data-intensive fields
This architecture is useful wherever edge systems collect large data volumes and central servers coordinate management and events. It can address the inefficiency of moving every record to one location before processing it.
The Future of Micro Data Lakes
If the remaining technical challenges can be resolved, I believe this architecture will become an important form of next-generation data processing. The benefits can outweigh the additional operational complexity, especially where enormous volumes originate at the edge. High-frequency analysis for manufacturing, fault diagnosis, and predictive maintenance is a particularly promising application.
Closing Thoughts
This article examined the architecture and its practical benefits and tradeoffs before moving into implementation. The next installment will explain how to build it.
