Store Machbase Backups in Amazon S3 and Access Data Easily by Mounting Them

2 hours ago

5 min read

Background

There is a question Machbase customers often ask:

If backup files from Machbase are stored in Amazon S3, can we access them directly?

Machbase’s backup and mount technology, patented in both the United States and Korea, is one of its most distinctive and powerful data management capabilities. When we originally designed the product, however, we had not envisioned storing backup files in a cloud service such as Amazon’s. If backups were stored in an inexpensive service such as S3, extracting their data required downloading or copying them and then restoring them.

This article therefore tests whether Machbase’s mount feature can be used in the environment of 2025, when cloud services are commonplace and many backup files are stored in inexpensive S3-compatible storage.

To learn more about Machbase’s backup and mount capabilities, see this article.

Test environment

Testing directly on Amazon’s infrastructure takes time and incurs costs. Instead, we will build an S3-compatible environment locally to verify S3-based mounting quickly and easily.

Operating system

We used Ubuntu 24.04, an accessible Linux distribution that is convenient for testing, running it through WSL on Windows. Linux has become remarkably easy to use even in a Windows environment.

S3 storage

Among the available open-source options, we chose MinIO because Docker makes it quick to test.

Because MinIO provides a locally hosted environment that is fully compatible with Amazon S3, we treated its behavior as equivalent for this test.

A filesystem for S3 — s3fs

For the Machbase database engine running on Linux to access portions of S3 files in the cloud or locally, that storage must be mounted as a filesystem in some form.

The files themselves remain in virtual storage. This is similar to accessing Google Drive through a filesystem: a particular local directory becomes a window into a specific part of a large S3 store.

We will use the s3fs package for this purpose.

We now have everything we need.

Starting MinIO

With Docker, anyone can start it easily using the following single command.

sudo docker run -d --name minio   -p 9000:9000 -p 9001:9001   -e "MINIO_ROOT_USER=admin"   -e "MINIO_ROOT_PASSWORD=password123"   quay.io/minio/minio server /data --console-address ":9001"

This command sets up the MinIO console at http://localhost:9001, with Access Key = admin and Secret Key = password123. You can log in and monitor the system in a browser, as shown below.

We have not uploaded any data yet, but the Upload button at the upper right of the image above lets us upload the contents of a Machbase backup folder.

Downloading the MinIO client

To create storage in MinIO’s S3 environment, download the MinIO client as follows.

wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
sudo mv mc /usr/local/bin/mc3

You can now manage buckets with mc3. We renamed it from mc to avoid a name collision with Midnight Commander.

Creating a test bucket

Create the bucket for the Machbase backup files with the following commands.

mc3 alias set local http://127.0.0.1:9000 admin password123
mc3 mb local/mybucket

A test bucket named mybucket has now been created, as shown below.

Uploading backup data

Use the Upload button above to upload the folder backed up by Machbase.

In this example, we uploaded the backup named sjkim to S3 storage, as shown below.

Installing s3fs

Install s3fs on Ubuntu as follows.

sudo apt update
sudo apt install s3fs

Preparing authentication for s3fs

To let s3fs connect to the S3 server, first create a file containing the username and password defined earlier.

Use the following commands.

echo "admin:password123" > ~/.passwd-s3fs
chmod 600 ~/.passwd-s3fs

Mounting S3 as a local Linux filesystem with s3fs

All the preparation is complete.

Now use s3fs to make the S3 storage containing the sjkim backup accessible through the local Linux filesystem.

Run the following command.

The mount directory is /home/sjkim2/work/s3.

sjkim2@PC:~/work$ s3fs mybucket ./s3    -o passwd_file=~/.passwd-s3fs   -o url=http://127.0.0.1:9000   -o use_path_request_style   -o dbglevel=info -f -o curldbg
2025-11-19T06:10:59.323Z [INF] s3fs version 1.93(unknown) : s3fs -o passwd_file=/home/sjkim2/.passwd-s3fs -o url=http://127.0.0.1:9000 -o use_path_request_style -o dbglevel=info -f -o curldbg mybucket ./s3
2025-11-19T06:10:59.323Z [CRT] s3fs_logger.cpp:LowSetLogLevel(239): change debug level from [CRT] to [INF]
....
2025-11-19T06:13:30.283Z [INF]       curl.cpp:RequestPerform(2520): HTTP response code 200
2025-11-19T06:13:30.283Z [INF] threadpoolman.cpp:StopThreads(195): Any threads are running now, then nothing to do.
2025-11-19T06:13:30.283Z [INF]       threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan.

Messages like these indicate that S3 has been mounted successfully. Let’s inspect the directory.

sjkim2@PC:~$ cd work/
sjkim2@PC:~/work$ ls -al s3
합계 4
drwx------  1 sjkim2 sjkim2    0  1월  1  1970 .
drwxrwxr-x 11 sjkim2 sjkim2 4096 11월 19 14:10 ..
drwxr-x---  1 sjkim2 sjkim2    0  1월  1  1970 sjkim
sjkim2@CEO-PC:~/work$ ls -al s3/sjkim
합계 24464
drwxr-x--- 1 sjkim2 sjkim2        0  1월  1  1970 .
drwx------ 1 sjkim2 sjkim2        0  1월  1  1970 ..
drwxr-x--- 1 sjkim2 sjkim2        0  1월  1  1970 TAG_TABLESPACE
-rw-r----- 1 sjkim2 sjkim2     1148 11월 19 14:54 backup.dat
-rw-r----- 1 sjkim2 sjkim2     6073 11월 19 14:54 backup.trc
drwxr-x--- 1 sjkim2 sjkim2        0  1월  1  1970 
Machbase_backup_19700101090000_20251113184807_130
-rw-r----- 1 sjkim2 sjkim2   159744 11월 19 14:54 meta.dbs-0
-rw-r----- 1 sjkim2 sjkim2   155648 11월 19 14:54 meta.dbs-1
-rw-r----- 1 sjkim2 sjkim2     8192 11월 19 14:54 meta.dbs-2
-rw-r----- 1 sjkim2 sjkim2 24707072 11월 19 14:54 meta.dbs-3
-rw-r----- 1 sjkim2 sjkim2        0 11월 19 14:54 meta.dbs-4
-rw-r----- 1 sjkim2 sjkim2    12288 11월 19 14:54 meta.dbs-5

All the files uploaded to S3 are visible.

Mounting the backup in Machbase

Can we mount and query a backup while it remains compressed in an S3 directory? Let’s run the following commands and find out.

MOUNT DATABASE '/home/sjkim2/work/s3/sjkim' to s3_backup;

We mounted the backup files in /home/sjkim2/work/s3/sjkim and checked which tables were available.

Mounting finished after about five seconds. We can see dozens of tables with S3_BACKUP in the DB_NAME field.

Let’s inspect the tag table.

The mounted database’s tag table contains 10 million records, with 10 tags containing one million records each.

Finally, let’s query a specific time range. We ran a query to extract a ten-second interval from the full set of 10 million records.

It works! Isn’t that satisfying?

By simply mounting a compressed Machbase backup stored in S3, we can now extract the data inside it directly.

The query takes roughly one second, which appears to be the time needed to load the backup file chunk containing the requested data. On a regular filesystem, it runs in less than 0.1 seconds.

The corresponding s3fs log is shown below.

The highlighted section shows that a database chunk of approximately 39 MB was loaded.

Because S3 access involves loading sizable files, here are some tips for improving costs and performance.

Tips for improving performance and reducing costs

S3 files are accessed over HTTP, so excessive file access can increase costs and reduce performance.

Backup files, however, have the following characteristics:

They consist of file-based chunks.
They are stored in compressed form.
They are read-only.

These characteristics suggest that caching in s3fs could substantially reduce costs by loading each file only once. Because the files are read-only, their contents do not change. With help from ChatGPT, we identified the following options for loading and reusing the files we need.

Cache type	Default	Description
File content cache (--enable_noobj_cache)	❌ Disabled	Caches information about nonexistent files
Directory cache (--stat_cache_*)	small	stat cache
Metadata cache (--enable_content_md5)	❌	Performance impact
Memory cache (-o use_cache)	❌	file-level local cache

Enabling these options instead of leaving the defaults may help reduce costs and improve performance.

Closing thoughts

At first, I had not imagined that backup data stored in S3 could be mounted and used directly. After trying different approaches, however, we confirmed that Machbase backup files in inexpensive storage such as S3 can be mounted without separately modifying or copying them, and that their contents can be queried with SQL.

I hope this article offers practical help to anyone considering how to manage, back up, and reuse data in a Machbase database.

For further questions or suggestions, please contact our official support address at support@machbase.com.

Thank you.