Run Ubuntu server from eMMC and limit write processes?

Support and Help
,
1

Ubuntu Support Template

Ubuntu Version:
Server 26.04 LTS,

Desktop Environment (if applicable):
headless

Problem Description:
I’m looking to run Ubuntu server 26.04 from eMMC storage on a x64 soc setup. To prolong the life of the installed eMMC module, I’d like to limit write processes and have the system run in ram.

What’s the most easiest way to limit writes, yet save log files and stuff on a periodical basis and on shutdowns?

Relevant System Information:
x64 system on a chip (Hardkernel Odroid H4+)

2

Deploying Ubuntu Server 26.04 on eMMC for an x86_64 SoC like the Odroid H4+ requires a strategic approach to I/O management to prevent premature flash degradation. To achieve a “run-from-RAM” behavior while maintaining periodic persistence for logs, I recommend the following implementation stack:

1. Root Filesystem Abstraction (OverlayFS)

The most robust method for your hardware is using OverlayFS. This allows the operating system to treat the eMMC as a read-only (RO) base layer, while all subsequent writes are redirected to a temporary upper layer in RAM (tmpfs).

You can utilize the overlayroot package available in the standard Ubuntu repositories.

Edit /etc/overlayroot.conf and set overlayroot="tmpfs". This ensures that upon reboot, the system reverts to its original state, effectively eliminating wear during runtime.

2. Persistent Log Management (Log2Ram)

Since you require logs to be saved periodically, a standard read-only mount is insufficient. I suggest implementing Log2Ram.

It creates a virtual block device in RAM for /var/log.

It includes a daily cron job and a systemd service that synchronizes the RAM-based logs back to the physical eMMC during shutdown or at a specified LOG2RAM_SCHEDULE.

sudo apt install git
git clone https://github.com/azlux/log2ram.git

3. Periodic Data Synchronization

For “periodic basis” saves outside of logs, you should utilize a custom rsync script coupled with a systemd timer.

Avoid frequent writes; a 6-hour or 12-hour interval is usually a healthy compromise between data safety and eMMC longevity.

Ensure you remount the partition as Read-Write (mount -o remount, rw /) only during the synchronization window, then immediately revert to Read-Only.

4. Swap Optimization

Ensure that vm. swappiness is set to a minimal value (e.g., 10) to prevent the kernel from utilizing the eMMC as virtual memory, which is a common cause of high write amplification.

3

I have some minor things to add, and a word of caution. Syncing logs to the lowerdirs of overlayfs, while it is mounted, i.e. by schedule, may exhibit undefined behavior. Maybe consider some external USB thumb drive as log target or some networked logging.

Also, I’d consider vm.swappiness=0, which forces the kernel to only consider swapping if there is no page cache to evict. It is usually discouraged to do that, but fits the bill here, I think. Or just use vm.swappiness=1.

Plus, zswap is your friend in avoiding swapping; consider the zstd compressor, which strikes a great balance between compression efficiency and throughput. You could also take a gander at swap on zRAM, but I don’t like that approach very much and neither do the kernel developers; some highly recommended reading on the matter, comparing both.

And last but not least, be aware that doing everything in tmpfs does put memory pressure on your system; and tmpfs is swappable. If you can live with an occasional hard crash due to an out of memory (OOM) condition, you might consider disabling swap altogether, but your logs might go down with that crash as well.

4

I’ve also just remembered that systemd-journald, which should be the major source of writes, I reckon, is quite sophisticated and can be set to use non-persistent storage in /run, which is a tmpfs. There could be a way to use a systemd.timer(5) to regularly flush that journal to a “remote” host, which is actually localhost. That way you run little risk of actual write amplification, because it will just be one big-ish blob to write to persistent storage.

Just throwing this in the mix of things to consider, to maybe avoid the undefined behavior of altering lowerdirs of overlayfs. There’d have to be a separate partition for /var/log, which can be excluded from the overlayfs. Mount that in /var/log/journal/remote/ on top of that overlayFS or whatever and use that as the log location of said “remote” journal. Maybe that logging partition should be an FFS (Flash-Friendly Filesystem).

I’m thinking something like this as the timer triggered flush service:

journalctl --output=export | /usr/lib/systemd/systemd-journal-remote -

May need this first:

sudo apt install systemd-journal-remote

Or you could just increase the SyncIntervalSec= setting of journald. That has the benefit that really critical messages still get flushed to persistent storage right away.

5

Thanks for the valuable suggestions, to which I’ve some questions. But first, some background on my use case that might be helpful:

I’m trying to setup a small, power-efficient home server that is going to run a very limited number of containers.

As a basis, I’d like to use Ubuntu Server because of the awesome long-term support of Ubuntu Pro and the low idle power values I was able to achieve with 26.04. Besides cockpit, powertop, snapraid and megerFS, all software is going to run in a Docker or Podman container.

For an easier backup setup, Docker/Podman is going to live on a secondary SSD and write appdata and configuration files to that drive and not the eMMC boot drive.

Back to my questions:

  1. How does OverlayFS handle required updates made by unattended-upgrades? Would those updates be gone by the next boot, when the entire system is in a read-only state? Or would this be a case where a systemd timer is required?
  2. Would running Log2Ram and lowering the swapiness-value alone be efficient enough to limit writes? Or is OverlayFS recommended?
  3. Does Log2Ram also cover the writes caused by systemd-journald @peterwhite23 mentioned? Or should you use both approaches at the same time?
6

But then why bother with a special setup to limit writes on the eMMC device? You can just put /var and whatever else is supposed to see write churn on a separate partition on that secondary SSD.

7

I wasn’t sure which parts of Ubuntu actually cause the most writes on the eMMC. So I could boot from the eMMC but “outsource” the folders with have write churn to the secondary SSD? Would system updates still be installed to the eMMC in such a setup?

Is there some kind of overview, which parts I should “outsource” besides /var? And do I need a dedicated partition for that to work?

1 Like
8

Yes, you absolutely can have different parts of the virtual filesystem (VFS) on different hardware; that’s the beauty of the VFS approach. :smiley:

I am not quite sure if there is such an overview, but maybe have a look at the Filesystem Hierarchy Standard and read between those lines. That’s also not quite an Ubuntu-specific question, so you could look for more general advice on the internet. But there shouldn’t be that many locations. Essentially all the software which is updated by APT is in /usr, so that should be on the eMMC. There is one exception to that, because Snaps are installed in /var/lib/snapd and /var/snap. I don’t know about Flatpak, though, but given it’s focus on the desktop that may not concern you at all. Some 3rd party software may also be installed in /opt.

You could also consider approaching this from the other direction, by installing / to that SSD and only have /{usr,opt}, and possibly /var/snap and /var/lib/snap on the eMMC.

9

This topic was automatically closed after 30 days. New replies are no longer allowed.