The Book of Container Myths

[cjdc]

The Book of Container Myths

Disclaimer: The following content contains facts about containers. Reader discretion is advised. Some beliefs you hold dear may be challenged, gently debunked, or — in extreme cases — shattered entirely.

I’ve been developing and using containers for many years now. I have seen tools come and go, ideas hatch and die, and myths calcify into gospel — repeated so often and so confidently that they stopped being questioned. This is my attempt to challenge them.

And don’t get me wrong — I’ve been complicit in spreading these dogmas, too. I’ve parroted them, shipped them, and defended them in situations where I shouldn’t have. But hey, what matters is that we keep evolving and ensure our containerization practices aren’t driven by dogmatic fallacies. To quote a great philosopher:

---

Myth I: Thou Shalt Have One Process Per Container

Why, oh why did we mistake a process count for a design principle?

This is the one I hear the most, and it’s a bastardized interpretation of a more nuanced container concept — “function isolation”. A rigid rule, turned into a developer myth, that often leads to fragile, overly complex, and difficult-to-maintain container stacks.

I’m less triggered when I hear “one service per container”, because it’s less mechanical and closer to the functional concept, but still…

Building by function aligns much better with how applications actually work. Sometimes the application needs helpers, or maybe has an adjacent component whose lifecycle is fully dependent on the application’s main process.

The funny thing is - developers actually try to containerize by function. The paradox is the ideology vs implementation - they design a container that follows the “one process per container” myth by creating a simple wrapper script as the entrypoint (commonly referred to as entrypoint.sh), and this script is equipped with some basic logic that allows the container to be started with the right application helpers and sometimes even toggle between application binaries, depending on the container’s runtime arguments.

Don’t get me wrong - my beef is not with the entrypoint.sh script itself, but rather with the frequent overengineered and brittle adoption of this script for containers that are more complex than your typical single-binary application.

You - the developer - need to understand the application you’re about to containerize. In many (I’d say most) cases, you’ll want to treat every process inside your container as a manageable and traceable unit. That includes being able to monitor each individual process, support multiple child processes, as well as reaping, signal and log forwarding, and graceful shutdowns. You see where I’m going with this…use a process manager as the container’s init system (PID 1)!

By “process manager” I’m not referring to a rudimentary transformation of the entrypoint.sh script into a controller of Bash jobs. Instead, you’ll find existing tools that have been specifically designed to act as the container’s PID 1 and as a process supervisor.

From minimal PID 1-specific tools like tini (natively supported by Docker and others like it, via docker run --init ...), to container-optimised lightweight service managers like s6-overlay and pebble that provide features like:

• declarative definition of services,
• support for multiple child processes,
• reaping and subreaping,
• signal forwarding,
• graceful shutdown,
• log rotation and log forwarding.

If you think about it, the introduction of a common and well-known PID 1 for your containers even brings the advantage of having a predictable and consistent UX across your entire container stack for typical image inspection and container management operations (e.g. describing the container services, defining health checks, getting individual logs, etc.).

TLDR:

• Don’t obsess over process count — design containers by function, not by process.
• Complex containers benefit from a proper init system (PID 1) — not a brittle entrypoint.sh.
• Tools like tini, s6-overlay, and pebble give you process supervision, signal forwarding, reaping, and graceful shutdown out of the box.
• A shared PID 1 across your container stack brings consistency and predictability.

Myth II: Thou Shalt Trust Distroless Images

Why, oh why did we start measuring container quality in megabytes?

Hey, “distroless” is great! It is a container concept that I believe has been instrumental in the way developers design their containers. At the same time, it has become one of the biggest logical fallacies I’ve seen around in the container space, where developers simply appeal to the authority of the distroless nature of their container image and its minimal size, to claim that therefore it is as secure as it can be.

Well, “the price of greatness is responsibility”. Distroless is an architectural design concept that doesn’t remove the need for developers to be disciplined and employ critical thinking while building their containers. And yet, somehow, “it’s distroless” has become the container equivalent of “it’s organic” — slap the label on and stop asking questions. The traditional way of approaching a distroless build is less straightforward than your typical non-distroless one. It normally involves a top-down approach with a multi-stage build where you first bloat a staging environment and then copy the runtime application and dependencies to another stage, with a pristine and scratched environment. Well, so far so good - just because the process is more involved doesn’t mean it cannot produce something great. It absolutely can. But complexity comes with costs and risks:

• Build recipes and processes become more complex and thus harder to maintain (sometimes including specialized tooling)
• There’s typically the need to have more distro expertise in order to go distroless
• Top-down builds often mean cherry-picking the contents for the final container image, which is, in my opinion, an error-prone process that may lead to the existence of CVE False Negatives — and this is my main gripe.

Being small and distroless isn’t all it takes to have a secure image. I honestly believe that not knowing about a security vulnerability is MUCH worse than being flagged for one that actually doesn’t exist (i.e. False Positive) or even having a real one. The bottom line is that all containers MUST be auditable!

Believe it or not, most container scanners rely on information that is inside the container in order to infer what’s inside and use that to find where your container may be vulnerable. In other words, if you manually delete your image’s dpkg metadata, your scanner may find Zero Packages and thus report Zero Vulnerabilities - great! Your CVE report is all green now…

🠮 These are CVE False Negatives, and they are dangerous! You can’t take precautions against risks you don’t know about.

Just imagine if this were a medical report. I’m sure you’d much rather be told “hey, it was a false positive! you’re all good ” than “I’m really sorry but our scanners missed a spot and actually it turns out you have Caffeine-Induced Jitter Buffer Overflow”.

Now, one of the main reasons why False Negatives happen is because during the process of building a distroless image, developers leave behind important package metadata that the scanners rely on for finding CVEs. And most times they don’t do it intentionally. Typically, the package manager is responsible for managing such metadata, but the traditional top-down distroless builds don’t rely on package managers to prime the desired software into the final image. And that’s a problem - the inability to detect software components not managed by package managers.

So what’s the takeaway? Well, if most scanners are designed to rely on distro-specific data, then you gotta carry that information when creating distroless images. You gotta have discipline. There are tools out there that will behave like a package manager for distroless filesystems, and thus ensure your image is shipped with the necessary metadata for it to be auditable. For example, Chisel - *shameless plug* - allows you to generate minimal, application-centric, slices of an Ubuntu filesystem, from the ground-up, while also generating a manifest file with all the metadata required for a scanner to detect the software inside (e.g. snyk container test <chiseled-image>).

TLDR:

• Distroless is a design concept, not a security guarantee.
• Small image size ≠ secure image. All containers must be auditable.
• CVE False Negatives (missed vulnerabilities) are far more dangerous than False Positives.
• Traditional top-down distroless builds often strip package metadata that scanners need.
• Use tools that preserve metadata (e.g. Chisel) so your images remain scannable.

Myth III: Thou Shalt Always Roll With The Latest

Why, oh why isn’t stability a de-facto standard for container tags?

Somewhat related to the previous “Cult of the Green CVE Dashboard” myth, there is this tendency for users to follow OCI tags that move really fast, usually alongside the upstream, without any sense of stability.

I am not strictly talking about the usual suspect - the latest tag - but rather about the inherent obscurity behind the stability associated with any given floating reference (aka OCI tag).

'Cause guess what, depending on the underlying release channel a latest tag refers to, I might accept it for non-production environments!

latest Definition Strategy	Example	Acceptable?
Latest Stable Release	The most recent official, generally available (GA) release (e.g., updated from v1.2.3 to v1.2.4).	but, while this ensures users pulling this tag receive the most recent, production-ready, and fully tested version of the software, it also allows for new major (and potentially disruptive) versions of the software to be introduced unwittingly.
LTS Rolling	The newest release of the currently supported Long Term Support (LTS) track, even if a newer major version exists (e.g., pointing to v18.x.x even when v20.x.x is out).	but, it is similar to the above. We can say it is more stable, since it may skip major versions that will stop receiving updates in the near future. But still not to be taken lightly.
Continuous / Bleeding Edge	Every single merge or commit to the default branch (main or master).	This is like playing Russian Roulette with your working environment. Highly unstable. Users pulling this tag are exposed to untested code, mid-feature commits, and unpredictable breaking changes. For clarity, many maintainers often refer to this as the edge tag (which I prefer, because it conveys the stability level).
Scheduled / Nightly Build	An automated build generated at a specific time interval (e.g., midnight) capturing the current development state.	It is unacceptable for the same reasons as the bleeding-edge strategy. This (often called nightly) tag violates the principle of least surprise, even if potentially introducing edge changes less regularly than edge.

It seems like I’m focusing on the latest tag, but in reality this applies to any floating tag, even if the tag is pinning a version! And this is the controversial part …

the 1.0.x tag is better than latest

Yes, absolutely. But if you think about it, that pinned tag isn’t actually pinned (for that you’d use digests). It’s still a floating tag, simply referring to a narrower stream of releases, still with no concept of stability.

Users who defend the stability and security of a 1.0.x tag often use the argument of “semantic versioning acting as an implicit public contract between developers and users”. But, think about it, “newer” isn’t always “safer”: when you pull a new version of a vulnerable image, you aren’t just pulling the CVE fix - you are pulling lines of code, patches, that relate to small refactors and bug fixes. And with each line of code, there’s an inherent risk - so, not only are you trusting developers to follow the semantic versioning contract to the letter, you are also trading a known CVE for a potentially unknown, undiscovered, and unfixed vulnerability.

This is why CVE backports are employed in many container supply chains (e.g. ubuntu/ images - *yet another shameless plug*). CVE backports are one of the backbones of stability. The image maintainers should not pull every single patch into the 1.0.x tag, but rather focus on rebasing the source with the minimal amount of changes to ensure image security and stability, fixing vulnerabilities without introducing volatility.

One of the strategies followed by the ubuntu/ Docker images is to tag every image with a notion of “stability” - i.e. tags will have a stability suffix, ranging from _edge to _stable, allowing a new version of the image to be quickly available in _edge, and progressively gain confidence to make its way into _stable without further changes.

TLDR:

• Most floating tags — even those like 1.0.x — miss a built-in guarantee of stability.
• “Newer” isn’t always “safer” — every update trades a known CVE for potentially unknown risks.
• CVE backports are the backbone of stability: fix vulnerabilities without introducing volatility.
• Look for tags with explicit stability signals (e.g. _edge, _stable suffixes).

Myth IV: Thou Shalt Label Only Thy Final Containers

Why, oh why is label usage treated as an exclusive privilege of the final application container?

This one is slightly related to the above, because something I’ve heard before is

Don’t label base images because then all other derivative ones will inherit their labels, and I can’t unset them

Again,

In this case, I see a couple of red flags:

1. The definition of “base” and “parent” images shouldn’t, on its own, be the decision-maker for whether an image should have labels or not
2. The fact that your tool can’t manipulate labels, doesn’t mean that they cannot be manipulated

To elaborate on these two things, you must remember that even the so-called “base” images have their own purpose beyond just being a “parent” for downstream builds.

Labels, by definition, are useful mechanisms for adding metadata to your container image. While one may argue that there is some overlap between labels and the more modern OCI annotations, the reality is that these sets of metadata live in different locations inside the OCI archive. Same house, different rooms. As such, in today’s state-of-the-art OCI specifications, they still serve different purposes.

While OCI annotations are much more registry-friendly, labels are still widely used for inspecting images locally and sometimes even for driving custom admission tooling.

I can see a case to push more strongly towards OCI annotations, and in fact I do this myself. I do favour OCI annotations. But I won’t go to the extent of completely getting rid of labels just yet, and definitely not because they are “hard to unset”. Redefine them in your Dockerfile, write a multi-stage build, use a tool like crane or umoci…the options are there.

TLDR:

• Use labels if you need them.
• Ensure labels serve a purpose.
• Follow a predictable labelling convention.
• Adopt OCI annotations.
• Expand your tooling and/or build processes to manipulate labels.

Myth V: Thou Shalt Settle For Any Non-Root User

Why, oh why aren’t we more disciplined about how non-root images are designed?

Non-root. Yes. Do it!

However, don’t just USER 1. Sure it’s non-root, but it’s also not the safest.

Understand that the goal behind running your containers as non-root is to protect yourself from an eventual container breakout where the malicious root process inside the container emerges onto the host machine with full root privileges. Enabling user namespaces is an extra protection, but employing non-root container users is still the ultimate goal, even from an application design perspective - the principle of least privilege.

So, what should you consider when making your container non-root?

It is not just about having a UID != 0. Containers are meant to be portable. Linux distributions evolve, and so do their policies. And while the exact user boundaries may vary slightly depending on the specific Linux distribution and its age, modern Linux systems generally follow a standard convention for allocating UID ranges. As a rule of thumb, today, you want to steer away from UID < 1000, as those are typically reserved for root (of course) and system users. This is why you’ll see many Docker images out there where the non-root user is 1000 or 1001.

If this is already the case for many Docker images, why am I talking about it?

Well, first, it’s not the case for all yet .

Second, I believe there’s still something to be said about your choice of UID. A UID >= 1000 refers to dynamically allocated user accounts. So it will be quite normal to find such UIDs already existing in your host system. For example, you’ll find Ubuntu instances (especially in cloud environments) where there is an ubuntu user with UID = 1000. So, would it still be safe to set your container user to UID = 1000 as well? Probably not, for the same reasons as before - sure, it’s not root, but it is still an existing host user that you don’t want to compromise.

So what’s the takeaway?

Choose a high, non-reserved UID with the lowest probability of collision with the host (see Rockcraft’s example - *another shameless plug*).

If you have User Namespaces enabled in Docker/Podman/Kubernetes, you’ll notice this already, since the default subordinate UID ranges go beyond UID 65536.

As a final remark, to maximize the portability and interoperability of your containers, you also want to be consistent in the choice of UIDs for your containers. Going non-root means there will be complexity regarding file permissions and volume mounts. You will struggle with permission denied errors when the non-root container attempts to access or modify mounted volumes or bind-mounts. This is hard to avoid, and in most cases it is a good thing, requiring you to be more prescriptive about the data management policies for your container deployments. Nonetheless, my suggestion to you is to strive for consistency in your choice of UID across your containers, especially the ones that need to share data.

TLDR:

• Non-root is essential.
• Avoid UID < 1000 (reserved for root and system users).
• Choose a high, non-reserved UID to minimize collision risk with host users.
• Be consistent in your UID choices across containers that share data.

Myth VI: Thou Shalt Not Embed Image Metadata

Why, oh why is it considered hearsay for a container application to know a little bit about its own image?

I want to finish with this one because it is actually quite close to me. Many times in the past, I found myself — not always for good reasons (I’ll admit) — trying to implement my container application in a way that it could read information about the image it started from.

The obvious question is “why would you want that”? At face value, I agree that separation of concerns is needed, and as such I wouldn’t recommend duplicating your OCI config file inside the image layers.

At the same time, I see value in having some internal metadata shipped alongside your container application, which may potentially be useful for things like enriched observability and debugging capabilities (e.g., app log forwarding with container build info (aka contextual logging)).

This is actually not so unusual, as you may have already found images where things like /etc/cloud/build.info are set within, with information about the build.

TLDR:

• Embedding some build metadata inside the image is not heresy — it can enrich observability and debugging.
• Don’t duplicate the full OCI config inside the layers — keep it lightweight and purposeful.
• Patterns like /etc/cloud/build.info already exist in the wild and serve this exact purpose.

---

Closing Note

If you’ve made it this far, congratulations — you’ve survived the sermon. These myths aren’t going away overnight, and honestly, some of them will keep being repeated long after this post. That’s fine. The goal was never to be the definitive authority on container best practices, but to get you to pause and question the defaults you’ve been carrying around.

Challenge your assumptions. Read the docs. Break things in staging. And if you catch me contradicting any of this in a future talk or commit — call me out.

Now go build something great.