Lachlan Cox

Wasting Time with Docker Save

Mid-way through the last post, somewhere between cosign and the local registry I swore I’d never run, I was sifting the Docker CLI Reference Docs and got reminded that docker save and docker load exist. Commands I have never needed once in my life. Commands that let you cram a Docker image (or multiple) into a tar file and yeet it at another machine. No registry. No auth. No infra. Just a file.

You have to understand the effect this has on me. The last post involved me standing up a local registry and generating cosign keys just to attach one JSON file to one image, and I felt like I was the problem. So a pair of commands whose entire value proposition is “what if not registry?” was always going to end with me neck-deep in a tarball at 1am recomputing sha256 digests by hand. This posts starts with the legitimately useful bit, and then documents the part where I started the second bottle of Nikka Whisky.

The Actually Useful Part

docker save flattens an image (or several) into a single tar file, and docker load turns that tar back into an image on whatever machine it lands on. That’s it. No registry in the middle, which is the entire appeal to me. Where this is genuinely useful:

For the deployment cases, let’s call this image we are having conniptions with rosie:latest. On our dev machine that builds the image we can run docker save rosie:latest > rosie.tar. Then you just need to copy that tar to the server and then docker load -i rosie.tar and just run it. You can even be a sicko and do:

# 1) Save and Load it onto the server
docker save rosie:latest | ssh deploy-server 'docker load'

# 2) Deploy the image
ssh deploy-server 'docker run -d --name rosie rosie:latest'

For the archiving case, assume you have a slow brain bleed and are relying on PHP 5.x for your legacy application. You might want to have an archive of php:5.4.39 before Docker Hub decides to delete it from their registry due to the security problems it poses; you can just do a docker save php:5.4.39 > php-archive.tar so you can load it at build time and use it in your Dockerfile.

Down the Rabbit Hole

So rosie is a small Golang program I wrote which just prints “Hello, World!” and exits. This gets statically compiled in a multi-stage Dockerfile and is deployable as a scratch container:

# Stage 1: Building the static binary
FROM golang:alpine AS builder
WORKDIR /app

COPY . .
RUN CGO_ENABLED=0 go build -ldflags '-s -w' -o rosie .

# Stage 2: Deployed Scratch Image
FROM scratch
WORKDIR /

COPY --from=builder /app/rosie /rosie
ENTRYPOINT ["/rosie"]

I built the image with two tags, rosie:latest and rosie:test (this will matter later, but also docker save letting you put multiple images in one tar is part of what I wanted to poke at), did a save of both into a single rosie.tar and untar’d it into a ./docker/ directory and these are the files I got:

docker/
├─ blobs/
│  └─ sha256
│     ├─ 725B 746aebd80568e25b97e2fd3d8cbb8098e5f3f057873ca5977af4a17a3c812069
│     ├─ 1.5M ca69737668d0e1d006dc44b945cfc68263e96bdf848c17ee37a212cac4adeb78
│     ├─ 748B e64b690412ecdb9bb2b1b60ab53489857252ec4d80a1391488b1d32691249e10
│     └─ 400B f802ec1ce6d815e5566884c20c956a2b42b617039f2778dfd674dfd3237bd698
├─ 361B index.json
├─ 456B manifest.json
├─  31B oci-layout
└─  88B repositories

3 directories, 8 files

As I don’t know what I’m doing, and because I have dabbled in web dev once or twice, I opened the index.json as it seemed like what would be a good starting point.

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:f802ec1ce6d815e5566884c20c956a2b42b617039f2778dfd674dfd3237bd698",
      "size": 400,
      "annotations": {
        "io.containerd.image.name": "docker.io/library/rosie:latest",
        "org.opencontainers.image.ref.name": "latest"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:f802ec1ce6d815e5566884c20c956a2b42b617039f2778dfd674dfd3237bd698",
      "size": 400,
      "annotations": {
        "io.containerd.image.name": "docker.io/library/rosie:test",
        "org.opencontainers.image.ref.name": "test"
      }
    }
  ]
}

Already something neat: both tags are in the one index, and because they are literally the same image, they point at the same manifest digest. No duplicate manifests, and as you’ll see in a second, no duplicate layers either. The tar dedupes by content. (I also tested this with a slightly modified image as a third entry, and it got its own manifest but still shared whatever layers were identical, which is very cool.)

NOTE: From here on I’m referring to blobs by the first 8 characters of their hash, e.g. f802ec1c, because life is short and these hashes aren’t.

So I guess the f802ec1c blob file is JSON and some form of manifest…

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:746aebd80568e25b97e2fd3d8cbb8098e5f3f057873ca5977af4a17a3c812069",
    "size": 725
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar",
      "digest": "sha256:ca69737668d0e1d006dc44b945cfc68263e96bdf848c17ee37a212cac4adeb78",
      "size": 1574912
    }
  ]
}

Ok, well that was JSON and I guess the 746aebd8 blob file is also JSON and some form of config:

{
  "architecture": "arm64",
  "config": {
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ],
    "Entrypoint": [
      "/rosie"
    ],
    "WorkingDir": "/"
  },
  "created": "2026-06-06T02:47:35.734847256Z",
  "history": [
    {
      "created": "2026-06-06T02:47:35.734847256Z",
      "created_by": "WORKDIR /",
      "comment": "buildkit.dockerfile.v0",
      "empty_layer": true
    },
    {
      "created": "2026-06-06T02:47:35.734847256Z",
      "created_by": "COPY /app/rosie /rosie # buildkit",
      "comment": "buildkit.dockerfile.v0"
    },
    {
      "created": "2026-06-06T02:47:35.734847256Z",
      "created_by": "ENTRYPOINT [\"/rosie\"]",
      "comment": "buildkit.dockerfile.v0",
      "empty_layer": true
    }
  ],
  "os": "linux",
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:ca69737668d0e1d006dc44b945cfc68263e96bdf848c17ee37a212cac4adeb78"
    ]
  }
}

At this point I pulled up the OCI Image Format Spec to see what I’m actually looking at. Apparently, the image.config is suppose to also have a user set, but I guess because the final image is a scratch container there is technically no users, so it is left blank. This file can get quite large with annotation labels, volumes, and a whole bunch of dead links in the docs.

The ca697376 is some form of layer and/or virtual file system. Doing a file ca697376... shows that the file is actually another tar file (POSIX tar archive). Which means I can do a tar with the -t flag to check the contents:

~/dev/rosie/docker
❯ tar -tvf blobs/sha256/ca697376...
-rwxr-xr-x  0 0      0     1573026  6 Jun 12:47 rosie

So the blob is quite literally just a tar of the file system of the final scratch. Rosie is the binary and it is executable. So this is all interesting. The main thing to note is that everything in blobs/sha256/ is named by the sha256 of its own contents. This gives a free dedupe side affect for being content addressable. This is also something that will bite me as I didn’t read the docs correctly.

Compression?

Reading the OCI spec showed that the image.layer tar files support a +gzip and +zstd compression which means that I could technically compress these tar files and then update the sha256 digests in the other files and have a compressed version of the archive. So let’s do that because why not, I like saving space.

~/dev/rosie/docker
❯ cd blobs/sha256

# 1) Compress the layer blob using zstd
❯ zstd -c ca697376... > new_layer
# 1.5M > 641K

# 2) Compute new Digest
❯ sha256sum new_layer
# digest = e61d959f2853a738064a3aaa818b52139ffe6d9fb7790b623aba0eb31fc7b3b9

# 3) Get the size in bytes
❯ stat new_layer
# size = 656127

# 4) Rename Layer to its digest and Remove Old Layer
❯ mv new_layer e61d959f2...
❯ rm ca697376...

This is when things get a bit more complicated. We need to update the image.manifest (f802ec1c) with the new layer:

{
  "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
  "digest": "sha256:e61d959f2853a738064a3aaa818b52139ffe6d9fb7790b623aba0eb31fc7b3b9",
  "size": 656127
}

We must NOT update the image.config rootfs diff_ids though. Those are defined as the digests of the uncompressed layer tars, on purpose, so the engine can verify it got the same content no matter what compression was used in transit. Which, when you think about it, is exactly the property that adds some form of validation to the system to make sure that you are running the right thing. I didn’t read the warning NOTE on the docs when doing this and had to back track.

I’ve edited the bytes of the f802ec1c manifest, which means its sha256 is no longer f802ec1c (the file is now lying about its own name). So just like the layer: recompute the digest, and rename the blob to it. For me that came out as e9f4f5d7de52c7045725d0901ed64c0410b7473ea79ff78b017e3fdafd39d91a, so f802ec1ce6d8 is gone and e9f4f5d7de52 now exists. Then update both manifest entries in index.json (one per tag, remember) with the new digest and size. Turtles, all the way up.

Other things I did was delete e64b6904, manifest.json, and repositories as they are the legacy format from before OCI. Assumptions and poorly worded documentation are great.

~/dev/rosie/docker
❯ tar -cf ../new-rosie.tar oci-layout index.json blobs

~/dev/rosie/docker
❯ tar -tf ../new-rosie.tar
oci-layout
index.json
blobs/
blobs/sha256/
blobs/sha256/746aebd80568e25b97e2fd3d8cbb8098e5f3f057873ca5977af4a17a3c812069
blobs/sha256/e61d959f2853a738064a3aaa818b52139ffe6d9fb7790b623aba0eb31fc7b3b9
blobs/sha256/e9f4f5d7de52c7045725d0901ed64c0410b7473ea79ff78b017e3fdafd39d91a

~/dev/rosie/docker
❯ docker load -i ../new-rosie.tar
lsetxattr /oci-layout: xattr "com.apple.provenance": operation not supported

~/dev/rosie/docker
❯ ffs apple ^C

~/dev/rosie/docker
❯ COPYFILE_DISABLE=1 tar --no-mac-metadata --no-xattrs -cf ../new-rosie.tar oci-layout index.json blobs

~/dev/rosie/docker
❯ docker load -i ../new-rosie.tar
invalid archive: does not contain a manifest.json

Well that’s annoying, it seems the version of Docker Engine I am running doesn’t have the containerd image store enabled. Apparently, it is supposed to be on by default since Docker Engine 29.0 but only for fresh installs. Without it, docker load ignores all my lovely OCI structure and wants the legacy manifest.json I just just deleted. So for now I just need to create the manifest.json from scratch:

[
  {
    "Config": "blobs/sha256/746aebd80568e25b97e2fd3d8cbb8098e5f3f057873ca5977af4a17a3c812069",
    "RepoTags": [
      "rosie:latest",
      "rosie:test"
    ],
    "Layers": [
      "blobs/sha256/e61d959f2853a738064a3aaa818b52139ffe6d9fb7790b623aba0eb31fc7b3b9"
    ],
    "LayerSources": {
      "sha256:e61d959f2853a738064a3aaa818b52139ffe6d9fb7790b623aba0eb31fc7b3b9": {
        "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
        "size": 656127,
        "digest": "sha256:e61d959f2853a738064a3aaa818b52139ffe6d9fb7790b623aba0eb31fc7b3b9"
      }
    }
  }
]

Now I can re-tar and load it:

~/dev/rosie/docker
❯ COPYFILE_DISABLE=1 tar --no-mac-metadata --no-xattrs -cf ../new-rosie.tar oci-layout index.json manifest.json blobs

~/dev/rosie/docker
❯ tar -tf ../new-rosie.tar
oci-layout
index.json
manifest.json
blobs/
blobs/sha256/
blobs/sha256/e9f4f5d7de52c7045725d0901ed64c0410b7473ea79ff78b017e3fdafd39d91a
blobs/sha256/e61d959f2853a738064a3aaa818b52139ffe6d9fb7790b623aba0eb31fc7b3b9
blobs/sha256/746aebd80568e25b97e2fd3d8cbb8098e5f3f057873ca5977af4a17a3c812069

~/dev/rosie/docker
❯ docker load -i ../new-rosie.tar
Loaded image: rosie:latest
Loaded image: rosie:test

~/dev/rosie/docker
❯ docker run --rm rosie:latest
Hello, World!

Both tags, one tar, hand-mangled manifest, still says hello. The new tar comes out at ~650KB instead of the original 1.5MB (around 40% of the size), for the price of one compressed layer, two recomputed digests, an index update, a resurrected legacy manifest, and a fight with macOS xattrs. Genuinely pretty chuffed with this. On a side note, I have no idea why it didnt care about the registry and other legacy layer I deleted but it works and I have given up caring.

Then, purely for completeness, I ran a plain gzip over the original, untouched rosie.tar:

~/dev/rosie
❯ gzip rosie.tar

~/dev/rosie
❯ ls -lh new-rosie.tar rosie.tar.gz
-rw-r--r--  1 lachlan  staff   650K  6 Jun 14:02 new-rosie.tar
-rw-r--r--  1 lachlan  staff   633K  6 Jun 14:03 rosie.tar.gz

633KB.

Smaller than my version. One command. No digest surgery, no manifest archaeology, no COPYFILE_DISABLE. So I completely wasted my time and yours.

In fairness to me, the in-tar compression has one real advantage: docker load eats it directly, whereas the outer gzip needs decompressing first. That advantage is worth approximately one shell pipe and dependencies.

I then looked into docker push and whether registries care about any of this. Turns out docker automatically gzips layers by default when pushing to a registry, so the transport is compressed regardless of what I do to my tars. I did notice the Docker Hardened Images from the last post ship their layers zstd-compressed though, and getting your pushes to do that involves spinning up a custom buildx builder with its own config, which is a mild pain. I did it. I didn’t want to.

Then I ran another local registry (count: two more than I ever wanted) to test pushing my hand-edited image. The regular rosie:latest got its layers gzip’d when stored, as expected. My manually edited one got completely butchered. The push recomputed the zstd layer back to gzip, like my surgery never happened.

Digging into why, it’s the same issue from earlier. The classic image store, the one that demanded the legacy manifest.json, doesn’t keep your original compressed blobs around. When you docker load, it unpacks the layers into its own internal format and quietly bins the blob I so carefully compressed; the zstd layer was dead the moment the load succeeded, and the push just re-compressed from the internal copy using its default, gzip. Apparently enabling the containerd image store fixes this too, it keeps the original blobs, so loads preserve them and pushes ship what you actually made. Same fix as the manifest.json problem. The classic store got me twice in one post and I only noticed the second time because I went looking.

So What was the Point?

I’m not sure. I would be more surprised if you made it this far. If anything it’s cool to know these random tech things, every blob is named by its own hash, edit anything and you have to re-hash your way up the tree, your docker engine might still demand a legacy JSON file out of spite, and that I would do absolutely anything to try and avoid using a registry.

<< Previous Post

|

Next Post >>

#Docker #Containers