At the end of the Grype post I added a section to harden an Alpine image
further by removing the package manager and deleting the /bin directory so
there is no shell. This helps reduce the busybox CVEs reported as busybox
cease to exist.
Around the same time as doing up the post, I was working on hardening the
UQCS Discord Bot production image. I first changed the debian image to
alpine then stripped Poetry and the build tooling (like pip and other setup
tools), removed the package manager, and forced the runtime to run as an
unprivileged user. This was all good, and the next step was to just delete the
busybox binaries and simlinks in the /bin directory, which I did, not really
thinking about anything but the attack surface of the production image. I
didn’t really think about how the image would run and what the possible
consequences about this.
In a work or product image this isn’t (shouldn’t) be a problem, and people
should probably push to have the shell removed from production images. An image
should be immutable and readonly, any write operations should be done to
attached volumes. But this discord bot was for an university club, and the
people who run the infra are very part time volunteers. The contirbutors are
typically new devs who haven’t done alot of software, and the infra team don’t
have any time to manage everything. So if something goes wrong and a container
restart doesn’t fix it, they just want to be able to docker exec into the
container and use a shell to quickly figure out what is going on and call it a
day.
This issue was outlined by James (one of the poor infra people) upon reviewing my PR (#236):
I’m not convinced it’s a good idea to yeet the entire
/binfolder? It leads to any possible debugging needs on prod to be a lot harder.
If this was a response for something similar at work, I would’ve pushed hard
on this. For the reason of already having centralised logging and existing
build tooling which doesnt need a shell. So I removed the /bin stripping from
the PR and it got merged in after another couple of people reviewing.
And then the universe decided to make James’ point for him. During the deploy we had a heart attack as github didn’t send the Azure App Service the webhook for over 15 minutes which was very stressful (typically it takes less than a minute). The Azure App Service dashboard is horid and provides no useful information of what is going on, and shelling into it was how we figured out what was going on. After 25 minutes we had the bot restarted and running the new image and confirmed.
Which is the joke, really. I’d spent the week arguing you don’t need a shell in
prod, and the first time something went sideways, the only thing that told us
what was happening was shelling in. But I don’t think that means the hardened
image is wrong. I think it means I want the shell to be something I bring to a
problem, not something that lives in the image waiting to become one. Hardened
by default, shell on demand. So the thought that wouldn’t leave me alone was:
“Is there a way to sideload a shell and debugging environment onto a running
Docker container?” Because once we remove the shell we can’t just do
docker exec -it uqcsbot sh without it throughing a fit.
For reference, I added the /bin removal back into the uqcs bot locally and
started playing around with testing.
Idea #1: Volume Mount a Shell In
My first idea was to bolt a directory that contains a busybox shell and then
mount a tmpfs onto a running container and exec via that directory.
For some reason I thought this would work. But turns out you cannot add a mount to an already-running container. These volume mounts are fixed at container creation time, so they are apart of the config the runtime sets up before the process starts.
Idea #2: Smuggle a Shell from a Donor
If I can’t volume mount one in, I know I can copy one in. The docker cp
command works on running containers, which means I can create a donor alpine
container and copy the /bin directory out of that and then copy that into
the hardened image:
# 1. Get busybox out of a donor to the host
docker create --name donor alpine:latest
docker cp donor:/bin/busybox ./busybox
docker rm donor
# 2. Copy busybox to the running shell-less container
docker cp ./busybox uqcsbot:/bin/busybox
# 3. Use the shell
docker exec -it --user root uqcsbot /bin/busybox sh
This worked, though I only copied the busybox binary as all the unix tooling
are apart of it (typically simlinked). There is something worth noting about
this method though. You must clean up the ./busybox binary once you are done.
This is important because docker cp writes to the container’s writable
layer, so it survives a docker restart and just sits there. It won’t survive
the container being recreated. A fresh image deploy throws the writable layer
away but on a long-lived prod container that you aren’t redeploying, it’ll
linger indefinitely and quietly undo the hardening you just did. They call this
a footgun.
Idea #3: nsenter and a tmpfs
The docker cp approach persists because it writes to the writable layer. What
I actually wanted was a more ephemeral version. Drop a shell into the runnig
container on a tmpfs so that it lives in RAM, dies when the container dies,
and never touches the core image. Docker won’t let me add a volume mount to a
running contianer, but docker is just a frontend tool. The mount namespace is
sitting right there, and nsenter is a linux tool that can be used to just run
the mount syscall myself.
docker inspect -f '{{.State.Pid}}' uqcsbot
nsenter -t <pid> -m -p -- mount -t tmpfs tmpfs /tmp/debug
The main pain about this is that I am running on a Mac, not Linux. So I don’t
have nsenter, and Docker Desktop seems to run the containers inside some form
of Linux VM, so even if I could run the nsenter from my mac it will need to
get into the VM first.
In hindsight this was a stupid idea, but I kept pushing through. What if I
just ran a privileged linux container that has nsenter, so I could run
the nsenter from inside the VM’s environment:
# 1. Get the process id of the uqcsbot running container
docker inspect -f '{{.State.Pid}}' uqcsbot
# 2. Launch an interactive terminal
docker run --rm -it --privileged --pid=host justincormack/nsenter1
# 2.1 Mount the tmpfs volume in the interactive terminal
nsenter -t <pid> -m -p -- mount -t tmpfs tmpfs /tmp/debug
This is alot of messing around, and it barely worked. But somewhere in the
middle of it I kept typing --pid=host, and that flag is the thing that
finally cracked it open. --pid=host shares a PID namespace. I’d just been
pointing it at the VM the entire time. It doesn’t have to be the host. Go
back to the docs and --pid will happily take a container: target instead.
Which was the whole problem. I’d been trying to smuggle tooling into the bot this entire time, when I could have just stood another container next to it and shared the namespace directly. That’s Idea #4, and it’s the one that actually works.
Idea #4: Namespaces, But the Right Way
The mistake I kept making, apart from reading the docker docs wrong, was that
I kept trying to put tooling into the image. But the --pid flag (and a few
others) take a container: form that points them at any container I want.
Which means I can stand a second container next to the bot, carrying all the
tooling, and have it share the bot’s namespaces.
docker run --rm -it \
--name sidecar \
--pid=container:uqcsbot \
--network=container:uqcsbot \
--cap-add=SYS_PTRACE \
alpine:latest sh
This spins up a throwawy sidecar Alpine container that shares two of the bot’s namespaces:
--pid=container:uqcsbot: Puts the sidecar in the bot’s PID namespace so runningps -afrom inside the sidecar lists the bot’s processes.--network=container:uqcsbot: Shares the network namespace, solocalhostin the sidecar is the bot’slocalhost. A simplecurl localhost:8080hits the bot’s listener as if I were inside it.--cap-add=SYS_PTRACE: Lets the sidecar inspect the bot’s processes. Sharing the PID namespace lets me see them, but reading/proc/<pid>/root/needs ptrace access, and the bot runs as the unprivileged user65532. Without this the kernel denies the peek and/proc/1/root/comes back permission denied.
The sidecar has it’s own shell, package manager and owns everything. It’s a
full Alpine image. The bot is untouched and stays hardened, and because we are
running with the --rm flag the sidecar cleans up the moment you leave the
interactive shell. No binary left behind in a writable layer, no footgun to
forget about, when you leave, it’s gone.
The only other thing you’d want is to see the bot’s filesystem. Notice that I
shared the PID and network namespaces but not the mount namespace, so the
sidecar keeps its own filesystem instead of the bot’s. That’s actually what you
want, the sidecar’s tooling stays separate, and the bot’s files are still
reachable. Every process exposes its own root at /proc/<pid>/root/, resolved
through that process’s mount namespace, so the bot’s entire filesystem is
sitting at /proc/1/root/:
/ # ps aux
PID USER COMMAND
1 65532 python -m uqcsbot
9 root sh
16 root ps -a
/ # ls /proc/1/root/app
.venv poetry.lock pyproject.toml uqcsbot
There’s the bot as PID 1, running as 65532, the unprivileged user I forced
back in the intro showing up right where it should, and the sidecar’s own root
shell as PID 9. And /proc/1/root/app is the bot’s app directory, read
straight out of a container that has no idea it’s being watched. (If you
actually wanted the bot’s filesystem as your own you could share the mount
namespace too, but then you’d lose the sidecar’s separate tooling, and
/proc/1/root gets you there without giving anything up.)
That’s it, a container with no shell and I am able to boot an ephermal sidecar and read its files.
I got this all working and then showed it to James. It was pretty cool. Then he explained to me that the problem was more Azure Application Service which just runs things weirdly. The other was that the members of the uqcs might want to add some features which need to call shell commands from python, we don’t know what these are but I shouldn’t deny young devs from fun.