It's a `->Technical<-` Blog

Intro

This is where I do little write-ups about technical things I encounter and find interesting. I'll try to make them more generally accessible than the notes I write to my future selves.

Article(s)

Dev Containers

So I like to use dev containers, as part of my developer workflow. Specifically I like to keep whatever funniness happens in a coding project (setup, experiments, etc.) contained to that project and not let it spill out into the rest of my system. As I don't do kernel-level things much Docker provides all the isolation I need.

As a side benefit it also keeps me honest in terms of being able to set up (and therefore also nuke) a dev environment quickly.

Mostly that boils down to two things:

Keeping my dotfiles well organized. Meaning they can configure almost-certainly-already-present base utilities found on almost every Linux machine (vi, sh, grep, less, etc.), but they can also pull down "nice-to-have" dev environment dependencies (vim, zsh, bat, other new-age, Rust CLI utitility replacements I have become so fond of) as needed. AKA I've gotten soft and if I'm going to be working in an environment for a while I want to be more comfortable than having a set of tools that will work, I want the tools that I like - with pretty colors and everything. So sue me.
Knowing how to connect two environments together securely & comfortably. Accessing a shell in one environment from my laptop, getting a GUI to display from one environment in another, being able to access a 'locally' hosted web service in one environment from another, sharing files - all of these would fall under this banner.

Lately, I've realized that while I've gotten pretty comfortable with Docker in terms of a number of use cases I do regularly, I didn't really understand its internals or how it did what it did. I was limited in my understanding because I didn't know what lower-level things it was built on and what sort of inherent limitations they would introduce to its functions. One concrete thing that brought this to light was needing to connect an X application in a dev container to the X server on the host (aka my laptop) which controlled its display/screen/etc.

This brought up a number of questions in my mind.

Should this be a port forwarding thing? Maybe SSH?
A couple of the dev-container utilities I had used (VSCode extension & Devpods) seemed to use those as their go-to solutions for connecting host to container.
What about mounts?
I mean the folder mount of a project's code to the "workspace" in a dev container is the fundamental connection between my host and dev container. Outside of that, named UNIX domain sockets were how SSH agent forwarding worked as far as I could tell (so I didn't have to go in and out of my container to do authenticated things like connect to AWS or sign my commits.)
Does this have something to do with cgroups?
I keep seeing that term crop up, but I have no idea what it is? How does Docker even work? What are its component systems and how do I trade-off between them and/or use them "correctly" and not just abuse them in the depths of my ignorance?

History & Context

Well that last question exposed the underlying problem of me not understanding how Docker or containers worked or what they were built on. As you can see that kicked off a whole stream of questions all having to do with How does any of this work?, so I decided to embark on a learning excursion.

Turns out, as far as I could tell, Docker is kinda a nice convenience layer on top of a bunch on different Linux kernel functionalities that ties them all together to give you a mostly isolated environment. In this respect, the most useful learning resource for me wasn't Docker (or container) specific, but was just a book on how Linux works.

I mean I've used Linux for nearly two decades now, but looking back on it everything I knew was hobbled together from disparate man pages and noticing things changing as I updated my distro. Oh look, /etc config files now have *.d/ folders that the main file tells me to use. Guess I'll do that.

I dual-booted Linux & Windows on my first laptop: a Dell with an Nvidia GPU. Everytime I would update the kernel I would lose access to my GUI and GNOME and only have a lonely TTY. My Wi-Fi was gone too as wpa-supplicant seemed to be tied to the GNOME display manager (as an applet? at least I remember it didn't start with its normal configuration outside of gdm .)

No Wi-Fi meant no searching the forums for how to fix things and even if I did regain Wi-Fi I couldn't start a GUI browser. This is when I learned how to write configuration for wpa-supplicant by hand (after using a university computer to search the forums) - which was no fun - and also how to navigate the TUI, ASCII-art-based, web browser Lynx - which was a lot of fun.

I eventually wrote a script for the unfun part. (As well as messing with X and kernel modules that normally re-enabled the GUI.) But still got to dive into Lynx a good bit as my Nvidia graphics fixes needed tweaking fairly frequently as the drivers were apparently undergoing big changes.

I knew specific parts of Linux, and more often than not, how to do specific things with those specific parts, but I never really got a this is how it is all connected design overview or the rational behind how things progressed and changed. This style of learning also meant that there just were parts of Linux that I straight-up didn't encounter and knew next to nothing about.

The Good Book

What solved this for me was the excellent book "How Linux Works, 3rd Edition" by Brian Ward and, hey, it finally answered what cgroups were.

Answers

Now that I knew that Docker is a convenience layer that bundles together a number of built-in Linux kernel features to make a cohesive API/environment that I can think of as a "container", I was able to start to answer some of the questions from above.

Should this be a port forwarding or SSH thing?
It could be. If I was going to possibly want dev containers on other machines connected through a network than this would make that easier. For SSH tunnelling I need an SSH server in my container which is a bit annoying to have to add to every dev container I use, so I would prefer it if I didn't need one.
Since X tends to use UNIX domain sockets locally and I was planning to do my development on my local machine, SSH seemed like too much and I might as well use the existing UNIX domain socket and not worry about port forwarding. Also opening up an X network port seemed iffy since it has all the naïve trusting tendencies of software developed in the 1980's. I mean that's why everybody always tunnels it with SSH (well that and because SSH has built-in, specialized, packet-sniffing support for X protocol tunnelling, sheesh - no wonder that's what we always use.)
Mounts?
Yeah, this is what I was looking for. X uses named UNIX domain sockets that are accessible via a filesystem path, I'm using this for local development, ... its all coming together.
As it turns out X hardcodes where it puts its UNIX domain sockets so I don't really need to do much fancy footwork (enforce a configuration or read an environment variable or anything except know that this path is the right and true directory for X server UNIX domain sockets.)
This does make mounting it in Docker a little annoying as other things like to mount a RAM tmpfs filesystem to /tmp. In particular the Docker in Docker feature for dev containers likes to do so in its container startup script . If I mount a subfolder (or file) of /tmp then it'll get kludged by this script. As a workaround I mount /tmp as a RAM filesystem ahead of time because the Docker in Docker container script is kind enough to check that /tmp itself isn't already a mount that it shouldn't kludge.
Control Groups?
As noted above, no this has nothing to do with what I was attempting. It is just the kernel feature that allows Docker to do host resource limiting.

Onwards

Well I found my solution and I got to learn more about how Docker & containers in general work so that's cool.

Now what to learn about next? Maybe one of the kernel features that are bundled up by Docker to make containers:

Capabilites
Mounts
Control Groups
Namespaces
Networking

I think I want to dig a little more into the networking bit next. I'd like to understand iptables (or its successor if that has hit prime time and taken over the spot light now?), maybe some traffic control? Basically I want to understand what all I can control using kernel networking utilities as they seem pretty powerful and I want that tool in my toolbelt.

Well that's that for now. TTYL.