Overview
Proof of concept on how to apply a Kubernetes manifest for QA internal tooling at my workplace. At the moment, this is an iteration that’s still ongoing experimentation, but it could go live in production once it’s proven capable. Plain manifests, no Helm/Kustomize, since we’ll only be serving four internal traffic tools that don’t need templating (yet).
Background & Consideration
No Uniformity
These four internal tools apparently didn’t grow up on the same or de-facto infrastructure. At various stages, they were built and developed on Cloudflare Workers, a couple of different VPS instances, and also Vercel. Whichever platform was fastest to deploy to at the time, with no proper decision on where internal QA tooling was supposed to live.
The issue started growing when the deployment and dependency management for all this stuff fell on the QA engineering team (which has become our full-fledged responsibility here). Gradually, we felt it had become a bottleneck as our development speed increased, and it became a growing, compounding cost in ways such as:
- Secret keys are fragmented across three or four different methods. Manual export conventions to each VPS or Vercel’s env variables feel bloated. Not to mention, say, needing to rotate credentials every six months
- Three different deployment mental models. At the initial stage, deploying felt straightforward enough, but after a round of discussion and iteration over the past 2-3 sprints, the mental models gradually became a bottleneck
- The underlying knowledge was scattered. Rather than making documentation for three or four different services, such as how to redeploy the X service, a single and unified piece of documentation would’ve been better for transferring knowledge afterward
honest or perhaps so-called bitter truth: it’s kind of ironic that the QA engineering team is fully responsible for all the ephemeral things like this, which relates to test infrastructure or infrastructure tooling, whereas our hands are already full with the important stuff, like ensuring reliability and quality across systems
Why Not Fully Adopt k8s?
Assuming this isn’t an interview-style answer, the honest truth is: we only have one minimum dedicated VPS instance that’s already been settled on with the higher-ups. It has specific requirements like 2 CPU cores, 4 GB RAM, and 40/50 GB storage. Using a full-fledged, high-availability control plane like etcd, an API server, a controller manager, and a full dashboard with RBAC is definitely over-engineering for low-traffic services like this.
And Why k3s?
A single binary with batteries included that perfectly fits the current problem statement we have right now. Traefik ingress ships by default with k3s, and it also has a low idle footprint, which means it doesn’t consume too many resources. It only runs on tens to a few hundred MB of RAM.
Architecture
At a glance, here’s the current architecture we’d deliver for this proof-of-concept, aiming for a “production-ready” environment for the QA workspace:
What’s the difference between a docker-compose setup and using this?
We dropped each app’s own nginx reverse proxy container, since Traefik ingress handles that job at the cluster level instead of per individual app container. One of the services, maritest, needs a PVC mounted into the deployment to persist run log history across pod restarts.