Our organising team set up accounts in STACKIT and set up a container registry via the STACKIT web portal. Anything any team would construct, they would have to put into a Docker image and upload it there.

We then set up a project fullstackathon and in it, under Runtime → Kubernetes Engine, we created a cluster. Under IAM, we created a custom role for our fullstackathon participants, allowing access to the Container Registry, adding a pod to a namespace and starting it. We weren’t very familiar with the particular access structure so we chose to be very restrictive and add permissions later if needed.

We used infrastructure as code to set up http-routes (using a Gateway API), limit-ranges, resource-quotas, and a service-account. This was all tied to together with Kustomize and its built-in secretGenerator in combination with STACKIT’s Robot Accounts to allow every team’s pod to use the image-puller Service Account to pull the uploaded Docker Image from the Container Registry and effectively deploy and run their application in Ludwighafen.

Ludwighafen was set up to be a website reachable via (the now defunct) ludwighafen.eu and any city service a team would build would have its own namespace and a corresponding subdomain. To keep a bit of control, we’d create the namespaces when a team indicated what service they were going to build and give them an upload key for it. We built a form for requesting it, where they could upload their visuals too. To ensure the image would fit the plot, we supplied an example AI-prompt for generating images in the right style and size. We added a shell-script for generating namespaces so that we’d be able to get teams going quickly during the fullstackathon.

All any team would have to do besides dockerizing their application and pushing it to the Ludwighafen (STACKIT) Container Registry was to create a deployment.yaml mentioning the namespace, selector, metadata and container image, and call kubectl apply -f deployment.yaml to deploy and activate their application in Ludwighafen.

We then decided to build the central application, that would visually be the city as well the castle at its center, and technically be charged with checking the availability of the other services in order to show the right image for it in the city map. Once a team requested to build a service and supplied us with the name (for the namespace and subdomain) and the visuals for it, the castle application would show it as under construction. Once the application gave an HTTP 200 OK on an HTTP request to its subdomain, the castle would consider it finished and show the appropriate visual. If after that the service ceased to be 200 OK, it would be rendered on fire and the king would make his concerns known.

The castle application was a Python application with a simple HTML/CSS/JavaScript frontend. And because the proof is in the pudding, we used the mechanisme we were going to unleash on our colleagues as the means to deploy it and make it available.

We tested one demo service: it showed up in the alloted plot in the browser, and clicking it opened the frontend at the intended subdomain.

We did run into some snags which we documented and that ended up becoming our tutorial. In the final minutes before the hackathon we decided to deploy that tutorial itself as an HTML page served via an Nginx application that ran at scroll.ludwighafen.eu, styled appropriately in medieval fashion.

We were ready for the event.

During the hackathon, people were surprisingly creative and productive off of a minimal set of requirements. However, not everything went smooth:

Especially the constrained resources were a miscalculation. People consequently resorted to more local development first, and as a result one or two teams failed to deploy their latest working application before the end, demonstrating it from their local machine instead. A reminder that proper resource allocation for delivery pipelines and test environments is crucial for timely feedback and the ability to deliver solutions.

The municipal records were a service we had indicated would be crucial. The team that picked this up had dubbed it the Blue Kremlin and went all-in on Big Brother features in a 90s style.

It featured registration of civilizans with name, age and the role & permissions they had. It offered an interface for other services to determine permissions (and no other details) for certain features they offered. The Blue Kremlin showed ads in true 90s fashion, and these actually led to other services in the city.

In the community hall you could log in as a civilian and create an avatar.

It featured multiple rooms to explore, like a neighbourhood cafe that served fresh beers from the brewery, a chatroom where you could request songs for the radiostation to play and a dark room which required a special permission, requested from the Blue Kremlin.

The bank allowed the creation of accounts for every registered civilian and also interfaced with the Blue Kremlin for checking that that civilian was indeed permitted to do so. Civilizans could make payments and transfer funds to each other and get an overview of their transactions.

Visually, the bank website had every feature frontend developers have nightmares of, like a bouncing logo with a lensflare, animations and misaligned boxes.

The AirDnD service did not offer a way to rent rooms for guests of Ludwighafen. Instead, it offered cheap uncomfortable dungeons to sleep in, some even featuring a dragon.

There was also the possibility for citizens to put up and rent out their own dungeons to visitors of the city.

The brewery was a neat gamified service for brewing bear. You could pick ingredients, set the temperature and choose a fermentation method to end up with unique beers.

These beers then went up for sale and became available in the Ludwighafen community hall.

Unfortunately, the radiostation service frontend was not caught on camera, as this was one team that had decided to focus entirely on local development first and ended up not being able to deploy it in time.

The radiostation could play songs from Spotify, as requested in the radiostation chat, or via the community hall. Between songs, AI-rendered commercial audiotracks spoke highly of Ludwighafen, its king and the available services.

The PortHub was the harbour and shipyard of Ludwighafen. Nothing but a means of uploading videos sharing the wholesome appreciation for hard labour and beautiful ships.

The idea behind the powerplant was that it kept an eye on the amount of services in the Ludwighafen cluster and calculated an amount of power needed to run those. If the power demand was too high, it would raise a loud alarm and eventually kill off a random pod. To increase the power, a small game was introduced reminiscent of the dinosaur jumping over cactuses in FireFox, but it instead had a knight jumping over beer barrels, each jump counting towards added power to run the city.

Actually killing off pods is something we implored the team to not implement, for reasons mentioned above.

As the hackathon participants hacked, they became thirsty. Every time they ordered drinks, they’d tell the bartender at the hostel a little more about their ideas and efforts. Eventually the bartender, a proud inhabitant of Soest, requested Ludwighafen to feature the Palace of Soestdijk. We were happy to oblige.

At the end of the presentations of the participants of the fullstackathon, the royal council showed that it had also secretly built an arcadehall with a powerful feature. Now that the work was complete and people were about to head home, we felt it was the right time to show we had planned some chaos engineering.

The arcade hall allowed you to play Doom. But it was a special version in which level 1 spawned demons that represented all the pods running in the Ludwighafen kubernetes cluster. Any demon shot meant a pod killed. As we demonstrated it and the buildings in the Ludwighafen city map caught fire, it was a testament to the fact that autohealing indeed did work and buildings would go back to being 200 OK.

That is, until we shot a demon that was some crucial pod of the kubernetes management plane itself, and everything grinded to a halt.

A fitting end of a successful fullstackathon.