The architecture of this humble abode

The page you are reading is served by one of three programs sitting behind a single nginx, with no orchestrator, no service mesh and no cluster anywhere in the arrangement: a Go binary, a Rust binary, a directory of static files, and a quantity of opinions about URLs that is entirely out of proportion to any of it.

Application knowledge about prefixes

The blog lives at /blog on the same domain as the static site, and while a reverse proxy will cheerfully move requests wherever you please, what it will not do is correct what the application says about itself on the way back out.

The tempting remedy is to rewrite the response, sub_filter across the HTML and be done with it, and it does not hold up for long, because it reaches HTML and nothing else. It does not reach the Atom and JSON feeds, nor the sitemap, nor the Location headers on redirects, nor the JSON-LD an ActivityPub actor serves, nor the short links the application hands to other software; each of those would want a rule of its own, every such rule is a substring match on a document whose structure it does not understand, and the failure mode throughout is silence, since a wrong URL renders perfectly well and simply points at something that was never served. That is a poor bargain for a prefix which could just as easily be a property of the program.

So the prefix became one. Configuration folds it into every blog path once at start-up, and a small helper prefixes whatever gets constructed later; the interesting part was not the change but the audit that followed it, because some sixty-odd call sites build an absolute address, and reading them one by one turned out to be a poor way of finding the four that were wrong.

What found them was a test that renders pages, collects every URL they contain, and asserts that none of them escapes the mount — a test which, before it worked, passed twice over while proving nothing at all. The first version matched attributes with a regular expression against markup the minifier had already stripped of its quotation marks; the second parsed the HTML properly but gathered only root-relative links, and the short link, as it happens, is emitted absolute. A test that cannot fail is worse than no test, not least because its green light discourages you from writing the one that would have caught something.

Once it could fail, it found five leaks, one of them in the health check, where the service had been up all along and the thing charged with asking whether it was up had been requesting a path that had not existed for a week.

WebFinger is the sole exception, and not one that is open to negotiation, since RFC 7033 places the resource at /.well-known/webfinger on the root of the host. That route is proxied separately, and the application is told as much.

Tail-sampled telemetry

The blog is instrumented with OpenTelemetry, and the first sampling policy was head-based, with the decision taken the moment the root span opens: anonymous page views yielded nothing, federation traffic was sampled at a low rate, and anything I did myself was always kept.

That policy carried a requirement it had no way of honouring, namely that every failure ought to be traceable, an anonymous read included, and when the root span opens the failure has not yet happened.

So the decision moved to the end of the request. Spans are now created unconditionally, a processor rules at export time, per trace, whether they ship at all, and an error anywhere in the request overturns whatever the earlier verdict would have been. The kept share of federation traffic is derived from the trace ID rather than drawn per span, so that every span of a trace meets the same fate and no child is orphaned by the dropping of its parent.

A related discipline governs the metrics. Labels are an enumeration declared in the code and never inferred from the router, which registers one route per hashed asset filename and would enlarge the series set with every deployment; concrete paths, slugs and peer hostnames belong in span attributes instead, where they cost nothing per value.

Watching Spotify, or how a Last.fm widget became a Rust service

The music widget is a Rust service that registers itself with Spotify as a Connect device. The obvious design would make it the playback device and have it report what it plays, which would be of no use whatever, given that I do not listen to music through my web server.

It registers, then, and afterwards it does nothing but watch. Connect broadcasts cluster updates to every device on the account, so a device that never plays anything nonetheless sees what the phone, the desktop and the speaker are up to, and visitors receive each new track over a WebSocket, with pause, resume and seek following as light delta frames. Play counts observe the old scrobbling convention of thirty seconds, on the reasonable ground that a track skipped after four is not a track anyone listened to.

Design consistency

The blog and the static site draw their colour tokens from a single file. They were formerly declared twice over, because the Hugo theme piped its values through Sass colour functions that will not accept a CSS custom property, and a merge of that sort is exactly the kind of change where one very easily believes oneself finished when one is not.

The check was to build the site before and after and compare the multiset of generated CSS rules, which came out as the same rules, the same count and the same bytes, with only the :root block relocated from the middle of the file to the front.

Three services, three languages, and one collector forwarding traces, metrics and logs to a Grafana stack on another machine, which is a good deal more machinery than a personal website requires and was, if I am honest, rather the point, because the interesting problems only present themselves once the thing is real enough to break.

The blog is a hard fork of GoBlog by Jan-Lukas Else, ported from SQLite to PostgreSQL. That port is a post of its own.