Implementing the homepage’s widget

The device playing a song knows perfectly well what it is playing; for anyone standing even one step removed, the same fact turns out to be oddly difficult to come by. The first post in this series granted the music widget on the home page four hurried paragraphs, and this one is the amends I owe it.

For a long while the widget was fed by Last.fm, by way of a third-party script that polled a third-party API to report what a third-party scrobbler had in turn been told by Spotify. Three intermediaries, each reasonable enough taken on its own, yet each adding its own delay to the one before and each failing in its own particular way — and not one of them a box I could open and look inside. What I wanted was for the page to know what I was listening to at the very moment I began listening to it, and the only party in possession of that knowledge is Spotify itself. I wanted, too, to own the machinery outright, so that whatever broke would be at once my fault and within my power to mend.

The watcher on the wall

Spotify Connect is the contrivance by which the phone in your pocket instructs the speaker in the kitchen what to play. For that to work at all, every device on an account must remain aware of what every other device is up to, and so Spotify maintains, one per account, what it calls a cluster: the roster of registered devices, the identity of whichever is active, and the full player state of that active one — track, position, the server time at which the position was taken, and whether it is paused. Whenever any of this shifts, the update is pushed to every device over a persistent connection that Spotify, with a certain candour, names the dealer.

Every device, note — not merely the one that happens to be playing.

Therein lies most of the trick. librespot, the open-source Connect client written in Rust, will cheerfully register a bare process as a device in its own right, and a device so registered receives the cluster updates of the entire account whether it is playing a note or sitting idle. nowplaying accordingly joins the account, subscribes to hm://connect-state/v1/cluster, and from that moment does nothing whatever but watch. It decodes not a single sample: the player is handed a sink whose write returns Ok(()) and is otherwise inert, and no audio library is so much as linked into the binary, there being nothing for one to do.

Two details proved to bear real weight. The first is a matter of order. librespot does not connect its session until every dealer listener is in place, and Spirc — the component that turns a librespot process into a Connect device — subscribes to that selfsame URI for purposes of its own; my subscription must therefore be registered before Spirc is ever constructed, and the arrangement holds together only because the dealer fans each message out to every subscriber of a URI rather than surrendering it to whoever happened to ask first. I satisfied myself of that last point in a throwaway binary before I wrote a line of anything else, on the principle that an architecture resting on an unexamined assumption about another man’s message router is no architecture at all, but a wager dressed as one.

The second is that a device, once registered, is also selectable. It presents itself in the device list on my phone as “NowPlaying Widget”, and ConnectConfig affords no means of concealing it. Choose it by mistake and Spotify will dutifully stream audio to a server that discards every packet as it arrives — silently, and scoring the whole futile exercise as a play. The service now recognises when the active device is itself and withdraws at once: Spirc::disconnect(true) halts playback and relinquishes the role, and it does so once per activation rather than once per update, the cluster being much given to announcing every change several times over.

What, precisely, is a play?

A cluster update is a snapshot and not an event. It reports “this track, at this position, as of this instant”, and it reports as much whenever anything in the cluster alters — a skip, a pause, a seek, and a fair number of occurrences that have nothing to do with the player at all. Discerning which of these snapshots amounts to a genuinely new play is the whole of the real work, and it falls to a small state machine that never so much as touches the network, precisely so that it may be unit-tested to exhaustion.

The anchor is the instant at which playback began — a quantity no snapshot states outright, yet every snapshot implies:

Seek forward by a minute and both the position and the implied start will move with it, unless one takes care to pin the start at the first observation and merely extrapolate thereafter, which is what the tracker does. A snapshot of the same track whose position falls within three seconds of where extrapolation says it ought to lie is the same playback continued; anything further afield is a seek; and a leap back into the first five seconds of a track that had already been running for more than thirty is a restart, and with it a new play.

That final rule harboured a bug from the very first release, and it is the most instructive bug the project has yet produced.

The original condition inquired whether the previously observed position lay past the thirty-second mark. The formulation reads impeccably, and every test I had written was in cheerful agreement with it — for the sufficient reason that every test fed the tracker a snapshot drawn from the middle of the track before sending it leaping back to the start. Real clusters are not so obliging. Updates arrive only when something occurs, and when a track runs from beginning to end with nobody laying a finger on it, nothing occurs: the last snapshot the tracker had seen was the one taken at position zero. Repeat-one, then, or the same track queued twice in succession, concluded with a jump from a remembered zero to an actual zero — which, under the old rule, was no jump whatsoever, so that the second listening simply never existed in the eyes of the statistics. The remedy is a single word: measure against the extrapolated position rather than the remembered one.

What would have caught the thing is a property, not an example. The tracker is now exercised against generated sequences as well: an arbitrary interleaving of pauses, resumptions and duplicated updates, so long as their positions are physically consistent, must yield exactly one play; seeks anywhere save the very start must yield none; and an untouched play-through followed by a return to the beginning must yield precisely two, whatever the length of the track and however belatedly the restart is reported. The last of these fails against the old code for every input the generator is capable of devising.

Thirty seconds

A play reaches the page’s visitors the instant it is detected, but it reaches the database only after thirty seconds of genuine listening — the old scrobbling convention, resting on the eminently reasonable premise that a track abandoned after four seconds is not a track anyone has listened to. “Thirty seconds of listening” is not, however, the same creature as “thirty seconds of wall clock”, and the whole of the difference is the pause: a track stopped at the twenty-second mark and taken up again an hour later has been listened to for twenty seconds, not for an hour.

Each detected play therefore carries a small clock of its own, one that accumulates only those intervals during which it was in fact playing, together with a timer armed against the time remaining — cancelled upon a pause, re-armed upon resumption, and thrown away altogether should the listener skip before the threshold is reached. A skip thus leaves no residue in the statistics, and there is nothing to undo, for the plain reason that nothing was ever committed to the database to begin with.

That arrangement concealed a race, and it took me an embarrassing while to catch sight of it. To discard a play was to abort its timer task; but the timer task, once its thirty seconds had run, would proceed to perform the insert itself — so that a skip arriving while the insert was still in flight would tear the insert off halfway and record the play as discarded, though by every measure that signifies it had earned its place. With a lone insert the window stood a few milliseconds wide and was harmless in practice. With the retries I meant to add next, it would have yawned open to several seconds and been anything but harmless. The timer now hands the play off and marks itself as fired; the insert proceeds detached, beyond the reach of any later skip, and a discard arriving after the firing is simply a no-op.

The retries themselves are nothing to marvel at — five attempts, spaced one, two, four and eight seconds apart — and they are safe for a reason that long predates them. Every insert is keyed on the pairing of track and start time and quietly ignores conflicts, and it is further guarded against any pre-existing play of the same track falling within ten seconds of the same start. Should an insert commit and its acknowledgement go astray on the journey home, the retry discovers its own row already written and stands down. It is the very same guard that allows a kill -9 and the restart behind it, which re-detects whatever is playing at that moment, to refrain from ever recording a single play twice.

A dropped connection to Spotify used, oddly enough, to cost rather more than an outright crash: each reconnection began afresh from a clean slate, and a play twenty seconds along would set out on its thirty seconds anew. The tracking state now outlives the connection entirely. For as long as the service stands disconnected the play’s clock is held in suspension — whatever I listened to across the outage is simply unknowable, and so, out of prudence, goes uncounted — and the first update to follow the reconnection, should it find the same track still playing, resumes it with the twenty seconds already served left standing to its credit.

Whose clock are we using, anyway?

The page displays a progress bar, and a progress bar is at bottom an extrapolation: the server furnishes a position together with the time at which it held, and the browser advances the position as the track plays on. Both that position and its timestamp, however, are reckoned by Spotify’s clock, whereas the extrapolation runs on mine — and a server whose clock has wandered a second out of true will publish positions that are wrong by a second.

librespot does keep an offset, drawn from the pings of its access-point connection, but only in whole seconds, which for the present purpose is no measurement at all. The cluster envelope, happily, carries a server timestamp to the millisecond, and this admits of a pleasingly compact piece of reasoning. Spotify stamps a message before dispatching it, and the message reaches me only after some network latency $L_i \geq 0$. If $\theta$ denotes the true offset between the two clocks, then every sample I am able to take is

Latency, in other words, can only ever drag a sample downward. The best estimate is therefore not the mean of the recent samples, which latency biases in one direction without fail, but their maximum — the sample that was least ill-used in transit; taken over a sliding window of eight, it tracks a host clock that drifts, into the bargain. Anything beyond five minutes is dismissed out of hand as implausible for a host that keeps time by NTP.

But what about podcasts?

The cluster reports whatever the active device is playing, and not everything a device plays is a song. A podcast episode arrives bearing the shape spotify:episode:…, a file from the phone’s own storage as spotify:local:… with its artist, album, title and duration form-encoded directly into the URI, and an advertisement as something else again.

The first version knew nothing of any of this, and the manner of its failure was a quiet one. librespot’s metadata lookup for a track declines any URI that is not in fact a track, so an episode slipped through to the cluster’s own metadata map, which offers a title and no artist name whatever — whereupon the episode was duly published and stored as a song possessed of no artist, linking out to open.spotify.com/track/ followed by the identifier of something that is no track at all. Nothing raised an error; the thing merely displayed the wrong answer with perfect composure.

Kinds are now modelled in the open. Episodes are resolved as episodes, linked to a page of their own, stored as such, and kept clear of the top lists; local files are shown but never stored, there being nothing public to link them to; and everything else is passed over, with a play interrupted by an advertisement held rather than discarded, so that a return to the same track takes up its count where it left off.

That still left the matter of the past. Every degraded row — anything set down from the cluster map’s threadbare fallback rather than from a full record — now bears a mark saying as much, and a repair pass runs after each connection and hourly thereafter, retrieving the full record and writing it over the row. The intriguing case is the old episode once stored as a track: asked for as a track, it is pronounced by Spotify not to exist, at which the repair pass asks a second time for an episode of that same identifier and, on success, relabels the row. The database thereby comes to learn, after the fact, something it was never originally told. A row that remains unknown under both readings is retired, lest it be interrogated hour upon hour for ever, and three unavailable answers in a row bring the pass to an early close, on the reasoning that a service already labouring has no need of my adding to its burden.

What leaves the server, and what does not

Visitors are sent two kinds of frame. A play frame conveys the full record of a new track; a state frame conveys no more than a playback state, a position, and the server time at which that position held, and it does so for every pause, resumption, seek and stop. Not every update, however, rises to the dignity of news: a seek is published only once it exceeds three seconds while the track plays, or half a second while it is paused — a frozen position suffers no extrapolation error, only the millisecond jitter of duplicated updates — and an identical update is never dispatched a second time.

These frames rest upon an invariant I had not originally troubled to state, though I ought to have: a state frame refers always to the track the visitor last received in full. The service was wont to violate it at start-up, when the first thing it chanced to observe was a track that had lain paused for hours — a state that would never ripen into a play, and hence a state frame concerning a track of which the page had never once been told. The service now ushers such a track in with a full frame first, and drops the state outright wherever no metadata can be found, a progress bar for an unnamed track being decidedly worse than none.

Two things take their leave of the server never at all. Should the playing device report a private session, nothing touching what it plays is so much as examined, let alone published or stored; the page merely watches the last public track draw to its close. That flag, too, I am taking upon the protocol’s word until the day I have watched it set from my own phone.

And the lyrics depart only in disguise. The page lays them behind the title as a washed-out backdrop, synchronised line by line, but what reaches it has first been scrambled on the server: the opening letter of each word is spared, every other letter is supplanted by a random one of the same case, and punctuation, word lengths, line structure and timestamps are left precisely as they stood. The randomness is seeded by a hash of the track’s identifier and the line’s index, so that the scramble comes out deterministic — a visitor who reconnects is met by the very same nonsense as before, rather than a freshly minted helping of it — and the words themselves never cross the boundary of the process. The shape of a song, it turns out, is backdrop enough; its substance is not mine to republish.

Instruments, and their limits

The service is instrumented as thoroughly as anything I run, its traces, metrics and logs departing over OTLP for the Grafana stack to which the first post of this series made passing allusion, so that a single cluster update may be followed through a trace from the moment of its arrival to the insert it occasions half a minute later. Two of the decisions taken in that quarter are worth the recounting, for both cut against the grain of what one would expect.

WebSocket sessions are given no span. A visitor may well keep the home page open for a whole afternoon, and a span is exported only once it has ended — so that an afternoon-long span is invisible for the length of that afternoon, forfeited altogether should the process expire in the interim, and ruinous to any honest view of durations. The upgrade request now receives instead a brief child span, covering the subscription and the first frame, and the close of the session is recorded as a structured log event bearing its reason, its duration, and the tally of frames it received.

And failures of authentication, alone among failures, do not bring the process down. Every other persistent inability to reach Spotify does, and by design — more than five of them inside ten minutes and the service exits, leaving systemd to start it afresh, which is the proper answer to a connection that has wedged. Rejected credentials, though, are mended by no restart; they are mended by me, running the login flow a second time, and a restart loop would do nothing but inter that fact beneath a drift of noise. The service parks itself instead, declares itself unhealthy, raises a metric kept for no purpose but to be alerted upon, and settles down to watch the credentials file, reconnecting of its own accord the instant the file is changed.

One Rust binary, then, enrolled as a loudspeaker that never emits a sound, eavesdropping on the private traffic of every device on an account for no grander end than to tell a web page which song is on. It is, by any sober accounting, a prodigious quantity of engineering to lavish upon a widget.

nowplaying is available on GitHub, and can be seen at work on the home page.