Anabasis, Book 1 - Θάλαττα! Θάλαττα!

Thálatta! Thálatta! The Sea! The Sea!

Words from an era long gone, and yet ones that had me sitting on a rainy morning in early May, wondering whether any of the projects I was turning over at the time could honestly be likened to the March of the Ten Thousand.

For the less philhellenic among you, a condensed history lesson. In 401 BC some ten thousand Greek mercenaries signed on with Cyrus the Younger, who wanted his brother’s throne and marched inland from the Aegean coast to take it. We call that march the Anabasis - the way up from the sea into the interior - which is also, less than aptly, the title of the whole book, though it covers only a small part of the journey and turns out to be, by far, the least interesting part: the men had pay, they had supplies, and above all they had a commander who was still alive.

At Cunaxa, a few days short of Babylon, Cyrus won his battle and died in the middle of it, which left ten thousand Greeks undefeated, unpaid, and on the wrong side of Mesopotamia. Their generals went to parley with the satrap Tissaphernes under safe conduct, were seized anyway, and Clearchus, who led them, was executed. So the army elected new commanders, and one of them was an Athenian who had tagged along as a friend of a friend and had held no command at all until that week. His name was Xenophon.

Then they walked north for months - over the Carduchian mountains, through an Armenian winter, losing men to the snow. And then this, which I’ll let Xenophon tell, since nobody has bettered it in two and a half thousand years:

On the fifth day they reached the mountain, the name of which was Theches. No sooner had the men in front ascended it and caught sight of the sea than a great cry arose, and Xenophon, in the rearguard, catching the sound of it, conjectured that another set of enemies must surely be attacking in front. […] But as the shout became louder and nearer, and those who from time to time came up began racing at the top of their speed towards the shouters […] Xenophon settled in his mind that something extraordinary must have happened, so he mounted his horse, and taking with him Lycius and the cavalry, he galloped to the rescue. Presently they could hear the soldiers shouting and passing on the joyful word, “The sea! the sea!”

The line everyone quotes is the last one. The part I keep coming back to is that Xenophon is at the back and sees none of it. He hears a noise from the head of the column and reads it as an ambush, because a noise from the head of the column has meant exactly that every other time. The man in charge, the one who will go on to write the history, is the last in the army to work out what is happening, and he only learns it by riding the whole length of the column and looking for himself.

(They didn’t reach Trapezus for a good while yet, incidentally - the Macrones, the Colchians and a memorable business with poisonous honey all come after the mountain. It’s the sighting that matters, not the arrival.)

What, may you now ask, is my personal Anabasis?

Auditing firewalls that nobody sells you as a firewall: nftables, on ordinary Linux hosts.

Before anyone accuses me of not having read the literature - the theory here is old, and it is undoubtedly good. FIREMAN, out of UC Davis and presented at Security and Privacy in 2006, takes a firewall configuration, treats it as a program, and checks every possible packet along every possible path through it, rather than throwing test traffic at it and hoping. It represents sets of packets as binary decision diagrams, which is going to matter a great deal later on. Margrave does something comparable in first-order logic, and will hand you a concrete packet that breaks your policy. The most serious Linux-specific work I’ve found came out of TU München around 2015, where Cornelius Diekmann and colleagues wrote a semantics for iptables in Isabelle/HOL, proved the thing, and could claim without blushing that they covered more of iptables than anyone before them.

All of it is about legacy iptables and Cisco ACLs.

Debian has defaulted to nftables since Buster; RHEL 8’s firewalld sits on an nftables backend; and the iptables binary on your machine is almost certainly iptables-nft, a translation layer that rewrites your 2004 syntax into the new subsystem before the kernel ever sees it. Twenty years of careful analysis, in other words, aimed squarely at a frontend that has quietly become a compatibility shim. And nftables isn’t iptables with tidier syntax - it has a different matching model, with sets and maps and verdict maps as first-class objects, intervals, concatenations, and arbitrary hooks in place of the old fixed tables.

The same TUM group wrote the sentence I’ve had lodged in my head ever since, describing the tools in their related work: they support only a limited subset of real firewalls, and when a firewall exceeds that subset, they either produce erroneous results or cannot continue. Those two outcomes are not equally bad. A tool that stops and admits it can’t model something has cost me an afternoon; a tool that misreads a meta l4proto match and reports everything green has cost me the very reason I installed it.

Two ways your firewall is wrong

Declarative config management is wonderful, and I do it for a living, so take what follows as a complaint from in-house: rolling out templates with Ansible or Terraform can’t see either of the two failure modes that actually happen.

The first is that the rules you wrote may not mean what you meant. The run was green, the file applied cleanly, every line is exactly what you typed - but whether all of that adds up to the policy in your head is a question about the rules in their entirety, and a template renders them one at a time. The second is drift, and this is where a host firewall stops resembling an appliance at all. Nobody else writes rules into your Palo Alto at three in the morning; on a Linux box, everything does. Docker installs its own chains and jumps into them, libvirt does the same for every virtual network, kube-proxy writes service NAT, and CrowdSec adds and expires ban entries around the clock, because that is its whole job. And of course, your colleague, under pressure to get critical infrastructure working again at 2AM, typed nft add rule with a broader-than-necessary rule, fully intending to properly check that rule into a versioned config afterwards and refine it to the most narrow variant of itself - which never happened. The file in Git is a statement of intent; the ruleset in the kernel is a fact, and the two are not the same object.

So diff it, then

That’s the obvious answer, and I’ve built it more than once: have an alert fire on whether the actual kernel ruleset differs from the config file deployed to the host. Here is what that actually gets you. This week I took a production ruleset recorded under nft 1.0.9, loaded it into a throwaway network namespace on a box running nft 1.1.7, and dumped it again - same rules, same kernel representation, same declared JSON schema version. The two files are not the same document. Objects come back in a different order, and five of the thirty-six rules spell their anonymous sets differently, because 1.0.9 emits them in whatever order the kernel’s hash table happened to hold them and 1.1.7 sorts them first. A set is a union, and a union is commutative, so precisely nothing has changed about what the firewall does - yet byte-comparing drift detection fires five times, and the afternoon you upgrade nft, every suppression you’ve ever written expires at once.

That, though, is the harmless failure mode. It just makes some people turn off the alert altogether, which is another failure mode in its own way.

The real problem, however, is that a list of rules isn’t a policy. What an auditor wants to know is whether anything outside the management network can reach tcp/22 on this host, and there is no single rule anywhere in the file that answers that. The answer to that particular question is a complex equation taking various truths about the rules into account: their order, which ones shadow which, the jumps between chains, what the base chain does when nothing matches, whether the packet is ct state established and skips your careful filtering entirely, and whether NAT rewrote the destination address at an earlier hook so that the filter chain is matching on something other than what arrived. You absolutely cannot read that off a list. You can even add a single rule that changes the meaning of the entire policy without another line moving a byte, and the diff will show you one added line and tell you nothing about the havoc wrought by it.

For a sense of scale: the ruleset I keep testing against is an unremarkable single server - three tables, six chains, thirty-six rules, one named set holding 23,621 addresses, 419 KB of JSON. Auditing it via diff is a futile endeavor, and even that ruleset carries some best-case preconditions, like a docker installation banned from editing the ruleset and, consequently, fixed bridge interface names which are manually encoded into the nftables.conf.

What nftables actually does give you

nft -j list ruleset. The entire ruleset - tables, chains, rules, sets, counters - as documented, versioned, machine-readable JSON, carrying a json_schema_version so a consumer can tell whether it’s looking at something it fully understands. Hand it to a JSON library and you have typed objects in microseconds.

What makes it useful isn’t the JSON, though. It’s that this comes out of the kernel - not your intent, not your nftables.conf re-read from disk, but what is loaded right now, after Docker and CrowdSec and the colleague at two in the morning have each had their turn, serialised by the very tool that speaks netlink.

All of the above leaves us with just one issue to be handled, and it’s the interesting, clawing-at-your-head-at-2AM one. Everything I want to ask is a question about sets of packets, and those sets are enormous. Source and destination address, protocol, both ports, connection state, ingress and egress interface, tcp flags, the firewall mark, icmp type - for IPv4, this yields at least 157 boolean variables, depending on the number of bridge interfaces, and 349 for IPv6. “Is there any packet at all that gets through to port 22” is a question about all of those variables at once. You can’t enumerate that. Sampling isn’t an option for a declarative compliance tool, either, since that’d just be testing for positive reachability with extra steps, while negative reachability could never be guaranteed at all.

How we resolve that issue is what we’ll be talking about in the next post of this series.