BankingSync v4 released into stable channel

Hear ye, hear ye: BankingSync v4 has gone out on the general availability channel.

Most of what went into it is correction rather than addition, which makes for a duller changelog than usual but, I would argue, a more useful post, since the interesting part of a release like this one is not what was added but what turned out to have been wrong until now.

Model validation

Version 3 shipped a Fellegi–Sunter matcher, and that design has not changed at all: each field yields a discrete comparison level, each level carries log₂(m/u) bits of evidence, the bits are summed into a probability, and two thresholds leave a band in the middle where you, esteemed user, decide instead of the machine.

What changed is the standard the implementation was held to. Version 4 is what came out of my ambition to stop testing the statistics against my own reasoning and to start testing them against the literature they were taken from — Platt’s own note on his method as well as Stephenson on the Brier decomposition, Houlsby on active learning, Zadrozny on what selection bias does to a naive-Bayes learner — under the premise that every claim the code made about itself had to be backed either by a source I had read or by a measurement I had actually run.

Eighteen assumptions in the previous version did not survive that treatment. Only four of them had been directly affecting decisions about real money, and those four are set out below; the remaining fourteen were corrected all the same, on the principle that an assumption which happens not to matter yet is still an assumption (and also because I am a perfectionist in that regard). Three further corrections suggested themselves along the way, and I measured those too, but they improved nothing and are therefore not part of this release.

A guard that had quietly stopped guarding

A truncating bank authorises a purchase as Da Luigi Roma and books it as VISA Da Luigi, the card scheme prefixed and the tail cut off to fit the field, and recognising that pair as one transaction is the very thing this project was created to do. Had this not been brought to my attention, release v3 and v4 would have focused on UI and utility improvements; as it is, there were more pressing matters to attend to.

The optional refit turned out to be capable of breaking the matching altogether. Thirty labelled matches drawn from a bank that truncates eight per cent of its payees will, about eight per cent of the time, contain no truncated payee at all, and the estimator read that silence as rarity, taking the level from +4.32 bits to −0.63. Da Luigi fell from P = 0.943 to P = 0.350, which is below the review threshold, and below the review threshold means written into the budget a second time rather than put in front of anybody, which is exactly the behavior we were trying to address in the first place.

All six behavioural anchors, which exist precisely to catch this sort of thing, passed regardless. The one named for the truncating bank paired a truncated payee with an amount agreeing to the cent, and an amount agreeing to the cent is worth +5.13 bits, enough to carry the case on its own; the failure the test was actually written for involves an amount agreeing only within tolerance, worth +0.74. Four and a half bits of slack lay between the two, and somewhere inside that slack the guard had stopped testing the case it was named after, without anyone being the wiser.

Both are now addressed - the refit now holds a level at its shipped ratio where the labels are merely silent about it and lets it shrink only where the silence is genuinely informative, the difference being whether the prior expected to see the level at all; and the anchor tests the case it describes in the first place.

A gate that opened too willingly

Refitted parameters take effect only through a gate, which requires the anchors unchanged, a better Brier score on evidence the fit never saw, and a person pulling the trigger via a manual backend override.

“Better”, on closer re-inspection of what I’d written during a sleepless night, was a bare >= between two noisy means over about a hundred decisions. To find out what that was worth I simulated a bank whose true parameters were already the ones in force, so that nothing should ever have been promotable, and the gate opened on 44 % of single looks - which was quite unacceptable.

That gate is now a Diebold–Mariano test on the paired loss differential, with the Harvey–Leybourne–Newbold small-sample correction, reported as a Brier skill score, and the same simulation opens it in none of forty attempts.

Platt scaling with the safeguards it was missing

The Platt scaling was textbook Newton’s method, without the protections that Lin, Lin and Weng published in 2007 for this exact problem. Across the range of weights this model produces, where the strongest comparison scores +10 bits and the weakest −9.5, it did one of two things: either it diverged to a slope near 10¹⁰ and scored worse than applying no calibration at all, or a saturated Hessian tripped the determinant test on the first iteration and it returned the identity. Neither outcome was visible from outside, which is the part that ought to bother anyone running it the most.

The solver has been replaced by Algorithm 1 from their paper, with sigmoids and logarithms branched on sign, a ridge on the Hessian, an Armijo line search, and a stopping rule on the gradient rather than on the step, and it recovers a true slope of 0.40 as somewhere between 0.37 and 0.42 at every spread from one bit to ten - a vast improvement over the previous generation.

Five more correctness fixes, found along the way

• Review answers were filed against the model’s best candidate even though the page lets you merge into any of them, so that choosing the second candidate caused the software to record a positive label against the very pair you had just turned down. Answers now carry the levels of the candidate you actually chose.
• The non-match sample skipped held windows and dropped the leading candidate everywhere else, which against known rates was worth nearly a full bit of unearned evidence on the strongest comparison there is. The exclusion now applies only where the matcher actually committed to a row.
• A truncated payee outweighed a verbatim one, with the result that a pair agreeing in full could land in the review queue while the cut-off spelling of the same pair merged without asking.
• The term-frequency floor scaled with statement length and therefore loosened as an account gathered history, until the payee term alone could reach +11.1 bits, which is more than the rest of the model put together at +6.54.
• The refit’s prior concentration was a number with nothing behind it. It is two now, chosen by simulation and measurement across four loss functions and four synthetic banks.

Additions

Seven metric series are new and three have changed shape. Three spans were added, one of them emitted per decision: it breaks the weight down term by term, names the comparison level each term came from, and records the gap to the runner-up, so that a merge can be explained a month later rather than merely asserted.

docs/telemetry.md now covers every metric, span and log record, and a test holds the document against the source, because an alert written against a series that has quietly stopped existing does not fail; it simply never fires again, and nobody finds out until they needed it.

Upgrading

Simply pull and restart. Tables and columns are created on open and nothing is re-imported; three things are worth knowing beforehand:

• Any parameter set you had promoted is discarded once, on first open, and the discard is logged. Sets fitted before this release came off a decision log with known defects in it, and nothing on record says which corpus produced which set, so there is no honest way to keep them.
• The shipped parameters changed, and param_version changes with them, which means every series carrying that label starts fresh. Neither threshold moved; 22.8 % of decisions now sit within a bit of one, against 25.8 % before.
• Dashboards will need work for the series above; the telemetry reference has the details.

If all of this works, you should not notice any of it, which is the nature of a matcher: it is only ever visible when it is wrong.

References

The model, and the sources this release was checked against.

Fellegi, I. P., & Sunter, A. B. (1969). A Theory for Record Linkage. Journal of the American Statistical Association 64(328), 1183–1210. — the matcher itself.

Platt, J. C. (1999). Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods. In Advances in Large Margin Classifiers, MIT Press, 61–74. — the calibration layer.

Lin, H.-T., Lin, C.-J., & Weng, R. C. (2007). A Note on Platt’s Probabilistic Outputs for Support Vector Machines. Machine Learning 68(3), 267–276. — the safeguards the solver was missing, and the algorithm it now uses.

Murphy, A. H. (1973). A New Vector Partition of the Probability Score. Journal of Applied Meteorology 12(4), 595–600. — the reliability–resolution–uncertainty decomposition.

Stephenson, D. B., Coelho, C. A. S., & Jolliffe, I. T. (2008). Two Extra Components in the Brier Score Decomposition. Weather and Forecasting 23(4), 752–757. — why three components do not add up once you bin by width.

Diebold, F. X., & Mariano, R. S. (1995). Comparing Predictive Accuracy. Journal of Business & Economic Statistics 13(3), 253–263. — the test the promotion gate now applies.

Harvey, D., Leybourne, S., & Newbold, P. (1997). Testing the equality of prediction mean squared errors. International Journal of Forecasting 13(2), 281–291. — the small-sample correction to it.

Houlsby, N., Huszár, F., Ghahramani, Z., & Lengyel, M. (2011). Bayesian Active Learning for Classification and Preference Learning. arXiv:1112.5745. — the criterion behind the optional confirmation question.

Zadrozny, B. (2004). Learning and Evaluating Classifiers under Sample Selection Bias. ICML. — why a naive-Bayes learner is not indifferent to where its labels come from.

Zhai, C., & Lafferty, J. (2004). A Study of Smoothing Methods for Language Models Applied to Information Retrieval. ACM Transactions on Information Systems 22(2), 179–214. — Dirichlet prior smoothing, which is what the refit is.