How we keep score

1 The task, in one paragraph

A small model, small enough to run in a browser, learns to read one person's handwriting from a pen tablet.

A cheap pen tablet records the pen as points: where it is, when, how hard it presses and how it tilts. A recogniser turns one line of those points into text.

Ours is small on purpose, so it can run in a browser: a bidirectional LSTMBiLSTM Two LSTMs over the same sequence, one forward and one backward, joined at every point. Changing it: Each point is read knowing what came before and after, like you do with a messy letter, at twice the cost of one direction. Here: 3 layers of them in v1 and v2. (a network that reads the line in both directions) of one to three layers, trained with CTCCTC A way to train and read a model that gives one guess per point, when nobody has marked which points belong to which letter. Changing it: It lets the model train on whole lines with only their text. Models trained this way say "blank" almost everywhere, with a short spike for each letter. Here: Merge repeats, then drop blanks: "hh_e_ll_l_oo" reads "hello"., which lets it learn from whole lines without knowing where each letter starts.

It learns first from synthetic inksynthetic data Training data written by code instead of collected from people: here, handwriting fonts turned into pen paths with timing. Changing it: It's unlimited and licence-clean, but if it doesn't look like real writing the model learns the wrong thing: the synthetic-to-real gap. Here: 300,000 lines for v2, from 134 fonts under open licences (OFL or Apache). drawn from fonts, then from one person's own lines: personalisingfine-tuning Continuing to train an already-trained model for a short while on new data: here, your own lines. Changing it: It adapts a general model to your hand from very little data. Too long, or with too high a learning rate, and it memorises your lines or forgets general writing. Here: 40 varied copies of each of your 25 lines, plus 2,000 synthetic lines replayed. It took v2 from 43.6% to 13.1% errors.. The question running through the series is how good a model that small, and that personal, can get.

2 The ruler: character error rate

We count the letters a person would have to fix, out of the letters they meant to write.

The CERCER Character error rate: substituted, deleted and inserted letters, divided by the number of letters in the right text. Changing it: The main score here; lower is better. It weighs every letter the same, and it can go over 100% when a reading has many extra letters. Here: The goal is 15% or less. v2 personalised: 13.1%. (character error rate) compares what was meant with what the model read. It counts the fewest one-letter changes that turn one into the other: the edit distance (Levenshtein, 1966).

A change is a substitution (wrong letter), a deletion (dropped letter) or an insertion (extra letter). WERWER Word error rate: the same as CER, counted in whole words. Changing it: Harsher than CER: one wrong letter makes the whole word wrong. Closer to how a reader judges a note. Here: v2 personalised: 40.2%, against 13.1% CER. is the same count over whole words.

CER = (substitutions + deletions + insertions) / letters meant

The fewest changes come from filling a table. Each cell is the distance between the first i letters of one and the first j of the other, and needs only its three neighbours.

Type into the boxes below: the lit path is one cheapest way through, and red cells are edits.

Two things follow.

• CER can go over 100%: a reading full of extra letters has more insertions than there are letters.
• Every letter weighs the same, so a missing accent (e for é) costs as much as a wrong letter. That matters for French.

3 Two gates, both required

A model is done only when it is both accurate enough and fast enough; one without the other doesn't count.

Accuracy

< 15%

CER on held-out lines, with the whole interval under it

Speed

< 50 ms

p95: 95% of lines read within it

Test lines

24

English, French, mixed and math, never trained on

Speed is read as the p95p95 The 95th percentile: the time within which 95% of lines are read. Changing it: It shows the slow lines a user notices, which an average hides. Here: Budget: 50 ms per line. v2 personalised: about 58 to 61 ms., the time within which 95% of lines are read, not the average. A note-taking tool that is usually fast but sometimes stalls feels slow.

A model that clears one gate and not the other isn't finished, and the series says so.

4 One score is one draw

With 24 test lines, a score is a rough guess: we report the range it could plausibly fall in.

A test set is a sample of lines, and a different 24 lines would give a different score. The bootstrapbootstrap Estimating how much a score would change on a different test set, by drawing the test lines again with replacement many times and re-scoring. Changing it: It tells a real improvement from luck. Paired means the same lines for both models, which removes the "some lines are harder" noise. Here: v2 beat v1 in all 10,000 paired resamples, by 5.0 to 15.6 points. measures how different.

Draw 24 lines from the 24 with replacement, so some come twice and some not at all. Score the draw, and repeat. The spread of those scores is how sure the number is.

Play it on our best model so far.

With 10,000 draws, the middle 95% runs from 7.2% to 13.9%. So a model only passes the accuracy gate when that whole range sits under 15%: the single score alone could be luck.

5 Comparing two models: pair the lines

Scoring two models on the same lines cancels out the hard ones, and shows differences that two separate scores hide.

Two models read the same lines, so compare them on the same draws. The paired bootstrap scores both models on each resample and keeps the difference.

A line that is hard for both cancels out, so the range on a difference is much narrower than the two separate ranges suggest.

Δ* = CER_A(lines*) − CER_B(lines*)

6 The protocol

Five rules every experiment follows, learned from the first two iterations that didn't.

The first two iterations didn't follow it, and the next episodes show what that cost. The protocol came out of those mistakes.

7 What carries over

Five habits for judging any model on a small test set.