One recipe, 136 training runs

1 The question

Episode 7 told all of iteration 3. This one follows one step of it, fitting the model to the writer, through every training run it took.

The base modelbase model The general reader, trained on lots of synthetic handwriting before it ever sees yours. Changing it: A better base gives a better personal model: it starts lower and learns your hand faster. Alone it can still read your writing badly. Here: From v1 on, trained on a cloud GPU from a recipe; v2's base alone gets 43.6% of your letters wrong, 13.1% once personalised. reads handwriting in general. Personalisingfine-tuning Continuing to train an already-trained model for a short while on new data: here, your own lines. Changing it: It adapts a general model to your hand from very little data. Too long, or with too high a learning rate, and it memorises your lines or forgets general writing. Here: 40 varied copies of each of your 25 lines, plus 2,000 synthetic lines replayed. It took v2 from 43.6% to 13.1% errors. trains it a little more on one writer's own lines, so it learns their hand. How base v3 was built is in episode 7; here it is the starting point.

Personalising has six settings, and the right value of each depends on the others. This episode is the lab notebook of how they were chosen: every search trial, every repeat, what each cost, and why the recipe says what it says.

Personal v3.1-stable, test lines

6.8%

41 of 600 letters; v2.1-personal: 62

Training runs to choose it

136

120 search trials, then 16 models

Rented GPU time

64 min

an A10G, rented by the second

On the bill

$1.99

for all of personalising

2 The day, step by step

Thirteen steps between the base model and the release. Most cost nothing; the two searches cost most; two steps went wrong and are kept here.

1. Pick the starting pointfree
   Base v3: 27.2% on the test lines before personalising, the same size and speed as v2.1-personal, and it has never seen a line of the writer's.
2. Join both recording roundsfree
   85 training lines (one round had been uploaded twice: 40 duplicates kept once). A fifth of each language, 17 lines, is set aside as dev lines; 68 are learned from.
3. Time the laptopfree
   At most 12.9 lines a second: a thorough search here would take 12 hours or more. So the search goes to a rented GPU (section 5).
4. First attempt stops before the GPUfree
   Making the synthetic lines to mix in failed: they needed the bank of hand-like words the base's own lines were made with. A bug, fixed, with a test that fails without the fix.
5. Search 1: 60 trials19 min · $0.52
   Best: 10.9% on the dev lines, against 13.4% for v3's own plan. But at the edge of two ranges (section 6).
6. Search 2: 60 trials, wider ranges31 min · $0.84
   Best: 10.4%, in the middle of the new ranges (section 7).
7. Make both bests again, test what they forget$0.04
   Two letters apart, so a rule written beforehand decides: the one that forgets less. The short plan becomes the recipe for v3.1 (section 8).
8. v3.1 from its recipe: 15.6%$0.01
   The same plan that had scored 10.9% twice. Nothing had changed (section 9).
9. Twelve repeats: three plans, four more seeds each$0.29
   The long plan has the best average (section 10). On the way, a connection slip (section 14).
10. Make v3.1-stable$0.03
    Its own recipe: the long plan, seed 0. 10.43% on the dev lines, epoch for epoch the same as before.
11. Tune the decoder on the dev linesfree, on the laptop
    10.2% → 9.3% on the dev lines.
12. Read the test lines, oncefree
    6.8% with the default decoder, 7.8% with the tuned one: the defaults stay (section 13).
13. Time it, export itfree
    Within the 50 ms budget on every timing run; an 8.6 MB file for the browser that reads exactly as the training code on 10 of 10 check lines.

3 Six dials, in plain words

Each setting trades one risk against another: learn the hand too little, or learn these 68 lines too well and forget the rest.

Personalising starts from a model that already reads well and nudges it toward one hand. Every setting decides how hard, how long, or with what, and each fails in both directions:

A hand-picked plan existed: v3's own, from the base's recipe. The question was whether a search could beat it, by how much, and whether the gain would hold.

4 A ruler for small differences

Settings are judged on 17 lines the model never learns from. On them one letter is 0.23 points, and a single reading wobbles by several letters.

The writer's lines are split three ways, and each pile has one job:

Learned from

68

both rounds, the fit

Dev lines

17

441 characters: settings are chosen on these

Test lines

24

600 characters: read once, at the end

The dev linesdev set A second set of held-out lines, for choosing settings and the best epoch. Changing it: It keeps the test set untouched, so the final score isn't flattered by choices made while looking at it. Here: A fifth of the writer's training lines, held out: the decoder and the personalising settings are tuned on them, never on the test lines. are a fifth of each language, picked by a fingerprint of each line's name, so every run of the day held out the same 17. One wrong letter of 441 is 0.23%: two plans half a point apart differ by two letters. Keep that ruler in mind for every number below.

The dev score is also noisy from epoch to epoch. The release's own run jumped by up to 10 letters between one epoch and the next (13.2% → 11.8% → 14.1% → 12.0% → 13.8% → 12.2% → 11.3% → 10.4%). So a single dev reading is a measurement with a wobble of several letters, not a fixed truth.

5 Measured first: why the search left the laptop

On this laptop personalising tops out near 13 lines a second, and more cores don't help. A rented GPU ran the same code 38 to 71 times faster.

Before choosing where to search, the laptop was timed. Personalising this model can't use more than about two processor cores, and running several at once barely adds up:

One model of the final plan reads 5,177 lines an epoch for 8 epochs. At the laptop's best that is at least 54 min of training; on the rented A10G it took 52 seconds. A 60-trial search here: 12 hours or more.

The GPU benchmark had promised about 1,500 lines a second, roughly 100 times the laptop. The personalising runs themselves read between 490 and 918 lines a second per epoch: 38 to 71 times faster. Less than promised, and still the difference between a search in minutes and one overnight.

6 Search 1: sixty trials draw a map

A search tried 60 plans, stopping weak ones early. Its best, 10.9%, sat at the edge of two ranges: the map was cut off where the treasure might be.

Each trial is a real personalising run, about a minute on the GPU, scored on the dev lines. Trial 0 is v3's own plan, so the search can't end worse than it. Optuna, an open-source search library, picks the rest:

Search 1's best used 2 epochs, the fewest it allowed, and a learning rate 95% of the way up its range, measured the way the search drew it. A best at the edge of a range says the real best may lie past it.

And one setting was settled for good: freezing never helped. Of the 14 trials across both searches that kept any of the base's layers frozen and ran every epoch, the best read 12.5%. With nothing frozen, each search did 7 and 9 letters better. The writer's hand needs the low layers too (episode 7 has the chart).

7 Search 2: a wider map, and a second valley

With room past both edges, the best landed in the middle instead: 8 gentle epochs, 10.4%. Two different recipes, two letters apart.

The second search allowed learning rates up to 0.01 and as few as 1 epoch. It was a study of its own: the search method only learns from trials that shared the same ranges.

Past the old edges there was nothing to find. The 6 trials above the old ceiling scored 13.2% at best, and the 3 one-epoch trials 13.2%. Instead, the search found a second valley:

Which setting mattered most? Optuna estimates the share of the spread in scores that each setting explains. The answer changed with the map:

8 What mixing in synthetic lines buys: a forgetting test

The long plan's batches are 93% the writer's ink, so it forgets more of the base's skill. A rule written beforehand used that to break the tie, for the short plan.

The two bests learn from very different diets. Counting what one epoch holds (the writer's 68 lines with their copies, beside the synthetic lines):

A plan that is nearly all the writer risks forgettingcatastrophic forgetting When a model trained on new data loses what it knew before. Changing it: A personalised model could read your hand and nothing else. Mixing old data back in (replay) and a small learning rate prevent it. Here: Checked every personalisation: error on synthetic lines stays flat. what the base knew. A few synthetic lines are held out of each run to check, but for the long plan that is 12 lines: far too few to see a change of a point or two. So both bests were read on 2,000 synthetic lines that neither had ever seen:

The two bests were 2 letters apart on the dev lines, inside the noise. The rule, written before measuring: within 3 letters, keep the plan that forgets less. The short plan won, and its values were written into the recipe for personal v3.1.

It was a sensible rule. It was about to be overturned by something no rule had foreseen.

9 Failure: the same recipe, 21 letters worse

Made from its recipe, the short plan scored 15.6% instead of 10.9%. GPU training isn't exactly repeatable, and a search's best is partly luck.

Personal v3.1 was trained from its recipe: same plan, same seed, same lines, same GPU type. It read the dev lines at 15.6%. The same plan had scored 10.9% twice, in the search and as a candidate.

Why: the loss this model learns from is added up in parallel on the GPU, in an order that can change from run to run, and nothing in training asked the GPU to be exactly repeatable. Rounding then differs in the last digit, and training can amplify that from the first epoch. Sometimes it doesn't (the long plan repeated exactly); sometimes it does, and a short plan at a high learning rate amplifies it most.

10 Five seeds each: judge a plan by its mean and its spread

Over five runs, the long plan averages 11.4%, v3's plan 12.8%, the short plan 13.9%. The lucky winner was the worst of the three on average.

A seedseed The starting number for every random choice: which font, which word, how much slant. Changing it: Same recipe and same seed give the same data, so a version can be rebuilt. Different seeds show how much a result is luck. Here: Pinned in each recipe. now picks the varied copies and seeds training, so a plan can be run several times on purpose. Each finalist and v3's own plan, as the reference, ran with four more seeds:

The long plan beats v3's plan by 1.4 points on average, and the short plan by 2.6. Divided by their own uncertainty (Welch's t), those gaps are 2.7 and 2.9: past 2, a gap is rarely luck, though five runs is still a small sample.

Note who came last on average: the short plan, the search's original winner, 1.1 points behind the hand-picked plan it was meant to replace. v3's own plan was the steadiest (spread 0.6 points), just not the best.

11 The recipe, value by value

Every value of v3.1-stable, what it means in lines and steps, and the evidence that put it there.

The pick got a recipe of its own, written before the model was made and never edited after. v3.1's recipe stays as the record of the lucky plan; v3.1-stable's keeps both searches, the superseded tie-break and all fifteen repeat runs with their scores.

The learning rate reads 0.0016710470002597796 in the recipe. Nobody chose those digits: the search draws from a continuous range, and the recipe keeps exactly what it drew, so the model can be made again from it.

12 Win: 6.8% on the test lines, read once

v3.1-stable gets 41 of 600 test letters wrong, against 62 for v2.1-personal: better on 13 lines, worse on 3.

Character error rate

6.8%

95% interval 4.6% to 9.2%

Word error rate

23.6%

from 32.9%

95% of lines read within

45.1 ms

budget 50 ms

The test lines are the same 24 that v2.1-personal was judged on, so the two can be compared line by line. Each square below is a letter of a test line; the shaded ones are how many that model got wrong on that line.

Paired line by line, the step is −3.5 points (−6.0 to −1.3), and the interval stays clear of zero. It was made from fewer of the writer's lines: 68, where v2.1-personal learned from all 85.

Math is the one kind of line read better without the language modellanguage model A model of which letters tend to follow which, learned from plain text. Changing it: It fixes readings the network finds ambiguous. Weighted too heavily, it overrules what you actually wrote, such as rare words and names. Here: A character model (each letter given the 4 before it) from 120,000 Tatoeba sentences per language, weighted 0.5. (8.2% against 9.4%): it learned which letters follow which from ordinary text, and pulls formulas toward words. What the remaining mistakes look like:

Symbols, units and names: Δ read as √, "max $40" as "max d40", "ASAP" as "49/12". The network alone gets π right, and "Yasmin" all but its capital, where the language model talks it out of them.

On all 39 test lines of both rounds, v3.1-stable reads 8.6%. The 8% goal is met on the 24, but the interval reaches 9.2%: 24 lines can't prove "under 8%" yet. The comparison with every earlier model is in episode 7.

13 Not kept: a decoder tuned on the dev lines

Tuning the decoder gained 4 letters on the dev lines and lost 6 on the test lines. The defaults stay, and they keep within the speed budget too.

The decoder turns the network's letter scores into text, guided by the language model. Its six settings were searched on the same dev lines (100 trials, on the laptop), keeping the fastest settings within half a point of the most accurate.

4 letters gained on 441, 6 lost on 600. Four letters is inside the dev lines' wobble (section 4), and settings fitted to 17 lines learn their quirks. Most of the search's attention went to two settings: how early to drop unlikely readings (38%) and how much to trust the language model (20%).

The tuned decoder was faster, though, and speed has a budget: 95% of lines read within 50 ms. The first timing of the defaults, 49.2 ms, was taken while the site was being built on the same machine. So the two decoders were timed in turn, three times each, so that any load fell on both:

14 What it cost: $1.99

Two searches took most of the money. A finished model costs about 3 cents and a minute; the whole of personalising cost 13% of the base model.

One finished model

$0.03

about a minute on the GPU

All of personalising

$1.99

13% of base v3's $15.06

The same search on the laptop

12 h +

and the laptop busy all along

15 Making it again, and what's next

The recipe makes the model again, within the noise of GPU training. Three things to try next, each with a measurement waiting.

Everything that made v3.1-stable is pinned: the base, both rounds of lines, the dev split, the plan and its seed, the GPU type. Made twice, the long plan gave the same score after every epoch, down to the same 46 wrong letters at the end. But GPU training isn't guaranteed to repeat exactly, so a fresh run of the plan should be expected anywhere between its best and worst of five: 10.4% to 12.7% on the dev lines.

16 What carries over

Six habits for choosing training settings when the data is small and every score is noisy.