Iteration 3, value by value

1 The question

Iteration 3 rebuilt the base model's data and searched for how to personalise it. Every value had to rest on a measurement.

Iteration 3 has two halves. The first rebuilds the base modelbase model The general reader, trained on lots of synthetic handwriting before it ever sees yours. Changing it: A better base gives a better personal model: it starts lower and learns your hand faster. Alone it can still read your writing badly. Here: From v1 on, trained on a cloud GPU from a recipe; v2's base alone gets 43.6% of your letters wrong, 13.1% once personalised.: the general reader, trained on 300,000 lines of synthetic inksynthetic data Training data written by code instead of collected from people: here, handwriting fonts turned into pen paths with timing. Changing it: It's unlimited and licence-clean, but if it doesn't look like real writing the model learns the wrong thing: the synthetic-to-real gap. Here: 300,000 lines for v2, from 134 fonts under open licences (OFL or Apache). before it sees a single line of the writer's. The second fits it to the writer: personalisingfine-tuning Continuing to train an already-trained model for a short while on new data: here, your own lines. Changing it: It adapts a general model to your hand from very little data. Too long, or with too high a learning rate, and it memorises your lines or forgets general writing. Here: 40 varied copies of each of your 25 lines, plus 2,000 synthetic lines replayed. It took v2 from 43.6% to 13.1% errors..

Each half is a list of values: how many words, which GPU, how many epochs, how big a step. Every value here was set one of three ways, and the episode says which:

• Measured: a small experiment or a count decided it before the big run.
• Searched: a tool tried many values on held-out lines and kept the best.
• Kept: the model's size, its training settings and the decoder are v2's, so the change can be read against v2.

Personal v3.1-stable, test lines

6.8%

from 10.3% (v2.1-personal)

Base v3, before personalising

27.2%

from 43.3% (base v2)

Personalising trials searched

120

$1.36 on a rented GPU

All of iteration 3

≈ $17

base $15.06, personalising $1.99

2 Five changes to what the base reads

v3's synthetic ink writes more like the writer and covers what the writer writes. Each change answers a measurement.

Episode 4 found that the old synthetic ink didn't write like a hand: it lifted the pen too often and drew jagged paths. A closer look at what the writer actually writes found more gaps. v3 fixes five things at once:

Before paying for the GPU, a sample of 3,000 v3 lines was checked against the writer's ink. v2's lines had raised five warnings in that check, on pen lifts, points and time per letter, a rare capital and characters never seen. v3's raised none.

3 How many words to ask InkSight for: 3,000, not 6,000

The first guess counted words. The model reads letters, and by that measure 3,000 words already show almost everything.

The first plan was 6,000 words, picked by eye from how much of the texts' words a bank covers. But the recogniser reads letters, not words. A word missing from the bank is still written, in font ink.

What InkSight adds is how a hand forms and joins letters. So the measure that matters is how many letters and letter pairs the model sees in InkSight style, and that saturates early:

And each extra word costs more than the one before. InkSight writes one point at a time, a batch of words finishes only when its longest word does, and rarer words are longer.

Images derendered

5,774

3,000 words in 2 fonts each

Faithful

97%

5,621 kept; 2,935 of 3,000 words

Cost

$13.29

4 h 31 min; $11 planned

A word counts as faithful when its strokes trace at least 80% of the letters' centre lines and at least 80% of the strokes stay on the ink. Midway through, the bill looked headed for $21 to $26; later batches held shorter words, and it ended at $13.29.

4 Which GPU: the cheapest epoch, not the fastest card

A 16-minute benchmark priced one epoch on six GPUs. The model is too small to fill any of them, so the mid-range card won.

Left to itself, the cloud provider (Modal) takes whichever of three GPUsGPU A graphics card: thousands of small cores doing arithmetic in parallel, which is what training mostly is. Changing it: Training is many times faster than on a CPU; rented by the second, so time is money. Here: A Modal T4 at about $0.59 an hour: v2's base cost $1.42. is free first. So before training, the same model, batch size and number format ran on six of them, and each was priced per epoch, including the processor cores and memory the provider also bills.

The provider's default tries an L4 first: 26% dearer and 53% slower per epoch than the A10G. The batch size and number format stayed as the recipe says, since they change what the model learns, not just how fast.

5 The training run: 30 minutes, 87 cents

12 epochs finished on the A10G in half an hour, ended by the plan rather than a clock.

The 300,000 lines were made on 60 processor machines at once, checked against the writer's ink, and only then was the GPU started. Episode 3 had found a clock that cut runs short, so the time limit was raised from 150 to 220 minutes: the 12 epochsepoch One full pass over every training line. Changing it: More epochs keep lowering the training error until it levels off. Past that point they cost time for nothing, or the model starts memorising the training lines. The best epoch is the one that scores best on held-out lines, not the last one. Here: v1's base ran 29 epochs of about 97,000 lines (about 1.5 minutes each on a T4); v2's ran 12 epochs of 290,000 lines (about 12 minutes each). Personalising stops after 6 epochs or 10 minutes. end training, not the clock.

Training

30 min

12 of 12 epochs, about 2,000 lines/s

GPU machine

$0.87

$1.39 with the 60 machines making lines

Data check

0 errors

2 small warnings, run kept going

6 Win: the base reads the writer 16 points better

Before seeing any of the writer's lines, base v3 gets 27.2% of their letters wrong, against 43.3% for base v2.

Both bases were read on the same 24 test lines (600 characters) with the same decoder. Neither has seen any of the writer's ink.

In letters: 260 wrong of 600 for base v2, 163 for base v3. Paired line by line, the step is −16.2 points (−21.8 to −10.6): base v3 is better on 20 lines and worse on 3. The gain shows with and without the language model, so it is the network reading better, not the decoder guessing better.

Base v3 alone now reads the writer about as well as v1 did after personalising (24.5%). Its worst lines are still symbols and units: "60 km/h × 1,5 h = 90 km", "x² − 9 = (x − 3)(x + 3)". Speed is unchanged: the same 3 × 192 network, and 1.4 ms is about the size of this laptop's run-to-run noise.

7 Tried and not kept: a word language model

A word-level language model was built and measured. It gained 3 or 4 letters of 600, inside the noise, so it stays out.

The decoder already uses a language modellanguage model A model of which letters tend to follow which, learned from plain text. Changing it: It fixes readings the network finds ambiguous. Weighted too heavily, it overrules what you actually wrote, such as rare words and names. Here: A character model (each letter given the 4 before it) from 120,000 Tatoeba sentences per language, weighted 0.5. over letters (which letters tend to follow which) and a bonus for the writer's own words. A model over whole words would prefer "meeting" to "meeling".

Two ready-made tools were also set aside: KenLM and pyctcdecode can't run in the browser, where the model reads. Instead, a brute-force check scores every possible reading of small made-up inputs and makes sure the beam search finds the best one, in 130 cases. It found readings that could never be produced still taking up room; removing them saved 1 to 2% of decoding time.

8 Personalising: where to start, and which lines to learn from

Start from base v3, learn from both recording rounds at once, and keep a fifth of the lines aside to choose settings with.

Three models could have been the starting point:

The last point matters most. To choose settings honestly, some of the writer's lines have to be ones the model hasn't learned from.

The writer has recorded two rounds: 25 training lines and 15 test lines, then 60 and 24. Personal v3 learns from both at once: 85 training lines, one copy of each (one round had been uploaded twice).

Lines it learns from

68

the fit

Dev lines

17

441 characters, to choose settings

Test lines

24

read once, at the very end

9 Searching the plan: 60 trials on a rented GPU

Six personalising settings searched together, each trial about a minute, all for 52 cents. Freezing layers always hurt.

Personalising has six settings, and they interact: a bigger learning ratelearning rate How big a step each weight takes after every batch. Changing it: Too small and training creeps, wasting GPU time. Too big and it overshoots: the loss bounces or blows up. It's usually the first setting to tune. Here: 0.002 for the base models; 0.001 for personalising, which starts near a good solution and shouldn't be knocked away from it. may want fewer epochs. So Optuna, an open-source search library, tried them together:

On this laptop a trial would take several minutes: personalising reads about 12.5 lines a second there, and can't use more than two processor cores. On an A10G the same model trains at about 1,500 lines a second, about 100 times faster. So the search ran on Modal, as the base had.

The search kept three rules:

• v3's own plan is trial 0. It read the dev lines at 13.4%, so every result is measured against it.
• Weak trials stop early: after each epoch, a trial worse than the middle of the others at that point is dropped.
• No clock cuts a trial short: each one may run an hour and needs about a minute. Not one was cut, where the v2.1 models had run 2 of their 6 epochs.

Trials

60

29 finished, 31 stopped early

On the A10G

19 min

$0.52

Best (trial 53)

10.9%

from 13.4%: 11 letters fewer

The idea behind freezing was that low layers read pen movement, which every hand shares, and only the top layers need to learn a new hand. The data says otherwise: this writer's hand needs the low layers too.

10 A best at the edge of the map: search wider

The first winner sat at the limit of two ranges, so a second search widened them. Its best landed in the middle.

The first search's best used 2 epochs, the fewest it allowed, and a learning rate 95% of the way to its ceiling. A best at the edge of a range says the real best may lie past it.

So a second search ran with room on both sides: down to 1 epoch, and learning rates up to 0.01. It was a study of its own, because the search method learns only from trials that shared the same ranges.

The second best was not past the old edges but in the middle of the new ranges: 8 epochs at a moderate rate, with its runners-up close by (11.3% to 12.0%). Two different recipes, one short and hot, one long and gentle, 2.5 and 3 points better than v3's plan.

The ranking changed between the searches. Importance belongs to the ranges searched as much as to the setting: widen the learning rate's range, and it has more room to make a difference.

The two bests differ by 2 letters of 441, inside the noise. A rule was written before measuring: within 3 letters, keep the plan that forgetscatastrophic forgetting When a model trained on new data loses what it knew before. Changing it: A personalised model could read your hand and nothing else. Mixing old data back in (replay) and a small learning rate prevent it. Here: Checked every personalisation: error on synthetic lines stays flat. less of what the base knew.

Measured on 2,000 synthetic lines neither had seen, base v3 read 1.4%, the short plan 3.6% and the long plan 4.2%. The short plan won the tie, and became the recipe for personal v3.1.

11 Failure: the 10.9% that didn't come back

Made again from its recipe, the short plan scored 15.6%. One run per plan can't rank plans.

Personal v3.1 was made from its recipe: same plan, same lines, same dev lines. It scored 15.6% on the dev lines, not 10.9%: 21 letters worse, with nothing changed.

Training on a GPU isn't exactly repeatable. The loss this model learns from adds up many small numbers in parallel, and the order changes from run to run. Rounding differs in the last digit, and training amplifies it from the first epoch. A short plan at a high learning rate amplifies it most.

12 Five seeds each: judge a plan by its mean and its spread

Over five runs each, the long plan wins: 11.4% on average, against 12.8% for v3's plan and 13.9% for the short one.

A seed now picks the varied copies and seeds training, so a plan can be run several times on purpose. The two finalists and v3's own plan, as the reference, were each trained with four more seeds, one call at a time.

The long plan is ahead of v3's plan by 1.4 points on average, and of the short plan by 2.6. Measured against the spread, those gaps are 2.7 and 2.9 times their own uncertainty: unlikely to be luck, though five runs is still a small sample.

The long plan, with seed 0, became the release v3.1-stable (every run of it, epoch by epoch, is in episode 8). Made from its recipe, it read the dev lines at 10.43%, exactly its earlier run: this time the run repeated.

Personalising calls to Modal

20

searches, finalists, repeats, releases

All of them

$1.99

one model: about 3 cents

On this laptop

12 h +

for one thorough search

13 Win: 6.8% on the test lines

Personal v3.1-stable gets 41 of 600 test letters wrong, a third fewer than v2.1-personal, from fewer of the writer's lines, and inside the speed budget.

The last steps kept their order: tune the decoder on the dev lines (section 14), then read the 24 test lines once. They are the same lines v2.1-personal was judged on.

Character error rate

6.8%

95% interval 4.6% to 9.2%

Word error rate

23.6%

from 32.9% (v2.1-personal)

95% of lines read within

45.1 ms

budget 50 ms: passed

Paired line by line against v2.1-personal, the step is −3.5 points (−6.0 to −1.3). v3.1-stable is better on 13 lines, worse on 3, and the interval stays clear of zero.

Every language improves. Math reads better without the language model (8.2% against 9.4%): that model learned which letters follow which from ordinary text, so on a formula it pushes the reading toward words. A math-aware language model, or none for math lines, is a cheap thing to try next.

On all 39 test lines of both rounds (1,036 characters), v3.1-stable reads 8.6% (5.3% to 13.4%), with a word error rate of 26.4%. The first round's 15 English lines are the harder ones: English reads 9.5% with them, against 5.7% without.

The model is exported for the browser (8.6 MB, the same reading as the training code on 10 of 10 check lines) and is now the one the app reads with.

14 Not kept: decoder settings tuned on the dev lines

Tuning the decoder won 4 letters on the dev lines and lost 6 on the test lines, so the defaults stay.

As in episode 5, the decoder's settings were searched on the dev lines: 100 trials, keeping the fastest settings within half a point of the most accurate. The rule set beforehand: keep tuned settings only if they also win on the test lines.

The tuned settings gained 4 letters on the 441 dev characters and lost 6 on the 600 test characters. A gain of four letters is inside the noise of 17 lines, and settings chosen on those lines fit their quirks. The defaults stay, and they also stayed within the 50 ms budget on every timing run, where the tuned ones went over once.

15 What carries over

Seven habits for anyone choosing values for a small model on little data.