Bigger, messier synthetic ink
Iteration 2: five changes at once. It got better, and we can't say why.
TL;DR
Win and failureThe question Do a bigger model and more varied, more realistic synthetic ink close the gap to a real hand?
The first model read fonts well and real handwriting badly, so we made the fake handwriting messier and the model bigger. Both versions got clearly better. But we changed five things at once, so we can't tell which one did it, and the bigger model got too slow.
- The base model went from about 1 letter in 2 wrong (51.7%) to 43.3%: better, but the gap to a real hand is still wide.
- Personalised on the writer's lines, it went from 24.5% to 16.7%, close to the 15% target.
- Both gains hold up line by line: they are not luck of the test set.
- Five changes went in together (fonts, words, messier lines, model size, more lines), so none can take the credit.
- Twice the weights made reading slower: the personalised model misses the 50 ms per line budget.
Change one thing at a time. A bundle of improvements can be real and still teach you nothing about which part to keep.
Why read on The surprising third most common English word in our training text, and why a win that teaches you nothing is half a failure.
1 The question
If the fake handwriting looks more like a real hand, does the model read the real hand better?
Iteration 1 ended with a clear diagnosis: the base read fonts almost perfectly and real ink badly. So the synthetic data had to look more like a hand.
Iteration 2 asked whether richer, messier synthetic ink and a bigger model to learn it would close that gap.
2 Five changes at once
More fonts, real words, untidy lines, a bigger model and three times the data, all in one go.
| Setting | v1 | v2 |
|---|---|---|
| Fonts | 51 | 134: school models, everyday print, casual cursive |
| Words | the project's own note words | 5,000 most used English and French words, in 75% of lines |
| Untidy lines | none | half: slant, scale, rotation, wobble, jitter, speed, dropped points |
| Model | 3 × 128, 973,490 weights | 3 × 192, 2,148,274 weights |
| Synthetic lines | 96,667 | 290,000 |
| Training | 45 min · $0.45 | 2.4 h · $1.42 |
The untidy half is augmentation Making extra training examples by changing existing ones in ways that keep the answer the same: slant, size, speed, jitter, dropped points. Changing it: It teaches the model which differences don't matter, so it generalises from few examples. Too extreme, and the examples stop looking like real writing. Here: Each of the writer's lines becomes 40 variations; half of v2's synthetic lines are augmented too.: each line is bent in ways a hand bends. That stops the model relying on font-perfect shapes.
The words follow Zipf's law The n-th most common word is used about 1/n as often as the most common one. Changing it: Drawing words by raw count would fill lines with "the" and "de"; drawing by the square root of the count keeps rare words in play. Here: Both English and French Tatoeba follow it closely.: a few words are everywhere and most are rare. So they are drawn by the square root of their frequency, which keeps common words common without drowning out the rest.
3 Better as a base and personalised
Both models improved for real, but the gap to a real hand is narrower, not closed.
| Row | Value | Interval |
|---|---|---|
| v1 base | 52% | 46% to 57% |
| v2 base | 43% | 38% to 49% |
| v1 personalised | 25% | 20% to 30% |
| v2 personalised | 17% | 13% to 21% |
The base still gets 43.3% of real letters wrong: the gap is narrower, not closed.
Personalised, v2 reached 16.7%, near the 15% target. But its range of plausible scores (12.8%–21.1%) straddles the target, so it hasn't cleared it yet.
4 Failure: too slow
The bigger model reads better but too slowly: it misses the time budget per line.
Twice the weights means more work per line. Reading is the network plus a beam search Decoding that keeps the best few partial readings at every step instead of committing to one letter at a time. Changing it: A wider beam gives the language model room to fix ambiguous letters, but costs time: the beam search is about 40% of v2's time per line. Here: Width 8, combining the network's scores with the language model and your lexicon. (weighing several candidate readings at once) guided by a language model A model of which letters tend to follow which, learned from plain text. Changing it: It fixes readings the network finds ambiguous. Weighted too heavily, it overrules what you actually wrote, such as rare words and names. Here: A character model (each letter given the 4 before it) from 120,000 Tatoeba sentences per language, weighted 0.5. (which knows what text usually looks like).
On the laptop, v2 personalised reads 95% of lines within 53 ms: over the 50 ms budget.
| Model | Median | p95 |
|---|---|---|
| v1 personalised | 32 ms | 46 ms |
| v2 personalised | 37 ms | 53 ms |
5 Failure: we can't say why it worked
Five changes at once means the win can't be credited to any of them.
Five things changed between v1 and v2. Any one of them could account for the whole gain, or two could pull in opposite directions.
Finding out would take an ablation An experiment that changes one thing, keeps the rest, and measures. Changing it: It's how you learn which change helped. Changing several at once gives a result you can't explain. Here: v2 changed five things at once; v3 changes one at a time.: retraining with each change undone in turn. That is five GPU runs for a question we should have asked before the first one.
- Messier synthetic ink and a bigger model help, as a bundle.
- Speed grows with size: a 3 × 192 model misses the budget.
- One change per experiment, from a named control.
- A bigger, more varied test set (iteration 3).
- The prediction written down before each run.
6 What carries over
Change one thing at a time, measure speed with accuracy, and read your data.
- Changing several things at once saves one run now and costs one run per change later, when you need to know which one mattered.
- A bigger model buys accuracy and spends latency. Measure both on every run, not only at the end.
- Read what your text sources actually say. Every corpus has quirks, and the model learns them.
- Sampling words by the square root of their frequency keeps common words common without drowning out the rare ones.