More real lines
Iteration 3: a real test set, the first model under 15%, a confound we caught, a clock that cut every run short, and a speed gate still shut.
TL;DR
Win and failureThe question With 2.4 times more of the writer's lines, in four kinds of writing, how much better does each model get? And does it matter where personalising starts?
We collected more of the writer's handwriting and built a proper test to grade against. Every model got better, and for the first time one cleared our accuracy target with room to spare. Then we found that our best-looking win had three explanations, that a timer had quietly stopped every run early, and that the best model was still a touch too slow.
- The best model now gets about 1 letter in 10 wrong (10.3%), and even its worst case stays under the 15% target.
- The new data: 84 real lines in English, French, mixed and math; 60 to learn from, 24 kept back for testing.
- Starting from an already-personalised model looked 2.7 points better, but it also had more data and more time, so we can't credit the starting point yet.
- A 10-minute timer let personalising finish only 2 of its 6 planned passes, while it was still improving.
- Speed missed: 95% of lines read within 52 ms against a 50 ms budget.
More, more varied real data beats almost everything else. But read every win for confounds, and check that training actually finished.
Why read on How a real, statistically solid win turned out to be three changes wearing one name, and the cheap trick that might buy back the missing milliseconds.
1 The question
Do more of the writer's own lines help, and does it matter which model we start personalising from?
Iteration 2 left two problems. The test set was too small to prove anything near the target, and every comparison mixed several changes.
So iteration 3 recorded more of the writer's lines and rebuilt the test set. Then it asked two questions, each with one change only: how much do more lines help, and does it matter which model personalising starts from?
2 A test set that can say something
84 new lines in four kinds of writing: 60 to learn from, 24 held back that no model ever trains on.
A guided, step-by-step recorder collected 84 lines. Each point also carries the pen's tilt, and each line the size of the pad.
Sixty are for training and 24 for testing, across English, French, mixed and math. No test line's text appears in any training set, old or new, so the test can't be memorised.
| Split | English | French | mixed | math | All |
|---|---|---|---|---|---|
| Train | 20 | 18 | 12 | 10 | 60 |
| Test | 7 | 7 | 4 | 6 | 24 |
3 Win: the first model under 15%
More real lines helped every model, and one now gets about 1 letter in 10 wrong: inside the target even at its worst.
| Row | Value | Interval |
|---|---|---|
| v2.1-personal | 10% | 7% to 14% |
| v2.1-base | 13% | 9% to 17% |
| v2 personalised | 17% | 13% to 21% |
| v1 personalised | 25% | 20% to 30% |
| v0.2 local base | 33% | 28% to 38% |
| v2 base | 43% | 38% to 49% |
| v0.1 local base | 50% | 47% to 54% |
| v1 base | 52% | 46% to 57% |
Compared with the same model family personalised on the 25 old lines, the gain is −6.3 points (−8.7 to −4.0). It was better on 20 lines and worse on 1.
The small laptop model gained too. v0.2 local base, the same 1 × 128 network retrained on the new lines, went from 50.0% to 32.8% (−17.2 points, −22.3 to −11.8).
Its time limit doubled as well, so that gain belongs to both changes until experiment E8 separates them.
4 Caught: a win that is really three
Starting from the already-personalised model scored better, but it also had more data and more time, so we can't say which one helped.
We personalised twice on the 60 new lines with identical settings, changing only where we started.
- v2.1-base starts from v2 base, which has only ever seen synthetic ink.
- v2.1-personal starts from v2 personalised, already fitted to the 25 old lines: a second round of fine-tuning Continuing to train an already-trained model for a short while on new data: here, your own lines. Changing it: It adapts a general model to your hand from very little data. Too long, or with too high a learning rate, and it memorises your lines or forgets general writing. Here: 40 varied copies of each of your 25 lines, plus 2,000 synthetic lines replayed. It took v2 from 43.6% to 13.1% errors. (extra training on the writer's own lines).
- v2.1-base
- 13.0%
- 9.2%–17.4%
- v2.1-personal
- 10.3%
- 7.2%–13.9%
- Paired difference
- −2.7
- −5.1 to −0.7 points
5 Failure: a clock that cut every run short
A 10-minute limit set for 25 lines stopped every run after 2 of its 6 passes, while it was still improving.
The plan was 6 passes over the data; the clock allowed 2. Personalising had a 10-minute limit, set back when there were only 25 lines.
Now there are 60 lines × 41 versions each (the line plus 40 varied copies), plus 2,000 synthetic lines replayed. One epoch One full pass over every training line. Changing it: More epochs keep lowering the training error until it levels off. Past that point they cost time for nothing, or the model starts memorising the training lines. The best epoch is the one that scores best on held-out lines, not the last one. Here: v1's base ran 29 epochs of about 97,000 lines (about 1.5 minutes each on a T4); v2's ran 12 epochs of 290,000 lines (about 12 minutes each). Personalising stops after 6 epochs or 10 minutes. (one pass over all of it) is about 4,460 lines and takes about 5 minutes on the laptop.
The error on the training lines was still falling when the clock stopped it.
6 Failure: accurate enough, not fast enough
The best model is just over the 50 ms budget, and a narrower search is the cheapest fix to try first.
| Model | Median | p95 |
|---|---|---|
| v0.2 local base | 26 ms | 40 ms |
| v2.1-base | 37 ms | 54 ms |
| v2.1-personal | 40 ms | 52 ms |
v2.1-personal reads a typical line in about 40 ms, but 95% of lines within 52 ms: just over the budget. Every 3 × 192 model sits there; the 1 × 128 local base reads at 40 ms.
The beam search Decoding that keeps the best few partial readings at every step instead of committing to one letter at a time. Changing it: A wider beam gives the language model room to fix ambiguous letters, but costs time: the beam search is about 40% of v2's time per line. Here: Width 8, combining the network's scores with the language model and your lexicon., which weighs several candidate readings before picking one, takes about 40% of the time per line. A narrower beam (E10) is the cheapest thing to try before a smaller model.
7 What it shows, and what it doesn't yet
More real data is a proven win; why the best model wins, and whether the trend continues, are still open.
- More, more varied lines helped every model.
- v2.1-personal is the most accurate model, and the first whose interval is under 15%.
- The small model gained 17.2 points from new lines and more time together.
- That the starting point is what helped (confounded; E5).
- That more lines keep helping at this rate: two points per family, from different rounds (E7).
- Run-to-run noise: one seed per run (E9).
- Browser speed: every timing here is Python on the laptop.
The small model trained 8 epochs on 9,667 synthetic lines and the writer's 60, and its curve was still falling too.
8 What carries over
Four checks for any project before trusting a win near its target.
- Before trusting gains near your target, rebuild the test set so it covers every kind of input you care about.
- A win from a comparison that differs in several ways is a hypothesis, not a result. Name what else changed, and plan the run that separates it.
- Time limits tuned for small data silently cut training short when the data grows. Check how many epochs ran, not just that the run finished.
- Accuracy and speed are separate gates. A model can pass one and fail the other, and the write-up should say which.