Base v4: the same lines, trained longer

1 The question

Can a base model read the writer at about 15% error before personalising? Base v3 is at 27.5%. First, a control: the same lines, trained longer.

The base modelbase model The general reader, trained on lots of synthetic handwriting before it ever sees yours. Changing it: A better base gives a better personal model: it starts lower and learns your hand faster. Alone it can still read your writing badly. Here: From v1 on, trained on a cloud GPU from a recipe; v2's base alone gets 43.6% of your letters wrong, 13.1% once personalised. learns from synthetic inksynthetic data Training data written by code instead of collected from people: here, handwriting fonts turned into pen paths with timing. Changing it: It's unlimited and licence-clean, but if it doesn't look like real writing the model learns the wrong thing: the synthetic-to-real gap. Here: 300,000 lines for v2, from 134 fonts under open licences (OFL or Apache). alone, then is fitted to the writer. The better it reads them before that fitting, the less the fitting has to do. Base v3, from episode 7, gets about one letter in four wrong. The goal for the next bases is one in seven: 15% or less, with none of the writer's lines in training.

The obvious next step is to change the data. But v3's error on its own synthetic lines was still falling when its training stopped. So base v4 changes nothing but the training: v3, trained longer on exactly its own lines. Whatever later bases gain beyond v4 is then their data's doing, not extra training's.

2 Where v3 goes wrong: letter shapes

The writer loops and joins letters three times as often as the synthetic ink does, and that is where the mistakes are. Preprocessing isn't the cause.

Base v3 has seen none of the writer's lines, so all 124 of them could test it: 27.9% wrong with the beam searchbeam search Decoding that keeps the best few partial readings at every step instead of committing to one letter at a time. Changing it: A wider beam gives the language model room to fix ambiguous letters, but costs time: the beam search is about 40% of v2's time per line. Here: Width 8, combining the network's scores with the language model and your lexicon., close to its official figure. Two thirds of the errors are lowercase letters, inside words; spaces are 90% right.

The most common mistakes are exactly those shapes:

t read as l

33

times, greedy

n read as u

25

times, greedy

h read as L

22

times, greedy

l read as b

16

times, greedy

A looped l reads as b, a looped h as L, a cursive n as u. Speed, size, slant and pen lifts already match the writer's. Preprocessing changes tried on the writer's ink (straightening the slant, smoothing, shortening pauses, reordering strokes, dropping tiny ones) moved the error by under a point or made it worse. Narrower ink read up to 2.6 points better: small, next to 27.

3 Measure first: dev lines choose, test lines judge

38 lines choose the epoch and decide whether v4 replaces v3; 84 other lines are read once, at the end. The rule was written before training.

A base model learns none of the writer's lines, so they can all be used to measure it, split by when they were recorded:

• Dev lines: the first recording round, 38 lines (919 characters). They choose which epoch to keep and whether v4 replaces v3.
• Test lines: the second round, 84 lines (2,250 characters), read once, for the model chosen.

On these test lines base v3 reads 27.5%, near the 27.9% on all lines. Two lines written across two rows are left out of both: the model reads one row.

4 The control run: 31 minutes, 89 cents

v3's own 300,000 lines, from v3's weights, the step size easing to zero. Every epoch came back to the laptop to be read.

The same 300,000 synthetic lines v3 learned from, still in the cloud's storage, so no new lines were made. Training starts from v3's weights, with the learning ratelearning rate How big a step each weight takes after every batch. Changing it: Too small and training creeps, wasting GPU time. Too big and it overshoots: the loss bounces or blows up. It's usually the first setting to tune. Here: 0.002 for the base models; 0.001 for personalising, which starts near a good solution and shouldn't be knocked away from it. starting at half v3's and easing to zero along a cosine curve, for 12 epochsepoch One full pass over every training line. Changing it: More epochs keep lowering the training error until it levels off. Past that point they cost time for nothing, or the model starts memorising the training lines. The best epoch is the one that scores best on held-out lines, not the last one. Here: v1's base ran 29 epochs of about 97,000 lines (about 1.5 minutes each on a T4); v2's ran 12 epochs of 290,000 lines (about 12 minutes each). Personalising stops after 6 epochs or 10 minutes..

The writer's ink never went to the cloud. Every epoch's weights came back, and the laptop read the dev lines with each, then kept the epoch that read them best.

Training

31 min

12 of 12 epochs on an A10G

GPU machine

$0.89

no line skipped

The day's v4 work

$1.90

with the lost launches (section 7)

5 Epoch by epoch: two curves part ways

The synthetic error kept falling, from 1.4% to 0.58%. The writer's dev error was best at epoch 2, then flat at 31–32%.

Chosen by its own synthetic lines, the run would have kept epoch 11. The model goes on learning v3's lines better, and from epoch 3 on that no longer reaches the writer's hand: the synthetic-to-real gapsynthetic-to-real gap The difference between how well a model reads the synthetic data it trained on and how well it reads real writing. Changing it: A model can score almost perfectly on synthetic lines and still read real ink badly, so only a test on real writing counts. Closing it means making the synthetic data look more real. Here: v1's base: 0.1% errors on synthetic lines, 58.3% on yours., measured epoch by epoch.

6 Win, but small: 2.4 points on the test lines

Base v4 reads the 84 test lines at 25.2%, from 27.5%: a real step, and far from 15%.

In letters: 619 wrong of 2,250 for base v3, 566 for base v4. Paired line by line, the step is −2.4 points (−4.1 to −0.6). It shows with and without the language model, so it is the network reading better. On the 24 test lines of earlier episodes, all from the second round, base v4 reads 24.2% against 27.2%.

What it says: more training alone is worth about 2 points on the writer's hand, and then it has run out. It took the edge off one shape: on all the writer's lines, t read as l fell from 33 times to 20. But n still reads as u, h as L, m as n, and lowercase letters are still most of the errors. What's left is what v3's lines don't show, looped and joined letters. The next base changes the data, and has to beat v4, not v3. Since the dev error peaked early, 4 to 6 epochs from v4 should be enough for it.

7 Failure: two launches lost before any GPU

Reusing v3's lines made every chunk finish at once, and 50 MB of statistics coming back together dropped the connection. Twice.

Before training, each of the 60 chunks of lines is measured in the cloud and its statistics are sent back, to check the data against the writer's ink. When v3 made its chunks, they finished at different times. Reused, all 60 finished together, about 0.9 MB each: roughly 50 MB at once.

Twice the connection dropped about 14 minutes in, before any GPU had started: about $0.20 together. The fix: each chunk writes its statistics to the cloud's storage, and the laptop reads them one by one. Measuring the 60 chunks then took 38 seconds, and the next launch ran through.

8 What carries over

Run the cheap control before the expensive change, and watch the real score, not the synthetic one.