Base v5: five new kinds of fake ink, and why v4 stays

1 The question

Base v4 reads the writer at 25.2%, and more training on the same lines has run out. What should change in the fake ink it learns from?

The base modelbase model The general reader, trained on lots of synthetic handwriting before it ever sees yours. Changing it: A better base gives a better personal model: it starts lower and learns your hand faster. Alone it can still read your writing badly. Here: From v1 on, trained on a cloud GPU from a recipe; v2's base alone gets 43.6% of your letters wrong, 13.1% once personalised. learns only from synthetic inksynthetic data Training data written by code instead of collected from people: here, handwriting fonts turned into pen paths with timing. Changing it: It's unlimited and licence-clean, but if it doesn't look like real writing the model learns the wrong thing: the synthetic-to-real gap. Here: 300,000 lines for v2, from 134 fonts under open licences (OFL or Apache).: handwriting faked from fonts and from another model's strokes. The goal is a base that reads the writer at 15% or less before it sees any of their lines. Episode 9 showed that training longer on the same lines stops paying after a couple of epochs. What's left has to come from the data.

Where v4's errors are, on all 124 of the writer's lines:

Lowercase letters

14.4 pts

470 errors

Punctuation, digits, accents, math

6.3 pts

207 errors

Capitals and spaces

2.7 pts

88 errors

Lowercase letter shapes are 14.4 of the 23.4 points: n read as u, h as L, t as l, m as n. The writer's pen crosses its own line 0.58 times per letter, in loops and joins; v4's synthetic ink 0.19 to 0.27 times. So the first idea was the loops.

2 Where the loops go

Fonts draw the loops. The tracer that turns them into pen strokes cuts them off, and smoothing closes half of what's left: 0.24 loops per letter against the writer's 0.58.

A font is a picture of a letter, not a pen path, so every font line is traced into strokes first. The fonts were fine: a rendered "lebthfkl" in a French school font has 9 closed rings, at any resolution. The loss is in the steps after.

• Tracing. A font draws a looped l, b or h as a ring on a stem, the up- and down-strokes overlapping. The old tracer visits each pixel once and stops when it meets ink it has drawn, so the ring is cut off and the letter splits in two: no loops at all.
• Smoothing. Font paths are 41% longer per letter than the writer's (1.23 against 0.87 line heights). Shortening them that much closes loops, however it is done.

A new tracer that keeps the rings ends at 0.38 loops per letter, closer to the writer's 0.58. Whether that helps is a question for the writer's lines, not for this chart.

3 Five new sources, the same words five ways

A loop-keeping tracer, single-line fonts, real sentences, lognormal pen movement, and a wider spread of width and speed. All built on the laptop, for free.

Here are the words "le bel hôtel" from each kind of ink: the old one and the new ones. Each pen-down stroke takes the next colour, so a change of colour is a pen lift. All of it is synthetic: the writer's own ink isn't shown.

• The loop-keeping tracer reads the font's skeleton as a map of junctions: the pen goes straight through a crossing, and instead of lifting it goes back down a stem, each pass a little to its own side. A looped l comes out as one stroke with a loop.
• Single-line fonts are drawn as pen paths in the first place, for plotters and engravers: eight looped script faces, among them Pepita, a French school hand, plus a casual hand and Hershey's script. They have accents, but no œ and no math.
• Lognormal movements write each stroke again, as a hand does: a run of overlapping movements between slightly jittered targets, each speeding up and slowing down in a bell. Corners round off, and the pen loops where it turns back.

Real sentences replace half the English and French lines. Until now a line was a string of frequent words drawn at random, so no two words belonged together. Now half are real sentences from a public collection, 12 to 50 characters, none of them one of the writer's lines.

Width and speed: each line 0.7 to 1.15 times as wide, and the time per letter spread wider. The writer's ink read better narrower and faster in episode 9's checks.

4 The pilot: one change per arm, two controls

Each arm starts from v4 and trains 2 epochs on 80,000 fresh lines with one change. Two arms change nothing, to show how far apart chance alone puts them.

A pilot costs a few minutes of GPU per arm: start from v4's weights, train 2 epochsepoch One full pass over every training line. Changing it: More epochs keep lowering the training error until it levels off. Past that point they cost time for nothing, or the model starts memorising the training lines. The best epoch is the one that scores best on held-out lines, not the last one. Here: v1's base ran 29 epochs of about 97,000 lines (about 1.5 minutes each on a T4); v2's ran 12 epochs of 290,000 lines (about 12 minutes each). Personalising stops after 6 epochs or 10 minutes. on 80,000 fresh lines, and read the writer's 38 dev lines. Every arm is v4's recipe with one change, as in an ablationablation An experiment that changes one thing, keeps the rest, and measures. Changing it: It's how you learn which change helped. Changing several at once gives a result you can't explain. Here: v2 changed five things at once; v3 changes one at a time.. A last arm has all five changes together.

The control is v4's recipe on fresh lines. A second control, the same recipe with another seed, shows the noise: whatever separates the two controls is chance.

First, a check that each arm's lines are new to v4 and still legible: 300 lines of each, read by v4. It reads its own kind of lines at 0.8% and the new ones at 1.3% to 6.9%.

5 Pilot 1: one worse, none clearly better

Single-line fonts cost +2.9 points (+0.8 to +5.0). Read greedily, lognormal movement helped and real sentences nearly did; with the beam search, nothing cleared the noise.

Single-line fonts made v4 worse: +2.9 points (+0.8 to +5.0), so they stay out. Read greedilygreedy decoding Reading the model's output by taking the top guess at every point, then applying CTC's rules. Changing it: Fast and simple, but it can't use spelling knowledge, so it produces things like "1%e" for "the". Here: Greedy CER is the network on its own: 18.4% for v2 personalised, against 13.1% with the beam search., lognormal movement was clearly better, −2.5 points (−4.0 to −0.8), and real sentences nearly, −1.5 points (−3.3 to +0.2). With the beam search neither shows: the language modellanguage model A model of which letters tend to follow which, learned from plain text. Changing it: It fixes readings the network finds ambiguous. Weighted too heavily, it overrules what you actually wrote, such as rare words and names. Here: A character model (each letter given the 4 before it) from 120,000 Tatoeba sentences per language, weighted 0.5. and the word listlexicon A list of known words that get a bonus when a reading completes one. Changing it: It pulls readings towards words you actually use; too big a bonus and it forces words that weren't written. Here: The words in your training lines, with a bonus of 1.5 per finished word. take most of that back.

6 Failure: the new tracer wrote its loops in the wrong order

A test found it: on a looped l the pen drew the exit tail first, came back, and went round the loop. Pilot 1's tracer arm trained on that.

The tracer's tests draw single letters and check the pen's path. On a looped l with a stroke in and a stroke out, the pen drew the tail out first, came back, went up and round the loop, and ended at its top. One stroke, as intended, but in an order no hand writes.

The fix: of two branches the pen has to come back from, it takes the one reaching less far right first, leaving the way on to the next letter for last. The l now goes in, up beside the stem, round, down the other side and out. Pilot 1's tracer arm, +1.4 points (−0.6 to +3.4), isn't a fair test of the idea, so it ran again.

7 Pilot 2: the two together, and v5's recipe

Lognormal movement and real sentences together: −1.3 points (−2.9 to +0.2) with the beam search. The fixed tracer: no better. Those two became base v5's recipe.

The same set-up, after the fix: the fixed tracer alone, the two sources pilot 1 favoured together, and the two with the tracer.

Lognormal movement and real sentences together are the only arm better than the control in both readings: −1.3 points (−2.9 to +0.2) with the beam search, −1.6 points (−3.1 to −0.1) greedy. The fixed tracer is no better (+1.6 points (−0.2 to +3.6) greedy), and the two with the tracer lose the gain: +0.1 points (−2.3 to +2.6).

So base v5 got those two and nothing else new. By the strict rule they weren't proven: their gain with the beam search is under the 1.1 points between two controls. They went in because they never hurt and agreed across both pilots, and because v5 has a rule of its own: it replaces v4 only if it reads the dev lines better with the beam search, interval below zero.

8 Pilot 3: looking again, the smoothing closes the e

Drawing the ink for this page showed the smoothing turning every e into an open hook. Switched off, the model alone reads the writer 2.5 points better; with the two, 4.0.

The statistics said the smoothing closes loops. The drawing in section 3 showed which: in its second line, the one v4 learned from, every e is an open hook and the ô is flattened. Small loops close along with the tall ones, and e is the most common letter in French and English.

So pilot 3 switched the smoothing off, while v5 trained: alone, with the loop-keeping tracer, and with real sentences and lognormal movement. Lines then keep a font's own length. The data check gave one warning more than usual, the expected one: their points per letter no longer match the writer's.

Read greedily, no smoothing helps as much as lognormal movement did: −2.5 points (−4.2 to −0.9). With real sentences and lognormal movement as well, −4.0 points (−6.4 to −1.8): 27.1% greedy, the best of all 16 arms read that way. With the beam search, every arm is 1.2 to 1.5 points better than the control, and every interval crosses zero. The tracer, even with its loops no longer smoothed away, helps neither way.

9 All the pilots, against four controls pooled

Read greedily, three changes help and add up: −4.1 points (−5.9 to −2.3) together. With the beam search, the way the app reads, none is clearly better.

The three pilots have 4 controls between them: v4's recipe, four seeds, all read on the same 38 lines. Averaged line by line, the four make a steadier control than any one: 23.8% with the beam search, 31.1% greedy. Every main arm, against it:

The model alone reads the writer better with each of three changes, and better still with all three. With the beam search, the helpful arms are 0.5 to 1.3 points better and every interval crosses zero. Single-line fonts are the one clear result there, and it is worse: +2.3 points (+0.4 to +4.1).

10 Failure: base v5 doesn't replace v4

By the rule set before training, v4 stays: v5 reads the dev lines with the beam search at 24.2% against v4's 23.0%, +1.2 points (−0.8 to +3.3).

Base v5 is v4 trained further, 5 epochs on 300,000 fresh lines, with the learning ratelearning rate How big a step each weight takes after every batch. Changing it: Too small and training creeps, wasting GPU time. Too big and it overshoots: the loss bounces or blows up. It's usually the first setting to tune. Here: 0.002 for the base models; 0.001 for personalising, which starts near a good solution and shouldn't be knocked away from it. easing to zero as in the pilots. Half the English and French lines are real sentences, and half of all lines are written again as lognormal movements. Everything else is v4's. The epoch kept is the one that reads the dev lines best, greedy: epoch 3, at 28.3%. The synthetic lines would have kept epoch 5.

The synthetic error fell every epoch; the writer's stayed within a point of v4 throughout, as in episode 9.

So the base stays on v4. As the rule says, v5's test lines were not read, and nothing was exported; v5 stays as a run.

Read greedily, v5 is a little better in English, French and mixed lines (25.9%, 29.1% and 27.9%, against 26.2%, 30.4% and 29.5%). With the beam search that is gone. What real sentences and lognormal movement teach the model alone, the language model already gives the beam search. The pilots showed the same at a small dose: clear greedy gains, beam gains of about a point.

Pilot 1

$1.65

8 arms

Pilot 2

$0.92

4 arms

Pilot 3

$0.74

4 arms

Base v5

$1.02

300,000 lines, 5 epochs

All of v5's work

$4.33

Modal's bill, per app

v5's training: 13 min on an A10G, $0.37 for the GPU machine by its own estimate; the pilots' GPU, by theirs: $0.33, $0.19 and $0.23.

11 What worked, what didn't

Three changes help the model alone; none helps the app's beam-search reading beyond noise. Font-made ink has reached its limit there.

Font-made synthetic ink has reached its limit for the beam search: 16 pilot arms, 4 of them controls, and a full run, and nothing beats v4 beyond noise. The best greedy arm, no smoothing with sentences and lognormal movement, is the one candidate left if another synthetic base is tried. But v5 just showed that greedy gains at a pilot's dose don't become beam gains at full scale, so a point at most.

What would move the score is new letter shapes, not more of the fonts': real handwriting from many writers. That is set aside for now, but it is where the evidence points, and it would be piloted the same way, against the same controls.

12 What carries over

Pilot new data against a control on the real thing, with a second control for the noise, and score it the way the product reads. And look at what you train on.