Base v6: real hands

1 The question

Five kinds of fake ink moved the app's reading by about a point, and base v5 didn't replace v4. Would real handwriting from other people move it more?

The base modelbase model The general reader, trained on lots of synthetic handwriting before it ever sees yours. Changing it: A better base gives a better personal model: it starts lower and learns your hand faster. Alone it can still read your writing badly. Here: From v1 on, trained on a cloud GPU from a recipe; v2's base alone gets 43.6% of your letters wrong, 13.1% once personalised. has learned only from synthetic inksynthetic data Training data written by code instead of collected from people: here, handwriting fonts turned into pen paths with timing. Changing it: It's unlimited and licence-clean, but if it doesn't look like real writing the model learns the wrong thing: the synthetic-to-real gap. Here: 300,000 lines for v2, from 134 fonts under open licences (OFL or Apache).: handwriting faked from fonts. The goal is a base that reads the writer at 15% or less before it sees any of their lines. Base v4 reads the writer's 38 dev lines at 23.0% with the beam searchbeam search Decoding that keeps the best few partial readings at every step instead of committing to one letter at a time. Changing it: A wider beam gives the language model room to fix ambiguous letters, but costs time: the beam search is about 40% of v2's time per line. Here: Width 8, combining the network's scores with the language model and your lexicon., as the app reads.

Episode 10 tried five new kinds of fake ink. The best of them moved the beam score about a point, inside the noise, and the base trained on them, v5, read the dev lines +1.2 points (−0.8 to +3.3) against v4. It wasn't adopted.

Two thirds of v4's errors are lowercase letter shapes, and fonts only draw so many. Real hands have shapes no font draws. So this round adds real handwriting, from other people.

2 Five other hands

BRUSH: 27,649 lines from 170 people, written with a stylus, a point every 10 ms. Most print; a few loop.

BRUSH is a public set of online handwriting: each line is the pen's path as it was written, with its timing, not a picture. 170 people wrote English sentences with a stylus; the pen's position was recorded every 10 ms. The model reads timing, so only the original lines were kept, not the copies that resample them and drop it.

3 Opening files that can run code

Every BRUSH line is a Python pickle, and opening a pickle can run any code it names. The reader allows numpy's arrays and refuses the rest.

The download is 567 MB, 89,068 files. Each line is stored as a pickle: Python's own way of saving objects. Opening one rebuilds whatever it describes, and it can describe a call to anything, such as running a shell command.

So the reader never opens a file the plain way: it allows only numpy's array and type constructors, and refuses any file that names anything else. Its tests are made of real attacks: files that do run code when opened plainly, which the reader refuses without running them. All 27,649 lines read, from 170 writers, and every character in them is one the model writes.

4 Failure, caught by a test: one damaged file would have stopped the reading

A file that opens into something other than a line raised an error nothing caught. Now any damaged file counts as one unreadable line, and the reading goes on.

A test pass over the new code wrote files the reader should refuse. One opened into something other than a line (nothing at all, say), and the reader raised an error the conversion didn't catch: it caught only the errors of a refused or broken file. One such file would have stopped the whole conversion of BRUSH.

Now the reader refuses anything that isn't a line, and any error a damaged file can raise counts that file as unreadable, and the reading goes on. No BRUSH file is like that, so all 27,649 lines read either way: the fix is for the next download.

5 Their ink against the writer's

BRUSH loops about as little as the fonts: 0.25 times per letter, the writer 0.58. 16 of 170 writers, about one in ten, loop nearly as much.

On average BRUSH looks like the fonts, not like the writer. And it is fast: 0.17 s per letter with the pen down, against the writer's 0.67 s, about four times faster.

Most people print. 16 of the 170 writers loop 0.55 times per letter or more, about one in ten; 13 as much as the writer or more. Over all their lines, the writers run from 0.12 loops per letter (the 10th percentile) through 0.27 to 0.50 (the 90th). The average hides the loopy few, so drawing writers evenly would mostly teach print.

6 How each line was prepared

Each choice comes from something an earlier round found: no smoothing, the writer's pace, a varied copy every draw, whole writers held out.

The varied copies are augmentationaugmentation Making extra training examples by changing existing ones in ways that keep the answer the same: slant, size, speed, jitter, dropped points. Changing it: It teaches the model which differences don't matter, so it generalises from few examples. Too extreme, and the examples stop looking like real writing. Here: Each of the writer's lines becomes 40 variations; half of v2's synthetic lines are augmented too., the same kind personalising uses, so 24,873 lines can be drawn hundreds of thousands of times without being memorisedoverfitting When a model learns its training examples themselves rather than the pattern behind them, so it does much better on lines it has seen than on new ones. Changing it: It makes training scores look better than reality. The cures are more or more varied data, augmentation, fewer epochs, and picking the epoch on held-out lines. Here: v2 personalised reads its own training lines with 4.5% errors and the test lines with 13.1%.. The two open questions, pace and which writers, became pilot arms.

17 whole writers, 2,776 lines, are never trained on. That leaves 24,873 lines from 153 writers to learn from, and a way to see whether the model reads hands it has never seen, not only the writer's.

7 Weighting towards the writer's loops

Drawing writers by how close they loop to the writer raises the loopy writers' share of lines from 14% to 40%. Every writer still appears.

Each writer's lines are drawn with a weight: highest for writers who loop about as much as the writer, never below 0.05, so the printers still appear, less often.

Whether that helps is for the pilot to say: one arm draws writers alike, one weighted, all else the same.

8 The pilot: real ink, four ways

From v4, 2 epochs, the same 80,000 fake lines; four arms add real ones. Every one reads the writer's dev lines 6.7 to 7.7 points better with the beam search.

As in episode 10: every arm starts from v4's weightsweights The numbers inside the model that training adjusts. A trained model is just these numbers plus the code that uses them. Changing it: More weights can learn more, but need more data and training time, make the file bigger and reading slower. They grow with the square of the layer width. Here: v0: 0.18 million (1 × 128). v1: 0.97 million (3 × 128). v2: 2.15 million (3 × 192), 8.6 MB in full precision., trains 2 epochsepoch One full pass over every training line. Changing it: More epochs keep lowering the training error until it levels off. Past that point they cost time for nothing, or the model starts memorising the training lines. The best epoch is the one that scores best on held-out lines, not the last one. Here: v1's base ran 29 epochs of about 97,000 lines (about 1.5 minutes each on a T4); v2's ran 12 epochs of 290,000 lines (about 12 minutes each). Personalising stops after 6 epochs or 10 minutes. on the same 80,000 synthetic lines (v3 and v4's own), and is read on the writer's 38 dev lines. The control adds nothing. The others add real lines:

• 25% real: 25% of each epoch real, the writer's pace, writers alike.
• 50% real: 50%, the writer's pace, writers alike.
• 50%, weighted by loops: 50%, the writer's pace, writers weighted towards the writer's loops.
• 50%, writers' own pace: 50%, each BRUSH writer's own pace, writers alike.

Every real arm is 6.7 to 7.7 points better with the beam search, every interval far below zero. With the language model, v4's 23.0% falls to 15.7% in the best arm, near the 15% goal, after two epochs at a small dose.

• More real ink helps a little: 16.6% with 25%, 16.3% with 50%.
• Weighting towards the writer's loops helps the writer's lines a little more (15.7%), and other hands a little less: 6.3% on the held-out writers against 5.6%. Expected: the writer's hand is the target.
• Pace barely matters with the beam search: each writer's own pace reads 15.9%, the writer's pace 16.3%; greedy, the other way round.

The real arms read their own synthetic lines a little worse than the control (0.38%): fonts are no longer all they see.

The pilot

$0.64

Modal's bill, 5 arms

Its GPU

$0.34

the runs' own estimates

All of v5's work

$4.33

three pilots and a run

Before it: nothing was running on Modal, and 525 leftovers of v5's pilots went from the storage volume. The pilot models are kept on the laptop; v3 and v4's synthetic lines stay, for reuse.

9 Fake ink moved it a point; real ink, seven

On the same 38 dev lines with the beam search: v5's fake-ink recipe −1.3 in its pilot, +1.2 at full scale. Two epochs of real lines: −6.7 to −7.7.

Three pilots of new fake ink, 16 arms, moved the beam score by about a point. Two epochs with real handwriting moved it seven, and the full run kept it. What the fonts taught, the language model already gave the beam search. Real hands bring letter shapes no font draws.

10 Other hands

Bases v3 to v5 read the 17 writers they never saw at 27 to 29%. A real arm, after 2 epochs, reads them at about 6%; base v6 at 2.9%, on an easy set.

A base that has seen only fonts reads a stranger's hand about as badly as the writer's. After two epochs with other people's, it reads people it never saw at about 6%.

11 Base v6 replaces v4

By the rule set before training, v6 replaces v4: 15.3% on the dev lines with the beam search against v4's 23.0%, −7.6 points (−10.0 to −5.1).

By the rule set before the pilot, v6 takes 50%, weighted by loops: real lines as half of each epoch, at the writer's pace, writers weighted towards the writer's loops.

Eight epochs, not five: the pilot was still improving at epoch 2, and an epoch now holds twice the examples.

8 epochs on a rented A10G, 37 min. Unlike v4 and v5, whose dev scores went flat after an epoch or two, v6 kept improving until epoch 7: 22.9% to 19.7% on the writer's dev lines, read greedily. Real ink gives the model more to learn. The synthetic lines would have kept epoch 8.

The test lines agree: 25.2% → 17.6%, −7.6 points (−9.9 to −5.2). The same size of gain as on the dev lines, so it isn't the luck of choosing an epoch on them. Against v3: −10.0 points (−12.6 to −7.3). Read greedily, the model alone, the test lines go from 30.3% to 21.4%.

The goal was 15% before personalising. v6 reads the writer's dev lines at 15.3% and the test lines at 17.6%. English lines are already below it. Personalising from v6, with the writer's own lines, is the next measurement.

Dev lines, beam

15.3%

v4 23.0%

Test lines, beam

17.6%

v4 25.2%, v3 27.5%

The 24 official lines

18.2%

v4 24.2%, v3 27.2%

Other writers, beam

2.9%

v4 26.9%

The pilot

$0.64

Modal's bill

Base v6's run

$1.13

the run's own GPU estimate

Pilot and run

$1.77

the run by its own estimate

Modal's per-app bill for the run lagged when this was written, so its cost is the GPU machine's own estimate: $1.13 for 37 min of training.

12 Where real ink helped

Lowercase errors nearly halved, 470 to 250, and English lines fell from 23.4% to 12.5%. The looped-l confusions are gone.

All 124 of the writer's lines, read by v4 and v6 with the beam search. The dev lines are among them, so this is for finding where the errors are, not a test figure.

Real hands fixed what fonts couldn't. Lowercase letters, two thirds of v4's errors, went from 470 errors to 250. English, the language BRUSH is written in, fell from 23.4% to 12.5%; French and mixed lines improved less.

Written → read as, with how often. The looped letters fonts drew badly, t read as l and h as L, are gone from the top. n read as u is still first, less often.

13 Failure: what real ink didn't fix

What BRUSH doesn't have got no better: math went from 33.8% to 37.7%, digits and accents rose, and the errors on spaces doubled.

BRUSH is English sentences, mostly two words a line. What it lacks, v6 didn't learn:

• Math: 33.8% → 37.7%, the only kind of line that got worse. Digits 58 → 67 errors, math symbols 40 → 41; 0 read as o is now a top confusion.
• Accents: 45 → 50. é read as e is now one of the most common confusions.
• Spaces: 44 → 87 errors, doubled. Lines of two words show few gaps between words of the writer's kind.

A public set fixes what it contains. French, math and word spacing are the next step. Real ink with accents, from French writers; longer lines; or synthetic lines kept for math while real ink carries the letters.

14 What the app now shows

v6 is the base: personalising starts from it, and the browser reads with it. The Models page says what each model was built on and learned from, and draws the family.

Base v6 is now the base. Personalising starts from it, and the browser's model is v6: 8.60 MB, reading 10 of 10 check lines exactly as the trained model does, at 45.6 ms per line (95th percentile), inside the 50 ms budget. A smaller 8-bit copy read 6 of the 10 differently, so the full model is used.

• Built on: trained further from a base's weights, a new model trained from scratch, or fitted to the writer's lines from a base. For v6: trained further from Base v4's weights.
• What it learned from, with what is new since its base marked. For v6: "Real handwriting from other writers: BRUSH, 24,873 lines from 153 writers (17 more kept out to read other hands), as 290,000 varied draws, 50% of each epoch, at your pace, writers weighted towards your loops", and BRUSH's research-only licence under it.
• Per-epoch curves for the writer's dev lines and for "Other writers (never trained on)".

• Base v3 new model
  • Base v4 trained further v6 was trained from it
    • Base v5 trained further not adopted
    • Base v6 trained further, with BRUSH current base

15 What carries over

When a synthetic source stops paying, real data from many people can move what rounds of synthetic changes couldn't. Pilot it, weight it towards the user, and check what it lacks.