Base v7: stitched hands

1 The question

Base v6 reads the writer's test lines at 17.6%. Its real ink was random pairs of English words, so what the writer writes and BRUSH doesn't, it reads worst.

In episode 11, real handwriting from 153 other people took the base modelbase model The general reader, trained on lots of synthetic handwriting before it ever sees yours. Changing it: A better base gives a better personal model: it starts lower and learns your hand faster. Alone it can still read your writing badly. Here: From v1 on, trained on a cloud GPU from a recipe; v2's base alone gets 43.6% of your letters wrong, 13.1% once personalised. from 25.2% to 17.6% on the writer's 84 test lines, read with the beam searchbeam search Decoding that keeps the best few partial readings at every step instead of committing to one letter at a time. Changing it: A wider beam gives the language model room to fix ambiguous letters, but costs time: the beam search is about 40% of v2's time per line. Here: Width 8, combining the network's scores with the language model and your lexicon. as the app reads. The goal is 15% before the model sees any of the writer's lines.

But BRUSH's writers wrote what BRUSH asked: random pairs of words, such as "Drawcansir Holroyd", in English. The writer writes notes of 5 to 7 words, French with accents, times, amounts and math. Here is where v6 goes wrong, on all 124 of the writer's lines:

Real ink taught v6 letters. It didn't teach the writer's texts. Accented letters are wrong 50% of the time, math symbols 46%, digits 40%. Of 87 errors on spaces, 64 are spaces it dropped: two-word lines show few gaps between words, and wider ones than the writer leaves.

2 The idea: cut real letters, join them into new lines

BRUSH labels every point of a line with its letter. So a writer's ink can be cut where letters meet and joined again into any text, in that writer's hand.

Each BRUSH line knows which letter every point of ink belongs to. That makes it possible to cut a writer's lines into pieces and join the pieces into new lines of the writer's kind of text: sentences, notes, math, accented words. We call them stitched lines.

• Pieces of up to 4 letters, cut whole from one writer's lines, so the joins inside a piece stay as the hand made them. Where both sides of a cut were joined in their lines, the pen stays down across it.
• One writer per line. Pieces are measured in their own line's letter height, sitting on its baseline, so pieces from different lines of one writer line up.
• Spacing is the writer's own: after each letter, the gap they left in the source. Word gaps too, then narrowed by a factor from 0.55 to 1.05 (why, in the next section).
• Late strokes stay late. i dots and t crosses come at the end of the word, as the writer wrote them.

What BRUSH lacks is built from the writer's own marks. Accents and symbols are made from strokes that writer did write:

Each writer has only about three of each of + = < > /. A symbol a writer never wrote is borrowed from another writer, at the same letter height. Letters are never borrowed: a line is one hand.

Cutting BRUSH into letters took 34 s on the laptop: 26,757 lines. 892 were left out, with too few letters to find their letter height. The 17 writers held out since episode 11 stay held out, and 2,000 of the texts are held out too. Every training run writes new stitched lines, in parallel on rented CPUs, at the writer's pace and varied as the real lines are.

3 Checks before any GPU

The writer's word gaps are 18% narrower than BRUSH's. v6 reads stitched lines about as badly as the writer's, and for the same reasons: dropped spaces, math, digits.

Word gaps, measured the same way everywhere. Per line, the gaps between blocks of ink, one per space, divided by the line's ink width per letter.

The writer's word gaps are about 18% narrower than BRUSH writers'. Stitched with the writers' own gaps, lines keep BRUSH's; narrowed by 0.55 to 1.05, they match the writer's.

What v6 makes of stitched lines. Read greedilygreedy decoding Reading the model's output by taking the top guess at every point, then applying CTC's rules. Changing it: Fast and simple, but it can't use spelling knowledge, so it produces things like "1%e" for "the". Here: Greedy CER is the network on its own: 18.4% for v2 personalised, against 13.1% with the beam search., on 1,000 lines of the held-out writers: their own BRUSH lines 2.8%, stitched lines in their hands 14.7%. By kind of text: sentences 12.1%, words 13.4%, notes 15.6%, math 25.3%. Its errors are the ones it makes on the writer's lines: dropped spaces ("onthe", "n'aiaucune"), math, digits. So stitched lines hold what v6 hasn't learned.

• Spacing is the cause. The narrower the gaps, the more spaces v6 drops: 83% found at BRUSH's gaps, 74% at the writer's, 62% narrower still. Lines with the writer's gaps should teach it what a word gap of that size looks like.
• Pieces of one letter read about a point worse than pieces of up to 4: the joins inside longer pieces look more like real writing.

4 A dead end: the long pieces aren't there to cut

Pieces may be up to 4 letters, but only about a third of the letters get one of 3 or 4. A cut that tries harder changed nothing v6 could read.

Each writer has about 160 BRUSH lines of random words: about 1,550 different runs of 3 letters. Most words of the writer's texts can't be cut into pieces of 3 or 4 from so little. The phrase in the figure above came out as 10 pieces, most of them one or two letters.

"3 or 4 letters linked" is what the cutter tries for, and about a third of the letters get it. Cutting with the fewest pieces raises that from 27% to 35%, and v6 reads both alike, within a few tenths across seeds. So v7 keeps the simpler cut. Whether piece length matters at all is a question for the pilot, with an arm of one-letter pieces.

5 The pilot: the stitching, or just more real ink?

Stitched lines beat the control by 2.5 points with the beam search. As many extra lines of real ink made it 2.0 points worse.

As in episode 11: every arm starts from v6's weightsweights The numbers inside the model that training adjusts. A trained model is just these numbers plus the code that uses them. Changing it: More weights can learn more, but need more data and training time, make the file bigger and reading slower. They grow with the square of the layer width. Here: v0: 0.18 million (1 × 128). v1: 0.97 million (3 × 128). v2: 2.15 million (3 × 192), 8.6 MB in full precision., trains 2 epochsepoch One full pass over every training line. Changing it: More epochs keep lowering the training error until it levels off. Past that point they cost time for nothing, or the model starts memorising the training lines. The best epoch is the one that scores best on held-out lines, not the last one. Here: v1's base ran 29 epochs of about 97,000 lines (about 1.5 minutes each on a T4); v2's ran 12 epochs of 290,000 lines (about 12 minutes each). Personalising stops after 6 epochs or 10 minutes. on the same 80,000 synthetic lines, and is read on the writer's 38 dev lines. Stitched arms make an epoch bigger, so one arm gets as many extra examples from real ink alone: if it does as well, the gain is just more real ink.

By the rule set before the pilot, stitched lines go into v7. The stitched arm beats the control, and the 65% real arm is worse than the control. The gain is the stitched lines, not more examples or more real ink.

• Stitching fixes what v6 lacked: accent errors 8 → 3, spaces 16 → 9 (mostly dropped spaces, 12 → 6), math 15 → 11, digits 15 → 12. Lowercase too: 71 → 64.
• Linked pieces matter for lowercase. One-letter pieces fix accents and spaces as well, but lowercase gets worse (74). Their gain, −1.5 points, is about half and isn't sure. So pieces stay up to 4 letters.
• Narrowed gaps or the writers' own: no sure difference. The same spaces (9 against 10) and a gain, −2.8, whose interval reaches +0.1. Not better by more than noise, so v7 keeps the gaps narrowed to the writer's, which also match the measured ones.

The pilot

$1.70

Modal's bill, 5 arms

Expected

$1.20

51 CPU containers wrote the stitched lines

Each GPU arm

$0.08–0.15

the runs' own estimates

6 Failure: the first pilot launch never started

A shared store made when the code is first loaded can't be shipped with the functions to the cloud. Now it is looked up when it is used.

The pilot's first launch failed before anything ran. The code made a shared store, Modal's dictionary that the laptop and the cloud both read, at the moment the code was loaded. To run in the cloud, the functions are packed up and sent, and that store can't be packed with them.

Now the store is looked up when it is first used, not when the code loads. The lesson is general: anything that holds a connection belongs inside the function that needs it, not at the top of the file.

7 Training with a watch on the laptop

The learning rate stays flat while a watch on the laptop reads every epoch with the beam search. When 3 epochs go by without a gain, the run decays and stops.

Episode 11 trained a set number of epochs along a falling learning ratelearning rate How big a step each weight takes after every batch. Changing it: Too small and training creeps, wasting GPU time. Too big and it overshoots: the loss bounces or blows up. It's usually the first setting to tune. Here: 0.002 for the base models; 0.001 for personalising, which starts near a good solution and shouldn't be knocked away from it., then kept the epoch that read the dev lines best greedily. This time the beam search, the way the app reads, picks the epoch, and the run decides its own length.

• The rate stays flat at 0.0005, so the run can go on as long as it pays.
• A watch on the laptop reads every epoch as it lands: the writer's 38 dev lines with the beam search, and the held-out writers. The writer's lines never leave the laptop: the weights come down, not the ink up.
• When it stops paying, decay. From epoch 4, after 3 epochs without beating the best by 0.3 points, the watch signals the run. The rate then falls to zero over 2 epochs, and the run stops at an epoch's end. 16 epochs at most.

8 Failure: the watch crashed, and took the GPU run with it

The first launch reached training, then the watch on the laptop hit an error it didn't expect. A local error stops the whole app, the GPU's training with it.

The first launch of v7 got through the data check, wrote 248,571 stitched lines in 63 containers, and began training. Then the watch looked for the first epoch's folder before it existed. It was ready for "file not found", but Modal raises its own "not found" error, a different one. A question to the run that isn't answered yet likewise raises Modal's own time-out, not the usual one.

An error on the laptop stops the app, and the training container with it: a watch meant to make the run safer stopped it. Before relaunching, the watch was changed to catch any error reading the storage or an epoch, log it, and try again on its next round. It was tried against fakes on the laptop, and the signal to decay was sent through Modal's dictionary and read back.

The cost: the CPU containers, a few GPU minutes, and writing the stitched lines a second time. It's in v7's bill below.

9 The watch, epoch by epoch

The dev lines bounced between 11.2% and 14.6%. The watch asked for the decay after epoch 9, and the run stopped itself after 12. Epoch 6 is kept.

The second launch: 828,571 examples an epoch (248,571 stitched lines written anew, 290,000 BRUSH copies, 290,000 synthetic lines), 12,947 batches, on an A10G: 527 s an epoch, 1.8 h in all.

1. After epoch 9, three epochs had gone by without beating epoch 6 by 0.3 points, so the watch asked for the decay.
2. The run was already in epoch 10, so it read the signal at that epoch's end. The rate fell over epochs 11–12, and training stopped after epoch 12 of 16, on its own, without being cut off mid-epoch.
3. The epoch kept is 6, the beam search's best on the dev lines.

• At a steady rate, the dev lines bounce from epoch to epoch. 38 lines are noisy, so one epoch's dip is partly luck. The kept epoch is the best of 12 on these same lines, so its dev figure, 11.2%, is flattered. That's why the test lines decide whether to believe it.
• The decay helped the other writers, not the writer. Stitched lines of the held-out writers went from 3.2% to 2.8%, their real lines from 2.8% to 2.4%, but the dev lines ended at 13.2%. The annealed model fits BRUSH-like ink a little better, not the writer's.

10 Base v7 replaces v6

By the rule set before training, v7 replaces v6. The test lines, read once, agree: 17.6% → 11.5%, below the 15% goal before personalising.

The test lines, read once, agree, and by more: 17.6% → 11.5%, −6.0 points (−8.6 to −3.6). That is below the 15% goal without personalising, and the dev figure's luck can't explain it. Read greedily, the network alone: dev lines 19.7% → 15.9%, test lines 21.4% → 16.0%.

Dev lines, beam

11.2%

v6 15.3%

Test lines, beam

11.5%

v6 17.6%

The 24 official lines

9.8%

v6 18.2%, v4 24.2%

Other writers, beam

3.5%

v6 2.9%

Base v7 is now the base. Personalising starts from it, and the browser reads with it: 8.60 MB, reading 10 of 10 check lines exactly as the trained model does. A smaller 8-bit copy read 7 of the 10 differently, so the full model is used.

The pilot

$1.70

Modal's bill

Base v7's run

$3.92

Modal's bill, with the failed launch; GPU $3.51

All of v7's work

$5.62

pilot and run

11 Where stitching helped

French went from 18.2% to 10.8% and math from 37.7% to 19.3%. Accent errors 50 → 20, dropped spaces 64 → 18.

All 124 of the writer's lines, read by v6 and v7 with the beam search. The dev lines are among them, so this is for finding where the errors are, not a test figure.

Stitched lines taught what BRUSH alone couldn't. French and math nearly halved. Accent errors fell from 50 to 20, and é read as e is gone from the top confusions. Digit errors halved. Dropped spaces went from 64 to 18, though added ones rose from 6 to 14.

Lowercase got better too, 250 errors to 178: real letters in real words and sentences, not only the two-word lines BRUSH wrote. English barely moved (12.5% → 12.3%): it was what BRUSH already had.

Written → read as, with how often.

12 Failure: what got worse

Punctuation errors went from 33 to 57, capitals that look like their lowercase are now top confusions, other writers read 2.9% → 3.5%, and a line takes 52.5 ms against a 50 ms budget.

• Punctuation: 33 → 57 errors. Commas and colons are dropped (8), read as a space (5) or as I. In stitched lines, marks are small and often borrowed from another writer. French puts a space before ":" and "!", and that space is narrowed like any word gap, so a small mark at a gap looks like a space.
• Capitals mistaken for lowercase: h→H, c→C, J→j are now among the top confusions. These are letters whose capital is a larger copy of the lowercase.
• Other writers: 2.9% → 3.5%, +0.6 points (+0.1 to +1.1): clearly worse, and small. BRUSH is 35% of each epoch instead of half.
• Speed: 52.5 ms a line for 95% of lines (two runs: 52.4 and 52.6), just over the 50 ms budget; v6 measured 47.0 ms in the same session. The network is the same size, so the extra time is in the beam search: more possible readings stay close to each other.

Next: size and place marks as their writers did, and don't narrow the gap before a mark. In BRUSH, marks are cut with their word ("dwelling."), so they can come with it. And the beam search's time is the next thing to look at.

13 What carries over

When real data has the right shapes but the wrong content, recombine it into the content your users make. Pilot that against more of the same real data, and measure what you actually get.