Sharpening the tools before iteration 3
Eleven experiments are waiting. Before paying for any of them: tune what's free, record everything, search instead of guess, and make each run cheaper.
TL;DR
MethodThe question Iteration 3 is a list of experiments, each one a training run. What should change in how runs are made, recorded and chosen before any of them starts?
Eleven experiments were lined up, each one a paid training run. Before starting any of them, we sharpened the tools: squeezed out accuracy that needs no training, kept every run in one place, and let a search pick settings instead of guessing them one at a time. Nothing here is a new model; it's how every model from now on gets made and chosen.
- Tuning how the model turns its output into text, with no retraining, took about a point off the error rate of both models.
- Every run, evaluation and exported model is now recorded in one local file: 11 past runs backfilled, no account, no ink uploaded.
- How to personalise is now searched over six settings at once, with hopeless trials stopped early, instead of four one-setting experiments.
- Checking the model during training got four times faster (9.7 s to 2.4 s) for the same score, and runs can now stop when they stop improving.
- Rereading the training code turned up four quiet bugs.
Before running a list of experiments, make each run cheaper, recorded and searchable. Then take the improvements that need no training at all.
Why read on The trade-off curve that answered an experiment before we ran it, and why the laptop's own GPU lost to its processor.
1 The question
Eleven paid experiments were waiting, and the old tools would have wasted part of each one.
Iteration 3 is a notebook of eleven experiments. Each one is a training run: an hour on the laptop or a couple of dollars of rented GPU A graphics card: thousands of small cores doing arithmetic in parallel, which is what training mostly is. Changing it: Training is many times faster than on a CPU; rented by the second, so time is money. Here: A Modal T4 at about $0.59 an hour: v2's base cost $1.42..
Run them with the tools of iterations 1 and 2, and four things go wrong:
- Settings that cost nothing to change are still picked by hand.
- Results end up in folders nobody compares.
- Each experiment changes one setting while the others stay guessed.
- Each run wastes time on work it didn't need to do.
So before any of them, the tools changed. Nothing here is a new model: it is what every model from now on is made, recorded and chosen with.
2 Free accuracy: tune how the model reads
Tuning how the model's output becomes text took about a point off the error, with no training at all.
For each point of the pen's path, the network says how likely each letter is. A beam search Decoding that keeps the best few partial readings at every step instead of committing to one letter at a time. Changing it: A wider beam gives the language model room to fix ambiguous letters, but costs time: the beam search is about 40% of v2's time per line. Here: Width 8, combining the network's scores with the language model and your lexicon. turns that into text: it follows several candidate readings at once and keeps the best. A language model A model of which letters tend to follow which, learned from plain text. Changing it: It fixes readings the network finds ambiguous. Weighted too heavily, it overrules what you actually wrote, such as rare words and names. Here: A character model (each letter given the 4 before it) from 120,000 Tatoeba sentences per language, weighted 0.5. (which letters tend to follow which) and a bonus for real words help it choose.
That search has six settings: how many readings it keeps, how many letters it tries at each point, when it drops a reading, and how much the language model and the two bonuses count. They had been set by hand once, and never changed.
Changing them needs no training: a trial is reading a few dozen lines again, which takes seconds. So an open-source search library, Optuna, ran 120 to 150 trials per model, looking for the lowest error rate and the fastest reading at once.
Two rules kept it honest:
- It is scored on dev set A second set of held-out lines, for choosing settings and the best epoch. Changing it: It keeps the test set untouched, so the final score isn't flattered by choices made while looking at it. Here: A fifth of the writer's training lines, held out: the decoder and the personalising settings are tuned on them, never on the test lines.: a fifth of the writer's training lines, put aside for tuning. The 84 test lines are only read once, at the end.
- The hand-picked settings are the first trial, so the pick can't be worse on the dev lines.
| Row | Value | Interval |
|---|---|---|
| v2 base, defaults | 43% | |
| v2 base, tuned | 42% | |
| v2 personalised, defaults | 14% | |
| v2 personalised, tuned | 13% |
| Model | Dev lines | Trials | Dev | Test | Picked |
|---|---|---|---|---|---|
| v2 base | 60 | 120 | 42.5% → 41.7% | 43.3% → 42.2% | beam 10, language model 0.72, word bonus 0.36 |
| v2 personalised | 12 | 150 | 14.1% → 11.7% | 14.2% → 13.3% | beam 9, language model 0.27, word bonus 1.70 |
The personal model wants about half the language model's weight the base wants. A network that knows the hand needs less help guessing, and too much help overrules what it read.
The tuned settings travel with the exported model, so the browser reads with them too.
The same search also shows the price of speed. Allow one point more error on the dev lines, and it picks a beam of 6 at 10 ms per line instead of 9 at 18 ms.
That is experiment E10, the speed gate, answered with a curve instead of one guess, before it was ever run.
3 Every run in one place
Every run since iteration 1 can now be compared side by side, without an account or uploading any ink.
Until now each run left a folder with a summary file, and comparing two meant opening both.
Now every run also goes into a local MLflow store: an open-source experiment tracker that keeps its records in one file on the laptop, with a browser view to compare runs side by side.
- Runs backfilled
- 11
- every model since iteration 1
- Accounts needed
- 0
- one file, no server
- Ink uploaded
- 0
- the writer's lines stay local
- Its settings, the code's version, and whether the code had unsaved changes.
- Loss and error rate per epoch One full pass over every training line. Changing it: More epochs keep lowering the training error until it levels off. Past that point they cost time for nothing, or the model starts memorising the training lines. The best epoch is the one that scores best on held-out lines, not the last one. Here: v1's base ran 29 epochs of about 97,000 lines (about 1.5 minutes each on a T4); v2's ran 12 epochs of 290,000 lines (about 12 minutes each). Personalising stops after 6 epochs or 10 minutes., learning rate, speed.
- Memory and processor use while it trained, on the laptop or on the GPU.
- Which lines it learned from and which it was checked on.
- Each evaluation: a trace per test line with the expected text and what was read.
- Readings can be marked wrong by hand, and the next evaluation lists them.
- Each tuning: every trial, and the one picked.
- Each exported model: which run, settings and scores it came from.
The summary file stays the source of truth: the store is a copy that can be rebuilt from the folders at any time. Cloud runs are copied in when they come back.
4 Search how to personalise, instead of guessing
One search over six settings at once replaces four experiments that each tested one setting.
Four of the notebook's experiments each changed one personalising setting and kept the rest: train longer (E1), replay no synthetic lines (E3), change the number of varied copies (E4), change the learning rate How big a step each weight takes after every batch. Changing it: Too small and training creeps, wasting GPU time. Too big and it overshoots: the loss bounces or blows up. It's usually the first setting to tune. Here: 0.002 for the base models; 0.001 for personalising, which starts near a good solution and shouldn't be knocked away from it. (E6, how big each training step is).
That is four runs, and each answer holds only for the other settings as they were. But settings interact: a higher learning rate may want fewer epochs.
So one Optuna study now searches six settings together on the dev lines: learning rate, copies, replayed Mixing some of the original training data back in while fine-tuning on new data. Changing it: It keeps the general skill while the model learns the new data; too much and the new data has less effect. Here: 2,000 synthetic lines alongside your lines., epochs, how much of each line is dropped at random, and a new one, how many of the base's lowest layers stay frozen.
Why freeze layers
The base's lowest LSTM A recurrent layer: it reads a sequence one step at a time and keeps a memory (its state) of what it has seen. Changing it: Well suited to pen paths, but each step waits for the one before, which limits how much a GPU can speed it up. Here: Each layer has four gates (input, forget, cell, output) deciding what to keep and what to forget. layers read pen movement: curves, turns, lifts. Every hand has those. The top ones turn them into letters, and that is where a hand differs.
Freezing the lowest layers keeps them exactly as the base learned them from 300,000 lines, and lets a few dozen real lines move only the rest.
Fewer weights to move can mean less overfitting When a model learns its training examples themselves rather than the pattern behind them, so it does much better on lines it has seen than on new ones. Changing it: It makes training scores look better than reality. The cures are more or more varied data, augmentation, fewer epochs, and picking the epoch on held-out lines. Here: v2 personalised reads its own training lines with 4.5% errors and the test lines with 13.1%. (memorising the training lines), or too little room to learn the hand. The search measures which.
- Today's plan is trial 0, so the report always says how far the best trial is from what we do now, in characters on the dev lines.
- Hopeless trials stop early: after each epoch, a trial worse than the median of the others at that point is dropped. Most of the time goes to promising ones.
- Every trial is rebuildable: same seed The starting number for every random choice: which font, which word, how much slant. Changing it: Same recipe and same seed give the same data, so a version can be rebuilt. Different seeds show how much a result is luck. Here: Pinned in each recipe., same data, so the picked trial's model can be made again exactly.
- Under 3 characters is not a verdict: a smaller gain is checked with a second seed and on the test lines before the plan changes.
5 A cheaper training loop
Rereading the training code line by line made checking four times faster and found four quiet bugs.
The training code is plain PyTorch: the model is a small network, and a framework would add more than it saves. Read again line by line, it did work it didn't need to.
- Validation, 300 lines
- 9.7 → 2.4 s
- lines read in batches, same score
- Loss read back
- 1 in 50
- batches, not every one
- Data workers on the GPU
- 4
- same batches as with none
- Validation reads lines in batches, as training does, instead of one at a time. It runs every epoch, so it adds up.
- The loss is read back every 50 batches. Reading it makes the processor wait for the GPU to finish, so doing it every batch left the GPU idle in between.
- Batches are prepared in parallel on the GPU machine while the GPU trains. Each batch draws its own seed, so a run still gives the same batches, however many workers prepare them.
- The optimiser updates all weights in one step on the GPU instead of one layer at a time.
- Runs can stop on their own. A recipe can now ask for a learning rate that warms up and then falls along a curve, and for training to stop after a few epochs without a better model (blank plateau An early phase of CTC training where the model outputs only blanks, because that's a safe guess. Changing it: Deep models can sit there a long time; smaller models, warm-up or more steps get out of it. Here: Our first 3-layer model on the CPU read every line as "1".). Left out, a recipe trains exactly as v1 to v3 did.
- Each epoch records its learning rate, the size of its updates and lines per second, so a slow or unstable run shows in the curves, not in the bill.
- A profiler can record a few batches on demand, to show where a batch's time goes: the network, the loss, the optimiser, or waiting for data. That says whether a bigger GPU or mixed precision (cheaper number formats) would pay, before paying for either.
Rereading also found four quiet bugs, none of which had failed loudly:
- Training reordered the lines it was given, so the caller's list changed under it.
- Evaluation left the library on one processor thread, slowing down whatever ran after it.
- A checkpoint cut off mid-write could be loaded as if whole.
- Scoring a model could leave it in training mode.
6 Failures: what we tried and didn't keep
A newer exporter and the laptop's own GPU both looked like upgrades, and both made things worse.
It fixed the length of the input to the example it was given, so the browser would read lines of one length only. The model stays on the older exporter, and a test now checks that the exported model reads lines of 1, 50 and 400 points.
On the laptop's M1 Pro, a batch of 32 lines took 1.67 s on its GPU against 1.45 s on its processor. The network is too small to gain from it, and the loss we train with isn't available there. The processor also gives the same result every run.
7 What runs next
The personalising search goes first, with its prediction written down before it runs.
The personalising search runs first: 30 trials or two hours on the laptop, whichever comes first, from v2 base on round v0.2's lines.
Then the decoder is tuned again on the model it picks, and only then are the test lines read.
The best trial beats today's plan by more than 3 characters on the dev lines. Learning rate and epochs matter more than copies and replay. Freezing the lowest layer helps or changes nothing; freezing all of them hurts.
Not run yet.
After that comes E11, the base trained on synthetic ink that writes like the writer, now with a run that can stop at its plateau and a profile of where its GPU time goes.
8 What carries over
Five habits worth setting up before any list of experiments.
- Look for settings that cost nothing to change, such as how a model's output is decoded, and tune them first. No training run is as cheap.
- Tune on a dev set, score once on the test set, and put today's settings in the search as the first trial so the result can only be measured against them.
- One search over interacting settings answers more than one experiment per setting, and stopping weak trials early pays for most of it.
- Record every run in one place before running the experiments, not after: the comparison you'll want is the one you didn't plan.
- Read the training loop like a bill. Work done every batch or every epoch adds up, and a rewrite is the cheapest time to find quiet bugs.