The synthetic ink didn't write like a hand
Checking the data before paying for the GPU: clean, but lifting the pen twice as often as a person does.
TL;DR
Open questionThe question The base model is only as good as its synthetic lines. Are they clean, and do they look like real ink?
Before paying for another GPU run, we checked the fake handwriting the model learns from. Nothing was broken, but it didn't write like a person: it lifted the pen far too often and drew jagged paths. So the generator now joins and smooths its strokes to match the real writer, and we wrote down what we expect before running it.
- Clean: no broken strokes, no copies, all 134 fonts used.
- But font ink lifts the pen about 1.8 times per letter; the writer lifts it about once.
- Its paths carry about a third more points per letter: a staircase a pen never draws.
- Some capitals, accents and symbols never appear at all, so the model can't read them.
- The fix brings both rates to the writer's own. The prediction: the base model drops from 43% to under 30% error.
Check synthetic data against real data before spending compute on it. Clean data can still be the wrong data.
Why read on Why a computer tracing a letter splits an "a" into pieces a hand never would, and the table where a quick prototype lands right on the writer's numbers.
1 The question
The model is only as good as its fake handwriting, and nobody had looked at it closely.
Every base model so far learned from synthetic lines, and nobody had checked them. A bad batch only shows up after a two-hour GPU run. So the data now gets checked before the GPU is asked for.
The first check was on the data v2 learned from: 20,000 of its 300,000 lines, compared with the writer's 85 recorded lines.
2 Errors stop the run, warnings don't
Broken data stops the run; data that is merely odd is flagged and kept.
- NaN ink: one such line turns every weight into NaN.
- Empty or backwards strokes.
- Lines too long for GPU memory, or too long for CTC A way to train and read a model that gives one guess per point, when nobody has marked which points belong to which letter. Changing it: It lets the model train on whole lines with only their text. Models trained this way say "blank" almost everywhere, with a short spike for each letter. Here: Merge repeats, then drop blanks: "hh_e_ll_l_oo" reads "hello". to place.
- Reused seeds, missing languages.
- Thin coverage of letters, digits and symbols.
- Lines that all look alike.
- Drift: synthetic ink measured unlike the writer's own.
The rules run over a table with one row per line, using an open-source data-validation library.
The writer's ink never leaves their machine: only measurements of it are compared.
3 Clean
Nothing was broken: no errors, every font used, no copies. The warnings told another story.
- Errors
- 0
- no NaN, empty or backwards strokes
- Fonts used
- 134
- every one it was given
- Distinct texts
- 97%
- no copies
Nothing would have stopped the run. But four warnings came back:
- capital letters or digits are rare (fewer than one line in 2,000)
- points per character is unlike the writer's ink (median outside the middle 80% of the writer's lines)
- pen lifts per character is unlike the writer's ink (median outside the middle 80% of the writer's lines)
- characters you write never appear in the synthetic lines (the base can't read them)
4 Drifting: font ink lifts the pen twice as often
Font ink lifts the pen almost twice as often as the writer and draws longer, jaggier paths.
Two simple counts, per letter, show the drift: how often the pen leaves the paper, and how many points the path is made of. On both, v2's lines sit well outside the writer's.
| Row | Value | Interval |
|---|---|---|
| the writer | 1.0 | 0.8 to 1.4 |
| v2 synthetic | 1.8 | 1.4 to 2.4 |
| Row | Value | Interval |
|---|---|---|
| the writer | 10 | 8 to 13 |
| v2 synthetic | 14 | 10 to 18 |
Why font ink drifts
Pen lifts. A glyph is drawn, thinned to its centre line and traced into strokes. The tracer starts a new stroke at every junction of a letter, where a hand would keep the pen down and go back over it. So an "a" or a "k" becomes two or three strokes.
Points. A traced path keeps the pixel staircase of the glyph image, and augmentation Making extra training examples by changing existing ones in ways that keep the answer the same: slant, size, speed, jitter, dropped points. Changing it: It teaches the model which differences don't matter, so it generalises from few examples. Too extreme, and the examples stop looking like real writing. Here: Each of the writer's lines becomes 40 variations; half of v2's synthetic lines are augmented too. adds jitter on top. The model reads the extra length as zig-zags a pen never makes.
5 The fix: synthetic ink that writes like the writer
Join strokes and smooth paths until the fake ink matches the writer's own rates; change nothing else.
The next version of the generator adds two targets, taken from the writer's own ink. Nothing else changes, so the old and new data compare cleanly.
- 0.96 pen lifts per letter: the two closest strokes are joined until the line reaches its target. The pen stays down over the short way between them.
- 10.1 points per letter: the path is smoothed until it has few enough points. Smoothing removes the staircase without shortening the letter.
Real lines vary, so each line draws its own target within ±20% of the rate. They don't all come out the same.
A quick prototype on 800 lines landed on the writer's numbers:
| Per letter | v2 as made | v2 joined and smoothed | The writer |
|---|---|---|---|
| Pen lifts | 1.33 / 1.83 / 2.38 | 0.74 / 0.94 / 1.23 | 0.78 / 0.96 / 1.42 |
| Points | 10.40 / 13.70 / 18.40 | 8.40 / 10.80 / 14.10 | 7.78 / 10.14 / 13.14 |
6 The prediction, before the run
We bet the base model drops under 30% error. Written down now, so a miss still teaches something.
This is experiment E11: a full base trained on the new data, about 2.5 hours on a rented GPU for about $1.50. It is then personalised exactly as v2.1-base was.
The control is v2 base, which reads the test lines at 43.3% error.
Base CER under 30% (from 43%), personalised CER falls with it, and the data check shows no drift warning.
Not run yet.
If the base doesn't move, the drift wasn't what held it back, and that will be the next episode.
7 What carries over
Five habits for anyone who trains on generated data.
- Check generated data before training on it, and split the checks in two: errors that stop the run, warnings that are recorded. Checks that only warn become noise.
- Compare synthetic and real data on a few simple, readable statistics, such as rates per character. Drift shows up there before it shows up in a score.
- The generator's artefacts (here, strokes split at every junction and pixel staircases) become features the model learns.
- Coverage matters as much as realism: a symbol the model never saw is one it can't read.
- Write the prediction down before an expensive run, so a disappointing result still teaches something.