Glossary

Terms

ablation

An experiment that changes one thing, keeps the rest, and measures.

Changing it: It's how you learn which change helped. Changing several at once gives a result you can't explain.

Here: v2 changed five things at once; v3 changes one at a time.

augmentation

Making extra training examples by changing existing ones in ways that keep the answer the same: slant, size, speed, jitter, dropped points.

Changing it: It teaches the model which differences don't matter, so it generalises from few examples. Too extreme, and the examples stop looking like real writing.

Here: Each of the writer's lines becomes 40 variations; half of v2's synthetic lines are augmented too.

base model

The general reader, trained on lots of synthetic handwriting before it ever sees yours.

Changing it: A better base gives a better personal model: it starts lower and learns your hand faster. Alone it can still read your writing badly.

Here: From v1 on, trained on a cloud GPU from a recipe; v2's base alone gets 43.6% of your letters wrong, 13.1% once personalised.

batch

The group of lines the model reads before each update of its weights.

Changing it: Bigger batches give a steadier direction and use the GPU better, but mean fewer updates per epoch and more memory. Smaller ones are noisier but update more often.

Here: 64 lines, sorted by length so little padding is wasted.

beam search

Decoding that keeps the best few partial readings at every step instead of committing to one letter at a time.

Changing it: A wider beam gives the language model room to fix ambiguous letters, but costs time: the beam search is about 40% of v2's time per line.

Here: Width 8, combining the network's scores with the language model and your lexicon.

BiLSTM

Two LSTMs over the same sequence, one forward and one backward, joined at every point.

Changing it: Each point is read knowing what came before and after, like you do with a messy letter, at twice the cost of one direction.

Here: 3 layers of them in v1 and v2.

blank

CTC's extra output, meaning "nothing new here".

Changing it: It separates real double letters (l_l is "ll", ll is "l"). Early in training, models can get stuck predicting only blanks: the blank plateau.

Here: One of the model's 178 outputs, next to its 177 characters.

blank plateau

An early phase of CTC training where the model outputs only blanks, because that's a safe guess.

Changing it: Deep models can sit there a long time; smaller models, warm-up or more steps get out of it.

Here: Our first 3-layer model on the CPU read every line as "1".

bootstrap

Estimating how much a score would change on a different test set, by drawing the test lines again with replacement many times and re-scoring.

Changing it: It tells a real improvement from luck. Paired means the same lines for both models, which removes the "some lines are harder" noise.

Here: v2 beat v1 in all 10,000 paired resamples, by 5.0 to 15.6 points.

calibration

Making a model's confidence match how often it's right: of readings at 90%, nine in ten should be correct.

Changing it: A calibrated confidence can decide when to ask "did you mean…?". Modern networks tend to be overconfident.

Here: Not calibrated yet; temperature scaling is the usual one-number fix.

catastrophic forgetting

When a model trained on new data loses what it knew before.

Changing it: A personalised model could read your hand and nothing else. Mixing old data back in (replay) and a small learning rate prevent it.

Here: Checked every personalisation: error on synthetic lines stays flat.

CER

Character error rate: substituted, deleted and inserted letters, divided by the number of letters in the right text.

Changing it: The main score here; lower is better. It weighs every letter the same, and it can go over 100% when a reading has many extra letters.

Here: The goal is 15% or less. v2 personalised: 13.1%.

confidence interval

The range the score would plausibly take on other, similar test lines; here the middle 95% of the bootstrap scores.

Changing it: It narrows with the square root of the number of test lines: four times the lines for half the width.

Here: v2 personalised: 6.7% to 21.4% on 15 lines, so the 15% goal isn't proven yet.

CTC

A way to train and read a model that gives one guess per point, when nobody has marked which points belong to which letter.

Changing it: It lets the model train on whole lines with only their text. Models trained this way say "blank" almost everywhere, with a short spike for each letter.

Here: Merge repeats, then drop blanks: "hh_e_ll_l_oo" reads "hello".

dev set

A second set of held-out lines, for choosing settings and the best epoch.

Changing it: It keeps the test set untouched, so the final score isn't flattered by choices made while looking at it.

Here: A fifth of the writer's training lines, held out: the decoder and the personalising settings are tuned on them, never on the test lines.

dropout

Hiding part of the input or the network at random, during training only.

Changing it: It stops the model leaning on any one point or feature, so it generalises better. Too much and it learns slowly.

Here: The time feature (dt) is hidden on 20% of points.

epoch

One full pass over every training line.

Changing it: More epochs keep lowering the training error until it levels off. Past that point they cost time for nothing, or the model starts memorising the training lines. The best epoch is the one that scores best on held-out lines, not the last one.

Here: v1's base ran 29 epochs of about 97,000 lines (about 1.5 minutes each on a T4); v2's ran 12 epochs of 290,000 lines (about 12 minutes each). Personalising stops after 6 epochs or 10 minutes.

fine-tuning

Continuing to train an already-trained model for a short while on new data: here, your own lines.

Changing it: It adapts a general model to your hand from very little data. Too long, or with too high a learning rate, and it memorises your lines or forgets general writing.

Here: 40 varied copies of each of your 25 lines, plus 2,000 synthetic lines replayed. It took v2 from 43.6% to 13.1% errors.

GPU

A graphics card: thousands of small cores doing arithmetic in parallel, which is what training mostly is.

Changing it: Training is many times faster than on a CPU; rented by the second, so time is money.

Here: A Modal T4 at about $0.59 an hour: v2's base cost $1.42.

gradient

For every weight, which direction and how strongly to change it to lower the loss. Backpropagation computes all of them in one backward pass.

Changing it: Gradient descent follows it downhill, one step per batch. A very large gradient from one bad batch can undo good training, which is why gradients are clipped.

Here: About 2.1 million gradients per step for v2, one per weight.

gradient clipping

Scaling down any gradient longer than a limit before the update.

Changing it: One unusual batch can't wreck the weights; the limit rarely matters otherwise.

Here: Norm 1.0.

greedy decoding

Reading the model's output by taking the top guess at every point, then applying CTC's rules.

Changing it: Fast and simple, but it can't use spelling knowledge, so it produces things like "1%e" for "the".

Here: Greedy CER is the network on its own: 18.4% for v2 personalised, against 13.1% with the beam search.

hidden size

The width of each LSTM layer: the size of its memory.

Changing it: Wider layers learn more but the weights grow with the square of the width, and reading gets slower.

Here: 128 in v1, 192 in v2: 1.5 times wider, 2.2 times the weights.

inference

Using a trained model to read new input, as opposed to training it.

Changing it: It has to be fast and run where the data is, which shapes the model's size.

Here: Every pen lift, in a background thread in the browser.

language model

A model of which letters tend to follow which, learned from plain text.

Changing it: It fixes readings the network finds ambiguous. Weighted too heavily, it overrules what you actually wrote, such as rare words and names.

Here: A character model (each letter given the 4 before it) from 120,000 Tatoeba sentences per language, weighted 0.5.

learning curve

A model's score plotted against how much training data it got.

Changing it: Still falling means more data will help; flat means the gains are elsewhere.

Here: Planned: personalise with 10, 25 and 50 of the writer's lines.

learning rate

How big a step each weight takes after every batch.

Changing it: Too small and training creeps, wasting GPU time. Too big and it overshoots: the loss bounces or blows up. It's usually the first setting to tune.

Here: 0.002 for the base models; 0.001 for personalising, which starts near a good solution and shouldn't be knocked away from it.

lexicon

A list of known words that get a bonus when a reading completes one.

Changing it: It pulls readings towards words you actually use; too big a bonus and it forces words that weren't written.

Here: The words in your training lines, with a bonus of 1.5 per finished word.

LoRA

Low-rank adaptation: freezing a model and learning a small add-on change for each weight matrix.

Changing it: Much less to store per person, and the base can't be damaged; it may learn less than full fine-tuning.

Here: Not used yet: a candidate for the personalisation layer.

loss

One number scoring how wrong the model's output is; training exists to make it smaller.

Changing it: It's the only thing training optimises, so it has to match what you care about. A falling loss with a flat test score means the model is getting better at the wrong thing.

Here: The CTC loss: minus the log of the probability the model gives the right text.

LSTM

A recurrent layer: it reads a sequence one step at a time and keeps a memory (its state) of what it has seen.

Changing it: Well suited to pen paths, but each step waits for the one before, which limits how much a GPU can speed it up.

Here: Each layer has four gates (input, forget, cell, output) deciding what to keep and what to forget.

ONNX

A standard file format for trained models, runnable without PyTorch.

Changing it: It's how the model gets from training in Python to the browser.

Here: Run with onnxruntime-web, in WebAssembly, inside a Web Worker.

optimiser

The rule that turns gradients into weight changes.

Changing it: AdamW gives every weight its own step size from its recent gradients, and shrinks weights slightly to keep them small. It's the default almost everywhere.

Here: AdamW, learning rate 0.002, gradients clipped at 1.0.

overfitting

When a model learns its training examples themselves rather than the pattern behind them, so it does much better on lines it has seen than on new ones.

Changing it: It makes training scores look better than reality. The cures are more or more varied data, augmentation, fewer epochs, and picking the epoch on held-out lines.

Here: v2 personalised reads its own training lines with 4.5% errors and the test lines with 13.1%.

p95

The 95th percentile: the time within which 95% of lines are read.

Changing it: It shows the slow lines a user notices, which an average hides.

Here: Budget: 50 ms per line. v2 personalised: about 58 to 61 ms.

Pareto front

The models that no other model beats on both accuracy and speed at once.

Changing it: Only models on it are worth keeping; the budgets decide where on it to be.

Here: Today: v0, v1 and v2 personalised.

perplexity

How surprised a language model is by new text: roughly, how many letters it hesitates between at each step.

Changing it: Lower means it predicts the text better and helps decoding more.

Here: About 10 for English and 174 for French: French text is far harder for it to predict.

quantisation

Storing each weight in 1 byte (int8) instead of 4 (float32).

Changing it: About 4 times smaller and often faster, but it can change readings, so it must be checked against the original.

Here: Only used if it reads every check line exactly like the original; so far it hasn't, so the browser loads the full-precision model.

recipe

A small file that pins everything that decides what a base model learns: fonts, words, mix, sizes, seed, training settings.

Changing it: It makes a version rebuildable and comparable; it's never edited once the version is built.

Here: One per base model (v1, v2, v3), each with its seed, so any of them can be rebuilt.

replayed

Mixing some of the original training data back in while fine-tuning on new data.

Changing it: It keeps the general skill while the model learns the new data; too much and the new data has less effect.

Here: 2,000 synthetic lines alongside your lines.

resampling

Keeping points at an even spacing along the pen path, instead of one per tablet report.

Changing it: Slow and fast writing give similar numbers of points; wider spacing means fewer points and faster reading, but finer detail is lost.

Here: Every 0.1 line heights.

seed

The starting number for every random choice: which font, which word, how much slant.

Changing it: Same recipe and same seed give the same data, so a version can be rebuilt. Different seeds show how much a result is luck.

Here: Pinned in each recipe.

synthetic data

Training data written by code instead of collected from people: here, handwriting fonts turned into pen paths with timing.

Changing it: It's unlimited and licence-clean, but if it doesn't look like real writing the model learns the wrong thing: the synthetic-to-real gap.

Here: 300,000 lines for v2, from 134 fonts under open licences (OFL or Apache).

synthetic-to-real gap

The difference between how well a model reads the synthetic data it trained on and how well it reads real writing.

Changing it: A model can score almost perfectly on synthetic lines and still read real ink badly, so only a test on real writing counts. Closing it means making the synthetic data look more real.

Here: v1's base: 0.1% errors on synthetic lines, 58.3% on yours.

test set

Examples kept aside and never trained on, used only to score the model.

Changing it: It's the only honest measure of how a model does on new writing. Too small and the score is noisy; used for tuning, and it stops being honest.

Here: 15 of your English lines today; 30 planned, in the same language mix as your notes.

WebAssembly

A compact code format that browsers run at near-native speed.

Changing it: It lets the model run in the page, on your machine; threads and SIMD make it faster.

Here: The browser p95 budget of 50 ms is measured in WebAssembly.

weights

The numbers inside the model that training adjusts. A trained model is just these numbers plus the code that uses them.

Changing it: More weights can learn more, but need more data and training time, make the file bigger and reading slower. They grow with the square of the layer width.

Here: v0: 0.18 million (1 × 128). v1: 0.97 million (3 × 128). v2: 2.15 million (3 × 192), 8.6 MB in full precision.

WER

Word error rate: the same as CER, counted in whole words.

Changing it: Harsher than CER: one wrong letter makes the whole word wrong. Closer to how a reader judges a note.

Here: v2 personalised: 40.2%, against 13.1% CER.

Zipf's law

The n-th most common word is used about 1/n as often as the most common one.

Changing it: Drawing words by raw count would fill lines with "the" and "de"; drawing by the square root of the count keeps rare words in play.

Here: Both English and French Tatoeba follow it closely.