BSLScribe

The idea

Automatic captions exist for speech, but not for sign language. I wanted to find out how far a small model can get at turning BSL into English captions, using public BSL vlogs and a hobby budget.

Building the corpus

The first part is Sign, a collector. It finds face-on BSL videos on YouTube and Instagram and downloads them with their captions. It keeps only clips with one frontal signer whose hands move. Then it turns each video into hand, arm, shoulder and mouth points for every frame, paired with the caption for that moment. Human-made captions come first. Burned-in captions are read with OCR. Every stage writes to a SQLite manifest, so a run picks up where it stopped.

The result is 13,653 segments from 327 videos and 101 channels: 25.6 hours of signing at 25 frames a second. One clean-up step found burned-in channel watermarks in 823 captions and removed them all.

How it works

1. CollectSign finds and downloads BSL vlogs, and keeps only clips with one face-on signer.
2. Extract landmarksMediaPipe turns each frame into hand, arm, shoulder and lip points.
3. Encode the motionA small transformer encoder (3.76 million parameters) reads the pose sequence.
4. Hand it to a language modelA projector turns the encoding into soft tokens for Llama 3.2 1B. LoRA trains about 1% of the model.
5. Train in stages on ModalOptional masked-pose pretraining, then the encoder, then LoRA, on L4 and A10G GPUs. A full run costs $4–10.

What the measurements say

A probe that has seen the training clips can tell two clips from the same recording apart 99.4% of the time. So the landmarks do carry the signing.

On recordings it has never seen, that falls to about 0.61, and the full pipeline scores 0.507, which is chance. Open-domain translation does not work yet.

What failed, and why

The project notes record twelve designs and fixes, and the measurement that ruled out each one. The most useful lessons:

• A test that read the clock, not the hands. To check that the model reads pose, I swapped in another clip’s pose and looked for a drop in score. But the other clip brings its own length, and clip length predicts caption length. Once the swapped clips were matched on length, the earlier gains disappeared. That one fault invalidated a large share of the early results.
• Signer identity is everywhere. A simple probe can name the signer from the pose alone. Fixing a real 3.2× distortion between landscape and portrait uploads, and normalising the shoulders and arms, barely moved it. Dozens of other cues remain.
• Richer inputs did not help. SHuBERT features, DINOv2 image crops and a re-extraction at 25 frames a second all scored the same as plain landmarks.
• Real bugs, found on the way. A 46× scale mismatch between the soft tokens and Llama’s own embeddings, and padding that leaked through the attention mask.

Where it goes next

Open translation from 25 hours of vlogs is a research programme, not a side project. A narrow domain is within reach: the PHOENIX weather-forecast dataset reaches useful scores with only eleven hours, because forecasts use a small vocabulary.

So the next step is smaller. Weakly supervised word spotting already finds some words on signers it has never seen, such as “welcome”, “learn” and “hello”. It is now the base for a tool that searches for a sign by example.

What I learned

• Test the test first. The most expensive bug was in the measurement, not in the model.
• Set the pass mark before the run. It stops a messy result from turning into one more small tweak.
• Treat GPU spend as a design input. At $4–10 a run on Modal, the whole project fits in a $100 budget.