What producing 50 training items a day taught me about LLMs

For four months at Zhipu I did something that sounds almost quaint next to agent frameworks and billion-parameter training runs: I made training data for GLM-series models. Roughly fifty curated items a day — multi-turn dialogues, instruction-following tasks, code problems, reasoning traces — each one written, validated, and passable enough to join a dataset that would eventually shape a model's behaviour.

Here's what surprised me, and what I'd tell anyone about to do the same.

The model learns your errors, not your intentions#

A single mislabelled example is invisible in aggregate metrics and extremely visible in the fine-tuned model three weeks later. The models are faithful students: if 3% of items reward a confident-but-hollow answer, you will meet a confidently hollow model.

This inverted my mental model of data work. The job isn't "produce examples" — it's produce a dataset whose distribution of errors is acceptable. That's an engineering problem, and it starts with tooling.

Dedup is not optional, and exact-match isn't enough#

Duplicated items skew the loss toward whatever they teach. My preprocessing pipeline ran:

def clean_batch(items: list[dict]) -> list[dict]:
    # 1. Format validation: every field present, structurally valid
    items = [i for i in items if validate_schema(i)]
 
    # 2. Exact dedup on prompt fingerprint
    seen: set[str] = set()
    unique = []
    for item in items:
        key = hashlib.sha256(normalize(item["prompt"]).encode()).hexdigest()
        if key not in seen:
            seen.add(key)
            unique.append(item)
 
    # 3. Near-dup check: paraphrased prompts with identical answers
    #    (flags copy-paste-with-a-tweak patterns for manual review)
    unique = flag_near_duplicates(unique, threshold=0.92)
 
    # 4. Quality heuristics: length bounds, refusal patterns, placeholder text
    return [i for i in unique if passes_heuristics(i)]

The near-duplicate stage caught the sneakiest class of junk: the same underlying task reworded five different ways. Individually fine; collectively a bias.

Build the error taxonomy early#

Sampling batches for manual review is where the real signal lives — but "reviewing" without structure degenerates into vibes. I kept a taxonomy with four families and forced every issue into it:

The taxonomy only earns its keep when it's actionable: each family mapped to a specific guideline change or filter rule. "Logical inconsistencies up 2× in the code subset this week" turned into a concrete fix — stronger step-verification examples in the batch mix — instead of a mood.

Distribution is a feature#

It's tempting to fill quotas with whatever's easiest to write. But dataset composition — the ratio of code to dialogue to reasoning, of Chinese to English, of easy to hard — is effectively a product decision about the model's personality. I started every batch from a target distribution and tracked drift through the week, because Friday-me drifting from Monday-me was a real phenomenon.

What carries over to building applications#

The internship changed how I build with LLMs day to day:

• Eval sets are the same discipline as data QA, smaller. A handful of labelled cases catches regressions.
• Failures cluster. When something breaks in production, classify it — the taxonomy is how you find the 20% of causes behind 80% of incidents.
• Models are downstream of their data forever. Fine-tuning on your own outputs without hygiene is how a model learns to launder its own mistakes.

Data work isn't the unglamorous cousin of model work. It is model work — just at the layer where a single well-designed example can still change something.