Synthetic Data

What it is

Synthetic data is information generated by algorithms — not captured from the real world — that replicates the statistical properties and structure of real data. In the AI context, it is used to train models, build evaluation benchmarks, and test systems when real data is insufficient, expensive to obtain, or contains sensitive information.

The practice has become central to modern LLM development. Models like DeepSeek-R1 and the reasoning families from OpenAI and Anthropic use synthetic data extensively during post-training, either as fine-tuning data or as judgments generated by evaluator models.

Generation methods

Model distillation

A large, capable model generates data used to train a smaller model. The "teacher" model produces high-quality responses that the "student" model learns to replicate.

Self-Instruct

Technique introduced by Wang et al. (2022) and popularized by Stanford Alpaca. An LLM generates instruction-response pairs from a seed set of examples. The process is iterative: each generated batch is filtered for quality and added to the pool to generate more variations.

import json
from openai import OpenAI
 
client = OpenAI()
 
SEED_TASKS = [
    {"instruction": "Explain what a load balancer is", "output": "..."},
    {"instruction": "Write a Python function that validates an email", "output": "..."},
]
 
def generate_instructions(seed_tasks: list[dict], n: int = 10) -> list[dict]:
    seed_text = "\n".join(
        f"- Instruction: {t['instruction']}" for t in seed_tasks
    )
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": (
                f"Here are example instructions:\n{seed_text}\n\n"
                f"Generate {n} new and diverse instructions in the same style. "
                "Cover different domains and difficulty levels. "
                "Respond in JSON: [{\"instruction\": \"...\", \"output\": \"...\"}]"
            ),
        }],
        response_format={"type": "json_object"},
    )
    return json.loads(response.choices[0].message.content)["instructions"]

Data augmentation

Transforming existing data to create variations: paraphrasing, translation, format changes, numerical value perturbation.

Adversarial generation

Creating data designed to expose model weaknesses: edge cases, adversarial prompts, security scenarios. This connects directly to AI safety practices.

Use cases

Quality validation

Generating synthetic data without validating it is a risk. A robust pipeline includes at least three checks:

1. Statistical fidelity: compare distributions of generated data against real data using metrics like Jensen-Shannon divergence or Maximum Mean Discrepancy (MMD).
2. Downstream utility: train a model on synthetic data and compare its performance against one trained on real data on the same benchmark. The acceptable gap depends on the domain, but a degradation greater than 5% on the primary metric usually indicates quality issues.
3. Leakage detection: verify that synthetic data does not contain verbatim copies from the generator model's training data. Techniques like membership inference tests help detect memorization.
4. Diversity: measure input space coverage using n-gram diversity metrics or embedding clustering to detect mode collapse.

Risks and limitations

• Model collapse: recursively training models on synthetic data can degrade quality generation after generation. Shumailov et al. (2023) demonstrated that iterative training on self-generated data produces a progressive loss of the tails of the original distribution.
• Amplified bias: synthetic data inherits and can amplify the biases of the generating model
• Lack of diversity: generated data tends to be less diverse than real-world data
• Hidden cost: generating high-quality data with large models has significant API cost — a 50K example dataset with GPT-4o can cost hundreds of dollars

Why it matters

As AI systems move from prototypes to production, the quality of training and evaluation data becomes the bottleneck. Real-world data is expensive, slow to collect, and often restricted by privacy regulations. Synthetic data breaks this dependency: teams can iterate on evaluation suites in hours instead of weeks, test edge cases that rarely occur naturally, and build datasets for domains where real data simply doesn't exist yet.

References

• Self-Instruct: Aligning Language Models with Self-Generated Instructions — Wang et al., 2022. Foundational method for generating instruction data with LLMs.
• The Curse of Recursion: Training on Generated Data Makes Models Forget — Shumailov et al., 2023. Demonstration of the model collapse phenomenon when training on synthetic data.
• A Deep Dive Into the Role of Synthetic Data in Post-Training — Analysis of synthetic data use in LLM post-training, 2025.
• Evaluating Language Models as Synthetic Data Generators — Evaluation of LLMs as synthetic data generators, 2024.
• Best Practices and Lessons Learned on Synthetic Data — Liu et al., 2024. Best practices for synthetic data generation.