What is training data?

The examples a model learns from. For a language model it is text; for a coding model it is source code, often with commit history and surrounding files. The model adjusts its parameters to predict that data, and everything it can do afterwards comes from what was in it.

Was my public GitHub code used as training data?

Almost certainly, if it was public for any length of time. Large permissively licensed code datasets have been assembled from public hosts for years, and they are a standard part of coding model training. Whether every use was lawful is being litigated; the collection itself is done.

Why do labs license data instead of scraping it?

Because the scrapable supply has been collected and does not contain what they need next. Private, production, long-maintained code was never on the public web. The only way to get it is from the people who own it, on terms.

Does synthetic data replace real code?

Not well. Models trained heavily on their own output learn the shape of clean code and lose the texture of real systems: the workarounds, the constraints, the decisions under pressure. Synthetic data is useful for augmentation and poor as a substitute.

Do labs pay for git history or just source?

Both, priced separately. History is the larger share of the volume and, in practice, of the payment. The dated figures are on the rates page.

Where does AI training data come from?

A model is a compressed version of what it was trained on. That makes the question of where the data came from the most important question about any model, and one that gets answered vaguely on purpose. Here is the concrete version, with the part that matters for code.

What training data is

The examples a model learns from. For a language model, text: books, web pages, forums, papers. For a coding model, source code, usually with the files around it and often with commit history.

Training adjusts the model’s parameters until it predicts that data well. Everything the model can do afterwards is a consequence of what was in the set. A model that has seen little of a kind of code is weak at that kind of code. That is not a limitation to be engineered around; it is what a model is.

The four sources

where training data comes from
  │
  ├── the public web ─────────► scraped. first, largest, now exhausted
  │
  ├── curated datasets ───────► assembled from public sources, filtered
  │
  ├── synthetic data ─────────► generated by models. cheap, weak for code
  │
  └── licensed from owners ───► negotiated. private, scarce, growing

The public web. Crawled at scale, filtered, deduplicated. For code, this means public repositories on the major hosts, package registries, forums, and documentation. It was the first source and by far the largest, and it has been collected continuously for years.

Curated datasets. Assembled from public sources with filtering by licence, quality, and language, and published for research use. Several large permissively licensed code datasets exist and are standard training inputs. They are still public code; curation changes the packaging, not the origin.

Synthetic data. Generated by an existing model, then used to train the next one. Cheap, unlimited, and controllable. Also, for code, weak, for reasons below.

Licensed from owners. Negotiated access to data that was never public. For text this means publishers and archives. For code it means the owners of private repositories, who grant a non-exclusive licence to train. This is the source that is growing, because it is the only one that contains what the others do not.

Why synthetic data works badly for code

The appeal is obvious: generate as much as you want, in whatever shape you want, with no licensing at all.

The problem is what it contains. A model generating code produces the code it thinks is likely, which is clean, idiomatic, and shaped like tutorials. Real production code is not shaped like that. It has the workaround for the vendor API that returns the wrong timezone. It has the config flag that nobody dares remove. It has the commit that tried the elegant approach, and the next commit that reverted it because the elegant approach did not survive contact with the data.

A model trained on synthetic code learns what good code looks like and learns nothing about how real code gets that way. Models trained heavily on their own output also degrade in measurable ways, converging on a narrow distribution and losing the tails. Synthetic data is useful for augmentation. As a substitute for real code it is poor, and the labs know it.

┌─ WORTH CHECKING ─────────────────────────────────────────────────────┐
│                                                                      │
│  Private, finished, owned by you: $500+ each to AI labs.             │
│                                                                      │
│  [ See what my repos are worth ]                                     │
│                                                                      │
└──────────────────────────────────────────────────────────────────────┘

Why public code is a biased sample

Public code is not a random sample of software. It is the software people chose to publish, and that choice is systematic.

It skews heavily towards libraries, frameworks, tooling, tutorials, and demonstrations. Things built to be shown. It under-represents, badly, the category most software actually belongs to: line-of-business applications, integrations, internal tools, and long-lived systems maintained for a decade by people who inherited them and never published a line.

A model trained on the public sample is good at the public sample. It writes a competent library. It is noticeably worse at maintaining a system whose context is not written down anywhere, which is most of what working developers do all day. That gap is the specific thing buyers of private code are paying to close.

The shift to licensing

So the sequence runs: the public web is collected, the curated datasets are collected, synthetic data is tried and found wanting, and the remaining source is the one that was never public.

That source cannot be scraped. It is on private hosts, behind access controls, owned by identifiable people and companies. The only way to get it is to license it, non-exclusively, from the owners. Which is what is now happening, at published per-token rates, with git history priced separately because history carries the how-it-got-that-way signal that a snapshot cannot.

What that comes to, dated and sourced, is on what AI labs pay for code. Why history in particular is where the value sits is in should you squash commits.

What this means if you own code

Two things, and they point in opposite directions.

Anything you published is in the first two sources already. It has been collected, it will be collected again, and there is nothing to license because nobody pays for what they have. That is done and it is not worth being upset about. It was the deal you accepted when you chose an open licence, and it is why your MIT-licensed repo is worth nothing.

Anything you never published is in the fourth source, or could be. It is scarce, it is representative of the code models are weakest at, and it has an owner who can be paid. That is most private repositories most developers have, and almost none of them have been valued. What has to be true for a repository to qualify is in can you license private repositories.