Datasets Need Debate

Extraction is not intelligence. A high-confidence dataset needs pressure: source review, disagreement, duplicate checks, and a final decision about what deserves to ship.

workspace % npx --yes alys-akusa@latest prepare ./knowledge-base --yes --workspace ~/Alys

TL;DR

Most bad datasets do not look bad. They look clean, structured, and perfectly valid until a model learns from them.

The fix is not prettier JSON. The fix is pressure:

Alys treats records as claims. A claim has to survive debate before it becomes training data, RAG context, or evaluation truth.

The Problem

Most dataset pipelines stop at the weakest step.

The basic extraction pipeline is simple: read files, pull facts, write JSONL. That is useful for demos, but it is not enough for production AI data. A model does not care that a row looked clean in a table. It cares what patterns the row teaches, and bad rows teach bad patterns with confidence.

Alys treats each output as a claim that needs evidence. The question is not only "can we extract this?" The better question is "should this survive into the dataset?"

The weak pipeline almost everyone starts with looks like this:

parse files
extract rows
save JSONL

That is three steps. It is fast. It ships demos. But it is missing the things that turn a raw scrape into infrastructure:

- source quality and provenance tracking
- contradiction detection and handling
- duplicate and near-duplicate pressure
- confidence scoring with actual reasons
- human review routing when machines disagree

A pipeline with these holes will produce JSONL files, but those files are not datasets you should hand to a training run or a RAG system. They are staging artifacts.

The real cost shows up later. A model trained on unsupported claims will hallucinate with authority. A RAG system retrieving ungrounded chunks will cite ghosts. An evaluation dataset with duplicated sources will inflate accuracy numbers and hide real failures. Fixing that downstream costs more than catching it upstream.

Alys exists to make the upstream catch the default, not the exception.

The Debate Loop

A record should win its place in the export.

Every row that lands in a final dataset should carry the burden of proof. Not just that it was extracted cleanly, but that it stood up to scrutiny. The debate loop is four explicit stages. Skip one, and the system is lying to itself about confidence.

1. Extract

Pull the candidate: the fact, instruction pair, chunk, or answer. This is the easy step. Extraction is a solved problem in the sense that any pipeline can open a PDF and emit text blocks. But extraction is only the admission ticket, not the verdict.

In practice, extraction should also capture the immediate context: the surrounding paragraph, the section header, the page number, the file path, the source URL. It is tempting to strip all of that away to save space. Do not strip it. That context is the first line of defense when a downstream reviewer asks later, "Where did this come from?"

2. Challenge

This is the step most pipelines skip entirely. Ask what would make the candidate wrong. Build explicit challenge checks into the pipeline, not as human review queues but as mechanical gates:

- Missing date or version: Does the claim depend on temporal information that is absent?
- Weak source: Is the originating document low credibility, outdated, or primary/secondary in the wrong direction?
- Duplicate or near-duplicate source: Is the same paragraph, slide, or page represented multiple times across files?
- Unsupported assumption: Does the claim interpolate, infer, or summarize beyond what the source actually says?
- Conflicting evidence: Is there another row or source that directly contradicts this one?
- Hallucinated framing: Does the extracted instruction-answer pair impose a premise that the source never agreed to?

Each challenge is a test. A healthy dataset pipeline logs the results for every row: passed challenge X, failed challenge Y, flagged for review on Z. When challenges are invisible, teams only learn about them when the model misbehaves in production.

3. Support

If a row survives the challenge stage, attach the source trail that justifies it. Not a vague citation. A real trail a human can follow:

- File path (if local)
- URL (if web)
- Document hash or version id (for immutable reference)
- Transcript offset, page number, or chunk id (for precise location)
- Source rank or relevance score (so downstream systems know how strong the link is)
- Related evidence (other chunks that support or contextualize the claim)

The goal is debuggability. When a bad row surfaces in eval or in production, the support trail should allow a team to reconstruct how that row entered the dataset, what source it rested on, and why the system accepted it.

Support fields also unlock downstream filtering. A RAG system can choose to only retrieve rows with source rank above a threshold. A fine-tuning run can exclude records whose source trails point to documents below a confidence cutoff. Without support data, the dataset is a black box.

4. Decide

Accept, lower confidence, route to human review, or reject. A pipeline that accepts every candidate is not a data pipeline. It is a file converter with delusions of rigor.

The decision can be automated or human, but it must be explicit. Alys uses decision rules based on challenge results and support quality:

- Pass all challenges plus strong support: accept into the export
- Pass challenges but weak support: accept with lowered confidence flag
- Fail minor challenges: route to human review queue
- Fail major challenges or show strong contradiction: reject

The important part is that the decision is recorded. A rejected row still lives in the database, with its rejection reason. Auditors can inspect what never made it, not just what shipped. This is how a dataset becomes defensible.

The full loop is intentionally uncomfortable:

If this feels slower than scraping and shipping, that is because it is. The point is to move the cost to the preparation step, where failures are still cheap.

What Breaks

Clean rows can still be low-trust rows.

The biggest trap in dataset building is confusing formatting quality with dataset quality. A row can have valid JSON, perfect casing, a well-formed schema, and a pleasant key order while still being entirely unsupported. The pipeline is happy. The model will suffer.

Debate is how the system refuses to be impressed by clean output.

If two sources disagree, the dataset should say so. It should not throw one away silently. Silence trains the model that consensus exists where it does not. The right behavior is to keep both, flag the contradiction, and expose that tension to downstream systems or reviewers.

If a claim depends on one stale page, confidence should fall. Not because the page is definitely wrong, but because the evidence is thin. The model should not learn that claim with the same weight as a triple-sourced, cross-validated fact.

If the same paragraph shows up ten times across scraped pages, the dataset should not count it as ten independent facts. That duplication inflates statistical weight and makes claims look better supported than they are. A deduplication pass is not a nice-to-have. It is a correctness requirement.

Common problems we see in real pipelines:

No source
The row may be true, but nobody can inspect why. In eval datasets, this makes error analysis impossible. In training data, this means unsupported patterns get baked into weights with no accountability.

One weak source
The model learns a claim without knowing it came from thin evidence. A single product changelog, a single blog post, a single Reddit thread: unsupported sources are everywhere. They need flagging, not auto-inclusion.

Duplicated evidence
Five copied pages can look like five independent confirmations. Statistical learners count instances. If the same evidence appears five times, it gets five times the weight. Deduplication at the source or chunk level prevents this skew.

No contradiction field
The export hides conflict instead of teaching the workflow to handle uncertainty. This is dangerous. It means eval datasets show false consensus. It means RAG systems retrieve answers without mentioning the other side. It means fine-tuning data implicitly resolves debates the source material never resolved.

Missing confidence reasons
A numeric score without an explanation is a dashboard toy, not engineering data. Teams need to know why a row scored 0.3 or 0.9. Was it source weakness? Contradiction? Near duplication? Sparse context? The score alone teaches nothing.

The Alys Position

Confidence is an argument, not a decoration.

Provenance

Each accepted row carries enough source context to inspect where it came from. Paths, URLs, chunks, hashes, version ids, and support chains make the dataset debuggable.

Provenance is not just for regulators. It is for the engineer who deploys a model and sees a weird behavior in staging. They need to trace the bad pattern back to its origin, fix the source, and regenerate the dataset without throwing away the whole pipeline. Without provenance, that trace takes days. With it, it takes minutes.

Contradictions

Conflicts do not disappear just because a model can summarize around them. Alys surfaces weak signals so downstream teams can decide whether to suppress, split, or review.

A contradiction is a feature, not a bug, when it is explicit. It tells you the data is honest about uncertainty. Teams can choose to build separate models per position, to train with uncertainty-aware loss, or to present both sides to a human user. Silenced contradictions remove those choices.

Readiness

Scores are not meant to make the dashboard pretty. They are meant to tell you whether the export is ready for:

- Retrieval-Augmented Generation (RAG): requires grounded, non-contradictory chunks with clear provenance
- Evaluation datasets: requires balanced, deduplicated rows with explicit confidence and support
- Fine-tuning: requires high-confidence, well-supported instruction-answer pairs with minimal noise
- Human review: anything that fails auto-accept but might be salvageable
- Nothing yet: contradictions, missing sources, or low credibility that should stay in draft until fixed

A score without a readiness category is a number without a contract. Alys ties the two together so teams know exactly what they can do with the export.

What This Means for Engineering Teams

If you are building or maintaining a dataset pipeline, there are practical questions to ask about its current state.

Can you trace any row in the export back to the exact source paragraph, file, or URL?
If the answer is no, your pipeline is a content mover, not a data system.

Does your pipeline have an explicit contradiction database?
If conflicts are resolved by arbitrary last-write-wins, your model is learning fake consensus.

How do you handle near-duplicates?
If duplicate text gets multiple unique IDs, your dataset is lying about independence.

What happens to rejected rows?
If they disappear, you cannot audit what the pipeline chose to exclude. That makes the export untrustworthy.

Are confidence scores explained?
If a reviewer sees 0.7 and cannot determine whether that is about source quality, context length, or duplication, the score is noise.

Is there a human review queue for borderline cases?
If every row is auto-accepted or auto-rejected, you will miss the cases where a human can fix a salvageable record in thirty seconds.

These questions are not about adding bureaucracy. They are about preventing the subtle, expensive failures that only show up after training or deployment.

Minimum Standard for a Serious Export

Before a dataset leaves staging, it should be able to answer this checklist:

This is the bar. Anything below it is not necessarily useless, but it should be labeled as a draft.

Takeaway

The dataset is the argument you hand to the model.

If a dataset cannot explain why a row exists, it is not ready to influence a model. It may still be a draft, a scrape, or a staging artifact, but it is not infrastructure.

Alys exists for the uncomfortable middle step between raw files and model-ready data: the step where records are challenged, sources are attached, duplicates are removed, warnings are preserved, and the export becomes something a serious team can audit.

Extraction is the beginning. Debate is the work. The dataset that ships should be the one that survived it.