RAG Corpora Without Slop

A useful RAG corpus is not a pile of chunks. It is a controlled export with clean boundaries, source trails, confidence scoring, and duplicate pressure.

Most teams figure this out too late. They build the retriever first, tune the prompt second, and treat the corpus as an afterthought. The result is a system that answers confidently and cites poorly, or retrieves aggressively and grounds weakly. The fix is not better prompting or a different embedding model. The fix is treating corpus preparation as dataset engineering.

workspace % npx --yes alys-akusa@latest prepare ./docs --yes --workspace ~/Alys

TL;DR

A RAG system is only as good as the corpus it retrieves from.

Bad RAG data usually fails in four places:

Alys prepares retrieval corpora as auditable data products: chunked, deduped, sourced, scored, and exported in a shape a retriever can consume directly.

What slop actually means

Slop is what happens when retrieval data has no discipline.

RAG fails quietly when the corpus is sloppy. The search layer may return something. The model may write something. The answer may even sound plausible. But if the chunk was badly cut, duplicated, stale, or unsourced, the system is running on weak ground. You will not catch this in a demo. You will catch it when a user asks a precise question and the retriever hands the model a fragment that contradicts itself, lacks context, or comes from an unverifiable source.

Alys treats RAG preparation as dataset engineering, not text splitting. The goal is not to maximize the number of chunks. The goal is to produce chunks that are retrievable, inspectable, and useful.

Bad corpus smell:

• answer looks confident
• source is unclear
• chunk starts mid-thought
• same paragraph appears 19 times
• retrieval cannot separate evidence from noise

If you have ever stared at a RAG response and wondered "where did it get that," you have smelled slop.

Chunk design is interface design

A chunk is a contract with the retriever. It promises: if you select me, I will contain enough information to be useful, and little enough noise to be specific.

This contract breaks when chunks are sized by字符 count instead of semantic boundaries. It breaks when overlap is treated as a padding parameter instead of a continuity strategy. It breaks when metadata is stripped because "the vector database only stores vectors."

Small enough to retrieve

A chunk should be focused enough that the retriever can select it for a specific question without dragging in unrelated sections.

In practice, this means a chunk about database connection pooling should not also contain a paragraph about Kubernetes networking. If it does, the retriever has no way to rank the chunk accurately for either topic. The embedding gets averaged across unrelated concepts. The similarity score becomes a compromise.

The right size depends on your content. API documentation tends to chunk cleanly at the endpoint level. Legal contracts need clause-level granularity. Transcripts need speaker-turn boundaries. One size does not fit all, and the splitter should know the difference.

Large enough to answer

A chunk should carry enough surrounding context to answer without forcing the model to guess what the source meant.

If a chunk contains "set the timeout to 5000" but omits that this applies only to the legacy API, the model will generalize incorrectly. If a chunk captures a warning but not the condition that triggers it, the model cannot use the warning accurately.

Context is not padding. It is the difference between a retrievable fact and a usable fact.

Overlapped deliberately

Overlap is not decoration. It keeps meaning intact when a boundary would otherwise split a sentence, procedure, or argument.

We overlap at semantic boundaries, not at fixed token offsets. If a boundary falls inside a code block, we extend the chunk to the closing fence. If it falls inside a table row, we complete the row. If it falls inside a legal cross-reference, we include the referenced clause.

This costs space. It is worth it. A chunk that starts with "however, this does not apply" because the previous sentence was cut off is a chunk that will mislead the model.

Tagged aggressively

Good chunks carry metadata that makes the corpus debuggable:

• file path: where the chunk came from
• source title: human-readable identifier
• URL: when the source is web-based
• page / section: when the source has structure
• timestamp: when the source was captured or updated
• hash: content checksum for deduplication and versioning
• domain: for filtering and provenance
• confidence: how strongly the source supports the claim
• support signals: whether the chunk is primary source, inferred, or quoted

Metadata is not overhead. It is how you answer the question "where did it get that" without reading the entire corpus.

The chunk contract looks like this:

If any step is missing, the vector database has to compensate for a data problem it cannot actually solve.

Common damage and how it hides

The wrong chunking strategy creates false confidence. The system looks like it works because it produces answers, and the answers look right because they are fluent. The problem is that fluency is not accuracy, and retrieval coverage is not retrieval quality.

Blind splitting

The corpus cuts through tables, legal clauses, speaker turns, or code blocks and loses meaning.

We see this when teams use a generic text splitter with a fixed token count. Tables become unreadable. Code blocks lose syntax. Dialogues attribute quotes to the wrong speaker. The embedding model receives garbage, and the generator receives worse.

The fix is format-aware splitting. Tables should be chunked as tables. Code should be chunked at function or block boundaries. Transcripts should be chunked at speaker turns. This requires the splitter to inspect the content, not just count tokens.

No deduplication

The same paragraph appears everywhere and retrieval overweights repeated text.

Duplication is common in documentation, where the same warning appears on every page. It is common in scraped data, where boilerplate headers and footers propagate across every article. It is common in logs, where the same error message repeats.

Without deduplication, the retriever sees five identical chunks and concludes the content is five times as important. The generator sees repeated text and treats it as stronger evidence. Neither is correct.

Deduplication should be content-based, not path-based. Two chunks from different URLs with identical text should be flagged. Two chunks with near-identical text should be downweighted or merged.

No provenance

The model can answer, but nobody can inspect or cite the source.

Provenance is not a compliance checkbox. It is a debugging necessity. When a RAG system hallucinates, the first question is always: what did it retrieve? If the answer is "chunk 847 from unknown source," you cannot fix the problem.

Every chunk should carry a source trail back to the original document. The trail should include URL or file path, section or page, and capture timestamp. The trail should be stable: if the source changes, the chunk hash should change, and the old chunk should be invalidated or versioned.

No suitability signal

The team ships a corpus without knowing whether it is usable for RAG, evals, or training.

Not every chunk in a corpus is suitable for retrieval. Some are too short to be meaningful. Some are too long to be specific. Some are noise: navigation links, copyright notices, cookie banners that survived scraping.

A corpus should carry suitability scores that tell the operator what fraction of chunks are ready to serve, what fraction need cleanup, and what fraction should be dropped. Without this signal, you are flying blind.

Why this damage is invisible

Bad RAG data does not always break loudly. It produces answers that are close enough to pass a demo and vague enough to be hard to debug. The failure shows up later, when users ask precise questions and the retriever returns source fragments that cannot support the answer.

You will notice it in the metrics first: answer relevance drops on edge cases, citation accuracy declines over time, users start asking the same question twice because the first answer was incomplete. By then, the corpus is large and the team is committed to the current pipeline. Fixing it requires rebuilding the dataset from scratch.

This is why corpus preparation has to expose the mechanics: what was chunked, how much was accepted, which sources support the row, what was deduplicated, and where confidence is weak.

What a serious export looks like

The RAG file should carry more than text. It should carry enough structure that a human can inspect it, a retriever can consume it, and a debugger can trace it.

Text

The chunk body is the model-facing content. It should be coherent, bounded, and stripped of obvious duplicate noise.

Coherent means the chunk reads as a complete thought or procedure. Bounded means it has a clear start and end, not an arbitrary cutoff. Stripped of noise means boilerplate, navigation, and duplicate warnings have been removed or separated.

Metadata

The retrieval layer needs source, path, URL, format, hash, chunk id, and timestamps when available. Metadata is how you debug the answer.

A minimal metadata schema:

{
  "chunk_id": "uuid-v4",
  "source_url": "https://docs.example.com/api/pool",
  "source_path": "/docs/api/pool.md",
  "source_title": "Connection Pool Configuration",
  "section": "Timeouts",
  "format": "markdown",
  "hash": "sha256:abc123...",
  "captured_at": "2026-05-06T10:00:00Z",
  "domain": "docs.example.com"
}

The schema can be extended with confidence scores, deduplication flags, and content-type tags. The key is that every field is populated automatically during preparation, not filled in by hand later.

Readiness

Confidence, grounding, dedupe, and suitability scores tell the operator whether this corpus is ready to serve an agent or still needs cleanup.

• Confidence: how certain the pipeline is that the chunk is accurate and complete
• Grounding: whether the chunk is a primary source, secondary reference, or inferred content
• Dedupe flag: whether this chunk is unique, a duplicate, or a near-duplicate
• Suitability: whether the chunk meets minimum length, quality, and relevance thresholds

These scores should surface in reports, not be buried in JSON. The operator should be able to run prepare, scan the summary, and know whether the corpus is ready to ship.

Export format

The export should be a shape your retriever can consume directly. For most teams, this means JSONL or Parquet with a predictable schema. For vector databases, it means chunks paired with embedding payloads and metadata. For evaluation, it means a versioned snapshot with source hashes.

Alys produces a single artifact that satisfies all three: clean chunks, rich metadata, and deterministic hashes in a format that loads into any retriever.

What Alys Adds Before the Vector Store

The vector store should receive a prepared corpus, not a cleanup project.

Pre-launch checklist

What a serious RAG corpus should prove before it serves a single query:

• Every chunk has a source trail. You can point to the original document, section, and capture time for any chunk.
• Duplicate chunks are removed or downweighted. The retriever does not overweight repeated text.
• Chunk size and overlap are intentional. They vary by content type and are documented, not guessed.
• Tables, transcripts, code, and prose are handled differently when needed. The splitter respects structure.
• Warnings survive into reports instead of being hidden. The operator knows if sources failed, chunks were dropped, or confidence is low.
• The export has a shape your retriever can consume directly. No post-processing transforms required after every run.

If you cannot check every box, the corpus is not ready. Ship it and you are shipping slop.

The real takeaway

RAG quality starts before the vector database.

Vector databases are not magic. They retrieve what you give them. If the corpus is duplicated, source-blind, poorly chunked, or unsupported, the retriever will faithfully serve that mess to the model. The model will then faithfully generate an answer from it. The result will look like intelligence and behave like coincidence.

Alys prepares RAG corpora as audited data products: chunked, deduped, sourced, scored, and exported with enough metadata to debug what the model says next.

The command is short. The discipline behind it is not.

workspace % npx --yes alys-akusa@latest prepare ./docs --yes --workspace ~/Alys