Documentation

user@machine repo % npx alys-akusa prepare ./massive-raw-data --profile all

Drop in massive raw data. Get AI-ready datasets.

Alys is terminal-native AI data preparation infrastructure. You point it at messy knowledge, Alys parses, chunks, deduplicates, grounds, evaluates, and exports AI-ready artifacts locally. The dashboard remains the observability and account layer.

Terminal is the runtime

Alys runs from your shell. The website does not execute preparation jobs; it observes what the CLI completed.

Dashboard is observability

Completed runs can sync usage, readiness scores, manifests, export formats, and preparation history to your account.

Local-first by default

Preparing files and folders runs locally, writes local artifacts, and does not require users to bring model API keys.

Outputs are local

AI-ready artifacts are written to your chosen workspace, usually ~/Alys/prepared/{run_id}.

Installation

Run Alys without installing a desktop app.

The recommended path is npx. It runs the published CLI package and writes prepared artifacts to your local filesystem.

snippet

npx alys-akusa

You can run explicit preparation commands with the same package. Login is useful for dashboard observability, but local preparation writes artifacts directly to your machine.

CLI Login

Connect the terminal once.

Alys uses a browser-based CLI login flow. The browser confirms the account; the terminal receives a CLI token and stores it locally.

Run login

Alys opens the browser and waits for the dashboard to authorize the terminal.

Token stored locally

The CLI token is stored at ~/.alys/config.json.

snippet

npx alys-akusa login
npx alys-akusa status

First preparation run

Prepare AI-ready data from a local folder.

Start with real source material: docs, PDFs, support exports, transcripts, CSVs, JSON files, or code. Alys turns the corpus into training, RAG, and eval artifacts.

snippet

npx alys-akusa prepare ./company-docs --profile all --yes

Use --yes for non-interactive runs. Without it, Alys asks for the input path and confirms the local preparation plan before writing files.

Usage limits

Local preparation is built for large corpora.

The preparation runtime runs on your machine, streams progress, and does not spend hosted credits. Account limits remain visible for dashboard and future cloud-sync workflows.

snippet

# Local preparation
npx alys-akusa prepare ./company-docs --concurrency 12

# Fine-tuning focused exports
npx alys-akusa finetune-ready ./support-tickets --profile openai,anthropic,qa

Runtime

How the Alys preparation runtime works, step by step.

This is the core local pipeline: ingestion, parsing, chunking, deduplication, grouping, grounding, structuring, evaluation, and export.

Alys indexes the input

The CLI accepts a file or folder and recursively indexes supported documents while skipping build artifacts, caches, node_modules, and hidden VCS folders.

npx alys-akusa prepare ./company-docs

Files are parsed in parallel

Markdown, TXT, JSON, JSONL, CSV, HTML, structured XML, logs, subtitles, code files, transcript-backed audio/video, and best-effort PDFs are normalized with source paths preserved.

documents + media -> normalized text

The corpus is chunked

Alys splits massive text into bounded chunks with overlap so downstream RAG and fine-tuning exports have usable context windows.

--chunk-tokens 420 --chunk-overlap 48

Duplicate content is removed

Exact content hashes and near-duplicate checks suppress repeated support answers, copied docs pages, boilerplate, and noisy repeated snippets.

hash + near-match dedupe

Chunks are semantically grouped

Keyword clustering groups related knowledge so the final exports are easier to inspect, score, retrieve, and evolve.

chunks -> semantic groups

Alys canonicalizes records

Low-quality chunks are filtered out and surviving chunks are converted into stable records with provenance, confidence, and metadata.

{ text, source_path, group, confidence }

Grounding is scored

Each chunk keeps its source path and gets a grounding score so users can trace output back to local knowledge instead of trusting a black box.

source_path + grounding_score

Export profiles are written

Alys writes OpenAI fine-tuning JSONL, Anthropic instruction JSONL, RAG chunks, QA records, conversation examples, eval datasets, and embeddings-ready corpora.

openai-finetune.jsonl + rag-chunks.jsonl

AI readiness is evaluated

The report scores knowledge quality, fine-tuning suitability, RAG suitability, eval suitability, grounding, structure, diversity, and throughput.

ai-readiness-report.json

Artifacts stay local

The CLI writes everything into your selected workspace so teams can use the files directly in training, retrieval, evaluation, and review pipelines.

~/Alys/prepared/{run_id}/

Modes

Choose the right preparation command.

Alys has focused commands for different source material and export goals.

prepare

Full local pipeline for messy folders and mixed corpora. Best default for company docs.

ingest

Lightweight corpus indexing for RAG chunks, embeddings, and eval preparation.

finetune-ready

Focused export path for OpenAI/Anthropic-style training and QA records.

simulate-rag

Deterministic retrieval test for hit rate, overlap failures, weak boundaries, and hallucination-prone gaps.

improve

Writes a safe optimized corpus copy with provenance headers, cleaner chunk boundaries, and improved RAG chunks.

benchmark

Compares original vs improved retrieval behavior, answer support, hallucination risk, and regressions.

Media + transcripts

Pair audio, video, captions, and transcripts before they poison the dataset.

Messy media corpora fail when clips, captions, speakers, timestamps, and transcript sidecars drift apart. Alys inventories the files, auto-pairs common transcript formats, chunks by time, and writes reviewable media artifacts.

snippet

npx alys-akusa media-debug ./call-recordings --yes
npx alys-akusa media-transcribe-plan ./call-recordings --gcs-input gs://bucket/in --gcs-output gs://bucket/out --yes

Auto-pair

Match audio/video with VTT, SRT, transcript text, and provider outputs.

Chunk by time

Keep cue indexes, speakers, time ranges, media fingerprints, and linked chunk IDs.

Review before export

Generate debug reports, redaction review rows, and dataset cards for governance.

Quality evaluation

Alys measures data worth keeping.

Every preparation run returns quality artifacts so the output is inspectable instead of a black box.

snippet

npx alys-akusa audit ./knowledge-base
npx alys-akusa simulate-rag ./knowledge-base --queries 12 --top-k 5
npx alys-akusa improve ./knowledge-base
npx alys-akusa benchmark ./knowledge-base --queries 16 --top-k 5

Audit mode runs the ingestion, parsing, chunking, dedupe, grounding, grouping, and evaluation stages, then skips bulky export profiles. Simulate RAG uses the prepared chunks to test whether realistic queries retrieve grounded, non-overlapping evidence before you ship a retrieval system. Improve writes a safe optimized copy of the corpus; it never edits your original files. Benchmark compares original vs improved behavior so you can prove whether the optimized corpus actually helped.

knowledge quality

before/after readiness benchmarks

recommended fixes

duplicate reduction

semantic grouping diversity

retrieval separation

hallucination risk proxy

eval coverage

RAG suitability

instruction tuning suitability

eval suitability

grounding score

throughput tokens per second

Credit accounting

Local preparation does not spend hosted credits.

The prepare, ingest, and fine-tune-ready commands operate on local files and write local artifacts. Hosted credits belong to account-linked cloud workflows, dashboards, and future sync features.

Individual

100 hosted credits

For one builder preparing clean RAG and training data from real files.

Professional

1,000 hosted credits

For serious operators running repeatable dataset, evaluation, and collaboration workflows.

Business/Enterprise

2,500 hosted credits

For organizations that need unlimited seats, governed pipelines, deployment control, and dedicated ownership.

snippet

local input
  -> parse files
  -> chunk corpus
  -> deduplicate
  -> evaluate readiness
  -> write local exports

Local storage

Your files stay in your workspace.

The dashboard stores history and metadata. The actual export files are written by the CLI to your machine.

snippet

~/Alys/
  prepared/
    {run_id}/
      openai-finetune.jsonl
      anthropic-instruction.jsonl
      rag-chunks.jsonl
      qa-dataset.jsonl
      conversations.jsonl
      eval-dataset.jsonl
      embeddings-corpus.jsonl
      media-manifest.json
      media-segments.jsonl
      media-redacted-segments.jsonl
      media-debug-report.json
      media-debug-report.md
      media-dataset-card.json
      media-dataset-card.md
      media-redaction-review.jsonl
      ai-readiness-report.json
      ai-readiness-report.md
      manifest.json
      sources.json
  media-transcription/
    {run_id}/
      media-transcription-plan.json
      media-transcription-plan.md
      google-media-upload-manifest.csv
      audio-extraction-commands.txt
      google-speech-submit-commands.txt
      google-transcript-import-map.jsonl
      google-speech-batch-requests/

CLI reference

Commands you actually need.

The CLI is intentionally small. Most users need prepare, ingest, finetune-ready, login, status, and datasets.

Show CLI helpnpx alys-akusa

Check accountnpx alys-akusa status

List synced runsnpx alys-akusa datasets

Audit readinessnpx alys-akusa audit ./knowledge-base

Simulate RAGnpx alys-akusa simulate-rag ./knowledge-base --queries 12 --top-k 5

Improve corpusnpx alys-akusa improve ./knowledge-base

Benchmark qualitynpx alys-akusa benchmark ./knowledge-base --queries 16 --top-k 5

Prepare a foldernpx alys-akusa prepare ./company-docs --profile all

Ingest a knowledge basenpx alys-akusa ingest ./knowledge-base --profile rag,embeddings,eval

Fine-tuning exportnpx alys-akusa finetune-ready ./support-tickets --profile openai,anthropic,qa

Debug media datasetsnpx alys-akusa media-debug ./call-recordings --yes

Plan Google media transcriptionnpx alys-akusa media-transcribe-plan ./call-recordings --gcs-input gs://bucket/in --gcs-output gs://bucket/out --yes

Fast non-interactive prepnpx alys-akusa prepare ./docs --chunk-tokens 420 --concurrency 12 --yes

Exports

What Alys writes after a run.

Each file exists for a reason: model training, RAG ingestion, human review, dashboard observability, or benchmark analysis.

openai-finetune.jsonl

Chat-style messages for OpenAI fine-tuning pipelines.

anthropic-instruction.jsonl

Anthropic-style instruction records built from grounded local context.

rag-chunks.jsonl

Retrieval-ready chunks with source path, title, group, and metadata.

qa-dataset.jsonl

Question/answer records with context and confidence metadata.

conversations.jsonl

Conversation-style examples for assistant behavior training.

eval-dataset.jsonl

Evaluation prompts, ideal answers, context, rubric, and difficulty.

embeddings-corpus.jsonl

Chunk text and metadata ready for embedding/indexing pipelines.

media-manifest.json

Audio/video dataset inventory with transcript coverage, cue counts, speakers, SHA-256 fingerprints, diagnostics, and chunk counts.

media-segments.jsonl

Timestamp-level media rows with cue indexes, speakers, time ranges, text, and linked chunk IDs.

media-redacted-segments.jsonl

Share-safer timestamp-level media rows with obvious direct identifiers redacted.

media-debug-report.json

Machine-readable media dataset debug report for missing transcripts, timing issues, probe failures, and readiness status.

media-debug-report.md

Human-readable media debug report with prioritized fixes for audio/video corpora.

media-dataset-card.json

Machine-readable media readiness card with governance, suitability, rights, privacy, and coverage summaries.

media-dataset-card.md

Human-readable media dataset card for downstream review and handoff.

media-redaction-review.jsonl

Transcript privacy findings for manual redaction review before training or sharing.

media-transcription-plan.json

Google Speech-to-Text V2 transcription plan with GCS prefixes, model settings, status, warnings, and media-to-caption mapping.

google-media-upload-manifest.csv

Upload checklist mapping local media or extracted audio inputs to Cloud Storage URIs.

google-speech-batch-requests/*.json

BatchRecognize request payloads for Google Speech-to-Text V2.

google-transcript-import-map.jsonl

Map from provider outputs back to local VTT/SRT sidecar targets.

ai-readiness-report.json

Scores for knowledge quality, tuning suitability, RAG suitability, grounding, structure, and diversity.

ai-readiness-report.md

Human-readable readiness summary for review.

manifest.json

Run metadata: profiles, chunking settings, metrics, scores, and output directory.

sources.json

The parsed local source list with file paths, extensions, byte sizes, token estimates, and parser type.

retrieval-simulation.json

Deterministic RAG simulation with hit rate, overlap failures, unsupported-answer risk, and hallucination risk.

retrieval-simulation.md

Human-readable retrieval simulation report with failed queries and recommended fixes.

improvement-report.json

Before/after improvement report for optimized chunks, provenance, grounding, and retrieval behavior.

improvement-report.md

Human-readable summary of what Alys changed and what to do next.

improved-rag-chunks.jsonl

Safe optimized RAG chunks with provenance headers and cleaner retrieval boundaries.

cleaned-corpus.md

A readable cleaned corpus copy. Original source files are never mutated.

knowledge-benchmark.json

Before/after benchmark for retrieval hit rate, answer support, grounding, hallucination risk, and regressions.

knowledge-benchmark.md

Human-readable benchmark report for proving whether improvement actually helped.

audits/{run_id}/

Audit-only output folder for diagnostics without writing bulky export profiles.

improved/{run_id}/

Improved output folder for safe optimized corpus copies.

benchmarks/{run_id}/

Benchmark output folder for original-vs-improved quality comparisons.

benchmark

Before/after readiness estimates for duplication, retrieval separation, grounding density, format consistency, and eval coverage.

recommendedFixes

Prioritized remediation actions with expected impact.

Manifest

The manifest is the run receipt.

It records what was prepared, which export profiles were requested, what metrics were produced, and where artifacts live locally.

snippet

{
  "id": "25b8cee79f04",
  "input": "./company-docs",
  "profiles": ["openai", "rag", "eval"],
  "metrics": {
    "filesParsed": 18221,
    "tokensParsed": 4100000,
    "chunksAccepted": 42112,
    "duplicateRate": 0.32
  },
  "scores": {
    "fineTuningSuitability": 91,
    "ragSuitability": 88
  },
  "outputDir": "~/Alys/prepared/25b8cee79f04"
}

Dashboard sync

The dashboard is the observability layer.

Alys keeps the browser focused on history, usage, readiness metrics, artifact metadata, and account state. Execution stays in the terminal.

Usage

Account state, remaining limits, and future billing state.

History

Completed preparation runs with status, formats, and timestamps.

Observability

Readiness scores, manifests, local export paths, and source summaries.

Troubleshooting

Common fixes.

Most issues are either an old npm package, unsupported source files, a huge noisy folder, or checking the wrong local output folder.

CLI opens login but terminal does not connect

Run npx alys-akusa login again, complete the browser flow in the same browser profile, then wait for the terminal poller to finish.

Preparation finds no files

Check that the input path exists and contains supported file types such as .md, .txt, .json, .csv, .html, .pdf, or code files.

npx still uses an old CLI

Publish a new npm version, then run npx alys-akusa@latest. npm will not replace an already published version.

The run is slow

Increase --concurrency, reduce huge binary files, or point Alys at a narrower folder first. Build artifacts and node_modules are skipped automatically.

Exports are missing

Check the output path printed by the CLI. By default files are under ~/Alys/prepared/{run_id}/.

Release notes

Current runtime notes.

The current priority is local data preparation: high-throughput ingestion, deduplication, readiness scoring, and AI-ready exports.

snippet

current runtime: prepare / ingest / finetune-ready / audit / simulate-rag / improve / benchmark
current exports: OpenAI, Anthropic, RAG, QA, conversations, evals, embeddings, retrieval simulation, improved corpus, quality benchmark
current dashboard role: account, usage, and preparation observability
current storage model: local artifacts under ~/Alys/prepared