Account-Tracked Research Runs

When you run npx --yes alys-akusa@latest prepare ./company-docs --yes --workspace ~/Alys, the files never leave your machine. The parsing, chunking, and export happen in your terminal, on your disk, under your permissions. But the moment the run completes, a structured record of everything that happened is sent to your account. That split: local execution, cloud memory. It is the core design decision behind Alys.

This post explains how that split works, why we chose it, and what the run ledger actually records.

TL;DR

Alys is built around a simple contract:

That separation is what makes the product useful. The CLI gets local trust and speed. The account gets memory and accountability. The dashboard gets visibility without becoming the runtime.

The Problem with All-in-One Tools

Most data preparation tools force a choice. Either they run in the browser, which means uploading sensitive files to someone else's server and waiting for a progress bar. Or they run entirely locally, which means no usage tracking, no history, no shared state between team members, and no way to answer "what did I run last Tuesday" without scrolling through three days of shell history.

Neither model works well for serious data work.

The browser-as-execution-environment breaks down quickly when you are processing:

• Private repositories with internal code
• Local PDFs under NDA
• Large media files that would take twenty minutes to upload
• Exports that need to land in a specific workspace folder with known permissions

A browser tab cannot read your local Git repository. It cannot inspect the file system to discover that your docs are spread across ./docs, ./specs, and ./README.md. It cannot write a JSONL export to ~/Alys/exports/2026-05-14/ so your downstream pipeline can pick it up.

On the other hand, a purely local CLI is a black box. You run a command. It produces files. A week later, you have a folder full of numbered exports and no idea which command produced which one, whether it consumed from your quota, what the source-to-record mapping was, or whether the output is actually ready for embeddings.

Alys solves this by separating the work from the memory.

Design Rule: Execution and Observability Are Different Surfaces

The terminal is for execution. The account is for observability.

This is not a compromise. It is a boundary drawn on purpose because the two surfaces have fundamentally different requirements.

Terminal responsibilities:

• Read local files with the user's actual permissions
• Walk repositories recursively and respect .gitignore
• Parse PDFs, markdown, code, and media using local tooling
• Chunk and transform documents deterministically
• Write exports to user-controlled paths in JSONL, CSV, markdown, or media formats
• Run fast by avoiding network round trips during processing

Account responsibilities:

• Authenticate the user and validate entitlements
• Track which runs happened, when, and what they produced
• Record usage: credits consumed, limits enforced, reservations held, refunds applied
• Store the structured manifest of each run
• Expose readiness scores and dataset state to the dashboard
• Serve as shared memory for teams

The CLI knows nothing about billing logic. The dashboard never sees your source files. The account layer sits between them, receiving a manifest after each run but never the raw data itself.

Another way to say it:

Only the manifest crosses the boundary. The raw files stay where they started.

The Run Ledger: What Gets Remembered

A run is a first-class object in Alys. When a preparation completes, the CLI assembles a structured manifest and POSTs it to the account API. Here is what that record contains.

Identity

• account_id: Which account initiated the run
• run_id: A unique identifier generated at the start of preparation
• workspace: The local path where exports were written
• cli_version: Exact version of alys-akusa that executed the run
• timestamp: ISO 8601 start and end times

This makes it possible to correlate a dashboard entry with the exact command and environment that produced it.

Usage

• generations_consumed: How many AI generations were used during parsing, chunking, or enrichment
• runs_consumed: Whether the run itself counted against a usage limit
• entitlement_applied: Which plan or quota was active
• remaining_balance: Quota left after the run
• unlimited_flag: Whether the account had no limit at the time

The CLI reports this. The account validates it. If a limit is hit mid-run, the CLI can query the account API and decide whether to halt or continue based on the entitlement response.

Manifest

• source_count: How many files were discovered and read
• record_count: How many structured records were produced after chunking
• formats: Which output formats were generated, e.g. ["jsonl", "csv", "markdown"]
• output_paths: Relative paths within the workspace for each artifact
• confidence: Per-document and aggregate confidence scores from parsing
• source_map: A mapping from source file to generated record IDs, so you can trace any output record back to its origin

This is the dataset story. Without it, you have a folder of anonymous exports.

Artifacts

• jsonl_files: Paths to structured record exports
• csv_files: Tabular exports if generated
• markdown_files: Human-readable transformation outputs
• media_files: Extracted or processed media references
• debug_logs: Verbose logs for troubleshooting
• review_files: Sidecar files flagging uncertain parses or low-confidence records

The artifact list lets the dashboard show a run and immediately answer: what did this command actually produce on disk?

Readiness

• rag_ready: Whether the output is suitable for retrieval-augmented generation
• eval_ready: Whether the dataset is structured enough for benchmark evaluation
• embedding_ready: Whether text chunks are sized and formatted for embedding pipelines
• fine_tune_ready: Whether the records are clean enough for supervised fine-tuning
• review_needed: Whether human review is recommended before downstream use

These flags are computed by the CLI based on heuristics: chunk size distribution, parse confidence thresholds, schema compliance, and error rates. The dashboard surfaces them so you can look at a run and know what it is good for without opening the files.

The ledger is not only a history table. It is the difference between "I generated some files" and "this dataset has an auditable preparation record."

Failure Modes We Are Avoiding

Without an account-tracked ledger, these scenarios happen constantly.

The Mystery Export Folder

You run preparation repeatedly over a week. Your workspace now contains:

None of them have metadata. Which one was run with the improved chunking strategy? Which one used the old parser? Which one consumed your quota? The only way to know is to run head and guess from the record shape.

With Alys, every file in that folder corresponds to a dashboard entry with a timestamp, version, and configuration hash.

The Overdrawn Quota

A purely local CLI has no way to enforce usage limits. You run a thousand-document preparation, it succeeds, and only later do you discover that your billing cycle ended three days ago and you are now over your plan. Or worse: the tool silently stops working and you spend an hour debugging the parser before realizing you hit a hidden limit.

Alys queries your entitlement before and during the run. If you are near a limit, the CLI shows a warning. If you exceed it, the account API returns a clear error and the run halts with a sensible message.

The Team Memory Gap

In a team, one person prepares a dataset on their laptop. They send a Slack message saying "ran prep on the specs, looks good." Two weeks later, someone else needs the same dataset for an eval pipeline. They do not know which command was used, whether any warnings were ignored, or where the output is. They re-run from scratch.

With account tracking, the run lives in the shared account. Any team member can open the dashboard, see the preparation history, inspect the manifest, and locate the exact export paths without bothering the original author.

The Phantom Warning

Your CLI run printed four pages of warnings: "low confidence on table parse," "skipped binary file," "deprecated field in schema." The output scrolled past and you did not save it. A month later, your evaluation scores are weird and you wonder if it was the parser or the data. But the terminal buffer is long gone.

Alys persists the warning summary in the run manifest and the full debug log as an artifact reference. The dashboard keeps the signal even when the terminal is silent.

What a Healthy Run Looks Like

The ideal path is boring:

Boring is the point. A dataset run should not depend on screenshots, Slack messages, or someone's shell history.

Authentication Flow: How the CLI Knows Who You Are

The CLI never stores your password. Instead, it uses a token-based flow that respects the same boundary.

1. You authenticate once via the browser dashboard, which sets a session cookie and issues a long-lived API token scoped to your account.
2. You run alys auth login, which opens a browser tab for confirmation and writes the token to ~/.alys/credentials with user-only permissions.
3. Every subsequent CLI command includes the token in the Authorization header when communicating with the account API.
4. The token is used only for metadata calls, not for file uploads. Your source files never leave your machine.

If you revoke the token from the dashboard, the CLI immediately loses access to usage queries and run recording. It can still execute locally, but the runs will not be tracked. This is by design: your data work should never depend on a network call.

The Dashboard as Control Plane

The browser is not a compute environment. It is a control plane.

The Alys dashboard shows:

• A chronological list of all runs for the account
• Per-run manifests with expandable detail
• Usage graphs showing quota consumption over time
• Readiness badges for each run so you can scan for RAG-ready or review-needed datasets
• Team activity if the account has multiple members
• Version history of the CLI that produced each run

What the dashboard does not do:

• Accept file uploads for processing
• Run parsing or chunking logic
• Store your source documents
• Act as a workspace for file management

This constraint is what makes the architecture trustworthy. The worst-case security breach of the account API exposes run metadata and manifests. It does not expose your private repositories, your PDFs, or your exported datasets, because those were never sent.

Solo Developer Mode

For a single developer, the account layer is personal memory.

You can run preparation on your laptop, shut it down, fly to a conference, open the dashboard on your phone in the airport, and see exactly what you produced yesterday. The source-to-record mapping is there. The readiness score is there. The output path is there.

You do not need to SSH back to your machine to check what happened. You do not need to keep a manual lab notebook of CLI commands. The dashboard is your lab notebook.

Team Mode

For a team, the account becomes shared state.

Multiple developers can run the CLI from their own machines with their own source files, and all the manifests aggregate into a single account history. This means:

• One person can prepare training data while another prepares eval data, and both results are visible to the whole team
• Managers can see usage trends and quota consumption without asking
• On-call engineers can locate the exact run that produced a problematic dataset and inspect its manifest without access to the original developer's laptop
• New team members can browse historical runs to understand what preparation strategies have been tried

The account is not a collaboration tool in the sense of Google Docs. It is a shared memory for a team that works locally.

Technical Implementation Notes

The manifest is a JSON document, typically 5 to 50 KB depending on source count. It is sent to the account API using a single POST request at the end of the run, or incrementally during long-running operations so that partial progress is recorded even if the terminal session is interrupted.

The CLI caches the last manifest locally at ~/.alys/runs/latest.json as a fallback in case the network request fails. If the POST succeeds, the cache is cleared. If it fails, the user sees a warning with instructions to retry manually or inspect the local cache.

Readiness scores are computed using domain-specific heuristics. For example, rag_ready requires that chunks are under a token limit, have associated metadata, and are stored in a format compatible with the Alys retrieval pipeline. fine_tune_ready requires that records contain both input and output fields, pass a basic schema check, and have a confidence score above a threshold. These heuristics are versioned with the CLI so that a dashboard entry from two months ago shows readiness flags that reflect the logic at the time of the run, not current logic.

The Core Principle

Local-first does not mean account-less.

The browser should not pretend to be a file system. The CLI should not pretend to be an identity provider. Alys works because those responsibilities are kept separate.

The terminal does the work. The account keeps the memory. The dashboard makes the memory useful.

That is account-tracked research runs.