Documentation

What TouchStone does

TouchStone turns requirements for a reinforcement learning environment into deterministic regression tests. A trusted adapter you write drives your environment, returns observations, and TouchStone checks those observations against assertions that a person has reviewed and approved. The same approved tests run again whenever the environment, grader or reward changes.

Suite

A directory with suite.json, your adapter.py, and ignored .touchstone/ evidence.

Requirement

A statement about expected behavior, such as “an invalid action must not earn reward”.

Case

One test: an input for the adapter, the requirement it supports, assertions on the observations, and provenance.

Approval

A reviewer’s decision bound to the exact content hash of the case and its requirement. Any edit invalidates it.

Run

One execution of the approved or pending cases, with observations, assertion results, adapter hash and diagnostics retained.

Candidates can come from the built-in starter, a Harbor task directory, your own JSON, or a model you invoke. All of them are hypotheses until reviewed. Only approved tests can fail CI; experimental runs never block anything.

Install and start

You need Linux or Ubuntu under WSL with Python 3.12 or newer. The package on PyPI is touchstone-us; the commands it installs are touchstone and touchstone-rl. No container engine, GPU or model API key is required for the starter example.

uv tool install touchstone-us
touchstone doctor
touchstone serve

Open http://127.0.0.1:4474/safety. The server listens on loopback only and uses port 4474 unless --port or TOUCHSTONE_PORT says otherwise. doctor reports whether this host can run the container-based audits; the reviewed test workflow does not need them.

The status page at /dashboard summarizes the same suite for a release owner: whether the latest approved run passed, which tests failed and what was observed, what changed since the previous approved run, requirements without an approved test, and run history grouped by adapter version. Failed assertions are counted separately from tests that could not run, each persisting failure shows how long it has failed unchanged, dismissed candidates stay listed with their reviewer, and every failing run carries the commands to reproduce it on that machine. It warns when tests or the adapter changed after the last run.

Suites, reviews and run history are written under .local/ in the directory where you run the commands, so add it to that project’s .gitignore. The web workspace uses .local/state/safety-suite by default; set TOUCHSTONE_SAFETY_SUITE to an existing suite directory before starting the server to work on your own. Restart the server after upgrading with uv tool upgrade touchstone-us; a running interpreter keeps the old version loaded.

From a source checkout, uv sync --frozen installs the locked dependencies and npm start runs the same server. Prefix every command on this page with uv run in that case.

Try the starter suite

The starter is an intentionally faulty checkout evaluator. It rewards a visible success marker even when no purchase happened. The walkthrough reproduces that defect, repairs it, and keeps the counterexample as a regression test.

1. Create the suite. Click Create starter suite. This writes suite.json and adapter.py locally; nothing runs yet.
2. Generate candidates. Two deterministic templates appear as pending: forged-success expects reward 0 and valid-checkout expects reward 1.
3. Review. Enter a reviewer name, open each candidate, inspect its requirement, input and assertions, then choose Approve for CI for both. The approval records the content hash you reviewed.
4. Run approved tests. The forged case fails: reward is 1 and purchased is false. The valid control passes, so the failure is a real defect rather than an evaluator that rejects everything.
5. Fix the evaluator. In the generated adapter.py, change REQUIRE_PURCHASE = False to True. No restart is needed; each run freezes the current adapter source.
6. Run again and compare. Both tests pass. Under Compare with an earlier run, pick the failing baseline: the forged case shows Fixed and the control shows Unchanged. Suite hashes match; only the adapter hash differs.

Experimental runs of pending candidates are recorded separately from approved runs. A changed assertion is labeled a changed test, not a fix.

Use the command line

Every browser action has a CLI equivalent, and all output is JSON. The same walkthrough from a terminal:

touchstone init .local/my-suite
touchstone generate .local/my-suite
touchstone report .local/my-suite
touchstone check .local/my-suite --experimental

In a terminal, review prints the case and its requirement, asks for confirmation, and records the hash it showed you. In scripts, pass the case_sha256 from report with --sha256; a stale hash is rejected.

touchstone review .local/my-suite forged-success \
  --decision approve --reviewer 'Your name'
touchstone review .local/my-suite valid-checkout \
  --decision approve --reviewer 'Your name' --sha256 <case-sha256>
touchstone check .local/my-suite
touchstone report .local/my-suite <run-id> > result.json

Every command accepts --evidence-dir PATH to store reviews and history outside the suite. Use the same path for review, check and report. Python callers get the same dictionaries from SafetyStore in touchstone.safety.store and check in touchstone.safety.engine.

Connect your environment

If your grader already answers HTTP requests or runs as a command, let init write the adapter for you:

touchstone init ./touchstone-suite \
  --requirement "Reward requires a paid backend order" \
  --http http://grader.internal:8080/evaluate

touchstone init ./touchstone-suite \
  --requirement "Reward requires a paid backend order" \
  --command "python grade.py"

The HTTP adapter POSTs each case input as JSON and treats the JSON reply as the observation. The command adapter sends the input as JSON on stdin and observes exit_code, stdout, stderr and output (stdout parsed as JSON, else null), so assertions such as exit_code eq 0, stdout contains "RESULT: PASS" or output.reward eq 1 need no Python. Requirement IDs are derived from their text. Without a target flag, init writes a skeleton adapter whose cases stay inconclusive until you implement it.

For a Harbor task directory, init --from-harbor ./tasks/my-task also drafts three pending probes: the reference solution earns reward 1, an empty submission earns 0, and a forged reward.txt earns 0. The generated adapter builds the task image with Podman or Docker, runs tests/test.sh as Harbor does and reads /logs/verifier/reward.txt. Run them with check --experimental --timeout 300, then review.

Generated or hand-written, your suite’s adapter.py is trusted operator code. It implements four functions; TouchStone calls them in order for each case in a fresh subprocess.

def setup(case):
    # Connect to or prepare your real test environment.
    return your_environment.start()

def exercise(context, inputs):
    context.perform(inputs)

def collect(context):
    # Return independent evidence, not the agent's claim of success.
    return {"reward": context.grade(), "purchased": context.admin_order_exists()}

def cleanup(context):
    if context is not None:
        context.close()

One adapter may call a grader once; another may replay an episode or invoke an existing evaluation script. Keep protected expectations, administrative evidence and the adapter itself outside anything the agent under test can write to, and never pass the full case, which contains the assertions, to that agent.

• Each case gets a default 30-second timeout. Exceptions, timeouts, invalid output and cleanup errors are inconclusive, never a pass.
• The subprocess is not a hostile-code sandbox. Forced termination cannot guarantee external cleanup, so use disposable resources.
• The entry point runs from a frozen temporary copy. Install adapter dependencies in the runner environment and reference support files by installed package or explicit path.
• Adapter changes between runs are allowed and recorded by hash, so a fix can be checked against the same approved test. Dependencies and external services are not frozen automatically.

Point the workspace at your suite with TOUCHSTONE_SAFETY_SUITE=path/to/suite touchstone serve. The web API never accepts arbitrary adapter paths.

Write or import tests

Cases are JSON. Each one references a requirement and declares assertions on the observation object your adapter returns.

{
  "cases": [{
    "id": "forged-checkout",
    "title": "A confirmation without an order earns no reward",
    "requirement_id": "verified-purchase",
    "input": {"action": "forge-marker"},
    "assertions": [
      {"path": "purchased", "operator": "eq", "expected": false},
      {"path": "reward", "operator": "eq", "expected": 0}
    ],
    "provenance": {"kind": "imported", "source": "Owner-authored regression"}
  }]
}

Import with touchstone generate DIR --from-json candidates.json, or paste the JSON into the workspace. Reusing an ID edits that case and invalidates its earlier approval. Rejected cases are excluded from approved and experimental runs.

Operators are eq, ne, lte, gte, contains and not_contains. Dot-separated paths address nested fields. Missing fields, wrong types and non-finite numbers cannot pass. There is no assertion eval; compute richer domain measurements inside your adapter and assert on the result.

Propose tests with a model

No hosted model is called by default. To ask a model for candidates, use your own signed-in Codex CLI or any trusted local command.

touchstone generate .local/my-suite --codex
touchstone generate .local/my-suite --command python my_author.py

Both send the suite, requirements, existing cases and adapter source to the authoring process. Run history and network captures are never included. Codex runs read-only in a temporary directory with a five-minute bound; a custom command receives JSON on stdin and must return a cases object within 120 seconds and 1 MiB. Proposals get new IDs, cannot overwrite existing cases and are never approved automatically. The Propose with Codex button in the workspace does the same thing.

Gate CI

Run touchstone check in your pipeline and branch on its exit code.

Never use --experimental as a gate: it exits 0 after recording its result even when a candidate fails. Treat 3 as a failure in CI so missing evidence cannot pass silently. A minimal GitHub Actions job on a Linux runner that can reach your environment:

jobs:
  regression:
    runs-on: [self-hosted, linux]
    steps:
      - uses: actions/checkout@v7
      - uses: astral-sh/setup-uv@v10
      - run: uv tool install touchstone-us
      - run: touchstone check touchstone-suite --evidence-dir /var/lib/touchstone/evidence

Commit the suite directory, including its reviews, so the gate runs the content that was approved. Pointing --evidence-dir at persistent storage keeps run history across jobs; a runner’s temporary directory is discarded.

The suite directory holds a shared .touchstone-suite.lock that init creates. If you mount a suite read-only in CI, create that lock in the writable source first and keep it for the suite’s lifetime. Execution uses a separate lock, so reviews stay available during a run. A runner container image is described in the repository’s RUNNER.md; it packages the CLI and SDK and is optional.

Share results with a team

Sharing is optional and always explicit. The control plane is a separate, separately licensed application that an organization operates for itself; it is not part of the PyPI package and no hosted instance is open to customers yet. Sign in with GitHub, create an organization API token, put it in TOUCHSTONE_API_TOKEN, then upload a run:

touchstone sync .local/my-suite <run-id> \
  --url http://127.0.0.1:4480 --suite checkout

• Default upload sends IDs, timestamps, hashes and statuses only. Reviewers can see results but cannot approve assertions they cannot read.
• Full evidence requires an organization owner to enable it first, then --include-evidence. This uploads requirements, inputs, adapter source, observations and logs.
• Pull reviews with --pull-reviews to apply team decisions locally. Only cases whose hashes still match are updated; stale reviews are skipped.

API tokens can upload but cannot approve. Owners can list and revoke them. Uploaded evidence is customer-reported execution, not server attestation.

The shared workspace walks a new organization through a five-step checklist, then shows usage and outcome metrics computed from uploaded summaries: runs, approved tests, failures caught, fixes verified and hours to fix, with a CSV export of every test result. An hours-saved figure appears only after a member enters the organization’s own hours-per-incident baseline, and is labeled as that assumption. Owners can set a retention window after which older uploaded runs are deleted.

Other tools

Two earlier workflows remain in the same local server with their own histories.

Grader audits

Freeze a bounded Python coding-task package, run reference, invalid and attack candidates in separate rootless containers, and compare outputs on the host. CLI: touchstone-rl run package.json --json.

Stack comparisons

Compare approved and candidate OCI images over repeated paired trials of a trusted workload. CLI: touchstone run manifest.json --json.

Both need rootless Podman. Their contracts live in RL_PROTOCOL.md and PROTOCOL.md in the repository.

Licensing

TouchStone is proprietary software. You may install and run it on infrastructure you control to evaluate it for up to thirty days; continued use needs a written agreement such as a pilot or subscription. Redistribution, modification and use in a competing product are not permitted. The full text ships as the LICENSE file in the package. The control plane is licensed separately under the Functional Source License.