CLI Commands
Reference for Aether CLI commands.
Global Options
aether [global-options] <command> ...--verbose, -v- Enable verbose logging--help, -h- Show help--version- Show version
Every subcommand that needs configuration takes the path to aether.yaml as its first positional argument. There is no --config flag and no implicit discovery of ./aether.yaml or ~/.config/aether/aether.yaml.
Pipeline Commands
aether pipeline start
Start a new pipeline job.
aether pipeline start <config> <crtdl> [input] [options]Arguments:
<config>- Path toaether.yaml(required, first positional)<crtdl>- CRTDL JSON file (required, second positional)[input]- Optional import-step input:- omitted: torch_import submits the CRTDL; local_import uses
--dir/config - local directory (local_import)
- HTTP(S) URL (http_import)
- TORCH result URL (torch_import; auto-detected from URL shape)
- omitted: torch_import submits the CRTDL; local_import uses
Options:
--no-progress- Disable progress indicators--dir PATH- Directory for local import (overrides config)--allow-http-crtdl- Acknowledge that HTTP data may not match the CRTDL query (required whenhttp_importis enabled)
Examples:
# TORCH extraction with CRTDL
aether pipeline start aether.yaml crtdl.json
# Direct TORCH URL (skip extraction, poll and download results)
aether pipeline start aether.yaml crtdl.json "https://torch.example.com/fhir/extraction/result-123"
# Local import with CRTDL for flattening (dir from flag)
aether pipeline start aether.yaml crtdl.json --dir /path/to/data
# Local import with CRTDL for flattening (dir as positional)
aether pipeline start aether.yaml crtdl.json /path/to/data
# HTTP download piped through flattening
aether pipeline start aether.yaml crtdl.json https://example.com/fhir/data.ndjson --allow-http-crtdl
# Disable progress bars
aether pipeline start aether.yaml crtdl.json --no-progressaether pipeline status
Check pipeline job status.
aether pipeline status <config> <job-id>Output:
- Job ID and status
- Current step
- Progress per step (files, bytes)
- Error messages if failed
Example:
aether pipeline status aether.yaml abc-123-defaether pipeline continue
Resume a paused or failed pipeline.
aether pipeline continue <config> <job-id> [options]Use cases:
- Resume after terminal close
- Continue after fixing errors
- Resume from wait step after placing files in wait directory
Options:
--no-progress- Disable progress indicators
Examples:
# Resume failed job
aether pipeline continue aether.yaml abc-123-def
# Resume from wait step (after placing files)
aether pipeline continue aether.yaml abc-123-def
# Resume without progress bars
aether pipeline continue aether.yaml abc-123-def --no-progressSend Command
aether send
Send the CSV files of a directory with the services.send settings.
aether send <config> <dir> [--job <job-id>]The command needs no CRTDL and no pipeline job. It reads the files in place and makes no copy of them. It writes nothing into <dir>. It ignores the pipeline section of the configuration, so a file with only services.send and jobs_dir is sufficient.
Arguments:
<config>- Path toaether.yaml(required)<dir>- Directory with the CSV files (required)
Rules:
- The command sends each file below
<dir>whose name ends in.csv, in any letter case. It does not send other files. - The command follows symbolic links to files and to directories. A link loop, a link to nothing, and a directory without CSV files are errors.
send_asmust betransfer_loadors3_upload.direct_resource_loadis an error.- With
transfer_load, the name of each zip entry is the path relative to<dir>. - With
s3_upload, the S3 key is the job ID and the path relative to<dir>.
Job directory:
Each run creates a job directory under jobs_dir. It holds source.json, job.log, and the upload manifest. source.json records the name, size, and SHA-256 hash of each file. The command prints the job ID and the command to resume the run. These jobs have no state.json, so aether job list does not show them.
Options:
--job <job-id>- Resume an earlier run. The command compares the files withsource.jsonand stops with an error when they differ. It sends nothing when the run is complete.
Examples:
# Send all CSV files below ./csv
aether send aether.yaml ./csv
# Resume an earlier run
aether send aether.yaml ./csv --job 20250101_1200_3f2a9c1e-7b4d-4e8a-9c6f-1d2e3f4a5b6cJob Commands
aether job list
List all pipeline jobs.
aether job list <config>Output columns:
- JOB ID - Job ID (
YYYYMMDD_HHMM_UUID) - STATUS - completed/in_progress/failed/pending/stopped/waiting
- STEP - Current step
- FILES - Total files processed
- AGE - Time since creation
Status symbols:
✓- Completed→- In progress✗- Failed○- Pending■- Stopped‖- Waiting at the wait step
Stopped jobs:
A job becomes stopped when its process ends before the pipeline is complete. Aether finds this in two ways:
- If you press Ctrl+C two times in 5 seconds, or the process receives SIGTERM, Aether writes the
stoppedstatus to the state file before it exits. The first Ctrl+C only showsPress Ctrl+C again within 5s to quit, and the pipeline continues. A Ctrl+C after more than 5 seconds shows the request again. The stop does not cancel the work on the services: a TORCH extraction continues on the TORCH server, andpipeline continuere-attaches to it. - If the process dies without a chance to write, for example from SIGKILL, a crash, or a power loss, the state file keeps
in_progress.job listandpipeline statusthen test the job lock. If no process holds the lock, they show the job asstopped.
Both commands only read. They do not change the state file in the second case.
A stopped job is not terminal. Continue it:
aether pipeline continue aether.yaml <job-id>A waiting job also has no live process, but that state is intended: the wait step paused the job. Continue it with the same command.
State file compatibility
The stopped and waiting job statuses are new. An older Aether binary rejects a state file that holds one of them, and job list skips that job with a warning. Do not change to an older version while jobs are in these states.
Example:
aether job list aether.yamlaether job run
Execute a specific pipeline step manually.
aether job run <config> <job-id> --step <step-name>Options:
--step- Step to execute (required)--force- Re-run the step even if it has already completed
Valid steps: torch, local_import, http_import, dimp, validation, flattening, send
Re-running completed steps: An already-completed step is not re-run unless --force is passed. Without --force, the command fails fast with step '<step>' already completed; pass --force to re-run and a non-zero exit. This is a pre-execution guard — nothing runs, so job state is left untouched.
With --force, the step is reset to pending and re-run. Every step ordered after it is also reset to pending, because re-running a step invalidates the output the later steps consumed (outputs are not diffed, so this is conservative). The command itself re-runs only the target step; the invalidated downstream steps stay pending until you re-run them — either with another job run --step or by resuming the pipeline.
Job status after a manual run: The job-level status is always derived from its steps. While a --force re-run runs, the job reads in_progress (visible in aether pipeline status). After it finishes, the status is re-derived: the job returns to completed only if every step is completed — so re-running the last step completes the job, while re-running an earlier step leaves it in_progress until the invalidated downstream steps re-run. Likewise, a manual run that finishes the last outstanding step sets the job to completed.
After the command exits, no process runs the job. Thus a job left mid-pipeline holds in_progress in the state file, but job list shows it as stopped.
Examples:
# Run DIMP step manually
aether job run aether.yaml abc-123-def --step dimp
# Run import step manually
aether job run aether.yaml abc-123-def --step local_import
# Re-run an already-completed step
aether job run aether.yaml abc-123-def --step dimp --forceaether job stop
Stop a job for good and cancel its TORCH job.
aether job stop <config> <job-id>If a process runs the job, the command asks that process to stop and waits until the process ends. The process records the status stopped, cancels the TORCH job, and exits with code 143. If the process does not record the stop in 30 seconds, the command fails. After the record, the command waits until the process ends. The command asks the process with a marker file .stop in the job directory. It does not send a signal, thus it also works on Windows. The command removes the marker when it ends.
When no process runs the job, the command holds the job lock and sets the status to stopped itself. This also applies when the process of the job ended without a record of the stop, for example after SIGKILL.
A running process stops a job in any state that is not completed or failed, also a job with the status waiting or pending.
If the torch step waits for a TORCH job, Aether cancels that TORCH job with POST /fhir/Task/{id}/$cancel. The cancel sends one request and waits at most 5 seconds, also for the OAuth2 token. After the process of the job ends, the command sends the cancel again, thus it knows the result. If TORCH answers 404, 409 or 410, the TORCH job does not run, and the cancel counts as done. If the cancel fails, the job stays stopped and the TORCH job runs until it ends. The command then prints an error and exits with a non-zero code.
A job that is completed or failed cannot be stopped. The command then prints an error and exits with a non-zero code. This also applies when the job ends on its own while the command waits.
aether pipeline continue starts the stopped job again. TORCH answers 410 Gone for the cancelled TORCH job, thus continue submits a new extraction.
Examples:
# Stop a job
aether job stop aether.yaml abc-123-def
# Start it again with a new TORCH extraction
aether pipeline continue aether.yaml abc-123-defOther Commands
aether completion
Generate shell completion scripts.
aether completion <shell>Shells: bash, zsh, fish, powershell
Examples:
# Generate bash completions
aether completion bash > /etc/bash_completion.d/aether
# Generate zsh completions
aether completion zsh > "${fpath[1]}/_aether"aether help
Show help information.
aether help [command]Examples:
aether help
aether help pipeline
aether help pipeline startConfiguration Loading
The path to aether.yaml is the first positional argument of every command that needs configuration; Aether does not auto-discover it. CLI flags and AETHER_* environment variables still override individual values from the file.
Environment Variables
Configuration values support environment variable substitution:
services:
torch:
username: "${TORCH_USERNAME}"
password: "${TORCH_PASSWORD}"Next Steps
- Configuration Reference - All configuration options
- Pipeline Steps - Step details