Skip to content
dev

CLI Commands ​

Reference for Aether CLI commands.

Global Options ​

bash
aether [global-options] <command> ...
  • --verbose, -v - Enable verbose logging
  • --help, -h - Show help
  • --version - Show version

Every subcommand that needs configuration takes the path to aether.yaml as its first positional argument. There is no --config flag and no implicit discovery of ./aether.yaml or ~/.config/aether/aether.yaml.

Pipeline Commands ​

aether pipeline start ​

Start a new pipeline job.

bash
aether pipeline start <config> <crtdl> [input] [options]

Arguments:

  • <config> - Path to aether.yaml (required, first positional)
  • <crtdl> - CRTDL JSON file (required, second positional)
  • [input] - Optional import-step input:
    • omitted: torch_import submits the CRTDL; local_import uses --dir/config
    • local directory (local_import)
    • HTTP(S) URL (http_import)
    • TORCH result URL (torch_import; auto-detected from URL shape)

Options:

  • --no-progress - Disable progress indicators
  • --dir PATH - Directory for local import (overrides config)
  • --allow-http-crtdl - Acknowledge that HTTP data may not match the CRTDL query (required when http_import is enabled)

Examples:

bash
# TORCH extraction with CRTDL
aether pipeline start aether.yaml crtdl.json

# Direct TORCH URL (skip extraction, poll and download results)
aether pipeline start aether.yaml crtdl.json "https://torch.example.com/fhir/extraction/result-123"

# Local import with CRTDL for flattening (dir from flag)
aether pipeline start aether.yaml crtdl.json --dir /path/to/data

# Local import with CRTDL for flattening (dir as positional)
aether pipeline start aether.yaml crtdl.json /path/to/data

# HTTP download piped through flattening
aether pipeline start aether.yaml crtdl.json https://example.com/fhir/data.ndjson --allow-http-crtdl

# Disable progress bars
aether pipeline start aether.yaml crtdl.json --no-progress

aether pipeline status ​

Check pipeline job status.

bash
aether pipeline status <config> <job-id>

Output:

  • Job ID and status
  • Current step
  • Progress per step (files, bytes)
  • Error messages if failed

Example:

bash
aether pipeline status aether.yaml abc-123-def

aether pipeline continue ​

Resume a paused or failed pipeline.

bash
aether pipeline continue <config> <job-id> [options]

Use cases:

  • Resume after terminal close
  • Continue after fixing errors
  • Resume from wait step after placing files in wait directory

Options:

  • --no-progress - Disable progress indicators

Examples:

bash
# Resume failed job
aether pipeline continue aether.yaml abc-123-def

# Resume from wait step (after placing files)
aether pipeline continue aether.yaml abc-123-def

# Resume without progress bars
aether pipeline continue aether.yaml abc-123-def --no-progress

Send Command ​

aether send ​

Send the CSV files of a directory with the services.send settings.

bash
aether send <config> <dir> [--job <job-id>]

The command needs no CRTDL and no pipeline job. It reads the files in place and makes no copy of them. It writes nothing into <dir>. It ignores the pipeline section of the configuration, so a file with only services.send and jobs_dir is sufficient.

Arguments:

  • <config> - Path to aether.yaml (required)
  • <dir> - Directory with the CSV files (required)

Rules:

  • The command sends each file below <dir> whose name ends in .csv, in any letter case. It does not send other files.
  • The command follows symbolic links to files and to directories. A link loop, a link to nothing, and a directory without CSV files are errors.
  • send_as must be transfer_load or s3_upload. direct_resource_load is an error.
  • With transfer_load, the name of each zip entry is the path relative to <dir>.
  • With s3_upload, the S3 key is the job ID and the path relative to <dir>.

Job directory:

Each run creates a job directory under jobs_dir. It holds source.json, job.log, and the upload manifest. source.json records the name, size, and SHA-256 hash of each file. The command prints the job ID and the command to resume the run. These jobs have no state.json, so aether job list does not show them.

Options:

  • --job <job-id> - Resume an earlier run. The command compares the files with source.json and stops with an error when they differ. It sends nothing when the run is complete.

Examples:

bash
# Send all CSV files below ./csv
aether send aether.yaml ./csv

# Resume an earlier run
aether send aether.yaml ./csv --job 20250101_1200_3f2a9c1e-7b4d-4e8a-9c6f-1d2e3f4a5b6c

Job Commands ​

aether job list ​

List all pipeline jobs.

bash
aether job list <config>

Output columns:

  • JOB ID - Job ID (YYYYMMDD_HHMM_UUID)
  • STATUS - completed/in_progress/failed/pending/stopped/waiting
  • STEP - Current step
  • FILES - Total files processed
  • AGE - Time since creation

Status symbols:

  • ✓ - Completed
  • → - In progress
  • ✗ - Failed
  • ○ - Pending
  • ■ - Stopped
  • ‖ - Waiting at the wait step

Stopped jobs:

A job becomes stopped when its process ends before the pipeline is complete. Aether finds this in two ways:

  1. If you press Ctrl+C two times in 5 seconds, or the process receives SIGTERM, Aether writes the stopped status to the state file before it exits. The first Ctrl+C only shows Press Ctrl+C again within 5s to quit, and the pipeline continues. A Ctrl+C after more than 5 seconds shows the request again. The stop does not cancel the work on the services: a TORCH extraction continues on the TORCH server, and pipeline continue re-attaches to it.
  2. If the process dies without a chance to write, for example from SIGKILL, a crash, or a power loss, the state file keeps in_progress. job list and pipeline status then test the job lock. If no process holds the lock, they show the job as stopped.

Both commands only read. They do not change the state file in the second case.

A stopped job is not terminal. Continue it:

bash
aether pipeline continue aether.yaml <job-id>

A waiting job also has no live process, but that state is intended: the wait step paused the job. Continue it with the same command.

State file compatibility

The stopped and waiting job statuses are new. An older Aether binary rejects a state file that holds one of them, and job list skips that job with a warning. Do not change to an older version while jobs are in these states.

Example:

bash
aether job list aether.yaml

aether job run ​

Execute a specific pipeline step manually.

bash
aether job run <config> <job-id> --step <step-name>

Options:

  • --step - Step to execute (required)
  • --force - Re-run the step even if it has already completed

Valid steps: torch, local_import, http_import, dimp, validation, flattening, send

Re-running completed steps: An already-completed step is not re-run unless --force is passed. Without --force, the command fails fast with step '<step>' already completed; pass --force to re-run and a non-zero exit. This is a pre-execution guard — nothing runs, so job state is left untouched.

With --force, the step is reset to pending and re-run. Every step ordered after it is also reset to pending, because re-running a step invalidates the output the later steps consumed (outputs are not diffed, so this is conservative). The command itself re-runs only the target step; the invalidated downstream steps stay pending until you re-run them — either with another job run --step or by resuming the pipeline.

Job status after a manual run: The job-level status is always derived from its steps. While a --force re-run runs, the job reads in_progress (visible in aether pipeline status). After it finishes, the status is re-derived: the job returns to completed only if every step is completed — so re-running the last step completes the job, while re-running an earlier step leaves it in_progress until the invalidated downstream steps re-run. Likewise, a manual run that finishes the last outstanding step sets the job to completed.

After the command exits, no process runs the job. Thus a job left mid-pipeline holds in_progress in the state file, but job list shows it as stopped.

Examples:

bash
# Run DIMP step manually
aether job run aether.yaml abc-123-def --step dimp

# Run import step manually
aether job run aether.yaml abc-123-def --step local_import

# Re-run an already-completed step
aether job run aether.yaml abc-123-def --step dimp --force

aether job stop ​

Stop a job for good and cancel its TORCH job.

bash
aether job stop <config> <job-id>

If a process runs the job, the command asks that process to stop and waits until the process ends. The process records the status stopped, cancels the TORCH job, and exits with code 143. If the process does not record the stop in 30 seconds, the command fails. After the record, the command waits until the process ends. The command asks the process with a marker file .stop in the job directory. It does not send a signal, thus it also works on Windows. The command removes the marker when it ends.

When no process runs the job, the command holds the job lock and sets the status to stopped itself. This also applies when the process of the job ended without a record of the stop, for example after SIGKILL.

A running process stops a job in any state that is not completed or failed, also a job with the status waiting or pending.

If the torch step waits for a TORCH job, Aether cancels that TORCH job with POST /fhir/Task/{id}/$cancel. The cancel sends one request and waits at most 5 seconds, also for the OAuth2 token. After the process of the job ends, the command sends the cancel again, thus it knows the result. If TORCH answers 404, 409 or 410, the TORCH job does not run, and the cancel counts as done. If the cancel fails, the job stays stopped and the TORCH job runs until it ends. The command then prints an error and exits with a non-zero code.

A job that is completed or failed cannot be stopped. The command then prints an error and exits with a non-zero code. This also applies when the job ends on its own while the command waits.

aether pipeline continue starts the stopped job again. TORCH answers 410 Gone for the cancelled TORCH job, thus continue submits a new extraction.

Examples:

bash
# Stop a job
aether job stop aether.yaml abc-123-def

# Start it again with a new TORCH extraction
aether pipeline continue aether.yaml abc-123-def

Other Commands ​

aether completion ​

Generate shell completion scripts.

bash
aether completion <shell>

Shells: bash, zsh, fish, powershell

Examples:

bash
# Generate bash completions
aether completion bash > /etc/bash_completion.d/aether

# Generate zsh completions
aether completion zsh > "${fpath[1]}/_aether"

aether help ​

Show help information.

bash
aether help [command]

Examples:

bash
aether help
aether help pipeline
aether help pipeline start

Configuration Loading ​

The path to aether.yaml is the first positional argument of every command that needs configuration; Aether does not auto-discover it. CLI flags and AETHER_* environment variables still override individual values from the file.

Environment Variables ​

Configuration values support environment variable substitution:

yaml
services:
  torch:
    username: "${TORCH_USERNAME}"
    password: "${TORCH_PASSWORD}"

Next Steps ​

Healthcare data integration made simple