TORCH Import

Extracts patient data from a TORCH server. Supports two modes: CRTDL-based extraction and direct TORCH URL import.

What it does

CRTDL mode (submit + poll + download):

• Sends CRTDL query to TORCH
• Polls for extraction completion
• Downloads FHIR NDJSON data

Direct URL mode (poll + download):

• Polls an existing TORCH extraction/result URL
• Downloads FHIR NDJSON data when ready
• Skips the extraction submission step

Start check

Before the first step starts, aether pipeline start sends GET {base_url}/fhir/metadata with the TORCH credentials and a 5-second timeout. The start stops if TORCH is unreachable, rejects the credentials (HTTP 401 or 403), sends a different status, or does not send a CapabilityStatement. The error shows the full URL and the HTTP status. If software.name in the statement does not contain Torch, aether logs a warning and continues.

Progress display

While the step polls, aether reads batch progress from the TORCH Task API (GET /fhir/Task/{jobId}, extension torch-job-progress) and shows it in three places:

• A progress line in the terminal, with a bar, a percent value, and the active batch stages: TORCH extraction [##########......] 66% — 1/3 batches (500/1200 patients), active: DIRECT_LOAD (2/5)
• A structured log line each time the progress changes.
• The pipeline status command, which reads the last persisted progress from the job file — also from a second terminal while the pipeline runs.

The percent value is an estimate: each active batch counts as stage index / 5 of a batch, and the five stages get equal weight.

The Task API extension needs TORCH v1.0.2 or later. Against an older TORCH, aether falls back to the OperationOutcome diagnostics text, as before.

Configuration

services:
  torch:
    base_url: "https://your-torch-server.org"
    username: "your-username"
    password: "your-password"
    extraction_timeout: PT30M    # default
    polling_interval: PT5S       # default
    max_polling_interval: PT30S  # default
    file_ready_retries: 10       # default
    file_ready_interval: PT10S   # default
    download_stall_timeout: PT1M # default
    request_timeout: PT1M        # default

pipeline:
  enabled_steps:
    - torch

Usage

With CRTDL file

aether pipeline start aether.yaml crtdl.json

With TORCH URL

aether pipeline start aether.yaml crtdl.json "https://torch.example.com/fhir/extraction/result-123"

URLs containing /fhir/extraction/ or /fhir/result/ are automatically recognized as TORCH URLs. See the TORCH Integration guide for details.

Configuration Options

Option	Type	Default	Description
base_url	string	-	TORCH server root URL (required). aether appends /fhir.
username	string	-	Authentication username
password	string	-	Authentication password
extraction_timeout	duration	PT30M	Max wait time for extraction
polling_interval	duration	PT5S	Initial status check interval
max_polling_interval	duration	PT30S	Max interval (exponential backoff)
file_ready_retries	int	10	Retries while waiting for files to appear after extraction completes
file_ready_interval	duration	PT10S	Interval between file-availability checks
download_stall_timeout	duration	PT1M	Inactivity window while streaming a result file; the download is canceled only if no bytes arrive for this long. 0 uses the built-in default.
request_timeout	duration	PT1M	Time limit for one request to TORCH, for example the extraction submit. aether does not send the submit again, because each submit starts a new extraction. 0 uses the built-in default.