Pipeline Steps

Aether processes data through configurable pipeline steps.

Import Steps

One import step must be first in the pipeline. Only enable one at a time.

Step	Description
TORCH Import	Extract data from TORCH server using CRTDL queries or a direct TORCH URL
Local Import	Import FHIR NDJSON files from local directory
HTTP Import	Download FHIR NDJSON files from HTTP URL

Processing Steps

Step	Description
DIMP	Pseudonymize FHIR data using DIMP service
Flattening	Transform FHIR NDJSON to CSV
Wait	Pause pipeline for manual inspection
Send	Upload data to FHIR server or DSF transfer

Typical Pipelines

Basic (TORCH + DIMP)

pipeline:
  enabled_steps:
    - torch
    - dimp

Local Data with Manual Review

pipeline:
  enabled_steps:
    - local_import
    - wait          # Review imported data
    - dimp
    - wait          # Review pseudonymized data
    - flattening

Full Pipeline with Send

pipeline:
  enabled_steps:
    - torch
    - dimp
    - flattening
    - send

Monitoring

# List all jobs
aether job list

# Check specific job
aether pipeline status <job-id>

Resuming

# Resume failed or paused job
aether pipeline continue <job-id>

Completed steps are not re-run.