Configuration

Aether uses a YAML configuration file. Create aether.yaml in your working directory.

Basic Configuration

services:
  torch:
    base_url: "https://your-torch-server.org"
    username: "your-username"
    password: "your-password"

  dimp:
    url: "http://your-dimp-server:32861/fhir"

pipeline:
  enabled_steps:
    - torch
    - dimp

jobs_dir: "./jobs"

Service Configuration

TORCH

services:
  torch:
    base_url: "https://your-torch-server.org"
    username: "your-username"
    password: "your-password"
    extraction_timeout_minutes: 30
    polling_interval_seconds: 5

DIMP

services:
  dimp:
    url: "http://your-dimp-server:32861/fhir"
    bundle_split_threshold_mb: 10  # Auto-split large bundles

Flattening

services:
  flattening:
    service_url: "http://fhir-flattener:8000"
    lookup_path: "/path/to/flatten-lookup.json"
    formats:
      - csv
    timeout: 30m

Send

Direct to FHIR server:

services:
  send:
    send_as: "direct_resource_load"
    url: "https://fhir-server.example.com/fhir"
    batch_size: 100
    auth:
      username: "${FHIR_USER}"
      password: "${FHIR_PASSWORD}"

DSF transfer:

services:
  send:
    send_as: "transfer_load"
    url: "https://transfer-server.example.com/fhir"
    auth:
      oauth_issuer_uri: "${OAUTH_ISSUER}"
      oauth_client_id: "${OAUTH_CLIENT_ID}"
      oauth_client_secret: "${OAUTH_CLIENT_SECRET}"
    transfer:
      project_identifier: "MII-PROJECT"
      organization_identifier: "your-org.example.de"

Local Import

services:
  local_import:
    dir: "/path/to/fhir/data"  # Override with --dir flag

Pipeline Steps

pipeline:
  enabled_steps:
    - torch         # OR local_import OR http_import
    - dimp          # Pseudonymization
    - wait          # Pause for inspection (optional)
    - flattening    # FHIR to CSV (requires CRTDL)
    - send          # Upload to destination

  # Max NDJSON line size in MB (default: 100)
  # Increase if you encounter "token too long" errors with large Bundles
  # max_ndjson_line_size_mb: 100

Step Placement Rules

Wait steps:

• Can be placed between any two steps
• Cannot be the first step (needs previous step output)
• Cannot be consecutive (redundant)
• Multiple wait steps are supported at different points in the pipeline

Processing steps (dimp, flattening):

• Should only appear once in the pipeline
• Multiple instances are not supported (output directories would be overwritten)

Import steps (torch, local_import, http_import):

• Must be first
• Only one import step allowed

Compression

compression:
  enabled: true        # default: true
  level: "default"     # fastest, default, better, best

Output files use .ndjson.zst extension when enabled.

Environment Variables

Use environment variables for sensitive data:

services:
  torch:
    username: "${TORCH_USERNAME}"
    password: "${TORCH_PASSWORD}"

export TORCH_USERNAME="researcher"
export TORCH_PASSWORD="secret"

Next Steps

• Quick Start - Run your first pipeline
• Pipeline Steps - Step details
• Configuration Reference - All options