Send

Uploads pipeline output to a destination. Supports three modes selected via send_as:

• direct_resource_load — POST FHIR resources to a FHIR server using transaction bundles
• transfer_load — wrap files in Binary / DocumentReference resources and POST them to a DSF transfer server
• s3_upload — upload files directly to an S3-compatible object store (AWS S3, MinIO, Ceph)

Direct Resource Load

Sends NDJSON directly to a FHIR server using transaction bundles.

services:
  send:
    send_as: "direct_resource_load"
    url: "https://fhir-server.example.com"  # server root; /fhir appended by client
    batch_size: 100  # resources per transaction (0-1000, default 100)
    auth:
      username: "${FHIR_USER}"
      password: "${FHIR_PASSWORD}"

pipeline:
  enabled_steps:
    - local_import
    - dimp
    - send

How it Works

1. Reads NDJSON files from input directory
2. Batches resources into transaction bundles
3. Uploads each bundle to the FHIR server
4. Uses PUT for resources with IDs, POST for resources without

Transfer Load

Packages files for DSF-based transfer using Binary/DocumentReference resources.

services:
  send:
    send_as: "transfer_load"
    url: "https://transfer-server.example.com"  # server root; /fhir appended by client
    auth:
      oauth_issuer_uri: "${OAUTH_ISSUER}"
      oauth_client_id: "${OAUTH_CLIENT_ID}"
      oauth_client_secret: "${OAUTH_CLIENT_SECRET}"
    transfer:
      project_identifier: "MII-PROJECT"
      organization_identifier: "your-org.example.de"

pipeline:
  enabled_steps:
    - local_import
    - dimp
    - send

How it Works

1. Reads all files from input directory
2. Zips each file and wraps in FHIR Binary resource
3. Creates DocumentReference linking all Binary resources
4. Uploads to transfer server

S3 Upload

Uploads pipeline output files directly to an S3-compatible bucket. Works with AWS S3, MinIO, Ceph, and any other S3-API-compatible store.

services:
  send:
    send_as: "s3_upload"
    s3:
      bucket: "${S3_BUCKET}"
      region: "eu-central-1"
      access_key_id: "${AWS_ACCESS_KEY_ID}"
      secret_access_key: "${AWS_SECRET_ACCESS_KEY}"

      # Optional: custom endpoint for non-AWS stores (MinIO, Ceph)
      # endpoint: "http://minio.example.com:9000"

      # Optional: required for MinIO and most non-AWS stores
      # use_path_style: true

      # Optional: per-upload timeout (default PT30M)
      # timeout: PT30M

pipeline:
  enabled_steps:
    - local_import
    - dimp
    - send

url is not required in s3_upload mode — it is only used by the FHIR send modes.

How it Works

1. Reads all files from input directory (recursively)
2. Uploads each file to the configured bucket using AWS SDK v2
3. Records the returned ETag per upload

Authentication

access_key_id and secret_access_key authenticate to the S3 API itself.

The optional top-level auth block is reused as proxy authentication for environments where the S3 endpoint sits behind an HTTP proxy that requires its own credentials. When auth.username and auth.password are set in s3_upload mode, Aether attaches a Proxy-Authorization: Basic … header to every S3 request without disturbing the AWS SDK's own Authorization signing header.

services:
  send:
    send_as: "s3_upload"
    auth:                              # only used as proxy auth in s3_upload mode
      username: "${PROXY_USER}"
      password: "${PROXY_PASS}"
    s3:
      bucket: "${S3_BUCKET}"
      region: "eu-central-1"
      access_key_id: "${AWS_ACCESS_KEY_ID}"
      secret_access_key: "${AWS_SECRET_ACCESS_KEY}"

Transient Errors

The S3 client classifies AWS responses such as SlowDown, ServiceUnavailable, InternalError, RequestTimeout, TooManyRequests, and connection-level failures (connection reset, connection refused, generic timeouts) as transient. These are retried according to the top-level retry settings; all other errors fail the step immediately.

Failed Requests

When the server refuses a request, the error message shows the server answer, but not the data that caused it. Set dump_failed_requests: true to keep that data:

services:
  send:
    dump_failed_requests: true

The step then writes two files to jobs/<job-id>/send/failed/ for the request that failed:

File	Content
<name>.request.json	The body sent to the server, unchanged. Send it again after you correct it.
<name>.response.txt	The HTTP status and the body of the answer, usually an OperationOutcome.

The name identifies the request:

• direct_resource_load — <file>-batch-<n>, the transaction Bundle for batch n of that file.
• transfer_load — binary-<id> or documentreference-<id>.

A Binary is written without its data element. That element only holds a copy of the input file, which stays in the input directory of the step.

The step stops at the first request that fails, so one run writes one pair of files.

s3_upload mode writes no dump. That mode sends no FHIR resources, and the file that the upload refused stays in the input directory of the step.

WARNING

The files hold pseudonymized patient data. Delete them when the analysis is complete, and include them in the retention rules for the job directory.

Configuration Options

Common

Option	Type	Default	Description
send_as	string	—	direct_resource_load, transfer_load, or s3_upload (required)
url	string	—	FHIR server root URL. Required for direct_resource_load and transfer_load. Do not include /fhir — the client appends it. Ignored for s3_upload.
batch_size	int	100	Resources per transaction (direct_resource_load only, 0-1000)
dump_failed_requests	bool	false	Write the body of a rejected request to disk (direct_resource_load and transfer_load only, see Failed Requests)
auth	object	—	Authentication (see below)

transfer (transfer_load only)

Option	Description
transfer.project_identifier	MII project identifier (used in DocumentReference.masterIdentifier)
transfer.organization_identifier	Organization identifier (used in DocumentReference.author)

s3 (s3_upload only)

Option	Type	Default	Description
s3.bucket	string	—	Target bucket name (required)
s3.region	string	—	AWS region (required, for example eu-central-1)
s3.access_key_id	string	—	S3 access key (required)
s3.secret_access_key	string	—	S3 secret key (required)
s3.endpoint	string	—	Custom endpoint URL for S3-compatible stores. Leave empty for AWS S3. Must be http:// or https://.
s3.use_path_style	bool	false	Use path-style addressing (required for MinIO and many S3-compatible stores)
s3.timeout	duration	PT30M	Per-request timeout (ISO 8601 or Go duration format)

Authentication

For FHIR send modes (direct_resource_load, transfer_load), choose one method:

Basic Auth:

auth:
  username: "${FHIR_USER}"
  password: "${FHIR_PASSWORD}"

OAuth 2.0 Client Credentials:

auth:
  oauth_issuer_uri: "${OAUTH_ISSUER}"
  oauth_client_id: "${OAUTH_CLIENT_ID}"
  oauth_client_secret: "${OAUTH_CLIENT_SECRET}"

API Key:

auth:
  api_key: "${FHIR_API_KEY}"
  api_key_header: "x-api-key"   # optional; this is the default

The API key goes in its own header, not in Authorization. Thus you can set it together with Basic Auth or OAuth 2.0. Basic Auth and OAuth 2.0 both set Authorization; use one of them only.

For s3_upload, the auth block is optional and only used for upstream proxy auth (see S3 Upload → Authentication). The S3 API itself is always authenticated via the keys under s3.*. Proxy auth supports username/password only: in s3_upload mode Aether ignores oauth_* and api_key.