Transfer Automation — Flows, Cron Schedules, Retries, Idempotency, and Recovery

Automation

Xferity replaces ad hoc scripts and manual transfer routines with configuration-driven flows.

Each flow defines:

what files to match
where files come from and where they go
which partner or endpoint to use
when the flow should run
which reliability and cleanup rules apply

Flow-driven execution

Flows are the core automation unit. A flow can be started:

once on demand
through run-service
from a schedule_cron expression
through the Web UI or HTTP API when enabled

In file-backed mode, triggered flows run in-process. In Postgres-backed mode, triggers can enqueue durable jobs that workers later claim.

What a flow controls

Flows can control:

direction
partner or inline endpoint selection
file matching rules
idempotency mode
retry behavior
locking behavior
delete, archive, and dead-letter behavior
notifications
payload handling such as encryption or decryption

CLI and service entry points

The CLI supports:

run <flow>
run-all
run-service <flow>
resume <flow> or resume

run-service can use either:

--interval-seconds
the flow’s schedule_cron

Cron parsing expects a six-field expression with seconds.

Reliability controls

Automation includes several runtime controls used to make repeated partner exchange safer than a cron script alone.

Retries

Retry behavior is configurable per flow. Xferity uses exponential backoff with a configurable base delay, delay cap, and maximum attempt count. Jitter is applied to spread retry timing and prevent coordinated retry storms across concurrent flows.

Retries are intended for transient transport or server-side conditions. The retry logic distinguishes transient from permanent failures. A partner refusing a connection is retryable; a cryptographic key mismatch is not.

Idempotency

Idempotency is implemented using SHA-256 content-hash tracking. When a file is successfully processed, its hash is persisted. On subsequent runs or retries, files whose hashes are already present are skipped.

This prevents duplicate uploads or duplicate processing when a flow is rerun after a partial failure, when a retry runs after a crash, or when an operator manually reruns via xferity resume.

The idempotency mode can be set per flow. For production partner exchange, content-hash idempotency is the recommended default.

Flow locking

Flow locking prevents two instances of the same flow from running concurrently. Flow locking includes:

lock acquisition at flow start
optional configurable wait time for an existing lock to release
stale-lock takeover when a lock exceeds its maximum age
ownership-aware release to prevent one process from releasing another’s lock

This matters for polling workflows where run-service is used with short intervals and transfers can occasionally run longer than expected.

Partial failure handling

File-level failure does not automatically abort the entire flow run. Flows can tolerate a configurable number of file-level errors and continue processing remaining files. The run result records how many files succeeded and how many failed.

This is useful for batch workflows where one partner file being invalid should not block the rest of the batch.

Dead-letter handling

When a file fails at any processing stage — encryption, upload, download, or decryption — and retry attempts are exhausted, Xferity can move the artifact to a configurable dead-letter directory for operator review.

This prevents file loss in recovery scenarios and gives operators a clear inventory of files that need manual attention.

Recovery workflows

Automation is only useful if recovery is predictable. Xferity provides explicit recovery paths:

xferity resume <flow> re-enters execution from the last committed state
idempotency tracking prevents re-processing files already completed for that run
xferity flow history <flow> shows per-run outcomes and retry counts
xferity trace <file> shows the file’s full lifecycle across all runs
dead-letter artifacts are inventoried and available for reinspection

Deployment differences

File-backed mode

File-backed mode is the simpler automation model. A single process loads config and runs flows directly. Flows execute in-process when triggered by run, run-service, or run-all.

Postgres-backed mode

Postgres-backed mode adds durable job enqueueing and worker-based execution. This is the mode to use when you need queued execution semantics, shared state, or authenticated UI/API access. Jobs are enqueued to Postgres and claimed by workers, which means a process restart does not lose queued work.

Boundaries and limits

To keep the description accurate:

automation does not guarantee zero duplicates in every failure scenario
automation does not replace upstream or downstream business validation
automation does not replace monitoring, backup, or runbook discipline

What Xferity provides is a structured automation model for secure file transfer and partner file exchange that is easier to review and operate than expanding scripts over time.