Core Concepts — Xferity Flow, Job, Run, Step, Provider, Transport, Crypto Provider

Core Concepts

This page defines the core concepts used throughout Xferity’s architecture, documentation, and runtime behavior.

Understanding these concepts is required to understand how configuration, execution, and operational behavior relate to each other.

Flow

A Flow is the primary unit of transfer automation in Xferity.

A flow is:

a named, versioned YAML file stored on disk
a complete definition of one transfer workflow
the source of truth for what runs, when it runs, and how it behaves

A flow defines:

direction — upload (local → remote) or download (remote → local)
partner or endpoint — which partner definition to use, or inline endpoint config
file matching — glob patterns, include/exclude rules, stability thresholds
schedule — six-field cron expression or interval-based polling
retry behavior — max attempts, base delay, cap, jitter
idempotency mode — hash-based (default) or none
locking — distributed flow lock behavior
cleanup — delete, archive, or dead-letter after transfer
payload handling — PGP encryption, decryption, signing, verification
notifications — per-flow channel overrides

Flows are loaded at startup and reloaded on change. Unknown YAML fields cause a startup failure — no silent misconfiguration.

Flow lifecycle

configured → validated → triggered → running → committed
                                           ↓
                                       failed → retry → dead-letter

Partner

A Partner is a reusable definition of a remote transfer endpoint.

Partners are stored as named YAML files separate from flow definitions. This separates connection and trust settings from scheduling and transfer logic.

A partner defines:

protocol — SFTP, FTPS, AS2, S3-compatible, or local filesystem
endpoint — hostname, port, path, or URL
authentication — credentials or key references
trust material — known_hosts, TLS fingerprint, or AS2 certificate roles
crypto policy — Partner Crypto Policy (which roles are configured)

One partner can be referenced by multiple flows.

Job

A Job is a durable execution record created when a flow is triggered in Postgres-backed mode.

Jobs are:

persisted in the PostgreSQL job queue before execution begins
claimed by workers using SELECT FOR UPDATE SKIP LOCKED
executed by a single worker at a time — no duplicate execution
durable: survive process restarts and crashes
updated with status, outcome, and retry count at each stage

A job fails for two reasons:

transient failure — retried with exponential backoff (network timeout, server unavailable)
permanent failure — not retried (wrong passphrase, bad signature, key not found)

In file-backed mode, there is no durable job queue — flows execute in-process.

Run

A Run is a single execution instance of a flow.

Every run has:

a unique run ID
a start timestamp and end timestamp
an outcome: success, partial, or failure
file counts: processed, skipped, failed
retry count for the overall run
an idempotency state (which files were already completed)
a link to audit records for each file in the run

Runs are queryable via:

xferity flow history <flow> — lists all runs with outcomes
HTTP API — query runs per flow
Audit records — trace individual files within a run

Step

A Step is an atomic file-level operation within a run.

Steps include:

file discovery (listing remote or local source)
file transfer (upload or download)
payload operation (encrypt, decrypt, sign, verify)
idempotency check (hash comparison)
cleanup (delete, archive, dead-letter)
audit event write

A run contains one step sequence per file. File-level failures do not automatically abort the run — flows can tolerate a configurable number of per-file errors.

Provider (Secret Provider)

A Provider in the secrets context is a runtime credential resolver.

Xferity supports 7 providers. Operators use secret references in config (env:MY_VAR, local-vault:key-name, etc.). The provider resolves the reference to the actual value at runtime.

Provider	Reference format
Environment variable	`env:MY_VAR`
File	`file:/run/secrets/password`
Local vault (AES-256)	`local-vault:key-name`
HashiCorp Vault KV v2	`vault:secret/data/path#field`
AWS Secrets Manager	`aws-sm:secret-id#field`
Azure Key Vault	`azure-kv:secret-name`
Literal (testing only)	`literal:value`

Secret references work across global config, partner config, flow config, and auth config.

Transport

A Transport is the protocol connector that performs the actual file transfer.

Xferity includes transports for:

SFTP — SSH File Transfer Protocol; directory-style partner exchange
FTPS — FTP over explicit TLS; legacy-compatible partner exchange
AS2 — certificate-based B2B message exchange with MDN receipts
S3-compatible — object storage (AWS S3, MinIO, Cloudflare R2)
Local — local filesystem (used in testing and pipeline staging)

Each transport implements a common transfer interface but with protocol-specific trust models, authentication, and failure profiles.

The transport used by a flow is determined by the partner type referenced in the flow config.

Crypto Provider

A Crypto Provider is the engine that executes OpenPGP operations.

Xferity supports three crypto provider modes:

Mode	Description
`gopenpgp`	Native Go OpenPGP library — no external binary
`gnupg`	GnuPG binary — full GnuPG pipeline
`auto`	Try native first; fall back to GnuPG for named compatibility cases

The crypto provider is selected per flow via the pgp.provider field.

When GnuPG is used, Xferity creates an isolated temporary GnuPG home per operation — no shared keyring, no agent side effects.

auto fallback is controlled — it only triggers for compat_enterprise_key_structure, not for general failures.

Control Plane

The Control Plane is the operator-facing part of Xferity.

It includes:

Operator Web UI
Authenticated HTTP API
Auth service
Certificate and PGP Key inventory
Partner Crypto Policy views
Flow Crypto Requirements views
Posture engine
Suppression manager
Notification manager

This is where operators review configuration, inventory, posture, and history.

Runtime Plane

The Runtime Plane is responsible for executing flows and resolving crypto and transport choices at execution time.

It includes:

Flow execution engine
Runtime preflight validation
Runtime crypto resolution
Protocol transports (SFTP, FTPS, AS2, S3)
Run and Job history updates
Audit event writer

State Backend

The State Backend is where Xferity persists all durable state.

Two backends are supported:

Backend	Use case
File	Single-node; local disk-backed state; minimal dependencies
Postgres	Production; durable job queue; full control plane features

The backend choice determines which capabilities are available:

Auth sessions, crypto inventory, suppressions, posture snapshots, regression alerts, and local vault require Postgres.
File backend provides local state, history, and basic flow execution.

Posture

Posture is Xferity’s continuous security evaluation of the runtime state.

The posture engine evaluates 6 domains:

Crypto — certificate expiry, PGP key bindings, AS2 cert roles
Secrets — plaintext credentials, missing secret references
Transport — SFTP host key status, FTPS TLS settings, AS2 TLS
Auth — UI authentication enforcement, rate limiting
Flow drift — scheduled flows with no recent execution
Platform — runtime health, required backend features

Posture produces:

Active Findings — current unresolved issues
Suppressed Findings — acknowledged (tracked, not erased)
Hourly snapshots (Postgres-backed only)
Trend data over time
Regression alerts when security state worsens

FlowRoleSpecs

FlowRoleSpecs() is the canonical definition of PGP role requirements for a flow.

It is intentionally reused by:

config validation
UI policy rendering
runtime preflight validation
runtime crypto resolution

This means: the same definition governs what the UI shows AND what the runtime enforces. There is no way for the UI to display a policy that differs from what runtime enforces.

This is one of the most important architectural decisions in Xferity — it structurally prevents role-definition drift between configuration review and execution.

Summary: concept relationships

Flow → references → Partner
Flow → triggers → Job (Postgres) or in-process run (File)
Job → executed by → Worker
Job or Run → contains → Steps
Steps → use → Transport (SFTP / FTPS / AS2 / S3)
Steps → use → Crypto Provider (gopenpgp / GnuPG)
Config fields → resolved by → Provider (env / file / vault / AWS SM / etc.)
Posture Engine → evaluates → Platform + Partners + Flows
FlowRoleSpecs() → governs → validation + UI + preflight + runtime