Skip to content
Xferity

Scaling Xferity — Multiple Workers, Parallel Files, and Throughput Tuning

Scaling

Section titled “Scaling”

Xferity scales by adding workers, tuning parallelism within flows, and separating the control plane (UI/API) from the data plane (job execution) when needed.

This page describes what scaling means in Xferity and what the real limits are.

What “scaling” means in Xferity

Section titled “What “scaling” means in Xferity”

Xferity is not a distributed system. Scaling in Xferity means:

  1. More worker processes — run multiple xferity processes with worker mode enabled, all sharing the same Postgres database
  2. More parallel files per flow — configure performance.max_parallel_files to process multiple files concurrently within a single flow run
  3. More flows running concurrently — configure performance.max_concurrent_flows in global config
  4. Tuning timeouts and poll intervals — reduce unnecessary wait time in high-volume scenarios

It does not mean:

  • horizontal auto-scaling by a Kubernetes operator
  • distributed control-plane management
  • automatic failover or leader election

Multiple workers with shared Postgres

Section titled “Multiple workers with shared Postgres”

The most practical scaling path is running multiple worker processes that share the same Postgres database.

Each worker:

  • polls the shared job queue independently
  • claims jobs with a select-for-update mechanism
  • executes them independently
  • writes results back to the shared database

Workers can run on separate hosts as long as they have access to the same Postgres database and the same configuration and key material (via shared filesystem, NFS, or config management).

worker:
  enabled: true
  poll_interval: 5s
  max_concurrent_jobs: 8
  job_execution_timeout: 300s

Increase max_concurrent_jobs to allow a single worker to handle more jobs in parallel. Tune based on host CPU, memory, and network capacity.

Separating the UI/API from workers

Section titled “Separating the UI/API from workers”

For higher-volume deployments, you can run one process as the UI/API server (without workers) and separate processes as workers (without the UI surface):

UI/API server:

ui:
  enabled: true
  port: 8080
worker:
  enabled: false

Worker process:

ui:
  enabled: false
worker:
  enabled: true
  max_concurrent_jobs: 16

Both share the same Postgres database and config. This separation keeps API response latency independent of job execution load.

Per-flow file parallelism

Section titled “Per-flow file parallelism”

For flows processing many files, enable parallel file handling:

performance:
  max_parallel_files: 8

This is a global setting. Per-flow overrides are available:

flows:
  bulk-invoice-upload:
    performance:
      max_parallel_files: 4

Higher parallelism increases throughput but also increases memory and connection usage. Tune based on the target partner’s endpoint capacity and your host resources.

Global concurrency limits

Section titled “Global concurrency limits”
performance:
  max_concurrent_flows: 10
  max_concurrent_files: 20

max_concurrent_flows limits how many flows can execute simultaneously across the instance.
max_concurrent_files limits the total number of file operations running in parallel across all flows.

Bandwidth limiting

Section titled “Bandwidth limiting”

For environments where egress bandwidth must be controlled:

performance:
  default_bandwidth_limit: 10485760  # 10 MB/s in bytes

File size limits

Section titled “File size limits”

Set global or per-partner file size limits to protect against large unexpected payloads:

security:
  max_file_size: 524288000  # 500 MB

Per-partner overrides are supported through the policy.max_file_size partner field.

Scale limits that currently apply

Section titled “Scale limits that currently apply”
  • Xferity does not auto-scale worker processes. Workers must be started and stopped manually or by your process management layer.
  • There is no built-in HA failover. If the Postgres database becomes unavailable, workers stop processing.
  • Worker-level redundancy via multiple worker processes on separate hosts does not protect against Postgres failure.
  • Very high volume flows (many thousands of files per run) may require tuning Postgres connection pool settings and host resources.

When to consider external scaling

Section titled “When to consider external scaling”

Consider adding infrastructure around Xferity when:

  • a single worker host cannot keep up with job volume despite max_concurrent_jobs tuning
  • you need geographic distribution for partner proximity
  • upstream or downstream systems require multiple ingress paths

In these cases, the typical pattern is multiple Xferity worker processes behind a shared Postgres database, with each worker responsible for a subset of flows by convention rather than automatic partition.

Section titled “Related pages”