Can You Automate File Transfers Without Losing Control?

In October 2020, a data-loading failure left 15,841 positive COVID-19 cases out of England’s reported daily figures. Public Health England traced the problem to files containing positive laboratory results that exceeded the maximum size the loading process could handle. The issue was identified on 2 October, and PHE announced the missing cases on 4 October. Its official statement confirmed that automated and manual checks were already in place throughout the process.

The incident exposed an important operational risk: a transfer can run automatically while the wider process still fails to deliver the expected result.

Automation reduces repetitive work. But as the number of file transfers grows, teams need to know what each flow does, whether it ran successfully, what changed and who is responsible when something goes wrong.

What does control mean when hundreds of flows are running?

Control means that a colleague who did not build a flow can quickly answer four questions:

• What is it supposed to do?
• What did it do last night?
• What changed since it last worked?
• Who is responsible for it?

These questions are more useful than counting the features in a transfer platform. They test whether a team can understand and operate its file transfers without depending on one person’s memory.

What is the flow supposed to do?

A flow should be understandable without a meeting with its original author.

In Xferity, a self-hosted managed file transfer platform, flows are defined in YAML files. Their schedules, destinations and retry policies can be kept together in a readable configuration. Strict validation rejects unknown configuration fields rather than silently accepting a misspelled setting.

This makes the flow easier to inspect and gives the team a clear definition of how it is intended to operate. A team can also keep YAML files in its existing version-control process, making changes easier to compare and review.

The principle is simple: the flow’s behaviour should be documented in a form that another person can understand.

What did it do last night?

Imagine a warehouse waiting for an order file before the morning shift begins. The first question is whether the file reached its destination. The next is whether the transfer completed on time and, if it failed, what happened during the attempts to deliver it.

Xferity provides structured audit records for file transfers, flow lifecycle events and sign-ins. Its audit records use a SHA-256 hash chain designed to make changes to recorded history detectable. In PostgreSQL-backed deployments, the Web UI provides a dashboard, while Prometheus metrics expose information such as runs, retries and queue depth.

Retries deserve particular attention. A flow that succeeds only after repeated attempts may still be creating an operational risk. Monitoring the number of retries helps teams spot recurring problems before they become failures.

Idempotency checks help prevent a retry from delivering the same file twice. If a file cannot be delivered, dead-letter handling provides a defined place for it instead of leaving the failure hidden in the normal flow.

With many flows in operation, teams need to know when a transfer requires attention - for example, when retries are exhausted, a file enters dead-letter handling or a scheduled flow stops executing as expected. Xferity supports configurable alert events and notification channels, allowing teams to decide which situations trigger alerts and where those alerts are sent.

Xferity’s security posture monitoring identifies flow drift when a scheduled flow is no longer executing as expected and reports the specific reason. This gives teams a clearer starting point for investigating the issue and deciding what needs attention.

There is an important limit to what transfer monitoring can prove. A record can show that a file was delivered intact. It cannot, by itself, prove that the receiving system loaded every row or completed the next business step.

The Public Health England incident illustrates why this distinction matters: the process that loaded the files could not handle their size. Successful delivery and successful processing are two different outcomes, and both need appropriate checks.

What changed since it last worked?

A transfer can stop working after a change: a partner rotates a key, a schedule is edited or an export format changes.

Google’s Site Reliability Engineering book explains why changes to live systems are a significant source of outages. Although this is not a statistic specific to file transfers, the principle applies: teams need a reliable way to understand what has changed.

Because Xferity flows are defined in YAML, teams can keep them in version control and compare revisions using their existing tools. Operator activity is also available for monitoring.

In PostgreSQL-backed deployments, Xferity supports role-aware access and OIDC sign-in. The company still needs to decide who should have access and which permissions each role requires. The platform provides capabilities; the organisation must apply them to its own operating model.

Who is responsible for the flow?

Every important flow should have a named owner and a backup. This can be recorded alongside the flow or in the team’s existing register. The team should also know where its alerts go and who is expected to respond.

Ownership needs to cover the business process, not just the connection. The person responsible for a payment file should understand the payment run it supports, not only whether the SFTP transfer works.

When an employee leaves or changes roles, the team should review the flows and access associated with that person and transfer responsibility where needed.

These steps are operational practices, not a substitute for technical controls. Together, they help ensure that a working flow remains understandable, maintainable and accountable over time.

So, can you automate file transfers without losing control?

Yes - if control is built into the way flows are defined, monitored and maintained, rather than held in the memory of the person who created them.

Audit records, retries, monitoring, access controls and clear ownership each answer a different operational question. Their value lies in helping a colleague understand what happened and decide what to do next, without having to track down the original author.

Start with one flow chosen at random. Give it to someone who did not build it and ask them to answer four questions: what it does, what happened during its last run, what changed and who is responsible.

How long does it take to get all four answers?