Skip to content
Xferity

Tutorial: Building a File Transfer Pipeline

Tutorial: Building a File Transfer Pipeline

Section titled “Tutorial: Building a File Transfer Pipeline”

A file transfer pipeline in Xferity means a sequence of flows where the output of one step becomes the input of the next.

This tutorial shows how to build a complete inbound-process-outbound pattern: download from a supplier SFTP, process in a staging directory, and upload the result to a bank.

Prerequisites: working SFTP flows for at least one partner

What you will build

Section titled “What you will build”

Three flows:

  1. supplier-inbound — download raw files from supplier SFTP
  2. bank-outbound — upload processed files to bank SFTP (triggered after processing)

The processing step (your business logic) runs between the two flows. Xferity handles the transfer steps.

Step 1: Design the directory layout

Section titled “Step 1: Design the directory layout”
storage/
  supplier/
    incoming/        # raw files land here (supplier-inbound output)
    processed/       # your processing logic outputs here
  bank/
    outgoing/        # bank-outbound input

Step 2: Create the supplier SFTP partner

Section titled “Step 2: Create the supplier SFTP partner”

partners/supplier-sftp.yaml — see the SFTP partner reference.

Step 3: Create the bank SFTP partner

Section titled “Step 3: Create the bank SFTP partner”

partners/bank-sftp.yaml:

id: bank-sftp
display_name: Bank SFTP
protocol: sftp
sftp:
  host: sftp.bank.example
  port: 22
  user: xferity-upload
  remote_dir: /incoming
  known_hosts: file:./keys/bank-known_hosts
  key_path: file:./keys/bank_upload_rsa
  key_passphrase: env:BANK_KEY_PASS

Step 4: Create the inbound flow

Section titled “Step 4: Create the inbound flow”

flows/supplier-inbound.yaml:

flows:
  supplier-inbound:
    direction: download
    enabled: true
    source:
      partner: supplier-sftp
      path: /outgoing
    local:
      path: ./storage/supplier/incoming
    files:
      - pattern: "*.csv"
    idempotency_mode: hash
    schedule_cron: "0 0 6 * * 1-5"

Step 5: Create the outbound flow

Section titled “Step 5: Create the outbound flow”

flows/bank-outbound.yaml:

flows:
  bank-outbound:
    direction: upload
    enabled: true
    target:
      partner: bank-sftp
      path: /incoming
    local:
      path: ./storage/bank/outgoing
    files:
      - pattern: "*.csv"
    idempotency_mode: hash
    delete_after_upload: true
    schedule_cron: "0 30 7 * * 1-5"

Step 6: Connect the processing step

Section titled “Step 6: Connect the processing step”

Your processing logic reads from ./storage/supplier/incoming and writes results to ./storage/bank/outgoing. This is external to Xferity — it could be a script, another service, or manual review.

The timing relationship:

  • supplier-inbound runs at 06:00
  • your processing step runs between 06:00 and 07:30
  • bank-outbound runs at 07:30 assuming processing is complete

Step 7: Validate and test

Section titled “Step 7: Validate and test”
Terminal window
xferity validate
xferity diag supplier-inbound
xferity diag bank-outbound
xferity run supplier-inbound
# (run your processing step)
xferity run bank-outbound

Review each flow’s history and audit trace.

Adding PGP to the pipeline

Section titled “Adding PGP to the pipeline”

If the bank requires encrypted files, add PGP encryption to the outbound flow:

flows:
  bank-outbound:
    # ... source/target/files as above
    pgp:
      provider: gopenpgp
      encrypt: true
      public_key_path: file:./keys/bank-public.asc
    delete_after_upload: true
    delete_encrypted_after_upload: true
Section titled “Related pages”