SFD Large File Support and "EOF" Markers

Secure File Delivery (SFD) is the Synctera product that delivers sensitive data at scale, via SFTP+PGP. It’s of chief interest to Synctera partners who are looking to ingest our platform data into their own data warehouses, data lakes or databases.

SFD takes each of the core tables of My Data and creates an encrypted file in our SFTP server, for partners to download.

In June, 2023, we introduced feature support for large table output in SFD. With this feature, when the file size for an output file would be > 1 GB, we split the output into a series of 50 MB files.

Once the table has completed export (whether as a single file or as multiple smaller files), we write an EOF marker file so that you know that all data for that table, as part of this export batch, have been written to disk.

This way, you know that you've received all the files that correspond to the source table.

An EOF file is generated for every file creation, irrespective of whether it's been split.

You don't need to take action on any of the "EOF" files. Indeed you can ignore them, as long as you keep track of all the files that you've processed. There's no row-wise ordering in the files that we create.