Salesforce to S3: Export Objects as Parquet and Capture Changes

Salesforce objects such as Account, Opportunity and custom objects (__c) can be landed in Amazon S3 as Parquet files and extended with their changes on a schedule. Four approaches are realistic: Amazon AppFlow, AWS Glue (a zero-ETL integration or an ETL job), a Python job, or a managed pipeline platform. AppFlow fits AWS-centred teams copying a handful of objects; Glue zero-ETL fits teams that want Apache Iceberg tables rather than loose files; Python fits custom logic; a managed platform fits many objects or several destinations.

Three decisions matter more than the tool: whether Parquet columns keep their Salesforce types, how formula values stay current when their changes don't move a timestamp, and whether deleted records reach the files. Copying Salesforce Files and attachments to S3, or taking backups, is a separate storage job. Estuary is one managed option for this route: its Salesforce connector backfills through Bulk API 2.0 and polls each object for changes on an interval, and its Amazon S3 Parquet materialization appends those changes to Parquet files on an upload interval.

Choosing a Salesforce to S3 method

Every row reads Salesforce through the same APIs as any other Salesforce integration, so the differences lie in typing, change detection and what each run writes to the bucket.

How Salesforce to S3 Parquet exports work

Salesforce keeps no change history you can replay from an object's first record, so each method copies an object in two phases and writes files that are never edited afterwards.

• Initial load. Each object is read in full. Salesforce recommends Bulk API 2.0 above 2,000 records; its query jobs return CSV only and can't select compound address or geolocation fields.
• Ongoing changes. Later runs ask the REST API for records whose SystemModstamp moved since the last run. It changes on automated updates as well as user edits, while LastModifiedDate follows user edits only, and a queryAll request also returns deleted records. The alternative is subscribing to Change Data Capture events.
• File writing. Each run, or each upload interval, adds Parquet files under the object's prefix. A changed record becomes a new row in a new file.
• Query. A catalog such as the AWS Glue Data Catalog describes the files, and Athena, Spark or Trino keeps the newest row per Id for current state.

Salesforce requirements

The org's daily API allocation is shared by every integration and installed package, and Bulk API 2.0 requests count toward it.

• Allocation. Enterprise Edition gets 100,000 requests a day plus 1,000 per Salesforce license; Unlimited and Performance get 5,000 per license. Bulk API 2.0 also caps query jobs at 10,000 and results at 1 TB per rolling 24 hours.
• An integration user with API Enabled, read access to each object and field-level security on each field; hidden fields never reach the API. Without View All Data, every API call sees only the records the user's sharing rules allow.
• An OAuth app. AppFlow creates its own connected app when you authorize with an authorization code. JWT bearer and Client Credentials sign-in need a Connected App or External Client App that you create.
• Change Data Capture, for event-driven flows only. An org can select 5 entities by default, and events are kept for 72 hours.

Amazon S3 requirements

Four S3 details decide whether files arrive and stay readable.

• Bucket and Region. AppFlow writes only to an S3 bucket in the flow's own Region and doesn't write across accounts. Keep the bucket in the Region where Athena runs.
• Writer permissions. Multipart uploads, which writers use for large files, need s3:PutObject. With SSE-KMS default encryption, the writer also needs kms:GenerateDataKey and kms:Decrypt on the key.
• A lifecycle rule with AbortIncompleteMultipartUpload. S3 keeps billing the parts of an interrupted upload until it is completed or aborted.
• One prefix per object, such as salesforce/opportunity/, and a Glue Data Catalog database. Athena reads every file under a table's location, so two objects must never share a folder.

Setting up Salesforce to S3 replication

These steps use AppFlow's settings where a choice is tool-specific; Glue jobs and Python face the same decisions.

1. Check the API budget. As the integration user, read DailyApiRequests and DailyBulkV2QueryJobs from the REST /limits resource before scheduling anything.
2. Prepare the bucket: a prefix per object, the lifecycle rule and a Glue database.
3. Pin the Salesforce API. AppFlow's default, Automatic, uses REST below 1,000,000 records and Bulk API 2.0 above, and the two format dates and nulls differently. Choose Bulk for large objects, or Standard (REST) for objects with address fields.
4. Write typed Parquet. Set preserveSourceDataTyping to true, or AppFlow writes every Parquet column as a string.
5. Load history first with a full on-demand run per object. A scheduled incremental flow's first run reaches back only 30 days.
6. Schedule incremental runs on SystemModstamp, include deleted records, and re-read objects with formula fields periodically.
7. Validate. Compare record counts. Edit a record, delete one and add a custom field, then check after the next run that the edit is a new row, the deleted record has IsDeleted true, and the new field is a typed column. Current state is the newest row per Id:

sql
SELECT *
FROM (
  SELECT o.*,
         ROW_NUMBER() OVER (PARTITION BY id ORDER BY systemmodstamp DESC) AS rn
  FROM salesforce.opportunity o
) latest
WHERE rn = 1 AND NOT isdeleted;

This query assumes complete rows. For Estuary with scheduled formula refreshes, use the formula-refresh view below: refreshes append partial rows with the same SystemModstamp, so this query can pick a partial row and omit the record.

What happens after the initial load?

• Updates arrive as new rows; older rows for the Id stay in older files.
• Deletes reach the files only through queryAll or an include-deleted option, which returns deleted records and the losing records of a merge with IsDeleted true, plus archived Task and Event records. Without it, the last active row stays current forever.
• Purged records can't be queried at all. The Recycle Bin keeps deleted records for 15 days, so a record deleted and purged between two runs never reaches S3.
• Formula fields hold no stored value; Salesforce computes them at query time. A result can change without the record's SystemModstamp moving, so an incremental copy keeps the old value until the object is re-read.
• New custom fields appear as columns in later files only. Athena reads Parquet columns by name, so that's fine there; Spark needs mergeSchema, which is off by default.
• Date folders record when the file was written. AppFlow's Date and time partition key is the flow run time in UTC, so it can't prune by CloseDate.

Salesforce to Parquet data type mapping

Parquet columns that arrive as strings

String columns look fine until a query compares them. Bulk API 2.0 returns CSV, so every Bulk-based tool must re-type each value, and AppFlow writes Parquet columns as strings unless preserveSourceDataTyping is true. A string CloseDate sorts as text and a string Amount can't be summed, so fix types at the writer rather than casting in every query. Currency and percent need a decision either way: the API returns them as doubles, which hold about 15 significant digits, so cast them to DECIMAL(precision, scale) from the field's describe wherever totals must reconcile.

Performance, latency and S3 cost

A Salesforce change reaches a query after up to one read interval, one file-writing interval and any catalog step. None of the four methods is real time: even Change Data Capture events wait for the next file to be written.

Where the API requests go

In AWS's own example, an hourly flow that reads five pages from Salesforce makes 120 API calls a day. Multiply by objects and shorter schedules, add each Bulk job's submit, status and result requests, and count formula re-reads, which read every record. Copy only the objects you query, as often as their consumers look.

Small files and compaction

Short schedules produce many small files, at least one per changed object per run. With AppFlow's Don't aggregate setting, file size follows the API's page size; Aggregate records into multiple files takes a target size instead. Compact older folders with a scheduled CREATE TABLE AS SELECT, or use Iceberg tables, which Glue can compact, and let a lifecycle rule move old folders to cheaper storage classes.

Production considerations

• Recovery. Incremental runs must restart from the last successful window, not from now. AppFlow's incremental transfer reads changes since the last successful run.
• Source impact. Every poll draws on the org-wide allocation that other integrations share. Alert on DailyApiRequests before it runs out.
• Re-loads duplicate. A full re-read appends a second copy of every record. Write it to a new prefix and switch the table, or deduplicate by Id and SystemModstamp.
• Monitoring. Alert on failed runs, Bulk job failures and the age of the newest file per object.

Common problems and troubleshooting

Every column is a string in Athena

AppFlow wrote the files with preserveSourceDataTyping false, so a crawler or CREATE TABLE sees only STRING columns. Check one file's schema with parquet-tools schema or DuckDB's DESCRIBE. Set the option to true and reload the object into a new prefix, so no table reads string and typed files together.

Only 30 days of history arrived

A scheduled incremental AppFlow flow's first run reads only the past 30 days. Run the flow on demand once with a filter covering all of history, then switch it to a schedule, so later runs read changes since the last successful run.

Address and geolocation fields missing

Bulk API 2.0 can't select compound fields: a Bulk query that names one fails with Selecting compound data not supported in Bulk Query, and AppFlow's Bulk runs drop them. Map the component fields instead (BillingStreet, BillingCity, BillingLatitude and so on), or read the object over REST.

Using Estuary for Salesforce to S3

On this route, Estuary's Salesforce capture connector writes each object to a collection keyed on Id, and the Amazon S3 Parquet materialization appends each collection to Parquet files under its own path.

• Reading Salesforce: each object is polled every Interval (5 minutes by default) with REST queryAll on SystemModstamp (or the object's next-best timestamp), re-describing the object first so new custom fields are picked up. Each poll stops 5 minutes short of the current time, allowing for Salesforce's eventual consistency. Initial loads run as Bulk API 2.0 jobs, apart from objects carrying compound fields, such as Account and User, which load over REST.
• Writing files: new files land about every Upload Interval (5 minutes by default) once the pipeline is caught up, and as fast as data arrives during a backfill. A file becomes visible only when its transaction commits.
• Rows: each file holds one row per changed Id per transaction, with a flow_published_at timestamp. A deleted record arrives as a row with IsDeleted true; records purged before a poll sees them never arrive.
• Formula fields: a per-object cron (default 23:55 UTC daily) re-reads them. Each refresh appends a row per record with only Id, the cursor field and formula values filled (see below).
• Types: standard fields land typed (BOOLEAN, INT64, DOUBLE, DATE, UTC TIMESTAMP). Custom fields are captured as text, because Salesforce's describe often mistypes them: custom numbers still land as DOUBLE or INT64, but custom checkboxes land as the strings true and false. Compound addresses land as JSON text beside their component columns. Nothing lands as DECIMAL; step 6 below shows how to keep exact digits.
• Limits: files are always Snappy-compressed. Testing the materialization doesn't contact S3, so a wrong bucket or credential shows up only when the task runs.
• Compliance: SOC 2 Type II, HIPAA and GDPR.

Implementing Salesforce to S3 with Estuary

These steps adapt Estuary's general data flow guide to this route.

1. Complete the prerequisites in Salesforce requirements and Amazon S3 requirements. Estuary also needs:
   • A read-only Salesforce user, plus an External Client App for Client Credentials sign-in.
   • An access key with s3:PutObject on the bucket, or a role in the bucket's account that Estuary assumes through AWS IAM authentication.
   • The multipart upload cleanup lifecycle rule.
2. Create the Salesforce capture. On the Sources page, click New Capture, choose Salesforce, name the capture, and fill in the endpoint properties:
   • Authentication: OAuth through Authenticate your Salesforce account; UserPass (username, password and security token), which fails under enforced SSO; or Client Credentials, with a Consumer Key and Consumer Secret.
   • My Domain: your login host, such as mycompany.my.salesforce.com. Client Credentials requires it.
   • Sandbox: on for a sandbox org.
   • Start Date: records with an older cursor timestamp are skipped. Blank means 1999-02-03, Salesforce's founding date.

3. Select objects. Click Next. Output Collections lists the supported objects, with 17 standard ones such as Account, Opportunity and User enabled; enable custom __c objects as needed. Per binding:
   • Interval: the poll interval, PT5M by default. Longer intervals spend fewer API requests.
   • Formula Field Refresh Schedule: a UTC cron, 55 23 * * * by default. Clear it for objects whose formulas you don't query.

Click Save and publish. Backfills start, with polling for changes running alongside them.

4. Create the Amazon S3 Parquet materialization. On the capture's details page, click Materialize, choose Amazon S3 Parquet, and fill in the endpoint properties:
   • Bucket and Region.
   • Authentication: Access Key (AWS Access Key ID, AWS Secret Access key) or AWS IAM (AWS Region, AWS Role ARN).
   • Prefix: a folder such as salesforce, with no leading slash and no s3://.
   • File Size Limit (10 GiB by default) and, under Parquet Configuration, Row Group Row Limit (1,000,000 rows) and Row Group Byte Limit (512 MiB).

5. Choose the Upload Interval.Upload Interval offers 1s, 1m, 5m, 15m, 30m and 1h, with 5m the default. Shorter intervals give fresher files and more of them; backfills don't wait for it.
6. Set paths and keep exact numbers. Click Next. Under Source Collections, each binding's Path defaults from the collection name, giving each object its own folder. Date patterns such as %Y/%m/%d group files by when Estuary wrote them (UTC), not by a Salesforce date. Keep CreatedDate and IsDeleted selected: the formula-refresh view below uses them to identify full rows and exclude deleted records. Custom number fields arrive as the text Salesforce sent, so castToString keeps their digits exactly in a STRING column: add Discount_Amount__c: { castToString: true } under the binding's fields.include in the specification. Standard Currency and Percent fields arrive as doubles; cast them to DECIMAL in your view.
7. Publish. Click Save and publish. Backfill files start landing without waiting for the upload interval.
8. Verify that data is landing.
   • On the capture details page, the Shard Information card reads PRIMARY; open a collection from the Collections list to see its Data Preview.
   • The materialization details page charts Data Read.
   • In S3, files arrive as <Prefix>/<Path>/v0000000000/00000000000000000000.parquet, numbered upward. Point an Athena table at a binding's v0000000000/ folder and build the formula-refresh view below. Run the record-count, edit and delete checks against that view, then confirm a formula refresh updates its formula value without blanking the other fields. If formula refreshes are disabled, use the simpler query in Setting up Salesforce to S3 replication.

How Estuary handles formula refreshes, re-backfills and extra destinations

Formula refresh rows in append-only files

A formula refresh selects only Id, the cursor field and the formula fields, for every record from Start Date on, including deleted records still in the Recycle Bin. A destination that merges by key folds those values into the stored record. S3 Parquet only appends, so each refresh lands as a second, mostly null row per record, newer than the record's last full row. A plain newest-row query returns it. Build current state from the newest full row and take formula values from the newest row of either kind:

sql
WITH full_rows AS (
  SELECT o.*,
         ROW_NUMBER() OVER (PARTITION BY id ORDER BY flow_published_at DESC) AS rn
  FROM salesforce.opportunity o
  WHERE createddate IS NOT NULL      -- refresh rows leave CreatedDate null
),
formulas AS (
  SELECT id, max_by(days_open__c, flow_published_at) AS days_open__c
  FROM salesforce.opportunity
  GROUP BY id
)
SELECT f.*, m.days_open__c AS days_open_current
FROM full_rows f JOIN formulas m ON f.id = m.id
WHERE f.rn = 1 AND NOT f.isdeleted;

Where formulas don't matter, clear the object's Formula Field Refresh Schedule and the extra rows stop.

One capture for S3 and a warehouse

Destinations read the collection, not Salesforce. Collections are stored as JSON files in your cloud storage bucket, so adding a warehouse beside S3 is a second materialization on the same collections: no second set of API requests and no second backfill against Salesforce.

Re-backfills and version folders

A materialization backfill rewrites a binding from the collection without calling Salesforce. It starts a new version folder (v0000000001/) numbered from zero and leaves the old one in place, so repoint the Athena table and delete the old folder when ready. A capture backfill re-reads Salesforce, spending Bulk jobs and API requests again, and can't remove rows for records purged meanwhile, because the capture emits no delete events for them.

When another approach may make more sense

• AWS-only, a few objects: AppFlow needs no extra vendor, at the cost of one flow per object and a 15 GB cap per run from Salesforce.
• Iceberg tables with no code: a Glue zero-ETL integration replicates Salesforce objects into Iceberg tables that Glue can compact, but doesn't support some objects, among them User, AccountHistory and OpportunityFieldHistory. For merged Iceberg tables from Estuary, the Apache Iceberg materialization merges updates with Spark jobs on your EMR Serverless application.
• One-off export: a Bulk API 2.0 queryAll job per object, converted to Parquet, with nothing left running. Results stay downloadable for 7 days.
• Spark or Airflow already in place: a Glue ETL job filters with Spark SQL (FILTER_PREDICATE), uses Bulk API 2.0 when TRANSFER_MODE is ASYNC, and reads deleted records only with IMPORT_DELETED_RECORDS. Airflow's SalesforceToS3Operator writes CSV, JSON or newline-delimited JSON, not Parquet. dlt's verified source covers 15 standard objects and reloads 8 of them in full every run.
• Files, attachments and backups: AppExchange apps or Apex callouts move documents, and backup products cover recovery; neither produces query-ready objects.

Related resources

• Estuary: Salesforce connector and its formula fields and field permissions notes, Amazon S3 Parquet materialization and its Parquet data types, delta updates, backfilling data
• Estuary pages: Salesforce to Amazon S3 Parquet integration, Salesforce connector, Amazon S3 Parquet connector
• Salesforce: Bulk API 2.0 query jobs, system fields, API request limits, Change Data Capture considerations
• AWS: AppFlow tutorial for Salesforce to S3, AppFlow Salesforce connector, AppFlow S3 destination, Glue zero-ETL refresh interval, aborting incomplete multipart uploads

To try the route, follow Estuary's data flow guide or create a Salesforce capture in the Estuary dashboard. Pricing: estuary.dev/pricing.