Salesforce to Iceberg: How to Load CRM Objects and Keep Tables Current

To copy Salesforce to Apache Iceberg and keep it current, load each object once with Bulk API 2.0, then keep querying for records whose SystemModstamp has moved and merge them into an Iceberg table on Id. Four tools do that: a Spark or dlt pipeline you build, AWS Glue zero-ETL, Snowflake's Openflow connector writing Iceberg tables (in Preview), or a managed pipeline platform. Build your own for a few objects, use Glue zero-ETL where hourly AWS syncs are fine, Openflow when Snowflake manages your tables, and a managed platform for many objects or destinations. Data 360 zero copy is different: it lets Data 360 read Iceberg tables you already have.

Salesforce exposes no database log, so every poll spends the org's API allocation, deletes and formulas need explicit handling, and every merge leaves files to maintain. Estuary is one managed option for this route: its Salesforce capture backfills through Bulk API 2.0 and polls for changed records, and its Apache Iceberg materialization merges them into tables with Spark jobs on your EMR Serverless application, on a sync schedule you set.

Choosing a Salesforce to Iceberg method

Amazon AppFlow exports Salesforce objects to S3 as JSONL, CSV or Parquet files, not Iceberg tables. The first, third and fourth methods all poll Salesforce's APIs, so most of this guide is about polling and merging well.

How Salesforce API sweeps become Iceberg snapshots

A poller keeps one Iceberg table per Salesforce object, and every cycle ends in a new snapshot.

1. Initial load. Salesforce steers jobs over 2,000 records to Bulk API 2.0, so each object starts with one Bulk query job. Results are CSV, and the SOQL must name address and geolocation components, because compound fields are off limits.
2. Change sweeps. REST queryAll over a SystemModstamp window. SystemModstamp moves when a user or an automated process changes a record (LastModifiedDate tracks users only), and queryAll also returns deleted and merged records, flagged IsDeleted = true.
3. Merge. Spark or PyIceberg runs MERGE INTO on Id from a staged batch, writing Parquet data files, or small delete files in merge-on-read mode.
4. Commit. The REST catalog points the table at its new metadata file, so Athena, Trino, Spark or Snowflake see each merge whole, as one snapshot.

Salesforce requirements

Every API-based copy draws on one daily allocation per org, and Bulk API 2.0 requests count against it too. Enterprise orgs, and Professional orgs with API access added, get 100,000 requests a day plus 1,000 per Salesforce license; Unlimited and Performance orgs get 5,000 per license on the same base, and Developer Edition 15,000. Bulk query jobs have their own rolling 24-hour caps of 10,000 jobs and 1 TB of results. The REST /limits resource reports both (DailyApiRequests, DailyBulkV2QueryJobs).

The integration user decides what the copy sees. Give it API Enabled, read access and field-level security on every object and field you copy, and View All Data, because API queries respect the sharing model. If login policy blocks login.salesforce.com, sign in through your My Domain host, such as acme.my.salesforce.com.

Iceberg requirements

• A catalog: the AWS Glue Data Catalog through its Iceberg REST endpoint, S3 Tables, Snowflake Open Catalog, Apache Polaris or another REST catalog. It decides which engines can find the tables.
• Storage and compute: an S3 bucket or table bucket, IAM permissions for the catalog and the writer, and Spark on Amazon EMR or EMR Serverless (PyIceberg for small volumes).
• One naming rule: Salesforce field names are mixed case (AccountId, SystemModstamp). S3 Tables needs lowercase table and column names for Athena and the Glue Data Catalog to see a table, and Snowflake resolves unquoted identifiers as uppercase, so fix the casing before the first table exists.

sql
CREATE TABLE lake.salesforce.opportunity (
  id             STRING,
  name           STRING,
  stagename      STRING,
  amount         DECIMAL(18,2),
  isdeleted      BOOLEAN,
  systemmodstamp TIMESTAMP
) USING iceberg
TBLPROPERTIES ('format-version' = '2');

Setting up Salesforce to Iceberg replication

These steps build the pipeline in Spark; dlt with PyIceberg follows the same order.

1. Create the integration user and check headroom in /limits.
2. Choose objects and read each describe, parents included, for every field's type, precision and scale.
3. Create one Iceberg table per object, keyed on Id, with decimal currency and percent columns, keeping IsDeleted and SystemModstamp.
4. Load existing records with one Bulk API 2.0 queryAll job per object, appended to the empty table.
5. Sweep and merge. Query a closed window that ends a few minutes before now, keep the newest row per Id, merge, and save the cursor only after the commit succeeds:

sql
-- SOQL, sent through REST queryAll
SELECT Id, Name, StageName, Amount, IsDeleted, SystemModstamp
FROM Opportunity
WHERE SystemModstamp > 2026-10-22T09:00:00Z
  AND SystemModstamp <= 2026-10-22T09:25:00Z

-- Spark SQL, with the sweep loaded into opportunity_changes
MERGE INTO lake.salesforce.opportunity t
USING opportunity_changes s
ON t.id = s.id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

6. Schedule formula re-reads and maintenance: re-read formula columns daily, expire snapshots, and compact tables that merge often.
7. Validate. Compare record counts with Salesforce. Then change one record of each kind (an edit, a delete, a new custom field, a parent value read by a formula) and diff the table VERSION AS OF the previous snapshot.

What happens after the initial load?

• Updates from users, flows and triggers alike replace the row with the same Id, each merge adding a snapshot.
• Deletes and merges come back from queryAll with IsDeleted = true, including a merge's losing record. UPDATE SET * keeps them: filter them in a view, or add WHEN MATCHED AND s.isdeleted THEN DELETE.
• Purged records never arrive as deletes: once a record leaves the Recycle Bin no API returns it, so its row stays until you compare Id lists.
• Formula fields are evaluated each time a query reads them. Editing a parent record moves the parent's SystemModstamp but not the child's, so a child formula that reads the parent keeps its old value in Iceberg until re-read.
• New custom fields become new columns through Iceberg schema evolution, a metadata-only change; earlier rows hold NULL until the object is re-read.

Salesforce to Iceberg data type mapping

Currency and percent precision

The API returns currency, percent and number values as doubles, which hold about 15 significant digits, so totals summed over a double column pick up rounding error. Describe gives each field's precision and scale: create the column as decimal(precision, scale) and cast at load.

Performance, latency and Iceberg cost

Freshness is the poll interval plus any safety lag plus the merge interval, so a polled table is minutes behind Salesforce. Cost comes from API requests, merge compute and the files merges leave behind.

Salesforce API budget

Polling cost scales with objects × polls per day × requests per poll, plus extra queries for wide objects, result pages and Bulk jobs. Polling faster than you merge buys no freshness, so set the two intervals together.

Copy-on-write vs merge-on-read

Iceberg's default for merges, updates and deletes is copy-on-write: every data file holding a changed row is rewritten whole, which suits Salesforce tables read far more often than they change. Merge-on-read (format v2 and later) writes small delete files instead, so a merge touching a few opportunities spread across large files writes far less, and readers apply the deletes until compaction. Set it per table with write.merge.mode, write.update.mode and write.delete.mode.

Table maintenance

Iceberg recommends expiring snapshots, removing old metadata files and deleting orphan files; it calls compaction optional, but tables merged every few minutes need it. Spark runs them as expire_snapshots, remove_orphan_files and rewrite_data_files; on AWS Glue, table optimizers can do the same. Expired snapshots are also gone for time travel.

Production considerations

• Cursor and commit order. Save the cursor after the merge commits: saved early, a crash skips records; saved late, it replays a window the Id merge absorbs.
• Concurrent writers. Compaction and a merge can touch the same files, and Iceberg fails whichever commit loses the race, so retry merges instead of failing the run.
• Monitoring. Alert on API usage against the allocation, cursor age per object, failed merge jobs and snapshot count per table.

Common problems and troubleshooting

Athena can't read mixed-case Salesforce columns in S3 Tables

Athena fails with GENERIC_INTERNAL_ERROR: Get table request failed: com.amazonaws.services.glue.model.ValidationException: Unsupported Federation Resource - Invalid table or column names. The table kept Salesforce's field names, such as SystemModstamp, and S3 Tables tables with capitals in their definition aren't supported by the Glue Data Catalog. Recreate the table with lowercase names and reload it, and lowercase every column before the first CREATE TABLE.

MERGE fails when a batch repeats an Id

Iceberg's Spark MERGE INTO lets only one source row update a given target row, and throws an error otherwise. A batch repeats an Id when a replayed window overlaps the last one, or a re-load page and a sweep land together. Keep the newest row per Id:

sql
CREATE OR REPLACE TEMP VIEW opportunity_changes AS
SELECT id, name, stagename, amount, isdeleted, systemmodstamp
FROM (
  SELECT *, ROW_NUMBER() OVER (PARTITION BY id ORDER BY systemmodstamp DESC) AS rn
  FROM opportunity_sweep
) WHERE rn = 1;

Queries slow down after weeks of merges

Scans slow down while the row count barely moves. Check the metadata tables:

sql
SELECT count(*) AS data_files, avg(file_size_in_bytes) AS avg_bytes
FROM lake.salesforce.opportunity.files;
SELECT count(*) FROM lake.salesforce.opportunity.snapshots;

Thousands of small files, or delete files on a merge-on-read table, mean merges have outpaced maintenance. Run CALL lake.system.rewrite_data_files(table => 'salesforce.opportunity'), then expire_snapshots, and schedule both from then on.

Using Estuary for Salesforce to Iceberg

Estuary pairs the Salesforce capture with the Apache Iceberg materialization: each object becomes a collection keyed on Id, merged into its own Iceberg table by Spark jobs on your EMR Serverless application. Against the concepts above:

• Polling: the capture never subscribes to Change Data Capture events. First loads use Bulk API 2.0, except for objects with a compound or base64 field: Account and User carry addresses, so they load through REST queries. Sweeps run REST queryAll every 5 minutes per object by default, skipping the latest 5 minutes.
• Merges: the materialization reads the stored rows for each batch's keys, runs MERGE INTO, and commits on the sync schedule, 30 minutes by default once caught up.
• Deletes: the capture writes a deleted record as an ordinary change whose IsDeleted is now true, never as a delete event, so the row survives even with Hard Delete on. Query with WHERE NOT IsDeleted.
• Formula fields: re-read on a per-object cron, 23:55 UTC daily by default, and merged into the stored rows.
• Types: currency, percent and custom number fields land as double, with no per-field decimal type, so cast in a view: CAST(amount AS DECIMAL(18,2)) is exact up to about 15 significant digits. Custom numbers are captured as decimal strings, which castToString: true stores unchanged. Addresses and the full document land in variant columns, so new tables are format v3 and need Spark 4 (emr-spark-8.0.0 or later); check that your query engine reads them, or set the no_variant_columns feature flag for JSON strings.
• Catalogs: AWS Glue through its Iceberg REST endpoint, S3 Tables, Snowflake Open Catalog, Dremio Cloud and other REST catalogs whose storage is Amazon S3, not Google Cloud or Azure.
• Limits: discovery offers custom objects and a built-in list of standard objects that support extends on request; external (__x) and big (__b) objects never appear. The append-only Amazon S3 Iceberg (delta updates) connector gets formula refreshes as rows empty apart from the key and formulas, so keep current-state tables on the merge connector.
• Compliance: Estuary is SOC 2 Type II certified, HIPAA compliant and GDPR compliant.

Implementing Salesforce to Iceberg with Estuary

These steps adapt Estuary's general data flow guide to this route.

1. Check the prerequisites. Beyond the Salesforce and Iceberg requirements, Estuary needs a read-only Salesforce user and an EMR Serverless application (Spark, emr-spark-8.1.0, x86_64) with outbound internet access, plus a staging bucket in its region, an execution role that trusts EMR Serverless, and an IAM user or role that submits jobs.
2. Create the Salesforce capture. On the Sources page, click New Capture, choose Salesforce, name the capture, and fill in the endpoint properties:
   • Authentication: OAuth through Authenticate your Salesforce account; a username, password and security token (not under enforced SSO); or Client Credentials, with a Consumer Key and Consumer Secret from an External Client App.
   • My Domain: your login host, such as acme.my.salesforce.com; Client Credentials requires it.
   • Sandbox for a sandbox org, and an optional Start Date.

3. Select objects. Click Next. Output Collections lists the supported objects, with Account, Opportunity, Case and others enabled; add custom __c objects. Each binding has an Interval (PT5M by default) and a UTC Formula Field Refresh Schedule (55 23 * * *). Click Save and publish to start the backfills, with sweeps alongside.
4. Create the Apache Iceberg materialization. On the capture's details page, click Materialize, choose Apache Iceberg, and fill in the endpoint properties:
   • URL: https://glue.<region>.amazonaws.com/iceberg for Glue, https://s3tables.<region>.amazonaws.com/iceberg for S3 Tables, or your REST catalog's base URL.
   • Warehouse: the AWS account ID without hyphens (Glue), the table bucket ARN (S3 Tables) or the catalog name.
   • Namespace, such as salesforce, and Base Location (s3://<bucket>/<prefix>/, required with Glue).
   • Catalog Authentication: AWS SigV4 Authentication or AWS IAM Authentication for Glue and S3 Tables; OAuth 2.0 Client Credentials for Snowflake Open Catalog, Dremio and other REST catalogs.
   • Hard Delete: leave off; it ignores IsDeleted updates.

5. Connect the EMR application. Under Compute, the AWS EMR Serverless option takes Region, Application ID, Execution Role ARN, and the staging Bucket and optional Bucket Path. For Authentication, choose AWS AccessKey, or Use Catalog Authentication with SigV4 or IAM catalog credentials. OAuth catalog credentials also need a System Manager Prefix, such as /estuary/.

6. Set the sync schedule and maintenance.Sync Frequency, under Sync Schedule, defaults to 30 minutes (choices 0s to 4h). Each sync runs Spark merges and commits a snapshot to every changed table, so shorter frequencies cost more EMR time and make more small files. A fast-sync window (Timezone, Fast Sync Start Time, Fast Sync Stop Time) limits it to business hours, with 4-hour syncs outside.
   • Glue Table Optimizers (Glue only): an Execution Role ARN, then Enable Compaction, Enable Snapshot Retention and Enable Orphan File Deletion.
   • Advanced Options, before any table exists: Field Name Caselowercase for Athena on S3 Tables, or Table Identifier Case and Field Name Caseuppercase for unquoted Snowflake queries.

7. Map collections to tables and publish. Click Next and pick a naming convention. Each binding under Source Collections has Table, Alternative Namespace and Additional Table Properties, applied only at creation, where the three write.*.mode properties can be set to merge-on-read. Click Save and publish; the backfill loads without waiting for the schedule.
8. Check the first loads.
   • The capture details page reads PRIMARY in its Shard Information card; open a collection from the Collections list to see its Data Preview.
   • On the materialization details page, Data Read rises as backfills land.
   • Compare SELECT COUNT(*) FROM salesforce.opportunity WHERE NOT isdeleted with Salesforce, then run the validation checks.

How Estuary handles formula refreshes, table maintenance and rebuilds

Prodege loads Iceberg tables on S3 with Estuary's Amazon S3 Iceberg (delta updates) connector, not the merge connector, and names no Salesforce source; its story reports a "60% Reduction in Replication Costs".

Formula refreshes merged into complete rows

A refresh selects only Id, the cursor and formula fields, skips objects without formulas, and emits partial records. The materialization reads each key's stored row before its MERGE, so the new values land in complete rows. Each refresh still makes a full pass over the object in Bulk jobs, which count toward the 24-hour caps.

Maintenance through Glue table optimizers

With a Glue catalog, the materialization registers Glue's compaction, snapshot retention and orphan file deletion optimizers on every table it creates and keeps them in step with your settings; merges retry when a maintenance commit conflicts. Elsewhere, run Iceberg's procedures yourself.

Rebuilding Iceberg tables without calling Salesforce

Collections are stored as JSON files in your cloud storage bucket. A materialization backfill rebuilds Iceberg tables from those files, not from Salesforce, as does a later second materialization such as a Snowflake copy. On this connector the backfill drops and re-creates each table, so earlier snapshots are gone for time travel. A Salesforce re-read costs API requests and can't clear rows for records purged since, as the capture has no delete event to send. Collections last as long as the bucket keeps them: about 20 days on Estuary's trial storage.

When another approach may make more sense

• Data 360 customers who only need Data 360 to use lakehouse data: its Apache Iceberg connector reads tables already in your storage through zero copy File Federation (GA), for example through the Glue Iceberg REST endpoint, and Snowflake can query Data 360 data products in place. Neither puts raw Account or Opportunity records in a catalog you control.
• All-AWS, and hourly is fine: Glue zero-ETL keeps the pipeline inside AWS, but it doesn't support User, ContentDocumentLink or many history and share objects, so check its unsupported list first.
• Iceberg tables managed by Snowflake: Openflow's Iceberg option (Preview) keeps deletes as isDeleted rows, can turn formulas into views, and fixes the table format and Iceberg version once ingestion starts.
• Reacting to individual changes: Salesforce Change Data Capture pushes events through the Pub/Sub API, for five objects by default, and keeps them 72 hours, so you still build a backfill and a merge layer. Estuary's connector polls instead.
• A one-time migration: one Bulk API 2.0 export per object, written once, needs no pipeline; Estuary's capture can also sweep on a long Interval.

Related resources

• Estuary: Salesforce connector and its formula fields notes, Apache Iceberg materialization and its table maintenance section, sync schedules, Salesforce to Apache Iceberg integration, Salesforce and Apache Iceberg connector pages
• Salesforce: API request limits, Bulk API 2.0 limits, queryAll, Data 360 Apache Iceberg connector
• Apache Iceberg: Spark writes, maintenance
• AWS: zero-ETL Salesforce source, S3 Tables naming rules, Data 360 zero copy to Iceberg in S3
• Snowflake: Openflow Iceberg destinations, About Salesforce Data Cloud and Snowflake; dlt: Iceberg destination

To try the route, follow Estuary's data flow guide or create a Salesforce capture in the Estuary dashboard. Pricing: estuary.dev/pricing.