Introducing the Native SQS Capture Connector
Read data from your queues in real time
Our latest source connector pulls directly from Amazon SQS, or Simple Queue Service, to stream your messages to any destination.
Capture messages as they occur, down to millisecond latency, to power real-time workflows. Or save those captured messages for a scheduled batch sync. Once the messages are in your Estuary data collection, you can replay them, route them to multiple destinations, or remix them with transformations.
Why It Matters
Estuary previously supported an SQS connector based on a third-party integration. What makes the new native connector different?
Building the integration from the ground up has allowed us to add some distinctly Estuary touches:
- Real-time capture: Message queues are often meant to support event-driven architecture, but event-driven analysis can’t occur when messages are delayed into two-hour batches. The new connector supports streaming messages down to 100 ms latency.
- Multiple queues, one connector: Manage data capture from all your queues in one place. The new connector discovers available queues based on the credentials you give it and stores data in a collection specific to each queue. This consolidates the previous configuration of one SQS queue per connector.
- Support for FIFO queues: Capture from both standard and FIFO queues. Messages within a message group will be captured and committed in queue order when using FIFO queues.
- IAM authentication: Besides AWS access key credentials, the new connector supports AWS IAM authentication so you can keep role and access management within your cloud provider.
Because these features are strict upgrades, the old third-party connector is considered deprecated in favor of this new native SQS connector. For the best experience, we recommend using the native connector going forward.
Getting Started
To begin streaming data from your Amazon SQS queues, all you’ll need is:
- Your AWS region
- An authentication method with SQS permissions; either:
- An AWS secret access key and its ID
- An AWS role ARN
Estuary will kick off auto-discovery. This will return all available SQS queues for the provided credentials in that region. You can review queue URLs and confirm the selection before publishing your capture.
Messages will start streaming into the associated data collections as they occur.
📚 Step-by-step setup instructions: Read the SQS Connector Documentation →
Better with Friends
SQS is only the latest in a line of recent connector releases. Check out what else is new:
Smartsheet Captures
Capture from your Smartsheet spreadsheets and store your data all in one place. All you need to get started is a Smartsheet API access token.
Don’t use Smartsheet for your spreadsheets? Estuary also has connectors for Google Sheets and CSVs in OneDrive, SharePoint, and more.
Dremio Materializations
Materialize your data to Dremio Cloud using this new Apache Iceberg connector variant. Fill in Dremio details for the catalog and use Spark jobs to merge updates into Iceberg tables.
Or you can always use AWS Glue or Bauplan as your Iceberg catalog.
Zuora Captures
Capture information on your recurring revenue with Zuora. Send subscription, invoice, and payment data downstream to keep your accounting and revenue analysis up to date. The capture connector takes credentials from a Zuora OAuth app to discover available objects.
Use Chargebee for your recurring revenue instead? We also support that.
Read messages from your SQS queues in real-time with Estuary, and combine it with a host of other connector options for a complete picture of your data.
