SocialHP Scales Real-time Analytics to the Next Level with Estuary and ElasticSearch
Challenge
- Struggled with analytics performance, scalability with their Postgres-based deployment.
- Getting more and more difficult to add new features and maintain.
- New features required much lower latency than current 1 minute aggregation intervals.
Solution
- Built a new change data capture (CDC) pipeline with Estuary in two days from Postgres to ElasticSearch, their new social analytics platform.
Results
- Done in 2 days. Everything from learning Estuary, to implementing CDC from Postgres to ElasticSearch, to testing and going into production took a few days to complete.
- 100% Reliable. According to SocialHP, Estuary just works. It has never gone down since day 1 and requires no maintenance. They have barely had to think about Estuary.
- Second-level latency. With Estuary, the data is always up to date. It streams in real-time from Postgres using CDC and loads directly into Elasticsearch within roughly a second.
- Enabled new analytics. Getting data into Elasticsearch in real-time made it easy for SocialHP to add new analytics to the SocialHP platform.
- High quality support. Whenever SocialHP did have questions, Estuary was very responsive.
About SocialHP
location
Toronto, Canada
INDUSTRY
Social Media
GOALS
Build a new social analytics platform to support new features and rapid growth.
SocialHP provides a single place for companies to manage and measure all their social media marketing and advocacy efforts by employees. This includes social media management, social advocacy features for employees, scheduling, and social performance reporting and analytics.
Challenge
As SocialHP continued to grow quickly and collect more data, it started to struggle with analytics performance, scalability, and adding new features. The main reason was that SocialHP had pushed the limits of their internal PostgreSQL deployment.
Postgres is a very capable database for writes and transactions, but SocialHP had hit several limits with using Postgres for analytics. They had built a system of event tables, triggers and multiple levels of rollup tables to proactively aggregate statistics once a minute. But this system was getting more complex and difficult to maintain. Data growth rates were increasing, and total volumes were making scaling harder with each new month. New features such as filtering needed lower latency and faster performance than what Postgres was able to provide.
Solution
Shane Iseminger, the CTO at SocialHP, led an effort to build a new social analytics platform that could support the new features and handle the growth. They chose ElasticSearch as their platform for social analytics. Now they needed to find the best way to stream real-time changes from Postgres with very low latency.
The team evaluated open source offerings, but found they weren’t robust enough for their needs, and would have required additional resources Shane felt should be focused on building new features. So the team shifted to evaluating managed cloud services.
SocialHP ended up choosing and implementing Estuary real-time change data capture (CDC) from Postgres to ElasticSearch.
Results
Social analytics is a core part of SocialHP’s value for customers. Estuary and Elasticsearch made it possible for SocialHP to increase that value by quickly adding new ways to analyze and track company executives’ social media performance vs their competitors.
We needed a platform that allowed us to give our customers more ways to analyze and visualize their social media performance vs their competitors, and scale with our rapid growth. With Estuary we were able to get enterprise-grade social analytics up in a few days with second-level latency and performance.
Shane Iseminger - CTO, SocialHP
What’s next for SocialHP? They’re in the process of launching a major revision to the SocialHP platform, and one of the biggest enhancements is in their analytics, including adding even more data for deeper analytics, all made possible by Estuary and ElasticSearch.