Google BigQuery Explained in 3 Minutes: An Overview

Try us free:
https://www.estuary.dev/

Join our Slack channel with a community of developers:
https://estuary-dev.slack.com/

0:00 Intro
0:12 What is BigQuery?
0:21 Petabytes of data
0:36 Characteristics
0:48 When did BigQuery come about?
1:00 Benefits
1:34 Architecture
2:14 Under the hood
2:49 Ingesting data

References:
https://estuary.dev/cloud-sql-to-bigquery/
https://en.wikipedia.org/wiki/BigQuery
https://cloud.google.com/blog/products/bigquery/bigquery-under-the-hood
https://medium.com/google-cloud/bigquery-explained-overview-357055ecfda3

#bigquery

What is BigQuery?

BigQuery is Google's fully managed, serverless data warehouse that enables scalable analysis over petabytes of data.

How much is petabytes?

One Petabyte is the equivalent of 20 million tall filing cabinets, or 500 billion pages of standard printed text.

It’s A LOT of data!

Also, BigQuery is a Platform as a Service (PaaS) that supports querying using dialects of SQL. It also has built-in machine learning capabilities, which everyone is going after these days.

When did BigQuery come about?

BigQuery was announced in May 2010 and made generally available in November 2011.[1] So it’s been around for over a decade!

What are some benefits of using BigQuery?

There are many, but here are just a few:

With BigQuery, you no longer have to provision and forecast compute and storage resources beforehand. BigQuery allocates all the resources based on usage, dynamically.

Also, BigQuery provides super fast analytics on a petabyte scale through its unique capabilities and architecture, which we’ll talk about.

And, since BigQuery uses a columnar data store, you can enjoy the highest data compression with minimized data scanning in the usual data warehouse deployments.

What is the BigQuery Architecture like?

BigQuery’s serverless architecture decouples storage and compute and allows them to scale independently on demand. This structure offers both flexibility and cost controls for users because they don’t need to keep their expensive compute resources up and running all the time.

This is very different from traditional node-based cloud data warehouse solutions or on-premise MPP systems.

This approach also allows users of any size to bring their data into the data warehouse and start analyzing their data using Standard SQL, without worrying about database operations and system engineering.

Under the hood, BigQuery employs a vast set of multi-tenant services driven by low-level Google infrastructure technologies like Dremel, Colossus, Jupiter and Borg.

Compute is Dremel, a large multi-tenant cluster that executes SQL queries.
Storage is Colossus, Google’s global storage system.
Compute and storage talk to each other through the petabit Jupiter network.
BigQuery is orchestrated via Borg, Google’s precursor to Kubernetes.

Ingesting Data

Now, at some point, you’ll probably need to ingest data into or from BigQuery because every organization today relies on multiple data sources, databases, data warehouses, so it’s likely that BigQuery alone cannot contain all of your data needs and pipelines.

The good news is: BigQuery supports several ways to ingest data into its managed storage. The specific ingestion method depends on the origin of the data. If your data sources are in GCP, then some of them support direct exports to BigQuery.

However, if your data sources are outside of GCP. Or, you don’t want to manually handle exports, there are several third-party ETL solutions in the market that can help you ingest data to and from BigQuery fairly easily. That’s what I’ll be going over in the next video. Stay tuned!