Microsoft Fabric: A Unified Data Platform with Power BI

With the proliferation of data tools along every step of the data process, from ingestion to transformation, storage to analysis, it can get overwhelming to keep up and keep everything wired together. A larger platform that consolidates different data tools can therefore help minimize the number of outside connections data engineers need to set up and maintain. Capitalizing on this idea, Microsoft unveiled the Microsoft Office Suite of data tools: Microsoft Fabric.

So, what is Fabric? When is it useful? And if it’s the “Office Suite of data tools,” what are all of the components woven into this platform? In this guide, we’ll explore all of these questions and more in detail so you can evaluate if Microsoft Fabric would benefit your use case.

What is Microsoft Fabric?

Microsoft Fabric is a unified data platform: essentially, Fabric groups together a collection of data tools, including Microsoft’s data visualization and reporting tool, Power BI. Other tools in the toolbox include a data lake and warehouses, ETL and ingestion tools, and real-time intelligence and other analytics.

Microsoft officially announced Fabric and released it for preview on May 23, 2023. The platform is therefore still a relatively new contender in the space, and much more recent than its signature component, Power BI, which was initially released a decade earlier in 2013. It’s so new, in fact, that a number of Fabric services are still marked as “preview” or “beta” features. If your use case would count on any of these preview features, you may want to carefully consider whether Fabric would meet your needs.

Microsoft Fabric’s name recalls the phrase “data fabric,” a term coined by Gartner that signifies a data management approach focusing on an integrated data view. Because the data platform is a Microsoft product, it easily integrates with the Microsoft Azure cloud platform, though it’s still possible to use Fabric for data collection and analysis even if you mainly use a different cloud platform. In these cases, it can be especially helpful to use an external data pipeline platform, like Estuary, to connect your data sources with Fabric. We’ll take a closer look at this option later in the article.

Microsoft Fabric’s Core Features

Being an amalgamation of tools, Fabric’s “core features” can map to the different services that make it up. Fabric comes with a handful of included “workloads” with several more available through third-party services. We’ll go over the native options and built-in Fabric features here.

1. OneLake

As might be guessed from the name, OneLake is a singular data lake that collects all known data sources for your Fabric tenant. It collects data from across workspaces, whether from warehouses, mirrored databases, or other data sources, to provide a holistic view of available data. This unified data lake is meant to reduce unnecessary copying and shuffling of data from one platform to another: you should be able to use one copy of data for analysis and reports.

2. Data Factory

The Data Factory workload provides tools to ingest and transform data. Dataflow connectors prioritize Azure sources, Microsoft formats like Excel and Access, and individual file sources in formats like Parquet and JSON. A visual interface allows users to merge and apply transformations on data and manage their pipelines.

3. Data Engineering

The Data Engineering workload aggregates item types like lakehouses, notebooks, and Spark jobs. Notebooks allow for collaborative data preparation and analysis using popular data science languages like Python, R, and Scala, without needing to switch out of the Fabric environment. Submit batch and streaming jobs to a Spark cluster to transform data hosted in a lakehouse that collects structured and unstructured data.

4. Data Science

ML models and experiments are core components of the Data Science workload. Track and score machine learning models to help ensure accuracy. Review the parameters that perform best to help tune your model.

5. Real-Time Intelligence

Monitor streaming data sources in Microsoft Fabric’s Real-Time Hub. A visual dashboard allows you to track event streams or drill down into incidents for investigations and swift response times. You can also set alerts or trigger actions based on certain event conditions.

6. Industry Solutions

Certain business verticals are able to implement additional solutions tailored to their use cases. See the Use Cases section below for more on these features.

7. Power BI

Likely the most widely known component of the Fabric platform, Power BI has been assisting with data visualization and reporting for more than a decade. Many data analysts are already proficient in Power BI; if your team already uses it, it may make sense to expand its capabilities and data sources with Fabric.

Visualization options include all the standards, such as area, bar, scatterplot, and line charts, as well as visualizations for more specific use cases, like various maps and custom visuals created with R scripts. These building blocks can create compelling reports and dashboards, with interactive and alerting capabilities.

Use Cases for Microsoft Fabric

It may make a lot of sense to manage your data with Microsoft Fabric if you’re already using Microsoft Azure as your cloud platform. Microsoft also offers extra features to support certain industry use cases. Businesses that touch on these use cases may want to give Fabric particular consideration.

Sustainability/ESG

Environmental, social, and governance (ESG) goals can be tricky to track, but important, and can affect businesses across many industries. There’s a growing trend of shareholders specifically picking investment options based on ESG scores. Tracking these metrics can therefore help a company give back to the community, save on energy costs, and be more attractive to investors, all at the same time.

Microsoft Fabric simplifies managing an ESG program by bundling related sustainability solutions, like ESG data estate, emissions insights, environmental analytics, and social and governance metrics.

Healthcare

Healthcare is a big industry with a lot of room for improvement. Data security, improving patient experience, and facilitating medical insights are all top-of-mind concerns. Microsoft Fabric provides a number of healthcare data solutions, such as data transformations to support medical formats like OMOP and DICOM, with previews for care management and patient outreach tools.

Retail

Retail industries are always awash in data, whether tracking inventory, keeping up with purchasing trends, or providing recommendations to shoppers. Fabric is in the process of growing its offerings in the retail realm. Current data solutions include a “frequently bought together” model and a retail industry data model that helps standardize data.

Microsoft Fabric vs. Competitors

Because Fabric is a collection of tools, it is difficult to point to a singular alternative. The closest comparison would be a collection of specific services in the other cloud platforms, the main difference being the necessity of selecting individual services rather than receiving them bundled in one smaller-than-the-entire-cloud-platform dashboard. There are also some other enterprise-grade data management systems that are comparable.

Google Cloud Platform

GCP offers a number of data services that can map to specific services within Fabric. Cloud Data Fusion is GCP’s data integration service with ETL capabilities, similar to Azure Data Factory. Looker provides business intelligence, like Microsoft Power BI. And BigQuery provides a petabyte-scale query service, similar to Microsoft Fabric’s warehouse and lakehouse querying capabilities.

Amazon Web Services

AWS also includes a host of data tools to choose between. AWS Glue is Amazon’s version of a data integration and discovery service, while Amazon QuickSight offers unified business intelligence. Amazon, of course, also has their own version of petabyte-scale querying in Amazon Athena, and various database and data storage options.

Informatica

Informatica is a mature enterprise platform for data integration, warehousing, and management. It provides data cleansing and enrichment options with a drag-and-drop interface for data ingestion. However, if you’re hoping for the robust data visualization and reporting that comes with Power BI, you may be disappointed with Informatica as a standalone solution. You can, of course, integrate with further services to fill these gaps, such as integrating Informatica with Tableau.

Snowflake

Alternatively, Snowflake provides more than just warehouse storage and querying. Notebooks, Streamlit dashboards, and collaborative functionality make Snowflake a more fully-featured data platform, comparable with Fabric. While Snowflake offers limited native data connectors, Snowflake’s marketplace and multiple third-party pipeline options fill this gap. However, Snowflake isn’t often used for fully real-time data and monitoring due to its pricing model.

Microsoft Fabric Challenges and Limitations

While Microsoft compiles a lot of functionality into Fabric, the data platform still faces some limitations and developers may find it challenging to work with for one reason or another.

Frustrating or Restrictive Usage

For those who are already set up with Azure, adding on a free trial of Fabric or outright licensing it shouldn’t require jumping through too many hoops. However, if you’re a data engineer who just wants to test out the platform or if your company has restrictions around its Azure setup and invited members, you may run into a few headaches: Currently, it is difficult to sign up for Fabric using a personal email address. Using a work email address, you must already be a member of your company’s Azure organization.

Once you do manage to try out Fabric, you may find it frustrating to navigate the sometimes-overloaded terminology Microsoft chooses to use. Individual item types, like Notebooks and Data pipelines, are collected in “Workloads,” which can have generic names like “Data Engineering.” This leads to documentation titles like “What is Data engineering in Microsoft Fabric?” that specifically relate to the Data Engineering Workload concept in Microsoft.

Limited Native Integration Options

As stated, Dataflow ingestion focuses on Microsoft-adjacent connectors. That’s not to say that’s all Fabric offers; there are certainly a plethora of other connector options, such as Snowflake, Databricks, and MySQL and PostgreSQL databases. There are even a few connectors for competing cloud platform data sources, like Amazon Redshift and Google BigQuery. However, for a platform featuring Power BI, there are surprisingly few connectors to support business intelligence. Marketing, ecommerce, and ad platform connectors are all but absent, as are connectors that facilitate internal insights and streamlined workflows, like Slack, Confluence, or monitoring tools like Datadog.

At least this limitation is more easily overcome than others: you can use an external data pipeline system to supplement Fabric’s functionality and draw closer to the vision of a unified data platform. For example, Estuary provides high-quality, low-latency capture connectors in all of the aforementioned categories, as well as additional cloud platform options. You can route all of these data sources to Microsoft Fabric using Estuary’s Azure Fabric Warehouse destination connector.

Even if Fabric includes a connector to a certain data source, you may want to consider using a separate data pipeline instead: at the time of this writing, a full quarter of Fabric’s listed connectors are still in beta. This brings us to our last challenge: the sheer number of “preview” or “beta” features.

Proliferation of Preview and Beta Features

As mentioned at the beginning of this article, Microsoft Fabric is a relatively new offering. This perhaps shows most clearly in the number of “(preview)” annotations and “beta” tags when browsing through the dashboard or documentation. Besides the large chunk of available connectors remaining in beta, some fairly core features are really previews, like mirrored Azure databases, SQL databases in Fabric, Python notebooks, and some industry-specific features.

While this isn’t automatically a deal-breaker, developers can justifiably be leery of building production products on the foundation of beta tools. It’s not uncommon for beta features to be discontinued, or to simply remain in a beta, sometimes broken, state for years. While Microsoft has more stability than most companies, it may still be a good idea to map out any features your use case requires to check whether those features are fully released on Fabric or not. After all, it’s best to avoid any “emperor’s new clothes” situations.

Conclusion: A Promising Unified Data Platform

Now you’re all caught up. We’ve gone over what Microsoft Fabric is and the services it gathers together in one collection. We’ve considered use cases where Fabric may be more beneficial and situations where it falls short.

If Microsoft Fabric sounds like a good fit for your data architecture except for the limited source connectors it natively provides, consider Estuary to fill the gap. If Estuary’s hundreds of connector options don’t cover your use case, we’re always open to requests for new connectors.

Contact us at Estuary, join our Slack community, and stay tuned for more content on Microsoft Fabric and other data platforms!