Databricks

Data governance tools – a comparative analysis

Posted by

Bhairab Ghosh

According to this Gartner prediction, 75% of individuals are concerned about the privacy of their data. Similarly, 70% of enterprises express their concerns about data security as they move their workloads to cloud platforms. Gartner estimates that 80% of digital initiatives fail when organizations don’t adopt a modern approach to data governance.

With expanding data volumes, data governance tools are gaining more importance. Depending on their data requirements, organizations must choose the right data governance tool based on:

User-friendliness and usability
Flexibility and scalability
Data security and privacy
Integration with existing systems

Here’s how the Databricks Unity Catalog tool performs against each of the following data governance tools:

Snowflake Polaris
Microsoft Purview
Apache Atlas

Snowflake Polaris and Databricks Unity Catalog – a comparative analysis

In June this year, Snowflake announced the release of its open-source data governance tool, Polaris Catalog, which is implemented on Apache Iceberg. The Databricks Unity Catalog open-source tool provides unified data governance.

Thanks to its use of open APIs and Apache 2.0 licensed open-source server, Unity Catalog provides a universal interface for the 3 major open table formats (OTFs), namely:

Iceberg
Delta
Hudi

Additionally, Unity Catalog enables interoperability across cloud platforms, query engines, and cloud tools.

Comparatively, Snowflake’s Polaris Catalog is fully based on Iceberg’s REST open-source protocol, thus allowing Snowflake users to access and retrieve data using any engine that supports Iceberg REST APIs, including:

Apache Flink
Apache Spark
Trino

Open-source does not translate into automatic compatibility. While Polaris only supports Iceberg, Databricks does comparatively better at supporting other OTFs besides its native Delta support.

MS Purview and Databricks Unity Catalog – a comparative analysis

Previously known as Azure Purview, MS Purview is a data governance solution for Azure tools such as:

Synapse Analytics
SQL Server
MS 365
Power BI

With MS Purview, enterprises can perform multiple applications such as:

Searching and finding Azure data assets
Obtaining a holistic view of their data
Tracking column-level data lineage
Monitoring the health of data assets

Comparatively, Unity Catalog provides cloud-independent governance of data lakes with functionalities such as:

Data searching and discovering
Granular access controls
AI-powered asset monitoring and data observability
Open data sharing

Here’s how Databricks Unity Catalog compares with MS Purview across these factors:

Data discovery and searches
Both Unity Catalog and MS Purview offer user-friendly search interfaces for users to search and discover their data assets – but differ in their approaches. For instance, both tools support natural language searches.

However, MS Purview implements this by grouping related data assets into products, thus facilitating bulk search requests. It also uses the AI-powered Copilot to intelligently recommend data assets and products.

On the other hand, Unity Catalog enables users to find data assets using natural language – with a much simpler user interface that focuses on metadata like table name, column name, and descriptions.

Data lineage
Data lineage is critical for troubleshooting data quality issues and improving trust. MS Purview offers data lineage with the following options:
- Entity level – captures a high-level graph of how data flows from its source to destination.
- Column or attribute level – provides a more detailed and granular view of how data attributes have changed from their source to target entity.
On the other hand, Unity Catalog automatically captures the data lineage across data assets in the Databricks workspace. These data assets include:
- Database tables and columns
- Dashboards
- Workflows
- External sources
Additionally, data lineage in Unity Catalog is more granular than in MS Purview.

Data security
MS Purview provides data security and governance with its role-based access control – along with a single pane to manage access control to Azure data sources. Unity Catalog provides data security and governance with a unified interface that defines access control to AI and data assets.
Third-party integration
MS Purview provides seamless integration with Azure tools. While this tool does offer APIs to connect with non-Azure tools, it often results in vendor lock-ins. Unity Catalog integrates well with data assets in the Databricks environment – but needs additional integration solutions for third-party systems.

Apache Atlas and Databricks Unity Catalog – a comparative analysis

How does Unity Catalog compare with Apache Atlas? Both offer exceptional data governance – but differ in their implementation approach.

Here’s a comparative analysis of Unity Catalog and Apache Atlas:

Metadata management
Unity Catalog has effectively transformed metadata management with its unified lakehouse-based architecture – along with data lakes and warehouses. With this architecture, enterprises can implement centralized governance across all Databricks workspaces.

On the other hand, Apache Atlas does offer flexibility with its customized metadata models, however this has its share of setup complexities.
Integration capabilities
Unity Catalog utilizes Delta Sharing to secure data sharing across multiple workspaces and organizations. Apache Atlas needs customized connectors for external cloud services – but also provides easy customization.
Data lineage
Unity Catalog provides real-time column-level data lineage tracking, which enables faster detection of any data quality issues. Apache Atlas supports complex relationships – but requires manual configuration and also lacks automated tracking.
Data security and access control
Unity Catalog can implement SQL-based permissions along with fine-grained access control. Apache Atlas only supports basic-level role-based access control – and is dependent on additional tools.

Conclusion

Before choosing the right data governance tool, organizations must weigh in on their data management requirements. Depending on the tool’s features, they can ensure robust data governance for all their business assets.

Which of these data governance tools are best suited for enterprises? Here’s our suggestion:

Unity Catalog is the best choice for companies looking to implement data governance and access control in their Databricks environment (being used for advanced data analytics and machine learning applications).
MS Purview is best suited for companies with data assets on MS Azure – or any on-premise database and SaaS platforms.
Apache Atlas can be a good choice for companies looking to build customized data governance solutions – or looking to integrate metadata management in their current ecosystem.

As a Databricks partner, Onix can help you implement data migration from traditional warehouses to the Databricks platform. Through this partnership, Onix enables its customers to leverage the capabilities of its Databricks lakehouse platform.

We can help you simplify your Databricks migration. Contact us today.