Data Technology Trend #8: Data Next — part 3

This article is a part of a multi-part series Data Technology Trends (parent article). Previous article — Link: Data Technology Trend #7: Monetized. Previous part of this trend article — Data Technology Trend #8: Data Next — part 2 and next part of this trend article — Data Technology Trend #8: Data Next — part 4.

Trend 8.1 Unified and Enriched Big Data and AI — Delta Lake

Few impressive features of Delta Lake: (Cont..)

3. Unity Catalog

Databricks Unity Catalog is a brand-new feature introduced in 2021 (this article is written when there is a waitlist to sign-up for the solution). The very reason Data Lakes becomes data swamps is the missing part of governance and the ever-growing data into the data lakes and creating multiple buckets and providing file-based access rights to the end-user. If this problem can be sorted, the data swamps can be avoided and data lakes can be retained taking care of data quality. In order to achieve this, instead of giving permission at the file level, provide permission at the query level! Users can be of 3 types in any organization (1) Simple user (2) Power user and (3) Super user. To any of these users, to access the data if restricted at the query / table level, then the data lake need not be disturbed and altered.

Data Lake starts all neat…

Unity Catalog, as per Data Bricks is the world’s first unified catalog for the lakehouse.

Key features

1. One interface to govern all data assets

2. One security model based on “ANSI SQL”

3. Full integration with existing catalogs

As Data Lake stores everything as files, there are a bunch of users who will need different access rights to different files. In order to separate it, we tend to create multiple buckets and provide permission at the file level. Data Bricks saw this to be the starting problem where Data Lake gets murkier and introduced permission at ANSI SQL level instead of at file level. If a user has permission to run a query on the table, then they will have access to that data set. This is not a new concept, we have been dealing…