Making a Modern Data Centric Organization — Part 4
Define a good Modern Data Technology Infrastructure
Last but not least,
To create a “MODERN DATA ORGANIZATION”
it is imperative
to choose the right data infrastructure.
1 The first trend is
Foundational
Data as a mainstream where the data related technology/processes has reached the plateau of productivity as per Gartner’s terms. Once thought dead is back into business due to cloud and modern data platforms. As part of this trend, Data Warehouse is back with modern cloud databases, Big Data is revived with Spark, increasing use of Artificial Intelligence with multiple PAAS and SAAS providers.
Gaining maturity: In Finance and health care, Data protection, security and privacy is not a new concept. While for the internet organizations the data protection and security is evolving.
2 The second trend is
Trusted
With Internet organization’s focusing on Data protection/security/protection, I would like to highlight “Trusted” as the initial trend — Ringfencing Data from generation till archival and. beyond.
In my opinion Trusted trend comprises of
1. Differential privacy — To cut the story short, differential privacy is about adding “noise” to the data, yet make it meaningful.
2. Authenticated / Data Provenance (aka lineage)
One of the upcoming trends is the “Authenticated Provenance” and how do you know the data is real and valid when it is created. Authenticated provenance is part of “Algorithmic Trust”, Blockchain helps to track the origin. This is especially useful for niche and costly and unique products in place or extremely sensitive/critical information that flows through. Garbage-in is Garbage-Out — Data Provenance is the key to understand the source and authenticity of data.
3 The third trend is
Strategic
The core of the Strategic trend is “Simplification of Data”. As with the modern data platforms, more and more organizations want to store and retrieve information from a single area and be able to have a simple yet efficient data architecture and platforms.
Data warehouse a big come back on Data Lake: eg., Redshift & Data Lake (Glue and S3) in AWS. Cloud is a given for Data & Analytics
Data Fabric: Worlds collide1. Cloud / On-prem 2. Relational / Non-Relational (structured / semi / unstructured)3. Data & Analytics
other features that are part of Strategic trend in my perspective includes Data Hub, DBAAS: Database as a service, Delta Lake, Data Mesh, Single Database for cross needs, Data Catalog, Self-Service data preparation, Data Management solutions, Data Integration tools.
4 The forth trend is
Accelerated
The core of this trend is “Optimisation of data”.
Long Live Streams (Realtime) and Dead ETL(?): Kafka and Kinesis have made streams on steroids. The ETL which would have been dead with the streams has given a come back with technologies such as Glue. ELT is more predominant than ETL. More managed services are in the spin for the Data ETL and Streams. key advantages of using the modern real-time streams / ETL is the ability to broker for multiple purposes, removing latency, improve performance, store, and access by multiple consumers.
Other technology includes Augmented Data Management, better Master Data Management, a better, cheaper, smarter, and faster conversion of Data into Business Insights.
5 The fifth trend is
Decentralized
There is always a constant debate about whether the Data should be centralized or decentralized. With the advent of modern data platforms, data as a service and data as a product, why this should be either decentralization or centralization. With organizations generating more and more data, there should be a proven ability to switch between centralization and decentralization — this should not be an after-thought and should be an inception point and must be created as part of the data architecture.
Modern technologies such as Data Mesh, Data Fabric, Data Virtualization and Blockchain Technologies enable this big time.
6 The sixth trend is
Democratized
The current and the next generation of data moves towards Data Democratization at a fast pace. There is no technology dependency or demarcation between which sources you retrieve the data and how you are putting up analytics on top of it. Most useful technology for the democratization of data including but not limited to DataOps or DataSecOps, MLOPS or AIOps, Data sharing using Delta Tables and Auto ML.
7 The seventh trend is
Monetized
If Data is the new oil, then we should be able to sell oil or create derivatives out of it and monetize Data. Best ways to go about, Data & AI Market Places and exchanges platforms, Data As A service, and Data as a product.
If you are intrested in this sub-topic, i have written a full blown article on “Modern Data Trends”.
Next: Extras, Benefits and Conclusion.
