Data Technology Trend #0: Foundational (Part 2)
This article is a section of a multi-part series Data Technology Trends (parent article). If you have not already read, please ready part 1 of this article Data Technology Trend #0.
Trend #0.3: AI As a (Cloud) Service
Most of the modern cloud providers provide AI as a Service — AWS, Microsoft, Azure, Google, IBM Watson, Data Robot, etc., of the few. Machine Learning As a Service (MLaaS) is an umbrella of various cloud-based platforms that provides a complete infrastructure and the ability to write python / R programs using the supplied notebooks.
AWS SageMaker Built in Algorithms: https://docs.aws.amazon.com/sagemaker/latest/dg/algos.html
Latest in AI As a Cloud Service: AWS Redshift ML, Databricks MLFlow.
Gaining Traction and Maturity:
Trend #0.4: Data Governance, Privacy, Security
General Data Governance:
While DAMA / DMBOK / DCAM has been there for a while, with the rise of data and importance has been shifted to data all again, the importance of Data Governance is a necessary one. Being from Finance, I understand the importance of data security and governance. Finance is one of the leaders in the space of Data governance, security, and privacy checks and has been adopting for more than 2–3 decades. Internet companies are new entrants to this space compared to Finance and Health Care. Data governance and security is not an afterthought but is embedded in the lifecycle of the data.
Depends on the maturity of the firm and how the Data Governance and security needs to be engineered, it is imperative to adopt or custom build Governance models. Below is a comparison between DMBOK and DCAM by Data Crossroads. The DCAM model has been predominantly developed for Financial Institutions. While both the models are equally mature, it is up to the organization’s culture as to how and where they position Data to pick a perfect model fit. For most of types of enterprises, data is a business asset (eg., finance) that makes both business and technology co-create a data governance strategy encompassing security, privacy, and protection.
A clear Data Strategy should always part of the Technology Strategy and Technology Risk management guideline of any firm. It is important for your firm to identify the maturity model it is currently at and where does it want to move and in by when?
A combined view of DMBOK or DAMA & DCAM:
Cloud Data Governance:
Isn’t it sufficient that we already have DMBOK, DCAM, etc., for our data governance? What is the need for one more section dedicated to the cloud? Cloud Data Governance is very important and will have to be dealt with in addition to on-premises security. While security and ringfencing the infrastructure is the responsibility of the Infrastructure and Data security is again a shared responsibility of the Application team and Infrastructure team, in the cloud, it is everyone’s responsibility and hence a clear Cloud Data Governance is imperative and using the appropriate technologies to govern your data is part of your architecture decision.
More on this will be discussed under “Data Ops” section.
Summary: As much as AI is wide-spread today and AI or ML As a Service is practically the new norm and with cloud providers like AWS Redshift ML, Databricks MLFlow etc. the only thing organizations must be ready with is “Clean Data” — Garbage in, Garbage out. Have a good data for practically limitless Business applications and improve business performance.
For other articles refer to luxananda.medium.com.
