The symbiotic relationship between data governance and AI

Generative AI has already began shaking the world of Data Governance, and it is about to maintain doing so.

It’s just been 6 months since ChatGPT’s release, but it surely looks like we’d like a retrospective already. On this piece, I’ll explore how generative AI is impacting data governance, and where it’s more likely to take us within the near future. Let me emphasize near because things evolve quickly, and so they can go a number of other ways. This text isn’t about forecasting the following 100 years of information governance, but somewhat a practical have a look at the changes happening now and people just on the horizon.

Before diving in, let’s remind ourselves of what data governance deals with.

Keeping things easy, data governance is the algorithm or processes that a company follows to make sure the info is trustworthy. It involves 5 key areas:

• Metadata and Documentation
• Search and Discovery
• Policies and Standards
• Data Privacy and Security
• Data Quality

On this piece, we’ll have a look at how each of those areas is about to evolve once we incorporate generative AI in the combo.

Let’s do that!

Metadata and documentation might be a very powerful part of information governance, and the opposite parts construct heavily of this one being done properly. AI has already began, and can proceed to vary the way in which we create data context. But I dont wish to get your hopes too high. We still need humans within the loop in the case of documentation.

Producing context around data, or documenting the info has two parts. The primary element, which makes up about 70% of the job, involves documenting general information, common for a lot of firms. A really basic example is the definition of “email” which is common to all firms. The second part is about writing down the precise know-how that’s unique to your organization.

Here’s the exciting part: AI can do a number of the heavy lifting for the primary 70%. It’s since the first element involves general knowledge, and generative AI is great at handling that.

Now, what about knowledge that’s peculiar to your organization? Every organization is exclusive, and this uniqueness gives rise to your individual specific company language. This language is your metrics, KPIs, and business definitions. And it isn’t something that might be imported from outside. It’s born from the individuals who know the business best = its employees.

In my conversations with data leaders, I often discuss tips on how to create a shared understanding of those business concepts. Many leaders share that to realize this alignment, they create domain teams in the identical room to speak, debate, and agree upon the definitions that best fit their business model.

Let’s take, for instance, the definition of a ‘customer.’ For a subscription-based business, a customer may very well be someone who’s currently subscribed to their service. But for a retail business, a customer is perhaps anyone who’s made a purchase order within the last 12 months. Each company defines ‘customer’ in a way that makes essentially the most sense for them, and this understanding often emerges from inside the organization.

Relating to such peculiar knowledge, AI, as smart because it is, can’t do that part just yet. It might probably’t sit in in your meetings, take part the discussion, or help latest concepts bloom. For Andreessen Horowitz, this might turn out to be possible when the second wave of AI hits. For now, we’re still at wave 1.

I’d also wish to touch on an issue posed by Benn Stancil. Benn asks: If a bot can write data documentation on demand for us, what’s the purpose of writing it down in any respect?

There’s some truth to this: if generative AI can generate content on demand, why not only generate it while you need it, as an alternative of bothering with documenting all the pieces? Unfortunately, it doesn’t work like this, for 2 reasons.

First, as I’ve previously explained, an element of documentation covers the unique facets of an organization that AI cannot capture yet. This calls for human expertise. It can’t be generated on the fly by AI.

Second, while AI is advanced, it’s not infallible. The information it generates isn’t at all times accurate. You’ll want to be sure that a human checks and confirms all AI-produced content.

Generative AI will not be just changing the way in which we create documentation but additionally how we devour it. Actually, we’re witnessing a paradigm shift in search and discovery methods. The standard methods, where analysts search through your data catalog looking for out relevant information, are quickly becoming outdated.

A real game changer lies in AI’s ability to turn out to be a personal data assistant to everyone in the corporate. In some data catalogs, you’ll be able to already approach the AI along with your specific data inquiries. You may ask questions resembling, “Is it possible to perform motion X with the info?”, “Why am I unable to make use of the info to realize Y?”, or “Will we possess data that illustrates Z?”. In case your data is enriched with the fitting context, AI will help disseminate this context across the entire company.

One other development we’re expecting is that AI will transform the info catalog from a passive entity to an lively helper. Give it some thought this fashion: if you happen to’re using a formula incorrectly, the AI assistant could offer you a heads-up. Likewise, if you happen to’re about to jot down a question that already exists, the AI could let you recognize and guide you to the prevailing piece of labor.

Prior to now, data catalogs just sat there, waiting so that you can sift through them for answers. But with AI, catalogs could start actively helping you, offering insights and solutions before you even realize you wish them. This may be complete shift in how we engage with data, and it is perhaps happening very soon.

Yet, there’s a condition for the AI assistant to work effectively: your data catalog have to be maintained. To make sure that the AI assistant provides reliable guidance to stakeholders, the underlying documentation have to be 100% trustworthy. If the catalog will not be properly maintained, or if the policies are usually not clearly defined, then the AI assistant will spread misinformation throughout the corporate. This may be more detrimental than having no information in any respect, because it may lead to poor decision-making based on the mistaken context.

You’ve probably understood it: AI and data governance are interdependent. AI can enhance data governance, but in turn, robust data governance is required to fuel the capabilities of AI. This leads to a virtuous cycle where each component boosts the opposite. But you must be mindful that no element can replace the opposite.

One other key component of information governance is the formulation and implementation of governance rules.

This often involves defining data ownership and domains inside the organization. Immediately, AI isn’t as much as the duty in the case of defining these policies and standards. AI shines in the case of executing rules or flagging infractions, but it’s lacking when tasked with creating the principles themselves.

That is for an easy reason. Defining ownership and domains pertains to human politics. For instance, ownership means deciding who inside the organization has the authority over specific datasets. This might include the facility to make decisions about how and when the info is used, who has access to it, and the way it’s maintained and secured. Making these decisions often involves negotiating between individuals, teams, or departments, each with their very own interests and perspectives. And human politic, for obvious reasons, cannot get replaced by AI.

We thus expect that humans will proceed to play a big role on this aspect of governance within the near future. Generative AI can play a task in drafting an ownership framework or suggesting data domains. Nevertheless, keeping humans within the loop still stays a must.

Nevertheless, generative AI is about to shake things up within the privacy department of governance. Managing privacy rights is a historically feared aspect of governance. No person enjoys it. It involves manually creating a fancy architecture of permissions to be sure that sensitive data is protected.

The excellent news is: AI can automate much of this process. Given parameters resembling the variety of users and their respective roles, AI can create rules for access rights. The architectural aspect of access rights, being fundamentally code-based, aligns well with AI’s capabilities. The AI system can process these parameters, generate relevant code, and apply it to administer data access efficiently.

One other area where AI could make a big effect is within the management of Personally Identifiable Information (PII). Today, PII tagging is normally done manually, making it a burden for the person in control of it. That is something AI can automate completely. By leveraging AI’s pattern recognition capabilities, PII tagging might be conducted more accurately than when it’s done by a human. On this sense, using AI could actually improve the way in which we we manage privacy protection.

This doesn’t imply that AI will completely replace human involvement. Despite AI’s capabilities, we still need human oversight to administer unexpected situations and make judgment calls when needed.

Let’s not ignore data quality, which is a very important pillar of governance. Data quality ensures that the data utilized by an organization is accurate, consistent, and reliable. Maintaining data quality has at all times been a fancy endeavor, but things are already changing with generative AI.

As I discussed above, AI is great at applying rules and flagging infractions. This makes it easy for algorithms to discover anomalies in the info. You could find an in depth account on how AI affects different facets of information quality in this text.

AI also can lower the technical barrier of information quality. That is something SODA is already setting up. Their latest tool, SodaGPT, offers a no-code approach to precise data quality checks, enabling users to perform quality checks using natural language alone. This permits data quality maintenance to turn out to be rather more intuitive and accessible.

We’ve seen that AI can supercharge Data Governance in a way that’s triggering the start of a paradigm shift. Quite a lot of changes are already happening, and so they are here to remain.

Nevertheless, AI can only construct on a foundation that’s already solid. For AI to vary the search and discovery experience in your organization, you have to already be maintaining your documentation. AI is powerful, but it will probably’t miraculously mend a system that’s flawed.

The second point to be mindful is that even when AI might be used to generate many of the context around data, it cannot replace the human element entirely. we still need humans within the loop for validation and for documenting the knowledge unique to every company. So our one sentence prediction for the longer term of governance: turbocharged by AI, anchored in human discernment and cognition.

At CastorDoc, we’re constructing a knowledge documentation tool for the Notion, Figma, Slack generation.

Want to examine it out? Reach out to us and we are going to show you a demo.‍