Dataplex fundamentals: Aspects & glossaries

Key Concepts

• Data Catalog: A tool for organizing and managing data assets.
• Metadata: Data that describes other data.
• Aspect Type: A schema or template for metadata, similar to a class in programming.
• Aspect: An instance of an Aspect Type attached to a data asset.
• Business Glossary: A centralized repository for defining business terms and ensuring consistent understanding.
• Strong Typing: Enforcing data types and constraints for metadata, similar to programming.
• Contracts: Defined structures and rules for metadata that must be adhered to.
• Governance as Code: Managing data governance policies and configurations through code (e.g., Terraform, Python).

Data Universal Catalog in Google Cloud: Moving Beyond Passive Inventory

This video introduces the concept of a "data universal catalog" within Google Cloud, advocating for a shift from passive data inventory and simple tagging to active governance driven by strong typing and defined contracts. The presenter, Hanuk, a developer advocate at Google Cloud, highlights a common pain point in growing data platforms: the unreliability and chaos that arises from unstructured, free-text tagging of metadata.

The Problem with Free-Text Tagging

The core argument against traditional tagging is its lack of "engineering rigor." Unlike defining database tables with schemas, types, constraints, and nullability, free-text tags are prone to inconsistencies. Examples cited include:

• Tags like "owner is Bob" becoming obsolete when Bob leaves.
• Variations in tag values (e.g., "contact" instead of "owner," "PII" vs. "PII" in all caps).
• This leads to a "catalog becoming untrusted noise."

The Solution: Strong Typing and Contracts

The proposed solution is to treat metadata with the same seriousness as code, employing a system of "strong typing and contracts." This is explained through an analogy to programming:

• Aspect Type: Analogous to a "class" in programming, defining the schema and constraints for metadata.
• Aspect: An "instance" of an Aspect Type, attached to a specific data asset.

The fundamental principle is that a schema must be defined before metadata can be attached, enforcing consistency by design.

Practical Implementation and Examples

The video demonstrates this concept using a BigQuery table with transaction data.

1. Addressing Ambiguity with a Business Glossary

• Problem: Ambiguous column names like "GMV" (Gross Merchandise Value) lack clear definition, leading to potential misinterpretations (e.g., whether it includes tax).
• Solution: A Business Glossary is introduced as a "single source of truth for definitions decoupled from the technical implementation."
• Action: The presenter defines "Gross Merchandise Value" in the glossary to ensure a consistent understanding.

2. Establishing Engineering Contracts with Aspect Types

• Concept: Custom Aspect Types are created to define structured metadata schemas.
• Example: A custom Aspect Type named "data asset governance" is created.
• Schema Definition:
  • data stores: A required text field.
  • sensitivity: An enum (restricted list) with options: public, internal, or confidential. This prevents the free-text chaos of inconsistent sensitivity labels.

3. Bringing it Together: Attaching Aspects to Data Assets

• Column Level: The ambiguous "GMV" column is linked to the formal glossary term for "Gross Merchandise Value." This ensures any data consumer understands its precise meaning.
• Table Level: The "data asset governance" aspect is attached to the BigQuery table.
  • This is not a free-text tag; the UI enforces the defined contract.
  • The presenter must provide a value for data stores and select sensitivity from the predefined enum list.

The Benefits of Structured Governance

The effort invested in structured governance yields significant advantages, particularly for tasks like security audits:

• Precision: Instead of fuzzy keyword searches (e.g., searching for "internal" and getting hundreds of irrelevant results), structured aspects allow for precise queries.
• Example Query: "Show me every asset where sensitivity is explicitly set to internal." This is the foundation of structured data governance on Google Cloud.

Conclusion and Next Steps

The video concludes by emphasizing the move from "message text to a strongly typed system." The presenter encourages viewers to engage with a linked codelab to build practical experience with this object model. The next logical step, discussed briefly, is governance as code, where schemas are managed through tools like Terraform or Python scripts rather than solely through the UI. This represents a more advanced stage of implementing robust data governance.