Validate your Looker workflow from data to dashboard

Key Concepts Looker Continuous Integration (Looker CI), Breaking Changes, Upstream, Downstream, LookML Development, Pull Request, SQL Validator, Assert Validator, LookML Validator, Content Validator, Primary Keys, Incremental Validation, False Positive.

---

Introduction to Looker Continuous Integration (Looker CI) Looker Continuous Integration (CI) is introduced as a crucial tool designed to protect LookML from "breaking changes" originating "upstream." Its primary goals are to enable developers to write more reliable code and to proactively alert them to updates that could disrupt "downstream" dashboards. By stopping the "domino effect" of small breaking changes, Looker CI fosters confidence in the development process. LookML development is described as the craft of transforming raw data into usable, governed models, with each stage (data connection, table joins, insight creation via Looks and Dashboards) presenting opportunities for issues. Looker CI aims to shift the paradigm from merely trusting the development process to actively controlling it, especially when making a pull request.

---

The Four Validators in Looker CI Suite The Looker CI suite comprises four distinct validators, each designed to protect against specific types of issues:

1. SQL Validator
2. Assert Validator
3. LookML Validator
4. Content Validator

These validators collectively check syntax, alert to disruptions, and safeguard content downstream, protecting against upstream changes.

---

1. SQL Validator The SQL Validator's core function is to ensure that dimensions defined in LookML accurately map to their corresponding database tables. It operates by running all dimensions within the LookML to identify:

• Naming issues: Discrepancies between LookML dimension names and actual database column names.
• Typos: Spelling errors in dimension definitions.
• Missing or broken joins: Issues that would prevent a query from successfully executing against the database.

Example: A real-world example demonstrated an error where a dimension's SQL specified user email while the database column was actually named email. The SQL Validator immediately flagged this mismatch, allowing the developer to correct the name before deployment, preventing a query failure.

Configuration: The SQL Validator offers flexible configuration options:

• It can be configured to exclude or include specific explorers and models.
• Individual dimensions can also be excluded. For instance, dimensions based on parameters will fail during SQL validation because parameters do not run during this process. To prevent "false positive" error messages for such dimensions, developers can add CI: ignore to their tags.

---

2. Assert Validator The Assert Validator addresses a more subtle category of errors: those "invisible to a query" where a column exists in the database, but the query returns incorrect or invalid data due to an "upstream model or data issue." This validator is particularly powerful when paired with well-written data tests.

Two Primary Use Cases:

1. General Data Tests: Used to verify fundamental data integrity, such as ensuring that primary keys are unique.
2. Custom Data Tests: Designed for more specific checks, like confirming that historical data returns an expected number (e.g., preventing "last year's revenue returning zero," which would indicate an upstream error).

Configuration: Similar to the SQL Validator, the Assert Validator can be configured to exclude or include specific explorers and models. Its effective use proactively safeguards data pipelines against hidden issues and protects content integrity.

---

3. LookML Validator The LookML Validator focuses on the structural correctness of LookML code. It checks for syntax errors, regardless of whether the LookML is written directly in the Looker UI or in an external environment. The configurable Looker CI version of this validator allows developers to set severity levels (info, warning, and error) for issues. This enables developers to identify and fix syntax problems before submitting a pull request, streamlining the development workflow.

---

4. Content Validator The Content Validator is specifically designed to ensure that changes made in LookML do not inadvertently break existing dashboards and Looks. It has been integrated into the Looker CI suite to provide this crucial "downstream" protection.

Configuration: The Content Validator can be configured to exclude or include specific models, explorers, or folders, allowing for targeted validation.

---

Incremental Validation A significant feature, "incremental validation," is available for both the SQL and Content validators. This functionality allows these validators to be configured to check only the development branch a developer is currently working on. This means developers can focus on fixing their own errors without being alerted to or dealing with errors present in the main branch. This targeted feedback loop is exemplified by a developer's internal monologue: "Oh, look, my change is going to break a dashboard. Thank goodness, Lici caught it. I'd better go address it." This highlights the immediate and actionable feedback provided by Looker CI.

---

Synthesis and Conclusion Integrating Looker Continuous Integration tools significantly boosts developer productivity by catching and correcting errors before they escalate into major problems. By leveraging these validators, organizations can build a more reliable and resilient codebase from the ground up. Developers are encouraged to consult the documentation for further configuration options and to create automated suites that run automatically upon pull requests, testing the workflow to maximize the benefits of Looker CI.