3 tips for stopping flaky tests

Key Concepts

• Flaky Tests: Tests that produce inconsistent results (passing or failing) despite no changes to the code.
• Deterministic Testing: The principle that a test should produce the same result every time it is run under the same conditions.
• Test Fakes: Objects or services that replace real dependencies (like databases or APIs) to provide predictable, fast responses.
• State Isolation: Ensuring that test environments do not share data or resources to prevent cross-test interference.

---

Strategies to Eliminate Flaky Tests

1. Isolate Test State

The primary cause of flakiness is often state collision, where multiple test runs attempt to access or modify the same resources simultaneously.

• The Problem: Hard-coded file paths or shared databases lead to race conditions where one test modifies data that another test relies on.
• The Solution: Implement unique temporary environments for every individual test run. By ensuring each test operates in a sandbox, you eliminate the risk of side effects from concurrent executions.

2. Fake External Dependencies

Relying on real-world infrastructure during unit testing introduces non-deterministic variables such as network latency or disk I/O errors.

• The Problem: Network calls and disk operations are inherently unpredictable and slow, making tests brittle.
• The Solution: Replace real dependencies with test fakes. These are lightweight, controlled implementations that simulate the behavior of the real system. This transition makes unit tests "lightning fast" and ensures they remain strictly deterministic by removing external variables.

3. Validate Environmental Assumptions

Tests often fail because they assume the environment is in a specific state that may have been altered by previous, failed, or incomplete test runs.

• The Problem: Assuming a "clean slate" without verifying it leads to false negatives when residual data from previous runs interferes with current execution.
• The Solution: Explicitly clear out existing data before executing any test logic. Never rely on the environment being "perfectly clean" by default; enforce the cleanliness as part of the test setup process.

---

Core Philosophy and Best Practices

• Reliable Signals: The ultimate goal of a testing suite is to provide a reliable signal regarding the health of the codebase. Flaky tests degrade trust in the CI (Continuous Integration) pipeline.
• Stop Ignoring Red Builds: A "red build" (a failed test) should never be ignored or dismissed as a "fluke." Ignoring these signals allows technical debt to accumulate and masks genuine bugs.
• Actionable Insight: Treat every test failure as a legitimate issue that requires investigation. If a test is flaky, it is a bug in the test suite itself that must be addressed immediately to maintain high-quality standards.

---

Conclusion

Flaky tests are a significant drain on developer productivity and CI efficiency. By focusing on state isolation, faking external dependencies, and explicitly managing environmental cleanliness, developers can transform inconsistent test suites into reliable, deterministic systems. The core takeaway is that a test suite is only as valuable as the trust developers place in its results; therefore, maintaining test stability is as critical as writing the production code itself.