Using RL Agent to Detect and Remediate ETL Pipeline Failures - Anna Marie Benzon

Key Concepts

• ETL (Extract, Transform, Load): The process of moving and preparing data for analysis.
• RL (Reinforcement Learning): A machine learning paradigm used here for contextual action selection.
• MTTR (Mean Time To Recovery): A key performance metric measuring the time taken to resolve a system failure.
• Deterministic Anomaly Rules: Rule-based logic used to establish observable facts (e.g., schema drift, null rate spikes).
• Safety Override: A hard-coded guardrail layer that sits outside the RL policy to prevent unsafe or unauthorized actions.
• Tabular Q-Learning: A reinforcement learning algorithm used for small state/action spaces where every decision is inspectable.
• Ablation Study: A method of evaluating a system by removing components to determine their individual contribution to performance.

---

1. System Architecture and Workflow

The system is designed as a closed operational loop triggered by AWS Glue job failures. The architecture follows these steps:

1. Event Trigger: Amazon EventBridge captures a "Job Failed" event from AWS Glue.
2. Evidence Gathering: A Lambda function collects logs from CloudWatch and metadata from the Glue Data Catalog.
3. Classification & Scoring: The system uses deterministic rules to classify the failure (e.g., schema drift, type change) and assigns an operational risk score.
4. Decision Engine: The state (failure category, risk level, drift severity) is passed to a Q-learning policy.
5. Safety Check: A safety layer evaluates the proposed action against constraints. If the action is deemed unsafe or the case is too novel, it is overridden or escalated.
6. Execution & Audit: The executor triggers the Glue API to remediate, and all actions/outcomes are logged in S3 for auditability.

2. The Intelligence Layer: Three-Tier Design

The project emphasizes a separation of concerns to ensure reliability and explainability:

• Deterministic Rules: Used for "facts." If a field is missing or a type has changed, the system uses explicit, auditable rules rather than opaque ML inferences.
• Q-Learning Policy: Handles "contextual action selection." It chooses from six actions: Retry, Coerce, Rollback, Quarantine, Escalate, or Log.
• Safety Guardrails: Acts as the final authority. If the RL policy suggests a passive action (like "Log") for a critical failure, the guardrail forces an "Escalate" action.

3. Performance and Evaluation

The system was evaluated using synthetic data across 30 experimental seeds to ensure robustness.

• MTTR Reduction: The system reduced the mean resolution time from a manual baseline of 2.5 working days (approx. 216,000 seconds) to roughly 5.24 minutes—a 99.85% reduction.
• Success Rate: The automated success rate was 74.63% (±1.51%).
• Precision/Recall: The anomaly detector achieved a precision of 1.0 and a recall of 0.8, indicating a conservative but highly accurate detection mechanism.
• Ablation Findings: The study revealed that the RL policy’s success was largely driven by the structured state and safety constraints rather than the complexity of the learning model itself.

4. Key Arguments and Perspectives

• "ML Ready is not the same as ML Required": The author argues that the simplest reliable component should own each decision. Deterministic rules are preferred for observable data conditions.
• Escalation as a Success: The system treats "Escalation" as a first-class outcome. A robust agent must recognize the limits of its own evidence; failing to act when uncertain is a feature, not a bug.
• Human-in-the-loop: The goal is not to replace human judgment but to automate the "fast path" for routine, recognizable failures, allowing engineers to focus on novel, high-risk incidents.

5. Notable Quotes

• "The central question is not simply whether an agent can act, but whether it can act usefully, explainably, and with boundaries that an operations team would actually trust."
• "If success is measured only by non-escalation, the optimization target is wrong."
• "A practical self-healing system does not need the largest possible model. It needs a clear state, bounded action, reproducible evaluation, observable decisions, and a disciplined stop when uncertainty exceeds its authority."

6. Synthesis and Conclusion

The project demonstrates that self-healing ETL systems are feasible when built on a foundation of deterministic facts and bounded, inspectable learning. By separating the "what happened" (rules) from the "what to do" (RL) and the "is it safe" (guardrails), the system achieves significant reductions in MTTR while maintaining high operational trust. The next phase for this architecture is "shadow mode" deployment, where the agent provides recommendations for human review before being granted full execution authority.