- Introduction
-
About this Course
- Health Checks and Check Groups
-
Configure Data Health Check Groups
-
Exercise Summary
- Applying Health Checks
-
Add a Schema Check from the Data Lineage Application
-
Add a Time-Based Check from the Data Health Application
-
Build vs. Job Checks
-
Install Schema and TSLU Checks Throughout your Pipelines
-
Exercise Summary
- Schedule Metrics and Checks
-
Using Metrics to Determine Alerting Thresholds
-
Setting Schedule Health Checks: Schedule Status
-
Setting Schedule Health Checks: Schedule Duration
-
Applying Schedule Checks to All Your Pipelines
-
Exercise Summary
- Conclusion
-
Key Takeaways
-
Next Steps
DATAENG 06: Monitoring Data Pipeline Health
Implement best practices for monitoring production pipelines using Foundry's Data Health service.
This tutorial is about giving you hands-on experience implementing best practices for monitoring production pipelines using Foundry’s Data Health service. The goal by the end of this training is to equip you with everything you need to apply the right checks at the right parts of your pipeline for optimal health and performance.
⚠️ Course prerequisites
- DATAENG 05: Transform Projects in Pipeline Builder: If you have not completed this course, please do so now.
📖 Learning Objectives
- Know where and how to apply data health checks.
- Learn and apply recommended data health checks to key parts of your pipeline.
- Know where to find metrics that might help you tune your checks.
- Understand the notification and alerting framework.
💪 Foundry Skills
- Configure dataset health checks in the Data Health and Data Lineage applications.
- Configure schedule health checks in the Scheduler application.
- Use schedule metrics to update your checks as needed.
- Configure group checks for batched alerting.