- Introduction
-
About this Course
- Add a Pipeline Schedule
-
Creating a Schedule
-
Defining what your Schedule will Build
-
Defining when your Schedule will Build
-
Defining how your Schedule will Build
-
Exercise Summary
- Conclusion
-
Key Takeaways
-
Next Steps
DATAENG 04: Scheduling Data Pipelines
Navigate the scheduler application and learn best practices for scheduling Foundry data pipelines.
This tutorial is aimed at conveying the basics and best practices for creating automated data pipeline schedules. Foundry’s Scheduler application abstracts pipeline configuration into an intuitive interface that enables you to set the execution conditions to maintain your data freshness SLAs and minimize the risk of wasting Spark computation resources.
After scheduling your pipeline, you’ll have the opportunity to add documentation about the features of the Datasource Project stage. Descriptive documentation about pipeline logic, SLAs, maintenance procedures, and troubleshooting history is a valuable prophylactic against entropy and sets your project up for long-term maintainability.
Course prerequisites
- DATAENG 03: If you have not completed the previous course in this track, please do so now.
Learning Objectives
- Navigate the Scheduler application.
- Learn best practices for scheduling Foundry data pipelines.
Foundry Skills
- Create a pipeline schedule.