- Introduction
-
About this course
- Cleaning your Data
-
Create your cleaning pipeline
-
Add your cleaning logic
-
Create a clean output
-
Making changes to your pipeline logic with branching
-
Exercise Summary
- Documenting your Pipeline
-
Documenting your Pipeline with a Data Lineage Graph
-
Documenting your Pipeline with Notepad
-
Exercise Summary
- Conclusion
-
Key Takeaways
-
Next Steps
DATAENG 03 (Builder): Creating a Project Output in Pipeline Builder
Engineer a clean output for your project to be consumed by downstream pipelines and use cases.
Foundry applications and project structures that support data pipelines provide ample opportunities for your to let your current and future team know the relevant facts about your data transformations. Having preprocessed your data, it’s time to clean it and prepare it for use downstream. This means airtight transform logic and documenting the scope every step of the way.
In this tutorial, you’ll create “clean” outputs for your project to be consumed by downstream pipelines and use cases. In doing so, you’ll get additional practice within a recommended pipeline project structure.
Course Prerequisites
- DATAENG 02 (Builder): If you have not completed the previous course in this track, please do so now.
Learning Objectives
- Distinguish between preprocessing and cleaning steps.
- Document the datasource stage of your pipeline.
- Gain additional practice transforming data in Pipeline Builder.
Foundry Skills
- Generate multiple outputs from a Pipeline Builder transform.
- Implement branching and pipeline documentation best practices.
- Generate a Data Lineage graph as documentation for the Datasource project segment of your production pipeline.