Prefect Tutorial and Workflow Development

[Jan-Willem]

Objective

Familiarize the team with Prefect by completing tutorials, developing a simple Prefect workflow based on the structure outlined in Figure 1 of the An example RADPS Workflow Decomposition, testing reliability and fault tolerance, and summarizing key learnings in a presentation.

---

Requirements

1. Explore and decide on suitable Prefect tutorials from the official Prefect documentation or community resources.
2. Complete the selected tutorials to understand key concepts, including:
   • Tasks and flows
   • State management
   • Scheduling
   • Adding conditional loops to workflows
3. Develop a Prefect workflow replicating the structure of Figure 1 in the An example RADPS Workflow Decomposition.
   • Tasks can be no-ops or simple sleep operations.
   • The workflow should be run locally to verify execution.
   • Include at least one conditional loop to demonstrate dynamic behavior.
4. Implement testing for:
   • Reliability: Ensure the workflow is robust by testing its ability to recover from failures (e.g., rerun on failure, retries).
   • Fault Tolerance: Introduce simulated failures (e.g., network or task errors) to verify how the workflow handles and recovers from errors.
5. Create slides summarizing:
   • Key learnings from the tutorials.
   • Implementation details of the workflow, including how reliability and fault tolerance were incorporated.
   • How conditional loops were implemented and their impact.

---

Definition of Done

Tutorial Completion

• Prefect tutorials are identified, agreed upon, and completed by all team members.
• Key learnings and notes from the tutorials are shared with the team.

Workflow Development

• A Prefect workflow mimicking the structure in Figure 1 is created.
• Tasks in the workflow use placeholders (e.g., no-op or sleep tasks) to demonstrate flow structure.
• At least one conditional loop is added to the workflow to illustrate dynamic behavior.
• The workflow is successfully executed in a local environment.
• The workflow includes basic reliability and fault tolerance features, such as retries and failure recovery.
• Simulated failures are tested, and the workflow handles them as expected (e.g., retrying or continuing on error).

Presentation

• Slides are created summarizing:
  • Core concepts learned during the tutorials.
  • The design and implementation of the workflow, including reliability and fault tolerance strategies.
  • Insights on using conditional loops in Prefect.

---

Key Decision Points

1. Tutorial Selection:
   • Choose relevant tutorials from Prefect's documentation or community guides.
2. Figure 1 Workflow Interpretation:
   • Agree on the structure and task definitions for the workflow based on Figure 1.
3. Conditional Loop Design:
   • Decide how and where conditional loops should be integrated into the workflow.
4. Testing Reliability and Fault Tolerance:
   • Determine how to test and incorporate failure recovery and retries in the workflow.

---

Artifacts

• Notes and resources from completed tutorials.
• A functional Prefect workflow script with reliability and fault tolerance features.
• Logs and outputs from local workflow execution, including tests for failure handling.
• Slide deck summarizing key learnings and implementation details.

---

Success Criteria

• The team has a foundational understanding of Prefect's core features, including reliability and fault tolerance.
• A working Prefect workflow, including conditional loops and failure handling, is successfully run locally.
• A comprehensive slide deck effectively communicates the learnings and workflow design.
• The team is equipped to design and implement more complex workflows in the future, with robust testing for reliability and fault tolerance.

---

Potential Challenges

• Identifying tutorials that are both comprehensive and time-efficient.
• Correctly interpreting and implementing Figure 1's workflow structure.
• Troubleshooting environment or execution issues during local workflow testing.
• Implementing conditional loops correctly and ensuring their functionality in the workflow.
• Simulating failures and ensuring that the workflow can recover as expected.