Upgrade from Prefect 2.0 to three.0 and use the brand new Push Work Pools that greatly simplify infrastructure management and reduce cost.
Prefect is an open-source orchestration tool for data engineering. The Python-based tool means that you can define, schedule, and monitor your data pipelines. Prefect is an amazing tool for data engineers and scientists who need to automate their data pipelines.
In an article titled Getting Began with Prefect: Powerful Orchestration for Your Data, I wrote about deploying Prefect to AWS EC2 to run what they called an Agent, which essentially ran a job after it was kicked off.
With the upgrade to Version 3.0, Prefect is retiring the longstanding Agent model in favor of a brand new architecture called Employees. The upgrade from v2 to v3 immediately broke my deployments, leaving me needing to determine an alternate.
Employees are tighter integrated with the infrastructure they run on. It is a fabulous improvement because, up to now, you needed to determine tips on how to host the Agent in your computer. It was like installing software on a classic hosted server and maintaining it to OS updates.
