1/8/2024 0 Comments Prefect vs airflow![]() Details matter – Prefect focuses on high-quality software, with impressive unit tests and documentation.Tasks are functions – We didn’t need to learn another way of doing things because any Python function can be a task.Prefect’s own “ Why Prefect” article provides a few more compelling reasons to use Prefect: If you want to build tasks in Airflow, you have to learn “the Airflow way.” To build tasks in Prefect, you can simply write Python. Specifically, Prefect lets you turn any Python function into a task using a simple Python decorator.īy contrast, platforms like Airflow use more verbose, tightly constrained tasks. It addresses many of the pain points common to more complicated tools like Airflow. We knew these Dask foundations would lead to a stable core and a strong community – neither of which we found with Kubeflow.įinally, we were attracted to Prefect because it’s familiar to Python engineers. Prefect is built on top of Dask, and they share some core contributors, so we were confident in Prefect from the start. So we needed a tool with native Kubernetes support. Unfortunately, Luigi doesn’t play well with Kubernetes – and although we’d patched this gap with some custom code in our older projects, we wanted Kubernetes to be a centerpiece in our new reference architecture. We’ve previously compared several other workflow orchestration tools, and we’ve used Luigi for many of our existing projects. Now we’re using Prefect.io instead, and we’re loving it. ![]() ![]() Our initial plan was to build this partly on top of Kubeflow, but that tool had certain shortcomings, so we decided to drop it. When it’s done, it’lll be a collection of our favorite machine learning tools, plus documentation and Terraform scripts to help you set up your full machine learning project quickly and easily. In short, we’re building a reference architecture for machine learning projects. You can read more about why we were less than impressed with Kubeflow in our previous post.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |