Lightweight DAG framework for local development

May 04, 2021 at 09:41

https://github.com/DavidTorpey/pydags

I worked on this a while back, and decide to release it in case it's useful. It's basically a lightweight Python-native DAG framework focused on simple use cases, with a Kubeflow-like API (but no reliance on Docker or k8s). You can think of it as a generic, more extensible version of scikit-learn's pipelines feature.

It supports Redis (and other key-value stores) as a way of passing data between nodes. It also supports DAG visualisation.

I may add more features if there's interest.