Lightweight DAG framework for local development
May 04, 2021 at 09:41
I worked on this a while back, and decide to release it in case it's useful. It's basically a lightweight Python-native DAG framework focused on simple use cases, with a Kubeflow-like API (but no reliance on Docker or k8s). You can think of it as a generic, more extensible version of scikit-learn's pipelines feature.
It supports Redis (and other key-value stores) as a way of passing data between nodes. It also supports DAG visualisation.
I may add more features if there's interest.