Metaflow has been used in production at Netflix since early 2018. The core Metaflow was open-sourced in December 2019. Some features of Metaflow are not available in open-source yet but we may open-source them later if there is sufficient external interest. These features are listed below.
Please click the link and comment / thumbs-up the corresponding GitHub issue if you want to see the feature open-sourced.
Bring all of Metaflow's capabilities to the Kubernetes universe (Github issue)
A variety of UI(s) for Metaflow - tracking flows, model monitoring, etc
Support for dependency management tools beyond conda and docker and address existing pain points (Github issue)
Support composing Metaflow flow from other flows (Github issue)
A Slack bot for Metaflow. Use it to ask questions about past runs (Github issue)
Update - Metaflowbot is now available in Open Source!
Support in-memory processing of large data sets (Github issue)
Provide advanced tutorials and documentation highlighting non-trivial use-cases (Github issue)
An easy-to-use Function-as-a-Service -style microservice hosting platform for artifacts (e.g. models) produced by Metaflow runs (Github issue)
Metaflow in the R language. Provide an idiomatic R API which uses the Python library as the backend (Github issue)
Update - Metaflow-R is now available!
Netflix uses an internal DAG scheduler to orchestrate most modeling and ETL pipelines in production. Metaflow flows can be deployed to the production scheduler with a single command. A similar integration could be provided e.g. for AWS Step Functions (Github issue)
Update - Metaflow 2.1.0 introduced integration with AWS Step Functions.