What is CodeReef?¶
CodeReef is an open platform to share and interconnect all the AI/ML/systems components (models, data sets, packages, scripts) from research projects in the common CK format required to enable portable, customizable and reproducible AI/ML workflows.
CodeReef client / CodeReef SDK is a small open-source Python library (Apache2 license) with a simple command line interface to automatically initialize, build, crowd-benchmark, test and customize CodeReef solutions with ML models across diverse platforms, AI frameworks, libraries, compilers and data sets.
CodeReef demo is our interactive live demo of the CodeReef solution and the CodeReef client to automatically build, run and crowd-benchmark object detection based on SSD-Mobilenets, TensorFlow and COCO dataset from the latest MLPerf inference benchmark across a wide range of platforms from Raspberry Pi, Android phones and IoT devices to HPC servers with powerful GPUs.
See our white-papers for more details: TBA
We created CodeReef to enable practical MLSysOps, i.e. cross-platform MLOps with the automated deployment of ML models in production across diverse systems from IoT to data centers in the most efficient way.
We also work with the community to keep track of the state-of-the-art AI, ML and systems research using automated and portable workflows with a common API (compile, run, benchmark, validate) to be able to reproduce and compare results (execution time, accuracy, latency, energy usage, memory consumption and so on).
It is motivated by our practical and painful experience validating the new technology from research papers at ML and systems conferences and moving them to production. We have noticed that while finding the code for ML models from ArXiv, PapersWithCode and MLPerf is not a problem nowadays, it is only a small tip of the iceberg. The major challenge afterwards is to figure out how to prepare the required software for such models across numerous and rapidly evolving heterogeneous systems with continuously changing interfaces and data formats, integrate them with complex applications, customize them and run them in the most efficient way.
Existing useful tools such as MLFlow, Kedro and Amazon SageMaker help to automate training and other high-level ML operations while Docker, Kubernetes and other container based tools can hide software chaos but there is a lack of intermediate tools to bridge the growing gap between ML and systems, particularly when targeting resource constrained embedded devices and IoT!
All these problems motivated us to develop CodeReef portal - a collaborative platform inspired by GitHub and PyPI to share portable ML components and workflows along with reproduced ML results.
It is based on the open-source Collective Knowledge technology (CK), with open standards to let users exchange all the basic blocks (components) needed to assemble portable ML workflows, collaboratively benchmark them and compare their performance using customizable dashboards.
Why open Collective Knowledge format?¶
We decided to use the open and non-virtualized CK format because it is a proven open-source technology used in many academic and industrial projects since 2015. For example, the authors of 18 research papers from different systems and ML conferences used CK to share their research artifacts with automated workflows.
CK concept is to gradually convert ad-hoc research projects into file-based databases of reusable components including code, data, models, pre-/post-processing scripts, experimental results, best research practices to reproduce results, and auto-generated papers with unified Python APIs, CLI-based automation actions and JSON meta information.
CK also features plugins to automatically detect different software, models and data sets on a user machine or install the missing ones while supporting different operating systems (Linux, Windows, MacOS, Android) and hardware (Nvidia, Arm, Intel, AMD ...).
Such approach allows researchers to create and share flexible APIs with JSON input/output for different AI/ML frameworks, libraries, compilers, models and data sets, connect them together into portable and unified workflows instead of hardwired scripts and data formats.
Our goals and plans¶
We continue improving the CodeReef platform as a centralized place to aggregate, version, test and create all components and workflows necessary to enable reproducible ML benchmarking and portable MLOps.
We are making a special effort to ensure that the CodeReef technology is non-intrusive - it complements, abstracts and interconnects all existing tools including MLFlow, MLPerf, Docker, Kubernetes, GitHub actions and make them more system aware rather than replacing or rewriting them.
Our long-term goal is to help the community quickly assemble portable ML workflows, participate in reproducible ML benchmarking and compare different ML/software/hardware stacks using online CodeReef dashboards, automate the creation of new ML benchmarks in collaboration with MLPerf, and share ML models as customizable and production-ready packages along with research papers.
Based on your feedback we plan to add the following features to our platform in 2020:
- GUI to add, test and interconnect ML components (software detection plugins, meta packages, pre/post-processing scripts)
- GUI to assemble portable ML/AI workflows from above components
- GUI for automated MLPerf benchmark submissions and validations across diverse systems, models and data set (see our proof-of-concept demo)
- Continuous testing of all shared components and portable workflows
- Improved dashboard to support reproducibility initiatives and optimization tournaments/hackathons
- Improved dashboard to reproduce and crowd-benchmark AI/ML solutions from SOTA papers.
Feedback and feature requests¶
We rely on your feedback to improve this open technology!
If something doesn't work as expected or you have new suggestions and feature requests please do not hesitate to open tickets on GitHub!