whisk documentation home¶
whisk is an open-source ML project framework that makes collaboration, reproducibility, and deployment “just work”. It combines a suite of lightweight tools with a logical and flexible project structure. Release your model to the world without a software engineer.
Whisk doesn’t lock you into a particular ML framework or require you to learn yet another ML packaging API. Instead, it lets you leverage the large Python ecosystem by structuring your ML project in a Pythonic-way. Whisk does the structuring while you focus on the data science.
Read more about our beliefs.
Getting Started¶
Start by creating a project. Begin a terminal session and run the commands below. Note: We use demo as the project name in the examples below. If you use a different project name, be sure to replace demo with the name of your project.
$ pip install whisk
$ whisk create demo
$ cd demo
$ source venv/bin/activate
The commands above do the following:
Create a project named “demo”
Activate the project’s venv
To try out all of the features, continue the quick tour of whisk →.
Examples¶
The whisk-ml GitHub org contains example whisk projects. Check out these examples and clone them locally. Since whisk makes reproducibility “just work”, in most cases you simply need to run whisk setup to use the models generated by the projects. Here are few examples to start with:
Text Classification with Keras and Tensorflow - A model that predicts which tweets are about real disasters and which ones are not. This project uses DVC to version control the data download and training stages.
Image Classification with Tensorflow - A classifier to determine if an image is of a Mountain bike or a Road bike.
Iris Classification with SKLearn (Coming soon)
Documentation¶
Beliefs¶
Most ML framework approaches start with a shaky foundation. Rather than building off software engineering best practices for structuring your work, they encourage quick hacks that result in ML projects that are hard to debug, collaborate on, reproduce, and deploy. The beliefs below inspired us to create whisk:
A Reproducible, collaborative project is a solved problem for classical software - We don’t need to re-invent the wheel for machine learning projects. Instead, we need guide rails to help data scientists structure projects without forcing them to also become software engineers.
A notebook is great for exploring, but not for production - A data science notebook is where experimentation starts, but you can’t create a reproducible, collaborative ML project with just a
*.ipynbfile.Optimize for debugging - 90% of writing software is fixing bugs. It should be fast and easy to debug your model logic locally. You should be able to search your error and find results, not sift through custom package source code or stop and restart Docker containers.
Python already has a good package manager - We don’t need overly abstracted solutions to package a trained ML model. A properly structured ML project lets you distribute the model via
pip, making it easy for anyone to benefit from your work.Version control is a requirement - You can’t have a reproducible project if the code and training data isn’t in version control.
Docker is an unsteady foundation - when we explicitly declare and isolate dependencies, we don’t need to rely on the implicit existence of packages installed in a Docker container. Python has solid native tools for dependency management.
Kubernetes is overkill - very few web applications require the complexity of a container-orchestration system. Your deployed model is no different. Most models can run on boring, reliable technology.