Experiment Tracking with Weights & Biases

Introduction

In this tutorial, we show how to set up experiment tracking via Weights & Biases on Neuro Platform using Neuro Project Template.

Creating Neuro Project

To create a new Neuro project, run:

neuro project init
cd <project-slug>
make setup

Connecting Weights & Biases

Now, connect your project with Weights & Biases:

  1. Find your API key (it is also called a token) on W&B’s settings page(section “API keys”). It should be a sequence like cf23df2207d99a74fbe169e3eba035e633b65d94.

  2. Save your API key (token) to a file in your local home directory ~ and protect it by setting appropriate permissions to make W&B available on Neu.ro platform:

export WANDB_SECRET_FILE=wandb-token.txt
echo "cf23df2207d99a74fbe169e3eba035e633b65d94" > ~/$WANDB_SECRET_FILE
chmod 600 ~/$WANDB_SECRET_FILE

After that, create a Neu.ro secret:

neuro secrets add wandb-token @~/$WANDB_SECRET_FILE

Next, replace this line in your Makefile:

SECRETS?=

With that one:

SECRETS?="-e WANDB_API_KEY=secret:wandb-token"

Now, you can start using W&B API in your code.

Testing

Run a development job and connect to the job's shell:

export PRESET=cpu-small # to avoid consuming GPU for this test
make develop
make connect-develop

In your job's shell, try to use wandb:

wandb status

You should see something like:

Logged in? True
Current Settings
{
"anonymous": "false",
"base_url": "https://api.wandb.ai",
"entity": null,
"git_remote": "origin",
"ignore_globs": [],
"project": null,
"run": "latest",
"section": "default"
}

Please find a real-world example of W&B usage on Neuro Platform in our ML Recipe for Hierarchical Attention. You can also find other examples on how to use experiment tracking and other features of W&B in official W&B's examples repo.

To close remote terminal session, press ^D or type exit.

Please don't forget to terminate your job when you don't need it anymore:

make kill-develop