Experiment Tracking with Weights and Biases

Introduction

In this tutorial, we show how to set up experiment tracking via Weights and Biases on Neuro Platform using Neuro Project Template.

Creating Neuro Project

To create a new Neuro project, run:

neuro project init
cd <project-slug>
make setup

Authenticating W&B

Register a W&B account (note: W&B offers free limited accounts for individual researchers, see W&B pricing policy). Then, download your API key on W&B settings page and save it to a local file ./config/wandb-token.txt (you can also use a different file name, but in this case you'd need to put the file name to the env var: export WANDB_SECRET_FILE=<file-name>).

Check that Neuro project detects the key file:

make wandb-check-auth

In case of success, this command should print something like: Weights & Biases will be authenticated via key file: '/project-path/config/wandb-token.txt'. Now, if you run a develop, train, or jupyter job, Neuro will authenticate W&B via your API key by running wandb login at job's startup.

Testing

Run a development job and connect to the job's shell:

export PRESET=cpu-small # to avoid consuming GPU for this test
make develop
make connect-develop

In your job's shell, try to use wandb:

wandb status

You should see something like:

Logged in? True
Current Settings
{
"anonymous": "false",
"base_url": "https://api.wandb.ai",
"entity": null,
"git_remote": "origin",
"ignore_globs": [],
"project": null,
"run": "latest",
"section": "default"
}

Please find a real-world example of W&B usage on Neuro Platform in our ML Recipe for Hierarchical Attention. You can also find other examples on how to use experiment tracking and other features of W&B in official W&B's examples repo.

To close remote terminal session, press ^D or type exit.

Please don't forget to terminate your job when you don't need it anymore:

make kill-develop