English
Neu.ro Core
Neu.ro Toolbox
ADMINISTRATION
Getting Started
Introduction
There are two things you will need to do before you start working with Neu.ro:
After this, you're free to explore the platform and it's functionality. As a good starting point, we've included a section about development on GPU with Jupyter Notebooks.
Installing the CLI
Web Terminal doesn't require installation and can quickly get you familiar with Neu.ro, allowing you to work with the platform in a browser.
However, installing Neu.ro CLI locally may prove more effective for long-term use:
- You won't need to pay for simply running the job like you do in Web UI.
- Your source code and other local files will be saved directly on your machine.
Installation instructions
Linux and Mac OS
Windows
Installing via pipx
Our neuro-all package available in pipx will automatically install all required components:
1
$ pip install pipx
2
$ pipx install neuro-all
3
$ pipx upgrade neuro-all
Copied!
Installing via pip
You can also install all of the components through pip.
Neu.ro CLI requires Python 3 installed (recommended: 3.8; required: 3.7.9 or newer). We suggest using the Anaconda Python 3.8 Distribution.
1
$ pip install -U neuro-cli neuro-extras neuro-flow
2
$ neuro login
Copied!
If your machine doesn't have GUI, use the following command instead of neuro login:
1
$ neuro config login-headless
Copied!
Installing via pipx
Our neuro-all package available in pipx will automatically install all required components:
1
$ pip install pipx
2
$ pipx install neuro-all
3
$ pipx upgrade neuro-all
Copied!
Installing via pip
You can also install all of the components through pip.
When you have it up and running, run the following commands in Conda Prompt:
1
$ conda install -c conda-forge make
2
$ conda install -c conda-forge git
3
$ pip install -U neuro-cli neuro-extras neuro-flow
4
$ pip install -U certifi
5
$ neuro login
Copied!
To make sure that all commands you can find in our documentation work properly, don't forget to run
bash every time you open Conda Prompt.Understanding the core concepts
On the Neu.ro Core level, you will work with jobs, environments, and storage. To be more specific, a job (an execution unit) runs in a given environment (Docker container) on a given preset (a combination of CPU, GPU, and memory resources allocated for this job) with several storage instances (block or object storage) attached.
Here are some examples.
Hello, World!
Run a job on CPU which prints “Hello, World!” and shuts down:
1
$ neuro run --preset cpu-small --name test ubuntu -- echo Hello, World!
Copied!
Executing this command will result in an output like this:
1
√ Job ID: job-7dd12c3c-ae8d-4492-bdb9-99509fda4f8c
2
√ Name: test
3
- Status: pending Creating
4
- Status: pending Scheduling
5
- Status: pending ContainerCreating
6
√ Http URL: https://test--jane-doe.jobs.default.org.neu.ro
7
√ The job will die in a day. See --life-span option documentation for details.
8
√ Status: succeeded
9
√ =========== Job is running in terminal mode ===========
10
√ (If you don't see a command prompt, try pressing enter)
11
√ (Use Ctrl-P Ctrl-Q key sequence to detach from the job)
12
Hello, World!
Copied!
A simple GPU job
Run a job on GPU in the default Neu.ro environment (
neuromation/base) that checks if CUDA is available in this environment:1
$ neuro run --preset gpu-small --name test neuromation/base -- python -c "import torch; print(torch.cuda.is_available());"
Copied!
We used the
gpu-small preset for this job. To see the full list of presets you can use, run the following command:1
$ neuro config show
Copied!
Working with platform storage
Create a new
demo directory in the root directory of your platform storage:1
$ neuro mkdir -p storage:demo
Copied!
Run a job that mounts the
demo directory from platform storage to the /demo directory in the job container and creates a file in it:1
$ neuro run --volume storage:demo:/demo:rw ubuntu -- bash -c "echo Hello >> /demo/hello.txt"
Copied!
Check that the file you have just created is actually on the storage:
1
$ neuro ls storage:demo
Copied!
Developing on GPU with Jupyter Notebooks
Development in Jupyter Notebooks is a good example of how the Neuro Platform can be used. While you can run a Jupyter Notebooks session in one command through CLI or in one click in the web UI, we recommend project-based development. To simplify the process, we provide a project template which is based on the cookiecutter package and is a part of the Neu.ro Toolbox. This template provides the basic necessary folder structure and integrations with several recommended tools.
Initializing a Neuro cookiecutter project
First, you will need to install the cookiecutter package via pip or pipx:
1
$ pipx install cookiecutter
Copied!
1
$ cookiecutter gh:neuro-inc/cookiecutter-neuro-project --checkout release
Copied!
This command will prompt you to enter some info about your new project:
1
project_name [Neuro Project]: New Cookiecutter Project
2
project_dir [new cookiecutter project]:
3
project_id [new_cookiecutter_project]:
4
code_directory [modules]:
5
preserve Neuro Flow template hints [yes]:
Copied!
Default values are indicated by square brackets [ ]. You can use them by pressing Enter.
To navigate to the project directory, run:
1
$ cd new-cookiecutter-project
Copied!
Project structure
The structure of the project's folder will look like this:
1
new-cookiecutter-project
2
├── .github/ <- Github workflows and a dependabot.yml file
3
├── .neuro/ <- neuro and neuro-flow CLI configuration files
4
├── config/ <- configuration files for various integrations
5
├── data/ <- training and testing datasets (we don't keep it under source control)
6
├── notebooks/ <- Jupyter notebooks
7
├── modules/ <- models' source code
8
├── results/ <- training artifacts
9
├── .gitignore <- default .gitignore file for a Python ML project
10
├── .neuro.toml <- autogenerated config file for Neuro CLI
11
├── .neuroignore <- a file telling Neuro CLI which files to ignore while uploading to the platform storage
12
├── HELP.md <- autogenerated template reference
13
├── README.md <- autogenerated informational file
14
├── Dockerfile <- description of the docker image used for training in your project
15
├── apt.txt <- list of system packages to be installed in the training environment
16
├── requirements.txt <- list of Python dependencies to be installed in the training environment
17
├── setup.cfg <- linter settings (Python code quality checking)
18
└── update_actions.py <- script used to update neuro-flow actions in one of the GitHub workflows
Copied!
The template contains the
neuro/live.yaml configuration file for neuro-flow. This file guarantees a proper connection between the project structure, the base environment that we provide, and actions with storage and jobs. For example, the upload command synchronizes sub-folders on your local machine with sub-folders on the persistent platform storage, and those sub-folders are synchronized with the corresponding sub-folders in job containers.Setting up the environment and running Jupyter
To set up the project environment, run:
1
$ neuro-flow build train
2
$ neuro-flow mkvolumes
Copied!
When these commands are executed, system packages from
apt.txt and pip dependencies from requirements.txt are installed to the base environment. It supports CUDA by default and contains the most popular ML/AI frameworks such as Tensorflow and Pytorch.For Jupyter Notebooks to run properly, the
train.py script and the notebook itself should be available on the storage. Upload the code directory containing this file to the storage by using the following command:1
$ neuro-flow upload ALL
Copied!
Now you need to choose a preset on which you want to run your Jupyter jobs. To view the list of presets available on the current cluster, run:
1
$ neuro config show
Copied!
To start a Jupyter Notebooks session run:
1
$ neuro-flow run jupyter
Copied!
This command will open Jupyter Notebooks interface in your default browser.
Also, you can adjust the jupyter job's preset configuration by specifying the preset argument to reflect your preferred preset:
1
jupyter:
2
action: gh:neuro-actions/[email protected]<version>
3
args:
4
...
5
preset: <gpu-preset-name>
6
...
Copied!
After this, each time you run a Jupyter job, it will use the specified by default without the need for you to provide it in a CLI command:
1
$ neuro-flow run jupyter
Copied!
Now, when you edit notebooks, they are updated on your platform storage. To download them locally (for example, to save them under a version control system), run:
1
$ neuro-flow download notebooks
Copied!
Don’t forget to terminate your job when you no longer need it (the files won’t disappear after that):
1
$ neuro-flow kill jupyter
Copied!
To check how many credits you have left, run:
1
$ neuro config show
Copied!
Last modified 4mo ago
Copy link