Accessing Object Storage in GCP
Introduction
This tutorial demonstrates how to access your Google Cloud Storage from the Neuro platform. You will create a new Neuro project, a new project in GCP, a service account, and a bucket that's accessible from a job on the Neuro platform.
Make sure you have Neu.ro CLI and cookiecutter installed.
Creating Neuro and GCP Projects
To create a new Neuro project and build an image, run:
It's a good practice to limit the scope of access to a specific GCP project. To create a new GCP Project, run:
Make sure to set a billing account for your GCP project. See Creating and Managing Projects for details.
Creating a Service Account and Uploading an Account Key
First, create a service account for the job:
See Creating and managing service accounts for more details.
Then, download the account key:
Make sure that the newly created key is located at ~/
.
Create a new secret for the file:
Open .neuro/live.yaml
, find the remote_debug
section within jobs
, and add the following lines at the end of remote_debug
:
Creating a Bucket and Granting Access
Now, create a new bucket. Remember: bucket names are globally unique (see more information on bucket naming conventions).
Grant access to the bucket:
Testing
Create a file and upload it into Google Cloud Storage Bucket:
Change the default preset to cpu-small
in .neuro/live.yaml
to avoid consuming GPU for this test:
Run a development job and connect to the job's shell:
In your job's shell, activate the service account for CLI:
And try to use gsutil
to access your bucket:
Please note that in remote_debug
, train
, and jupyter
jobs the environment variable GOOGLE_APPLICATION_CREDENTIALS
points to your key file. So you can use it to authenticate other libraries.
For instance, you can access your bucket via Python API provided by package google-cloud-storage
:
To close remote terminal session, press ^D
or type exit
.
Please don't forget to terminate your job when you don't need it anymore:
Last updated