Accessing Object Storage in AWS

Search…
Introduction
This tutorial demonstrates how to access your AWS S3 from Neuro Platform. You will set up a new Neuro project, create an S3 bucket, and make it is accessible from Neuro Platform jobs.
Make sure you have Neu.ro CLI installed.
Creating Neuro Project
To create a new Neuro project, run:
1
neuro project init
2
cd <project-slug>
3
neuro-flow build myimage
Copied!
Creating an AWS IAM User
Follow Creating an IAM User in Your AWS Account.
In AWS Console, go to "Services" drop-down list, "IAM" (Identity and Access Management). On the left-hand panel, choose "Access management" -> "Users", click the "Add user" button, go through the wizard, and as a result you'll have a new user added:
Ensure that this user has "AmazonS3FullAccess" in the list of permissions.
Then, you'll need to create an access key for the newly created user. For that, go to the user description, then to the "Security credentials" tab, and press the "Create access key" button:
Put these credentials to the local file in the home directory ~/aws-credentials.txt. For example:
1
[default]
2
aws_access_key_id=AKIAIOSFODNN7EXAMPLE
3
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Copied!
Set appropriate permissions to the secret file:
1
chmod 600 ~/aws-credentials.txt
Copied!
Set up the Neuro Platform to use this file and check that your Neuro project detects it:
1
neuro secret add aws-key @~/aws-credentials.txt
Copied!
Open .neuro/live.yaml, find remote_debug section within jobs in it and add the following lines at the end of remote_debug:
1
jobs:
2
  remote_debug:
3
     ...
4
     secret_files: '["secret:aws-key:/var/secrets/aws.txt"]'
5
     additional_env_vars: '{"AWS_CONFIG_FILE": "/var/secrets/aws.txt"}'
Copied!
Creating a Bucket and Granting Access
Now, create a new S3 bucket. Remember: bucket names are globally unique.
1
BUCKET_NAME="my-neuro-bucket-42"
2
aws s3 mb s3://$BUCKET_NAME/
Copied!
Testing
Create a file and upload it into S3 Bucket:
1
echo "Hello World" | aws s3 cp - s3://$BUCKET_NAME/hello.txt
Copied!
Change default preset to cpu-small in .neuro/live.yamlto avoid consuming GPU for this test:
1
defaults:
2
  preset: cpu-small
Copied!
Run a development job and connect to the job's shell:
1
neuro-flow run remote_debug
Copied!
In your job's shell, try to use s3 to access your bucket:
1
aws s3 cp s3://my-neuro-bucket-42/hello.txt -
Copied!
To close the remote terminal session, press ^D or type exit.
Please don't forget to terminate the job when you've done working with it:
1
neuro-flow kill remote-debug
Copied!
Accessing Object Storage in GCP
Next - Neu.ro Toolbox
Remote Debugging
Last modified 9mo ago
Copy link
Contents
Introduction
Creating Neuro Project
Creating an AWS IAM User
Creating a Bucket and Granting Access
Testing