Serving Models with TorchServe

Search…
Introduction
This tutorial demonstrates how to use TorchServe (a flexible tool for serving PyTorch models) for serving models on Neu.ro. 
Before moving forward with the tutorial, make sure you have Neu.ro CLI installed.
Preparing Local Files
First, clone the TorchServe repository:
1
$ git clone [email protected]:pytorch/serve.git
Copied!
You can also use the following version of this command:
1
$ git clone https://github.com/pytorch/serve.git
Copied!
Then, copy the model you would like to serve to your local repository. In this example, we will use DenseNet:
1
$ curl -o serve/examples/image_classifier/densenet161-8d451a50.pth https://download.pytorch.org/models/densenet161-8d451a50.pth
Copied!
You can now copy the files from your local repository to the platform storage:
1
$ neuro cp -r serve storage:
Copied!
Serving the Model
As you have all necessary files on the platform storage, you can mount them as a volume to your jobs. Run the following command:
1
$ neuro run --name serve --volume storage:serve:/home/serve:rw --preset gpu-small --http 8080 --no-http-auth --detach pytorch/torchserve:0.1.1-cuda10.1-cudnn7-runtime
Copied!
Now, run a bash terminal from within this job:
1
$ neuro exec serve bash
Copied!
Create the model-store folder:
1
$ mkdir /home/serve/model-store
Copied!
Run Torch Model Archiver on your model:
1
$ torch-model-archiver --model-name densenet161 --version 1.0 --model-file /home/serve/examples/image_classifier/densenet_161/model.py --serialized-file /home/serve/examples/image_classifier/densenet161-8d451a50.pth --export-path /home/serve/model-store --extra-files /home/serve/examples/image_classifier/index_to_name.json --handler image_classifier
Copied!
Before serving the model, run:
1
$ torchserve --stop
Copied!
Now, you can serve your model. Here are the commands you can use for densenet161:
1
$ torchserve --start --ncs --model-store /home/serve/model-store --models densenet161.mar
2
  curl -O https://raw.githubusercontent.com/pytorch/serve/master/docs/images/kitten_small.jpg
3
  curl -v https://serve--your-user-name.jobs.default.org.neu.ro/predictions/densenet161 -T kitten_small.jpg
Copied!
Accessing the Management API
To access the serving job's management API, you would first need to port-forward the serving job:
1
$ neuro port-forward serve 8081:8081
Copied!
After that, you can copy the densenet161 folder to your local machine:
1
$ curl http://localhost:8081/models/densenet161
Copied!
​
Deploying with Seldon
Next - Neu.ro Toolbox
Working With the Sandbox
Last modified 6mo ago
Copy link
Contents
Introduction
Preparing Local Files
Serving the Model
Accessing the Management API