Comment on page

Serving Models with TorchServe


This tutorial demonstrates how to use TorchServe (a flexible tool for serving PyTorch models) for serving models on
Before moving forward with the tutorial, make sure you have CLI installed.

Preparing Local Files

First, clone the TorchServe repository:
$ git clone [email protected]:pytorch/serve.git
You can also use the following version of this command:
$ git clone
Then, copy the model you would like to serve to your local repository. In this example, we will use DenseNet:
$ curl -o serve/examples/image_classifier/densenet161-8d451a50.pth
You can now copy the files from your local repository to the platform storage:
$ neuro cp -r serve storage:

Serving the Model

As you have all necessary files on the platform storage, you can mount them as a volume to your jobs. Run the following command:
$ neuro run --name serve --volume storage:serve:/home/serve:rw --preset gpu-small --http 8080 --no-http-auth --detach pytorch/torchserve:0.1.1-cuda10.1-cudnn7-runtime
Now, run a bash terminal from within this job:
$ neuro exec serve bash
Create the model-store folder:
$ mkdir /home/serve/model-store
Run Torch Model Archiver on your model:
$ torch-model-archiver --model-name densenet161 --version 1.0 --model-file /home/serve/examples/image_classifier/densenet_161/ --serialized-file /home/serve/examples/image_classifier/densenet161-8d451a50.pth --export-path /home/serve/model-store --extra-files /home/serve/examples/image_classifier/index_to_name.json --handler image_classifier
Before serving the model, run:
$ torchserve --stop
Now, you can serve your model. Here are the commands you can use for densenet161:
$ torchserve --start --ncs --model-store /home/serve/model-store --models densenet161.mar
curl -O
curl -v -T kitten_small.jpg

Accessing the Management API

To access the serving job's management API, you would first need to port-forward the serving job:
$ neuro port-forward serve 8081:8081
After that, you can copy the densenet161 folder to your local machine:
$ curl http://localhost:8081/models/densenet161