Comment on page
Serving Models with TorchServe
This tutorial demonstrates how to use TorchServe (a flexible tool for serving PyTorch models) for serving models on Neu.ro.
$ git clone [email protected]:pytorch/serve.git
You can also use the following version of this command:
$ git clone https://github.com/pytorch/serve.git
Then, copy the model you would like to serve to your local repository. In this example, we will use DenseNet:
$ curl -o serve/examples/image_classifier/densenet161-8d451a50.pth https://download.pytorch.org/models/densenet161-8d451a50.pth
You can now copy the files from your local repository to the platform storage:
$ neuro cp -r serve storage:
As you have all necessary files on the platform storage, you can mount them as a volume to your jobs. Run the following command:
$ neuro run --name serve --volume storage:serve:/home/serve:rw --preset gpu-small --http 8080 --no-http-auth --detach pytorch/torchserve:0.1.1-cuda10.1-cudnn7-runtime
Now, run a bash terminal from within this job:
$ neuro exec serve bash
Create the
model-store folder:$ mkdir /home/serve/model-store
$ torch-model-archiver --model-name densenet161 --version 1.0 --model-file /home/serve/examples/image_classifier/densenet_161/model.py --serialized-file /home/serve/examples/image_classifier/densenet161-8d451a50.pth --export-path /home/serve/model-store --extra-files /home/serve/examples/image_classifier/index_to_name.json --handler image_classifier
Before serving the model, run:
$ torchserve --stop
Now, you can serve your model. Here are the commands you can use for
densenet161:$ torchserve --start --ncs --model-store /home/serve/model-store --models densenet161.mar
curl -O https://raw.githubusercontent.com/pytorch/serve/master/docs/images/kitten_small.jpg
curl -v https://serve--your-user-name.jobs.default.org.neu.ro/predictions/densenet161 -T kitten_small.jpg
To access the serving job's management API, you would first need to port-forward the serving job:
$ neuro port-forward serve 8081:8081
After that, you can copy the
densenet161 folder to your local machine:$ curl http://localhost:8081/models/densenet161
Last modified 1yr ago