2

What is the advantage of SageMaker Neo vs the specialized native runtimes that every ML accelerator provides, such as NVIDIA TensorRT, Intel OpenVINO, DeepView RT, CoreML, ArmNN, etc.? I understand that Neo uses some of these frameworks, like TensorRT, under the hood, but what is the advantage of having the model compiled for Neo instead or TensorRT directly?

I suppose companies with edge ML workloads will standardize on a given platform, eg NVIDIA, ARM or Intel, and each vendor is probably the best positioned to provide an optimized runtime for its own platform with cross-compiling tools from other frameworks (everybody seems to support TensorFlow and ONNX). Is this correct? Have you seen different cases in the field?

Another point is that according to the official documentation, support for different ML model is limited in frameworks other than MXnet. Why would then a company choose Neo if it has models in PyTorch or Tensorflow/Keras?

juvchan
  • 6,113
  • 2
  • 22
  • 35

1 Answers1

2

Thanks for your questions, here are some of my thoughts on the value SageMaker Neo provides to the customers.

  1. Directly programming specialized ML accelerator runtime frameworks (such as TensorRT), requires a steep learning curve. Most companies either do not have the resources to do that or would not want to invest resources in this area as it is not a core area of interest for them. They would rather use a managed service on a managed platform like SageMaker Neo.
  2. People normally use ONNX to deploy to TensorRT but as our experience from customer anecdotes, ONNX is quite limited in terms of supporting dynamic models and even fails on converting some static shape models. So, model coverage is an issue with ONNX.
  3. Most other accelerator libraries such as OpenVINO and ArmNN have even less supported operators so some advanced models will just fail. However, Neo can fallback those operators to TVM implementations and run inference.
  4. Neo supports many models in the currently popular ML frameworks like TensorFlow and PyTorch, please look at the list here: https://docs.aws.amazon.com/sagemaker/latest/dg/neo-supported-devices-edge-frameworks.html).

Please let me know if you have further questions.

Jian Sheng
  • 21
  • 1