How can one control the number of threads Windows ML is using for evaluation

Question

I'm trying to benchmark Windows ML against other backends and see some weird distribution of inference times (see plot). This is with the CPU backend using the ARM64 architecture. On ARM there's no bimodal distribution.

I don't have a good intuition on why there's two modes in the distribution of inference times. There doesn't seem to be a temporal correlation, I run the network once per second and it switches between the "slow" and "fast" mode seemingly randomly.

One of the guesses I have is that perhaps sometimes Windows ML decides to use two threads and sometimes one, possibly depending on estimated device load. However, unlike with TensorFlow Lite or Caffe2 I haven't found a way to control the number of threads Windows ML uses. So the question is:

Is there a way to control the number of threads Windows ML is using for evaluation in CPU mode, or is it guaranteed to use only one thread for computation in any case?

Other pointers to what could cause this weird behavior are also welcome.

Answer 1

Please see this Windows Machine Learning API that allows users of the API to set the number of intra-operator threads : https://learn.microsoft.com/en-us/windows/ai/windows-ml/native-apis/intraopnumthreads

NOTE: Setting this value higher than the number of logical cores on the CPU may result in an inefficient threadpool and a slower evaluation. Here is a code snippet on how to use the API:

void SetIntraOpNumThreads(LearningModel model) {
    // Create LearningModelSessionOptions
    auto options = LearningModelSessionOptions();
    auto nativeOptions = options.as<ILearningModelSessionOptionsNative>();
 
    // Set the number of intra op threads to half of logical cores.
    uint32_t desiredThreads = std::thread::hardware_concurrency() / 2;
    nativeOptions->SetIntraOpNumThreadsOverride(desiredThreads);
 
    // Create session
    LearningModelSession session = nullptr;
    WINML_EXPECT_NO_THROW(session = LearningModelSession(model, LearningModelDeviceKind::Cpu, options));
}