How to convert a Neural Network model to a standalone function source code

Question

I have to run source code in an environment where I do not have access to third-party libraries and I would like to be able to use a Neural Network model for predictions in that environment. I cannot run compiled code, it has to be source code.

I'd like to train my Neural Network using a popular library like Keras, Pytorch, Tensorflow etc... and convert that model to a source code function that can run in an environment that doesn't have access to that library. So the generated code can't be a wrapper around calls to the library. It needs to have everything it needs inside it to be able to run the Neural Network without external dependencies. And it has to be source code, not compiled code.

While researching this I realised that most libraries will have APIs to save the model to different kinds of serialized format but no way to generate code that could be run on its own.

How would I go about doing that?

Just to be extremely clear here is an example of the kind of code I would like to generate from a neural network model:

function predict(input) {
  // Here is all the code that contains the topology and weights of the network 
  // along with the code needed to load it up into memory and exercise the network, 
  // using no external libraries.

  return output;
}

So what I'm looking for is a library, or a technique that would allow me to do:

var neuralNetworkSourceCode = myNeuralNetworkModel.toSourceCode();
// neuralNetworkSourceCode is a string containing the source code described in the previous example

It could save the source code to a file instead of just producing a string, makes no difference to me. At this point I also don't care about the language it is producing source code in, but ideally it would be in one of these: c, c++, c#, java, python, javascript, go or rust.

Is there a library that does this? If not, how should I go about implementing this functionality.

If you can use tree models (trees, forests, GBDT), `treelite` emits C that you can run just about anywhere that supports c99 and malloc ([docs](https://treelite.readthedocs.io/en/latest/tutorials/deploy.html#prediction-instructions)). `glow` is the closest I'm familiar with for NNs, but it's designed to emit machine code rather than high-level languages like C/C++/Rust ([code](https://github.com/pytorch/glow), [paper](https://arxiv.org/abs/1805.00907)). — Alexander L. Hayes, Dec 19 '22 at 21:22

score 1 · Answer 1 · answered Dec 24 '22 at 18:30

Something similar was asked a while back: Convert Keras model to C++ or Convert Keras model to C, and those threads have some advice that may still be relevant. Neither mention keras2c (paper, code), which provides a k2c(model, ...) function after setting the library up.

Calling k2c on the model created from the Simple MNIST convnet example like this produces a .csv files with the weights and some code to set up the predictions:

import numpy as np
from tensorflow import keras
from tensorflow.keras import layers
from keras2c import k2c

(x_train, y_train), (_, _) = keras.datasets.mnist.load_data()
x_train = np.expand_dims(x_train.astype("float32") / 255, -1)
y_train = keras.utils.to_categorical(y_train, 10)

model = keras.Sequential([
    keras.Input(shape=(28, 28, 1)),
    layers.Conv2D(32, kernel_size=(3, 3), activation="relu"),
    layers.MaxPooling2D(pool_size=(2, 2)),
    layers.Conv2D(64, kernel_size=(3, 3), activation="relu"),
    layers.MaxPooling2D(pool_size=(2, 2)),
    layers.Flatten(),
    layers.Dropout(0.5),
    layers.Dense(10, activation="softmax")])

model.compile(loss="categorical_crossentropy", optimizer="adam", metrics=["accuracy"])
model.fit(x_train, y_train, batch_size=128, epochs=15, validation_split=0.1)
k2c(model, "MNIST", malloc=True, num_tests=10, verbose=True)

Compile the contents of include/ with make, then:

gcc -std=c99 -I./include/ -o predict_mnist MNIST.c MNIST_test_suite.c -L./include/ -l:libkeras2c.a -lm

And running the executable shows a quick benchmark:

$ ./predict_mnist
Average time over 10 tests: 6.059000e-04 s
Max absolute error for 10 tests: 3.576279e-07

How to convert a Neural Network model to a standalone function source code

1 Answers1