Add Custom NN Layers to MIVisionX

This page shows a step by step guide about adding a custom layer to OpenVX Neural Network Extension Library (vx_nn)

Refer to the Suggested development workflow before you plan your development.

A Crop Layer is used as an example for each step.

1. Implement your custom layer

Once the implementation is done, put your layer source code under src/ directory.

In addition, add your layer(s) in

• src/kernels.cpp

//register kernels
...
#176:  ERROR_CHECK_STATUS(publishCropLayer(context));
...

• src/kernels.h

enum user_kernel_e
{
...
#88: VX_KERNEL_CROP_LAYER_AMD = VX_KERNEL_BASE(VX_ID_AMD, NN_EXTENSION_LIBRARY) + 0x00d,
...
#134: vx_status publishCropLayer(vx_context context);
...

• include/vx_amd_nn.h

...
#198: VX_API_ENTRY vx_node VX_API_CALL vxCropLayer(vx_graph graph, vx_tensor input, vx_tensor ref, vx_tensor output, vx_scalar axis, vx_scalar offset1, vx_scalar offset2, vx_scalar offset3, vx_scalar offset4);
...

• CMakeLists.txt

...
list(APPEND SOURCES
...
#81: src/crop_layer.cpp
...

Now build the MIVisionX library again.

cd $(MIVisionX_ROOT)/build
cmake ../
make
sudo make install

Once the library builds without errors, now is the time to test your layer.

2. Test your layer with gdf

Refer to runvx on testing gdf file with runvx.

If you are to test a crop layer,

1. Create input files for crop layer.
2. Run gdf and dump the output.

Example

import vx_nn

data input = tensor:4,{w,h,c,n},FLOAT32,0
read input /directory/input.f32

data ref = tensor:4,{w,h,c,n},FLOAT32,0
read ref /directory/ref.f32

data output = tensor:4,{w,h,c,n},FLOAT32,0

data axis = scalar:INT32,2
data offset = scalar:INT32,0
data offset2 = scalar:INT32,0
data offset3 = scalar:INT32,2
data offset4 = scalar:INT32,2

node com.amd.nn_extension.crop_layer input ref output axis offset offset2 offset3 offset4

write output /directory/output.f32

3. Run caffe's crop layer with the same inputs and dump the output.

4. Compare two outputs and see if they are identical.

Run several unit tests to make sure it passes on every case including edge cases.

3. Add your layer in the model compiler

The model compiler converts a pre-trained neural-net model (caffe, onnx, nnef) to MIVisionX run-time code for optimized inference. Add your layer into the model compiler python code so that your layer can be converted into MIVisionX run-time code.

Example

Crop layer is used in the caffe framework. Therefore, the following python scripts should be modified:

• caffe_to_nnir.py
• nnir.py
• nnir_to_openvx.py

Look into the above codes and see how crop layer is being converted from caffe to OpenVX code.

4. Model flow testing

Once you add your layer to the model compiler, test it with pre-trained models that include your custom layer.

1. Train or get a model that includes your custom layer. For example, crop layer is used in Fully Convolutional Network for Semantic Segmentation. If you can't find a model, you may train your own model.

2. Run the model compiler.

For example, with pre-trained Caffe model,

python caffe_to_nnir.py <net.caffeModel> <nnirOutputFolder> --input-dims <n,c,h,w> [--verbose <0|1>]
python nnir_to_openvx.py [OPTIONS] <nnirInputFolder> <outputFolder>
cd <outputFolder>
cmake .
make
./anntest weights.bin

3. If the model is compiled and anntest is successful, that means your layer is ready to be used.

5. Create a pull request.

Create a pull request so that it can be merged into MIVisionX.

Refer to Suggested development workflow for creating a pull request.