Tidy up of VGG10 RadioML model

[satishkumar538]

"the ONNX model has been tidied up by removing the input quantization (See the last paragraph of attached screenshot of readme file provided with finn-example of vgg10-radionl)" How I can perform this removal of input quantization on the onnx file of the sandbox repository https://github.com/Xilinx/brevitas-radioml-challenge-21 . Please suggest steps/ codes.

_

[fpjentzsch]

Hi,

I attached some Python code in a .txt file: network_surgery.txt

We used this code to perform the necessary "tidy up" (aka "network surgery") to make the model ready for FINN, but I have not tested it on the latest Brevitas/FINN versions. It is not very elegant because it works on the .onnx model and not on the Pytorch model level, but it gets the job done.

As you see, we remove all nodes before the first "Mul" node. In our case, this includes the (MultiThreshold->Add) sequence that represents the input quantization. We also replace the final Softmax/LogSoftmax node with a TopK node. The rest of it is mostly cleanup steps, some of which might be redundant or not needed anymore.

[fpjentzsch]

Could you show me the ONNX model after the network surgery and after the "convert_to_hls" step?

I see that your model does not include a nn.LogSoftmax(dim=1) layer at the end (you probably do this within the loss function). The code I provided assumes existence of this node, so maybe that's where things go wrong?

[Aakasha01Agarwal]

Thanks for your reply. See my ONNX model after network surgery and after the "convert_to_hls" step. Still I am not able to resolve the errors.
Please suggest.

ONNX After Network_Surgery

ONNX after convert_to_hls

[fpjentzsch]

I see, the problem is the feature map dimension across your model, which prevents the "MaxPoolNHWC" layers from being converted to "StreamingMaxPool_Batch" HLS layers.

Here you see that the kernel dimension (2 or 32 in your case) must divide the input dimension evenly. You would need to add additional padding or change your conv layers, so that you no longer have odd dimensions (e.g. 1025, 513, 257, ...) going into the MaxPool.

[Aakasha01Agarwal]

Dear fpjentzsch
Thanks for your continuous support. In my model, I have modified the kernel dimension and now I have even dimensions only. After running I get some other errors (see screenshot)

Error

)

ONNX after Network_Surgery

[fpjentzsch]

This means that the folding_config_file you set in the build_cfg defines folding factors (PE & SIMD) that are incompatible with the dimensions of a layer (e.g. if you have 16 output channels you cannot parallelize PE>16). You can:
a) Adjust the folding factors for each layer in this file, e.g. set PE to divide the MH=16 for the offending layer in this example.
b) Do not define a folding_config_file in your build_cfg, but set a target_fps value. FINN will then try to automatically determine folding factors. In some cases though, even the automatic variant will produce illegal folding factors. A new version is on its way: #438

[Aakasha01Agarwal]

Hello @fpjentzsch , as directed by you here, I added "AveragePool" to the list of valid nodes and here . The network surgery operations works fine. But when I observe the .onnx file of the model, after the surgery, I can see that an extra layer named "PAD" is generated before the "AveragePool" layer. We should note that whenever we use a "MaxPool" layer, no such "PAD" layer is generated before it.

Using this model for hardware deployment I am getting the following error which I am pretty sure is caused by this extra "PAD" layer only.

I am attaching the error screenshot and the log file screenshot too.

[fpjentzsch]

Weird, if you see the "Pad" node this early in the flow (before streamlining), that means it must be inserted during the Brevitas export. I'm not sure why the export would create such a node. Since it does nothing, maybe you can remove it manually during network surgery with a few lines of code?

[Aakasha01Agarwal]

Thanks for the suggestion. I removed the "Pad" node successfully. Now after running the hardware deployment file I am still getting an error (screenshot attached).

I am also attaching the .onnx file after the "convert_to_hls" step. I found that the "AveragePool" layer is not getting converted like "MaxPool" is getting converted to "StreamingMaxPool_Batch" HLS layer. I think this is caused because "AveragePool" layer is not defined in FINN library.

Please suggest as to how to resolve this issue or let me know which FINN file needs to be updated for incorporating this AveragePool Layer.

[fpjentzsch]

Oh, I never worked with AvgPool myself, so I missed the following:
AveragePool is supported via the Pool_Batch HLSCustomOp and the InferPool_Batch() transformation instead of InferStreamingMaxPool(). Also, it only seems to work with the special Brevitas layer QuantAvgPool2d and not nn.AvgPool1d. Pool_Batch was recently updated (123892f) to support non-square inputs (you could set one dimension of QuantAvgPool2d to 1), but I'm not sure if it actually works in the current dev branch. Might be worth a try.