fix Unable to find data type for weight_name='/encoder/layer.0/attention/output/dense/MatMul_output_0'

[VTrngNghia]

What does this PR do?

Adds a keyword argument to allow passing extra_options to ORTQuantizer.quantize()

To fix RuntimeError:

Unable to find data type for weight_name='/encoder/layer.0/attention/output/dense/MatMul_output_0'. shape_inference failed to return a type probably this node is from a different domain or using an input produced by such an operator. This may happen if you quantize a model already quantized. You may use extra_options DefaultTensorType to indicate the default weight type, usually onnx.TensorProto.FLOAT.

Maybe it can be added to AutoQuantizationConfig, but there any many @staticmethod for that, so maybe this quick fix is simpler.

Who can review?

It's very simple. Anyone can review.

• ONNX / ONNX Runtime : @fxmarty, @echarlaix, @JingyaHuang, @michaelbenayoun
• ONNX Runtime Training: @JingyaHuang
• BetterTransformer: @fxmarty
• GPTQ, quantization: @fxmarty, @SunMarc
• TFLite export: @michaelbenayoun

[severinsimmler]

+1

Thanks for fixing this @VTrngNghia

[Whadup]

Is there any harm in just hard-coding "DefaultTensorType": onnx.TensorProto.FLOAT in the extra_options dict?

[home15c6]

Do you mean hard-coding in your package code? Then no "harm" except every time you setup your environment (say, in Docker container), you'll have to apply that change again.
There are tools to automatically apply local changes (and I'm using them until they merge this PR). They're a bit hassle to setup.

[Whadup]

I meant to change the code in this PR to

 "extra_options": {
      "WeightSymmetric": quantization_config.weights_symmetric,
      "ActivationSymmetric": quantization_config.activations_symmetric,
      "EnableSubgraph": has_subgraphs,
      "ForceSymmetric": quantization_config.activations_symmetric and quantization_config.weights_symmetric,
      "AddQDQPairToWeight": quantization_config.qdq_add_pair_to_weight,
      "DedicatedQDQPair": quantization_config.qdq_dedicated_pair,
      "QDQOpTypePerChannelSupportToAxis": quantization_config.qdq_op_type_per_channel_support_to_axis,
      "DefaultTensorType": onnx.TensorProto.FLOAT
  },
and not pipe through the extra_options keyword argument

[michaelbenayoun]

Can you set the default to None and then create an empty dict in the method please?
Dict as default value is not good because they are mutable.

[michaelbenayoun]

cc @echarlaix