Improve protobuf support for BigQuery Storage Write API

[alvioshki]

The feature request in short: to implement a better support for/remove the protobuf restrictions for BigQuery Storage Write API in the C# client.

I've tried to migrate from BigQuery legacy streaming API - to use the BigQuery Storage Write API with a C# client library instead.
However the protobuf handling restrictions have prevented me from taking this path to the end:
https://cloud.google.com/bigquery/docs/write-api#proto_buffer_handling

I have existing proto3 contracts. They use package specifiers, imports for wrapper types and other features. However, these are restricted according to documentation. I've tried using it anyway, I have hit some limitations with imports, enums and default values (I've managed to worked around this one).

Changing the proto3 contracts is not an option for me, as I have too many of them. Writing custom proto2 contracts and mapping from one to another is a no-go as well. I've considered implementing a dynamic mapper, but I am not confident my investment would work in the end.

The documentation states that "The Java and Go clients support arbitrary protocol buffers, because the client library normalizes the protocol buffer schema.".

What does this mean? My guesss is that this means that these client libraries have built-in mechanisms to handle and adapt arbitrary protocol buffer schemas, allowing more flexibility compared to other languages. Perhaps adapt the schema to conform to the restrictions imposed by the BigQuery Write API, such as removing or adjusting package specifiers, flattening nested messages or enums to meet the top-level definition requirements, resolving external references and embedding them into a single schema, etc.

Given I understand it correctly, could this be implemented in the C# client?

I would like to know if such an improvement arrives. Until then I will be using the legacy streamin API.

[jskeet]

Thanks for the clear description. We'll talk to the BigQuery team internally and discuss the next steps.

[amanda-tarafa]

@alvioshki We've discussed internally with the Bigquery team. And as you described, it's a little more complex than converting from proto3 to proto2, although that's also part of the problem. See the documentation for the NormalizeDescriptor function in Go for more details.

Unfortunately, we have higher priority work at the moment, so we cannot inmediately look into implementing similar normalization logic for the .NET client. We'll keep this request in our backlog for future consideration.

[alvioshki]

Thank you for informing me

…

On Wed, 19 Feb 2025 at 23:24, Amanda Tarafa Mas ***@***.***> wrote: @alvioshki <https://github.com/alvioshki> We've discussed internally with the Bigquery team. And as you described, it's a little more complex than converting from proto3 to proto2, although that's also part of the problem. See the documentation for the NormalizeDescriptor <https://pkg.go.dev/cloud.google.com/go/bigquery/storage/managedwriter/adapt#NormalizeDescriptor> function in Go for more details. Unfortunately, we have higher priority work at the moment, so we cannot inmediately look into implementing similar normalization logic for the .NET client. We'll keep this request in our backlog for future consideration. — Reply to this email directly, view it on GitHub <#13873 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIATVOXQQRYWGBOJLAC2G32QTLAVAVCNFSM6AAAAABSN2LUPOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRZGU3DANBSG4> . You are receiving this because you were mentioned.Message ID: ***@***.***> [image: amanda-tarafa]*amanda-tarafa* left a comment (googleapis/google-cloud-dotnet#13873) <#13873 (comment)> @alvioshki <https://github.com/alvioshki> We've discussed internally with the Bigquery team. And as you described, it's a little more complex than converting from proto3 to proto2, although that's also part of the problem. See the documentation for the NormalizeDescriptor <https://pkg.go.dev/cloud.google.com/go/bigquery/storage/managedwriter/adapt#NormalizeDescriptor> function in Go for more details. Unfortunately, we have higher priority work at the moment, so we cannot inmediately look into implementing similar normalization logic for the .NET client. We'll keep this request in our backlog for future consideration. — Reply to this email directly, view it on GitHub <#13873 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIATVOXQQRYWGBOJLAC2G32QTLAVAVCNFSM6AAAAABSN2LUPOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRZGU3DANBSG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>