Support for OTLP/UDP Span Exporter

[jj22ee]

What are you trying to achieve?

We want to mitigate some of the performance issues in an AWS Lambda Environment when using the OTel Lambda Layers to send spans to AWS X-Ray. One mitigation involves avoid deploying the OTel Collector during a Lambda Function invocation to reduce cold start time and memory allocated. As a workaround for having no Collector, we are trying to send Spans to a UDP endpoint (AWS_XRAY_DAEMON_ADDRESS) in an AWS Lambda Environment. This UDP endpoint was originally designed for the X-Ray SDKs to send X-Ray Trace Segments to AWS X-Ray. Today in AWS Lambda, this UDP endpoint also supports receiving OTel Spans and send them to AWS X-Ray.

What did you expect to see?

We expect the OTel SDKs to provide a Span Exporter that can export spans via UDP protocol.

• This exporter can be configured via OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=udp.
• The endpoint is configurable when instantiating the UDP Exporter or via OTEL_EXPORTER_OTLP_TRACES_ENDPOINT env var.

Additional context.

• Slow start-up? opentelemetry-lambda#263 - issue summarizes the OTel performance impact in Lambda that we want to mitigate
• While the purpose of this UDP Exporter was originally for this Lambda Performance mitigation use case to vend to AWS X-Ray, it is quite generic w/ configurable endpoints. As such, what are the thoughts of the OTel community on the addition of this UDP Exporter?
• Today, AWS Observability's JS instrumentation is leveraging a UDP Exporter to export traces to Lambda's UDP endpoint to send traces to AWS X-Ray. This is the UDP Exporter's implementation in JavaScript.
• Related issue opened in opentelemetry-lambda

[dmathieu]

While there could always be hacky ways to do this, OTLP relies on protobuf and gRPC, and the gRPC folks have explicitly said support for UDP is not on their roadmap.
grpc/grpc#9493

[jj22ee]

OTLP relies on protobuf and gRPC

I thought OTLP doesn't fully rely on gRPC, for example there's a protocol for OTLP/HTTP. Can you clarify your implication of OTLP not being able to rely on UDP (Why not use UDP outside of gRPC)?

We have an example of using OTLP/UDP exporter that uses a Protobuf Trace Serializer. When you say "there could hacky ways to do this", could you clarify? Do you refer to the absence of an "OTLP/UDP" response when using a UDP Exporter, or are there any faulty implications on the receiver side?

---

There are is a good use case for UDP Exporters, mainly in Function as a Service (FaaS) environments, where:

• If HTTP or gRPC exporters are used, the HTTP/gRPC call must be a blocking call before the FaaS execution call returns, otherwise the HTTP/gRPC call is frozen along with the FaaS environment which is frozen when its execution completes
• Since the FaaS needs to wait for the HTTP/gRPC OTLP response from a server (given avoidance of Collector usage for performance), that can incur more billing for FaaS execution time

Where minimization of FaaS execution times is needed, this UDP Exporter would be desired tradeoff solution to not wait on server responses while using a lossy protocol.

[danielgblanco]

During spec triage @svrnm, @mx-psi, @jpkrohling and myself considered this issue. We think that there's no immediate need for UDP to part of the OTLP spec, since exporters and receivers can be created without being part of the spec (e.g. OTLP Arrow).

It'd be helpful to have a prototype and bringing it up for discussion again after evaluating the benefits.