docs: produce message size rfc #4202
Conversation
fraidev
force-pushed
the
rfc_produce_msg_size
branch
3 times, most recently
from
8b495a7 to
7efe6bb
Compare
fraidev
force-pushed
the
rfc_produce_msg_size
branch
2 times, most recently
from
f60994d to
eaab9a5
Compare
rfc/produce-message-size.md
Outdated
|
|
||
| 1. Handling Larger Messages than Batch Size | ||
|
|
||
| If a single record exceeds the defined `batch_size`, Fluvio will process the record as a standalone request, ensuring that larger messages are not discarded or delayed. If the record does not exceed the `batch_size`, Fluvio will process the record as part of an already existing batch or create a new one if the batch is full. |
There was a problem hiding this comment.
Hmmm, is this true? Customers are complaining of packets being dropped if they are larger than the batch size.
There was a problem hiding this comment.
I wrote the expectation, I'll rewrite it with what is doing now and what should be changed.
There was a problem hiding this comment.
not sure why this behavior is needed. should move to enhancement if this doesn't applies to original problem
There was a problem hiding this comment.
this can be subsumed into exiting batching behavior. if batch_size is exceed, then batch is flushed
|
Never mind previous comments. Here is clarifying semantics.
|
rfc/produce-message-size.md
Outdated
|
|
||
| ### Batch Size | ||
|
|
||
| `batch_size` will define the maximum size of a batch of records that can be sent by the producer. If a record exceeds this size, Fluvio will process the record as a standalone message. |
There was a problem hiding this comment.
batch_size should be ultimate raw limit. again not sure what this address
There was a problem hiding this comment.
I think this behavior will be confused, other products has the same parameter config batch_size. In other products, Its behavior is to just check if the record will be inside a batch or not.
I think that we should use the compressed size for
Sure, Kafka for example uses 16kb for |
|
|
||
| ### Compression | ||
|
|
||
| `batch_size` and `max_request_size` will only use the uncompressed message size. |
There was a problem hiding this comment.
see suggestion. Having batch_size applies to final raw batch should simplify calculation.
fraidev
force-pushed
the
rfc_produce_msg_size
branch
from
eaab9a5 to
57a0bd7
Compare
fraidev
force-pushed
the
rfc_produce_msg_size
branch
from
fe81e0f to
bc03205
Compare
TL;DR
batch_sizeproducer config must not reject large records, just send them directly.Create a new
max_request_sizeproducer config that must reject large messages.I am using
max_request_sizebecause Kafka usesmax.request.sizebut we can change it to other config name.Compression sizes should not be used for these producer configs.
Related: