fix(storage): fix stream termination in MRD.

[shubham-diwakar]

1. Make CloseSend() call before releasing resource.
2. Drain inbound response from the stream.

[BrennaEpp]

Should we also drain the stream after the CloseSend (as we do in drainInboundStream(), receiving from stream until we get a non-nil error) to make sure its resources are released? See grpc/grpc-go@365770f

[shubham-diwakar]

Thanks for the best practice, applied this.
Using stream.recv just for determination of error.

[shubham-diwakar]

Actually i am not sure that will be get any outputs here if we call stream.Recv post CloseSend(). What if all the responses where consumed by streamReceiver go routine?

Although there are some cases when we close the stream even with requests added i have drained responses there.

LMK your thoughts.

[cjc25]

I think Recv() continually returns the err (io.EOF or something else) once the stream is done? This should be easy to check with a toy program.

However, multiple concurrent calls to Recv() are not allowed, so if streamReceiver goroutine may be calling Recv(), you have to be careful not to call Recv() elsewhere until streamReceiver is done. I think it's probably easiest for all Recv() calls to live on that one goroutine.

It's maybe easiest to call CloseSend() on the same goroutine which calls Send()? Then you have one goroutine for Send/CloseSend, and another for Recv, and user code can cancel the context then call Close() to trigger a cancellation.

[shubham-diwakar]

Thanks Chris for the suggestion.
That kind of simplifies structure too.

Now one go routine has all Recv() calls and other has all Send()/CloseSend() calls.