Blob lease renewal is not rejected after expiration and modification

[bluenote10]

Which service(blob, file, queue, table) does this issue concern?

blob

Which version of the Azurite was used?

3.29.0

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

DockerHub

What's the Node.js version?

None (or the one shipped with the Docker container)

What problem was encountered?

According to Azure's Lease Blob specification, a renew should be rejection if the lease has expired and the blob has been modified in the meantime. This is also the behavior we can see on Azure itself, i.e., the renew gets properly rejected with an error in this case.

With Azurite however, the renew is not rejected, and thus, diverges from the Azure specs.

Steps to reproduce the issue?

The following Python snippet can be used to replay the test scenario:

import time

from azure.storage.blob import BlobServiceClient


def check_expire_with_modify_behavior(
    blob_service_client: BlobServiceClient,
    container: str,
    blob_path: str,
):
    print("Writing initial blob...")
    blob_service_client.get_container_client(container).upload_blob(
        blob_path, "original content", overwrite=True
    )

    print("Acquiring lease...")
    blob_client = blob_service_client.get_blob_client(container, blob_path)
    lease = blob_client.acquire_lease(lease_duration=15)

    print("Waiting for lease to expire...")
    time.sleep(20)

    print("Modifying blob without specifying the lease...")
    # This should make it impossible to renew the lease, because it gets modified
    # after the lease expiration.
    blob_service_client.get_container_client(container).upload_blob(
        blob_path, "modified content", overwrite=True
    )

    print("Renewing the lease...")
    lease.renew()

When giving the snippet a BlobServiceClient pointing to real Azure the output is:

Getting client for: Azure
Writing initial blob...
Acquiring lease...
Waiting for lease to expire...
Modifying blob without specifying the lease...
Renewing the lease...
Traceback (most recent call last):
  File "/home/me/debug_azurite_renew.py", line 31, in check_expire_with_modify_behavior
    lease.renew()
  File "/path/to/venv/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/path/to/venv/lib/python3.10/site-packages/azure/storage/blob/_lease.py", line 179, in renew
    process_storage_error(error)
  File "/path/to/venv/lib/python3.10/site-packages/azure/storage/blob/_shared/response_handlers.py", line 184, in process_storage_error
    exec("raise error from None")   # pylint: disable=exec-used # nosec
  File "<string>", line 1, in <module>
  File "/path/to/venv/lib/python3.10/site-packages/azure/storage/blob/_lease.py", line 172, in renew
    response = self._client.renew_lease(
  File "/path/to/venv/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/path/to/venv/lib/python3.10/site-packages/azure/storage/blob/_generated/operations/_blob_operations.py", line 3153, in renew_lease
    map_error(status_code=response.status_code, response=response, error_map=error_map)
  File "/path/to/venv/lib/python3.10/site-packages/azure/core/exceptions.py", line 164, in map_error
    raise error
azure.core.exceptions.ResourceExistsError: The lease ID specified did not match the lease ID for the blob.
RequestId:9a7c1feb-e01e-0092-809d-539416000000
Time:2024-01-30T16:55:19.8648998Z
ErrorCode:LeaseIdMismatchWithLeaseOperation
Content: <?xml version="1.0" encoding="utf-8"?><Error><Code>LeaseIdMismatchWithLeaseOperation</Code><Message>The lease ID specified did not match the lease ID for the blob.
RequestId:9a7c1feb-e01e-0092-809d-539416000000
Time:2024-01-30T16:55:19.8648998Z</Message></Error>

I.e., the renew gets rejected properly. With Azurite there is no exception, and the renewal silently passes.

I'll try to figure out how to get to the Azurite logs and attach them hopefully soon... EDIT: I'm mainly seeing just:

172.20.0.1 - - [30/Jan/2024:16:55:50 +0000] "PUT /local_user/azuritecontainer/test_folder/foo HTTP/1.1" 201 -
172.20.0.1 - - [30/Jan/2024:16:55:50 +0000] "PUT /local_user/azuritecontainer/test_folder/foo?comp=lease HTTP/1.1" 200 -

Have you found a mitigation/solution?

No.

[EmmaZhu]

Hi @bluenote10 ,

Thanks a lot for bringing this to us. We'll fix it.

[EmmaZhu]

#2354

[bluenote10]

@EmmaZhu Thanks for looking into this issue!

We actually noticed another inconsistency, which is closely related. The reproduction snippet would be:

import time

from azure.storage.blob import BlobServiceClient

def check_write_to_blob_after_lease_expire(
    blob_service_client: BlobServiceClient,
    container: str,
    blob_path: str,
):
    print("Writing initial blob...")
    blob_service_client.get_container_client(container).upload_blob(
        blob_path, "original content", overwrite=True
    )

    print("Acquiring lease...")
    blob_client = blob_service_client.get_blob_client(container, blob_path)
    lease = blob_client.acquire_lease(lease_duration=15)

    print("Waiting for lease to expire...")
    time.sleep(20)

    print("Writing to blob with lease, after lease has expired...")
    blob_client.upload_blob("modified_content", lease=lease, overwrite=True)

Using Azurite this errors with a LeaseNotPresentWithBlobOperation code:

azure.core.exceptions.HttpResponseError: A lease ID was specified, but the lease for the blob has expired.
RequestId:c7aad088-ff38-4c4b-9593-f37324bf43a1
Time:2024-02-12T16:24:08.275Z
ErrorCode:LeaseNotPresentWithBlobOperation
Content: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error>
  <Code>LeaseNotPresentWithBlobOperation</Code>
  <Message>A lease ID was specified, but the lease for the blob has expired.
RequestId:c7aad088-ff38-4c4b-9593-f37324bf43a1
Time:2024-02-12T16:24:08.275Z</Message>
</Error>

But on Azure it actually errors with code LeaseLost:

azure.core.exceptions.HttpResponseError: A lease ID was specified, but the lease for the blob has expired.
RequestId:4cd75829-801e-004a-2cd0-5dd089000000
Time:2024-02-12T16:27:59.4300167Z
ErrorCode:LeaseLost
Content: <?xml version="1.0" encoding="utf-8"?><Error><Code>LeaseLost</Code><Message>A lease ID was specified, but the lease for the blob has expired.
RequestId:4cd75829-801e-004a-2cd0-5dd089000000
Time:2024-02-12T16:27:59.4300167Z</Message></Error>

The inconsistency in the error codes makes it awkward to handle that case in a way that works for both Azurite and real Azure. Also LeaseLost sounds a bit more appropriate than LeaseNotPresentWithBlobOperation (in fact the lease is present with the blob operation, and that is even the problem, because the blob would even be writable if the blob operation would omit the lease...).

@EmmaZhu Is this covered by #2354 as well? Or do you want me to open a separate issue for that?