Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for DefaultAzureCredential with pl.read/scan_delta() #20808

Open
drumtorben opened this issue Jan 20, 2025 · 0 comments
Open

Support for DefaultAzureCredential with pl.read/scan_delta() #20808

drumtorben opened this issue Jan 20, 2025 · 0 comments
Assignees
Labels
A-io-cloud Area: reading/writing to cloud storage A-io-delta Area: reading/writing Delta tables enhancement New feature or an improvement of an existing feature python Related to Python Polars

Comments

@drumtorben
Copy link

With the latest Polars version 1.20.0, it should be possible to read tables from Azure directly using DefaultAzureCredential under the hood.

However, I encountered an issue when trying to use pl.read/scan_delta(). These functions seem to still require me to explicitly provide storage_options={"bearer_token": "<token>"} for authentication.

Here are the details of my setup:

  • Storage format: The path starts with "abfs://...".
  • Question: Could the issue be related to this specific storage format? Or is DefaultAzureCredential support not yet implemented for this type of storage?

Steps to Reproduce:

  1. Attempt to use pl.read/scan_delta() on an Azure storage account using a path that starts with "abfs://...".
  2. Ensure DefaultAzureCredential is configured and available in the environment.
  3. Observe whether it works without providing storage_options manually.

Expected Behavior:
DefaultAzureCredential should handle authentication seamlessly, as described for other storage types in the documentation.

Observed Behavior:
Authentication fails unless storage_options are explicitly provided with a bearer token.

Environment Details:

  • Polars Version: 1.20.0
  • Azure Storage Type: abfs://

Additional Context:
If this feature isn’t supported yet for "abfs://...", it would be great to have clarity on whether it’s planned or if there are workarounds.

@nameexhaustion nameexhaustion added python Related to Python Polars enhancement New feature or an improvement of an existing feature A-io-cloud Area: reading/writing to cloud storage A-io-delta Area: reading/writing Delta tables labels Jan 21, 2025
@nameexhaustion nameexhaustion self-assigned this Jan 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-io-cloud Area: reading/writing to cloud storage A-io-delta Area: reading/writing Delta tables enhancement New feature or an improvement of an existing feature python Related to Python Polars
Projects
None yet
Development

No branches or pull requests

2 participants