Native ABFS scans fail every task instead of falling back when Hadoop declines the auth type #7202

Describe the bug

Since #6059, the native Parquet scan resolves ABFS authentication through Hadoop on the driver. When the driver declines an auth type it does not model, the scan still plans natively, and every task then fails with the native "unsupported" auth error. The scan does not fall back to Spark's own reader.

Hadoop declines these, so the task fails:

• fs.azure.account.auth.type=Custom
• UserboundSASWithOAuth
• a custom fs.azure.sas.token.provider.type
• OAuth with RefreshTokenBasedTokenProvider or UserPasswordTokenProvider
• the hadoop-azure 3.5.0 Workload Identity client-assertion provider

1.1.0 ran these scans natively. It ignored fs.azure.account.auth.type and the provider class, and used the environment instead: AKS Workload Identity AZURE_* variables, or the IMDS managed identity. So 1.1.0 worked whenever the ambient identity could read the container, and Spark works through hadoop-azure. With 1.2.0, those jobs fail.

The flow is:

1. AbfsAuthResolver.resolve returns Declined.
2. NativeConfig.extractScanObjectStoreOptions still returns Right for a single-container scan.
3. CometNativeScan stays native.
4. builder_for in native/core/src/parquet/objectstore/azure.rs raises unsupported for the declined marker. It does the same in resolved_settings for RefreshToken and UserPassword.

There is a second case. When the driver cannot reflect into hadoop-azure, every task fails as well, through the Failed outcome and the error marker. That happens with a hadoop-azure whose AbfsConfiguration constructor or getTokenProviderClass signature differs from 3.3.4 to 3.5.0 (for example a vendor fork), or with a conf class loader that cannot see hadoop-azure. The handles are resolved before the Unconfigured check, so it also hits users who configure Azure only through environment variables and set no fs.azure.* keys. On 1.1.0 those users read natively from the environment.

datasources.md documents the failure ("the scan fails ... keep the path on Spark's own scan"). Not authenticating as an identity Spark would not use is the right goal. But a decline that is known at planning time could fall back to Spark instead of failing the job.

Steps to reproduce

Found by reading the code; not run end to end.

1. Run on AKS with Workload Identity environment variables that can read container c.
2. Set spark.hadoop.fs.azure.account.auth.type=Custom and spark.hadoop.fs.azure.account.oauth.provider.type=<a custom AccessTokenProvider>.
3. Run spark.read.parquet("abfss://c@acct.dfs.core.windows.net/t").count().

Expected behavior

The scan falls back to Spark whenever AbfsAuthResolver declines or fails on the driver, as it already does for multi-account scans and for ABFS mixed with other schemes. Spark returns the count through the custom provider, as it did on 1.1.0, where Comet returned it natively through the ambient identity.

Additional context

This is a regression since 1.1.0, found in the branch-1.2 audit. #6605 covers other differences between Hadoop and native ABFS handling.