[rizzabhdwaj]

Which issue does this PR close?

issue - #27342

Rationale for this change

Spark-specific configuration was stored in DataFusion core even though it is only consumed by Spark-compatible functions. This caused Spark options to appear in every DataFusion session and required new Spark options to wait for a DataFusion release.

What changes are included in this PR?

• Removed SparkOptions and MapKeyDedupPolicy from DataFusion core.
• Added them to the Spark crate as a ConfigExtension using the spark namespace.
• Registered the Spark configuration extension when Spark features are enabled.
• Updated Spark map functions to read SparkOptions from configuration extensions.
• Changed the map deduplication configuration key from datafusion.spark.map_key_dedup_policy to spark.map_key_dedup_policy.
• Preserved the existing EXCEPTION and LAST_WIN duplicate-key behavior.
• Updated SQL logic tests, configuration documentation, and upgrade documentation.
• Left enable_ansi_mode unchanged because its ownership between DataFusion core and Spark remains unresolved.

What is the testing strategy for this PR?

• Verified the Spark crate with:

  cargo check -p datafusion-spark

• Ran formatting and diff checks:
  cargo fmt --all -- --check
  git diff --check
• Updated Spark map SQL logic tests to cover the new spark.map_key_dedup_policy key and existing duplicate-key behavior.
  Are there any user-facing changes?
  Yes. The map deduplication option is no longer a built-in DataFusion option.
  The old key:
  datafusion.spark.map_key_dedup_policy
  is replaced by:
  spark.map_key_dedup_policy
  The new key is available when Spark features are enabled. The default behavior remains EXCEPTION, and LAST_WIN remains available for preserving the previous permissive behavior.

[andygrove]

Thanks for working on this, @rizzabhdwaj. #27337 removed datafusion/spark from this repository, so the crate's half of this PR now belongs in apache/datafusion-spark. Would you be willing to open it there?

That would be step 1 of #27342: add SparkOptions as a ConfigExtension in datafusion-spark, register it in with_spark_features(), and have the map functions read it from config_options.extensions, falling back to the defaults when the extension isn't registered. apache/datafusion-spark builds against branch-56, where datafusion.spark.map_key_dedup_policy still exists, so the crate can't remove or rename anything in core yet. The removal from datafusion-common, information_schema.slt and the configs docs can stay in this PR for DataFusion 57, once the crate no longer reads the old key.

There is some urgency: I'd like to add more options to that extension, including Spark-version behavior for the CAST port (apache/datafusion-spark#88). #27260 and #27273 are also waiting on it to add spark.sql.legacy.* flags for encode. @AnuragRaut08 offered to help on #27342 too, so please coordinate there if you'd rather hand it off.

[rizzabhdwaj]

Yeah, absolutely . I'll open a separate PR in apache/datafusion-spark shortly for the changes that belong there.

I've also reached out to @AnuragRaut08 about collaborating on this. Hopefully, we can work together and get this issue resolved soon. Thanks for the guidance!

[AnuragRaut08]

@rizzabhdwaj Can you resolve the conflicts? Rebase if needed.

[rizzabhdwaj]

I've resolved all the conflicts.