Partitioned writes with multiple columns creates wrong directory structure if child output columns is not in same order

[ayushi-agarwal]

Backend

VL (Velox)

Bug description

val data = Seq(
      ("b1", 1, "a1"),
      ("b2", 2, "a2"),
      ("b1", 3, "a1"),
      ("b2", 4, "a2")
    )

    // Define schema explicitly
    val schema = StructType(Seq(
      StructField("b", StringType, nullable = false),
      StructField("c", IntegerType, nullable = false),
      StructField("a", StringType, nullable = false),
    ))

    val rdd = spark.sparkContext.parallelize(data).map {
      case (b, c, a) => Row(b, c, a)
    }

    val df = spark.createDataFrame(rdd, schema)

    // Write DataFrame as Parquet with partitioning
    df.write
      .format("parquet")
      .partitionBy("a", "b")  // Partition by columns a and b
      .mode("overwrite")      // Overwrite if output exists
      .save("file:///tmp/partitioned_output")  

This creates directory structure as
b=b1/a=a1
b=b2/a=a2

instead of
a=a1/b=b1
a=a2/b=b2

@zhouyuan @JkSelf @rui-mo

Spark version

None

Spark configurations

spark - 3.5.1

System information

No response

Relevant logs

[JkSelf]

@ayushi-agarwal @FelixYBW I will look into this issue after I return to work on February 10th.