[VL] Should convert kSpillReadBufferSize and kShuffleSpillDiskWriteBufferSize to number

[boneanxs]

What changes were proposed in this pull request?

Fix the issue introduced by #8045, we could meet the error if manually set spark.unsafe.sorter.spill.reader.buffer.size to value like 2m

org.apache.gluten.exception.GlutenException: Non-whitespace character found after end of conversion: "m"
	at org.apache.gluten.vectorized.PlanEvaluatorJniWrapper.nativeCreateKernelWithIterator(Native Method)
	at org.apache.gluten.vectorized.NativePlanEvaluator.createKernelWithBatchIterator(NativePlanEvaluator.java:68)
	at org.apache.gluten.backendsapi.velox.VeloxIteratorApi.genFirstStageIterator(VeloxIteratorApi.scala:204)
	at org.apache.gluten.execution.GlutenWholeStageColumnarRDD.$anonfun$compute$1(GlutenWholeStageColumnarRDD.scala:88)
	at org.apache.gluten.utils.Arm$.withResource(Arm.scala:25)
	at org.apache.gluten.metrics.GlutenTimeMetric$.millis(GlutenTimeMetric.scala:37)
	at org.apache.gluten.execution.GlutenWholeStageColumnarRDD.compute(GlutenWholeStageColumnarRDD.scala:77)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:380)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:344)
	at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:106)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:380)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:344)

We should parse value to number in bytes before put to nativeConf

(Fixes: #ISSUE-ID)

How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

• Other pull requests

[github-actions]

Run Gluten Clickhouse CI on x86

[boneanxs]

@jinchengchenghh @FelixYBW Hey, could you please help review this? Thanks!

[Yohahaha]

incubator-gluten/shims/common/src/main/scala/org/apache/spark/sql/internal/GlutenConfigUtil.scala

Lines 21 to 45 in e009208

	object GlutenConfigUtil {
	private def getConfString(configProvider: ConfigProvider, key: String, value: String): String = {
	Option(ConfigEntry.findEntry(key))
	.map {
	_.readFrom(configProvider) match {
	case o: Option[_] => o.map(_.toString).getOrElse(value)
	case null => value
	case v => v.toString
	}
	}
	.getOrElse(value)
	}
	def parseConfig(conf: Map[String, String]): Map[String, String] = {
	val provider = new MapProvider(conf.filter(_._1.startsWith("spark.gluten.")))
	conf.map {
	case (k, v) =>
	if (k.startsWith("spark.gluten.")) {
	(k, getConfString(provider, k, v))
	} else {
	(k, v)
	}
	}.toMap
	}
	}

spark.unsafe.sorter.spill.reader.buffer.size=2m should be converted by above codes, could you investigate why it not works?

[Yohahaha]

incubator-gluten/shims/common/src/main/scala/org/apache/spark/sql/internal/GlutenConfigUtil.scala

Lines 21 to 45 in e009208

	object GlutenConfigUtil {
	private def getConfString(configProvider: ConfigProvider, key: String, value: String): String = {
	Option(ConfigEntry.findEntry(key))
	.map {
	_.readFrom(configProvider) match {
	case o: Option[_] => o.map(_.toString).getOrElse(value)
	case null => value
	case v => v.toString
	}
	}
	.getOrElse(value)
	}
	def parseConfig(conf: Map[String, String]): Map[String, String] = {
	val provider = new MapProvider(conf.filter(_._1.startsWith("spark.gluten.")))
	conf.map {
	case (k, v) =>
	if (k.startsWith("spark.gluten.")) {
	(k, getConfString(provider, k, v))
	} else {
	(k, v)
	}
	}.toMap
	}
	}

spark.unsafe.sorter.spill.reader.buffer.size=2m should be converted by above codes, could you investigate why it not works?

oh, GlutenConfigUtil only process the config which prefix is 'spark.gluten'.

[boneanxs]

incubator-gluten/shims/common/src/main/scala/org/apache/spark/sql/internal/GlutenConfigUtil.scala

Lines 21 to 45 in e009208

	object GlutenConfigUtil {
	private def getConfString(configProvider: ConfigProvider, key: String, value: String): String = {
	Option(ConfigEntry.findEntry(key))
	.map {
	_.readFrom(configProvider) match {
	case o: Option[_] => o.map(_.toString).getOrElse(value)
	case null => value
	case v => v.toString
	}
	}
	.getOrElse(value)
	}
	def parseConfig(conf: Map[String, String]): Map[String, String] = {
	val provider = new MapProvider(conf.filter(_._1.startsWith("spark.gluten.")))
	conf.map {
	case (k, v) =>
	if (k.startsWith("spark.gluten.")) {
	(k, getConfString(provider, k, v))
	} else {
	(k, v)
	}
	}.toMap
	}
	}

spark.unsafe.sorter.spill.reader.buffer.size=2m should be converted by above codes, could you investigate why it not works?

oh, GlutenConfigUtil only process the config which prefix is 'spark.gluten'.

Do we need to extend this function to allow all configures?

[Yohahaha]

Do we need to extend this function to allow all configures?

yeah, we may need add new method GlutenConfigUtil#get(ConfigEntry) to process non-sql configs.

[boneanxs]

Do we need to extend this function to allow all configures?

yeah, we may need add new method GlutenConfigUtil#get(ConfigEntry) to process non-sql configs.

Hey @Yohahaha We cannot use ConfigEntry since they are private in Spark package only

https://github.com/apache/spark/blob/cea79dc1918b7f03870fe1cb189da9a152e3bbaf/core/src/main/scala/org/apache/spark/internal/config/package.scala#L1892-L1899

How about only extract a specific method that handle bytes value only to reduce duplicates?

[Yohahaha]

Do we need to extend this function to allow all configures?

yeah, we may need add new method GlutenConfigUtil#get(ConfigEntry) to process non-sql configs.

Hey @Yohahaha We cannot use ConfigEntry since they are private in Spark package only

https://github.com/apache/spark/blob/cea79dc1918b7f03870fe1cb189da9a152e3bbaf/core/src/main/scala/org/apache/spark/internal/config/package.scala#L1892-L1899

How about only extract a specific method that handle bytes value only to reduce duplicates?

sounds good to me.

[github-actions]

Run Gluten Clickhouse CI on x86