Does spark-operator support volumeDevices?

[zzylydx]

What question do you want to ask?

Does spark-operator support volumeDevices?

Additional context

No response

Have the same question?

Give it a 👍 We prioritize the question with most 👍

[ChenYi015]

@zzvara Yes, see examples here:

• examples/spark-pi-configmap.yaml
• examples/spark-pi-pod-template.yaml

[zzylydx]

@zzvara Yes, see examples here:

• examples/spark-pi-configmap.yaml
• examples/spark-pi-pod-template.yaml

In your example I only see volumeMounts, but no volumeDevices. Does it really support mounting block devices?

[ChenYi015]

@zzylydx You can use the pod template feature to customize volumeDevices the same way as volumeMounts.

[zzylydx]

@zzylydx You can use the pod template feature to customize volumeDevices the same way as volumeMounts.

VolumeDevices only supports ReadWriteOnly mode, so multiple executors cannot share a PVC. You need to specify the storageClass to let the spark driver dynamically create a PV and then bind it to each executor. Is this scenario supported?

In spark-submit, there is a configuration similar to the following, but spark-submit does not support mount block device.

${SPARK_PATH}/bin/spark-submit \
    --master k8s://https://$K8S_URL \
    --deploy-mode cluster \
    --name zzy-pyspark \
    --conf spark.driver.cores=8 \
    --conf spark.executor.cores=8 \
    --conf spark.cores.max=16 \
    --conf spark.executor.instances=7 \
    --conf spark.kubernetes.container.image=$CONTAINER_IMAGE \
    --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
    --conf spark.kubernetes.executor.deleteOnTermination=false \
    --conf spark.kubernetes.driver.podTemplateFile=./driver-processed.yaml \
    --conf spark.kubernetes.executor.podTemplateFile=./executor.yaml \
    --conf spark.ssl.enabled=false \
    --conf spark.executor.memory=16g \
    --conf spark.driver.memory=8g \
    --conf spark.file.transferTo=false \
    --conf spark.kubernetes.driver.volumes.persistentVolumeClaim.spark-local-dir-1.options.claimName=OnDemand \
    --conf spark.kubernetes.driver.volumes.persistentVolumeClaim.spark-local-dir-1.options.storageClass=nfs-client \
    --conf spark.kubernetes.driver.volumes.persistentVolumeClaim.spark-local-dir-1.options.sizeLimit=20Gi \
    --conf spark.kubernetes.driver.volumes.persistentVolumeClaim.spark-local-dir-1.mount.path=/data \
    --conf spark.kubernetes.driver.volumes.persistentVolumeClaim.spark-local-dir-1.mount.readOnly=false \
    --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.claimName=OnDemand \
    --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.storageClass=nfs-client \
    --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.sizeLimit=20Gi \
    --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.mount.path=/data \
    --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.mount.readOnly=false \
    --verbose \
    local:/opt/spark/examples/src/main/python/transitive_closure.py