providing extra jar file using --jars is not working for pyspark jobs

[shaleena]

Trying to pass a jar dependency file with spark submit as mentioned in the examples for pyspark jar is not working

"sparkSubmitParameters": "--jars=s3://<data-integration-artifacts>/spark-credentials-provider/idealo-spark-credentials-provider-1.3.0.jar",

Also noticed that the jar doesn't end up on "spark.driver.extraClassPath" or "spark.executor.extraClassPath" even if it's downloaded from s3 into tmp folder

Files s3://<data-integration-artifacts>/spark-credentials-provider/idealo-spark-credentials-provider-1.3.0.jar from /tmp/spark-112c49ee-7811-43bf-82ee-587a2d188f19/idealo-spark-credentials-provider-1.3.0.jar to /home/hadoop/./idealo-spark-credentials-provider-1.3.0.jar
Tried the above with both EMR version 7.0.0 and 6.14.0

copying the jar to /usr/share/aws/emr/emrfs/auxlib/ with a Docker build worked.

Is this a known issue and is there any solution to fix this without using Docker ?