Driver executor disk space fills up with temp files

[warwickhadley]

We have been successful in using this library with smaller parameter spaces however as the parameter space increases we are running out of space on the driver executor. It appears that temp files are created in the container for each job and are never removed.e.g. the folder /mnt/yarn/usercache/hadoop/appcache/application_1624531736036_0001/spark-9d614b58-4959-4b9e-a0ef-da525c803263/pyspark-6e9409d1-d631-41f0-a2b4-c3d9060c23d9 just fills up even if we increase the size of the volume mounted, it will eventually fill up and take down the driver.

I am not sure if this is a problem with joblib, joblik-spark or the spark configuration of our cluster. If anyone has any suggestions I would be grateful.

[fritshermans]

is there already a fix for this problem? it takes down the driver for me as well

[arezki1990]

i have the same issue here on databricks any fix