Replies: 1 comment
-
So finally figured out that the JOB_NAME needs to reference an existing glue job. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I am using the code fragment below and the job.init call is failing with an error while getting security configuration. (see log below). I'm not suprised by the error since I can't find documentation as to the args that need to be supplied. I did find https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html but the essential parameters are missing: (IAM_ROLE, connections, etc.) . When I look at the AWS Glue Studio job editor there is a lot of Job context established in the "Job Details" tab. I expect that I need to pass similar information to job.init.
I'm trying to do local debugging of glue jobs as suggested by https://aws.amazon.com/blogs/big-data/developing-aws-glue-etl-jobs-locally-using-a-container/
Any help would be appreciated.
Code:
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
LOG:
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Traceback (most recent call last):
File "/home/glue_user/spark/python/pyspark/sql/utils.py", line 111, in deco
return f(*a, **kw)
File "/home/glue_user/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling z:com.amazonaws.services.glue.util.Job.init.
: java.lang.RuntimeException: error while getting security configuration for None
at com.amazonaws.services.glue.util.SecurityConfig$.get(SecurityConfig.scala:29)
at com.amazonaws.services.glue.util.Job$.init(Job.scala:67)
at com.amazonaws.services.glue.util.Job.init(Job.scala)
Beta Was this translation helpful? Give feedback.
All reactions