You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
rm: cannot remove 'PyGlue.zip': No such file or directory
**/home/my_user/aws-glue-libs/bin/glue-setup.sh: line 14: zip: command not found**
[INFO] Scanning for projects...
[INFO]
[INFO] -------------------< com.amazonaws:AWSGlueETLPython >-------------------
[INFO] Building AWSGlueETLPython 4.0.0
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.080 s
[INFO] Finished at: 2023-06-20T16:02:27Z
[INFO] ------------------------------------------------------------------------
mkdir: cannot create directory ‘/home/my_user/aws-glue-libs/conf’: File exists
/home/my_user/aws-glue-libs/bin
Jun 20, 2023 4:02:28 PM org.apache.spark.launcher.Log4jHotPatchOption staticJavaAgentOption
WARNING: spark.log4jHotPatch.enabled is set to true, but /usr/share/log4j-cve-2021-44228-hotpatch/jdk17/Log4jHotPatchFat.jar does not exist at the configured location
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/my_user/install_files/spark/jars/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/my_user/install_files/spark/jars/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/my_user/aws-glue-libs/jarsv1/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/my_user/aws-glue-libs/jarsv1/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
log4j:WARN No appenders could be found for logger (org.apache.spark.util.Utils).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Traceback (most recent call last):
**File "/home/my_user/aws-glue-libs/glue_script.py", line 9, in <module>
sc = SparkContext()
File "/home/my_user/install_files/spark/python/lib/pyspark.zip/pyspark/context.py", line 197, in __init__
File "/home/my_user/install_files/spark/python/lib/pyspark.zip/pyspark/context.py", line 282, in _do_init
File "/home/my_user/install_files/spark/python/lib/pyspark.zip/pyspark/context.py", line 410, in _initialize_context
File "/home/my_user/install_files/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1585, in __call__
File "/home/my_user/install_files/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.io.FileNotFoundException: File file:/home/my_user/aws-glue-libs/PyGlue.zip does not exist**
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:784)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:1105)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:774)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:472)
at org.apache.spark.SparkContext.addFile(SparkContext.scala:1621)
at org.apache.spark.SparkContext.$anonfun$new$13(SparkContext.scala:515)
at org.apache.spark.SparkContext.$anonfun$new$13$adapted(SparkContext.scala:515)
at scala.collection.immutable.List.foreach(List.scala:431)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:515)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.base/java.lang.Thread.run(Thread.java:829)
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
I'm following provided documents for Python local development:
https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-libraries.html#develop-using-etl-library
https://github.com/awslabs/aws-glue-libs/tree/master
Getting below error when runing my script
Beta Was this translation helpful? Give feedback.
All reactions