A Python Stack Trace Error Occurs if the cdk-assets NPM Package is out of date

[X-Guardian]

Subject of the issue

If the installed cdk-assets NPM package is out of date, the CDK handler errors with a full Python stack trace.

Your environment

• sceptre-cdk-handler v2.0.0

Expected Behaviour

The handler should return a managed error like:

[2023-03-03 17:09:21] - dev/cdk - Launching Stack
[2023-03-03 17:09:22] - dev/cdk - Stack is in the PENDING state
[2023-03-03 17:09:22] - dev/cdk - Creating Stack
sceptre-example@ C:\Users\Simon\Documents\GitHub\Sceptre\sceptre-cdk-handler\sceptre-example
└── (empty)

C:\Program Files\nodejs -> .\
└── [email protected]

[2023-03-03 17:09:50] - Publishing CDK assets
[2023-03-03 17:09:50] - Error: Canot read asset manifest 'C:\Users\Simon\AppData\Local\Temp\cdk.outNA6wTy\CDKStack.assets.json': Cloud assembly schema version mismatch: Maximum schema version supported is 22.0.0, but found 30.1.0. The cdk-assets package is probably out of date.

Actual behaviour

[2023-03-03 17:09:21] - dev/cdk - Launching Stack
[2023-03-03 17:09:22] - dev/cdk - Stack is in the PENDING state
[2023-03-03 17:09:22] - dev/cdk - Creating Stack
sceptre-example@ C:\Users\Simon\Documents\GitHub\Sceptre\sceptre-cdk-handler\sceptre-example
└── (empty)

C:\Program Files\nodejs -> .\
└── [email protected]

[2023-03-03 17:09:50] - Publishing CDK assets
Error: Canot read asset manifest 'C:\Users\Simon\AppData\Local\Temp\cdk.outNA6wTy\CDKStack.assets.json': Cloud assembly schema version mismatch: Maximum schema version supported is 22.0.0, but found 30.1.0
    at AssetManifest.fromFile (C:\Users\Simon\AppData\Roaming\nvm\v18.12.1\node_modules\cdk-assets\lib\asset-manifest.js:24:19)
    at AssetManifest.fromPath (C:\Users\Simon\AppData\Roaming\nvm\v18.12.1\node_modules\cdk-assets\lib\asset-manifest.js:43:30)
    at Object.publish (C:\Users\Simon\AppData\Roaming\nvm\v18.12.1\node_modules\cdk-assets\bin\publish.js:7:40)
    at C:\Users\Simon\AppData\Roaming\nvm\v18.12.1\node_modules\cdk-assets\bin\cdk-assets.js:32:25
    at Object.handler (C:\Users\Simon\AppData\Roaming\nvm\v18.12.1\node_modules\cdk-assets\bin\cdk-assets.js:56:15)
    at Object.runCommand (C:\Users\Simon\AppData\Roaming\nvm\v18.12.1\node_modules\cdk-assets\node_modules\yargs\build\index.cjs:446:48)
    at Object.parseArgs [as _parseArgs] (C:\Users\Simon\AppData\Roaming\nvm\v18.12.1\node_modules\cdk-assets\node_modules\yargs\build\index.cjs:2697:57)
    at Object.get [as argv] (C:\Users\Simon\AppData\Roaming\nvm\v18.12.1\node_modules\cdk-assets\node_modules\yargs\build\index.cjs:2651:25)
    at main (C:\Users\Simon\AppData\Roaming\nvm\v18.12.1\node_modules\cdk-assets\bin\cdk-assets.js:43:9)
    at Object.<anonymous> (C:\Users\Simon\AppData\Roaming\nvm\v18.12.1\node_modules\cdk-assets\bin\cdk-assets.js:59:1)
Traceback (most recent call last):
  File "C:\Users\Simon\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\Simon\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\Simon\Documents\GitHub\Sceptre\sceptre-cdk-handler\.venv\Scripts\sceptre.exe\__main__.py", line 7, in <module>
    sys.exit(cli())
  File "c:\Users\Simon\Documents\GitHub\Sceptre\sceptre-cdk-handler\.venv\lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "c:\Users\Simon\Documents\GitHub\Sceptre\sceptre-cdk-handler\.venv\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "c:\Users\Simon\Documents\GitHub\Sceptre\sceptre-cdk-handler\.venv\lib\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\Users\Simon\Documents\GitHub\Sceptre\sceptre-cdk-handler\.venv\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\Users\Simon\Documents\GitHub\Sceptre\sceptre-cdk-handler\.venv\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "c:\Users\Simon\Documents\GitHub\Sceptre\sceptre-cdk-handler\.venv\lib\site-packages\click\decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\Users\Simon\Documents\GitHub\Sceptre\sceptre-cdk-handler\.venv\lib\site-packages\sceptre\cli\helpers.py", line 43, in decorated
    return func(*args, **kwargs)
  File "c:\Users\Simon\Documents\GitHub\Sceptre\sceptre-cdk-handler\.venv\lib\site-packages\sceptre\cli\launch.py", line 62, in launch_command
    exit_code = launcher.launch(prune)
  File "c:\Users\Simon\Documents\GitHub\Sceptre\sceptre-cdk-handler\.venv\lib\site-packages\sceptre\cli\launch.py", line 105, in launch
    code = code or self._deploy(deploy_plan)
  File "c:\Users\Simon\Documents\GitHub\Sceptre\sceptre-cdk-handler\.venv\lib\site-packages\sceptre\cli\launch.py", line 197, in _deploy
    result = deploy_plan.launch()
  File "c:\Users\Simon\Documents\GitHub\Sceptre\sceptre-cdk-handler\.venv\lib\site-packages\sceptre\plan\plan.py", line 170, in launch
    return self._execute(*args)
  File "c:\Users\Simon\Documents\GitHub\Sceptre\sceptre-cdk-handler\.venv\lib\site-packages\sceptre\plan\plan.py", line 29, in wrapped
    return func(self, *args, **kwargs)
  File "c:\Users\Simon\Documents\GitHub\Sceptre\sceptre-cdk-handler\.venv\lib\site-packages\sceptre\plan\plan.py", line 55, in _execute
    return executor.execute(*args)
  File "c:\Users\Simon\Documents\GitHub\Sceptre\sceptre-cdk-handler\.venv\lib\site-packages\sceptre\plan\executor.py", line 52, in execute
    stack, status = future.result()
  File "C:\Users\Simon\AppData\Local\Programs\Python\Python39\lib\concurrent\futures\_base.py", line 439, in result
    return self.__get_result()
  File "C:\Users\Simon\AppData\Local\Programs\Python\Python39\lib\concurrent\futures\_base.py", line 391, in __get_result
    raise self._exception
  File "C:\Users\Simon\AppData\Local\Programs\Python\Python39\lib\concurrent\futures\thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "c:\Users\Simon\Documents\GitHub\Sceptre\sceptre-cdk-handler\.venv\lib\site-packages\sceptre\plan\executor.py", line 59, in _execute
    result = getattr(actions, self.command)(*args)
  File "c:\Users\Simon\Documents\GitHub\Sceptre\sceptre-cdk-handler\.venv\lib\site-packages\sceptre\hooks\__init__.py", line 107, in decorated
    response = func(self, *args, **kwargs)
  File "c:\Users\Simon\Documents\GitHub\Sceptre\sceptre-cdk-handler\.venv\lib\site-packages\sceptre\plan\actions.py", line 213, in launch
    status = self.create()
    response = func(self, *args, **kwargs)
  File "c:\Users\Simon\Documents\GitHub\Sceptre\sceptre-cdk-handler\.venv\lib\site-packages\sceptre\plan\actions.py", line 91, in create
    create_stack_kwargs.update(self.stack.template.get_boto_call_parameter())
  File "c:\Users\Simon\Documents\GitHub\Sceptre\sceptre-cdk-handler\.venv\lib\site-packages\sceptre\template.py", line 224, in get_boto_call_parameter
    return {"TemplateBody": self.body}
  File "c:\Users\Simon\Documents\GitHub\Sceptre\sceptre-cdk-handler\.venv\lib\site-packages\sceptre\template.py", line 97, in body
    body = handler.handle()
  File "c:\Users\Simon\Documents\GitHub\Sceptre\sceptre-cdk-handler\.venv\lib\site-packages\sceptre_cdk_handler\cdk.py", line 230, in handle
    template_dict = builder.build_template(context, self.sceptre_user_data)
  File "c:\Users\Simon\Documents\GitHub\Sceptre\sceptre-cdk-handler\.venv\lib\site-packages\sceptre_cdk_handler\cdk_builder.py", line 141, in build_template
    self._publish_artifacts(manifest_artifact.file, environment_variables)
  File "c:\Users\Simon\Documents\GitHub\Sceptre\sceptre-cdk-handler\.venv\lib\site-packages\sceptre_cdk_handler\cdk_builder.py", line 56, in _publish_artifacts
    self._run_command(
  File "c:\Users\Simon\Documents\GitHub\Sceptre\sceptre-cdk-handler\.venv\lib\site-packages\sceptre_cdk_handler\cdk_builder.py", line 64, in _run_command
    result = self._subprocess_run(
  File "C:\Users\Simon\AppData\Local\Programs\Python\Python39\lib\subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'npx cdk-assets -v publish --path C:\Users\Simon\AppData\Local\Temp\cdk.outNA6wTy\CDKStack.assets.json' returned non-zero exit status 1.

[jfalkenstein]

The challenge with providing the sort of error messaging you're requesting here is that the originating error message has been raised in Node resulting in a nonzero exit code. As far as the Python code knows, its subprocess has returned a nonzero. That's about it.

The only way to be more specific would be to read the output of the node stacktrace, parse it (somehow) or search for certain strings in it, and then reraise an error. I don't think it's appropriate for this handler (and definitely isn't maintainable) for us to keep trying to check for all the possible errors that Node/CDK/CDK-assets might raise and reraise specific errors accordingly.

As to wanting to bury the stacktraces, I also don't think that's a good idea. Since we're invoking a subprocess here, there's a number of issues that might take place and, without that stacktrace, it'd be very difficult to diagnose what's going on here. With this stacktrace info, you can see the EXACT command that's failing and even retry it directly for triage. Furthermore, with the Node stacktrace that's pushed out above it, you can even see what was going on within Node. All of that is invaluable information for diagnosing any number of issues that might arise when using this handler.

[jfalkenstein]

Actually, I thought a bit more about what you're saying here. The only part of the Python stacktrace that's actually useful is the message, not the traceback itself. I've updated this in my PR #10 in this commit: 00efe56. We won't hide the JS stacktrace (because that's actually the really helpful bit), but it won't include the Python stacktrace.

If Sceptre is run with the --debug flag, though, it should still show the full stacktrace, which would be helpful.