Enable rocm and vulkan build in CI workflow for PJRT plugin #19279
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: PragmaTwice <[email protected]>
Signed-off-by: PragmaTwice <[email protected]>
PragmaTwice
force-pushed
the
pjrt-ci-rocm
branch
from
c30acce
to
02c6694
Compare
Signed-off-by: PragmaTwice <[email protected]>
PragmaTwice
force-pushed
the
pjrt-ci-rocm
branch
from
02c6694
to
bf355b0
Compare
Signed-off-by: PragmaTwice <[email protected]>
Signed-off-by: PragmaTwice <[email protected]>
Signed-off-by: PragmaTwice <[email protected]>
Signed-off-by: PragmaTwice <[email protected]>
Signed-off-by: PragmaTwice <[email protected]>
PragmaTwice
force-pushed
the
pjrt-ci-rocm
branch
5 times, most recently
from
311bd17
to
f39168f
Compare
Signed-off-by: PragmaTwice <[email protected]>
PragmaTwice
force-pushed
the
pjrt-ci-rocm
branch
from
f39168f
to
f0258ad
Compare
ScottTodd
reviewed
Nov 25, 2024
Comment on lines
-16
to
+17
| iree::experimental::rocm | ||
| iree::experimental::rocm::registration | ||
| iree::hal::drivers::hip | ||
| iree::hal::drivers::hip::registration |
There was a problem hiding this comment.
Oof sorry that this code has bit-rotted so much, and thanks for finding what needed to be updated.
Signed-off-by: PragmaTwice <[email protected]>
Hmm the error in rocm plugin comes with a hint ( I'll investigate these errors in these days. |
ScottTodd
reviewed
Nov 26, 2024
Comment on lines
+48
to
+65
| // TODO: here we just use the target name of the first available device, | ||
| // but ideally we should find the device which will run the program | ||
| if (device_info_count > 0) { | ||
| hipDeviceProp_tR0000 props; | ||
| IREE_RETURN_IF_ERROR(iree_hal_hip_get_device_properties( | ||
| *out_driver, device_infos->device_id, &props)); | ||
|
|
||
| // `gcnArchName` comes back like gfx90a:sramecc+:xnack- for a fully | ||
| // specified target. However the IREE target-chip flag only expects the | ||
| // prefix. refer to | ||
| // https://github.com/iree-org/iree-turbine/blob/965247e/iree/turbine/runtime/device.py#L495 | ||
| std::string_view target = props.gcnArchName; | ||
| if (auto pos = target.find(':'); pos != target.npos) { | ||
| target = target.substr(0, pos); | ||
| } | ||
|
|
||
| hip_target_ = target; | ||
| logger().debug("HIP target detected: " + hip_target_); |
There was a problem hiding this comment.
+ python test/test_add.py WARNING:jax._src.xla_bridge:Platform 'iree_rocm' is experimental and not all JAX functionality may be correctly supported! [IREE-PJRT] DEBUG: Using IREE compiler binary: /home/esaimana/actions-runner/_work/iree/iree/.venv/lib/python3.11/site-packages/iree/compiler/_mlir_libs/libIREECompiler.so [IREE-PJRT] DEBUG: Compiler Version: 3.1.0.dev+da45650503b955e[35](https://github.com/iree-org/iree/actions/runs/12031928938/job/33543044454?pr=19279#step:7:36)a45ea8e34b5f10b30a2f912 @ da45650503b955e35a45ea8e34b5f10b30a2f912 (API version 1.4) [IREE-PJRT] DEBUG: Partitioner was not enabled. The partitioner can be enabled by setting the 'PARTITIONER_LIB_PATH' config var ('IREE_PJRT_PARTITIONER_LIB_PATH' env var) [IREE-PJRT] DEBUG: ROCM driver created [IREE-PJRT] DEBUG: HIP target detected: gfx1100 iree/runtime/src/iree/hal/drivers/hip/native_executable.c:306: UNKNOWN; HIP driver error 'hipErrorFileNotFound' (301): file not found; mismatched target chip? missing/wrong bitcode directory?; while invoking native function hal.executable.create; while calling import; [ 0] bytecode jit__add.__init:1284 "jit(_add)/jit(main)/add"("<module>"(/home/esaimana/actions-runner/_work/iree/iree/integrations/pjrt/test/test_add.py:9:6)) build_tools/testing/run_jax_tests.sh: line 31: 888283 Aborted (core dumped) JAX_PLATFORMS=$actual_jax_platform python $test_py_file > $actual_tmp_outHmm the error in rocm plugin comes with a hint (
mismatched target chip? missing/wrong bitcode directory?) but it seems weird that I've already add some new logic to detect the HIP target and pass to IREE (via--iree-hip-target) before the compilation phase.I'll investigate these errors in these days.
For test workflows, we typically set an environment variable explicitly, matched with the hardware installed into the runner chosen using the runs-on property:
In PJRT, matching the code from iree-turbine that detects available devices makes sense to me, instead of that environment variable approach. This is a JIT scenario, where the compiler is being used to generate code for the current device, not an arbitrary deployment target.
Can you have the PJRT client log the compile command used, including all flags?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR trys to enable rocm and vulkan PJRT plugin in CI workflow (via the runner
nodai-amdgpu-w7900-x86-64, mentioned here: #19222 (comment)).Currently it's marked as a draft PR for potentially CI debuging.
ci-exactly: build_packages, test_pjrt