[Bug] Build - Cross Compilation Failure #3188

jwinarske · 2022-10-18T14:41:24Z

Describe the bug

When cross compiling firecracker the secompiler-bin execution step fails.

To Reproduce

Cross compile using Yocto kirkstone build system using this recipe

SUMMARY = "Firecracker"
HOMEPAGE = "https://github.com/firecracker-microvm/firecracker"
ISSUES = "https://github.com/firecracker-microvm/firecracker/issues"
LICENSE = "Apache-2.0"

LIC_FILES_CHKSUM = "\
    file://${COREBASE}/meta/files/common-licenses/Apache-2.0;md5=89aea4e17d99a7cacdbeed46a0096b10 \
"

DEPENDS += "\
    compiler-rt \
    libcxx \
    seccompiler-native \
    "

S = "${WORKDIR}/git"

SRC_URI = "git://github.com/firecracker-microvm/firecracker.git;protocol=https;nobranch=1"
SRCREV = "df69983f6d57abd5df986fea11de2952a1beae2c"

COMPATIBLE_MACHINE = "(-)"
COMPATIBLE_MACHINE:aarch64 = "(.*)"
COMPATIBLE_MACHINE:x86_64 = "(.*)"

do_compile[network] = "1"

SRC_URI += " \
    git://github.com/firecracker-microvm/kvm-bindings;lfs=0;nobranch=1;protocol=https;destsuffix=git/kvm-bindings>
    git://github.com/firecracker-microvm/micro-http;lfs=0;nobranch=1;protocol=https;destsuffix=git/micro-http;nam>
    crate://crates.io/aead/0.4.3 \
    crate://crates.io/aes/0.7.5 \
    crate://crates.io/aes-gcm/0.9.4 \
    crate://crates.io/atty/0.2.14 \
    crate://crates.io/autocfg/1.0.1 \
    crate://crates.io/base64/0.13.0 \
    crate://crates.io/bincode/1.3.3 \
    crate://crates.io/bindgen/0.59.2 \
    crate://crates.io/bitflags/1.3.2 \
    crate://crates.io/bstr/0.2.17 \
    crate://crates.io/bumpalo/3.9.1 \
    crate://crates.io/byteorder/1.4.3 \
    crate://crates.io/cast/0.2.7 \
    crate://crates.io/cc/1.0.73 \
    crate://crates.io/cexpr/0.6.0 \
    crate://crates.io/cfg-if/0.1.10 \
    crate://crates.io/cfg-if/1.0.0 \
    crate://crates.io/cipher/0.3.0 \
    crate://crates.io/clang-sys/1.3.1 \
    crate://crates.io/clap/2.34.0 \
    crate://crates.io/cpufeatures/0.2.1 \
    crate://crates.io/crc64/1.0.0 \
    crate://crates.io/criterion/0.3.5 \
    crate://crates.io/criterion-plot/0.4.4 \
    crate://crates.io/crossbeam-channel/0.5.4 \
    crate://crates.io/crossbeam-deque/0.8.1 \
    crate://crates.io/crossbeam-epoch/0.9.7 \
    crate://crates.io/crossbeam-utils/0.8.7 \
    crate://crates.io/csv/1.1.6 \
    crate://crates.io/csv-core/0.1.10 \
    crate://crates.io/ctr/0.8.0 \
    crate://crates.io/derive_more/0.99.17 \
    crate://crates.io/device_tree/1.1.0 \
    crate://crates.io/either/1.6.1 \
    crate://crates.io/event-manager/0.2.1 \
    crate://crates.io/generic-array/0.14.5 \
    crate://crates.io/getrandom/0.2.4 \
    crate://crates.io/ghash/0.4.4 \
    crate://crates.io/glob/0.3.0 \
    crate://crates.io/half/1.8.2 \
    crate://crates.io/hermit-abi/0.1.19 \
    crate://crates.io/itertools/0.10.3 \
    crate://crates.io/itoa/0.4.8 \
    crate://crates.io/itoa/1.0.1 \
    crate://crates.io/js-sys/0.3.56 \
    crate://crates.io/kvm-ioctls/0.11.0 \
    crate://crates.io/lazy_static/1.4.0 \
    crate://crates.io/lazycell/1.3.0 \
    crate://crates.io/libc/0.2.117 \
    crate://crates.io/libloading/0.7.3 \
    crate://crates.io/linux-loader/0.4.0 \
    crate://crates.io/log/0.4.14 \
    crate://crates.io/memchr/2.4.1 \
    crate://crates.io/memoffset/0.6.5 \
    crate://crates.io/minimal-lexical/0.2.1 \
    crate://crates.io/nix/0.23.1 \
    crate://crates.io/nom/7.1.0 \
    crate://crates.io/num-traits/0.2.14 \
    crate://crates.io/num_cpus/1.13.1 \
    crate://crates.io/oorandom/11.1.3 \
    crate://crates.io/opaque-debug/0.3.0 \
    crate://crates.io/peeking_take_while/0.1.2 \
    crate://crates.io/plotters/0.3.1 \
    crate://crates.io/plotters-backend/0.3.2 \
    crate://crates.io/plotters-svg/0.3.1 \
    crate://crates.io/polyval/0.5.3 \
    crate://crates.io/ppv-lite86/0.2.16 \
    crate://crates.io/proc-macro2/1.0.36 \
    crate://crates.io/proptest/1.0.0 \
    crate://crates.io/quick-error/2.0.1 \
    crate://crates.io/quote/1.0.15 \
    crate://crates.io/rand/0.8.4 \
    crate://crates.io/rand_chacha/0.3.1 \
    crate://crates.io/rand_core/0.6.3 \
    crate://crates.io/rand_xorshift/0.3.0 \
    crate://crates.io/rayon/1.5.1 \
    crate://crates.io/rayon-core/1.9.1 \
    crate://crates.io/regex/1.5.5 \
    crate://crates.io/regex-automata/0.1.10 \
    crate://crates.io/regex-syntax/0.6.25 \
    crate://crates.io/rustc-hash/1.1.0 \
    crate://crates.io/rustc_version/0.4.0 \
    crate://crates.io/ryu/1.0.9 \
    crate://crates.io/same-file/1.0.6 \
    crate://crates.io/scopeguard/1.1.0 \
    crate://crates.io/semver/1.0.5 \
    crate://crates.io/serde/1.0.136 \
    crate://crates.io/serde_cbor/0.11.2 \
    crate://crates.io/serde_derive/1.0.136 \
    crate://crates.io/serde_json/1.0.78 \
    crate://crates.io/shlex/1.1.0 \
    crate://crates.io/subtle/2.4.1 \
    crate://crates.io/syn/1.0.86 \
    crate://crates.io/textwrap/0.11.0 \
    crate://crates.io/thiserror/1.0.32 \
    crate://crates.io/thiserror-impl/1.0.32 \
    crate://crates.io/timerfd/1.2.0 \
    crate://crates.io/tinytemplate/1.2.1 \
    crate://crates.io/typenum/1.15.0 \
    crate://crates.io/unicode-width/0.1.9 \
    crate://crates.io/unicode-xid/0.2.2 \
    crate://crates.io/universal-hash/0.4.1 \
    crate://crates.io/userfaultfd/0.4.2 \
    crate://crates.io/userfaultfd-sys/0.4.1 \
    crate://crates.io/version_check/0.9.4 \
    crate://crates.io/versionize/0.1.6 \
    crate://crates.io/versionize_derive/0.1.4 \
    crate://crates.io/vm-allocator/0.1.0 \
    crate://crates.io/vm-fdt/0.1.0 \
    crate://crates.io/vm-memory/0.8.0 \
    crate://crates.io/vm-superio/0.5.0 \
    crate://crates.io/vmm-sys-util/0.9.0 \
    crate://crates.io/walkdir/2.3.2 \
    crate://crates.io/wasi/0.10.2+wasi-snapshot-preview1 \
    crate://crates.io/wasm-bindgen/0.2.79 \
    crate://crates.io/wasm-bindgen-backend/0.2.79 \
    crate://crates.io/wasm-bindgen-macro/0.2.79 \
    crate://crates.io/wasm-bindgen-macro-support/0.2.79 \
    crate://crates.io/wasm-bindgen-shared/0.2.79 \
    crate://crates.io/web-sys/0.3.56 \
    crate://crates.io/winapi/0.3.9 \
    crate://crates.io/winapi-i686-pc-windows-gnu/0.4.0 \
    crate://crates.io/winapi-util/0.1.5 \
    crate://crates.io/winapi-x86_64-pc-windows-gnu/0.4.0 \
    crate://crates.io/libc/0.2.135 \
    crate://crates.io/proc-macro2/1.0.47 \
    crate://crates.io/quote/1.0.21 \
    crate://crates.io/serde/1.0.145 \
    crate://crates.io/serde_derive/1.0.145 \
    crate://crates.io/syn/1.0.102 \
    crate://crates.io/unicode-ident/1.0.5 \
    crate://crates.io/vmm-sys-util/0.10.0 \
    crate://crates.io/bitflags/1.2.1 \
    crate://crates.io/libc/0.2.66 \
    crate://crates.io/vmm-sys-util/0.8.0 \
"

SRCREV_FORMAT .= "_kvm-bindings"
SRCREV_kvm-bindings = "4569d3f5b7746b66fc58a14cd05e5dbf9368932b"
SRCREV_FORMAT .= "_micro-http"
SRCREV_micro-http = "0a58eb1ece68e326e68365c4297d0a7c08ecd9bc"

EXTRA_OECARGO_PATHS += "\
    ${WORKDIR}/git/kvm-bindings \
    ${WORKDIR}/git/micro-http \
"

CARGO_SRC_DIR = "src/firecracker"

RUNTIME = "llvm"
TOOLCHAIN = "clang"
PREFERRED_PROVIDER:libgcc = "compiler-rt"

inherit cargo

do_configure:prepend:x86_64() {
    cp ${S}/resources/seccomp/x86_64-unknown-linux-musl.json \
        ${S}/resources/seccomp/x86_64-poky-linux-musl.json
}

do_configure:prepend:aarch64() {
    cp ${S}/resources/seccomp/aarch64-unknown-linux-musl.json \
        ${S}/resources/seccomp/aarch64-poky-linux-musl.json

local.conf requires

TCLIBC="musl"
RUST_PANIC_STRATEGY="abort"

layer deps: meta-clang

Expected behaviour

That seccompiler-bin works when cross compiled.
One solution would be to check the system path for it before attempting to build.

Environment

[ - Firecracker version.]
Tip of tree
[ - Host and guest kernel versions.]
Yocto (Poky) Linux - kirkstone
[ - Rootfs used.]
Yocto generated
[ - Architecture.]
x86_64-poky-linux-musl + aarch64-poky-linux-musl
[ - Any other relevant software versions.]
meta-rust tip of tree (1.64)

Additional context

[How has this bug affected you?]
It's no longer possible to cross-compile build without patching the source tree.

[What are you trying to achieve?]
Build firecracker and tools in a cross compilation environment.

[Do you have any idea of what the solution might be?]
Using this patch and pre-building seccompiler, firecracker builds and installs without error.

SRC_URI = "\
    git://github.com/firecracker-microvm/firecracker.git;protocol=https;nobranch=1 \
    file://0001-use-prebuilt-seccompiler.patch \
"

0001-use-prebuilt-seccompiler.patch:

From 6b4a80fc38899d6ac5198c73349a0d3ffd0712f8 Mon Sep 17 00:00:00 2001
From: Joel Winarske <[email protected]>
Date: Tue, 18 Oct 2022 07:12:09 -0700
Subject: [PATCH] use prebuilt seccompiler

Signed-off-by: Joel Winarske <[email protected]>
---
 src/vmm/build.rs | 14 +-------------
 1 file changed, 1 insertion(+), 13 deletions(-)

diff --git a/src/vmm/build.rs b/src/vmm/build.rs
index bacc4e93..9b82ef25 100644
--- a/src/vmm/build.rs
+++ b/src/vmm/build.rs
@@ -58,20 +58,8 @@ fn run_seccompiler_bin(cargo_target: &str, json_path: &str, out_path: &str) {
     let target_arch = env::var("CARGO_CFG_TARGET_ARCH").expect("Missing target arch.");
 
     // Command for running seccompiler-bin
-    let mut command = Command::new("cargo");
+    let mut command = Command::new("seccompiler-bin");
     command.args(&[
-        "run",
-        "-p",
-        "seccompiler",
-        "--verbose",
-        "--target",
-        cargo_target,
-        // We need to specify a separate build directory for seccompiler-bin. Otherwise, cargo will
-        // deadlock waiting to acquire a lock on the build folder that the parent cargo process is
-        // holding.
-        "--target-dir",
-        SECCOMPILER_BUILD_DIR,
-        "--",
         "--input-file",
         json_path,
         "--target-arch",
-- 
2.34.3

[joel@fedora core2_64]$ rpm -qlp firecracker-0.1.0-r0.core2_64.rpm
/usr
/usr/bin
/usr/bin/firecracker
[joel@fedora core2_64]$ rpm -qlp jailer-0.1.0-r0.core2_64.rpm 
/usr
/usr/bin
/usr/bin/jailer
[joel@fedora core2_64]$ rpm -qlp rebase-snap-0.1.0-r0.core2_64.rpm 
/usr
/usr/bin
/usr/bin/rebase-snap
[joel@fedora core2_64]$ rpm -qlp seccompiler-0.1.0-r0.core2_64.rpm 
/usr
/usr/bin
/usr/bin/seccompiler-bin
[joel@fedora core2_64]$  rpm2cpio firecracker-0.1.0-r0.core2_64.rpm | cpio -idmv
./usr
./usr/bin
./usr/bin/firecracker
3997 blocks
[joel@fedora core2_64]$ ls -la ./usr/bin/firecracker 
-rwxr-xr-x. 1 joel joel 2045840 Oct 16 23:48 ./usr/bin/firecracker
[joel@fedora core2_64]$ readelf -d ./usr/bin/firecracker 

Dynamic section at offset 0x1f2a38 contains 28 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so]
 0x000000000000000c (INIT)               0x1a000

Build Output without patch:

| NOTE: rustc =
| NOTE: cargo build -v --target x86_64-poky-linux-musl --release --manifest-path=/b/yocto/linux-dummy/build/tmp/work/core2-64-poky-linux-musl/firecracker/0.1.0-r0/git/src/firecracker/Cargo.toml
| warning: patch for `kvm-bindings` uses the features mechanism. default-features and features will not take effect because the patch dependency does not support this mechanism
| warning: path override for crate `kvm-bindings` has altered the original list of
| dependencies; the dependency on `versionize` was either added or
| modified to not match the previously resolved version
| 
| This is currently allowed but is known to produce buggy behavior with spurious
| recompiles and changes to the crate graph. Path overrides unfortunately were
| never intended to support this feature, so for now this message is just a
| warning. In the future, however, this message will become a hard error.
| 
| To change the dependency graph via an override it's recommended to use the
| `[patch]` feature of Cargo instead of the path override feature. This is
| documented online at the url below for more information.
...
|       Building [=========================> ] 31/32: seccompiler-bin(bin)
|      Running `rustc --crate-name seccompiler_bin --edition=2018 src/seccompiler/src/seccompiler_bin.rs --error-format=json --json=diagnostic-rendered-ansi,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 -C metadata=dd2fcea93bc9a13b -C extra-filename=-dd2fcea93bc9a13b --out-dir /b/yocto/linux-dummy/build/tmp/work/core2-64-poky-linux-musl/firecracker/0.1.0-r0/git/src/vmm/../../build/seccompiler/x86_64-poky-linux-musl/debug/deps --target x86_64-poky-linux-musl -C linker=/b/yocto/linux-dummy/build/tmp/work/core2-64-poky-linux-musl/firecracker/0.1.0-r0/wrapper/target-rust-ccld -C incremental=/b/yocto/linux-dummy/build/tmp/work/core2-64-poky-linux-musl/firecracker/0.1.0-r0/git/src/vmm/../../build/seccompiler/x86_64-poky-linux-musl/debug/incremental -L dependency=/b/yocto/linux-dummy/build/tmp/work/core2-64-poky-linux-musl/firecracker/0.1.0-r0/git/src/vmm/../../build/seccompiler/x86_64-poky-linux-musl/debug/deps -L dependency=/b/yocto/linux-dummy/build/tmp/work/core2-64-poky-linux-musl/firecracker/0.1.0-r0/git/src/vmm/../../build/seccompiler/debug/deps --extern bincode=/b/yocto/linux-dummy/build/tmp/work/core2-64-poky-linux-musl/firecracker/0.1.0-r0/git/src/vmm/../../build/seccompiler/x86_64-poky-linux-musl/debug/deps/libbincode-7a5f304127b1d08c.rlib --extern derive_more=/b/yocto/linux-dummy/build/tmp/work/core2-64-poky-linux-musl/firecracker/0.1.0-r0/git/src/vmm/../../build/seccompiler/debug/deps/libderive_more-113bd1c92cf88c6b.so --extern libc=/b/yocto/linux-dummy/build/tmp/work/core2-64-poky-linux-musl/firecracker/0.1.0-r0/git/src/vmm/../../build/seccompiler/x86_64-poky-linux-musl/debug/deps/liblibc-6fa2d2f9be58e321.rlib --extern seccompiler=/b/yocto/linux-dummy/build/tmp/work/core2-64-poky-linux-musl/firecracker/0.1.0-r0/git/src/vmm/../../build/seccompiler/x86_64-poky-linux-musl/debug/deps/libseccompiler-bb60495074d5fa52.rlib --extern serde=/b/yocto/linux-dummy/build/tmp/work/core2-64-poky-linux-musl/firecracker/0.1.0-r0/git/src/vmm/../../build/seccompiler/x86_64-poky-linux-musl/debug/deps/libserde-249ecdcc3ff8adc7.rlib --extern serde_json=/b/yocto/linux-dummy/build/tmp/work/core2-64-poky-linux-musl/firecracker/0.1.0-r0/git/src/vmm/../../build/seccompiler/x86_64-poky-linux-musl/debug/deps/libserde_json-84da451989bd128a.rlib --extern utils=/b/yocto/linux-dummy/build/tmp/work/core2-64-poky-linux-musl/firecracker/0.1.0-r0/git/src/vmm/../../build/seccompiler/x86_64-poky-linux-musl/debug/deps/libutils-355040f56a8cd184.rlib -L /b/yocto/linux-dummy/build/tmp/work/core2-64-poky-linux-musl/firecracker/0.1.0-r0/recipe-sysroot/usr/lib/rust --remap-path-prefix=/b/yocto/linux-dummy/build/tmp/work/core2-64-poky-linux-musl/firecracker/0.1.0-r0=/usr/src/debug/firecracker/0.1.0-r0`
|   warning: `seccompiler` (bin "seccompiler-bin") generated 2 warnings (2 duplicates)
|       Finished dev [unoptimized + debuginfo] target(s) in 14.51s
|        Running `../../build/seccompiler/x86_64-poky-linux-musl/debug/seccompiler-bin --input-file ../../resources/seccomp/x86_64-poky-linux-musl.json --target-arch x86_64 --output-file /b/yocto/linux-dummy/build/tmp/work/core2-64-poky-linux-musl/firecracker/0.1.0-r0/build/target/x86_64-poky-linux-musl/release/build/vmm-9e1cd72b8f1b959f/out/seccomp_filter.bpf`
|   error: could not execute process `../../build/seccompiler/x86_64-poky-linux-musl/debug/seccompiler-bin --input-file ../../resources/seccomp/x86_64-poky-linux-musl.json --target-arch x86_64 --output-file /b/yocto/linux-dummy/build/tmp/work/core2-64-poky-linux-musl/firecracker/0.1.0-r0/build/target/x86_64-poky-linux-musl/release/build/vmm-9e1cd72b8f1b959f/out/seccomp_filter.bpf` (never executed)

../../build/seccompiler/x86_64-poky-linux-musl/debug/seccompiler-bin is not correctly being built for HOST.

Checks

Have you searched the Firecracker Issues database for similar problems?
Have you read the existing relevant Firecracker documentation?
Are you certain the bug being reported is a Firecracker issue?

The text was updated successfully, but these errors were encountered:

bchalios · 2022-10-25T08:30:03Z

Hello, thanks for opening an issue.

We are not really familiar with Yocto. Could you give us more details on your use-case? Why do you specifically need to build with Yocto and cross-compile?

Also, could you provide more details on the compilation failure?

jwinarske · 2022-10-25T14:01:33Z

@bchalios The use case is around building a custom OS image that includes firecracker components. Instead of building on a host and copying to a target to run, one uses Yocto to build a custom Linux OS image and flash/boot device. It allows building reproducible custom Linux OS images. It does not use the firecracker build script, but above "recipe".

The build invocation and error are included at the tail of my original post. There is nothing else. The gist is that the current method of building seccompiler (main) is not compatible when Cross compiling firecracker.

https://www.yoctoproject.org/

roypat · 2022-10-28T10:15:32Z

Hey Joel,
We just merged a fix for another cross-compilation failure wrt seccompiler-bin that we were investigating. Could you please try whether this also fixes your issue?

jwinarske · 2022-10-29T19:04:03Z

Will do, thanks

…

On Fri, Oct 28, 2022, 3:15 AM Patrick Roy ***@***.***> wrote: Hey Joel, We just merged a fix for another cross-compilation failure wrt seccompiler-bin that we were investigating. Could you please try whether this also fixes your issue? — Reply to this email directly, view it on GitHub <#3188 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC5R3WZC72GS5KC47SO6CCTWFORU7ANCNFSM6AAAAAARIFHDQU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

roypat · 2022-11-07T09:55:12Z

Hi Joel,
just wanted to follow up to check in if the cross-compilation issue has been fixed for you?

jwinarske · 2023-01-18T01:00:24Z

@roypat Problem is not resolved. I just checked on tip of tree.

jwinarske · 2023-01-18T01:36:23Z

Working Yocto kirkstone recipes for tip of tree firecracker are located here:
https://github.com/meta-flutter/meta-flutter-vm/tree/kirkstone/recipes-vm/firecracker

roypat · 2023-09-18T09:24:18Z

Hi, we have some plans to replace the cargo run step in the build script with just calling into seccompiler as a library. Hopefully that should also fix this issue (since it should have the same effect as using a pre-compiled binary)

roypat · 2023-09-21T16:30:24Z

#4120 changed our build script setup to no longer invoke cargo run. Could you check if this resolves your issue?

wearyzen · 2023-10-02T12:30:18Z

Closing the issue since the issue seems to be fixed with #4120.

jwinarske · 2023-10-06T05:14:18Z

@roypat I'll confirm the change.

@sudanl0 that makes no sense.

wearyzen · 2023-10-23T21:07:58Z

Hi @jwinarske, there was no reply on the issue for a long time so we concluded this was fixed. However, if this is not the case please feel free to reopen the issue and we will have a look at it.

jwinarske added the Type: Bug Indicates an unexpected problem or unintended behavior label Oct 18, 2022

roypat mentioned this issue Oct 27, 2022

Fix cross compiling firecracker for different architecture #3213

Merged

9 tasks

roypat closed this as completed in #3213 Oct 28, 2022

roypat reopened this Oct 28, 2022

jwinarske mentioned this issue Feb 2, 2023

Enable KVM for raspberrypi4-64 agherzan/meta-raspberrypi#1141

Closed

roypat added the Status: Parked Indicates that an issues or pull request will be revisited later label Sep 18, 2023

ShadowCurse mentioned this issue Sep 21, 2023

Moving seccomp trigger #4120

Merged

9 tasks

wearyzen closed this as completed Oct 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug] Build - Cross Compilation Failure #3188

[Bug] Build - Cross Compilation Failure #3188

jwinarske commented Oct 18, 2022 •

edited

Loading

bchalios commented Oct 25, 2022

jwinarske commented Oct 25, 2022

roypat commented Oct 28, 2022

jwinarske commented Oct 29, 2022 via email

roypat commented Nov 7, 2022

jwinarske commented Jan 18, 2023

jwinarske commented Jan 18, 2023

roypat commented Sep 18, 2023

roypat commented Sep 21, 2023

wearyzen commented Oct 2, 2023

jwinarske commented Oct 6, 2023

wearyzen commented Oct 23, 2023

[Bug] Build - Cross Compilation Failure #3188

[Bug] Build - Cross Compilation Failure #3188

Comments

jwinarske commented Oct 18, 2022 • edited Loading

Describe the bug

To Reproduce

Expected behaviour

Environment

Additional context

Checks

bchalios commented Oct 25, 2022

jwinarske commented Oct 25, 2022

roypat commented Oct 28, 2022

jwinarske commented Oct 29, 2022 via email

roypat commented Nov 7, 2022

jwinarske commented Jan 18, 2023

jwinarske commented Jan 18, 2023

roypat commented Sep 18, 2023

roypat commented Sep 21, 2023

wearyzen commented Oct 2, 2023

jwinarske commented Oct 6, 2023

wearyzen commented Oct 23, 2023

jwinarske commented Oct 18, 2022 •

edited

Loading