HashMap executes UB on aarch64_be-unknown-linux-gnu_ilp32

[cathy-sjh]

I tried this code:

use std::collections::HashMap;

fn main() {
    let mut map = HashMap::new();
    map.insert(1, "a");
    
    println!("map = {:?}", map);
}

I expected to see this happen: map = {1: "a"}

Instead, error happened:

# ./hello_world_ilp32 
thread 'main' panicked at /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:136:5:
unsafe precondition(s) violated: NonNull::new_unchecked requires that the pointer is non-null
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread caused non-unwinding panic. aborting.
Aborted

Sometimes it causes Segmentation fault:

# ./hello_world_ilp32 
Segmentation fault

Meta

rustc --version --verbose:

rustc 1.74.0-nightly (3223b0b5e 2023-09-20)
binary: rustc
commit-hash: 3223b0b5e8dadda3f76c3fd1a8d6c5addc09599e
commit-date: 2023-09-20
host: x86_64-unknown-linux-gnu
release: 1.74.0-nightly
LLVM version: 17.0.0

compilation command:

cargo +nightly build --target aarch64_be-unknown-linux-gnu_ilp32 -Zbuild-std

.cargo/config:

[target.aarch64_be-unknown-linux-gnu_ilp32]
linker = "/opt/buildtools/rtos/opt/RTOS/207.7.0/arm64be_4.4_ek/bin/aarch64_be-linux-gnuilp32-gcc"

Backtrace

stack backtrace:
   0: rust_begin_unwind
             at /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:597:5
   1: core::panicking::panic_nounwind_fmt
             at /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:106:14
   2: core::panicking::panic_nounwind
             at /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:136:5
   3: core::ptr::const_ptr::<impl *const T>::sub_ptr::runtime
             at /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/intrinsics.rs:2531:21
   4: core::ptr::const_ptr::<impl *const T>::sub_ptr
             at /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/const_ptr.rs:792:13
   5: core::ptr::non_null::NonNull<T>::sub_ptr
             at /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/non_null.rs:507:18
   6: <core::slice::iter::Iter<T> as core::iter::traits::exact_size::ExactSizeIterator>::len
             at /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/slice/iter/macros.rs:142:17
   7: <core::str::iter::CharIndices as core::iter::traits::iterator::Iterator>::next
             at /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/str/iter.rs:139:23
   8: <str as core::fmt::Debug>::fmt
             at /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/fmt/mod.rs:2336:23
   9: <&T as core::fmt::Debug>::fmt
             at /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/fmt/mod.rs:2288:62
  10: <&T as core::fmt::Debug>::fmt
             at /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/fmt/mod.rs:2288:62
  11: core::fmt::builders::DebugMap::value::{{closure}}
             at /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/fmt/builders.rs:850:17
  12: core::result::Result<T,E>::and_then
             at /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/result.rs:1320:22
  13: core::fmt::builders::DebugMap::value
             at /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/fmt/builders.rs:841:23
  14: core::fmt::builders::DebugMap::entry
             at /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/fmt/builders.rs:741:9
  15: core::fmt::builders::DebugMap::entries
             at /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/fmt/builders.rs:893:13
  16: <std::collections::hash::map::HashMap<K,V,S> as core::fmt::Debug>::fmt
             at /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/collections/hash/map.rs:1309:9
  17: core::fmt::rt::Argument::fmt
             at /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/fmt/rt.rs:138:9
  18: core::fmt::write
             at /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/fmt/mod.rs:1114:17
  19: std::io::Write::write_fmt
             at /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/io/mod.rs:1714:15
  20: <&std::io::stdio::Stdout as std::io::Write>::write_fmt
             at /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/io/stdio.rs:726:9
  21: <std::io::stdio::Stdout as std::io::Write>::write_fmt
             at /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/io/stdio.rs:700:9
  22: std::io::stdio::print_to
             at /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/io/stdio.rs:1018:21
  23: std::io::stdio::_print
             at /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/io/stdio.rs:1095:5
  24: hello_world_ilp32::main
             at /usr3/s00659936/rust_program/helloworld_ilp32/src/main.rs:7:5
  25: core::ops::function::FnOnce::call_once
             at /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5

[saethlin]

Miri reports UB with the target in question:

error: Undefined Behavior: out-of-bounds pointer arithmetic: alloc1009 has size 60, so pointer to 12 bytes starting at offset -12 is out-of-bounds
   --> /home/ben/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hashbrown-0.14.0/src/raw/mod.rs:510:22
    |
510 |             unsafe { self.ptr.as_ptr().sub(1) }
    |                      ^^^^^^^^^^^^^^^^^^^^^^^^ out-of-bounds pointer arithmetic: alloc1009 has size 60, so pointer to 12 bytes starting at offset -12 is out-of-bounds
    |
    = help: this indicates a bug in the program: it performed an invalid operation, and caused Undefined Behavior
    = help: see https://doc.rust-lang.org/nightly/reference/behavior-considered-undefined.html for further information
help: alloc1009 was allocated here:
   --> src/main.rs:5:5
    |
5   |     map.insert(1, "a");
    |     ^^^^^^^^^^^^^^^^^^
    = note: BACKTRACE (of the first span):
    = note: inside `hashbrown::raw::Bucket::<(i32, &str)>::as_ptr` at /home/ben/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hashbrown-0.14.0/src/raw/mod.rs:510:22: 510:46
    = note: inside `hashbrown::raw::Bucket::<(i32, &str)>::as_ref::<'_>` at /home/ben/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hashbrown-0.14.0/src/raw/mod.rs:674:11: 674:24
    = note: inside `<hashbrown::map::Iter<'_, i32, &str> as std::iter::Iterator>::next` at /home/ben/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hashbrown-0.14.0/src/map.rs:4737:25: 4737:35
    = note: inside `<std::collections::hash_map::Iter<'_, i32, &str> as std::iter::Iterator>::next` at /home/ben/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/collections/hash/map.rs:2232:9: 2232:25
    = note: inside `std::fmt::DebugMap::<'_, '_>::entries::<&i32, &&str, std::collections::hash_map::Iter<'_, i32, &str>>` at /home/ben/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/fmt/builders.rs:893:23: 893:30
    = note: inside `<std::collections::HashMap<i32, &str> as std::fmt::Debug>::fmt` at /home/ben/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/collections/hash/map.rs:1309:9: 1309:43

(sorry for reflexively blaming LLVM, it didn't deserve that)

[Amanieu]

The root cause is that the stdarch SIMD intrinsics are broken on big-endian targets. For now I've disabled the use of NEON in hashbrown on big-endian ARM: rust-lang/hashbrown#475

We should probably disable/de-stabilize the NEON intrinsics on big-endian targets. They are currently all producing incorrect results.

[RalfJung]

The root cause is that the stdarch SIMD intrinsics are broken on big-endian targets.

All of them, or specifically for ARM? (Not sure if we have stdarch support for any other big-endian target.)

Is there an issue tracking that and explaining in a bit more detail how they are broken?

[Amanieu]

This only applies to arm & aarch64 targets. Here is the issue in stdarch: rust-lang/stdarch#1484

[RalfJung]

Miri reports UB with the target in question:

Miri doesn't support the NEON SIMD intrinsics though. So if Miri reports UB then it seems there are other problems as well?

[bjorn3]

Some are implemented using the same simd_* intrinsics as portable-simd.

[Amanieu]

It's a combination of a few things:

• We are missing a cfg(not(miri)) check for the NEON implementation.
• All the NEON intrinsics we happen to use in hashbrown are implemented without LLVM builtins.

[RalfJung]

Ah so I guess what happens is that Miri correctly implements the portable SIMD intrinsics, but stdarch implements the NEON intrinsics on top of them and it's doing that incorrectly? Miri and codegen/LLVM are fine?

[Amanieu]

Ah so I guess what happens is that Miri correctly implements the portable SIMD intrinsics, but stdarch implements the NEON intrinsics on top of them and it's doing that incorrectly? Miri and codegen/LLVM are fine?

Yes, that's my understanding.