Use the glibc 64bit time APIs to mitigate the Y2038 problem on 32bit platforms

[snogge]

The GNU libc (glibc) provides alternative 64bit entry points for many of the file and time related functions on 32bit platforms. For C programs these are activated by setting the -D_TIME_BITS=64 and -D_FILE_OFFSET_BITS=64 preprocessor flags. What happens is that all the relevant types (time_t, blksize_t, ...) are defined as 64bit integers and the called symbols are redirected from the standard 32-bit variant like gmtime to a 64bit variant like __gmtime64.

In many ways this is the same as -D_LARGEFILE_SOURCE flag except that this will not add new types and symbols but replace the existing ones. Note that it is possible to use -D_LARGEFILE_SOURCE and -D_FILE_OFFSET_BITS=64 at the same time.

For full 64bit time support on llnux you also need a kernel newer than 5.6, but the 64bit glibc functions should be safe to use anyway and no worse than the 32bit versions.

The 64bit time support is needed when building embedded linux systems on 32bit platforms that are expected to live for 10 to 15 years.

[snogge]

I've submitted #3175 as a possible solution.

[safinaskar]

If somebody needs something like crate libc, but without time32 and lfs problems, I suggest using crate rustix. You can read its code and see that rustix put a lot of effort to deal with time32 and lfs problems

[snogge]

I can see how that would work if I control which crates I want to use. But I'm not designing these projects from scratch, I'm trying to build existing projects with plenty of dependencies that use the libc crate.
But if there is a way to drop rustix in as a replacement I'm all ears.

If it is not obvious: I don't know much about rust, I'm approaching this from a distro building perspective using the Yocto project.

[nekopsykose]

regardless of whether rustix works or not, libc has to be fixed anyway (i.e. this is lacking platform support and a goal of libc), so that is more of an offtopic suggestion, even if it's what somebody wants to use for a usecase.

[lvella]

I think the sensible solution here is to simply use 64 bits version of these types in every platform that supports them. In this time and age, I think it makes no sense for Rust's libc to support 32 bits off_t, time_t, etc.

It is not like we have to worry about not breaking a huge ecosystem of 30 years old code that assumes these types to be 32 bits, like they have to in C.

[snogge]

Will this be part of the 0.3 release? I see a lot of talk about the same problem on musl.

[JohnTitor]

cc @joshtriplett as you facilitate the libc 0.3 here: #3248

[Amanieu]

The main concern here is compatibility with C code that we link to using FFI. If some C code uses time_t, we want to ensure that our FFI bindings in Rust use the same definition of time_t as the C code. The general policy for the libc crate is to match the behavior of C with no preprocessor options defined. In the case of musl 1.2 this would default to 64-bit time but for glibc it would default to 32-bit time.

This is similar to the situation with 64-bit off_t: we default to 32-bit off_t but also expose LFS types and functions such as stat64. This won't work for 64-bit time since there are no alternate type/function names, but one possible solution would be to expose the 64-bit time_t types and functions under a time64 module.

[snogge]

I understand your concern. But won't that be just as big a problem if the library has been built with the 64-bit flags set even if that is not the default for that library?
I suspect that these flags will be set on a distro level on Linux. How would Rust applications that use libc handle that?

Note that as long as the C library does not use any of the affected types such as time_t in their ABI, there is no conflict.

My usecase is an embedded linux distribution created with Yocto. Everything is built from source and I control all the compiler settings. I need to make any Rust app build against 64-bit libs, and right now it seems my only option is to patch all uses of libc. For this, I would be happy with a switch I could set globally but I can see how that is not usable by everyone.

[Amanieu]

Unfortunately there is just no way for Rust to know what flags any particular C code was compiled with. My recommendation would be to expose the time64 APIs in a time64 module, and only for glibc. Then Rust programs using the libc crate can choose which API to use depending on the flags use to compile the C code (e.g. using the cc crate from build scripts).

Most Rust programs will not interact with the libc crate directly though: most will use the standard library's SystemTime instead. This already supports 64-bit time APIs on glibc by calling symbols such as __clock_gettime64 directly.

[kanavin]

SystemTime and libc crate aren't using musl correctly either. I got the following on a 32 bit musl system after setting the system clock to 2040.

I didn't yet look into details of what is happening, but please take this seriously: rust simply can't be used on 32 bit systems as of today, not if you want them to keep working after 2038:

root@qemux86:~/hello# RUST_BACKTRACE=full cargo build
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 75, kind: Uncategorized, message: "Value too large for data type" }', library/std/src/sys/unix/time.rs:404:72
stack backtrace:
   0:  0x103b76e - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h298f061022fcd35f
   1:  0x10932f0 - core::fmt::write::h35b33882d9394d02
   2:  0x1062721 - std::io::Write::write_fmt::haa08496d0a09f55a
   3:  0x103b534 - std::sys_common::backtrace::print::hdd15c84bc6d639a0
   4:  0x1040e22 - std::panicking::default_hook::{{closure}}::h5a904d4c613603d1
   5:  0x1040a8e - std::panicking::default_hook::h151573c2103f9e1d
   6:  0x10419cd - std::panicking::rust_panic_with_hook::h253407a2d246e6f4
   7:  0x103bacd - std::panicking::begin_panic_handler::{{closure}}::hc37c74e9bd7e9112
   8:  0x103b8a9 - std::sys_common::backtrace::__rust_end_short_backtrace::ha4ef9ac410a9a589
   9:  0x1041512 - rust_begin_unwind
  10:   0x51ea15 - core::panicking::panic_fmt::h2d2cbb4e20fa6b71
  11:   0x51e912 - core::result::unwrap_failed::h8814d485edf5763b
  12:  0x1049f1d - std::sys::unix::time::inner::<impl std::sys::unix::time::Timespec>::now::h558a463fd73b0e89
  13:  0x10707ac - std::time::SystemTime::now::h6c0b823bbb84a570
  14:   0x5fba60 - cargo::core::compiler::timings::Timings::new::h120d8638ea051b42
  15:   0x8e2106 - cargo::core::compiler::job_queue::JobQueue::new::h9433228ad310e0d9
  16:   0x751aa7 - cargo::core::compiler::context::Context::compile::hc66564059807438c
  17:   0xa4b65f - cargo::ops::cargo_compile::compile_ws::hc837eacdf826dd77
  18:   0xa4b352 - cargo::ops::cargo_compile::compile::h7008d233c91a9342
  19:   0x536221 - cargo::commands::build::exec::h451c313ba133f7a0
  20:   0x5465c3 - cargo::cli::execute_subcommand::h10bbc4a45aa39bf7
  21:   0x543e2f - cargo::cli::main::h90c338691b998293
  22:   0x5a6231 - cargo::main::ha655f162b7ead5e5
  23:   0x524634 - std::sys_common::backtrace::__rust_begin_short_backtrace::h80d0852799ad4e70
  24:   0x59387e - std::rt::lang_start::{{closure}}::h1188bc695e7dc705
  25:  0x10412c7 - std::panicking::try::hdc84a4a49c4e71bf
  26:  0x1061b8e - std::rt::lang_start_internal::hac224e5bce96cd89
  27:   0x5a8c38 - main
  28: 0xb7efc1bf - libc_start_main_stage2
                       at /usr/src/debug/musl/1.2.4+gitAUTOINC+83b858f83b-r0/src/env/__libc_start_main.c:95:2

[snogge]

Wouldn't using a time64 module require source code changes to all users of the libc crate?
Is there any way we can use a configuration option to the libc crate and make it use only 64bit time?

[kanavin]

Libc crate could check whether _TIME_BITS=64 is in CFLAGS (as prescribed by glibc), and then build with 64 bit time. Same as what happens via glibc header define magic in C components. I’m not yet sure where things go wrong with musl (see above), need to look into it.

[kanavin]

After some poking at musl code, I think it has the same ABI as as glibc: clock_gettime() is the 32 bit version, and if you want 64 bits, you must call into __clock_gettime64(). C headers would take care of this automatically, but rust won't use them, so 🤷

musl's version of _TIME_BITS define is _REDIR_TIME64, and it's set to 1 by default in its headers (unlike glibc where you need to opt into 64 bit time via externally set -DTIME_BITS=64).

[kanavin]

@snogge you can change the title of this issue, I suppose. It's the same on glibc and musl. 32 bit api in use on both :(

[snogge]

@kanavin, switching musl to 64bit time_t is part of #1848 (PR #3068) and is also mentioned in #3248 .

[kanavin]

@kanavin, switching musl to 64bit time_t is part of #1848 (PR #3068) and is also mentioned in #3248 .

Thanks, I left a comment there. #3068 could serve as a template for adding glibc time64 support as well: enable cargo:rustc-cfg=glibc_time64_abi, if -D_TIME_BITS=64 is in CFLAGS. Then redirect everything to time64 symbols subject to that.

Then rust-libc consumers need not know or care about which ABI to select (I strongly object to the above suggestion that they should - one will never be able to fix or validate correct usage in a complete system that way).

[snogge]

I'll have a look at that later. I'm working on rebasing the PR right now.

[safinaskar]

@snogge , do you still want this? If yes, then now is perfect time for making such changes, because breaking libc 1.0 (aka 0.3) will be released soon: #3248

[snogge]

I do, and I'm working on rebasing and cleaning up my PR as we speak.

[snogge]

@safinaskar the PR has been updated and is ready for review.

[sdroege]

There's also the problem here that Linux distributions are switching over to using -D_FILE_OFFSET_BITS=64 -D_TIME_BITS=64 by default, e.g. Debian: https://wiki.debian.org/ReleaseGoals/64bit-time *

This means that somehow the libc crate would have to be switched to the 64 bit definitions depending on not just on what a certain piece of C code usually uses, but also what the default of the whole system are. During the transition period, that effectively depends also on which build of a C library is currently in use.

---

Quoting from the Debian wiki

4963 are golang-*, librust-* and libghc-* which we can ignore, leaving 5360 packages

that sounds like they actually missed this problem here.

[kanavin]

There's also the problem here that Linux distributions are switching over to using -D_FILE_OFFSET_BITS=64 -D_TIME_BITS=64 by default, e.g. Debian: https://wiki.debian.org/ReleaseGoals/64bit-time *

This means that somehow the libc crate would have to be switched to the 64 bit definitions depending on not just on what a certain piece of C code usually uses, but also what the default of the whole system are. During the transition period, that effectively depends also on which build of a C library is currently in use.

I'm not sure why this is a complication. This libc crate should check CFLAGS from the unix environment at build time, if -D_TIME_BITS=64 is there, enable that, otherwise don't. Then the OS build system would set CFLAGS depending on which libraries would be used (ideally it's just a system wide setting with no need to provide 32 bit time compatibility items). Is there some scenario where it wouldn't work?

[sdroege]

If we ignore the transitional aspect where some libraries are using 64 bit and others 32 bit time_t *, a user building some Rust software interfacing with C libraries via the libc crate would have to set something in the environment for the libc crate to do the right thing, and otherwise there are going to be annoying ABI problems at runtime. (Note that CFLAGS="-D_TIME_BITS=64" is not just set globally!) This seems like something that is going to cause a lot of problems to users unaware of that, and a lot of issues being reported that waste everybody's time.

* Not sure if it's safe to ignore the transitional aspect though. It doesn't seem like an impossible situation that users running a 64 bit time_t Debian will have some 3rd party libraries that use 32 bit time_t still. As long as there is no incompatible mixing in the same build, this would work fine in C but adds an additional challenge to the libc crate. It doesn't just depend on the system but on the exact library configuration, and theoretically you could have both variants in the same build but libc can only be compiled with one configuration.

---

Maybe I'm missing something here and this is all somehow taken care off. I didn't look too closely into this issue and only thought of the interactions with the libc crate today. (And I guess bindgen for example will have similar problems)

[snogge]

The compiled GNU libc library always support both 32bit and 64bit time APIs and ABI. A C program that is built with the proper preprocessing options will use 64bit types and redirect to the 64bit APIs.
You could actually build different C files in the same program using different options, and it will work fine as long as the affected types are not part of the interaction between the two translation units.
Where I think it may become a problem for Rust programs is when a program uses some other C library that has not been built with the -D_TIME_BITS=64 flags, and the relevant types are part of the ABI. If the library API/ABI doesn't use these types I think it will be fine in the same way as the C translation units mentioned above.
But note that I'm not a Rust programmer, there could very well be issues I'm not aware of.

[kanavin]

For what it's worth, I'm coming from the Yocto project, which builds complete system images and associated package feeds completely from source. So we have the luxury of using global CFLAGS to enforce 64 bit time across the entire stack, with no exceptions and transitions and potential mixing of libraries. In this scenarion, we'd still want rust libc crate to honor that global setting, as otherwise it becomes impossible to tell if all components using that crate are doing the right thing.

[Amanieu]

I don't see a good way to solve this. The fundamental issue is that we don't know what pre-processor definitions were used to build arbitrary C code we may want to link with. It could be system libraries built with -D_TIME_BITS=64, it could be C code built with build.rs with the default settings (so 32-bit time), or it could even be an external cross-compiling toolchain which was built with options that are unrelated to what the host system uses.

We could do some probing in build.rs to check if the C compiler for the current target generates a 64-bit time_t by default, but I really don't want libc to depend on having a C compiler available at build time considering how it is used in the ecosystem.

This unfortunately leaves us with very few options. I think the best way forward for libc is, for glibc targets, to:

• Expose 64-bit off_t functions under the existing aliases used by glibc (e.g. fseeko64).
• Expose 64-bit time_t functions and types under a time64 sub-module. Glibc doesn't publicly expose separate function names for opting into 64-bit time_t, you can only do it with _TIME_BITS (internally, these use mangled symbols like __clock_gettime64).
• Expose 32-bit time_t/off_t functions under the default symbol names (e.g. fseek, clock_gettime). However these should be marked as deprecated, with the deprecation message pointing to the preferred function to use instead (fseeko64/libc::time64::clock_gettime).

It then becomes the user's responsibility to ensure that if they use the 64-bit symbols, they need to ensure that the C code they are using is built with the proper flags.

The downside of this approach is that we are intentionally shipping deprecated functions, but I feel it is better this way because it gives users a chance to think twice in cases where there might be an incompatibility with C code that is being linked into the same binary.

[sdroege]

I don't see a good way to solve this.

That was also my conclusion. It seems unsolveable in general, and even if you have a C compiler at hand you still don't know which of the two variants that C library in front of you was compiled with.

It then becomes the user's responsibility to ensure that if they use the 64-bit symbols, they need to ensure that the C code they are using is built with the proper flags.

Unless I misunderstand you, I think the problem with this approach is that you actually make it the developer's responsibility (selecting which of the variants to use in their code) to solve a problem that depends on the user's choices (what the user's C libraries are compiled with).

Let's say you're the maintainer of openssl-sys (OpenSSL has a couple of functions that work with time_t, e.g. ASN1_TIME_set). You can't possibly know if the user that actually builds something that somewhere down in the dependency tree has openssl-sys happens to have a libSSL.so that expects 32 bit or 64 bit time_t, so how would you select the correct variant of the API in your code?

Apart from a build-time configuration and making this very clear in the documentation, (and having Debian/etc set this correctly for all their builds!), I don't see another way out of here. Of course a user building some random crate that somewhere deep in its dependency has openssl-sys or a similar case likely wouldn't read these docs.

---

Also as a side-note, this whole thing also affects off_t. -D_TIME_BITS=64 implies -D_FILE_OFFSET_BITS=64, which does the same thing to off_t. But -D_FILE_OFFSET_BITS=64 can also be set independently. The PR for this handles off_t in parallel with time_t.

[safinaskar]

Note that Debian plans to switch to time64 for many 32-bit archs, but not for i386, according to wiki:

https://wiki.debian.org/ReleaseGoals/64bit-time

[snogge]

glibc is probably unique in supporting both variants. Eventually we will be in the situation that some versions of some distros on some archs use 64 bit time_t while the rest stick to 32 bit. And the libc crate will be incompatible with one of the sets.
The openssl library will be built with different options on these variants. How will openssl-sys cope?
#3175 tries to use a single switch to control which variant to use. This will at least make is relatively simple for distros to change to the non-default variant.
One thing that would be useful for Yocto and similar projects is to merge this inactivated. Patching just the activation switch would have much lower impact on such projects. Keeping both variants working would require more tests in the libc crate.

[kanavin]

glibc is probably unique in supporting both variants. Eventually we will be in the situation that some versions of some distros on some archs use 64 bit time_t while the rest stick to 32 bit. And the libc crate will be incompatible with one of the sets. The openssl library will be built with different options on these variants. How will openssl-sys cope? #3175 tries to use a single switch to control which variant to use. This will at least make is relatively simple for distros to change to the non-default variant. One thing that would be useful for Yocto and similar projects is to merge this inactivated. Patching just the activation switch would have much lower impact on such projects. Keeping both variants working would require more tests in the libc crate.

I'm not sure how this would be activated, but from Yocto perspective what is needed is that we can set a 'global setting' inside a complete system build that would be automatically picked up by all of the components that contain the libc crate. Right now that setting is having the -D_TIME_BITS=64 in CFLAGS environment variable, and if libc crate would pick up and respect that then that'd be perfect.

[plugwash]

Eventually we will be in the situation that some versions of some distros on some archs use 64 bit time_t

Where, if a google search for Ubuntu noble release date is to be belived "Eventually" is in four days. Ubuntu noble (24.04) will be using time64 on armhf but not on i386.

I would agree with Anvin that enabling this based on CFLAGS seems a reasonable first step, it's unlikely to break anything that isn't already broken and will fix things for some (but admittedly not all) usecases. Whether to add more elaborate detection is IMO a question that can be asked after the basic support is included.

[snogge]

I could update the PR to check the CFLAGS environment variable, but wouldn't that only work in distro-like builders? Most users wont have CFLAGS set in their environments. And the correct flags might not be in CFLAGS anyway. The Yocto Projects puts them in the CC variable.

[kanavin]

I could update the PR to check the CFLAGS environment variable, but wouldn't that only work in distro-like builders? Most users wont have CFLAGS set in their environments. And the correct flags might not be in CFLAGS anyway. The Yocto Projects puts them in the CC variable.

This is correct, CFLAGS is not the canonical place to put these, and I was confused about where yocto puts them. There are various way to prescribe what ends up in C preprocessor, e.g. environment variables with command line flags, Makefiles, per-project config includes etc.

That said, I would really appreciate at least having some environment variable that can be set by distro builders to force 64 bit time in rust libc in a complete system build. You can name it RUST_LIBC_USE_64BIT_TIME. We (the yocto project) willl take care of setting it as appropriate.

[sdroege]

AFAICS the most user-friendly way would be to check the target triplet, and Linux distro version from build.rs and based on that and a table auto-detect the variant of time_t (and off_t) that should be used. Plus an override for the auto-detection.

That probably won't work for Yocto though as there it's actually a configuration (can that be retrieved somehow easily?) and not a simple matter of checking a version.

[snogge]

Isn't that what cargo is doing by passing the various CARGO_CFG variables? See the is_gnu_time64_abi function in build.rs.
I don't think it is reasonable for the libc crate to carry a list of all distro/triplet combinations that should configure things one way or another. Auto-detect by cargo with a possibility to override sounds OK.

[kanavin]

Isn't that what cargo is doing by passing the various CARGO_CFG variables? See the is_gnu_time64_abi function in build.rs. I don't think it is reasonable for the libc crate to carry a list of all distro/triplet combinations that should configure things one way or another. Auto-detect by cargo with a possibility to override sounds OK.

I'm also not keen on the hardcoded distro/version table idea. As far as I understand, the only remaining binary distro that has any interest at all in 32 bit computing (and fixing y2038 in it) is Debian. For yocto such auto-detection is not useful as it is a generator of custom distros, and especially if build.rs would try to read things in /etc to determine 'distro (and its version) it would make completely incorrect decisions.

is_gnu_time64_abi() as it is has two problems:

• riscv32 has had 64 bit time from the beginning. There may be other targets like that.
• no ability to override its decision from an environment variable

I'd consider not trying to guess this at all, and just setting from environment variable only, as an opt-in.

[snogge]

The riscv32 variant does not look at the gnu_time64 config option, or does so in a safe way, so is_gnu_time_64 does not have to handle it.
An override to force either variant seems like a good idea to me.

[snogge]

I've updated #3175 to handle the cases listed above.

• The environment variable RUST_LIBC_TIME_BITS can be set to 32 to force 32-bit time_t
• gnu_time64_abi is not set for riscv32.

[safinaskar]

Possible solution to "how to distinguish 32 and 64" problem: rust-lang/rfcs#3716 ("[RFC] Target Modifiers")