How to embed 2 types in user-ptr, when one of them is lifetime-constrained by the other?

[justinbarclay]

I'm trying to convert a rust library to be able to run in Emacs but I've run into an issue when trying to upgrade from 0.9.0 to 0.10.0+. I get an error about rust being able to infer appropriate lifetimes for a struct. This code compiles perfectly fine in 0.9.0, but once you added the static lifetimes requirement to user_ptrin0.10.0`, Rust has refused to compile. Unfortunately, I haven't been able to figure out a work around and I was wondering if you had any suggestions for how I could resolve or work around this issue.

If you would like to see where this is happening in my code, you can find it here. But I've created a minimal program that recreates the same error message. Which can be found below:

I think the issue is because I've got a struct that has an explicit lifetime 'a, and it's getting this lifetime from another user_ptr being passed into the function and rust is getting confused about how to ensure that the struct with lifetime 'a has a static lifetime.

#[macro_use]
extern crate emacs;

use emacs::{Result};
use std::borrow::Cow;
struct Foo{
  string: String
}

struct Bar <'a>{
  sub_str: Cow<'a, str>,
}

#[defun(user_ptr)]
fn run_function(foo: &Foo)-> Result<Bar>{
  Ok(Bar{
    sub_str: Cow::from(foo.string),
  })
}

error[E0495]: cannot infer an appropriate lifetime for autoref due to conflicting requirements
--> src/main.rs:15:17
|
15 | fn run_function(foo: &Foo)-> Result{
| ^^^
|
note: first, the lifetime cannot outlive the anonymous lifetime #1 defined on the function body at 14:1...
--> src/main.rs:14:1
|
14 | #[defun(user_ptr)]
| ^^^^^^^^^^^^^^^^^^
note: ...so that reference does not outlive borrowed content
--> src/main.rs:15:17
|
15 | fn run_function(foo: &Foo)-> Result{
| ^^^
= note: but, the lifetime must be valid for the static lifetime...
note: ...so that the type Bar<'_> will meet its required lifetime bounds
--> src/main.rs:15:30
|
15 | fn run_function(foo: &Foo)-> Result{

[ubolonton]

The change was introduced in 0.10.0. Before that, the following code could be compiled, but would result in undefined behavior:

struct Foo {
    string: String,
}

struct Bar<'a> {
    sub_str: Cow<'a, str>,
}

#[defun(user_ptr)]
fn make_foo(string: String) -> Result<Foo> {
    Ok(Foo { string })
}

#[defun(user_ptr)]
fn make_bar(foo: &Foo) -> Result<Bar> {
    Ok(Bar { sub_str: Cow::from(&foo.string) })
}

#[defun]
fn use_bar(bar: &Bar) -> Result<String> {
    match bar.sub_str {
        Cow::Borrowed(s) => Ok(s.to_owned()),
        _ => unreachable!()
    }
}

The undefined behavior can be seen with something like the following:

(let ((bar (let* ((foo (t/make-foo "abc abc"))
                  (bar-with-foo (t/make-bar foo)))
             (message "before: %s" (t/use-bar bar-with-foo))
             bar-with-foo)))
  (garbage-collect)
  (message "after : %s" (t/use-bar bar)))

On my machine, it printed this:

before: abc abc
after : <

The problem is that bar keeps a reference to foo.string, so it cannot outlive foo. However, foo is owned by the GC, because it's embedded in a user-ptr. Its liveness is decided by the GC. When foo is garbage-collected, the reference bar keeps becomes invalid.

It can show up more easily (with a segfault) if the type being referenced has more complex Drop logic.

Making it work requires tracking the reference at run time, in a way that's visible to the GC. This can be done with an Rc (as done here), which involves a bit of tricky unsafe code. Alternatively, you can use make_global_ref (not yet exposed in this binding). However, that may require careful disposal to avoid memory leaks, as there's an issue with free_global_ref (described in #8).

[justinbarclay]

Awesome, this helped a lot.

However, I have ran into one issue. I went with your first suggestion, using Rc, and I was able to make it a lot further! Rust was able to compile the library, but unfortunately when I tried to call my execute function, emacs-module-rs tosses an error saying

Wrong type user-ptr: "expected: Rc"

type AliasedRequest<'a> = &'a SharedRequest;

#[defun(user_ptr, mod_in_name = false)]
fn execute(shared_request: AliasedRequest) -> Result<WrappedAnswer> {
  let answer = parinfer::rc_process(&shared_request);
  let wrapped_answer = unsafe{WrappedAnswer::new(shared_request.clone(), answer)};
  Ok(wrapped_answer)
}

I'm wondering if you have any ideas of why it's expecting an Rc and not getting one? Is it because it's a reference to an Rc and not an Rc in and of itself?

If it helps, I was able to replicate this error here

[ubolonton]

#[defun(user_ptr)] is short for #[defun(user_ptr(refcell))], which means wrapping the returned value in a RefCell, then embedding it. It's for embedding arbitrary types, with mutability enabled.

SharedRequest is an Rc, which has an IntoLisp implementation that does automatic "user-ptr-embedding", so you can just use #[defun] (or #[defun(user_ptr(direct))], to be explicit).

[ubolonton]

https://github.com/jpernst/rental is possibly a good option to avoid having to write unsafe code
directly (letting the rental! macro deal with it). I haven't looked deeply into it yet.

[justinbarclay]

That worked! Thank you so much for your patience in walking me through this.

[ubolonton]

I tried out the rental crate. It seems to be a good fit for this use case.

The basic idea is similar to the unsafe hacks above: eliding the lifetime parameter. The good thing is that you can push the burden of safety proof to the rental crate, so you don't have to use unsafe in your own code:

#[macro_use]
extern crate rental;

#[derive(Debug, Clone)]
pub struct Foo {
    string: String,
}

#[derive(Debug, Clone)]
pub struct Bar<'a> {
    sub_str: Cow<'a, str>,
}

#[defun]
fn make_foo(string: String) -> Result<Rc<Foo>> {
    Ok(Rc::new(Foo { string }))
}

rental! {
    pub mod rental {
        use std::rc::Rc;

        #[rental(debug, clone, covariant)]
        pub struct Bar {
            foo: Rc<super::Foo>,
            bar: super::Bar<'foo>,
        }
    }
}

#[defun(user_ptr)]
fn make_bar(foo: Value) -> Result<rental::Bar> {
    let rc: &Rc<Foo> = foo.into_rust()?;
    Ok(rental::Bar::new(
        rc.clone(), |foo| Bar { sub_str: Cow::from(&foo.string) }
    ))
}

#[defun]
fn use_bar(bar: &rental::Bar) -> Result<String> {
    match bar.suffix().sub_str {
        Cow::Borrowed(s) => Ok(s.to_owned()),
        _ => unreachable!()
    }
}

I'll update the guide with this soon.