Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: riscv-interrupt-{m,s} calling conventions #111891

Merged
merged 3 commits into from
Aug 10, 2023

Conversation

sethp
Copy link
Contributor

@sethp sethp commented May 23, 2023

Similar to prior support added for the mips430, avr, and x86 targets this change implements the rough equivalent of clang's __attribute__((interrupt)) for riscv targets, enabling e.g.

static mut CNT: usize = 0;

pub extern "riscv-interrupt-m" fn isr_m() {
    unsafe {
        CNT += 1;
    }
}

to produce highly effective assembly like:

pub extern "riscv-interrupt-m" fn isr_m() {
420003a0:       1141                    addi    sp,sp,-16
    unsafe {
        CNT += 1;
420003a2:       c62a                    sw      a0,12(sp)
420003a4:       c42e                    sw      a1,8(sp)
420003a6:       3fc80537                lui     a0,0x3fc80
420003aa:       63c52583                lw      a1,1596(a0) # 3fc8063c <_ZN12esp_riscv_rt3CNT17hcec3e3a214887d53E.0>
420003ae:       0585                    addi    a1,a1,1
420003b0:       62b52e23                sw      a1,1596(a0)
    }
}
420003b4:       4532                    lw      a0,12(sp)
420003b6:       45a2                    lw      a1,8(sp)
420003b8:       0141                    addi    sp,sp,16
420003ba:       30200073                mret

(disassembly via riscv64-unknown-elf-objdump -C -S --disassemble ./esp32c3-hal/target/riscv32imc-unknown-none-elf/release/examples/gpio_interrupt)

This outcome is superior to hand-coded interrupt routines which, lacking visibility into any non-assembly body of the interrupt handler, have to be very conservative and save the entire CPU state to the stack frame. By instead asking LLVM to only save the registers that it uses, we defer the decision to the tool with the best context: it can more accurately account for the cost of spills if it knows that every additional register used is already at the cost of an implicit spill.

At the LLVM level, this is apparently implemented by marking every register as "callee-save," matching the semantics of an interrupt handler nicely (it has to leave the CPU state just as it found it after its {m|s}ret).

This approach is not suitable for every interrupt handler, as it makes no attempt to e.g. save the state in a user-accessible stack frame. For a full discussion of those challenges and tradeoffs, please refer to the interrupt calling conventions RFC.

Inside rustc, this implementation differs from prior art because LLVM does not expose the "all-saved" function flavor as a calling convention directly, instead preferring to use an attribute that allows for differentiating between "machine-mode" and "superivsor-mode" interrupts.

Finally, some effort has been made to guide those who may not yet be aware of the differences between machine-mode and supervisor-mode interrupts as to why no riscv-interrupt calling convention is exposed through rustc, and similarly for why riscv-interrupt-u makes no appearance (as it would complicate future LLVM upgrades).

@rustbot
Copy link
Collaborator

rustbot commented May 23, 2023

r? @jackh726

(rustbot has picked a reviewer for you, use r? to override)

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels May 23, 2023
@rustbot
Copy link
Collaborator

rustbot commented May 23, 2023

Some changes occurred in compiler/rustc_codegen_cranelift

cc @bjorn3

Some changes occurred to MIR optimizations

cc @rust-lang/wg-mir-opt

Some changes occurred in src/tools/rust-analyzer

cc @rust-lang/rust-analyzer

These commits modify compiler targets.
(See the Target Tier Policy.)

@rust-log-analyzer

This comment has been minimized.

@rust-log-analyzer

This comment has been minimized.

@rust-log-analyzer

This comment has been minimized.

@rust-log-analyzer

This comment has been minimized.

@rustbot rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels May 25, 2023
@sethp
Copy link
Contributor Author

sethp commented May 25, 2023

So the riscv UI test failures have taken some debugging. It seems that these are the first riscv-specific UI tests, which have been subtly broken since #110884: on any version of LLVM that does not recognize that feature (i.e. any before rust-lang/llvm-project@da7c712, which appears to have landed in ~15.0.0), the act of filtering down the total list of features known to rustc attempts to apply each feature to a short-lived feature set, causing LLVM to emit this error as a side effect:

+ '+unaligned-scalar-mem' is not a recognized feature for this target (ignoring feature)

Since this goes directly to stderr, the UI tests pick it up and observe "hey, on LLVM 14 there's this extra output that's not in the .stderr file": and on versions that do support the feature, no such message is printed. (I suspect, but have not confirmed, that this means every feature listed in that file is supported across every tested version of LLVM, except apparently for this one).

As a potential fix, in 70b2188 I've modified the C++ side of the binding to search the feature table directly rather than delegating to the side-effecting class member, but this requires access to the feature table. Accessors were added to the Rust fork in rust-lang/llvm-project@da7c712 , but the failing test is being built against Ubuntu's system LLVM that lacks any rust-specific patches, so that accessor doesn't exist. Instead, I've had to resort to, uh, cleverness to access the private field in a way that the internet claims ought to be portable across LLVM and language versions, and C++ compilers. Feedback gladly accepted on where else there might be some leverage here that I'm missing, either within the test suite or elsewhere.

@rust-log-analyzer

This comment has been minimized.

@sethp sethp force-pushed the feat/riscv-isr-cconv branch from 70b2188 to 605858b Compare May 26, 2023 15:22
@rust-log-analyzer

This comment has been minimized.

@jackh726
Copy link
Member

This likely needs some lang and/or compiler team okay

@sethp
Copy link
Contributor Author

sethp commented Jun 11, 2023

Thanks for taking a look, @jackh726! Is this something the lang and/or compiler teams are going to see, or is there an action I should be taking to discuss it with them?

@jackh726
Copy link
Member

I'm going to start by tagging @rust-lang/lang. I guess adding a new calling convention needs at least someone on lang to okay (it's behind a feature gate, so likely doesn't need full FCP). For compiler, not sure - maybe an MCP is warranted, but probably not.

For lang team: Is it okay to add a feature gate for new risvc-interrupt-{m,s} calling conventions?

@jackh726
Copy link
Member

jackh726 commented Jul 1, 2023

Lang nominating. See above comment.

@jackh726 jackh726 added the I-lang-nominated Nominated for discussion during a lang team meeting. label Jul 1, 2023
@nikomatsakis
Copy link
Contributor

nikomatsakis commented Jul 25, 2023

@jackh726 this sounds like an experimental feature gate, although I think it would ultimately be covered under rust-lang/rfcs#3246. We discussed in meeting and are happy.

@tmandry can serve as liaison.

@joshtriplett joshtriplett removed the I-lang-nominated Nominated for discussion during a lang team meeting. label Jul 25, 2023
src/ci/run.sh Outdated Show resolved Hide resolved
sethp added a commit to sethp/rust that referenced this pull request Jul 29, 2023
Previously, re-running `run.sh` in the same container would fail at the useradd step, because the user already exists. Instead, change that step to "create if not exists" semantics to ease interactive debugging of CI failures.

Split out from rust-lang#111891 per request by @jackh726
@rust-log-analyzer

This comment has been minimized.

@sethp sethp force-pushed the feat/riscv-isr-cconv branch from a550ef7 to 1350d89 Compare July 29, 2023 18:01
@sethp sethp requested a review from jackh726 August 5, 2023 01:03
@jackh726
Copy link
Member

jackh726 commented Aug 5, 2023

Can you squash your commits?

@sethp sethp force-pushed the feat/riscv-isr-cconv branch from 7112eb8 to 4182a0e Compare August 5, 2023 21:41
@sethp
Copy link
Contributor Author

sethp commented Aug 5, 2023

Can you squash your commits?

Sure, I did a little bit of editing down to cut out the stuff that's only useful for test usability. Let me know if you want me to do any more!

tests/ui/proc-macro/meta-macro-hygiene.rs Show resolved Hide resolved
@@ -177,11 +177,64 @@ extern "C" void LLVMTimeTraceProfilerFinish(const char* FileName) {
GEN_SUBTARGETS
#undef SUBTARGET

// Thanks to https://gist.github.com/dabrahams/1528856
namespace access_retrofit {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this change nice-to-have, or required? If it's just nice-to-have, I'd rather it be split into a separate PR, since I'm not personally comfortable reviewing this alone.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wrote more about this in #111891 (comment) and 7161c5e : prior to this change, it was impossible to introduce riscv UI tests to the compiler; I'm not super confident either in whatever is happening here, but it does seem important to me to include in this context. Could we ask someone else to help us take a look here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's tag @nikic and @rust-lang/wg-llvm

It might still be worth to split this into a separate PR that can have a separate reviewer assigned.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is obviously not acceptable. You probably no longer need this because LLVM 14 is no longer supported.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah! Dropping LLVM 14 support may indeed decrease the urgency here, though I'm not totally comfortable abandoning it entirely. The side-effect-ful filter call will cause any architecture's UI tests to fail any time any supported version of LLVM doesn't recognize a feature that rustc knows about, thereby seeming to defeat the purpose of the feature filter (at least as far as UI tests go)?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for removing this change here. If you feel strongly about this, a separate PR is likely the right way to go about this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, the tests pass, so the good news is that means there's currently no features at the LLVM level that exist in any newer versions, which does indeed mean these changes can be decoupled. I did want to leave it up to @nikic what the resolution ought to be: I searched for a few hours before coming up with the approach here, and this was the only one I found that worked against system-provided LLVM. I'm open to an alternative approach, but if y'all would rather do nothing, then perhaps an issue documenting the problem would be a more fitting outcome than a PR with this change in it. What do you think?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opening an issue to document the problem is a reasonable approach to move forward with this.

compiler/rustc_feature/src/active.rs Outdated Show resolved Hide resolved
bors added a commit to rust-lang-ci/rust that referenced this pull request Aug 7, 2023
fix(ci): Ensure idempotence of user creation

Previously, re-running `run.sh` in the same container would fail at the useradd step, because the user already exists. Instead, change that step to "create if not exists" semantics to ease interactive debugging of CI failures.

Split out from rust-lang#111891 per request by `@jackh726`
@bors
Copy link
Contributor

bors commented Aug 7, 2023

☔ The latest upstream changes (presumably #114569) made this pull request unmergeable. Please resolve the merge conflicts.

github-actions bot pushed a commit to rust-lang/miri that referenced this pull request Aug 8, 2023
fix(ci): Ensure idempotence of user creation

Previously, re-running `run.sh` in the same container would fail at the useradd step, because the user already exists. Instead, change that step to "create if not exists" semantics to ease interactive debugging of CI failures.

Split out from rust-lang/rust#111891 per request by `@jackh726`
@sethp sethp force-pushed the feat/riscv-isr-cconv branch from 4182a0e to 01ae0a6 Compare August 8, 2023 22:16
@rust-log-analyzer

This comment has been minimized.

@sethp sethp force-pushed the feat/riscv-isr-cconv branch from 01ae0a6 to dbc030f Compare August 8, 2023 23:14
@jackh726
Copy link
Member

jackh726 commented Aug 8, 2023

Please remove the llvm changes/revert from the commit history, and squash the final test change commit into the previous ones. After that, I think this should be good to go.

sethp added 3 commits August 8, 2023 18:09
Similar to prior support added for the mips430, avr, and x86 targets
this change implements the rough equivalent of clang's
[`__attribute__((interrupt))`][clang-attr] for riscv targets, enabling
e.g.

```rust
static mut CNT: usize = 0;

pub extern "riscv-interrupt-m" fn isr_m() {
    unsafe {
        CNT += 1;
    }
}
```

to produce highly effective assembly like:

```asm
pub extern "riscv-interrupt-m" fn isr_m() {
420003a0:       1141                    addi    sp,sp,-16
    unsafe {
        CNT += 1;
420003a2:       c62a                    sw      a0,12(sp)
420003a4:       c42e                    sw      a1,8(sp)
420003a6:       3fc80537                lui     a0,0x3fc80
420003aa:       63c52583                lw      a1,1596(a0) # 3fc8063c <_ZN12esp_riscv_rt3CNT17hcec3e3a214887d53E.0>
420003ae:       0585                    addi    a1,a1,1
420003b0:       62b52e23                sw      a1,1596(a0)
    }
}
420003b4:       4532                    lw      a0,12(sp)
420003b6:       45a2                    lw      a1,8(sp)
420003b8:       0141                    addi    sp,sp,16
420003ba:       30200073                mret
```

(disassembly via `riscv64-unknown-elf-objdump -C -S --disassemble ./esp32c3-hal/target/riscv32imc-unknown-none-elf/release/examples/gpio_interrupt`)

This outcome is superior to hand-coded interrupt routines which, lacking
visibility into any non-assembly body of the interrupt handler, have to
be very conservative and save the [entire CPU state to the stack
frame][full-frame-save]. By instead asking LLVM to only save the
registers that it uses, we defer the decision to the tool with the best
context: it can more accurately account for the cost of spills if it
knows that every additional register used is already at the cost of an
implicit spill.

At the LLVM level, this is apparently [implemented by] marking every
register as "[callee-save]," matching the semantics of an interrupt
handler nicely (it has to leave the CPU state just as it found it after
its `{m|s}ret`).

This approach is not suitable for every interrupt handler, as it makes
no attempt to e.g. save the state in a user-accessible stack frame. For
a full discussion of those challenges and tradeoffs, please refer to
[the interrupt calling conventions RFC][rfc].

Inside rustc, this implementation differs from prior art because LLVM
does not expose the "all-saved" function flavor as a calling convention
directly, instead preferring to use an attribute that allows for
differentiating between "machine-mode" and "superivsor-mode" interrupts.

Finally, some effort has been made to guide those who may not yet be
aware of the differences between machine-mode and supervisor-mode
interrupts as to why no `riscv-interrupt` calling convention is exposed
through rustc, and similarly for why `riscv-interrupt-u` makes no
appearance (as it would complicate future LLVM upgrades).

[clang-attr]: https://clang.llvm.org/docs/AttributeReference.html#interrupt-risc-v
[full-frame-save]: https://github.com/esp-rs/esp-riscv-rt/blob/9281af2ecffe13e40992917316f36920c26acaf3/src/lib.rs#L440-L469
[implemented by]: https://github.com/llvm/llvm-project/blob/b7fb2a3fec7c187d58a6d338ab512d9173bca987/llvm/lib/Target/RISCV/RISCVRegisterInfo.cpp#L61-L67
[callee-save]: https://github.com/llvm/llvm-project/blob/973f1fe7a8591c7af148e573491ab68cc15b6ecf/llvm/lib/Target/RISCV/RISCVCallingConv.td#L30-L37
[rfc]: rust-lang/rfcs#3246
These new interrupt calling conventions are not themselves stabilized,
but there are other unstable calling conventions present in the SMIR
mapping (e.g. AVR interrupts) and the mapping appears to be "complete"
so far, with no obvious way to represent unstable conventions separately
from the stable ones.
The change in 07f855d introduced a
trailing numeral of some kind after the `extern crate
compiler_builtins`, which appears to have caused at least two false
negatives (654b924 and 657fd24). Instead, this change normalizes the
test output to ignore the number (of symbols rustc recognizes?) to avoid
needing to re-`--bless` these two tests for unrelated changes.
@sethp sethp force-pushed the feat/riscv-isr-cconv branch from dbc030f to 26bd86d Compare August 9, 2023 01:10
@jackh726
Copy link
Member

jackh726 commented Aug 9, 2023

Thanks for your patience on this review!

@bors r+

@bors
Copy link
Contributor

bors commented Aug 9, 2023

📌 Commit 26bd86d has been approved by jackh726

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 9, 2023
@sethp
Copy link
Contributor Author

sethp commented Aug 9, 2023

Thanks @jackh726 for your help and guidance! star wars voice almooooost there, stay on target

bors added a commit to rust-lang-ci/rust that referenced this pull request Aug 9, 2023
…iaskrgr

Rollup of 6 pull requests

Successful merges:

 - rust-lang#110435 (rustdoc-json: Add test for field ordering.)
 - rust-lang#111891 (feat: `riscv-interrupt-{m,s}` calling conventions)
 - rust-lang#114377 (test_get_dbpath_for_term(): handle non-utf8 paths (fix FIXME))
 - rust-lang#114469 (Detect method not found on arbitrary self type with different mutability)
 - rust-lang#114587 (Convert Const to Allocation in smir)
 - rust-lang#114670 (Don't use `type_of` to determine if item has intrinsic shim)

Failed merges:

 - rust-lang#114599 (Add impl trait declarations to SMIR)

r? `@ghost`
`@rustbot` modify labels: rollup
@bors bors merged commit 7d78885 into rust-lang:master Aug 10, 2023
@rustbot rustbot added this to the 1.73.0 milestone Aug 10, 2023
bjorn3 pushed a commit to bjorn3/rust that referenced this pull request Sep 6, 2023
…iaskrgr

Rollup of 6 pull requests

Successful merges:

 - rust-lang#110435 (rustdoc-json: Add test for field ordering.)
 - rust-lang#111891 (feat: `riscv-interrupt-{m,s}` calling conventions)
 - rust-lang#114377 (test_get_dbpath_for_term(): handle non-utf8 paths (fix FIXME))
 - rust-lang#114469 (Detect method not found on arbitrary self type with different mutability)
 - rust-lang#114587 (Convert Const to Allocation in smir)
 - rust-lang#114670 (Don't use `type_of` to determine if item has intrinsic shim)

Failed merges:

 - rust-lang#114599 (Add impl trait declarations to SMIR)

r? `@ghost`
`@rustbot` modify labels: rollup
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants