T-compiler Supplemental Tool Policy (Linker Scripts, etc.)

[workingjubilee]

Proposal

Periodically, people use the Rust toolchain with binary-manipulating tools that we do not ship. In addition, tools that we do ship are not always developed in an "in-house" manner. Some targets, in order to perform realistic builds for them, require a complex user-provided tool, like a linker script, to be executed in order to produce a working binary. Others are more predictable but no less exciting, like the BOLT post-link-optimizer.

There are many operational semantic concerns that these tools raise, because they manipulate a binary's contents in poorly-specified ways. This may have undermined the assumptions that were made as part of the compilation process. None of these concerns are easy to address.

Fortunately, I believe we do not yet have to address any of them directly! They are a distraction from a more fundamental decision that T-compiler can make: a more boring, fundamental correctness requirement on their inputs and outputs, and a contract about our own. I propose we adopt something in the mold of the following Supplemental Tool Policy, which is intended to be more of a formal crystallization of something like our current informal policy, rather than a change. The primary intent is to provide a default answer to many tiny triaging decisions about whether or not we should do something or consider it an open issue, so that we can close issues without having to defer decisions to meetings potentially several thousand times, reserving triage meeting time for more ambiguous or high-touch cases.

tl;dr: "If you ship a broken strip, we may treat it as damage and route around it. If you throw in a bad linker script, then you break it, you bought it. If you want the compiler sandboxed, then you can do it yourself. If you want a diagnostic in LLD or something and upstream rejects it, then tough luck, I guess?"

Supplemental Tool Policy

T-compiler's mandate is to do what is necessary to ship a functioning Rust compiler (rustc), but rustc has never worked in isolation. It has always required additional tools to use, whether provided with the compiler toolchain or expected in the environment. The following is an attempt to clarify some of the expectations around rustc and the tools it may be used with or require, and make explicit some expectations that are already implicit.

• To "invoke" a "compiler tool" should be interpreted in the broad sense of running the code of any artifact, especially those shipped as part of the Rust toolchain, as part of the process of compiling and linking code into fully-functional binary executables or libraries.
• A "provided binary tool" is a compiler tool that is shipped as part of the Rust toolchain with the support of T-compiler and T-release for the purpose of emitting or editing binary objects. This includes examples like rustc, rust-lld, and rust-objcopy, amongst others. Notably, this includes provided binary tools developed by the Rust Project and provided binary tools developed by external sources.
• A "supplemental binary tool" is a compiler tool that we do not ship, but that may be used independently or in direct conjunction with provided binary tools to emit or edit binary objects. This includes examples like ld.bfd, link.exe, llvm-bolt, and scripts using the Link Editor Command Language.
• The "compilation environment" is the state of the host system that a Rust toolchain is invoked on, including things like the filesystem and environment variables.
• Any provided compiler tool of the Rust toolchain may implicitly invoke a supplemental binary tool without requiring explicit user input, by examining the compilation environment. This does not alter any requirements for supplemental binary tools.
• A "good-faith" assumption may be made about supplemental binary tools, regardless of whether they are available in the compilation environment or provided by explicit arguments to compiler tools. This means that e.g. a path set for the environment variable $CC may be assumed to be a useful C compiler, or a program named link.exe may be assumed to be the program provided by Microsoft. No attempt is required to verify the correctness of these assumptions, only a reasonable attempt to correctly locate the tools in question.
• If the output of a supplemental binary tool is used as an input to any other binary tool used as part of the compilation process, then it must remain a valid binary object in the input's expected object file format.
• The final output of the compilation process shall include a valid binary object for applicable output object file formats (including e.g. none if a compiler argument indicates to not emit any). These are not required to fulfill additional criteria that a supplemental binary tool may expect of such binary objects if such exceed the minimum spec of the format, even if other toolchains produce binaries that meet such criteria, except insofar as T-compiler prioritizes such.
• For some targets, their object file formats are poorly-understood and poorly-specified. In the case of ambiguity, this policy may be interpreted as narrowly or broadly as T-compiler pleases. Further, a binary object used as an input to the compilation process may be expected to fulfill additional requirements beyond those of strictly adhering to an object file format's specification to be considered valid, such as being a dynamically loadable or executable object, having a specific section, or similar.
• When it becomes apparent an assumption the compiler makes is not obvious (e.g. that a tool is at least a specific version, and it is not "the minimum version shipped with the operating system") then T-compiler should document such before resolving an issue dependent on it. When an assumption the compiler makes becomes infeasible or unrealistic to continue holding, we may revisit it.

For all cases where some behavior is not required, but remains desirable, it may still be pursued at the discretion of T-compiler. It's a guideline, not a straitjacket.

Mentors or Reviewers

uhh I guess I'd PR this to the forge or dev guide or something...? does that still require reviewers?

Process

The main points of the Major Change Process are as follows:

• File an issue describing the proposal.
• A compiler team member or contributor who is knowledgeable in the area can second by writing @rustbot second.
  • Finding a "second" suffices for internal changes. If however, you are proposing a new public-facing feature, such as a -C flag, then full team check-off is required.
  • Compiler team members can initiate a check-off via @rfcbot fcp merge on either the MCP or the PR.
• Once an MCP is seconded, the Final Comment Period begins. If no objections are raised after 10 days, the MCP is considered approved.

You can read more about Major Change Proposals on forge.

Comments

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

[rustbot]

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

Concerns or objections to the proposal should be discussed on Zulip and formally registered here by adding a comment with the following syntax:

@rfcbot concern reason-for-concern 
<description of the concern> 

Concerns can be lifted with:

@rfcbot resolve reason-for-concern 

See documentation at https://forge.rust-lang.org

cc @rust-lang/compiler