tschuett:

As an outsider: Could you teach libObject in LLVM to read rlib files and teach the lld linker in LLVM to link rlib files?

The issue is not that lld or other linkers can't read rlib files in their current implementation (they are standard ar archives), but that this isn't guaranteed to be the case and that there are object files and link args essential for linking that rustc only generates when it invokes the linker by itself. Without these object files liballoc and libstd simply fail to link due to missing symbols and without the link args we may be missing essential libraries or in the past fail to link due to cyclic dependencies (we solved this by producing a symbols.o file referencing all symbols we need in advance, but that file is again only produced right before rustc invokes the linker)

One purly API related alternative could also consider is, rather them adding a new compilation flag, rlib-version , a new crate type, e.g. ( staticlib-nobundle or object-rlib ) is added that has the semantics of the v0-rlibs described in this proposal (including the ability to be linkable into other rust code.). This would avoid having to add an API switch, only usefull for one single crate type. Also this would allow for a future use case where users want to generate a MIR rlib first and the build an v0-rlib from it.

Unrelated to this: Would there be any advantages of somehow matching up the layout to C++ compiled modules?

So, I'm going to preface this by saying that I am in favor of the general idea of a compatibility initiative, however there are some problems with just stablizing the rlib format.

As part of the lccc project, I have begun specifying a considerable portion of the associated formats (including rlibs and the rmanifest file) and abi details, and considerable is indeed the proper adjective here. To be useful, beyond for inspection by curious programmers (which far from requires a specified format), many portions of what are currently internal compiler details must be made public, at the very least on a by-version basis.

pcwalton:

A global symbol is a symbol with the BFD BSF_GLOBAL flag set. Those symbols, and only those symbols, that the target crate defines must be defined in the archive symbol table.

This definition does not address what must be a global symbol. Do #[inline] definitions need to be global? And of course, if they are, do they only need to be defined in the object that includes them (this falls short, as by design, they are generated in multiple cgus to be inlined, and are compiled as weak symbols that are presumably placed into a linkonce COMDAT group). What about (instantiated) generic symbols. What about destructors (the drop_in_place definition, not the Drop::drop function specifically). Additional shims? I think you'd find an exhaustive list to be nearly impossible to specify without nailing down many of the abi details. Even with those details, the lcrust abi lacks any sort of exhaustive list, and only has a few cases that it says "must be generated" (implying the symbol is global). It's entirely possible that a symbol could be "public" but not "global" and require rustc to make it available when linking downstream. This specification does not prevent that, and I would assume it would require a full ABI specification to do so.

pcwalton:

System linkers generally have no concept of "metadata" beyond the symbol table and ar pseudo-filesystem, so there would be nothing to specify even if we wanted to.

It's entirely possible that an .rmeta file may be an object file with the metadata in a section, and could contain salient definitions (including symbols containing no data, but are checked by code in other object files). This would preclude doing that.

nacaclanga:

Unrelated to this: Would there be any advantages of somehow matching up the layout to C++ compiled modules?

Are you talking about C++ standard modules? Or are you referring to compiled TU objects that get put into some container ("library")? All of these are platform specific (the former is even compiler-specific) and wholly unspecified in any language spec.

As far as I understand it C++20 modules as implemented by clang and gcc are basically dumps of the internal compiler structures of the respective C++ frontend. This format is fundamentally incompatible with most non-C++ languages, differs between clang and gcc and does not capture any of the complexity that makes it hard/impossible for rlibs to be linked without rustc wrapping the linker.

nacaclanga:

a new crate type, e.g. ( staticlib-nobundle or object-rlib ) is added that has the semantics of the v0-rlibs described in this proposal (including the ability to be linkable into other rust code.).

How does that differ from this proposal in practice? Other than trying to convince all the existing Rust tooling to emit a different --crate-type option to rustc ?

(I think a related issue is that the roles of the --crate-type and --emit options are pretty confused and entangled, and could do with a solid bout of rationalization.)

This would avoid having to add an API switch, only usefull for one single crate type.

I'm not sure what you mean by this - specifically how "API" comes into this, beyond the very general sense of an object file being part of the "API" of a toolchain.

jsgf:

(I think a related issue is that the roles of the --crate-type and --emit options are pretty confused and entangled, and could do with a solid bout of rationalization.)

There are three mutually exclusive crate type families: proc-macro , bin and the various library types. Selecting one family will have affects at the frontend, unlike --emit . --emit affects the backend with the various library crate types kind of being sub types of --emit link . Now this is not an entirely accurate view as which --emit and which library types are used can affect what the crate metadata contains and the library types have an effect on the crate metadata loading as well as some other things, but I think it is a usable mental model despite these limitations.

jsgf:

How does that differ from this proposal in practice? Other than trying to convince all the existing Rust tooling to emit a different --crate-type option to rustc ?

To make the stable rlib format useful you did have to have the standard library available in this format too. Cargo doesn't specifying the crate types of crates on the commandline as would be necessary for this. It hard codes them in Cargo.toml .

I'm not sure what you mean by this - specifically how "API" comes into this, beyond the very general sense of an object file being part of the "API" of a toolchain.

API is ment in the sense of "the choice of available command line arguments of rustc".

What I mean here is that the suggested API in the pre-RFC, suggests that -C rlib-version=v0 is somehow orthogonal to crate type, e.g. that something like: -crate-type=staticlib -C rlib-version=v0 could be somehow usefull. However choosing to specify rlib-version , somehow implies, -crate-type=rlib . My suggestion removes this "redundancy" by spliting the crate type rather them adding a new switch.

How does that differ from this proposal in practice? Other than trying to convince all the existing Rust tooling to emit a different --crate-type option to rustc ?

There should be little to no difference in practice, this is mostly an command API layout proposal. The effect on tools will also be similar, in both cases tools would need to be ajusted to make use of the stablized format.

nacaclanga:

However choosing to specify rlib-version , somehow implies, -crate-type=rlib .

I don't think so. I would expect that you can use RUSTFLAGS="-Crlib-version=v0" and then have all rlibs use this version, but have all non-rlib crate be unaffected.

InfernoDeity:

This definition does not address what must be a global symbol. Do #[inline] definitions need to be global? And of course, if they are, do they only need to be defined in the object that includes them (this falls short, as by design, they are generated in multiple cgus to be inlined, and are compiled as weak symbols that are presumably placed into a linkonce COMDAT group). What about (instantiated) generic symbols. What about destructors (the drop_in_place definition, not the Drop::drop function specifically). Additional shims? I think you'd find an exhaustive list to be nearly impossible to specify without nailing down many of the abi details. Even with those details, the lcrust abi lacks any sort of exhaustive list, and only has a few cases that it says "must be generated" (implying the symbol is global). It's entirely possible that a symbol could be "public" but not "global" and require rustc to make it available when linking downstream. This specification does not prevent that, and I would assume it would require a full ABI specification to do so.

So I've been thinking about this, and I think we can get away without specifying anything beyond the bare minimum of API details by doing the following, and no more:

Specify that, if any Rust crate B depends on Rust crate A, A must export whatever symbols are necessary for B to successfully link. We don't need to specify what those symbols are, or even whether they're weak or strong; that's an implementation detail that can and will change from rustc version to rustc version. We just need to say that enough symbols are exported, whatever those are, to avoid missing symbol errors in the final link.

Specify that if any Rust crate C depends on Rust crate A, there will never be duplicate symbol errors in the final link, as long as there is only one copy of C and A in the link line.

Specify that all pub extern functions are exported, under their mangled name (deferring to the name mangling RFC) or under their unmangled name if appropriate. This ensures that C++ code can link to Rust code as necessary.

This spec language is a trick that lets us avoid specifying ABIs.

It's entirely possible that an .rmeta file may be an object file with the metadata in a section, and could contain salient definitions (including symbols containing no data, but are checked by code in other object files). This would preclude doing that.

Can you elaborate as to what "checked by code in other object files" means? For any object files A and B (in this case B would be the .rmeta file), either A contains a symbol reference to B or it does not. If it does, then the linker will detect that and will find the object file for linking. If it doesn't, then there should be no problems.

pcwalton:

Specify that all pub extern functions are exported, under their mangled name (deferring to the name mangling RFC) or under their unmangled name if appropriate. This ensures that C++ code can link to Rust code as necessary.

I think only #[no_mangle] functions should be guaranteed to be exported. No function with a mangled name can be called anyway by non-rust code as the mangled name contains an unpredictable hash (both for the legacy and v0 mangling scheme). Furthermore cdylib only exports #[no_mangle] functions anyway.

pcwalton:

External tools such as linkers should ignore any non-object files, as their contents are unstable.

Several linkers entirely forbid non-object files. That is the reason we wrap the crate metadata in an object file.

pcwalton:

The object files inside a version 0 rlib must collectively contain global definitions for all the non-generic functions and statics defined by the crate being compiled.

#[inline] functions are never exported from object files.

pcwalton:

The primary reason why unstable is left unspecified is so as not to preclude the possibility of MIR-only rlib s in the future.

In that case should we disallow mixing multiple rlib versions in the same crate graph?

pcwalton:

In particular, all supported targets begin their archive format with the string !<arch> followed by a newline character: i.e. the bytes 0x21 0x3C 0x61 0x72 0x63 0x68 0x3E 0x0A.

AIX begins it with <bigaf> . By the way should we guarantee a specific archive format if multiple are supported on a target? On almost all platforms we default to the gnu archive format (32/64bit variant depending on archive size), even on bsd targets that have their own archive format. Pretty much only macOS and AIX would use a different archive format.

pcwalton:

Any set of crates in .rlib format compiled by the same Rust compiler (including compiler version) must be linkable together as long as the following conditions are fulfilled:

a. For each crate, all dependencies of that crate must be in the set.

b. All conditions specified in the "additional linking requirements" section are met.

c. The set contains each .rlib file no more than once.

There are two exceptions to this rule:

(i) Multiple crates that define the same language item may not be linkable together.

(ii) Multiple crates that define identically-named items marked with #[no_mangle] may not be linkable together.

This misses #[global_allocator] and the alloc error handler. In addition it misses the rule that two libraries with identical crate name need to have different -Cmetadata arguments to disambiguate symbols.

pcwalton:

Definitions of additional std -internal symbols that the compiler generates calls to may be required in order to link a Rust target. The names of such symbols must begin with __ (a double underscore).

Some internal unmangled symbols don't start with __. For example rust_eh_personality. Not sure if any such symbols are generated by the compiler though.

pcwalton:

An issue regarding static initializers was raised during the discussion: they don't reliably work unless --whole-archive is provided when linking the rlib . However, --whole-archive is not available on AIX. AIX is currently not a supported platform for Rust, however; if and when it becomes one, the RFC for support for that platform can specify what to do here. Additionally, this is an issue that would be present regardless of whether the rlib format is specified.

It isn't an issue for rustc as it generates a symbols.o file containing references to all symbols exported by any rlib (with the exception of those defined by bundled static libs) to force all object files in all rlibs to be linked.

pcwalton:

It would also be incompatible with any other language wanting to "take over" linking in this way; only one language can be in charge of the last linking stage, and the advantage of system ld is that it's language-neutral.

What if rustc were to have a mode in which it becomes a transparent wrapper around whichever linker except for also handling rust rlibs such that you can just squeeze it in between the other language and gcc/clang? Then it did just be a matter of stacking the linker drivers for all involved languages on top of each other, right? Also the gcc/clang linker drivers are not language neutral. They contain a lot of code to handle C/C++ peculiarities (like determining which system libraries or compiler specific libraries (like libgcc_s or libcompiler_rt) to link or handling static initializers) before ever running the actual ld linker. This is why rustc uses gcc/clang rather than the linker directly.

By the way how should the standard library be handled? It can't be built on stable, yet it did need to use the v0 rlib version and the build system needs some way to get the location of standard library crates and their dependency lists.

The code for determining what libraries to link and with which linker arguments and what to object files generate is specific to each rustc version and generating those object files literally runs the codegen backend. However we need to work with gcc and clang versions that are both older and newer than rustc. Additionally I would expect toolchains to be mutually exclusive in clang, but rustc would need to be additive as you can still link C code. And finally gcc doesn't have this concept of runtime switchable toolchains.

This misses #[global_allocator] and the alloc error handler.

I recall that there have been discussions about generalizing that pattern to a sort of provided-impl mechanism where a crate can require that a type implementing some trait must exist and will be provided exactly once in the compilation graph, usually but not necessarily by the final bin crate.

Those would probably run into similar issues?