Add Rust for Linux auto CI job

[Kobzol]

This PR adds an auto CI job that checks if Rust for Linux (RfL) still compiles with the latest version of the compiler and the standard library. If not, we should ideally ping the RfL ping group.

[ojeda]

Thanks for this again! Some more comments/thoughts below...

I guess depending on how things go, i.e. how many failures we get and where they fail most, we could consider reducing or increasing the scope.

[Kobzol]

I think that the last thing required here is to select some reasonable initial commit. @ojeda any suggestions? I tried the commit that you have sent me before, but it was not found in the repo I think (maybe git just couldn't work with the shorter SHA though).

Is there any more up-to-date commit that I could use instead of master?

[ojeda]

I tried the commit that you have sent me before, but it was not found in the repo I think (maybe git just couldn't work with the shorter SHA though).

The commit exists, but you probably need something like:

mkdir linux
git -C linux init
git -C linux remote add origin https://github.com/torvalds/linux.git
git -C linux fetch --depth 1 origin 8f5b5f78113e881cb8570c961b0dc42b218a1b9e
git -C linux checkout FETCH_HEAD

instead of --branch. Another way is using the archive URL, e.g. https://github.com/torvalds/linux/archive/8f5b5f78113e.tar.gz.

Also, later today Linus will likely tag v6.10-rc1, which we could also use.

[ojeda]

Is there any more up-to-date commit that I could use instead of master?

Not sure what you mean, master is the latest in Linus' (and generally should not be used, since it will move). Do you mean something in the RFL repository?

[Kobzol]

Not sure what you mean, master is the latest in Linus' (and generally should not be used, since it will move). Do you mean something in the RFL repository?

Sorry, I meant "what is the most up-to-date commit SHA that works with RfL". master is not a stable ref, ofc.

[ojeda]

Both should work (i.e. the one I gave you as well as the current master, i.e. c13320499ba0efd93174ef6462ae8a7a2933f6e7).

[ojeda]

If you want to use the RFL repository as the base URL, I can push those commits there too.

[Kobzol]

No need for now, if it works with upstream Linux. I changed the commit SHA (thanks for the code to checkout a single commit!), let's see if everything still works.

[ojeda]

My pleasure!

[Kobzol]

Ok, looks like it is working, this should be ready for a review now.

r? @Mark-Simulacrum

@rustbot ready

Successful CI run: https://github.com/rust-lang/rust/actions/runs/9243326050/job/25427322274?pr=125209 (~12 minutes on a 8 core CI runner).

[vincenzopalazzo]

I am not an infra expert, but this patch LGMT.

[Mark-Simulacrum]

Is the choice of LLVM important / why are we using LLVM 17 here vs. 18 (current latest release?) And why use an external LLVM at all?

[ojeda]

I asked Jakub about this part when we working on this; and he said it was taken from the original job, but that it would be useful to avoid building our own LLVM. I don't think the version matters much from the kernel point of view, especially for the few things we are actually doing here.

[Mark-Simulacrum]

If this isn't part of PR CI (which I think makes sense, hopefully we're not breaking RfL that often that we want the early warning), then building or not building LLVM shouldn't really matter since that'll be cached anyway.

[Kobzol]

Currently the build takes about 12 minutes, even a fully (sccache-)cached LLVM build will take similar time (at least 10 minutes including linking, IIRC), so it would double the build duration. But if you don't mind that, I can switch it to locally built LLVM. Or we can just use download-ci-llvm.

[Mark-Simulacrum]

Test builders are usually using download-ci-llvm today, so I think we'd get the faster build?

[Mark-Simulacrum]

This step worries me. We're using a random stable here that's going to get bumped whenever this Docker image gets rebuilt, not at some "good" time. Can we pin this to a particular stable, similar to the pinned LINUX_VERSION above?

Or actually, maybe this stable isn't used for anything? If so, can we explicitly uninstall it?

[ojeda]

If it is not used, it could perhaps be a good idea to use a given stable one anyway for the step of building bindgen-cli, especially if we find it does give trouble. Otherwise, yeah, uninstalling it would be best.

If it is used, it would be nice to add a comment to clarify.

[Kobzol]

Rustup is used for two things:

1. Allow easy installation of bindgen
2. Have an easy way of providing our built rustc to RfL

We don't really need the stable toolchain for anything, so I can change this to --default-toolchain none.

[Mark-Simulacrum]

It looks like this is largely copied from the llvm-17 builder. Can we make note of that? I wonder if we should be reusing the builder/Dockerfile somehow...

[Kobzol]

It is indeed copied from there. I plan to separate the Docker images from the scripts that we run in CI scripts, but that is a non-trivial refactoring, I'd prefer not to block this PR on that.

[Mark-Simulacrum]

It sounds like if we switch to download-ci-llvm or sccache, we'd avoid most of the copying, right?

[Kobzol]

Well, we'll still need to copy-paste some Dockerfile, since we don't really have a way to share Dockerfiles at the moment, right?

[Mark-Simulacrum]

Well, yes, but at least there's less going on. e.g., src/ci/docker/host-x86_64/x86_64-gnu/Dockerfile looks "simpler" to me and the stuff there that's shared is already copied into lots of files. So I think it would be simpler, and also doesn't need edits as we continue advancing the llvm versions we support.

[Kobzol]

Ok, fair enough, I'll reuse some simpler Dockerfile then.

[Kobzol]

Although I remember now that the RfL build did actually require some extra tools that weren't even in the original LLVM 17 Dockerfile, so it probaly won't be much shorter.