Build Linux on M1 with macOS

Some time ago, I just used the CLion IDE from jetbrains. That IDE is capable of loading the kernel project, although the initial loading is quite slow. Clion supports uploading the project on every change to a remote host, and also calling make on the remote host. But the initial upload to a remote host took multiple hours and sometimes CLion lost its state, therefor requiering a full rebuild.

CLion also supports a rust language plugin. But this plugin is not based on rust-analyzer but on a costum rust AST parser, written in Kotlin. This results in the plugin only beeing able to load projects based on cargo, and is not able to read the rust-project.json file, that rust-analyzer supports. Playing a bit with that, I got the plugin to read a dummy Cargo.toml, and generate some autocompletion hints.

Another thing I briefly looked at is the Emacs TRAMP mode. This package allowes me to connectect to a remote host via ssh, and open a project like the linux kernel there.

According to guids I found online DOOM Emacs with TRAMP mode is able to use clangd on a remote host, but special config could be required and is required to also have rust-analyzer support.

Both methodes required some sort of remote host. Either on a remote host (which could even be x86_64 if required for something) or a local VM on the mac.

Searching a bit online, I found this KDK which seems to be able to build the linux kernel on the m1, targeting aarch64. But I never installed homebrew since I know of nix, and therfore could not really use the project.

But its is. nix wrappes all compilers with the wrapper, to link to packages in the local environment/shell. As make menuconfig depends on ncurses and the build process itself required some tools like libelf, I needed a nix environment which provides those packages. Entering this nix shell which also provided the linux headers from the KDK I executed make CC=clang ARCH=arm64 the build script greeted me with the error message stating that it found a mach-o file instead of an elf file. Looking at the kernel Makefiles I found that using clang with the arch arm64 passes --target aarch64-unknown-linux-gnu to clang, which hints LLVM to produce elf files, even on macOS, which usually uses mach-o files.

Using the Xcode provided clang this flag worked, and produced the expected elf file, but using the clang provided by nix still produced an mach-o file.

After some searching in the env, I found that the clang wrapper resets the --target flag, and always sets the default option for macOS.

After asking in IRC, I got the hint to use the unwrapped clang exposed as honest-clang. This resulted in the following nix-shell (already including dependencies for rust):

with import <nixpkgs> {};
let
	prefixBins = prefix: orig: runCommand "${orig.name}-prefix-${prefix}" { inherit prefix orig; } ''
		mkdir -p $out/bin
		for b in $(find "$orig/bin" -type f,l -maxdepth 1 ); do
			ln -s $b $out/bin/$prefix$(basename "$b")
		done
	'';

	inherit (pkgs.rustPlatform.rust) rustc;
in
stdenv.mkDerivation {
	name = "test";

	hardeningDisable = [ "all" ];

	RUST_LIB_SRC = rustPlatform.rustLibSrc;

	buildInputs = [ gcc libelf ncurses libyaml pkg-config openssl # dt-schema
# (python3.withPackages (pkgs: [pkgs.ply pkgs.GitPython])) ]
									llvmPackages.bintools clang
									rustc rust-bindgen rustfmt libiconv
								] ++
		[(runCommand "glibc-headers" {} ''
			mkdir -p $out/include/asm
			sed 's/^#include <features.h>/#include <stdint.h>/' < ${./frankenstein/elf.h} > $out/include/elf.h
			cat > $out/include/endian.h <<- EOF
				#define __LITTLE_ENDIAN 1234
				#define __BIG_ENDIAN 4321
				#define __BYTE_ORDER __LITTLE_ENDIAN
			EOF
			cat > $out/include/asm/types.h <<- EOF
				typedef char __s8;
				typedef unsigned char __u8;
				typedef short __s16;
				typedef unsigned short __u16;
				typedef int __s32;
				typedef unsigned int __u32;
				typedef long long __s64;
				typedef unsigned long long __u64;
				#define BITS_PER_LONG (__CHAR_BIT__ * __SIZEOF_LONG__)
			EOF
			touch $out/include/byteswap.h $out/include/asm/posix_types.h
		'')] ++ [
			(prefixBins "honest-" llvmPackages.clang-unwrapped)
			(prefixBins "honest-" llvmPackages.bintools-unwrapped)
			(prefixBins "honest-" rust-bindgen-unwrapped)
		];
}

(./frankenstein contains the headers from the Homebrew KDK)

From the IRC is also got a small shell wrapper, setting some always required make flags to make this work:

make CC=honest-clang-11 LD=honest-ld.lld AR=llvm-ar NM=llvm-nm STRIP=llvm-strip OBJCOPY=llvm-objcopy OBJDUMP=llvm-objdump READELF=llvm-readelf HOSTCC=clang HOSTCXX=clang++ HOSTAR=llvm-ar HOSTLD=ld.lld -j8 ARCH=arm64 "$@"

Seperating the HOST build tools, so the clang wrapper can find ncuses to build the menuconfig program and using the honest versions to build the actuall Kernel objects.

But using just this still did not let me compile rust code, as bindgen has a wrapper, which also overrited the target, and the kernel objects complaint about a missing arm neon header.

(This also required a path to scripts/mod/file2alias.c in the linux tree.)

Fixing all this I changed the clang-build helper script to this:

make CC=honest-clang LD=honest-ld.lld BINDGEN=honest-bindgen \
    AR=llvm-ar NM=llvm-nm STRIP=llvm-strip OBJCOPY=llvm-objcopy OBJDUMP=llvm-objdump READELF=llvm-readelf \
    HOSTCC=clang HOSTCXX=clang++ HOSTAR=llvm-ar HOSTLD=ld.lld \
    ARCH=arm64 \
    HOSTCFLAGS="-D_UUID_T -D__GETHOSTUUID_H" \
    KCFLAGS="-isystem /nix/store/an2h2ic7isza22j0dfdjixkjak849k0z-clang-11.1.0-lib/lib/clang/11.1.0/include/" \
    -j8 "$@"

This sadly has a hard coded path into the nix store rigth now, which I did not fix yet. The path provides the missing arm_neon.h which was still required for me to build the kernel.

Using this (and nixpkgs staging at time of writing as rustc 1.62.0 is required) the linux kernel build the CI config used for rust for linux in about 1 minute 30 seconds.

[nix-shell:/Volumes/build/kernel/linux]$ time ./clang-build

real	1m22.354s
user	0m59.946s
sys     0m43.720s