OS Overview - XNU
"X is not UNIX" apparently. My thought is that X stands for Mac OS X, so XNU stands for OS X is not UNIX. That's something that's confused me, and probably isn't the best way to begin this section so let me try that again. XNU is an acronym for "X is not UNIX", it's also the Kernel of Apple's *OS operating systems, it's also something i'm significantly interested in so writing this should be as fun as reading it.
XNU is a combination of a Monolithic and Micro Kernel. A Monolithic Kernel is a architecture where the entire operating system runs in kernel space, this includes device drivers, filesystems, etc. The advantages of Monolithic kernel's are that they're able to load executable modules at runtime. XNU get's its Monolithic components from BSD, more on that later. A Microkernel is a architecture where the core functionality is isolated from everything else. The advantages of a Microkernel are that there are separate kernel and user spaces.
XNU is quite cool in the fact that it combines the advantages of these two kernel architectures into one. XNU uses OSFMK 7.3 (Mach) and FreeBSD 4.3 to gain the best of both worlds, in this case the best of Micro and Monolithic Kernel architectures. XNU also includes a C++ API called IOKit that's used for writing device drivers.
Some History
XNU was originally developed for the NeXTStep operating system by NeXT and used 4.3BSD and Mach 2.5 (OSF). Also, IOKit wasn't always called IOKit, it was originally called Driver Kit and was written in Objective-C. When Apple bought NeXT in March of 1998, they changed and improved a great deal of XNU. They upgraded Mach 2.5 (OSF) to OSMFK 7.3, upgraded BSD to FreeBSD, rewrote DriverKit in C++ and renamed it IOKit and finally, begun Darwin.
XNU is also an open-source project. The source code can be viewed at opensource.apple.com or github.com/apple/darwin-xnu. Recently, Apple have been very quick to release the source for XNU and, as of September 28 2017, both x86_64 and ARM components of XNU are available.
Building XNU
Now, as i'm writing about XNU, feel like i haven't wrote enough and have struggled with the lack of guides for this in the past, i shall explain how to compile XNU.
Step 1
First, we need a few things. Create a folder for all this because theres a lot and we don't want to get confused
$ cd Desktop/; mkdir BuildingXNU/; cd BuildingXNU/
There're a few packages we need: XNU, DTrace, AvailabilityVersions, libdispatch and libplatform. We can get these with the following commands
$ curl -O https://opensource.apple.com/tarballs/xnu/xnu-<VERSION>.tar.gz
$ curl -O https://opensource.apple.com/tarballs/dtrace/dtrace-<VERSION>.tar.gz
$ curl -O https://opensource.apple.com/tarballs/AvailabilityVersions/AvailabilityVersions-<VERSION>.tar.gz
$ curl -O https://opensource.apple.com/tarballs/libdispatch-<VERSION>.tar.gz
$ curl -O https://opensource.apple.com/tarballs/libplatform-<VERSION>.tar.gz
Now, there is 2 options for the next part, 1 of which never works for me but you can try anyway. The first is to extract all these tar's via shell
$ tar xvf *.tar.gz
Or just do it via Finder.app (i do hope you're not expecting a demonstration for that)
Step 2
The fun (not) part, compiling all the dependencies that we downloaded. Do the following, remember, replace <VERSION> with whatever version you are using.
First, to make life easier:
export XNUPATH=$(pwd)/xnu
Compiling dtrace:
$ cd dtrace-<VERSION>
$ mkdir -p obj sym dst
$ xcodebuild install -target ctfconvert -target ctfdump -target ctfmerge ARCHS="x86_64" SRCROOT=$PWD OBJROOT=$PWD/obj SYMROOT=$PWD/sym DSTROOT=$PWD/dst
$ sudo ditto $PWD/dst/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain
Compiling Availability:
$ cd ../AvailabilityVersions-<VERSION>/
$ mkdir -p dst
$ make install SRCROOT=$PWD DSTROOT=$PWD/dst
$ sudo ditto $PWD/dst/usr/local `xcrun -sdk macosx -show-sdk-path`/usr/local
Installing Headers:
$ cd $XNUPATH
$ mkdir -p BUILD.hdrs/obj BUILD.hdrs/sym BUILD.hdrs/dst
$ make installhdrs SDKROOT=macosx ARCH_CONFIGS=X86_64 SRCROOT=$PWD OBJROOT=$PWD/BUILD.hdrs/obj SYMROOT=$PWD/BUILD.hdrs/sym DSTROOT=$PWD/BUILD.hdrs/dst
$ sudo xcodebuild installhdrs -project libsyscall/Libsyscall.xcodeproj -sdk macosx ARCHS='x86_64 i386' SRCROOT=$PWD/libsyscall OBJROOT=$PWD/BUILD.hdrs/obj SYMROOT=$PWD/BUILD.hdrs/sym DSTROOT=$PWD/BUILD.hdrs/dst
$ sudo ditto BUILD.hdrs/dst `xcrun -sdk macosx -show-sdk-path`
Compiling libplatform:
$ cd ../libplatform-<VERSION>
$ sudo ditto $PWD/include `xcrun -sdk macosx -show-sdk-path`/usr/local/include
Compiling libdispatch
$ cd ../libdispatch-<VERSION>
$ mkdir -p BUILD.hdrs/obj BUILD.hdrs/sym BUILD.hdrs/dst
$ sudo xcodebuild install -project libdispatch.xcodeproj -target libfirehose_kernel -sdk macosx ARCHS='x86_64 i386' SRCROOT=$PWD OBJROOT=$PWD/obj SYMROOT=$PWD/sym DSTROOT=$PWD/dst
$ sudo ditto $PWD/dst/usr/local `xcrun -sdk macosx -show-sdk-path`/usr/local
Compile XNU
Now to actually compile XNU:
$ cd ../xnu
$ sudo make SDKROOT=macosx ARCH_CONFIGS=X86_64 KERNEL_CONFIGS="RELEASE DEBUG DEVELOPMENT"
Letting that run for a while we should get 3 folders in BUILD/obj/ corresponding to the build you want to use. (i'm using DEVELOPMENT as an example)
BSDKernel.symbolset kernel.development.unstripped
IOKit.symbolset lastkernelconstructor.d
Libkern.symbolset lastkernelconstructor.o
MACFramework.symbolset lastkerneldataconst.d
Mach.symbolset lastkerneldataconst.o
Private.symbolset libkern
SupportedKPIs-all-archs.txt libsa
SupportedKPIs-x86_64.txt link.filelist
Unsupported.symbolset lto.o
all-kpi.exp osfmk
allsymbols pexpert
bsd san
config security
iokit tools
kernel.development version.c
kernel.development.ctfdata version.d
kernel.development.dSYM version.o
We can check the build information of kernel.development like so:
$ strings kernel.development | grep 'Darwin'
Darwin Kernel Version 17.0.0: Sat 7 Oct 2017 15:11:40 BST; root:xnu-4570.1.46/BUILD/obj/DEVELOPMENT_X86_64
Darwin
And, well, that's it! The real challenge is getting it to compile for ARM, which, despite the source code being there, is a bloody hard task.
Summary
I have to say i've enjoyed writing this section a lot, mainly because of compiling XNU, but it's quite interesting how the XNU kernel came to be, and it's quite cool to know that Apple didn't want to settle for a single type of kernel and instead took the best of both worlds and created XNU.