CI20: User space!

This is a CI20 bare-metal project post.

Big update this time as we enter user mode for the first time. Before we get started, it’s worth checking out the code to follow along. The git tag is ‘userspace’:

$ git clone https://github.com/nfd/ci20-os
$ cd ci20-os
$ git checkout tags/userspace 

Here’s what we’re going to do, overall:

1. Implement system call handling
2. Implement TLB management.
3. Write a user-mode program
4. Get the user-mode program into memory somehow
5. Run everything

Surprisingly, the most complicated part of the whole process is step 4 — getting the user-mode program into memory. That stage is also where most of the design is in this instalment, so let’s start with that.

4. Get the user-mode program into memory

As you know, the boot process at the moment goes like this:

1. Use CI20’s built-in USB loader to load stage1, which initialises the DDR RAM.
2. Use the same built-in USB loader to load and run the kernel.

Fundamentally, we just want to load another ELF. We’re already loading two ELF files to start the system. This leads to obvious solution 1.

Obvious solution 1: add a step 1.5: “Use the USB loader to load the user-space program”.

You’ve probably guessed, since this heading is called “obvious solution 1”, that I didn’t go with it. I think there are several problems with this approach.

The first problem is that you need to tell the kernel where the user-space program is loaded and what the entry point is, so that the kernel can run it. The second problem is that you need to link the user-space program at some address which is not already occupied by the kernel (or, to put it another way: the kernel loads at 0x80000000, where does the user-space program load?). 

Both of the above problems can be avoided by making the kernel deal with them. That leads to obvious solution 2.

Obvious solution 2: store the user-space ELF as an uninterpreted binary blob inside the kernel ELF. Then write an ELF loader in the kernel.

This solution has its own problems. ELF loaders are complicated and a potential source of security vulnerabilities. It’s also not very obvious (to put it politely) that ELF loading needs to be something the kernel should do — perhaps a user-space program could be in charge of ELF loading. So all we need to do is to get that one user-space ELF loaded, and then it could do the rest.

Possibly non-obvious solution which I actually used

My chosen solution is to combine the kernel and user-space programs into a single ELF file according to the following diagram:

In words: the kernel is compiled and linked normally (at a base address of kseg0, i.e. 0x80000000). Then the user-mode program, which is called sigma0 for reasons we’ll get back to, is linked. The base address of sigma0 is the highest virtual address of the kernel. In other words, if the kernel is linked to occupy addresses 0x80000000 to 0x80004FFF, then sigma0 will be loaded from 0x80005000 up*. The two files are then combined into one ELF file, which works because the loadable segments of each file do not overlap (but are contiguous). Finally, a data structure in the combined ELF file is patched up to contain the entrypoint of Sigma0, as well as some other details.

This adds a lot of complexity, but the complexity is in building the system: the running code is very simple.

This horrible ELF manipulation is achieved through a new toolchain called Saruman. Saruman is now a required part of the CI20 build process.

In your CI20 directory, download and compile Saruman:

$ git clone https://github.com/nfd/saruman
$ cd saruman
$ cmake .
$ make -j 

That covers the loading part. The rest is simple!

* Side note: Why 0x80005000? I thought this was user space? Actually, the link address would be 0x5000 in this example, but the load address inside the ELF file is in kseg0 so that the USB loader can upload it without needing TLB refill support. What sorcery is this? Saruman again.

1. Implement system call handling

There is nothing CI20-specific about this — we implement a very simple MIPS syscall handler which saves all registers (to a structure called “current”), runs a C function to handle the syscall, and then returns to user mode using eret. There is one syscall implemented (implementation in architecture/mips/exception.c) which writes a character to the serial port.

2. Implement TLB management

Like many modern MIPS processors, the JZ4780 provides a separate happy-path TLB exception handler entrypoint. The implementation of this is in start.S (label _exc_tlb_refill) and it simply direct-maps user space — giving vaddrs from 0 to 0x7FFFFFFF the same treatment that vaddrs in kseg0 get (except, of course, that user-mode code can only access the former).

3. Write the user-mode program

This was a relaxing part! Implementation in the sigma0 directory. It does almost nothing at this point. sigma0/main.c includes some commented-out code to access kseg0. If you uncomment this code, you should see the kernel… well, panic with an unhandled exception, which is admittedly not very nice, but which does demonstrate that the program is indeed running in user mode.

4. Get the user-mode program into memory somehow

We did this bit!

5. Run everything

If you’ve downloaded and built Saruman, you should be able to compile and load everything the normal way.

You should see two messages from sigma0 — “Sup0” and “Sup1”. These correspond to the two cores we are running. Each core is running its own user-space program. This is all part of the grand plan to eliminate inter-core communication on the kernel, and we’ll discuss it in more detail in the next post.