CI20: User-space cache coherency

This is a CI20 bare-metal project post.

Back in the dual-core post we talked a bit about multicore cache coherency — specifically, that kseg0 didn’t have it. This meant that if two cores wanted to communicate using shared memory in kseg0 they would have to use explicit cache flushing, or turn caching off entirely.

Now that we have a user mode, we can run basically the same test in user space, where things are very different. To do this, we modify sigma0’s main.c to do one of two things:

• If it’s core 0, increment a counter
• If it’s core 1, periodically print the counter.

The code for this is in the userspace-cache-coherency branch. If you build and run that, you should see a changing number when the kseg0 version repeatedly printed zeroes. The user-space version is somehow magically keeping the caches coherent between the two cores.

The magic is that the hardware implements the MSI protocol: when one core modifies a cache, the other cores find out somehow (either by bus snooping or by consulting a central directory) so they can invalidate their own state. This is very convenient for us, so why isn’t it implemented in kseg0 as well? Actually, on some architectures, it can be (by setting the CCA bits in the Config register of CP0). But this is implementation-specific: the standard “cacheable” setting (3) of CCA is incoherent.

As noted in the “When You Say Nothing At All” post, the philosophy we’ll adopt with this operating system is to leave inter-core communication to user space, as much as possible, and cache coherence is one of the reasons why.