Mnemos moment - Search for a Shell

2023-07-10

A screenshot of a monochrome simulated display, showing forth code

This post continues catching up on the current status of mnemos, a hobby-grade OS written in Rust.

The last post covered the basics of the OS and why it exists. This post explores the work that gave mnemos an interactive, forth-inspired, user shell.

Re-inspiration

In order to get mnemos to a state where it was fun to play around with, and not just something to be developed, it needed some way to make it interactive.

Until we have more complex graphical capabilities, the easiest option was to add some kind of text interface. Text interfaces usually fall into one of two categories:

They could be a shell, like Bash or DOS' Command Prompt. Or, they could be some kind of interpreter with a REPL, like BASIC or Python. There is probably some delicate distinction between the two approaches, but we needed something that could:

Luckily, one of my other research projects over the last months was building a forth interpreter in Rust, as well as a library to manage rendering directly to a framebuffer.

What luck!

Forth and Forth3

Forth is a very simple language from the 1970s. It consists of only a few core concepts:

In the old days, you would hand-write the core of your forth system in assembly. The scope was small enough that it could be done reasonably quickly in both developer time as well as performance. Once you had a simple system, you could begin building more complex business logic in forth. If a forth function ever became a "hot spot", you could replace the forth definition with an assembly one, without much hassle.

I originally wrote forth3 as an interpreter in Rust, to allow me to build the "core" in Rust, and to allow me to have embedded systems that provided some kind of interactivity. Because of this, it was written as a no-std library, to make it easy to "drop it in" wherever I might need it.

Although forth3 is very much in the spirit of the larger forth language, it certainly makes no attempt at being "standards compliant", so I generally call it forth inspired instead.

In forth3, you could define "native" functions (in Rust, instead of assembly), by providing a function pointer and a text name. These functions would be given mutable access to the forth "virtual machine", allowing them to access the internal state to implement their functionality.

For example, the . function in forth pops the first number off the top of the "data stack", and prints it to the console as a number. In Rust, it looks like this:

pub fn pop_print<T: 'static>(forth: &mut Forth<T>) -> Result<(), Error> {
    let a = forth.data_stack.try_pop()?;
    write!(&mut forth.output, "{} ", a.as_i32())?;
    Ok(())
}

You could then add this builtin to the VM like this:

let mut forth: Forth<Context> = Forth::new();
// This turbofish monomorphizes the free function to a specific concrete type
forth.add_builtin(".", pop_print::<Context>)?;

Which means inside of forth, you can now call the . function, which will call the native Rust function.

The Forth<T> type is generic over one parameter, which is a "user context" field, allowing for custom words that access special data that the rest of the VM has no idea about.

This is great, and very close to what we needed for mnemos! But there was one small problem:

forth3 wasn't async, and mnemos very much is async.

We would need some way to define async builtins for things like I/O operations, and we'd need some way to run the interpreter in an async context!

I had tried a couple of ideas to make this work, but nothing ended up working (or if it did, it wasn't very pleasant to use). Luckily, I have friends who are smarter than I am.

This is where Eliza comes in

After posting about my recently refreshed plans for mnemos, I was lucky enough to have Eliza Weisman reach out. Eliza (best known for working on libraries like Tokio and tracing) was already a contributor to mnemos both directly as well as indirectly, as mnemos uses a number of her projects, including cordyceps, an intrusive linked-list library, as well as maitake, a no-std compatible executor that is used for mnemos' kernel.

The promise of running an OS on physical hardware in the near future was enough to pull her in, and she very quickly figured out a way to provide async builtins. The full async VM docs are a good read, but the trick boils down to providing a trait that looks like this:

pub trait AsyncBuiltins<'forth, T: 'static> {
    type Future: Future<Output = Result<(), Error>>;

    // All of the builtins that should be handled async
    const BUILTINS: &'static [AsyncBuiltinEntry<T>];

    fn dispatch_async(
        &self,
        // The name of the builtin being called
        id: &'static FaStr,
        // The context of the VM
        forth: &'forth mut Forth<T>
    ) -> Self::Future;
}

Which defines three important things:

Inside the kernel, we can now implement this trait like so:

impl<'forth> AsyncBuiltins<'forth, MnemosContext> for Dispatcher {
    type Future = impl Future<Output = Result<(), forth3::Error>> + 'forth;

    const BUILTINS: &'static [AsyncBuiltinEntry<MnemosContext>] = &[
        // open a sermux port
        async_builtin!("sermux::open_port"),
        // write the current out buffer to the sermux port
        async_builtin!("sermux::write_outbuf"),
        // spawn a child interpreter
        async_builtin!("spawn"),
        // sleep for a number of microseconds
        async_builtin!("sleep::us"),
        // ...
    ];

    fn dispatch_async(
        &self,
        id: &'static FaStr,
        forth: &'forth mut forth3::Forth<MnemosContext>,
    ) -> Self::Future {
        async {
            match id.as_str() {
                "sermux::open_port" => sermux_open_port(forth).await,
                "sermux::write_outbuf" => sermux_write_outbuf(forth).await,
                "spawn" => spawn_forth_task(forth).await,
                "sleep::us" => sleep(forth, Duration::from_micros).await,
                // ...
                _ => {
                    tracing::warn!("unimplemented async builtin: {}", id.as_str());
                    Err(forth3::Error::WordNotInDict)
                }
            }?;
            Ok(())
        }
    }
}

This ends up with a very similar level of capability to the non-async builtins, with the one "nit" that the dispatch_async must handle dispatching itself, in order to avoid the issue of all async functions having a unique type. We could have boxed the futures instead, with something like Box<dyn Future<...>>, but this approach seems preferable so far, as we'd like to avoid heap allocations where possible in the kernel.

Okay but what does that mean?

It means that a user in the shell can call functions that would take some amount of time - like flushing to disk, or sleeping, and NOT block the entire kernel while doing it!

We can write synchronous-looking forth code, which under the hood automagically plays nice with Rust's async/await capabilities. This also means that the kernel can run multiple forth interpreters at the same time, which will all work cooperatively with each other to share the available CPU resources.

We also added something we like to call the Bag Of Holding, a way to turn arbitrary rust objects into type-safe, i32 "object IDs" which the forth code can use when calling built in functions.

For example:


( create a port handle for port 5 with a 1024-byte buffer. )
( the bag of holding token for the handle is returned on   )
( the stack. This could cause the interpreter to yield!    )
5 1024 sermux::open_port

( write some data to the output buffer )
." Hello, world!"

( copy the token on the stack, since write_outbuf "consumes" it )
dup

( call write_outbuf, which will use the token from the stack    )
( the contents of the output buffer will be sent over the port. )
( this could also cause the interpreter to yield!               )
sermux::write_outbuf

Beyond that, Eliza also gave the forth interpreter the ability to spawn additional forth interpreter instances, sort of like fork() on other operating systems. However, this post is getting long already, so we'll leave the details of that as a reading exercise for another time.

The forth3 crate is still standalone, and not specifically tied to mnemos, though we've moved it into the mnemos monorepo for ease of development. It could be integrated into other environments (async or blocking), and hopefully the mnemos kernel integration gives some inspiration on how that would be possible.

On the next episode...

At this point I have to admit the blog has fallen behind the actual development progress. We've already gotten the OS running on top of the beepy, our first hardware target.

In the next blog post, we'll share a bit more on what the process has looked like to get mnemos running on the Allwinner D1, and add the necessary drivers to support the beepy.

Interested in hacking on mnemos, or think it could fit something you need to do in the future? Let me know, I'm always happy to chat.

Need help with building something (else) in Rust? Maybe I can help!.

A picture of a SQFMI Beepy, a device with a blackberry keyboard and monochrome screen, showing a forth shell

OneVariable — Imprint — Privacy