This episode is coming from the archives, originally recorded in May 2022 - James talks with Steve Klabnik from Oxide Computer about knowledge transfer within the Rust community, how learning-by-doing and reading datasheets help you develop, and how limits and regulations are put in place across many different fields.
Originally recorded on 2022-05-04
Reposted from jamesmunns.com/podcast.
Audio
FLAC
M4A
MP3
Show Notes
- Introduction Steve Klabnik, Oxide
- Rust, Book on Rust
- Ruby, Ruby on Rails, Ruby World
- Rustfest Berlin 2016, Simon Sapin, Cargo
- Rust All Hands, Rustconf
- Miscellaneous chit chat, gear acquisition syndrome, Schiphol Airport
- Transition from Rust to bare metal systems
- Solving problems using a transfer of knowledge from different backgrounds
- Learning vocabulary & intimidating aspects of programming, impossibility of one person understanding all aspects
- Limits & restrictions, putting guidelines in place vs. people following them, safety critical and reliability / failure
- Questions from others: How do you draw the rest of the owl? Reading data sheets & knowing what to look for
- Just do, make mistakes, you will learn! Not understanding everything on day 1 is ok.
- Video games growing up
- First jobs, goodbyes
Transcript
James Munns
I got this mic because I needed something vaguely better than my laptop mic… especially because—
Steve Klabnik
Yeah…
James Munns
— like just tapping on the keys is super loud.
Steve Klabnik
Absolutely.
James Munns
I tried… like I actually got a webcam and I was hoping— because it's one of those Logitech ones where it's got a stereo mic and stuff and I was hoping that'd be enough for noise canceling and it's better than my laptop mic but it is not— like this is like a 50 buck USB mic and it blows the water off of ah both my laptop and the Webcam.
Steve Klabnik
Yeah, absolutely- I because I'm using a desktop I didn't have a built-in mic and I think I have like the Logitech Brio or whatever and maybe it has a microphone… I think maybe even my, ah, monitor has a microphone built into it or something but I like turned that off.
James Munns
Really?
Steve Klabnik
Yeah I was starting to get this weird echo sometimes, ah and in a recording so I was like, “What the hell is going on?” and then I realized like, “Wait a minute, it says there's like something in my monitor? Like that seems weirdly CIA or something, hahaha — Why- Why does monitor listening to what I'm saying this is weird…”
James Munns
What monitor do you have that has a mic built in?
Steve Klabnik
Ah it's- it's just like a Dell whatever like Dell IPS flat screen. But, I don't know, it was really weird. But yeah, so I need to get something and so I'd been using like the classic, ah, whatever the regular podcast mic is that like literally every podcast person seems to have…
James Munns
Like the Blue Yeti?
Steve Klabnik
Yeah, the Blue Yeti, and it was totally fine but I wanted something is like, a little fancier because it was getting into gear — you know when you start doing- recording this stuff and whatever, you start being like, “All right, how can I, like, upgrade all the things…”
James Munns
It's gear acquisition syndrome. At least that's what the audio folks call it.
Steve Klabnik
Yeah, exactly that makes total sense. But so I'm not- I haven't done the full XLR setup upgrade yet because this is totally good enough for my purposes, and it's been a couple months since I've recorded anything anyway and this is definitely overkill for the morning standup meeting or whatever. So.. not that we do literal standups at at Oxide but like yeah.
James Munns
Yeah, every time I get in a call with a client they're like, “Wow, fancy setup!” and I'm like, “For 50 bucks, you can have it too!”
Steve Klabnik
Yeah, absolutely..
James Munns
But before we get too far, do you want to give yourself a quick introduction?
Steve Klabnik
Yeah, absolutely. Hey everybody, I'm Steve. I work at a company called Oxide Computer, writing firmware and Rust for some computer stuff. Before that ah, did a whole bunch of other things but I spent a really long time working on the Rust project in general, and before that it's been a long time working on Ruby on Rails and all sorts of things. I'm getting old, I've done a lot at this point… I co-authored the Book on Rust. Ah yeah, I don't know, known you for a long time at this point.
James Munns
Yeah I was going to say I think I probably met you right around the time I was getting involved with Rust, because like for me the big like change of going from like idly watching Rust to actively being involved was around Rustfest Berlin, which I think is 2016 or something like that? like I had been…
Steve Klabnik
Yeah I think it sounds about right.
James Munns
I'd been messing around with it. But I admit I was in that stage where like some stuff sort of made sense and some stuff just did not… and it was actually like Rustfest because a bunch of Rust folks came into town and I got to met meet, um, Simon Subin or Simone Sapin and ah, we were at a meet up at I think the Mozilla headquarters in Berlin before the thing— and just sitting down with him for like an hour, he showed me how cargo worked and showed me how a couple things, and I was like, “Oh okay,” so all the stuff from like just idly reading the book a couple hours a night going from like actually using it was sort of for me like the big changeover moment of like getting things and then getting involved especially with embedded Rust and stuff.
Steve Klabnik
Yeah, it's great. Yeah, that was a really good time. I miss Berlin a lot.
James Munns
But yeah, it's a cool city. I've definitely seen you when you’ve been in town. It's interesting because Steve is one of those people that I talked to online a decent amount but the only time we see each other in real life is at some kind of Rust conference or some kind of Rust event like, Rust all hands or whatever and you'd come to Berlin or if we went to like Rustconf in Portland or something like that. So you're in- you're one of the few people that I only see at events or conferences and things like that in real life and I feel like- especially because you've had a couple generations of different things you've and been involved with I'm sure you have like, conference friends from Ruby and Rust and things like that who are like that all over the world. Yeah.
Steve Klabnik
Yeah, absolutely and obviously the pandemic had changed that a lot but before that is totally the case like— literally the reason why I started keeping track of frequent flyer miles is because I was like sitting waiting for a plane with a friend from the Ruby World and they were like, “Oh, how are you doing on your like you know, frequent flyer miles this year?” and I was like, “Oh man, I don't fly often enough to like, keep track of that,” and he was like, “I see you at all the conferences and I'm keeping track of that. You absolutely do actually and I was like, “Oh!” and that was like the moment when I realized I was traveling like a lot a lot. You know like I always thought like, I'm not traveling literally every week— I'm traveling like once or twice a month. So I probably am not traveling as much as that stuff requires or whatever and so yeah, but you sort of- a lot of the folks who do a lot of that kind of thing, you end up seeing people around or… I think one of my favorite ones ah was one time I was- there's a friend of mine who lived in Berlin at the time and he was flying to somewhere on the west coast of the US and I was flying from the US somewhere else in Europe and we both were in Schiphol in Amsterdam at the same time, and like that's a really big airport with lots of people for anyone who hasn't been there before. And we were like walking past each other in the hallway and we're like, “Oh my god, wait is that- like say hi, like, I got to go run to my plane!” and be just like totally you know randomly run into people. Ah sometimes for things. It's really cool it's it's it's one of the nice things about being able to have done all that kind of stuff, is making all these friends all around the world and like running into people and talking to them and seeing them. And it's nice that the internet can keep us connected whenever- it's not the like three days a year whenever I'm in whatever city. So yeah.
James Munns
I yeah, Schiphol by the way like, 1 of my favorite airports in the world—
Steve Klabnik
Same.
James Munns
I have like an oddly- like I have such weird like association and memories of it because when you fly from the US to Europe, Berlin always had 2 small airports. Now it has a bigger one, so maybe it'll be better after the pandemic, but you almost always had to connect through either Munich or Frankfurt or Amsterdam depending on which carrier you were going. But when you fly from the US to Europe, most of the time those flights leave in the afternoon or evening like 5, 6, 7 PM US time. And then land at like 5, 6, 7, in the morning in Amsterdam so I have like- because I flew back and forth either to visit family or for a while I was doing consulting and a lot of my clients were in the US and stuff like that. So I was flying back and forth between the US and Europe not super often, but every couple months and just this like… recurring habit of landing in Amsterdam at about 5:30 in the morning before- the airport was basically empty, because the only other people there were the people who just came in off an, ah, international flight. And it's a really nice, like, green airport where they've got trees indoors and you can go out on a balcony and stuff like that and… just wandering around chilling for the 2 hours until my connecting flight back to Berlin is like a really- and usually getting like a coffee and a stroop waffle on my way in the airport is just like such a weird like memory, that after Corona started I realized I started having, like, occasional dreams and I'm like, “I miss Schiphol…” like which is such a weird thing to have I feel like, but yeah…
Steve Klabnik
Yeah I have one other very funny story about Schiphol I'll tell relatively quickly. But ah I the very first time I went to Europe I was transferring back home through Schiphol because as you said many flights to the US either come through or go buy there. And I had those couple hours of layover and I had a friend who was living in Amsterdam and he was like, “Oh, maybe we could like say hi!” and I was like, “Yeah, that seems cool…” Ah, but I was traveling and was really tired and so I didn't have that much time, and so he agreed to like come meet me at the airport. So I like left security, we went and we sat down at the airport and got some coffee— which you know was a Starbucks because you're at the airport and there's no like small independent you know shops or whatever and it's totally fine. We get some coffee I get back on the plane, I do that flight back to the US and I hadn't traveled very much, you know up to that point. So I was really tired and really jet lagged and not used to it. When I landed back in the US and they're asking me these questions about like, “Oh hey, you know, I see you made this trip-” you know, “What you do?” and I told them, you know, conference, whatever, “I see you stopped in Amsterdam for a couple hours on the way home,” like, “What did you do there?” and I was like, “Oh I, ah, you know, I met up with my friend at, ah, at a coffee shop and it was fine…” And he is already laughing… Ah you can’t see the video but ah already laughing about that because if you don't know that means something very different in Amsterdam which is a place you can buy weed. Ah, but I had not known this because I had not like actually gone into Amsterdam, and in the US we call them coffee shops as instead of cafes. There's like no distinction generally. And so then he was like, “Did you buy any weed?” and I was like, “What?!” —
James Munns
At the airport?!
Steve Klabnik
— he's like,“Did- did you smoke any weed?” And yeah I was like, “At the Starbucks in the airport?!” and he kind of gets this annoyed look on his face, he’s like, “Go ahead and stand for passport,” and then I like got on Wikipedia and I was like, “Oh my god, I'm never going to say that to US customs ever again actually regardless…” of but I just was that naive of a person I just like literally had no idea, so that's always one of my enduring memories of Schiphol as being a totally new traveler just like not knowing what I'm saying to customs.
James Munns
Yeah, so when when I first met you, I mean you were- I think you were definitely involved in the book already and you were involved in in Rust for significant amount of time I'm sure, like I passively followed it from 2013-2014 but really didn't get super invested until about 1.0… but um, yeah I always knew you as like the person who was doing the book and the person who is working behind the scenes and things like that, and one of the people who was always very welcoming and and warm and like, introduced me to people. Like after I met you at Rustfest, you introduced me to other people… But at the time I was super into embedded and I didn't tag you as an embedded programmer.
Steve Klabnik
I was not.
James Munns
But as you mentioned, when yeah when you left Mozilla and the other things that you were doing, you ended up at Oxide and your primary responsibility as far as I know was to work on bare metal or in-house operating system systems. So like how did that transition happen for you?
Steve Klabnik
Yeah, so when I first learned programming as a kid, computers were much more simple than they are today. Ah, the first computer I ever played with was like a Mac plus but then the first computer I ever owned was this like, computer we got out of the Sears catalog that you could like hook up to a TV and many decades later I found out it was like a Z80 basically and it had a BASIC interpreter on it and so I like learned BASIC on that, but eventually I learned C as my second language. And like when back in that like late 80s early 90s kind of era, everything was a little lower level because there wasn't as many like towers of abstractions built up and like obviously you know I'm not going to claim that like I was handwriting machine code like my uncle was, you know the generation before me or whatever, but like you know I still like remember like, using ResEdit to like edit the resource files of the keyboard definition or whatever, and like Mac OS 6 and like setting my own keyboard shortcuts and like feeling super like- like a, you know, wizard about it and so I really… when I went to college I wanted to either do compilers or operating systems, and I had set my like my class schedule to be like: how do I take compilers and operating systems as like soon as possible and that's because at the time I kind of like thought that almost no one knew how to write operating systems anymore and I knew this is like kind of not true, but it was also like a fun story and I basically was like, you know, there's a zillion people writing web applications and like doing desktop software but like, it seems like only Mac and Windows exists and like so does Linux sort of kind of but it was much smaller back then and so I was like if there's an an earthquake in Cupertino, ah, you know like we're going to lose like half the people that know how to build an OS and like now I know that it's like obviously very silly and the world of this kind of programming is like much larger than you like normally think but when you're like on- when you're on web forums and mostly talking to web programmers and you never really talk to systems folks, you're like, “Oh my god, no one does that work anymore,” and I thought it was really interesting and cool and I liked the idea, so I really wanted to do it. And so ah, me and some of my friends worked on an operating system written in D in college for like a long time and like we spent a lot of times doing like OS stuff and some of my friends continued on doing systems work after that. But basically I got an internship at a company that was doing podcast hosting software and so I got into like PHP and doing that kind of work and so I saw how many more jobs there were doing that kind of thing than system stuff and ended up doing a startup and all this other kind of crap but I kind of like, like that stuff but sort of stopped doing it after college and so professionally then like I became known as this like Rails / Ruby person, but I'd always still like like that stuff and so part of me seeing Rust and deciding to get interested in Rust was because I was kind of like, not totally sick of that world, but I wanted to make a big change for various reasons and so when I saw Rust I looked at it and I thought like, wow okay, so D like sort of kind of was a thing, but not really, and by that point in time I didn't think it was going to succeed any further than it already has and… You know, I don't want to be rude to the D people, but I don't think that my estimate was like entirely off really, exactly, at that point, and so I- there wasn't really any challenger in that space at all and so I was like: okay, you know, like, this language seems to be offering something legitimately new in the low-level space, and critically like, it being supported by Mozilla was like 85% of the reason why I wanted to like, get involved and that's not even because I had any particular love for Mozilla, though I do have the like New York Times, ah, ad for 1.0 like in a frame like up on my wall somewhere but like, it wasn't—
James Munns
I've seen a couple Rust folks with that up on their walls.
Steve Klabnik
Yeah, exactly like- it's not because I have a particular love for Mozilla, but just because I knew that having a big name associated with it would cause enough other people to take it seriously that maybe it would have a shot, because getting a programming language from like nothing to something is like a very large task and so I was like: okay, if this like thing is a thing, and it's in this area where I think it has a real shot, because it's legitimately better than what's come before, and it has a big name backing it, then like this seems something that's like worth my time getting invested in. And so then I tried it out, and when I tried it out, the people were like so ridiculously nice and good. I was like yeah like this is really where I want to spend my time and so that's kind of like how I got involved in Rust, and so even though I wasn't like doing embedded stuff with Rust initially, because I thought the docs work was like the most important work that would be needed to get Rust adopted, and was like something that I had like strength in, like both necessary and a thing that I was good at, and so I focused on that and I didn't really get into the embedded stuff. I still kind of like always wanted to, and so, you know, once I had some free time that was not purely, you know, working on getting Rust actually going, I started messing around with like: can I write my own little OS again. And it was specifically Phil Oppermann started writing those tutorials that I was like, I got back into it and I was like oh yeah, like this is not the OSDev Wiki which is like, has a bunch of information and is useful, but is also definitely, you know, as someone who is a 20 year old kid who likes to run his mouth about computers — there's a lot of 20 year olds who like to run their mouths about computers on the OSDev Wiki and it's not exactly, like an inclusive and welcoming space. But Phil's work was like: oh my god, yeah I could like, I could like start doing this again and like, that's the thing I really liked. And so I did a bunch of that sort of thing like, kind of as a hobby, and it was mostly on X86 just because I didn't know anything about ARM or like other embedded stuff. And so the Oxide move is kind of like both a interest in building something out of Rust as opposed to just building Rust, like okay, like we're going to build an entire computer system and use Rust to write as most- as much of it as we possibly can. So, you know that seems like a fun challenge and idea and like validates all this work that I've been doing for the last couple years, but also did give me the opportunity to like do some low-level embedded stuff. And when I first joined I kind of like said to Brian and the other folks there, like, I don't know if I want to do the embedded work or I want to just work on the control plane, because like, the control plane stuff is more in my wheelhouse but I'm also like looking to grow professionally and so, you know: I would, I think, I would like to do the embedded stuff. But I'm also a little terrified that I, like, can't do it at, like, a, you know, level that's good enough for you to justify paying me for it. And sort of the attitude there was like basically like hey you know we have a lot of people who are professional embedded folks, but they're not Rust experts and so you can be a Rust expert and an embedded newbie and it's no different than them. It just means we got to, like, skillshare and cross-train and you'll all have to, like, teach other each other a whole bunch of stuff and that's, like, totally great for us. And so I was like cool. And so that's kind of like what I did was I had some knowledge on my own and some stuff I learned on my own but I also had really supportive coworkers who were like willing to take the time to, like, help me out when I had questions and, like, do that kind of stuff, and so and vice versa like I spend- so like while the team I work on is the embedded stuff and I was- early on writing a lot of like actual code in the sort of kind of things, I kind of found a niche helping people with their Rust questions and sometimes that's doing the embedded stuff and sometimes it's in other parts of the company too; and then also I've really gotten more like personally interested in, like, the build system that we use, because we layered a build system on top of Cargo because Cargo is not really sufficient for like the kind of work that we're doing and like also kind of like… The- the sacrificial lamb at the altar of GitHub actions. I spent a lot of time doing that kind of like opsy stuff because it's, like, a fun way to combine this new knowledge with something I've, you know, done for a long time in a totally different context, and so… If I can let the hardcore firmware people continue to write the firmware that's necessary for bring up while I can, like, you know, fight with GitHub actions, like, that's a tradeoff I'm, like, happy to make. And so I- lately my work has shifted more to these kind of, like, Meta-tasks than directly writing firmware. But that's partially because I, ah, have just really enjoyed it and so, you know, when it's like, oh like, there's this stuff to do and so yeah, so I'm I'm doing a lot of that kind of work lately and I'm actually- I'm working on one interesting task around binary sizes and then also I'm adding a REPL to our like debugger that we've written from scratch humility. And so that's also kind of a fun way of getting that kind of like- not that it's really a programming language but you know when you start writing a REPL you eventually end up with a programming language. So like it'll happen at some point. I'm trying to avoid it because it's, like, not with a work that really needs to be done, but there'll be little aspects of that and so it's kind of a fun way to combine this kind of stuff together. Yeah so.
James Munns
It starts with a CLI and then it becomes a DSL and then it becomes a full blown language eventually because you just go, “Oh well 1 more thing…” and at some point… I ran into this with a customer project I was doing at Ferrous at some point where I was doing some message serialization stuff where the customer was, ah, needing some way to talk between different systems of different sizes with different programming language because one group used let's say Python, and one used .NET and one used Rust and things like that, and they needed some way to have a standard message interchange format but also be fast enough where it could run on a bare metal system because that was one of the entities that was talking. So things like Protobuf we tried, but it was too slow for what we were doing. And I ended up building them a tool that was at first just kind of an ugly script that was doing some hacky things and- and building these definitions and generating some code for different languages; and at some point, I just sort of painted myself into a corner and I sat down with Jonas who's someone who's been involved with the Rust project for a very long time and he was just like, “Look, you're building a compiler… like, you- you are building- like it's building libraries and it's generating code. You are- you are building a compiler,” so I actually carved out some time for him to come in and be like, “Instead of this just being sort of these hacky, ad hoc, code gen tool that James built-” because that's very much how my brain works because I never took the compiler's class. My knowledge of compilers has only come from talking to people on Twitter about Rust who share their knowledge and share how things work and are willing to answer questions when you're like, “Wait, but how does Rust do this?” and sitting down with Jonas was like that, where he split it into, you know, the parsing, the lexing, the code generation, and then- or then something on the internal representation and then code generation at the end.
Steve Klabnik
There's like 4 so yeah.
James Munns
Yeah, so it's like it had the actual defined stages, and that made the tool way easier to work with, because when you wanted to add a feature to the front end, you didn't have to rework the entire middle system because there was a more defined boundary. And- and that's one of those things where operating systems are something that I've learned on the fly, just from working on bare metal systems, because bare metal systems are really just… your application is the operating system. So you end up learning a lot of these operating system concepts or why you need these concepts indirectly, because you're essentially building your own operating system every time you write a program. And learning these compiler things just from people talking on Twitter, because I’m colleagues or or friends with folks who are working on the Rust project and I'm just interested because people in the Rust project tend to be pretty excited about the stuff they're doing and are usually like you said pretty welcoming and interested to share how and why they are doing things, or at least there's documentation that explains why certain things were done. So yeah, it's been interesting to come at it from like operating systems are a fairly recent development for me and I talked a lot with Eliza about this… We came at this from very different perspectives because she was working on, you know, applications and things like that, and again, this sort of interesting thing about Rust is that the domains wrap around— like whether you're talking about a high performance database or an operating system kernel or an embedded system or any of these things, you get to the point where you just need that level of control and what you need to control is a little bit different but in Rust you use the same techniques. So, you know, knowledge will transfer. So I'm sure a lot of the stuff that you worked on applied when you were moving to embedded; and embedded particularly in the last 5 to 10 years has been in a pretty interesting inflection point, where it's more or less speed running the last thirty years of knowledge that desktop computers and operating systems have learned about being persistently connected to the internet and having multiple cores and having a ton of RAM, where if you talk to old school embedded programmers or people who are still active but on more legacy or classic style systems, they're missing a lot of that domain knowledge that is becoming not an option anymore. Well, about networking stacks and um, how to manage these things with only certain compromises and things like that. So I'm sure even you bringing your Rust knowledge from a higher part in the stack to sort of the meeting point of writing Rust on embedded— I'm sure there was a lot of sort of meeting in the middle of either concepts or practices and things like that of figuring out a different way to solve problems that have been solved just one way and embedded classically.
Steve Klabnik
Absolutely… I think a lot of Javascript folks actually would have a much easier time learning embedded than in other systems because, like, so much of hardware is asynchronous and the idea of callbacks is, like, kind of pervasive like when you're dealing with hardware? It's responding to real events that happen in the real world, and so you have to like start getting into that mental model in many cases— I mean obviously you can paper over it— it's always been kind of funny to me that like hardware is often asynchronous, and then we paper a synchronous system over top of it because we think it's an easier mental model and then in userland, we repaper an async system on top of it because it's like actually more performant because turns out that's how stuff actually works. And you know, we got to like build these like towers of abstractions and… and yeah, and sometimes that's also because like too, when you learn things in one domain or even if it's not the same stuff like… when I did a lot of this like operating system stuff, it was an X86, which is like a giant pile of legacy garbage compared to ARM. Like dear god, it's like so much nicer to write an embedded ARM thing because you don't have to like deal with… like for those of you don't know, like when you turn on a X86 processor it like boots up in like 16 bit mode, and then you have to like teach it how to like coax it into the 32 bit mode, and then finally into 64 bit mode, and then you could start doing useful work and so you know you made this joke about speed running the history of like application development or whatever but like you sort of also have to do that with like all of the legacy of that entire architecture, whereas on ARM, it's like I start executing at this memory location.
James Munns
and.. go!
Steve Klabnik
Here's- here's the point where I jump into it, and we're good. Like there's no, you know, like doing all that kind of stuff and things like that and so… It's a deal. So it's always interesting how yeah, these like different kinds of like perspectives on things, you know, happen and like… Compilers like, are such a generic idea like we sort of have particular things that kind of like “make a compiler” according to classic sense. But it's really like, just computation, like, I'm taking some sort of format in, I do some work on it, and I put some sort of format out and you know's like when you describe it generically enough like that, it's like that can represent almost everything and so you know like … I also- I feel like compiler knowledge is useful and so many other domains are like if you start getting involved in compilers, you're like, “Oh, my templating engine that I'm using, like that's actually a compiler,” and it's like, “Yep!” Like, a lot of what people don't think about, that I, like— Oh, if you've- if you've written, you know, a template thing, then, you know, you've implemented a compiler before, even if you don't think about it that way. So, you know, there's a lot of that kind of shenanigans.
James Munns
Steve, you can't just point at things and call them compilers.
Steve Klabnik
I know. But I will, and I can. I love that meme so much.
James Munns
Yeah, I was going to say, I guess that's the- the meme of the episode I think… When I was with ah Eliza, we were talking about Spiders Georg,
Steve Klabnik
Um, absolutely.
James Munns
and you know, statistically unlikely things and when you're working at scale, you meet Spiders Georg a couple times a day so you can't really ignore spiders Georg. But yeah, I mean like you've talked a lot about Dev Tooling, like whether that's CI or build systems or even like, programming language.
Steve Klabnik
Rust Doc yeah.
James Munns
Yeah, exactly of of yeah you- you keep running into these patterns of knowing a little bit about compilers and how they work knowing a little bit about programming languages and how they work can help you even if those aren't your- like you aren't working on Rust C or you aren't working on, you know, a programming language of your own or anything where there's just certain patterns in established practices that are good to at least vaguely know because even if you don't do the theoretical perfect cutting edge research one, it at least helps you structure the problem in a way that the programming community has gone, “Okay, this is a way of structuring that problem and thinking about the independent stages, and if you don't have a better idea. It's probably good to split up your compiler into a, you know, a lexer a parser ah something that handles the internal format and then generates that because it's at least a proven way of breaking up that problem into manageable chunks,” where before I knew about compilers I knew what all those words were, and I knew vaguely that compilers had stages but I guess the- the lesson didn't ring true to me why it was so important to keep those separate, but then you get to the cutting edge research like what folks at Rust Analyzer or Rosalind are doing where they go “Well, that model works when you're talking about a totally batch compiler,” But there's other models like a more reactive on-demand compiler that in some cases could be more performant or have less latency because you're not just trying to job queue the entire task and get through that. So, I mean, like, there's always more to learn in all of these domains. So.
Steve Klabnik
Totally. And even learning about that stuff too, like, you know you said you've heard the words before but you didn't know what they mean like even just learning what the words are can be helpful, like gaining a vocabulary is what gets you access to the ability to learn. So, you know, if you've never seen compiler internals before it may look like magic. But once you hear the word parser or lexer now you have like a key, you can go type that into Google or whatever and you know, learn as much as you want about that stuff. It's also always interesting how different people perceive different things differently in terms of like difficulty. Gonna say diff like a zillion times in 1 sentence there… I am always like… mentally for me for some reason parsing is the hardest part of a compiler— which is like hilarious because from a like theory or like research perspective, it is like the most solved part of like all of this, like there's no novel research in parsers like virtually every real compiler in the first place uses like handwritten recursive descent parsers anyway and so like it's like in many ways the simplest part of the whole thing; but it was the part that was least interesting to me, and so it's the part I focused on the least when I was learning about these topics. And so it's always been something that I, like, have been like, yeah I don't really want to write a parser for that and then ah, there's like for most people it's like oh yeah, like totally writing a parser and generating an AST is like the simple part, but like, I don't want to deal with all the, like, transformations or optimizations of code generating those kind of things, and- and so it's also interesting how different people perceive different parts of these things to be hard or easy, or like in similar kinds of like tasks, like a lot of web people think that embedded is kind of like intimidating, because they think it's like something that's super hard and like kind of impossible, and it's like the amount of like crap web programmers have to deal with and juggle with and like the complexity of APIs and the complexity of the platform, like ARM is a simpler platform than the browser and so if you can manage the complexity of the browser and you feel like you have a handle on like all the stuff that's involved in like building a web application, like, I guarantee you could learn how to write embedded stuff if you wanted to because— while everything is new and different and it's its own challenges I don't want to say that it's always easier but I will say that I think it's conceptually simpler in many ways or at least it's simple enough to the point where— If you get your head around the complexity of the web development, you definitely can get your head around the complexity of embedded development if if you want to.
James Munns
Yeah I feel like embedded has not done itself many favors historically, because it was that same kind of crowd as the OSDev Wiki, where there was the folks that are like, “It's hard and it builds character, and it's awful because it has to be awful, and if you can't handle it then… that's on you,” where it doesn't really have to be, especially like you said in more- historically embedded has had a problem where every CPU manufacturer does things in totally different ways and their compilers are very bespoke special things that are really terrible and usually you had to pay a lot of money to get them and things like that. But at least in the last five or ten years is, ARM has become the sort of default option for this and the ARM architecture is fairly standard and is supported by open source tools and things like that. The accessibility of that has gotten exponentially easier, but the community around it hadn't really grown— like I think the- the biggest milestone in in my eyes for a long time was Arduino where you could pay thirty, forty bucks and get a board, and there was an open source compiler that supported that board and there were code examples that could let you sit down and go I want to blink a light and I don't need to know theory of how my interrupt controller, or how stacks work or things like that— I want to blink a light and there's a lot of things that like you know… A lot of engineers will point fingers at Arduino because it does make a lot of abstractions in the name of simplicity rather than efficiency, and you get a crowd that just won't put up with that— because for embedded, when you only had a 1 Megahertz CPU with 512 bytes of RAM, you couldn't afford abstractions— but those aren't really the days that we're living in anymore like…. My last customer project was on a 32 Mega- or excuse me 300 Megahertz CPU with I think 512k of Ram and 2 meg of flash or something like that, and I was building a TCP stack on it. So we really don't live in that super constrained world anymore. But I feel like those communities are still populated by the people that they were up until maybe the last four or five years, where open source really started to make headway beyond Arduino, and whether that's the real-time operating system communities like Zephyr or FreeRTOS and things like that, or languages like Rust or Zig or TinyGo and things like that who are aimed at, “Hey! If you can write web applications in Rust, you know what you need to know,” and Rust has done a good job of saying— we won't have a one-size-fits-all option for things, like what async executor you have will leave an executor shaped hole in the language. And that way, on the desktop you might have Tokio or aysnc standard or small but on embedded you could write an executor and the one that's most popular today is ah Embassy. And Embassy is really neat because, exactly like you said, embedded systems are a reactive environment. Interrupts are literally immediate callbacks of an event happening, the same way that a signal comes in from your operating system and tells you a packet has been received. It's a signal that says you know you got some data on the serial port or something like that, and it ends up being the natural model. But like you said, because languages didn't help us deal with complexity before, the only way you could deal with it was by putting a very simple abstraction on top of it… and I know Dirbaio, the author of Embassy, before he wrote Embassy was trying to take a C real-time operating system— I think it was Zephyr or FreeRTOS— and run an async executor on top of it, and he was laughing at exactly the reason you said. He said: well, the embedded hardware is async, the operating system in C puts it all synchronous and with threads, and then I was trying to put an async executor on top of that, so it was doing this whole round trip of of functionality for no reason other than I guess existing code and things like that. But, that's what kind of prompted him to really sit down and write Embassy because he goes, “That that doesn't make sense like I want to do something different and why at all of these towers of complexity in between What the hardware is and what I want to be doing.”
Steve Klabnik
Totally. I mean, this happens at every layer of the stack we like rebuild these abstractions on top of each other and then eventually somebody comes and like smashes through several layers and it's like— that doesn't matter anymore like… If you look at the- the, you know, the rise of like ah, people talking about, “Oh, like, are you doing like server-side template generation for your web app nowadays?” and like you know, it's like— wait, like what year is this again? Like, isn't that how, like, these things start or whatever? But, you know like, it's like okay, like, we moved everything into the front end, and it sounds like— well, maybe that's like not a good idea. So let's rebuild the concept of like okay: it's a frontend-first app that you can like, server-side generates some templates for, you know, but it's like running on a thing that like is you know, still originally generating your first response anyway, and it also happens on the lower level of stack. So like one of the things that we're doing at work is like rewriting all of the super low-level stuff because you know… I think it was Brian that put it this way recently, but he's like: an operating system, if you ask someone what’s an operating systems' job, is it would be to abstract the hardware. But the problem is, is that on a modern like computer, and operating systems- operating systems are written like for BIOS or UEFI who actually abstract the hardware, like, OSes are like not even actually always- like they are abstracting the hardware in a certain sense right? But like, they still aren't the thing that's, like, doing the lowest lowest level parts of, like, the boot process. There's some other software that you use that is going from like, you know, turn the power on to, like, handing it off to k-main. And like you know, in- in the old days that was entirely part of your operating system. But we've- we've also grown a layer under OSes that abstract hardwares. And like, UEFI like definitely is an operating system if you squint at it, as so many services like BIOS you could make a case like: okay, this is just like a standard spec or something, but UEFI is super moved into the land of like: this is an OS you build your OS on top of, as opposed to like anything else, and so um I think eventually one of my colleagues is going to be doing a talk on how we're doing this with ah like Illumos is like the unix that we use for internal stuff, but like we're basically getting rid of all those layers and just like making the operating system actually boot the hardware again. And, you know, it like- stuff boots way faster when it turns out when you kill multiple abstraction levels. We have the the privilege of knowing what our hardware is, and so it's not as simple like the reason those abstractions grew is for good reason, which is you have to support all these different configurations and like we don't have to do that, and so everything is tradeoffs, but just like the point is like: up and down the stack we reintroduce abstractions for good reasons and then we had more abstractions and then we realized that like, wait a minute: we're building layers and layers and like— let's maybe like consolidate these again or like try something else. It's, like, different and yeah, just like happens everywhere constantly. It's like- it's like, kind of cool and kind of frustrating.
James Munns
I mean there's also trends and things like that, I mean my running— I don't know if it's a joke but… it's my understanding is that serverless is just CGI-bin V2, where it's like it's someone else's hardware, and you have some code that you want to run when a request happens, and the way that we do that is totally different today, but the net effect is the same— where maybe you don't want to run a PHP script on every request on shared hardware with no isolation anymore, but I mean you're running a, you know, bottle rocket instance or something like that, because all you need to do is wake up talk to the database and then kill the program and we're essentially letting the cloud providers provide their own operating system basically in a way.
Steve Klabnik
Yeah, it's definitely related to unikernels and stuff. I do think that joke is funny, I do think it's slightly misleading because serverless is more about the deployment model than about the way that your program interacts with the nets stack or like whatever but is definitely like hilarious situation or like it's always it's fun to like, tease apart why people make the jokes the way that they do and like what they mean underneath it, and so I think that's definitely yeah, like… There's definitely some truth there. I just think it also kind of misses the point, which is why I think it's funny and not, like, an insightful decisive criticism like some people seem to think it is or whatever. Yeah, totally.
James Munns
No, and I don't think it's necessarily a bad thing because like you said: the- the main deployment is a really interesting part of that is: the programming model might be very similar, conceptually similar, but the reason that you do a lot of this serverless stuff is because you can run these much closer to the edge, in that you can run this code in the region that your end user is in, rather than forcing all of that traffic to come to US East or whatever.
Steve Klabnik
Yeah, like when I was deploying CGI bin applications I was writing a script to like SSH into my 5 servers and copy stuff into a directory, and then like restart my web server, and start putting load-balancing stuff over, and like then rolling back if it failed, and like dealing with all that like manually… And with serverless I, like, click a button, and now my code is running everywhere in the world. And that's like the stronger distinction for me about that programming model, is not the, like, way that my application interfaces with the infrastructure. It's that- and that's what people are like, “Oh, it’s serverless, it has servers.” It's like yeah. Not the point. The point is you don't have to think about those servers. In the old days, I had to think about those servers and how many there were and how that stuff works and so you know that's like the significant difference is the ‘click a button and you're done’ instead of ‘oh god, did my rollback fail, and now my application's in a weird state, and everything's crashing, and like you know I now have downtime,’ or like whatever.
James Munns
That's one of those hard things is, is sometimes complexity exists because it needs to, and sometimes complexity exists because that's the path of least resistance and in in some cases that complexity exists where people will make something simple or they'll make a programming language that is very simple, or they'll make abstractions that are very simple, but they're not necessarily right. Like, strings are always the one that I think of as, as many languages just call a string a string, but Rust some people get frustrated with because there are so many flavors of strings, and that's because Rust is going, “Well no, really, there are many kinds of strings and they all have different rules” and I don't mess around with kinda. But- I mean there are certain things where like you said that complexity of having a load balancer and having rollbacks and having those kind of things: that's complexity that exists for a reason and up to a certain scale, you could probably ignore that or up to a certain reliability guarantee you can probably ignore that, and it's fine. But once you get to the point of, you know, pushing things out to a certain edge or or pushing things up to a certain reliability, you need those concepts because they need to exist, versus when you're talking about these operating systems models, sometimes it's just because that's such a wide span that no one really understands the whole picture, or how all the pieces relate to each other, and that's just because people have a limited cache size of their brain and there's only- you could only be a specialist at so many things, and taking a very systems approach to that requires either a lot of luck that you've talked to the right people and been in the right places or the right collection of people or the right team that you have to happen to be on where you can go well no, if you squint at this, this looks like this and why are we doing- doing this and undoing this and doing this and undoing this all the way up and down the stack, and do I have the ability to impact that kind of change because like you said, if you have to support 50 different CPU models from different vendors and, you know, different versions of the bios and different versions of UEFI you may not have a choice because that's the only way that you're going to work everywhere is if you play the complicated game and go up and down the stack. But sometimes you have the ability to change that, if you're lucky enough to be in the right position at the right time and the right combination of circumstances, like it seems like Oxide is for for its products.
Steve Klabnik
Yeah, but even then there's also a difference too like- like Steve Wozniak could fully understand an Apple II, like he understood the hardware. He understood the software. He basically wrote all of it almost in my understanding. I think he even said one time he believed it has no bugs, which I think was mostly a joke more than something serious, but it was like a statement about- that was like a level of complexity that a human could understand. But like we left that world shortly after like I'm sure he was not literally the last person to fully understand a complete hardware software system, but it was definitely still in the late 70s or early 80s when that was the case. And so even though while Oxide like we as an institution, we'll be able to understand our whole system, there's no individual at Oxide that is fully currently able to understand all of the details of everything; and you know obviously at a high level, we understand the whole system but like nobody is working on every single one of those codebases because it's 40 people and one person can't understand all the things that 40 people can, basically by definition… We've reached the point with this kind of like reliability and complexity and like you know an Apple II didn't need to be on the web 24/7 responding to millions of customer requests or whatever and like because it couldn't do that and so you know we just like— We have higher expectations of our tools in our systems now, and so you know we have to think about them. This is also I think is a good segue into something else I've been totally obsessed with lately which is traffic laws and road safety and transportation. I swear this actually makes sense— but like I've been getting really into that stuff lately and, what's really interesting about it— this also kind of wraps back around to to Schiphol and- and Amsterdam— the Netherlands is really good at all this stuff and part of the reason why is that they basically acknowledge that traffic and transportation and city management is a systems issue, not an individual issue. When I'm learning about this stuff, at some point I watched a video and I was like- on how the Dutch design roads, and I was like, “Oh my god. This is like Rust in a different domain.” Basically… like the attitude in the US- like literally… do you know anything about how we set speed limits in the US?
James Munns
And I know for certain things like curves it's limited by, you know, the ‘are you going to fall off the edge’… but in genera,l no not really.
Steve Klabnik
Yeah, so basically, what- what they do in most places and this started since like the 60s, is like handbook someone wrote, and it became sort of like national policy: you- you send your cheapest intern with a clipboard to the side of the road on a bright day, like normal weather conditions, and you count- you take them up like a speed gun and you're like, “How fast did everyone drive on the road?” and they write them down and then you chop off the top 15th percentile, and you round it off to five miles an hour and that's what the speed limit is set at. And so like sure that works and like it is safe in a certain sense but like… you can understand why the US is often such a pedestrian-hostile place, when like the speed of cars on the road is entirely determined by the actions of drivers and not on what the like actual safety properties of those cars/the drivers are… Or like you know, if there's like mixed-use like with cyclists or- or pedestrians or crossings, like any of that kind of stuff. And so you know, once I like started thinking about this principle, I like see it everywhere: like there's ah, there's ah, a street kind of near my house that's nice and big and wide because we decided safety means lots of gaps, so we made roads large but that just means people feel safer to drive faster— and so to combat this, we put up signs that are like ‘Please watch for pedestrians.’ Ah, or like there's also a road that's like in a different part near my house that has like signs every like 500 feet or so that says ‘This is a very dangerous road, please slow down.’ And like, the idea that those signs are a thing that anyone reads and pays attention to and then slows down is just like ludicrous, because like it's the exact same thing as letting people write C with pointers. Sure, we can say, you know, please don't do the bad thing— but people are going to ignore all those signs, and they're going to make bad decisions that like cause harm to other people. And so what the Dutch do and like what has been a little more trendy amongst ah people that like do this kind of work lately, and it hasn't really taken off in the US but people are kind of arguing for it is okay: We look at like all the uses of the road— who's going to be using this and what's the situation? Oh, this is gonna be an area where a lot of people are walking? Like, great! That means we need to design a road where the maximum speed that someone feels okay driving at is also a safe speed for people to be walking at. They don't rely on speed limit signs to- for people to like understand how fast the road is they simply design the road such that you driving down it feel like that you're driving at the correct speed, and that means things like they use speed bumps more than we do. They have smaller narrower streets because narrower streets aren't more dangerous, they're only more dangerous if people drive faster but people don't drive faster on narrower streets. So like, you know you kind of like shape the environment around pedestrians and cyclists and cars so that they all sort of like- they have an affordance to do the safe thing as opposed to the dangerous thing, and like yeah you can still speed in Amsterdam if you want to, but it's going to be harder and feel dangerous and weird. Just like in Rust, you can write a pointer to integer cast and then dereference some arbitrary memory if you want to, but it's not going to be as ergonomic as you know x+=1 or like whatever. And so there's kind of like this similar attitude of systems thinking and safety. Now we're acknowledging like like part of the reason these older systems languages let you do whatever you wanted is because as you mentioned people didn't have the- the resources like CPU and memory was so small you couldn't afford extra you know, sort of checks or whatever. We're not in that world anymore, and we're a world where complexity is so big and so strong that it's like basically required that we start thinking about things as like a systems way, and like how do I not rely on my limited individual brain power, because not only am I probably working with a team, but I'm working with me 6 months from now when I forget how I wrote that system and what all the variants are and so we need to start thinking about like— How do we nudge people towards making the right choices by the systems design, as opposed to hoping that people make the right decisions all the time? And so you know I think there's a lot of interesting things to learn from like both traffic and programming languages basically.
James Munns
Growing up in the US you have this concept of the Netherlands being this bike paradise and things like that, where you know everyone rides their bikes everywhere, and not realizing that that is much more modern than you think it is where I've seen pictures of- of these neighborhoods in the Netherlands where this happened in the late 60s or early 70s or something like that and there was huge blowback of ‘No, you can't take away our cars’ There were- there's this video of someone getting out of their car and tearing down ah like a barricade or something like that and getting into a fight on the street about it. That happened, but that was in recent history, you know, probably in the life of many of the people who are listening to this and things like that where that change that is now taken for- forgranted in the Netherlands was something that had to be a fight at some time.
Steve Klabnik
Yeah, it was a deliberate choice to make- to say like, “We know there can be a better world, and we're going to do it, even though it's hard and tough and controversial to convince everyone.” Yeah.
James Munns
Yeah, yeah, and I think Rust is- is sort of like that too, where you get- I personally can remember the shift in- in opinion of when Rust 1.0 came out, they go, “Oh well, that's great, well, it's kind of niche. You know, it's- it does a thing, and that's interesting. But it'll never be adopted for hyper for you know, embedded or whoever, these industries just don't change, and it's always been like that…” But in the last 5 years, you see that opinion in online discussions especially changing, where there certainly are still people who have those opinions and are not afraid to share them. But there's usually a counterpoint, and often people going, “No, we switched over, is fine.” It's noncontroversial and non-surprising anymore and I think that's really the the sign of success is when it- it will be controversial when you switch, but it's amazing how quickly that becomes uncontroversial, and that's just in the span of the last 5 years, watching it go from I guess 1.0 was more than 5 years ago at this point… Ah, I can't keep track of time, but…
Steve Klabnik
Yeah, 7, 7… yeah, time doesn't make sense anymore. It's also similar to watch like people talk about Covid and masks and- and like I've seen so many people tweet things that are like, “When we said you couldn't smoke in public anymore, did people like get upset about that?” and you're like, “Yes! Yes, they did. Very, very much so!” Like drunk driving— like, do you know how many people got like upset when we said you couldn't drive drunk anymore? Like that was like an actual like- that was older than I'm alive, but like you know, I've like heard about those things, but I I remember, the smoking thing happened when I was a kid and like yeah like.
James Munns
Yeah I remember going to restaurants that had a smoking section that was- you know they have a one and a half meter like four or five foot tall glass wall and they go “Don't worry, the smoke's over there…” and you're like, “There's no circulation or windows in this Denny's!” like…
Steve Klabnik
Or the same thing— when I started flying, the first time I got on a plane, I'm like, “Why do they keep harping on like, ‘you're not allowed to smoke here’ like, who would- who would want to smoke in a plane?” and then you like find out like, oh yeah, people used to smoke in planes all the time! You're like what! like… And I'm even somebody who smoked for a while, and I'm still just like, I wouldn’t smoke in a plane, like are you serious, like what's going on… There will always be this pushback against like, you know, trying to to design from systems first. But… It's interesting because I think a lot of people who get upset about this see it as something that's, like, deeply anti-human like, they're like— you know in the US, we'd say freedom, but like in many places it might be creativity, is like going to be taken away by these like constraints or whatever. But I'd like find it to be actually like deeply human instead, like we're acknowledging that we're fallible. Like sometimes I'll joke that like I like Rust because I can write it when I'm drunk, because like the compiler will stop me from doing anything too terrible. Or like whatever, I'm sleepy or tired is like a more like— not silly ridiculous scenario— is like yeah, like, I can still do this kind of stuff and, you know, it'll check my work for me and like you know that helps— or you know anything else or like I said earlier if it's 6 months later and I've just forgotten the context, then I do something that's like pretty ridiculous like oh yeah, like you know, I totally mess this up. And so I think that like, acknowledging that we as humans are fallible, and so you know we don't want to totally cut off the ability for you to say like, “I know better, please let me do this thing here,” because I don't think that works either, like I think that would be, you know— it's not not always terrible but like in some circumstances that's going a little too far, but like I don't think that inherently flipping the default from unsafe to safe is like… an abridgment of everything that makes me human, or like whatever some people try to like get upset about… I think it's more of an acknowledgement of the way that we operate and using tools to help us do better is like a very pro-human stance.
James Munns
Yeah I mean like in working at safety critical, that's the whole reason you have these processes: not because it will be perfect, because even in this safety standards, you're never allowed to say something is defect free. You only look at defects from a perspective of statistical likeliness. And you can do certain things that are shown or believed to reduce defects, or to make them more predictable, or to force them into certain categories that are more easily visible than unexpected defects and things like that. But there is no, you're not allowed to put. Expect to defect rate 0 on anything you say, “Okay I have 5 9s of reliability,” or, “6 9s, or my meantime between failures are X,” and then you are forced to look at that and go the the safety standards are not measured in, ‘will you have bugs.’ They go, ‘how many bugs are an acceptable level’ where you say I will have a million of these units in the field operating for 12 hours a day simultaneously. We expect to have n number of incidents of like near hits basically of- of something went wrong, but it didn't impact anyone. Or X number of incidents that will happen, and we know that that can't get down to 0 but if that happens once a year, pragmatically we might say, “Okay, well, the positive impact is X, and we can't get this to 0, but this is a net benefit,” for things like you know, adding a piece of equipment that in the normal case helps you a lot, like GPS or something like that. But if it fails, it might end up causing a problem of its own because it gave you wrong data or incorrect data that wasn't detected as wrong. But you say that the net benefit is: It's likely to save more lives than not doing this in that impact. And usually it's a much more slanted equation, where to change something you you need to be visibly better than the what was there previously. But I mean yeah, it's like seatbelts. Yes, there's probably 1 case in a billion where someone will get trapped in their car because of the seatbelt. But for the tens of thousands of accidents that happen a year, where that prevented you from going through the windshield that's worth making that that the default.
Steve Klabnik
Definitely. And it is true that like safety critical systems is another thing that people that don't do at a ton like have these projections about what they wish that it was like, and they like don't actually understand what it is, and I say that also partially because I was that person and I distinctly remember when that changed for me. Like I had a conversation with someone one time that was like, I was like talking about this and I wasn't saying like, “Oh yeah, like we should always do only safety critical stuff,” or whatever, but someone was like, “Okay Steve, here's the thing: what do you think about the current state of being able to prove properties and software is?” And I’m like, “Well, you know, we can prove some limited stuff, but like it's sort of, you know difficult to…” like, it's like nice sometimes, it can prove things about some systems, but it's definitely not perfect, because like how do you know your proof is correct and like proving properties is really complicated and expensive and and takes a long time, so doing it for like a whole application would maybe not be great. So like hopefully you can prove some things about some stuff and like you know, look at seL4 and like it took years to verify and like there still were bugs afterwards like all this good stuff and they were like, “Cool.” So, why do you think that like software and airplanes or cars is formally proven? If they have some secret way that was like able to prove a full industrial application, don't you think you'd be using it in your web application too? And it's like “Oh. Yeah.” Like, they don't have some sort of like secret ultra safety, no bugs writing techniques or tools like locked up that they like don't share with the rest of the class, they're just using the same stuff that we're doing. And maybe they're doing it more rigorously or they're using more of those tools or whatever but it's not like: Okay, you say that like you don't think that Rust is going to be able to be work on a car because like there's no spec and C has a spec. Do you know any bug reports there are open against the C-spec right now? Where some of them are like contradictory language where 1 section says X and the other section says Y, and those 2 things can't be important simultaneously. Don't you know that would like, would make make the proof false right? Because it's contradictory and then you're like, “Oh.” And then all that like- things that you wish— and it's like kind of terrifying, like I understand why people wish that car software was formally proven, because like, dear god like that's software that's going to be deciding if I live or die in someday in some circumstance, but it's just like not the case. Instead, you acknowledge that people are fallible and systems are fallible and you're like, “How do I reduce the failure rate, understanding that it can literally never be 0,” and so you know like a specification is not a guarantee of correctness, and so like you know it's not really about having that exactly, and like you know there's all these things that people wish were true about this stuff and then people perpetuate the story about how like you know, oh yeah, like you can't even be considered to be using a car because there's you know, no particular like spec for Rust and it's like then why do these 3 auto manufacturers have Rust jobs open right now? Like what- like where- how- how do you reconcile these beliefs that you have about the world and it's just you know, not not true in that way.
James Munns
I mean, it's good to have heroes. But I mean there- like you said sometimes, it becomes legend more than actual knowledge or it's easier to succinctly say, “Oh no, you can't do X,” but like you say, a lot of the people on the internet have no like- there's no qualifications required to make a comment on the internet for better or worse.
Steve Klabnik
I had these beliefs about safety critical systems and I've never worked in a formally safety critical system, I've only learned more about that environment over the last 4 or 5 years but I've never done it, but I would still totally say things about it, because I'd heard them before from people I thought made sense and it sounded like it was right to me, so I said it again. You know like yeah.
James Munns
Yeah I mean it's an easy to defend position. Especially if you're talking to someone else who also doesn't have proof that that's not true. But I love your your mention about formal modeling and things like that. So most of these safety standards will have sort of a graduated approach for exactly that reason of the amount of formalism they make you go through is proportional, at least in aviation which is like my primary area of expertise for safety critical. The levels go from D to A, where below D would be considered not safety relevant. D is the lowest, level A is the most strict, but you measure— not quite this simply— but you measure which standard your piece of software has, or your system has to fall into, by what happens if your system goes wrong.
Steve Klabnik
“If my- if my movie skips, then like it doesn't matter and that's like the below D part. But if it's like ‘I tell it to go right and it goes left’ that would be the A part” is my lay- that's would be like, my lay guess, is like what you're talking about right?
James Munns
Yeah, So it's- I mean, it's something like D would be ‘would cause a minor if not noticeable inconvenience,’ C would cause a major inconvenience and possibly some injury, B would be some death and definitely injury, and A is like ‘complete loss of the aircraft.’ So for example, I worked on for a number of years a system called TCAS II which is Traffic Collision Avoidance System. So its whole job is to keep you from running into another plane. That was level B. Level A, like you said, you tell the system to go left, and not just it goes predictably right, it goes wherever it feels like. You have lost control authority on the system and in most cases, formal verification is only practically used at level A. Like when I worked on a level B system or level C systems in the company that I worked at, we had very strict rigorous process like you said and we had high requirement review and basically a lot of testing, like it prescribed a mechanism of of showing you how deeply you needed to specify and validate your system, but not formally like it's writing unit tests and writing system tests and writing ah robustness tests or stress tests and things like that and making sure that you go through those processes and do those things but at the end of the day they're just best practices they're not- they're not anything that you wouldn't do if you were writing Amazon's routing back plane because you lose millions of dollars and a minute that that system is down or something like that.
Steve Klabnik
Right, and they use more formal modeling than many internet companies do for exactly that reason right? Like they're they're known for using TLA+ to do stuff in AOS to like check properties of those systems for exactly their reason. Yeah yeah.
James Munns
They have a whole group. What is it? Amazon's automated reasoning group. They have a whole, not just like practice but they have a whole research and application team that figure out what parts of formal verification can we apply practically for a net benefit to our system and things like that. But yeah, I mean the the spec thing is definitely- I mean I think this is how we specifically we've been talking about having one of these chats for a while but what actually came up is someone was talking about- I think it was Mara was like—
Steve Klabnik
Mara- Mara asked if we should have a spec and if it should be an Iso or like C or C++ yeah
James Munns
Yeah, and you and I sort of both jumped in there and I provided a little bit of context, and someone chimed in with me and said- I said yes, it should with some caveats and things like that and someone goes, “Yeah you- James would definitely know, you definitely need a spec for safety critical.” My own answer was like, “Oh, Ah, it's complicated.”
Steve Klabnik
You- I think you said, “Sorry, I'm tired, but that's not correct,” and I was like, “I am not tired. Let me elaborate for James!”
James Munns
Yeah I mean my ability to sit down and write a blog post and like have structured thoughts is particularly difficult and that was a question that would require, I guess it's just the internet. I know that if I post an opinion on the internet and put it in written and form I need to come correct and have sources for all that and I didn't have the energy to sit down and write a well-sourced and founded—
Steve Klabnik
A thousand percent yeah.
James Munns
Ah yeah, so I mean I'm glad you came in, because if I was going to write it on Twitter you wrote it the way that I was going to write it on Twitter which is basically: you want it because it helps. It is something that makes things more predictable, because most of what safety critical systems focus on is predictability rather than perfection if that makes sense, because if your system fails in a predictable way, you can tell a pilot, “Don't do this,” or you can tell a programmer, “You're not allowed to use variable length arrays, because they're just known to have defects in many compilers.” You just say- like that's what the MISRA spec is. Is it's a list of things of ‘don't do this because we know it's bad,’ or it is- ‘we know it is easy to get wrong.’ So you put that constraint on, and that's almost a workaround, so people will often ask me “What do you think that the Rust MISRA will be?” and I go, “Safe Rust.” Like it's it's you try and make sure that robustly you make the the default mode as predictable as possible, and even when people talk about MISRA and stuff like that a lot of them don't realize: you can break MISRA rules. You can break whatever coding rules. It just means that it's not the default, and often what this means in aviation or automotive is, if you break or bend the rule, you have to write essentially a justification for why this is acceptable for this purposes, and that's always more work than just rewriting the code to not do that, unless you really need the code to work in that way.
Steve Klabnik
Right? As an example I guess if people aren't familiar with MISRA: my understanding— if I remember correctly because I've never paid for the spec, so I've only seen drafts and it's been a while— But one example is like, there can only be 1 return point from a function thing, you get 1 return and you have to write your code says that there's no early returns. And so it's like, sure, you could write some code with an early return, and I personally think that's a bad rule because I like early returns. I understand why it’s in MISRA or whatever, like the point is not to say that that rule is good, but just like would you rather rewrite your code so that's a little more awkward but returns once or write a 5 page paper on why you're allowed to write an early return for that function and then it’s fine? And like the answer is the former for sure like for 99.9 of people and so therefore most of the time you've now you've now written a development system that will get people to have the outcome that you want even if, you know, they don't necessarily want to or like whatever else.
James Munns
And I'll admit, working at companies where we wrote a blanket exception for that. But we wrote a separate rule that says early returns can be used at the top of the function for error checking. So basically the rule was you can have returns at the top or at the bottom. You couldn't have it like mid-loop or something like that, because I understand the justification that they're they're making is that it makes control flow harder to reason about because you do have multiple ways that you can enter and exit- may not enter, but multiple ways you can exit a function and things like that and I get that. But like you said, if you're doing two null checks at the top of the file, you don't need to have your entire function nested 8 spaces over just to do that, and the alternative would either be not checking nulls, which would be bad or to yeah to have an awkward function. So, but that's- that's within the spec is that if you as an organization say, “We have an in-house coding rule that supersedes MISRA and we say “MISRA, accept these 2 exceptions and here's the 5 page paper of why our coding standard is considered acceptable, and here's how we will consistently apply that rule within our organization.” Whatever: like all the rules can be bent and broken. But. You've got to with a straight face take it to a regulator at some point or an auditor after something has gone wrong and go tell me why you did this because you bent the rules and your system crashed, and we'd love to know why that wasn't the problem. And you'll either be doing that with lawyers or in aviation, it's more upfront where you'll have a regulator look at it beforehand. In automotive in most places in the world,
Steve Klabnik
After the fact, yeah.
James Munns
— you do it with lawyers after the fact. But yeah, I mean the the net approach is the same, is that they just want you to have a system, because consistency and having a system is is more important than the specifics, if that makes sense.
Steve Klabnik
Yep, definitely.
James Munns
I know there was also a couple of other other questions, because I did ping people of “Hey, I'm going to be talking to Steve, what should we talk about…” Okay, so Walter asked a really good question that- this is one of those hard questions that I don't know how to answer of: “How do you draw the rest of the owl?” Because when you learn a little bit about embedded, you learn how microcontrollers work and you learn what a serial port is and you learn about those kind of things and you blink an LED. And then something happens in your learning process and then you've built a real-time system or an operating system or something like that. So, since you've been through this more recently than me, what was that process like for you in that going from— you mentioned, having helpful coworkers— but: what kind of process did you go through with those co-workers where you go, “Look, I know Rust, and you know embedded systems— how do I get to the point where I understand what DMA is, or what a specific kind of serial port is, like and how do I learn how to apply this practically in the system that I'm building today?”
Steve Klabnik
Yeah, so the first thing that I did was go through The Embedded Rust Intro Book, which is very light but at least gets you doing hello world and you like have a board and you know you you do some stuff, and it kind of like gets you going. So that was like sort of like step 0 was like, okay like setting up an environment, like the original purpose of hello world was not to get something printed on the screen, but to demonstrate that you could like build and run an application and see this successful result, and so I think The Embedded Rust Book is really great at that. Like you know here's the thing, get the light to blink, like you're good. But then I think a lot of the the pointers were more like that were good, not just from like actual conversations but with coworkers but the things that I could like replicate and tell you about are more like um, okay. This is the— I totally am forgetting the title and it's not on the bookshelf behind me, so maybe you'll remember the actual title— but there's basically a book that's like, “This is the ARM architecture,” and you can like read it, and it will explain the general concepts, because like ARM while there’s like obviously different versions and stuff is a little different, there's like still consistent principles across all those different architectures, and there's like kind of this like conceptual understanding and it'll layout like okay like, here's what happens when one of these machines boots. You know it looks up, you know, the the vector table is like located at this part of memory and like that and it kind of like walks you through all the different parts and pieces. And so like I read through that, and that gave me kind of like a good enough conceptual understanding of how the platform worked at a more high-level— I mean obviously it's still low-level in the grand scheme of things— but it's like not to the degree of like here's my particular chip or board I have or peripheral or whatever. It's like here's- here's how an ARM system kind of like works and is roughly like laid out in that kind of stuff. And then from there it was pointing out like how to find this like documentation for the given like bit of hardware that I'm using. And this kind of this interesting thing, I had a really good conversation with Esteban on Twitter the last couple days about like ah there's an interesting relationship between error messages and compilers and docs in that like people have always thought they're bad, so they don't read the error messages or read the docs because they don't trust that they're good, and so for those of us that have put in a lot of time and effort to make those things good for people, can be very frustrating whenever someone's like, “Oh yeah, I didn't read the docs, because everyone knows docs are bad.” And it's like: I spent years of my life making sure that they were far and above, you know, what you expected, and so there's that, but there's also a really interesting like… it's not actually the Uncanny Valley but there's a similar kind of effect… Ever get like a bad Rust error message? And you're like, “Man, this error message sucks!” And then you realize, like no— this error message is still good. It's just like, my expectations are so high now, like if I got this- if I got this error 10 years ago in a C compiler, I would be like, “My god. This is the best error message I've ever seen.” But I'm so spoiled by being really good, that when I get one that's like bad from my new perspective, it like feels bad even though it's like kind of crappy. It's like way better than things that came before. This is the way that I feel about data sheets. Like data sheets for like- so when you- when you have like a particular, you know, board or chip or whatever, the manufacturer provides a datasheet that like gives you all the information you need to know about how to interact with that peripheral, and there's definitely a wide spectrum of good to bad in those kinds of docs, but it's still like better. There's no like Ruby on Rails datasheet. Yeah, there's a collection of docs people wrote but there's nothing that's like as comparable and comprehensive I think is the way the data sheets are, and therefore like on some level like how you reconcile like how do I do this stuff is you're like, “Okay, I want to generate a random number. Let me look in the table of contents to see what my RNG hardware is,” and then it'll tell me go to page 930, and I go to page 930 and it's like, “Here are all of the, you know like- put this value in this register to get this particular kind of RNG generator, and then, you know, call this function to make it do its job, and then it'll call back to you with bla bla blah or whatever. But it'll all explain that in text in the datasheet and I'm like, not used to that level of documentation for stuff that's not you know, like the Rust language itself like any sort of tooling or external service or whatever. Like the fact that there's a giant-ass PDF that basically has all of the you know details in it is like pretty good, but the the downside of that as newbie is: those are also written by humans who are also fallible and sometimes they're wrong or incomplete or have inaccuracies, and so I think that like the second layer of drawing the owl is like being able to figure out like— okay, am I missing something here, or is this- that thing actually wrong. And I think you can guard against that in sort of 2 ways, and the first one is using less exotic peripherals. So like you know, buying the STM32F4 discovery board that like so many people have used, like they've sanded those docs down really well because that's really popular, whereas if you buy some sort of like you know $2 tiny MCU off of Alibaba where the spec sheet comes like only in Mandarin, like you're going to have a much much harder time and it doesn't mean that that chip is bad. It's just like you don't have the skills to like be able to deal with as averse of an environment you know. And so ah like I think like using better known stuff where you have a lot of support like ADA fruit stuff or like you know, whatever is kind of like a thing that's like more commonly used by hobbyists will tend to have those cowpaths paved for you already, and that will make that task easier, and then like as you as you get more comfortable with the terminology and doing that kind of stuff, you can start to branch out into things that are like a little harder and a little less documented, or like a little more difficult in that sense. But, it's really like understanding that like you're not going to be Googling for blog posts or going on stack overflow, you're going to be reading giant PDFs and then trying to like figure it out. And I think the other thing that's like the hard part about drawing the owl is like debugging, because I don't really use a debugger in Rust very often, and with the degree that we've run into with the brokenness of tests for the dwarf Metadata Rust C produces, I don't think many other people use Rust debuggers like that often or at least not in the way that we use them at Oxide very much but like the thing about it is you will need to get more familiar with a debugger and the- the tools and like that kind of stuff because you don't have the same capacity to understand what's going on in the system than you did before, and so that's I think the other part that's like a little challenging is getting used to like debugging stuff. Yeah.
James Munns
I think the document that you're talking about is called the ARM ARM. It's the “ARM Architecture Reference Manual” which is hilarious that they call it the ARM ARM.
Steve Klabnik
Yes, well the other thing it's amazing is like ARM is great with these puns right? Because you have the like A profile, the R profile, and the M profile right? When I first like realized that- that was also the case, I was like, “Oh my god, this is like amazing…”
James Munns
And the instruction set for smaller ARM chips is “thumb.”
Steve Klabnik
Yes, ah totally. It's like very very funny.
James Munns
Yeah, but I think that's a really good point I think reading a datasheet is probably the most critical skill for embedded, because like you said, that's what you're going to spend a huge- It also depends, because embedded can be weird because you could be on embedded…
Steve Klabnik
Linux!
James Munns
— if you're not writing drivers…
Steve Klabnik
Yeah, like you be on an embedded Linux system and you're basically writing userland software. Yeah, totally.
James Munns
Yeah, yeah, then it then it becomes, even if it's not Linux, even if it's on an RTOS, or you're at a company where someone else has written the drivers for you, it just feels like writing code on ah on a desktop but when you get to the point where you have to write your own driver, either for a peripheral that's inside of the chip like you said if there's a hardware random number generator on the chip that you're on, or if you're talking to an external device like a temperature sensor or something like that. Both of those are going to have data sheets and those are the source of truth, like when I was writing this code for the customer recently where I was doing ethernet bringup, that was what I referred to: I went and read that multiple thousand page document, I went to the ethernet set— well, first I went to a couple other sections of like ‘How do you get the clocks up and going, how do you get flash memory working correctly, how do you then get the ethernet turned on and map to certain pins’ and things like that, and that data all came from the datasheet and figuring out how to read those because even though they're written by different people at different companies, there's a style. Like in the same way that I'm sure like if you're in the Ruby ecosystem or in the Rust ecosystem, you get used to reading docs.rs, or you get used to reading a book that comes with the project and things like that. And you you get trained where to look for examples, where to look for types, where to look for functions and things like that, and it starts- you start seeing the meta-pattern after a while because people just mimic what they've seen, and hopefully the best things they've seen are the best things they were able to produce. But, most of the time, they're fairly consistent, like you said— varying quality, like you mentioned ST, ST tends to have not always my favorite, like the Raspberry Pi RP20 has excellent datasheet docs and examples in my experience or Nordic's and NF family have really good docs in my example and still they sometimes have errors or errota, either in the hardware or in the docs, but you can, once you start learning how to read them, you start seeing the patterns of— when I want to know this information I need to look for a section called “this,” like if I want to know where my memory in flash is, I know I need to look for the word ‘memory map’ because everyone calls it ‘memory map.’ So, if you go to the table of contents and look for ‘memory map’ that will take you to the one table that you need to figure out how much flash or how much RAM my chip has. So I definitely recommend reading data sheets and getting used to them, and they’ll be a little bit different for microcontrollers and external sensors and stuff like that. But you'll start seeing the patterns just like get comfortable reading that or even if you have an existing driver, go read the datasheet for that, and try and figure out how that driver works versus— like see if you can make the information line up. The other thing is, like I said there's patterns. All of these microcontrollers will have certain functionalities, like a serial port or a spy port or I2C and at some point it makes sense to go and learn how those protocols or those procedures work, because the protocol is always the same, but every chip will implement the hardware support for that in different ways. But you'll know what sort of the minimum thing you need to do is like— I want to send this kind of message on this kind of bus and all of a sudden the data sheets will start making more sense, because you'll realize they've structured the datasheet, because they know those are the questions that you're going to ask, and you'll start understanding why the specific examples that are there are there because they usually are- they're aimed at developers who are going, “Okay, my boss told me I needed an I2C driver for speaking to this temperature chip. Skim Skim Skim Skim— here's the example I want, I will copy this exactly and write my code for my driver exactly like that.” And getting used to those patterns and habits is really just the difference of- of how much time it takes you, of knowing what to look for when helps you move a lot faster when you're trying to get something done, and those patterns will start to make sense after a while.
Steve Klabnik
Definitely. I think there's 1 other thing too, I would say, which is more of a pep talk than specific advice, which I know the ‘how do you square the owl’ is like asking for specific advice in some senses, but Brian, whenever I was having this interview— so Brian Kanttrell is the CTO of Oxide— when I was saying like, “I don't know if I could do this professionally or not.” He was like, “Okay, here's the thing,” he's like, “I joined son out of college and I showed up to work on the first day and I was like, ‘What are you going to do- like what you know… what's my task?’ and and my boss was like, ‘I need this thing fixed in the kernel, like go do that,’” and he's like, “I'm working on the kernel my first day!” and he's like- what you told me was: “Hey you learned- are- you learned data structures, and you learned algorithms, and like maybe you didn't always use those in a kernel context, but like it's still all just data structures and algorithms. They may be different ones than you've used before, and maybe in a different context and used before, but like: you know about data structures and you know about algorithms, you'll be fine.” And so he's like, “That's the same advice I would pass on to you,” and that's in- I think that's been super relevant and helpful is like: there's been things about web dev stuff, that sometimes when I've learned things about embedded I'm like, “Oh, I kind of like know how this works!” like you mentioned I2C: when I learned about I2C I was like, “This is this simple, like I feel like this is what I would have designed if you would said like, ‘Hey Steve, how do you communicate this information over to the thing,’” and that's the- the wonderful and terrible thing about I2C specifically right is it's like kind of crappy but like kind of great because of its simplicity, but like when you first learn that you're kind of like, “Oh, this is that thing that I thought was mystical and super hard is actually just like just this like. Wow. Okay sure. Yeah, absolutely.” And so I think once you get started doing certain things, like and if you keep at it, like I'm not going to say that you're immediately going to think it's all easy or something, but I think if you if you keep at it at some point you'll realize like, “Oh yeah, like, if I understand how an http message gets sent between a client and server, I can understand how to use spy to communicate between 2 peripherals, like it's the same general idea of like I need to send some bytes from here to there, and you to be reliable- you know what protocol tools I use to make sure that's reliable, and like all that kind of stuff. And like yeah, the context is a little different but conceptually, you know, you're doing the same sorts of things, just in slightly different ways, and as long as you can like remember that you do know things and you do have expertise and maybe it looks a little different but you just kind of like got to adapt what you know to what you're doing, you'll- you'll be fine in the end.
James Munns
It's also really hard when you're when you're contact switching so hard of like you said, if- if you're someone coming from web development and learning about embedded for the first time. Things like Rust and being able to use async and use those concepts helps a lot, because it gives you one stable foot. But it's also important to remember that especially for developers, sometimes if you've been doing one thing for a while people forget how to be bad at stuff. And that you were bad at stuff when you started, whatever you're good at right now, and like you said you probably didn't understand all of CSS and you didn't understand all of HTML and you didn't understand all or any of Javascript or how HTP works or how TCP works or how a load balancer works or any of that when you started, but you were still able to make that little website and that was an achievement at the time. Embedded is a lot like that too. Don't worry about going super deep and understanding how all the protocols work. Like build up a simple model that's useful enough for what you're doing, and at some point that will break. Your hardware will stop working or you thought you wrote a working driver, and it won't, and that will be a time for learning where you get to learn like like Steve was saying is it's- I feel like I2C is like the JSON of embedded communication protocols and that it is very simple and it's very convenient. It's useful for a ton of stuff, but it's so simple that there's room for a lot of edge cases. And as you go on in your experience, you'll learn what those edge cases are, and when stuff starts not working you go, “Well, it's 1 of the 3 cursed things,” like with JSON you go like, “Well, my numbers are floating points and they don't round-trip and turns out that causes problems,” like—
Steve Klabnik
Literally, the example I was thinking of whatever you mentioned JSON, I was like, “Okay, we're going to talk about how if you put a number with more than 50 whatever bits of significant values as an integer, like, does it work…” Yeah absolutely yeah.
James Munns
Yeah, and that goes and ruins your day because now these numbers are not equal, and your system is spinning forever because it's waiting for the response that is equal or something like that like, but you know I2C has its own problems like that where like sometimes devices just get locked up because they missed one special signal in the simple protocol, and then they go, “Now I'm waiting forever!” And that just means that there's nothing you can do now, and there are certain ways to avoid that or reset it and things like that. But those are things that you learn over time and you don't need those on day one. Like you said, especially if you're using fairly well trodden components where people will know where the sharp edges are most of the time.
Steve Klabnik
Yeah, and this is— for the people who ask this question, this won't make any sense, but you'll probably giggle at this— I ran into that the equivalent in the low-level stuff when doing I2C stuff because I had written it synchronously initially and it was time to like make it interrupt driven instead, and so I was rewriting my code to do that. And I totally got it working, and then when I went to go deploy it, it didn't actually work. And the reason was because I was using semihosting to print out the messages I was using for my printf debugging, and I didn't know how semihosting worked, and so it had like so made enough time go by that it totally worked in that way, but my code, once you removed that big, you know, giant pause basically in the middle of things, it now was misfiring. And so it was like infuriating to be like- when I remove my printf, why does the behavior change? And it's like, “Oh, because this is timing sensitive actually,” and like you know… You were like pausing for like an hour in the middle of all of the things, and that's why it worked out. And so that's like another classic like— you start off by learning that like semihosting is really cool and then shortly thereafter you're like, “Oh god, this is actually totally unusable.” For certain actual tasks or whatever… I always think of those problems as isomorphic. Yes, like relying too much on semi-hosting and then also yeah floats in JSON.
James Munns
Or it's like the first time you learn what Nagle's Algorithm is when you're trying to do latency sensitive things in networking and things like that where Nagle's Algorithm is something that aims to make TCP more efficient by: if you just send a little message, it'll sit around and wait to see if you're going to send another one, for some amount of time out. But if you don't, then it'll just sit around and wait and hold on your packet, and if you're expecting something to respond immediately, but you only sent 5 bytes to get the immediate message to go out, you go, “Why is this taking 500 milliseconds or 3 seconds?!” or something like that, and then you get to learn what TCP no delay or Nagle's Algorithm is, and these are just things you learn when you start peeling back abstractions, and they're not necessarily something you need to know on day one, but it's part of the learning experience of: stuff just won't work for reasons you don't understand to begin with. But that's okay, and your mental model will evolve as you get more experience.
Steve Klabnik
Yeah I think that is also a very funny example to pick because it goes back to something we talked about at the very beginning of our conversation which is: John Nagle is one of those people who was like, “Eh, Rust’ll never be used by anybody for anything serious,” on Hacker News for years, and I used to get really really annoyed at him specifically for his Rust post on Hacker News. And now he's been like running Rust only for the last like couple years, and so like watching that change over time… But it's also like- I'm always like, “Why am I arguing with literally John Nagle on the internet right now, like what is- what- what choices of my life have I made to like make this happen?” Ah but it was so funny when I found out, I was like, “Wait… that guy? Is that guy- gah, ah ugh…” But anyway, just a
James Munns
I actually had no idea who the person behind Nagle's Algorithm was… I mean, I guess it it makes sense and tech is a weird small world where… for a long time I was looking for the name of this game that I used to play on my 286 back when I was a kid like, and it was this ASCII art game where- it's kind of like Civilization, but all of your cities and units were just characters, and you'd draw them around on the map, and you'd had submarines and stuff. I was trying to figure out what it was, and I even posted on like Reddit/tipofmytongue or whatever, like people had suggestions that looked sort of similar, but it wasn't the one I was looking for. And I was on Hacker News at some point, and Walter Bright— the author of the D language— was talking about something totally unrelated. He's like, “Oh, I ended up in compilers because I wanted to make my games like Empire faster,” and I went and looked at Empire, and I was like, “That's the game! Like, that's the one that I've been looking for, for years now!” And I just saw the person who is this and I also just realized that it's the person who wrote the D language who happens to be commenting on this, and he ended up making the D language because he wanted to make his game Empire faster. So you get these like weird world where the people… but you assume is sort of like always there established everything it was written just by someone like 30 or 40 years ago and like—
Steve Klabnik
Right place at the right time…. Yeah yeah, John Nagle is the 12th most highest karma person on Hacker News.
James Munns
Oh really! I've probably seen him around, but like I- I don't parse usernames usually on Hacker news. Really.
Steve Klabnik
So absolutely yeah, totally. And I also don't advocate anyone reads or posts on Hacker News, but it's just- it's just very funny when that kind of thing happens.
James Munns
I'm also supposed to ask you about baking, because I remember when the pandemic started you—
Steve Klabnik
The baking sub's good! I haven't cooked as much bread lately. But… so I worked at a pizza shop through high school and college and so I really liked making pizza specifically, and then I kind of like didn't do a whole lot. But yeah, once once the pandemic started we all had to be inside all the time, and other people started baking I was like, “Yeah, I really haven't baked anything a long time and I think that would be a lot of fun,” and so I got really into bread stuff. And I've been doing more cooking lately than baking. So I haven't been posting as much about the baking, because I already eat a like relatively higher carb diet than I want to, and so when you're baking loaves of bread for yourself, ah it is not conducive towards eating less carbs. And so I've slowed down a little bit on it as, I've like, I've put more work into like exercise and cooking for myself rather than like eating out and baking and so I've slowed down on that a little bit but... It's definitely been really nice to have an activity that's like not purely about programming, and you know you get to learn a whole ‘nother set of terminology and get a whole different set of gear. Ah you know, like like things like that for sure and so that was definitely like yeah like a lot of fun.
James Munns
When you were in a pizza place: did you work like back a house or did you do delivery?
Steve Klabnik
I did almost every single job at various points of time. I started off as like a person answering the phones because there was no online ordering in those days, because I'm that old. Ah, and working the line and making pizza, and then when I turned 16 I got driver's license, I started doing some delivery stuff and then eventually became a shift manager, and then eventually I became the like assistant general manager of like the busiest store in the franchise, and then eventually they wanted me to go to corporate and I was like, “I'm going to college instead. Sorry, I want to do computer stuff, and I love the pizza job, but like computers are like actually want to do for a career. So I'm not going to do that,” but I was even- right before college I was, like, helping new franchises open. I would actually go travel to new places and train up their staff, and help them run through their first like Friday night rush or whatever kind of thing so as also a lot of the like Rust teaching- like the teaching stuff goes like very deep for me like, before Rust I was teaching Ruby stuff to people, and before that I was like teaching making pizza to people, so it's definitely been like kind of like a lifelong interest of mine.
James Munns
Nice yeah, I worked delivery but I never worked ah in-house my favorite job growing up where while- like while I was in university and stuff like that was: I worked a night shift at a gas station but it was- it's just a weirdly different thing of just: you get in at like 11 PM, it's super busy until about 1 AM— because where I lived 1 AM is when you stop selling liquor at the gas station, and you could sell liquor at the gas station because we weren't on the East Coast or anything like that— and then it's just silent until about 5 AM. And then you get to know, in order, all of the mail and newspaper delivery people, all of the construction workers, and then everyone who was going to work as like a teacher or something else like that. So like, it was always like a graduated progression of groups of people, and then my shift ended at 7 when when the next crew came in and started picking up the rest of the morning rush. It's always interesting to hear what jobs people had before what they're doing now, because I find people either like remember that super fondly and they liked the things that they were doing, or they're like, “I learned that I hated that,” and if nothing else, you have respect for that. But it… you know that that's just never something you're going to be doing again.
Steve Klabnik
Totally. If like money was no object, I probably would like work shifts at the pizza shop, because I did really love it. I was usually like closing shift manager, so we were open til 2 every day, and 3 on Friday and Saturday, because bars in PA close at 2, so we'd make sure to be open later than the bars. So like, I had a similar kind of thing where you get to learn all these like regular customers who like come in after they've been drinking too long, and like you know they're like, “Oh hey like pizza guy!” like it's everything with delivery— like, I had a driver one time get a $2000 tip on a delivery, and that's because we could accept Carnegie Melon's food program, you know like if you had the meal plan through the school, you could like buy pizza from us. And so there was these kids at this frat house that one of my drivers would deliver pizza to every single Friday, and so their final Friday before they graduated, they basically all pooled together all the extra money that was left over in their accounts, and were like, “Hey man, like you literally have given us pizza every Friday through all of college, and so we really felt like we should just like give you all the extra money that I have,” and like you know, you got to end up developing those relationships with people. Or like you know, you go to places you like, “Oh yeah, like you know that's the first new always comes in on a Thursday afternoon in orders like ah- a large pizza with ham and sausage!” or like whatever and you know you kind of like… It's not the same as like a friendship, but you, kind of like you said, you get to know the the delivery people that do a thing. One time I got myself in a little bit of trouble at a concert I was in because there was a mosh pit and someone punched me and I punched them and the bouncer only saw me punch them and it was the whole thing, but I didn't have a cell phone and I didn't know- I was there with my boss at the time because he also liked metal music, and I didn't know his phone number off the top of my head, and so I called my store because I did know their phone number and I was like, “Hey,” you know, “Susan, it's, uh, Steve from the Oakland Store? Could you put me through to Frank please?” and she's like, “Sure, is everything okay?” and I'm like, “Yeah totally…” and I'm like actually, like you know, in the like little police station or whatever. Yeah, and is like because like—
James Munns
Bleeding out the nose or something.
Steve Klabnik
Yeah, like I got carted off by security into like- the not real cop thing you know, but like whatever like, it was like sort of I got a summary chart… Whatever but like…
James Munns
Ah, you got taken to the holding room.
Steve Klabnik
Yeah, basically, and they're like, “Do you have anybody to like call to like, you know, do about this?” and I was just like, “Ah no,” but like I ended up calling the pizza shop and talking to these people because that was the… I didn't want to call my mom. I did call her afterwards, but I was like— she can't actually help me in this situation, I need to like talk to my coworkers who are here. So: what do you? You call a pizza shop! And so that was like very very funny. Those kinds of jobs… there's like a level of camaraderie that I had at my pizza job that I've never had in any tech job, and it's because like you're in something that's like kind of shitty together. And that like builds this sort of like shared attitude towards something that like I don't think is always good, but it's a thing that I look back on fondly. Is there a thing that I necessarily loved while I was in it, and am I romanticizing it a little bit? Absolutely, but it is like- that's the way I think about that old pizza job, is like it was kind of terrible in many ways, but it was terrible with a lot of other people and so we were able to bond over that, and that was like a good experience that you know I still ultimately am glad that I did, you know, even though when 20 people would walk in at 5 minutes to close and order 30 large pizzas, and I had just cleaned everything and counted all the money in the till, and had to redo everything and went at home at 6 AM instead of 4 AM, you know, I still like you know… those were still good times at the end of the day.
James Munns
Very cool. Well, I think we'll probably wrap up here. Is there anything that you want to plug, anything you want people to look at, people to go see any other thing awesome that we didn't hit that you want to talk about before we ah wrap up?
Steve Klabnik
I don't think so in particular, other than ah, the little OS that I work on is technically open source at Oxide. So if you go to github.com/oxidecomputer/hubris, our hilariously named rewrite and operating system in Rust is a hubris thing, and so ah, yeah, that's that's all I'd have to say!
James Munns
Okay, thanks so much, and I'll talk to you later.
Steve Klabnik
See ya! Bye.
Credits
This podcast is brought to you by OneVariable UG — a consultancy focused on advising and development services in the areas of systems engineering, embedded systems, and software development in the Rust programming language, based in Berlin, Germany. Check out our website at onevariable.com or send an email to contact@onevariable.com.
Audio recording done by James Munns, edited and produced by Amanda Majorowicz. Special thanks to Louie Zong for the music.
Thanks for listening!