Chats with James 017 - On the Road: RustNL & Oxidize

Part 1: RustNL

James: So right now we're sitting outside of RustNL 2024. It's the second day. I gave my talk in my workshop yesterday. Okay. And I put out a call of folks that might want to talk at RustNL and mentioned the organizers and immediately two of the organizers showed up and were interested to talk.

So would you two mind introducing yourselves?

Laura: Yeah. So I'm Laura. I'm 24. I live here in Delft and, yeah, I'm super excited to be here.

Jonathan: Yeah. And I'm Jonathan, another co organizer, I guess. We did this with the four of us. I am 23 myself, and I work here at the university because RustNL is just right next to the university.

I guess that's partially because of us that we live here, that we sort of, convinced a lot of people to do it here, I guess.

James: So I know the Tweede golf folks are involved with the organizing. How did you two get involved? I mean, you're here, but how did that go from being here to being part of the organizing crew?

Jonathan: How did that go again?

Laura: So we attended a Rust meetup, back in 2022 in Amsterdam,

Jonathan: Which they organized...

Laura: Which they organized. And then we thought because you were working part time as a teacher for the university and then we really thought, hey, this could be a really cool thing to organize in Delft because the student population would find this really interesting.

Jonathan: Yes, because the one in Amsterdam was at the University of Amsterdam, so that's how we sort of got the idea, I guess.

Laura: Then we organized a meetup, and then we organized another meetup later that year, and both of them were really, really successful. And then we volunteered at the RustNL 2023.

Jonathan: Yes, we basically offered, do you need help? Can we help out?

James: I would say, as someone who's run an event before, they always need help. They may not be prepared to accept help.

Jonathan: But to be quite honest, it was an easy way to get free tickets. That was the real reason.

Laura: "Hey, we'd really like not to pay 200 euros for this."

James: Yeah.

Laura: And then we met Erik again at EuroRust last year, and that was really when the concept of, RustNL in Delft started, yeah, popping into our collective conversations.

Jonathan: The biggest conference on Rust ever held in Europe was the plan, that I think... we were just under, but.

James: Okay, because I know it was somewhere 400, 450?

Laura: 450 people, I think, was.

James: So who has the record still then?

Jonathan: EuroRust was slightly bigger, but I think RustNation is mostly the bigger one.

Laura: But of course, that still counts as in Europe, I believe.

James: It's in Europe, it's not in the EU. So maybe you could be the largest conference in the EU.

Jonathan: It doesn't matter so much. Though Mara was talking yesterday about the whole "Rust Week" next year? We'll have to see how much is just the thought of Mara, and actually something we can arrange.

Laura: But no, yeah, the idea was to have an event that would be open to everyone and really affordable and accessible to students, and have it be here in Delft.

James: I'm not from the Netherlands, I'm from the U. S., but I've been living in Germany for ten years. There's a lot of cities that I didn't realize were very university-driven cities, like Delft, for example, I'd heard of before, but not ...

Laura: Not as a university city.

James: So I met up with the Tweede golf folks when they were coming to look at this venue. I didn't come to the venue, but I met them up here in Delft, and then I realized it's, you know, a university in town, and that was a pretty huge part of today's version of the town.

Jonathan: I guess so, I think a good portion, maybe, is it half? Well, at least a third of the population of Delft is students, I think.

Laura: Yeah, and I think even if you're not a student per se, you're still like, your life is shaped by the university being here, because for example, I work as a software developer for a small company that started off as a university research group.

James: Okay, yeah.

Laura: So yeah, I'm still affiliated with the university even though...

James: It's hard to get away when you're inside of the city.

Laura: Yeah, exactly, exactly.

Jonathan: Mm hmm.

James: So I'm betting that quite a few of the people who listen have been to conferences before, whether it's a Rust conference or other tech conferences, but what goes in behind the scenes of setting up a conference?

Because I think everyone sees the front side of that, which is: it gets announced with some level of detail. They put up ticket sales, some level of detail goes out about who's going to be the speaker. Maybe they've applied to be a speaker before and then they had the conference and they have a great time for a couple of days and then they leave.

That's the timeline from someone who's outside of it, but what does it look like behind the scenes?

Laura: I think at the end of the day, it starts with a couple of people going like, "We want to make this happen."

Jonathan: And then nobody really knows how to make it happen yet. So we sort of start improvising and working some things out, I guess.

It is quite stressful at times.

Laura: Yeah, I think, the first thing that needs to get worked out is when do we want the thing to happen. And then the second thing that needs to be worked out is where do we want the thing to happen. And then suddenly you have a venue and you have a date and you have an event.

Jonathan: Well, all those things need to align. That's the big issue, right? Finding the right place, but the venue is not available and the other way around.

James: There were quite a few events this year. So I looked it up earlier in the year and especially for Embedded in Rust which are kind of the intersections of my interest. I made a list and it was something like every two weeks there was Embedded World and then RustNL and then Oxidize in Berlin.

How much work goes ahead of time of avoiding similar topics- so like you said picking the date- And then how much of that is surprise because if you're all kind of quietly, secretly organizing your own plans, is there any coordination behind the scenes?

Laura: So, we did coordinate with I think the RustNation organizers, and also with RustLabs, I believe, in Florence.

Jonathan: Yes. Yeah, we met those at EuroRust, and then we all said, okay, let's pick a different month, basically.

James: So none of these conferences are affiliated, but the same way that like, Rust developers, we all talk to each other when you know you're working on a similar project, I would imagine conference organizers are the same of like, "Hey, we're, we're going to do another one this year. What dates are you thinking? What dates are we thinking?"

Laura: No, it's a, it's sort of like that where we all still keep in contact.

Jonathan: Well, it's a careful balance, right? Because we don't want to overwhelm the people because like at some point people don't come to conference because they went to it last year and the one next week as well.

James: And if you have to travel internationally, it's- if there's two weeks in between conferences. It's very hard to take one week, come back and then take another week.

Laura: It's also I feel difficult to convince your employee to get your tickets to RustNation and RustLabs and RustNL and Embedded World and whatever.

Jonathan: So it is sometimes complicated to plan that, although we do really want to keep doing RustNL because it's a lot of fun!

Laura: Yeah, we think it really adds a lot of value to the local community.

James: I will say the Netherlands is also easier to get to from a lot of places. With such a large airport here.

Laura: Yeah,

James: A lot of the times you fly through the Netherlands anyway to get to where you're going in Europe if you're coming from North America or South America and things like that. So, I imagine being that well connected makes it easier for people who are coming from out of town. Do you have any idea of what the breakdown of like: in the Netherlands, in Europe-

Laura: Where are people coming from?

James: Yeah...

Jonathan: I only have estimates. I didn't actually look this up, but if I look around here, then about half of the people speak Dutch I think.

James: Really?

Jonathan: Not all from around here necessarily, because even though the Netherlands is a small country, I mean-

James: There's lots of cities.

Jonathan: Yes,

James: Yeah, I've talked to a lot of people who are students here in the Netherlands who have come who say, "I go to the university five minutes from here, and so I'm excited, I've just started with Rust, or I'm interested in Rust," and it's cool that they're able to see what other people are doing. It's sort of a very quick way to jumpstart them into all the stuff that's going on in Rust.

Jonathan: I try to get some of my students to come as well because I teach here at the university in Delft, right, and I teach them Rust. So I try to get a few of my students to attend because it's fun if they also get a bit of a bigger picture of what's going on. At least the ones that are interested in that.

James: I know Henk during his teaching talk yesterday mentioned that there was one school in the Netherlands that he knew that was officially doing Rust. Are you in the school doing Rust? Excellent. So how did that process go of starting to teach Rust?

Jonathan: We have been doing this for two years now. We're doing the third year after the summer.

James: And what university is this?

Jonathan: Delft. The University of Delft. I guess it started- when I did my bachelor's here, which is like six years ago now. I wanted to do my projects in Rust, and I kept complaining to my teacher about the fact that I really wanted to do it, and he said no.

But I guess three years later they were redesigning part of the Master of Embedded Systems, and the same teacher who said no to me, came to me, "Well actually we're trying to do Rust now. Can you help us like, like work some things out, because we heard you liked it?"

James: That's a fun turnaround.

Jonathan: It is.

Laura: And then you got all of your friends on board.

Jonathan: Yes. There may have- yes.

James: Mild social engineering and a crowd.

Jonathan: Yeah. But, it has worked, I guess, because it was a plan then. And I just was helping out a little. And in the end I was teaching two lectures, and then the next year I was teaching four lectures because the other lecture.

James: They became more popular?

Jonathan: No, yeah, well they had some other duties to do, the other lectures. So I gave a few more lectures and I guess I sort of rolled into that job, right? Now we're doing it as a core part of our Embedded Systems Master. You learn some C as well, because there's still C in the world.

But we like to at least teach the students also about new technologies. We're in a university where part of it is to try and also be ahead of the industry a little bit.

Laura: Yeah, kind of push that boundary.

Jonathan: Yeah. Yes.

James: Learning your second language is challenging as well. So particularly for university students, getting them used to learning C and Rust, I think learning your first programming language, I think is definitely the hardest because you're learning a lot of these concepts that are not the same but somewhat similar in different languages.

But then learning that second language I feel like is almost like learning in the real world a third language because it stops being like primary and then the other one of, "Oh wait, now I actually have to think how I start grouping these logically."

Laura: I mean, I think for me at least... so I learned C first because I finished university a little bit before the Embedded Systems Masters here at the TU Delft started really being a thing. So I learned C first and then I learned Rust basically on my own. But for me the two languages kind of complete each other very well in that learning C makes you appreciate the things that Rust does better, and then using Rust and then going back to C gives you like a certain degree of freedom that you don't really appreciate. Yeah, you're programming in Rust. So I find that learning both at the same time can be very educational. Then again, I'm not, uh,

Jonathan: Yeah, but you're not alone in this at all. My students also claim that, some of them you see before and they say, well, I got a better C programmer now.

James: Yeah.

Jonathan: Cause I know what crimes I was committing.

James: It yells at you when you do things and you go, "Oh, I- oh, I should never have done that." Yeah, for sure. I understand that. When I was working in safety critical at the time when I first heard of Rust. So I was doing Safety Critical C for Embedded Systems, and I started reading, like, the promises that they have and what kind of things they allowed and what they didn't allow, and I went, "Oh, if this continues growing, it's going to be the thing!" And that's really what got me into Rust was I was in Safety Critical, and we had wanted better tools, where we had come up with a process for always making sure we checked everything, but it was very tedious and very easy to misstep, and you usually had someone catching you, but I was like, "If you can just not do that in the first place..."

Laura: I feel like that still takes a lot of manpower to, yeah.

James: You do it because you have to for safety critical.

Jonathan: But you, Laura, you also use Rust at work, right? So how did that start out? Because they must have had some reason to switch.

Laura: I'm not entirely sure. We're a fairly new company. So I think everything was built in Rust from the very beginning. At my work, we build key delivery systems for quantum computers. So, yeah, thanks. A lot of the firmware we build in Rust, about all of the firmware we build in Rust and all of the, yeah, kind of application layer software, we also build in Rust. So yeah.

Jonathan: And does it help that those are the same? Like, do you interact?

Laura: Yeah. Yeah. That does really help because a lot of the interfaces are just smooth, the same.

James: I don't know if you've seen in the Rust Embedded group, there's the folks from... I forget the company, but they make a project called Stabilizer as well as another one where they're making equipment for precision laser control for quantum computers or something like that? They're building scientific equipment on embedded systems for that.

Laura: Yeah, it's sort of the same kind of thing. I am familiar with Stabilizer, but I'm also blanking on the company name.

James: Yeah, yeah. It's a collaboration. I think there's a German university and company that does it. It's not just one company. There's quite a few collaborating. So I was wondering if you were-

Laura: Um, We're not actually affiliated with them.

James: Okay.

Laura: It does sound like sort of the same thing.

James: It's just impressive to hear people doing Rust, embedded and quantum computing altogether and how common that's becoming? I have no idea the size of the quantum computing industry, but-

Jonathan: You talk a lot to the physicists as well...

Laura: Yeah, yeah... well, in our company, it's at the very least, there's no single person that does all of these things. It's very much a collaboration. So the quantum physicists will write a Python script and we'll go like, "This is what we need it to do." And then we will take that and then translate it into actually safe, secure, usable Rust code.

Jonathan: And hopefully a little faster.

Laura: And hopefully a little faster, yeah.

James: Yeah, I was going to say, talking to the Stabilizer folks, I spoke to specifically Ryan, and he was saying that's a big difference of developing- because a lot of embedded systems, you're building them to do one purpose. They might have some tunables or settings and things like that, but usually, they're doing a purpose, but when you're building tools for scientists, they're still figuring out what needs to be done. Before you productionize it, so he was saying how configurable you have to make everything and how accessible you have to make that, because they're writing a Python script and they want to be able to tweak everything.

Laura: I mean, one of the first things that I worked on at my current company was actually a... just we have certain debug interfaces. And the debug interface exposes a lot of functionality of the FPGA to the scientists because the scientists need to be able to read a certain register or tweak a certain value or stuff like that.

And it was really tedious to have to add those registers to the debug interface manually. So one of the first things I did was actually automating that process and making it super easy and fast to update those debug interfaces. And Rust just makes that sort of thing really easy with build scripts and that sort of infrastructure.

James: So it's been interesting to hear both of you talk about something that isn't conference organizing. Because this is something that I find with a lot of conference organizers: is that they're not necessarily doing this as their full time job.

Laura: No.

James: It's just a passion that they have and they go, "Well, I'll sacrifice all of my free time for a couple months out of the year to make that happen."

Laura: Yeah.

Jonathan: That's what it's been. Yes. Exactly. Yeah.

James: Any notes you have for people who are really excited about that are in a university where something's going on of: how to start the momentum going towards having that? Is it starting a smaller event like a meetup and then rolling towards that or is it really defined?

Laura: It's definitely a good idea but I think as long as you're excited about the thing that you're organizing, you can figure out a lot of stuff on the way.

Jonathan: It's lovely. I saw a post on Mastodon, I think, yesterday or the day before, of some people in the United States. They said, we are having a Rust meetup, and it was a small room with five people in it, and they were just sharing ideas.

Laura: Yeah.

Jonathan: It was just a picture of them, them five, working something out. But if you just announce that you're doing that, and five people show up, and you have a fun talk, maybe next time ten people show up. Before you know it, I guess that's what happened two years ago to us. And it's only two years ago. Which is a bit, but in two years you can go to organizing a 400 people event.

James: Yeah, and I think that's one of those really important things for momentum, is if you sit there and you ask, is anyone else interested in that? You'll get a lot of, "Well... maybe... yeah, if you'd like to." But it's impressive how different the response is when you say, "I am doing this-"

Laura: Yeah.

James: -people go, "Oh. I'm in."

Jonathan: "I'll take my friends."

Laura: Yeah, exactly.

Jonathan: Yeah,

Laura: It's also kind of insane to me that so many people have flown here for- yeah I don't know- California, and yeah and New Zealand. And just everywhere across the globe because we kind of went like, "Yeah! We're doing this in Delft on these and these dates." Yeah.

James: It's one of those things when you just say it's real, it starts becoming real. And it's always a challenge because you never know what the response is going to be. So definitely from when we ran Oxidize in Berlin for the first time, I was involved in the organizing of that. We kind of had the backup plan of, we chose a venue where it was cheap enough that even if it wasn't that popular and only 20 people showed up, we'd still have a good time.

It ended up being about as big as we wanted, which was a little over 100 people or so if I remember correctly. But, we kind of had: if this is way more popular than we want, we kind of know what venue we're gonna try and switch to and we make sure we had a good refund policy at the first place. I think actually we had connection to someone who was willing to sponsor, or like front it, if something happened. Because you never know what the response is going to be on the first one.

Laura: No, yeah.

James: But then once you have the second one, people know what it is. I think it's way easier to go bigger on the second one. Yeah... that's really cool. This is the first portable recording we've had and, we're sitting outside of the conference. So there's probably been some audio texture of some cars driving through in the background and it's starting to sprinkle, so I don't want us to get caught in the rain. So- I wanted to thank both of you for taking some time to chat with me today.

Jonathan: No worries, it was fun.

Laura: No, it was really fun. Thank you for having us.

James: Yeah, and enjoy the rest of the conference. I hope the wrap up goes as smooth as it seems to have gone so far.

Laura: Yeah, I hope you have fun at the Unconference.

James: Thank you.

Part 2: Oxidize

James: So we're at Oxidize right now. We're kind of right in the middle of Berlin. So in between like Hallesches Tor and Mitte. I live sort of down more in the Neukölln area. minute drive or so, but... Always depends on whether there's a connection or not for it.

Jonathan: I love the public transit in Berlin. I left my hotel. I took the SBahn, I jumped on the U6 and yeah, came out at Checkpoint Charlie and it's, yeah, it's always really smooth getting around in Berlin.

James: Yeah! Do you want to give yourself a quick introduction?

Jonathan: Yeah, so my name's Jonathan Pallant, Senior Engineer at Ferrous Systems, a full time Rust trainer. You may know me from such daft projects as writing a FAT32 file system in Rust. I wrote a keyboard decoding library in Rust. and I'm in charge of a project called the Neotron.

James: The Neotron is, I think, it's gotta be the most conference talked about embedded project in Rust. I feel like I've seen you give a talk about the Monotron or the Neotron at three, four conferences now?

Jonathan: So yeah, well let me see if I can remember a potted history. The first was RustFest Paris. Where it was basically a bunch of resistors hanging off a dev board. Then there was RustBelt Rust 2018, in Ann Arbor, and that was on a Veroboard so I, you know, upgraded a bit. And then there was RustConf 2019, the last one before the thing happened. And there, I think I had PCBs. And then it was- I think it was Oxidize 2019. I was talking to some people and were like, "This project has run its course." It was a demo built on top of a demo built on top of a demo. And none of it was really any good. And it kind of... It did a thing, but it wasn't something I was really excited about going back to.

And that was the point I thought, I'm going to throw this all in the bin and start again. Which was then the name of my 2023 EuroRust talk where I kind of disappeared for four years and switched processors and re-architected everything and the thing happened and I was mayor and I've changed jobs and then yeah, kind of came back and went, ta da, look, I made a thing. I spent ages on it. Um, I don't know why, it's kind of weird. So yeah, kicked off again with EuroRust. It was on display at RustNL. I didn't have a talk. Oh, I'd previously given talks about it at ACCU and NDC TechTown. That was before Eurorust. Yeah, it was probably 2022, I think. I forget. Time's going by. I'm getting old.

And then, yeah, here we are, middle of 2024. Not talking about it. It's on display at Oxidize. Yeah, and people come along and they look at it and they're like, "You brought a computer I don't recognize." And then you explain, "Ah, it's not just a computer, it's bare metal Rust." And you ask them questions like, "What is an embedded system?" Alright, I'm going to ask you the question, we're going to flip roles for a minute. James, what is an embedded system?

James: It's any computer you don't necessarily sit in front of. I haven't come up with anything more concise with that. My RustNL talk, I always end up having to put in two or three slides of what do I mean by embedded systems, and I think my favorite image is I found a stock image of someone holding a key fob in front of a car, and I went, "An embedded system could either be the key fob or the car, or both."

Because it ends up being everything that ends up having just purpose specific behavior, and I think that's really the difference-

Jonathan: You've changed the rules then.

James: You actually, you gave a really good definition and I'm struggling to remember it, but the line between embedded systems and operating systems and computers is very very fuzzy on the best day. But your definition was something like: when someone can sit down and run software that was not envisioned when the hardware and software shipped together, then it becomes a computer, because you can have someone else write programs that run on your system. And I think that's the Rubicon. It's hard to explain that succinctly, but I think like in my heart, that's the differentiation of -

Jonathan: Yeah, for me it's about the control. Cause the problem with proximity is you say, "Well, what about the Windows server VM I just spun up on AWS. That's in Virginia East One." It's still a computer, but I'm definitely not in front of it.

James: Yeah.

Jonathan: But I am able to run programs on it that I chose and that Amazon and Microsoft did not choose. So it's a computer.

James: Or you might even say that- well, once I've put one application on it, and it's just running, and then it's a purpose built system, so then does it become a embedded VM system, or does it become...

Jonathan: So it, I think it does if you take the console away. So a ticket machine running Android: it's an embedded system provided, you know, they didn't leave the keyboard. If they left the USB ports on it, I'd go, "This is now a computer, because look, watch me Alt-Tab away from the ticket program and you know, start running things."

James: I mean, Windows embedded was a thing for a very long time.

Jonathan: Yeah, it was, and that was taking away the opportunity for people to load arbitrary software on it of my choosing, not of the designers choosing. Where it gets really interesting is when you ask the question, is the iPhone an embedded system? Can I walk up to an iPhone and run software on it of my choosing? Not of Apple's choosing.

James: Yeah, I think Apple would much prefer it was an embedded system, but I think practically with an app store, there is still the ability, like, it's a vetted, or it's a walled garden or whatever you would say.

Jonathan: So then you get into things like: can you write computer programs in JavaScript?

James: I would say yes.

Jonathan: In that case, then, this is a computer, because I can just open my browser and execute an arbitrary program. It just has to be served somewhere on the internet. But that's okay. I could probably give- Visual Studio Code has like a web version, right? I could probably load that in Safari and edit my program.

James: You absolutely can. I was saying even the editor itself running on your machine when you have plugins, if it has the ability to load and run arbitrary plugins that interface with the user. That's one of those things where a browser becomes an operating system or anything with a plugin system becomes an operating system. I don't know.

Jonathan: You definitely get this slope, right? This sort of gradation from: this is definitely an embedded system, you know, it's a PIC microcontroller and it's in a microwave. I'm not walking up to that and reprogramming it to play Snake on the screen. It's doing a job, it does it very well.

To: this is a laptop I bought and let's hope they continue to be general purpose computers and, you know, I'm free to run on it what I want. iPads, iPhones, kind of live in this interesting area in the middle.

James: This gets especially hard because the way that I actually understand things is by saying this is like this, which is both useful and totally confusing for category theory. Like, I was describing TCP is a network protocol, but it really has two things: it's an infinite line of bytes, and a way of synchronizing the read and write head in either direction on those bytes. Where TCP does a lot more than that, but if you end up looking at it, you go- well, TCP is a ring buffer, or I wrote a blog post awhile ago that was, interrupts is threads, because at least in embedded Rust, we treat them a lot like threads for various reasons, but they're not generally the same thing, but it's useful for me to comprehend things-

Jonathan: They have sufficient things in common that it is useful to put them in the same bucket at least temporarily.

I teach Rust. Pretty much full time these days. I mean, I have a full time job. I just don't teach Rust every day. Sometimes I'm writing training material or coming to conferences and doing stuff like that. But yeah, it's kind of interesting when you have to start to reformulate these ideas in a way you can express to other people who have paid to come and hear them.

And you can't have wooly understandings of a concept if you're going to teach it. You've got to try and find ways of being sufficiently precise that you can then build on those definitions and teach them more and more of the subject. I found if you don't get precise at the beginning, then, you know, you can have, you can have problems, but you can't be infinitely precise.

You can't just do tangent after tangent on day one, because you'll never get anywhere useful. So you've got to kind of build a layer that's sufficiently good, sufficiently accurate. Maybe tomorrow I'll tell you that we were lying. This is kind of how school felt. You learn physics and they're like, "The atom is kind of like a football with things spinning around it."

And then you do like A level physics. And they're like, "Yeah, no, it's not really like that. It's like a tiny thing in the middle of a stadium. And the electrons are kind of buzzing around at the edge." And then, you know, you get to university like, "Yeah, we have no idea what this stuff actually looks like. It's more like a fuzzy cloud." And you kind of have to admit that the previous level of abstraction was a convenient lie to help you understand more.

James: Was it "All models are wrong, but some models are useful"?

And I think that's exactly the case. So then, more concretely... the Neotron is a system where you've built a motherboard shaped PCB, with expansion card slots, with a socket for a CPU module, and many of the niceties that you'd expect. PS2 ports, there's USB for power, but it looks more like a computer than many other hobbyist systems- my own included- end up looking. So what was it about making something that looked and felt like a computer that people were used to that was more motivating than running this in QEMU or on a virtual machine or on existing dev boards?

Jonathan: There's two kind of aspects to that. The first is real world things are just more tangible. Ever since I first put wheels on my Commodore 64 in 1995 and turned it into a line following robot. There's just something very tangible about it. I wrote this and then a physical real world thing happened.

And you could argue that displaying characters on a monitor is a physical real world thing. It just doesn't feel the same as making an LED come on and blink. So there's something very tangible about: I can see it, I can touch it, and I can see it did a thing.

The second thing is specifically why it's ATX shaped, microATX shaped, is I was tired of trying to find cases. The first version, the Monotron, didn't have a box. It was just a PCB with connectors around the edge, kind of laid out randomly. Then there was a machine called the Neotron 32. which was the same Texas Instruments processor as the Monotron. Basically the same firmware but split apart, reorganized, rewritten from scratch really.

But it was laid out to fit in a Hammond instrument case. You've seen those lab equipment that's clearly kind of homebrew lab equipment and it's a plastic clamshell with a top half and a bottom half that fit neatly together and at the front and the back of the panels. And those are the panels you're supposed to drill the holes in for the connectors, the serial ports, the power connector, whatever.

And then it's very easy to make a product and, you know, many firms where I'm from in the Cambridge area got started making lab equipment for universities and labs and you just like get a bunch of stuff, put it in the case, buy the case for, you know, 10, 15, 20 dollars, Hammond do them in a bunch of different sizes.

And the faceplates, you can either mill some thin steel, you could laser cut some Perspex. These days, you just go to make a PCB. You just go to JLC and say, Well, I'd like this piece of fiberglass in these colors, these bits in gold, put some holes over there, these bits should be white, and, you know, they work perfectly well.

James: It's amazing that it's become circuit board manufacturers who have made it possible to go: well, I would like custom cut fiberglass with CNC holes and screen printing on it.

Jonathan: Yeah, and a selection of gold accents. Yeah.

James: Yeah. And it becomes cheaper than any other-

Jonathan: Than laser cutting Perspex. Yeah. It's incredible.

The problem with the Hammond cases is, you're kind of limited in the sizes that are available and some of them start to get to the same price as the cheapest PC case you can get. I just bought a new PC case. With a smoked glass window, and that's where my Neotron lives, actually in the ATX case. And the power supply hole is empty, and the drive sleds are all empty.

But it sits there, and it mounts because the holes are all in the correct place, and it's just a nice place for it to live. And that case cost me, shipped, I think around 30 pounds, just a little bit less, including shipping to my house. And at that point you're like, "I might as well let people just choose a case of their own whatever they want, rather than force them into this kind of lab equipment form factor." And it's broadly the same price, so just give people the flexibility. Plus, you know, it looks kind of funny.

James: This is the interesting thing about embedded, or at least embedded hobbyists: particular when you're doing quantities one through ten, you end up having to pick one of two routes. Am I going to become an expert at the mechanical, the electrical, and the software all at once? Do I use off the shelf dev boards? Do I use off the shelf cases? Do I focus just on the software? Or do I become an expert at all three? Am I designing my own hardware? Am I going to 3D print my own case? Am I going to do a kind of- not kitbash, but you know- I've laser cut all the pieces and they Lego themselves together or things like that.

Not everyone chooses all three. Some people end up punting on some of them, but you end up having to answer that question for all three, where when I do rapid prototyping, usually the answer is you beg, borrow and steal as much as you can. And if you go: well, I can get something that's way overkill for my use case, like a well shielded and grounded case for a PC and room and mounting holes for everything like that is way overkill for the three sockets that I have but you go: yeah, but they make a hundred million of those a year. Which means that volume pricing I could get the one that's exactly what I need but it costs twice as much as just making a sort of egregiously over spec'd or oversized or things like that and you learn to sort of sail behind volume prices where it becomes easier to go with the wind than against the wind, unless you really have to do something weird.

Jonathan: Yeah, I know you know, as we said with the with the panels made from PCBs, that's not a sensible choice by any metric, except for the fact that there are factories in China that just make seemingly bazillions of these a day, and they've got them down to, you know, extraordinary price points.

Yeah, choosing your- choosing your discipline is an interesting one, right? Because it's, it's always fun to do things that are not your day job. You're like, "I want to learn a new skill. I'd love to know how KiCad works and how to lay out a PCB!" And in the past, that was much harder. Like if you just woke up one day in the 1800s and went, "I'd really like to become a blacksmith!" Well, you kind of got to go and find a blacksmith because you're not really going down to the library and getting blacksmith books. Maybe in the 1900s. Yeah. You go to the library, maybe get a book.

But these days there's great tools available at basically zero cost like KiCad and FreeCAD and we had some good talks at Oxidize about this. And there's great learning materials written by people for no reason other than they kind of wanted to make the world better. They learned a thing they were excited about a thing and they wanted to teach other people the thing, and instead of teaching them face to face they wrote it down and they published it online. And yeah it's revolutionized the tools and the technologies people can have access to.

And it's how I learned electronics. I watched videos on YouTube. I read online guides. I used KiCad because it's free. And I was able to make, you know, a pretty good quality product. I know people doing professional great stuff. And that's then kind of interesting for the people who are selling the pro tools, because, you know, the Altiums of the world, it's still a piece of software.

James: It does do things you can't do with KiCad.

Jonathan: Oh, it definitely does more than KiCad. Yeah. But it's still just software, and it's very expensive. And KiCad's not gonna take over anytime soon. Linux flattened the Unix market.

James: Even closer to home, the one that I can think of in between Linux and KiCad is Blender. Blender used to be sort of the weird oddball open source option and everyone was doing design on SGX workstations with bespoke tools. Over the years, Blender just kept getting incrementally better. To the point where I'm sure someone will disagree with me and we go, "We don't use Blender at work." But there are still a lot of professionals using Blender full time. And like you said, if you're not doing crazy RF stuff or impedance mismatches or dealing with an existing giant process at your company.

Jonathan: And if you can deal with the absence of a PLM, a product life cycle manager, because KiCad is kind of not great at managing the fact that my transistors got out of stock. And now I need to go and find a different one that's compatible, that kind of BoM management. It can't do that at all.

James: Or I'd like to check the warehouse stock of my transistor in my tool before I spin it because then I'll know whether I need to wait six weeks for the parts to arrive.

Jonathan: Which is why JLC have their own online electronic CAD tool, because that definitely can just dip into the JLC warehouse and tell you what's available.

James: And people have written plugins for KiCad that you can live load that or at least on a daily resolution. Because I've done that where you sit down you go, "Oh, I'm gonna reuse the same parts as last time!" Then you go and check and you go, "I need 20 of these LEDs!"

And they go, "We have four."

And you go, "Okay, yeah. Do I redesign my boards, or try and find a knockoff part, or do I just wait and hope they restock?"

Jonathan: On the Neotron, the audio codec was obsoleted by TI after the second batch was made. And for TI that means all traces of it disappear from the TI website. You can't search for the part number, they just come up with zero results found. The hot links I'd made to their PDF data sheet no longer work. and DigiKey and the other vendors no longer have them in stock. So now I'm like, "Ugh, do I redesign the audio system? How do I progress?" I don't really want to go to gray market vendors. But actually, another board manufacturer reached out, Elecrow. And they said, "We're interested in, you know, helping people make boards." And, you know, well, they want to get into the business of, you know, working with hobbyists. I guess PCBWay is the common one because they seem to sponsor everyone on YouTube.

Elecrow reached out and said, "Yeah, is there anything we can help with?"

I said, "Well, I've got this product. Here are all the files on GitHub. How much would it be to make these? By the way, you won't be able to buy these parts."

And they went, "No. No, we found everything." I don't know where they got them from, but-

James: There's a warehouse somewhere.

Jonathan: They soldered it to the board and it works great. And they gave me a good price. So I think I'm back in business with that one, at least for a while, at least until they tell me, "Actually, yeah, all the grey warehouses have now, you know, run out of those as well," but.

James: You've also made a jump where you've gone from having a hobby project that isn't just one board or a series of boards that's sitting on your desk. You worked with the Rust Foundation in the past and got some funding to build 20 boards if I remember correctly?

Jonathan: Yeah, 25 I think. Yeah, that was a weird one because the Rust Foundation put out a call for grant proposals And it was just like- lol, they're never going to pay for hardware, but whatever, we'll help them out. I'll fill in a proposal. And I said, "Yeah, give me two and a half thousand US dollars and I will make 25 boards and give them away free of charge. And then more people can do Embedded Rust and kind of hack on this platform." And they came back and went, "That sounds great. We don't think you've asked for enough money. Have 3 thousand dollars." So I got 3 thousand dollars and then I suddenly went, "Crap. In the UK this kind of counts as income and I'm supposed to pay tax on it."

So I then had to really do accounting for a year and track all of my expenses in that tax year and all the money I'd spent on having the PCBs made. I had them made surface mount with all the surface mount parts loaded and then all of the through hole came separately. So I'd ship it to you in a box with the through hole in a bag you fit that yourself because at JLC the through hole costs quite a bit more.

And yeah, I basically got into this little business where I had my big bag of 200 capacitors from Farnell and I put three in this bag and three in this bag and three in this bag for 25 bags and then the keyboard connector, they got one each and I decided I really didn't like this. This was like my first electronics kit business and I decided I didn't like it.

So, yeah, probably not gonna- I fulfilled the grant, I got the 25 done, I actually made 30. Some I kept, some I've sold. The 25 got given away, people are using them, they've sent me, you know, pull requests and changes based on what they've done with them. But now Elecrow say they can do the surface mount and the through hole, and they will do drop shipping from their warehouse, so I will pay them to make the stuff.

And then they will sell them on their website and send me the profit. With the assumption, I guess, that every so often I'll just go, you know what? I'm going to spend all the profit on making twice as many as I did last time. So we'll give that a go. Which is kind of a shame, because through hole soldering is fun, I just don't want to put all the bits in bags.

James: So how's that process gone from being a very successful personal project, where you were able to show it off to a lot of people and get a lot of excitement from that; versus now you have multiple users, which I think is one of those- a lot of people do hobbies to scratch their own itch, or just for the purpose of learning, and I know I have countless PCBs in my office. I really wanted to make one, but I bought five, because I have to buy five, and then I used it for a while, and I learned the things that I wanted to learn, and then I put it down and walked away from it.

And I'm only just starting to get to projects where I've become okay with putting it down for six months and coming back around, but how does having that order of magnitude more people change the good and the bad of a project like that?

Jonathan: Yeah, that's interesting. I kind of ran into that earlier because I published one of very few SD card libraries written in Rust to a FAT32 file system and an implementation of the SD card protocol as executed over SPI. And there's mine, there's an async fork of mine. We get into long discussions about whether mine should be async or not. Currently, it can't be both. So I'm sticking with the way mine's designed. And I think there's a separate one that was fat32-rs, developed completely independently.

But there's not many of them. So whenever anyone wants an SD card library, they go to crates.io and they type the stuff in and they find mine. I was in University Politehnica Bucharest this week, judging some student projects. The students there, I think they were second years, they're learning embedded for the first time, and they do it in Rust.

And so they're making all these projects: smart cat flaps, so the door only opens if the cat's wearing the right Bluetooth tag. I saw home burglar alarm systems with keypads and screens, remote control cars, loads of really amazing stuff. And they put it together by learning a bit of Rust, going to crates.io and jamming all of this stuff together. It was Embassy on the RP2040 and then, you know, drivers for the various screens and on whatever they found. And I go around to talk to some people and say, "Oh, there's an SD card on this board. It's playing WAV file samples or whatever. There was a laser piano."

And I'd look at the source code and be like, "Oh no! I recognize that. You're using my SD card driver."

And they're like, "Wow, you wrote that?"

And I'm like, "Yes, I'm very sorry."

Like, "Your API was really hard to use."

Like, "I know... I didn't really intend for other people to use my driver. I wrote it for my dumb retro computer and I just published it in case it was useful."

So, you know, that's the latest instance. There have been plenty of other people turn up in the issue tracker for that and go, " I was using your library to log sensor data on this remote thing I made. And there seems to be an issue, you know, in flushing rights or occasionally I get disk corruption..."

And I'm like, "Oh no! Don't! Don't actually save data you care about!" This-

James: Accidentally load bearing?

Jonathan: I am a very firm believer in the MIT license, which says "No warranty is given, express or implied. This may catch fire and delete all of your data! Really hope you did your due diligence before you started using it." That's kind of an interesting problem, right? If you go to crates.io How do you really sniff a project and work out whether it smells good or bad?

How do you tell the difference between, "This is something I wrote as I was learning, and I wrote bad code as an educational process," with, "Here is a project that I genuinely believe may be useful to people, and I am open to hearing your opinions on it, and how we can make it better."

Because I'm not sure we always clearly make a distinction between those, and if you get it wrong, if there's an impedance mismatch, people are going to end up upset in your issue tracker. Nobody has an obligation to make their open source project do a thing.

Like, "No warranty given, express are implied," but, Ah, sometimes people feel there's an obligation, and that's an interesting one.

James: Yeah, I think expectation management is something where I've recently seen people, at least in the embedded Rust world, start to go, "Look, I'm publishing this, but it is what it is. If you like it, you have my blessing to fork it. Please have a great time. I will not be maintaining it. It scratches my itch, and I'm done."

And I think that's very healthy. I actually went through the same process, because I have some libraries, like Postcard, where I've taken money for it. And I know that there are people using in production and I'm gearing up for a 2.0 release. And so I try and be good about, you know, informing people about what is breaking and what isn't breaking.

And then there are other things that I either wrote in an afternoon or I was really enamored with for a while, but it just doesn't fit into the problems that I'm facing today. And I really thought about sitting down and adopting something like NASA's technical readiness levels. So NASA has this whole system of basically saying: how serious is this component? Where the idea is essentially the maximum level of seriousness is: we've used it for multiple space flight missions. No, one's going to look at you sideways. If you reuse something that is well tested versus the other one is: someone proposed it on a forum somewhere, and they actually have sort of a graded difference that you're expected to move up either as prototypes or make sure that it's in a non-load-bearing position or it's the secondary system of a flight and it's the way that they sort of gauge how ready it is.

Unlike safety critical a lot of the times, when you're shipping satellites, yes it's very expensive if the mission fails but there's a difference between cost and human life. So there's always a different level of how much you have to get done, but they are still cognizant of they only have so much funding and they want to make sure that they are generally successful.

And I've thought about trying to sit down and write a- I called it the Mnemos, the operating system that I was working on, we had a bunch of components and other libraries that I wrote and thought about just trying to convey that more clearly of: let me give it an A, B, C, D, F or a 1 through 10 scale where I go, "This is absolutely something you can expect to work. If you really want long term support, please pay me. But you shouldn't expect it to just change or break or for me to disappear and things like that."

Jonathan: Wouldn't it be nice if there were crates.io badges and filters for that? I would like to search for a thing on crates.io but only show me things where paid support is available.

James: And I think that's one of those challenges with an open package management system is there's no differentiation. There's only soft signals. The difference of how many downloads, how many releases, how active is that?

Jonathan: Read the README. Yeah. When was it last updated? Is the README three lines or is it like, you know, 16 pages? There's no hard and fast rules. You just learn to give it a sniff and go, does this smell bad? You know, is this something I can work with or not?

James: Yeah, we've snuck outside for this discussion at the conference and a talk just ended so people are filing out. So I think we are going to uh, rejoin the crowd, but I -

Jonathan: Going to grab some coffee.

James: Yeah, it's time to grab some coffee.

So thank you very much for taking some time to chat with me!

Jonathan: Thank YOU! Yeah. Really appreciate it. See you around.