OpenMind, The OS for Intelligent Machines

GM GM. Welcome to what's going to be another great episode of DevNTell. So if you didn't know, DevNTell is a 30-minute podcast held every week, allowing founders, hackers, and anyone in between to come on the show and showcase what they've built. Uh, so today I am ecstatic to welcome Jan Liphardt, who's an associate professor of bioengineering at Stanford University as well as the founder of OpenMind. OpenMind is a... is building AI architecture for intelligent robotics, enabling an open and connected future for powerful human and machine collaboration. So if you stick around for today's episode, you'll see Jan give us a... an overview of OpenMind, why it exists, what problems it solves, and how you can get started with it today. Alright, let's get into it. GM GM. Welcome to the show, Jan. Pleasure to have you.

Narb, it's uh wonderful to be here.

Yeah, yeah. Exactly. And uh today's episode is going to be really interesting. Uh we're going to be diving into the world of AI and robotics and how all of that will also uh connect to human society going forward. But um before we get into all that uh interesting stuff, uh would you perhaps like to give an introduction about yourself?

Sure. Well, based on that introduction, uh it's too bad that we only have like 30 minutes because those are uh many big topics. Um, I'm a... uh at Stanford, I'm in the School of Engineering and also the School of Medicine. I'm in bioengineering. And mostly what I do on campus is I think about how to uh use data um collected by people uh to make decisions about uh problems in healthcare. For example, in COVID, um I uh built a system to collect uh COVID symptom data and uh we had something like 30 million data submissions from 93 countries. So that was an example of using... of... of working with many, many people all around the world for them to uh submit data uh to then hopefully make better decisions on the public health side. But um I've been in love with robots probably uh since I was uh a little kid and I'm sure many of us have had that experience where... remember the books you read when you were a little kid or the cartoons uh we looked at where when we were a little bit older we'd be surrounded by robots and rockets and flying cars and things like that. And uh that love has uh never left me.

Awesome. And yeah, just... just interesting you... you seem to be uh all over uh in terms of uh the science side as well as the tech side. Uh just curious what inspired you um to go into bioengineering? Was there a particular uh um piece that... that really interested you early on that you're like, 'Oh I gotta keep going down this hole' or?

Oh, well, I always thought in terms of the things um we understand. One of the biggest areas of uh ignorance we have has to do with living things. And a great example would be like how... how do our brains work? If you look at many areas of science and technology and areas like physics, like particle physics or cosmology, um those are all areas where um it seems like the... the problems that um you know a lot of the big problems have been at least partially solved. But it struck me that uh biology and living things, living matter, is something where uh we're still relatively uh ignorant about. And of course if you're curious about figuring things out, it makes sense to look for... for areas where um where there's still a lot hopefully to discover and so I was really fascinated by... by living things.

Right on. And uh obviously you also have a uh love for the tech side um and kind of both of these things are now overlapping in the work you do. Um what got you interested in technology uh in the first place?

Oh, well, um I was one of those kids that um got into trouble a lot because uh I would always take things apart. And uh for example, I took apart my uh my parents' car radio. Remember in the old days where you could like take a car radio out of the car so it wouldn't get stolen in New York? Um, and of course I had to take it apart to figure out what was inside. And so um I... I don't have a uh good answer for that. That's sort of been there for a long time. And um uh so I can't... that's just been um so I don't have a simple answer for you. It's something I just uh like enjoy doing.

Part of your DNA. Part of your DNA. And that's... that's so fire. Um and uh you are also uh dabbling in the Web3 space um from what I've seen. Is there a particular uh um point in you discovering uh perhaps the Bitcoin whitepaper or the Ethereum whitepaper that really got you hooked?

Yeah, it was... well, um I... I have a lot of uh concerns and qualifications with um uh with what is meant by Web3, uh because there's incredible diversity of opinion what that even is. And what draws me to blockchains and distributed systems um has to do with uh the math and the game theory and what really got my attention was of course the uh the Bitcoin whitepaper and then Ethereum as a uh decentralized uh Turing complete computer uh is... is really interesting. And for many years I've been keeping my eyes peeled for uh for really compelling use cases of decentralized event ordering and timestamping, aka Bitcoin. Um and there... I think there's now a little bit more clarity on what the technology uh is most suited for and we can probably dig into that uh in a little bit.

Certainly. Uh yeah, and uh yeah it's... it's quite... quite the umbrella of uh different applications, but yeah the most prominent use case that seems to have stuck um has been the crypto... cryptocurrency angle. Um but uh yeah we'll... we'll see. Uh there's starting to be some um AI uh integrations as well. Obviously you are... you'll be quite familiar with that. Um but what I want to get into um is you are actually a co-author of a uh ERC... or EIP uh proposal actually. Uh ERC 7777. Um governance for human-robotic societies. Um which I'll share in the chat here with folks if they ever want to uh check it out. But yeah, uh this is very interesting. I've... I've read uh some of this um and it's... it's pretty... pretty wild stuff, I'll say. Um would you like to... would you care to dive into this a little bit and explain to the audience?

Sure, sure. Yeah, so some people when they hear this, they're going to think, 'Oh, Jan is completely insane.' Um, so... but... so let me put this in context. Um uh when the large language models um started to appear, um I was curious to see the extent to which large language models speak robotics. Uh large language models um speak code uh they're... they're very good at coding, and that gives you, you know, the... like GitHub Copilot and things like that. Um they're very good for uh students to cheat on their engineering problem sets. Um so those are like standard use cases of large language models. But I was very curious to see if they could be used to control robots. And the simple answer is absolutely. So if you ask me about what are compelling use cases for large language models, uh one of the best ones in my mind is to control robots because all of a sudden you can now talk to a robot and it can talk to you and you can tell it, you know, 'Help me do this' and you have the ability to speak to a machine and the machine uh tells you what's going on and listens to you and does stuff. And that's really an awesome capability for robotics. But when I was writing the software that then led to the open-source Android for robotics OM1, uh when I was writing the software and um taking my robotic dogs for a walk where I live, I noticed two things. The first thing is that most people have not interacted with smart machines. So they think of uh a robot, you know, as this like remote-controlled toy. But of course if the robot is now uh interacting with multiple um LLMs simultaneously on the vision side, on the speech side, on the spatial awareness side, um you get really interesting capabilities. So just picture yourself walking down the street, there's a pack of robotic dogs walking with you. It's a small town in California called Los Altos. There's little kids running around. Kids are fascinated um by the quadrupeds and they'll hug them and drool on them and play with them. Um but their parents um uh have this look on their face that's a combination of like curiosity and concern. Because naturally, you know as parents, you're like worried, like yeah, is this even safe? So they ask me, 'Hey Jan, like how come you're not scared?' And I say, 'Well, uh it's because um I wrote the software and it's open-source, so there's like no funny stuff in the software.' And then when the dogs wake up, um they download Asimov's three laws from Ethereum, which is immutable. So you don't have to trust me, I can give you um a link to a smart contract and you can go there and you can read the guardrails that are immutably written into a public ledger that the machines are using when they wake up in the morning um as their guardrails. So um that was just uh like a uh practical observation based on walking around a small town in the US with a pack of um speaking quadrupeds. Hehe. And of course it sounds completely crazy when I just sort of narrate the story, but um the technology is moving so extremely quickly that what used to be science fiction um three, four, five, six years ago uh is no longer science fiction. And of course you see that in San Francisco with a lot of people using Waymos, which are wheeled robots, and so uh increasingly people are trusting their lives uh to wheeled robots to get from one place to another. And so in terms of ERC 7777, uh we were writing guardrails onto Ethereum anyway so then we decided to formalize that. But the... the other observation was that um people um who build payment systems and payment infrastructure uh such as Coinbase um have noted interesting um uses for crypto. And um Brian from Coinbase told me, 'Hey Jan, we're seeing agents um spontaneously decide to use crypto to coordinate.' And of course in retrospect, uh that's... like makes total sense because if you're a thinking machine and you walk into Bank of America and say, 'Hey I'd like a bank account please,' uh the bank teller will just like call security. Hehe. Right? I mean um just imagine uh you're a machine and you want to interact with um human infrastructure uh laws, banks. Um we had a real struggle of getting our humanoid to Denver because um Southwest Airlines said, 'Oh you know um unfortunately Iris can't get on the airplane because her lithium batteries are too big.' So um Iris the humanoid had to uh be driven to Denver. So there... there's all these pain points when it comes to um thinking machines interacting with infrastructure built largely for humans. And what the ERC 7777 is designed to do is really alleviate some of those pain points.

Fascinating. And it seems like um it's a little bit more um towards giving uh robotics perhaps some sort of... some sort of rights uh at the same kind of level as humans, correct?

Absolutely. And of course um that's exactly uh what's happening. Um and this is like... this is like a philosophical point or something um but it really goes to um as machines around us get really smart, um what are they? Are they toys? Are they tools? Are they disposable inert objects? Um or shall we think of them more as team members and partners and coaches and teachers and helpers and friends? And if you think more along those lines, then I think it's only natural to um to also treat them a little bit differently in terms of helping them and making their lives easier and more convenient and things like that. And you could of course say, 'Oh Jan this is just like a bunch of inert machinery and software. You're being ridiculous.' Um 'Thinking machines will never have true agency. Um they're really these inert objects.' But I've had all these funny experiences where I see humans interact with robots and humans get uh emotionally really attached to the machines. A great example would be a Australian healthcare company that's deploying humanoid form factors into memory care units. And in most memory care units, most people don't realize this, but most people in a memory care unit um have not been visited by a relative in years. And if you deploy a humanoid into a memory care unit and the humanoid looks at you and laughs with you and interacts with you and remembers your name, uh what the nurses have to do, the human nurses have to do in the evening is they have to wipe off lipstick from the head of the humanoid because patients are kissing the humanoid. And so I wouldn't at all be surprised if um uh more people get quite strongly emotionally uh engaged uh with machines. And by virtue of the conversations you can have with a tool like Deep Research, um I wouldn't be surprised at all if that line between uh you know, is this like an inert piece an inert set of wires and motors um and uh is this more like a trusted friend and colleague? That line is definitely blurring very quickly. And um yeah, uh these machines should have their own agency and then also their own governance system and their own payment system.

Very interesting. Yeah, I feel like this can be its own like two or three-hour podcast. Hehe. Yeah, um but yeah, uh that... that's really, really interesting stuff and I agree with you. I think like we're going to see a lot of this start to pop up even more. Um like the Optimus robots from Tesla, like as soon as people start bringing them into their homes, right? It's like they're technically kind of sort of part of your family at that point, right? So um the... yeah, it'll be interesting developments indeed. Um but yeah, uh wanting to get into the topic of OpenMind, um which I think will kind of bring all... all of this together. Um why don't you tell us what OpenMind is, I guess?

Sure. Well, if you look at most phones on Earth today, um most of them run Android. Of course there's companies like Apple which um have a lock on the um uh the cloud, the software, the hardware and they've integrated beautifully and a lot of us use iPhones. Um but the vast majority of phones run Android because uh companies want to focus on um you know special features of their product as opposed to sort of basic enabling parts of the stack, which is why they use Android. And it struck me that uh there's a major software gap and that's Android for robots. Um of course there's going to be companies like Optimus and Amazon um that when they deploy humanoids, um they're going to own the whole stack. Um software, hardware, everything and you know there's no way that Amazon would allow um Optimus to do their logistics operation inside an Amazon warehouse. So the... the winners are likely going to be um Amazon and... and Optimus and they're going to do their own software thing. But there's going to be an incredibly long tail of companies um that are looking for uh I think the equivalent to Android. And so that... that's what we're trying to build. We're really proud of the fact that it's open-source because as you imagine the possible implications of thinking machines, um I think humans can make a really, really strong argument that we want to be able to understand what's going on, we want to be able to read the code, and we want to make sure that an entire generation of like new developers and new engineers and everyone um can interact with the technology at the deepest possible level. I do not think we want to be surrounded by thinking machines that, you know, have their own bank accounts and know a lot about you and for that software to be closed source. So like it's important for me personally that if we're working on a technology that powerful, well then everyone should be able to look inside and see everything that's going on. Um in terms of what the... what the software does, we live one step above motion policies and foundational models. Um foundational models have typically been messaged as you know the one-stop solution to making machines smart, and I'm approaching this a little bit more from a like really pragmatic perspective. Um I've spent uh also a lot of time on ArduPilot, uh open-source software for drones, and I'm generally a proponent of modular software where individual pieces can be independently tested, upgraded, improved and that you then snap together so that in aggregate you end up with awesome software. And so OM1 is modular. It's a series of inputs that connect to different types of sensors. Um it is a data fuser that takes information and turns it into a description of the world around you, and that description then flows to multiple interacting AIs that are operating on different time scales. Some of those AIs are really simple and they're designed to um be done with their token generation in 300 milliseconds. So they're used for things like not bumping into stuff. Um and then we have other AIs that are mentoring and they produce a page worth of critique of the human-robot interaction every 30 seconds. So it's like having your mother look over your shoulder and say, 'Hey Narb, stand up straight. Um what about this joke? Um have you thought about this? Why do you keep doing that?' And the mentor AI, which is producing this critique every 30 seconds, that then flows into the core LLM which is doing the usual um motion, speech, emotion generation every two seconds. That's a little bit what the stack looks like and uh we just had our first big hackathon in San Francisco, SF Hacks, with 300 people. And that was really awesome to see um what high school students and undergraduates build once they're brought into direct contact with humanoids and can start writing non-trivial functionality uh into humanoids.

Awesome. So basically at a high level, um would you say that this is essentially an operating system for robotics?

Yeah, well um that sounds like a little bit... a little bit much because um there's so many pieces of software that need to come together for a machine to be smart. For example, um you have the basic operating system like Ubuntu and on top of that you have to run middleware and that is responsible for moving data around the system. And then you need to run foundational models and motion policies. And then on top of those basic components you then need the decision-making and the memory and the cognitive part and the um uh the speech-to-text and all those things running to then give you an interactive machine that is good at you know tactics and planning and engagement and looking at people and making them laugh and things like that. So there's so many different parts of the stack that need to come together. Um I'm a little bit uncomfortable with you know the operating system for robots um simply because um also ROS exists and that's called you know Robot Operating System things like that. But really what we're focusing on software-wise is the very top of the stack, which is the thinking, making decisions, articulating what the machine wants and... and interacting with humans.

Right on, right on. And uh you... I was going to ask and you kind of touched on it. Um are there any interesting robotics that are equipped with OpenMind uh today? Maybe something you saw at the hackathon or... or something you have working in the background?

Oh! Um well given that we have four minutes left um uh there's a bunch of things coming out and typically um they have one thing in common: they're not focused on fine motor skills. Our software is horrible at fine motor skills. So if you want to do brain surgery do not use our software because that's not what we're emphasizing. Um we're emphasizing um uh decision-making, interactions with humans, conversation, contextual awareness. And so a lot of the use cases that are coming out have to do with things like... imagine you go to the ER and you're greeted by a humanoid. The humanoid asks you 10 questions: 'Why are you here? Have you had COVID?' And then there's a sensor mounted on the humanoid that does remote vitals assessment. So that's a situation where the machine is able to go up to patients, say hello, ask questions, do vitals assessment and that information then can flow into a hospital backend. So that would be... that's an example of probably 10 or 15 different use cases that are coming out over the next few months. But do not use our software to do uh brain surgery. Please don't.

Hehe. You heard it here first, folks. Uh yeah. Um amazing. Uh yeah and uh this being a uh developer-oriented show, um I looked through your docs and you do offer a... an API. Uh would you care to speak uh about its capabilities and what people can build with it?

Sure. Well um the by far the best thing to do is um to uh to look at the actual software. And um if you go to openmind.org um then there should be various links to get you started or point you to the GitHub and things like that. Um in terms of the APIs, um some of those are more for enterprise, um for example a HIPAA-compliant endpoint for healthcare robots and that's designed for hospitals and memory care units and old people's homes. Um and another API endpoint is for example a spatial RAG that um takes data coming from the robot, turns it into a inventory of objects around it and allows it to reason over that. But the core software will run on your laptop and you don't need to talk to any of our APIs. Uh you can spin up your own instance of Llama or DeepSeek or whatever your favorite LLM is and you can run the whole thing on your laptop and you don't need us at all.

Awesome. Uh and I guess is there uh as we're nearing uh the end of the show unfortunately. I feel like we could have gone on for another hour or so. Um but uh what... what are some interesting trends in the AI robotics and blockchain space that excite you the most? Obviously what you're building but is there any uh particular application of this or... or that you are very keen to see um come into play?

Well, one thing I'd really love to see is um robots more broadly deployed for good. Um so you can think of robots as this... as these tools that make it easier to like manufacture or uh eliminate humans in the workplace or something. And that's a very um sort of sad perspective. Um a more optimistic perspective is that thinking machines um could birth... could bring with them incredible benefit to so many areas. And I would like to see this all play out of most benefit to people and not simply ending up with a technology that uh you know like replaces human workers. I think of all those situations where we're currently limited by not enough people. A great example would be mental health. The reason most people in the US have to wait for three, four, five, six months um to see uh to get a mental health consult simply reflects the fact that there aren't enough doctors. And there's many industries where that's true, where we're not necessarily limited by money, we're just limited by there aren't enough people to do this. And so I'm most um excited about areas like healthcare and education where I see an incredible role um for teaching and helping um by smart machines.

Yeah, I... I agree. Um I think those are going to be some prominent areas, I'll also be um keen to see develop as well. Um and I guess uh to close us off is there anything you can share about the OpenMind roadmap? Uh perhaps features that are going to be coming out sometime this year or... or yeah?

When we ask robots what they want, um they'll frequently say that they want to talk to other machines. They don't want to be alone. Um they want to be able to um get data from other machines and also learn skills from other machines. And a very large part of what we're working on is software to help machines connect. Um as humans we have many technologies for doing that like this podcast or we have Zoom or we can meet in person and we can get a coffee and all these things. Um but the entire stack that relates to machine-machine communications um really remains to be built. We love middleware like Zenoh, but that type of middleware was never designed to accommodate uh dynamic teams of machines um that are transacting and sharing many different types of data for lots of different reasons, for security and identity reasons and many other reasons. Um but that's the main... that's... that's the main missing part of the stack that needs to be built next. It's basically: what does Zoom look like for thinking machines? What does Instagram look like for thinking machines? What does Facebook look like for thinking machines? And that entire part of the stack um is the next thing that needs to be built.

Yeah, well well said, well said. And yeah, definitely um this space is very fascinating, um so I'm quite keen to see its development and... and unfortunately uh this brings us to time, but um this was a very fascinating and interesting chat, Jan. Thank you so much for taking the time to come on today. Narb, um thank you so much for having me. Yeah, my pleasure anytime. And um for folks who were watching us live and are into collectibles, I'm dropping a link to uh bit badges uh for you to be able to... for you to be able to collect your uh collectible for indeed watching us live. Um so if you're into that kind of thing, definitely check that out. Um but with that I just want to wish everybody a very happy Friday, happy weekend wherever you may be and that will... we'll see you back here for another great episode of DevNTell next week. Alrighty. Have a good one. Cheers. Bye.