Exploring Real-Time Market Data Delivery with DoubleZero Edge feat. Ben Marx

GM GM. Welcome to what's going to be another fantastic episode of DevNTell. So if you didn't know, DevNTell is a 30-minute podcast held every week, allowing founders, hackers, and anyone in between the opportunity to come on the show and showcase what they built. And today I'm ecstatic to welcome Ben Marx, who is the Head of Engineering at Malbec Labs. So if you didn't know, Malbec Labs are the core contributors behind DoubleZero. A dedicated network of high-performance links designed to provide low-latency connectivity to decentralized applications. So if you stick around for today's episode, you'll get to meet Ben, learn about DoubleZero, learn about one of their latest products called DoubleZero Edge, and a bit more. All right, let's get into it.

GM GM. Welcome to the show, Ben. I'm excited to have you on today, man.

Well, thanks very much for having me. I likewise, it's great to be here.

Yeah, yeah, for sure. Uh, yeah, we, we had the pleasure of being introduced to DoubleZero late last year, if I'm not mistaken, by Austin. But I'm ecstatic to get you on to get more of the technical side of things, technical perspective, as you are the Head of Engineering behind Malbec Labs. But before we get into all that, would you just like to give an introduction about yourself?

Sure. So, I live in Venice, California. I've been working in different industries in the tech world for the last 15 or 20 years. One of the, I think, the interesting things about the tech world is that you can sort of jump from industry to industry. So I've worked from sports, which I don't really care about, I'll talk a little bit more later about why I joined that company. And then I worked at another networking company where we built a global network. And then I worked at a space company where we built like a ground system and built that in Elixir, which we can talk about a little bit more. And then I found myself here at Malbec Labs. And I think the line that connects them all, all these different companies together, is that they're trying to do new things, new innovations, and they're trying to do something at the bleeding edge. And that's something that's appealed to me since I was a teenager messing around with Linux and, you know, you think about in the mid to early '90s it was Microsoft, this massive corporation, versus Linux, the underdog. And that's something that's stuck with me throughout my entire career.

Amazing, man. Yeah, I'm the same. I love being on the bleeding edge. Sometimes it's painful because not everything works. But that's why you gotta be somewhat of a pioneer and help push that forward. Yeah.

I mean, when I was younger I would try to get the bleeding edge of Linux or Arch Linux or Debian or whatever it was, and all of the hours and days spent fighting with various drivers not working. So I'm not quite as bleeding edge in that sense anymore, but I still like to be on the forefront of technology and building cool things.

Certainly, this is the way. And as I understand, you are quite the prominent figure in the Elixir community and you even co-authored a book. So yeah, I'm just curious to dig into that a bit and learn a little more about that.

Sure. Well, I'll dispute the prominent figure. I did write a book, co-authored a book called Adopting Elixir with Jose Valim and Bruce Tate. Jose is the creator of Elixir. And so I got into Elixir because I was, I'd been working with Ruby on Rails and Ruby in general for a few years. And I wanted to get into functional programming. There was something that appealed to me about that. It seems a lot simpler to me. And so I started learning Clojure. And then sort of happenstance, my boss at the time was like, 'Hey, have you heard of this language called Elixir?' And then I was like, 'Wow, this looks like Ruby. This is very easy to digest and it kind of hides some of the trickier functional programming paradigms away from you at the start.' And then I found a job at Bleacher Report, which is this, I think it's like the second largest sports company in the world in terms of users. And they adopted Elixir way before they should, but it was because they were going through this sort of transitional phase. And this gave me the opportunity to learn Elixir at scale because we had, you know, millions of users, hundreds of thousands of concurrent users, sending out something like a quarter of a million or a quarter of a billion push notifications every month. And we were doing all of this in Elixir at the bleeding edge. And so that's how I ended up meeting Jose and I used to speak pretty regularly at conferences around the world about how to use Elixir, how we did it at scale. And again, this was sort of serendipity, right? Like if I hadn't been at the right place at the right time, I wouldn't have had these opportunities or if I hadn't said yes to these things, I wouldn't have been able to do these. And so it was really great to experience that.

Amazing. And do you think all that experience there kind of set the foundations of you jumping into the distributed systems crypto scene?

Definitely. So two things. So there's something, so Elixir is built on top of Erlang and Erlang has been around since like the '90s. And Erlang is designed for fault-tolerant, highly distributed concurrent systems. And so they have these behaviors built into the system so that you can take advantage of concurrency in a really easy and convenient way. And so we don't use much Elixir at Malbec. We have one tiny little thing that I wrote, but the same paradigms about how to build fault-tolerant, highly scalable systems, we do apply those things. Like one of the core tenets of Erlang and Elixir is 'let it crash', which is sort of a, it's the idea of we don't want your system to crash, but actually sometimes you do because the best way to get back into a good state is to turn something off and turn it back on. You know, like the cliché of 'turn it off and turn it on again' is actually a really good way to build a reliable, scalable system. And when you think about something like DoubleZero or, you know, a global network or a system that has millions of users or hundreds of thousands of concurrent users, these things are complex systems and they don't actually, they're never actually fully healthy. You know, just like your body is never fully healthy, like my foot hurts, but I can still function. And so in the same way you want your system to say, 'Okay, this part of the system isn't functioning well. How do we work around it? Or how do we get that back into a good state?' And so that's done in Elixir and Erlang through something called supervision. And we try to take those processes and carry them over to what we're doing here. And even there's a Rust version of the Erlang virtual machine whose name escapes me. But all this is to say that the ideas that have been battle-tested over the last 20, 30 years in Elixir and Erlang are now being applied to other languages and other paradigms as well.

Fascinating, fascinating. And I like that metaphor. Yes, it certainly makes sense when you kind of think about it. But yeah, I guess kind of transitioning into DoubleZero itself. Like I mentioned at the start, we did have Austin Fedora on the show earlier or late last year, but for people who might have missed that episode, can you just give us a high-level overview of what DoubleZero is?

Sure. So DoubleZero is a high-performance, dedicated, decentralized network. And so this again, like if you harken back to my 13-year-old days, like the decentralization aspect of DoubleZero is incredibly appealing to me because it means that no one person owns it, no one entity owns it, and it's, you know, censorship-resistant and all of that kind of stuff is really nice. But the idea behind DoubleZero, that you should have a network that is dedicated for a specific purpose, is not new. All big tech companies have their own private networks. Google, Meta, on and on, Netflix, they all have their own. High-frequency trading firms likewise also have their own private networks. And the reason is pretty obvious because if you need to get something from point A to point B that has to get there in the quickest possible time, and I think in terms of high-frequency trading, that can be like nanoseconds, like those, that's like the timescale that high-frequency trading firms focus on, and you just can't do that over the public internet, right? Because the public internet ISPs have different incentives than what a private company would have because they want to get as much data through the network as possible. And if it gets congested and if you have a little bit of packet loss or a little bit of jitter, it's not a big deal. Your Netflix show is probably going to be fine. You're going to be able to load a website fine. But if you're making trades that are on the order of these millions of dollars, if you can get that the fastest, then you're going to spend the money to do that. And so DoubleZero is taking the same approach and open-sourcing it to the community to say, 'Hey, what if we built this thing that can do the same thing as these high-frequency trading networks do, but we can bring it to Solana and basically any other use case that you can think of? If you can send a packet over the network, you can probably take advantage of DoubleZero.' So that's sort of a rambly way of saying that DoubleZero is trying to democratize these private networks to be used by anyone that would want to take advantage of having a reliable and dedicated network.

Yeah, it's certainly very fascinating when you start to really dig into the tech. And you mentioned trading. Yeah, trading is very high stakes, like a few milliseconds makes the biggest difference in the world from you getting like a good entry price versus like a bad fill, right? Exactly. And I mean, one of the prominent use cases in the crypto industry is trading, arguably. So yeah, I mean, having that parallel there in this side of the world, and I guess both sides of those worlds are kind of like colliding now, mixing together. But yeah, certainly the time is quite crucial. And this is a nice segue into one of your most recent product releases, I think from a couple months ago if I'm not mistaken, called DoubleZero Edge. Which is, yeah, I'll let you explain it.

Sure. So DoubleZero Edge is built for traders to take advantage of our decentralized high-performance network. So one of the great advantages of building this global high-performance network is that you can start to build applications on top of it. So that it can be very clear, it also lowers the barrier to entry for how to use the network and what you can get out of it. So again, like this same ethos that we have for decentralization and for democratization is why we built DoubleZero Edge so that you can get shreds as quickly as possible. And so you can go to fastshreds.com to get more information about that. And without our community, this amounts to nothing, right? So every validator who joins DoubleZero and publishes shreds into it gets rewards. And so if you publish a leader shred, you get some percentage of rewards for that contribution. And the more validators that join, the more likely that the network will get the shred faster and you'll also be able to probably get more rewards as well.

And then I want to also emphasize that the way that the rewards distribution is done is not a zero-sum game. It's not one person take all. We use something called Shapley values. And I actually studied economics. I have a degree in economics, which I've never used until this job where I, the chief economist and I like to argue about economics, which is a lot of fun. But Shapley values is a game theory concept and basically what it says in the way that we use it is, so you have these networks of links and if one of those links isn't there, how much worse is the network? And so what, how does the network suffer if you're not there? And we use some calculations to determine the value of your link that way. And so this way everything is evenly and equitably distributed. So the more you participate in the network or the more shreds that you publish, the more rewards you get. So it's a pretty easy concept to understand, although there's a lot of complicated math behind it. And I think it's a really nice way to be as fair as you can be to make sure that everyone who participates can get rewards for what they're doing.

Awesome, awesome. And just to kind of step back one step here. You mentioned the word shreds. For people who aren't so familiar with the Solana ecosystem terminology, could you just explain what that means?

Sure. So whenever, like shreds are what you build up a block in the Solana ecosystem, right? So those little chunks are what you get to help you build up that block. So the faster that you get those, the faster you can build a block and the faster that you can maybe do something with that block. So again, this is all about timing and making sure that you can get things faster and you can distribute them faster, then you can likely make tighter spreads and get more rewards for, or you know, not necessarily rewards, but get more for that value.

Oh, awesome. Awesome. And thank you for that. And just to, I guess, go back down a level. So I've read up a bit on DoubleZero Edge and it seems you guys are using multicast, which is also what a bunch of the big trading firms also use. And I actually learned this yesterday, it seems like the data is transferred for the high-stakes trading over microwaves. Did you guys take a similar approach to data transfer with DoubleZero Edge or...?

Well, we're not using, we're not using microwaves. We haven't gotten to that point yet. But I think the multicast is, yes, that's sort of the underpinning technology of what we're using for DoubleZero Edge. And there's an, it's probably like 10 years old now, but there's a presentation, you can find it online, I'll send you the link, but it's called Boring Technology Club. And the idea is that for every startup you have a certain number of innovation tokens. So you can, like say you wanted to use one esoteric technology or you want to change something. But the core of your business and the core of your technology should be boring. And in that sense, boring means like battle-tested, stuff that you can depend on. And especially as an infrastructure, we want our product to be as boring as possible because if you're an infrastructure provider and your product isn't boring, then that probably means it's on fire. Right? And like the last thing you want is for someone, especially with an infrastructure project, to wonder if it's going to stay up or down. That's like a death knell for a product if you can't depend on your infrastructure. And so the way that we've built DoubleZero is through that same, although this was never discussed through this presentation, but we've basically taken all of these tried-and-true internet technologies that have existed for 20 years or more and the same with the high-frequency trading and big tech company, these private networks. We've taken all of that because we know it works and we've put a novel approach on it by being decentralized, by us owning none of the hardware or the links. And by fostering this community of network contributors, that's the innovation that we've done with DoubleZero to make this new thing. And that's also why we've been able to iterate so quickly and build it up because we didn't come up with like a DoubleZero IP protocol. We're just using the same, you know, boring technology that works. And it's worked really well for us so far.

Yeah, man. Boring works. And like you said, if you're an infra company and you're somewhat exciting, yeah, something's on fire or yeah, you're not having a good day. And as I understand, you have a dashboard as well that kind of showcases some of this. Yeah, be awesome to kind of pull that up.

Yeah, let me get that set up. Okay. So this is pretty cool, I think, because you know, if you were having a problem with your internet connection, you would maybe go to fast.com or you go to Cloudflare and see what's going on. But you're not going to be able to go to your ISP and say, 'Show me how congested your links are or show me how your traffic flows are.' But everything we have about the network is here. It's open-source. It's all on-chain as well, so you can go and verify what's going on. And so as we walk through this, feel free to ask about what things you'd like to see, but I'm going to take it from a top view and then dig into some of the smaller pieces as we go on. But here's the network itself. And this shows the network that we've built, all of the places that we have devices, the links that exist. You know, you can see it on different views, globe view and so on. We can also see, you talked about multicast, we can see the multicast groups that we have. And let's just pick one. This might take a sec to load. But again, what this does is showing it's pulling data from on-chain, pulling data from ClickHouse, and everyone can see this as well. And we can, actually, let's maybe go down here and then to the status. Because and this goes sort of an overview of the whole system, right? Like I talked about earlier about how every complex system is never fully healthy. And so you can see here we have all systems operational with three drained links. And the drained links means that the traffic is no longer flowing through them for one reason or another. It looks like we have packet loss or ISIS down. But the system is still operational. I think this is the point to underscore, is that we've built a system that is resilient and redundant and can handle links that are down because you know, you can, when you design a system, you know, with the CAP theorem, consistency, availability, and partition tolerance, you can't have a consistent and available system. You have to have either a consistent and partition-tolerant or an available and partition-tolerant system because physics, like things break. You cannot, so you have to choose one or the other. And I think this is a good demonstration of that from a high view.

And then we can go down here and see some of the devices that we have. So load in a second, hopefully. There they are. And so here you can see we have 97 devices. And again, we don't own any of this infrastructure. The network contributors are the core of DoubleZero. Like we wouldn't, we just Malbec Labs provide the software which is a protocol for users to run and for network contributors to run on their devices. But without the network contributors, we have no network. And so this is a really incredible thing that the network contributors and us have built over the last year and a half to get this network up and running.

And then if we go here to the shreds, this is a view into I talked about DoubleZero Edge and this is part of the shreds are part of that. And so you can see here just showing that we have the our win rates by these different regions and DoubleZero is doing pretty well on the winning rate. Like you can see here it goes the slot leader as it goes for each one and pretty generally we're winning almost all of the time. And then we have about 44.5% of the publisher stake weight and I think the last time I checked we have about 49% of Solana on the network as well, which you can see in here, yeah, stake share about 49.37%. So I think this is really, you know, speaking back to the ethos of why I joined DoubleZero is this transparency in the fact that we have this information that anyone can see. And I'm not even signed in, right? Like anyone can check this out. Like there this isn't hidden behind some gate that no one else can see. And let's see. Yeah, so is there anything else you'd like to look here or we can move on?

I was just going to ask, and I'll mention the link to this dashboard for anybody viewing listening today will be in the description so you can go poke around. But in terms of, I guess, supporting the network from a hardware perspective, what does it take for somebody to run the software and support the network?

Sure. So we have on our website docs.malbeclabs.com, there is a site, there's a section on how to join as a network contributor, the devices that we support, and that, but in a very simplified way, you need to have two devices, right? A point A and a point B, and you need to have some fiber that you can connect between them. And then once you have those, you run our software on your devices and we have ways of configuring the devices automatically for what we need to do. So this also helps the network stay in sync. And then we run some tests with the network contributors when a link is ready and once it looks good, then that comes back into the network, or it comes into the network and then traffic starts flowing over it. So we try to make this as seamless as possible. And we have a, the team at Malbec has quite an interesting set of backgrounds, right? Like I we have people who come from high-frequency trading, we have people who come from networking, we have people who come from both, and then people like me who have just a sort of random background. But because we have all this experience in these different industries, we can say how do we make this concept of having people who we have some network contributors who have never managed a network before who are quite competent now in managing the network, which is a really great sign. And so the way that because we've had this rich experience, we've been able to take all of these different people and put this into this into this new project. And I think we've made sort of the best of all worlds because we've had this experience and because we've been able to make this this thing that we know we all want. Like our goal is for us to not have to manage the network, right? We want to make sure that the network is healthy, but our goal is not to go in and fix these problems because we can't. Like the network providers own their devices. We can't actually get into a device. So if there's a problem with a device, like we have to work with them or what's more likely is they notify us that, 'Hey, we're going to have maintenance on this box or there's some outage for here.' And then they'll just let us know and they'll drain the link and traffic will continue to flow seamlessly.

Well said, well said. And I guess building something like this must've been quite complicated, but you mentioned it's the sum of many established internet technologies that have been around for quite a while. I guess were there any kind of big gotchas or anything like that as you were building it? Just curious.

Yeah, no, it's a great question. And I mean, I'm trying to think, we knock on wood or whatever you want to do to guard against bad things. We haven't had any major outage since we've launched. I think we had an issue in testnet maybe I think we launched testnet in last April and I think we had some small issue in June or July. But other than that, it's been pretty smooth sailing. I think the thing that we need to do or continue to do is just because is show that the network is reliable and resilient and highly performant. And so because you're asking someone to take people say, 'Oh, the internet's good enough.' Right? But DoubleZero is better, but you're asking people to say like, 'Why should we move to something better that we don't know?' And so I think we've shown over the last year that we know what we're doing. The network is sound. The network is resilient and it's high high-performant. There wasn't a bit that I didn't show you on the labs or on the data side that shows where we are better than the public internet. And like we're doing pretty well against the public internet. And so again, all this data is public. So anyone can see this and they can see that the network is a high-performance network. I can bring it back up if you want to quickly.

Oh, it's okay. We can keep going. Okay. Okay, sure, sure. And I guess in terms of, so we are a developer-esque show. For the developer watching today who might want to tap into this high-speed market data that DoubleZero Edge provides, how can they start utilizing it?

So they can start by just signing up. We have if you go to doublezero.xyz, there's a sign-up page there. Also on the docs.malbeclabs.com, there's a way for you to sign up. So we're also trying different ways of signing up for people. So there's a way that you can just sign up through the web and start getting access there, or you can do it through the CLI. So there are different levels for different developers who prefer different ways of doing things. All of our code is open-source as I think I might have mentioned. And so you can also just go in there and audit your audit yourself. And we also take security very seriously. So we've had pen testing audits and we have security audits for our smart contracts as well because we want to make sure that we can be as, you know, as transparent and responsible as possible when we deliver software for people to use.

Excellent. And I guess speaking from all your experience and kind of going through all the trials and tribulations and kind of ending up where you are now, for some of our audience members who might be thinking about starting their own company or maybe switching it up and changing careers, I mean no better time to build than now. I guess is there any advice off the top of your head maybe you'd want to give to your younger self perhaps?

Yeah, I think I mean, this is almost like a cliché, but I guess most sincere things are clichés because people say them all the time, right? But it's if an opportunity comes up that sounds good and you like, do it. Like what's the opportunity cost of not taking that chance? Because like say you're starting your own thing, in the like software isn't what it was in 2010 or maybe 2020 in terms of job availability. But like if you're a good developer and you try something on your own, like the worst thing that's going to happen is you get another job. You know? So if you have that idea, go for it. You know, like there's there's no reason not to.

Yeah, exactly, exactly. And especially in this day and age where maybe maybe it was like time, you didn't have time to do it or maybe you didn't have the skills in the like a particular language that catered to whatever you were building. But now you have this magical thing called AI that can spin up whatever you want. At least an MVP version of it in in a day. Exactly, exactly. Yeah.

Yeah, I mean we only have a few minutes left, but we could talk about AI for a long time. Yeah.

Yeah, that that can be a whole other episode for sure. Um, I guess in the few minutes that we have left here, I guess could you just speak to kind of what the future vision of of DoubleZero, I guess from your perspective, what that is and if there's anything that you can share from the roadmap?

Sure. So I can so the immediate goals are to continue to improve the Edge product, right? Like making sure that that's the best that that it can be. We released geolocation, which is a way for you to be to verify a location of an IP or a device. And there are a lot of interesting novel use cases for that. And we just like I think we just released the page maybe this week or last week, so it's pretty hot off the presses. And then as far as like the vision of DoubleZero, one of the exciting things about DoubleZero is that we don't know what what it's going to become. Like when you build a network, especially because we operate on the, you know, 1, 2, 3 level layer of the OSI stack. And so again, like if you can send a packet over a network, we can probably handle that. And so we don't really know what what people will use the network for. And I think that's one of the really exciting things is and my hope is that as we continue to push towards decentralization and that we can bring up different parts of the world that are under-represented. And this is a way for people to, you know, in maybe in South Africa or in different parts of MENA or different parts of South America, if they can have the same connectivity that we have in the US or Western Europe or other parts of the world, who knows what they'll build. And that will really level the playing field. And that to me again, if we go back to this idea of why joining DoubleZero in the first place, it's decentralization and sort of democratization and providing resources to people that didn't have access to them before. If we can do all of those, then that I think is the realization of DoubleZero in its in its full potential.

Well said, well said. And yeah, as we as we kind of come to time. What's the best way for people to kind of keep up with all things DoubleZero and yourself?

Yeah, so I'm on Twitter at BGMarx, although I don't tweet at all. But we have, of course, on docs.malbeclabs.com, the DoubleZero website, and we have a Malbec Labs account on Twitter, and we have a DoubleZero account on Twitter as well.

And all that information will be linked in the episode description below, as well as the dashboard that Ben showed and we'll also throw in that geolocation feature that was just off the hot off the presses as well. So you can dig into all of that. And with that, Ben, thank you so much for taking the time out of your busy day to come chat with us today. It was amazing to learn more about DoubleZero, DoubleZero Edge, and yeah, I'm sure our audience also appreciated it as well.

Yeah, well, thanks so much for having me on. I can show you the DoubleZero versus the internet if you want to do that as a as a last thing before we go.

Sure. Yeah, sure. Let's do it. All right.

Okay. Should be there. All right. So again, anyone can check this out. But you can see here we have all these savings over the public internet from 4 milliseconds to 55 to 18 and the RTT round trip time improvement. I mean, these are pretty significant improvements. And so again, like if you think back to we're talking about landing trades or doing something more reliably and more efficiently than someone else, if you can get a 60% improvement over your competition, there's a lot of room for you to make a lot of decisions there. Right? Like if you're saving 55 milliseconds, I mean, that's that's like a low-earth orbit ping in terms of time saved. So you can do a lot with that if you think about sending a a ping to space versus sending something across the world.

That's crazy. Yeah. I'm going to I'm going to poke more into this dashboard to to kind of see see more for myself, but that's amazing. Awesome. Awesome. Um, and yeah, I guess with that, just want to wish everybody a very happy Friday, happy weekend where you may be, and we'll catch you back here next week for another great episode of DevNTell. Till then...

Yeah, thanks so much for having me again.

My pleasure, my pleasure. Till then, have a good one, folks. Cheers.