Exploring Web3 DIY infrastructure with Kleomedes

GM, GM. Welcome to what's going to be another fantastic episode of DevNTell. So if you didn't know, DevNTell is a 30-minute podcast held every week, allowing founders, hackers, and anyone in between the opportunity to come on the show and showcase what they built. And today, I'm ecstatic to welcome both Marco and Avi of Kleomedes. So if you don't know, Kleomedes is a Web3 sovereign infrastructure provider specializing in do-it-yourself, bare-metal blockchain infrastructure and more. So if you stick around for today's episode, you'll get to meet Avi and Marco, learn all about Kleomedes, and how you can get started using them today. All right, let's get into it.

GM, GM. Welcome to the show, Marco and Avi. I'm ecstatic to have you on today, guys. I know you guys have been cooking up a storm with Kleomedes, taking a peek around all the stuff you guys have built, and I'm really ecstatic for our audience to learn all about it. But I guess before we kind of get into that, do you guys just want to give an introduction about yourselves?

Yeah, sure. So my name is Avi. I guess not really full-time crypto; I actually was in medical school and recently started residency, and this is kind of something I do for fun. It's like a passion project on the side. But Marco and I actually met on Twitter. He used to run nodes as Full Stake, I believe was his moniker at the time, and I was a mere delegator and was basically just interested in what he was doing. I know that he was trying to create a DAO-run validator, and that was like a really cool idea. So I just reached out to him on Twitter, expressed my interest in helping him out, and that's how we first got started working together. And then I'll let Marco kind of introduce himself.

Hi, I'm Marco. I'm from Naples, Italy. I've been messing around with servers and hardware basically all my life. And it was 2020 when I found out there was the possibility for me to make money with my passion, which was running nodes. I've never been a big corporation fan, so the settings that crypto had at the time were perfect for me because it was just enthusiasts doing things, and I was an enthusiast, a crypto enthusiast, really a fan of the decentralization philosophy. So years go through, but I still really like what I do.

Amazing. And yeah, it's awesome how you guys kind of found each other. Marco was also telling me the story before we started the recording, so that's awesome. And congratulations on your residency as well, Avi. Good stuff, man. I don't know how you find the time. I found the time to come on the podcast today, but ecstatic that you guys did. Yeah, I guess we can start getting into it a little bit. I guess you guys kind of mentioned it briefly, but yeah, just interested to hear kind of how Kleomedes kind of all came together and what the origin story is there.

Yeah. I can maybe start and Marco can kind of add on to it because I guess from the outsider's perspective, I just saw that Marco was a validator who was basically advertising an airdrop, which I think has a lot of feelings around it. At the time, it was not really like a popular approach, but I think to his credit as a new node operator, it's very hard to attract delegations, and so this was like one of the ways people were trying to attract delegations. But we've seen a lot of DAOs that aren't true DAOs. I think there's a lot of centralization still that exists, whether it be things being run by a multisig or whatever. And this was basically a way to distribute tokens, and the majority of actually the supply to delegators. And so we worked on that together. There's actually a few more of us that were involved at the time, people from all over the world that basically all met each other online. And we distributed the tokens, started the DAO, it ran in Cosmos under some tooling called DAO DAO, which made it really easy to be able to create proposals and execute code in like a truly decentralized manner. And we basically did that for several years, hoping that we would be able to onboard new contributors with incentives. And basically what we were doing was we were using the validator revenue to bootstrap liquidity for the token, to pay obviously for operating expenses, and then also to give profit share to the token holders. And so we were kind of splitting the revenue in those three different ways, with hopes that if we created proper incentives and onboarded more team members around the world, that we'd be able to have like a big team of contributors that were basically going out and recruiting new chains for us to validate and things like that. So I think the idea was sound, and we actually were able to distribute profit for a while, and there were several months that we did really well. There were a series of airdrops on chain previously called Kujira. There was like a month that I think we made like almost $30,000 in airdrops and distributed all that profit across like the staked token holders. But, you know, as the market has kind of dwindled, especially in the Cosmos ecosystem, that became kind of untenable. So over time, we did our best to continue and try and make it work, but it didn't work. So then we ended up winding down the DAO. We tallied up basically all of the assets that we had and basically put forward a proposal to the community like, hey, we tried to do this, didn't scale like we thought, we were still able to distribute a decent treasury that was left over without having to like sell the hardware and things like that because people wanted us to be able to continue on and try and succeed as a private company. And so that's kind of where we're at now. But I don't know, Marco, if you have anything to add to that, because it was originally your vision to do this.

Yeah, as I said before, what I more like about crypto, it was the possibility for everyone to build something. Nowadays it has become more complicated, but at the time, you still could get the feeling that alone you could make an impact. And that's how I started. And yeah, we were successful for a little bit, and it's been like really exciting doing it. But we were working a little bit of like open-source contributor works, but of course, after some time, people want to get paid, and we could not always get people paid. So this created a little bit of instability into the DAO. And yeah, I feel like the decision to pivot to a private company has been like good for us. And I feel like our older DAO members still wish us to succeed, so I think that we close that chapter well. Right now we have different problems, to say one, the RAM shortage, so we would really like to scale because we're profitable and we're seeing that our product actually works. We would like to buy and build more machines, but unfortunately with the rising of artificial intelligence, there is no more hardware available. So right now, that's like our biggest challenge.

Yeah, that's a problem I think basically everyone from retail to enterprise are going through. It's just very, very expensive to get your hands on hardware, and when you do, it also costs a lot to actually run that hardware. So that pain is felt by everyone, and yeah, I'm curious to see how it's going to unfold over the next couple of years, whether those models will keep getting bigger and you'll need more power or they'll shrink and you won't need as much hardware. But we'll see. But yeah, that's certainly quite the origin story there. And yeah, DAOs are always very interesting in terms of bringing people together and wanting to do meaningful work, but then that governmenty type of feel to it where things have to be voted on and it kind of stutters that growth, that acceleration that a private company, a startup, would need. So yeah, I definitely resonate with that as well. But I guess kind of double-clicking into Kleomedes a bit more, I guess what is it exactly that you guys provide?

Marco, you want to take this one and talk about it a little bit, like maybe the philosophy behind it as well?

Yeah, so as I said before, it all started because this is a thing that I like. I like building machines, putting them online, and keeping them online with excellent performance. And the thing is that the cloud wasn't really built for crypto; it has been built for regular web. And of course, modern blockchains have resource requirements that normal web apps don't have. So with time, I found out that my machines were actually more powerful than the servers I used to rent. And the fun thing is that we still use regular consumer hardware, regular consumer drives. We have fiber lines, of course, but it's all stuff that is available to everyone. So, of course, you just need to have the knowledge to put it together, but it's all out there, anyone can do it. And that's exactly the spirit of cryptocurrency. Anyone can spin up a node and participate. So the fact that we actually are more performant than others is like, I didn't know at the time, it's something that I found out with time. And so we're taking, I started with one little data center, and we're trying to scale like a franchise. So not like a big data center, but a lot of smaller data centers so that we don't depend on any single point of failure. Even if one of these data centers goes down, there are another two or three that can take over the job. So this lets us not need super measures for backup; we have batteries, we have backup power lines, but we still offer redundancy through multiple data centers. And also this costs us like a small amount compared to building a regular data center with all the certifications and stuff. So we are more performant, but we are also actually cheaper than the competition. So again, we really, really, really like to expand and we hope to do it very soon.

Yeah, I think the scaling is an interesting thing because you hear about like banks that become too big to fail. And it's interesting because I feel like it gets flipped on its head when you think about cloud servers because you have these massive servers like AWS and Google and things like that, and they've gotten so big that they've actually created these massive dependencies. And we've even seen, you know, when one of these big servers goes down, it's like so many node operators are just non-functional. And it's actually scary because if you think about the original Bitcoin thesis and the idea of like nodes distributed across the world, it was very antithetical to what is happening now in proof-of-stake networks and like L2 nodes and things like that. And I think one of the big reasons why we're doing what we're doing is because we actually want to motivate other people to also do this. I think there's enough pie to go around, so while we want to scale and we want to, you know, get bigger, I think the whole situation will only improve if more people do what we do, and there are, you know, several people in the space that are doing what we're doing. But we never view that as like a threat to us, and so we just focus on what we're doing. I think one of the bigger things that we could talk about too is like we're pivoting. We're still doing validators, we still run a bunch of proof-of-stake nodes, mostly in the Cosmos ecosystem, and we still look to support new projects there. But I think we found that relying on token prices for revenues can be rough when the market conditions change. And so pivoting into the RPC space is another way that we're trying to scale our revenues, and I think it's going really well. I think that alone is what is enabling us to continue on at this point. And so anyone who has node ops experience that is trying to make a living out of doing this, I think RPCs are 1000% the way to go. And we support like Lava and Pocket Network, which are two decentralized RPC options. We're integrated with RouteMesh and a couple of others, but pretty soon here we'll be offering like direct to consumer as well, and that way people can reach out to us even now and express their interest and we can give them an API key and they can get set up right away. And we kind of just use like a credit system so they just need to send like a small deposit and then we will bill them monthly, and as long as they're paying their bills, you know, we continue serving relays without any limits. So I think like that's the direction we're moving into, supporting more networks, continuing to scale so we make sure that we have like proper redundancy. Obviously there's some challenges with that, but I think, you know, we can do it in a sustainable way. But that's where we've been and where we're going, I would say, in a nutshell.

Amazing. Yeah, I definitely resonate with this. And an interesting kind of tangent to this, and I know you wanted to talk about this as well, but just like the resilience and the redundancy of a network and the fragility behind it if say some kind of go down. And kind of what we saw with the LayerZero exploit with Kelp DAO, I think last week if I'm not mistaken. Yeah, I guess for our audience who might not be familiar with that, just curious to kind of get your guys' take on kind of what happened there and why it happened.

Yeah, about this, I wanted to particularly talk about the fact that people feel safe because they partner with like a big company. So they think like, oh, I'm using Infura, I don't have to worry about anything. But then this was the exact reason for the exploit because they were using not a big number of providers, just a small number. So the hackers could actually do like a man-in-the-middle attack and serve like wrong data. I don't think that this would have been possible if like the people behind LayerZero didn't feel like safe. So I think this feeling of safety, like I'm working with a big company, I'm safe, this is what is behind a lot of exploits in crypto right now. And I would argue that I would be safer with a smaller company because it is less known, is using less known practices. And you know, most of the hacks is actually social engineering. So if I don't know the company, I don't know who works for the company, I cannot engineer them. And maybe this is a weird point, but I actually think that going with smaller companies would really help a lot in this case, in this particular case, of course. And we talked a little bit with Avi what could we do to prevent another attack like this, and of course it isn't easy because the big infra companies almost are like a wrapper to AWS. And you guys as a developer really know that most infra companies nowadays they haven't got nothing, it's just a scheme that you put on someone else's services. And I know it's the modern way of doing things, but it's also why we don't have like real alternatives to this service because nobody wants really to build something from scratch. And I get that you don't need to reinvent the wheel, but I also feel that there are no alternatives there for very basic services.

Yeah. I mean, I think everyone knows about the DVNs that LayerZero uses. Like there's been a lot of articles that focus on like, oh, it was a 1-of-1 DVN, like of course if you're going to trust just one party to verify the origin transaction and the destination transaction, and then that's the point of failure. But I think what Marco said, and I think this is a narrative that's now finally catching on, as people are starting to realize that like, yes, that is for sure a sticking point. Like they should not just be using a 1-of-1, they should definitely be using 4-of-4, 3-of-3, or however many is possible, even if it sacrifices maybe speed to a small degree, because I mean, this is a $300 million hack. Like I think if you had to wait a few more seconds, no one would really mind. But that said, even I think they were using, if it was a total of maybe six or so different RPC services, and if I recall correctly, there was a DDoS attack to four of those services that then went offline, and then the one that they actually defaulted back to was two RPC services that had been compromised. And so I think the concern here, especially even for someone like us and what we want to learn from this, is like how do we make our own systems more resilient so that we don't suffer from something like this in the future when we do get big? Because Marco said it appropriately, like we're not a big provider right now, so like we're maybe not like a big target. But if the company grows over time, you add more people and you want to hire people, then it's like how do you know you're not hiring a North Korean hacker as part of your team? And so I think the key here is if there were only two RPC services left to verify the data, and if both of those had been compromised and fed faulty data where then the DVN looked at the data through those services and agreed that it was accurate and then processed the transaction, then what you're really talking about is back to like proof-of-stake consensus principles of like in Cosmos chains, like all of the nodes that are signing blocks need to agree. And there are other people monitoring that system. So if someone's not proposing the block themselves, all the other nodes look at that block and verify like, okay, yeah, this is the correct version of the chain. And if they think that a malicious version of the chain is now being forked or built, then it's their job to act as a good actor and like point that out. And there are obviously negative financial ramifications for people who try to fork the chain and create an alternative truth basically. And so really that's what we need. It's interesting, Pocket Network, this is a bit of a plug for them, they utilize a bunch of different providers like us of varying sizes. And what they do is they actually have sessions that are active. So like it's a round-robin system where everyone who has active RPC nodes gets selected for a session, and that session runs for a finite period of time. And then after that time period is done, a new session begins and a new set of node providers are brought in. And so there are solutions that exist to kind of avoid something like this from happening again. I think the industry now is like acutely aware of the problem, but it's tough to like implement changes like this on a large level. So that's kind of our take on the whole thing.

100%. Yeah, it's certainly a very weird time in terms of these exploits, like you guys mentioned. We've seen quite a bit of them this year, late last year. I personally think we're going to continue to see more of them, especially the stronger and stronger these models kind of become and become accessible by basically anyone. I think there was some sort of outage on GitHub today with some merges and PRs or something going haywire, so there's always going to be something. I think we'll see a spike and then kind of people will also the people that get kind of affected by these AI hacks will probably use the same AI to kind of make their systems more resilient and then maybe we'll see it kind of die off. But yeah, to your point, it's really hard even just beyond just like a retail user using like a SaaS or something. People who are looking for jobs, you don't know if the person who's interviewing you is like a North Korean hacker or some other fraudulent person, right? Especially again with AI, you can kind of spoof your appearance, your voice, is very, very dangerous. So you've got to be very vigilant. But I guess kind of bringing the convo back to Kleomedes and around some of the different product offerings you have, you mentioned your RPC. Just curious what other things does Kleomedes offer just for our audience to learn more about you guys.

Yeah, I can talk a little bit more about like StakeHub, which is basically one of the like tools that we kind of provide for some of the chains that we work with. Narb, you wanted to bring up the screen?

Great. So StakeHub is basically a website that one of our developers built to be able to monitor chain performance. I think this is like really valuable for foundations and chains, so maybe not so much like users exactly. On the user-facing side, it's basically just like a staking platform. So the way it works is if you need to stake tokens on like a proof-of-stake network, you can use this for that. But I guess more importantly for chains and like people who are using this from like a foundation perspective, they can actually go and look at these performance metrics and be able to tell like whether or not their RPCs are like functioning properly. So like if you look here, you can see that there's like some chain health metrics, there's governance health, uptime health, tells you like the Nakamoto coefficient for each chain, like how many nodes basically need to get compromised before the chain could get attacked. And then it looks also at services health, so like RPC, REST, gRPC endpoints, how many there are, what the aggregate score for those are. There's also like a decentralization score that basically tells you like how distributed the geography of these nodes are. The decentralization data is also interesting, so this takes like a minute to load, but basically it looks at a map that then shows you like exactly where the nodes are located. And then it also shows you the provider distribution. So you can see like Hetzner and Amazon and OVH basically have like a massive percentage of nodes, and Hetzner alone has like 39%. And so this is interesting to go through if you're trying to see like which chains are maybe like at risk versus not at risk. This RPC health score basically records like whether or not there are errors, and if there are errors, it kind of describes how many of each there are, what the success rate of that RPC is, what the average latency is. So this was kind of like the first thing we ever built to add value to the chains that we validate, and so they basically look at this and if they have a foundation delegation program where they support nodes, this is the data that they then use to decide how to stake their tokens with the validators they want to support.

So that's StakeHub. And then we actually just recently built this dashboard for Pocket. This is all Marco's doing, so all credit to him. But basically, we thought, you know, Pocket is like a great place to make money as an RPC provider, but they didn't have a great way of demonstrating that to other people. So we essentially created a dashboard that shows like market leaders, created a little bit of like competition here, like who wants to get to the top of the leaderboard, who's serving the most amount of relays making the most amount of money. And then it also shows you here exactly like how much provider-side revenue there's been for the defined timeframe. So I think for now it's 24 hours, but you can change this to 7 days, 30 days, and you can see like what the network revenue pool looks like. You can see how many relays have been served in that time, so 5 billion relays. And then you can see, let's say for example if you're trying to decide like which chains to go on, you can actually see here like okay, there are eight providers on this chain, this many relays, and obviously the less providers there are the more relays there are, it'd be a great place for you to kind of like expand to.

And so there's like here exactly, it says like high-growth opportunities, and so BNB is right now like the hot place to be, but there's no way for people to know that unless they actually like can see it visually here. And so that's kind of another thing that people can look at if they're node operators that want to get involved.

We also have like our own internal monitoring here, so like our health dashboard that basically shows like all of our nodes. So for Ethereum mainnet, we have multiple backup nodes, I think we have at least two machines for every chain and so there's always redundancy in case one goes down. Like here on Base, you can see there's like one unhealthy node but the other nodes are all healthy and so it's continuing to serve relays. And then we have, you know, a slew of Cosmos nodes, and so like all these different networks are networks that we support in Cosmos. And then we also utilize load balancers that are able to kind of like funnel relays to whichever node is able to receive them more quickly. So that's kind of like just a little bit of a look. I don't think there's too much else to show, it's mostly like dashboards monitoring things like that. But maybe Marco can speak a little bit more to how our machines are capable of being like better or like how our configuration puts us like at an advantage.

Yeah, of course the main problem with the cloud is that of course it's built for scale. So to say one, the drives are not linked to a machine by hardware but by network. So there has to be some more latency. Our setup is simpler because it's built with just one purpose in mind: to serve blockchains. So once you find, this is the sixth year that I do this, so I have a package, I tried all the drives, every possible combination so that now I can select perfectly the right hardware not for the machine but for the blockchain. Like I know that I don't know, Ethereum has lower block times, so I can go with as lower drives. I know that Base or maybe Arbitrum has sub-one-second block times, so it's hand-crafted for the chain. So the concept is really simple, but of course you have to put the work to have a great result. Of course, as I said multiple times in here today, the difficult thing is scaling because maybe that drive that you selected is not available anymore. I don't know, Crucial doesn't manufacture anymore for consumers now, just for companies. So that is becoming really difficult. But as long as you have all the hardware available, it's not really difficult. I would not say that anyone can do it, but if you're passionate enough, I think you can build something similar even if not like a rocket scientist.

Well said. And yeah, that's amazing to hear you guys take the optimization very seriously and kind of hand-craft each of the servers to that particular blockchain how it behaves. That's pretty neat. I guess kind of on, this is kind of on like the developer side, I suppose. For the developers watching today, what kinds of APIs and SDKs and whatnot do you have available for them to plug into Kleomedes and use that RPC?

So the thing is that since we cannot scale, we cannot offer like a real consumer platform right now. But we have like a private beta going on, so if someone wants to try our service, they can contact us and we can give an API key. They can like maybe confront us with the other providers and see if they get really a benefit. Right now our pricing is $1 for a million relays, that I think is really good since other providers are doing $6, $8, $10 for one million relays, so it's really, really cheaper. I don't know that we can give this pricing forever, but the good thing that even at $1, we are profitable. We need like a lot of customers of course to get the $1, but like we are profitable. We're not giving away stuff, so it's sustainable also for us. And I really look forward from someone that wants to try us and I really appreciate for feedback. And you know, I just want us to get better and better, so really appreciate.

Absolutely. And yeah, folks watching today who want to give Kleomedes a try, the contact information to get a hold of Avi and Marco is found in the description below, so definitely if this peaks your interest, reach out. And we always love our alpha on the show, so thank you for sharing that with the audience. And I see as we're kind of approaching time, I just want to kind of peek your guys' brain on a couple more things here. I guess we've been speaking about it throughout the show, but we've seen quite a number of outages from the big boys, the AWSs, the Azures, what have you, the Googles. I guess during those times, those outages, how did your guys' servers perform? I suspect they kept humming since you guys are running your own hardware, but yeah, just wanted to kind of get your guys' perspective on that.

Yeah, we of course are completely independent from them, so our services were completely unaffected. There was actually a moment where we got down for a few minutes, and it was for Cloudflare when it went down. And of course we learned from that and we cut the ties even with that. So since the beginning, we haven't had a major incident. Everything going really, maybe too well, so there is a little bit of preoccupation, but it's been six months I think, perfect uptime, like 100%. So I'm really, really satisfied with our work, and we'll keep it up.

Yeah. Six months specifically for the RPC service. Many more years with validator nodes. I think when Marco and I first met, he was still renting servers from Hetzner if I remember correctly. And we actually together as a team, when we were like still the DAO, made a decision to pivot away from that. And that was actually when Hetzner first started advertising that they were going to start kicking out node operators. Because I think to Marco's point, people who rent servers in these different data server farms, they utilize the crap out of these disks, and they have to replace disks probably more frequently than they do with other clients. And I think there's like a financial incentive for them to also like create terms of service that exclude people, especially for Solana, because Solana's network requirements are like insane. So once that happened, we decided to take everything in-house. But to your point, Narb, I mean, if you're a centralized service going down, like that's obviously not a good thing. But even for a smaller service like us, and we rely on our machines for income, we had to make sure that we had proper redundancy before making that massive pivot. And so we waited, we saved up a ton of money, in addition to buying all of the machines, we still have like backups readily available so if something does happen, like if Marco's house or data pod goes up in flames or something like that, we definitely have the ability to pivot and deploy backups within I think it's like 15-10 minutes, something like that. But even in addition to that, it's like we have, you know, backup internet connectivity, we have backup power sources. Italy gets very hot, and we've seen times where like power outages have happened in Italy, and we've actually done well through those times too. Like Marco has had to power up the generator before and make things run on that, and then eventually backup nodes elsewhere and make sure they continue running. So I think the important thing is that even though we're small, it's important to take this very seriously because it's our reputation on the line, and so we've kind of prepared for all of those types of outcomes as well.

Excellent. That's awesome to hear. And I guess in terms of the long-term vision for Kleomedes, I guess can you guys kind of share anything around the roadmap around what you have planned for this year and maybe beyond?

Yeah, I mean, I think we talked about scaling. The biggest priority for us, even though it's a challenge right now, is to expand beyond Europe and to have data pods in other continents. Obviously, our latency in Europe is excellent, but if you want to run like a performant node for like Hyperliquid, whether it be a validator or RPCs, like you pretty much have to be in Tokyo to do that. And renting servers in Asia is like especially expensive. And so that's kind of one thing that we're looking at, to expand to other countries. And then I think one of the other things that we're now looking at because of what happened with LayerZero is just looking at developing software to help prevent these sorts of attacks in the future and just partnering with more blockchain projects and foundations that, you know, want a highly performant node operator that can provide like metrics, insights, dashboards that they need so that we're contributing more than just signing blocks.

Marco, you have anything to add to that?

No, I think you've been perfect. And yeah, I think the thing that I would like the most is to replicate the setup that I have here in other cities, in other countries. And it would make us like really, really, really reliable also in terms of latencies. So I think that will be the next step.

Promising. Yeah, I guess to keep up with all the great stuff you guys are doing, what's the best place for people watching or listening today to keep up to date with all things Kleomedes or get in touch with yourselves?

Yeah, I think the Twitter DMs should be open. We also have a Discord that people can join by going to our website. We have a Telegram group. I think both me and Marco's personal Telegram handles will also be included in the description with this. So you know, the benefit of being a smaller operation is you guys can reach out directly to us if you want, and it's easy to get a hold of us. We're both very responsive. I think just to kind of maybe insulate Marco of the inflow of DMs, like I am technically like the BD guy, so I love people, he loves machines, and that's kind of how we've delegated tasks. So if you want to hear back, I'm probably your guy. But Marco's always around and he's always helping users out too. Like people in our Discord will ask questions and either of us, whoever is around, will reply. I guess the best thing about having people in Italy and the States and then even in Australia is that there's always someone who is awake regardless of what time it is. But it makes it hard to meet because we have a very narrow window of time to meet.

Yeah, those time zones, man, I know the pain. But yeah, that's wonderful to hear. And just as we reach time here, just want to thank both of you, Avi, Marco, so much for taking the time out of your busy days to come chat with us. I know I really enjoyed our convo, and I'm sure our audience did too.

Thanks, Narb.

Thank you, Narb.

For sure, for sure. And yeah, with that, I just want to wish everybody a very happy Friday, happy weekend wherever you may be, and that will catch you back here for another great episode of DevNTell next week. All right, have a good one folks. Cheers.