SQD: The Web3 Data Layer

GM, GM, and we're live. Welcome, everybody, to what is going to be another great DevNTell. So if you didn't know, DevNTell is a 30-minute podcast for builders to showcase something they're passionate about and have been working on in Web3. So whether you're a big-time Web3 company or a weekend hacker or anything in between, this is the platform for you to showcase what you built. And today, I'm ecstatic to welcome Dimitri Zhelezov, who's co-founder of SQD, the Web3 data layer. SQD ingests and stores petabytes of raw blockchain data, unlocking indexing and data access at a speed, scale, and price never seen before. If you stick around for today's show, you'll learn what SQD is and how you can get started using it today. Alright, let's get into it.

GM, GM, welcome Dimitri, or do you prefer Dima?

Yeah, GM, GM. Both work. I guess Dima is just a little bit more pleasant for a Western ear, so otherwise, yeah. Thanks a lot for having me.

Our pleasure. It's wonderful to have you on, and I'm really interested to learn all about SQD. You guys were previously Subsquid and have rebranded recently, so just in case people are wondering, Subsquid and SQD are one and the same, but I'm sure Dima will get into those details as well. But yeah, excited to have you on the show. I guess before we really get into the meat of the content, did you want to give a brief introduction about yourself?

Yeah, sure. So, I'm Dimitri, or Dima. I have an academic background. I graduated with a PhD in Sweden. Then I did my postdoc in fundamental math, working with Endre Szemerédi, one of the greatest minds of our time, I think. And around that time, I discovered Bitcoin. I definitely heard about it before, but wanted to get a little bit more into the way it works, and finally, I understood the actual value of being able to transfer anonymously, and it kind of clicked to me why it actually has value. Before, I just thought it was magic internet money. Around that time, I started to do independent research, started to do some blogging, and just traveled around the world at conferences. Back then in Berlin, I just met some dudes at one of the rooftop parties and it turned out they were building some protocol for data-intensive IoT applications, basically making IOTA great again, a fork of IOTA that was pretty big back then. Right now, it's more like a dead coin, but the project was really interesting. So I joined it first as a part-time researcher, then took over pretty much all the R&D on the protocol side. That didn't quite take off back then, but unfortunately, the bear market started and we ran out of funds. But still, it was a pretty interesting experience, and I just understood that I would never look back from Web3; I got red-pilled with Web3. So yeah, I started to work at another company called Joystream, and I was set to build an indexing solution. They had a lot of data stored on-chain, and back then it just struck me that it's very difficult to work with this data. And this is basically where I first realized how one can improve this. The rest of the story is how we came up with this idea of building SQD, but I guess we will get there soon.

Excellent, yeah, that's awesome. I always enjoy hearing everybody's origin story, and I'm sure a lot of other folks watching today are also pretty red-pilled on Web3 like I am. So that's awesome to hear. Cool. I guess without further ado, let's get into the content.

Yeah, absolutely. So, SQD, formerly Subsquid, in a nutshell, is a way to access data the Web3 way. It all started from a very simple thing, basically just from the concrete problem and my frustration about RPC nodes, and in particular that taking data from RPC is just incredibly slow. I was amazed by how much research and effort is put into making blockchains performant, but no one really took any effort to make data easily accessible. Everything that is built on the RPC side is really much of an afterthought. And when you actually start building something, you end up doing a lot of weird contraptions that in no way are aligned with the Web3 ethos. So, here we go. I start with the slides that I prepared for today.

And this is where the SQD Network really starts. We built the Web3 data layer, but before we proceed, I would like to start with some first principles, I would say, and also look back at the history of Web3 even before it was called Web3, right? So, what we really believe in and what we think about when we say Web3 is universal, anonymous access; the dogma of 'don't trust, verify,' meaning that everything that is not on your laptop is assumed to be against you; user ownership, meaning that no one, even the government, can stop you from what you are doing. If you want to transfer the funds, no one can really steal them—not your keys, not your money. You have full responsibility and full custody of what you do online. And finally, an often overlooked feature is what I call grassroots liveness, meaning that there is no centralized point which can be just switched off. And as long as there are people who want to be part of the protocol, the protocol is live. For example, as long as there are miners in the Bitcoin network, the network itself is live. So even if you shut down all the major miners around the world today, there will be some enthusiasts around the world who might be with less powerful equipment but still be able to produce blocks, and if the hash rate drops down, they can actually still keep the network alive. It's a very genius design which basically ensures that as long as there is an interest in the project, it's alive.

So, okay. Bitcoin is really what kind of started it all. And it really checks all the marks that we mentioned before. So it's censorship-resistant, it's grassroots live, it gives you permissionless access and self-custody. But it solves a very specific problem: peer-to-peer transfers. It has its own drawbacks: fees, block times, etc. I'm not going to dive into this, but to me, it's more like a canonical example of these cypherpunk ideas put into a complete piece of software. When it comes to accessing the data from Bitcoin, Satoshi envisioned that already back then, it is kind of hard to assume that everyone is going to run the full node of Bitcoin. But instead, you can actually run a very small piece of software, which is an SPV client, and do some cryptography magic that would guarantee that even if the full node that you're connected to somewhere on the internet is trying to fool you, you will be able to detect and figure out that the data is malicious. So it is full custody, it's still very easy to run, no one can cut your access because you can admittedly find some kind of full node somewhere, and no one can really cut your access from Bitcoin and you get the actual data from it.

Now comes 2015. A new era started. The Ethereum mainnet, envisioned by Vitalik and company, launched. And it's really an iPhone moment of Web3 because this is the first time when from simple transfers we leapfrogged into applications. It's really like how an iPhone made a revelation that a phone is not really just for calls, but for something a lot more complex, and with a lot of possibilities to put things together, to make a Turing-complete virtual machine one can build virtually anything. And at the same time, it, of course, increases complexity of the software and the network. Years later, we are now in the boom of L2s, L3s, and Solana because the concept of a programmable blockchain is so powerful that people started to build applications, primarily DeFi at the moment, but it is clear that the power that you get from this programmable blockchain is just enormous. But on the other hand, the on-chain data and the complexity of this data is just growing exponentially, right?

Because one thing is just keeping a ledger of how many tokens belong to each user in the network, which is basically the Bitcoin ledger. Another thing is all the transactions and application states that are essentially encoded on-chain in a very intricate way and there are now almost probably around a petabyte of on-chain data combined from all the chains, and what is even more annoying, it's fragmented across different pieces. So, this picture really encompasses all this complexity, all this predicament when a user finds himself basically saying, okay, there are hundreds of chains right now. There is like this monster of Solana, there are like BNB, and we just end up getting control of how we actually work with it, right?

The real modus operandi at the moment is essentially just fall back on the centralized setup where we have a bunch of centralized RPC providers that we use in order to send transactions, which is kind of okay because this transaction is signed and we just say, okay, probably some of them will land this transaction on-chain. But when it comes to reading this data, the situation is completely unacceptable simply because most of the applications really rely on a centralized indexer that combines, collects, and aggregates this data for you, and you just consume a ready-to-use API without really any assumption about the nature of this indexer. So it's completely possible that the data is bogus, there is no way to verify it. And at the moment, only a small fraction of applications can really pass this Web3 philosophy check, like a Web3 fundamentals check. And typical concern here is that, okay, it is always possible to run a node and just self-host the application and just interact directly with the chain, but this is really a wishful thinking. So you cannot in practice access the data, especially the historical one, and make any insight without relying on a centralized provider. And in general, there is no way to verify any meaningful complex state of an application.

And this is at large the problem that we are solving with the SQD Network, and it is envisioned as the Web3 data layer. Essentially, the SQD Network is like a data ocean. It is a decentralized network which contains all the data across all the networks, including Solana, EVM chains, soon there will be also Cosmos chains and the other ones. And the promise is that you can run a very lightweight portal application on your laptop or even on your mobile phone, and access and query all this information across all the chains with a simple interface. Get as much data as you want, as fast as your internet bandwidth connection allows. And the cryptography magic guarantees that this data is valid, and essentially the guarantees are comparable to what you would get if you were running a full node. But instead of running like hundreds of nodes across all the networks out there, you just get a simple, single application, which is the SQD portal, that gives you access to pretty much all the Web3 data out there. So far, the focus was really on the on-chain data, but this design extends to off-chain and Web2 data as well. So we would really want to think about the SQD Network as a Web3 data ocean, which at the same time passes the Web3 fundamentals check.

Let's revisit how this Web3 philosophy is matched with the SQD Network. So first of all, permissionless access. The way it works is that in order to access the network, you have to lock SQD tokens, and basically the more SQD tokens you lock, the more bandwidth you get. SQD is available on DEXes, so it's a very natural mechanism to allocate resources so that the network can serve as many consumers as possible in a fair way, and essentially no one can prevent you from acquiring SQD and getting access to the data. The 'don't trust, verify' component are economic and ZK, basically the crypto-magic part guarantee, that guarantees you that the data is valid, and even with a very small piece of software, you can leverage like petabytes of data that were stored in the SQD Network. User ownership, again, this thing is completely self-hosted, and the portals are lightweight so that you can run it even on your mobile phone. So there is no way to cut or shut down your account. For example, there are these scary stories from Google removing accounts from organizations or Cloudflare just closing someone down. It's completely impossible by design. And finally, the liveness, right? So as long as there are nodes in the network, the data is going to be accessible. And there are no centralized choke points that are going to fail. So all the connections and the data is consumed in a peer-to-peer fashion. So basically your portal is a node, like a small access point in the whole network, and it connects to the nodes in the network through a peer-to-peer fashion. So there is like no centralized intermediary that basically can break the traffic in between.

This is all good, and these are like pretty big words, right? And I prepared a very early preview of how the SQD portal actually looks. This is run against the testnet, which is called Tethys. And just for the sake of ensuring that everything works and doesn't break online, I just made screenshots to walk you through the actual process that is already available on the testnet. So it all starts with a very simple process of generating local libp2p PeerID keys locally. You get your public PeerID, which you then register in a special smart contract in a gateway application. Gateway is another name for the portal. And there are two things: so first you need to register your PeerID, you can see it on the screen here, and the second step is to lock some SQD, and the way it works is that the more SQD you lock, the more throughput, meaning like the more queries per epoch, you can make. So I here locked 100 tSQD, and that's more than enough to like test and query the data in SQD Tethys testnet.

Now, like once you have the key, you start the portal, in this case, it's just like a simple Docker Compose application but there are also binaries. And then you have a local endpoint that you can query, and there is an already pretty powerful query engine that allows you to specify which block range you want, which data you want, which transaction filter, traces, etc. I'm not going to dive into this, but to me, it's more like a canonical example of these cypherpunk ideas put into a complete piece of software. You can query the data specific to a contract or specific to an address. And then just query this localhost endpoint, and the query is propagated to the portal, and the portal locates the nodes in the network, and piece by piece, chunk by chunk, completes the answer. At the end of the day, you get a stream of data matching your filter, and these are JSON lines that can be then piped into any kind of downstream application. So it can be SQD SDK, which is designed to work with this data and you get a very performant, completely self-hosted indexer without relying on RPC. It could be just something like Kafka or any other database like Postgres. And without running any kind of node software, you get access to the data across multiple networks, including even traces for most of them.

So, this is currently in beta. We are preparing for a release that is slated for Q3 24. This in general is called Data Streams, which means that you can get historical and on-chain data streamed from the SQD Network on demand. Later by the end of the year, hopefully we get to the point where not just the pre-configured data sets are available through the SQD Network, but actually it would be possible to submit a user dataset. So one great example would be to upload shadow logs for specific contracts, for example, as it's getting a little bit more attention at the moment and it's a nice way to get more instrumented and more useful logs. And for example, you can generate these shadow logs locally and then upload to the SQD Network so they're available for querying for everyone. And 2025 will be built around like building this vision of powerful cryptography to guarantee validity and bringing this data into smart contracts so that you can actually use these queries against the SQD Network even in smart contracts—a very powerful feature. And finally, the on-chain SQL, meaning that actually you can build from these pieces a fully powered, fully-fledged SQL engine that can execute a lot more complex queries. Because right now, it's more like a simple filtering across multiple fields, but in many cases, you for example want to aggregate or have full scans of the network, and instead of relying on centralized providers like Dune or Allium or some other, for example, BigQuery things, you can actually run this stuff locally without allocating a lot of resources. So I think this is going to be like really amazing and really in this spirit of Web3. And finally, the indexing endgame is having a different kind of network, perhaps on the re-staking AVS stack, that would be serving APIs specific for the applications. So this is more about R&D at the moment, but the idea is that like once we have this universal data access layer that can power extremely fast and powerful indexers, we still need the way to distribute this data and basically build APIs specific for applications. The Graph kind of made the first attempt to build a decentralized indexing network, but we see that due to the technical limitations, it kind of falls short of having of getting this mass adoption, like once they migrated into the decentralized network. I believe that with the right combination of modularity and using SQD as a universal data layer, one can build a lot faster and scalable indexers on this modular design.

Meanwhile, the SQD stack is already used in production by almost 100 projects, and I encourage you to apply for credits to try it out and also, of course, be the first to try the portal's beta. Scan this QR code, fill in this simple form, and we will reach out to you. And meanwhile, you can check all the information at sqd.dev. Thanks for attention, I think this is now the time for the questions. Yeah, excellent questions. I didn't quite cover this part, but the network is fully live, the SQD mainnet is live. So the easiest is to go to our website, sqd.dev, then go to the network page. And then there is a page specifically for node operators, basically for infrastructure providers who run nodes. So you need to acquire 100,000 SQD in order to bond as an economic deposit, and then you can join the network and get the rewards. You get like around 20% APY for joining the network. At the same time, if you're more of an SQD holder, you can delegate to one of the networks and get around 8 to 10%. This is, we have a pretty much novel design for delegation, meaning that if you delegate too much, the API, the yield is going to go lower because by design, you have to actively participate in the creation of the network, right? So you have to actively monitor which nodes are more online, do their work properly, and then you get the best yield. Otherwise, you just like risk not getting any rewards at all.

Gotcha, gotcha. Excellent. Yeah. And I guess in terms of kind of aligning with your grand vision—we know, we all know like data is king—what are some kind of interesting ideas or things that you hope people will build using SQD? Like just beyond the traditional like analytics dashboard and whatnot, is there some other thing you're hoping people kind of build or things you have in mind that you don't have time to do yourself?

I am super bullish on device-first and local-first indexing, and this is something that we are looking into ourselves internally, but I would really like to see mobile-first or browser-first applications which don't really depend on anything but for the actual code. So I think like the endgame here is really having SQD network for historical data access plus lightweight client to get real-time data from the network, basically just like connecting to the peers, and getting rid completely of RPC providers, and really utilizing to the max the power of the client. Because actually, both mobile and browsers became increasingly powerful, and we still have a lot of inertia designing the applications through this client-server architecture. But I believe that the future is really more of like data plus the client without really having kind of a server intermediary. And actually in Web2, there is kind of a renaissance of SQLite at the moment. People are super excited about like putting the data on the client—I mean the data that you can use. Yeah, absolutely. This is like a very good point because I started to draw this like human on my diagrams, right? But in reality, I would think that in 5 to 10 years, there will be no human there, right? So it will be more like an AI assistant, and the human will be just like talking to something on their laptop and everything is going to be happening under the hood. And absolutely, the way that we think about SQD network is that it is one of the data sources that is going to be used seamlessly in any kind of AI-powered applications, be it RAG or an AI bot. And we think that this portal approach is a move in the right direction of moving the responsibility of how to handle query and aggregate data on the client. So the SQD network is designed to be very unopinionated, very simple, at the same time the client can build upon pretty complex data primitives and for example even like join their local Spark cluster, so like what not, or if it's on mobile, like join some private data which is not available, of course, on the SQD Network. So yeah, it's really about making what we do good, like what we do well, in a very convenient way.

Yeah, yeah, I've certainly seen the same sentiments as well, and yeah, I'm really curious to see what people or how people start to innovate using the data that your platform is going to provide. Unfortunately, we're running out of time here. The last couple questions for me is: how can people get started if they want to spend their weekend hacking with SQD, and what's the best way for people to get in touch with you if they have questions or just want to learn more?

Yeah, absolutely. So docs is the first page to look at, it's docs.sqd.dev. You can also find a link there on our webpage. And we have a dev channel on Telegram, SQD Devs, Squid Devs. So we are pretty active there and answer questions pretty much real-time. You can build pretty much any kind of indexer applications, scraper, analytics across more than 200 chains, including Solana, already today. So yeah, this is already something that you can play with, and if you want to host your indexer, of course, don't forget to apply for credits, and if you have any like idea in mind, we're very happy to get in touch at any moment.

Excellent, love it, love it. Yeah, that was a wonderful session. Thank you so much, Dima. I really appreciate you taking the time to come on today and enlightening us on all things SQD. I'm really looking forward to seeing how folks start using it and and people to get on board hosting nodes there as well.

Thanks a lot, Narb, for having me. It was a pleasure and yeah, wishing a really good weekend, good hacking maybe on Friday and on Sunday and Saturday, and yeah. Thanks a lot, cheers.

Yeah, yeah, likewise. And we'll catch you all here next week for another great DevNTell. Until then, have a good one. Cheers.