SurfDB

Welcome to what is going to be another great DevNTell. So if you didn't know, DevNTell is a 30-minute window for members of the DAO to showcase something they are passionate about or have been working on. This can be an awesome project they've been working on, demonstrating unit testing best practices, automation goodies, smart contracts, how to structure a project, etc. Basically, if you've got a passion for something, this is your opportunity to share it with the community. And today I am happy to introduce to you Aditya, who will be demoing the Partical project. Take it away, Aditya.

Hi, hi everyone. I'm Aditya, and today I'm going to be showing something I've been working on over the past three weeks. So this is like a project I started with it in the ETHOnline hackathon, and yeah, I'm pretty excited to show you all. So I'll share my screen.

Please let me know if it's visible.

Yep, I can see it.

Okay, yeah. So actually my project is actually SurfDB, not Partical. I mean, it was called Partical before, but then I had to pivot, and yeah, it's called SurfDB now. And SurfDB is basically a database made for dapps.

So the problems which we are trying to solve is like currently with the decentralized databases like Ceramic, IPFS, Arweave, and Filecoin, etc., they cannot be used as a database for dapps. As a database, I mean to store app data like we do usually in MongoDB and SQL. So that's why developers usually are using Web2 databases like MongoDB and SQL, which is not really Web3-native, and why we can't have Web3 features. And another problem is that if we are using these Web2 databases, we can't really store files on them. So usually we use IPFS to store files. And IPFS again, then there's this problem of which provider to use. So Infura charges like $60 for 100GB, and free solutions like Web3 Storage by IPFS only is extremely slow because there are a lot of people using the same API, the same gateway, which makes it extremely slow. Then another issue is that some of these databases, they don't have smart contract access control conditions. So what I mean by this is ability to have read-write access condition based on some on-chain condition. So for example, an on-chain condition could be like a Gnosis Safe owner. So something like that. Then some of these databases like Tableland and Arweave, we have to pay for actually transactions, so if you want to write some data, you need to pay, which is not really ideal user experience. Then the decentralized databases, yeah, the performance is obviously you have to sacrifice on performance to get the decentralization, and yeah, that's also one issue. Next is that if apps are using these centralized databases, then it creates these silos where you, like, I can explain with this picture here. So in the example of DAO tools, so each DAO tool will be using their own sort of centralized database, and DAOs using multiple DAO tools now have separate data on these DAO tools, and there's no way to port data from this to that because the app owns the data since it's centralized.

So this is what I wanted to solve with SurfDB. So it's basically a decentralized database built on top of Ceramic and cached using Redis for fast response times. You can have contract-level access control conditions. So as I said, you can have a condition such that only the owners of a Gnosis Safe can write and update data, or owners of an NFT can write and update data, or even just your address. This also can be decentralized by hosting multiple nodes. You can get real-time updates. So if you're creating data or updating data, you can subscribe to an event and you can get the update. Then you can even have your own dedicated IPFS and Ceramic node and use that for file uploads instead of relying on other IPFS providers. You can have private and encrypted data. Since it's stored on Ceramic, which is a decentralized database, it's open, like the data is open, anyone can read it. So to avoid that, you can even encrypt it. Then the main thing is that you can eliminate the data islands by, instead of creating your own data from scratch, you can fork an existing node. So what I mean by this is that if let's say there's an existing app using SurfDB and it has some data regarding whatever it's doing, let's say Gitcoin. Gitcoin is using SurfDB and it has this grants data, and you want to build your own grants app, what you can do is you can fork the Gitcoin's node and you can easily just create your own frontend on their data. And the good part is that any new data being created on the original node, the Gitcoin node, it will automatically be synced to every other node, and vice versa. So even the Gitcoin app would benefit if other nodes are getting new users. So yeah, this was the background. Let me show you a demo.

So this is sort of like the dashboard page where you can create your instance, a SurfDB instance. So this instance is sort of deployed on Digital Ocean. So for that, before this, you sort of need to add your Digital Ocean access API keys. I have already added, so I'm not showing. But yeah, once your Digital Ocean access keys are added, you can create your instance. You can select the regions offered by Digital Ocean. So I'll select Bangalore since I'm from Bangalore. Then you can select the size, medium. And what you can do now is you can either create just a SurfDB instance or you can create it with the Ceramic and IPFS node. You'll get your dedicated sort of IPFS and Ceramic node. So I'll do this. This process takes a while, it can take up to three minutes, because first we need to sort of create a, in Digital Ocean they call it a droplet, which is like a VM, a bare minimum VM. So right now it's basically, we are on creating droplet stage, where Digital Ocean instantiates a VM, and then once that VM is instantiated, we have to set up the Surf client and all the other IPFS node and Ceramic node.

While this is loading, we can see what's actually happening in the background. So how it works is, so this thing is also using SurfDB as its database, this app. So what I did was encrypting the Digital Ocean keys and storing it on SurfDB. Then when we create an instance, we decrypt the key and then call the Digital Ocean API to create a droplet and a space. So space is basically like their S3 storage stuff. Then when the droplet is ready, we call an API to the Surf controller which installs the Surf client and sets up the IPFS node and the database on the droplet. Once that is done, we can query the Surf client on the IP, so the IP is assigned. So on this IP and port 3000.

Once we get the IP, this means that the droplet is ready but now we are initializing the server on the droplet. This will take a couple of minutes. Till then I can use this instance I had created earlier and demonstrate the capabilities. So using Postman, I can just ping it to check, yeah, it's running. So the first basic thing we need to see is the authentication. So for SurfDB, we always need to authenticate before we can do anything, which is slightly different from traditional databases. We need to authenticate with the sign-in with Ethereum sort of format. So basically we need to sign a message and send that signature so that we can verify that we own this address, like this is my address. So without signing in, without authentication to SurfDB, we can't do any creation or, I mean, we can read but we can't create and update because every create and update is checked if that address is allowed to do it. Every operation is based on your wallet's address. This is where this on-chain sort of conditions can come into play.

I have already logged in, I've just copied like the signature from the client and since you can't use Postman to sign messages, so I've just copied this and I've logged in. So what we can do is let's see how we can create some data. So if you see the API here, it's 3000/data/page. Page is like this schema name, so you can create whatever schema name you want. So in this case I have page schema name and then here in the data you can give any JSON data, and entity address is basically which address has the access to this data. So right now I'm just giving my own address, so only my address can update this data after it's created. And so in this address, you can give contract addresses, NFT token addresses, anything. So here, this is just some random data and I can send the request. Yeah, it has created it and you can see we get this stream ID, which is sort of the Ceramic stream ID. And now if we do a GET call on this, we'll get all the rows created on this schema.

So yeah, these were the rows which I created earlier and this was the row which we just created. And this all, like some of it is stored on cache so we get it very quickly, just in 180 milliseconds we got all the rows. Then the next thing was the file upload. So you can see here, I'll upload a file. So if we do a POST call to the 3000/file, yeah, it's uploaded instantly almost. If you'd use public IPFS providers, it'll take at least a couple of seconds. So we get, we got this hash, we can copy it and see. So the IPFS gateway will basically be your IP but it will be on port 8080.

So we'll go there. Yeah, like instantly we got the image. Using Web3 storage and stuff, it sometimes even takes a few seconds to load the image itself. But since you have your own dedicated IPFS gateway, it will load instantly. Now so this instance I was, let's see, yeah, so it's active now. So this basically we have deployed it, but you can even create locally. So I have, we can go for that, we have created these docs where you can sort of set it up locally. And it's very easy. So first you just need to install the binary, and I've already installed so I won't go through that. Then the next is you can, we have created this package called Create-Next-Surf-App, which is basically it will scaffold like a Next.js app with RainbowKit and SurfDB already set up. So I'll do this.

And sorry, this is creating a frontend scaffold for you to be able to connect to a frontend to SurfDB?

Yeah, yeah, it's a Next.js app. Once your instance is ready, what you need to do is just copy this IP and add it to the Surf client file and yeah, let's run it. So yeah, this is sort of the default page and to get started by connecting a wallet. This authentication, it's all like the Surf client, so authentication using RainbowKit, it's not that we are authenticating just like the frontend, it's actually being authenticated to the Surf. Now we can, we can create or get data.

So let's click on create data. We just created some random data, like hardcoded. So we can even like see the stream ID on Serscan, which is like the Etherscan for Ceramic. So this was the stream which was created. And yeah, like you can even get data and instantly you'll get rows. So whenever you create data, it gets cached on Redis so you get it like instantly.

Let's see what is in the code. Here what it comes with is the API setup. So AUTH API is already set up which is sort of integrated with RainbowKit. Then the data API. So using this, you can have this SDK, our Surf client sort of JavaScript SDK which you can use to do all the operations, create, update, read. So this is what is set up in this app. Then so like after this, after you install this, you can go through the docs and all the operations are listed here, create, read, update, real-time and even like the Next.js example. So yeah, this was the demo.

Yeah, this is awesome man. This is yeah, this is an area I've been dabbling in as well as there's nothing really built on top of the existing decentralized storage solutions we have now. But this is fantastic. So the question I had was, when you were doing the create POST, you were creating data it looked like, but what's the process to create your actual schema definition?

So it's right now there is no, it's NoSQL, so there's no schema as such. It doesn't matter what the data is in this JSON. It's stored directly on Ceramic. I mean, unless in the future you could create like, in Ceramic there's this thing called schemas where you could create streams out of a schema. But right now it's all freeform, you can just create anything. This schema name is basically to group, to help in grouping it. So it's indexed based on this schema name. Like if you want to get all the pages, you can just simply GET call on this. That's why the schema name is used but it doesn't mean anything, there's no actual schema defined.

I see, so this is kind of like a grouping key of sorts. Gotcha, gotcha. Crowd, any questions from you?

This is so awesome. I had just one question. Do you have any plans to move away from Digital Ocean? I do like that it fits into like the SurfDB theme, but just curious.

So I started with Digital Ocean because I first of all wanted to avoid AWS because I personally find it very complicated and also reading feedback from others, it seems it's very difficult to maintain and keep track of the bills and stuff. And that's why I went with Digital Ocean, it's like extremely simple even from like a developer side. You just need to add your one API key and using that key you can just create a virtual machine very easily. So that's why I started with Digital Ocean, but yeah, like if developers want, we can easily support, scale it to other providers like Google Cloud, AWS. But for now it's on Digital Ocean.

And is it only deployed on one node or is there some fallbacks also that Digital Ocean provides?

Digital Ocean does not provide any, like Digital Ocean is just providing a VM. Everything else I'm doing like setting it up and adding like the nodes and stuff. As I said, you can if you really want to decentralize you can host multiple nodes. You have to do it. From the app you can create multiple nodes if you want and in each node in different regions and you have to then decide how to sort of direct the traffic between the nodes. But you can easily like create multiple nodes, you can just create like a safety node and just not use it, use it only if something happens to the main node. The data will always be in sync.

Right. Cool. Cool. Any other questions from the crowd? If not, I do have a few more. I guess what inspired you to create SurfDB?

Yeah, so because I was facing the problems which I mentioned here. I am sort of also working on like this other project where we are building like a DAO tool for DAOs operation management and like task management. So while building that, we sort of internally had a discussion of how we can decentralize our infrastructure because we were using MongoDB and we just couldn't find a easy solution. Like there was no way to use any of these databases which I mentioned as simple as how we are using the traditional Web2 databases. So I'm sure all other DAO tools which want to decentralize themselves, they can't decentralize because there's no tool out there to even help them. So that's why I felt like this is a missing point. And SurfDB also has like very easy deployment. Right now if you see IPFS nodes and Ceramic nodes, it would encourage more people to deploy their own IPFS nodes which help in the overall decentralization of IPFS and Ceramic. So there's like two benefits to it, like you get your own node and you also contribute to the decentralization aspect. Because right now I'm sure at least for Ceramic only the core team has nodes. I highly doubt anyone is operating their own Ceramic node because it's hard, it's a bit complicated. But what I want from SurfDB is just one-click deployment, you know, no difficulties and just straightforward easy solution.

Yeah, yeah, definitely. We gotta definitely lower the technical barriers for people to start using this tech more freely. So this is wonderful and thank you for building something like this. And on the topic of building, is this open-source? Do you need any particular helping hands from the DAO to continue building it out?

Yeah, it is open-source. Like the core client is not open-source, but all the other libraries and stuff which help with the orchestration is open-source. And yeah, definitely I could use some help there. And basically I think mostly I need help is in feedback. Right now I'm assuming devs need this kind of solution but I don't know. So I would really want some developers to try it out and like our docs are very easy to get started with. I will definitely share this and just please try it out. I really want to know what other features we should have. So that feedback would be really helpful.

And is there any process to sign up for like a beta test or something or anybody can go use it today?

So like this deployment is not live yet and this might take some time because this is like slightly critical because we are handling the user keys and like deploying it in production. I need to do some more checking, so this might take some time. But you can install it locally, you can run it locally. It's very easy to do it, you just need to install the binary and run it. So you will have all the features that I showed.

Excellent. We only have a few minutes left. Are there any questions from the crowd? Going once. Going twice. Sold. Well, alright. Thank you very much again Aditya, this was a fantastic demo and something that I'm sure a lot of folks have been clamoring for but haven't been able to find. But this is fantastic that you've taken the initiative to build this out. So fantastic work and yeah, really looking forward to see how this develops and really rooting for mass adoption for this. So good on you man.

Yeah, thank you, thank you so much.

So where can I share like the relevant links, the docs?

Yeah, so I was actually going to follow up with you after the fact. So you can send those to me and I'll be posting them in the blog, but or alternatively you can post them in this chat for everybody else to have beforehand as well, in the developer voice chat I mean. Cool. All right gang, with that I want to wish everybody a fantastic Friday and weekend, and yeah, we will see you back here next week. And a reminder, we have new DAO-level roles, so if you want to get notified for things like DevNTell workshops and whatnot, you want to head over to the DAO-level channel right at the top of the server and click the corresponding roles. Thank you Kemster. Click the corresponding role for notifications you're interested in. And with that, we will stop the recording.