Ceramic Network

GM, GE, GA everybody. Welcome to what's going to be another great DevNTell. If you didn't know, DevNTell is a 30-minute window for members of the DAO to showcase something they're passionate about or have been working on. This can be an awesome project you've been working on, demonstrating unit testing best practices, automation goodies, smart contracts, how to structure a project, etc. Basically, if you've got a passion for something, this is your opportunity to share it with the community. Today, I am happy to introduce to you Matt from the Ceramic Network, who will be going over composable data with us. Take it away, Matt.

Excellent. Thank you for the introduction. Hi, I'm Matthew. I'll share my screen really quick and we will get going. Why is screen sharing always a challenge? You'd think after doing this so much that it'd be easy, but no. So we're going to talk about Ceramic a little bit. I'm not sure if you guys are super familiar with it or not, so hopefully after the end of this you'll have an understanding of what we're doing over on our side of things and how you can fold it into your app. If I breeze through this quickly, I have a couple demo apps that I can pull up and show you.

Outside of that, let's talk about who we are. First and foremost, my name's Matthew. You can find me over on the Ceramic Discord. I'm the developer advocate for our team. The rest of our team, however, is way more exciting than I am. We have a crazy group of developers that come from large-scale enterprise Web2 companies. So we have people working on this protocol from Facebook, from Mongo, from GitHub, as well as other financial institutions like Deloitte. Some of the engineers on the team developed UPort, which was the first Web3 identity solution. We've got people that have authored core Web3 standards such as decentralized identifiers. There's been a ton of developer-centric products that have been shipped and available to use across Web3 applications by our team. So we've definitely been well-versed in the old way as well as the new way of doing things and we couldn't be more excited to be solving this data problem that exists.

Getting into what Ceramic is: easily we could just say, hey, we're a decentralized network for composable data. That's somewhat helpful, somewhat not. I think it's more helpful to break this into two parts. The first part will be composable data. Imagine I'm a brand new end user and I land on Twitter. The first thing I need to do is create an account, right? And when I do that, I'm asked to enter my name, my phone number, my birthday. But then I also want an Instagram account because 140 characters of text just isn't as good as a picture of somebody's really bad plate of food. So I sign up to Instagram and I'm asked for my phone number, my name, a username, and password. And then I realize well, if I'm going to go down this rabbit hole of social media, I might as well be in the most evil of social media companies, so I sign up for Facebook too, right? And it asks for very similar information. And you can see there's a lot of area where we could very easily just reuse old information that is still valid information. Name is a good one, or phone number is a good one. They all ask for it, but as the end user, I've had to put it in three different times. So this is problem number one: as a user, I have to do this too often.

Problem number two is once I do that, where does it go? The moment I put my information into Twitter's form, it goes to Twitter's database, right? The moment I do that with Facebook, it goes to Facebook's database. So these pipelines of end user to application are all kind of housed and locked away inside of an application-specific database. So problem number two is it becomes Twitter's proprietary property the moment I give them my data. Then we have to kind of trust Twitter and trust Facebook and trust all these other applications that when we request a view that uses that data, that we're actually getting the proper view, or something that is true and that they're not manipulating it on the way or doing other things with it. Then there's the developer problem, not just the end user problem, but the developer problem of if I need to integrate with one of these applications, I have to implement some sort of one-off API. And if my application starts depending on that API, then all of a sudden Twitter says nope, we're going to cut you off, you're no longer going to have access to the API. Like what are the business effects on my application the moment I lose the ability to interact with these APIs that somehow get closed down, right?

So there's a lot of different problems that are boiling up with the current Web2 system in data and it all points to the same thing and that's the moment you provide your data somewhere, it's no longer yours. You have no agency over it. We've seen this a lot with how data is sold and how it's used to manipulate all sorts of ad campaigns and stuff. It's not yours anymore. Even requesting to delete your data isn't an easy thing and you then still have to trust that it got deleted.

So at Ceramic, we're trying to say hey, how can we solve some of this, if not all of this, all these problems that we just talked about? So we look at this from a composable data point of view. So we could take our little developer here and maybe that developer creates a schema for what a profile looks like and we call it a data model currently. And this contains data that a user would enter, right? Their name, their birthday, their phone number, everything we just looked at. And then whatever applications are built aren't reading or logging that in their own database, rather they're querying the Ceramic network for that profile data and then filling the page or the signup forms or whatever it might be with that data as the application then needs it. What we get out of this is the user now has a central place where if they decide to change their name or they change their phone number, they can update that in one central location, which is on the Ceramic network. And then that immediately gets reflected across all the Web3 applications they use that read from that data model. And we're not limited to one data model per application, we could pull in a hundred different data models per application and we also keep the APIs open, they're not going to get shut down, it's an entire decentralized network, so this data is also always available.

And we store this data in something called streams. So if you're digging into the documentation, you're like I want to implement a data model, what I keep seeing all this stuff about streams, well this is kind of what a stream is. You can think of a stream as this append-only log of commits, right? And they all get reduced down into a stream type. So it's the history of changes to that data. I've updated my name, I've changed my phone number, I've added a new field that didn't exist before. All of that gets captured and lives alongside of that data interaction in that data model forever, which is kind of cool. And it's append-only, right? So we can't remove stuff out of it, we can very definitively say like this is the history of the data. So each one of those commits is actually a content identifier that is an IPLD object, right? So you can start to see now that Ceramic layers on top of IPFS. You need to be running IPFS in order for the Ceramic protocol to kind of interact with that.

These stream types aren't just a series of commits like the stream is. They also provide logic for like what can happen with that stream. Currently we support two different types of stream types, the most common one is a TileDocument and you can think of that as like a MongoDB document-style database: big object of key-value pairs, right? There's another one that deals more with CAIP-10 links and then there's a couple that are in the works. Stream types are extensible. And then all of those things get combined, right? We get the log of commits and at some point when a change happens, we make what's called an anchor commit and we say all right, let's go ahead and write those updates down to a blockchain. So we're keeping track of the commits on-chain, not necessarily the data on-chain, so this is really small on-chain interactions. And again, the things that are being anchored to the blockchain are encoded using IPLD. So again, on top of IPFS.

All of that is great, but it kind of sets us in this weird realm of all the streams that exist are globally readable. Which is kind of cool and also kind of scary at the same time and something to be aware of. If everything is globally readable, then it's probably not a good idea to store credit card numbers on the Ceramic network, right? It's not encrypted by default, you would need to add some sort of encryption layer to it like Lit Protocol in order for everything to be encrypted. It's globally readable which means I just need to know the name of the data model and when somebody connects a wallet, if they have data associated with that data model and that decentralized ID that they have, then we can read that in. But applications can't change it. Only the owner of the stream, the controller of the stream, can change it. And we get that control through this authentication mechanism that exists. We typically authenticate one of three ways. Right now if you're an end user, the most common way to connect is through something called 3ID Connect. This is a little modal that will use your MetaMask to create a decentralized identifier which is then used as the controller for a stream. It's a little clunky, you have to sign a couple messages, so it takes a little bit more time than we'd like it to, but there's a solution for that. The second common way is a Key DID method, which is you're setting that up more at build time. Developers are probably going to use these as they create and publish data models for themselves. Or if I had a service that needed to write to the Ceramic network, I might give that service its own key. It's not going to have a wallet to interact with, right? So I would use this type of method. And the most common and preferred way right now that we want you to use going forward is sign-in with X flow. Right, sign-in with Ethereum, sign-in with Solana. These authentication flows are becoming more and more prominent and we have factored them into something called CACAO, which is a chain-agnostic capabilities object, which will allow you to use your wallet, sign less messages, you authenticate into the application that way and then the application gets to act on your behalf for a dedicated session time.

It all kind of looks like this at the end of the day. We have a developer using a DID, probably Key DID, to create the definition of a data model. In this case it's a basic profile and he's got a schema outlined and it's just a basic JSON schema. Then we have two users, they've created DIDs most likely using CACAO or 3ID Connect, in which case they can write data to their own version, like a copy of this object for them. So they can write their individual records and everybody can read these individual records but only the controller can make changes. So the entire table of a user's data is now with a user rather than with the application and every application they go to could potentially read from that. What we get out of this is user agency. Now users own the data. It's not stored in an application-specific database and I need a DID to use as a controller to make any changes to that data. So applications can't change the data unless we're using CACAO and we're giving the application a very specific capability to be able to change that data.

That's the data composability piece. Let's talk about the network side of it. Traditional blockchain stuff, in a very high-level view, blockchains give us this single global execution environment, think the EVM. Every transaction that we want to take or that we want to carry out on Ethereum or every smart contract we want to execute goes through this EVM and it's kind of a single point of failure to some degree. There's a limited number of transactions EVM can process at any given time and the more congested it gets, the more expensive it is to perform one of these actions. And really what it's responsible for is taking the state of this single global ledger and publishing it out to everything that's participating in its network. So we can think of this as the traditional way and by understanding that, we can understand what Ceramic's doing.

So instead of there being one giant global ledger that needs to be broadcast out to every node that's participating in its network, each of our nodes, like when you run a Ceramic node, each individual node has its own individual execution environment. So there is no global ledger to keep track of and a node becomes responsible for validating all the commits and transactions that take place on the streams that it cares about. This allows us to scale horizontally. If we need more processing power, we just add a node to the network. So there is not a single point of execution on this, we can add more points of execution by adding nodes. This all works because it's kind of offline-first. So a Ceramic node, it does all of its stream transactions locally to itself and then it syncs those updates with the rest of the network. It does that over libp2p, so that's what all the communication between the nodes uses to take place as a pub-sub topic that they all kind of subscribe to. This is awesome because if for any reason you lose connection to the network, it doesn't mean your application that is connected to your node needs to stop working. It will work and then when the node comes back online, everybody gets that update available. And this works because everything is done in a global namespace.

Each node has some responsibilities and a very key characteristic. The first major responsibility of a node is to manage the state of a stream, like to keep track of that. So a node only keeps track of a stream that it cares about. It's called pinning. So if you look in the documentation, you're seeing stuff about pinning a stream, it just means you're telling a node like I want you to pay attention to this stream. That means if there's five nodes, they might not all care about the same streams. So one stream the state can be maintained by all five nodes or two of those nodes may update or keep the state up to date. This commit log storage, what they're responsible for is just really maintaining a local copy of all the commits that happen to the streams that they care about. We're working on something called persistence connectors and this will allow you to say hey, I need to back up that log. Very soon you'll be able to plug in like Filecoin or Arweave or something in there to do all your backups of your commit logs. The last two things that nodes are really responsible for is that query and response and like broadcasting those transactions. Again, when your client, like when your application says hey, can I get Matt's basic profile, the Ceramic node goes oh, let me see if I care about that stream, if I care about it then I return the stream and if I don't have it pinned, then I broadcast out that I need that and some other node on the network will deliver it to me and then I can show it in my application. Nodes can exist on multiple networks, not at the same time, but there's a lot of different networks that you can use when working with Ceramic. The first one is a local network, you can run a node locally in your terminal, it doesn't take a lot of resources to do so. We have the testnet which is called Clay. This is the preferred way for you guys to jump in and get started with Ceramic is just do your dev work on testnet.

We've provided a couple suites of tools for you. If you're going through our docs or landing on our GitHub or talking to us in Discord, you might be trying to use something called Self.ID or maybe Glaze or the Ceramic stuff. If you want your low-level interactions with Ceramic, like a raw HTTP client or you want to interact with TileDocument stream types directly, then you're going to use the core Ceramic package. If you want to abstract that out a little bit and use data models rather than one-off structures of data, then you're going to need to step up to this package called Glaze. Honestly, this is a really nice wrapper around the low-level stuff. Glaze is probably the easiest one of these to use. It offers a really nice blend between customizability and usability. If you really don't care about how any of it works because you're just writing an app that only needs to consume, then Self.ID might be the place for you. We got a bunch of like React hooks and everything already pre-built for you, so if you just wanted to plug that into an application and get reading data from the network, Self.ID is your answer. So this can kind of get confusing because you might see a blend of these things if you're doing some Google searches to try to figure out how to implement this.

The final thing I can throw at you is we got all these resources here for you. If you want to see what's up next for Ceramic, this roadmap is super important. Documentation is always your friend. And then I want to point out that we do have our Discord, we welcome you over there so come join us, we can answer your questions directly. We do everything in our power to spend a lot of time with the community and help you guys out. Last and certainly not least, we're hiring. So if you're interested, have data background, have DevOps background, just go look at the job postings that exist. We are still actively looking for team members. And that kind of wraps it up with about five minutes to go, so I will leave it up to you guys for questions.

Awesome, Matthew. Thank you so much for that wonderful overview. Yeah, gang, does anybody have any questions for Matt?

Sorry, my mind kind of wandered so I may have blanked on this, but you were mentioning that everything is global read and you have to do your own encryption to make the data private. Is that correct?

Yeah, we definitely don't suggest that you store sensitive stuff on Ceramic. There is no native encryption when it comes to that. However, you can tie in like Lit Protocol, super easily, to encrypt things prior to getting it stored on Ceramic if you really needed to. But as things stand right now, I would highly suggest against storing super sensitive things at the moment. Awesome, thanks.

Great question. Anybody else in the crowd have any questions? Okay, I guess I could ask mine. Are there any plans to introduce say a GraphQL component to be able to query some of the data on Ceramic?

Yeah, right now that is the biggest change that is taking place. So that's probably the number one thing that's going to stand out on the roadmap if you do take a look. We are working really hard to release very soon an entire revamp on how Ceramic queries happen to all be GraphQL. Data models are changing in a good way. Right now they're a little bit clunky and hard to work with unless you really understand the difference between build time and runtime with Ceramic. There's a big GraphQL change that's coming that's going to make data models a million times easier to use and to develop and to integrate with all because it's going to be native GraphQL. Awesome, can't wait to play with it. Yeah, me too.

We have about two-three minutes left here gang. Any final questions for Matt? Going once, going twice, sold. Well, all right gang. Again, Matthew, thank you so much for coming on and giving us that awesome overview of Ceramic. I know I'll probably be taking some time over the weekend to go play around with it. It's been on my list for a long time, as I'm sure the rest of the gang here is. And yeah, I just want to wish everybody a happy Friday, happy Saturday, happy weekend, whatever it may be for you. And I'll be posting a link to the attendees' POAPs in the developers' voice text and Matt, I'll forward you your special DevNTell speaker POAP as well. Excellent, thank you guys for having me and enjoy your weekend. All right gang, all the best. Cheers.