Unifying AI Model Access with LLM Gateway

GM GM. Welcome to what's going to be another fantastic episode of DevNTell. So if you didn't know, DevNTell is a 30-minute podcast held every week, allowing founders, hackers, and anyone in between an opportunity to come on the platform and showcase their product. And today I'm stoked to welcome Ismail and Luca, who are the co-founders of LLM Gateway. So if you didn't know, LLM Gateway is an open-source API gateway for large language models, LLMs, acting as a middleware between your applications and various LLM providers. So if you stick around for today's episode, you'll see Ismail and Luca give us a talk about LLM Gateway, how and why it was built, and how you can use it today. All right, let's get into it.

[Music Playback] GM GM. Welcome to the show Ismail and Luca. Pleasure to have you on, guys.

Hello, everyone.

Hello, and thanks for having us.

Yeah, for sure, for sure. And thank you for taking the time out of your day to come on the podcast today and give us a rundown of LLM Gateway. I've been poking around at it the last couple of days. It looks really cool and I can't actually believe it's open-source, which is awesome. But before we kind of get into what it is, did you guys want to just give a brief introduction about yourself?

Yeah, sure. You can start, Luca.

All right. So I've been into programming and software development since I was a kid, basically, and just building things and working for software companies. And in the recent years, I've been working on lots of AI stuff because everyone wants to do it, it's super interesting. And we've noticed some things which led us eventually into building the LLM Gateway.

Awesome, awesome. Ismail?

Yeah, on my side I started initially as a graphic UI/UX designer, and then decided to get into web development. At some point in late 2017, I started learning about neural networks. There is an article on my blog where I created a perceptron from scratch with help from my dad because he is a mathematician teacher. And then decided to stick with web development because it was hard to finance my living, but it was easier with web development. So I've been working for the last eight years now, mostly on the front-end side. I met Luca initially in 2016. That's when we knew each other. He was one of my first clients, which is funny, but he was more than a client because because of him I got to learn Node.js, Git, GitHub, and lots of interesting stuff. He made me switch from PHP to Node.js.

Nice. That's quite the switch, and good on you, Luca. I think you saved Ismail. And I guess after you guys met, did you guys kind of start various other companies before doing LLM Gateway, or was it kind of like a back-and-forth kind of thing until something kind of spurred you on to start the company?

So for us, it was mostly just some side projects, but nothing too serious. And for me, I'm very technical, so I'd like to work for like as a DevOps is my main role right now. So LLM Gateway is still very new and more like a side thing for us, but we are kind of working on it right now to transform it into a full-time real company.

Cool. And I guess was there any particular thing or reason that kind of inspired you to start the project?

It was mainly because we were building applications with AI and just using it on small tools, or with Cursor or Cloud Code, you know? And we realized that there were some things which are not optimal when you have your own when you have an project using AI and you want to support some AI models, either for yourself as an internal tool or for your end users. But you realize very quickly that switching to different provider if you also want to offer Anthropic models is like they have a different API and you need to adapt it, or you need to use some tools to adapt it, but then you still need to manage API keys and setup billing and so on. And so I think these are the two at the same time the two major reasons to to use an to use a gateway and the reason we built it. And the main reason is that some of these exist already like OpenRouter, which is the most popular one, they are very big, but it's they just have a cloud version and you cannot self-host it and it's not open-source either. So it's a nice product. I use it myself as well, but it's very limited and not much you can do about the fact that it's closed. Kind of funny because it's called OpenRouter.

Yeah, exactly. I was just going to say. And yeah, I guess this is a good segue. Like maybe what's the elevator pitch of what LLM Gateway is? What is it, why was it built?

So an easy answer is like it's an alternative, an open-source alternative to OpenRouter. A better description would be it's a unified API to access all different models and providers. But if you compare to OpenRouter, we offer more than a unified API. We focus as well on observability and analytics. So our dashboard looks way better than what OpenRouter offers. And Luca can give more details on this.

Yeah, I think maybe this for later because there's some lots of stuff around this but yeah, lots of things you can do.

Yeah, for sure. And it's really cool that you can actually self-host this. But before we get into that, I guess Luca, did you want to kind of take us on a tour of LLM Gateway?

Sure. I will share my screen and give a quick overview of it. One sec. So you can basically use go to LLMgateway.io. And I'm doing it self-hosted now, so you can also just clone it to your machine if you prefer to host it yourself. Let me just refresh this. So the first time you will see this thing where you sign up, which helps you to get started to create an API key. So you can create your API key here. And in the cloud version, when you do it on the production domain, then in order to get started, you will have to buy some credits because obviously we will have the AI costs if you will use it. And we will also have a pro plan which comes with some features I can share that later. But basically that's that's already it and you will see nothing is going on here now. But with this API key, you can go to your console or you can go to whatever app you already have and you paste this API key. And in your app you would also change the base URL to api.llmgateway.io. This is all in the docs in self-hosting. So this is just a local URL because it is running locally on my machine right now. And we will run this and you will see we have asked meaning of life and it will give us a response of the content actually it's quite long. You can also see the and you can choose the you can set the model here in this instance we used the new open source model from OpenAI. You can see it was even reasoning. And when we go back to the dashboard and refresh here, you see that you have now the log and you can see how much this call costs you, how many tokens it was. And we can even have all the details into how many tokens it was, the cost and so forth, some metadata and basically also the context. You can also choose to not log the actual contents in case you cannot do it for legal reasons if you don't want to log the user's content, but this is very useful if it's your own app and you want to debug which like something went wrong or if the responses make sense to evaluate if you can maybe improve your prompt.

And yeah, so this doesn't look very interesting now, but you can have insights into how much all of this cost. Maybe let me show you the production version of this actually, because it will look a bit more interesting. So you will gain insights into not just the individual requests, but also at a glance on how how many what models you used and how many tokens all of these used and effectively the cost. So we can switch to cost here and we can see oh, actually sonar, so this is a Perplexity API. It was actually quite expensive, you know, five cents on this day. And yeah, you can see all this stuff and volume you can see which models you use or this is more like for your end users. For example, we have Noemi.ai here, which is built on LLM Gateway and you can just basically search for all the models here. So they are all provided by LLM Gateway and you have to just, you know, add one line and you're pretty much done, you know. And even though that all of these models have a different API and different like different details and for Google it works completely different, but you basically have access to everything here. You can do Anthropic, all the cloud models, you know. And you have insights into the error rate in case something went wrong. You can do caching which helps you save costs.

Yeah, just to add regarding provide provider keys, you're required to be on the pro plan in order to bring your own API keys. But this is for the cloud version.

Yeah. So basically on the self-hosting, you can do everything without paying anything. You just need to set up the keys yourself, of course. But on the cloud, the easiest way to get started is buying credits and optionally to bring your keys we have the pro plan, which also comes with a lot of other features. I think maybe we can talk about this later, but up to you.

And then for the self-hosted version, are you also able to plug in local models as well? Like if you download from Hugging Face or Ollama? Are you able to plug those in as well?

So right now this is not our focus because we wanted to get the we wanted to focus on making the quick start very easy, so the hosted version so that you can access to all the cloud models. But technically you can go to provider keys, you add provider key and then you can select custom. Where is it here, custom? And then basically you can set up anything you want. It just needs to be OpenAI compatible, which the Ollama models, for example, are. So you can set the base URL to your local URL and then you can use even the local ones. I think we will also add Ollama here just to make it a bit more easy. But yeah, you can use local models if you have it running locally.

Nice. And if you have multiple providers that provide you the same model, what kind of goes into the magic of LLM Gateway choosing or load balancing which provider it uses?

Yeah, very good question actually. So you can see on the models page, I'm still you still have my screen showing, right? Okay. So you can see that for example, for some other open source models like Llama, we support different providers. So I'm you can go to the models page where we list all the models and this is constantly expanding. When we look for Llama, for example, this one is on Nebius, Inference, and Together AI. So you can copy this model ID and we can basically try this out here even. Let's say we want to use this one and then okay, this one does not support reasoning, so we have to remove that. I really hope it works. But anyway, you can use this. And what our algorithm does is that it uses the cheapest model available, cheapest provider available, which means that you basically get the best price. So you can see that this is priced very differently and we use the average of input-output price for now. I think there's like lots to do later because you can kind of like check even for like a small payload input payload, it would be maybe better to use a different provider which has very low input price on this model for example. So yeah, right now if you choose a model, we just choose the cheapest one for you. There's lots more coming in terms of that we also monitor the service in case one is down so it would fall back to the other, which is better than which is though an more expensive price is better than having none at all. But this is all this will be all customizable, but this is also something we still need to think more about and work on.

Awesome, awesome. Yeah, this looks fantastic. Sorry if I interrupted there, I think you wanted to go through the GitHub repo perhaps.

Maybe the GitHub repository is in itself not interesting to show. I think our idea is here that all of this what you see here, even the dashboard and the login and so forth and all of the credit system, is all open source. So it's not just that our gateway is open source, for example, which some people do and is also valuable, but really you can clone the whole thing and you have the same experience on your machine or on your own domain if you want than what you see here, which I think is very cool. And so all of this is in the GitHub repository. I will not show the details here because I don't think it's so interesting, but it's all it's all there. And I think the docs are worth to mention that you can go to self-host and you can run this single like this is just one line, you know, you copy-paste it and you can run it locally and you can just add your keys if you want and yeah. And here we have also more advanced getting started sections where you can run it on a server with Docker Compose, for example, because you know you would want to have your database, we use Postgres, so you would have that separate of course in a separate container and you can set all the environment variables with your keys and stuff and then you point your domain to it and it's pretty much done. Also something we are expanding continuously to have some more examples on how to get started with a specific provider or like on a VPS or on whatever the cloud platforms are called.

I think this is like really this is really fantastic, the fact that you guys actually decided to open source all of this. Like you didn't have to do that but you did it anyway, exactly. I'm continuously impressed by the open source AI community and just like the sheer willingness to kind of bring that to somewhat parity of like the closed consumer model versions of things, like models and and apps and services like LLM Gateway. I guess will like if people do want to self-host, like you showed it all it took was like a simple Docker command. But in terms of like resourcing, like how much how much do you think it would cost for a person to host this on a cloud instance?

I mean if you do it on a VPS, which is always the cheapest way to host things, you can basically spend, I don't know, three or maybe five dollars a month for a simple VPS which has like, you know, 1 GB of RAM or something and it will totally work fine. It doesn't need much resources, especially if you don't have much traffic on it. So I think really the cheapest way is you buy a three dollar VPS, you install it there and you're good, you know. There are some other services which host it for you. So then you can also it also creates a database for you which has backups and so on, but this will be more expensive. I'm not sure about how much.

I mean that's still cheaper than a cup of coffee, so that's pretty good. And one thing I want to also mention, I'm not sure if you can still see my screen.

Yeah, we can see it. Okay. Okay. So I just want to mention this because I kind of forgot about it before is if you self-host it or if you use the cloud version, doesn't matter. We have also this getting started where you can just try out with different integrations. And then there's the AI SDK from Vercel, which is kind of a standard by now I would say for chat apps, which makes it very easy to just switch to LLM Gateway. So you can if you already use OpenAI, you can just set the base URL, maybe I can zoom in here actually, you just set the base URL and you, you know, make your account and you provide LLM Gateway API key and then it's done. And then now all of a sudden you can use any model you want. It also works for this one and Python and yeah.

Yeah, but we also have our own Vercel AI SDK provider, which is also open source. So you could use that as well. We'll be adding it to the docs after after this podcast. So you can also use the LLM Gateway AI SDK provider and use the AI NPM module to just use it and you can use any model you want with one integration.

Right on. So it's just a matter of grabbing an API key and switching some configs and you're up and running. That's pretty much it then. Excellent. And you get all the observability with it which I think is cool. And we support all the common features like tool calls, reasoning.

Do you guys think you'll ever expand to basicly beyond text-to-text models? Like maybe text-to-image or image-to-text? Do you see that in the horizon or you think just sticking to text-to-text that niche is the way to go?

Definitely we will support it eventually. That is the plan for me at least. It's just the matter of managing the workload we already have.

Fair enough, fair enough. Yeah, we can actually talk briefly about the roadmap, both short term and long term. For the short term we're currently working on revamping the playground. So we do have a chat playground which we use and we recommend people to use to to test out the gateway instantly. So we're working on a V2 which will be using the new Vercel AI SDK version 5 with probably AI elements from Vercel. We'll be adding dynamic pricing for providers models which price like depending on context size. The failover feature that Luca briefly talked talked about. So if there is an error with the specific model or provider, we will fall back to the same model in a different provider. We'll be offering some models free of charge by collaborating with specific providers. OpenRouter already does this with some providers. And we're currently really working on providing guidelines, tutorials how to self-host the gateway and how to use it with the Vercel AI SDK with Cursor, etc.

For the long term, we're working on more observability, like public statistics, private insights for fine-grained querying, filtering and segmenting traces. We're also working on smart routing. So if if you're the heavy user, like power user, we'll use your data to to train our algorithm to decide which provider and model fits your use case. Yeah. And the last thing I want to talk about regarding the long term roadmap is if we secure a good amount of funding in the long term, we will have our own servers and infra to deploy all these open source models like Llama 2 and DeepSeek. And we will be offering ourselves as a provider alongside the others and we will always fall back to our providers in case the the other ones fail over.

Awesome. Sounds like you got quite the roadmap there. And yeah, like all of that sounds really amazing. And I guess for the developers that are watching today, since the project is open source, is there a open issues or is there a list of issues that people can help out with today if they're interested in contributing to the project or how would you recommend people get into into the project?

Yeah. So we have some people who added some features already. So I think the best way, I mean if there's someone actually interested, there's the best way to open an issue first and then we can kind of agree that it makes sense. But there's also some issues with the help wanted tag, which people can look into and which would be highly appreciated to to implement or fix those. And but yeah, we are very open to adding new features especially because we think that there is still lots to be added. So opening an issue and then opening pull request will probably get your PR merged if anyone's interested. So we're very open for that.

Sweet. Awesome. And yeah, we will list the the repo in the description below. So if you're somebody who wants to get into the AI space with an open source contribution, this is a really fantastic project to kind of dip your toes into. And like we mentioned, open source AI is really really an interesting rabbit hole and lots of great projects and models are kind of coming out of this space and becoming very competitive with the closed source versions of themselves. I guess like last week, for example, we saw GPT-5 come out, the GPT open source models, we got Opus 4.1, and we got Kwen image model as well. I guess with all that after the dust has settled a bit, like what are your guys's thoughts on all the models that kind of came out last week and the open source model landscape in general?

So before we answer that question, I think just the fact that this is something people talk about a lot is proof that an LLM Gateway is so valuable because you don't have to care about who launches what, you know. It's just the moment it's out there, we add it to our code base as well and anyone can use it. And users who use the cloud version, they don't have to worry about API keys and limits. And and for example for the which model was it? I for GPT-5, it required some verification on OpenAI. So a lot of people actually kind of kind of use it. But we had it very quickly available as soon as they made it available basically. Awesome. And for the models itself, if you want to say something about that, Ismail, but.

Yeah. So like what we built would not be possible if there was no like Cursor and these different LLM providers. Because two people out of nowhere come and build this like in the matter of like two weeks or three, would take longer than that. So we we use AI heavily. If you go to our GitHub repo, you will see some PRs written by AI, reviewed by PR and merged by by AI. For so we use heavily Devin AI initially. It used to raise good PRs and then we use CodeRabbit for reviews. And we use Cloud Code for marketing strategy, growth hacking, etc. So we use like AI heavily, but in a smart way so that we are still reviewing the generated code or generated strategies that it tells us. And it's good that OpenAI finally decided to be to stand for for its name and came up with these open weight models. Personally, I've been using Claude on Cursor. It's I switched to GPT-5 when it came out, but it didn't it didn't impress me. So I I reverted back to Claude. Yeah, and Luca, I don't know which model are you using.

I do I do want to give a shout out to Cloud Code. So I don't mean the model itself, which is also the best one, I think for coding, still. But the Cloud Code CLI. So if anyone has never tried, I think really it's the best one for me because it's so customizable and it's just so smart. I feel like for me it was way better than what Cursor does these days. So Cloud Code, which comes with the Cloud models basically. Really good tool. I use this a lot for LLM Gateway as well. So this is actually a nice if you use other tools such as Cursor, you could also try the all the Chinese models, for example, and with LLM Gateway it's so easy to switch and then if it doesn't work out, you just switch back.

Yeah, exactly. And you guys you guys touched on it. But yeah, I still think there isn't like a one model fits all model yet, or I don't know if there ever will be. But it seems like there's like models that fit particular niches very well, like Claude is I think still the king of like the coding side. GPT is very good for like writing and like the marketing and SEO optimization and all that. And yeah, I think personally I believe that that trend is going to kind of continue and unless Grok kind of blows everything away, but we're still kind of waiting on the open source version of that. But yeah, it's a very very interesting time to be alive to say the least.

Yeah. And I forgot to mention V0. So we used V0 initially heavily to generate all the user interfaces on LLM Gateway. So shout out to the team behind V0 for it's the best out there for building like user interfaces. It's on my bucket list of things to play with. But yeah, definitely they're bringing a lot of good things to the table as well.

And I guess as we're closing to time, what's the best way for people to keep tabs on all the latest ongoings with LLM Gateway? Is there a Discord people can join, is there a X handle or something people can follow?

Yeah, we have Twitter, X. That's we post almost all updates there I think, so that's probably the best way. For more in-depth conversations or to be notified, you can join the Discord and also ask questions we help everyone who who needs some help getting started or if you discover a bug, happy to fix it. And we have a change log on our website in case you just want to have a look on what we published recently and there's also lots of stuff coming for next week, very interesting features around observability where you can have like fine-grained insights into how your users use your own apps and and like how the cost spreads to different users or which users which use model and so forth. So lots of upcoming announcements there. So definitely worth to check or follow.

Excellent. Excellent. Yeah, there you go gang. Definitely give LLM Gateway a follow on X. It should be linked in the description below. And with that, Luca, Ismail, thank you so much for coming on the podcast today.

I do want sorry, I do want to mention one more thing. So the easiest way to get started is to go to LLMgateway.io and you just make an account and buy five dollar, ten dollar worth of credits, just try it out. If you don't want to pay, you can run Docker on your local machine for free, you put your own API keys, you can run it locally and try out. And we do have a pro plan for which we have a code which is DevNTell. If you can Ismail, if you can put it in the chat please, which gives you lifetime 50 percent off of our pro plan which as I said will have lots of features that we will not have in the free version with just the credits. So you have in the free version you have access to all the credits, all the models sorry all the providers all the models. But in the pro plan we plan to offer all these advanced observability which not everyone may need, but if you do need it as a lot of stuff you can do and you will have 50 percent off lifetime even if he bumped the pricing for new customers. Wow. That's so awesome. Well there you go gang. And how how long can people use that promo code for, or how long is it good for? Next two weeks I think.

All right. There you go. There you go. What else like only on DevNTell. There you go. Yeah and yeah I'd like to thank Kim for reaching out on on Twitter. He was the the guy who part of Developer DAO who suggested that we come to the podcast and talk about LLM Gateway. Appreciate all the support and shout out to our first customer who bought credits, Kino, he's based in Morocco as well. And to Motek who's our our overpowered user. If you go to his Twitter account follow him he's building an open source chat playground to support multi-model multi-provider. Awesome. Do we have the link for for that?

Yep, I got it. I'll I'll share it.

Yeah, we will share all these goodies and yeah. And yeah there you go. What an action packed episode. Thank you again so much guys. Really appreciate you taking the time out of your day. And we will link all the all the things that were mentioned at the end of the episode here, including the promo code and Discord and X accounts as well. And with that, just want to wish everybody a very happy Wednesday, happy rest of the week to you and we will catch you back here on Friday for another great episode of DevNTell. Till then, have a good one. Thanks very much for having us. My pleasure. All right, all. Have a good one. Bye. Bye.