Local Data: Sqlite, LocalStorage, Session, Cookies and IndexDB

Welcome to Syntax. On this Monday Sanity treat, we're gonna be talking about local data. Locally, right now, Wes, I'm trying to get a snack in. I don't know if you've ever had these rice mochi balls from, Trader Joe's, but holy cow, they're my favorite thing in the whole world.

It takes great restraint for me not to eat them.

That's why whole bag. Yeah. Yeah. I don't keep them locally in my office usually because then I'll need to access them all the time.

But if, you need to access your data locally, we're gonna be talking about that in this episode. But what happens if you need to access your data locally and that data's not there and, you're not checking to see if it's there correctly? Well, you're gonna get a bug in your site. And oftentimes, you know, with this stuff, especially if we're talking about syncing engines, making sure the data is where it's supposed to be, Yeah. We can often get in these situations where we think we're in the happy path all the time. And, oftentimes, we might not even realize what that happy path is. I experience that all the time on my HabitPath app because I use local data and data syncing. And, man, it is hard to solve those bugs, but not with Sentry because Sentry lets me know who has what issues at any given point in my application, and I know what's happening beyond the happy path. I it works for me. All my data's local. It works. Nothing but sad paths in the in your century. Just a sad place to be. But You know what? The best part is too. You know, Courtney also uses Habit Path. And, just like most things, when other people use your stuff, it's really nice for them to be like, this is broken. And then I'll go check the Sentry, see what's going on, fix the problems. Head on over to sentry.i0forward/ syntax. Sign up again 2 months for free with the coupon code Sanity treat.

Sentry saves my bacon all the time. Wes, what's up, my man? How's it going? Hey. Let's talk about,

storing data locally. There's lots of ways you can store data in the browser, but let's talk about why you might wanna store data in the browser first. Right? Probably the first one is, user preferences and user settings.

You don't necessarily always have an account for somebody to store that in, or it doesn't make sense to always send that data back to the database every single time that it changes. Right? So one example I'll have is on my course platform, as people are watching the videos, every I forget how long. Every every time the video current time updates, which can be every 2 to 3 seconds, I'm putting that data in local storage. And same same with, playback rate, volume level, like, all those settings. Right? I'll stick that in local storage, and then, periodically, they'll they'll get synced to to the back end. But, it's nice to stick that in local storage or or in the browser when you need it immediately.

Yeah. Another thing too often that, like I think you see this a lot more native apps, which is something that people, might be doing on the web more and more these days is letting you use the thing before signing up for an account or even maybe paying for data syncing. That's like a thing where people pay to sync the data, and they say, oh, actually, I need to have this available on all my devices.

You can store that data. You can let the the people use the application even before signing up for an account or even choosing to sync it to a server somewhere. And that's just, like, straight up usage of your application.

If it is like a to do app, you're physically saving all the mutations and doing everything in the local data. And in that same regard, you might be saving that data locally from the server into local storage or into your IndexedDB or local storage or literally anything.

So that way, you can access it faster. Because if the data's already on the client, you don't need to go to the server to get it. So it's a way of accessing faster data as well. Sometimes it can it can even just be a privacy thing. Yes. So I built a

background removal tool.

And the thing with people's photos is they don't want to upload them to some random sketchy service that they're they're gonna have. Right? So this background removal tool will just do it entirely in the browser.

And the thing about that is if you refresh the page, they're gone. So you can stick those files and stick all the data related to that in the browser in in one of the places that we're gonna be talking about. In that way, when you come back and refresh the page or, or even, like I was on JS bench the other day writing some benchmark tests for JavaScript, and I, like, refreshed the page before I had, like, saved it.

Or, TypeScript Playground is another good example.

Before I've saved it, right, before it's in in their database, if they even save that in their database, then it's it's gone.

So it's nice to just persist it locally in the browser before you even have decided to send that to to a server.

Yeah. I think anytime, like you said, anytime that you would expect your application to, if you hit the refresh button for it to remain in the same state as when you left off, you're filling out a form, you're shopping at a store, you're you're you have a draft of a tweet, those types of things. Right? Anytime you want that state to persist is a perfect opportunity to save your users some headaches by putting that that information in local data. Let me tell you. There is nothing more frustrating than filling out a big ass form and having the page refresh for some reason and having to start over.

If you are doing job application things Wes they ask you to fill out every single one of your experiences or any of that stuff, you gotta have that data, storing locally or something so that way the user isn't losing that. That drives me absolutely crazy as either. Yeah.

And your users will delight be delighted for it. I'll tell you that. Another Node is auth tokens. You know, we don't often think about this, but anytime you log in pnpm anything, it's either storing it in a, cookie or local storage typically, whether it's a session token or a JWT, as an access token, those types of things. These are stored locally on your client.

Reason being is, anytime you have a request to that server, you need that information there at the initial request. That way, the server can be able to authenticate it JS you go or authorize it. Man, I don't know about you, but I I flip flop authorization and authentication so much. I'm I'm talking with the XeroSync folks all the time, and I'll use the word auth authentication, and they'll be like, you mean authorization. Right? I'm like, yes. I do. Gosh darn it again.

Flip flops to my dad for something. That I would use the word authorization.

Authentication is is where I'm at. Authorization is like a like a credit card thing to me. No. No. No. They're they're they're,

very distinct in web development or even any sort of login system. Right? And I'm gonna mess this up. I, authentication is logging in. Right? It's the process of logging in, and authorization is the process of making sure you have access to the things you have access to. That makes sense. Yeah. Totally. I always just goof it up, though, when I'm I'm trying to say it quickly.

Let's talk about where you can store your data. Alright. So

first one is cookies. Storing data in your user's browser, you can throw them in cookies. There are HTTP cookies, which can only be set and read via the server and server Wes, and then there are just regular cookies, which can be set and read via JavaScript as well.

Cookies have a limit to how much data you can stick in them. So, generally, cookies are only used for things like language settings, session IDs, just like identifiers in short little text settings for that specific user.

Yes. Next up is local storage. Local storage is often more used for single values like, a JWT, which could then be encoded,

decoded, those types of things, or a simple string value. Or you might use a local storage for preferences, anything that you need to just save. Again, typically, just a single string, save it, retrieve it, those types of deals. Yeah. Local storage is you can fit a lot in there, but the the thing about local storage is that it's it's only text. Right? It's key value store. So if you are trying to store something that is not text, you have to convert that to text. So if it's an object, you JSON stringify it. If it's an image, you you can store images in local storage by converting to base 64.

It's probably not the best bet, but it's Mhmm. It's a really simple API where you can simply just say local storage Scott set item, local storage dot get item. You can pull it out.

The only thing that that gets me sometimes is I'll pull something out of local storage and assume it's an object and assume you can JSON parse it, but it's not, not always the case. Right? Sometimes you accidentally set undefined and then try to JSON parse undefined, and you're in trouble. I'm almost always saving things in cookies over local storage,

for no particular reason other than I just do. But nowadays, if I'm storing data, like, beyond single values or whatever, I'm I'm using, IndexedDB typically.

There's also session storage. So session storage, just like local storage. However, if you close the browser, you close the tab, that's the end of the session, and it goes away. I I've never used session storage. I'm going to be straight up about that. I just haven't hit a situation where I needed session storage, but I I'm sure there's plenty of of these reasons to use it. I think a a good example of session storage would be

you were just talking about like, you're filling out a form or job application.

You wanna be able to refresh the page and that data to persist.

Session storage will persist in the same tab if you refresh the page. It'll not if you open up a secondary tab, and that's really important if you Yeah. Somebody is potentially on a shared computer. So you imagine a job application. Somebody goes to apply for a job.

They fill out the thing, maybe they don't finish it, and then they close the tab. If you put that in local storage, somebody else comes around and and opens up the same website, then there's a bit of a privacy issue because now you're sharing your your data between users of the same device, whereas session storage would as long as they close the tab, you will

you'll be fine. It's cleaned up. Yeah. That makes way too much sense. So, thank you for that.

Next up is IndexedDB, which, like a lot of people, I think, they say that it's hard to work with. And, honestly, I've used it just, you know, raw straight up API, and I didn't find it to be that awful.

It's it's not like the most fun API. It's not as simple as just, get set whatever. However Mhmm. I I do use Dexi quite a bit. Dexi is a a small library on top of IndexedDB, and it's basically just giving you a nice little API for creating, getting updating data. And, you know, you have to do a little bit of setup, but it's more like working with things that your typical, like, normal databases that you work with. And for the most part, I found Dexi to be Node to work in. And and pretty much anytime I need to store structured data, and objective data, I'm not picking cookies or local storage. I'm picking, IndexedDB,

obviously, because that's what it's made for. Query local storage. Right? You can simply just check if a key is there or Node, whereas IndexedDB is a database, and you can query it and filter and do what all the stuff you're used to. And I I picked up Dexi. Js for that background removal project I did, both to store files, which the browser also has file storage. We did a whole episode on that, but you can also just store, like, blobs of data in IndexedDB. Yeah. And the cool thing about it is it has, like, React hooks and and Svelte hooks so that as long as you save it to IndexedDB, the your application is reactive, and it's it's it's your whole state store.

And it's it's a really cool

project. I'm a big fan of it. I wanna introduce you to a library from, Kyle Simpson, of you don't know JavaScript fame. He has a library. He has, like, a kind of a a bunch of new packages called BYO JS that are all kind of cool. And one of these is a storage package called, storage, byojs.devforward/storage.

And the stated goal of this project is to have a simple key value storage API for all of the storage mechanisms.

So you have a basic git pnpm, and the only difference is is where you import those from. So you import it from IDB, for IndexedDB, from local storage, your session storage cookie, o p f s, which is, the private file system.

And so, yeah, basically, it just gives you the same API to work within all of this stuff. And I thought this was a really neat API that you don't really hear a lot of people talking about. That's cool. It seems like it's only key value, though. Right? So you wouldn't be able to key value. To query. You know what I really like about some key value stores? Deno's key value does this, is,

you can set an array of keys.

So you could say, like, user and user 123.

And that way, you'd be able to query all the users or just query user 123.

Yeah. I'm I'm sure more key value databases are like that. Sure. But that's Deno's key value store is is 1. It's built on top of SQLite. Oh,

there's also a host of really, wild new options that, people are exploring in this area right now. As we start to get, like, more down this rabbit hole of using local data more frequently in applications, People are wondering, what can we do that is beyond IndexedDB? So a couple of these things are are using WASM. So one of them is SQLite via WASM, is increasingly more of an interesting option.

Although, you know, who said, I I believe there is, like, a penalty, essentially, like an initial download Sanity, where the 1st time you you load it up, there is a pretty large hit to use something like this. You have to download so the way

what it's doing is it's literally running SQLite in the browser. Mhmm. So you have to download the entire binary to run SQLite via WASM, and I'm curious how large that actually is.

Oh, holy smokes. Four 10 KBs? How big? 410 kilobytes.

Yeah. 558 k kilobytes, by a 1.14 megs if you want the async version. I'm not sure what that means, but that's much smaller than I would have thought to be able to run SQLite in the browser via WebAssembly. I I think there's also, like, going between the browser and WebAssembly, there's, like, a cost you pay for going across that bridge.

But, we had Johannes Schickling on, and he's one of the devs behind, WebAssembly SQLite.

And he knows the stuff, so I kinda kinda trust him with this. Yeah. There's also, another one, pg lite, which is Postgres

in WASM.

And, that's being created by the folks over at Electric SQL, which is one of these cool new local first platforms.

And I think Drizzle even has support for pg lite now. Man.

Yeah.

So, man,

That's what I want. I don't wanna learn IndexedDB.

Oh, come on. Dexi I already know. Well, I, like, I I can learn it, but you know what I mean? Like, I wanna be able to just use the same database queries everywhere. Totally. And especially even if you're sharing schemas.

ESLint drizz schemas.

Drizzle does have they even have a whole page, which we'll link to, which is, Drizzle and PG lite, how how you can use the 2 of them together, again, to do your drizzle DB selects and all that stuff, directly from PG light, which you know, the big question I have with some of these still is that because I haven't used any of these more intense ones JS is migrations and how do you handle that kind of stuff in a client side level? I guess you're running the migrations on boot up or something. Yeah? I don't know. Yeah. Probably. I I bet it's you try to keep them

a little bit more flat.

So I I guess you don't have to, but yeah. That's a a great question. You'd have to run it in the in the user's browser because that's

that's where it exists. Yes. So that's all great. Right? You can store data in any of those numbers of ways. So, you know and and primarily Wes you're doing this stuff, especially in applications that are loading this data, you can think of a lot of the times that this data, since it's living on the client, is a use case for not waiting for a long server response.

Because if the data JS living on the client, right, you don't necessarily always wanna wait for the server to hit the database, to do all the database stuff on the server before returning HTML.

You wanna get the client side HTML and JavaScript as fast as physically possible, then the data's there. It doesn't take any time to read it from the local database. It's infinitesimally small to load from a local database compared to, loading from a database on a server and building the HTML. So you're doing a lot more client side rendering, which is one of the reasons why I've been talking about client side rendering more. It's that you you're sending the information to the client. The client is then grabbing the data, loading it up, getting you all ready to go. And then, occasionally, if you need it in the background, you could be fetching the data from the database. That database process on your your server side database could be kicked off right away. So that way, by the time that that comes back, you you initially have data. You initially have a loading thing that's beyond a blank skeleton screen, and then the rest of the data can be filtered in in any sort of way. Now you could have, like, a little syncing message to let people know that data's coming in, or you could just pop it in, which might not be a a great solution if you're you're getting your UI popping all over the place. Or you could have a button that says, you know, there's new data refresh, those types of things. But in reality, I think what it comes down to is that once you get into syncing data to a server, you're gonna wanna do it a couple of ways. Here's how I've done it specifically with storing things in Dexi.

You store things in Dexi, and it's stored primarily in your your client side, database. When you make a change, you push that change up to the server. When you load your application, again, you fire off a message that gets then pushed into your, your local database. That database updates the UI, and then that information is in your your local database. You're just trying to keep those 2 in sync. At a basic level, for an application that 1 person is using, you're sending a message off on 1 device, whatever. That's very simple.

We talked about how it can get really complex with conflict resolution.

But the way these things will work, I believe, in the future, specifically, a platform like XeroSync does this really well. It stores your information locally, and it handles that sync process.

And you're only ever writing 1 Node set of queries or you're writing 1 set of, mutations all from the client side,

and it just knows what to do. It's sending messages back and forth. You're not being like, oh, you're not writing the glue code yourself to sync it and query from the right place because that sounds like a a really hard thing to to do yourself.

Yes. And one of the cooler things that many of these new libraries are doing is that instead of, like, the naive way I've done it the naive way a whole bunch of times. If you need this sync, by the way, there's plenty of application where you just save things locally, and that's it.

That that's definitely super valid, so you don't always need a sync. The the naive way is just to send all the information every single time, and that's too much information to send every single time. But if you're not dealing with a ton of data, not a huge deal. But the more and more data you add to the system, the larger and larger those sync messages are going to be. So a lot of the things that the the new platforms are doing is would be, like, patch messages.

So it's only sending the data that's changed. So it's asking for what's changed since this version. Oh, this has changed. That way, that sync message becomes smaller every sing Scott every single time, but it's smaller than it would be if you were sending all the data, meaning that that trip becomes much less, meaning that you don't have that crazy pop in or anything like that because it's not sending a massive bundle of data that then needs to get updated into your local database.

So that is how you get it done with local data. Local data, I think, JS, widely underused, but I think it could be, you know, greatly more used in terms of, like we mentioned, keeping persistent state for things like forms, surveys, things that would piss the user off if, if they lost their data in between save or refresh.

Can we rattle through a couple of the the possible options for this? I know we did a whole show on it, but, the author of tiny Bos, when you were explaining it to me, I was like, I don't really understand where this JS, and he said he explicitly added a diagram to the docs for us.

And it's exactly what you're just explaining. Right? Like, it's it's it's local, but you can also synchronize.

Like, it's a local database ish, but then it will also synchronize, and it can also persist to, like, a Cloudflare or something like that.

Yes.

And, I I believe James was on the Local First FM, Local First podcast, and gave a really great explanation about how tiny base came to be. So that's well worth your time to listen to. But, yeah, tiny base is an option here.

RepliCache, I I've talked about Replicash before. It is tough. Replicash is becoming 0 sync, which, in my opinion, is way, way, way easier than Replicash.

PowerSync, man, PowerSync is one of these ones that handles the syncing. But every single time I look at this platform, again, I I there's too many graphs and diagrams to the point where I'm like, alright. What are you doing for me here? There's yeah. It's tough.

Rx DB is a cool one as an option for again, the ones that make this the easiest are the ones that kind of combine the 2.

Instant DB is a closed sourced Firebase competitor that does local storage by default. That's pretty amazing.

Electric SQL has kind of undergone some changes lately Wes they're they're kind of, like, they're changing somewhat significantly. I I know it was, like, become version 2 now, and I haven't checked in on it, since they announced that it's going to to change a little bit. But Electric SQL, like we mentioned earlier in this episode with the, PG lite has definitely been one of the bigger, innovators in this space along with the RepliCash folks and many of these platforms. Honestly, tiny base as well, for sure. There are a lot of these options that we've talked about in here. Yeah. I'm gonna paste in local first web dot dev. It has a whole section on different software for these things.

Yeah. I I think there's a lot of different opportunity here. My the most exciting one to me is Xero because I've been, it's kind of the thing I've been, like, really hoping and wishing for. Working with. Yeah. And I've been using it for a little bit. Granted, there could be other ones. Like, Instant DB is a really cool option, but it isn't like I can't spin up an Instant DB server myself. And and that's the only bummer about that. Right? You're you're tapping into their ecosystem to to get going. But when I wrote an app with the Instant DB, it was probably the closest thing I had to get 0 to 100, in the space because it comes with, auth and MagicLink auth and all this stuff. Like, you could have it up and running in no time. You're just not able to run it yourself at the moment.

Awesome. Well, that's local data. Hopefully, you learned a thing or 2. Let us know if you have anything to add. Tweet us, Scott, email us, whatever you got. We're at syntax f m. And,

that's it. Yeah. And let us know what you're using local data for. If there's anything that we didn't mention in this episode that you think is a good use case for storing local data, Leave a comment below, YouTube, Spotify, wherever you're at.

Alright. Peace.