Next Level Web APIs. Bluetooth, File Access, Thomas Steiner - Project Fugu

Welcome to Syntax. Today, we've got a good one on good one for you. We have Thomas Steiner on who is a developer relations engineer at Google, And he's on the Google Chrome team.

And he's here to talk to us today about project Fugu, which is this sort of effort To I'll let him explain it. But, essentially, there's a lot of really cool stuff in the browser from Bluetooth.

I'll tell you the story of how this podcast came about is I have this little label printer that I bought on Amazon. It's called the Nimbot D110, and it is really cool because you can use your phone to type and it'll print labels. And I thought, That's cool. I had it for about a day, and then I go, of course, can I hack it? You know? Like like, how does this thing work? I always try to dip into that. So, I spent an evening dipping into the web Bluetooth API, and I was amazed that, a, the web Bluetooth API is The best API for working with Bluetooth, full stop. I was looking at, like, the Python API for working with Bluetooth. That sucks. It's so simple with with using JavaScript.

And, b, I was amazed that I was able to connect to it and see all of its capabilities. And I didn't get it to print a label just yet because there's some Bluetooth devices don't want you to they want you to use their app, and some of them wanna be open. But, Thomas sent me a message. He's like, love seeing this, like, because that's what that's what you use. So welcome, Thomas. Thanks for coming on.

Tell us who you are and and what you do. Sure. Yeah. Thanks for having me.

So, yeah, as you said, I'm part of the Chrome team at Google. I'm focused on what we call project FUGO And also responsible for web assembly.

Yeah. So I was listening to this episode with, Pharos. He did a couple days ago weeks ago maybe.

I hear project food, and I was like, wait. That's something that I'm working on and that I am very interested in.

And, yeah, then I sort of just reached out on Twitter, sent you a DM.

And, yeah, luckily, you were welcoming enough of my self invitation

to be part of the show and, yeah, chat about project Fugu. So, yeah, thanks for having me, and, glad to be here. Yeah. You're You're welcome. Fugu is one of those things that comes up from time to time on this show along with themes things like, you know, Houdini Wes we're we're saying, like, alright. These are these big, Big overarching plans. Right? I guess from an outsider's perspective, they're they always seem very ambitious.

What is the the goal overall goal of Project Fugu? CSS, essentially, what our goal is,

if you have an app idea, You should be able to realize this app idea on the web. And, typically, when it comes to app ideas, it's like, yeah, straightforward, hack, hack, hack, hack. Oh, How do how do I do that? And then, in the worst case, you hit this, oh, there's an API gap, as we call it. There's just no way to do, whatever.

Get a x signal onto y.

So it's like, oh, no. I can't realize this app idea on the web. I need to go native or I need to go electron Or whatever, you name it. So, Project FUBU, is essentially this vision that you should be able to Do and realize all your app ideas on the web. And, if that means, inventing new web APIs, then sure. We accept the challenge. We We try to do it. We try to, standardize it. And, yeah, in a nutshell, this is what project Fugu is about. That's that's awesome. And I'm gonna recommend that you go to fugudashtracker.web.app.

I'll I'll link it up. And there's a couple of websites that Scott of detail it, but just scrolling through the list of web APIs that are either shipped In some browsers or are sort of being thought about is is so cool because, like, at the very basics, I posted a screenshot of me working on the Bluetooth thing, and everyone was like, you can access Bluetooth in the browser? And it's like, well, yeah. In in Chrome, they have it, and they've they've had it for for quite a while. And, there's also a a web serial port API, which JS, on the electronics show, I talked about how I bought these little Wes 30 twos.

And the process for flashing them JS plug it in via USB and visit a website, and it will flash your hardware from it. And, my Bose headphones, I can I can update the firmware on my Bose headphones via a web browser? And, like, that experience Like, let alone people wanna build desktop apps with with Wes tech. That's probably a big part of this as well. But just that experience of people don't have to download some Sketchy ass Windows only EXE to and run it to to be able to update your software is such a nice experience. We think so too. Yeah. So there's a a couple of

new APIs that enable this, experiences like this and others.

We had the Vercel bug buds recently, that now have, this app that allows you to configure the equalizer and stuff.

Because, yeah, as I said, people feel bad if they have to give up whatever their location history just to Change whatever equalizer setting in their headphones.

And the web just makes sure that these kind of things, yeah, are very Visible, so, you don't really need to, accept this permission if you don't want to.

They can obviously still program it in a way, that requires, giving giving up your location history. But, like, in the worst case, there's just no way to, track background location on the Wes. So you're not becoming an active target for whoever, tracks this data.

So, yeah, just going web only for these kind of things makes a lot of sense. It's, it's more secure. It's easier to roll out on many devices. You don't have to have, whatever, Windows 11 to patch your Bose headphones, whatever.

So, yeah, it's just way more accessible for a lot of people. And, a lot of companies also realize that this is actually, making a ton of sense.

We just got out of the Google IO season, so we had a bunch of events all around the planet. And, What we showed at Google IO connect was, Lego. And, Lego has this really, really cool set of, well, it's kind of It's not a toy, but, it somewhat is. So it's an educational toy, I guess. And, they just they just realized, all these schools and universities, they have typically very low powered Chromebooks. They have relatively simple Windows laptops.

They have Especially a very mixed architecture of Vercel.

And, for them, it was one of the best solutions, that, yeah, was on the market to just say, hey. Let's go, web with our application.

It's essentially Something like a scratch editor where you can, you know, move around, blocks and then flash them onto the device so that, you can then run your LEGO model, program the hub, and so on. And, yeah, we just showed that, LEGO uses this For their, yeah, production web applications today shipped with, whatever LEGO kicked you by. Wow. Is is that called the LEGO Mindstorm? Mindstorm was before that. So the new, generation is called, Spike. Okay. LEGO SPIKE Education.

That's cool. Speaking on, like, companies getting on board, I think one of the bigger companies that people criticize for maybe not getting on board with APIs that could maybe reduce the influence of apps would be Apple.

Do you see the Apple ecosystem taking these things seriously in in your and, like, if I I guess maybe speak a little bit on, like, what's the likelihood that all of this stuff comes to all the brands?

Yeah. So that's the the elephant in the room.

So Wes USB, web Bluetooth are some of the oldest APIs that we're talking about in this context. So They were launched on Chrome in 2015 ish.

And, yeah, to your question, how serious do they take them? I would say relatively Sirius.

So, there's W3C TPAK, a yearly event where all the browser vendors and, industry and researchers and whoever come together.

And, we discuss hardware APIs regularly there.

There's a bunch of browser vendor representatives from Mozilla, from WebKit, from Google, obviously.

And, we talk about these APIs. And, it's typically, a bunch of engineers with Node that want to, yeah, bring these needs forward on behalf of the users.

And, what we want is we want to Get other browser vendors to see that these use cases exist and that the, use cases need to be addressed more safely on the web rather than On native applications, as we said before.

So, we discuss these APIs regularly at events like TPACK.

And, All of these APIs have very, very different threat models. So something something Bluetooth JS a different threat model from something web USB, From something Vercel, from something, HID or head.

So all of these discussions need to be very, very detailed, which, ironically, Pnpm is kind of the Bos, menu for because Mhmm. You bring a bunch of people into a room, They discussed. Half of the people don't really understand the problem just because it's a very specific problem Yeah. When it comes to security of USB, whatever.

So half of the people zoom out, and you have a bunch of people very engagedly discussing.

And, it becomes a relatively, I would say heated ish debate relatively quickly just because, it's a topic that Moves people and, makes people take very strong opinions, that they hold very weakly and sometimes very strongly.

So, yeah, I would say it is a very, very interesting discussion all the time.

And, of course, we do hope that eventually we can convince vendors like WebKit, like, Mozilla to eventually ship those APIs.

And the first success was had, finally with WebMidi.

So An API for talking to media devices over the web. And, Mozilla was the first to, ship this after Chrome. And what they did is, they came up with a solution where they essentially create an on the fly extension whenever you, wanna talk to a web MIDI device. And this extension then, acts JS, like, the the barrier between you and, the API low level signals that you can, send to the MIDI device.

So we were hoping maybe, something like this could be a way forward to, allow other vendors, like, WebKit maybe, To consider their position and say, yeah, this might be something that we can, allow for our browsers as well. So, Yeah. That's a very interesting discussion.

Still ongoing. There's no final solution yet. But, yeah, that's the elephant in the room

Yeah. I'm glad you brought up WebMidi because, that was one that I think got a lot of press for. Google implemented this, And Safari and Firefox are both choosing not to implement this.

But then you said Firefox did come up with a solution there. So do you think that Safari will reverse their position now that there's a a change there, or do you think there needs to be more I I don't know. The reason was what Sanity of fingerprinting was the the reason why Safari declined to add the midi support. So What do you think is gonna happen there? Do you think that will eventually get reversed? So what we saw works

mostly with, Apple, and of course, this is only speculating, it's, very much a black box.

Yeah. But when a big company asks for something, And they are in a position of saying, to their users, hey, we are big company X. You can use our product in Chrome. And if you use our product, you know, Safari, please switch browsers.

This is, of course, something that, no vendor wants. So they try to, yeah, make The use case, happy and possible on the on the device in question, which means sometimes, yeah, Going through extra hoops like like Mozilla did, will this ever happen? We don't know. So a big company like Lego asking for this is Obviously, something different. But then also, they, of course, do have, a very strong presence on the, Bos App Store. So you could say, people who wanna talk to Lego devices and, who have in their schools, let's say, a fleet of iPads, They could just download, the, iPad app for Allego and talk to devices like this. So,

it's interesting. Yeah. Definitely. Yeah. So we talked to Eric Meyer who works at Agalia.

Agalia is hired by all the browser vendors to implement Specific features. And I thought the most interesting one was that they were hired by Adblock to implement CSS ESLint the browser because, like, that would be super helpful for blocking elements that have ads.

Is is a lot of this stuff also driven by specific companies? Like like you said, Lego. But, like, do you have any other examples of companies that Say, like, you know what? We need this API because not necessarily because someone wants to visit the URL in the browser and use this thing, but our app runs on Chromium.

And we need this API in order for it to work.

Yeah. So, one of the big cases that we had was Adobe. And, Adobe brought Photoshop to the web, which is kind of mind blowing in itself.

And the way Photoshop works, is Essentially, they have a big, big swap file on their native application. And in the swap file, they host things like, mipmaps.

So I didn't know what a mip mipmap Wes, so I guess, most, listeners probably would be happy for me to explain at least try to explain what it is. So, essentially, if you have images and you have images of different sizes so that you can zoom in, a mipmap is a way of Just precalculating various resolutions of a certain zoomed image so that it can then very quickly, Change zoom levels. So Uh-huh. Maps, serve for this and probably other purposes. And I'm, I've probably done the worst job of explaining it. But essentially, The requirement there is you need to talk very performantly to a file and, frequently so. Because whenever you make even 1 pixel change, all your zoom levels need to reflect this change because you wanna zoom in. You wanna see the change reflected everywhere. Right? So, What, they needed on the web was a way to sort of replicate this behavior so that, you could, create a Photoshop Image or open a Photoshop image, make some changes, and then be able to zoom in very quickly.

And, this Eventually led to the creation of what we now have as a cross browser API called origin private file system.

It was born Under the name storage foundations, so people who have been following our blogs have maybe first read the storage foundation blog, and then eventually, they read the origin private file system, blog post.

And, yeah. So this is where we could see this kind of behavior really driving cross browser adoption because Adobe is obviously a very, very big company on the web.

And, yeah, they needed this API. They wanted to bring Photoshop to the web and not just to Chrome, which JS, something that I always applaud. So if you build applications, you should be building web applications and not, applications that only run-in Chrome.

And that that's me working on the Chrome Vercel team. So we actually wanna get web applications, not Chrome, running in Chrome applications.

So, yeah, Wes, saw that Adobe probably pulled some strings behind the back of, Most, yeah, curtains that people see, and they just somehow made Apple implement the o p s so our original private set file system, OPFS.

And, eventually this also led, Mozilla to implement. And, now we Yarn in the situation where we have this new API.

It's a WhatWIC standard, the file system standard, that, implements, yeah, essentially a very performant file system that is modeled mostly after POSIX APIs so that you could, get Synchronous access to, write and read operations, which makes them very performant.

It's hidden from the origin, Or, actually, hidden in the origin of the, of the web application that you're loading, which means, the files that you create there are not opposed to the user visible file system. And this means, browsers don't have to implement things like, safe browsing. So, whenever you download an as a a file from, an application, your browser, typically does something, like virus checks and make sure that you don't, I don't know, get a Trojan horse. So, this is called safe browsing, on Chrome.

Other browser vendors have Other options, but, yeah, mostly, the idea is the same. We wanna make sure that you don't download anything, bad onto your device. And, if these files are not discoverable by the user, by just going somewhere on the file explorer and seeing them, we don't need to run these checks, because we can just be sure the application deals with these files alone. There's no way, that anything else can get access to those files. And this is why, yeah, the access to these files can be very performant. Mhmm.

Let's go through a couple more of of these APIs because there's there's literally hundreds of them, ranging from a crazy idea someone thought of one day to Actual ones that are going through the entire, standards process.

So one kind of in the middle that I've used a couple of times in my courses is the face detection API, and I don't think that's a browser standard or anything like that. It only works in Chrome, but it's kind of cool to see that That type of thing is being, sampled. So do you have any other ones that are kinda interesting or or some of your favorites? So, face detection Actually JS a really interesting example, that we could maybe briefly touch upon. Sure. Yeah. So face detection

forms part of an API called shape detection API.

The one thing that is, exposed on the, like, Scott behind the flag, just, shipped It's the bar code detection Yarn, and Wes actually see Apple's implementing that as well, because it's considered to be a generally useful, API.

But, yeah, so face detection is also, in this family of shape detection.

And, Obviously, if you look at, devices like iPhones, Android phones, and so on, they tend to have, face detection as part of the operating system. So The camera is able to, spot where JS the face, so that it can zoom on you and stuff.

And, So today, I listened to, your show on AI, which, obviously AI face detection, face landmark detection, and so on is a very interesting computer vision Ask. Mhmm. So with many of these APIs, the question arises.

Wait, does this make sense as a high level API? So should there be, Whatever detect face something something API, and it does exactly that and nothing else? Or should there be, like a generic Way of saying, hey, we give you the freedom to load whatever model in the browser and then run whatever camera image through it and you get, the detected faces and landmarks and whatever, or should this be somehow be tied into, what the operating system gives? Which then, gives another interesting, yeah.

Which then shows another interesting problem, which is, not all operating systems have the same level of support. So If you think, a Windows laptop, maybe face detection is part of the operating system, but maybe not. For Android, it is. So Where do you start? Where do you stop JS a very interesting Wes to ask with many of these APIs. So how high level should it be? How Generally useful. Should it be, how should you just implement something that is super low Vercel? And, if you wanna get To a higher level, you need to, whatever, pull in some model that does the task for you from Hugging Face or whatever.

So, yeah, it's, it's an endless discussion. And, I guess Wes don't really have an answer yet. So more recently and now continuing to new interesting API proposals, there was a proposal from Intel for, a blurring API. So, looking at your image, Scott, so you have your background blurred.

I think you, Wes, too.

So this is Yarn of the operating system in many cases, but, of course, it's also something that you can do with, segmentation. So you can just use a model to segment, the foreground from the background and then apply some blur effect to it. So, yeah, Pnpm have proposed this, as an API that is high level, but it can obviously be done as a low level API as well. So

Very interesting discussion. Yeah. I I went down this exact route yesterday, via I was trying to run A bunch of models in the browser kind of seeing which ones are small enough and fast enough that you can actually run it in the browser at no cost It's just the CPU.

And there's a really good library called Transformers.

Js, which is being worked on by a dev Hugging Face, and he has adapted many of the popular models to being able to run-in the browser. I was I was really surprised, like everything from, like, image detection, elephant detection, to Contact detection.

Hot dog detection.

And I did that one of my types, of course, is the hot dog detection, but also things like sentiment analysis and creating embeddings directly in the browser.

And I went into the, like, the issues to see what people are asking for, and it's not It's not, oh, we need, like, a browser based AI.

Like, I can load a model into the browser and use it because I don't know if, like, in 3 years, we'll be like, well, The browser one sucks. You know? It doesn't work. But what people are asking for is even a lower level one JS just we want Wes GPU support for this library so that it'll run faster. So it's kinda interesting. Like you said, where does the line draw of, What API should we have or should we just provide lower level primitives like a like a web GPU support that makes Fast as hell. This is sort of blurring the lines with the AI discussion.

So we have the WebNN proposal by Intel, which obviously makes sense if you're a processor maker that you want to get things onto the processor.

But, yeah, we also obviously have things like TensorFlow, TensorFlow.

Scott, TensorFlow running in Wasm.

Wes do have, as you said, the higher level APIs.

So, something that I found very interesting is, connecting all the dots, making very cool applications By combining all these different APIs and, if you wanna, sweep, sweep swap, the implementation, you could easily do so. And Something that I'm very interested in is also, if you think global, globally, downloading a model can be Free for you if you are on a high speed Internet and I'm 50 max, 100 max, 500 max, whatever. Mhmm. It's essentially just done in a second, or it can be very, very expensive.

But, But, download cost is Node thing. Storage is then the next. Like, how many different, let's say, video image editing applications to use? Let's say 3.

And all of those would need to download some sort of model that you need to store somewhere and ideally store it in such a way that it's Persisted, which brings us back to the origin private file system, where you could store such a model.

But then, yeah, because Recently, we had a lot more, privacy focus on the on the web.

None of these Sanity be cached and stored, in a way that, they could be reused by different origins. You have to, in the worst case, download the same model For every single application that you're using. So, that's another interesting problem there. So how do you make sure that, your applications are useful and accessible Across the globe with, all these different, constraints that you have on network, on device, yeah, power Node device memory.

How do you make sure that this works? Another opt option is, of course, to just say, hey. Just bring your own AI AI key.

You plug your AI key into the settings, and then you can use something that runs somewhere in a cloud, so you can Get the same, experience across the globe to everyone.

It will be consistent, because The cloud will determine how fast or not your model can run and not the device.

You can think hybrid so you can see, How performant is the, device? We have an API for compute pressures. You could dynamically even, see, Oh, how busy is the processor? Is it running in the red zone? Is it, still green? Maybe if I notice, I'm running suddenly in the red zone, I could then dynamically swap and say, I go from on device to in the cloud. So there's a whole bunch of interesting ways to combine all these new APIs, and, yeah, just make applications that work universally for everyone, wherever they may be and whatever, device they might be using. Yeah. I CapCut is

a desktop video editor that's built in.

I I downloaded the thing and I was like, this is amazing. Like, it's a full blown video editor, HTML and CSS.

We've had the Devon from Descript as well. That's kind of similar idea. And I dove deep into the, the bowels of CapCut, and you go into the library application support, and They have a lot of their models downloaded