Supper Club × Dylan Jhaveri - Video for the Web and MUX

Welcome to Syntax.

On this supper club, we're gonna be talking to Dylan Javeri, about mux and video for the web, the challenges of video streaming, modern video streaming, future video streaming, and all Kinds of things in between.

Do video. A lot of developers learn through video. I'm a self taught developer, so video was a big part of how I learned to code. So, yeah, super stoked to be on and talk about it today. Nice.

Do you wanna give maybe just a a little background about who you are and what you're doing?

Yeah.

I'm, so I work at Mux, mux, m u x, .com.

I'm for kind of 10 plus years, I've been a software engineer building products, kind of switched off between working at companies, starting my own companies. But for the past almost 4 years now, I've been at Mux. Mux's video infrastructure for developers.

The last startup I had before this was building a product.

That Video was a component of that. It was a SaaS product, called Crowdcast and did, video streaming.

So I came into that kind of with no background as With video in particular, I just was building products on the web, building with a lot of the same stuff you guys talk about, and, I had to kinda learn about video for that start up. And that's how I started learning about the space, Learned about Mux.

Kinda got involved with the folks here and then joined joined here for the past 4 years. And right now at Mux, I, the director of open source and community, one of our main products is Mux Player, and that's open source. So that's, like, a team that, that I work on that that, builds that that video player and the tech behind that video player. Oh, nice. So as a as a fan of

Mux and a not fan of most video players on the web. I was so excited to see you guys put this out.

And, Wes, if you haven't used this thing, it's a web Components for video that make working in video feel like a way more modern modern thing was When you when you set out to to build, like, a web component video player, was it like was the intention that all of the other video players out there are just Not meeting the kind of modern needs that we have, or, was it just like Mux needs something in this space? Let's let's get something out. Yeah.

It was really it It was really practical, honestly. Like, probably the most the biggest video player that you know and heard about is Video JS, and that was started by, actually, one of the founders of Mux back in, you know, 2010, and that was kind of built almost just feels like for a little bit of a different era. It's, like, kinda jQuery based, kinda feels like you, you know, you have a constructor. You pass in, like, a dumb query query string, and it, you know, kinda builds it Throws throws in a bunch of dibs, builds out the player, and, that was kind of the player that most people kinda defaulted to using. There's some other open source ones out there, but Video. Js was kind of the most common. And a lot of our customers, they'd they'd use Video JS, and they'd try to use some of the some of the plugins out there. And, they'd usually find a way to get it to work, but we what we found and Since Mux is just video infrastructure, like, from the beginning, we we use open protocols for delivering video. Our our stance is, like, You could bring your own player. Like, we're not trying to be a closed ecosystem of you have to use our player. Like, you know, some other platforms, it's it's sort of proprietary, and they kinda force you to use your that that player. And then if you're kinda familiar, you you you just recognize that player when you when you see it in the wild, and and Mux very much tries to be transparent. Like, If you're if you're a Mux customer, we don't we don't want your your customers to know you're using Mux. That's kind of a you know, that's between us and you that that you're using, Mike. And we're and we're just we're just the infrastructure. So you you're you we wanna make sure you can build whatever experience you want fully custom.

So coming over from that, it didn't really make sense for us to have a video player first. Right? We're just video infrastructure. There's plenty of players out there people can use.

But what we found is that customers were, running into problems with video players out there. They wanted an easier solution. They were, they're like, I I love everything I'm doing with mux, but why am I having a hard time with the player? We'd have kinda support issues, and then we're we're debugging other players out there. And it's like, you know, we're we're dealing with a bunch of the issues ourself, that at the end of the day weren't weren't necessarily exactly our fault, and we're like, okay. We can document this better. You know? We can We can try to make it easier around the edges for these players. We can we can put in pull requests to fix issues and other players out there that aren't even ours, and and we were doing all that kind of stuff.

But then it came time to, hey. What you know, what have we gave our customers an yeah. A player that they could drop in and just use and get and get all the functionality out of that that they want, and,

and that and that's how we approached it. And I can I can talk more about the the kind of web components angle and stuff like that? Happy to happy to talk all about that. Yeah. I'd love to get into to some of that stuff. It it does feel like instead of supporting 1 video player, you were having to support all of the Video players out there do support for them. So but, in addition to I guess, we can talk more about the web component stuff. But in addition to a player, You guys also have, like, uploaders now, and an audio element, and a viewer count element. So it seems like Y'all have, like, a a focus on building UI elements for video as well, around here. What was the was the choice to go with web components A challenging one, or was it an obvious one? It was

it was kind of obvious when we kinda weighed all the All the things we had to consider. So first of all, our our opinion is that we're we're we're not of the opinion at least, I'm not of the opinion that, like, kind of web components are the solution to everything. It's going to kinda, like, replace React or anything like that. Like, I I think web component's just another Another tool, and it's good for specific use cases. Yeah. I think a video player is a really good use case for for a web component.

You know, we Could've we could've spent a bunch of time building a React specific player and doing that, but then we'd also have to build of you 1, and, and and we have to build, you know, a specific player for every single every single framework. And as As you as you all know, frameworks come and go. They they get popular, then they then they, you know, wax and wane in popularity.

And we wanted to build something that can be used for, you know, for the next 10 years by by by developers and and not have to be constantly chasing the the latest trends. And When we evaluated the options, web components was actually finally at a point where we could it had brought enough support across the evergreen browsers, and we could actually build something As a web component, feels native to the web.

Like, kind of the principles we approached it with is that when you are using one of our s one of our elements in your application. It should feel like the rest of your application. It should feel like it should fit in with all your application code. It shouldn't feel like this weird work of your application. You should feel comfortable with it. You should like using it, and, web components met that for us. And it it it plays really nice in in all the frameworks, And we have to have this special React wrapper to do to to make it work to make it work in React, and that's just kind of the state of things now. I think React will get Better at web components support. It remains to be seen if the community will kind of embrace it. It it kinda feels like maybe not or maybe maybe at least not yet. So so we're happy to, you know, do what we have to do for React to make React developers happy, but it seems like so far, every other framework, people are pretty happy just using web components Out of the box and and and all the other frameworks. So it's been it's been working out well for us. Yeah. We use the video element for level up tutorials,

And, we use just the straight up web component in a SvelteKit instance, and it was, just couldn't be any more seamless. I often use it as, like, An example of the way that web components can be done. Right? Because too oftentimes, we are, like, getting into arguments about web components, in terms of having to take over your entire stack or something like that, would the the the mux video player is the perfect web component for me because It replaces something that feels like it should just work in the browser, which is video streaming. Right? It feels like video you should just be able to pass in an h l or m three eight u whatever. M m three u eight. Is that it? Yep. You got it. M 3 u eight. Wow.

M 3 u eight Into a video element source, and it should just work. And that and that's really what it feels like using this element. If you need to customize it, you can. So, like, Bravo on on finding what I consider to be like a really good use case for

web components here. Let's talk a little bit more about, Like what Mux is as well. So, like, obviously, you have the player and it looks like you have stats and stuff to actually consume it.

But that is only there because Mux is an entire platform for doing video. So tell us, What does Mux do? Why can't I just put video source equals my video dot m p 4? And, like, where does it sit amongst Something like a Vimeo or a YouTube. Yeah. Totally. So Mux is broadly just video for developers. Our mission is, like, we wanna power

all video on the Internet for developers.

Video probably the first thing to understand about video, there's kind of 3 main buckets of video on the Internet.

So you have on demand video, which is, You know, you go watch a video on YouTube or Netflix, and that's just a static video that's that's what was recorded before and put on the Internet. So you have on demand video.

You have live video, which is kind of a live stream that usually has a little bit of a delay, so it can have anywhere from, like, 5 to 30 seconds of delay. That's like if you're watching if you're watching YouTube TV, you know, you're kinda like cable replacement, and you're watching that, that's You do or if you're on Twitch, you're watching Twitch, that's like live video. And then you have a 3rd bucket, which is real time video. So real time video is, like, You're on a Zoom call. You're on a Google Meet, and you're talking in real time. Now these are 3 this this is how we kinda think about the 3 segments of video. Now 2 of them are very similar, and one of them is completely different technology.

The 2 that are similar is on demand and livestream.

So the way the way that on demand video and the way live stream video works is with what Scott was talking about with the m three u eight. So that's, like, HLS protocol. It's a standard for delivering on demand and and live streams. And what HLS said and the reason the reason why is you can't just kind of, like, Put an MP 4 in a video element. You can do that. Right? But, typically, the MP 4 that you have is gonna be the kind of, like, highest quality, highest definition MP 4 that your Your editor exported and gave to you, and that's gonna be the the biggest bandwidth version of that video.

And the problem With using that is that users have all kinds of various Internet connection speeds. They might be on Yeah. Mobile devices. They might be in different parts of the world. They just have different Internet connection speeds. And they're not gonna have enough bandwidth to download that large video to watch it.

So the solution is Using a an adaptive bit rate, streaming technology. So that's what we use is HLS. It it's the the URL end ends with dotm3u8.

And what, adaptive bit rate and what HLS does is you take that high quality definition video, like the MP 4, And you break it up into little small chunks. So you break it up into, like, 6 second chunks, for example. Mhmm. And then in each chunk, you process it into maybe 5 or 6 different quality levels all the way from the from the highest quality level all the way down to the lowest.

And then you throw that all on on CDNs, and you and you distribute that.

And what that does is you give, you give the player the dotm3u8url, and it's basically A surprising amount of this is just text files that are, like, not not formatted like JSON. They're just, like, text files with line breaks that you have to that you have understand and parse, and that's what the player understands and parses. So the player will look at m 3 u 8 and see, like, okay. I have 6 different quality levels, and each one requires a certain Amount of bandwidth to stream. So it'll start it'll pick, like, the middle one, for example, and it'll just start trying to play the middle one. And it'll start downloading 6 seconds of video at a time. And while it's downloading each segment of video, it's measuring how long is it taking to download that video. And if it's taking too long, then it'll it'll downgrade to a lower quality. If it's if it's downloading fast, then it'll scale up to the higher quality. So you might have experienced this if you're watching Netflix at home or you're watching a YouTube video. And if your Internet connection just suddenly slows down, you'll still be able to watch the video, but it'll just start to look visually like a lower quality. And then if your Internet catches up, then it'll it'll ramp back up to the to the higher quality level. So so that's the way kind of HLS delivery works. And you can do that with an on demand video Because you have kind of that MP 4, you could process it. You can do that with a live stream. So with a live stream, typically, you'll you'll ingest that video over RTMP.

If you've and maybe you've seen RTMP, like, if you've ever used OBS and you put in the RTMP. So what what that happens is you'll be sending your video and audio to an RTMP server. The RTMP server will ingest that video, process it, and create these little 6 second chunks of video, and then push them out over CDNs.

And then, end users can download those those chunks and integrate it with a player and just be able to watch Watch that video and consume it. So if you are a developer and you're wanting to put video in your application, you have a few options. You can build all this stuff yourself.

Amazon has, like, some services you can piece together to do that.

You know, you can use all the FFmpeg, use all open source source stuff, and Figure out how to do that, integrate with CDNs, and and and do all that stuff.

And Mux tries to be kind of the the simple answer where You 2 who have used Mux, you probably you you never had to think about how many quality levels am I doing. Am I am I choosing the right bit rates for each video? So Mux tries to be the simple answer where you just Throw your video at us. We'll process it. We integrate with multiple CDNs. We give you a playback URL. You just drop that into a player, either our player or any player you want That works with HLS, and you can just get that get that adaptive beat bit rate streaming experience.

And then Mux gives you other things, like if you wanna add Captions files to that video. If you wanna pull out thumbnails from that video, you we have APIs for for doing all that kind of stuff. What what about something like,

Tacking on. So something I have to do before I launch a course. I have to go through every video,

and I have to put a little 3 second bumper in front of every single one. Is that something you could Take a sec. Alright. I've Bumper Dime before. Slap that in right before you start playing it. That's stitching. We don't we don't have that yet, but I'm gonna put in that feature request We have the opposite of that, which is clipping. So we have the opposite of it. If you have a video and you wanna clip part of it, we don't have it yet where you can kind of, like, add parts onto the video. But,

Yeah. Those are the kinds of things. Oh, yeah. Yeah. That would be pretty cool. I it's funny. I my then my next point I have here is clipping question mark, because I was gonna ask, like, can you take something that's larger and clip down to a a smaller value? And Scott starts writing underneath it. You sound fine. I was like, no. I'm gonna ask about clipping video, not I'm not asking my audios. Okay?

Yeah. I I thought that was what we were talking about because we we have our show notes open all the time, and we'll, like if we need to, like, communicate with each other. We'll do we'll write write little notes. Well, you wrote clipping. I was like, no. I think it sounds okay.

Before we get off of this, HLS stuff, though, I I do wanna, like I I wanna dive into a little bit more about HLS, because when you when you get into video streaming, there's there's other, streaming techniques like Dash. Is Dash still exist? Do you know if that's something that is still around? Because it looks like it was in Chromium at some point, and

It's nothing but red. Oh, yeah. Dash is still around. I mean, you need to use, similarly, it's not supported by browsers natively, so you need to use Dash. Js or something similar too. And it it's a very similar concept To HLS, it takes a video, breaks it up into segments. It's just the the files look a little different. Like, HLS is basically represented in just text files with line breaks. Dash the the manifest files for Dash look more like HTML. So just the the format of it is a little different for how the player understands what to do. But Underlying, it's the same the same idea of taking a video, breaking it up into small bits, and then break into into different quality levels and then streaming having that adaptive streaming Based on based on the bandwidth. And the 1 browser that does support HLS natively is Safari. So Safari is the only one where you can, like, throw an HLS playlist into a video element, and it just works.

All the other browsers don't. Is that why the Apple announcements used to only work on Yes. On Safari?

They didn't wanna put the dash in there. They didn't they didn't wanna put HLS JS or whatever. Oh, yeah. Yeah. They didn't wanna load, like, a third party HLS library to make it work in the other browsers. So, yeah, they you had to go to Safari and use Yeah. Yeah. So that's interesting. I wonder, like, do you think

and I know you you might not have any inside information, but just like Purely speculation.

Speculation. I could only come with spectacles. Like, do you think HLS will ever be natively supported in Chromium or Firefox? You know, I haven't heard much about that yet. I I don't really know one way or the other, to be honest.

The one thing that like, the The way we get video into those browsers is through media source extension. So that that's kind of the important API, MSE, media source extension, which is how you can take Media, push it into a buffer that and then have the browser, like, render it. So that's what HLS JS does. That's what Dash JS does. It's kinda like Interesting. Pareses all video processes in it and then subs it shoves it into a source buffer. And, the 1 browser that doesn't support media source extension is iOS browsers. So it's like the Safari based iOS browsers. They don't source support media source extensions. So the only way to play HLS on iOS browsers is with Apple's HLS, which means you have no control over the bandwidth switching. You have no control over, anything that's going on in the internals of the player. You just have to say, play this HLS video, and Apple just just plays it. Interesting. Desktop Safari, even though it supports HLS natively, it Desktop Safari also supports media source extensions, so you don't have to use Apple's Native HLS on Safari. You can use your own you can use HLS JS, or you can use your own and and use that media source extension. Wild. Yeah. It's so funny because we went from, like, a world where,

you know, you did every video in Flash and then you have HTML 5 video comes out. But it it feels like This this kind of streaming based video is really the only way to do video nowadays.

Do do you know if, like, Like, companies like TikTok or Instagram, are they streaming all of that video? Is it is it using

HLS, do you think? Oh, yeah. Yeah. Yeah. I haven't checked for sure, but they're definitely doing some something like that. They're probably using either HLS or Dash, and they have that integrated with the player. The thing that TikTok does really well and those apps do really well that Everyone is trying to do now, which is is quite hard to replicate, is how aggressively they kind of, like, preload videos.

Mhmm. So it's like, We we get users all the time where they're, you know, they're building an iOS app or they're they're using, like, React Native or something, and they're, like, you know, scrolling through a feed of videos, and they're like like, It's and it it starts up in, like, maybe maybe half a second, which is, like, you know, scrolling. It's, like, loading everything everything fresh every time it scrolls, and it's, like, You know, it's kind of expected. Maybe, like, 100 milliseconds or 400 milliseconds. It's like sort of they're like, no. I wanna be fast like TikTok. And it's like, Well, in order to do that, you have to like, TikTok does some really advanced stuff where they preload not the entire video, but just, like, the beginning of each video for, like, the next 10 or 20 videos in the feed. Because the thing about video and the thing that for me was a learning was, like, video is expensive. Like, if you're if you're just building a SaaS app that's just doing, like, basic CRUD or whatever, it's like, you never really need to think about Bandwidth. Right? You deploy it. You deploy your your your app somewhere. You you're never thinking about, oh, are users using a lot of bandwidth on my servers? Like, it's it's not something that really crosses your mind in, like, Modern web development. But as soon as you start doing something with video, it, like, becomes a serious expense of the amount of bandwidth that you're using. And So, you know, even a product like TikTok, like, the they're they're very finely optimizing how much they're preloading to make sure, you know, they're not paying too much of the bandwidth, but also that users are able to Kinda see that constant content instantly and not bounce. And, like, as of right now, iOS and Android and stuff like you, that's all very custom. Like, they don't really have APIs available for just kind of off the shelf, especially if you're using React Native because React Native is even, like, 1 layer of abstraction higher, where Where it's like, okay. Load a video player. Give it a URL. It doesn't really have easy ways to, in a performant way Yeah. Preload, you know, preload just small segments of video and and do Stuff like that. When

I when I send you a video, let's say I send you a video that's in 4 k, and you guys you guys will then process it into How many versions of that video? 5. 5. Okay. Yep. Yep. How long does that take? Like like what's happening? Do you have, like, a Someone's PC sitting on their desk at Mux that has a sweet graphics card in it. Like, are you using some sort of, like, GPU heavy server?

How does that work? It is a bunch of big servers, and we do something that is very that's a bit of our secret sauce. That's a little special to Mux, which is called dress just in time transcoding.

Like, we make your videos ready really fast. So even if you send us an hour long video, what we do is you send that video to us. There's kind of 2 paths that goes down. We have, like, a we determine whether it's, like, a standard input or a nonstandard input. If it's a standard input, it just means, like, It's, it's using codecs that that we recognize. It's it's, you know, pixel density, and all and all these factors make it standard. And if it's standard, Then we're able to process it really fast. So a, you know, hour long video, it could be ready in, like, a matter of seconds and ready ready for playback. So we'll do, like, 1 processing pass on it and basically Put it into a format where we can, transcode it just in time.

And and after that, after we tell you it's ready, like, It's ready. You can play that video.

And then what we do on our end is when users start requesting That video and start watching that video, that's when we actually transcode it and actually make the HLS segments.

And Then and then we just cache them over over the CDNs, like, as so if you if you go to a video and no one watches it, then we never have to actually Transcode it and save those segments of video. We just save it in this, like,

specific format that's, like, ready to be just in time transcoded. And that makes so much sense. Yeah. Because I was getting into before I went with Mux, I did a bit of a tour on a lot of the different, like, video hosting things. And one of things that I would really looked at was the AWS, like, VOD workflow that they have. And one one thing that was always, like, kind of shocking to me within that was the fact that By default, in in that workflow, when you put something into a bucket, it's it kicks off that transcoding process immediately and spits out, like, 8 versions of the video For everything you upload all the time, even if no one's watching those. And, like, that always felt, like, extremely wasteful to me. So it it Makes perfect sense as long as that the transcoding process can be fast enough. The is is FFmpeg

still, like, the the thing in this space, or is there Fancier things? Yep. Yep. It's it's FFmpeg. There's there's, the other thing is g streamer, but we we use a lot of FFmpeg. Wow. Somebody somebody at Mux knows all the dashes,

that you can pass All the flags?

Yeah.

For sure.

Oh, for sure. That's amazing because, like, freaking YouTube. Sometimes I'll upload a video to YouTube, and I'm sitting there for, like, an hour waiting for that thing to be HD ready before I can post it. Because otherwise, you post it, and everyone's like, the the quality is awful. It's like, yeah, I know. But the other day, I posted 1 to YouTube, And it's like, it'll be 55 minutes to get 4 k.

And then, like, 6 minutes, not even, like, 2 minutes later, it was it was ready. And I was wondering, like, What are they doing differently? Like, couldn't they do that on demand as well? Maybe it'd be too expensive. Yeah. It's a good question. I'm not I'm not sure the internals of how of how YouTube YouTube manages that. Yeah. They should use mugs.

We we would love

it. Oh, that's super interesting. And Is it special servers that have GPUs in them or, like, is the is that just done on the CPU? I don't actually know that. Oh, okay. I'm always it would have to be. Yeah. Like, right? I don't know. Maybe I'm yeah. I think it's I think it's big CPUs, honestly, but I I'd have to check. Yeah. When I export my video, there's like a couple options. And like you can do like hardware accelerated, or you can just use CPU. And and by default, Like ScreenFlow is just CPU export.

And then the the hardware accelerated version is really not any faster. I don't know if it's wrong with it, but, yeah, I was surprised that I was like, oh, it's for sure a GPU because I was looking into, like, building, like, a custom hackintosh with, like, a $8,000

graphics card to go faster. I did that. Yeah. And I talked to the folks at ScreenFlow, and they're like, I won't make it any faster. It's it's really didn't really? Yeah. Yeah. Yeah. That's what I ran for a long time was a beefed up Hackintosh, and it was a Yeah. In the butt. It did not get me that fast. And it didn't help? Yeah. It was like not it was not very much fat or not fasted. Like, you'd expect something like that to be enough faster. You'd be like, woah. But it just it never really was.

So has Mux done any anything with web assembly at all? You know, it feels like this might be a a case where Some of these heavier tools, I know, like FFmpeg.

I I don't know what the status of FFmpeg

in the browser is, but is this something that y'all have looked into at all? Yeah. There's a really interesting project called FFmpeg WASM. It gives you, like, the f, it's like a WebAssembly version of FFmpeg that you can run-in the browser, and you could do things like Like transcoding and stuff like that. We we've kind of looked into it and and played around with it and kinda like hackathon kind of kind of ideas, but, we're not doing anything with it yet. But I could see it being handy, especially when we as you kind of alluded to, we're you know, we have the player. We have the uploader. We have, like, other elements we're building for the kinda application layer. And, as we do more of that, I could see us Exploring some of that stuff where if you're if you're actually, like, recording your video in the browser or you're trying to, like, compose video in the browser, you're trying to, like, do a live stream from the browser.

So we are we are starting to play with that with that kind of stuff to see to see what's possible. Yeah. No. It feels like a a space. Just in general, video for the web feels like a space that there's enough exciting things available in the future or potential future that You could probably sustain yourself in all sorts of ways of interesting things to work on. What about, like, has the AI stuff Hit you guys at all in terms of changing how you're thinking about some things? I know we've been looking into a lot of Transcripts and subtitle stuff with AI. Is that something that you all have hit, or have you found that it's not necessary? Yeah. We've we

Our kind of exposure to AI is more like through our customers.

So we're like again, we're just trying to be video infrastructure. Right. So if you're an AI company that's doing something like generative video and you and you build all the stuff where you can, like, type in a prompt and generate And and get this, like, video that's generated.

We wanna power the actual video when you have to distribute that video and actually have to send it to people and want to stream it or Turn it into a livestream and be that. So, we we're not really using AI as directly.

We have We have tried stuff in the past, and and we do have some stuff, that we've experimented with with being able to Optimally, like, transcode versions of your video that makes sense based on, like, the content of the video or based on the audience of who's who's who's viewing that video. So If your video, for example, is mostly viewed by by users who are on, you know, low power devices that are watching on their phone, you you There's probably a different bit rate ladder that makes sense for your videos that would that would minimize the amount of bandwidth and and And, maximize the quality that your users are getting. So there's a lot of kinda, like, machine learning and things that can go into into optimizing the What we're actually outputting and what we're what we're delivering so it's so that you're always trying to make this trade off of having the least amount of bandwidth with the highest quality video, And that there's a lot of factors that go into that. It can depend on the actual content that's in your video. If it's like a sports game and there's a lot of action and moving back and forth versus, like, a static talking head versus and then depending on, like, where users are and what kinds of devices they're they're viewing on, there's a lot of, like, kind of, yeah, machine learning and things that can go into To optimizing that that, that we've kind of explored in the past and done some stuff with. But yeah. So we don't have, like, a a direct kind of, like, how are we using AI AI to power to power video,

today. Yeah. It feels like something you gotta ask everybody now, though. It's like,

have you have you seen this stuff? What what are you doing with it? Yeah. Yeah. Yeah. Our bet is kinda like, as long as developers are building stuff and putting video in their applications, then then that's, then that then that's where Mux fits in, and I don't think That trend is is going away. Yeah. You guys are lucky in that,

like, you're like I I almost see it as as, like, electricity. You know? Like, you're not, You're not gonna be out AI'd because somebody uses AI to build a better platform or something like that as people still need to get video to the end user.

And you're almost like a utility in that regard.

I don't mean that in a bad way. I mean that in an awesome way. You know, like, you are the backbone Bone to getting the video to your user. Exactly. And, one thing for sure is, like, as these AI tools develop and as it's like you can build applications with AI and people are you know, describe try, but, you know, you've seen the demos where people, like, describe an application and kinda, like, build it for them where they can build a back end.

We would definitely like to Plugged into those tools so that when someone builds a builds an application video, it's using Mux behind the scenes. They put in their Mux API keys or whatever, and it, like, does the API calls and sets up the webhooks and just, like, Handles all that for them. That's actually something that we were just playing with on Friday is,

you know, those, like, like, podcast clips where people put text Over top of it, and they highlight the current word and whatnot. So we we piped 1 of our podcasts into, like, a transcription thing, and they gave us back The transcription as well as, like, timestamps for every single word.

So then I pipe that into some JavaScript and based on where you're playing the audio, I'm showing the transcript and highlighting the current word. And then I was like, well, could we record this somehow and kick that out to Kick that out to like an MP4 that that is then shareable. But who would do that on demand for people that are like, oh, this Scott says something really cool. Can I share that little bit? And so we're I was playing around with, or the APIs.

Media recorder API in the browser and the screen Quarter no. Media recorder is a screen recorder. Yeah. Media recorder, you can capture camera or screen. That's awesome. So we're trying to figure out, like, Now can we do that headless? So you're doing, like, get user media

and then capturing it with media recorder? We use get display media to capture

the user's, Uh-huh. Screen screen screen recording. And then we use media recorder to pipe the stream into like a stream could come from a webcam, come from your Yep. From audio, and then you pipe that into a meter recorder. And then when you're you got all these chunks of data, and then you you could send those chunks to mux, I assume. Right? And were you sending those to

another tool to do the thing with the audio and the text?

No. No. We were Right now, it just manually took one of our like transcript outputs and and manually import it. But ideally, that's the end game is that you could You could just clip it, bring in the audio and the transcript, record the screen as it plays it back, And then kick that out to MP 4 or some Yeah. So we actually have something,

we just wrote a blog post about this recently, but it's called a web broadcast that we're experimenting with, which is If anything you could create in a browser, and you could just broadcast that as a live stream or, you know, you get the recording after a live stream. So if you were able to create Just a page that has you know, it could have video coming in over WebRTC.

Mux has a has a WebRTC product now I can talk about. That was kind of the 3rd bucket of video, is WebRTC, like, real time video. So Oh, yeah. If you're so so we so we have a, API now. It's in, like, private beta where Anything you can just anything you can compose in a browser. So you're just using your JavaScript and CSS and piecing different tools together, and then we and then you just send us the URL to that to that page, and we'll just record it. And Wow. Broadcast broadcast as live stream, or you could just save the recording.

Yeah. So that's kind of like a general purpose thing we have that I think I'm really excited to see people get kind of creative with that. That's Yeah. Because you you could bake, like, a OBS in the browser if you wanted to. Right? Exactly. Yeah. Yeah. And that's what a number of people, you know, building these live streaming platforms have done some some version of, and that that's what we've we've kinda seen peep people kind of repeat building that kind of thing. And, so, yeah, this is a this is a, like, API. I'm happy to happy to let you guys try it out. Yeah. There's a lot there for us, Especially because we've been experimenting

with live streaming or even, like, when we did our live episode, we were Trying to broadcast, and we ended up having to jump on Zoom to do a video call. And, like,

it feels like there's a lot here for us In terms of potential APIs. That's really cool. I'm excited about that. And how does the how does that work? Do you have, like, a

you must have, like, a, Like a browser running somewhere that just captures the output? Yep. It's basically a headless Chrome that it's basically a headless Chromium browser that spins up Wow. And Captured the video and audio and piece it together, and it is, quite it it it's sounds simple, and it's one of those things you dead dive into. And then it's You can never get the audio and and video in sync. You can you know, there's all kinds of problems you run into as soon as you start start trying that.

Great. And Yeah. So that's that's what that's what we have to to play around with. Wow. When did you guys redo your website? Because I,

it is gorgeous. The new one, it it reminds me of That was

on Friday. It was kinda, like, soft launch on Friday. Oh, wow. People started, like, kinda talking about it over the weekend Ken, before before we kind of, like, fully announce it. Yeah. My gosh. It's great. The colors are great. It reminds me very much of, like, Ableton Live kind of Design, in a good way. There's so many cool things here. What are, like, teenage engineering style design in a a very positive way? Yeah. The the team really, really crushed the new, like, brand brand design.

We're really excited to get it out. Yeah. It it feels it feels great.

Yeah. There there's a lot of really interesting stuff here that I I can't wait to dive into. So I've been using, the audience.

You know, we we maybe talked a little bit about I've been using just Straight up only the the video stuff. But we also use the live streaming at some point for one of our I think our 300th episode we broadcast Sent the syntax website and did it live, and it really functioned just like you do do a live stream anywhere else. You you pass in that The the the code that what is the Playback ID? Yeah. The playback ID and and just stream away, Tossed into, an HLS video player, and next thing you know, you're you're up and running. Yeah. I gotta wonder about, like, video formats a little bit. I know I know we keep going I I keep going back to that stuff. But when we talked about, like, HLS, you said it's, like, kind of like a a map for, like, which video Clip. You're essentially you're loading. Right? Is that dependent on the actual video file itself? Like, do those have to be dotmp fours behind the scenes? And, like, I I wonder a lot about, like, video format changes. Like, there's WebM and things like this. Are does does that Work at all? Does that fit into play here at all, or or or is it always dotmp4

behind the scenes? Yeah. So as far as The original format of the video, that matters like, Muck kind of abstracts that away. So it's like whether you send us an MP 4 or a dot MOB or any any like, our stance is, like, send us any video, and we will process it and kind of prepare it for the standard HLS output. So We're completely agnostic to any video format that you send us.

Now on the other side of that, you have, Different containers. So, like, with the HLS, that mux that mux outputs.

Right now, we do dotts files is what you'll see. So those are those are transport streams. Dot ts is the actual, like, video format, and, that's just a container for what's Actually, inside the video, which is h 264, codec, and AAC, and But now the kind of video world is is moving to to fragmented MP fours, which is f m p 4, which is just, like, It's basically the same m p 4 container format, but in a in a fragment. So it's not the full it's not the full m p 4, but it's just a way of breaking it up. So, right now, if you do mux If you do a mux, low latency livestream, you'll get fragmented MP 4 out on the other side. But that's sort of that's sort of just a detail That, again, that kinda like the end users end developers, you know, using mux never have to really, you know, think about or worry about. Yeah. Totally. And who wants to worry about that stuff anyways. Right? The TS files reminds me of my days of

burning DVDs.

Yeah. Right. And, that's also why VLC always wants to open my TypeScript files. Yep. Totally. Totally. You're poor developers who have to use both.

Yeah. Slack always thinks that TS TS files are TypeScript.

Oh, yeah. That's funny. Yeah. The TypeScript kinda stole that one. I guess that's why we have MIME types though, not just extensions. I'm having so much fun playing around on your new website. I love the,

the grid the grid buttons, the how they expand to be from Border radius tonight. Sorry about that. It's it's, like, not not relevant at all. But, man, this new website's great, so shout out to the Mux designers. That's awesome. I'm super I'm super happy to to To hear that. Our our team worked they they worked really hard to get that out,

last week, and everyone's everyone's super stoked about it. I'm just looking what fonts, these are using. I know. The fonts are

good. There's 1 more thing I could talk about, related to the web components and the player, that I wanna touch on, which is So you're familiar with Mux Player, just the web component that you drop in.

Well, we actually built Mux Player. Like, we we kinda built a web component framework for building players.

So that that project is called Mediacrome, mediadashchrome.org, and it's on our GitHub, muxing slash media chrome media dash chrome. And what that that project is really kind of, is is where kind of All of our efforts going into when it comes to comes to the player work.

So MediaChrome is just a general project for anyone to build video players, And what it does is it gives you all the things that are missing from the browser when it comes to building a video player.

So you're familiar with, like, building a form. Right? You have a select element. You have, like, Radio, radio buttons. You have checkboxes.

So what MediaChrome does is it gives you a play button, a volume range, A mute button, a full screen element, a picture in picture element. It gives you all these, like, kind of low level controls, And they feel and our goal with that project is they feel like you're just building them into the you're you're just using the browser, just bill using HTML to kind of The same way you would build and style a form, you can just you do that to build a video player. What we found is that all the kind of all the video players out there, when it comes to customizing them, It's it's like, okay. Change the colors or, like, change this config, and they all have different syntaxes. Or take the CSS file and copy and paste it into your project and then just, like, Edit the crap out of it and, like, override override all the styles. Massive CSS file. Yeah. Exactly. It didn't really feel like you were building anything custom. It felt like you were just kind of constrained to what the player gave you.

So MediaChrome is kinda like one level lower, when if just, like, the individual components for building building a video player, and That project is meant to be compatible with with anywhere your video is hosted. It could be an MP 4. It could be it could be on Vimeo. It could be on wherever it is, and you could build a a video player for with Mediacrome and have it work with any of those video sources.

And then we use Mediacrome to build MuxPlayer. So, everything everything that we built with MuxPlayer was is is built with MediaChrome, and, we and the latest iteration of MediaChrome is that this idea of themes where we have a very simple kind of templating language that just feels like HTML that you can just write HTML and build a theme and then use that player in various places. So you can you can put a play button, then let's say you have your own button for, like, for, like, a a syntax FM, like, call to action button. Button. You could just put that in your player right next to the other buttons, and it just you're just writing HTML. So it just feels it feels natural, and we have things we handle things like she uses Scott? Like break points so you can you know, when a when it's in a small container, you know, you can lay out the buttons a particular way. When the container gets bigger, you can Change change change the layout of it the same way you would design design a web page. So that's really kind of the the secret sauce behind the MuxPlayer work. MuxPlayer is actually Doesn't do a whole lot beyond Metochrome. It it uses it just use it uses the Metochrome theme that we've built to be to look like Mux Player and swaps out the icons and stuff. But but, really, what we wanna do is enable any developer building a video player to just use that project, build a video player, And design it the way they want and just, you know, be able to ship that and run and run with that. Even if they're not using Mux. Right? They're today, they just have some MP fours they wanna play. They could build a player with that, and then, you know, maybe later down the road, they they decide they need mux for live streaming or real time or they wanna do more video, then they can start they can start using mux, and they can, you know, then then become,

Mux customers in a different way. Yeah. So does this work with audio too? Is that

Yeah. It's, like, media. Right? So it's general video and audio. Yeah. You can build an audio player. Exactly. Yeah. Oh, man. We need to we need to build the news. We're so we're rebuilding Syntax website right now.

And one of those is You need, like, a full featured audio player with, like, every feature you could imagine, AirPlay, you know, like, all of those things. Yeah. We should totally use this. We would love to help you. We would love to help you. And and we wanna get we wanna get into some really interesting stuff too with it too. We've been talking about

adding reactions or Just fun UI things to it. There's a lot of, like, really kind of niche interesting topics I think Wes and I will wanna dive into there. Yeah. Totally. We'd we'd love to help you,

build the audio player, make it make it awesome, make it look like syntax. We can make it a theme, so So you can so it can be and when it's a theme, it can be kinda portable. So you can put it on different if you have different websites you wanna put it on, you can use it in other places.

Yeah. So that that's really where the The bulk of the work for the for the player stuff is going into. And and I'm just looking at the mux,

like, stats.

Like, you can hook What's it called? Oh, that's data. Mux data. You could say you hook that up to a video and you could see where are people scrubbing back. You know, on YouTube, you can see the little blue, but where people are scrubbing back to. Is that what it's that in the show? You're like, all right, you skip this, like, 8 minute intro of somebody telling you, like, Welcome. What's up, space heads? Today, we're with the other day. Yeah. Usually, like, I'm like, I was, like, fixing a snowblower the other day, Laying in the snow minus 20, and I'm, like, watching a YouTube video on how to thread a belt. And the guy has, like, an 8 minute intro, but, like, what's up, belt heads? Get past this.

Anyways, I'll let you talk. That that's what that's for, the Muxed Data? Yeah. So Muxed Data was actually Muxed's first

Product.

And Muxed Data was first built as it's a tool for, quality of experience. So you can imagine if you are and it really it's it's kind of originally started as kind of more of an enterprise product where you can imagine if you're A media company, and you're streaming your videos over the Internet, and they're they're streaming on your website, they're streaming on your Apple TV, and they're going out to, you know, Samsung TV and everywhere ever it every everywhere you're distributing your video, maybe it's like sports live sports games or on demand video, whatever it may be, you need a way to make sure that your users your customers are having a good time watching your video. Like, are they seeing, like, buffering spinners all the time? Are they, does it is it the right resolution for their screen? How quick is the video starting up when they hit play? Is it is it skipping? Are you having all these kinds of quality of experience issues. So Mux Data, was our very first product back in, like, 2015 I built and it's an SDK that gets installed at the player level. So you just install the SDK in the player, and it monitors the playback Experience, and it's constantly sending sending beacons to the Muxed Data back end and that and Muxed Data's processing that and then giving you all these metrics. So you see, startup time, rebuffering, exit before video start, all these metrics. And you can break down these metrics by region, where they are in what part of the world, what What their Internet provider is, is it like is Comcast, like, slow in the northeast right now, or is your CDN in Australia slowed. You need to fall back to another CDN. So video engineering teams use Mux Data in order to monitor Their video infrastructure. And that was Mux's first product, and, Mux Data has done really well. And now since then, you know, we've added on Mux Video where it's like, okay. Instead having your own video engineering team, why don't you can just we could be your video engineering team, and you can, you know, give us your video. And now we've sort of sort of bridged the gap, and we started to add, what you're referring to, Wes, is more like engagement metrics. So beyond just the, quality of experience, engagement engagement. So, so So you can kind of take all the data out of Mux Data and process it and, put it in your own own data warehouse to to process that to to come up with those kinds of Insights are, like, granular level level, data, like, per video.

Awesome. Is it I I think I probably know the answer. Is it blocked by

Most ad blockers in the browser? Yeah. There's just no way to get around that. No. That's everything we do. Yeah. We don't collect any, like, Personal information or whatever, and it's like, we we tried to we tried to make the case for ourself, but the all the all those blockers are, like, very, very aggressive. Yeah. We have, like, custom domains and stuff, so you can use custom domains to, you know, get around that. But, I think actually custom domains I saw Fathom recently Said they're they're discontinuing their custom domain product because it's it, like, stopped working. Yeah. I think the ad blockers are trying to get around the custom domains. Yeah. I think they just resolve the DNS and, like, figure out that, where that domain your that domain is pointing to, and then they just block it. Yeah. I even saw like a Google Analytics, like a hot potato service that you could run. You take your data, you throw it to your server, and your server will Then hot potato throw it like form it to like other servers.

But I would imagine that the ad blockers were gonna say, well, that looks like Tracking data, especially now with AI, you know, like, then They start expecting the payloads, and then they can tell from the payload. Mhmm. That's that will definitely mess some stuff up, though. Some stuff might look like tracking it. Yeah. It actually might. Well, it already does. I had a I had a call with, I sell ads a couple weeks ago, And, they have some They have some pro they they own web developer.com.

They have this new, like, thing where, like, you can write articles And, like, tip people with crypto, like, you you get, like, Satoshis, which is like a fraction of a Bitcoin.

And, you can keep thousandth of a Bitcoin.

Yeah.

So, I had a call with them just to see, like, what it what it was all about, and I could not Get on the Zoom for the longest time. I could not open up the Zoom link they sent me. And finally, I realized, oh, My ad blocker was blocking ad blocker.

Yeah. Yeah. The domain was I sell ads? What was it? Buy buy. Buy, sell ads.

Oh, buy, sell ads. Gotcha. Yeah. Probably their whole domain name is is busted. Yeah. Right. Yeah. Cool. Well, I think that,

unless you have anything else, there is one thing I I do wanna shout out once more about the new website because I'm obsessed with it.

I think the website does a great job of showing video used in non typical ways.

So if you scroll down from, like, the square video, there's all these little video screens. You have, like, a fake screen, a fake Zoom chat, like, another fake mobile screen. The like, that video is an actual video in the In the mobile screen using mux player, behind the scenes if you inspect the source. So, like, you you guys are really you're not faking it hear it all on the website, just tossing in a a GIF or something that looks like a movie. Y'all are really consuming the video in a

Modern way, and and it's really, really, really sweet. That makes me so so happy to hear. We, we were definitely expecting people like you to inspect Inspect the dom and, like, see what's going on. So we we didn't wanna cut any corners.

Even the CSS grid here is pretty neat, West. The the little, like, One platform video for everything section has a CSS grid section where they did a, like, really fine grid squares And used it like a typical web grid where you're placing things on the web grid rather than we often think of, like, alright. You have these big squares and you're moving big things around. Right. Like the Twelve column or whatever. You guys made an actual grid and then placed things on that grid like Oh. The web of yesteryear in in, like, a really modern way. It's so good. Love that. What do you think about all the Tailwind on the site, Scott? Oh, I don't think anything of it.

I don't have no thoughts there. It's a Sweet sight. So Tailwind, Next. Js, I'm just poking around as well. Some really cool stuff on there. The Next 13 with the, you know, server components and stuff. Oh, man.

Yeah. We we did we rooted the docs site.

It's actually it's 2 separate sites, mux.com and docs.mux.com.

And the the doc site we rooted on Next, 13 with serving components as well. Darius on our team wrote wrote a blog post about that, that that transition and And move that to Tailwind as well. Nice. We will, so nice and fast. Link to your blog as well. Cool. Well, anything else From, you at Bucks? No. That's it. Super super stoked to talk to you guys today.

Yeah. Where I know, I know that one of the main developers who worked on the on a lot of the mikes.com stuff. I know he listens to the pod, and he'll be he'll be really excited to hear hear hear what you have to say. Sweet. Yeah. We're we're happy to have you on because,

Honestly, it's a service that I've personally been really enjoying for a long time and, glad to to pick the brains of I mean, you you guys are seem like the the forefront of modern video on the web in a lot of ways. So it's, really, really excited to to hear from you. Do you have a sick pick or

a shameless plug? I do. I I have 2 sick picks. Cool. Those are, like, You know, recent technological advancements that I'm stoked about. What first is obvious is the obvious one, AI.

I think in some ways, The stuff I see on Twitter, AI is, like, overhyped.

Like, I tried something like the auto g p t stuff and tried to do it, and it's like, I can never get it to work as well as what people are saying it can do.

So in some in some ways, I feel like it's maybe I'm just not prompting it right, but in some ways, I feel like, a, it's been overhyped. And then in other ways, I feel like it's completely underhyped. And now I'm finding that, like, you know, Basically, throughout my day, I have chat g p t open, and I found it really helpful with debugging.

Yeah. Like, saving me so many steps of, You know, Stack Overflow documentation, this, that. Like, I I actually find it very handy to, like, post and paste in error messages, and I pretty much have it open all day. And it's kind of, like, has now become kind of the 1st place I go to, and it's and it's definitely helped me definitely feels like it's it's better my my day to day workflow. And I'm just super excited about to see kinda, like, how it still just feels so so young, so I'm just super excited to see how how things develop in this space, over the next, I don't know, few years. The second one is, is self driving cars. So everyone in the everyone, you know, loves to shit on San Francisco, Cisco, and it's mostly mostly well earned. But there's a miracle that's happening in San Francisco every single night, which is from 10 PM to 5 AM. They're self driving cars taking passengers all around the city. And, like, normal people, like, instead of pulling out an app and order an Uber, you order a self driving car and just comes and picks you up, Takes you to where you need to go, drops you off.

It's, like, closed. I guess it's beta. So, like, it's not just public yet, but a lot of people have access to it. I've done a bunch of times. It's, like, Blowing my mind, and it feels like no one's talking about it that, like, after after 10 PM Yeah. Like, there's a lot of people just around self driving cars, like, with passengers going all over the place, and, like, no one's talking about it. And it's, like, it's kinda blowing my mind that This is happening. Like, the future of self the future is, like, here. Yeah. Yeah. That's a it's, like, honestly amazing that you think about that that Literally, a car drove somebody from start to finish. And the craziest thing is you start riding and after in, like, 3 minutes, you completely forget that no one's driving. It just feels it feels so normal, and then it just drops you off. And then you're like you turn around. You're like, bye. And then you're like, but no one's even in Car. And then did it feel weird, or was it like If I swear it felt weird for, like, 3 minutes. And then it's like and then you just kinda, like, you know, whatever. Look at your phone and just, like, kinda forget, Like, that that no one's driving. It's it's it's completely wild, and it feels like it's so crazy, and I feel like no one's, Like, acknowledging how how crazy it is.

It still scares the life out of me for some reason. I saw a video the other day of, Somebody was in 1, and a cop is like, alright, pull over. Like, there was an accident, and the cops, like, pull over. And the guy's like, I'm not driving. There's nobody in the driving scene, and it didn't know what to do. Oh my gosh. That's wild, though. I can't I wanna try what's the company? Is it Waymo that you you use? It's I don't think Waymo's allowed to take passengers yet. The one that can take passengers is Cruise. Okay.

Yeah. That's the one that GM bought.

They're they're you see them all around. You see during the day, you see the Waymo and the cruise, like, both of them cruising around With no passengers with no passengers, no driver, no passengers, empty cars, like, cruising around during the day, but after 10 PM is when Cruise is allowed to take passengers, and there's an app. It's just like it's just like Uber, Lyft. You just, you know, put in your destination. There and There's certain sections of the city it's allowed to be in, and it's it's quite a big area now. Oh my gosh.

That's significant. There, you know, I have the Ioniq five, and there's there apparently is the Ioniq five robo taxis in Las Vegas that are the same, but I don't know anything about Who, what app, or anything like that, but they look so cool, decked out with all their cameras and sensors and all this stuff. I gotta I gotta get in one of these at some point, but I'm also very scared. You gotta get that update. Somebody's somebody who's listening, slide Sky USB with the firmware for a thing. Here's a robo.

Yeah. Robo taxi.

That's right up my alley.

Cool. Shameless Plug anything you would like to plug. Just the new mux.com, the new the new mux the new mux brand site. Check it out. Yeah. Definitely check it out. Alright. Thank you so much coming on. We appreciate all your time and,

insight. I thought this was a fantastic episode. Nice chatting with you guys. Thanks. Yeah. Thanks, Dylan.

Head on over to syntax.fm for a full archive of all of our shows, and don't forget to subscribe in your podcast player Or drop a review if you like this show.