Supper Club × 70,000 Serverless Functions with Kristi Perreault of Liberty Mutual

Welcome to Citac's supper club. Today, we have a Christy Perrault here, And she's going to be talking about some epic serverless stuff where it is seriously, next level And in terms of how many functions and the types of coordination that's going on here. So, West just added 70,000 serverless functions here. I'm gonna Click bait, the title of this podcast. Yeah. Yes. We're gonna be talking big time serverless with Christie. This episode is sponsored by 3 amazing companies, Hasura, which is The Instant GraphQL API for your Postgres database, the Stack Overflow podcast, which is the podcast of Everyone's favorite website, Stack Overflow, and LightStep incident response, intelligent on call scheduling and escalation.

So welcome to the show, Christy. My name is Scott Talinski. I'm a developer from Denver. And with me, as always, is Wes Bos.

Hey, everybody. Excited.

I met Christy, Reactathon where you were doing a talk on, the serverless in the park. And, basically, we were just, like, walking to the speaker dinner.

I was like, hey. So what do you do? And then she's like, yo, I do 70,000 serverless functions. I was like, what? So, immediately, I was like, can you please come on? Because Scott and I have been talking about serverless for quite a while on the podcast, but, we have not talked to anybody that does it at a larger scale in a big corporation. So, Christy, thank you so much for coming on. Appreciate it. Yeah. I'm, super excited to be here.

So you missed the key part where I was actually super creepy, and you were checking into the hotel. And I was like, oh, I think I know you. I was like, you're you're Speaking at this event too. Right? And I was like, you wanna walk with me? So I was being super sketchy about it. So, you know, how to get on a podcast 101, just hang around the front desk For a bit.

Yeah. There you go. Who's that, Zach? What is that? Hey. Oh, hey. What's your yeah. No.

It's a good way. Like, that's the weird thing about conferences is that you show up, and you're like, I know nobody. And you just have to start making friends with random people that look like developers In the in the lobby?

Yeah. Literally. And the funny part is, like, I so I am not a React person. I don't know, like, much about React at all, and here I am at Reactathon, but they had an extra day that was serverless in the park, which was pretty cool. So it was kind of a a weird mix of, like, Folks that were really, like, front end React developers and then, like, a whole group of us that were just, like, very serverless focused.

And, like, in my line of work, Especially, like, I'm in I'm in serverless dev ops, so I'm really focused on that that bigger picture and, like, the infrastructure and supporting, roughly, like, 5,000

developers that we have. So it it was an interesting contrast, that conference. Okay. Well, I think that's a good lead in. It it would be maybe awesome if you could give us a little bit about, who you are and what you do for your career may be in a little bit more than than that, and we can, take it from there. Yeah. Definitely.

So I am a principal software engineer at Liberty Mutual Insurance, and actually you guys are getting a little bit of an exclusive here. I was Just recently announced as an AWS serverless hero as well, last week. So very exciting stuff.

So, sorry. I kinda know what I'm talking about, I guess.

Yeah.

But, I'm focused specifically on, so I work on a 4 person team. We do serverless DevOps. So we help support our 5,000 or so engineers, kinda get to the cloud as as easily, as cost effectively as possible, and we embrace this concept of of the serverless first initiative. So, you know, kinda getting there. We're gonna try serverless right off the bat, see if that works with your system, with your architecture. Have those conversations, hopefully. And in most use cases, it does, and then we kinda Help you get to that point. So I do everything from creating kinda starter patterns to get teams going for to consulting with teams 1 on 1, to now we're focused on, kind of the well architected review and well architected framework. So we're making sure that your products are, kinda Focusing on those pillars of cost, of performance optimization, of reliability, sustainability, so making sure that we are hitting all those points when you're architecting your applications.

So Liberty Mutual, you said 5,000 developers? Is that how many people are are working on

Stuff at Liberty Mutual? Yeah. It's it's kinda crazy.

So we actually joke a little bit that the the company tries to brand itself as, You know, we're a tech company that happens to sell insurance.

So a lot of folks don't realize yeah. Yeah. A lot of folks don't realize that all of our Our code, everything is done, in house.

We have some contractors, but we're global. We're we're in 30 different economies and countries, you know, worldwide, and we have folks supporting everything from our our claims and quoting systems to internal applications to, folks like me working on our our cloud infrastructure and development.

So, yeah, we do have, reaction developers, front end developers, application developers,

kinda everything, in between. Wow. So okay. So that was gonna be my next question is about, like, what does Liberty Mutual need to do with, 70,000 serverless functions? Like, is it Primarily just all about claims and and managing those types of, customer interactions, or is it, like you said, internal tooling? Or Is it all of the above? It's a lot of different stuff. Yeah. I'd say it's all of the above. Right? Like, there's definitely, like, customer facing applications, like, you know,

Mean, you want you you want a new, car insurance policy. Right? Like, there's a process to go through that. You know? There's a process to submit claims, to get a quote, to do those things. So we have developers that are supporting, you know, you as the customer, on the mobile app, or on your desktop, as far as, yeah, Like, internal tooling and systems that we use internally, for our employees to kinda do their jobs, for call centers to handle things, and then folks like myself doing more infrastructure DevOps kind of work, and then dev experience tools for for those folks. And it may it makes sense that these things need to be fast considering

Most of the times, people are contacting insurance companies. It's when something bad has happened or they're maybe not feeling great about man. There's so many things so many things I wanna I wanna ask.

Man, there's so many things so many things I wanna I wanna ask here. So before you had serverless functions, I'm assuming you had Mainframes are huge servers and and, like, lots of on-site people and lots of on-site hardware.

Is that how it worked? Yep. So we did have several data centers, I think in the time that I've been here. So I've been with the company for about, 5 years now.

I interned with them, and it was my 1st job out of college too. So, I I will admit I don't I I don't know a ton about the the pre, times before that, I was kinda lucky. Oh, yeah. Got to skip a lot of that and jump right into the cloud and and to roll this work.

But we did have and still do have some, data centers up and running that that we maintain. We do have some applications that are still, on prem. So, we have a I'm in a really unique spot where there's some areas of the organization that are really mature and are churning out, you know, Lambda functions, API gateways, working with AWS services.

And then there are areas of our company that are still Really on prem, still working on monolith applications.

We tend to acquire a lot of companies than we have over the past you know, We've we've been around for, what, a 110 years now.

So we have a lot of legacy code. We we've acquired a lot of companies and inherited tech debt, and some of those things too. So, You know, we have folks that are really experienced in in cloud and serverless and are super excited about the latest and greatest and then folks that are still kinda Trying to get on board with this this cloud idea. Wow. So what's if for those people, like, why why are you using serverless functions? There's probably people listening Right now, I say, okay. I'm building my application as a regular one. Why would I want to use serverless? Why are you at Liberty Mutual using serverless? Yeah. So some of the biggest Things for serverless that we really like. 1st of all, pay cost. Obviously, we're an insurance business. That's very important to us. So you only pay for for what you need and what you're using. Funny enough is also the Liberty Mutual tagline, only pay for what you need to be. That. That's that's it's so Oh, familiar.

That's great.

That's my favorite lead up right there.

It's amazing. Yeah. So we're doing it, like, even bare bones. Right? So With with serverless, you only pay for what you're using. Right? Like, if you're have an on prem solution, like, you're still paying to manage, to house all those servers, to have them, to maintain them.

Whereas with serverless, someone's maintaining that for you.

So I think that that's an important differentiation, like, you know, serverless Doesn't mean that there's no servers completely.

It's just that someone else and usually one of these cloud providers is managing that for you.

So that helps a lot with cost. That helps a lot with, performance as well. So it will scale up and down as you need things.

And it also helps with, you know, just just time and resources. You know, that's less that we need to worry about. That's less that we need to think about. That's less that we need to manage, and we can focus Way more on optimizing business code.

Makes that that makes that makes total sense, especially, like, you think about the the scale that you're at.

And some I'm sure there are some, code that only gets run once a year or something like that. Right? Like like, would it would it make sense to To run a server for 365 days a year so that 1 guy in accounting can get a report on people from Idaho.

You know, like, I'm I'm sure that exists somewhere within those 70,000. Right? So so the it takes, what, like, 30¢ or or 10¢ So run that once a year versus maintaining a server to run that app year round, is that the kind of thing that you're Exactly. Looking at here? Yeah. I mean, that's definitely part of it. And then some of it's the reverse side of Things right. Like, we have, you know, a catalog or a library being hit constantly,

for people single sign on or, You know, folks, pulling reports every day for stuff, and that cost adds up too. And you're gonna run some things, kinda into the ground and have know, you're gonna have issues with your physical servers and stuff, and you don't have to worry about that going down all the time. Right? Like, you can put that burden on someone else a little bit there.

And that's almost a a downside in terms of, we're not used to thinking like that. Especially developers that have been around for a while, we're not used to putting that responsibility on someone else, And that that is a scary kinda risk for a company and a business as large as we are. But for a developer,

you know, it's one less thing that we have to worry about or maintain, And we can kinda put that trust in the cloud. It's there when you need it too. Like, I'm thinking a couple weeks ago, we had a major almost a tornado roll through here, and, the power was out for days at at all over Ontario, and and, like, everybody was filing insurance claims. And, like, the 2 websites that you need the most, the power website and the insurance websites, a lot of them went down, and they were just out at the time that you need them. It's like, well, Like, of course, you can you can optimize for regular use. But when people need you the most, you can't just Immediately, oh, the power's out. Let's go rack up a couple more servers so we can get this thing going. Right? Right. I mean,

what I'm learning from that is, don't move to Canada. But, Yeah. Right. Right. But, I mean, you it's true. Like, you have these massive cloud companies. Right? And they have Multiple locations with different data centers. So, like, if if a tornado does hit a data center, you know, there's backups, there's Edge locations. There's other places to kinda default to and to fall back on. So there's some of that fault tolerance and reliability in place But, again, you don't have to think about or worry about. Yeah. That's pretty amazing. So okay. So we have a lot a lot of functions being are they all Mostly the same language that developers are writing here, or is it a number of different languages? Are different teams able to kind of work in their own,

the the own way they want or or or, like, what are we talking about for

programming languages across these functions here? Yep. So that's a great question.

So kinda in DevOps, and to focus on the developer experience, you know, I like to say that my job's a lot easier when, I can just support what the developer wants to do. So we allow them to kinda gravitate towards what the skill set is on their team, what makes sense for their group.

We see the most, TypeScript, is one of the most, popular ones, and Python's a really close second.

The biggest the the bigger thing is that we're, kind of locked into using infrastructure as code. So in order to deploy any Sources, you have to go through our pipelines. You can't just go out into the console and the fancy UI and create whatever you wanna create that way. It all has to be done through code.

So there's also a number of different developer tools you can use for that, which would be AWS CDK, AWS SAM, CloudFormation, TerraForm, the Serverless framework.

So there's a number of different tools.

We do support and allow for all of those. I would say that our most popular that has bubbled The top over the last year or 2 has been AWS CDK, because you can define that infrastructure in TypeScript or Python rather Then like a YAML or a JSON. Okay.

And I think that's an important thing to touch on here for our listeners because we hit this in the What was it? The episode about the no code, is that setting up these services, if you're just clicking buttons to configure something, You're gonna have a bad day when that gets accidentally wiped out because you're not gonna remember what buttons you clicked and what knobs you turned and what Keys you selected from a drop down, but, like, infrastructure as code, like you said, that means that you literally have,

Like, committed to your repo, you have a description of all of the infrastructure. Is that right? Is that what infrastructure as code is? Yeah. And I I think the the big thing for us as a company and a large organization too is, by by pushing developers through our pipelines, we can add checks and certain balances and security to some of those things. So, like, we have a process where, you know, we'll check your code and say, like, hey. You know, you're trying to push up this s 3 bucket or this I'm role, and you don't have this permission on it, or you haven't tagged this properly so we know who to reach out to, when there's tech debt, when there's a security patch, you know, if we notice you're spiking something.

So that's, like, a huge part of why we like To go through and use infrastructure as code and kind of force folks into our pipeline service.

So it it's big on the security front, on the tech debt front, And kind of on the the management and governance part, so that's why it's kind of a nice balance where we're gonna allow you to use whatever tool you're comfortable with, and whatever language you're comfortable with. But, like, you know, no matter what you're using, you still have to make sure that you have these policies, these tools, We'll see things in place. Oh. Mhmm. Never thought about that way. That's that's interesting. Okay. This episode is sponsored by Hasura, which is the instant GraphQL for your Data that's using any Postgres or Postgres family of database.

You don't need to write your GraphQL server by hand anymore Because all you have to do is point Hasura to your data source and let it auto generate your GraphQL API within minutes.

You have granular authorization baked in so you can secure access to all of your data sources, databases, or any other external GraphQL API.

You can even have your own GraphQL API sources bringing into Hasura and then add complex authorization rules Down to the individual row and column level, you can also simultaneously share GraphQL and REST APIs and create Styles of APIs from one configuration to support new and existing architectures.

Because let's face it, building and maintaining your own GraphQL servers It's time consuming, and it's difficult. And with HESARA, you no longer need to do that. You can save half of that time That you would be spent building your API and maintaining it, and you can put that into actually doing real work. So sign up for HasuraCloud's free tier.

It just takes a moment and you can get started. And you can get real time GraphQL instantly.

So head on over to hasura.infoforward/ Free trial. You can use the coupon code, try Hasura, and you'll get 3 months of Hasura cloud standard free tier. Only the first 100 people get this, so please, head on over there right now and give this a try. Thanks so much for Hasura for sponsoring.

So where does all this code live? Does it live in, like, a big mono repo? Are there mini repos? Is it is this the type of thing where everybody's kinda working on the same The more or you you just have a flow of

being able to plug in. So it's a bit of a mix. Like I mentioned earlier, there are some teams and things working On monolithic applications, we're trying to get away from that model.

There there are some folks that are doing microservices.

There are some folks that are just maintaining a couple refills. Some of them maintain 1 refill that has a lot of things in it.

Typically, you know, You would define your infrastructure based on, a stack. So, you know, like, we've done, stacks. We have, like, applications that have Your Lambda functions as well as maybe like a dynamo table for your storage or an s three bucket for your storage, and those can all be Be deployed up into 1 application if you wanted to, or you can separate them out into separate kinda stacks or groups, depending on how you wanna do it, what makes sense for your architecture, what you're trying to accomplish.

Maybe some things need a lot of uptime, like, you know, maybe you're really worried about the storage, and everything that's in your dynamo table, but you don't really care that much about the functions that run it. You know, if those go down, it's not a big deal.

So it it's kinda like trade off decisions in that space. But, as I said, like, 5,000 developers, I cannot possibly keep track of everyone is doing all of the time, and there is a little bit of everything going on. Yeah. I mean, what is the, like, like, local local workflow

Work like for these types of things. I know that's, like, kind of a question people get into with serverless functions in general. Like, what's this like, when I'm working locally with functions, I think that can be a bit of a barrier to entry for some people with serverless.

Is it is, like, the the local workflow all button buttoned up to, and just like a normal normal flow that you guys have going on, is it is it

I I don't know what I'm trying to get at here, but, Like, what what does the local development actually look like in this type of workflow? Yeah. So that's another reason why CDK is a pretty popular one because there are some really good tools there to to Help, with local testing, local development. So, like, you know, typically when you're using, AWS, all of these things roll up to a cloud formation template. So it's just 1 massive, like, YAML template that shows you everything you're provisioning and how it's configured.

So if you're using something like CDK and you're Finding all that, infrastructure in Python or TypeScript.

You can run a CDK synth command locally, and it shows you what the template looks like. You can go through and check it and be like, yep. This is exactly what I wanted to create. Like, this is how I wanted it configured.

And then there's some other, cool kinda tools and The things that along with, like, CDK, like, there is, like, a Python Lambda local, command that you can run-in your CLI and and run your functions locally. And you just define your events Send JSON files, and say, like, hey. I'm passing it this event. This is what I would expect it to get in the cloud.

So let's mock it out, kinda here locally.

And then the other thing too is, you know, we definitely use multiple environments still. So we have a company wide, like, sandbox account that's kind of a A free for all. People can experiment, POC stuff, push up whatever. It gets cleaned out, like, every 90 days.

And then you also have a dev account.

You can have a test environment, a non prod environment.

So it does help out a lot to push up to a development environment,

and kinda test from there as well too. Neat. You you have me wanting to try CDK.

Yes. Because, like, Scott and I, I don't know. Big big serverless you're not big serverless users. We don't have 70,000, but, like, we've we've tried a lot of them. And most of what Scott and I use our, products products as a service that sit on top of Amazon Lambda. Begin.com, Vercel, Netlify, all of these. They sit on top. And I I had a joke the other day I tweeted out that says, like, Serverless is so cheap, he says as he cries and, spends another $19 a month for a service. And, a lot of people are like, yeah. Like, serverless is supposed to be cheap, but when you're just hosting a little website, you have to pay $19 a month. And people are like, well, that's not Serverless, you're using a platform as a service, and there's added value there.

But there's sometimes I'm just like, this like, I don't wanna pay Whatever for this every this thing, like, service is supposed to be cheap. And I have stayed away from all AWS Direct stuff for probably 6 or 7 years since when I when I first tried it out because of how I had a poor experience with it. And and you're You're telling me about it here, and I'm looking at the CDK docs.

It seems like it's come a long way, and and now you're now you're making me wanna try it again.

Yeah. It definitely has. And I mean, even, we they actually released CDK v 2, as part of re:Invent this past year, like, the same week.

So they have. They've definitely iterated on it. And one of the things I really like about it is it is very community driven. Right? Like, it's developers coming together and saying, hey. I needed this, so I'm gonna Submit this construct because somebody else can also use this. So it's really driven by the priority of the community and and what they're lacking and what they're seeing and what they're needing.

And, you know, we're we're mostly an AWS shop. We still also support, you know, Azure and and GCP as well if If folks need it, we're definitely not as mature in those spaces, and the majority of our users are going to AWS.

So we've had a lot of Support from the vendor as well, which has helped us out a lot. And I have found at an enterprise level, at least CDK has been a really great tool to kinda utilize there.

Some of those other, ones that you mentioned are are great products and might be good kinda more One offer individually, but in terms of an enterprise as large as we are and as many things as we're covering, a platform is not gonna cover our use cases that we're looking at developer tools more than anything, but I highly recommend it. I've had great experiences with it. I found it's really easy to get started with. If you try something like there's Lair, I think it's literally called cdkworkshop.com.

Walks you through a hello world

super easily. I was just gonna ask you. Yeah. Is that your recommended, Like people wanna pick this up for the 1st time, CDK workshop? Yeah. Yeah. Start with the docs. Start with CDK workshop.

CDK constructs And CDK patterns are also 2, great spots to kinda get started as well, and those are, contributed from by the community. So, patterns will get you started with, you know you kinda know a basic structure of what you're looking for, what you wanna do. You can browse there and start with a pattern right out of the box.

And then constructs are just kinda those building blocks that'll help you build those things up, if you do wanna start a little bit more bare bones or aren't seeing something in the patterns that make sense. Oh, that's awesome. I'm just looking

like, they have TypeScript on the CDK workshop right out of the box, which I love. Like, it's always With a lot of these things, it's just like, here's here's a JavaScript example. And then if you want to use TypeScript, well, buckle up because it's going to be, like, 8 more things. And I I love it when TypeScript is deal with the output and then re

synchronize it. Yeah. TypeScript is probably the most popular CDK language. And, actually, if you do wanna contribute to that construct hub, you will have to do it in TypeScript. So it's it's the Okay. The one that people are rallying around. And Python is a close check-in. That one's that one's near and dear to my heart. That's my favorite. So

Python? Did you say? Typon. Yeah. And he said, Typon.

I was like, what is Typon?

I never heard of that. Yeah. That sounds bad.

You just combine the 2 of them. Type on.

Well, we we should also say that, like, if you are thinking like, oh, I don't wanna learn another thing is If you've ever used a serverless function or even a regular, like, express function, you'll feel very at home in all of the Lambda, code as well. Because If you've used Netlify functions or Vercel functions or any of these, they're all bay they're all built on top of the same signature, and you can move code from 1 to another, which is always why I'm I tell people, like, it's not a big deal to lock into Netlify functions or Gatsby functions or Next. Js functions because you it's just JavaScript at the end of the day, and very rarely do they have Specific stuff that you wouldn't let you move somewhere else. This episode is also sponsored today by LightStep Incident Response, The all in one incident response platform for DevOps and SREs.

LightStep instant response is built On the ServiceNow platform, which is used by over 6,000 companies, you pay for the services you actually use, not the number of the people on your team, which is important because you can scale your incident response team without adding to your bill. You get intelligent on call scheduling and escalations.

You can get the full context on your service health active alerts and who's on call. LightStep incident response immediately pinpoints issues And groups alerts with machine learning, reducing your time to respond. And this means less noise and more personalization for precise notifications where you can take control On declaring incidents, automatically notifying the right teams. And you can get unified incident response by seamlessly orchestrating alerts An incident triage with on call scheduling across Slack, Teams, Zoom, desktop, and mobile. So what you'll want to do is head on over to light step.comforward/ syntax, and you'll get a free 30 day trial. Listeners of syntax will also Receive a free LightStep incident response T shirt after firing an alert or incident.

So check it out. Are you in need of incident response? Check out lightstep.comforward/syntax.

Can can I ask what you went to school for? I I was just reading your, AWS hero thing here. I'm very impressed. So can you tell us about that? Yeah.

So I I like to joke because I feel like every time somebody in tech themselves, they're like, oh, I'm from a nontraditional background, and I I feel like I'm nontraditional now for going to school for engineering because, like, everyone else? I was like, this is cool. Like, why That's why I'm asking. Before code, and I'm just like, no. Yeah. No.

So when I I went to college, undergrad for for Computer engineering. I actually thought I wanted to be a, architect like, a building architect.

And then I did, like, a programmed for it, and I was like, I did not realize how much art was involved here, and I do not have this kind of vision.

So I just gravitated towards, like, the math and science.

And I knew I liked hardware as well as software, so I went for this computer engineering degree. And in my undergrad, I actually focused quite a bit on, like, circuits and, Actually built an autonomous vehicle with a team and competed with it What? Across the country. Yeah. It was pretty cool.