Spooky Web Dev Stories — Part 2

Welcome to Syntax. Today, we've got part 2 of a spooky web development stories. These are stories that people have sent in, and we're gonna read them off to you. Just horror stories of deleting the database or accidentally pushing something to production or, cringe worthy stories of web development where it all went wrong. Today, we are sponsored by 2 awesome companies. First1 is LogRocket, and second one is Netlify. We'll talk about them partway through the episode. Oh, I forgot to do spooky voices.

Oh.

How are you doing today, Scott?

Doing good. Tired. Tired style. I had meetings because I have meetings now. I don't know if you know this. I have meetings because there's people on my dev team that I we we meet. And so I've been just tucking into a microphone all day. Just meetings for the rest of your life. Yeah. I know. Right?

I just went to the dentist, and they use, like, a little mini sandblaster on my teeth. Have you ever had that happen? Yeah. Cool. My dentist is is, like, hypermodern.

They do X rays with this thing that looks like a literal 19 fifties depiction of a ray gun. Like, it looks just like that, and you just, like, click it on the side of your head, and you're done. It takes, like, 2 seconds. But the the cleaning, they do almost entirely via a, like, a water, like, a water sprayer. Is that weird? Like, a water blaster? I I was much more of that over the scratching with the little pick.

Oh my gosh. It's so efficient and so good. I have, like, a a receding gum on the inside because I used to have a tongue twister. So it, like, receded my gum, and I hate going to the dentist. Berate me all day long at the dentist. Oh. I can't take it. They don't berate me. They, use me as a shining example because I started flossing in the past, like, 5 or 6 years more more routinely, and I use an app to do it. And they so now they tell all of their other clients. They're like, he used an app. I gotta use an app. They're getting you flossing.

They're they're trying to get me to buy a, electric toothbrush.

So Oh, yeah. Dude, you gotta get an electric toothbrush. Sonicare all the way day, all day. I may getting ready for some sick pics in the future.

I think I'm gonna pull the trigger. We got we got the the latest Sonicare.

Got the the wireless charging. You just pop it in. It buzzes on your teeth. It tells you when you're done. It couldn't be Sanity easier, and my teeth have been way better for it. I'm doing it. I'm doing it. Yeah. I'm I'm an adult now. Alright. Well, so we've got the stories for you today. Banter. Lots of banter. You want banter? We got it. And none of it is about the weather, so you're welcome.

Listen to this. We've Scott was chit chatting with his bud, Jack Resider, over on Twitter. And as you know, Jack Resider runs the Darknet Diaries podcast, which is probably one of my favorite podcasts. Fantastic. So he he just talks about, like like, what? Like, security issues, dark web, breaches, things like that. And he just, like, tells it. My favorite ones are the pen testers warp people, like The pen tester interview. Into, like, real world situations, and it's just an awesome podcast. And he agreed to take one of these stories and read it, and, like, he stayed true to the story, but he just the way he embellished upon it and, is amazing. So we are going to play that for you right now, and then we'll come back with just regular Scott and Wes, which is not as good.

Have you ever thought about email storms? Like, for instance, suppose you have your out of office reply on and you send an email to someone who soon as you send them an email, their auto reply sends you an email right back saying they're out of office. But wait. You also have the same auto reply on too. So shouldn't your inbox respond saying, hey. I'm out of office too. And then as soon as that email gets sent to them, their responders comes back saying, I'm still out of office. Why are you trying to email me? And so on and so on.

Luckily, that doesn't happen because someone must have made a mistake at some point in the past.

But this is a story about an email storm that didn't end so well.

I was working on an app which included email integration, and I needed to test SendGrid's email API.

And I needed to see what was in the notification, which triggered when something was sent through the API.

So I wrote a webhook for SendGrid to send a post request, and that would forward the contents of the post request to myself and another developer by email using SendGrid.

So I sent the post request and got the email notification.

But then I got 2 more, and then 4 more, and then 8 more, then 16, 32, 64, a 128, 256, Wes, 1,024 emails just showed up in my inbox and so on and so on. It only took me a minute or so to realize my mistake. But by that time, my mistake had triggered more than 75,000 emails.

See, every time me or that other developer would get an email, SendGrid would trigger that notification email and tell me that emails were sent successfully.

So for every email sent to us, 2 more notifications were generated and also sent to us by email. This email storm ripped through our office like a category 4 hurricane.

See, our email servers were hosted on premises in our offices. So not only did I knock out email for everyone in our office, but the exponential growth of inbound emails dust the office's Internet connection for the rest of the day.

Immediately, we called the IT support company who managed this email Vercel. But every time they would try to remote into it and clear the queue, the remote session would just freeze up. Over and over, they desperately tried to weather the storm, but they couldn't.

Eventually, they had to get in the car and come do an on-site visit and fix the problem. And for a few days after that, our offices still didn't have a functional email server.

This was a nightmare scenario which has haunted me for years after.

I'm Jack Rhysider, and I hope you enjoyed the story.

Thanks, Jack, for, recording this for us, by the way. It's, fantastic.

Yeah. Definitely check out Darknet Diaries. I really appreciate it.

Let's get going with the different stories.

In almost all of these, we didn't actually mention the name of the company or the Vercel. It's just that we get no one in trouble. But this person, it's kind of important which company it was, and they said it was fine. This one is called Dirty Dick's JSON, which is from Dick's Sporting Goods. I worked for Dick's Sporting Goods for several years. While I was there, the structure of the engineering tier was a bit odd to say the least. The back end engineering team and the front end engineering team were completely separate teams. They didn't have any direct interaction and didn't even touch some of the code Bos. Due to the separation, my team JS front end engineers often had no clue how about the back end was actually set up. It sounds ridiculous, but it's true. Well, one day, I was monitoring XHR requests on the website and saw an odd JSON called banned words Scott JSON get loaded in. So I opened up that bad boy to see what was inside. And to my surprise, it was hundreds of words that I simply cannot repeat on this platform.

I actually have used that JSON file before. There there is a file on the web of every bad word you could possibly ever imagine.

You think dicks is in there? I they probably had to do a custom. They probably had to to tweak it.

They had to really fork it and release their own bad words, that JSON.

I was absolutely shocked. It turns out the back end team was using this to filter out search requests that had bad words listed in the file. I couldn't believe what was in this list was loaded into the client side. What if somebody found it? Well, a month later so I didn't have absolutely nothing about it.

Sure enough, a month later, I pop open Reddit, and I see trending on our web dev, Dick's Sporting Goods, bad words Scott JSON. It was an interesting day

at the office.

That's very funny. I have a a a close relationship with Dick's. I think I've been there a bunch of times because I bought all my workout equipment there on Black Friday. So I've been I pop into DICK'S all the time and and, get plates or that's a cheap workout equipment there. So shout out to DICK'S Sporting Goods if you are looking to buy a squat rack on the cheap. Ours has held up really nicely.

So advertisement for DICK'S. Shout out to DICK'S. Alright. Next question. CMS or next story.

CSS disaster.

I moved it from design to engineering. So I had some code experience, but never really dealt with environments.

After a grueling few hours of setting up the AMP stack natively on my machine, httpd.conf

and all, I got XPression ESLint up and running. AMP is Apache, MySQL, and PHP, not the Google

attempt to control the web. Not yes. Yeah. Yeah. AMP pages or whatever that is. Shout out to expression engines too. I haven't I haven't used that in a little bit.

The last step is to replace some global Yarn, the local points you did the dev URL instead of prod URL, but not yet being trained to pay close attention if I was looking at the site .dev rather than site .com. I swapped them out on prod.

The site completely tanked.

The marketing site was the main login for the app, so support blew up. The company basically stopped. Senior leadership team was doing rounds trying to get to the bottom of it. Because the break happened via fields inside of the CMS, the CMS was down JS well. A senior eventually had to go in and manually change the values from the database to get things up and running again.

Pure, live edit the database style. Again, this is my 1st day moving into engineering. I was mortified, and soon they were going to boot me right in back into Design Forever. Yes.

I think this one's very relatable. I've done this. I don't know if I've done this. I feel like I've done this in WordPress. I've done WordPress. Maybe even Expression ESLint. But, like, there is, like, a lot of specific situations when you can edit something in a WYSIWYG and bring down the whole site.

Like, that's very scary. You want me to tell you a good UI for this JS I've done it on WordPress before. You change the site URL on WordPress, and then you save it, and then it's just white screen. And, like, you can't you have to go into the database to fix it. Yeah. My router has an awesome UI for this, and when you hit save and it makes Bos possibly a breaking change, like, you change the the the URL of or you change the firewall rules or something like that, it will save.

And then if it doesn't get a ping back from the browser within a minute, it will revert the change back because, obviously, you went missing to the browser,

and, it will roll back that change for you, which I thought was was awesome. That's neat. It's sort of like that, you know, when you change the resolution That's it. It's like, do you want to keep this resolution?

Yeah. Or Yarn am I upside down right now? Yes.

Next one is the Oh Node Hotel. Oh, Node hotel.

I accidentally mounted an endpoint in production that pointed to a staging route for hotel reservations.

I think about 400 people made a reservation that was wiped after a database cleanse. That was a bad day. Not sure how I get my job.

Oh my god.

Next Node, FTP.

Once in my career, I needed to create an FTP user for a client's website.

Due to some wonky permissions, I had set up their home directory to the root of the site so I could FTP into the web root. Once I was done, I removed the FTP user and left the also delete user home directory.

About 15 minutes later, it dawned on me what I did. Thankfully, the server host had a very decent backup. So so what they

they'd left to this bit that deletes the user's home directory, but they It's when you have a app. App? You deleted the FTP user. So what did it do? It, like, went to the app then or something? When you add a FTP user, usually what it does is it gives that new user its own, like, subdirectory. But he created an FTP for the root. And then when you delete it, it says, hey. You're getting rid of this user. You also wanna delete their files, but it was the root. I got rid of the the root. Gotcha. Seems like there should be some production for this. Yeah. Rough.

Next 1, push notification hell. A new to me codebase was working on some push notification issues, and there was a difference between the local setup and the production setup while doing that. I sent a dozen increasingly more frustrated test push notifications.

Turns out, it wasn't just me that was getting the messages, and the entire team was notified.

I notified everyone on the team that I was just getting Wes messages out. Shoot. Not a big deal. It's a tech company, and s h I t happens until the tweets start coming in and emails from investors that not only was it the internal team, but the entire user base.

Woah.

Last time, I I swore, wrote witty, despising push notification messages. I think that's sort of a trend of these is don't ever write a swear in any of your console logs or images or You just never know.

Man. Yeah. You never know when people are gonna see it.

Back in 2005, I worked on a web app for a DVD retail chain, moving on from the server plus client apps required by dial up to a single full time online app.

On the launch day, I discovered that the central d b sync from the client d Bos was broken.

Nobody tested data integrity.

Yeah. Yeah. Yeah. Oopsies.

People from branches just called. We see people from different cities in our system and ours are missing.

Hashtag ghost.

Node little hashtag there. I spent the day VNC ing 2 local PCs, dumping the databases through MySQL admins, and compiling the central DB manually, all that after a brutal 2 day launch. All of those 3 days, I listened to David Bowie's album, Let's Dance, 70 times. Let's dance.

Yeah. He probably whenever you hear that music, you just it's true. I remember I have that with, Owl City. Remember Owl City? You know what? Actually, back when I was trying to become a musician, I had a lot of contacts with the Owl City guy because Really? We were doing similar types of music. And there was a website.

It was called the if this is not what it's called, let me, let me Node. The 60 you might not have ever heard of this. It's called the 61, all lowercase letters, v 61. I don't know if it still exists. It looks like it's closed. But this was like a music streaming app where people would upvote songs. And so you could enter them under hashtags and people upvote them or whatever. And the Owl City guy, he, like, kind of blew up on there first and foremost. Wow.

And we were in the same communities of making songs, and he sent me a couple of messages because he liked some of my music. And then he got really popular and I never heard from him again.

That's awesome. I I just associate that music with learning to code Myspaces.

And every time I hear it, it brings me right back. Yeah. That's very Myspace era music. That's funny how music can do that. Next one is called bad words again. So this is another one about my words.

I wrote a bad regex to check for bad words in a Node, and it alerted the user humorously of bad words, and they said not to say it. Unfortunately, it matched substrings in words and alerted banking HR managers that they were saying ass and t I t when they were writing assistant and title.

Oh, I remember when I was a kid, I was writing a Microsoft Word shopping list for my mom, and one of the things on the list was fruit cocktail.

And every time I wrote it, this huge thing would pop up on our my computer, and I would be like, what? Like, what's bad about fruit cocktail?

What about

JS it just fruitcon? No. It just it just, like it was, like, matching part of the word. Oh, it did. Wasn't it wasn't the software probably wasn't smart enough to I thought I Or maybe it's because I was typing it. As I was typing it, it realized the word, and I hadn't had time to finish it.

That's why we need

AI. Well, we need it.

This next story is Mo Money. As a junior dev in my 1st web dev job, I left a variable as 0 instead of 1 in the payment gateway.

This stops certain cards from being allowed to pay. No one noticed for a year.

Yeah. I got a final written warning and confidence destroyed.

Don't assume it works. Prove it does, which is a great point. Yeah. Check that. So this is the first time I heard of someone actually getting canned for a mistake like that. A year.

They left it in for a year. And how did Node knows for a year? That seems like operational issues. Yeah. Yeah. That's crazy.

Next Node, Bos Ackwards.

My first job out of college was at a prominent review and feedback management Scott.

If someone bought a product from one of our clients, we would send them an email asking them for feedback and a review. In other words, we spam people with emails.

We were having a formatting issue in some of our templates. A senior dev pushed up a change, and we marked the bug as fixed. The next day, we received several frustrated calls when they found out all their emails are being sent backwards.

Every single word backwards. Woah. And so example, dear customer became r e m o s. It's it's like Scott just, like, customer dear. It's the actual word was backwards. Remote suck read.

Remote suck read. The commit was reverted, but thousands of emails had already been sent. It was an interesting day, and I always remember watching our senior dev furiously revert code for the 1st time. I love the show in y'all's course.

That's I wonder what that was. Do you think that was like a what do you think? Like a CSS right to left?

I have no idea.

Yeah. I don't think CSS right to left ESLint? Does it reverse the No. I don't think it does. Well, I guess it does. You know, I don't know.

I don't know, man. I don't know. I have no idea on this one. Alright. Next Node, taxi coding.

I made an Electron game to run on in store screens in 5 major brand stores for the launch of a new flagship device.

Installation happened the night before, So I went to the 1st store with the crew to oversee.

I found a breaking bug and pnpm the next 12 hours coding in a taxi next to the client on three g. So much pressure, I couldn't remember if a JS filter removed or kept values in a true condition.

Dancing with the Devil with 5% battery and 3 docker images to build and publish, managed to find a pub at 8 AM to celebrate the job well done. Yeah. If if yeah.

I've had actually my share of pullover. I had to once one time I was on the ski hill, I was on Winter Park Mary Jane, and I had to ski all the way down the hill to my car to go get my computer to fix a bug was in production at that very moment that I had gotten an email about while I was on the chairlift. So, yeah, shout out to you. I had a similar situation. Man, I this one, I thought about all weekend. It's just I just kept thinking back to the poor guy sitting in the back of a cab that's probably just sitting there running

and fixing it. Wow. That's that's rough. Glad that you can find a pub at 8 AM. That would not happen in Canada.

Next one is bad e n v. Not sure if that's the kind of story you're after, but in a company I work for, we had an absolute madness of packages with weird dependencies to each other.

So every time we had a switch between projects, which is regular because we worked on 6 different sites, we had to yarn unlink r m r f, the whole Yarn link directory, yarn install, and yarn link again. This, on average, would take at least half an hour out of each dev's day. Oh, and another nightmare that there was was there JS no hot reloading for the sites. Every time you made a change, even if it was just a change of margin and a SaaS issue, it would take 40 to 50 seconds to rebuild.

I miss the Ruby SaaS times where the refresh took less than 10 seconds.

Yeah. Let me describe to you a nightmare. Here's a nightmare.

Yarn link. That's it. That's the nightmare.

That's the whole nightmare. It is very hard, and you never know if what you're doing is working or not working. It's it's very frustrating.

I don't know if you've done any Yarn linking or Npm. I have. But in a lot of cases, if I have to edit a like, a module that I'm using, I just go into node modules and start hacking away at it and then start hacking. Yeah, it's usually you're working on the bundled version, not on the main version, but it's it's a pain.

That's exactly what I do. Yeah. I just had to Node because yeah. Especially if the parent repo is like a mono Deno, and then you're just, like, asking to, make yourself very frustrated.

Those are probably the biggest pains to me is, like, when development is slow. You're like, I can code and think much faster, but this thing is getting in my way, and it's I'm so slow and frustrated with it.

Yeah, I know. I'm spending some time implementing Snowpack right Node. And I'm just I'm very looking forward to the future where I save something and it's just updated that next instant.

Alright. Next story is log in as I once left some debug code in when checking an error for a user that logged everybody into his account.

So that account has been compromised, I would say. I think that's safe to say that account's been compromised.

Everybody gets logged in JS Oh. That sounds fun. Sucks. Email subscribers plug in. So my task was transferring a WordPress site from 1 server to another. We warp pretty rookie, so we're setting it all up manually.

We set up all the plug ins and then imported the content years' worth of pull posts. Unfortunately, we didn't notice there was an email subscribers plugin that emails a subscriber every single time that a post is published. So this is pretty common. People say, yeah. I wanna get an email when there's a new blog post. There's a plug in. You throw it in, and it works like that. So when we imported 100 plus posts, it ended up sending an email for every single post to every single subscriber.

Oh, you think it would detect that? There's a lot of these. It's like, oh, man. There should be Check some some system downplays here.

Alright. Next 1. 1 in the 300 chance of the c word. I once wrote a pronounceable password generator. The theory being that it generated passwords made up of random vowels combined with consonant pairs would create a string, which wasn't a real word, but could be pronounced and therefore would be easier to remember.

It was used during the password reset flow of a web app whose demographic skewed towards very non tech savvy, many of which would have problems even copy and pasting.

It was in production for several months and had generated thousands of passwords before another dev received a new password, which had contained the c word. And then he wrote, yes, that c word.

One of the consonant pairs I had used was n t.

The function had roughly a 1 in 300 chance of including the c word somewhere within the generated string. 1 in 300.

Yeah. Not great odds for the first the fucking c word into your your generator. That's pretty good. Scott some I have random,

store names in one of my courses, and it just takes, like, an adjective and a couple vowels.

And, someone got angry, unsightly women.

And, oh, jeez. Like, you should probably check this. I'm like, yeah.

I probably shouldn't have that possibility, but it's also random. So Yeah. Like, what yeah. Yeah. How do companies like Heroku or you know, how do they deal with that stuff? I don't know. That's that's a good question. Like, Netlify does it as Wes, or they, like, generate random possible pairs. And, like, how do you go through all the permutations? Because there's millions of possible combos.

I don't know. That's that's a good question.

Next Node, production target. When I joined as a new grad, I was asked to learn about load testing and load testing our staging servers. I learned about Gatling, which I just looked that up, and that is an open source load testing framework.

And after I figured out how it works, I ran my Gatling load test script. I think I set it about a 100,000 concurrent users.

That's kinda cool, actually. Maybe we should do a show about that. I think that'd be kind of a cool tool. I'm interested. Yeah.

3 minutes later, we all start getting alert emails and calls saying someone Wes trying to take our servers down. I also didn't realize it was me until I noticed that I set the production script to be target the production servers. I did not get fired, thankfully.

Hey.

That's great. And let me tell you, you also will not get fired if you use one of our sponsors. I'm talking about LogRocket because LogRocket allows you to find bugs in your application very easily and very quickly by giving you a session replay. Now what is a session replay? Well, it's a video scrubbable video that gives you the network tabs. It shows you the user's mouse Wes they clicked on, how they, were able to do all this stuff. So this way, you would have seen potentially somebody firing off this, Gatling and seeing that it was your mouse. It was the the the killers coming from inside of the house, that whole thing. So you'll wanna check out LogRocket at logrocket.comforward/syntax, and you'll get 14 days for free. Again, you get a scrubbable video replay that includes not only the video of what happened, but also the network tab and the console and all sorts of fantastic things to help you find and solve those bugs.

So thank you so much for LogRocket for sponsoring this spooky episode. Oh, this next one is a happy SEO ending.

There are definitely Wes things to read from others, but the worst thing in my career was to block all search engines from Europe's largest cooking community by accident. Oh, yeah.

In the good old days, let's deploy those changes meant to log in to a server by SSH, run 30 commands, and check to see if everything is fine. I remember those days, Wes. I remember those days.

There was no visible problem on the website, so we all left it for the weekend.

Yeah. I know not to deploy large changes on a Friday.

The source of all bad things that happened was the idea of a colleague to remove our testing installations from Google.

Sadly, he pushed it to the wrong branch.

As I was already thinking about the weekend, I skipped the check every file of all commits if they're reasonable on the checklist for each go live because, you know, it always works. Yeah. This this kind of happens to me too. Like, you get complacent with things working all the time and that you lack your guard a little bit and something breaks.

So setting deny all in the robots dot t x t did not only lead to 50% loss of visibility in Google measured by Sysrix and millions of pages removed from the index, but also lost revenue from ads.

If you block Google's crawler, it will not deliver content related ads on websites, and this burned 1,000 of euros per day.

Luckily, the SEO saw that something was wrong on Monday, and we could fix it very fast. But it burned lots of money, and I was in fear for a full week until we figured out that it was a good accident.

Google started to crawl the new content that was better than the old one, and rankings had been higher after 2 weeks than it had been before. Impressive. One line of code, a lot of money burned, a horrible week to wait for new search result measurements, and the day that we started to implement automated deployments with security checks that prevented some of the possible mistakes.

So you just ran the, like, probably the largest SEO test

of the time Yes. Which is hilarious. I'm glad it worked out. Next one, I call it just oof. I couldn't even think of a name for this one. Horror story. I'm surprised you named all these. I I named a whole bunch from last time. I did not. I didn't get a chance to name any of these, although I think it's probably for the better because your names are awful.

I got the names. You got the ad transitions.

Yes. So this Node, horror story. I wrote masquerade, a com command line tool to anonymize databases with. In one of the earliest versions, I did an array merge, but got the order of the arrays wrong. The content of these arrays was database connection config, the one from a config file and the other from CLI params passed to the binary. I meant for it to have the CLI params take precedence so a cron would run to anonymize a replica a Oh, we found out a few days later that we had to parse order confirmation emails from said grid in order to retrieve the data.

I've since learned not to run a process like this on a production machine, and now we all use GitLab runners with a scheduled task to anonymize.

Those ones where you can screw it up by accidentally typing the wrong thing or hitting the wrong button or running a command in the wrong directory.

Spooky. Spooky.

I fell in, and I can't get up. Alright. This Node this one is, yeah, pretty rough. At my 1st job, I worked for a company that built a system for monitoring the vulnerable folks in air homes.

It worked by motion sensors placed on the wall of the house. I pushed a change to the charts that I thought fixed a bug. 2 days later, we've received a call that a woman was convinced her elderly father had fallen in the bathroom.

She had called him in a panic at 1 AM. Oh, gosh. That's scary. Turns out, because of a time Node error, the charts were missing data from 12 AM to 1 AM.

Certainly, my most shameful and costly bug. Yeah. That is frightening. That's very frightening, to know that you could write some software that that would have that kind of consequence.

Very, very scary. Yeah. Like, that that's that's real, man. I can't even laugh at that one. That's real. I'm glad that Right. I know. I'm like, that's like, that hurts. Like, that person was probably so super rattled. So good story. Yeah. For real.

Pretty yeah. That happens. Next TypeScript.

I used to work for a Scott up company that had an ICO back when it was a thing, TM. So ICO JS initial coin offering. So when, like, a new type of crypto comes out, they have an ICO where people can buy in.

Launch day is coming, and we're preparing for months. The smart contract is already out there on the e Ethereum network. No going back. Our website had a countdown and everything for all the investors we spent months acquiring and the potential big ones we had on our list. Big day finally comes, and I finally get the wallet address just before launch.

It's a QR code I uploaded to the website in time just, and the countdown and excitement.

We did it. Not long after, I'm informed the QR code was a placeholder, and nobody was able to pay.

Yikes.

The crypto craze died right after that. So who cares about how much, quote, unquote, money we lost in those few hours? Mhmm. The company went under the next year before I left.

They still owe me some money.

So that's, like, under the same yikes as deploying that last one where you have the gambling and you forgot the Boolean of test of true. There's there's so many, like, money ones like that. Like, lots of money lost or potential for money to lost. I still do. Like, when I launch my courses, I still even though I I test on Stripe all day long, I still generate myself a coupon.

I set it for, like, a dollar, and then I buy the course myself just to make sure it actually works with a real.

Buy the course Vercel. I refund Vercel. Whatever. Yeah. I do that. R m r f. My Halloween horror story. Friday afternoon in the office working on a Magento site, frightening enough in itself. I love that note that, this author added because it is frightening, working on Magento.

But anyway, we used to do beer Fridays, but I'm a woman who doesn't drink.

So I sat at my desk drinking a glass of Prosecco and ran r m r f r m hyphen r f to, for those of you don't know, that command removes recursively all directories of which you specify.

Command to delete a folder so I could recompile and accidentally left out the forward slash. Somehow, I deleted that the company's entire development server, which contained probably about 60 to 70 websites.

The poor DevOps guy spent the entire weekend trying to recover it by some miracle.

I still work for the company 5 years later.

60 to 70? Oh, my Node. Yikes. I would delete, like, Node development site and freak out. Sixty or 70 production sites. Yay. Yikes.

Next one we have is never on Fridays. I work for an ecommerce agency at the time. I made some small updates to our site. I forgot exactly what it was, and I thought I'd update the project's dependencies at the same time. So I made an update and deployed on a Friday afternoon. I know. I know. Yeah. So this is what I refer to as a YOLO update, where you're like, 400 updates to my package, Jason? Sure. It's fine.

As you should after any deployment, I went ahead and checked the site for any possible issues. Searched for some products, added it to the basket, went through the checkout, all seemed well. I go through the test a second time just to be sure. I deployed on a Friday after all. Better make sure it works. Right? No issues spotted. I closed up my lap laptop and headed home for the weekend. Monday, I arrive at my desk. The mood in the office is more somber than usual. I'm grabbing whatever preferred choice of caffeine was for the day, and the team lead pulls me aside. He does not look like a happy chap.

Props to whoever wrote this. This is very well done. Happy chappy. Can I see you in the meeting room, please? Oh, no.

It turns out that the update updates I so bravely deployed caused a basket issue that prevented customers from adding more than 1 item to the basket at any time. Of course, I never spotted it as I only added 1 item during my test. The issue was not reported until Sunday.

I never got told how much money my this cost my client, only that I really, really did not want to know.

Deployments had been done in pairs from there on out. Never deploy on Fridays. Oh.

Yeah. A lot of these people never deploy on Fridays. Or have a, like, a good set of tests would have or a pretty basic test would have caught that. I'm sure they have a test for it now.

Yeah. Yeah. That's how you learn.

Next 1, $1,000,000 scramble. Hey. You know what? I have, like, $1,000,000 scrambled eggs right about now. That sounds good. These aren't scrambled eggs. Okay.

We were demoing a product Wes were building to a potential client, our 1st ever demo of the product, and it was for a deal worth 1,000,000 of dollars. So we got a $1,000,000 deal here. A a DevOps engineer just so happened to be going through what they thought were our old clusters and deleting them.

Our deployment got deleted about an hour before the demo. Oh. It was the maddest scramble I've ever seen to try to recover. Yeah. Yeah. Yeah.

So, if there's any major appointments, don't start deleting stuff. Just don't. Just wait. Just wait. You never know what you're deleting. Know. We literally probably had, like, 3 or 4 stories like that so far.

Speaking of deleting production, I deleted the production database thinking it was my local because my database client and prod local looked the same. The most recent backup was at 10 AM. I deleted it at 4 PM, and there was around 600 orders we had without a backup.

We managed to recover 99% of these using email logs, spreadsheets, and bulk inserting. The process of deleting to recovering was from 4 PM to 2 AM with 4 engineers.

To this day, I use 2 database clients, 1 for local database and 1 for production staging, and I will only use a read only access for production. Thankfully, it didn't get me fired.

My work handled it very professionally or understanding.

Spicy. That's a spicy one.

Sanity. Spicy.

Alright.

500,000 concurrent problems.

That's a lot of problems.

A couple Yarn ago, I was working for a well known company that offers a very widely used website chat widget.

I was working on a few subtle UX changes to the widget that would give users a much more intuitive interactions in the behavior of the chat.

After emerging and deploying the changes, we started to see a couple small errors.

Turns out that anytime anybody clicked on the chat widget, even if there was a friendly message notification inviting them to join a conversation, it would just disappear completely.

At this point in the company, we had around 500,000 concurrent users, and around 5% of them would engage with the widget. Upon my discovery of this bug, my heart sank into my stomach and a cold sweat started. I had deployed this and then gone into an hour long meeting.

Resolution. Early on, the company decided that deploys and rollback would be easy and instant.

Luckily for me, rolling backwards is instantaneous and only required clicking a button. I made sure to test my code more thoroughly from now on.

Do you have any rollback set up in your in your deployment process? Because I do, and it's it, like, saves my butt all the time.

Don't on my DigitalOcean.

Like, I could just roll back a commit and and redeploy, and it would fix it. You would have to redeploy it. Yeah.

But no. I I think I I would like to have something like that.

Yeah. I host on a Meteor Galaxy, right, because it's the Meteor host. And they have, like, just like a history of every version you've ever deployed.

And so at any given point, it'll tell you, Node, use the current Vercel, whatever this is, the past version. Any given point, if I have a mistake, I just click the old version and click, you know, go back to this Node. And I can go back to any prior Vercel, and it just redeploys it for free and, like, instantly.

It just cut cuts over the domain. Now, like, Netlfi's, I they'll all do that as well because they just have multiple versions of your app

instead of one where they You can take different update it. Yeah.

That's good to know. Next Node, deleting a government website. Many years ago, I worked for a company that ran the state of, and I blanked this out already, Wes website. So, this person asked to for us to anonymize it because they're worried about the government getting litigious.

I that's such a good word.

Litigious. I was trying to debug an error on our staging server, and anytime I touch the files on the server, the changes didn't seem to be Wes the website was there.

I bounced the server thinking cash, still nothing.

Finally, my office make mate goes, hey. Did you know the whole website is down? Which website? The production website. It's 404ing everything.

Face palm.

Oh, and this this sounds like this was done way back before version control and things like that, where if you if you deleted it, it's gone.

Also, r r m r f is just so dangerous. Didn't you have something where you use some of the the I feel like you had this picture? It's trash or something. Install

dash g trash dash CLI and that will put it in your trash instead of r m. I never use r m r f, just because it's scary.

Interesting.

Next story, you've ruined the surprise.

My worst production related offense happened while I was at a Scott up in LA. They were trying to become a more modern Evite.

What is Evite? Do they oh, they do, like,

they do, like, gift cards and stuff like are not great. Like It's like you if you get invited to, like, a birthday party or, like, a wedding, you can RSVP, and, like, it comes in over the email. It was pretty popular, like, 5, 10 years ago. Did you get my Evite to to the Halloween party? Did you get my Yeah.

Basically, if you were having a party, wedding, whatever, you wanted to have a digital experience, you would use us or a Paperless Post.

Never heard of it. Paperless Post is another good one. So, anyways, I had this pretty amateur rails web app API that we had built, typical MVC architecture using active model. We had a bunch of callback methods into models themselves that would fire on create save events.

I actually thought this was pretty slick at the time, and, yeah, that's pretty sick. We had just deployed a change to run a bunch of DB migrations and it went off without a hitch. We then had a migration script that would walk the DB and upgrade each record with default fields.

This script failed to skip the app permission, the callbacks, and basically firing off emails left and right for every single event 10 d in this system. Oh. Shooting off emails.

Yeah. So, oh, that is so funny. That is very funny. So for those of you who who aren't picking up, basically, they had a hook on DB change that would send an email or something, and their crawling script just fired off against every record.

Now imagine the horror when you're in the middle of America planning a surprise birthday party for your father's 50th with over a 100 people, and you're waiting to announce until 2 weeks before. And all of a sudden, a bunch of emails start going out to everyone attending.

Or you are a bride planning a wedding and you send out a bunch of emails to your attendees ESLint prematurely.

Or you just had a funeral for a family member a month ago, and you send out a please attend TED's funeral email to everybody again. Oh, that one is very painful, very painful.

It was all out pandemonium.

Company was freaking out. I won't say for certain that this was the final nail in the coffin for the company, but it certainly didn't help.

Oh, gosh. To this day, I'm hypervigilant when it comes when and where back end Node sends emails. I hope this is what you're looking for. This person may have single handedly killed Evite.

Oh, that's that's even why, like, when I use, like, a local development, I'll make sure that I'm using, like, a mail catcher or temporal email or something just because if you accidentally trigger 500,000 emails, that's kind of a nightmare to come back from. Kind of a nightmare. Yeah.

What's not a nightmare, though, Scott?

Is it Netlify, our sponsor for this episode? Netlify is certainly not a nightmare. Wes, I don't even know why you put them in the same sentence.

It was actually funny. A little bit ago, you you started one of these, and you're like, you know what? I thought you're gonna do a Netlify ad transition, and then you, like, started reading the next story.

So you've got me. So ESLint, now at Netlify.com JS the fastest way to build the fastest sites JS in you can deploy your front end code on this thing and just with a git push. Look at TypeScript and Node git push. Your entire site builds and is constantly deploying anytime you push a commit to a specific branch, all for free, easy to use. There JS awesome, awesome, awesome features that are added to Netlify constantly over a 1000000 developers currently using Netlify. Isn't that crazy? 1000000 developers? That's a lot. A lot of people using this thing. There is just a ton of features that allow you to really, really gain productivity like crazy. I host my site in Netlify. I know Wes host his site in Netlify, and it is fantastic.

You can do all sorts of things like serverless functions or or Deno based analytics, which are actually very, very nice compared to normal script based analytics. Node there is even an identity platform that allows you to create a login user account on your front end code site, all just through Netlify itself, all this and more. So check out netlify.comforward/syntax and see what everyone's talking about because trust me, this is the place to host your front end code. It is so dang easy and simple. You know what? Now if I have it, I just checked. If you go to one of your commits

and you can view that commit, it's already built. You can just publish it. You're gonna roll it back if you accidentally screw it up. Hey. Cool.

Mister d hole.

D hole?

One time, I was working on a client site and running tests to try to debug some email template issues going out from my CMS.

Again, I had a Mailtrap installed locally, so no real emails got sent from PHP's mail function. Okay. Okay. So good. Okay. Okay. Unfortunately oh, the module I was using uses its own SMTP implementation and bypasses the PHP mail function. So it was funny when the client called our office and asked if my business partner if he thought his order for mister d hole was real or not.

Guys, stop Stop putting bad words in testing.

Oh, it Wes not so what the what's kind of lost here is that,

the the the word is is actually spelled out. They're not shredding d. No. It's the same word as the sporting goods store from earlier, which we were allowed to say.

So that that is much funnier to me that it's, another bad word. It seems hilarious that so many people have made this very same mistake.

Very funny.

Also, these kind of things are are a little more harmless than, like, you know, know, sending out a notification about some of these funerals. Not not exactly harmless.

Alright.

One expensive race condition. This just happened today, so it's fresh in my mind. Oh, fresh one.

I had built a samples request wizard for an international flooring company on WordPress using jQuery steps and Sanity forms. At the end of the wizard, you submit the form by clicking the finish button on the wizard.

My code in a WordPress template catches the form finish event, submits the form, then location Scott h ref equals thank you very much, page.

I come to find out that I had built a race condition that sometimes made the page change without submitting the form.

Client misses 60% of their leads. Boss is furious and chews me out over Slack.

So glad I wasn't in the office. Tester forms in all browsers at all network speeds. So the form was firing an event early and submitting the form without collecting all of the information.

Yeah. People people don't like that, especially any sales teams that need to keep track of their their, their people. They don't like that. I don't like it when you you take away their leads. Like, huge loss in sales. Like, sometimes I submit a form on a website that's, like, a local business. I'm like, I'm never getting a reply to this. And sometimes you feel that way. Sometimes you're filling out these, like Yeah. Junky forms. That was rough. Like, this is definitely a waste of my time. A perfect example. People always ask, like, what's a race condition? Like, that is a perfect example where they hit submit, and then the the the code after submit the form was just window Scott location, and they didn't await for the submit to come back properly.

So you could forget 1108 or you put a you don't put the window location in a callback. You're pooched.

Pooched.

And it also works a 100% of the time in development because it's fast as out. Right?

Right. That that is the actually, that is the big problem there is that it does. It works always warp like, race conditions are so often not apparent in development just because it's super fast.

Next one is just called yikes.

That's that's all I could say to some of these Wes these stories that came in. I am a developer in a consulting firm in Sweden rating c sharp on the back end and using React with either JavaScript or TypeScript and hosting everything in Azure 99% of the time, 1% SharePoint.

I was in my last week at my last job, and I was due to start my new job. I worked a 12 hour day to keep up with all the handovers, etcetera, so colleagues could have a chance to continue working on the solutions that I had taken care of. Node project was a process tool hosted in SharePoint online. The guy who would oversee it had negative 1% experience with SharePoint, which I pointed out to my bosses. But to make things easier, I made it a JavaScript to ease things a bit.

Starts with the terminal and runs the script warp environment. Umpteen million pnpm errors appear appeared, which is strange because there should only be about 20 commands.

I log in to the environment and double check if I accidentally entered the wrong values in the script, which looks okay according to me, but I get a four zero four error when trying to reach the environment.

I log in to the admin interface, and I discovered the site is gone. Also checking the trash can, there are no things there. Very strange. I find that I'm in a different folder than the one where I saved my script. In that folder, there is an old deploy script that was used when the project was started a 1000 years ago, which was not used after this project was finished.

The first thing the script does is force delete the site and then try to create a new empty site.

Oh, no.

The site is gone with lists and everything. Lists are like a SharePoint thing, sort of like SQLite.

There are no backups of the acceptance environment, although that is very important. I feel just a little bit panicked. How am I going to solve this? However, I remember testing a tool 6 months ago to copy entire environments Wes my first intent was made here in the acceptance environment.

I find the clone environment. It can be used in the same tool to clone it back. It only took 8 to 12 hours of work to create all the new things done in the environment in the last 6 months instead of the x number of hours to rebuild everything from scratch.

Yeah. This is a great use case for delete your old stuff if you don't need it. No kidding. You can get it. You can get it back from You can get it back. Node you need to. Cool. Next Node, always be closing. When I was in my twenties, I forgot a closing table tag in the mail, shot that went 2,000.

This resulted in the 1st mail containing 1 mail's content. The 2nd mail contained 2 mails' content.

The mail servers of the multinational company crashed as a result.

Yeah. So con dangers of, recursive information, I suppose. Possibly leaking

a sensitive information from 1 email to another. Right? Yikes.

Big big big gas. Alright. Last one we have here is Adidas.

All day, I delete a site.

I deleted the Adidas Facebook page at 8 PM the night before a $3,000,000 spend for ESPN and YouTube homepage takeover. This was back when Facebook apps were big and there was no tiered permissions.

I was a tech director in clearing out all the designer project manager accounts so no one would delete anything.

Accidentally deleted all the accounts and then deleted the entire page.

Our sister media agency called Facebook in the UK, and some engineer found the deleted page and readded it in crisis averted. My boss did not fire me.

Yikes.

Hi. It's it's days like this where I am very happy that my job is as low stakes as it is. I can spend a month working on a course and I get that course out and I get I get a chance to review and watch it 800 times before it goes out.

If I do, by chance, happen to release that course with a a tiny little typo in it, that is not a problem of the magnitude of any of Node that we have read on this show. I it just man, be safe and be careful, y'all.

Write tests. Do not run our MRF unless you are very, very sure of it. Although, I did that once where I ran git clean in a non git directory and deleted half the computer. I talked about that last year.

So don't do those things, and, make sure you take good backups. Man. Good backups.

Good tests.

Be very careful. Don't push on Friday. Any other parting pieces of advice

that you've gained from us? I just keep thinking about the one with the 500,000 concurrent users because you, like, do an image tag incorrectly, and all of a sudden, your support team blows up. And, like, you're you're causing support, extra work, or maybe you have to bring in more people and not be able to pay them. And, it's just the mistakes in those environments are just so high stakes. So that's what I So high stakes. Obviously, like, backups and automated testing and, like, maybe 2 keys to deploy sites probably would have hurt a lot of these, but then we wouldn't have the show every Halloween. So

So please continue to make major mistakes. Of course, none that will result in the permanent harm of anybody. But if 800 people get the whole sent to them in their text messages, then I think that's probably okay as a as a bug. That sounds that sounds pretty good. Keep sending us those. Oh, that's great. Alright. Let's move into some sick picks.

I'm in a sick pick, something I've sick picked in the past, but I just got a a second one of them. And I was just reminiscing about how much I love this thing. So this is a, instant read meat thermometer. Really, not necessarily just meat. I gotta get Node of these. Like you said this last time I sickened it. And so if you wanna be a good cook, Yarn of why people aren't good cooks is because they don't know when something is cooked all the way through, so they overcook it so they don't die of salmonella poisoning or or something like that. And you can for, like, $15, you can go get a really good thermometer that you just poke into your food that you're working, and it tells you what it is. And, like, I feel like it made me a better cook when I got a really not not necessarily, like, a really nice version because there's, like, thermal pens that are, like, $100. But I got this one. It's called mister Sheffer, which is hilarious to me, but has backlight. The the numbers are huge. Most importantly, it reads very fast. So some of these cheap ones you get at the grocery store, you gotta leave it stuck in for 15 seconds before it actually reads accurately.

And that's too hot if you're on the barbecue or something because you key you have to keep your hand on it, or you got this plastic thing in the way of the heat. So go get yourself a nice thermometer. I have got 2 of these now. I've the mister cheffer I had at the cottage. I left it outside. It poured rain on it for days. I dropped it off the deck.

Going strong, so I really like it. I'll put a link for it in the show notes.

Mister Sheffer will be coming to, mister Tolinski house at some point because I I said that at some point and, like, yeah, I I do need to get one. And and just, like, it was, like, 2 days ago, we were cooking some pork and we were using the stick and we got a way for it. And it just is, like, you're looking at it. You're like, I don't even know if this is accurate. Like, I have no idea. Pork is the best one because at least in Canada, a couple years ago, they changed

the safe temperature for eating pork to 145.

And if you if you pull up piece nice piece of pork off the barbecue at 1:45, it'll go up to 1 540 or sorry. You pull it off at 1 40, goes up to 1 45 after ESLint, You cut it open, you think that that is still raw. And it's like there's red in it, and it it doesn't look like it's cooked all the way through, but just, like, knowing, like, yes. It's it's cooked. I I'm did the temperature on it. It is safe for my family to eat. And it's funny because some like, I have my, like, parents over there. Like, are you sure that's cooked all the way? Because they grew up in the the age of the higher pork temperature, and they probably went over that because they didn't have

thermometers at that time. I gotta I gotta get me a good meat thermometer, especially a mister Jaffer.

Sometimes I just buy things based on the name, and that is a great name.

I'm very into the name of mister Jaffer.

So what am I gonna pick today?

Oh, man. I'm just looking

online, and it's not called mister Sheffer in the States. Oh, why? Why did they do that to us? Why would they possibly do that to us? Can I get the Canadian version? The American version comes with a bottle opener. I would rather get the mister Shepherd than the one with the bottle opener.

Oh, no. I found the I found the exact one that I have. It's not called mister Shepherd, unfortunately.

That's a huge disappointment, Wes. Very big disappointment. Okay. I am going to, pnpm pack a library, JavaScript library, and I'm talking about Fastify. Have you heard of Fastify? Fastify dot io, Wes? No.

So I've been diving into the world of Node servers lately because I wanted to see, like, what's changed since the last time I looked. I know a lot of people talk about Nest JS, not Next, but Nest.

And there's, like, Nest. There's Happy. There's Express. There's Koa.

Fastify was the one that's kept on popping up over and over again for me. And I started looking into this, and it's a really neat server. So I will see pick this. I've been given it a test out just to see. There's some really neat GraphQL stuff in here too. And they basically did some neat little, benchmarking against Express and Apollo for this GraphQL thing to find that it's, like, very, very fast. Okay. So here here's how fast this thing is. So on their benchmarks page and this is their own benchmark. So take take that for what it is. But according to this, they have some code here, and they got 76835 requests per second, nearly 77,000 requests per second on Fastify, where Express with the same code was only able to get 385 one Deno. So nearly double the amount of requests per second that it was expressed was able to deliver with the same Node. And same with, happy. Koa was a little bit more, but Bestify was still, like, 20,000 requests a second more. So, obviously, this is their benchmark. So, you know, take that with a grain of Scott. Do your own benchmarks. But Fastify, for me, has been not only very fast but very easy. The logging, all the stuff, set up the plug in, the whole environment has been very cool. There's a lot of neat plug ins involved here. So check this Scott. If you're looking for a Node server,

right now, check out Fastify Scott I o. It's one that I've been having my eyeballs on quite a bit lately. Cool. I'll have to check that. It looks like they have a lot of middleware as well, which is something you wanna you wanna think about if you're picking a new server. Like, you probably are gonna need a bunch of plug ins or middleware. It looks like they've got hundreds of community ones, so sick. And there's some that, like, Yarn, like, make not only GraphQL servers easy, but also make I don't know if you've ever gotten into DataLoader.

No. There's, like, an issue in GraphQL where, like, let's use Vercel up tutorials as an example. I say I want all of the playlists on the site, and then I want all of the tutorials on the site. And if you set up your GraphQL server in a normal way where you have your resolver and then another resolver and then one calls the other, you could end up in a situation where you're getting, like, 500 database queries. Because, let's say, you have 20 tutorials. Each tutorial has 20 videos in it. And, like, the way your system might be set up, it's not gonna do all of those requests in Node fell swoop. It's gonna do the 1 and then the individuals and loop in and whatever.

So DataLoader is like a caching mechanism that prevents large queries like that from being a problem. It's basically a caching solution that will make, your n plus 1 queries way more performant, like, exceedingly more performant. And so there's a really neat, GraphQL server in here that has, like, data loader essentially built into it. They make it, like, part like a first class citizen, and, it really solves a lot of the the pain points I was having personally with DataLoader. So I'm into this. It's very cool. Sweet.

Have to check that out. The API looks similar to Lambda instead of, Express like, which is kinda cool. It's it's not that much different, but Yeah. I just noted that. Cool. Shameless plugs.

I am going to shamelessly plug all of my courses Wes, which is my new website. Check it out. It's forward slash courses. Has a list of all my courses, most recent one being Master Gatsby. Make sure you use a coupon code syntax for $10 off.

I'm gonna shamelessly plug level up tutorial.com Wes you can sign up to become a pro member and gain access to a new tutorial series every single month along with our entire catalog, which is constantly growing. And let me tell you, I am really excited for the next upcoming year. We have 3 or 4 guest teachers lined up that you are all going to be very excited about. I can't talk too much about it just yet, but we have some guest creators coming on. I have some new courses coming out. I have one that just came out on Svelte animations. That is fantastic.

We have new course every single month. It's sort of like a magazine subscription. Check out what the latest course is. So level up tutorials.comforward/pro.

Sign up for the year and save 25%.

Beautiful. Alright. Thanks so much for tuning in, and we will catch you on Monday. Have a spooky Halloween.

Head on over to syntax.fm for a full archive of all of our shows, and don't forget to subscribe in your podcast player or drop a review if you like this show.