October 27th, 2021 × #webdev#horror#halloween
Horror Web Dev Stories - 2021
Scott and Wes share horror stories submitted by developers about production bugs, data loss, performance issues, and more.
- Collecting horror stories
- Wes deletes Denver emails
- Scott leaks XFL cities
- $90 million crypto bug
- Accidentally deleting server
- Accidentally emailing customers
- Mass website outage on Friday
- Performance issues in production
- Company outage from button
- Deleting site with script
- Breaking site with dependencies
- Developer photos leaked
- Accidentally deleting code
- Bug from old code
- Updating all records
- Bad interview answer
Transcript
Announcer
You're listening to Syntax, the podcast with the tastiest web development treats out there. Strap yourself in and get ready. Here is Scott Talinski and Wes Bosch.
Guest 1
Welcome
Guest 2
to syntax, rid The spookiest web development show in the world.
Guest 2
With me, as always, is Spooky Dalinsky.
Scott Tolinski
Ready. It's Spooky Talinsky here with West Booth.
Wes Bos
So today, we've got rid. Our horror web dev stories for you, which is pretty exciting. We do this every Halloween. It's it's become a yearly thing where we ask the audience rid To submit their most stick your head in the sand, oof horror stories of things that went wrong as a developer, whether it's rid. Deleting an entire database or shipping a swear word or all there's all kinds of hilarious stories. So I think this is the 3rd year that we've done it now, and rid The stories are awful, but, usually, they are in the past, so we can laugh about them now.
Scott Tolinski
They're tremendous. And I I almost feel like you could have a whole podcast that's just web dev oopsies. It's like, here's a story about some web dev oopsies, and we'll talk about it.
Wes Bos
Rid. So these are always a blast. I'm I'm super psyched. We are sponsored by 3 spooky companies today. First 1 is LogRocket JavaScript Session replay.
Wes Bos
So if you have ghosts in your application, they'll be able to track them down.
Wes Bos
And the next one is Mux. Re They do video hosting and transcoding for your application.
Scott Tolinski
So if you have a spooky video, they could transcode it. Okay. What we gotta do is set something different than spooky for the third one. What do you got? 3rd one is
Wes Bos
Linode. They are the spookiest rid. Smookiest cloud computing and Linux servers that is out there, they if you've got Some gremlins in your server.
Wes Bos
Mhmm. And it's going down. You know who you should be trusting is Linode.
Scott Tolinski
How's that? Rid Yeah. Sure. Yeah. There's no there's no bats in their bell bell tower. That's for sure, gang. Yeah. I don't really know. So so, Wes, I think we are ready to kick it off with some spooky stories.
Collecting horror stories
Guest 2
Woah.
Guest 2
How did you do that?
Guest 5
How did I do what?
Wes Bos
Oh my gosh. You didn't even tell me he was doing this. This is amazing.
Scott Tolinski
You would rid not believe how much time I had to put in just for that one little
Guest 5
spooky story.
Scott Tolinski
Get in here and actually sound decent. It's been a voyage, Wes.
Wes Bos
That was well worth it. You'll have to read out the titles in that.
Scott Tolinski
Yes. As long as it doesn't crash my computer, I'm very good. So I will get started with this first one that I found on, I found on Twitter. I thought this was very funny. Says, hey, guys. Love the show. I wanted to share with you something that happened just the other day. It was October 4th. I was starting my new job at a large tech company.
Scott Tolinski
They use React for everything, even DNS.
Scott Tolinski
Don't ask me how it's complicated.
Scott Tolinski
I figured I'd celebrate my 1st day and push some code to prod.
Scott Tolinski
How hard could a use effect be? Right? Next thing you know, rid I end up bringing down the entire company, all of their servers, and they ended up bringing in a guy with an angle grinder to access the server cage. Rid Wait a second, Wes. Wait a second.
Wes Bos
What's going on here?
Scott Tolinski
This is fake. Obviously, this is fake. Sorry. I Started off with the fake one. The rest are real. Yeah. The rest are real, I promise. For those of you don't know, they actually needed to use an angle grinder to get into the to access the The physical servers for the Facebook outage? Yeah. Yeah. So that's what this joke is about. Basically, the Facebook, they had, like, goofed up something in their DNS settings. Not just like, oopsies, we added a period here in the DNS. No. They were, like, doing some crazy self managed DNS stuff, and it, like, pulled all their records from the web. And then Yeah. They, like, locked themselves out. Yeah. So they brought an angle grinder and tore down the door. Oh, wow.
Wes Bos
That's insane.
Wes deletes Denver emails
Wes Bos
I have a bit of a ready. Story myself that I figured I'd start the thing off with is this is about maybe about a year ago that this happened, and re Listener of the show, Tim, a Denver coder on Twitter, he messaged me one day, and he was like, hey. I'm trying to buy your course, and it's just not working. Like, It's, like, not giving me an error message or anything. And I was like, that's super weird. Like, what what could possibly be going on? And luckily, he was nice enough to, like, give me all this information.
Wes Bos
Rid. And I was, like, sitting there being like, what is going on? Like, I'm selling all kinds of other courses. Everything is working fine. Like, 0 things are showing up in rid and my error tracking. And I was like, what is going on here? And finally, I was just, like, going as I was reading, like, the code base line by line just rid Just trying to figure out what could be possibly happening, and then I realized they came to me. So probably about a year and a half or, like, 6 months before this, rid. I was getting slammed by some spammer. I was trying to just slam my thing with a bunch of fake credit cards, and they were Stripe was handling it and whatnot. But I just wanted to, rid Stop this from happening. So I fairly quickly just jumped in, and I wrote, like, a block. I said any email address that has Denver in it, just reject it. Rid Right? And because the the spammer was specifically using, like, an email address with Denver in it, and I it was just enough to just stop this guy from from hammering the system. Right? I just threw a little if statement in there and returned if it had the email Denver in there, and it went along with my life. Oops. It did. Yep. So the bug was rid. Literally nobody with the Denver in their email address, according to a RegEx,
Scott Tolinski
could buy any of my courses. And it had, like, never popped up. Like, who's got Denver in their email address? That can't be, like, that common. I I'm you know, somebody lives in Denver. It's not like We all we all got that email address or something. That is very funny. I I actually wanna see if I can see how many people in level up re Orioles database have the email Denver in their email address. None. There's not a single one. There's not a single one. I'm curious about that too. Well, let me look at mine. 63.
Wes Bos
What?
Scott Tolinski
Yeah. Where are my people at? My people gotta come to, Level up tutorials here.
Wes Bos
Probably be about 20% of them are from a university in Denver. Oh, okay. Okay. Okay. But, rid. Surprisingly, a lot of people have Denver in their mail address, so that was definitely a issue.
Scott Tolinski
That's super funny. Here's another real world one, not one that was sent to us, I figured this one would be good to share because it turns out, like, the week we decided we were gonna start pulling stories for this episode, There's a ton of, like, high profile dev oopsies. Here's another dev oopsie horror story here that I thought was even just reading that the headline might might get you shaking in your boots. DeFi bug accidentally gives $90,000,000 to its users. Founder begs them to return it.
Scott leaks XFL cities
Scott Tolinski
So this is this crypto protocol, or I don't know. Honestly, I don't know the the language enough to speak on this, but, basically, rid They gave $90,000,000 to their users because they goofed up, and then the guy went on Twitter and, like, really? I think begging is He threatened them. He was saying he was gonna send the IRS after them and all sorts of stuff. It's like, dude, I know it's $90,000,000.
Scott Tolinski
You made that oopsie. You gave the money away. I don't think threatening them is going to give the money back, so That's a big time oopsie. I I think, $90,000,000, there's there's very few developers who are gonna hit a $90,000,000 oopsie in their career. Oh,
Wes Bos
so rid The guy who tweeted it out says, if you received a large incorrect amount from a protocol error, please return it. Rid. Keep 10% as white hat. Otherwise, you're being report as income to the IRS, and most of you doxed. Well, If you report it as income to the IRS, you get to keep 50%.
Scott Tolinski
And, also, you have to pay taxes on the IRS for that regardless.
Scott Tolinski
You'll have to pay it for the 10%. Even if you gain whatever you gain, you're gonna have to pay taxes on it. So, like, nah.
Wes Bos
Yikes. Rid That's a bad day. That's a bad day.
Wes Bos
Oh, man. Alright. This one and by the way, we keeping everybody anonymous in this one Just to protect the innocent. Most people ask to be named anonymous, but, literally, everybody is anonymous, so we don't get any people asking us to claw back their story.
$90 million crypto bug
Wes Bos
Rid. So the story goes, hi. This is my contribution to these stories. I was a developer at a consulting firm in Sweden writing c sharp rid. On the back end and using React with either JavaScript or TypeScript and hosting everything in Azure on the 99% of the time and 1% in SharePoint.
Wes Bos
Rid. It was my last week of my job before I was due to start a new job. I worked 12 hours a day to keep up with all the handovers to colleagues so rid. They would have a chance to continue working on all the solutions I had taken care of. The project was a process tool hosted on SharePoint online. Rid. The guy who would oversee it had negative 1% experience with SharePoint to which I pointed out to my bosses.
Wes Bos
But To make things a bit easier, I wrote a deploy script to ease things out a bit. Starts in the terminal and runs a script towards the acceptance environment. Rid. Umpteen million errors appear, which is strange because there should be only about 20 commands which can cause errors like these. Rid. I log in to the environment to double check now. I accidentally entered the wrong values in the script, which looked okay according to me, but I got a four zero four error when I was trying to reach the environment. I log in to the admin interface.
Wes Bos
I discover the site is gone. In checking the trash can, there are no things there.
Wes Bos
Very strange. Very curious.
Wes Bos
I find that I'm in a different folder than was the one where I saved my script. In that folder, there was an old deploy script that was used when the project rid. Was started 1000 of years ago. The first thing the script does is force delete the site and then to create a new empty site. The site is gone with lists In everything. Listen to SharePoint are sort of, like, SQL lites, so the database is gone, essentially, here. Oh. There are no back ends of the acceptance environment, rid. Although it is very important, I feel just a little bit panic of how I'm going to solve this. However, I remember a testing tool 6 months ago to copy the entire environments, which re A temp was made to the acceptance environment.
Wes Bos
Finds the clone environment, and we can use the same tool to clone it back. It only took 8 to 12 hours of work to recreate the new things in the environment in the last 6 rid months instead of x number of hours to build everything from scratch. Oh my.
Scott Tolinski
Oh.
Scott Tolinski
So lesson here the lesson here, backups, rid Frequent backups, if you have things like this, dang. Yeah. Frequent backups for stuff like this. I mean, it seems like they got lucky with a backup, not even, like, one that was prepared to be there. Oh, a little bit of bonus here. Once I updated the a feature that saves accessories on order, same solution. However, I rid. To add all the new fields to the production environment,
Wes Bos
which meant that accessories were not saved at all, which was discovered after a week. I fixed the air in 5 minutes, and the sellers Had to contact x number of customers to double check what kind of accessories they would want on all their orders.
Scott Tolinski
Oh. Couple of doozies. Confirm. Confirm. Ridged. Confirm. Yeah. Yikes. Yep. Good good story, though. Couple of doozies here. Next one we have here is called External HD.
Accidentally deleting server
Scott Tolinski
External HD. Okay.
Scott Tolinski
One time, I needed to format a server.
Scott Tolinski
It was an outdated Windows Server. Read. I selected and copied and pasted to an external hard drive. My drive was pretty fast, and it looked like a minute. And I was like, wow. This is a great external HD.
Scott Tolinski
Formatted the server, and as soon as I realized it didn't copy 10% of the files, rid I had that face. We all know the face. Yes. We absolutely do. Anyways, I tried to restore the files using some HD recovery tools, But they were all corrupted not by the formatting itself, but by the installation of the new OS.
Scott Tolinski
My boss was pissed.
Scott Tolinski
Rid I was very young, so I had to blame it on the server. I'm not proud of it. What'd the server do? Good. The the server just got corrupted.
Scott Tolinski
But why the heck Would they ask a developer to format a server in the 1st place? By the way, my birthday is on Halloween.
Scott Tolinski
John finishes Love it. Spooky.
Scott Tolinski
So yeah. That one's pretty bad. You know, I'm so very paranoid about that. Whenever I'm I'm formatting a hard drive Yeah. You know, when they say, like, you know, the measure twice, cut rid One thing. I'm measuring, like, 18 times. If I have moved over precious files or backed up something, it's like, I need to confirm that those things are where they are. Read. I'm from the the days of a LaCie 200 gigabit drive storing your entire life on it, and that drive just waking up one day and saying, not today. Rid No thanks today. I'm done. So, yeah, I gotta have those backups.
Wes Bos
Yeah. Always have 2 backups of everything and confirm that rid They are working periodically. That's just a awful thing to happen. Yeah. Next one we have here is called Hey, Loser. You wanna say it in your spooky voice?
Guest 5
Hey, loser.
Wes Bos
I was testing some new code to automate mass mailing to our customers. Oh, these ones always start so good. Whenever it includes the word mass in it, it's, Yeah. Whenever it includes the word mass, it's like, oh, man. Who knows what demon drove me? But I wrote the test mailings like ransom notes. Dear loser, fork over your dollars or else. Well, all was looking great, And I was feeling pretty pleased with myself. Progress bars were sliding and counters were spinning, but I could hear the rising commotion from the marketing guys behind me.
Accidentally emailing customers
Wes Bos
Rid Phones were ringing. Voices were raised. Turns out, I had moronically wired myself to the production database.
Wes Bos
Even worse for me, I'd only been with the company a month or 2.
Wes Bos
I thought my goose was cooked, and the big boss was Plenty mad, but I owned up right away and apologize.
Wes Bos
We put out the cover story that we've been hacked and always forgiven.
Wes Bos
Love your show.
Scott Tolinski
Woah. What? Woah. Is saying you got hacked better than saying we accidentally sent these mailings out that were a test? At least blame it on the intern. That's what HBO did. No kidding. HBO is like, I mean, like I said, there's we got this person just Push the button. I'm so sorry. We can't do anything about it. Oh, man. Never put anything,
Wes Bos
swear words, or silly things into tests. Always just write test, test, test. Yeah. Even loremipsum.
Wes Bos
Better than that. Yeah. I would probably just put test, test, test. This is a test. Yeah. Because, Like, many times, I get little push notifications on my phone that say this is a test. I'm like, oh, some Okay. Some developer somewhere just goofed up, but at At least I don't have to look into it. You know, like, loremipsum, especially if you have just, like, regular users, they're gonna be like, what is this? Oh, yeah. They're gonna think it was hacked. Rid Then maybe if the hack is gonna be your excuse, then maybe that's the, that's the answer.
Scott Tolinski
Alright. Next question here or next question. Next Horror story. Let's see. We gotta get a come up with a good title for this one. Extreme
Guest 5
developer fumble.
Mass website outage on Friday
Scott Tolinski
Horror story. Rid I was part of the developer team that accidentally leaked the 8 cities in the XFL, an alternate football league. And for those of you who are not Americans, They're talking about American football here. The XFL was the wrestling XFL one, where they hit people hard and they had some fancy new rules. Either way, A week before their press conference, that's when all 8 cities were leaked. We were using Contentful and Gatsby.
Scott Tolinski
A ready. Junior dev entered the information into the prod space of the UAT space. And when we released some bug fixes, re It picked up the content update. So as you can imagine, there's new data available, and so the Gatsby site was rebuilt With that new data, that new data then went live.
Scott Tolinski
I found out after seeing the stories pop up in Google News when I was about to go to sleep, was content down and started getting calls from the CEO or CIO of the WWE.
Scott Tolinski
Rid OP is about to get his head in a a choke slam here. The league went bust because of COVID. Yes. Re If this is used, please don't use my Twitter name. You can be safe. You hate me. We did not expose you on here, And I am so sorry that you you accidentally were part of the team that leaked the aid cities. That's actually a pretty major leak if you think about it. I'm sure you could find the news stories about that when that was leaked. Chances are, if you did something where there, like, becomes a news story about whatever you did, rid. That's a pretty big oopsie. He'd linked us to a ewrestling.com
Wes Bos
article where all of the names of the things had leaked, and, like, rid Imagine, like, a little Gatsby error and, like, the entire league names are leaked out. That's kinda interesting that, like, a lot of us rid. Developers have access to information that is confidential. And sometimes, probably even, like, I wonder A lot of developers have to go through, like, trading stocks and things like that. You have information to it's kinda crazy. When I worked at Ford, I had
Scott Tolinski
sign mountains of documents about unreleased vehicles or all sorts of stuff. Oh, yeah. Or, like, when we were doing demos that people were going to be seeing, We had to make sure that we were using, like, specific model. I told you about that one time we were doing a q and a, and that guy was, like, super upset that the Ford Mustang, like, wasn't in the thing. Re Ruck, listen, man. We've been given very specific assets.
Scott Tolinski
I'm sorry that the site does not have a Ford Mustang for you on it. For those of you who have never heard that story, We were doing, like, a q and a with, like, real people, and this guy was, like, so hung up. He's like, I hate this design. This design is terrible, and we're just like, why? There's no Mustang.
Scott Tolinski
Gotta have the Mustang.
Scott Tolinski
They're just psycho.
Scott Tolinski
So if you are the type of person who has made a spooky mistake in your code. Everything's just gonna be spooky from now on, unless it's just spooky. Spooky. If you've made a spooky mistake in your code and you don't know why, rid. But you get reports of users seeing some unusual behavior in your application. You might be concerned.
Scott Tolinski
You might be hiring a a ghost hunter, but we We will hire LogRocket because it allows you to see what's going on in your site, and that's like an error and exception handling service that gives you an actual visual replay. Imagine if someone was stealing your cookies, and you could go in there and and you fire up your video camera and you say, oh, that's not a ghost. I thought that was a ghost rid stealing my cookies. It's not a ghost. It's actually my my 4 year old child opening the thing. And you could see your 4 year old child sneaking in those cookies with LogRocket.
Performance issues in production
Scott Tolinski
Rid LogRocket, again, is a service to monitor errors and exceptions on your site. So if a user is causing some issues, they're stealing your cookies. You're gonna get a video of that. You're gonna be able to see how they did it, When they did it, where they did it, all that error log, network requests, all that and more. Check it out. Log rocket.comforward/ rid Syntax, sign up, and you'll get 14 days for free. And thank you so much for LogRocket for sponsoring this very spooky episode. This is This is actually honestly, as for a sponsorship read, this is like a dream episode for LogRocket, because you you could've gone anywhere with it. You could've said, is what's the wolf man doing? Is the wolf man is he rid. Destroying your header is Dracula causing errors in your console. Who knows? I could just keep going. Yeah. It's a good episode for LogRocket to sponsor, so sure ready. Sure they'll be happy with that.
Wes Bos
Next one we have here is called, I don't have memory of this.
Wes Bos
I had 2 rid. Pretty bad code changes that only showed their problems when they went live into production. The worst kind of bugs are bugs that show up When they only hit production, especially ones that are, like, take the entire site down.
Wes Bos
Around 6 years ago, I was running a large into a large rid. Issue with some of our queries running slowly against our date DB.
Wes Bos
We're using jpa/ hibernate and had a bunch of joins that were done lazily. I switched rid them to eager so they would create a single SQL statement instead of a bunch or 1,000. The change worked fine in my dev environment, QA, and staging. Staging was supposed to be representative of production, so we went live with and within a few minutes, our entire system went down And was out of memory.
Wes Bos
We quickly switched back to lazy joins. We found out that staging had more memory and fewer DB records Then production, and they were supposed to be exactly the same. Oh, ain't that funny? Like, you run all these tests. You think about everything, but rid. If the environment is not to a t exactly, both in terms of specs of the server and specs of how many pieces of databases you have how many items in your database you have, things can go south. Do you know what? There's just about nothing in the entire world that scares me more than modifying a production database.
Scott Tolinski
Yeah. Like, rid I take backups. I do everything you could possibly do to be secure about it, do dry runs and all that stuff. And we we did some database migrations recently, and I was just like, Dear gosh, please work no matter what. Even though I knew it was gonna work, it's still like, just hold on to your butts. Next story here is called,
Guest 5
Your performance is slowing us down.
Company outage from button
Guest 5
No. No. No. No. No. Okay. Back when VMware
Scott Tolinski
was becoming a thing, like 2010 or so, I was working at an ecom site, and we were seeing slow performance between the app server and some data services. Rid I decided to build a little multi headed logger that could track when a query to Oracle Financials was running too slow and generate a warning. Rid. Multithreaded.
Scott Tolinski
Multi headed sounds spooky. Oh, yeah. You know that multithreaded is a tough word? There are 2 t's in there, and they're very quite close succession. Re Multi threaded logger that could track when a query to Oracle Financials was running too slow and generate a warning. Oracle Financials was doing the credit card transaction orders and the rest of the site's DB work. The code had no impact on my dev, QA, And staging environments, we were hitting well over our minimum number of concurrent users. We deployed it to production, and then the system got slower and slower, but never crashed.
Scott Tolinski
Again, production and staging were set up differently. Staging was a bare metal server. Production was running on an ESXI XI server host.
Scott Tolinski
Server on a host that was split 4 ways. The multi threaded code was meant to detect performance degradations, rid was slowing the whole system down when it tried to synchronize the data across threads.
Scott Tolinski
I was pretty embarrassed by these 2 issues. It went to show that That production is its own special thing and that you really don't know, and your server side code is going to work until it starts running there. One, I am so glad I never have to worry about multi threaded anything in my data g job. If I had to worry about that, that would probably be Not a good situation for the company I was working on. This seems like there's, like, a lot of attention to detail when dealing with when dealing with performance of that kind of nature, but this sounds intense. I don't follow anybody for goofing up something like this and just seeing that performance degradation. At least you caught it before you sent out an email to 20,000 users or something or leaked something to the re WWE's news press kit. So I don't think this one this one's not too bad. Next one we have here is
Wes Bos
From my dad, actually. My dad's worked in tech for many, many years. Bert? No. Not Bert. Not the creator of CSS.
Wes Bos
This one is called dead button.
Wes Bos
Way back when mainframes were king, A guy I work with pushed a button in that, if released, would immediately take down the entire company. He stood there for 4 hours.
Deleting site with script
Guest 2
Why? He stood there
Wes Bos
he stood there for 4 hours holding the button in until we could let it crash after business hours. We gave him a chair rid After 2 hours.
Wes Bos
Oh, that sucks. I remember doing this exact same thing, not Taking entire business down, but when I was in grade 3, remember Windows 95 would have that black and Orange screen, that was that is now safe to turn off your computer. Yeah. I remember the teacher's like, alright. Just shut down the computers because that's what you did when you were done with the computers. You turned rid So I just push the button in, and I immediately recognize, oh, shoot. It's not safe to turn off your computer that way. Right? So I push the button in, And then I start, like, looking around and being like, oh, man. Like, who who here is smart enough to know how to shut down a computer? Right? Because, like, I was the tech kid. Because I I couldn't do it with my left hand, so I called over a buddy of mine. He shut down the thing, and, like, Windows 95 took, like, 3 minutes to shut down. So I was sitting there holding the button in until we finally got to that black and orange screen that said it's now safe to shut down your computer, and I I let of go, the button. That's so funny. I don't honestly even know what it's doing in that shutdown process. I've never, like, taken the time to learn rid that actually was about. What's the deal with that shutdown screen? That's a a really good question. Probably
Scott Tolinski
something with the hard drive. Yeah. It's the same thing with, like, safely ejecting your rid SD card or whatever. I don't do that. No one's got time for that. No one's got time for that. Yank it. Okay. So I gotta do the spooky voice. Don't deploys on Fridays.
Scott Tolinski
Re Okay. I was a junior dev working on our company's website. They were HTML plus Nunchucks templates. I'd never used Nunchucks West. Did you? Rid No. No. I was, it was more popular in, I think, the the Ruby on Rails world. Okay. HTML and Nuntrux Templates that were later integrated with the back end using some Python witchcraft. Oh, we had snakes and witchcraft.
Scott Tolinski
There was also a metric, boop, Ton of JS libraries added, like Babra or page transitions, 3 JS for cool interactive animations on the landing page, rid Didn't really get much of all this package dot JSON stuff at that seniority level. So after running Yarn or NPM or whatever And seeing a bunch of warnings I think that's typical, by the way. Seeing a bunch of warnings about a couple packages being outdated, I decided to update some of them. It rang great locally, But I didn't build the prod version as I didn't know there could be any differences. I was working on some minor feature, And the PM decided that there was no time for a code review, so I pushed the repo.
Breaking site with dependencies
Scott Tolinski
The back end guy did his integration, and it launched on prop. As it turned out, There were some breaking changes in one of the libraries that decided to update and crashed the entire site on Friday at 4:30, and that, kids, is why you don't deploy on Friday. Yeah. You think? You know what? I actually don't mind deploying on Friday. Personally, as long as the test passed, I've manually tested immediately afterwards, and Everything is, like, this is a small, small update, but to push some code to production that you did not test after updating dependencies, rid. That's on you.
Scott Tolinski
That's not on Friday. Don't blame Friday for you.
Wes Bos
Yeah. Risky. Risky. Risky. I definitely would not rid Do that myself on a Friday. No matter how much I tested, it's just a recipe for disaster. Hopefully, they had, like, some, Like, lock files of of the previous dependencies, they were able to roll it back or fix the thing pretty quickly. But Yeah. I love the modern host now just have, like, the rollback.
Wes Bos
Yeah. But and it's like, redeploy this to play, please. Yeah. Thank you. Yeah. Honestly, like, that's sort of the way to do it as you deploy the thing on another server And make sure it's all running fine. And then when it is, you cut over to the new one, and instead of having to, like, break the entire thing.
Wes Bos
Next one we have here is stupid selfie horror story for you, Wes. I work one of the biggest retailers in the UK, and we were working on an app That would go on a media wall in their flagship store in London. Cool. Basically, a giant 200 inch screen In the middle of a store that would show social content can go on to, I actually built a couple of those for a forget what it was, but it was on, like, Part of, like, a Christmas thing, and they would show tweets where you could hashtag it. And I remember having to, like, filter them for swear words and everything. Right on. Turns out, rid. I left my local dev version connected to the production API when I uploaded a couple of my stupid selfies of my big head in the office. Rid. I get a call the next day to ask why my face is all over the media wall.
Developer photos leaked
Wes Bos
Oh. Oh, that sucks. That is funny. Rid That's very nice. Like walking into this, like, high end store in London, and there's just pictures of some guy Goofy developer. Rid With a Coke can beside them?
Scott Tolinski
Yeah. That is very funny. Alright. This next one is called rid Souda.
Scott Tolinski
I was a computer operator back in the late 19 sixties operating a Honeywell mainframe.
Scott Tolinski
The consoles were huge, and the size, About the size of a dishwashing machine, also known as a dishwasher, with the console typewriter and printer inset in the middle on top, rid I had a soft drink on the console next to the typewriter mechanism.
Scott Tolinski
We were told never to bring drink into the room, but we all did it.
Scott Tolinski
You can see where this is going, especially on the 3rd shift. Long story short, someone called my name. I turned around and knocked the glass of soda onto the console. It had to be completely replaced. The machine was down for 2 days, and my boss was not happy. Well, do you think? You're
Wes Bos
Yeah. That's a rough one. Spilling anything, not so good. Yep. No. Thank you. Can you imagine, like, every time I knock something over, I specifically have my laptop rid. Up on a little stand just so that it yeah. Your keyboard, if it gets a coffee on it, you might lose it. That's okay.
Wes Bos
But, like, your laptop, rid Too scary. And, like, imagine a mainframe running an entire business, I would just go stick my head in the sand. I would stick my head in quicksand or yeah. Yes. Absolutely something.
Scott Tolinski
Speaking of mainframes, speaking of mainframes, You wanna host your website in a place where you don't have to worry about accidentally spilling a Coke on your mainframe to bring it down? You might wanna check out one of our sponsors, which is Linode, which has been a cloud hosting service that's been around for a long time and does the job way good. Wes, do you wanna talk about Linode? Ready. I do wanna talk about Linode. Linode is a cloud hosting company for your applications,
Wes Bos
and they have all kinds of different services. They're rid. Most popular for their Linux boxes that you can host the node app or Ruby on Rails app or whatever on. But I'm gonna talk about something else rid They do today is they have dedicated compute for CPU intensive workloads. So if you need a mainframe To maybe do a whole bunch of CPU intensive workloads, whether that's, like, resizing data or going over a 1000000000 records and and Updating those records or parsing up music or anything that is CPU intensive, and you need to do a whole bunch of that work at once, they have dedicated processors rid. On there so you're not sharing it with anybody. You can use the whole go yourself. Check it out at linode.comforward/syntax, rid. And that's gonna get you a $100 in free credit, which is pretty sweet. You could use that for some dedicated CPU intensive workloads Or just hosting your your own website. Thank you, Linode, for sponsoring.
Scott Tolinski
Yeah. You can also use it to buy Coke with some that money you saved. Okay. Oof. Rid A big horror story.
Scott Tolinski
I had my own software companies in the nineties, and it was in Singapore, customizing my software package rid For Johnson and Higgins insurance brokers, I had their Asian contract for my insurance broker accounting package. I spent a good 40 hours on Saturday and Sunday making all the changes that were asked for, getting ready for a demo on Monday morning.
Scott Tolinski
I finished up about 4 AM on Monday morning. I was cleaning out my files. All of this work was being done on a Novell server. Print Files had an extension of PRT, and I had a ton of them in the main directory from all the testing I had done. I was getting to cleaning out old files, getting ready to break everything up, and then I thought I would delete all of the print files. I mistakenly key erased asterisk.prg instead of dotprt.
Accidentally deleting code
Scott Tolinski
Re whatever the delete command was, I don't remember. Now programming files have a dot p r g Extension. I deleted all of my updated files from the weekend.
Scott Tolinski
In desperation, I called the Nobel in Utah Hoping that they could recover the files, but it was no go, and the demo Monday morning was not fun. Oh. Re Failed demo's never any good. I've been in a room with a failed demo where the dev was just sitting there sweating the whole time, And it was not a lot of fun to be in, and I wasn't even the person who worked on it. So yeah. No. Thanks for that one. Oh, man. Like, rid. This is why you should use that trash command. Yes. Yeah. Trash. No matter what,
Wes Bos
one little hitting a g instead of a t, rid Just erase 40 hours and a bad demo. Yikes. Yikes. Yikes.
Scott Tolinski
Remember my horror story, which is running git clean on the root directory rid of a machine. Oh, yeah. Yeah. These all remind me of that. No. Thank you. I'm stressed out just reading these stories. I know. I think I I have a certain degree of tension in my body after this episode.
Wes Bos
Next 1 we have here is called a young dev. I was a young dev right out of cottage. My first job Was that a child support company where we had desktop apps that would handle all the information more efficiently than using Excel? My first project was to write a proof of concept that would later be implemented into a new bigger app that consolidated all of the proof of concept for various parts of the child support process. For some odd reason, I still don't know why to this day. My boss wanted to write me through me to write this new app on top of an older app With a bunch of legacy code, I never understood why. But as a young dev fresh out of school, you tend to just do what you're told.
Wes Bos
In school, I mainly use PHP, HTML, and CSS for learning how to work with a database, but this job was c sharp in .net for their desktop apps.
Wes Bos
So I was doing a lot of learning as well. I remember finally learning how to connect to the database and run some SQL after fighting this old pile of legacy code. In the early versions, rid I chose to handle create updates for those records in the same function.
Wes Bos
My young dumb self wrote a try catch Statement that it would attempt to create a record, and if it failed, it would update the record. That's called an upsert in a lot of programming languages. Just update it. If it's not there, then insert it. Re Before the 1st production release, I updated the flow to handle create updates in separate functions, but I never removed the update in the catch block of the original function now used for re creates only. Somehow, I or any other PM or QA never failed to create or to hit this catch block while testing.
Wes Bos
We talk about test coverage a lot. That that's a a missing case in your your test, where if you write some code and that line of code is never run-in all of your tests, rid Then you don't have very good test coverage. Fast forward to probably 9 to 12 months. Sorry. I'm just trying to add some value to the listeners.
Wes Bos
So this Fast forward to probably 12 months later, I got a ticket to investigate why every case's data looked the same in production.
Bug from old code
Wes Bos
I log in to the app, Search a few case numbers, and sure enough, every case's data is exactly the same. I began freaking out as I had no clue how this could have happened. I mean, it could never have happened. All the dev work testing and months of lies production use. After I investigated with a senior dev, we realized the rid. Try block had failed, and the update query in the catch block ran for that record.
Wes Bos
We also realized that it left of the where clause in the related SQL query to specify which records needed updating.
Wes Bos
Rid. So all the records got updated with this data. Oh, thankfully, we kept the regular backups and were able to restore the data to a recent time frame without losing a ton of work. Rid. We commented out that database update and called redeploy of the code ASAP. Also, senior dev was cool, and he's like, hey. It happens to us, all of us at some point. Rid Let's just say I've learned a ton since then and definitely steer clear of writing code like that. You live, you learn, I suppose.
Scott Tolinski
You live, laugh, laugh. You know what? It is it's so funny. You get, like, a couple sentences into each of these, and it's, like, almost like there's, like, one little bit of foreshadowing that, like, Yeah. If you've been around the block a little bit, you can be like, oh, I know it's coming.
Wes Bos
I know it's gonna happen. It's true. There's probably only, like, rid I don't know. 5 or 10 things that go wrong in all of these stories,
Scott Tolinski
and it's helpful just to know these things so that you when you are in a similar situation, hopefully, these Spooky stories will come eking back at you. Yeah. Seriously, like, what have we learned here? Have a a staging environment that mimics your real production environment for testing. Don't have something that's only only only production. Yeah. What else are we learning? Don't swear. Don't write swears. Don't write silly stuff. Test your code. Yeah. Rid. Read things twice. Yeah. Do a dry run of database migrations.
Scott Tolinski
Hey. A safe run.
Scott Tolinski
This episode is also sponsored by Mux. Now Mux is the perfect place for you to host your video and has some of the most awesome video streaming functionality tools around. Mux, re Basically, it's a CDN for your videos that gives you on demand transformed transcoded video To suit the needs of any of your audience depending on where they're listening to it, if they are if they're watching your video in some place, their Internet's not good, it's gonna serve them up at a lower quality version automatically, and it's gonna do this all via streaming. Now, take it from me, I've done a lot of video streaming, and it is not an easy problem to solve. You end up having to write all of these re Transcoders, and you have to write your whole ingest flow and move things around and have a database for all your files. No. Thank you. Re and Mux gets to do all of that for you and so much more. Thumbnails, animated GIF previews, the little thumbnail scrubber thingy, all of that is solved with a simple get request From Mux. So check it out. Mux Video is the API first video platform that makes it insanely easy for developers to build beautiful video. So head on over to mux.com and sign up today.
Scott Tolinski
Give it a try, especially if you are the type Who doesn't want to deal with all of that annoying stuff around video is one of those things that we use personally over at level up tutorials. And let me tell you, I absolutely adore this service, so check it out, mux.com.
Scott Tolinski
Thank you so much for Mux for sponsoring.
Scott Tolinski
Alright. Next one here is re Sleepy set up.
Updating all records
Scott Tolinski
Here's my development, tale of terror.
Scott Tolinski
Oh, tale of terror.
Scott Tolinski
1 night, I was burning the midnight oil trying to get caught up on the never ending workload.
Scott Tolinski
At the time, I was working on an online travel booking site. It was after 11, and the last thing I had to do for the night was to rename one of the hotels in our production database.
Scott Tolinski
So I wrote my query, update. Hotels, set.
Scott Tolinski
Name some hotel chain. Oh, boy. I bet you can all see where this is going.
Scott Tolinski
One problem, I forgot the were clause. Yo. We should name this title of this episode werewolf because it seems like so many people are forgetting the were
Wes Bos
Oh, that's a good one. I like that. Werewolf. Werewolf.
Scott Tolinski
See, that wasn't even the spooky voice. That was just me trying to do it. Suddenly, over 5,000 hotels in our database all had the same name. This is around 2003, so well before the point in time restores, and they're only backing up the database rid Every week at that point, I was panicking.
Scott Tolinski
Fortunately, I had a dump of the production database that I created only a few hours earlier sitting on my local hard rid So, thankfully, I was able to restore almost all of the hotel names, save for a couple that signed up for the data dump, rid. And my boss was none the wiser.
Scott Tolinski
That's when I learned that working late hours is not worth it because at some point, You get too tired, and you no longer make good decisions.
Scott Tolinski
That's actually a great lesson because I personally I learn that all the time. You know, you're you had a couple beers or something. You're working on some almost never productive, or you, like, end up hitting a a bug that you, like, solve, and then you spend hours and you stay up all night trying to solve it. And then you solve it first thing in the morning because that's just how your brain works. Very good. Yeah. Burning the midnight oil is not always a great idea.
Wes Bos
We should say, though, like, is this a bit of a issue with the SQL Yes.
Wes Bos
Language that this happens. Like, I know people don't like MongoDB, but if you wanna literally update many records in MongoDB, you have to rid Supposedly type that in one of a few different ways. You have to pass it as a Boolean or use one of the methods that update all.
Scott Tolinski
So it's Such a common thing. It seems like a common thing, and it seems like something that I would do if I wrote a sequel.
Wes Bos
Yeah. I wonder there must be, like, a rid flag where you don't you have to explicitly have to pass there because it's gotta be the to stop this type of thing from happening. Otherwise, Maybe it shouldn't because then we would not have this hilarious stories.
Scott Tolinski
Yeah. Let's keep it for the stories.
Wes Bos
Next one, we are going to call Ivanti's job.
Wes Bos
There we go, Wes. When I first started out working for a consultancy, they trained us in sales meetings to help managers get promoted because we were coming in to make them look good. This was okay because, obviously, we're coming in as contractor. However, after being rid Laid off due to 911. Yes. That was about 20 years ago. There was a couple stories here that were I think we've this is our 4th story that's, Like, 20, 30 years old. I was looking for a new job, and during the interview, I was asked where I'd like to be in x years. I mentioned the hiring manager that I wanted to eventually do what he was doing.
Wes Bos
I want your job.
Wes Bos
Well, I guess he didn't take of that as I wanted to make him get promoted and take his spot. So safe to say I didn't get hired. Oh, that's so funny.
Bad interview answer
Wes Bos
Re You know what? Preferably, what I would do is I would be kicking you out of your chair, sir. That's what I would do. The guy probably didn't want a new job. That's so funny. That is good. Alright. That's rid All of the long form horror stories that we have for you today, those are some pretty good. And thank you everybody for submitting them. If you have a horror story of your own, feel free to email, [email protected], and we'll put it on the list for next year because these are hilarious.
Scott Tolinski
We love ourselves some good dev horror stories, And we've all done it. Everybody's been there. Everybody's had that. Them person says, you know the face. We all know that face, that, oh, boy. What did I just do face? I'm in double face, that big double.
Scott Tolinski
It's very funny. These are both simultaneously wonderful and scary and hilarious all at the same time. So thank you if you've submitted your your horror story, and rid I look forward to seeing the next ones for next year, and happy Halloween for those of you who are are celebrating the spooky Halloween.
Wes Bos
Alright. Let's move into some sick pics. I got Sick pics.
Wes Bos
Sick pic. I have a sick pic for you today, and I recently upgraded ready. The new iPhone 13 Pro Max all wheel drive edition, whatever it's called.
Wes Bos
And it has the rid MagSafe charging, and my wife has it as well. So we just said, okay. Let's update the car mount from we previously I probably 3 or 4 years ago, I picked rid. The magnetic car mount that goes in your CD player, which is awesome, but now we're like, we want the one that's magnet mount, rid. And it charges the phone because then you just have to click it into it, and you're you're up and running. So I asked on Twitter, like, is it any good? Like, is it gonna fall off? Like, rid. You see the the MagSafe, and it comes off, like, super easy. I was like, there's no way that's gonna go over our neglected streets here in Hamilton. Rid So I found this one called the ESR Halo Lock Magnetic Wireless Car Charger.
Wes Bos
And the way that it works is if you have a case rid That's not MagSafe or whatever. Like, if you just clip your phone to it, it works great. But if you have a case like most people do What if you have a phone that doesn't have rid. MagSafe. Could I put this magnet on the back of my phone? Yeah. Well, it's so it's a little stainless steel halo So I can that goes on the back rid your phone. I can do that. Yeah. And it makes the magnet super, super strong. I went over some huge speed bumps pretty fast and all kinds of stuff and no problem. Rid. And I was shaking the thing. It did eventually fall off, but much more than it would. I was so skeptical that it would be good, and it is rid. Awesome. Very, very strong, and it charges the phone. I am waiting on a faster or higher voltage, Like cigarette adapter, so I'll probably sick pick that in the coming episode once I give it a shot, but I was very impressed with with this. So it's called the ESR Halo Lock. Rid. You have to buy the it's a vent mount one, which is we're getting a new car tomorrow, so I was, like, fine. I the vent mount didn't work well in our rid. Our BMW that we have now, but I hope it does in the in the new van that we get. And they also have one that, like, sticks to the windshield as well, but I was pretty happy with it. Rid I need to get one of these. Yeah. It's a good one. I don't have the MagSafe on my phone, but I have a wireless charging. And I don't have any mount in my car right now. Not that I drive that much, but still. Yeah. Like, honestly, like, it's it's also still a good like, if you don't have it, I probably it's $35, and you could get, like, a decent just magnet mount rid. For probably less than that. But if you just wanna think about, like, future forward, then that's good. It's also it's not MagSafe.
Wes Bos
Rid MedSafe charges at, like, 5th 18 watts or something like that, but it should charge fast enough once I have the proper rid Cigarette adapter thing. Probably charges as fast as anything else in my car right now. Yeah. We've got these, like, gas station rid. 5 volt ones and, like, they've been fine for us all this time.
Wes Bos
So getting a proper one, we do have, like, 1 little car adapter that is it detects rid How fast it can charge? Like, it's one of those, like, Smart IQ ones. Yeah. But I think it's from Anker or something like that, and it it charges the kids' iPads super fast. Wow. That was in the back for the kids, not in the front for us. Of my new car, whenever it exists, whenever that may be Yep. It has plugs in it, Like, AC plugs in it. Yeah. Yeah. Like, several of them. I'm like, oh, yeah. That's gonna be the best. That's awesome. We our van, We're getting tomorrow, fingers crossed, has those as well. And I just Yeah. Do think sometimes I see people have I see pictures of people who have those, and they stick the, like, rid Regular iPhone wall adapter into that, and it drives me crazy because the power coming from your your battery and you're alternating your car's DC, And then you're running that through a Converter. Converter or inverter that changes it to AC, and then you run it through your wall wart Outlet. It turns it back into DC.
Wes Bos
That's, like, the most inefficient way to charge your phone ever. Yep.
Wes Bos
Don't do it.
Scott Tolinski
Rid I have a a sick pick today, and it's a portable air compressor, which I didn't think this thing would be any good. I needed a bike pump rid For when he got a soccer ball, he's got some bike tires that need pumping up. We didn't have a bike pump. And, you know, the bike pumps are are you know, they're, like, 10, $12 anyways. So I found this one that was a electric one, a portable mini air inflator with a tire indicator, and it charges via USB.
Scott Tolinski
And I I I thought there's no way this thing could be any good because this is a little $34 air compressor, a $34 little air compressor that doesn't even plug into the wall. Read. And what was so neat is that you charged it up, you turn it on, I plugged it into the soccer ball, I selected ball, and then it filled it up. And it stopped automatically when it was supposed to stop, and it it I didn't have to even, like, check to see, like, what the pressure level was. It just filled it up and then stopped. I was like, well, this is, like, super handy for bike tires or anything you got around the house. It says in here that you can use it to Fill up all sorts of things, including a car tire. And I'm just thinking, like Oh, yeah. That'd be wild. Nice to have in your car. The auto cutoff is re He because, like, if you can tell it to go to, 33TSI Right. That's gonna be awesome. Then it it just cuts off. Like, that's You're saving your tires because sometimes people are like, ah, a little over, a little under, whatever, and then you wear your tires unevenly. Hey. Can we talk about those? The air tire pressure gauges. You push it, and it says one thing. You take it out. You push it, and it says something else. You take it out. You put it, and you're like, I don't even know. I'm just guessing at this point. I'm better off just using my eyes and pretending that I know what it is. Rid Yeah. Those things are the worst. So Man, I need this because we I have at the cottage, we've got a tractor and a ATV and kids' bikes, and All of them are far away from where you actually need to pump it up. Yeah. And I always have to drag an extension cord out or bring the thing close enough to the cigarette adapter, But just being able to, like, walk over, this is awesome. It's it's pretty cheap. I was shocked that this thing existed and for as cheap as it did. That's rid Yeah. And it worked very well. 1st shot, I was not expecting. I was, like, $34. It's a good bet. I'll I'll give it a try. Whatever. It's gotta at least work to fill up a soccer ball, And it definitely did. It was definitely succeeded there. Alright.
Scott Tolinski
Shameless plugs. I'm going to shamelessly plug the latest course on level uptutorialsdot rid com, which is actually, I'll I'll plug 2 of them. We have 2 courses now. 1 that came out last month from Ryan Olinska and is on Prisma, Modern GraphQL database with the Prisma, and he works with even, all sorts of really neat APIs. It's a really substantial series, and Ryan is an excellent, excellent, excellent teacher. Rid. If you're interested in learning a little bit more about GraphQL and Prisma, which is a really neat tech, head on over to level up tutorials dot rid com forward slash pro. Sign up for the year and save 25%. Also, the course that's coming out for this month, October, just about any day now, is going to be rid On Astro. So if you've never heard of Astro, Astro's a neat new tool that allows you to write in any front end framework and have ship way less JavaScript. Ready. It's very, very cool. So if you're interested in learning Astro or Prisma or Svelte or anything that we teach, level up tutorials .com.
Scott Tolinski
Thank you so much, and,
Wes Bos
check it out. I'll shamelessly plug my advanced react course at advanced react.com, and I have just updated it ready. For I added a whole bunch of the whole testing module, and the deployment module is coming probably by the time you listen to it. That will be there as well. So check it out if you wanna learn. Rid React, Stripe, Checkout, Ecommerce, GraphQL, Apollo, all of that good stuff. Advancedreact.com.
Wes Bos
You can use coupon code syntax for an extra $10 off. Rid Alright. That's it. Thank you for tuning in to the spooky episode of Syntax. We will catch you later.
Scott Tolinski
Head on over to syntax.fm for a full archive of all of our shows. And don't forget to subscribe in your podcast player or rid Drop
Guest 1
a review if you like this show.