Spooky Coding Horror Stories 2023 - Part 2

Welcome to Syntax.

This is our annual rid Spooky coding horror stories episode.

You may have heard actually the Hasty before this, but this is the tasty full of rid Coding horror stories. So every year, we ask people to submit their,

my wife just texted me, is Dracula in your office? Good. I nailed it then. You were really loud. In fact, I had my headphones turned up, and you started that intro so loudly that I, like, and almost shot the earbuds out of my ear.

So these are stories that make you want to put your head in the sand. Stories of Caution stories of people totally dropping the ball on customers websites with their tech, with their database. It's just awful stuff.

So it's a good lesson both for entertainment as well as

there's something like Valuable stories to be learned here. Yeah. These stories will make you wish that you had some brains. Right? If if you had any brains, You'd be using [email protected] to track your errors and exceptions. Because when these types of spooky situations pop up, You wanna make sure that you have a, well, a non spooky pal to help you out solving those issues. So head on over to century. Io.

This podcast is presented by Century. So thanks. Let's get into these spooky stories. Wes, you wanna hit that first one? Yes. This one is wild. So, by the way, we should say

All of these stories we are keeping anonymous whether the person has asked or not just because some of the best ones are anonymous because they're total I don't want them to see the light of day, but we don't want to put anyone's name on these because if you tweeted it, you can delete the tweet. But if it's on this podcast, you can't delete the podcast. So rid thank you everybody for submitting. The first one is a crypto copy paste horror story. I worked for a crypto company in 2020. And while building the React Native app, one of my colleagues made a PR to solve a small copy to clipboard issue Where it didn't alert the user that the copied value was actually copied, he, quotes, fixed it. And I threw a a classic LGTM, looks good to me, and it was deployed.

A few hours later, 1 angry customer made a huge thread On Twitter about how he made a mistake and sent $60,000 to the wrong wallet as he thought he had copied the address rid To the right place. The UI showed that it has copied successfully, but it wasn't for some specific Android phones.

Rid Oh, we patched the fix, and the company paid him after all the buzz. TLDR always test your code.

LGTM.

Also, I thought LGTM stood for let's get This money.

So every single time I've read LGTM, I read it as let's get this money, which It makes so much more sense as looks good to me. That is hilarious. So when I first saw this in this context, I was like, oh, that's so that's even,

rid so you think every time someone, like, sends a PR and they say LGTM, they said, let's get this money, like like, that's slang for deploy the sucker?

Yeah. Like, what? This is this is our product. It's going live. I I've I'm, like, I'm having a moment of, like, I'm so stupid right now. So this is my spooky story.

And I honestly I'm having that moment too. Extremely red in the face right now. I wish y'all could see because,

I'm very embarrassed to admit that, and I feel really dumb. So rid Alright. This next one, I think, might be the best one. I'm gonna let you read it, but, like Oh, yeah. Buckle up. Maybe if you're driving, pull over. This is a big one.

Re Buckle up, strap yourself in, and get ready.

Big Brother bug. In my early career, mid 2000, my 1st week on the job as a software dev Was for a local big brother production.

I won't say what country. Big brother is in the, the TV show. Right? So is a TV show where people are in house, and it's a competition reality show in case you haven't heard of Big Brother.

I was Tasked with writing an algorithm for that week's house vote.

The logic was intricate. Something like The public had a percentage of the vote determined by, text message count. The house contestants had a percentage.

The previous week's winner had a percentage, and Big Brother, the production team, had their own percentage.

I wrote the algorithm and was pretty happy about it. However, come Sunday, as the votes were tallied and the auditor checked the results, He wrote down the name of the loser and then walked on stage to hand the envelope to the presenter.

The contestant to leave the house was announced.

And the announcement was made, I double checked my code and spotted a bug. Rid After correcting it and running the code again, the results flipped, And the person announced as the loser was actually the winner.

Essentially, the most popular house guest was leaving.

I kept this to myself for years.

I had night sweats for months.

I still think about the butterfly effect it would have caused As it was the number one show in quite a number of countries.

I can only laugh about it now. Oh. Oh, man. Oh. Like like that's

Like, how much do you win at Big Brother? Isn't it like $1,000,000

if you win? Oh, so this is glad that this one is anonymous because That is a very rough situation.

You know what? You know, we've given some of these scores in the past. Rid. Now I'm going to be giving this, like, the golden the golden spider award for their being the the spookiest story I've ever seen. Oh. Oh my gosh. Brutal. You know what? I guarantee

the bug was, you know, when you're trying to, like, find a percentage or something and you have to, like, take 1 minus that percentage, I bet rid. I bet that was the bug. That bug. Yes. Because when you're working with percentages,

it is easy to have that flip flop happen. I wonder everyone must have been outraged with the that I I wrote myself. I wrote That's crazy.

For a voting platform before for a breakdance competition, where judges' votes were tallied and then put through a thing, and it would reveal the winner. And the 1st time they used it was in, like, a, A battle in South Korea. And I remember, like, thinking to myself, I am so pooched if this messes up. Rid.

Luckily, it was fine. So, yeah. So many of these stories, the stakes are so high and they have to do with People's lives or a significant amount of money.

So next one. For 2 years, I had one of the 6 laptops That could fix npm when it fell over. So this one was submitted anonymously, but from somebody who was a major player at npm in the rid. Not the early days, but in the mid days, the feeling of dread when I would occasionally misplace that computer in my apartment was true horror. So I asked him, like, that's wild.

How come there's only 6 laptops in the world that could fix NPM if it fell over? And he said there was rid. SSH keys on those computers that were not backed up anywhere for security purposes.

Oh, the dread. The feeling of dread. Both both losing it, but also somebody getting their hands on that. You know? Yeah. You'd have to know what you had, but still, Yeah. That is, that's that's spooky.

Alright. Next one. I once implemented a useEffect wrong and was auto submitting users' payments before they could submit before they click the buy now CTA.

Rid. Dear gosh. Auto submitting payments. Here's another case. When when, just just for those of you out there, if you're working rid. Ever with a payment system, and before I read the rest of this, and you do have an action to submit a payment, the very first thing that should be done on that action to submit the payment should be to disable the button, especially if you're you're working in some sort of click, but I guess this might have been outside of that. So, rid Okay. I once implement implemented a useEffect wrong and was auto submitting user payments before they clicked the buy now CTA.

It was a full purchase flow. And near the end, they put their payment info in, but there were some other steps for them to manually Click a button where they would actually make the purchase.

But instead, they put their payment info in, then got redirected to the final rid Confirmation stage without them manually confirming.

I think, like, 100 orders got through. Our traffic wasn't super high.

There were there was a big meeting with the business people to be made aware about it, and the higher ups were basically like, well, let's just see how it Hands out and see if these people contact us. Oh, no. That's not right. That's not the right thing. I think all but 4 of those orders activated, And it was our highest conversion month. I like to brag that my bugs are run of you revenue generators. Oh my gosh.

No, people. Man.

The thing is the right thing to do is to refund everybody without them having to contact you. That's

Oh, that's wrong. Next one. So I accidentally turned on a feature before it was ready. Nobody noticed for 4 days. By the time we did, forty rid.

$1,000. No. Not $40,000.

40,000 orders had been shipped to the customers, but then also refunded.

It took another week to notice that nothing we can do about it. 40,000 people just got a whole bunch of free stuff. A few days later, I was fine. The total cost of the company was £700,000.

That's that might be the 1st $1,000,000

mistake. Is that is that a 1000000 dollar company? We've had a lot of expensive mistakes on these spooky stories. Dollar mistake. But that one is you know that, did you watch that TV show on Netflix, The $1,000,000 menu.

It's like oh, there's a $1,000,000 man million pound menu.

That that's right there. Holy cow. $700,000.

£13,000. That's a $1,000,000

US. Pounds. You're right. Oh my gosh. I I, like, I recoil reading these. This is This is simultaneously my favorite episode of the year because I feel like it's the most laughs per second,

just because of how Bad. These are but they're cringey laughs. They're they're laughs that come from a deep place of Yeah. I also I spent all morning going through these, and I'm so stressed out right now just from, like, Oh. Oh. Oh. Oh.

I have so much tension in my body reading these. Alright. On the on the 1st week of my internship, I dropped the analytics database for the past 6 months.

I was trying to build a view with just the data I needed, And I dropped the source data instead of the view. Honestly, I blame them for giving the intern full access to the database. Yeah. Right. You are. Yeah. Don't give the intern The ability to drop it. Yeah.

Data phase.

2nd week on the job was testing SQL injection attacks on our dev site.

Rid. Example. So SQL injection is like you have like an input and and you're like, what's your name? And you say my name is drop DB dev. So he typed in, like, semicolon t equals t drop d v dev.

I got a support email saying they couldn't access the site anymore. Rid. Turns out our dev DB was named dev_ dev. Dev was actually production.

Happened at 11:57.

The next DB backup was at 12. Oh.

My own.

I wanna say, I I also feel like, accidentally having That dev to production thing, I accidentally did that with PlanetScale when we did our coupon voucher site because they have, like, an interface for swapping them, and I accidentally rid Swapped them myself. I fixed it, and I put it back to where it was. Oh, yeah? But that is easier than you might expect, so I feel for you. This is from a past guest. You'll know who this is because of of the website, but past guest on the show. Yeah. Oh, from the website.

Yes. Okay. When updating the DNS for, polyfill dot I o. There's your hand. I got 1 digit incorrect and took it completely offline for 2 hours.

As after updating the DNS, I immediately got on the train and had no signal.

So I was unaware that people were trying to contact me about it being offline. Now this is a service folks, if you don't know about polyfill dye, this is a service that people are regularly hitting. So It being just straight up down for 2 hours.

30,000,000 hits a day or something like that.

Like a good chunk of the Internet. It's like things polyfill.

Io to polyfill APIs that are not available in some browsers.

So rid. It not being available means people's entire applications break. That's like AWS going down without, you know, being a hosting thing. I wrote a large application performance management software similar to Century.

This happened about 10 years ago, But one of our offerings was so called synthetic tests, basically a way to ensure for a company that Their website works when accessed all over the world. A customer can configure a URL and optionally some JavaScript to click a few buttons. And if not nothing breaks, it test rid. Passes. We would then periodically run these tests on all our machines all over the globe, which we had hosted on AWS. If The test would fail. The customer would get an alert. Everything was fine until 1 weekend when a team came back to find lots of the AWS alerts on high cost. Rid. Turns out a trial customer configured lots of synthetic tests to hit his website and stay there for some seconds.

Well, in the background of his website was a Bitcoin miner running. So basically, he used our AWS machines to mine bit We lost $50,000 due to this exploit on 1 weekend and quickly had to add some detection rid To fix the exploits as well as we could simply not turn off the feature altogether.

Oof. That those are the worst bugs where rid It's being used legitimately in a specific way, but also it could be used nefariously. And it's always Bitcoin miners. If you let somebody run code anywhere. Even if you let somebody hit a URL and like, sit there for 3 seconds or 15 seconds.

Then they have 15 seconds of your compute running, and they can do

Nefarious things with that compute. Yeah. Bitcoin miners are like, it's like life. They will find a way. Right? Did you know That Mongo's dollar sign out aggregation stage clears the collection you're saving too.

That's That's a good start. What a good hook, this person. This is a person who's, like, writing a story.

I didn't I didn't. Not the 1st time I used it. I cleared 80,000,000 records in one swift move with 0 backup. No. Stop.

It wasn't user data, though, But a dataset to train AI that took me months to build and recover.

Oh, please. 80,000,000 records With no backup, if you got 8,000,000 records, he gotta make a backup somewhere.

I asked him if it was time consuming or expensive, and he said rid Both. Yeah. Yeah. Oh. Next one, web chat DDoS. I'm a front end At a company that does software for customer service many years ago is working on a prototyping web chat feature. Once it seems rid. Once it seems stable enough, we would let our customers put chat on their websites. That's when we realized our code was now loaded onto a lot more computers than our main app, And it was making way too many calls to our server. Basically, we were DDoSing ourselves with our new web chat product. We took it down as soon as we could, but it was already loaded on a bunch of pages that were still hammering us. So like, I imagine they have rid A piece of JavaScript somewhere that people load up a chat, and every 5 seconds, it's pinging an endpoint to rid. Say, hey. Are there new messages? Are the new messages? Are the new messages? So even if you take it down, that JavaScript is still running In somebody's browser and pinging them. So it goes on to say, I'm sure there were people out there with a bunch tabs open in Chrome, not realizing it was making a bunch of people sweat in a conference room somewhere.

Dear gosh.

We we see kind of the same stories over and over again in a in a different way. Right? Accidental DDoS, rid drop the database. Accidentally charge a 1000000000 people. It's, like, so funny how the these same patterns in in I mean, in software.

Oh my gosh. And I'm I'm I'm seeing the next one right now, and I'm already in tears looking at part of the next one. So okay. Rid. The this one is a a URL shortener.

This is great. Okay.

A while later a while later, we made a is this the same person? I gotta ask. Yeah. Because they say a while later. Submitted too. I chopped it in. No. That's great. This is a double dose then.

Good for this person. A while later, we made a URL shortening service to our URL.

Someone from the marketing team was going to demo this and Slacked us About our prank, none of us knew what he meant.

So he sent us a screenshot of the URL that the app had generated for him. The URL was and then there was a URL shortener forward slash, Capital f, capital u, lowercase c, capital k, capital m, lowercase e.

I'm gonna let you, spell that out yourself, but it is beep me.

I know you can't read it on air, but you get the idea.

This wasn't a prank. It was actually randomly generated.

But then we had to do some extra work to make sure the shortened URLs that didn't contain profanity.

Fortunately, this only happened to someone at our company and not to a customer during a demo.

Yeah. I think that's Gotta be a concern with any random generated word. You think about, like, the name generator on Heroku, Dinos, or anytime you have, like, dynamically generated anything.

Got to put a profanity filter on that.

Ontario puts out a list of rejected personalized license plates.

Oh, yes. And

it's hilarious because they have to try to decipher. Are you trying To sneak 1 by me and spell something out on your license plate that I don't know. So it's this, like, game of trying to figure out, Is this a misspelled bad word? Or and also is it like a new slang that I have not yet heard of? And It's hilarious. If you go on the Ontario website, you can look at the list of denied plates, and there's so many good ones on there. Speaking of that,

there's a podcast I listen to. It's like an Australian comedy podcast, Bunta Vista, and, That is, like, a segment of 1 of theirs is reading rejected plates.

So the they just have a segment where they're reading rejected plates from various places. Butt of everything. That's such a good plate, chicken butt. There's so many worst ones in the out west. There are they get so rough, and it is very funny. Have you ever considered getting a personalized license plate? I have

many times.

I don't know. I I don't wanna be, if I cut somebody off. I'm a pretty good driver, I I think, but Every now and then, you make the wrong You don't want to do that easily. Yeah. You identify the wrong person mad. You know? Maybe I would guess something funny, but every time I look at it, it's, like, $300.

And I was like, that I've never found one that's $300 funny. You know? Like, it's gotta be

Yeah. Really, really Honey, we have, like, new black and white license plates that are really slick in Colorado because the other ones are, like, green with mountains on them.

But I I thought of, like, a black and white one that says syntax. It's the perfect amount of characters. Our license plates are 6 characters.

It would look pretty cool. Be cool. Like on brand. What's his name on Twitter? I forgot his name right now. He has no JS. That's a custom Iowa plate.

Next one.

Rid I sent an email to 20,000 users with the wrong username and password. Basically, Excel import rid had a 1 row shift. The client was not happy. We had a terrible meeting with shouting, and we had to write an apology email to all 20,000 users.

For some reason, my boss was chill the entire time and didn't say anything to me. Everybody on Twitter replied to that, like veteran.

You know? He's seen it all. Next one, dodged it.

My 1st web dev gig set in the heart of downtown Vancouver.

I was a newbie navigating the tech jungle.

When it came to making my 1st change to my code base, my boss mentioned I needed to commit my changes to CVS.

They say not the pharmacy. For all you new folks, CVS is a, version control system that's not Git.

Instead of drowning in CVS confusion. I had a light bulb moment.

Let's move our code base to GitHub.

My boss gave the knob. He was planning to make this move for years now but did not have the experience himself.

Timing, they say, is everything.

A weekend later, our office got hit by real world thieves who swiped our servers. Oh, no.

Rid Little did I know our CVS server held our entire code base.

I became an office hero, sans cape. I'll never I'll forever be the code savior who spared our code from literally being robbed.

Oh my gosh.

Rid. Yeah. That is, off-site backups. Man, off-site backups are a thing.

CVS, by the way, stands for concurrent rid Versions systems. Concurrent version system. Concurrent versions system.

I I've never never used anything other than rid get myself. So, shout out to all of you folks who have had to deal with non Git version control. Whether it's Mercurial And

what does WordPress use? SVN? That's the big one. Subversion was the big one before Git. A lot of people still use that. And then I know, like, Google and Facebook have their own versions of it because it's much bigger than Git can even handle.

Next one we have here is called Lorem Sale. Okay. So this one time back in the day, I was showing a new dev around a dev instance on a website we had built and managed For massive national airline. I love how people have to redact. And it's kind of funny because the people that emailed it to me, if you look at Where they've worked in the past, it makes these even more like, oh, the site also ran Company's APIs. It was a Drupal monolith. The site is basically a sales funny funnel for the company's separate booking engine site. Rid There was a feature where you could put a sale entity live, and it would push a notification to all of the company's double digit percentage Of the national population.

Double.

So, like, even 10% of the country at at at a minimum.

Anyways, showing the dev around, I put a sale live. Keep talking.

Put it off live then on again.

Rid As I'm talking through how it works and what happens when a sale goes live. So I just kinda explain, alright, you turn this thing on and off, and this is what happens if a sale goes live. All of a sudden, I get a handful of push notifications from the company's sales app indicating a sale went live. Ugh. Client Call incoming.

As it turns out, the ops guy had refreshed in air quotes, rid. Dev with prod without changing the DB connection script back to the dev DB. So the dev site was connected to the production database, And I was putting sales live with the dummy data in production.

Still one of my worst mistakes.

1,000,000 push notifications to customers saying Lauren Sale is now live.

A few points of policy were developed.

Oh, yeah. How much is sending a 1,000,000 push notifications cost? 1,000,000 push notifications.

Rid Do do push notifications cost money? I don't know if they do. I guess it that's a good question. I don't think they do. I think it's just SMS. I could be wrong because I I don't work that much in push notifications. Frankly, I find them to be awful. I think it depends if you have your own push notification

service or not. Of problems. Like, if Clari is doing I mean, something this large has their own infrastructure.

Hokey. Yeah. Yeah. Hokey is right. If you get a Hokey out of West, That's one hokey. Oak. That's that's pretty important. Yeah. Hokey. There might even be a couple hokey doodles in here. Yeah. We get into hokey doodle territory. You know you know it's serious biz.

Rid I wrote a bug that sent an SMS to a customer of my client 2,000 plus times in the loop, And we took 3 hours to find out. So another massive amount of notifications. Although, As we just mentioned, I think SMS, this one's gonna be quite a bit more expensive. Holy cow. Imagine getting 2,000

rid. Push notifications to your phone or text messages. That's a mess. That's that's wild. Like yeah. What what does that do? When I sold my stickers, rid. I got, I think, like, 5,000, 6,000 notifications in, rid. Like, 6:6 hours or so, and it was just constant. My phone battery was dead. Oh, okay. Yeah.

Crazy.

Rid. We, I actually strongly opposed myself to that, shipped hundreds of kiosk hardware and software in the US From France without any remote control solution, boss had to ask a cousin living there to travel and check rid For a crash computers and to manually reboot it.

That would be so scary to me. Like, imagine You are pushing an update to ecobee or some sort of hardware that is in someone's thing. Like, it has to download the firmware, put it on itself, and then reboot itself. Like, what happens if you break it rid At that point, you know? And, like, hopefully, there's a recovery mode, but then you got support telling people how to do recovery mode. But if they're rid. If you can't get into recovery mode or you need to do some weird, like, USB stick thing, can you oh, that's sweaty moms doing that kind of stuff. Hardware is another level. Yeah. I know. Whenever it's like

whenever you you hear an alert of, like, iPhones are being bricked or this is being bricked, The first thing I do is think about all those poor devs. Whoever whoever pushed out the software that's bricking hardware.

Okay. Next one. While at a booking engine company, I made a change on prod with no staging or Git back then just as the boss walked in for a meeting.

Afterwards, I found out that I had missed a semicolon and crashed prod For a whole hour, losing 1,000 of euros.

Broke out in a cold sweat, offered to resign, and he said no. Yeah. We've all been there. Rid. You know what? If you haven't taken down prod at some point in your career

I took down Are you a real dub? Instant Tax podcast feed

rid. Yeah. You did. With DNS or what?

You know what I it was? I was using Cloudflare to do the syntax meetup URLs.

And I made the redirects. I was like, these aren't working. I was like, oh, it's because we're not you. I had Cloudflare gray clouded because we weren't using any of the other Cloudflare features. We were going straight to Purcell. Okay. So I was like, oh, we need it on. So I flipped it on and I tested the website and immediately the website broke. And I was like, I know what this is. You got to like the CloudFlare default is like flexible SSL, And that causes the HTTPS redirect loop.

So you just flip it to full strict SSL and it fixes everything.

But I had failed to remember that we also host we don't host it, but we proxy the RSS feed at feed syntax.

Fm so that if we ever change podcast providers, we own the URL.

And something happened there with the SSL, And it was doing the redirect loop.

So I Quickly threw a Cloudflare rule on there that says if it's the feed, change the SSL, the flexible, and that it fixed it right away. But our podcast was a couple of hours late and from going out, I was worried that Spotify wouldn't pick it up, but literally, like, 3 minutes later, it popped up in our feed on Spotify. So it was smooth.

Rid. They they must another weird thing, like, talking about RSS for a second, is that Spotify downloads our RSS feed Probably every 5 minutes. Right.

And it's 9 megs, and we have 600 episodes.

RSS Podcast feed doesn't get paginated. There's no spec for that. So you think about how much bandwidth these companies do. Just rid Parsing RSS feeds must be unreal. Yeah. It's not like we're the only podcast with 600 episodes. Oh, there's a lot that are are much larger. All right. This next one again, I'm keeping them anonymous, but this guy works for a very large ticketing company.

Rid. I charged credit cards and updated balances within the same database transaction, all inside of a batch job with retry logic.

Customers had their credit cards charged dozens of times until they eventually hit their credit limit. Created a terrible mess, But a few critical learnings to retry logic that kept retrying.

Rid. Oh, man.

Yikes. It can you imagine checking your credit card and seeing rid $10 worth of charges run up. Yeah. No. Thank you.

That is, Yeah. And the last thing you wanna do is have to get on The phone and do chargebacks for all that stuff are oh my gosh. Yeah. I swear I didn't order $10 worth of one thing over and over again.

Next one. I accidentally I accidentally implemented an infinite redirect loop that sent a 110,000 emails to a group of about rid 6 users causing their IT department to think they were under attack.

6 people got absolutely

Buried under a mountain of email. Oh, man. That is and if they host their own, like, rid. Mail servers, like, they're probably choking under all of that. You also can't

you can't put that one back in the bag. There's no one due for that. Undo a 100 and like, yeah. What do you do? You're just gonna have to swipe them away or whatever rid Oh, man. Even I I was curious. I looked it up. A 110,000

emails depending on, like, what they're sending. But if you're a medium sized company, It was $100 and just sending emails.

Next 1, 1st commit at a new company brought down the entire site, rid. And I had just boarded a plane for a cross country flight.

How how many times rid. Have we heard this where someone says, I did something on a Friday afternoon, and then I became totally uncontactable?

And, turns out I pushed to prod, and then I went to get into a submarine to visit the Titanic.

Next one here. Back in the day, I created an augmented reality game in Flash AS 3, that's ActionScript 3, for a customer to use in public places such as malls.

Every winner receives a small prize. All of the data was already randomly generated in an XML with the exact time when people can win. And I was saving the index to the last 1 item into a shared object, which is like local storage or cookies these days. But the problem is is that you need to call Flash for the data to persist on the hard drive, which I obviously didn't.

So whenever the computer crashes or they take a break or they reload the app, the index will be reset to 0, And all the previously won prizes will appear again. So the customer is handing prizes all day Long. Yeah. A lot of big winners that day. Hey.

That one's actually good for the user for a change. All of these ones, like, They they cause havoc to the user, but this one, hey. Free prizes. I'm pretty sure in in Canada, we had

roll up The Rim, which is Tim Hortons. You roll the rim up and you see if you win, like, a doughnut or a car or something like that.

And they have rid. Stupidly moved it to some app based that you you do, like, a fake role. During COVID, they nixed it. Now they have an app so they can track you and rid. Oh, stuff is awful.

It's like the one, like, nostalgic thing I have about Tim Hortons. But, ready.

At one point, all these people were winning. I think, like, I think it was a car or something significant, you know, like rid. 1,000 of dollars of prize at TV, and, they had to, like, claw them back because they were accidentally, rid. You know, like, for every one of these stories, people will tell me there's probably 10 more that people will never they're tight lipped about.

Next one, a $20,000 hour. I borked a logical expression in a condition rid. For some credit card processing and caused easily $20,000 of damages in a few hours by making everything free.

Rid. If you're not if you're not following along, an if statement was goofed up, Probably some sort of multiple and or the parentheses in the wrong spot And literally

everything. It would be very interesting for somebody to tell tally up the amount of dollars lost over every one of these stories.

Oh, okay. Next one is 3 years ago, I did a major release, pushed hundreds of commits from Staging to production after extensive testing.

The site went down for 3 days. Oh. I had no idea what went wrong Like a needle in the haystack. I mean, that's the worst feeling. Right? Things down, you have no idea why. Then I found a single line regex was bottlenecking All queries.

Did not sleep for 48 hours straight trying to look for the issue. It was a different time and place. Rid Man. Man. Yeah.

48 hours.

Just I you know, imagine, like, the amount of Stress you're under in that amount of time. I've I've I've had some sleepless nights where I'm up coding trying to fix a bug myself. Yeah. And there's really not a whole lot. Like, You know, your your partner can be like, hey. You doing okay? Is everything okay? No. It's not okay. I'm I'm on fire.

Rid. I I don't wanna be a a century, Schmuck here, but Yeah. Slow DB query detection, you know? And like, there's lots of tools out there that will like I'm not sure if the regex was in the query or if the regex was somewhere, but It probably was something weird where someone rejects a piece of text and that text was not rid. You can't do that just in the DB. So it probably had to query every single record into memory and loop over them. You know, That is crazy.

But, yeah, you certainly should have some tooling around telling you when rid a query and what the query is that is slow.

Next one. Accidentally truncated the production database one. We had a weird setup where we needed to connect the fraud because debugging was near impossible.

So, yeah, that happened. We changed all the rights of the database User so we didn't have the problem anymore.

Brutal. And then somebody followed up, said, didn't you have a bunch of medical equipment delivered to a customer At one point, and he says, yes, lots of bed hooks. However, that wasn't my fault. Someone made the test test form point to live.

That one like dropping a database. Yeah, we've heard it. But like we're customers.

We had that one with toilet paper a couple of years ago. Somebody literally shipped, like, like 15 bundles of toilet paper to somebody's random house, and it just started showing up. When product actually arrives at random people's houses, that is just that's hilarious to me. It's unfortunately hilarious, especially if it's large amounts of things like toilet paper.

Next one. I once made an off by 1 error writing a high efficiency load balancer for a platform as a service So all domains served the wrong traffic domain sites index for food.com was actually served rid Sites index for Food.com

plus 1. So you can imagine, like like, Netlify is a platform as a service. I don't this probably wasn't Netlify. I don't know I don't know who submitted it, but imagine Netlify had has a load balancer, and every single request They get in for westboss.com is off by 1. Is off by 1, so I get, like, diapers.net.

You know? Yeah. Oh, yeah. Diapers.net.

Rid Oh, yeah. There was a I I know this isn't the same thing, but there was, like, a thing with, like, Steam, the gaming platform, Where they have, like, a similar issue that was serving up the wrong cache to the wrong user, and it was a similar bug, I believe.

So yeah. Broke everything, and people thought Every site had been hacked or discontinued for a day or so. I was ex was extremely confusing as it wasn't a simple index lookup. Yeah. No kidding. Yeah. And, also, you know, you give people somebody else's anything, and, like, that that that hurts customer trust pretty badly. Right? Yeah. No kidding.

Rid. In 2007, the author, who was 15 at the time, worked as a freelancer, and developed a web store for a local music company.

The client's tech wizard son who served as the project's reviewer and administrator inadvertently removed removed exception handling rid Handling.

So causing an error message containing the database credentials to be displayed on the website's home page. Rid Unfortunately, this error was indexed by search engines during a maintenance window, exposing sensitive credentials in search results.

Despite The Sun's responsibility for the issue, the client unfairly held the author accountable, Demanding a refund equivalent to the amount paid for the freelance work. Regrettably, due to their youth fear and lack of knowledge of their rights, the author rid complied with this demand. Subsequently, the author never had any further contact with the client. Oh, that sucks. Bro. Oh, that's not even funny. That's sad. That's sad. You know what sucks about that is, like I mean, you think about the horror of committing

Your secrets to GitHub. I mean, this is like I mean, GitHub's permanent. You can just, like, GitHub, change your seat, but to have it indexed by search engines. That's why it is so hard to

like, in remix, Next. Js, Gatsby, all these frameworks, For you to explicitly put an environmental variable into your template, you have to prefix it with Next underscore because they don't want anything ever accidentally happening where you possibly leak, especially with these, like, full stack. You're not sure if your back end or front end, you have to explicitly put them in To make say, yes. I want this to be on the client side. Yeah. It's the same concept as behind

dangerously set inner HTML or whatever. It's like, We're putting these words here or putting this extra step in here so that you think about it. Just so you you think about it. Rid. Alright. Next one here. I just put a note here. I've heard this story a several times over the years, and this is This is a cautionary tale. I wasn't sure if this was an author's note or a West Boss note. So this is a West Boss note. That's a West Boss note. Yeah. This is a cautionary tale. A few years back, I SSH'd into a prod server. And when trying to delete a temp directory, I accidentally executed r m hyphen r f till date forward slash. Till date forward slash for those of you who don't know, it's the home directory.

The temp directory was supposed to autocomplete, but before it did, rid. But before it did, I pressed enter too soon, and the data was gone. And, you you know, people Keep all sorts of stuff in their, home directories.

Could be keys. Could be I mean, some people put, like, sites and stuff. The entire computer is in tilde.

Rid. That's the root of the hard drive. Oh, the So No. Till day till day isn't the root of the hard drive. Forward slash is the root. Sorry. It's the root of the user. Rid. Yeah. You're a user home. Yeah. And especially if you have, like, a droplet or something, I never put anything in my user directory. But I know what I'm saying is I know some people put a lot of stuff in their user rid. And on a, like, a Mac computer, there's a lot of stuff on your user directory, including system files. But yeah.

Rid Woof. Yeah. If you were to do that on a Mac, your entire computer folder. Would be gone. All of your apps, all of your data, All of your set like, pretty much the entire computer unless somebody else had another account on that computer as well, but that's rid. That's unlikely.

Yeah. Totally. Oh.

Oh. That is it for today.

Thank you, everybody, for submitting your spooky stories. If you Still have a spooky story. A lot of the ones we have here are from last year. People submitted them after listening to this thing. So send me an email, [email protected], And I will put them in the queue for next year. We love doing this, so please send us your horror story.

Yeah. If anything, The world can learn from your mistakes,

and, you can have a little chuckle. I think that's the key of this. This isn't just us laughing at other people's misfortune, But, like, hey.

You know, what do they say? You know, people who know the History Channel, don't repeat the History Channel. You know what I mean? So, rid I I think that is an important important thing that we all can learn from each other's mistakes. We can laugh at these things. It is a unifying aspect of being a software developer.

However many times you you really think you have everything dialed in, Everybody makes mistakes, and it's important to really celebrate those mistakes in a fun fun, laugh y way where we all get to cringe together. So, yeah, That's it for this year's horror stories.

Spooky spooky stories, all that stuff. My bones are rattling. I'm ready to get out of here. West, do you have a sick pick for us today? I do. My sick pick is

100 pound magnetic hooks.

Oh, wow.

Rid. I've been using it for 2 things. So first, in the gym, I have all these attachments that I need. I got handles, I got bars, rid. I've got what else? Other like bands and things like that. And you need to be able to store them somewhere. Right. And The whole rack that I have is metal. So I bought these a 100 pound. They sell really cheap ones, like 25 pound ones.

And I'll say, I don't think those are good for anything that you would be.

They get knocked off all the time. I hate One of my biggest pet peeves in life is a weak magnet where things don't stay on properly. You know, come on, make this like, I want to be able to pry this thing off. I want to be able to pinch my finger in between it.

So I got these 100 pound ones and they are awesome.

So I literally am holding up like a I don't know. It's probably a 15 pound bar. It says a £100.

I don't know if I believe that That's like Amazon. All this crap is just a bunch of lies, bro. Bunch of lies. I guess you probably have a story about that. I do. But They're super handy. And I also put them on my whiteboard in our kitchen and I hang my keys on it. And it's like the best little hook to hang keys on. You can move it around and Put them on the side of the fridge as well. So if you are looking for decent

magnetic hooks, check these out. Yeah. At first, I laughed at your weak magnet thing, but then, like, as I was thinking about it, I bought, like, this, like, magnet based pop socket competitor.

I'm gonna return it because it stinks.

And my biggest complaint is that the magnet on it's weak, so it doesn't you know, it's not you can't, like, trust it if you're using it as a pop Socket, which the PopSocket branded ones actually do have a stronger magnet. So, yeah, you're right about that. My my, Amazon story is that I got a a waterproof speaker that was 300 watts on Amazon, and, it stopped working. And when I opened it up, obviously, the thing's full of water. There there's no waterproofing in it. In fact, the only waterproofing even being done like, there's no there's not even a gasket or a seal around the plastic, which, You know, you can't tell from just looking at it. So when I opened it up, I was like, alright. There's not even a gasket here. And the only bit of waterproofing was, like, quota was, like, a It was like a, really crappy kind of, like, cotton ball material in there that they were probably just hoping would suck up some water. I have no idea.

And or it could have been an urge for prevent riling. Either way, there was not an ounce of waterproofing done to these speakers, and it said they were 300 watts.

And the amplifier I I googled the board. The amplifier was 40 watts. It was a 40 watt board. And And it's just like I went to Amazon. I was like, listen. This is just completely misleading. It's, like, it's not that it stopped working. I don't care about that. Yeah. I mean, I do. But, like, rid. It's not even close to the product that it says it is.

So much of that Amazon stuff are just, like, straight up lies, You know? And you can straight up lies. Yeah. Sometimes you see one with, like, like, like, the magnet hooks, and you'll see one for £105.

You're like, yeah. Right. Rid Yeah. Right.

Yeah. I know.

I have a sick pick that is a documentary on Netflix, rid. And it's exactly my type of documentary.

It is the mountain climbing, mountaineering documentary. You know I love those.

And rid. This one is really fascinating because it's just like any of these other, like, really modern mountaineering documentaries, this is a 2003 one. It's called Race to the Summit. Just like any of these modern one, your jaw's just kind of on the floor the whole time.

This one is about speed climbers, and It's about, like, 2 2 different climbers who are doing several different alpine climbs as well as Himalayan climbs, but they're rid. Trying to break the records to see how fast they can do them. And some of these climbs like the are, like, historically Very scary climbs that, like, take people a long time, and these guys are doing it in, like, a couple of hours. And is he's like, this section has ropes, but, like, If I really wanna move fast, I, you know, I I don't wanna use the ropes. I wanna do this entirely without ropes. So not only is he doing the scariest climb of all time, he's doing it Without ropes entirely and basically running up the mountain to the point where, like, there there's times where rid He's, like, walking around. I mean, there's 2 different guys. They're both crazy. Walking around corners really fast, and you're just thinking, man, I couldn't even you couldn't even catch me rid Walking around a corner, like, 30 feet up that looks like this. And he's Wow. On the side of a mountain, and he's basically, like, running around it. You're just like, rid. This is absurd.

And, you know, with any of these things, you kinda have these wild kind of personalities. So the people involved in this documentary are super good. It's an hour 30, so it's a really nice nice watch. We put it on last night and had a great time watching it. So if you see this on Netflix, throw it on race to the summit. Alright. Thanks everybody for tuning in. Hopefully, you have a spooky

Halloween.

Head on over to syntax.fm for a full archive of all of our shows.

And don't forget to subscribe in your podcast player Or drop a review if you like this show.

Rid.