Spooky Coding Horror Stories 2023 - Part 1

Am I talking about spooky stories? We're talking about Spooky dev stories. This is one of our annual episodes where we do where we talk about spooky stories. I'm not gonna do the spooky voice the whole time, but This is part 1. This is the hasty version of spooky stories that we're gonna be telling on the podcast where basically people write in. They talk about Spooky dev stories.

Dev stories that are very spooky, spooky, spooky, spooky stuff like dropping the whole database or maybe charging 10,000 users at, You know, who knows? My name is Scott Tolinski. I'm a developer from Denver, and with me as always is Wes.

Wes, oh, I like that. Yes. Thank you. Freestyle. Alright. So we have 2 shows for you. This is gonna be the first one, a little hazy. Just a couple. We're going to get you wet your whistle.

And then on Wednesday coming out, we have the big one, which has some some doozies in it. You certainly don't want to miss that one either. These are going to be fantastic stories of Wanting to stick your head in the sand, ways people have goofed up in tech, deploying, deleting databases, all that

awful, awful stuff. We should talk about what you do if you're scared, because maybe you're frightened. You it's Spooky Saturday, and you pushed a code on Friday, and, you don't have error in exception handling tracking. You gotta have that. So check it out at century.i0.

And, yeah. Just just let them know that you you heard about it from Syntax. That'd be that'd be pretty rad of you. So let's get into it here. Alright. First1 we have here, my dev horror story was I was only 1 month into my 1st job as a junior web developer. These are always the most awful ones. There's so many that are, like, on my 1st day on the job, my 1st week on the job, I was working on-site updates for one of the largest furniture retailers in the country.

I was still very inexperienced and wrote some JavaScript. We worked on the page I was making updates to. However, I had no null checks in the JavaScript bundle to run on every other page. So if the element didn't exist on the pages, JavaScript would break.

Very common thing. You got to check if the element exists there before you go ahead and use it. And of course, it runs if the element's there. But if that JavaScript runs on another page, thing will break. Somehow this code passed PR review in internal testing as they were only testing the page updates had been made on.

Interesting.

So the code was deployed to production completely breaking JavaScript across the site right before the retailer's biggest Online sale of the year.

What's worse is the deployment to production happened on a Thursday, and this issue was not Notice until late Friday afternoon.

Uh-huh. I realized what had happened and told my manager, but I was a junior developer, and my manager said I must be wrong. There's no way something small as a missing null check could have caused a What issue? I was told to go home so I can't stop laughing and that others would have to stay late on Friday. That's not funny at all to attempt to resolve or roll back a deployment.

I never was blamed or called out for causing the issues, but the guilt of causing the team members to work late on a Friday night for my mistake still haunts me to this day. Oh, brutal, brutal, brutal. Hopefully, it was just a, like, quick little if statement in there, but Who knows? Yeah. Woof.

Alright. Next one. The embarrassing test page incident. Many years ago in my 1st web developer job, published a test page with a silly message to a WordPress site. Folks, keep your silly messages for for Slack. Never. Never. I've I've done it myself many times. They will get out to to real people. We we've all done it. Yeah. Little did I know, the website Site had a live Twitter feed that displayed the latest tweets from the company's account. To my horror, the test message appeared on the feed Visible to 12,000 followers.

Although I promptly deleted the page, the tweet remained.

We had We had to seek help from another SEO firm to remove the embarrassing tweet. Oh, dang, folks. You gotta get control of your own Twitter. I think it's a little bit of an own goal here that you can't delete tweets from your own Twitter account.

Yeah. No kidding. That's wild. Yeah. Wow. That is Crazy. Yeah. You think those things are kind of funny where they set off a chain chain reaction. You know, you think, oh, I'll just delete This thing and everything will be fixed. But those things unravel and they go like even sometimes we Publish a podcast before it should be out.

Do we hear about it, man? Like, even if you turn it off, There's it's still in caches around the world, and they still ripple through. And it's one of those things where it sets off a chain reaction. It can be impossible to put the genie back back into the bottle. Yeah. I was still getting messages about the episode that didn't work, like yesterday even after you had fixed it. Yeah. The Quest oversight. As a lead developer for a web established SaaS product, my role included reviewing and deploying pull requests to production. During a busy period, I Received numerous Pro requests with substantial code changes.

One particular request had extensive occasions, spending multiple files and lines of code.

Feeling overwhelmed, I briefly skimmed the change and tested them on my end. Satisfied with the results, I merged and deployed the request. However, that's much better than the last one we read, which was LGTM.

Looks good to me. Well, a few days later money.

We received a support ticket reporting a concerning issue.

Users were able to log in with incorrect passwords and even access other users' accounts.

Tips. Upon investigation, I discovered that the authentication code had been mistakenly altered. The developer responsible confessed that the disabled passwords Do we do to forgetting their own? Oh, man. Yeah. Uh-huh. Sometimes Sometimes you think, you know what? I I'm working on this feature. Let's just disable this other thing that gets in the way so I can focus on the feature.

Don't forget. I fixed it. Pushing it high.

Oh, from that point on, I implemented stricter review protocols and sought assistance from the team to prevent such oversights in the future. Honestly, that is such a human mistake that the only thing that can save you from that is automated test of testing every single make sure that it logs in. Make sure that it incorrect password doesn't work. Make sure that a blank password doesn't work.

Oh, woof.

That's a bad one too because that take a straight up security incident. The email is gonna be going out.

If this is, like, a a public company that's, you know, money off the stock price or whatever, could you imagine the the value That would actually have an a real, like, big scenario. Yeah. Luckily, hopefully, this was a a small scenario a small time company.

Alright. Next 1 is this is a developer horror story from my 1st professional role. I was tasked with making changes to a JSON file, essentially a lookup table We're integrating a learning management system with a student management system that would later pass data to our national education registry.

LMS to c SMS to NER.

I I intentionally didn't read this one because I could Tell that it was very well put together,

so I didn't read this one. And I'm cringing at the very first thing, which is 2 systems put together Pushing data to a national registry.

Yes. Right. Yeah. You see the word national in any of these, and it, definitely gets your ears perking a little bit.

We had a script that converted a CSV into a structured JSON tree that our integration used to determine what national Competencies was to be awarded to learners as they progress through their degrees.

My responsibility was to add new competencies to this table for learners.

Without a complete understanding of how it worked, I noticed duplicate entries in the lookup table. I discussed this with a colleague, and together, we decided to remove them entirely to clean it up in quotes due to the substantial Increase in lines of both the CSV and JSON files. The differences between the 2 became significant.

When it came time for my review, my manager Glance it over without much concern. Alright, folks.

Fatal flaw number 1 here.

I decided to clean this up Without fully understanding what it does. That's like looking at, like, a box full of wires and then, like, I'm just gonna pull all these Yeah. In your does it. Yeah. Probably nothing.

Yeah. Hey. Probably nothing.

Double data? I I don't think it's running anywhere. I'll just unplug it. Alright. I mean, I don't need it. That's like when I build something, I have all these extra bolts Probably for nothing. Just yeah. Who needs them? Right? Okay. We gotta clean this up. So we pushed it to production. After another review, I Celebrated my 1st professional poll request had been deployed.

8 months later, I received an email from our client stating that over 3,000 students had incorrect competencies awarded to them, and some even had their degrees aspire to missing information.

Oh, gosh.

As my colleagues and I tried to unravel the mystery, we discovered that the system actually required those duplicates to award historical competencies to learners.

Okay. So the duplications had a meaning. Right? Believe it or not.

Reverting the changes And reintroducing the duplicates to the lookup table was straightforward, but now we needed to write a script to review approximately 60 50,000 individuals in the system ensuring they had received the correct national competencies for their degrees.

As it turns out, about 7,500 learners nationally were affected.

We created this script to simulate these updates, Had our client review and approve the differences and then nervously initiated the integration to correct the students.

After processing about 500 students, we encountered a rate limit issue.

After negotiation with the systems provider, they temporarily lifted the limits allowing us to proceed.

After in the end, we resolved the issue. I didn't get fired, though my managers were understandably frustrated.

They admitted they should have provided more oversight.

Regardless, It was an exhausting few weeks working around the clock to rectify this monumental screw up. I have to say discovering that your Pull request was the root cause of the problem after opening a ticket searching through the logs and finding the issue Yeah. Was the worst feeling I've had in my professional career.

On the surface, it might not seem like a big deal because we fixed it, but the worst part about incidents like this is knowing that your code caused significant disruptions to other people's work weeks.

Yeah. Yeah. I'd say that's And, like Like, possibly people's lives. You know? Like, if someone's degree Expired? You know, like, you you plan or or if you're like, oh, do I have all the credits? You log in. Oh, I got I got them all, and then they're clawed back.

You're like, oh, well, I just moved to Denver. I can't go back. I can't go back. I had a a nontechnical

scrub. My My, adviser in college once told me that I had enough credits to graduate, and then, like, 2 months before graduation, he was like, oh, oopsies. You were you're missing this one thing.

You're missing. So I did a a special, like, a a project where I I created a bunch of music for him, and he just gave me the credits I needed. So it worked out.

But some of these stories are like, I broke something, and we immediately realized how awful it was, and some of these stories are like, we realized on a slow burn, you know, over 8 months. This next one I'm all done with it. Slow burn.

I deployed a disallow everything robots dot txt. So robots. Txt is the file that search engines use to say what pages am I allowed to crawl.

I didn't manage analytics, so it went unnoticed for 3 months. This is for a huge ecommerce platform, and it tanked Their search rankings for a while after. Oh, you cannot quick fix that. That takes forever To come back from.

Oh, that it and, yeah, messing with search search rank can be devastating for a number of reasons. And like you said, it's not a quick fix. You can't just call up Google and say, oopsies. I made a whoopsies, and, can you fix this for me? Yeah. You're you're Definitely a big trouble at that point. You know what?

I'll I'll tell you right now for everybody looking for lessons learned on this one is sign up for I think it's like Google search indexing or something like that. Yeah. And they will email you anytime web messages, anything anytime anything weird comes up. Like, I got an email this morning of a website that I have. I have, like, a little demo video on one of my websites, and it's, like, Like, 500 pixels by 300 pixels. This is tiny. Right? And I got an email from Google saying we can't index this video because it's too small, And I went, oh, like, that's helpful to know. Like, I don't want it indexed. It's not like a video that should go in as a video thing. It's just A video for the website to show a demo, but,

yeah, that's that that's good to know. Yeah. Holy cow. Alright. Next one. I had worked On implementing GDPR deletion of customers for a few months and was ready to launch.

One of the things to delete was Customer uploaded files in AWS s three.

I meant to delete files starting with, for example, customer forward slash 123, Where 123 was the customer ID, but I accidentally forgot the trailing slash.

People who are experienced can see where this is going. Right? This means that not only files starting with customer 123 were removed, but files Starting with customer 1234 and 12345 or anything after, etcetera. In other words, deleting 1 customer could remove files from other customers.

Be because, yes, they weren't checking the end.

Luckily, we had another problem with GDPR Deletion, so we had to roll that back quickly.

A few days later, we found out about the wrongly deleted files. Thanks to that early rollback, Only about 400 files were deleted wrongly instead of tens of thousands, a blessing in disguise. Okay.

You know that you messed up When you can say that accidentally deleting 400 files was a blessing in disguise because the alternative is is is much greater than that. It's 10,000.

Yeah. This next one comes from, again, somebody who I we can't say. We keep them all anonymous, But this person was worked for some very large fang type companies.

I dropped the backing disk for the production Postgres, Then fell horribly sick. So basically, you deleted entire Postgres deleted a hard drive, which had a Postgres database on it.

While I was unable to respond on Slack that whole weekend, the rest of the team realized their credentials were on my desktop, And they couldn't deploy without contacting Amazon support.

Mistake number 2 is Only having 1 person with the magic keys, I came in still a little sick on Monday to a bunch of tired faces and dozens of DMs on Slack and no clue what had happened.

Yeah.

Being blissfully unaware.

Next one. I accidentally pushed staging code as an update through code push to production Android app of my company.

Gave me a production Android app of my company.

The rollout was also to 100% of users. Hey. Nice. Because you can do a, stage rollout on Android like that. I'm sure you can on Ios. Luckily, we realized this soon as in the production testing, users were automatically to be logged out, And the user does not exist in the staging database. Token return error. I checked the branch name and it was developed.

I still remember this moment.

We immediately halted the rollout and pushed a new update with the production code to all users. It was the worst day of my life as developer.

Only me and the QA leader are aware of this, and thanks to her, she had not told this to the morning stand up the next day. The next day, I wrote a Bash script that double checks the git branch before pushing an update to production environment. And since then, nothing like this has happened as the script Access and intermediate. Yes, folks. This is why people often have, continuous deployment, continuous integration setups because They do this stuff for you. It's like, yeah. If you push to main, it will run production. Now I don't understand exactly how that would work in an Android world. I don't do that kind of stuff.

Yeah. Pushing production pushing dev to production is definitely something I think a fair number of us have done at some point in our our careers by accident.

Next one here. Biggest tech fail. I forgot to take out a test condition and sent around 10,000 emails to 200 people each. Oh.

Wait. That's 10,000 times 200 people is I don't know how much, but a ton Wait. Ton of emails. 10,000 times 200? Oh, I know. I know what that is.

What is it? It is 2,000,000. No. It's no. It's not. 10,000 times 200.

Is it 2,000,000? 10,000.

Oh, yeah. I was thinking Oh, no. Try to call me out for math, mister. 1, 2, 3 times 200. It is 2,000,000. Now I am mister Magoo. I got the big old mister Magoo award.

Never try to do math in front of people. You always make a

Make a pool of yourself. Make a mister Magoo out of your

Next one. A previous employer, we made white label mobile app. So White label means where, like, you make, like, an app that's, like, kind of generic, and then different companies will use that app and just brand it themselves as whatever they want. One of the features we had a calling a configured phone number. Well, 1 client was a religious group with a toll three number for a prayer line.

One day, they emailed support to change it. I guess someone misread 8884 866 for the area code. And instead of a prayer line, it went to an adult hotline.

Oh, wow.

That's a great that's a the exact wrong place you'd want that call to go.

I didn't know. Maybe they had some support there. You know? Yeah. Right. Yeah. No one admitted the mistake, and we didn't try, Hard enough to find anyone who had done it. Another more technically consequential one was accidentally clearing out a varnish cache Across every single instance, which had thousands of large RSS feeds cached, this served millions of requests daily.

Back end wasn't happy. Yeah, because a Varnish cache will RSS feed might be a massive database query, So you cache the results of that. So maybe that query runs once every 10 minutes, or maybe it runs once Hey. Who knows? Right? But if you if you clear them all and they all need to rerun at the exactly same time, you're probably gonna run out of CPU or some resources, or you're gonna get a bill that is massive from your cloud service. Anyways, he has he goes on to say related to RSS feeds, you may or may not know, but if you change the GUID, GUID, guaranteed unique identifier for an item, it will cause clients to automatically redownload the associated file.

This is kind of we want to move our podcast RSS feed, but I need to look into that because That scares me of waking up one day. Somebody says, hey. I got 600 episodes of New episodes. Yeah. Yeah. Well, aren't we just we're just reserving

Our hosts.

Oh, you're saying if we move our host, if we move our podcast. If we move our host. Yeah. Yeah. Totally. Something I'm sure guarantee we're not the 1st people in the world. Luckily, we own the RSS feed. We just have to look into it. Yeah. And and what I I know is that at least the some of those services that we're looking into have specific you you tell them what service you're coming from, and we will Hopefully, white glove migrate you. So Yeah. If you accidentally change every

guaranteed unique identifier on Very popular feed that lists a ton of large audio files. It turns out that your entire back end dies a horrible death, and Amazon sends you a huge

Bill. It was just like you said, both of the, both of the instances there. Oh. That's hilarious.

Oh, man.

One time, I ran an update statement on a production ACH database making all of the routing and account numbers bogus.

We had to restore the backup from a previous night, and the data entry team had to key in all of the HCH, ACH info again.

Needless to say, they didn't like me anymore.

Yeah. Your solution to the problem is having to Key stuff in by hand, and if you have, you know, 1,000 users if you have more than a 1000 users I mean, even if you have a 100 users, that's That's like a awful time. Then Yeah. What? You can break that into task for a certain amount of people to do, but, gosh, anything that ends up in manual data entry, Brutal. Not my favorite type of thing. We should say for non Americans,

ACH is automatic clearinghouse.

That is how people in the US send Money between accounts and banks, right? So not only was this a major screw up, But, like, it's it's people's money at and, like, at stake here. You know, you don't wanna accidentally send that to the wrong person.

Yeah. Right? Can you imagine manual data entry of these bank account numbers, and then you accidentally you know, you you just goof up looking at the wrong?

Yeah. Frog wine. I once made a, the first time I ever did stickers, I had to wire The printer $10,000 to print them all.

And don't at the time, my bank didn't have the ability to do a wire online, So I had to take the invoice and the payment details from the printer.

And I walked to the bank and I sat down and it was just like, I had never seen this teller very young, and I sat down and I said, hey. Like, I need to wire $10.

And I was wiring it to China. So, like, I wasn't wiring this to another Canadian bank or maybe in the States. I was wiring it to a bank that you could not read the name of.

And so she I give her the piece of paper. She sits there and punches all the numbers in and hits the submit button.

And then all of a sudden, I don't like, a couple days later, I asked the printer, hey. Did you get the money? And they said, no.

We didn't get anything. Like, we contacted the bank. Mhmm.

And so so I took the The little piece of paper that they give you when you make a wire transfer at the bank. Yeah. I I scanned it in. I sent it on. Like, look. This is proof of payment. He said that's the wrong account number. So I looked at it, and I go, oh, no.

I just sent $10 Oh. To a random and a Chinese bank account that I do not know.

So I went into the bank. I was like, look.

You typed this number wrong, and this Poor girl was shaking.

You know? Luckily, the bank sent it back.

Like, it took, like, a week and a half. So there was, like, a week and a half of just, like, did I just send $10 into the Yeah. Into the ether? It's gone forever. It's gone forever, But luckily the bank returned it and everything was fine. But the next time I had to go in, I got the same teller and she was you could tell she was just like, Can you triple check this? You know? And we sat there, and we're like, 7, 4, 3. They're making sure every and, like, it's so funny because a copy paste solution would have fixed that. Although, in the next episode, we have a $60,000 copy paste mistake. So listen to that one.

That's a, yeah, a great one for the headline there. 60,000 copy paste mistake. I used to have to, when I worked at a record label, I used to have to wire royalties out all the time, and that was by far the most stressful part of the job. You're you're you're walking out of the bank. You're filling out that piece of paper and having to do it, like, once a week. I remember the 1st time I did it. I felt like, oh, I cannot believe this company is putting me in charge of sending out their royalties like this.

Alright. Last one here. I was once working for an agency on a high profile customer.

We were preparing for one of their peak days and needed To make some changes to their payment gateway integration so they could offer better service to their customers.

Working on the payment integration is Always scary. Very. But, yeah, as we have just talked about.

But all went fine and worked as expected.

We let it rest and waited a few days before shipping it to the website. Next morning, we deployed and started working on other features.

Within an hour, we suddenly got a call from their CMO who told us that customer service was getting calls from a customer Who couldn't pay? We sprang into action and looked into the changes I had shipped.

I made a hard coded switch to make sure I was Testing with the gateway.

Oh, this is this is classic. Oh my gosh.

I had made a hard coded switch To make sure I was testing with the test gateway, which I had forgotten to remove again. OMG.

This rejected all payment from the production site.

Fortunately, we could make the correction and deploy it, but this was back when deployments could easily take half an hour. Oh, yeah. So by the time it was out, they had lost several 100 orders.

Nothing to do but own up to my mistake and make sure it wouldn't happen again.

Oh. Man. Hey. Guess who's done this? I'm raising my hand right now. I used to, you know, level up tutorials. You're managing those the sandbox environments. You're managing the production environments. You're testing your payments. Next thing you know, you know, there there's some sort of bug. You gotta get it up and fixed. Next thing you know, that Production key and, sandbox key gets swapped somehow in your e n v, and you're you're pooched. Yeah. Because

I don't I've I don't think I've ever done that. I've almost accidentally done it. I I think we had this last year. Somebody submitted a a thing where they deployed it. And if you put real credit cards into a test environment, They'll work, you know, but they won't actually charge your credit card anything. Depends on who the provider is.

Brain brain the way Braintree works, you actually have to have a specific type of credit to get a certain response. Oh, man. He's trying to the provider is. Yeah. Yeah.

It certainly depends on who the provider is. I know there's some of them that do a range, so there's a certainly a chance it could work. My situation, Wes, was I had a Staging site that had used the sandbox. Oh, yeah. The staging site was an entirely different code base. And when I was ready to swap over, I was like, oh, yeah. I'll just make the staging site live. Oh, shoot. I forgot about the most important part, which was v and v variable change. Oh, I did it once Where

when I before I release a course, I usually let, like, 100 people buy it and and go through it and whatnot. So I had released it to the like a 100 people that wanted, like, early access and they all got emails with, like, a local domain in the email, and they weren't but they had paid real money for it, but they weren't able to, like, activate it Because the URLs were whatever. And luckily, thank goodness, they're all web developers, so I was able to say, hey.

Replace local hosts with this URL and you're good to go. But imagine that times Not having smart customers

and Yeah.

Like, times a 1000000 customers.

Yeah. Nightmare. So if If you enjoyed this, hopefully you're not too stressed out. We got another one coming for you on Wednesday. Some doozies doozies in there, so make sure you tune in to that one as well. Please submit your own. We'll we'll put them on the show for next year.

Email me at [email protected].

Email Scott. Tweet us. Whatever you gotta do, We love hearing them. Yeah. And in fact,

I have a new email, [email protected], so you can hit me up at a syntax email. That's pretty sweet. I I do wanna say, just like you said, Wes, at the jump of this episode, this is just a taster for what you're about to get on Wednesday, because What you will get on Wednesday is a barrage of of some of the spookiest stories we've ever had on this podcast, including, one of them that cost, Well The 1st $1,000,000 mistake 1st $1,000,000 mistake is syntax, spooky story history. So if that doesn't get you, hyped up, I don't know what will because that

that one to me had me rolling. It it, yeah. No. It gets really bad. $1,000,000 in, like, potential revenue. Like, literally He lost $1,000,000.

Alright. Well, buckle up, folks. It's about to get really spooky on Wednesday. So Thanks so much for listening, and we'll catch you on Wednesday. Peace. Peace.

Head on over to syntax dotfm for a full archive of all of our shows.

And don't forget to subscribe in your podcast player or drop a review if you like this show.

My name is Wes. My dog eats food on

the moon.