Spooky Web Dev Stories — Part 1

Welcome to Cinta.

Thanks.

I'm very glad you didn't you didn't hint that you were going to do that. I'm very happy about that. I didn't even think to do it until we started.

Welcome to syntax.

Today, we have the spookiest syntax ever. Yes.

We will tell you stories from web developers where it all went wrong.

With me today is Spooky Tolinski.

Who's spooky Dalinsky.

Welcome to Zidax, everyone. We are doing spooky web dev stories. I am very happy because I think the closest Wes has ever come to quitting the podcast was when I did a spooky voices for half of the podcast, and he asked me to stop.

So this has made me very happy, Wes. I'm I'm very you've you've totally made my day. Thank you so much for doing a spooky voice here. Okay. This is our spooky web dev stories episode where we are going to be reading to you stories sent into an sent into us anonymously about various spooky, web dev situations that have had taken place in the depths of the hell that is our career choice sometimes. So

Today What? We have 2 spooky sponsors, Scott.

Okay. Who who are our spooky sponsors? Wes Boo, boss.

Well,

if you if you have a headless CSS, a CMS without a HUD, it was cut off. Where might you host your Jamstack

website? Not only at Netlify.

And Node sound like a grandma now.

I don't. I love your spooky voice because it's not really how it is. It's just like it's like your it's like witch voice. Is that what you're trying to do, witch voice? And the other

spooky thing could be bugs.

Imagine having bugs in your application.

A reboot. Those with Sanity. We'll talk about them partway through the episode. Alright. I'm back to normal voice now. Three big bugs.

Alright. So these are stories that we asked you, the general public, to submit. And, man, we got a ton of them. So these are just stories where people are working in web development, and you'd accidentally delete a database or send out an email that you shouldn't have or just like these, like, stories that you hear in web development where something went wrong and you just wanna stick your head in the sand or or say, oh, oh, that was that was rough. So it's Sanity. Unfortunately, a lot of these stories could have been avoided with a lot of, testing or proper proper staging environment. So maybe this is both spooky as well as, like, cautionary tale of, what to do with production.

Yes. A cautionary tale.

Alright.

Do we want to get into the first one? Do you wanna do it, or you would would you like me to do it because this the second one is one of my favorites, and I wanna read that one. Okay. So this Node, Wes has settled the most expensive Boolean ever. I might have some alternate titles for these as well.

Early in my career, I was working at an online casino company. As with all paid services, it was crucial to make it easy for our customers to pay us. Yes. Many payment providers, however, such as PayPal, chose not to work with the companies in the gambling industry. So it's a big part of our job was to implement more niche payment providers for customers in various countries that we served.

Yeah. This this all seems like one of those issues so far that you hit, and you're like, oh, great. Nobody wants to support gambling.

Yeah. So

while I was still new at the company, our CTO integrated 1 such payment provider.

Everything went smoothly for a couple of weeks until Node tiny but also important error surfaced. You see, most providers offer some sort of way of testing their integration service without having to make any real money transaction.

Here, each transaction will be initiated with among all the usual payment information such as credit card number, a bullion flag called the test. Right? So a money deposit with this flag set to true would act like any other transaction except for a small details that the card would not actually be charged. Right? This is very common. Or they might have, test cards or things like that to to run the payment.

At deploy time, our CTO forgot about this flag, and its value remained hardcoded as true.

This one bit of information enabled customers customers to make free deposits, then withdrawal actual money.

Oh, boy.

As we discovered the error, a big salvaging operation began. Our operational staff deducted credits from affected customers who still had them and halted withdrawal requests.

But with most times, it was simply too late. With no reasonable way of asking individual customers deposit money that they owed us, we had to accept a massive loss. Our CTO handled the situation by letting the dev team know what happened and humbly took blame without attempts to minimize his Node. A gesture that turned out to be valuable for me as soon thereafter, assumed responsibility for our payment provider integrations and could sleep easier knowing that mistakes happen to everyone.

I'm happy to report I made it through without collecting one of these stories for myself. So this is a, a non self issue. This, this person's luckily, the CTO did it because that would be a spicy meatball for any individual small employee to make at a company like that. Wow. So, basically, if you didn't get it from what happened JS they went live with their test payment processor, but this payment processor, it didn't use fake credit cards for their test surface. It used real information for their test surface, allowing them to make deposits for free and pull out real money is the big, big one. That is an expensive, expensive bug. Holy cow.

That might be the most, if we're doing this in dollar signs, like Yelp dollar signs, that's like a $5 sign fail right now. Would I really yeah. He didn't say how many, but, like, sounds like it was it was quite a bit. Yeah.

Next one we have here JS, I've titled it, an actual logging issue.

One thing we should say is that we Yarn have made all of these anonymous. Most people Wes, obviously, please keep me anonymous, but just keeping absolutely everybody anonymous for this. So thank you to if you did submit 1, but we don't want anyone to get fired.

That's fine.

I work for a logistic company in Johannesburg. And about 8 months ago, we warp rolling out a new system for clients who deal in timber. So, like, trees and pieces of wood and stuff like that. Our system was simple on paper. Track timber logs using QR codes as they move around the world. Each time a QR code was scanned, you could see 1 log's entire journey on our app JS well as the entire container's details, kinda like tracking for logs. Right? We had spent a year working on system. We're just about all done. All we had to do is generate production QR codes and ship them off to our men men on the ground in Durban.

I was given that task. My duty was to write up a script and generate the QR codes, print them onto an a four file, and, eventually, they'd be printed in laminate. Each QR code was basically what we call a log ID that when you wrote data to, it would update the database with whatever details you put in.

How it would work was a QR code was generated. The log ID was written to a database as well as the QR code. That way, we would know how many logs were scanned and could keep track of all the other information, like dimensions and so on. So I wrote the script overnight fixing some bugs and seeing if it was populating in the database, which I clearly remember it was. Important important note. As the daylight sailed in, I ran Node last Wes, and the QR codes were generated. I ran a quick ESLint, and voila, there they were. I pushed the code, went home. 3 days later, the team were on the ground celebrating the 2000th log scanned into the system.

The first the 1st batch was ready to go to China, but deep in the tech department, there was a secret.

Somehow, I had commented out the code that was supposed to write the log IDs to the database, meaning only 1 log was actually ever saved to our database. Oh, no. And because of an issue with the QR code, that log was discarded, which brings us down to 0. Oh, no. Needless to say, the client eventually spotted the issue, and we had to fly down to the depot before the logs were packed, and replace every QR code with a new one. It was hot and humid. The average temperature was 41 degrees Celsius.

It took us 2 full days of working nonstop to fix the issue, and we delayed departure by a full week. The only saving grace was we had a good time. Durban was has some really beautiful beaches and some awesome spots.

Oh my god. Catch it, this guy commented about a line of code and resulted in him flying around the world and stapling pieces of paper to actual odds.

Isn't that funny? It's like, you're flying around the world because of a a little messed up bug. Oh. You know, they go take a big trip. That is rough. That is definitely some some real world results from just a tiny little mistake.

Interesting.

Oh, man.

Oh, man. Alright. This next one, it says, here's my worst coding story. I was working in Adobe Experience Manager Npm, and it was getting to the end of my nerves.

I was adding some comments so I could quickly get back to the next day. The comment was, if this code ever ships and I am not around, tell my wife that I love her and that tell my son, snitches get stitches.

It was on the site for 6 months before I got an email asking about it.

The plus side of that was I had to learn Git Hooks to make sure this would not happen again.

So with this content, this made it to the actual, like, live site. I you know, is is AEM like a CMS, Like, that's a lot of I think so. So this content made it to the site. That is amazing.

What I wonder is, like, what what type of get hooked did you have to add that like, if the if the Wes snitches ever gets in? Just remove all yeah. Right. That's hilarious.

Oh, that is funny.

Alright. Next 1 is the spooky office skeleton.

This one made me laugh super hard. At a previous job, I was just laughing before I even read this. At a previous job, I was working on a scheduled email task that went to a mailing list of a given organization.

Now when I was running my integration test, our staging database only had test data.

How I was wrong? We had pulled in production into staging very early into the company's history, and I had emailed several of our longest standing most loyal customers, and I quote, hello. I'm redacted name of the oldest man in the office. Let's call him Phil. Hello. I'm Phil, the spooky office skeleton.

They put animated GIF of a skeleton playing a trombone.

Oh my god. And that's a trumpet. That is not a trumpet.

Yes. So I had to own up to the failure with the 1 on 1 emails to each of these customers who thought they or we had been hacked. Node great. So not only did he, like, goof it up with his customers, but he also, like,

called out the oldest guy in the in the office. Which is that that might be the the funniest part to me JS that, like and the the GIF is fantastic.

It's very spooky, very skeleton, and it is right at home on the web. That is great. Oh my gosh. Next Node is prevent the default of making money. I have 1 horror story. 3 years ago, I was starting my new job. A few weeks into it, I was asked to create an ROI, return on investment, calculator for the company website. I had a simple on click function for a button. It all worked, and I published it. 1 week later, my CEO and marketing team were panicking because there were no inbound form submissions from any of the forms, newsletter, or website.

I quickly checked everything and realized that I had grabbed the elements using input and then the attribute selector type submit without any specific class. So it ended up on all forms.

And since the function had prevent default, none of the forms work for 1 whole week. This is a specificity issue in new selectors.

That's great. Oh, so, basically, they just prevented all of the forms from on their entire site from working, and that could have been a very costly mistake. I know that a lot of industries really, really heavily rely on their forms for intake and customer acquisition. No kidding. So far, what we're learning, a lot of the a lot of people have made some very Stick my head. Like, I would as these were coming in, I would just, like, stick my head in the sand and be like, I this is not even me, and I can hardly read this. I know. Yeah. I know. It's funny because non Node devs, you know, people might not cringe that Yarn, but knowing that, like, what goes into these things and how they end up being pushed live and who the stakeholders are, like, well, I cannot I cannot imagine experiencing the shame of some of these.

Alright. Next 1. This is one that we got computer magic trick. Way early on, I set up an endpoint in PHP that took an argument to to delete a directory. So, like, probably, URL forward slash remove dir dir, and then as a param, it says dir name equals whatever.

I once hit it with no argument, and the computer disappeared.

Yeah. Defeated the entire computer.

Because it's probably running in, like, root, And then if you run r m, whatever, and it just deletes the entire computer.

Oh my gosh.

Oh, man. Yeah. That is a lesson in API design. Definitely, lesson in API design. Sane defaults or checking for required arguments? Base base cases. Yeah.

This next Node is update user set score equals new Scott, where I forgot Wes ID. So this is a database operation here where the person updated the user's score without specifying a user ID, which, I don't I'm not a SQL person myself. The SQL code doesn't mean a whole lot to me, but the root result here is every time I was updating the score for 1 user, I overrid for everyone else's score as well. So, basically, that's great. Imagine doing any sort of database operation instead of performing it on 1 user. You're performing it on all users. Every user. How do you recover from that? Well, hopefully, you're keeping backups, and you roll back the database because there's there's not a whole lot of hope from, fixing that one otherwise.

We got that story probably 5 times from people. I'm not we're not gonna read every single one, but pretty much people forgot the Wes clause in SQL.

And I guess that that seems like a a design flaw in the lang in SQL in general, where if you don't provide a where, it just assumes all of them. Is that yeah. Is that I I don't know SQL enough to to It is. Get it to you. That's how it works. Yeah. I I used to sling some SQL myself in the I never knew.

Alright. Next one is called Change Reaction.

It was the 2nd year of me working at a company full time doing PHP. We managed to get a big project, which is way over our heads. We still managed to get most of it right. I don't know how we did it. Fake it till you make it. One part of the system that we did was a newsletter. There was, first, a newsletter software, which we use through an API.

There was the new website that we built. There was the client's IT infrastructure where we had an old and crappy API, and there was a middleware which communicated between the new website, the old, and the old client IT infrastructure.

As a result, those systems were all connected, and a change on the website needed to trigger a change on the client data side, etcetera. Long story Scott, when I was testing a new feature and triggered a new newsletter Wes which was sent to all 30,000 production users of my closet, but it didn't end here. Oh my gosh.

Because of the bug, half of those users receive this email every minute for about an hour.

Oh.

An hour an hour because I was at a conference, and I needed to drive back to the office to find a way to fix this horror loop, which I was responsible for. Because of some hooks, one change triggered another and resulted in an endless horror loop.

It also sends it was just some emails I accidentally sent, but I felt like people were dying because of me. What I learned is So many emails.

I could have prevented it somehow being better at what I do, but being careful. But in the end, this stuff happens to everyone. This is true. I, the other day, sent a Black Friday email out, and this is during November. Or no. Sorry. This was in September, and I sent out a Black Friday email, which is almost a year old.

And the worst part about it is that you can't stop it because it goes out to everybody, and then you start getting phone calls and emails from everybody being like Uh-huh. Hey. Did you know? Like, yeah. I knew. I I know what happens.

So Oh my gosh.

Before we ESLint this next one, I do I wanted I wanted to bring in a personal one. Yeah. This isn't a mistake I had made, but one one time we had worked for, this conference, and we were it's a major conference.

It was at the Anaheim convention site. It's big, big, big conference. I'm not going to reveal the name of said conference, but we were tasked with building a scheduler app. We built it. However, there was, like, the whole real time scheduler thing, and somebody we didn't have, like, the the time to do it, so we contracted it out to, like, a third party. And this JS somebody with that we knew, and he built basically, like, it was really not a whole lot of stuff, but it was just JavaScript for, like, infinite, loading when you scrolled on the calendar.

And, we did not check his code well enough, because if we would have, we would have seen that he had schedule. You open up the scheduler.

Every single person that logs on to that thing is just hammering our server Wes Vercel? DDoS ed ourselves, took down the conference scheduler for an entire day during a major conference, and everybody had used the pamphlet once. We were in crap pants mode, needless to say. Unfortunately, it was not my project. Well, fortunately, it was not my project. So, all I had to do was just sort of stand there and watch in horror as everybody else had to freak out about it. That's hilarious.

Alright. Next Wes is personal MongoDB.

I once left a, oh, man. This is so good. I I read this one already. Woah. I once left the company, then a month later, went into my MongoDB Atlas account and cleared up some old clusters. So, well, it turns out at some point, I'd used a personal cluster for a work project that previous company had even gotten to use in production.

Needless to say, I got a call from my old boss. So, basically, this is the exact type of thing that I would do being like, I I like, I'm a big fan of deleting things and getting rid of things that I no longer want or use. And sometimes I'm just like, oh, right. I actually did need that. Like, oh, oh, great. This is like basically, you come in. You just start nuking clearing out old stuff, only to find out that one of them was used in production by a client application.

Rough. Rough stuff. Nightmare.

Next Node is called, hello, Rob.

While working on the BBC iPlayer, while rubber duck debugging to a colleague, and that's just sort of talking to a, a colleague about what's broken and how you're fixing it. I stuck a log message into the site wide alt text helper. Hello, Rob. And I forgot to take it out. It made it into production, and every single image on the website had an alt text of hello, Rob.

Wes, that reminds me of remember last year, there was the someone that crashed their server by writing a 10 gig file of

success.

Oh, that's good.

That is very good. You know what else is good, Wes? What's good? The spookiest way to solve all of your spooky bugs, your creepy crawly bugs in your site. I'm talking about Sentry at Sanity dot I o. Sentry is the perfect place to keep track of log and, really just manage any errors and exceptions that happen in your application.

You got creepy collies. You got ghouls. You got what else? We've got skeletons in there. They're marching in with their pitchforks. Well, we can assign those skeletons and say, well, that skeleton really only showed up once I started committing the line of code about awakening the Deno. So maybe I should create an issue that says, please remove code that awakens the dead, and then that will allow us to track that as we then vanquish the undead from our site. So head on over to century at century.i0.

Use the coupon code tasty treat, all lowercase, all one word, and you will get 2 spooky months for free on century.i0.

K. Next one is SSN regex.

I was writing a regex function to validate a Social Security number before being hashed into a DB.

So you you wanna tell me a spooky story? You start with, I was writing regex because that spooky me.

However, before doing validation, I forgot to unhash the SSN numbers and ended up deleting close to 10,000 patient SSN numbers from our production DB.

I spent the next 2 days recovering and reinserting 10,000 patient Social Security numbers.

Yikes. Because like, what do you a little bit what if you don't have backups or whatever? You end up just being like, can you please reenter your Social Security number into this line? Like, yeah, Wes lost your Social Security number. The

most interesting thing about all these stories is how people had to get the data back. Well, once your emails or logs or because in a lot of these cases, people don't have proper backups running.

Next one is called WordPress plugins.

I deleted the WordPress plugins directory on the server not tracked in Git before going on vacation for 2 weeks. Sounds like a good idea. Yeah. I fixed the white screen of death and then left. I couldn't push updates to the site until I reinitialized the repo and tracked plug ins, Especially odd because of how WP Engine did get at the time. So it sounds like a website was down. Like, even if you throw them in, you have to log in and set them up and stuff like that. So

for 2 weeks. No. Thanks. No. Thanks. Taking the website down right before you go on vacation, always the best thing. Pro move. Yeah. It's a pro Wes, it's a pro move. 2, it's it's the best way to have a super relaxing vacation.

Alright. Logging you in, you effing dummy.

I had this JS, quote, logging you in, you effing dummy, And I don't know if this was actually effing or the, the full word here.

Once It was the full word. I amended it.

Wes amended it. We are a PG podcast. I had I had logging in, you effing dummy, once go to prod. It was the test auth stuff I was working on, and I never got rid of it until the client saw it.

Yeah. So dropping f bombs in your Sanity. Yeah. It probably not something you wanna do.

I actually I used to be, like, very, very liberal with my use of Node not like f bombs, but, like, just slang or funny things or little little blips and blurbs in the code that that, like, would not be considered to be professional, and that you definitely learn your lesson when something like this happens. Happens to everybody. You you don't do that. Just keep it pro. Keep it pro. And then if you make a mistake, at least the mistake is pro.

Next one is called a hostile coding environment.

I launched a site a day before I went on a 2 month backpacking trip in 2009.

So Yeah. Good idea. There's another useful theme here to to pick up on. Yeah. Wes, in a hostel on the 1st night, I checked my email only to find out there was a major bug.

I spent the next 5 hours writing PHP and a non mobile friendly editor on Safari on my 1st gen iPad Touch.

Fixed it.

Fuck. It was 2009 was a long time ago. Like, I guess I had a laptop at the time, but it doesn't sound like he took it along writing PHP in a non mobile friendly editor on a iPad Touch.

Amazing. Yeah. IPad Touch, it'll be imagine what if you did it in, like, a click wheel? I I brought the set back up in the click wheel iPad. It Wes t nine on a on a phone.

T 9. Yo. I used to be fast at t 9. I was very quick at t 9. I remember when when Wes went to the full size keyboards, I was like, definitely got slower.

Alright.

A graceful exit. I took the master PSD template for a monthly email blast and destroyed it while building out assets back when slicing was commonplace. Yeah. To get a background, this is what you did because we had largely raster based sites. Right? So you would do the site in Photoshop. You'd slice it all up, and then the process of that oh, I don't know about US, but I would, like, do what this user is saying, and I would I would actually destructively manipulate this thing to get at like, a a rat getting at the cheese. I'm scratching and claw until I get all of the graphics I need to the site. Yeah. Because you sometimes you had to, like, rearrange people's, like, folders and stuff like that. Yeah. You had to get to that that content, and it wasn't as easy as getting the CSS to just do it. So, this this user, they basically destroyed the master email PSD template.

No biggie until I hit save on the master, quietly closed the file in hopes no one would figure out it was me. I left the company a few weeks later. I left the company a few weeks later.

So they basically destroyed the master PSD template and was just like, alright. I'm out.

This was, like, big back then. Like, back a lot of companies just use Dropbox or, like, a local file server to sync files, and there was no a local file store. Yeah. There was no backup or version control. And if you goof something up, it was gone.

The backup and version control Wes final final

master PST underscore final 2. You know? So good. Next one is a favicon DDoS. I DDoS ed their own website when I forgot to include a Favicon back in 2010.

This in turn gave a fully fledged 404 page on every Favicon Wes, increasing bandwidth needed to serve the site in an order of gigabits per second. So this Node this one puzzled me a little because I don't remember having this issue. But, like, when a favicon is not there, it obviously serves up your 404 page.

If your 404 page had significant resources on it that then requested more resources, you could DDoS yourself rough.

Rough. Rough. Any self DDoS. Not so good. Alright. Next Node, common sense. I didn't use sense to store how much a client should be charged.

Accounting was trying to figure out why we are only getting a 10% of what we were owed for months. So, this is what do they call these? Like, a shifting decimal error or something like that where, like Yeah. In fact, this was the plot point of, the movie Office Space when they were trying to steal pennies or fractions of a penny, and they accidentally shifted a mundane detail like a decimal point a couple of places. Yeah. Resulting in a much different figure than that was accounted for. Or they stored it with, like,

a well, like, you're using, like, a Scott of as a string, and you try to parse that into a number. And, the base is not 10, but 8 and whatever. There's Scott a lot of bad things that can happen when you don't serve you don't store how much things cost in a nondecimal value.

Next one is I got permission. This one's for all the Canadians out there. I got permission to say which company this was because I I'll I'll tell you why it was hilarious in just a second because of people in Canada will know who this is.

So indigo.ca, which is indigo.ca is the biggest bookstore in Canada.

They've got bookstores everywhere and online. They're they're they're huge for that. Indigo.ca once went down because the Cloudflare CDN JS had a bug that served a random piece of content instead of whatever you were requesting. So before we had, like, bundlers and stuff, it was popular to load in your libraries in via a, a CDN URL. And the reason behind that is because if if somebody loaded jQuery from the same URL that your website, then they would already have it in their cache, and it would go quicker.

So people always use the CDN JS website for a long time. In our case, we were requesting require JS. Since this request was valid, h t d p two hundred, there was no easy way to know, if it was failing aside from checking the contents of the of the module that worked.

The entire site failed to load white page, so, like, the whole thing just didn't work at all. And Heather yes. That Heather. So, this is only funny to Canadians, but if you go into Indigo, there'll be a sticker on the books right when you come in. There's a there's a there's books, like, right there, and it says Heather's pick. And it's this big gold sticker. I have 1 on my laptop because I think it's hilarious. It says Heather's pick. So Heather, yes, that Heather, wanted to know which vendor was responsible for taking down her site. Oh, no. Heather. Can only imagine the CTO and VPs at that time trying to explain open source CDN to her. Needless to say, we changed it to a local copy and checked its contents after that.

Oh.

That's the thing that people say you could get burned by, but I had never actually heard somebody getting burned by this. So that's great. Yeah. Although, I'm I'm just gonna say out there for one of our sponsors today, if they had Sentry,

they would be able to see the actual syntax error that happened when they tried to load

Require. Js. Look at you bringing our, sponsor background.

Yeah.

Well, if if we wanna maybe maybe do another spooky sponsored transition, one of our sponsors today will give you some very spooky behavior when you push to your Git repo from your site. It automatically builds it without you having to set it up, and I'm talking about Netlify at netlify.com.

Netlify is the host for all of your front end content and more. Wes, do you wanna talk about Netlify?

Yeah. So Netlify is the host for your Jamstack website. And what's kind of cool is that Netlify also has a whole host of other features that it gives you alongside the building and deploying and and caching and all that good stuff that we probably know about, but they do things like contact forms. So, like, that's that's a tricky thing. If you're doing a Jamstack website, how do you do a contact form? Right? So they offer that. They've got plug ins for AMP, Algolia. If you wanna if you wanna implement search in a Jamstack website, you can use Algolia. They've got plug ins for all of that stuff. They've got all kinds of really neat analytics stuff built in. So it's kinda cool, like because whenever you mention the amazingness of Jamstack, inevitably, that comes along. People say, what about contact forms, serverless functions, search, all of these things, log in, CMS, all of that. And Netlify gives you offers all of that as well, which is pretty cool. So check it out. Netlify.com forward slash syntax.

You can sign up and check it out right on there, whether you're using a Gatsby or Next or it's it's kinda cool. They have, Sanity, Stripe, Eleventy, which is a really cool static site generator. So check it out. Netlify.com/ syntax.

Alright. Next Node is don't leave the country.

Hey, Wes. I'm gonna put Anne Scott in here because he didn't say Anne Scott in a Oh, yeah. This one is specifically a friend of mine, which I was so surprised to hear the story.

That's great. They should also say, hey, Scott. That'd be nice.

Just I'll put it in. Here we go. Hey, Wes. Saw your post about horror stories. Oh, yeah. Wes just added Scott with an exclamation point, and I appreciate that. Hey, Wes and Scott.

Saw your post about horror stories. A few years ago, my client's husband was arrested for embezzling.

He was high up at a Canadian telecom company, and the feds called me in to and told me the website I built were being paid with illegal money, so my business was pnpm laundering.

I had to hire a criminal lawyer and was told not to leave the country or I could face arrest. Needless to say, there were additional clauses that I added to my contracts after that. If we wanna have, like, a tier of, you know, how big of an f up or or I guess this isn't on the user's part or whatever, you warp it like Yeah. Consequences for building our websites and stuff. This is very high up there on very extreme consequences being investigated by the feds.

Yeah. That is a scary situation.

I ended up following up with my contact with the story, and, I said, oh, like, like, what did you add to your contracts to say, like, don't involve me in your embezzlement? And, they said I added that if the client does anything that requires me to pay any professionals, like legal fees, they would be charged back to them.

And they said, ultimately, it was okay because I had all of my books in order, could prove my sources of income from 2 years prior.

Lesson learned, keep your books in order. Oh my gosh. That that's a good story for a FreshBooks ad transition.

Are you involved in embezzlement? Apparently, you can prove it. Wes, we'll have your finances around. Yep.

Oh my gosh. Oh, that's good.

Alright. Next one we have here is called the apostrophe catastrophe.

I ran a contest for a radio show, created a form for users to submit answers to a weekly question.

1 week's answer included an apostrophe.

No entries were recorded that week. Oh, no.

Oh, that's something you don't even think to write in your tests.

Right? That's the kind of test that you write after you after the error pops up.

Yeah. There's a lot of tests that are ever right after the error pops up. Oh, boy. Oopsies.

The second one's an oopsie. It's called the e n v. I posted a dot e n v file one time. Enough said.

So, yeah, they put in a dot e n v file to GitHub, I would assume, or somewhere public, and, oh, they posted it. So what, they posted in, like, Reddit or Twitter or Discord or something. Either way, that e n v, lots of secrets. Node, thanks.

Next one, a Christmas miracle.

I'm not sure if I told you one this last year, which you didn't.

I told I took down a production server at Christmas time with some poorly optimized SQL and stored procs.

The app was for low income families to get assistance purchasing holiday dinner.

The production server hosted over a dozen database, all that which became unavailable.

Oh. Can you imagine taking away Christmas dinner

from families in need? Dude. Yeah. Some of these the stakes are so high on these. It's just it's very unlike cringed into the back of my my chair here. The $1,000,000 bill. When I'm interviewing a candidate, I like to ask them, tell me about a time they made a mistake and how they resolved it.

To make that vulnerability easier for a nervous candidate, I tell them a time where I wrote in a Vercel update statement without the where argument, which destroyed 13 records on production.

Node candidate we interviewed shared his story.

He was configuring a utility that would spool up an AWS server, upload a video, transcoded, transfer the video, and then delete the server.

It was going well until 1 Friday, he pushed some code that broke the delete server step. He got back to work on Monday with a 1,000,000 dollar bill from Amazon dotcom.

So let me tell you. I also woke up one day with a very, very costly AWS bill. I don't know if this has ever happened to you, but I don't I don't know what happened, if it was a, an access issue or something. But I I had been and I don't know if there was, like, hacks necessarily, but somebody got access to an API key that allowed them to spool up new servers on my AWS account. And I woke up, and there were hundreds of servers doing mining Bitcoin or something. Who knows? Amazon had a bill for it. It was, like, $30,000, and I had to email them and be like, and and so they were like, well, if you can get rid of these and secure your stuff, then we can we can move on. That happened to me. I think it's a few years ago, but it was a situation. I was, like, looking at that bill like, holy cow.

And luckily, AWS is reasonable enough that if you Yeah. Fix up your stuff, they they rescind any of that. That just goes to show the power of the cloud, really. Yeah. Like, the fact that you can

spin up $1,000,000 worth of equipment over a weekend. And, obviously, so they he he kept those servers running, which means that no one else can use those servers. So you're paying. Yep. Usually, with the serverless function, you pay for 3 seconds, and then they shut the server down. But if you're especially with, like, video servers that probably have graphics cards hooked up to them, and you're running them for, like, 50 hours instead of, like, 12 seconds.

That's gonna be expensive.

Yeah. $1,000,000.

1,000,000.

Next Node, the hacker who dropped the DB. So I was around 2 to 3 months into my 1st web dev job and was meaning to truncate the order table for a pretty big shop locally.

Instead, I did it on production.

Oh, 2 minutes later, my boss walks in and tells me that there's a bug in the admin interface, and orders are not showing up.

That's when I realized what I had done. I got so scared. I command q'd the SQL Pro and acted like I didn't know anything.

I know it. I didn't

see anything. In the office until 11 PM fixing it up, and luckily, we could restore all data since we had query logs enabled for replication.

So that's that's how a lot of times people, like, fix these database issues JS they go back into emails that were sent or logs, and they have to parse the data out of those logs or emails and put them back into the database.

Then they assumed it was a hacking attempt and malicious script, and spent quite some time to try to find the hacker.

Well, I acted like I didn't know anything. Oh.

I never confessed except to 2 very close colleagues. I quit the job after 3 otherwise wonderful years. Sweat emoji.

Oh, yeah. Alright. We got a hefty one here.

So my story takes place in 2016 to 2017. I was hired by a consulting company to work on AppServer.

It's a CMS implementation. I've never heard of AppServer. No. My skill level at the time was probably just barely above mid level, but mainly c sharp and .net tech.

Maybe why. I haven't heard of it. Not so much JavaScript.

I was a pretty fair hand with the internals of the CMS, though, so I thought I could figure it out. Also, I've reasoned a bit that my tech lead had a decade of experience so I could lean on him a bit. Arriving on-site was the first thing, and the client told us was no frameworks, plain that JS or jQuery only, but then we'll be Ajax ing in the widgets from the CMS to other projects so the JavaScript can't conflict. So okay. So you got immediately Scott to deal with a whole bunch of version stuff going on here.

So when I asked what version of jQuery the other projects warp using, he said, oh, various some version 2, others on version 1.9.

That's not a problem. Right? Sounded bad to me.

Wouldn't that be rough to know whether your code will work? Yes. I think that that does sound rough to me as well. My other tech lead thought, no problem. We can do it. Okay. Phew. At least my tech lead had the goods.

Over the next week or so, while we familiarized Vercel with project, the client would make some more restrictions around the JavaScript and CSS to use, like no build steps, no jQuery plugins, etcetera, etcetera. The tech lead would sagely Scott his head. No problem.

Okay. This tech Deno. No. Not a lot of problems with this tech lead. So we start working, and a few weeks in, I'm struggling with a problem. Trying to reason through a code should work Vercel the written requirements.

I asked my tech lead to come over, and I'm explaining the Node, and he says, what is this tool you're using here? I had a dual monitor set up with Node screen that had Chrome DevTools on the other, and I had the Wes page I was working on. He was gesturing at the DevTools.

I didn't have any fancy plug ins or anything.

It was standard dev tools. I mean, this the console? No. What's this tool? It seems pretty useful.

You mean the dev tools? Oh, yes. That's built into Chrome.

Oh. My heart wasn't in my stomach. Oh, he doesn't know Chrome Dev Tools? What else might he not know? Am I the one who's strong, or am I the one who's wrong? Maybe DevTools aren't standard tooling.

Oh, no.

From a place of pure self doubt without holding judgment on the expert tools, I hit. So what do you use? And he shows me his setup. He uses an elaborate system of alert statements to debug his code.

I keep a straight face, but internally, I know I'm in danger.

I soon learned this is this is very good. This is like this is spooky stories. So when you're when you're a dev lead who's been saying yes to everything does not debugs through alert statements.

I soon learned that he doesn't know a lot about certain tech that was necessary for the project, like AJAX or MVC pattern. What sorts of things? I Scott picking up more and more user stories as he struggles with his work. Pretty soon, the client is asking me to plan and implement these features, and that's when the real horror begins. I start writing JavaScript.

I, the junior level coder who starts writing a lot of JavaScript, I need to have reactive components, but I can't use a framework for libraries. I can't count on jQuery because the I can't use plug ins, and we're not even sure what jQuery JS being used at any given time.

Oh, no. I I figured

I can't so I can't get through this. I'm so sorry. I figured out I can reliably use the AJAX functions at jQuery. So I start to set something up reasonable for the Ajax, and that's about it.

Then I read about this regular JavaScript called a mutation observer.

So I start working writing a custom framework. I can create functional representations of components and start overloading the prototypes Wes of these functions.

I feed into an engine in quotes that has MutationObserver at its heart. It's not as bad as you're imagining.

It's worse.

Things happen on the site, but we don't know why. Race conditions abound, Node off by errors, off by pnpm errors. And then I realized my errors are fundamental to the architecture of the framework that I wrote. I I work long hours. I become bitter.

This is so beautifully written.

I work long hours. I become bitter.

Yeah. It's very funny, and I'm oh my god. Desperate.

I start introducing Vue. Js into the project knowing that the framework is not used elsewhere. Maybe I can get away with it, and I do get away with it. Technically, things work over budget and out of time Wes released a production. I roll off the project 1 a month later, but I hear stories. The company spends months and 1,000 of dollars to fix the project.

They formally adopt Vue. Js and everything to that. I am so sorry, but I learned a lot, and thank you.

So, anyway, there were just so many What a train wreck. What a train wreck. You can't look away.

Many No. Like, once you get to the many versions of jQuery and you don't even know which one is, like, useful,

yeah. Yikes. Yikes. Oh, man. That is just, like, every possible thing to go wrong, the the tech lead who uses alert statements. And, like Yeah. Normally, I wouldn't rip on somebody for not knowing how to do something, but the guy Wes a tech lead of a large JavaScript project. And having other people underneath you and just been like, yeah. Sure.

Yeah.

Oh, that was a good one. So thank you to whoever wrote that. And he said, I'm inspired to write a blog post about it myself, so please please do that. Yes.

Laura Mipsum.

The horror story comes from when I was working on a feature to break a cache on our customers' RSS feeds stored in s three to reduce a delay when pulling for updates. So this is pretty common thing. You have an RSS feed. They probably just save the save the RSS feed to a text file instead of having to do all the maybe hundreds of database queries underneath.

I found out the hard way that our development environments also wrote to the same s 3 buckets at the same time. And instead of just breaking cache files, I rewrote production records with lorem ipsum.

It doesn't say how they fixed it here, but that was, probably had to rerun that script with the production.

Oh, yikes. Big gags.

Malicious compliance.

I once had to embed an Ember app into an Iframe in a React React

app. This is so real world.

Yes. I know. And and implement a browser side to PubSub so they could interact with each other. To make matters worth, the OAuth sign in modal opened from an element in the eyes.

Node day, our author randomly stopped working because of a major browser implemented a security update to block events originating from the Iframe. So we hastily added an add button to the React app that published an event so that the Amber app could subscribe to it. This will be why this

is too unplug it.

We quickly shipped a fix, but our leadership said it was unacceptably ugly in order us to undo it. We warned them that undoing it would prevent users from being able to log in. They didn't believe us. We reverted the fix, and within minutes of the deploy completing, our support team started receiving reports of the login being broken. After a couple of hours of broken log reports, the leadership told us to put it back.

So this this is a major Frankenstein right here. Like, this is this is, you know, 10 neck bolts out of 10 neck bolts Frankenstein.

This is this is a horrifying.

Oh, that is there's a whole Reddit called malicious compliance, and it's just stories like that where people are like, alright. I'm telling you it's not gonna work, but I'll comply.

Whatever. Do it. Cool. Ship it. Alright.

We're actually actually, it's pretty good. We're about halfway halfway through the stories now, so we are going to, put a cap on that and pick it up next week with a whole another hours of additional stories. And let me tell you, there is some good ones in there. Yeah.

Oh my gosh. I'm very excited.

Sick pics. This is the brother of the show where we talk about things that we find to be

sick. Wes, do you have any sick pics for us? I do. And this has this is actually something I saw on one of Adam Savage's one day builds a couple about a year ago. You're Adam savaging these sick pics? Because Adam's ice Adam Sanity last week as well. So this is a LED flame effect light bulb. So this is just a light bulb that goes into any regular light fixture, and it has a actual flame effect.

And it actually looks real. Like, I'm I'm this do you think, oh, it looks cheesy, But, like, this looks like a real candle or a real campfire.

And what I'm doing is I'm gonna put it into the light that is on the front of our porch. Because, like, if you ever been to New Orleans, you know how they have these, like, gas lanterns that are sort of flickering? I love the love the look of that. So what I ended up doing, you can get them in just like a regular light bulb, which is called E60 2, or you can get them in the really small sockets that go into our porch light has a small socket. So I got a a couple of each of them because I I think it looks really cool. The kids love it, and it just gives a really nice, like, ambiance without having the danger of real flame. So the one I you can just go on Amazon and search for. The one I got was called Hanseang LED flame light effect.

And also the cool thing about it is it if your light fixture is upside down, it's smart enough to know that it's upside down,

and it will revert the effect so that the flame isn't going downwards but up. That's cool. That's very cool. Yeah. My second pick is going to be a video game that I've been playing almost obsessively.

You know what? I play games, and I I play games on the the weekly. Like, I like, Sunday is my game night, right, where I play video games, but this game is just so fantastic. It's called Hades. It's by Supergiant Games. I've played several of their other games before, and I've always been a fan. And this is a they they call it a rogue like game. I don't even really even I've never played with roguelikes before. I don't know very much about that, but let's say Hades is this this game that's themed out of, like, Greek mythology. Right? And you play a character who is stuck in basically hell, and their father is Hades, and you're attempting to escape. That's, like, the long and short of it. You're attempting to escape these, like, layers of Wes, essentially, to get out into the world. And so the way it works is that there's, like, you you play the 1st time and you maybe make it to, like, level level 10 or something, and then you die. And then you have to start at the beginning. And the next time you're a little bit better at the game, you you've gotten a couple more power ups. You get to, like, level 15, and you keep going, and you you end up playing almost in, like, a loop. But the coolest thing about this game, and and I urge you as, like, a computer programmer for to play this game and, like, think about it in a programming sense. There are so many variables at play here. So, like, as you go, there's just like a 1,000 there's there's 5 different weapons. There's, hundreds of different upgrades and whatever, and you use the Greek gods to empower yourself in various ways. But as you end up going, it is so finely balanced, so so incredible that, like, you Sanity tweak your setup and play the same game. I've I've maybe made, like, 50 attempts to escape, maybe even more than that. I've escaped, you know, many times, and I still wanna pick it up again just because every single time you play, it's, like, permeated, and it's different, and it's, random enough that, like, you can have just entirely different experiences even amongst the the weapons. Like, you could have, like, a sword, and there's, like, 4 different variations of the sword, and each one substantially changes the gameplay. And what I found it to be just absolutely fascinating about this game programming wise is how the heck did they get so many different variations and permutations and and random ability without breaking, like, the challenge of the game at some point. It's always very challenging no matter what you do, and it's always very engaging. So Hades been my favorite game of the year by far, and I'm very into this game. So, check it out. It's on Switch PC and that, that kind of stuff. So that's my sick pick.

Sick.

Shameless plugs, master Gatsby Scott. It's my latest course on learning to build websites with Gatsby and friends, and, we do all the CSS in it. We use a headless CMS. It's pretty good. People are really enjoying it. Use it's at mastergatsby.com.

Use a coupon code syntax for $10 off.

Sick. My shameless plug is going to be my animating Svelte course, which is coming out, and it's gonna teach you how to animate things in Svelte. Normally, we get into the gesture stuff, and we we show you how to write really neat custom animations and really take control of the look and feel of your site using Sanity.

Check out animating Svelte only on level up tutorials

story. Hopefully, you have a good Halloween, and we will catch you on Monday.

Peace. Peace.

Head on over to syntax.fm for a full archive of all of our shows. And don't forget to subscribe in your podcast player or drop a review if you like this show.