Three scary developer histories for Halloween

🎃 I deleted a production database

Ana had less than a year of experience as a programmer.

Her boss at the time managed multiple client sites, mostly landing pages. He was a fan of shared hosting, since all the options to configure the “server” are there through a web interface and you hardly have to use the terminal, a big mistake in the long run in my opinion.

The boss shared database access with Ana, and since Ana didn’t want to use phpMyAdmin that came with Cpanel, she used a Mac software on this occasion.

When opening the program, she listed all the tables without any problem. She executed some queries to check if the connection was working correctly.

That day there were no problems, the worst would come a couple of weeks later.

Her boss asked her to manually delete some customer records. Yes, they didn’t use a soft delete, so deleting the records forever from the database was the only viable option.

Among the database instructions she had saved for her local environment, she had the classic delete purchases from. She copied and pasted the query, but when trying to edit the filter parameters, she pressed Ctrl + Enter.

5300 records updated. 30ms

And that’s when she realized she had made a mistake. When she refreshed the table, everything was gone. Moreover, it was a table that had no dependency on others, not even a foreign key error could save her.

She proceeded to call her boss. He wasn’t angry, but he did mention that he assumed it could happen at any time. Her boss used an automatic backup from the previous night and, since it was a not-so-used site, they proceeded to use the SQL script to delete, create, and refill the database again.

Being a junior, she was very scared, thinking that would be her last day in the company and that she wasn’t born to be a programmer in this life.

But everything was fixed, and from that moment on, Ana was ten times more careful with any script she needed to execute on a database.

Precautionary measures

• In all databases for production environments, try to create a read-only connection. This will help you avoid executing unwanted instructions by accident.
• If you use software with a graphical interface (GUI) to connect to a database, always add a background color to your connections. Personally, I usually use the following color range: green for local databases, yellow for staging or QA versions, and red for production databases. If I have a read-only version for production, I usually use orange.
• Always try to use soft deletes as a deletion measure. Usually, it’s a field called deleted_at. The NULL value is interpreted as the record being active. If it’s deleted, it will have a value with the date and time when it was deleted. However, your application logic must support hiding these records from the end user.

🎃 A hacked e-commerce

Several years ago, a colleague from a previous job referred Max to a client with a WordPress site developed by a development agency.

The horror story wasn’t that the site even had the WordPress core modified (oh my god), but also that the code of the plugins themselves was modified, making it impossible to apply a future update without fear of breaking everything.

Over the years, it was possible to migrate to custom software, made with a framework. After the migration, everything was fine for a whole year.

Until one day the site owner called Max.

• Max, we’ve been hacked
• There are no hacks, only security breaches from the user side - Max replied, assuming that maybe someone had shared their password by mistake.
• Look at the website! It’s full of images in Russian!

And it was true, they had replaced the product images and call-to-action banners on the website with a bunch of propaganda that Max neither understood nor wanted to understand.

• Oops - Max said as he observed the damage.

Max proceeded to block the website. He logged in via SSH to the server and there was no strange or encoded file that made him suspect they had access to the database. After a while of thinking about how the intruder could have entered, he only came to one conclusion.

“The admin panel for uploading images”

And confirming with the workers, it turned out that to quickly access the admin panel, they had changed the password to a simple one, like “test123”.

No images were lost, as they were uploaded to an object storage (similar to AWS S3), and when images are replaced, they are NEVER deleted. This is to emulate the versioning feature that AWS S3 has.

Max also sends us the solution measures and future precautions to avoid possible failures.

Precautionary measures

• All passwords were changed to a more complicated version of them, for now I removed the option to change the password for all user types.
• Do not allow the application to delete images once used. Always store things outside the server.
• The database was also outside the server, which allowed reverting to a version from 3 hours before the incident. Some records of some purchases were lost, but they were already saved in an external billing software.
• Divide the users who can update the site information with roles and permissions. With this, the attackers didn’t have access to customer information either.

🎃 No money, I had to fire the developers

Back in 2017, John was in charge of a couple of junior developers. He was working for a startup at that time. They received Venture Capital funds and sales were “tight”.

It was then that he discovered that when companies don’t get money, it’s difficult to maintain a stable salary for all workers.

It’s hard to fire someone from a company. As co-founders, we pride ourselves on our workers loving our company, forgetting that they also do it to support a family, pay a house mortgage or a car, etc.

After a streak of more than 3 months without reaching the expected sales, John and his partners decided to fire the developers who had been hired for a period of 6 months.

The other two co-founders told John that he would be responsible for communicating such a decision, as they still didn’t have a human resources manager.

John arrived on a Friday and communicated the news to both developers. The expression of both was one of total disappointment. John could understand in part, as he was unaware of the difficult situations each one might be going through.

The most he could offer was to prepare a letter of recommendation.

There are no precautionary measures written by John, but from the business side, I feel free to write some.

Precautionary measures

• Don’t hire more than you can maintain. There’s a popular phrase: “hire slow, fire fast”.
• Don’t make exaggerated promises to workers; show positivity, not exaggerated lies. It’s better to say: “We’re a startup, how far we go depends on us” instead of: “We’re going to be the best in this country, we’re going to reach the moon”.
• As a founder, remember that entrusting your company to your workers is like giving a stranger the care of your child. No one will love it more than you.