The clever hackers

[RAYMBO]

I do not know the vector used to gain the level of access that was gained: root. This is the first time I have ever seen that. I suspect it has something to do with recent kernel bugs that allow privilege escalation. But I have not found the evidence of how it was done.

I have many many lines of custom code (plugins, etc) that all could be suspect. But, I am very security conscious when it comes to AXAX, API calls, public data input, and coding in general. Been coding too many years to not defend against the easy hacker tricks. My code is not perfect and exploitable bugs could exist. I am generally protected from that since my code is not open source (no public access). So the hacker will be basically be trying to hack a black box.

Hackers are the laziest coders, as all criminals are lazy. Even though the vBulletin code is also not open source, but it is readily available. So hackers can let AI work over the code base and look for exploits.

Normally hacks are from spammer hackers, and they just want to get some kind of redirection in place. Or a spam page or fake login. As sending a spam email that has a legitimate site link helps trick you. Checkout pitmaster tools here! With the link to Pitmaster that gets redirected to some other site, and these normally have several redirection jumps via several hacked sites. Landing on some casino site or some fake login page.

However, this time they gained root access and setup a CRON to overwrite the /root/.ssh/authorized_keys file with their key (which was pulled from a pastebin site via CURL). Since I do not allow password logins, only pubkey logins, this was the only way they could retain access. And this happened in real time. As I was logged in SSH (after noticing that the site was messed up) and was deleting the hacker files. I then saw the files returned and tried to SSH a second console, so I could have a spare, and that failed. As I tried to use the first console to figure out what was happening, they rebooted (not before corrupting the boot config).

I honestly think the break and run (and reboot) was a panic move as I was locking them out when it happened. I have my own tricks.

The rest of the hack was installing the crypto miner (in /tmp) and setting up the code to allow them to use it remotely.

The only reason I could regain access is because Linode has a web console feature (most virtual server hosting has this). It is not optimal as the server could not be booted and I had to boot into rescue mode, mount the main server, and do my work from there.

I do have daily backups. In the last one was only a few hours old. Not up to the minute. I was able to do a complete database dump minutes before the hackers rebooted. This is always my first action on a suspected hack, dump the database. So that was available. Now how to get the data? The attachments directory is currently 65GB. I could not set up pubkey login on the rescue disk. So I temporarily allowed password login on another server and use sftp to transfer all files and the database dump. That took about 4 hours, the vast majority of that was the attachments. Also took that long to transfer all the files to the new server.

So that's it. Summary: use exploit to get a file on the server, use file to escalate privileges, install crypto miner, use crontab to lock out other logins and ensure you can login.

The two things that bother me about all this. The first is the vector, how they got in to start with. But the other thing is, they did not stay stealthy. Hackers do not want you to know your site or server was hacked, as you will then take steps to clean it up. The first thing they did was break the site, deleted the file that handles rendering all assets (CSS, JS, and Media). Then breaking my ability to login at all. Very concerning. Lazy, stupid, or bold?

[gcdmd]

Thanks for all your great work, Ray. You are much respected and appreciated here.

[LA Pork Butt]

I am happy you understand this and can respond. I appreciate your explanation, but I have no idea what you were talking about.

[Richard Chrz]

Agreed, on both points!

[texastweeter]

Have ski mask and Louisville slugger, will travel. Just find the location.

[Purc]

And a BBQ accessory, just because....

[shush]

raywjohnson
A few things:
1. I think you are spot on w/r/t "lazy" or "easy" as the path the hackers took. It is not uncommon for "script kiddies" to scan a site, enumerate it, and then dump a large number of expected hacks against that site. I learned a very difficult lesson about 20 years ago... not all exploits have been published yet. For instance, ssh is a very well known, and very easily used, method of connectivity and exploit. There are always new exploits against it. It is software that needs to provide connectivity and use modules (pam, libcrypt, libssh, etc.) that are not all fully within their control. Different *nixes have their own dependencies. I would keep an eye out for any upcoming CVEs that are tied to SSH or any of the libraries that ssh uses.
2. mitigation is tough. While most hackers are stealthy, beware that just because this looks like they were sloppy... they may be hiding something else. Real suggestion, blow away the instance completely and start fresh.
3. chattr is your friend. While it isn't a failsafe, it does provide an extra layer of frustration. If the script cannot modify .authorized_users, the hacker may not know the next step and give up.
4. There is a POC in the wild from 3 days ago... One and Two -- may want to check with vBulletin and see if they have a hotfix.
5. Keep up the good work. It doesn't really matter how tough you make it for bad guys... with AI and faster computers, and a user base that may not always understand why passwords should be difficult, (I am not calling anyone out, just stating personal experience facts), the bad guys will eventually find a way in. Depending on where this is hosted, A WAF may be a good idea to either implement or get more granular rules applied.

I appreciate your work (because I do similar things) and know it is typically a thankless job.
"When people see you calm, they ask 'What do they do all day?' and when they see you in a frenzy, trying to fix things, they ask 'What do they do all day?'"
Thank you and keep on keeping on.

[RAYMBO]

1. Agreed.

2. I built a whole new server.

3. I have learned this recently. And I have locked specific files with it. And set up a service to monitor changes to those files and restore them if changed.

4. That is for v6, we are in v5

5. Thanks.

[jfmorris]

raywjohnson thank you so much for your hard work and patience in dealing with this hack. I run a much much smaller forum site, originally on vBulletin, and now on phpBB, and know that the hackers and spammers are a constant job to fight. I have nightly backups in my case, to a remote server, and have had to rebuild several times over the past several years. It is very frustrating.

The absolutely WORST thing I ever ran on a server for getting hacked was Wordpress. In fact, I have a disabled Wordpess site at the moment that I need to do something about. It was a tribute to a local Bible teacher who passed away, with links to audio MP3 files for a series she taught. I'm probably going to redo it as static HTML and forego Wordpress entirely. EVERY Wordpress site I've setup over the years has been defaced or crashed. I had a blog site, my daughter had one. Both gone. Very frustrating.

[dpearce]

Yeah, Wordpress is just about the last option you want to host with, pity that it's so popular. I had a hosting provider that decided all sites were going to Wordpress, I quickly jumped off.

[RAYMBO]

I manage and protect 90+ Wordpress based sites. It is a constant battle. I have been trying to work on a few methods to lock WP down. No simple solution.

[Oak Smoke]

Thank goodness for Ray! I have no idea what was even said in describing the problem or the fix. I was somewhat concerned that AR was gone. It’s a part of my daily routine and I have friends here. My only link to the group was the fact that I have two of them’s phone numbers. I text one of them. He was as bewildered as I was. We need an AR phone book!

[RAYMBO]

There are currently 3 main servers and two staging servers. I was moving to consolidate, but I am concerned that if one site gets attacked, then it brings them all down. So rethinking that. I will be setting up a special contact page on the extra server, using amazingribs.net, so that members can contact us when the site is down. I also want to add a status page so that when there is an issue, you can see what's up quickly. The lack of info is the part of all this.

[RonB]

I am very glad you unnerstand this stuff 'cause I sure don't. Thanks for all you do.

[Steve R.]

Me reading that:

[texastweeter]

Pretty much. "Where do you add the gas in this thing?"

[RAYMBO]

My apologies to the muggles (non techie folk). The post was for those of us crazy enough to learn the arcane knowledge of tech.

[Sweaty Paul]

+1

[Dan Deter]

We appreciate all your work raywjohnson! While I did understand and appreciate your synopsis, I am very glad I'm not the one having to deal with that, and appreciate you all the more because of it!

[Duanessmokedmeats]

To bad you couldn't find their location to call in a missile strike.

[RAYMBO]

Well, I am hoping that AI will soon make this a thing.

[RAYMBO]

When I first started coding (late 80's) I had a odd mental issue. As you literally have to juggle 2, 3, or more concepts in your mind at the same time. At first, the result was that I would start to laugh. Like out loud. It was so ridiculous that my brain thought it hilarious. I got past that quickly, a few weeks before it stopped (still happens from time to time). The juggling act remains. Code, database, HTML, CSS, Javascript, security, and how to get it all to work together without making it a huge mess. Not to mention, internet and networks, operating systems, protocols, command line, and on and on...

So no worries of you do not get it. You have to be a little "special" in the mind to even begin to understand any of it.

[CaptainMike]

Practitioners of the Dark Arts are a society unto themselves. They have their own language, codes, and culture. Such strange goings-on...

Thank you for all you do for us, kind Wizard!

[Donw]

raywjohnson Boy do I identify with your last post. My first computer was a Altair 8800B bought in the mid 70’s Hardest part was assembly and then learning opcode for programming. After that things got easier as I learned Assembly, BASIC,C, C++, PASCAL, Objective C, and other languages, all because of documentation. When I purchased Borland’s Delphi first version it came with 2 thick books for documentation.

I struggled, and still do at times, when it comes to Linux and the internet, and its seemingly Wild West of things that all need to work together to get things working correctly. Personally I blame the hodgepodge of the often terse, conflicting or outdated online documentation. However thinking about it, maybe I’ve just gotten too damn old.

My hat is off to you, and the other members here, who have approached fluency in this new era of computing. And I hope the AI gods are kind to you in the future.

[Henrik]

Thanks for the summary raywjohnson, and for your hard work. I understood everything, as I'm a software developer since 25 years. I think they were just lazy, and were trying to hijack as many servers as possible, not realizing (or looking into) what AmazingRibs is all about (size-wize I mean). Thanks again for being to fix it so soon, and do let us know if you figure out how they got in to begin with.

[Jerod Broussard]

Kinda like the Spammers; just dropping something and gone to the next www.

[RAYMBO]

Yep. I have been battling this for so many years. The most infuriating thing is that they hit and run. Breaking 10,000 sites to get one or two spam redirects in place.