Server busy notification

[J-Melt]

I had it happen about an hour or two ago.

Then it happened as I attempted to make this post.

[Sweaty Paul]

Happened to me today at 1743 CDT. Obviously the server doesn't recognize how important and cool I am!

LOL!

[HawkerXP]

It is making me feel bad.


I haven't been busy for two years.

[WI Bubba]

Got a new one tonight.



I'm sure those words mean something to someone, but it's all Greek to me.

I've been getting a lot of "server busy" errors to the last couple of days, but I'm sure they will go away as they usually do.

[Donw]

I’ve seen that one on and off for several days.

[RonB]

I just got the server busy notice 10 min ago.

[Willy]

I'm seeing it often too.

[RAYMBO]

Sorry, I missed the "server too busy" part.

That is from a setting that checks the server loading. If the server CPU loading (average for 5 minutes) is above 20, then is shows that message. And basically will not allow any posts, search, etc.

A load of 20 is massively excessive. Normal loading values fluctuate between 0.50 and 1.50. A value of 1 (in general) means the CPUs (there are 6) are working at 100%. That is actually highly efficient. So values up to 6 will not affect much. But once you start getting above 15, things slow down, and processes start to pile up.

I should actually set the cutoff to 10, but short spikes of 20 or more should normally not be an issue.

Now I need to figure out what is causing the spikes. I cannot be the backups as the entire site is not accessible during backups. Those only last about 8 minutes.

[RAYMBO]

I am up to my freakin' ears with these bots. I tried to login about 40 minutes ago and could not reach the site.

I logged in via SSH to check the CPU Load. It was high, but not bad. However, there were 30 PHP processed (the limit) continuously active.

Under normal conditions, up to 10 (maybe) spiking to 25/30.

I scanned the log for bots. Some new one I have never herd of his hammering the crap out of server. Not even trying to do it slow enough not to cause issues.

Called SleepBot/1.0; +http://sleepbot.com/ -- today, 107,400+ hits in the log.

Count = User Agent
107405 = Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; SleepBot/1.0; +http://sleepbot.com/) Chrome/131.0.0.0 Safari/537.36

To be fair, the total for all none bots is about 140,000. So nearly half from one bot.

I have blocked a bunch of bots. A soon as I did that, the server returned to normal.

For the record, I have this same problem on other servers, not just here.

And finally, I think this is bot is using a fake user-agent string. The site (http://sleepbot.com/) has not been updated since 2008. And is not a bot site. You can set the user-agent string to whatever you want.

[RAYMBO]

Follow up funny.... I hope no one read my last post before I fixed it.

I am the typo master.

This:
"I logged in via SSH to check the CPU Load. It was high, but not bad....."

Was this:
"I logged in via SSH to check the CPU Load. I was high, but not bad...."

[Jerod Broussard]

"I Was high" no problem....."I is high" would have raised flags, "But I got I" more flags, "Anybody wanna buy the best weed for getting high" all the flags

[RAYMBO]

[SheilaAnn]

“I was high”

You do you, friend! JK 🤣🤣🤣🤣🤣🤣🤣🤣

[RAYMBO]

Follow up #2: If you see a page that is blank except for "404", please let me know. That is the page I am redirecting bad bots to. So there could be false positives and I will need to adjust the blockers.

[RAYMBO]

Follow up #3: I am on a mission. Knife in mouth, gun in hand, hunt for ways to put a stop to these ass hat, jerkwater, filthy scammers.

I have found that there are new features in Apache just for protecting against these kinds of actions. I will working on this today.

My clumsy attempts to block them are just not reliable enough.

[Jerod Broussard]

So they jam up the server, raspberry perhaps?

[RAYMBO]

Alrighty then!

I have implemented a feature on the web server called: mod_qos

Per AI: Summary of mod_qos - A "Quality of Service" module for the Apache web server. Its main job is to ensure your website stays available to real users even when it is under heavy load or being attacked.

Basically, when the hackers start heavy scans, they get blocked automatically. I have set Google and Bing on the allow list. They generally do not hammer the server.

I will be monitoring and tweaking the configuration as needed.

[Ace]

Outstanding... Thank you sir.

[Sweaty Paul]

Thank you for your indefatigable efforts!

[Mosca]

Thank you, Ray. It’s the stuff you do behind the scenes that keeps the clubhouse running!

[jfmorris]

raywjohnson love the new monikor! But when I typed it in it reverted to your pre-Raymbo name.

I've been dealing with this exact issue in recent months on a small phpBB based forum site, and have had to block LARGE blocks of mis-behaved bot IP's recently. I finally addressed it, for now, a couple of weeks ago.

It took a while of me bouncing Apache access logs, fail2ban logs and such off of Copilot (AI) for analysis, to determine the IP blocks running the bots. I then had to setup system level firewall rules.

If interested in my discovered list of botnet IP's to block at the firewall level, send me a private message, and I can pull the list, including what each range is blocking.

Jim

[RAYMBO]

jfmorris

Display name vs User name. I changed the display name only. But the @ system uses the user name. Not sure why, the display name system is supposed to hide the username.

I will send you a message. I tried the IP based blocks. Those work for a time, until they change there IPs or new bots/botnets are create. Then you have to start again. And that is only for IPv4. IPv6 will make IP blocking basically obsolete.

Try a search for Apache mod_qos and how to leverage fail2ban with it. That is my next step. Adding fail2ban to the mix.