Server busy notifications

[synodog]

0730 Mountain Standard Time (AZ doesn’t do Daylight Savings Time) or PDT or 1430 GMT lol.

[RichieB]

10:18 AM East.

[RAYMBO]

Oops! Forgot. I disabled cloudflare for now. See if that helps.

[PGH_RAM]

14:39 EST today

[Alan Brice]

HP Laptop. This has been the message most of today. So I thought I would try my mobile. No prob.
mobil= I phone 12. Laptop = HP, Duck Duck Go.
Now the mouse pointer is glued to the upper right corner of the open program. (AmazingRibs)
when I try to move the pointer the whole window shrinks or expands depending which way the mouse pointer goes.

[acorgihouse]

Got same screen as Alan Brice just above here, 7pm Eastern on Thursday. I had had the slow running thing for days, not "accepting" a like, freezing, but just now first time I think I got the server notice.

[Carolyn]

Server busy 8.03 pm.

[RAYMBO]

Results of my investigations.

I initially did not find anything that stood out that could be the cause. The analytics are showing spikes in CPU and Drive access every two to three ours. But no correlation with Network usage, no spikes. Which means the overload is not from too many users (bots, hackers, DOS/DDOS attacks), but from the server (apache, mariadb, php).

Drive access (Disk I/O Rate) is showing a correlated swap usage. Swap is the term for a "swap file" on the hard drive that is used like Memory when the regular memory gets used up (the server has 16GB and should never get used up, so swap should never need to be used).

That was the clue. I found that the database (mariadb) was using 80% of memory (12GB, it's allowed max) leaving 4GB for everything else. This caused the swap file to be used. Which is an order of magnitude slower than memory. Like 700ms read/write vs 7ms read/right.

That can very easily cause the system to get backed up and overloaded (i.e. busy).

The server runs on Akamai (Linode) and I do have alerts for high usage (CPU, Disk, Network). None were triggered as the length of time for the overloading was very short. A few minutes. Just enough to be annoying! I did set the CPU alert down a bit, but not the Disk I/O as that was never high 1/10 the alert value. Unfortunately there is no alert for swap usage.

The fixes. I made minor adjustments to the mariadb settings and restarted mariadb. Memory usage by mariadb seems to have stabilized at 45%. I will keep an eye on that.

Keep posting "server busy" reports if you get that notice.

[Panhead John]

I was just getting ready to PM you Ray, to see if you checked for that ….🥸

[Draznnl]

Thank you Ray.

[RichieB]

Thanks and whatever you said I agree.

[Donw]

Unfortunately still slow at times this morning. And now getting Error 0 often when liking a comment.

[Panhead John]

Liking comments is giving me a hard time too, along with a few delays in posting comments.

[DaveD]

I've been on several times today and everything has been nominal. Had one server error about 9am EDT but all good since then, FWIW.

[RAYMBO]

I did discover that the site was been scraped by bots. Hammering the server. Scrapy.org was the main culprit. I added some directives to block that (and other) bad bots. The block only works if the bot uses its correct user agent string. That can easily be changed to get around the block.

But, since enabling that, the spikes in CPU and Disk I/O have stopped. That was about 10am Central time yesterday.

[Panhead John]

I figured that! 🥸

[fkrall]

After getting off, couldn't immediately reaccess the site around 0940 Eastern 06/07

[RAYMBO]

There was a CPU/Disk IO spike just before 0940 Eastern.

[fkrall]

Thanks, Ray. After searching for and learning about "bot," I don't envy you. Thanks for hanging in there!

[RAYMBO]

I am still working to get the bots to go away, I added a few more blocks. I have verified the they are working. Returning "403 Forbidden" to the bot.

The last 24 hours of activity, bot vs normal traffic. The # is the count of times the bot accessed the site.

15960 meta-externalagent (this is facebook)
11172 GPTBot
8308 GoogleOther
3472 Googlebot
306 Googlebot-Image
1528 bingbot
1404 SemrushBot
1367 DataForSeoBot
966 ChatGPT
520 ClaudeBot
424 Amazonbot
311 DotBot
45738 Total

vs

7285 regular traffic

So normal traffic being about 16% of the total. Bots are taking over the world!

[Michael_in_TX]

This mimics what we see at the University on our main web site. The rise in the LLM bots has been nothing short of shocking. We also see the Meta bot in the top five easily.

[Jerod Broussard]

what a pain in the bot

[RiverJeff]

Does this happen when face book articles and the others are listed to open and view?

[Donw]

raywjohnson I believe some of the bots may be back as the working popup and error 0 dialog are showing up again often. I also noted the post by smokenoob about the photo attached to a post being out of sync and displaying another member Thought I would let you know. Thanks.

[RAYMBO]

Thanks for letting me know. I will see what I can do.

[Donw]

raywjohnson I know you are not fond of Cloudflare but I just read that they are going to block A.I scrapers by default now in their settings. Hope this helps in your fight against the scrapers. Just my 2¢

https://www.nytimes.com/2025/07/01/technology/cloudflare-introduces-default-blocking-of-ai-data-scrapers.html?unlocked_article_code=1.TE8.7mNh.X2b qf9FDtxPk&smid=url-share