The Forum Is Wonky

[Meathead]

Ray says it is fixed:

I figured it out.

It was vbulletin caching causing the problem.

I recently tweaked the cache settings to fix another issue.

But that just caused vbulletin to generate a 72GB cache file. The database dump is normally less than 3GB, but the bloomed to 75GB.

So the backup system was soaking up all the resources and choking out the server.

I have fixed the cache settings and I will be monitoring.

[WI Bubba]

That peaky law of unintended consequences....

[smokenoob]

ah the ol’ “for every bug you fix you create 2 bugs”

[J-Melt]

Great job raywjohnson! Thank you for all you do to keep this great community up and running!

[RiverJeff]

raywjohnson This morning I got an error saying I could not save a like to a post?

[Panhead John]

Not a problem for me…..🤷♂️

[Mosca]

Tell you what, it’s lickety split now!

[ComfortablyNumb]

Deja vu? I got scores of 'server busy' just now trying to check the forum

[Huskee]

It was a good 12 hrs!

[Panhead John]

Same here!

[RichieB]

It's back.

[RonB]

raywjohnson - it's doing it again. I had to use a roundabout way to get here...

[RAYMBO]

Getting reports of server issue over the last hour or so. I am checking things out now.

[RAYMBO]

Hmmm... so the same loading issue as before, but I cannot blame the cache table or the backup this time as the loading is happening now and the backup will not start until about 4a (central time).

I did find 16 IPs hammering the server (all hidden by cloudflare), but after blocking them, nothing changed. So not the bots hitting the server.

I will be tweaking some configuration settings (in an hour or so, after pacific time goes past midnight). I will also do a server reboot.

I will solve this mystery! Eventually.

[Jerod Broussard]

If you need help I once stayed in a Holiday Inn Express.

[RAYMBO]

[RAYMBO]

I went ahead and did a reboot (a minute ago). I wanted to see if that fixed the issue. Which would indicate some process or system getting stuck or corrupted. But the reboot changed nothing. The mystery continues.

[RickyBobby]

Having server issues also. Additionally, received this.

[RAYMBO]

Thanks for that report. I do need the link to the topic.

[J-Melt]

I got this at 5:05am central time.

Thank you for your work on this raywjohnson!

[RAYMBO]

Still working on it. It is a DOS/DDOS attack. CloudFlare is supposed to protect against that, but is actually preventing the real IPs from being logged. I just discovered this issue and I will have to work on the Apache configuration to get and log real IP addresses.

---

This is a real head scratcher. I cannot find any issues with the server or services.

Even though the CPU keeps spiking, as high as 300%, that should not be a problem. There are 6 CPUs so you have to divide by that. If the server was at 600% CPU (or 6 CPUs at 100% each), the server would still be fine.

• Checked Apache (web service) Configuration. Updated some options. All good.
• Checked PHP-FPM (script service) Configuration. Updated some options. All good.
• Checked MariaDB (database service) Configuration. Updated some options. All good.
• Checked Performance data on MariaDB, no issues.
• Checked all tables on MariaDB, no issues.

There were 3 tables that were still the old/obsolete format (MyISAM), converted to InnoDB, no issues.

Checked for database tables without indexes. None found. Non-indexed tables are slow as MariaDB then has to generate the index on the fly for every query on the table.

I activated the Slow Query Log for queries that take longer than 1 second. One logged in 2 hours. So not the problem.

However, when I activated the "log_queries_not_using_indexes" option, many queries logged. I cannot fix vBulletins poorly implemented queries.

So far, nothing has fixed the issue. And there is no indication of any problem with the server, services, or settings.

There is the one issue: 56 IPs, that have hit the server 10,000 times or more, in today's access log. For a total of: 703,046 hits. That is about 10 hits per second.

All are CloudFlare IPs, all have the same (fake) User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36

And these IP cannot be blocked as doing so will block everyone. As I mention, I just discovered this and will work on a fix.

---

Been a long day...

[Panhead John]

After waking up this morning, the forum has been working fine for me since 7am CST if this helps. Hope I don’t jinx it!

[DaveD]

I've had many error message and server-too-busy pages all morning. Just as bad as before. Can't be good if Ray is scratching his head like this...