The Forum Is Wonky

[RAYMBO]

All is well as of now.

Of course, just after I fix the configurations to start login real IP addresses, the bots vanished.

Red line is where the configuration was fixed...



Figures.

I am now certain that this was either a direct DOS/DDOS attack or someone scraping the site (and being a jerk since you can set your scraper/crawler to slowly hit the site, even once a second would not be noticeable). One other bit of info, found via CloudFlare, most of the hits were from Hong Kong.

I created access log scanner script to pull the offending IPs when (not if) this happens again. I might work on an automated blocker. If more that x hits in x minutes, block for x hours.

[Panhead John]

That’s exactly what I was going to recommend! 🙄

Thanks Ray!

[jayjordan]

raywjohnson - Thanks for the many hours you put into finding and resolving the issues!

[Duanessmokedmeats]

I recommend an airstrike on Hong Kong!

[RAYMBO]

Updated Graph: I realize that the times were UTC. I set the account Timezone to Central. And my red line was off a bit...

[Donw]

All this scraping of data, often copyrighted, to produce AI insightful analysis and easy to understand information to educate people. I feel smarter already. 🙄

[Huskee]

This pic is hilarious!

[Michael_in_TX]

Ray,

As I work in infosec, I appreciate you taking the time to share the technical details. I wish I had a useful suggestion to offer. At the risk of suggesting something utterly unpractical, have you considered nginx? It seems to be a little more DDoS resilient in my experience. Granted, that would be a major architecture change and something that would be far down the road, of course.

Horizontal scaling is also probably out of the picture due to expense, I imagine. And that would just mitigate, but not solve, the problem.

DDoS attacks are insidious. We certainly get them from time to time at the University. For us, when they occur our procedure is to go get a cup of coffee and take a walk around the datacenter. We have such a huge inbound pipe that there effectively is no defense.

My favorite DDoS story is when we DDoS'ed ourselves. It was Wednesday, September 19, 2012. Approximate mid-day, we couldn't get to external web sites. Then we "fell off the internet." Rice University soon followed.

We quickly realized the traffic flood was coming from inside our network, nearly all from our wireless network. We were being attacked from within! A few packet captures later we saw what was happening and couldn't believe it.

Apple had released its much-anticipated iOS 6 update that day. 20,000+ iPhones were all trying to download it at once. Due to a horrendous misconfiguration with Akamai, it flooded our network and, as at the time, we went through Rice to go to the outside world, we brought them down, too!

Fun day.

[DaveD]

The call is coming from INSIDE THE HOUSE....!!

[Huskee]

Takeaway: even bots and nefarious ne'er-do-wellers can't stay away from us.

[Jerod Broussard]

You've pointed out Panhead John what else ya got

[Panhead John]

Hey, I’m no bot! A nefarious ne’er-do-weller? Well, occasionally…..🥸

[Smoker_Boy]

I can't even spell nefarious.

[RAYMBO]

As I work in infosec, I appreciate you taking the time to share the technical details. I wish I had a useful suggestion to offer. At the risk of suggesting something utterly unpractical, have you considered nginx? It seems to be a little more DDoS resilient in my experience. Granted, that would be a major architecture change and something that would be far down the road, of course.

I am not a fan of nginx (for those that do not know, it is pronounced Engine X). The configuration for anything complex is always difficult. And no .htaccess files. Everything has to be in the config file. It is currently used on the main (free) site. I dread having to make changes to the config.

Horizontal scaling is also probably out of the picture due to expense, I imagine. And that would just mitigate, but not solve, the problem.

Yep. I robust solution would be to have 7 servers, 2 for load balancing, 4 to serve the site, and one for the database. 2 load balancing so that if one goes down, the other will still be operational (they could also be uses as caching proxies for static content using.... nginx! But easy configure and you rarely have to make changes). All this makes the costs x 7. Not to mention the extra complexity and managment.

DDoS attacks are insidious. We certainly get them from time to time at the University. For us, when they occur our procedure is to go get a cup of coffee and take a walk around the datacenter. We have such a huge inbound pipe that there effectively is no defense.

My favorite DDoS story is when we DDoS'ed ourselves. It was Wednesday, September 19, 2012. Approximate mid-day, we couldn't get to external web sites. Then we "fell off the internet." Rice University soon followed.

We quickly realized the traffic flood was coming from inside our network, nearly all from our wireless network. We were being attacked from within! A few packet captures later we saw what was happening and couldn't believe it.

Apple had released its much-anticipated iOS 6 update that day. 20,000+ iPhones were all trying to download it at once. Due to a horrendous misconfiguration with Akamai, it flooded our network and, as at the time, we went through Rice to go to the outside world, we brought them down, too!

Fun day.

Fun day indeed!

[DaveD]

Actual unretouched photo taken on the outskirts of Hong Kong of Ray conducting a commando op from his home in Thailand to root out the cyberattackers...

[STEbbq]

Seriously, it Ray doesn’t make this his avatar…it is perfect.

[SheilaAnn]

Do it ray!!

Avatar!
Avatar!
Avatar!

[RAYMBO]

Done!

[DaveD]

Yes... it is Raymbo: First Blood, part 37. We just need that voice-over dude from back in the day... "In a world where Hong Kong hackers attack a barbecue forum... Only one man can stop them: RAYMBO."

[Donw]

Okay Ray. New member name!

RAYMBO
Code Ninja

[DaveD]

Yeah!!! Raymbo in da houuuuuuuuse!!!

[STEbbq]

Woohoo!!!!!

[Donw]

raywjohnson Last night, East coast time, and continuing into this morning I’m getting a lot of “Working” messages and also Error 0 messages. Basically a slowdown.

[Panhead John]

It’s starting to mess up for me also. The last few hours have been slow, very slow, to load pages or to post something. @RAYMBO

[RAYMBO]

I checked the load data. There was a spike for about 2 hours on CPU and DISK, but not network. So something on the server. The daily backup lasts about 6 minutes. Let me know if the site is still slow for you.

[RichieB]

raywjohnson I have not had the issues reported this time. I must be special.

[RAYMBO]

special special

[DavidNorcross]

Painfully slow today for some reason. Anyone else seeing this same issue?

[Sid P]

Yep, a lot of "Working".

[Panhead John]

Same here. It’s off and on sloooow. Been going on for a few days now. Occasionally it’s taken 30 seconds to a minute to load or change pages.

[Smoker_Boy]

20 minutes ago I couldn't do anything.
Now it's fine.

[RichieB]

I had a couple of burps today.

[Panhead John]

When I just now clicked on this page in Notifications, it took about 30-45 seconds to get here. Does it on my iPad and iPhone running the latest update.

[Huskee]

The blocks I set up to stop you are failing and it's letting you in!! Drat!

[Panhead John]

With all my skills I should put my name in the hat for the next administrators position…..🥸

[Huskee]

You just may have more skills than I!