The Forum Is Wonky

[captainlee]

Worked fine here yesterday and today.

[dpearce]

raywjohnson Maybe you've answered this before, but does Amazingribs.com and pitmaster.amazingribs.com share the same server, or are they split up on different servers? And do you guys disallow the AI scrapers?

[dpearce]

SheilaAnn Haha! Ray and I are in the same boat, although different technology bases. Years ago I dabbled in PHP sites (20 years now?), but have since migrated through about 3-4 presentation scripting languages. At this point, I'm dealing with Vue.js paired with Tailwinds (migrating off PrimeVue), with a C# API backend and SQL Server database. I was just curious how he and his cohorts structured the sites. Nothing nefarious, I promise!

[SheilaAnn]

No nefariousness implied, sir. Is that even a word? 🤣🤣🤣

[Johnny Booth]

SheilaAnn At least I understood what you said. 🤣

[RAYMBO]

raywjohnson Maybe you've answered this before, but does Amazingribs.com and pitmaster.amazingribs.com share the same server, or are they split up on different servers? And do you guys disallow the AI scrapers?

They are in different servers. However, in the near future, they both might get moved to the same server. That idea has not been fully vetted as of yet. I am petitioning to change the domain name to https://pitmaster.club/

I did implement a method to block AI scarpers, but was directed to undo that. Since both sites are protected by CloudFlare, I could disallow AI from all sites with a few clicks. AI hammering servers and overloading them is such a problem now that CloudFlare added the ability to block them before they even get to the server.

There is the usual mindset on AI (as with all new tech) is get on board or be left behind. I disagree with that. As it is an idea that is promoted by those that want you to use the new technology.

[dpearce]

We've toyed with the idea of blocking AI from our sites, but ultimately we decided to let them have access. For us, it's all about SEO at this point, and AI's getting more and more integrated with search engines, so we're reluctant to restrict them. Obviously, we don't have near the amount of info you guys have here, and that's probably the only thing that saves us from getting hammered to the point of our sites being unusable.

[RAYMBO]

Yep. That is the fact. If you block the AI bots, your SEO tanks.

[dpearce]

I did implement a method to block AI scarpers, but was directed to undo that. Since both sites are protected by CloudFlare, I could disallow AI from all sites with a few clicks. AI hammering servers and overloading them is such a problem now that CloudFlare added the ability to block them before they even get to the server.

Just did a little research, and according to cloudflare's developers docs, apparently there's entries you can add into your robots.txt file to limit the speed at which they crawl. Obviously, you'll need a list of all the current agents though.

Example Robots.txt Configuration

You can also allow or disallow different AI bots and search agents, for instance:

User-agent: GPTBot Disallow: /

User-agent: PerplexityBot Allow: /
Crawl-delay: 5

User-agent: ClaudeBot
Crawl-delay: 8

User-agent: *
Crawl-delay: 10

Disallow: /admin/
Disallow: /internal/

This configuration blocks some bots, allows others, and sets different crawl rates.

I don't know if this helps or not, or if you already knew this info, but thought I'd post it anyway.

Thanks again for all you do Ray (and everyone else!)

[RAYMBO]

- You are welcome.

It should help, it does not. I have tested it. I had Crawl-delay of 3. I thought a reasonable delay. No bot honored it.

Just for everyone's info, robots.txt does not block, it is a request only. Bots are free to ignore it completely. archive.org and stated publicly that it will NOT honor robots.txt directives.

[dpearce]

Ah yeah, doesn't help when the crawlers don't honor it. Well, the good news is, you do tools on CloudFlare at least, so if they get too aggressive you can always shut them off at the source!