15 second search wait

[Dutch]

This is just out of personal curiosity, I'm not bitching whatsoever, because I'm sure there's a good reason for it...but why is there a 15 second wait between forum searches? Something to do with the ad bots?

Probly been answered before, couldn't figure it out.

[Solecord]

What are you searching for when it takes a long time? I just ran a couple of searches and got results in less than a second?

[Dutch]

I should have worded it better, sorry. It's not a delay in displaying search results, it's a timer that won't allow subsequent searches within 15 seconds. I was looking up threads pertaining to MD3 models. First I searched keyword MD3, then tried .md3 to see if anything different would pop up, and got this message:

"The following errors occurred with your search:

This forum requires that you wait 15 seconds between searches. Please try again in 12 seconds."

The 12 will change depending on how many seconds are left.

[Solecord]

Oh yea, sorry... that's due to bots hammering the search which in turn causes the entire server to run slow.

[Mindf!3ldzX]

Oh yea, sorry... that's due to bots hammering the search which in turn causes the entire server to run slow.

I'd love to jam my hard disk in a bot owners eyesocket, because that 15 second wait period always pisses me off so I leave Quakeone.com and just end up utilizing Google's fantastic indexing via Google search , lol.

[Phenom]

This is just out of personal curiosity, I'm not bitching whatsoever, because I'm sure there's a good reason for it...but why is there a 15 second wait between forum searches? Something to do with the ad bots?

Probly been answered before, couldn't figure it out.

Because Solecord has to get up, search thousands of threads, then create a page with a list of variable threads for your search criteria, he's just one guy common!

At least that's how I think the internet works. :]

Oh yea, sorry... that's due to bots hammering the search which in turn causes the entire server to run slow.

Can't you block bots for the search via robots.txt?

Something like...

User-agent: *
Disallow: /search.php

Or are we talking about the bad bots that ignore robots.txt?

[Dutch]

Oh yea, sorry... that's due to bots hammering the search which in turn causes the entire server to run slow.

Yeah was my first guess...appreciate the answer, thanks! Was just curious, I haven't been a member of very many different forums, but never saw that before.

Because Solecord has to get up, search thousands of threads, then create a page with a list of variable threads for your search criteria, he's just one guy common!

At least that's how I think the internet works. :]

LMAO! Boy, all of a sudden, 15 seconds seems pretty damn fast!

That reminds me of the "If Google was a Guy" videos on youtube...

[Solecord]

Can't you block bots for the search via robots.txt?

Or are we talking about the bad bots that ignore robots.txt?

Yea, talking about bad bots.

I had that off back when we first launched the site and my server's CPU was hammered because of the bots. I can try turning it off again and see how it goes... my server did get upgraded last winter so perhaps it can handle it again...

Alright, I'm turning off the timing... hold on to your butts...

[MadGypsy]

hmmm, that sounds like a big gamble just to save impatient people a few seconds. I get that same message from time to time and I simply wait <15 seconds. Did I (or do I even need to) mention that I am the most impatient person on the planet? I could think of numerous things on this site that need attention far more than waiting <15 seconds.

[Dutch]

hmmm, that sounds like a big gamble just to save impatient people a few seconds. I get that same message from time to time and I simply wait <15 seconds. Did I (or do I even need to) mention that I am the most impatient person on the planet? I could think of numerous things on this site that need attention far more than waiting <15 seconds.

I never brought it up cause I was impatient or wanted to see it changed, I was just curious as to the reason behind it. And I was bored at the time so I thought I'd just ask and find out. Lol.

[Phenom]

Yea, talking about bad bots.

Couldn't you edit the template file for the search box and do something like:

if($_SERVER['HTTP_USER_AGENT'] = '$bad_bot_array') {
header('Location: http://quakeone.net');
} else {
echo 'Hello I'm a search box!';
}

It would redirect all the bad bots in the list to 'some other website' and if the $_SERVER['HTTP_USER_AGENT'] doesn't qualify as a 'bad_bot' it gets to access the search box?

The $bad_bot_array would obviously have to be a list either preset and added onto over time which may cause pinholes in this security screening or else a start from scratch list which would become more effective over time but be more efficient.

[Solecord]

lol @ quakeone.net

[MadGypsy]

if you are going to use a list to block bots that list should be in your .htaccess, not a php array with a redirect. You just check your logs daily and add bots as you discover them. It's not an elegant solution, nor is it ideal. Unfortunately, blocking bad bots with a list is a pain in the ass no matter how you approach it. To make matters worse, this site is a bit international so, blocking entire countries is not really an option.

Alternately, but using the same approach, you could simply write a bot that checks your logs for you and appends to your .htaccess as it discovers bad bots. A long .htaccess can also cause a performance hit on your server though. You can easily check if this is happening with a site like www.gtmetrix.com

probably needs work but, it's a decent place to start, I guess.
https://github.com/bluedragonz/bad-b...ster/.htaccess

found this too
http://www.allthingsdemocrat.com/blo...n-htaccess.txt

yet another .htaccess starter kit
http://www.javascriptkit.com/howto/htaccess13.shtml

[Phenom]

if you are going to use a list to block bots that list should be in your .htaccess, not a php array with a redirect. You just check your logs daily and add bots as you discover them. It's not an elegant solution, nor is it ideal. Unfortunately, blocking bad bots with a list is a pain in the ass no matter how you approach it. To make matters worse, this site is a bit international so, blocking entire countries is not really an option.

Actually, that's a much better way to do it, what was I thinking?

It would be diligent work but it would work.

[MadGypsy]

Actually, that's a much better way to do it, what was I thinking?

Doesn't matter what you were thinking. It just matters that you were thinking. It was your use of a list that sparked me to think of the .htaccess file. I've never worried about bots cause I've never bothered to make any of my sites popular but, I did mess around with appending IP addresses to the .htaccess file as a ban solution within one of my projects. Your idea wasn't absolutely terrible, it just needed a tweak. Technically there is still a redirect and a list, just no PHP and they never get to the search box in the first place.

PHP "arrays" are hash tables and they are slow. In other words, in order to access an element, PHP has to calculate a hash and traverse a linked list. A bad bot list in PHP with any real data in it would add significantly to page load times. To make matters worse, using something like in_array will add even more overhead.