Hey guys, I was always impressed by the number of guests browsing the forum until I clicked "Quick Links" and "Who's Online"...the majority of the guests are indexing spiders, mostly from Yahoo.
I know spiders need to crawl website content to keep their indexes up to date but their has to be a way for forum software logic to exclude these "guests" from the "Who's Online" list.
...a simple robots.txt file with the appropriate parameters limits their presense.
I know spiders need to crawl website content to keep their indexes up to date but their has to be a way for forum software logic to exclude these "guests" from the "Who's Online" list.
...a simple robots.txt file with the appropriate parameters limits their presense.
Comment