Several times when your browsing you find that Google throws up search results which lead to pages which sit behind a username/password protection. You need to register to view the content. Websites like New York Times and Wallstreet Journal and many more require you to register and/or pay to view their content. Google manages to index that password protected content.. how? It’s because a lot of websites leave a little backdoor for the Google bot to enter through. This way the Google bot can index their content and their content shows up within search results.
There are couple of different ways to solve this problem.
- It’s to convince the website that you are the Google Search Bot rather than an ordinary user by changing your browser’s userAgent name. To do this you can use the following registry edit:
[HKEY_LOCAL_MACHINE\SOFTWARE\Micro$oft\Windows\CurrentVersion\Internet Settings\5.0\User Agent]@=”Googlebot/2.1″”Compatible”=”+http://www.googlebot.com/bot.html”
Save it as useragent.reg and merge it into your registry. All done.
For switching back to your browser’s userAgent use this hack:
[HKEY_LOCAL_MACHINE\SOFTWARE\Micro$oft\Windows\CurrentVersion\Internet Settings\5.0\User Agent]@=”Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)”
- Otherwise you can use Firefox extensions like BugMeNot. BugMeNot is a online database of username and passwords which people have contributed and you can bypass the registration by a simple right click.
BugMeNot will work in a lot of situations, however someone should have added a username/pw to the database. Also BugMeNot won’t work where you need to pay to get in. Thats where Google’s Backdoor is perfect.