Hyper Dog Denver SEO Blog | Colorado Web Site Development » Security

5 web development techniques to prevent Google from crawling your HTML forms

Jim — Fri, 18 Apr 2008 21:36:39 +0000

Google has recently decided to let it’s Googlebot crawl through forms in an effort to index the “Deep Web”. There are numerous stories about wayward crawlers deleting and changing content through submitting forms, and it’s about to get worse. Googlebot is about to start submitting forms in an effort to get to your website’s deeper data. So what’s a web developer to do?

1. Use GET and POST requests correctly
Use GET requests in forms to look up information, use POST requests to make changes. Google will only be crawling forms via GET requests, so following this “Best Practice” for forms is vital.

2. Make sure your POST forms do not respond to GET requests
It sounds so simple, but many sites are being exploited for XSS (Cross Site Scripting) vulnerabilities because they respond (and return HTML) to both GET and POST requests. Be sure to check your form input carefully on the backend, and for heaven’s sake – do not use globals!

3. Use robots.txt to keep robots OUT
robots.txt file keeps Googlebot out of where it doesn’t belong. Luckily, Googlebot will continue it’s excellent support of robots.txt directives when it goes crawling through forms. Be sure not to accidentally restrict your website too much, however. Keep the directives simple, excluding by directory if possible. And test, test, test in Google’s Webmaster Tools!

4. Use robots metatag directives
Using the robots metatag directives for more refined control. We recommend “nofollow” and “noindex” directives for both the form submission page and search results pages you want Google to stay out of, even though Google says disallowing the form submission page is enough. Consider using tags and category pages that are Google friendly instead.

5. Use a CAPTCHA where possible
Googlebot isn’t going to fill out a CAPTCHA, so it’s an easy way to make sure some bot isn’t filling out your form.

Googlebot is, of course, the nicest bot you can hope to have visit your website. This provides a chance to secure forms and take necessary precautions before other – not so polite – bots visit your forms.

Tags: denver web design, seo, website development, denver web development, search engine optimization, website design

Tweet This Post

Keeping track of multiple passwords

Jim — Wed, 13 Sep 2006 21:28:26 +0000

RSA Security’s newest password management survey found that one of the greatest threats to corporate security is the weak password. Employees that change their too often, or have to juggle too many passwords for login to various services, are likely to choose weak passwords or even write them on a scrap of paper near their station. I am a little suspicious a survey that highlights RSA security as the solution to this problem, but it is valuable to stop and ask yourself “Do I have too many passwords to keep track of?”.
Sure, too many passwords lead to “irresponsible password behavior”. A single login and password for every service is usually a bad idea, too. Once an intruder has access, they could wreak tremendous havoc.

A sensible alternative is to choose Four passwords that you can actually remember. Make each password incrementally more random, if possible. Choose the weakest password, and use it to sign up for services that only need a password for the most rudimentary of tasks. Use the “second level” password for sites that may have some personal information – your name, address, etc. Save the “third level” password for sites that have your credit card on file. The final password is to be used only in online banking and/or paypal.

Gee, so simple. But who can keep track of Four passwords, anyway?! Good luck out there – no one ever said good security was easy!

(More information on the password survey)

Tweet This Post

Electronic Voting Machines: More troubles

Jim — Wed, 13 Sep 2006 06:02:22 +0000

In Maryland, the electronic voting machines were inoperable when polling places opened. It was human error, though: Plastic wallet sized cards needed ro operate the machines were forgotten and a scramble ensued. By the end of the day, election workers had voters casting on scraps of paper, sealed inside envelopes purchased at the local pharmacy. Despite the cases of security issues(once hacked by a monkey?!), electronic voting is here to stay. Unfortunately, it seems the machines are being phased in too quickly – without the necessary fallbacks to paper balloting systems when needed. Track the latest issues during the primaries at www.blackboxvoting.org.

Tweet This Post