disallow all crawlers access to certain files.

User-agent: *
Allow: /homeopathy/
Allow: /homeopathic/
Disallow: /custom/
Disallow: /manual/
Disallow: /m/
Disallow: /amazon/