# Crawlers Setup User-agent: * # Directories Disallow: /app/ Disallow: /var/ # Paths (clean URLs) Disallow: /catalog/product_compare/ Disallow: /en/catalog/product_compare/ Disallow: /se/catalog/product_compare/ Disallow: /catalogsearch/ Disallow: /en/catalogsearch/ Disallow: /se/catalogsearch/ Disallow: /checkout/ Disallow: /en/checkout/ Disallow: /se/checkout/ Disallow: /customer/ Disallow: /en/customer/ Disallow: /se/customer/ Disallow: /review/ Disallow: /en/review/ Disallow: /se/review/ Disallow: /wishlist/ Disallow: /en/wishlist/ Disallow: /se/wishlist/ # Files Disallow: /LICENSE.html Disallow: /LICENSE.txt Disallow: /LICENSE_AFL.txt # Paths (no clean URLs) Disallow: /*?SID= Disallow: /*?price= Disallow: /*?powIdt= Disallow: /*?manufacturer= Disallow: /*?SID= Disallow: /*&SID= Disallow: *&SID= Disallow: &SID= Disallow: /*?price= Disallow: /*&price= Disallow: *&price= Disallow: &price= Disallow /*?powldt= Disallow /*&powldt= Disallow *&powldt= Disallow &powldt= Disallow: /*?manufacturer= Disallow: /*&manufacturer= Disallow: *&manufacturer= Disallow: &manufacturer=