Search Engine Friendly Robots.txt and Sitemap.xml

Making it Easy for the Internet to Find Your ecommerce Products

What is Robots.txt and Why Does My Website Need It?

 

A Robots.txt file is the standardized way in which your ecommerce website tells search engines how and what to crawl and index on your site. This is very useful if you have pages which, for various reasons, you do not want search engines to index and crawl. Common applications include blocking crawlers from non-public portions of your site, print versions of HTML pages (and other duplicate content), and auto discovery of XML Sitemaps. Having a robots.txt makes indexing a website and its content much more efficient for the search robot.

Clarity's ecommerce solution automatically generates this file and keeps it updated as you add new content to your website. Additionally, editing the robots.txt file is very easy for individual pages through Clarity's user friendly platform. Furthermore, Clarity's automatically generated

It is important to note two common mistakes people make in regards to the use of robots.txt. First, malicious robots will ignore the file, so DO NOT try and use it as a security measure. Second, sometimes it is better to tell a search crawler not to index the page in the meta description of that page rather than the robots.txt. This way, other areas of your site still receive 'link juice,' or links from the noindex page, that help boost the SEO value of other areas of your site.

For more information on Robots.txt best pracitces for SEO, visit SEO MOZ's article here. 

 

---

What is Sitemap.xml and Why is it Important for SEO?

 

Whereas the Robots text's purpose is to efficiently tell search crawlers what to steer clear from within your site, the Sitemap.xml is used to tell search crawlers which URLs within your site are the most important. This bit of Extensible Markup Language (XML) helps search engines recognize the hierarchical structure of your site, giving precedence to your homepage first, then your main navigation pages, next their subtopics and so on. This process ensures your most important pages are getting the attention from search engines that they need in order to rank high in search results.

Again, Clarity's ecommerce platform makes this easy by automatically generating new content within their appropriate place wihtin the XML Sitemap. The Sitemap Clarity's solution generates is clean, from both a users and a search crawlers standpoint, and follows best practices. These two documents are essential to any website, but are a pain to generate manually. The automated generation of these pages is just one more example of how Clarity's ecommerce solution makes the jobs of your team both easier and more efficient.