andyfrench.info

Search engines such as Google can make use of XML sitemaps to discover content. Such sitemaps are useful if a site:

• Has dynamic content
• Has pages that aren't easily discovered during the crawl process (e.g. rich media)
• Is new and has few links to it
• Has a large archive of content pages that are not well linked to each other, or are not linked at all

The XML sitemap protocol is an open standard defined by http://www.sitemaps.org.

According to the specification:

• You can provide multiple sitemap files
• Each sitemap file can only contain 50,000 URLs
• Site map files must be < 10 MB
• Compression is allowed
• Multiple sitemap files should be listed in a sitemap index file

The location of a sitemap file is important to what URLs can be contained in it.

“A Sitemap file located at http://example.com/catalog/sitemap.xml can include any URLs starting with http://example.com/catalog/ but can not include URLs starting with http://example.com/images/.” – see http://www.sitemaps.org/protocol.php#location

See http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156184 for Google specifics.

Search engines may down-rank pages if the same content is referenced by multiple URLs. This would apply in cases where URL parameters are added to the URL to put the page in different modes (e.g. sort order). It would also apply in the case of pages that are referenced by different portals (i.e. different domains).

For example the following URLs could all point to the same content:

• http://<domain name here>/mypage.aspx
• http://<domain name here>/mypage.aspx?sort=desc
• http://<domain name here>/mypage.aspx?sort=desc&page=2
• http://<another domain name here>/mypage.aspx

To assist search engines in identifying the canonical version of the content add a <link> tag to the <head> section of the duplicate content:

<link rel="canonical" href="http://<domain name here>/mypage.aspx" />

For Google’s guidance go here: http://googlewebmastercentral.blogspot.com/2009/02/specify-your-canonical.html

More on duplicate content:

• http://googlewebmastercentral.blogspot.com/2008/09/demystifying-duplicate-content-penalty.html
• http://www.google.com/support/webmasters/bin/answer.py?answer=66359