Redirects are helpful and are overlooked many times in important transitions. Many businesses redesign their websites and go live with a completely new website and website structure yet forget to add redirects. In forgetting this, they lose the rankings of their old pages and quickly lose visitors to the website.
When a website is redesigned and the URLs of directories and pages change, a redirect maintains the old URL in the rankings and forwards both visitors and search engines to the new versions of the website’s pages. There are many types of redirects, but two are used more than others. A 301 redirect is considered a permanent redirect (or change of URL address) and is usually recommended. A 302 redirect is a temporary redirect and can be used in some cases.
Robots.txt
The Robots.txt, as shown in Figure 4.11, is a text file that sits at the root level of your website’s hosting server. This is requested by the search engines at the initiation of every crawl. The robots.txt file informs the search engines which content is off‐limits and should not be indexed.
The Robots.txt file is not a security measure! Not all robots will follow these instructions. It is simply a protocol that excludes directories or files that are not important or critical for indexing. Some also use this file to manage duplicate content by listing duplicated files or directories for exclusion.
This little file has probably created more headaches for site owners than any other factor. This is because with one little slash, it can remove your website from the search results! In Figure 4.11, the difference between full site access and blocking the search engines completely is a forward slash.
Many times, developers will add this protocol to keep development sites from being indexed and ranking in the search engines. Yet when the site goes live, this file is often forgotten! This causes the site to drop out of the rankings because the search engine spiders are being instructed not to index the website. Each of the webmaster tools provided by Google and Bing will verify the format of your Robots.txt file and ensure that it is properly developed for your website.
Leave a Reply