SEO Robots.txt File for WordPress

Self-hosted WordPress blog is advised by WordPress.org to use Robots.txt file to tell search engine about what to crawl and what not.

Here’s Robots.txt file example which is good for SEO (search engine optimization) according to WP

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /feed
Disallow: /comments
Disallow: /category/*/*
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Disallow: /*?*
Disallow: /*?
Allow: /wp-content/uploads

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*

# digg mirror
User-agent: duggmirror
Disallow: /

Sitemap: http://www.example.com/sitemap.xml

Don’t know how to create robots.txt file? See here.

Another sample of a good robots.txt file is here.

What is User Agents?

What is User Agents? User agents are the programs bot visitors (Google bot, Yahoo search bot, Bing bot, etc) use to see your content. Human visitors will normally use a web browser like Internet Explorer, Firefox, Chrome, etc.

Search spiders and other computer programs (grouped together as “bots”) go through your web pages by scanning the source code – What you see if you right-click on a web page and select “View Source”.

How to Create Robot.txt File

How to Create Robot.txt to allow all search engine robots to access your website / blog

You may create a robot.txt file in a notepad as follow:

1. Click Notepad from your computer
2. Write the following:
User-agent: *
Disallow:
3. Save it.
4. Upload it to the root directory. For example, if you blog is website.com, the file should be uploaded in public_html/–upload here–
5. Done.

Another format of robot.txt is as follow:

User-agent: *
Allow: /

Note: You cannot create Robot.txt file in a Blogspot or WordPress.com blog as you don’t have access to the root level directory. It can be done in a self-hosted blog only.

How to Remove Unwanted Character in End of URLs

How to Remove Unwanted Character in the end of URLs of your WordPress homepage / frontpage URL, post url and page url.

If you check your website through Google Webmaster Tools, you’ll find there are some URLs which are ended in unwanted variables / characters such as http://www.fatihsyuhud.com/seo/EygMYqwqH instead of http://www.fatihsyuhud.com/seo/

In such case, what you have to do is to install Permalink Redirect plugin by Yoast, activate it. That’s it. From now on anyone who come to the URL http://www.fatihsyuhud.com/seo/EygMYqwqH would be redirected to the true URL that is http://www.fatihsyuhud.com/seo/

If you’re interested, you may download it here for free.