Self-hosted WordPress blog is advised by WordPress.org to use Robots.txt file to tell search engine about what to crawl and what not.
Here’s Robots.txt file example which is good for SEO (search engine optimization) according to WP
User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /feed
Disallow: /comments
Disallow: /category/*/*
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Disallow: /*?*
Disallow: /*?
Allow: /wp-content/uploads# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*# digg mirror
User-agent: duggmirror
Disallow: /Sitemap: http://www.example.com/sitemap.xml
Don’t know how to create robots.txt file? See here.
Another sample of a good robots.txt file is here.
Mas Apa jadinya Klo Kita Tidak Membatasi Directory Tersebut ?
Matur suwun nggih sam,,,,langsung praktek dh,,
Not sure you really need one, as I think they are auto generated. But if you want to customize things like this it would be necessary.
- Robert
You can use “PC Robots.txt” WordPress Plugin, it Auto Generated Robots.txt and you can easily change it in your dashboard..
nice post..
i will introduce this tutorial..
I am very lucky to find your web site. Your article is very useful. Thank you for share
kalo dah pake robot txt ini crawl gugel dah gag bisa masuk ke wp admin beneran ya gan?
Iya gan, fungsi robots.txt itu sendiri kan untuk membatasi crawl search engine ke blog kita
info bagus, bisa praktek langsung