博主用 robots.txt 与 meta 标签让 Google 完成去索引
I've Been Deindexed by Google
一位博主今年早些时候决定不再让搜索引擎索引自己的网站，先设置 robots.txt 全站 disallow，等待数周无效后又加上 robots meta 标签，并特意允许 Googlebot 抓取以读取该标签。最后一次调整距今至少 3 个月，他今日确认网站已被 Google 去索引。
Citron
07 Oct, 2026
I don't remember exactly when but, earlier in the year, I decided I didn't want this website to be indexed by search engines anymore. A lot of it has to do with AI and the relentless scraping that plagues the internet now. I know that as long as my blog is accessible by the public internet, there are no real measures I can put in to stop it but at least some of it will be respected.
I started out by setting a blanket disallow in the robots.txt.
User-agent: * Disallow: /
When I realised that didn't do anything after a few weeks of waiting and checking, I set the robots meta tag.
<meta name="robots" content="noindex, nofollow" />
I thought that would do the trick. What I didn't realise is that you have to actually, ironically, allow the bot that will do the deindexing to visit your website so that it can see the meta tag that tells it to not index your page anymore.
User-agent: Googlebot Allow: / User-agent: * Disallow: /
I also did this with several other common bots and removed them from the robots.txt whenever I found that my website had been deindexed.
It's been at least 3 months since I did my last change to have Google deindex me and, after checking today, I'm happy to announce that this website is free from Google! 🥳
来源：Hacker News · kennyqin.com