.Google.com's John Mueller answered a concern about why Google.com marks web pages that are prohibited coming from crawling by robots.txt and why the it is actually safe to neglect the related Search Console reports about those crawls.Crawler Traffic To Concern Specification URLs.The individual talking to the inquiry recorded that bots were actually developing links to non-existent query criterion Links (? q= xyz) to pages with noindex meta tags that are likewise blocked in robots.txt. What triggered the concern is that Google is actually crawling the hyperlinks to those webpages, acquiring blocked out by robots.txt (without noticing a noindex robots meta tag) after that receiving shown up in Google Look Console as "Indexed, though blocked out through robots.txt.".The person inquired the following concern:." Yet listed here is actually the major inquiry: why would Google.com mark web pages when they can not even see the web content? What is actually the advantage in that?".Google.com's John Mueller affirmed that if they can not creep the page they can't observe the noindex meta tag. He also produces an exciting acknowledgment of the website: search driver, encouraging to disregard the results since the "typical" consumers will not observe those outcomes.He composed:." Yes, you are actually correct: if we can't crawl the web page, our company can't see the noindex. That claimed, if our experts can't creep the webpages, then there's certainly not a lot for us to mark. Thus while you could view a few of those pages with a targeted web site:- query, the average user won't observe them, so I definitely would not bother it. Noindex is actually additionally fine (without robots.txt disallow), it merely suggests the Links will end up being crept (and also end up in the Explore Console document for crawled/not listed-- neither of these standings cause problems to the rest of the site). The integral part is that you do not make all of them crawlable + indexable.".Takeaways:.1. Mueller's answer validates the limitations in operation the Site: hunt advanced hunt driver for analysis reasons. One of those factors is actually because it is actually certainly not hooked up to the routine hunt mark, it is actually a distinct trait completely.Google.com's John Mueller talked about the website hunt driver in 2021:." The short solution is actually that an internet site: inquiry is actually certainly not suggested to become comprehensive, nor utilized for diagnostics reasons.An internet site inquiry is a particular type of hunt that limits the end results to a certain internet site. It is actually generally simply words web site, a digestive tract, and afterwards the web site's domain.This concern confines the outcomes to a details website. It is actually not meant to become a detailed selection of all the pages from that website.".2. Noindex tag without utilizing a robots.txt is actually fine for these kinds of situations where a bot is connecting to non-existent web pages that are actually acquiring found out through Googlebot.3. Links along with the noindex tag will certainly generate a "crawled/not catalogued" item in Look Console which those will not possess a negative result on the remainder of the internet site.Review the question and answer on LinkedIn:.Why would certainly Google index webpages when they can not even see the web content?Featured Image through Shutterstock/Krakenimages. com.