Tag Archives: GoogleBot

15MB Only for Googlebot Today

When Google decided to update its official documents in June 2022, SEO practitioners and specialists started worrying about a particular piece of content that’s been added to the Googlebot crawl limit. Stated within Googlebot’s official Google documentation is the following statement:

“Googlebot can crawl the first 15MB of an HTML or support text-based file. Any resources referenced in the HTML such as images, videos, CSS, and JavaScript are fetched separately. After the first 15MB of the file, Googlebot stops crawling and only considers the first 15MB of the file for indexing. The file size limit is applied to the uncompressed data…”

Source: Googlebot

GoogleBot User Agent’s Update Rolls Out in December

GoogleBot optimization can be tricky especially since you cannot control how the spider perceives your site. As Google announced that they are going to update the user agent of GoogleBot come December, this is also a call for us SEOs to rethink how we can optimize for Google crawlers.

The Complete Guide to Robots.txt and Noindex Meta Tag

How do I use Robots.txt and the noindex meta tag?

Quick Answer: To create a Robots.txt file, you can use any text editor (such as Notepad). Make sure to save the file with UTF-8 encoding during the save file dialog. This file must also be named “robots.txt,” and your site can only have one such file. This file must also be located at the root of the website host you’re applying it to. To use a noindex tag for pages that you do not want to be included in search results, add “<meta name=”robots” content=”noindex”>” to the <head> section. Or, you can add the noindex tag using a X-Robots-Tag in the HTTP header: “X-Robots-Tag: noindex”.

Using Log File Analysis to Optimize Your SEO Strategy

The challenge for webmasters everywhere is to accurately know what GoogleBot is doing on your website. GoogleBot’s primary job whenever they enter a website is to crawl a specific number of pages set by the website’s crawl budget. After crawling, they save the pages they crawled to Google’s database.

Send us an Email