Screaming Frog’s Crawl Comparison Feature Update

Google has already installed Rankbrain, BERT, and MUM into its algorithms to improve its search results for its users. However, how did Screaming Frog stay on top of things? With every Google update, this SEO spider tool adds new features that are worth checking out.

One of its feature updates that I look forward to trying out for our SEO Hacker team’s future website audits is its crawl comparison feature. Screaming Frog has a tutorial on how to use this feature to compare crawls and analyze how crawl data has changed over time.

So, what did this feature do when it came to doing website audits? I explored the feature and found some neat and time-saving “tricks” that will help me and my team audit websites. Let’s get right into it.

SEO Spider Tools’ Crawl Comparison Feature Update

The reason why I look forward to Screaming Frog’s crawl comparison feature is how it will help you in the same way how it has helped us with our clients’ website audit reports.

Over the years, my team and I have done our clients’ website audits in a pretty straightforward manner. We put the URL of our client’s website in the Screaming Frog tool, then we let it crawl the site. Afterward, we save and export the data in a spreadsheet to check out and identify any errors or issues for optimization opportunities. Then we create and design an easy-to-understand PDF about the strategy and fixes that must be done to improve rankings. That’s pretty much it.

The crawl comparison feature update aims to help monitor the progress of a website’s SEO issues and opportunities. This feature lets you compare crawls over time and see the changes in data, issues, and opportunities in bite-sized information.

To see how Screaming Frog’s crawl comparison feature performs, I have done some tinkering with the feature and explored it myself.

In the tool’s interface, you will find that the Mode tab is in Spider by default. Switch it from Spider to Compare mode so you start your crawl comparison analysis.

To start your crawl comparison, select the files you want to compare side by side. When you click on the compare button, the crawl comparison analysis will run.

Once it’s done with the analysis, the overview tab on the right-hand of the interface will populate, showing your current and previous crawled data and changes. For illustration purposes, I used my stored crawled data in the tool from a new website we’re currently working on.

Since both crawled files were from the same month, there isn’t much to compare with the crawled data. Nevertheless, having explored the feature I find this to be extremely convenient for my and my team’s future website audits.

I’ve also observed that before I switched over to the database, I was using RAM and it was slowing the computer down. After the switch, my computer’s performance significantly increased while Screaming Frog was running its crawl comparison analysis at the same time.

Essentially, the SEO spider tool is in Memory storage by default. This means every crawled data is automatically stored in your computer’s short-term memory (RAM). What actually happens when the crawled data is stored in RAM is that it slows down your computer.

This explains why Screaming Frog strongly recommends configuring your storage mode to Database. When I switched my licensed tool’s storage mode from Memory to Database prior to running the analysis, my computer significantly performed better.

Benefits of Screaming Frog’s Crawl Comparison Feature

Truth be told, my team and I have been identifying technical SEO issues and monitoring our optimization efforts’ progress manually since day one.

Having experimented with Screaming Frog’s crawl comparison feature, I find it efficient in producing the data I need for my technical SEO sweep checks. While its basic crawling feature does a fine job already, I recommend you get the licensed version of the tool to access and make the most of the feature.

So, what led me to explore this little-known feature in the first place? Here are some reasons why I believe this feature might be beneficial to you as well.

It helps you save time

Screaming Frog’s crawl comparison feature makes comparing data a lot easier. It eliminates the need to manually transfer or export data from the tool into a spreadsheet. Consequently, the only tool to check your website’s technical problems and monitor its progress is the tool itself.

In addition, this feature also makes monitoring your optimization efforts a breeze. Because your crawled data is imported in the tool, you will only have to click on the File>Info tab via interface and find the crawl data you wish to compare.

It helps save your computer’s memory storage

Whenever you run crawl comparison analysis, the tool switches its processing from Memory to Database by default.

Screaming Frog is keen on the allocation of a computer’s memory. This is because you can only do so much with its free version. Switching to the licensed version of the tool will give you access to Database storage. Database storage not only lets you crawl beyond 500 URLs (especially if you’re performing audits for larger websites), but it also keeps my computer from slowing down while the tool is running in the background.

It detects the website’s other important on-page elements

The overview tab found on the right side of the interface lets you see its default key metrics including the total number of URLs, its internal (same) and external (different) subdomains, response codes, and changes between crawls. However, it doesn’t show you if other key elements have been changed, such as title tags, meta descriptions, internal links, etc. by default.

You can quickly change this by adjusting the comparison configuration for change detection in the Screaming Frog tool. Configuring this setting lets the tool perform analysis around your key metrics of choice, such as H1, page title, crawl depth, meta description, internal links, and more. For your website to rank in search engines, you need to optimize these elements.

Click on the cogwheel icon in “Compare mode” so you can select which item you would like to detect for change.

Doing this is extremely useful when checking whether the title page, meta descriptions, or other SEO recommendations were already updated or not.

Key Takeaway

For years, my team and I have used Screaming Frog SEO spider to crawl and analyze to improve our clientele’s search rankings. To me, this is one of the most necessary SEO tools out there. Even more so now because of its crawl comparison feature update.

Dabbling in the tool’s crawl comparison feature is just one of its many little-known features that I have recently immersed myself in. If you have a licensed version of the tool, I recommend you try it out today and experiment with how you can consolidate it in your general website health checks and website audits.

Have you utilized Screaming Frog’s crawl comparison feature on your website audits? How did it go? Let me know by leaving a comment below!