Let's Chat! 1-(888) 262-6687
1st on the List Promotion Inc.
  • Home
  • Services
    • SEO
      • Custom SEO Services
      • Local SEO Services
      • SEO Consulting Services
      • SEO Page Experience
      • Technical SEO Audit
      • Content
      • Link Acquisition SEO
      • Social Media
      • Analytics and Reports
      • Additional SEO Services
    • Website Design
      • WordPress Web Design
      • Ecommerce Website Design
      • Website Content
      • Conversion Rate Optimization
      • Security and Maintenance
    • PPC
      • Custom PPC
      • Local PPC Mgmt
      • PPC Audits
      • PPC Ad Retargeting
      • PPC Social Media Ads
    • Content
      • SEO Content
      • Copywriting
      • Content Audits
      • Strategy Development
      • Training
  • Our Work
    • Case Studies
    • Website Design Portfolio
    • Client Results
  • About
    • What Sets Us Apart
    • Our Headquarters
    • Our Team
    • Who We Work With
    • Testimonials
    • FAQ
  • Blog
  • Contact
  • Start a Project
  • Menu Menu
Google & Bing, Technical SEO

Understanding Crawl Budgets and How to Increase Your Website Crawl Budget with Google

Reading Time: 4 minutes

Google admitted in their Webmaster Central Blog on January 16, 2017, that they don’t have an official “crawl budget definition”, indicating that a single term would not suffice. They went on to indicate that they would clarify what they actually have and what it means for Googlebot.

In this article, we will explore what Google has told us about site crawl budget and how it operates.

In This Article

  • What is Crawl Budget?
  • Should I Be Concerned About My Site Crawl Budget?
  • What are Google’s Limits to Site Crawl Rates?
  • What is Crawl Health?
  • How Frequently are Crawl Demands Made?
  • 6 Ways to Improve Your Website Crawl Budget According to Google

What is Crawl Budget?

As an SEO, when we talk about crawl budget we are referring to the resources Google will allocate to crawling (or discovering) the pages on your website. The budget could be determined by number of pages and or the time Google will spend crawling. That’s right – Google has limits on how long and how many pages of your website they will crawl!

Should I Be Concerned About My Site Crawl Budget?

For pages crawled on a daily basis, Crawl Budget is not a high priority concern for websites with fewer than 1,000 URLs. As long as there isn’t anything blocking Google out of the site there shouldn’t be issues crawling these pages.

Google stresses that website crawl budgets are more improtant for larger sites.

What are Google’s Limits to Site Crawl Rates?

Crawling is a priority, but Google claims it does so without degrading user experience on the site, instituting crawl rate limits for given sites to minimize that possibility. They define crawl rate limits as the number of simultaneous connections used by Googlebot to crawl, and the wait-time between fetches.

What is Crawl Health?

On websites that respond rapidly to Google crawling its pages, the crawl limits go up, using more connections, and allowing Google to crawl and discover more and more pages. This allows Google to index and rank more and more pages.

For slower sites or those with server errors, the crawl limit is throttled back and fewer pages are crawled as the Googlebot has to wait longer to crawl each page.

Google also reminds website owners you can manually set limits for crawling inside your Google Search Console, but that setting higher limits does not mean they will automatically increase the crawl on your site.

How Frequently are Crawl Demands Made?

If there is little demand from indexing (even when peak crawl rates haven’t been reached) there will be little Googlebot crawl activity.

More popular URLs are usually crawled more consistently to keep Google’s index up-to-date, as another objective for the crawler is to prevent URLs from becoming stale.

Crawls can also be triggered if a website is moved and new URLs need to be re-indexed.

Ultimately, the rate and demand are what define a crawl budget in Google’s eyes.

6 Ways to Improve Your Website Crawl Budget According to Google

Google says that crawl rate is negatively affected by websites with numerous low-value-add pages and categorizes these types of websites into 6 main groups, listed by order of importance.

If your site is large enough to have a Crawl Budget, or if you want to prevent your site from falling under a Crawl Budget, address the following prioritized issues, as applicable.

1. Address URLs with naturally duplicated content.

Faceted navigation (ability to filter pages by price, colour, size, etc.) affects the crawl budget because they contain many combinations of a URL with duplicated content. These prevent Google from crawling new and unique content as quickly or index pages correctly as a result of diluted signals between versions that have been duplicated.

Session identifiers also fall under this list. User info and tracking information stored in these URLs cause duplicated content through the numerous URLs used to access a single page.

2. Minimize duplicate text content from page to page.

On websites with content duplicated across several pages, Google uses algorithms designed to prevent this duplicated content from adversely affecting user-experience or webmaster-experience. Here’s how Google deals with this duplicate content:

  1. When duplicate content is found, the URLs are grouped into a cluster.
  2. The best URL that represents the cluster is chosen and presented in search results.
  3. Properties such as link popularity within the cluster are consolidated and applied to the chosen URL that represents the cluster.

3. Mark old deleted pages with 404 Not Found response.

Crawl rate can be affected by soft error pages that occur when a server responds with a 200 OK response if a page does not exist instead of a 404 Not Found which is more appropriate. This limits site crawling because these old deleted pages might be crawled instead of other live pages on the site.

4.  Deal with any hacked website pages and content ASAP.

If your site has any hacked pages, well, just don’t expect Google to crawl it anytime soon.

5.  Make sure that each URL on your website has its own unique purpose.

Sites affectionately called infinite spaces (sites with an excessive number of URLs) are not high on Google’s list of crawlable content. Make sure that each page you want Google to crawl has its own unique purpose and message. What need does it meet for your visitor?

6. Purge any low quality or spam pages.

Pages containing low quality and spam are right up there with the hacked pages and will negatively affect the crawling rate of the site.

Google determines all six of these areas as a waste of their resources and delays them from discovering the great content of a website.

If you have any questions about Crawl Budget, whether or not your site contains any of the items that might negatively affect your crawl budget, give 1st on the List a call at 1-888-262-6687. You can also reach us through email at [email protected].

Learn more about How Search Engines Work to crawl, index, and rank your website!


Carissa Krause

Carissa Krause is a Digital Marketing and Project Specialist at 1st on the List. Over the last 12+ years she has worked in our Abbotsford office with clients on a wide range of projects that include areas like local SEO, project reporting, backlink profile review, content development, strategic planning, and more. Whatever the project may be Carissa focuses on achieving greater efficiencies and putting plans into action. When away from her desk you’ll likely find her drinking all the coffee while sitting on the floor driving cars with her two young boys. Learn more about Carissa.
View all posts
February 4, 2017/by Carissa Krause
Share this entry
  • Share on Facebook
  • Share on Twitter
  • Share on WhatsApp
  • Share on Pinterest
  • Share on LinkedIn
  • Share on Reddit
You might also like
Silver Chain with one Golden link7 Reasons Internal Link Structure of Your Website Matters for SEO

Recent Posts

  • How to Attract Local Customers
  • Optimize Your Google Business Profile
  • How Mobile Local SEO Can Help Your Business
  • The Perfect SEO Page
  • What is Local SEO?

Explore Topics

  • Social Media Blogs
  • mobile SEO
  • GBP Profile
  • Business & Marketing
  • Company News
  • Content & Keywords
  • e-commerce
  • Google & Bing
  • Lead Generation
  • Link Building
  • Local PPC
  • Local Search
  • Mobile Search
  • Search Engine Marketing (PPC)
  • SEO in the News
  • Social Optimization
  • Technical SEO
  • SEO
  • Conversion Optimization
  • Web Design

1st on the List celebrate 25 years

Trusted SEO Experts, Delivering Results Since 1997. We’ve spent 25 years getting the best rankings and branded exposure for our clients in Canada and the US through winning SEO, Web Design, and PPC Strategies. Join our growing list of happy clients today!

GET PROPOSAL

COMPANY

About
Team
What Makes Us Different
Who We Work With

OUR WORK

Client Results
Case Studies
Web Design Portfolio
Testimonials

SERVICES

SEO
Local SEO
Technical SEO Audits
SEO Consulting
Web Design
Content
Pay Per Click
Social Media

CONTACT

201 – 33119 South Fraser Way
Abbotsford BC, Canada
V2S 2B1

34878 Millar Crescent
Abbotsford BC, Canada
V2S 7K5
1-888-262-6687
 [email protected]

© Copyright 1997-2023 - 1st on the List Promotion Inc. All Rights Reserved.
  • Facebook
  • Instagram
  • Twitter
  • LinkedIn
  • Youtube
  • Blog
  • Press and Partners
  • Resources
  • FAQ’s
  • Privacy Policy
  • Sitemap
Scroll to top