Tactical Crawling Large Sites Effectively

@rachellcostello brightonSEO
TALK ABOUT WHAT YOU KNOW
What we do every day:
Help people to crawl websites
to get insights, no matter how
many URLs they have.

Recommended fixes for a better site.
WHAT WE’LL COVER
Tips and tactics for crawling large sites.
Common pitfalls of the biggest brands.

CRAWLING LARGE SITES
Effectively & Efficiently

The sheer scale of enterprise sites and
knowing where to start can be daunting.

That’s why you need to start smaller.

Targeted crawls can run regularly, allowing
even the largest sites to bypass resource
and time constraints.

Good news: you don’t
need to crawl every
URL every time.
TACTICAL CRAWLING

You only need enough
data to validate issues.
TACTICAL CRAWLING

TACTICAL CRAWLING =
Getting the data you need as
quickly as possible.1
Building the bigger picture
from smaller parts.2

With tactical crawling methods, you can
use segments to give a representative
picture of the whole site.

Learn more about a site by breaking it into
manageable, predictable chunks.

Depending on the vertical, each site should
be able to be split into 5-7 categories:

Category PagesProduct Pages Blog Pages Vanity Pages
Navigational
Pages
Transactional
Pages
FAQ Pages

SAMPLING SLICING

SAMPLING

SAMPLING
This method is about
collecting a percentage
of the whole site.

SAMPLING METHOD #1
Start a small crawl and find
areas of crawl waste to
exclude for whole crawl.

Homepage
Category Page
Subcategory
Page
Tracking URLs
Pagination URLs
Product Page Parameter URLs
Social Tag URLs

SAMPLING METHOD #2
Use level limitation to
assess site breadth without
getting lost in the weeds.

Homepage
Category Page
Subcategory
Page
Category Page Category Page
Subcategory
Page
Subcategory
Page
Level 1
Level 2
Level 3
Product Page Product Page Product PageLevel 4

SAMPLING METHOD #3
Crawl a certain number of
examples of each page
type.

Product Page 4Category Page 4 Blog Post 4
Category Page 3
Category Page 2
Category Page 1
Product Page 3
Product Page 2
Product Page 1
Blog Post 3
Blog Post 2
Blog Post 1

SLICING

SLICING
This method is about
analysing an isolated
section of the site.

SLICING METHOD #1
All of the products but none
of the categories
(horizontal slice).

SLICING METHOD #2
Crawl a distinct section that
serves a separate function
(e.g. blog or separate
mobile site).

Homepage
Category Page
Subcategory
Page
Category Page Blog
Blog Category
Subcategory
Page
Level 1
Level 2
Level 3
Product Page Product Page Blog PostLevel 4

1. A strong history of organic traffic.
2. Consistent levels of customer engagement.
+
3. Close monitoring of all changes made.
+
HAVE SUBSETS WITH:

Because when you see the needle move
it will be more meaningful.

Supplement your segments in external data sources too.
Use custom properties,
inclusion rules and
filtered reports to
pre-filter data in:
Analytics tools
Log files
Sitemaps
Google Search Console

Tactical crawling allows you to get to know
your website and its structure better.

Establishing patterns means you can crawl
more efficiently in future.

But, remember to also run annual or
semi-annual full crawls and tech SEO
audits.

The takeaways from these are the primary
baselines for planning tactical crawls.

HOW THE BIGGEST
BRANDS MEASURE UP

I analysed the most recent audits
we’ve completed using tactical
crawling methods for our biggest
enterprise clients.
Here are the key takeaways of 3...

FOOD
DELIVERY
TRANSPORTATION
ONLINE
CLASSIFIEDS

The key overlapping problem
areas...

1
2
3
4
5 Mobile
Internal Linking
Site Speed
Indexing
Internationalisation

MAIN
INTERNATIONALISATION
FIXES

Removing URLs
for nonsensical
locales.

Removing URLs
for nonsensical
locales.
Use location to
show relevant
results.

Removing URLs
for nonsensical
locales.
Use location to
show relevant
results.
Allow users to
toggle between
languages.

Removing URLs
for nonsensical
locales.
Use location to
show relevant
results.
Allow users to
toggle between
languages.
Add x-default
for non-
regionalised
URLs.

Removing URLs
for nonsensical
locales.
Use location to
show relevant
results.
Allow users to
toggle between
languages.
Add x-default
for non-
regionalised
URLs.
Translate all
titles and
descriptions to
the correct
language.

MAIN INDEXING FIXES

Replace 302
redirects with
301 redirects.

Replace 302
redirects with
301 redirects.
Fix any server
errors in Google
Search Console
‘Crawl Errors’
report.

Replace 302
redirects with
301 redirects.
Fix any server
errors in Google
Search Console
‘Crawl Errors’
report.
Ensure all
content
elements are
unique and
relevant.

Replace 302
redirects with
301 redirects.
Fix any server
errors in Google
Search Console
‘Crawl Errors’
report.
Ensure all
content
elements are
unique and
relevant.
Increase
pagination to
show more
results per
page.

Replace 302
redirects with
301 redirects.
Fix any server
errors in Google
Search Console
‘Crawl Errors’
report.
Ensure all
content
elements are
unique and
relevant.
Increase
pagination to
show more
results per
page.
Review
orphaned URLs
in the sitemaps.

MAIN SITE SPEED FIXES

Lazy-load
below-the-fold
images.

Lazy-load
below-the-fold
images.
Eliminate use of
unnecessary
plugins.

Lazy-load
below-the-fold
images.
Eliminate use of
unnecessary
plugins.
Remove unused
and/or
render-blocking
scripts.

Lazy-load
below-the-fold
images.
Eliminate use of
unnecessary
plugins.
Remove unused
and/or
render-blocking
scripts.
Reduce page
HTML
size for high
priority
templates.

Lazy-load
below-the-fold
images.
Eliminate use of
unnecessary
plugins.
Remove unused
and/or
render-blocking
scripts.
Reduce page
HTML
size for high
priority
templates.
Increase
caching time for
static pages
from minutes to
days.

MAIN INTERNAL LINKING
FIXES

Remove
redirect loops
and chains.

Remove
redirect loops
and chains.
Reinstate links
to pages with
10+ impressions
and/or visits.

Remove
redirect loops
and chains.
Reinstate links
to pages with
10+ impressions
and/or visits.
Remove
internal links to
nofollow pages.

Remove
redirect loops
and chains.
Reinstate links
to pages with
10+ impressions
and/or visits.
Remove
internal links to
nofollow pages.
Avoid using
parameter URLs
for tracking.

Remove
redirect loops
and chains.
Reinstate links
to pages with
10+ impressions
and/or visits.
Remove
internal links to
nofollow pages.
Avoid using
parameter URLs
for tracking.
Remove
internal links to
canonicalised
search pages.

MAIN MOBILE FIXES

Avoid using
‘maximum-scale
=1’ in viewport
settings.

Avoid using
‘maximum-scale
=1’ in viewport
settings.
Change
‘user-scalable=
no’ to ‘yes’ in
viewport
settings.

Avoid using
‘maximum-scale
=1’ in viewport
settings.
Change
‘user-scalable=
viewport
settings.
Migrate all
pages to
responsive
ASAP.

Avoid using
‘maximum-scale
=1’ in viewport
settings.
Change
‘user-scalable=
viewport
settings.
Migrate all
pages to
responsive
ASAP.
Ensure each
mobile-desktop
reciprocal
URL pair has
200 status.

Avoid using
‘maximum-scale
=1’ in viewport
settings.
Change
‘user-scalable=
viewport
settings.
Migrate all
pages to
responsive
ASAP.
Ensure each
mobile-desktop
reciprocal
URL pair has
200 status.
Fix content
mismatches
between
mobile and
desktop URLs.

Luckily, there are already
resources on these topics!

Internationalisation - bit.ly/internationalisation-dc
Indexing - bit.ly/indexing-dc
Site Speed - bit.ly/site-speed-dc
Internal Linking - bit.ly/internal-linking-dc
Mobile - bit.ly/mobile-dc
RESOURCES

TO WIN IN ENTERPRISE SEO:
Run targeted, agile crawls on smaller sections
of your site.
Better understand internationalisation, indexing,
site speed, internal linking and mobile.
Use the findings to deliver quicker and more
impactful insights into site health.
1
2
3

THANK YOU!
Any questions? Send me a tweet:
@DeepCrawl www.deepcrawl.com

Tactical Crawling Large Sites Effectively

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Similar to Tactical Crawling Large Sites Effectively

Similar to Tactical Crawling Large Sites Effectively (20)

More from DeepCrawl

More from DeepCrawl (16)

Recently uploaded

Recently uploaded (20)

Tactical Crawling Large Sites Effectively