Deep dive into session isolation and why search engines render pages in isolated rendering sessions to avoid having the rendering of one web page affect the functionality or the content of another. Web crawling tools aim to replicate search engines' crawling and rendering behaviours by implementing and using web rendering systems. This offers insights into what search engines might see when they are crawling and rendering web pages. While there is no defined standard for an automated rendering process, search engines (e.g. Google, Bing, Yandex) render pages in isolated rendering sessions. This way, they avoid having the rendering of one web page affect the functionality or the content of another. Isolated rendering sessions should have isolated storage and avoid cross-tab talking.

1. RESEARCH: Validating session isolation for
web crawling to provide data integrity

2. ● Web Rendering, Search Engines, and Web Crawlers
● Research context
● What is Session Isolation?
● Session Isolation in the wild
● Solving Session Isolation
● Tests
● Conclusion
Table of contents
@giacomozecchini

3. Web Rendering, Search
Engines, and Web Crawlers

4. We are not in the 90s anymore
As new web rendering patterns got traction on the
web, we moved from static HTML pages to more
complex ways of rendering content.
@giacomozecchini
https://www.patterns.dev/posts/rendering-introduction/

5. New rendering patterns emerged
With the massive use of rendering patterns such
as Client-Side Rendering and Progressive
Hydration, search engines were somehow forced
to start rendering web pages and retrieve almost
as much content as the users would get with their
browsers.
@giacomozecchini
https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics

6. Web Rendering Systems to save the day
Search Engines have developed their own web
rendering systems (or web rendering services).
These are a piece of software that is able to
render a large number of web pages by using
automated browsers.
@giacomozecchini
“Googlebot & JavaScript: A Closer Look at the WRS” by Martin Splitt: https://www.youtube.com/watch?v=Qxd_d9m9vzo

7. Web Crawler tools followed Search Engines
Web crawling tools also started to build
rendering systems to keep up with the evolution
of the web and mimic search engines'
capabilities.
@giacomozecchini

8. But rendering is hard!
There is no industry standard for rendering
pages, which means that not even leading search
engines such as Google are doing it in the
“correct” way.
Each web rendering system is built to serve
specific use cases, which results in inevitable
tradeoffs.
@giacomozecchini

9. Research Context

10. At Merj we’ve been happy users of many web
crawling tools and during the years we probably
used all of them at least once.
We’ve been using web crawling tools for years
@giacomozecchini

11. We’ve been building custom WRS solutions
For use cases such as custom data sources in
complex data pipelines for enterprises, we have
been building our own web crawling systems .
@giacomozecchini

12. Data integrity assurances
The starting point of this research was a recent
project that required us to provide assurances to
a legal and compliance team about the data
quality and integrity of a data source (rendered
pages). These were to be ingested into a machine
learning model.
@giacomozecchini

13. Data validation process
In addition to other checks present in our data
integrity validation process, we tested the output
of multiple web crawling tools.
We found some unexpected values which varied
across tools.
@giacomozecchini

14. What is Session Isolation?

15. What is Session Isolation?
While rendering a page in an isolated rendering
session, the page must not be able to get any
data from previous rendering sessions and be
influenced by other pages' renderings.
@giacomozecchini

16. Stateless is a similar concept
This is similar to the concept of “stateless” as
used for web crawlers, where all fetches are
completed without reusing cookies and without
keeping in memory any specific data.
@giacomozecchini

17. Session Isolation in the Wild

18. Content customisations based on navigation
Session isolation real world problems can be
found by observing the rendering of pages that
have content customisations based on user
navigation.
@giacomozecchini

19. The “Recently view products” feature
A practical examples are the "Recently viewed
products" boxes.
These boxes show the user's recent browsing
history, with links to various products, and can be
found on many websites.
@giacomozecchini

20. Ikea.com
@giacomozecchini

21. Asos.com
@giacomozecchini

22. Adidas.com
@giacomozecchini

23. Visited pages are saved in memory
For all three of the previous examples, the
"Recently viewed products" box is implemented
by saving the pages visited by the user in the
browser memory.
@giacomozecchini

24. Saved data may affect rendering
For those web crawlers that render web pages
without isolating the rendering sessions, the data
saved in the browser's memory may affect the
rendering of other web pages of the same
website.
@giacomozecchini

25. @giacomozecchini

26. @giacomozecchini

27. @giacomozecchini

28. @giacomozecchini

29. @giacomozecchini

30. @giacomozecchini

31. @giacomozecchini

32. Tools with session isolation behave differently
The result is different if we look at how Search
Engines or Web Crawlers that implement correct
session isolation are rendering pages.
@giacomozecchini

33. @giacomozecchini

34. @giacomozecchini

35. @giacomozecchini

36. @giacomozecchini

37. @giacomozecchini

38. @giacomozecchini

39. @giacomozecchini

40. Additional content and “ghost links”
These different ways of rendering pages will
produce additional content and a considerable
percentage of “ghost links”, only visible by web
crawlers affected by session isolation issues.
@giacomozecchini

41. @giacomozecchini
Without session isolation
With session isolation

42. Crawling/rendering order matters
Depending on the crawling/rendering order, a web
crawling tool with session isolation issues may
create arbitrary HTML content that changes every
time.
@giacomozecchini

43. @giacomozecchini
Starting from PAGE 1
Starting from PAGE 3

44. Three main implications
● Lack of data integrity
● The rendered pages are not an accurate
representation of what search engines will
render and use
● Developers may waste time (and money)
investigating issues which are not present
@giacomozecchini

45. Analyses are based on wrong data, for example:
● Content Analysis with additional content
● Internal linking analysis with X% arbitrary links
* those additional content and links are not visible to Google & Co
Effects on SEOs’ day-to-day
@giacomozecchini

46. These wrong analyses often translate into:
● Waste of time & money
● Wrong choices
Effects on SEOs’ day-to-day
@giacomozecchini

47. Session isolation isn’t limited to web crawlers
All systems that use browser-based
functionalities might be affected such as dynamic
rendering services, web performance analysis
tools, and CI/CD pipeline tests.
@giacomozecchini

48. If it’s an option, it should be clear
There are some cases where you need to keep
data for specific tests, but that option should be
really clear and intended, not a side effect of a
hidden problem.
@giacomozecchini

49. Solving Session Isolation

50. Partial or incorrect solutions
There are many partial or incorrect ways of
tackling session isolation for web crawling
purposes, let’s have a look at some of them.
@giacomozecchini

51. Partial or incorrect solution #1
Clearing cookies manually after the rendering of a
page. The problem here is that Cookies are not
the only Web API that can store data.
@giacomozecchini

52. Partial or incorrect solution #2
Opening and closing the browser for each page
you want to render, manually deleting the folders
where the browser stores data. This option is not
efficient at all.
@giacomozecchini

53. Partial or incorrect solution #3
Using the incognito profile hides some possible
pitfalls as well. Within an incognito profile the
rendered pages might share storage and
cross-tab communication is possible. This option
would solve our problem only if, again, we don’t
render pages in parallel and we start/stop the
browser for each page.
@giacomozecchini

54. The optimal solution
Introduced at BlinkOn 6, Browser Context is an
efficient way to have correct session isolation.
Every Browser Context session runs in a separate
renderer process, isolating the storage (cookies,
cache, local storage, etc.) and preventing
cross-tab communication.
@giacomozecchini

55. @giacomozecchini

56. How to use Browser Context effectively
Rendering a single page per Browser Context,
closing it at the end of the rendering, and then
opening a new Browser Context for the next page
will guarantee isolated rendering sessions
without the need to restart the browser every
time.
@giacomozecchini

57. Data integrity > Performance
Using this solution will have a minimal effect on
the web crawlers' performance. In most
real-world cases, the majority of web crawling
tools users would not compromise data integrity
caused by session isolation for an overall
performance difference of a few seconds.
@giacomozecchini

58. Documentation and example
Additional documentation and examples on the use of
Browser Context can be found here:
● https://chromedevtools.github.io/devtools-protocol/tot/
Target/#method-createBrowserContext
● https://pptr.dev/next/api/puppeteer.browser.createinco
gnitobrowsercontext
● https://playwright.dev/docs/api/class-browsercontext
@giacomozecchini

59. Tests

60. @giacomozecchini

61. Methodology
We set up a testing environment with 1,000 pages
that try to communicate with each other using the
storage and cross-tab communication.
@giacomozecchini

62. Avoiding false negatives
Rendering 1,000 pages will increase the chances
of having two or more pages rendered at the
same time in parallel or by the same browser,
using fewer pages may cause false negatives if
the tested web rendering system uses a high
number of machines in parallel.
@giacomozecchini

63. Storage isolation tests
Storage isolation tests are focused on Web APIs
that save or access data from the browser's
memory. The goal of each test is to find race
conditions in accessing data saved from previous
or parallel page renderings.
@giacomozecchini

64. Test #1 - Cookies
Cookies don’t need presentation. The Cookie interface lets you read
and write small pieces of information in the browser storage.
Test explanation: When the rendering starts the page creates and
saves a Cookie, then reads if there are cookies saved from other pages.
Fail criterion: if there are cookies other than the ones created for the
rendered page, the test fails.
@giacomozecchini

65. Test #2 - IndexedDB
IndexedDB is a transactional database system that lets you store and
retrieve objects from Browser memory.
Test explanation: When the rendering starts the page, it creates or
connects to an IndexedDB database. Then, it creates and saves a
record in the database to eventually start reading if there are records
saved from other pages.
Fail criterion: If there are records other than the ones created for the
rendered page, the test fails.
@giacomozecchini

66. Test #3 - LocalStorage
LocalStorage is a mechanism that uses the Web Storage API by which
browsers can store key/value pairs. Data persists when the browser is
closed and reopened.
Test explanation: When the rendering starts, the page creates or saves
a data item in the Local Storage, and then it reads if there are data
items saved from other pages.
Fail criterion: If there are data items other than the ones created for the
rendered page, the test fails.
@giacomozecchini

67. Test #4 - SessionStorage
SessionStorage is a mechanism that uses the Web Storage API by
which browsers can store key/value pairs. Data lasts as long as the tab
or the browser is open and survives over page reloads and restores.
Test explanation: When the rendering service starts the page, creates,
or saves a data item in the Session Storage, and then it reads if there
are data items saved from other pages.
Fail criterion: If there are data items other than the ones created for the
rendered page, the test fails.
@giacomozecchini

68. Cross-tab communication tests
Cross-tab communication tests are focused on
Web APIs that send or receive data. The goal of
each test is to find if during rendering a page can
receive messages from other pages rendered in
parallel.
@giacomozecchini

69. Test #5 - Broadcast Channel
The Broadcast Channel API allows communication between windows,
tabs, frames, iframes, and workers of the same origin.
Test explanation: When the rendering starts the page connects to the
channel and then starts sending its page title as a message to the
channel. If there are other pages connected that are sending messages
through the channel the page gets and saves them.
Fail criterion: If the rendered page gets even a single message from the
Broadcast Channel sent by other pages, the test fails.
@giacomozecchini

70. Test #6 - Shared Worker
The Shared Worker is a Worker that allows communication between
windows, tabs, frames, iframes, and workers on the same origin.
Test explanation: When the rendering starts the page connects to the
Shared Worker, then it starts sending messages to the Worker and
eventually starts listening for messages from other pages sent through
the worker.
Fail criterion: If the rendered page gets even a single message from the
Shared Worker sent by other pages, the test fails.
@giacomozecchini

71. 71% of web crawlers failed at least one test
@giacomozecchini
Test Results
Cookie 29% of web crawlers failed this test
IndexedDB 64% of web crawlers failed this test
LocalStorage 71% of web crawlers failed this test
SessionStorage 21% of web crawlers failed this test
Broadcast Channel 14% of web crawlers failed this test
Shared Worker 14% of web crawlers failed this test

72. Source Code on GitHub
Replicate the testing
environment using the
following code.
https://github.com/merj/test-
crawl-session-isolation
@giacomozecchini

73. Conclusion

74. Cause of storage isolation problems
It’s complex to predict what’s causing the storage
isolation issue. The implementation may
drastically vary and we can only speculating on
the cause.
@giacomozecchini

75. Cause of cross-tab communication problems
A possible cause for failing the cross-tab
communication tests (Broadcast Channel and
Shared Worker) is having the same browser used
to render pages in parallel using multiple
windows and/or tabs.
@giacomozecchini

76. Don’t clean it manually!
Web crawlers might pass all tests included in this
research by manually cleaning every single
storage at the end of every page rendering
session, but this approach is not a secure and
viable solution to guarantee data integrity.
@giacomozecchini

77. Workarounds are not future proof solutions
Web APIs and browser interfaces included in the
research aren't the only ones that might have
access to browser memory/cache and trying to
keep up with the development of all new
standards and web features is a complex and
time-consuming process.
@giacomozecchini

78. Our goal is to improve web crawling!
Not all web crawlers have been able to fix the
session isolation issues yet while they investigate
further. The Docker crawling test framework was
able to support those who have fixed the session
isolation and might be included in their future
release checks. Some web crawlers included us
through the entire remediation process.
@giacomozecchini

79. @giacomozecchini
Web Crawler Status
Ahrefs Fixed - 15 Nov 2022
Botify Passed all tests
ContentKing Fixed - 27 Oct 2022
FandangoSEO Looking into this
JetOctopus Looking into this
Lumar (formerly Deepcrawl) Passed all tests
Netpeak Spider Looking into this
OnCrawl Passed all tests
Ryte Fixed - 10 Oct 2022
Screaming Frog Fixed - 17 Aug 2022
SEO PowerSuite WebSite Auditor Looking into this
SEOClarity Looking into this
Sistrix Passed all tests
Sitebulb Looking into this
Last update: 15 Nov 2022

80. Final thoughts
● Rendering is hard, we hope that in the future
there will be an industry standard
● Make sure you validate your data
@giacomozecchini

81. Blog
We published the research on
our blog:
https://merj.com/blog/validat
ing-session-isolation-for-web-
crawling-to-provide-data-integ
rity
@giacomozecchini

82. Thank you for your time!
@GiacomoZecchini on Twitter, Slideshare & Speakerdeck
We work with enterprise clients to support them with SEO Innovation,
Research, & Development. Want to work with us?
rfp@merj.com +44 (0) 203 322 2660
7 Pancras Square
London, N1C 4AG