Java Collections Framework represents one of the key building blocks of any Java application. Although the standard JDK devoted a lot of attention to developing a coherent and easy to use collections framework one important aspect remains overlooked – collection element eviction. Collection memory footprint can not grow indefinitely because we would eventually run out of memory; we either have to remove elements from a collection or somehow periodically evict certain elements according to a chosen eviction algorithm. Since day one eviction has been a key feature of Infinispan, and in the latest 4.1 release thanks to event update batching, Infinispan has reduced the eviction overhead to such an extent that it hardly affects application performance. On top of that, Infinispan implements LIRS, a more precise eviction algorithm compared to the traditional LRU, making it the first open source project to implement this revolutionary algorithm in the data grid space. In this session, Galder and Vladimir will present to the details behind these changes, performance measurements and third-party use case testimonies.
2. Keeping Infinispan In Shape:
Highly-Precise, Scalable
Data Eviction
Galder Zamarreño & Vladimir Blagojevic
Senior Engineer, Red Hat
7th October 2010, JUDCon - Berlin
galder@jboss.org | twitter.com/galderz | zamarreno.com
Thursday, October 7, 2010
3. Who is Galder?
• R&D engineer (Red Hat Inc):
• Infinispan developer
• JBoss Cache developer
• Contributor and committer:
• JBoss AS, Hibernate, JGroups, JBoss Portal,...etc
• Blog: zamarreno.com
• Twitter: @galderz
galder@jboss.org | twitter.com/galderz | zamarreno.com
Thursday, October 7, 2010
5. Java Collections Framework
• Key building block of any Java app
• Introduced in Java 1.2
• Extended with concurrent collections in Java 1.5
• Collection element eviction ??
galder@jboss.org | twitter.com/galderz | zamarreno.com
Thursday, October 7, 2010
7. If using a collection as cache
• Forces clients to either:
• Remove elements proactively
• Or run a periodic cleanup process
• Which can be a PITA...
galder@jboss.org | twitter.com/galderz | zamarreno.com
Thursday, October 7, 2010
8. Eviction in JBoss Cache days
galder@jboss.org | twitter.com/galderz | zamarreno.com
Thursday, October 7, 2010
9. JBoss Cache eviction issues
• Under heavy load, eviction queues could fill up - bottleneck
• Possibly a side effect of MVCC’s non-blocking reads
• Separating data into regions could alleviate the issue
• But it’s really a hack!
galder@jboss.org | twitter.com/galderz | zamarreno.com
Thursday, October 7, 2010
10. Original Infinispan 4.0 eviction
• Tried a different approach:
• Avoid using queues to hold events
• Taking advantage of tree to map change:
• Attempt to maintain map ordered as per eviction rules
• Could we use ConcurrentSkipListMap ?
• No. O(1) desired for all map operations
galder@jboss.org | twitter.com/galderz | zamarreno.com
Thursday, October 7, 2010
11. Doubly Linked
ConcurrentHashMap
• Based on H.Sundell and P.Tsigas paper:
• Lock-Free Deques and Doubly Linked Lists (2008)
• With each cache access or update update links
• Eviction just a matter of walking the linked list one way
galder@jboss.org | twitter.com/galderz | zamarreno.com
Thursday, October 7, 2010
12. Issues under heavy load
• Algorithm always uses same node (the tail) to append stuff
• With high concurrency, CAS stress lead to loads of retrying
• So much retrying, we’re getting infinite loops
• Before 4.0.0.Final, reverted to a more conservative algorithm
galder@jboss.org | twitter.com/galderz | zamarreno.com
Thursday, October 7, 2010
14. LRU eviction algorithm issues
• Weak access locality
• One-time accessed keys not evicted timely
• In loops, soon to be accessed keys might get evicted first
• With distinct access frequencies, frequently accessed keys
can unfortunately get evicted
• LRU’s working set limited to cache size
galder@jboss.org | twitter.com/galderz | zamarreno.com
Thursday, October 7, 2010
15. Enter the room... LIRS
• Eviction algorithm that can cope with weak access locality
• Based on S.Jiang and X.Zhang’s 2002 paper:
• LIRS: An efficient low inter-reference recency set
replacement policy to improve buffer cache performance
• LIRS based around two concepts: IRR and Recency
galder@jboss.org | twitter.com/galderz | zamarreno.com
Thursday, October 7, 2010
16. IRR and Recency
• Inter-Reference Recency (IRR):
• Number of other unique keys accessed between two
consecutive accesses to same key
• Recency (R)
• Number of other unique keys accessed from last reference
until now
galder@jboss.org | twitter.com/galderz | zamarreno.com
Thursday, October 7, 2010
17. How does LIRS work?
• If key has a high IRR, it’s next IRR is likely to be high again
• Keys with highest IRR are considered for eviction
• Once IRR is out of date, we start relying on Recency
• LIRS = Low Inter-reference Recency Set
galder@jboss.org | twitter.com/galderz | zamarreno.com
Thursday, October 7, 2010
18. How is LIRS implemented?
• Low IRR (LIR) area
• Holds hot keys!
• High IRR (HIR) area
• Holds recently accessed keys
• Keys here might get promoted to LIR area
galder@jboss.org | twitter.com/galderz | zamarreno.com
Thursday, October 7, 2010
19. Cache Hit - LIR area
galder@jboss.org | twitter.com/galderz | zamarreno.com
Thursday, October 7, 2010
20. Cache Hit - LIR area
galder@jboss.org | twitter.com/galderz | zamarreno.com
Thursday, October 7, 2010
21. Hit - HIR area and in LIR Q ...
galder@jboss.org | twitter.com/galderz | zamarreno.com
Thursday, October 7, 2010
22. Hit - HIR area and in LIR Q
galder@jboss.org | twitter.com/galderz | zamarreno.com
Thursday, October 7, 2010
23. Hit - HIR area and not in LIR Q
galder@jboss.org | twitter.com/galderz | zamarreno.com
Thursday, October 7, 2010
24. Cache Miss
galder@jboss.org | twitter.com/galderz | zamarreno.com
Thursday, October 7, 2010
25. Cache Miss
galder@jboss.org | twitter.com/galderz | zamarreno.com
Thursday, October 7, 2010
26. Cache Miss
galder@jboss.org | twitter.com/galderz | zamarreno.com
Thursday, October 7, 2010
27. LIRS implementation hurdles
• LIRS requires a lot of key shifting around
• It can lead to high contention
• Unless you can implement it in scalable way, it’s useless
• Low contended way to implement a high precision eviction
algorithm? Is it possible?
galder@jboss.org | twitter.com/galderz | zamarreno.com
Thursday, October 7, 2010
28. Batching Eviction Updates
• Original idea: X.Ding, S.Jiang and X.Zhang’s 2009 paper:
• BP-Wrapper: A System Framework Making Any
Replacement Algorithms (Almost) Lock Contention Free
• Keeping cache access per thread in a queue
• If queue reaches a threshold:
• Acquire locks and execute eviction as per algorithm
• Batching updates significantly lowers lock contention
galder@jboss.org | twitter.com/galderz | zamarreno.com
Thursday, October 7, 2010
29. Batching Updates in Infinispan
• Decided against recording access per thread
• 100s threads could be hitting cache; some short lived
• Created BoundedConcurrentHashMap
• Based on Doug Lea's ConcurrentHashMap
• Records accesses in a lock-free queue in each segment
• When threshold passed, acquire lock for segment and evict
galder@jboss.org | twitter.com/galderz | zamarreno.com
Thursday, October 7, 2010
30. Precision and Performance
• Segment level eviction does not affect overall precision
• Community member run some performance tests:
• After swapping their own LRU cache with BoundedCHM:
• Berlin SPARQL Benchmark performance increased
55-60% for both cold and hot caches
galder@jboss.org | twitter.com/galderz | zamarreno.com
Thursday, October 7, 2010
31. Summary
• In JBoss Cache, eviction can become a bottleneck
• Infinispan 4.0 uses conservative eviction
• Infinispan 4.1 has more precise eviction algorithm (LIRS)
• Batching updates, present in 4.1, significantly lowers lock
contention
• Result = Highly-concurrent, highly-precise implicit eviction
galder@jboss.org | twitter.com/galderz | zamarreno.com
Thursday, October 7, 2010