Organizations continue to adopt Solr because of its ability to scale to meet even the most demanding workflows. Recently, LucidWorks has been leading the effort to identify, measure, and expand the limits of Solr. As part of this effort, we've learned a few things along the way that should prove useful for any organization wanting to scale Solr. Attendees will come away with a better understanding of how sharding and replication impact performance. Also, no benchmark is useful without being repeatable; Tim will also cover how to perform similar tests using the Solr-Scale-Toolkit in Amazon EC2.
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Benchmarking Solr Performance at Scale
2. About Me
• Lucene/Solr committer. Work for Lucidworks; focus on hardening
SolrCloud, devops, big data architecture / deployments
• Operated smallish cluster in AWS for Dachis Group (1.5 years ago, 18
shards ~900M docs)
• Solr Scale Toolkit: Fabric/boto framework for deploying and managing
clusters in EC2
• Co-author of Solr In Action with Trey Grainger
3. Agenda
1. Quick review of the SolrCloud architecture
2. Indexing & Query performance tests
3. Solr Scale Toolkit (quick overview)
4. Q & A
4. Solr in the wild …
https://twitter.com/bretthoerner/status/476830302430437376
5. SolrCloud distilled
Subset of optional features in Solr to enable and
simplify horizontal scaling a search index using
sharding and replication.
Goals
performance, scalability, high-availability,
simplicity, elasticity, and
community-driven!
6. Collection == distributed index
A collection is a distributed index defined by:
• named configuration stored in ZooKeeper
• number of shards: documents are distributed across N partitions of the index
• document routing strategy: how documents get assigned to shards
• replication factor: how many copies of each document in the collection
Collections API:
curl "http://localhost:8983/solr/admin/collections?
action=CREATE&name=logstash4solr&replicationFactor=2&
numShards=2&collection.configName=logs"
8. ZooKeeper
• Is a very good thing ... clusters are a zoo!
• Centralized configuration management
• Cluster state management
• Leader election (shard leader and overseer)
• Overseer distributed work queue
• Live Nodes
• Ephemeral znodes used to signal a server is gone
• Needs at least 3 nodes for quorum in production
9. ZooKeeper: State Management
• Keep track of live nodes /live_nodes znode
• ephemeral nodes
• ZooKeeper client timeout
• Collection metadata and replica state in /clusterstate.json
• Every Solr node has watchers for /live_nodes and /clusterstate.json
• Leader election
• ZooKeeper sequence number on ephemeral znodes
10. Scalability Highlights
• No split-brain problems (b/c of ZooKeeper)
• All nodes in cluster perform indexing and execute queries; no master node
• Distributed indexing: No SPoF, high throughput via direct updates to
leaders, automated failover to new leader
• Distributed queries: Add replicas to scale-out qps; parallelize complex query
computations; fault-tolerance
• Indexing / queries continue so long as there is 1 healthy replica per shard
11. Cluster sizing
How many servers do I need to index X docs?
... shards ... ?
... replicas ... ?
I need N queries per second over M docs, how many
servers do I need?
It depends!
12. Testing Methodology
• Transparent repeatable results
• Ideally hoping for something owned by the community
• Synthetic docs ~ 1K each on disk, mix of field types
• Data set created using code borrowed from PigMix
• English text fields generated using a Zipfian distribution
• Java 1.7u67, Amazon Linux, r3.2xlarge nodes
• enhanced networking enabled, placement group, same AZ
• Stock Solr (cloud) 4.10
• Using custom GC tuning parameters and auto-commit settings
• Use Elastic MapReduce to generate indexing load
• As many nodes as I need to drive Solr!
17. Indexing Performance Lessons
• Solr has no built-in throttling support – will accept work until it falls over; need to build this into
your indexing application logic
• Oversharding helps parallelize indexing work and gives you an easy way to add more
hardware to your cluster
• GC tuning is critical (more below)
• Auto-hard commit to keep transaction logs manageable
• Auto soft-commit to see docs as they are indexed
• Replication is expensive! (more work needed here)
18. GC Tuning
• Stop-the-world GC pauses can lead to ZooKeeper session expiration (which is bad)
• More JVMs with smaller heap sizes are better! (12-16GB max per JVM ~ less if you can)
• MMapDirectory relies on sufficient memory available to the OS cache (off-heap)
• GC activity during Solr indexing is stable and generally doesn’t cause any stop-the-world
collections … queries are a different story
• Enable verbose GC logging (even in prod) so you can troubleshoot issues:
-verbose:gc –Xloggc:gc.log -XX:+PrintHeapAtGC -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps
-XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCApplicationConcurrentTime
19. GC Flags I use with Solr
-Xss256k
-XX:+UseConcMarkSweepGC -XX:+UseParNewGC
-XX:MaxTenuringThreshold=8 -XX:NewRatio=3
-XX:CMSInitiatingOccupancyFraction=40
-XX:ConcGCThreads=4 -XX:ParallelGCThreads=4
-XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90
-XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=12m
-XX:CMSFullGCsBeforeCompaction=1
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSTriggerPermRatio=80
-XX:CMSMaxAbortablePrecleanTime=6000
-XX:+CMSParallelRemarkEnabled
-XX:+ParallelRefProcEnabled
-XX:+UseLargePages -XX:+AggressiveOpts
21. Query Performance
• Still a work in progress!
• Sustained QPS & Execution time of 99th Percentile (coda hale metrics is good for this)
• Stable: ~5,000 QPS / 99th at 300ms while indexing ~10,000 docs / sec
• Using the TermsComponent to build queries based on the terms in each field.
• Harder to accurately simulate user queries over synthetic data
• Need mix of faceting, paging, sorting, grouping, boolean clauses, range queries, boosting, filters (some
cached, some not), etc ...
• Does the randomness in your test queries model (expected) user behavior?
• Start with one server (1 shard) to determine baseline query performance.
• Look for inefficiencies in your schema and other config settings
22. Query Performance, cont.
• Higher risk of full GC pauses (facets, filters, sorting)
• Use optimized data structures (DocValues) for facet / sort fields, Trie-based numeric fields for
range queries, facet.method=enum for low cardinality fields
• Check sizing of caches, esp. filterCache in solrconfig.xml
• Add more replicas; load-balance; Solr can set HTTP headers to work with caching proxies like
Squid
• -Dhttp.maxConnections=## (default = 5, increase to accommodate more threads sending
queries)
• Avoid increasing ZooKeeper client timeout ~ 15000 (15 seconds) is about right
• Don’t just keep throwing more memory at Java! –Xmx128G
23. Call me maybe - Jepsen
https://github.com/aphyr/jepsen
• Solr tests being developed by Lucene/Solr committer Shalin
Mangar (@shalinmanger)
• Prototype in place:
• No ack’d writes were lost!
• No un-ack’d writes succeeded
See: https://github.com/LucidWorks/jepsen/tree/solr-jepsen
24. Solr Scale Toolkit
• Open source: https://github.com/LucidWorks/solr-scale-tk
• Fabric (Python) toolset for deploying and managing SolrCloud clusters in the cloud
• Code to support benchmark tests (Pig script for data generation / indexing, JMeter samplers)
• EC2 for now, more cloud providers coming soon via Apache libcloud
• Contributors welcome!
• More info: http://searchhub.org/2014/06/03/introducing-the-solr-scale-toolkit/
25. Provisioning cluster nodes
fab new_ec2_instances:test1,n=3,instance_type=m3.xlarge
• Custom built AMI (one for PV instances and one for HVM instances) –
Amazon Linux
• Block device mapping
• dedicated disk per Solr node
• Launch and then poll status until they are live
• verify SSH connectivity
• Tag each instance with a cluster ID and username
26. Deploy ZooKeeper ensemble
fab new_zk_ensemble:zk1,n=3
• Two options:
• provision 1 to N nodes when you launch Solr cluster
• use existing named ensemble
• Fabric command simply creates the myid files and zoo.cfg file for the
ensemble
• and some cron scripts for managing snapshots
• Basic health checking of ZooKeeper status:
echo srvr | nc localhost 2181
27. Deploy SolrCloud cluster
fab new_solrcloud:test1,zk=zk1,nodesPerHost=2
• Uses bin/solr in Solr 4.10 to control Solr nodes
• Set system props: jetty.port, host, zkHost, JVM opts
• One or more Solr nodes per machine
• JVM mem opts dependent on instance type and # of Solr nodes
per instance
• Optionally configure log4j.properties to append messages to
Rabbitmq for SiLK integration
28. Automate day-to-day cluster management tasks
• Deploy a configuration directory to ZooKeeper
• Create a new collection
• Attach a local JConsole/VisualVM to a remote JVM
• Rolling restart (with Overseer awareness)
• Build Solr locally and patch remote
• Use a relay server to scp the JARs to Amazon network once and then scp them to other nodes
from within the network
• Put/get files
• Grep over all log files (across the cluster)
29. Wrap-up and Q & A
• LucidWorks: http://www.lucidworks.com -- We’re hiring!
• Solr Scale Toolkit: https://github.com/LucidWorks/solr-scale-tk
• SiLK: http://www.lucidworks.com/lucidworks-silk/
• Solr In Action: http://www.manning.com/grainger/
• Connect: @thelabdude / tim.potter@lucidworks.com
Editor's Notes
Brett is at Spredfast (ATX), 12-hr sharding scheme (180 shards)
ZooKeeper: Distributed coordination service that provides centralized configuration, cluster state management, and leader election
Node: JVM process bound to a specific port on a machine; hosts the Solr web application
Collection: Search index distributed across multiple nodes; each collection has a name, shard count, and replication factor
Replication Factor: Number of copies of a document in a collection
Shard: Logical slice of a collection; each shard has a name, hash range, leader, and replication factor. Documents are assigned to one and only one shard per collection using a hash-based document routing strategy.
Replica: Solr index that hosts a copy of a shard in a collection; behind the scenes, each replica is implemented as a Solr core
Leader: Replica in a shard that assumes special duties needed to support distributed indexing in Solr; each shard has one and only one leader at any time and leaders are elected using ZooKeeper
You’re not going to tune your way out of every query problem!