Presented at Web 2.0 Expo, Oct. 13 2011

1. Metrics-Driven Engineering
Mike Brittain @ mikebrittain
Director of engineering, Infrastructure
October 13, 2011

2. Tools and Process at Etsy

3. How many new visits?
How many listings created?
How many registrations?
How do people use Etsy?
How many convos sent?
How many purchases?
How many new shops?

4. Search indexing?
How fast are pages generating?
Async tasks currently in queue?
What is the application doing?
Developer API auth and rate limiting?
Images resized and stored?
Error and warning rates?

5. Replication slave lag?
Memcache hits/misses?
Available connections?
Are the servers in good shape ?
Database queries per second?
Total outgoing bandwidth?
CPU, Memory, I/O?

6. Business Metrics

7. Application Metrics

8. System Metrics

9. Visibility EVERYWHERE

10. Constant Change

12. $314 Million GMS 2010
$180 Million GMS 2009
$87 Million GMS 2008
$26 Million GMS 2007
credit: pentarux (flickr)

13. 25 Million Unique Visitors
1 Billion page views per month
credit: pentarux (flickr)

14. Engineering team grew 500%
over 18 months
credit: martin_heigan (flickr)

15. Less talk, more do.

16. Always Be Shipping
credit: ibailemon (flickr)

17. Always Be Shipping
(even if it’s your first day)
credit: ibailemon (flickr)

19. 90+ Engineers
40+ Deploys / day
credit: misswired (flickr)

20. credit: digidave (flickr)

21. Code Reviews

22. Automated Tests

23. $cfg = array(
'checkout' => array('enabled' => 'on'),
'homepage' => array('enabled' => 'on'),
'profiles' => array('enabled' => 'on'),
'new_search' => array('enabled' => 'off'),
);
Config Flags
Enable and disable features quickly

24. $cfg = array(
'checkout' => array('enabled' => 'on'),
'homepage' => array('enabled' => 'on'),
'profiles' => array('enabled' => 'on'),
'new_search' => array('enabled' => 'off'),
);
Config Flags
Enable and disable features quickly
Plus “admin-only,” percentage ramp-up, A/B testing,
whitelists, blacklists, etc...

25. Failure is not an option

26. inevitable!
Failure is not an option

27. inevitable!
Failure is not an option
a learning opportunity!

28. inevitable!
Failure is not an option
a learning opportunity!
DETECTABLE!

29. Access

33. Detect problems quickly

34. CONFIDENCE

36. A: Well, the Ops team manages the network, racks
the servers, installed the monitoring tools, wears
the pagers, blah, blah, blah...

37. Engineers build the application

38. Logging
Graphing
OPS ENG
Trending
Alerting

39. “Engineers are too busy writing
features to build metrics.”

40. Metrics are part of every feature
...and so are config flags

41. Dead Simple

42. Simple, open source tools

43. Cacti (network, SNMP)
Ganglia (machines)
Graphite (application)
Splunk (log analysis, nightly reports)
Nagios (alerting)
Logging
Logster
StatsD

44. Ganglia

45. Ganglia
Cluster-oriented
Huge community contributed recipes
Custom metrics (gmetad)

46. Graphite

47. Graphite
Single-instance
Create new metrics on-the-fly
Customize via URLs and display functions

48. Logging

49. It’s 2:48 PM.
Do you know where your
logs are?

50. Logger::log_error("User login failed.
Reason: $msg for $username", “login”);

51. Logger::log_error("User login failed.
Reason: $msg for $username", “login”);

52. web0054 [Fri Mar 04 16:27:48 2011]
[error] [login] [mk04gw1p71] User login
failed. Reason: wrong password for ...

53. web0054 [Fri Mar 04 16:27:48 2011]
[error] [login] [mk04gw1p71] User login
failed. Reason: wrong password for ...

54. web0054 [Fri Mar 04 16:27:48 2011]
[error] [login] [mk04gw1p71] User login
failed. Reason: wrong password for ...

55. web0054 [Fri Mar 04 16:27:48 2011]
[error] [login] [mk04gw1p71] User login
failed. Reason: wrong password for ...

56. web0054 [Fri Mar 04 16:27:48 2011]
[error] [login] [mk04gw1p71] User login
failed. Reason: wrong password for ...

57. web0054 [Fri Mar 04 16:27:48 2011]
[error] [login] [mk04gw1p71] User login
failed. Reason: wrong password for ...

58. LogFormat "%h %l %u %t "%r" %>s %b"
common

59. LogFormat %{True-Client-IP}i %l %t "%r
" %>s %b "%{Referer}i"
"%{User-Agent}i"
%{etsy_shop_id}n %{etsy_uaid}n %V
%{etsy_ab_selections}n
%{etsy_request_uuid}n
%{etsy_api_consumer_key}n
%{etsy_api_method_name}n
%{php_memory_usage_bytes}n
%{php_time_microsec}n %D" combined

60. apache_note()

61. LogFormat %{True-Client-IP}i %l %t "%r
" %>s %b "%{Referer}i"
"%{User-Agent}i"
%{etsy_shop_id}n %{etsy_uaid}n %V
%{etsy_ab_selections}n
%{etsy_request_uuid}n
%{etsy_api_consumer_key}n
%{etsy_api_method_name}n
%{php_memory_usage_bytes}n
%{php_time_microsec}n %D" combined

62. LogFormat %{True-Client-IP}i %l %t "%r
" %>s %b "%{Referer}i"
"%{User-Agent}i"
%{etsy_shop_id}n %{etsy_uaid}n %V
%{etsy_ab_selections}n
%{etsy_request_uuid}n
%{etsy_api_consumer_key}n
%{etsy_api_method_name}n
%{php_memory_usage_bytes}n
%{php_time_microsec}n %D" combined

63. LogFormat %{True-Client-IP}i %l %t "%r
" %>s %b "%{Referer}i"
"%{User-Agent}i"
%{etsy_shop_id}n %{etsy_uaid}n %V
%{etsy_ab_selections}n
%{etsy_request_uuid}n
%{etsy_api_consumer_key}n
%{etsy_api_method_name}n
%{php_memory_usage_bytes}n
%{php_time_microsec}n %D" combined

64. grep "/listing/" access.log |
awk '{sum=sum+$(NF-2)} END {print sum/NR}'

65. web0001 [04:28:54 2011] [error] [client 10.101.x.x] Help me, Rhonda.
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Heeeeeeellllllllllllllppppp!
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!
web0001 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!
web0201 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!
web0034 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web1101 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0201 [04:28:54 2011] [error] [client 10.101.x.x] You've been eaten by a grue.
web0055 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!!
web0002 [04:28:54 2011] [warning] [client 10.101.x.x] Sky is falling.
web0089 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling.
web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!
web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!
web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0034 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0087 [04:28:54 2011] [fatal] [client 10.101.x.x] Sky is falling.
web0002 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!
web0201 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!
web0077 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!
web0355 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo
web0052 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0003 [04:28:54 2011] [error] [client 10.101.x.x] You've been eaten by a grue.
web0066 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!!

66. Logster
Fatals Errors Warnings

67. Logster
Run by cron
Keeps a cursor on your log file
Aggregate lines anyway you want
Output to Ganglia or Graphite
Simple parsers
github.com/etsy

68. web0054 [Fri Mar 04 16:27:48 2011]
[error] [login] [mk04gw1p71] User login
failed. Reason: wrong password for ...

69. ^.+ [.+] [(?P<log_level>.+)]

70. if (fields['log_level'] == “fatal”):
self.fatals += 1
elif (fields['log_level'] == “error”):
self.errors += 1
elif (fields['log_level'] == “warning”):
self.warnings += 1
...

71. MetricObject("fatals",
(self.fatals / self.duration), "per sec")
MetricObject("errors",
(self.errors / self.duration), "per sec")
MetricObject("warning",
(self.warnings / self.duration), "per sec")

72. Fatals Errors Warnings

73. StatsD

74. StatsD
Network daemon (node.js)
Accepts data over UDP
Flushes to Graphite every 10 sec
One-line of code
github.com/etsy

75. StatsD::increment("logins.success");

76. StatsD::increment("logins.success");
logins

77. StatsD::timing("gearman.time", $msec);

78. StatsD::timing("gearman.time", $msec);
90th pct
average
lower

79. Ad hoc
name value timestamp

80. echo "events.deploy.site 1 `date +%s`"
| nc graphite.etsycorp.com 2003

81. Vertical Line Technology!
target=drawAsInfinite(events.deploy.site)

83. We could stare at graphs all day...

84. http://graphite/render?
from=-1hours&width=600&height=200
&target=webs.errorLog.warning&rawData=1

85. http://graphite/render?
from=-1hours&width=600&height=200
&target=webs.errorLog.warning&rawData=1
webs.errorLog.warning,1318444930,1318448530,60|
5.0,1.0,3.0,1.0,0.0,9.0,0.0,1.0,3.0,2.0,1.0,6.0,2.0,6.0,3.0,6.0,4.0,4.0,2.0,
1.0,1.0,8.0,2.0,3.0,6.0,3.0,5.0,3.0,0.0,4.0,6.0,2.0,0.0,2.0,0.0,4.0,0.0,3.0,
1.0,3.0,4.0,2.0,10.0,3.0,0.0,6.0,0.0,4.0,2.0,5.0,18.0,1.0,1.0,2.0,1.0,8.0,5.
0,1.0,1.0,None

86. Holt-Winters Confidence Bands
upper
lower

87. Holt-Winters Aberration

88. Business metrics
+ Confidence bands
_____________
Alertable metrics

89. 40,000+ metrics at Etsy
Systems, Applications, Business

90. Dashboards

91. Dashboards

92. Kind of Hard :-/
<a href="http://graphite.etsycorp.com/render?from=-1hours&width=800&height=600&title=File+or
+Script+Not+Found&yMin=0&target=webs.errorLog.notExist&target=drawAsInfinite
%28deploys.config.production%29&target=drawAsInfinite%28deploys.web.production
%29&target=drawAsInfinite%28deploys.search.production%29&target=drawAsInfinite
%28deploys.imagestorage.other%29&colorList=%2300cc00,%230000ff,
%23ff0000,%23006633,%23cc6600">
<img src="http://graphite.etsycorp.com/render?
from=-1hours&width=280&height=220&title=File+or+Script+Not
+Found&hideLegend=1&yMin=0&target=webs.errorLog.notExist&target=drawAsInfinite
%28deploys.config.production%29&target=drawAsInfinite%28deploys.web.production
%29&target=drawAsInfinite%28deploys.search.production%29&target=drawAsInfinite
%28deploys.imagestorage.other%29&colorList=%2300cc00,%230000ff,
%23ff0000,%23006633,%23cc6600">
</a>

93. Super Easy!
$g = new Graphite($time);
$g->setTitle('File Not Found');
$g->addMetric('webs.errorLog.notExist', '#00cc00');
echo $g->getDashboardHTML(280, 220);

94. Metrics!

95. Metrics!
Metrics + Events

96. Metrics!
Metrics + Events
Metrics + Alerts

97. Metrics!
Metrics + Events
Metrics + Alerts
Metrics + Metrics

98. High-level, real-time visibility

99. Detect problems quickly

100. CONFIDENCE

101. Make them required features

102. Make them dead simple

103. Make them accessible

104. Make them!

105. Homework
codeascraft.etsy.com
github.com/etsy Get in touch
mike @ etsy . com
We’re always looking for people @ mikebrittain
who are interested in this kind of
stuff...
Thank You
etsy.com/careers