Deployment and Continous Integration of a Zope/Plone application
1. Deployment & Continuous integration
of a Plone/Zope application
Julien Pivotto
PyconFR’13
October 26, 2013
;
2. whoami
sysadmin @ inuits
• open-source defender for 7+ years
• devops believer
• @roidelapluie on twitter/github
•
Julien Pivotto
Deployment and CI of a Plone/Zope application
;
3. •
•
•
Plone is an Open-Source CMS
Written in python
Vibrant community
Julien Pivotto
Deployment and CI of a Plone/Zope application
;
4. •
•
•
Open-Source Web application server
Written in Python
Used by Plone
Julien Pivotto
Deployment and CI of a Plone/Zope application
;
7. The old days
•
•
•
•
Manual work
Failover HA
Supervisord
No overview
Julien Pivotto
Deployment and CI of a Plone/Zope application
;
8. BUT
Run buildout on each server/environment
• Compilation error in the middle
• Pypi is down
• Which commit is faulty?
•
Julien Pivotto
Deployment and CI of a Plone/Zope application
;
9. The goal
•
•
•
•
•
Less failure during builds
More testing
Puppetize all the things
Reproduce all the builds
More env: dev, test, demo, prod
Julien Pivotto
Deployment and CI of a Plone/Zope application
;
10. Infrastructure changes
• DRBD for PostgreSQL ⇒ PostgreSQL 9 native replication
• DRBD for files ⇒ GlusterFS for data, RPM for code
• Supervisord to launch instances ⇒ Puppet to start services
• Pound as reverse proxy ⇒ Haproxy
• Internal mirror of pypi
Julien Pivotto
Deployment and CI of a Plone/Zope application
;
11. Puppetization
•
•
•
•
From nothing to everything
Plone, but also apache, haproxy
. . . and anything else
Mcollective to orchestrate
Julien Pivotto
Deployment and CI of a Plone/Zope application
;
13. It takes time
•
•
•
•
Cultural shift
Legacy stuff
A lot of surprises
Compromises between Dev and Ops
Julien Pivotto
Deployment and CI of a Plone/Zope application
;
14. Jenkins
•
•
•
•
An open-source CI server
More than 600 plugins
Flexible
Jobs & pipelines
Julien Pivotto
Deployment and CI of a Plone/Zope application
;
17. Pipeline
•
•
•
Build is only part of the process
Need to package, deploy, . . .
Running tests
Julien Pivotto
Deployment and CI of a Plone/Zope application
;
18. Tests
•
•
•
•
•
•
Running tests on each module
Feedback: graphes, mails
Code coverage
Violations
Unit tests
zc.recipe.testrunner
Julien Pivotto
Deployment and CI of a Plone/Zope application
;
19. Translation in Jobs
virtualenv --no-site-packages .
bin/python bootstrap.py
bin/buildout -c jenkins.cfg buildout:eggs-directory=
/var/lib/jenkins/buildout-shared-eggs
bin/test-all
Julien Pivotto
Deployment and CI of a Plone/Zope application
;
21. Buildout
•
•
•
•
Running buildout in a Spec file
Creating a RPM
Using virtualenv
Starting from scratch each time
Julien Pivotto
Deployment and CI of a Plone/Zope application
;
22. Ops like RPM
•
•
•
•
•
•
Easy to deploy anywhere
Consistency and dependencies
Find where a file comes from
Orchestrate with puppet/mcollective
Version number
Conf does not belong to package
Julien Pivotto
Deployment and CI of a Plone/Zope application
;
24. Deploying Zope
•
•
•
•
•
Building RPM
Adding them to a yum repo (with pulp)
Cleaning yum cache (with mcollective)
Running puppet (with mcollective)
dev -> staging -> demo and prod
Julien Pivotto
Deployment and CI of a Plone/Zope application
;
26. Role of Puppet
•
•
•
•
Puppet creates the configuration
Update the package
Restarts the instances
Puppet (hiera) creates password files
Julien Pivotto
Deployment and CI of a Plone/Zope application
;
27. Database upgrades
•
•
•
WIP, no solution chosen yet
Postscript in RPM?
Commands trigerred by puppet?
Julien Pivotto
Deployment and CI of a Plone/Zope application
;
28. Logs
Zope/Plone: logs files/instances
• 10 instances * 2 servers
⇒ 20 log files
• Plus apache logs
• Plus system logs
•
Julien Pivotto
Deployment and CI of a Plone/Zope application
;
29. The deprecated way
•
•
•
tail -f
multitail
clusterssh
Julien Pivotto
Deployment and CI of a Plone/Zope application
;
30. Logstash
•
•
•
•
•
An open source tool
Manage your logs
Input from anywhere
Filter
Output to anywhere
Julien Pivotto
Deployment and CI of a Plone/Zope application
;
32. Plone and Logstash
•
•
•
•
Plone output to syslog
Mangle to add good fields
Logstash output to ElasticSearch
Kibana3 creates dashboards
Julien Pivotto
Deployment and CI of a Plone/Zope application
;
33. zope.conf
<syslog>
address 172.16.28.55:5544
facility local3
format %(asctime)s ZopApp-Server zope[%(process)s]:
instance1 [%(levelname)s] %(name)s | %(message)s
dateformat %b %d %H:%M:%S
level info
</syslog>
Julien Pivotto
Deployment and CI of a Plone/Zope application
;
34. filter {
grok {
type
=> "syslog"
pattern => ["(?m)<%= "<%" %>{POSINT:syslog_pri}>
%{SYSLOGTIMESTAMP:syslog_timestamp}
%{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}
(?:[%{POSINT:syslog_pid}])?:
%{GREEDYDATA:syslog_message}"]
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{@source_host}" ]
add_tag => "syslog-%{syslog_program}"
}
}
Julien Pivotto
Deployment and CI of a Plone/Zope application
;
36. Kibana3
•
•
•
•
•
Plays with ElasticSearch and Logstash
Easily build dashboards
Share the metric and logs with anyone
Filter by time, instance, http code. . .
View all the logs at the same place
Julien Pivotto
Deployment and CI of a Plone/Zope application
;
38. Conclusions
This is a sysadmin point of view
• Still a lot of work to be done
• From a "black box"
. . . To a fully automated environment
• Not a unique solution
• You need to get it working for you
. . . in your way
•
Julien Pivotto
Deployment and CI of a Plone/Zope application
;
39. This work is being realized with
Affinitic SPRL, Plone developers in Belgium
http://affinitic.be
Julien Pivotto
Deployment and CI of a Plone/Zope application
;