Presentation for PloneConf2017 in Barcelona. Backend tools used to develop and deploy the Diazo theme engine, for front- and back-end developers. Toolchain including build, tests, continuous integration, and deployment to a high-availability AWS cloud cluster. AWS cloudformation creation of the cluster.

1. Second Skin:
Real-time Retheming a Legacy Web Application
with Diazo in the Cloud
Chris Shenton
CTO V! Studios
chris@v-studios.com
@shentonfreude

2. Talk Overview
● Tech talk for developer rather than designers
● The Challenge
● The Solution: Diazo
● Architecture Drill Down
● Toolchain
● Results
● A Warning

3. Servers, Software, Development (not theming)
● This talk focuses on Servers, Software, Development and Deployment
environments
○ Tools for front-end and full-stack developers
○ Building the AWS network and server infrastructure
○ Nginx configuration
○ XSLT module (patched for Diazo)
○ Deploying into AWS in a high-availability cluster of inexpensive servers
● We will not address building Diazo themes here
○ That’s already covered by training and talks at this conference, and docs online

4. The Challenge: site did not age well
● 15-year old ASP/VBScript
● We proposed a rewrite
○ Did a prototype in Pyramid
○ Customer didn’t have the time or budget
● Legacy back-end application was working fine

5. Client: “Can’t we just give it a new look and feel??”
● No: ASP is like PHP and JSP:
○ UI and code are intertwingled
○ Can’t separate front-end from logic

6. Wait, We’ve Done This Before!
● We used Diazo to create mobile.nasa.gov
● Gave a mobile friendly face to the
non-responsive origin content
● Without changing www.nasa.gov

7. “Can’t we just give it a new look and feel?” Yes, with Diazo!
● Modern theme
● Responsive
● Mobile-Friendly
● Faster
● Redundant front-end
● IPv6 accessible
● No changes to ASP
application

8. 1000 Foot Meter View: Overall Architecture, Data Flow
DNS
TTT Diazo:
rules, theme
TTT
Origin
tradetotravel.com
GET /search GET /search

9. 100 Meter View: Cloud Architecture
TTT Diazo
TTT
Origin
AWS
ELB
Auto-Scaling Group
EC2: t2.micro
Ubuntu
Nginx: cache, XSLT
Diazo rules, theme
EC2: t2.micro
Ubuntu
Nginx: cache, XSLT
Diazo rules, theme

10. 10 Meter View: Nginx Cache and XSLT
Ubuntu
TTT
Origin
AWS
ELB
Nginx
80, 443: Cache
server_name tttdiazo_cache
proxy_pass http://127.0.0.1:8888
8888: XSLT
server_name tttdiazo
root theme/
xslt_stylesheet etc/theme.xsl

11. Tools Make Building Easier, More Reliable
● AWS Cloudformation and Troposphere
● Make
● Buildout
● CircleCI
● AWS CodeDeploy

12. Troposphere, CloudFormation: Infrastructure as Code
● Troposphere: Python library to create CloudFormation templates
● CloudFormation: AWS mechanism to define cloud resources in JSON/YAML
● net-infra.py
○ Builds the network the application components live in
○ VPC; Security Groups for SSH, ELB; Gateway; Routing Table; Default Route; Subnets; IAM Roles
and Policies; DNS; CodeDeploy
● app-infra.py
○ Defines the application server cluster
○ Security Group; IAM Roles and Policies; auto-scaling LaunchConfig, Groups, policies, alarms,
actions; SNS notifications; ELB; DNS; S3; CodeDeploy role, deploymentgroup, IAM user policy

13. Makefile
Developer-friendly wrappers around buildout, docker, paster, testing
$ make help
Front-end developer targets: clean, build, run, test, test_browser
Fullstack developer targets: clean, fullstack, fullstack_run, fullstack_test,
fullstack_stop
Production targets: clean, prod, prod_run, prod_test
Docker targets: docker, docker_start, docker_curl, docker_stop
If you just say 'make' it will run the 'build' target.
"make test" should test paster, fullstack and docker runs, as they should all
listen on 5000.
You don't need to install or invoke the virtualenv, it's done for you.

14. Buildout: Builds Everything for Development, Production
● buildout.cfg
○ Default for buildout, extends buildout-base
● buildout-base.cfg
○ Defines everything, used by fullstack and prod configs
○ Installs libxml2, libxslt
○ Compiles diazo theme
○ Builds Nginx configuration
○ Compiles Nginx with patches to support XSLT for HTML
(thanks to Lawrence Rowe and David Beitey)
● buildout-fullstack.cfg
○ Builds all the parts defined in base needed for development
● buildout-prod.cfg
○ Extends fullstack to add Nginx log rotation

15. Nginx Config
● Created by buildout
○ For local development
○ For stage and prod deployment
● Main server on 80 and 443
○ Redirects DNS aliases
○ Creates a cache
○ Proxy_cache to XSLT engine
● XSLT engine on 8888
○ Enable xslt_html_parser for HTML
○ proxy_pass to origin server (ASP application)
○ Avoid theming /static-images, /fonts, /scripts, /styles, origin’s sitemap.xml

16. CircleCI: Continuous Integration and Deployment
● circle.yml
● Uses “make docker_build” to build on Circle’s servers
● Runs tox and integration tests
● Instructs Circle CI to deploy commits to servers via AWS CodeDeploy
○ only if tests pass!
○ branch ‘develop’ gets sent to Stage deployment group
○ branch ‘master’ gets sent to Prod deployment group

17. AWS CodeDeploy
● Deploy code to existing servers
● Separate Stage and Prod server groups
● appspec.yml
1. ApplicationStop.sh: kills nginx
2. BeforeInstall.sh: apt-get installs prerequisites, virtualenv
3. AfterInstall.sh: uses “make prod_build” to build diazo, nginx
4. ApplicationStart.sh: uses “make prod_run” to start nginx
5. ValidateService.sh: uses “make prod_test”

18. Results: Toolchain helps a lot
● CloudFormation, Troposphere
○ Repeatable, fast deployment of AWS cloud infrastructure
● Makefile
○ Greatly simplifies developer chores
○ Leveraged by Docker and CircleCI
● Docker
○ Really helpful for local full-stack development
○ Used for CircleCI builds
● CircleCI
○ Makes testing and deployment easy (with CodeDeploy)
● CodeDeploy
○ Reliable deployment to AWS for Stage and Prod

19. Results: Performance is Excellent (mostly)
● Running in Docker is surprisingly fast (compiled theme)
● Running in Production is very fast (compiled theme)
○ Sometimes faster than the origin server itself
○ Probably due to Nginx cache we’re using
○ Very little CPU load on server
○ Even on a tiny t2.micro EC2
● Compiled theme.xsl
○ 855 KB
● Running under Paster, for front-end devs, is disappointingly slow :-(
○ About 5 seconds to render a page
○ Likely due to 28MB of theme, HTML, CSS, JavaSCript, images
● Legacy back-end is still a single point of failure

20. Warning: Too Much Flexibility
● Diazo is intended to theme content living on the origin server
○ all content changes should be done on origin
● Diazo themes can also contain content
○ “Our admin is out, can’t we just insert a new paragraph?”
○ “Can we add a new page or two here, and here?”
○ We should have said “No.”
● Now the content is partly on the origin and partly in the Diazo theme
● This means we’re now in the content editing business
● And all our theme content updates require a deployment to Production
● Don’t do this, be firm: it’s OK to say “No.” if you explain why

21. Code in GitHub, Reach Out
We’ve cloned the repo minus the client’s theme so you can see how
all the pieces fit together. You should be able to run it under paster,
local full-stack build (only tested on Mac), or local Docker full-stack
build.
https://github.com/v-studios/PloneConf2017-second-skin-diazo
Have any questions? need some help? Reach out and let’s chat:
● Email: chris@v-studios.com
● Twitter/GitHub/LinkedIn: shentonfreude