Ian Watson - DevOps at Callcredit….a year nearer awesome
•Download as PPTX, PDF•
2 likes•900 views
Agile Tour Kaunas 2015 http://agileturas.lt/kaunas
Recommended
More Related Content
More from Agile Lietuva
More from Agile Lietuva (20)
Recently uploaded
Recently uploaded (20)
Ian Watson - DevOps at Callcredit….a year nearer awesome
- 1. DevOps at Callcredit …..a year nearer awesome
- 2. Ian Watson Head of DevOps at Callcredit Email : ian.watson@callcreditgroup.com Twitter : @purplemarauder I am….
- 5. Actual Deployment plan Check if access to the deployment packages is available. • R:Callcredit Development DMLLLBuildsRelease- v6.1.56.14856.zip Extract the relevant (Web or App) files zip within the release to c:program filescallcreditLL on each Web and App servers. Ensure the correct rollback version location is known. Deployment Steps Instruction Done (1,2&3) Done (4,5&6) Run script to take the first 3 pairs of Web and App servers out of the pool “C:scriptsscripts” on WIN**WB01 server….(run Phase 1 script). ENSURE YOU ARE NOT REMOVING ALL SERVERS FROM THE POOL On the app server: • Open a command prompt in the "Release" folder, e.g. "C:Program FilescallcreditLL v6.1.56.14856Release" • Enter the command: • Teardown_APP On the app server: • Open a command prompt in the "Release" folder, e.g. "C:Program FilescallcreditLL v6.1.56.14856Release" • Enter the command: • nant -D:"config=siteconfigid" -D:"encrypt.key=yourencryptionkey“ ptionkey = see app support passwordsafe) (where siteconfigid: • Live = “LV” • Site 2 QA = “qa” • Site 2 CT = “ct” On the web server, open INETMGR and delete the following: From IIS ? Callcredit Web Site ? services • Admin • Call • DL1 • DL2 • LL From IIS ? Callcredit Web Site • Call • DL • LL On the web server: • Open a command prompt in the "Release" folder, e.g. "C:Program FilescallcreditLL v6.1.56.14856Release" • Enter the command: • Teardown_WEB On the web server: • Open a command prompt in the "Release" folder, e.g. "C:Program FilesLL v6.1.56.14856Release" • Enter the command: • nant -D:"config=siteconfigid"-D:"encrypt.key=yourencryptionkey" (where yourencryptionkey= see app support passwordsafe) (where siteconfigid: • Live = “LV” • Site 2 QA = “qa” • Site 2 CT = “ct” On each of the App servers: Under component servicescomputersCOM+ApplicationsCAST Job controllerRolesPermittedUsers Add the user: ServerName_LL_CS_WebSvc e.g. OLSWINxxAP0xLL_CS_WebSvc the user should already be present under computer Managementusers On each of the App servers: Open DCOMCNFG -> servicescomputersCOM+ApplicationsCAST Job controller Right click on the component and: • Shut down • Disable • Enable DBA should run following script on ML_Lookup_Vws5 databse: UPDATE vw_IDType SET [Desc] = 'Passport' WHERE [ID] = 5 Rerun the pairing scripts to update COM+ configs (if COM+ has been updated) ENSURE YOU ARE RUNNING THE CORRECT “PHASE” SCRIPT TO MAINTAIN SEPERATION BETWEEN UPGRADED AND NOT UPGRADED VERSIONS OF THE RELEASE!! QA can then verify the web and app tier install Add the 3 pairs back into the pool and remove next 3 pairs (run phase 2 script). ENSURE YOU ARE RUNNING THE CORRECT “PHASE” SCRIPT Repeat above steps for each pair Once paired testing is complete, add all servers back into pool (run Phase 3 script) ENSURE YOU ARE RUNNING THE CORRECT “PHASE” SCRIPT QA will perform a final BIGIP test.
- 6. We needed to change
- 9. Universal Truths You can’t ‘do’ DevOps… …. in a windows Environment …. on an Enterprise scale …. in a database-centric company
- 10. Topologies
- 11. The Teams – System Build
- 12. The Teams – Platform Build
- 14. Death to Manual Deployments! Numbers from : OctopusDeployment AutomationSurvey 2013, State of SoftwareDelivery: Trends in the market now and in the future (2014), PuppetLabs State of DevOps Report 2014
- 15. Death to Manual Deployments! Numbers from : OctopusDeployment AutomationSurvey 2013, State of SoftwareDelivery: Trends in the market now and in the future (2014), PuppetLabs State of DevOps Report 2014
- 16. Death to Manual Deployments! Numbers from : OctopusDeployment AutomationSurvey 2013, State of SoftwareDelivery: Trends in the market now and in the future (2014), PuppetLabs State of DevOps Report 2014
- 17. Deployment automation - Picking a tool https://xebialabs.com/periodic-table-of-devops-tools/
- 18. Pilot teams
- 19. Continuous Delivery What : We make sure our software is always potentially shippable throughout its entire lifecycle and that any build could potentially be released to users at the touch of a button using a fully automated process. Why : We can put the release schedule in the hands of the business rather than IS. Be this with a regular rhythm or on demand.
- 20. Deploying Databases isn’t easy But it doesn’t have to be difficult either
- 21. 0 10 20 30 40 50 60 70 80 Aug Sept Oct Nov Dec Jan Feb Mar Apr Test Pre-Prod Prod Deployments/Month
- 22. Meanwhile – Platform & Infrastructure…. Tooling, tooling, tooling
- 25. Achievements so far… People actually understand what DevOps is!
- 26. In fact… We now have an active DevOps Community
- 27. Achievements so far… A new definition of done “it’s been coded and tested” “it’s working in Production”
- 28. Achievements so far… Packaged, scripted deployments are now the standard rather than the exception 40% of products deploying via Octopus Deploy
- 29. Achievements so far… Able to build dozens of version controlled servers in very short timescales
- 30. Not all smooth sailing Culture change is HARD!
- 32. Too fast?
- 33. Still plenty of work to do to get to….
- 34. Next steps along the road - Culture - Complete the move from Project Teams to Product Teams - Really treating Pre-Production and Production the same - We need to breed more champions across the company – Development, Security, Operations, Business
- 35. Next steps along the road - Monitoring - MUCH more monitoring – proactive rather than reactive and visible to everyone - Log aggregation – choose a tool and allow access to Dev and Ops - Get development teams delivering monitoring as a standard part of any system
- 36. Next steps along the road - Environments - IaaS – internally and externally - MUCH more monitoring – proactive rather than reactive and visible to everyone - The Cloud – use the best technology for the required solution
- 37. Next steps along the road – Delivery - Blue/Green deployments with zero downtime - Continuous performance testing
- 38. Anyone can ‘do’ DevOps If we can…..
- 39. Just keep aiming at Awesome!
Editor's Notes
- Introduce me! Introduce Callcredit …this is the story of the last 18 months at Callcredit as we’ve worked to move into a DevOps way of working First a bit of background for those of you that don’t know us. CC has only existed as a company for about 15 years, when the Skipton Building society decided it would be a cunning plan to set up a 3rd Credit Reference Agency to challenge the big 2 What followed was a decade and a half of rapid growth – and it’s that velocity that had created us some….challenges. So…
- Traditionally with Agile development we have a beautiful process of iterative development – with actual working functionality at the end of each sprint… GREAT! and then you hit this….. ....in fact from the Ops perspective this is what they see…
- By way of example – Who fancies following this manually… on 12 servers…. at 4am… and expect not to make a single mistake or miss a single line???
- So we’d kind of ended up with the classic Dev vs Ops Silos. With Developers constantly ‘sabotaging’ the live service and Ops “not letting us deploy changes” Server builds took in the region of a day to just build a server - let alone provisioning storage and configuring firewalls Bottleneck on DBA and Application Support teams when they're also trying to keep live services running Can't get people from Ops to contribute to projects (especially since Live is a dark hidden mystery to the dev teams) Can't get time from Ops to do deployments Painful deployments So once you did manage to secure someone from Ops to do a deployment for you , what was presented to them (often for the first time) was an, often long, list of manual instruction So, with new owners and a stated aim to double the revenue of the company in the next 5 years we clearly needed to stop sprinting to stay put and start getting to a place where we’re actually moving somewhere At the same time the then IT Director moved 'across the floor' into delivery Director role. It had already been flagged up by the Delivery teams struggled to get IT resource - mostly just deployment monkeys - so some members of IT Ops moved with him. This was the moment of opportunity and DevOps was flagged as a thing we should look at…. So we did….
- We did A LOT of reading and talking to people Trying to understand what we were really trying to do. Fortunately there are Millions of websites, blogs and books … (As an aside – If you’re looking into this sort of transformation read the Phoenix Project, and get everyone else to read it too. From the CEO downwards ideally)
- It’s a lot of different things – Most obviously it’s a simple combination of Development and Operations and really bringing those two disciplines together is what we’re talking about. That’s why the biggest word up there is ‘culture’… which is also, incidentally, the biggest challenge
- Some things you’ll also sometimes hear about DevOps ….fortunately we didn’t know enough about DevOps at this point to be aware of these! And in reality that’s‘ completely…. Wrong. Anyone can work towards a DevOps culture and gain the associated benefits
- Looked at the potential topologies - we know what's ideal, but that's too big a leap for us. So we aimed for (Venn diagram) Created 2 new teams *alarm bells* for anyone who's read into DevOps before, but don't panic - remember the initial topology we're aiming for. Type A – where we started Type 1 – where we’d like to be Type Y – a good start
- System Build Embedded in Development teams Bringing the Ops perspective into the Development teams ‘non-functional’ requirements understanding of live environments Championing other DevOps principles – automation, collaboration Allow the team to be autonomous in terms of getting code from idea all the way into production
- Platform Build Focus on providing services to help rapidly build the infrastructure, OS and middleware platforms which underpin these Operating in a “Platform-as-a-Service” Aiming at Infrastructure as code Also provide guidance and advice on infrastructure design for new systems and solutions
- Once we had a decent idea what it was we were talking about we looked at where we were currently and then at (a potentially unattainable) AWESOME Nirvana state Broken down into 4 areas Culture, Monitoring, Environments and Continuous Delivery…. Broadcast, broadcast, broadcast is SO important I can’t over-emphasize Speaking of sharing information – another thing that worked well for us was a small weekly email sent to all the IT and delivery teams. To feed them information whether they like it or not. Regardless it was just another way of getting DevOps to start seeping into their everyday working life
- During of research we’d come across some interesting surveys – and they pretty much showed us that we weren’t alone in our challenges around software delivery
- During of research we’d come across some interesting surveys – and they pretty much showed us that we weren’t alone in our challenges around software delivery
- During of research we’d come across some interesting surveys – and they pretty much showed us that we weren’t alone in our challenges around software delivery
- One of the first things System Build did was to look at a deployment automation tool – and be in no doubt there are a LOT of tools out there It’s IMPORTANT TO PICK THE ONE THAT’S RIGHT FOR YOU RIGHT NOW We specifically looked for ease of Integration Ease of use Visualisation Traceability Costs/Support
- Focus on automated pipelines (using Octopus Deploy) small deployments delivered often all the way to live Product rather than project work with change management to define a new Standard Release process – in fact we ended up with an actual DevOps champion in change management – side point ITIL and DevOps absolutely DO play nicely together In short we focussed on….
- In many ways this is the reason we’re doing all of this – and is also one of the easier sells to the business side of the world
- What’s the thing that makes databases trickier than web or app tiers? It’s that pesky data…. When we looked we found that, actually, once your dB is established most of the changes aren't destructive to data. In which case we can mostly treat database deployment like we do web or app…. As long as we ALWAYS keep thinking about the effect changes have on data. For our SQL Server deployments there only really seemed to be 2 tools – Microsoft’s SSDT and Redgate’s SQL deploy. Both work just fine DATABASES CAN BE BOTH VERSION CONTROLLED AND DEPLOYED IN A PACKAGED FASHION
- Would anyone like to guess when we put in the deployment automation tool?
- Tools used So, we had our deployment automation tool of choice for software, at the same time we looked at the same question for infrastructure There are a lot of “DevOps tools“ out there, but the realisation that platform came to was that there was a long way we could go with the ones we already had and just weren’t leveraging
- Powershell – Because if you’re automating in Windows, why wouldn’t you, fast, powerful scripting language SCCM – was used to deliver our base image, rather than templates Why: Because we could port it between hypervisors, physical servers and cloud providers easily, means we are adaptable, yet could maintain an identical base image System Center Orchestrator – Combining Powershell, SCCM and C#. Community of orchestrator guru’s delivering integration packs SCOM – immediately able to give devs a view of live server performance leading to areas quickly identified for code improvements Leankit – to manage our demand, and capture metrics
- Who meet regularly to discuss
- Finally the infrastructure wasn’t always the bottleneck to delivery! Statistics showed us to be 17x faster at deploying servers than the old manual method, this fact was advertised to senior IT Leadership, the result: an ovation!!!
- There’s still some teams and individuals unwilling to change how they work We’re still periodically asking Ops to stop ‘fixing’ live manually And there’s still developers who don’t think it’s their job to worry about if their code works in Production environments so long as it works on their computer
- Infrastructure as code almost instantly showed up every little mismatch in our environments, forcing us to fix and improve along the way Automating our complex legacy network architecture was going to be no small task In short we’re learning a whole lot about our own environments that were previously hidden
- Another challenge was - especially with the infrastructure - we were now starting to move at a pace that the service teams were struggling to support Such as… Being able to support an increasing number of appearing and disappearing VMs and Purchase and provision of storage were starting to be the new bottlenecks – so clearly we’ll now need to focus on helping them mitigate these challenges
- While we have made huge leaps, there are still challenges around working together. We got some great grass roots support – as one engineer noted, if we do this right then being On-call is FREE MONEY! We’re now working with Ops leadership to also drive this from the top down as well – especially around aligning objectives around business requirements rather than clashing things like uptime vs change
- We’re in a highly regulated industry Working with large amounts of very sensitive data And some very complex legacy systems and infrastructure Every company’s DevOps strategy will be unique, and every company’s plan for overcoming challenges will be just as unique. Whatever the circumstances, communication will be crucial to any adoption, as it ensures that everyone understands why such a change is necessary… So, we’ll keep pushing towards our ever-changing definition of “Awesome” I assume eventually it’ll look like this…