SlideShare a Scribd company logo

Gaming dev ops - Eduardo Saito

•
•
D
Devopsdays
TechnologyDesign
1 of 20
Download to read offline
Alert workflow in Gaming DevOps Eduardo Saito Director of Engineering - Server Operations GREE International November 2013
Traditional Alert workflow Ops NOC SME (Network, DBA,…) Dev
Traditional Alert workflow Ops NOC SME (Network, DBA,…) Dev
Alert workflow – previous Critical
Alert workflow – previous Ops Critical Dev
Alert workflow – previous Ops: where’s the runbook for this? Ops: app bug or system issue? Ops: who’s the devel of this game? Phone #? Ops: I can’t find the developer… who’s his manager? Critical Critical NonCritical Ops Dev
Alert workflow 2.0 Ops: where’s the runbook for this? Ops: app bug or system issue? Ops: who’s the devel of this game? Phone #? Ops: I can’t find the developer… who’s his manager? Ops Critical Dev
Alert Workflow 3.0 - current Ops Dev, Project X, Server
Alert Workflow 3.0 - current Ops Dev, Project X, Server Each alert go directly to the right team that can resolve it ! Dev, Project Y, Client, Android Dev, …
Alerts go to the person that can resolve Type Scope Checked by Who to page? ELB Load balancer health-check ELB No one – email alert only System-level Check cpu / disk / memory / network Pingdom / Nagios Ops team App-level Application issues / bugs Pingdom Dev and Ops teams
Alerts go to the person that can resolve Type Scope Checked by Who to page? ELB Load balancer health-check ELB No one – email alert only System-level Check cpu / disk / memory / network Pingdom / Nagios Ops team App-level Application issues / bugs Pingdom Dev and Ops teams
Alerts go to the person that can resolve Type Scope Checked by Who to page? ELB Load balancer health-check ELB No one – email alert only System-level Check cpu / disk / memory / network Pingdom / Nagios Ops team App-level Application issues / bugs Pingdom Dev and Ops teams
Alerts go to the person that can resolve Type App-level alerts can beChecked byby issuesto page? triggered Scope Who in: ELB System-level • Load balancer ELB Server-side • health-check Client-side •  iOS Check cpu / •  Android Pingdom / disk / memory / network App-level Pingdom Ops team Nagios Application issues / bugs No one – email alert only Dev and Ops teams
Dev and Ops are responsible Team On-call Ops 8 Dev 32, from 20 games (Serverside or client-side Android or iOS) Analytics 5
Big display dashboard = quick status
Big display dashboard = quick status
IM Bot = better communication Skype Bot informs in the game channel that an alert was triggered
IM Bot = better communication Ops and Dev receive the alert, and troubleshoot
IM Bot = better communication Skype Bot detects issue is resolved and send allclear
Thank You! eduardo.saito@gree.net We’re hiring! Vancouver and San Francisco http://gree-corp.com/jobs

Recommended

Mobile application testing
Mobile application testingMobile application testing
Mobile application testingshanu_123
 
Apps Qualification for the NAO Store
Apps Qualification for the NAO StoreApps Qualification for the NAO Store
Apps Qualification for the NAO StoreNicolas Rigaud
 
Mobile First: How To Approach Mobile Testing!
Mobile First: How To Approach Mobile Testing!Mobile First: How To Approach Mobile Testing!
Mobile First: How To Approach Mobile Testing!Usersnap
 
Presentation (software engineering)
Presentation (software engineering)Presentation (software engineering)
Presentation (software engineering)Abdullah Al Shiam
 
KrankyGeek WeRTC Conference 2014
KrankyGeek WeRTC Conference 2014KrankyGeek WeRTC Conference 2014
KrankyGeek WeRTC Conference 2014Jonathan Nolen
 
Mobile app testing
Mobile app testingMobile app testing
Mobile app testingBugRaptors
 
.NET Debugging tricks you wish you knew tamir dresher
.NET Debugging tricks you wish you knew tamir dresher.NET Debugging tricks you wish you knew tamir dresher
.NET Debugging tricks you wish you knew tamir dresherTamir Dresher
 
AWS re:Invent 2013 - MBL303 Gaming Ops - Running High-performance Ops for Mob...
AWS re:Invent 2013 - MBL303 Gaming Ops - Running High-performance Ops for Mob...AWS re:Invent 2013 - MBL303 Gaming Ops - Running High-performance Ops for Mob...
AWS re:Invent 2013 - MBL303 Gaming Ops - Running High-performance Ops for Mob...Eduardo Saito
 

Recommended

Mobile application testing
Mobile application testingMobile application testing
Mobile application testingshanu_123
 
Apps Qualification for the NAO Store
Apps Qualification for the NAO StoreApps Qualification for the NAO Store
Apps Qualification for the NAO StoreNicolas Rigaud
 
Mobile First: How To Approach Mobile Testing!
Mobile First: How To Approach Mobile Testing!Mobile First: How To Approach Mobile Testing!
Mobile First: How To Approach Mobile Testing!Usersnap
 
Presentation (software engineering)
Presentation (software engineering)Presentation (software engineering)
Presentation (software engineering)Abdullah Al Shiam
 
KrankyGeek WeRTC Conference 2014
KrankyGeek WeRTC Conference 2014KrankyGeek WeRTC Conference 2014
KrankyGeek WeRTC Conference 2014Jonathan Nolen
 
Mobile app testing
Mobile app testingMobile app testing
Mobile app testingBugRaptors
 
.NET Debugging tricks you wish you knew tamir dresher
.NET Debugging tricks you wish you knew tamir dresher.NET Debugging tricks you wish you knew tamir dresher
.NET Debugging tricks you wish you knew tamir dresherTamir Dresher
 
AWS re:Invent 2013 - MBL303 Gaming Ops - Running High-performance Ops for Mob...
AWS re:Invent 2013 - MBL303 Gaming Ops - Running High-performance Ops for Mob...AWS re:Invent 2013 - MBL303 Gaming Ops - Running High-performance Ops for Mob...
AWS re:Invent 2013 - MBL303 Gaming Ops - Running High-performance Ops for Mob...Eduardo Saito
 
451 and Cylance - The Roadmap To Better Endpoint Security
451 and Cylance - The Roadmap To Better Endpoint Security451 and Cylance - The Roadmap To Better Endpoint Security
451 and Cylance - The Roadmap To Better Endpoint SecurityAdrian Sanabria
 
Software Analytics: Data Analytics for Software Engineering and Security
Software Analytics: Data Analytics for Software Engineering and SecuritySoftware Analytics: Data Analytics for Software Engineering and Security
Software Analytics: Data Analytics for Software Engineering and SecurityTao Xie
 
Android Internals (This is not the droid you’re loking for...)
Android Internals (This is not the droid you’re loking for...)Android Internals (This is not the droid you’re loking for...)
Android Internals (This is not the droid you’re loking for...)Giacomo Bergami
 
The Last Mile Continued: Incident Management
The Last Mile Continued: Incident Management The Last Mile Continued: Incident Management
The Last Mile Continued: Incident Management Rundeck
 
Incident Management in the Age of DevOps and SRE
Incident Management in the Age of DevOps and SRE Incident Management in the Age of DevOps and SRE
Incident Management in the Age of DevOps and SRE Rundeck
 
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping ProgrammersAre Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping ProgrammersChris Parnin
 
Incident Management in the Age of DevOps and SRE
Incident Management in the Age of DevOps and SRE Incident Management in the Age of DevOps and SRE
Incident Management in the Age of DevOps and SRE Rundeck
 
File hippo
File hippoFile hippo
File hippoSandeep Kamath
 
Designing with capabilities (DDD-EU 2017)
Designing with capabilities (DDD-EU 2017)Designing with capabilities (DDD-EU 2017)
Designing with capabilities (DDD-EU 2017)Scott Wlaschin
 
Do Bugs Reside in Complex Code?
Do Bugs Reside in Complex Code?Do Bugs Reside in Complex Code?
Do Bugs Reside in Complex Code?CISPA Helmholtz Center for Information Security
 
Machine programming
Machine programmingMachine programming
Machine programmingDESMOND YUEN
 
Let's Make the PAIN Visible!
Let's Make the PAIN Visible!Let's Make the PAIN Visible!
Let's Make the PAIN Visible!Arty Starr
 
Cracking the mobile application code
Cracking the mobile application codeCracking the mobile application code
Cracking the mobile application codeSreenarayan A
 
AppSec Threat Modeling with 5 Agile Design Diagrams Every Project Should Have
AppSec Threat Modeling with 5 Agile Design Diagrams Every Project Should HaveAppSec Threat Modeling with 5 Agile Design Diagrams Every Project Should Have
AppSec Threat Modeling with 5 Agile Design Diagrams Every Project Should HaveRobert Grupe, CSSLP CISSP PE PMP
 
PHP, AWS, and Sleep - Hampton Roads DevFest 2016
PHP, AWS, and Sleep - Hampton Roads DevFest 2016PHP, AWS, and Sleep - Hampton Roads DevFest 2016
PHP, AWS, and Sleep - Hampton Roads DevFest 2016Guillermo A. Fisher
 
SAD15 - Maintenance
SAD15 - MaintenanceSAD15 - Maintenance
SAD15 - MaintenanceMichael Heron
 
Catch Me If You Can- Evaluating Android Anti-Malware Against Transformation A...
Catch Me If You Can- Evaluating Android Anti-Malware Against Transformation A...Catch Me If You Can- Evaluating Android Anti-Malware Against Transformation A...
Catch Me If You Can- Evaluating Android Anti-Malware Against Transformation A...Papitha Velumani
 
The Hacking Games - Operation System Vulnerabilities Meetup 29112022
The Hacking Games - Operation System Vulnerabilities Meetup 29112022The Hacking Games - Operation System Vulnerabilities Meetup 29112022
The Hacking Games - Operation System Vulnerabilities Meetup 29112022lior mazor
 
How to Manage the Risk of your Polyglot Environments
How to Manage the Risk of your Polyglot EnvironmentsHow to Manage the Risk of your Polyglot Environments
How to Manage the Risk of your Polyglot EnvironmentsDevOps.com
 
Ch01-whyTest.pptx
Ch01-whyTest.pptxCh01-whyTest.pptx
Ch01-whyTest.pptxSteveIrwin25
 
Dev opsdays scriptcode
Dev opsdays scriptcodeDev opsdays scriptcode
Dev opsdays scriptcodeDevopsdays
 
Zero to hero - Geoff Webb
Zero to hero - Geoff WebbZero to hero - Geoff Webb
Zero to hero - Geoff WebbDevopsdays
 

More Related Content

Similar to Gaming dev ops - Eduardo Saito

451 and Cylance - The Roadmap To Better Endpoint Security
451 and Cylance - The Roadmap To Better Endpoint Security451 and Cylance - The Roadmap To Better Endpoint Security
451 and Cylance - The Roadmap To Better Endpoint SecurityAdrian Sanabria
 
Software Analytics: Data Analytics for Software Engineering and Security
Software Analytics: Data Analytics for Software Engineering and SecuritySoftware Analytics: Data Analytics for Software Engineering and Security
Software Analytics: Data Analytics for Software Engineering and SecurityTao Xie
 
Android Internals (This is not the droid you’re loking for...)
Android Internals (This is not the droid you’re loking for...)Android Internals (This is not the droid you’re loking for...)
Android Internals (This is not the droid you’re loking for...)Giacomo Bergami
 
The Last Mile Continued: Incident Management
The Last Mile Continued: Incident Management The Last Mile Continued: Incident Management
The Last Mile Continued: Incident Management Rundeck
 
Incident Management in the Age of DevOps and SRE
Incident Management in the Age of DevOps and SRE Incident Management in the Age of DevOps and SRE
Incident Management in the Age of DevOps and SRE Rundeck
 
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping ProgrammersAre Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping ProgrammersChris Parnin
 
Incident Management in the Age of DevOps and SRE
Incident Management in the Age of DevOps and SRE Incident Management in the Age of DevOps and SRE
Incident Management in the Age of DevOps and SRE Rundeck
 
Designing with capabilities (DDD-EU 2017)
Designing with capabilities (DDD-EU 2017)Designing with capabilities (DDD-EU 2017)
Designing with capabilities (DDD-EU 2017)Scott Wlaschin
 
Machine programming
Machine programmingMachine programming
Machine programmingDESMOND YUEN
 
Let's Make the PAIN Visible!
Let's Make the PAIN Visible!Let's Make the PAIN Visible!
Let's Make the PAIN Visible!Arty Starr
 
Cracking the mobile application code
Cracking the mobile application codeCracking the mobile application code
Cracking the mobile application codeSreenarayan A
 
AppSec Threat Modeling with 5 Agile Design Diagrams Every Project Should Have
AppSec Threat Modeling with 5 Agile Design Diagrams Every Project Should HaveAppSec Threat Modeling with 5 Agile Design Diagrams Every Project Should Have
AppSec Threat Modeling with 5 Agile Design Diagrams Every Project Should HaveRobert Grupe, CSSLP CISSP PE PMP
 
PHP, AWS, and Sleep - Hampton Roads DevFest 2016
PHP, AWS, and Sleep - Hampton Roads DevFest 2016PHP, AWS, and Sleep - Hampton Roads DevFest 2016
PHP, AWS, and Sleep - Hampton Roads DevFest 2016Guillermo A. Fisher
 
Catch Me If You Can- Evaluating Android Anti-Malware Against Transformation A...
Catch Me If You Can- Evaluating Android Anti-Malware Against Transformation A...Catch Me If You Can- Evaluating Android Anti-Malware Against Transformation A...
Catch Me If You Can- Evaluating Android Anti-Malware Against Transformation A...Papitha Velumani
 
The Hacking Games - Operation System Vulnerabilities Meetup 29112022
The Hacking Games - Operation System Vulnerabilities Meetup 29112022The Hacking Games - Operation System Vulnerabilities Meetup 29112022
The Hacking Games - Operation System Vulnerabilities Meetup 29112022lior mazor
 
How to Manage the Risk of your Polyglot Environments
How to Manage the Risk of your Polyglot EnvironmentsHow to Manage the Risk of your Polyglot Environments
How to Manage the Risk of your Polyglot EnvironmentsDevOps.com
 

Similar to Gaming dev ops - Eduardo Saito (20)

451 and Cylance - The Roadmap To Better Endpoint Security
451 and Cylance - The Roadmap To Better Endpoint Security451 and Cylance - The Roadmap To Better Endpoint Security
451 and Cylance - The Roadmap To Better Endpoint Security
 
Software Analytics: Data Analytics for Software Engineering and Security
Software Analytics: Data Analytics for Software Engineering and SecuritySoftware Analytics: Data Analytics for Software Engineering and Security
Software Analytics: Data Analytics for Software Engineering and Security
 
Android Internals (This is not the droid you’re loking for...)
Android Internals (This is not the droid you’re loking for...)Android Internals (This is not the droid you’re loking for...)
Android Internals (This is not the droid you’re loking for...)
 
The Last Mile Continued: Incident Management
The Last Mile Continued: Incident Management The Last Mile Continued: Incident Management
The Last Mile Continued: Incident Management
 
Incident Management in the Age of DevOps and SRE
Incident Management in the Age of DevOps and SRE Incident Management in the Age of DevOps and SRE
Incident Management in the Age of DevOps and SRE
 
Are Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping ProgrammersAre Automated Debugging Techniques Actually Helping Programmers
Are Automated Debugging Techniques Actually Helping Programmers
 
Incident Management in the Age of DevOps and SRE
Incident Management in the Age of DevOps and SRE Incident Management in the Age of DevOps and SRE
Incident Management in the Age of DevOps and SRE
 
File hippo
File hippoFile hippo
File hippo
 
Designing with capabilities (DDD-EU 2017)
Designing with capabilities (DDD-EU 2017)Designing with capabilities (DDD-EU 2017)
Designing with capabilities (DDD-EU 2017)
 
Do Bugs Reside in Complex Code?
Do Bugs Reside in Complex Code?Do Bugs Reside in Complex Code?
Do Bugs Reside in Complex Code?
 
Machine programming
Machine programmingMachine programming
Machine programming
 
Let's Make the PAIN Visible!
Let's Make the PAIN Visible!Let's Make the PAIN Visible!
Let's Make the PAIN Visible!
 
Cracking the mobile application code
Cracking the mobile application codeCracking the mobile application code
Cracking the mobile application code
 
AppSec Threat Modeling with 5 Agile Design Diagrams Every Project Should Have
AppSec Threat Modeling with 5 Agile Design Diagrams Every Project Should HaveAppSec Threat Modeling with 5 Agile Design Diagrams Every Project Should Have
AppSec Threat Modeling with 5 Agile Design Diagrams Every Project Should Have
 
PHP, AWS, and Sleep - Hampton Roads DevFest 2016
PHP, AWS, and Sleep - Hampton Roads DevFest 2016PHP, AWS, and Sleep - Hampton Roads DevFest 2016
PHP, AWS, and Sleep - Hampton Roads DevFest 2016
 
SAD15 - Maintenance
SAD15 - MaintenanceSAD15 - Maintenance
SAD15 - Maintenance
 
Catch Me If You Can- Evaluating Android Anti-Malware Against Transformation A...
Catch Me If You Can- Evaluating Android Anti-Malware Against Transformation A...Catch Me If You Can- Evaluating Android Anti-Malware Against Transformation A...
Catch Me If You Can- Evaluating Android Anti-Malware Against Transformation A...
 
The Hacking Games - Operation System Vulnerabilities Meetup 29112022
The Hacking Games - Operation System Vulnerabilities Meetup 29112022The Hacking Games - Operation System Vulnerabilities Meetup 29112022
The Hacking Games - Operation System Vulnerabilities Meetup 29112022
 
How to Manage the Risk of your Polyglot Environments
How to Manage the Risk of your Polyglot EnvironmentsHow to Manage the Risk of your Polyglot Environments
How to Manage the Risk of your Polyglot Environments
 
Ch01-whyTest.pptx
Ch01-whyTest.pptxCh01-whyTest.pptx
Ch01-whyTest.pptx
 

More from Devopsdays

Dev opsdays scriptcode
Dev opsdays scriptcodeDev opsdays scriptcode
Dev opsdays scriptcodeDevopsdays
 
Zero to hero - Geoff Webb
Zero to hero - Geoff WebbZero to hero - Geoff Webb
Zero to hero - Geoff WebbDevopsdays
 
Letting go gavin - Mc Donald
Letting go gavin - Mc DonaldLetting go gavin - Mc Donald
Letting go gavin - Mc DonaldDevopsdays
 
Dw tpain - Gordon Klok
Dw tpain - Gordon KlokDw tpain - Gordon Klok
Dw tpain - Gordon KlokDevopsdays
 
Dev ops finishes what agile started - Manfred Moser
Dev ops finishes what agile started - Manfred MoserDev ops finishes what agile started - Manfred Moser
Dev ops finishes what agile started - Manfred MoserDevopsdays
 
Game of thrones - Jonathan Thorpe
Game of thrones - Jonathan ThorpeGame of thrones - Jonathan Thorpe
Game of thrones - Jonathan ThorpeDevopsdays
 
Heka - Rob Miller
Heka - Rob MillerHeka - Rob Miller
Heka - Rob MillerDevopsdays
 
From the classroom to the cloud a journey with node.js - Christopher Hogue
From the classroom to the cloud a journey with node.js - Christopher HogueFrom the classroom to the cloud a journey with node.js - Christopher Hogue
From the classroom to the cloud a journey with node.js - Christopher HogueDevopsdays
 
Dev ops at mobify - Kyle Young
Dev ops at mobify - Kyle YoungDev ops at mobify - Kyle Young
Dev ops at mobify - Kyle YoungDevopsdays
 
Your business needs devops, so don’t follow - Brian johnson
Your business needs devops, so don’t follow - Brian johnson Your business needs devops, so don’t follow - Brian johnson
Your business needs devops, so don’t follow - Brian johnson Devopsdays
 
Test kitchen 1.0 - Fletcher Nichol
Test kitchen 1.0 - Fletcher NicholTest kitchen 1.0 - Fletcher Nichol
Test kitchen 1.0 - Fletcher NicholDevopsdays
 
Living system or build factory - Chris Maxwell
Living system or build factory - Chris MaxwellLiving system or build factory - Chris Maxwell
Living system or build factory - Chris MaxwellDevopsdays
 
From vagrant to production - Mark Eijsermans
From vagrant to production - Mark EijsermansFrom vagrant to production - Mark Eijsermans
From vagrant to production - Mark EijsermansDevopsdays
 
Dev ops lessons learned - Michael Collins
Dev ops lessons learned - Michael CollinsDev ops lessons learned - Michael Collins
Dev ops lessons learned - Michael CollinsDevopsdays
 
Building for operations - Reinhardt Quelle
Building for operations - Reinhardt QuelleBuilding for operations - Reinhardt Quelle
Building for operations - Reinhardt QuelleDevopsdays
 
Taking devops to the Next Level - Max Martin
Taking devops to the Next Level - Max MartinTaking devops to the Next Level - Max Martin
Taking devops to the Next Level - Max MartinDevopsdays
 
Sensu intro - Sean Porter
Sensu intro - Sean PorterSensu intro - Sean Porter
Sensu intro - Sean PorterDevopsdays
 
Ops for everyone - John Britton
Ops for everyone - John BrittonOps for everyone - John Britton
Ops for everyone - John BrittonDevopsdays
 
Effective monitoring with statsd - Alexis lê-quôc
Effective monitoring with statsd - Alexis lê-quôcEffective monitoring with statsd - Alexis lê-quôc
Effective monitoring with statsd - Alexis lê-quôcDevopsdays
 
Being healthy dev and ops in cookpad - Issei Naruta
Being healthy dev and ops in cookpad - Issei NarutaBeing healthy dev and ops in cookpad - Issei Naruta
Being healthy dev and ops in cookpad - Issei NarutaDevopsdays
 

More from Devopsdays (20)

Dev opsdays scriptcode
Dev opsdays scriptcodeDev opsdays scriptcode
Dev opsdays scriptcode
 
Zero to hero - Geoff Webb
Zero to hero - Geoff WebbZero to hero - Geoff Webb
Zero to hero - Geoff Webb
 
Letting go gavin - Mc Donald
Letting go gavin - Mc DonaldLetting go gavin - Mc Donald
Letting go gavin - Mc Donald
 
Dw tpain - Gordon Klok
Dw tpain - Gordon KlokDw tpain - Gordon Klok
Dw tpain - Gordon Klok
 
Dev ops finishes what agile started - Manfred Moser
Dev ops finishes what agile started - Manfred MoserDev ops finishes what agile started - Manfred Moser
Dev ops finishes what agile started - Manfred Moser
 
Game of thrones - Jonathan Thorpe
Game of thrones - Jonathan ThorpeGame of thrones - Jonathan Thorpe
Game of thrones - Jonathan Thorpe
 
Heka - Rob Miller
Heka - Rob MillerHeka - Rob Miller
Heka - Rob Miller
 
From the classroom to the cloud a journey with node.js - Christopher Hogue
From the classroom to the cloud a journey with node.js - Christopher HogueFrom the classroom to the cloud a journey with node.js - Christopher Hogue
From the classroom to the cloud a journey with node.js - Christopher Hogue
 
Dev ops at mobify - Kyle Young
Dev ops at mobify - Kyle YoungDev ops at mobify - Kyle Young
Dev ops at mobify - Kyle Young
 
Your business needs devops, so don’t follow - Brian johnson
Your business needs devops, so don’t follow - Brian johnson Your business needs devops, so don’t follow - Brian johnson
Your business needs devops, so don’t follow - Brian johnson
 
Test kitchen 1.0 - Fletcher Nichol
Test kitchen 1.0 - Fletcher NicholTest kitchen 1.0 - Fletcher Nichol
Test kitchen 1.0 - Fletcher Nichol
 
Living system or build factory - Chris Maxwell
Living system or build factory - Chris MaxwellLiving system or build factory - Chris Maxwell
Living system or build factory - Chris Maxwell
 
From vagrant to production - Mark Eijsermans
From vagrant to production - Mark EijsermansFrom vagrant to production - Mark Eijsermans
From vagrant to production - Mark Eijsermans
 
Dev ops lessons learned - Michael Collins
Dev ops lessons learned - Michael CollinsDev ops lessons learned - Michael Collins
Dev ops lessons learned - Michael Collins
 
Building for operations - Reinhardt Quelle
Building for operations - Reinhardt QuelleBuilding for operations - Reinhardt Quelle
Building for operations - Reinhardt Quelle
 
Taking devops to the Next Level - Max Martin
Taking devops to the Next Level - Max MartinTaking devops to the Next Level - Max Martin
Taking devops to the Next Level - Max Martin
 
Sensu intro - Sean Porter
Sensu intro - Sean PorterSensu intro - Sean Porter
Sensu intro - Sean Porter
 
Ops for everyone - John Britton
Ops for everyone - John BrittonOps for everyone - John Britton
Ops for everyone - John Britton
 
Effective monitoring with statsd - Alexis lê-quôc
Effective monitoring with statsd - Alexis lê-quôcEffective monitoring with statsd - Alexis lê-quôc
Effective monitoring with statsd - Alexis lê-quôc
 
Being healthy dev and ops in cookpad - Issei Naruta
Being healthy dev and ops in cookpad - Issei NarutaBeing healthy dev and ops in cookpad - Issei Naruta
Being healthy dev and ops in cookpad - Issei Naruta
 

Recently uploaded

DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsSergiu Bodiu
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxLoriGlavin3
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024Stephanie Beckett
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii SoldatenkoFwdays
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024Lonnie McRorey
 
Generative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersGenerative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersRaghuram Pandurangan
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxLoriGlavin3
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 3652toLead Limited
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Mattias Andersson
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity PlanDatabarracks
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningLars Bell
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.Curtis Poe
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .Alan Dix
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfMounikaPolabathina
 
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESmohitsingh558521
 
SAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxSAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxNavinnSomaal
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 

Recently uploaded (20)

DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platforms
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024
 
Generative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersGenerative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information Developers
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity Plan
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine Tuning
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdf
 
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
 
SAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxSAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptx
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL Certs
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 

Gaming dev ops - Eduardo Saito

  • 1. Alert workflow in Gaming DevOps Eduardo Saito Director of Engineering - Server Operations GREE International November 2013
  • 2. Traditional Alert workflow Ops NOC SME (Network, DBA,…) Dev
  • 3. Traditional Alert workflow Ops NOC SME (Network, DBA,…) Dev
  • 4. Alert workflow – previous Critical
  • 5. Alert workflow – previous Ops Critical Dev
  • 6. Alert workflow – previous Ops: where’s the runbook for this? Ops: app bug or system issue? Ops: who’s the devel of this game? Phone #? Ops: I can’t find the developer… who’s his manager? Critical Critical NonCritical Ops Dev
  • 7. Alert workflow 2.0 Ops: where’s the runbook for this? Ops: app bug or system issue? Ops: who’s the devel of this game? Phone #? Ops: I can’t find the developer… who’s his manager? Ops Critical Dev
  • 8. Alert Workflow 3.0 - current Ops Dev, Project X, Server
  • 9. Alert Workflow 3.0 - current Ops Dev, Project X, Server Each alert go directly to the right team that can resolve it ! Dev, Project Y, Client, Android Dev, …
  • 10. Alerts go to the person that can resolve Type Scope Checked by Who to page? ELB Load balancer health-check ELB No one – email alert only System-level Check cpu / disk / memory / network Pingdom / Nagios Ops team App-level Application issues / bugs Pingdom Dev and Ops teams
  • 11. Alerts go to the person that can resolve Type Scope Checked by Who to page? ELB Load balancer health-check ELB No one – email alert only System-level Check cpu / disk / memory / network Pingdom / Nagios Ops team App-level Application issues / bugs Pingdom Dev and Ops teams
  • 12. Alerts go to the person that can resolve Type Scope Checked by Who to page? ELB Load balancer health-check ELB No one – email alert only System-level Check cpu / disk / memory / network Pingdom / Nagios Ops team App-level Application issues / bugs Pingdom Dev and Ops teams
  • 13. Alerts go to the person that can resolve Type App-level alerts can beChecked byby issuesto page? triggered Scope Who in: ELB System-level • Load balancer ELB Server-side • health-check Client-side •  iOS Check cpu / •  Android Pingdom / disk / memory / network App-level Pingdom Ops team Nagios Application issues / bugs No one – email alert only Dev and Ops teams
  • 14. Dev and Ops are responsible Team On-call Ops 8 Dev 32, from 20 games (Serverside or client-side Android or iOS) Analytics 5
  • 15. Big display dashboard = quick status
  • 16. Big display dashboard = quick status
  • 17. IM Bot = better communication Skype Bot informs in the game channel that an alert was triggered
  • 18. IM Bot = better communication Ops and Dev receive the alert, and troubleshoot
  • 19. IM Bot = better communication Skype Bot detects issue is resolved and send allclear
  • 20. Thank You! eduardo.saito@gree.net We’re hiring! Vancouver and San Francisco http://gree-corp.com/jobs