The document discusses concerns around designing microservices. It begins with an overview of microservices, noting they are an approach to building distributed applications that are independently developed and deployed, contain a single context or responsibility, and communicate simply using technologies like HTTP or message queues. The document then covers advantages like loose coupling, ability to use the right tool for each job, and facilitating continuous delivery. It also discusses challenges like the complexity of distributed systems and potential for services to fail. Finally, it outlines considerations for service design, including having each service represent a single bounded context and authority, being resilient, fast, and efficient.
IBM Bluemix OpenWhisk: IBM Seminar 2016, Tokyo, Japan: The Future of Cloud Pr...OpenWhisk
Learn more about the IBM Bluemix OpenWhisk, a serverless event-driven compute platform, which quickly executes application logic in response to events or direct invocations from web/mobile apps or other endpoints.
Respond to and troubleshoot production incidents like an saTom Cudd
So it's 4 AM and you just got a call from a panicked executive that the system is down! Oh noes! What do you do? Troubleshoot LIKE AN SA. I know "Systems Administrator" is not the cool industry term anymore, but that mentality for fixing the big live problem, like RIGHT NOW can still help today.
You're probably in the job you're in because you're AWESOME at figuring out what's wrong and fixing problems. But your projects have grown, your team has grown, and the expectations grow with them. How do you deal with these new found responsibilities? LIKE AN SA. There are some simple processes you can put in place to help make your life easier. We'll discuss a framework for incident response, a step-by-step guide for troubleshooting production issues, and how to then learn from these outages to prevent problems from happening again.
Interact Differently: Get More From Your Tools Through Exposed APIsKevin Fealey
Most tools are designed with a single function in mind, but can often be leveraged for additional workflows as well. For example, SonarQube is a marketed as a platform to manage code quality, but custom plugins can also enable security testing. At its heart, Jenkins is a build tool, but it can be used for security testing as well. Don't like the security dashboard applications bundled with your commercial security tools? Either Jenkins or SonarQube can replace them! Can't stand viewing your tool exports in Excel? A simple XML transformation can allow you to see your data in a way that makes sense to you.
Welcome to the age of APIs. Nearly every major software product now exposes RESTful APIs, an SDK, command line, or plugin interface. Don't limit yourself to out-of-the-box functionality - customize your tools to work best for you.
This talk will describe some ways exposed APIs and plugin interfaces in commercial and open source products have been used to make work more effective and efficient. We'll demonstrate simple tools that interact with common software packages, like Jenkins, SonarQube, and OWASP ZAP, to streamline workflows and provide better visibility on what matters most to us. And we'll tell you how to get started getting more out of the tools you already use.
Next-gen Network Telemetry is Within Your Packets: In-band OAMFrank Brockners
While troubleshooting or planning, did you ever wish to get full insight into which paths *all* your packets take in your network or were you ever asked to prove that your traffic really follows the path you specified by service chaining or traffic engineering? We approach this problem by adding meta-data to *all* packets - "In-band OAM for IPv6" and "path/service-chain verification" are the associated technologies. In-band OAM adds forwarding path information and other information/stats to every data packet - as opposed to relying on probe packets, which is the traditional method that tools like ping or traceroute use. In-band OAM information can either be accessed directly on the router or be available via Netflow. The presentation introduces in-band OAM as a technology and discuss a series of use-cases and deployment scenarios, ranging from proving that all packets traverse a specific path and troubleshooting forwarding issues in networks which use ECMP, over simple approaches to deriving the network traffic matrix, or trend analysis on network parameters such as delay or packet loss, to using iOAM as a tool to optimize forwarding in your network. The technology discussion is complemented references to demos (using Cisco IOS, FD.io/VPP, OpenDaylight Controller etc.) which showcase this new technology at work.
IBM Bluemix OpenWhisk: IBM Seminar 2016, Tokyo, Japan: The Future of Cloud Pr...OpenWhisk
Learn more about the IBM Bluemix OpenWhisk, a serverless event-driven compute platform, which quickly executes application logic in response to events or direct invocations from web/mobile apps or other endpoints.
Respond to and troubleshoot production incidents like an saTom Cudd
So it's 4 AM and you just got a call from a panicked executive that the system is down! Oh noes! What do you do? Troubleshoot LIKE AN SA. I know "Systems Administrator" is not the cool industry term anymore, but that mentality for fixing the big live problem, like RIGHT NOW can still help today.
You're probably in the job you're in because you're AWESOME at figuring out what's wrong and fixing problems. But your projects have grown, your team has grown, and the expectations grow with them. How do you deal with these new found responsibilities? LIKE AN SA. There are some simple processes you can put in place to help make your life easier. We'll discuss a framework for incident response, a step-by-step guide for troubleshooting production issues, and how to then learn from these outages to prevent problems from happening again.
Interact Differently: Get More From Your Tools Through Exposed APIsKevin Fealey
Most tools are designed with a single function in mind, but can often be leveraged for additional workflows as well. For example, SonarQube is a marketed as a platform to manage code quality, but custom plugins can also enable security testing. At its heart, Jenkins is a build tool, but it can be used for security testing as well. Don't like the security dashboard applications bundled with your commercial security tools? Either Jenkins or SonarQube can replace them! Can't stand viewing your tool exports in Excel? A simple XML transformation can allow you to see your data in a way that makes sense to you.
Welcome to the age of APIs. Nearly every major software product now exposes RESTful APIs, an SDK, command line, or plugin interface. Don't limit yourself to out-of-the-box functionality - customize your tools to work best for you.
This talk will describe some ways exposed APIs and plugin interfaces in commercial and open source products have been used to make work more effective and efficient. We'll demonstrate simple tools that interact with common software packages, like Jenkins, SonarQube, and OWASP ZAP, to streamline workflows and provide better visibility on what matters most to us. And we'll tell you how to get started getting more out of the tools you already use.
Next-gen Network Telemetry is Within Your Packets: In-band OAMFrank Brockners
While troubleshooting or planning, did you ever wish to get full insight into which paths *all* your packets take in your network or were you ever asked to prove that your traffic really follows the path you specified by service chaining or traffic engineering? We approach this problem by adding meta-data to *all* packets - "In-band OAM for IPv6" and "path/service-chain verification" are the associated technologies. In-band OAM adds forwarding path information and other information/stats to every data packet - as opposed to relying on probe packets, which is the traditional method that tools like ping or traceroute use. In-band OAM information can either be accessed directly on the router or be available via Netflow. The presentation introduces in-band OAM as a technology and discuss a series of use-cases and deployment scenarios, ranging from proving that all packets traverse a specific path and troubleshooting forwarding issues in networks which use ECMP, over simple approaches to deriving the network traffic matrix, or trend analysis on network parameters such as delay or packet loss, to using iOAM as a tool to optimize forwarding in your network. The technology discussion is complemented references to demos (using Cisco IOS, FD.io/VPP, OpenDaylight Controller etc.) which showcase this new technology at work.
Dashboards: Using data to find out what's really going onrouanw
How many people are using your website right now? Which features are their favourite? Are they experiencing errors? Getting stuck? How are your servers performing? Is your code easy to work with? Are you making money? Dashboards are a way to have the answers to these questions all around you, all the time.
Join Rouan as he shows you why building dashboards will change the way you look at software. He’ll share concrete examples of the kinds of dashboards you could build and will show you the tools with which you can build them. He’ll introduce you to principles that will guide you as you make decisions about what you dashboard, how you treat data and how you use data to make decisions.
EXECUTIVE SUMMARY
Building off Veterans Ombudsman reports and other reports which support systematic change to the federal government process involving injured and/or disabled veterans, the Foundation for Democratic Advancement (FDA) examines closely the federal government veteran processes. Through that examination, the FDA uncovers serve deficiencies which compromise the federal government’s service to injured and/or disabled veterans and ultimately its obligation to Canadians in the Canadian Forces and Royal Mounted Police who put their lives at risk to protect this country. In particular, the FDA documents a system of patronage appointments, mismanagement of the Veterans Review and Appeal Board (VRAB), and a veterans review and appeal process defined significantly by unreasonable interpretation and application of relevant Acts of Parliament. Using its expertise in democracy and government, the FDA believes that these deficiencies stem from a failure of Canadian democracy and shortcomings in the Canadian federal electoral system. The FDA uncovers evidence that elected officials including the Prime Minister are putting their self-interests above the interests of injured and/or disabled veterans and Canadians as a whole. The Brian Bradley case study, an ongoing egregious seventeen-year legal struggle in which Brian battles for just care for his spinal cord injury, shows that the deficiencies with the VRAB and failings of the federal government have gone on far too long. The FDA recommends a number of reforms, including the elimination of the VRAB and delegation of its responsibilities and duties to the Federal Court, implementation of a non-partisan federal government appointment system, and correction of biases and unfairness in the federal electoral system to help ensure that representatives who represent the broad public good are elected.
"If liberty and equality, as is thought by some, are chiefly to be found in democracy, they will be best attained when all persons alike share in government to the utmost."
- Aristotle
For more information on veteran Brian Bradley and the Veterans Review and Appeal Board (VRAB) please go to the FDA url:
http://democracychange.org/2013/10/fdas-brian-bradley-case-study-and-process-review-of-the-veterans-review-and-appeal-board/
Broken up into three core sections, this presentation tries to help explain why you monitor software, platforms and your products. What you can look for, and how to best get that information out of your code and finally how Sensu can be used to achieve this in a scalable platform.
This presentation we showed how security products fail to protect or detect against some type of flaws. Products includes web scanners , log analysis, ids, waf as Snort, OSSEC, Qualys, Acunetix, W3AF and so on. Problems aren't related only with those tools, we just use them to create our PoCs. Possible problems could be a Compliance bypass creating reports "without" SQLi for example.
Lorenzo Nicora makes an introductory presentation on event sourcing, what you want to achieve and how to use CQRS to implement event sourcing for your microservices.
Home Brewing R.U.M - Analyzing application performance with real user monitoringAnkit Rastogi
It is a true fact that performance experienced by real world users has direct impact on our application adoption rate.Good performance increases retention and conversion rates. Our primary aim should be to deliver value to an user, and make sure that they get the best possible user experience that delights them, and keeps them coming back for more. In this regard, monitoring user behaviour becomes imperative as it provides key metrics for web application performance. While developing web applications, developers test in their local environments, and also do user acceptance testing. But what happens once the application goes out onto the real world? The case I’m making here talks about analysing application performance once its in the hands of real users.
This is where real user monitoring a.k.a R.U.M comes in picture. R.U.M captures performance metrics such as bandwidth and page/view load times, user location, device type, carrier speed, application errors, Ajax request and application usage along with custom performance metrics that provides actionable business intelligence.
In R.U.M, we can then visualize performance over time for key metrics based on different average, geometric mean, median or percentile calculations. The performance data can be drilled down further on the basis of geography, device, error encountered, speed etc.
Presentattie Joomladagen 2015:
Net als bij welk CMS dan ook laden Joomla sites soms traag. Echter, ook als je site al wel redelijk snel laadt kan de laadtijd nog steeds drastisch versneld worden. je zult er versteld van staan hoe makkelijk je veel winst kan behalen. De winst valt op en aantal punten te behalen:
Zoek een goede host en optimaliseer deze omgeving
Optimaliseer Joomla
Simpele regels in .htaccess
Beperk je in extensies / externe bronnen
Optimaliseer CSS en Javascript
Gebruik optimalisatie plugins
Optimaliseer plaatjes
Optimaliseer voor mobiele devices
http://joomlaseo.com/Blog/talking-at-the-dutch-joomladagen-2015
Detail agenda + links to slides & videos from vodQA Pune - Innovations in Testing, conducted on Sat, 6th June 2015 at ThoughtWorks, Pune
http://vodqa-pune.weebly.com
http://essenceoftesting.blogspot.com/search/label/vodQA
“ The Microservices architecture has many appealing qualities, but the road towards it has painful traps for the unwary. This book will help you figure out if this path is for you, and how to avoid those traps on your journey.”
—Martin Fowler Chief Scientist, ThoughtWorks
These are my summarized notes from all the microservices session I attended at QCon 2015. These sessions had tons of learning around how to scale microservices and avoid common pitfalls
Dashboards: Using data to find out what's really going onrouanw
How many people are using your website right now? Which features are their favourite? Are they experiencing errors? Getting stuck? How are your servers performing? Is your code easy to work with? Are you making money? Dashboards are a way to have the answers to these questions all around you, all the time.
Join Rouan as he shows you why building dashboards will change the way you look at software. He’ll share concrete examples of the kinds of dashboards you could build and will show you the tools with which you can build them. He’ll introduce you to principles that will guide you as you make decisions about what you dashboard, how you treat data and how you use data to make decisions.
EXECUTIVE SUMMARY
Building off Veterans Ombudsman reports and other reports which support systematic change to the federal government process involving injured and/or disabled veterans, the Foundation for Democratic Advancement (FDA) examines closely the federal government veteran processes. Through that examination, the FDA uncovers serve deficiencies which compromise the federal government’s service to injured and/or disabled veterans and ultimately its obligation to Canadians in the Canadian Forces and Royal Mounted Police who put their lives at risk to protect this country. In particular, the FDA documents a system of patronage appointments, mismanagement of the Veterans Review and Appeal Board (VRAB), and a veterans review and appeal process defined significantly by unreasonable interpretation and application of relevant Acts of Parliament. Using its expertise in democracy and government, the FDA believes that these deficiencies stem from a failure of Canadian democracy and shortcomings in the Canadian federal electoral system. The FDA uncovers evidence that elected officials including the Prime Minister are putting their self-interests above the interests of injured and/or disabled veterans and Canadians as a whole. The Brian Bradley case study, an ongoing egregious seventeen-year legal struggle in which Brian battles for just care for his spinal cord injury, shows that the deficiencies with the VRAB and failings of the federal government have gone on far too long. The FDA recommends a number of reforms, including the elimination of the VRAB and delegation of its responsibilities and duties to the Federal Court, implementation of a non-partisan federal government appointment system, and correction of biases and unfairness in the federal electoral system to help ensure that representatives who represent the broad public good are elected.
"If liberty and equality, as is thought by some, are chiefly to be found in democracy, they will be best attained when all persons alike share in government to the utmost."
- Aristotle
For more information on veteran Brian Bradley and the Veterans Review and Appeal Board (VRAB) please go to the FDA url:
http://democracychange.org/2013/10/fdas-brian-bradley-case-study-and-process-review-of-the-veterans-review-and-appeal-board/
Broken up into three core sections, this presentation tries to help explain why you monitor software, platforms and your products. What you can look for, and how to best get that information out of your code and finally how Sensu can be used to achieve this in a scalable platform.
This presentation we showed how security products fail to protect or detect against some type of flaws. Products includes web scanners , log analysis, ids, waf as Snort, OSSEC, Qualys, Acunetix, W3AF and so on. Problems aren't related only with those tools, we just use them to create our PoCs. Possible problems could be a Compliance bypass creating reports "without" SQLi for example.
Lorenzo Nicora makes an introductory presentation on event sourcing, what you want to achieve and how to use CQRS to implement event sourcing for your microservices.
Home Brewing R.U.M - Analyzing application performance with real user monitoringAnkit Rastogi
It is a true fact that performance experienced by real world users has direct impact on our application adoption rate.Good performance increases retention and conversion rates. Our primary aim should be to deliver value to an user, and make sure that they get the best possible user experience that delights them, and keeps them coming back for more. In this regard, monitoring user behaviour becomes imperative as it provides key metrics for web application performance. While developing web applications, developers test in their local environments, and also do user acceptance testing. But what happens once the application goes out onto the real world? The case I’m making here talks about analysing application performance once its in the hands of real users.
This is where real user monitoring a.k.a R.U.M comes in picture. R.U.M captures performance metrics such as bandwidth and page/view load times, user location, device type, carrier speed, application errors, Ajax request and application usage along with custom performance metrics that provides actionable business intelligence.
In R.U.M, we can then visualize performance over time for key metrics based on different average, geometric mean, median or percentile calculations. The performance data can be drilled down further on the basis of geography, device, error encountered, speed etc.
Presentattie Joomladagen 2015:
Net als bij welk CMS dan ook laden Joomla sites soms traag. Echter, ook als je site al wel redelijk snel laadt kan de laadtijd nog steeds drastisch versneld worden. je zult er versteld van staan hoe makkelijk je veel winst kan behalen. De winst valt op en aantal punten te behalen:
Zoek een goede host en optimaliseer deze omgeving
Optimaliseer Joomla
Simpele regels in .htaccess
Beperk je in extensies / externe bronnen
Optimaliseer CSS en Javascript
Gebruik optimalisatie plugins
Optimaliseer plaatjes
Optimaliseer voor mobiele devices
http://joomlaseo.com/Blog/talking-at-the-dutch-joomladagen-2015
Detail agenda + links to slides & videos from vodQA Pune - Innovations in Testing, conducted on Sat, 6th June 2015 at ThoughtWorks, Pune
http://vodqa-pune.weebly.com
http://essenceoftesting.blogspot.com/search/label/vodQA
“ The Microservices architecture has many appealing qualities, but the road towards it has painful traps for the unwary. This book will help you figure out if this path is for you, and how to avoid those traps on your journey.”
—Martin Fowler Chief Scientist, ThoughtWorks
These are my summarized notes from all the microservices session I attended at QCon 2015. These sessions had tons of learning around how to scale microservices and avoid common pitfalls
Microservices for Mortals by Bert Ertman at Codemotion DubaiCodemotion Dubai
With popular poster children such as Netflix and Amazon, Microservices based architecture seems to be the killer approach to 21st century architectures. But are they only for Hollywood Coders pioneering on the bleeding edge of our profession? Or are they ready to be used for your projects and your customers? I will go over the benefits, but more so the pitfalls, of using a Microservices based architecture. What impact does it have on your organization, your applications, on dealing with scale and failures, and how do you prevent your landscape from becoming an unmaintainable nightmare.
Microservices is the new popular kid on the block. Crowd pleaser at many conferences. With popular poster children such as Netflix and Amazon it seems to be the killer approach to 21st century architectures, right? But is this stuff only for Hollywood Coders pioneering on the bleeding edge of our profession? Or is this stuff ready to be used for your projects and your customers? This presentation is a warning. Microservices don't fix broken organizations and distributed computing is still hard. I will go over the benefits, but more so the pitfalls, of using a Microservices based architecture. What impact does it have on your applications, on dealing with scale and failures, and how do you prevent your systems landscape from becoming an unmaintainable nightmare.
Understanding MicroSERVICE Architecture with Java & Spring BootKashif Ali Siddiqui
This is a deep journey into the realm of "microservice architecture", and in that I will try to cover each inch of it, but with a fixed tech stack of Java with Spring Cloud. Hence in the end, you will be get know each and every aspect of this distributed design, and will develop an understanding of each and every concern regarding distributed system construct.
OrteliusMicroserviceVisionaries2022_Why do you need a microservice catalog to...Turja Narayan Chaudhuri
It is indeed a valid argument that microservices allows engineering organization to grow and scale, providing better constructs for isolation and independence. But, microservices do not only add technical complexity -they also introduce organizational complexity and pose a serious threat to scaling at the enterprise level.
Over time, with hundreds of services and no governance or oversight, when something breaks, nobody knows who to reach out to. A microservice catalog is a record/list of all the microservices that an enterprise has in its ecosystem, providing powerful information at your fingertips.
In this talk, you will learn how a microservice catalog facilitates a successful microservice at scale architecture by helping developers find and share services, providing DevOps teams metadata about services needed to make solid decisions, and support teams ownership and consumption information needed when something goes wrong.
It's easy to say... Microservices! Reality is we need to learn and apply concepts coming from many disciplines like SOA, EDA and DDD just to name a few! Mix them with some ALM and technical processes around Packaging and Deploying... and maybe then you get a true Microservices solution.
Whar are microservices and microservices architecture (MSA) How we reach them? Are they the same or SoA or not? When to use them? What are the key characteristics?
Slides of my talk given in #Gapand2017 in Andorra
Microservices – or micro-services – may simply be a term for something already being used, but by giving something a name and defining it, we sometimes give ourselves the advantage of recognizing a tool we perhaps should be including in our systems.
Here you find some points to think about if you're considering to use a microservice architecture in your next project. In the first part you'll find some management considerations and then some points about technologies and solutions for MS problems.
Comparing and contrasting monolithic systems to Lego pieces (microservices) at the 50,000 foot view. In this presentation we will compare and contrast monolithic systems to microservices. We will then take a look at some of the down sides to microservices. And then we will discuss some strategies for building microservices.
Similar to Advanced Microservices - Greach 2015 (20)
Spring I_O 2024 - Flexible Spring with Event Sourcing.pptxSteve Pember
Event Sourcing is a modern but non-trivial data model for building scalable and powerful systems. Instead of mapping a single Entity to a single row in a datastore, in an Event Sourced system we persist all changes for an Entity in an append-only journal. This design provides a wealth of benefits: a built-in Audit Trail, Time-Based reporting, powerful Error Recovery, and more. It creates flexible, scalable systems and can easily evolve to meet changing organizational demands. That is, once you have some experience with it. Event Sourcing is straightforward in concept, but it does bring additional complexity and a learning curve that can be intimidating. People coming from traditional ORM systems often wonder: how does one model relations between Entities? How is Optimistic Locking handled? What about datastore constraints?
Based on over eight years of experience with building ES systems in Spring applications, we will demonstrate the basics of Event Sourcing and some of the common patterns. We will see how simple it can be to model events with available tools like Spring Data JPA, JOOQ, and the integration between Spring and Axon. We’ll walk through sample code in an application that demonstrates many of these techniques. However, it’s also not strictly about the code; we’ll see how a process called Event Modeling can be a powerful design tool to align Subject Matter Experts, Product, and Engineering. Attendees will leave with an understanding of the basic Event Sourcing patterns, and hopefully a desire to start creating their own Journals.
Anatomy of a Spring Boot App with Clean Architecture - Spring I/O 2023Steve Pember
In this presentation we will present the general philosophy of Clean Architecture, Hexagonal Architecture, and Ports & Adapters: discussing why these approaches are useful and general guidelines for introducing them to your code. Chiefly, we will show how to implement these patterns within your Spring (Boot) Applications. Through a publicly available reference app, we will demonstrate what these concepts can look like within Spring and walkthrough a handful of scenarios: isolating core business logic, ease of testing, and adding a new feature or two.
SACon 2019 - Surviving in a Microservices EnvironmentSteve Pember
Many presentations on microservices offer a high-level view of the architecture; rarely do you hear what it’s like to work in such an environment. Stephen Pember shares his experience migrating from a monolith to microservices across several companies, highlighting the mistakes made along the way and offering advice.
Surviving in a Microservices environment -abridgedSteve Pember
Many presentations on Microservices offer a high-level view; rarely does one hear what it’s like to work in such an environment. Individual services are somewhat trivial to develop, but now you suddenly have countless others to track. You’ll become obsessed over how they communicate. You’ll have to start referring to the whole thing as “the Platform”. You will have to take on some considerable DevOps work and start learning about deployment pipelines, metrics, and logging.
Don’t panic. In this presentation we’ll discuss what we learned over the past four years by highlighting our mistakes. We’ll examine what a development lifecycle might look like for adding a new service, developing a feature, or fixing bugs. We’ll see how team communication is more important than one might realize. Most importantly, we’ll show how - while an individual service is simple - the infrastructure demands are now much more complicated: your organization will need to introduce and become increasingly dependent on various technologies, procedures, and tools - ranging from the ELK stack to Grafana to Kubernetes. Lastly, you’ll come away with the understanding that your resident SREs will become the most valued members of your team.
Over the past few years, Gradle has become a popular build tool in the JVM space. This is not surprising, considering the power and the features it brings, compared with its competitors. However, one thing Gradle lacks is history and the collective knowledge at the same level of other alternatives: how does one organize a Gradle project in an ‘idiomatic’ fashion?
We feel that we’ve put together a decent build pipeline for each of our microservices over the years, and each one starts with their build.gradle file(s). We’d like to share it, although we’re not sure if it’s the ‘correct’ way.
In this talk, we’ll walk through a sample project structure and build process. We’ll discuss the various checks and tools we use (e.g. Sonar, CodeNarc, Jenkins) at each step of the build. We’ll explain how each of the components in the process work for us, and share samples of our Groovy scripts. Most importantly, though, we’d like to hear what the audience are using in their builds!
Many presentations on Microservices offer a high level view; rarely does one hear what it’s like to work in such an environment. Individual services are somewhat trivial to develop, but now you suddenly have countless others to track. You’ll become obsessed over how they communicate. You’ll have to start referring to the whole thing as “the Platform”. You will have to take on some DevOps work and start learning about deployment pipelines, metrics, and logging.
Don’t panic. In this presentation we’ll discuss what we, at ThirdChannel, learned over the past four years. We’ll examine what a development lifecycle might look like for adding a new service, developing a feature, or fixing bugs. We’ll dive a bit into DevOps and see how one will become dependent on various metric and centralized logging tools, like Kubernetes and the ELK stack. Finally we’ll talk about team communication and organization… and how they are likely the most important tool for surviving a Microservices development team.
Reactive applications & reactive programming result in flexible, concise, performant code and are a superior alternative to the old thread-based programming model. The reactive approach has gained popularity for a simple reason: we need alternative designs and architectures to meet today’s demands. However, it can be difficult to shift one’s mind to think in reactive terms, particularly when one realizes that we must be Reactive up and down the entire programming stack.
In this talk we’ll explore what it means to be ‘Reactive’. We’ll examine some of the more interesting tools available to us, some of which come from the Groovy community. Specifically we’ll cover Ratpack, RxGroovy, React, and RabbitMq - along with examples and a sample implementation. We’ll demonstrate how effectively they can work together at each level of the stack - from the front end, to the back end, to handling http requests and message queue events - and how easy it can be to go Reactive all the way down.
Event storage offers many practical benefits to distributed systems providing complete state changes over time, but there are a number of challenges when building an event store mechanism. Stephen Pember explores some of the problems you may encounter and shares real-world patterns for working with event storage.
Harnessing Spark and Cassandra with GroovySteve Pember
This talk is an introduction to a powerful combination in the big data space: Apache Spark and Cassandra. Spark is a cluster-computing framework that allows users to perform calculations against resilient in-memory datasets using a functional programming interface. Cassandra is a linearly scalable, fault tolerant, decentralized datastore. These two technologies are complicated, but integrate well and provide such a level of utility that whole companies have formed around them.
In this talk we’ll learn how Spark and Cassandra can be leveraged within your Groovy Application: Spark normally asks for a Scala environment. We’ll talk about Spark and Cassandra from a high level and walk through code examples. We’ll discuss the pitfalls of working with these technologies - like modeling your data appropriately to ensure even distribution in Cassandra and general packaging woes with Spark - and ways to avoid them. Finally, we’ll explore how we at ThirdChannel are using these technologies.
Surviving in a microservices environmentSteve Pember
Many presentations on Microservices offer a high level view; rarely does one hear what it’s like to work in such an environment. Individual services are somewhat trivial to develop, but now you suddenly have countless others to track. You’ll become obsessed over how they communicate. You’ll have to start referring to the whole thing as “the Platform”. You will have to take on some DevOps work and start learning about deployment pipelines, metrics, and logging.
Don’t panic. In this presentation we’ll discuss what we learned over the past three years. We’ll examine what a development lifecycle might look like for adding a new service, developing a feature, or fixing bugs. We’ll dive a bit into DevOps and see how one will become dependent on various metric and centralized logging tools, like Kubernetes and the ELK stack. Finally we’ll talk about team communication and organization... and how they are likely the most important tool for surviving a Microservices development team.
Surviving in a Microservices EnvironmentSteve Pember
Cloud Native Microservice architectures have become increasingly popular over the past few years, and for good reasons: smaller, efficient codebases, finely targeted scaling options, and the ability to do continuous deployment along with continuous integration, among others. All potentially very powerful features. However - as with most things - Microservices bring tradeoffs in terms of application complexity: working with an individual service is easy; overall application development becomes increasingly complex. Perhaps too complex for your average web application.
Many presentations on the Microservice phenomena offer either a high level view on what it is, compare and contrast it with the Monolith pattern, or discuss how to migrate from a Monolith to Microservices, but rarely does one hear what it’s like to actually work in such an environment. Frankly, it can be intimidating for someone accustomed to a traditional monolithic development experience. Individual services are somewhat trivial to develop, but now you suddenly have countless others to keep track of. You may become lost: with all these services, is anyone directing the overall development? You’ll become obsessed over how and when they communicate. You’ll have to start referring to the application on the whole as “the Platform”. It’ll soon become difficult or even impossible to run the whole Platform on a development laptop. You may even have to take on some DevOps work, and start learning about deployment pipelines, and whole new worlds of metrics and logging.
Don’t panic. In this presentation we’ll discuss what we learned working with a Microservice platform for the past three years. We’ll cover what to expect when joining a Microservice team and what the situation will look like as the team size grows. We’ll see how critical inter-service testing strategies are to the success of the team. We’ll examine what a development lifecycle might look like for adding a new service, developing a new feature, or fixing bugs. We’ll dive a bit into DevOps and see how one will become dependent and various metric and centralized logging tools, like Kubernetes and the ELK stack. Finally we’ll talk about communication, team organization strategies, and how they are likely the most important tool for surviving a Microservices development team.
An introduction to Reactive applications, Reactive Streams, and options for t...Steve Pember
The term “reactive” has lately become a buzzword, with a variety of definitions around the Web. When you hear “reactive,” what do you think of? Reactive Streams? The Reactive Manifesto? ReactJS? These terms may seem unrelated, but they share a common core concept.
Reactive applications and Reactive programming result in flexible, concise, performant code and are a superior alternative to the old, standard thread-based imperative programming model. The Reactive approach has gained popularity recently for one simple reason: we need alternative designs and architectures to meet today’s demands. However, it can be difficult to shift one’s mind to think in Reactive terms due to how accustomed we’ve become to the imperative style.
Stephen Pember explores the various definitions of Reactive and Reactive programming with the goal of providing techniques for building efficient, scalable applications. Steve dives into the key concepts of Reactive Streams and examines some sample implementations—including how ThirdChannel is currently using reactive libraries in production code. Steve looks at some of the open source options available in the JVM—including Reactor, RxJava, and Ratpack—giving you an idea of where to begin with the reactive ecosystem. If Reactive is new to you, this should be an excellent introduction.
Are you tired of Hibernate? Is GORM is too heavy for your current project? Do you like having more control over your SQL? Do you like flexible DSLs? Try JOOQ!
JOOQ (Java Object Oriented Querying) is light-weight alternative to classic data access solutions or ORMs like Hibernate, JPA, JDBC, and GORM. JOOQ's goal is to give the developer a flexible DSL for building typesafe, database agnostic SQL queries, and attempts to convince the developer of a ‘database-first’ approach to building their application. In this talk we’ll quickly present an introduction to JOOQ from a high level, discuss its features, and see several examples of how we’re using JOOQ to great effect with many Spring Boot and Ratpack apps within our platform.
Reactive Streams and the Wide World of GroovySteve Pember
The concept of Reactive Streams (aka Reactive Extensions, Reactive Functional Programming, or simply Rx) has become increasingly popular recently, and with good reason. The Reactive Streams specification provides a universal abstraction for asynchronously processing data received across multiple sources (e.g. database, user input, third-party services), and includes mechanisms for controlling the rate at which data is received. This makes it a powerful tool within a Microservice platform. And did we mention that the Groovy lang community is quite involved?
In this talk we’ll explore the various features and concepts of Reactive Streams. We’ll talk about some typical use cases for Rx and more importantly, how to implement them. We’ll focus primarily on RxGroovy and Ratpack, then provide example implementations that show you how to get started with this powerful technique.
As businesses grow, so does the complexity of their software. New features, new models, and new background processes all continue to be added. . .and developers struggle to make sense of it all. Yet the end user demands a swift and functional experience when interacting with your application. It is paramount to be open to alternative patterns that help tame complex, high-demand services. Two such patterns are command-query responsibility segregation (CQRS) and event sourcing (ES).
Command-query responsibility segregation is an architectural pattern for user-facing applications that extends from the now standard Model-View-Controller (MVC) pattern and is an alternative to the CRUD pattern. At its core, CQRS is about changing how we think of and work with our data by introducing two types of models: all user actions become commands, and a read-only query model powers our views. Commands and queries are logistically separated, providing additional decoupling of our application. CQRS also calls for changes in how we store and structure our data.
Enter event sourcing. Instead of persisting the current state of our domain objects or entities, we record historical events about our data. The key advantage is that we can examine our application data at any point in time, rather than just the current state. This pattern changes how we persist and process our data but is surprisingly efficient.
While each of the two patterns can be used exclusively, they complement each other beautifully and facilitate the construction of decoupled, scalable applications or individual services. Stephen Pember explores the fundamentals of each pattern and offers several examples and demonstration code to show how one might actually go about implementing CQRS and ES. Steve discusses task-based UIs and domain-driven design as he outlines some of the advantages—and challenges—that ThirdChannel has seen when developing systems using CQRS and ES over the past year.
An Introduction to Reactive Application, Reactive Streams, and options for JVMSteve Pember
The term “reactive” has lately become a buzzword, with a variety of definitions around the Web. When you hear reactive, what do you think of? Reactive Streams? The Reactive Manifesto? ReactJS? These terms may seem unrelated, but they share a common core concept.
Reactive applications and reactive programming result in flexible, concise, performant code and are a superior alternative to the old standard thread-based imperative programming model. The reactive approach has gained popularity recently for one simple reason: we need alternative designs and architectures to meet today’s demands. However, it can be difficult to shift one’s mind to think in reactive terms due to how accustomed we’ve become to the imperative style.
Stephen Pember explores the various definitions of reactive and reactive programming with the goal of providing techniques for building efficient, scalable applications. Steve dives into the key concepts of Reactive Streams and examines some sample implementations—including how ThirdChannel is currently using reactive libraries in production code. Steve looks at some of the open source options available in the JVM—including Reactor, RxJava, and Ratpack—giving attendees an idea of where to begin with the reactive ecosystem. If reactive is new to you, this should be an excellent introduction.
Richer Data History with Event Sourcing (SpringOne 2GX 2015Steve Pember
A common pattern in application development is to build systems where the data is directly linked to the current state of the application; one row in the database equates to one entity’s current state. Only ever knowing the current state of the data is adequate for many systems, but imagine the possibilities if one had access to the state of the data at any point in time. Enter Event Sourcing: instead of persisting the current state of our Domain Objects or Entities, we record historical events about our data. This pattern changes how we persist and process our data, but is surprisingly lightweight. In this talk I will present the basic concepts of Event Sourcing and the positive effects it can have on analytics and performance. We’ll discuss how storing historical events provides extremely powerful views into our data at any point in time. We’ll see how naturally it couples with the Event-oriented world of modern Reactive systems, and how easily it can be implemented in Groovy. We’ll examine some practical use cases and example implementations in Ratpack. Event Sourcing will change how you think about your data.
Springone2gx 2015 Reactive Options for GroovySteve Pember
Reactive applications and Reactive programming are an alternative to the standard thread-based imperative programming model that can result in flexible, concise code. The Reactive approach has gained popularity recently for one simple reason: we need alternative designs and architectures to meet today’s demands. However, it can be difficult to shift one’s mind to think in Reactive terms. It doesn’t help that the descriptions around the web can be contradictory and the library documentation can be obscure. In this talk, we’ll explore the concepts of Reactive and Reactive Programming. We’ll demonstrate some of the useful Reactive functions and examine some practical implementations - including how we’re currently using Reactive libraries in production code. Most importantly, we’ll look at some of the open source options available to us in the Groovy community, including Reactor, RxJava, and Ratpack. If Reactive is new to you, this should be an excellent introduction.
Gr8conf US 2015 - Intro to Event Sourcing with GroovySteve Pember
As Grails developers we generally build systems where our data is directly linked to the current state of our application; one row in the database equates to one entity’s current state. Only ever knowing the current state of the data is adequate for many systems, but imagine the possibilities if one had access to the state of the data at any point in time.
Enter Event Sourcing: instead of persisting the current state of our Domain Objects, we store historical events about our data. This pattern changes how we store and process our data, but is surprisingly lightweight and performant. In this talk I will present the basic concepts of Event Sourcing and the positive effects it can have on analytics and performance. We’ll see how naturally it couples with the Event-oriented world of modern Reactive systems, and how easily it can be implemented in Groovy. We’ll examine some practical use cases and example implementations. Event Sourcing will change how you think about your data.
Utilocate offers a comprehensive solution for locate ticket management by automating and streamlining the entire process. By integrating with Geospatial Information Systems (GIS), it provides accurate mapping and visualization of utility locations, enhancing decision-making and reducing the risk of errors. The system's advanced data analytics tools help identify trends, predict potential issues, and optimize resource allocation, making the locate ticket management process smarter and more efficient. Additionally, automated ticket management ensures consistency and reduces human error, while real-time notifications keep all relevant personnel informed and ready to respond promptly.
The system's ability to streamline workflows and automate ticket routing significantly reduces the time taken to process each ticket, making the process faster and more efficient. Mobile access allows field technicians to update ticket information on the go, ensuring that the latest information is always available and accelerating the locate process. Overall, Utilocate not only enhances the efficiency and accuracy of locate ticket management but also improves safety by minimizing the risk of utility damage through precise and timely locates.
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus
As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and we describe a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.
Large Language Models and the End of ProgrammingMatt Welsh
Talk by Matt Welsh at Craft Conference 2024 on the impact that Large Language Models will have on the future of software development. In this talk, I discuss the ways in which LLMs will impact the software industry, from replacing human software developers with AI, to replacing conventional software with models that perform reasoning, computation, and problem-solving.
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...Globus
Large Language Models (LLMs) are currently the center of attention in the tech world, particularly for their potential to advance research. In this presentation, we'll explore a straightforward and effective method for quickly initiating inference runs on supercomputers using the vLLM tool with Globus Compute, specifically on the Polaris system at ALCF. We'll begin by briefly discussing the popularity and applications of LLMs in various fields. Following this, we will introduce the vLLM tool, and explain how it integrates with Globus Compute to efficiently manage LLM operations on Polaris. Attendees will learn the practical aspects of setting up and remotely triggering LLMs from local machines, focusing on ease of use and efficiency. This talk is ideal for researchers and practitioners looking to leverage the power of LLMs in their work, offering a clear guide to harnessing supercomputing resources for quick and effective LLM inference.
Globus Connect Server Deep Dive - GlobusWorld 2024Globus
We explore the Globus Connect Server (GCS) architecture and experiment with advanced configuration options and use cases. This content is targeted at system administrators who are familiar with GCS and currently operate—or are planning to operate—broader deployments at their institution.
AI Pilot Review: The World’s First Virtual Assistant Marketing SuiteGoogle
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
👉👉 Click Here To Get More Info 👇👇
https://sumonreview.com/ai-pilot-review/
AI Pilot Review: Key Features
✅Deploy AI expert bots in Any Niche With Just A Click
✅With one keyword, generate complete funnels, websites, landing pages, and more.
✅More than 85 AI features are included in the AI pilot.
✅No setup or configuration; use your voice (like Siri) to do whatever you want.
✅You Can Use AI Pilot To Create your version of AI Pilot And Charge People For It…
✅ZERO Manual Work With AI Pilot. Never write, Design, Or Code Again.
✅ZERO Limits On Features Or Usages
✅Use Our AI-powered Traffic To Get Hundreds Of Customers
✅No Complicated Setup: Get Up And Running In 2 Minutes
✅99.99% Up-Time Guaranteed
✅30 Days Money-Back Guarantee
✅ZERO Upfront Cost
See My Other Reviews Article:
(1) TubeTrivia AI Review: https://sumonreview.com/tubetrivia-ai-review
(2) SocioWave Review: https://sumonreview.com/sociowave-review
(3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review
(4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review
Quarkus Hidden and Forbidden ExtensionsMax Andersen
Quarkus has a vast extension ecosystem and is known for its subsonic and subatomic feature set. Some of these features are not as well known, and some extensions are less talked about, but that does not make them less interesting - quite the opposite.
Come join this talk to see some tips and tricks for using Quarkus and some of the lesser known features, extensions and development techniques.
Understanding Nidhi Software Pricing: A Quick Guide 🌟
Choosing the right software is vital for Nidhi companies to streamline operations. Our latest presentation covers Nidhi software pricing, key factors, costs, and negotiation tips.
📊 What You’ll Learn:
Key factors influencing Nidhi software price
Understanding the true cost beyond the initial price
Tips for negotiating the best deal
Affordable and customizable pricing options with Vector Nidhi Software
🔗 Learn more at: www.vectornidhisoftware.com/software-for-nidhi-company/
#NidhiSoftwarePrice #NidhiSoftware #VectorNidhi
GraphSummit Paris - The art of the possible with Graph TechnologyNeo4j
Sudhir Hasbe, Chief Product Officer, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Zoom is a comprehensive platform designed to connect individuals and teams efficiently. With its user-friendly interface and powerful features, Zoom has become a go-to solution for virtual communication and collaboration. It offers a range of tools, including virtual meetings, team chat, VoIP phone systems, online whiteboards, and AI companions, to streamline workflows and enhance productivity.
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxrickgrimesss22
Discover the essential features to incorporate in your Winzo clone app to boost business growth, enhance user engagement, and drive revenue. Learn how to create a compelling gaming experience that stands out in the competitive market.
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisGlobus
JASMIN is the UK’s high-performance data analysis platform for environmental science, operated by STFC on behalf of the UK Natural Environment Research Council (NERC). In addition to its role in hosting the CEDA Archive (NERC’s long-term repository for climate, atmospheric science & Earth observation data in the UK), JASMIN provides a collaborative platform to a community of around 2,000 scientists in the UK and beyond, providing nearly 400 environmental science projects with working space, compute resources and tools to facilitate their work. High-performance data transfer into and out of JASMIN has always been a key feature, with many scientists bringing model outputs from supercomputers elsewhere in the UK, to analyse against observational or other model data in the CEDA Archive. A growing number of JASMIN users are now realising the benefits of using the Globus service to provide reliable and efficient data movement and other tasks in this and other contexts. Further use cases involve long-distance (intercontinental) transfers to and from JASMIN, and collecting results from a mobile atmospheric radar system, pushing data to JASMIN via a lightweight Globus deployment. We provide details of how Globus fits into our current infrastructure, our experience of the recent migration to GCSv5.4, and of our interest in developing use of the wider ecosystem of Globus services for the benefit of our user community.
Enhancing Research Orchestration Capabilities at ORNL.pdfGlobus
Cross-facility research orchestration comes with ever-changing constraints regarding the availability and suitability of various compute and data resources. In short, a flexible data and processing fabric is needed to enable the dynamic redirection of data and compute tasks throughout the lifecycle of an experiment. In this talk, we illustrate how we easily leveraged Globus services to instrument the ACE research testbed at the Oak Ridge Leadership Computing Facility with flexible data and task orchestration capabilities.
2. THIRDCHANNEL
* Before I begin, I should probably point out there’s not much in this talk that specifically relates to Groovy
* In fact, there’s no code in this talk at all. Mostly just theory.
* Still, I believe the information is useful. Microservices are all the rage.
* Incidentally, At 3C we love microservices, and Groovy is our most used language in our services
3. THIRDCHANNEL
Agenda
• Microservices: Overview
• Microservices: Advantages
• Service Design
• Organizational Structure
• Dev Ops
The agenda for this presentation first is to briefly cover the microservice architectural pattern, some of it’s advantages,
and then get into some of the advanced concerns, where I’ve grouped them by overall topic… Service Design, Organizational Structure, and Dev ops!
4. THIRDCHANNEL
Agenda
• Microservices: Overview
* To begin:
* Who here has heard of ‘Microservices’? Just trying to measure how to talk about this
* Who here is currently working with or has worked on a Microservices platform?
5. THIRDCHANNEL
Microservices
• Fight the Monolith
So let’s get started with a quick overview of what Microservices actually are.
I believe microservices are an attempt to fight against a distressing pattern in software architecture: the monolithic application
6. THIRDCHANNEL
Building a monolith has become, I think, the natural response for many developers or companies when they first start out on a project
7. THIRDCHANNEL
-A monolithic app is: Single, logical executable unit, often comprised of 3 components - at least for a web app: the View layer (CSS/HTML/JS), the middle
tier (contains business logic) and the data storage.
- here’s an example of an e-comm app’s middle tier: product browsing, catalog management, order processing, shopping cart
- as I add features to my application, I simply tack them on to the existing code base. Feature upon Feature upon feature
8. Feels Natural
The danger with the monolith I think, is that this feels like the natural thing to do.
You start a new project, choose a framework and begin building.
9. start out easy
quick wins as you add new features
“Oh, I need a shopping cart now. Great, let’s shove in a Cart model and a Controller. Maybe service”
…in the long term, though, you’ll run into some problems
NEED TO GO BACK AND ADD BITS ABOUT HOW MONOLITHS GROW
10. complexity grows quickly as you keep adding new features!
Even if you’re very, very careful… application will still inevitably become a mess
11. –Johnny Appleseed
“Type a quote here.”
To the point where it leads to Developer Misery.
Open up your IDE, only to be bogged down by the countless controllers or View objects popping out at you. Quick hacks and other Technical Debt that
promised yourself you would go back and fix … and you never do… make the system hard to develop.
When adding new features or fixing bugs, developers will spend huge amounts of time figuring out how something works before they can even start fixing a
problem. May require dragging in other people who worked on it before to help.
12. Microservices to the Rescue!
This frustration - of working on large, monstrous applications - , I think, is why Microservices have become so popular.
13. THIRDCHANNEL
Microservices
• Fight the Monolith
• Distributed
So, when we use that term “Microservice”, what exactly are we talking about?
Well, fundamentally what it means is that we’re building a distributed application
15. What About ‘Service Oriented
Architecture’?
Now you may be saying:
“Wait, isn’t this just Service Oriented Architecture? People have done this before.”
16. * It’s true, this has been done before. SOA has been around for years now
* Microservices, though, are a more focused and disciplined version of SOA.
* REASON BEING: Involves technical and organizational changes not defined in SOA. Let’s go over a few of them:
17. THIRDCHANNEL
Microservices
• Fight the Monolith
• Distributed
• Independent
First: Independence.
In a Microservice system, Each service should be completely independent of each other
18. THIRDCHANNEL
Independently developed. Each with their own separate code repository.
Independently deployed.
- allows for independent history
- results in isolated code which is great for adhering to design principles like separation of concerns.
-
19. THIRDCHANNEL
Microservices
• Fight the Monolith
• Distributed
• Independent
• Single Context
Next.
Each Service contains a single context…. Or rather, each service is responsible for One thing.
This one thing could be a particular set of data or a particular process.
This approach naturally leads to smaller, less complicated codebases
20. THIRDCHANNEL
Let’s say I was building an e-commerce application. I may have… <name services>
Each one responsible for a single object or a single process
Each one of them, will be developed separately from each other and will be each far less complicated than if we tried to jam each service together into a
single monolith
21. THIRDCHANNEL
Microservices
• Fight the Monolith
• Distributed
• Independent
• Single Context
• Simple, “Dumb”, Communication Technology
Your services should communicate with each other using very simplistic channels, but be very smart about how they do so.
<slow down>
22. No Enterprise Service Buses
Which basically means “No enterprise Service Buses” or ESBs.
You would often find ESBs in Service Oriented Architecture or SOA implementations. They are complicated services that are responsible for figuring out
how to route messages to the various services.
In practice, they Would often be very complex with custom logic; another thing to maintain
23. Instead, Use HTTP or Message Queues
Instead, use direct HTTP calls or Message Queues
* Http: Already in the web, so why not use a web technology? HTTP allows for Direct communication between services, but it’s synchronous… so, while
the request is active, both services have resources that locked in the communication
* Message Queues on the other hand: Asynchronous, but require some additional code or library for messaging formats
* Regardless, Each approach does require a lightweight program to help with the communication.
24. THIRDCHANNEL
So with HTTP based comms and a handful of services, you may end up with something like this:
web of inter service communications. Each service needs to know the ‘address’ of the other services, along with when it needs to talk to the others.
So, if a user places an order, that code needs to know to talk to the order history service and the billing service in a particular order.
25. THIRDCHANNEL
* With HTTP, you’ll need a service to help you discover other services, otherwise each service would also have to maintain a huge internal map of the
locations of the other services.
* A “Service Discovery Service”
* Netfix has open sourced one, called “Eureka” which is very useful.
* With all of that, HTTP is very commonly used in Microservice Architectures.
26. THIRDCHANNEL
With a message Oriented approach on the other hand..
- services don’t know where each other are, nor do they care where the other services are.
- communicate entirely through the the Message Broker
- When an event needs to go out, it is simply generated. The services do not need to individually worry about which other service needs this information.
- Personally, I prefer this approach for a variety of reasons, which I’d be happy to talk about after.
28. THIRDCHANNEL
Microservices
• Fight the Monolith
• Distributed
• Independent
• Single Context
• Simple, “Dumb”, Communication Technology
• REST-ful Communications
Your services should communicate with each other using very simplistic channels, but be very smart about how they do so.
29. REST != CRUD != HTTP
I feel that when many people think of REST or say that they have a “RESTful api”, they’re thinking of performing CRUD operations (create / read / update /
delete) over HTTP, using some of the HTTP verbs.
30. THIRDCHANNEL
There are many more aspects to REST than that.
Primarily, I think, no State should be shared across services.
Attempting to keep shared state across distributed services WILL drive you crazy.
31. Caching, Layering, and Uniform
Interface
the REST specification also calls for caching your data, layering your services, and ensuring they have a uniform interface.
- Layering: when your services ask for or push information, they shouldn’t care whether who they’re talking to is the actual service or some intermediary
that’s then passing it on. As long as the data is handled, the services should trust that the handlers know what to do.
- Uniform interface implies many things… I’m going to take a shortcut and just say that each of your services should communicate with each other via the
same technique. If anyone wants to argue about HATEOAS later, I’d be happy to.
32. THIRDCHANNEL
Microservices
• Fight the Monolith
• Distributed
• Independent
• Single Context
• Simple, “Dumb”, Communication Technology
• REST-ful Communications
• Dedicated, Cross-Purpose Teams
lastly,
… from an organizational standpoint, the teams building the service should be cross-purpose.
33. THIRDCHANNEL
Teams in charge of a service should come from a variety of disciplines. Engineering, QA, Art & Design, Project Management, perhaps a deployment
engineer, etc.
34. THIRDCHANNEL
Agenda
• Microservices: Overview
• Microservices: Advantages
* Next on the agenda: Microservice architectures have been gaining huge followings in the past few years due to the benefits they bring.
35. THIRDCHANNEL
Microservices: Advantages
• Loose Coupling!
Because our systems are independent of each other, this promotes natural Loose Coupling. Individual teams can develop the services independently,
which keeps complexity down.
36. THIRDCHANNEL
Microservices: Advantages
• Loose Coupling!
• Right Tool for the Job
* Each service can be built according to the technology best suited to it.
* Not locked into one language or framework
37. Always use Groovy, naturally
But of course, Groovy is always the right tool for the job!
38. THIRDCHANNEL
Microservices: Advantages
• Loose Coupling!
• Right Tool for the Job
• Easy to Scale Services
It’s easy to scale Microservice applications to meet the particular user demand
39. –Johnny Appleseed
“Type a quote here.”
Borrowed this from Martin Fowler’s website
As demand on the application grows, I can scale just the components that are needed, rather than needing to replicate the whole monolith
* targeted growth results in your team using fewer resources
* which saves $$$ and makes your team more efficient
40. THIRDCHANNEL
Microservices: Advantages
• Loose Coupling!
• Right Tool for the Job
• Easy to Scale Services
• Easy to Scale Development
Speaking of scaling…
Scaling development in general is very difficult. With a microservices architecture, you have an advantage where it’s relatively easy for developers.
Because the code base is smaller, your developers can hold the context of a service in their head at once (something impossible with a monolith). This
allows them to find bugs quickly and identify exactly where new features should go.
41. THIRDCHANNEL
Microservices: Advantages
• Loose Coupling!
• Right Tool for the Job
• Easy to Scale Services
• Easy to Scale Development
• Fast to Fail / Start Over
Microservices allow us to fail fast.
Find out quickly of a technology or a service isn’t working.
Easy to refactor or replace a service
…
42. THIRDCHANNEL
Microservices: Advantages
• Loose Coupling!
• Right Tool for the Job
• Easy to Scale Services
• Easy to Scale Development
• Fast to Fail / Start Over
• Allows for Continuous Delivery
Finally, Adopting a Microservices approach allows for your team to Continuously Deliver your code.
43. THIRDCHANNEL
Which, I think, is the absolute ultimate goal of any development team. I think it’s the main reason to use microservices to begin with.
The ability to basically make a change or implement a feature and then immediately have it running in production? It’s thrilling, and powerful, and liberating
for your developers.
Furthermore, It essentially eliminates the notion of a Sprint and dramatically changes Agile Development.
44. Goodbye, Release Windows
Who here has worked in teams where you have a development release window or a ‘release train’? I have. Who here was worked in companies where you
release a new build every 3 or 4 weeks… or longer? I certainly
It was miserable, it was depressing, to stack up a bunch of features for release. Everyone was a little unsure if the changes they made broke other people’s
code. If a bug was detected in the wild, it wouldn’t be fixed until the next release. Madness.
45. Microservices Have
Disadvantages
That all being said, Microservices do have their downsides. Like any new buzzword or new hot thing in tech, people will complain about it or point out any
problems it has.
Microservices do have a few complaints, although I don’t agree with all of these.
Some of the complaints I’ve seen have been:
46. THIRDCHANNEL
Microservices: Problems
• Distributed Systems are Hard
Working with distributed systems is difficult. Or really, it’s difficult to wrap your mind around a distributed system.
And yeah, it can be hard to worry about multiple small services.
47. THIRDCHANNEL
Microservices: Problems
• Distributed Systems are Hard
• Added Complexity
Splitting your code into services can add additional operational complexity, as now we have more moving parts in our platform.
I disagree with this one, breaking your code down reduces programming & computational complexity
48. THIRDCHANNEL
Microservices: Problems
• Distributed Systems are Hard
• Added Complexity
• “What if a service goes down?”
You may here someone in your organization say something like “What if service X goes down? It’s central to our business!”
Well, That person is not ready for microservices.
49. THIRDCHANNEL
Microservices: Problems
• Distributed Systems are Hard
• Added Complexity
• “What if a service goes down?”
• Significant Operations / Dev Ops Work
Microservices will require some additional focus on dev ops related work, like streamlining deployments.
50. These Can All Be Mitigated
These points, while valid, can be addressed, reduced, or handled pretty well. These next sections cover some of these ‘Advanced’ concerns when working
with Microservices. These are things that everyone should be aware of.
51. THIRDCHANNEL
Agenda
• Microservices: Overview
• Microservices: Advantages
• Service Design
* First up: Service design!
* As we’re building our individual services, what should we be aware of?
52. THIRDCHANNEL
Service Design
• Single Bounded Context
I’ll reiterate here that each service needs to be a single bounded context.
It’s a bit tricky, but this point basically states how large a service should be… which is: a service should be small enough that a single developer can hold
the entire functionality of it within their head.
54. they should embrace and welcome that errors will occur within the platform… and they should know how to handle any potential error.
For example, if our order billing system goes down… instead of shutting down the whole website, we simply accumulate orders, allow our users to go on
as normal, and hope the system comes back up eventually.
57. –Johnny Appleseed
“Type a quote here.”
You should strive to return a result to your Request as fast as you can. In the US we have an idiom called “hot potato” which means to get read of
something as fast as you can.
It’s important that your back end services return a result as quickly as possible in order to
Speed is king.
Several years ago, an engineer with Amazon named Greg Linden revealed that the company ran experiment where they intentionally slowed down site
performance
- 100ms delay resulted in 1 percent drop in sales.
58. THIRDCHANNEL
Service Design
• Single Bounded Context
• Resilient
• Fast and Efficient
• Each Service is an Authority
Each service should be responsible for one particular thing within your application. This relates to single bounded context.
Each service should be an Authority on some data… or an authority on a particular process. For example, I may have a ‘User Service’ that is the source of
information on Users within my system… or I could have an Email Notification Service which is responsible for sending emails
59. THIRDCHANNEL
Service Design
• Single Bounded Context
• Resilient
• Fast and Efficient
• Each Service is an Authority
• Embrace Eventual Consistency
Embrace Eventual consistency. Does everyone know about this? basically, in a distributed system, you must accept that an individual client or service may
not have the most recent view of the data… but that eventually, every service will have data consistent with the authority service.
This is a very large subject and I’d love to talk about some of the nuances about it later.
60. THIRDCHANNEL
Service Design
• Single Bounded Context
• Resilient
• Fast and Efficient
• Each Service is an Authority
• Embrace Eventual Consistency
• CAP Theorem
These next two topics are perhaps the most difficult in this talk. At least, very difficult to explain.
First up, the CAP Theorem
61. THIRDCHANNEL
CAP Theorem
• (C)onsistency
• (A)vailability
• (P)artition Tolerance
The CAP Theorem states that there are 3 three attributes or guarantees that a Distributed System can have:
- consistency: when an update occurs in our system, all services know about it at the same time (which is the opposite of eventual consistency)
- availability: when information is asked of the service, it responds as quickly as it can
- partition tolerance: a service is able to handle or deal with failures occurring in other services
62. THIRDCHANNEL
CAP Theorem
• (C)onsistency
• (A)vailability
• (P)artition Tolerance
• Pick 2!
Well, the CAP Theorem says that’s great, and in fact you can only have two of these. A distributed system can only meet two of these guarantees.
63. THIRDCHANNEL
CAP Theorem
• (C)onsistency
• (A)vailability
• (P)artition Tolerance
• Pick 2!
• Theorem from Paper which proved “Brewer’s Conjecture”
Originally called Brewer’s Conjecture
64. THIRDCHANNEL
In 2012, this paper contained a formal proof of the Conjecture and it graduated to full-on theorem.
65. THIRDCHANNEL
CAP Theorem
• (C)onsistency
• (A)vailability
• (P)artition Tolerance
• Pick 2!
• Theorem from Paper which proved “Brewer’s Conjecture”
• CAP Decisions can be Service-Specific
67. THIRDCHANNEL
Service Design
• Single Bounded Design
• Resilient
• Fast and Efficient
• Each Service is an Authority
• Embrace Eventual Consistency
• CAP Theorem
• Locality of Reference (Data Locality)
Last… let’s talk about Data Locality.
You should be very concerned about Data Locality
68. THIRDCHANNEL
Data Locality
• Spatial: “How far away or how distant is our data?”
There are two aspects of data locality:
1. Spatial. How proximal is our data? In a single data base, data can be considered highly spatial if it’s in the same table. Data which is reachable by joins is
less spatial and perhaps less efficient to reach
69. THIRDCHANNEL
Data Locality
• Spatial: “How far away or how distant is our data?”
• Temporal: “How often is our data accessed?”
2. Temporal. Being highly temporal means that data is read frequently. Highly temporal data - is an excellent candidate for caching.
These terms take on slightly different meanings in a distributed system, though.
70. THIRDCHANNEL
In an ideal distributed environment, each system would be completely separate.
-However, there will always be some conceptual overlap between domain objects. For example, let’s say we have an ecommerce system where
<describe system>.
-Red lines are anchors between data localities
- Here, both the user service and the O history service maintain a concept of user. But, users’ uuid is a shared natural key which acts as each
services’ anchor to understanding the concept of a User.
- Our user service is the authoritative source on a User
- Our order history system also understands the User, but simply maintains the order history along with the user uuid of the user placing the order
71. THIRDCHANNEL
Same chart from before, but another service, communication via email. This service also conceptualizes the user, but with a different subset of
information. Namely, the email address.
- Email address is stored in the user information service, but is *also* used extensively by the communication service. Which one should be the
authoritative source of the email address? The communication service uses it more (and thus has higher Temporal locality), but email is a key piece
of information which should be managed by the user service thus keeping it Spatial close to the other user data.
- Tough Choice
- One Compromise is to (unfortunately) synchronize the email address across multiple services
- Or, communication service has no concept of User’s email address, but simply blindly sends emails out. The email addresses involved must be
placed into each message received by the Communication service
- Or, third option, perhaps we cache the email address on the communication service and wait for an event or notification from the user service
when it changes (Eventual consistency, eh? When the user changes their email address, the communication service eventually, busts the cache and
resets the email address. The user may miss an email until that happens, and that may be ok).
72. THIRDCHANNEL
Data Locality
• Spatial: “How far away or how distant is our data?”
• Temporal: “How often is our data accessed?”
• Embrace Eventual Consistency, Do not sync data!
Remember the point from earlier though… embrace eventual consistency
My recommendation is to never sync data. It will likely be inevitable that you’ll eventually have to synchronize something
73. Do NOT Synchronize
I believe that synchronizing data will eventually always break down and cause no end of problems. Inevitably data will become out of sync; you will have to
create some jobs or processes to re-sync any data which has ‘drifted’.
We’ve tried synchronizing data at ThirdChannel in the past, and data discrepancies were the the primary cause of bug reports. It was terrible.
If you overhear anyone on your team say “Well, we *could* just synchronize these two things”, roll up a newspaper and whack them on the nose like a dog.
74. THIRDCHANNEL
Data Locality
• Spatial: “How far away or how distant is our data?”
• Temporal: “How often is our data accessed?”
• Embrace Eventual Consistency, Do not sync data!
• Goal: Have Highly Spatial Data & Efficiently Handle Temporal Data
With a MS, your goal is to have..
77. THIRDCHANNEL
DBA
Engineers
QA
UX
In many companies, it’s a common pattern to have your staff broken up into massive departments: a QA department, a UX team, your engineers, and the
dreaded DBAs…
Instead, with microservices, it is encouraged to break up that thinking. Build teams around your services such that it’s comprised of multiple people of
multiple disciplines.
One of the best working experiences I had was with an awesome QA person that sat right next to me… finish story
78. THIRDCHANNEL
Organizational Changes
• Cross-Discipline Teams
• Dedicated & Responsible
With that, each team is ultimately responsible for their individual service(s).
When I say I responsible, I mean it. If the service malfunctions over the weekend, the people on that team should be receiving notifications about it. There’s
not a team of junior engineers or interns around to fix it. If you commit code, you’re responsible for fixing it.
What ends up happening, is that the engineers end up writing a hell of a lot of unit and integration tests in order to avoid the dreaded early am phone call.
80. –Johnny Appleseed
“Type a quote here.”
-But, They also need to be aligned well with the business. Essentially, provide the end goal, and let the team figure out how best to get there.
-I borrowed this drawing from a Spotify blog post on their engineering culture, and it shows the concept of a highly Aligned, highly Autonomous team.
Describe the top right
82. * These people will need to have the vision to direct people. To guide them. This leadership team will need to have a shared vision and design from which
they can guide the rest of the company.
*If you start down the Microservices path, you will have some people in house who disagree with it. They’ll complain for various reasons I mentioned earlier.
*What they’re really saying is that this new approach is too hard and they’re afraid. The leadership team must be there to guide people forward and to
educate.
* And to encourage communication
83. THIRDCHANNEL
Organizational Changes
• Cross-Discipline Teams
• Dedicated & Responsible
• Autonomous
• Microservice Champion(s)
• Conway’s Law
which brings us to Conway’s Law.
I talked about Conway’s law last year.. I think at Great.
I was at a conference a couple weeks ago, and it was the subject of several talks.
So… I just want to tell you all I’m a bit of a trendsetter, leading the pack.
84. -Melvin Conway, 1968
http://en.wikipedia.org/wiki/Conway's_law
“organizations which design systems ... are constrained to
produce designs which are copies of the communication
structures of these organizations”
Somewhat tongue and cheek, but basically says that the structure of any given computer system will begin to reflect the social structures of the
company that built it.
While yes, teams should be organized around services, these services still need to communicate with one another.
85. If Conway’s Law is not observed, some interesting things can happen.
You may end up with situations where:
- intra service APIs not matching up at all
- teams expecting data to be in services, that may not be there. A misunderstanding of what a service provides
- team A waiting for Team B to finish service without bothering to help them
- eventually you may get a point where teams start blaming or attacking each other for problems in the system. There can be arguments. “Man, the team
working on the Shopping Cart Service is terrible. Their code is always broken”
86. Hard Problem to Solve
Avoiding Conway’s Law is not easy, and there’s no clear way to do it. Try your best to encourage communication and collaboration between teams.
Rotate people between service teams
Get everyone on a shared communication channel (Slack is a popular choice)
Ensure that each service’s API is published and highly visible to other teams.
88. Because each team is responsible for their deployments, nearly everyone in your team is going to get some Dev Ops experience.
Whether they like it or not….
However, there are several things you can do to make the Dev Ops experience better, and make it easier to manage your application
90. THIRDCHANNEL
Receives Metrics, Logs and Health checks
One absolute requirement of a successful microservices deployment is to have detailed monitoring of your services.
This monitoring service is responsible for performing heartbeat checks on your services and gathering metrics about them. It should also be responsible
acting as a centralized logging platform for each of your services.
If you think this may end up being an absolute ton of data… well, you’re right.
91. THIRDCHANNEL
There’s many applications or paid services available to store and visualize all of this data. However, many people I speak to recommend the Open Source
‘ELK stack’. Which is LogStash for parsing your logs, elastic search for enabling search on your data, and Kibana for visualization
92. No SSH == You’re doing it right
If you can get data from services and be aware of what they’re doing… and you *never* have to ssh into your services… well, then, you’re doing it right.
I read somewhere that some companies will actually block the ssh ports (22?) on their instances or their Amazon VPC to prove this point
93. THIRDCHANNEL
Dev Ops
• Centralized Service Monitoring & Logging
• Standardize your Deployment Process
Even if your services use different languages or frameworks, do your best to make sure everyone is using the same general approach to deployment.
97. THIRDCHANNEL
Dev Ops
• Centralized Service Monitoring & Logging
• Standardize your Deployment Process
• Small, Fast Deployments
• Continuous Deployment!
With continuous Deployment, your engineers should be releasing as soon as things are ready. Push a feature when it’s complete.
98. Beware “I have to deploy Service A before
Service C or else Service B will break”
Wasn’t sure how to diagram this.
The only thing that should concern your team are the moments where a release has dependencies on other services… E.g. I have to deploy Service C
before I deploy service A or else service B will break.
However, these cases should be RARE
99. THIRDCHANNEL
The day to day deployments, though, will be amazing.
It may be a lot of work to get to the point where you can continuous deploy, but once you get there and are able to do it… well, it feels great.
100. In Summary:
So that’s really all I have, but before I stop, here is a brief summary
101. THIRDCHANNEL
Laws of the Microservice
1. Be Independent
2. Communicate with your coworkers
3. Keep Data Close and Cache it Well
4. Do Not Sync
5. Monitor and Visualize
6. Work towards Continuous Deployment
Be Independent, but at the same time, Communicate with your coworkers. A well designed system requires constant communication
Keep your data close and cache it … but do not sync any data
Get some excellent monitoring in place
Reach for Continuous Deployment!