Integrating microservices and taming distributed systems is hard. In this talk I will present three challenges we observed in real-life projects and discuss how to avoid them using Open Source orchestration. Communication is complex. With everything being distributed failures are normal, so you need sophisticated failure handling strategies (e.g. stateful retry). A synchronicity requires you to handle timeouts. This is not only about milliseconds, systems get much more resilient when you can wait for minutes, hours or even longer. Distributed transactions cannot simply be delegated to protocols like XA. So, you need to solve the requirement to retain consistency in case of failures. I will not only use slides but demonstrate concrete source code examples available on GitHub.

1. The Benefits of Orchestration
in Distributed Systems
@NiallDeehan

2. Who I am?
Niall Deehan

3. Distributed systems
Challenges of
asynchronicity
Distributed
Transactions
Communication
is complex

4. Do A
Do B
All or
nothing
+
try {
tx.begin();
doA();
doB();
tx.commit();
} catch (Exception e) {
tx.rollback();
}
@Transactional
public void createCustomer(Customer cust) {
// ...
}
Or simply:
Once upon a time:

5. A
C
I
D
Atomicity
Consistency
Isolation
Durability

6. Distributed systems

7. Distributed systems

8. Some
Service
Some
Service
Some
Service
Some
Service
Some
Service
Some
Service
Some
Service
Failure will happen.
Accept it!
But keep it local!
Be resilient.

9. Communication
is complex

10. Let‘s start with a simple example
Credit
Card
Payment
REST

11. Live hacking
https://github.com/flowing/flowing-retail/blob/master/payment-
rest/src/main/java/io/flowing/retail/payment/port/resthacks/PaymentRestHacksControllerV1.java

12. It‘s a blocking request.
It doesn‘t scale

13. Starbucks does not use two phase commit
Gregor Hohpe https://www.enterpriseintegrationpatterns.com/ramblings/18_starbucks.html
Photo by John Ingle

14. That means
Do A
Do B
Temporarily
inconsistent
Eventually
consistent
again
t
Consistent
Local
ACID
Local
ACID
1 (micro-)service
1 aggregate
1 program
1 resource
Violates „I“
of ACID

15. Youmightknowthisfrom:
Do A
Do B
Temporarily
inconsistent
Eventually
consistent
again
t
Consistent
Photo by Gerhard51, available under Creative Commons CC0 1.0 license.

16. It is impossible to
differentiate certain
failure scenarios:
Independant of
communication style!
Service
Provider
Client

17. Network problems
Credit
Card
Payment
charge

18. Let‘s start with a simple example
Credit
Card
Payment
REST

19. Circuit
Breaker
Photo by CITYEDV, available under Creative Commons CC0 1.0 license.

20. Live hacking
https://github.com/flowing/flowing-retail/blob/master/payment-
rest/src/main/java/io/flowing/retail/payment/port/resthacks/PaymentRestHacksControllerV2.java

21. Fail fast
is important

22. Fail fast
is important
but not enough!

23. Berlin, Germany
bernd.ruecker@camunda.com
@berndruecker
Bernd Ruecker
Co-founder and
Developer Advocate of
Camunda

24. Photo by Tookapic, available under Creative Commons CC0 1.0 license.

25. Translation :
„There was an error
while sending your
boarding pass“

26. Check-in
Web-UI
Bernd
Current situation

27. Check-in
Barcode
Generator
Web-UI
Output
Mgmt
Current situation
Bernd

28. Check-in
Barcode
Generator
Web-UI
Output
Mgmt
Current situation
Bernd

29. Check-in
Barcode
Generator
Web-UI
Output
Mgmt
Current situation
Circuit
breaker
Bernd

31. Check-in
Barcode
Generator
Web-UI
Output
Mgmt
Current situation – the bad part
Bernd

32. Check-in
Barcode
Generator
Web-UI
Output
Mgmt
Current situation – the bad part
Bernd

33. Check-in
Barcode
Generator
Web-UI
Output
Mgmt
Current situation – the bad part
Stateful
Retry
Bernd

34. Translation:
It‘s your problem
now.

35. We are having some technical
difficulties and cannot present you
your boarding pass right away.
But we do actively retry ourselves, so
lean back, relax and we will send it
on time.

36. Check-in
Barcode
Generator
Web-UI
Output
Mgmt
Possible situation – much better!
Bernd

37. Check-in
Barcode
Generator
Web-UI
Output
Mgmt
Possible situation – much better!
Stateful
Retry
Bernd

38. Check-in
Barcode
Generator
Web-UI
Output
Mgmt
Stateful
Retry
Possible situation – much better!
The failure
never leaves
this scope!
Bernd

39. Persist thing
(Entity, Document, Actor, …)
State machine or
workflow engine
Typical
concerns
DIY = effort,
accidental
complexity
Complex, proprietary,
heavyweight, slow,
don‘t scale,
developer adverse
Scheduling, Versioning,
operating, visibility,
scalability, …
Handling
State

40. Warning:
Contains Opinion

41. Workflow engines,
state machines
It is
relevant
in modern
architectures

42. What about open source…?

43. CADENCE
Silicon valley
has recognized
Workflow engines,
state machines

44. What about purpose built and open source…

45. CADENCE
Workflow engines,
state machines

46. CADENCE
also at scale
Workflow engines,
state machines

47. CADENCE
for todays demo
Workflow engines,
state machines

48. Live hacking
https://github.com/flowing/flowing-retail/blob/master/payment-
rest/src/main/java/io/flowing/retail/payment/port/resthacks/PaymentRestHacksControllerV3.java

49. Challenges of
asynchronicity

50. Payment
Now you have a state machine!
Credit
Card
REST

51. has to implement
Retry
has to implement
Idempotency
Client Service Provider

52. We are processing your payment.
Do not leave this page.
And for god sake – do not reload!
It is a business
problem
anyway!

53. We are processing your payment.
Do not leave this page.
And for god sake – do not reload!
It is a business
problem
anyway!
We are currently processing your request.
Don‘t worry, it will happen safely –
even if you loose connection.
Feel free to reload this page any time!

54. Distributed systems introduce complexity you have to tackle!
Credit
Card
Payment
REST

55. Distributed
systems

56. It is impossible to
differentiate certain
failure scenarios.
Independant of
communication style!
Service
Provider
Client

57. Distributed systems introduce complexity you have to tackle!
Credit
Card
Payment
REST

58. Distributed systems introduce complexity you have to tackle!
Credit
Card
Payment
REST

59. Check-in
Barcode
Generator
Web-UI
Bernd
Output
Mgmt
Workflows live within
service boundaries

60. „The customer wants a synchronous response“
Check-in
Barcode
Generator
Web-UI
Bernd
Output
Mgmt
„Eh – no!“

61. generateBoardingPass
HTTP 200 OK
HTTP 202 ACCEPTED
Check-in
A synchronous response is possible in the
happy case, otherwise it is switched to
asynchronous processing.

62. Live hacking
https://github.com/flowing/flowing-retail/blob/master/payment-
rest/src/main/java/io/flowing/retail/payment/port/resthacks/PaymentRestHacksControllerV4.java

63. Challenges of
asynchronicity

64. Check-in
Barcode
Generator
Web-UI
Bernd
Output
Mgmt
Asynchronous communication
You need to
monitor
timeouts

65. Check-in
Barcode
Generator
Web-UI
Bernd
Output
Mgmt
Remember…
The failure
never leaves
this scope!

66. Workflow…

67. Workflow…

68. BPMN
Business Process
Model and Notation
ISO Standard

69. BPMN
Executable
and mature
Easy to
understand*
Widespread
*(for Biz and Dev)

70. Graphical models?

71. Clemens Vasters
Architect at Microsoft
http://vasters.com/archive/Sagas.html

72. Clemens Vasters
Architect at Microsoft
http://vasters.com/archive/Sagas.html

73. Clemens Vasters
Architect at Microsoft
http://vasters.com/archive/Sagas.html

74. Living documentation for long-running behaviour

75. Visual HTML reports for test cases

76. Proper
Operations
Visibility + Context

77. Compare to e.g. Step Functions

78. Imagine more complex stuff

79. has to implement
Timeout, Retry
has to implement
Idempotency
Client Service Provider

80. Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

81. Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

82. Avoids thundering herds
Photo by Gopal Vijayaraghavan, available under Creative Commons BY 2.0 license.

83. Photo by Gopal Vijayaraghavan, available under Creative Commons BY 2.0 license.
https://aws.amazon.com/de/message/5467D2/

84. Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

85. Distributed
Transactions
Distributed
Transactions

86. Compensation – the classical example
Saga
book
hotel
book
car
book
flight
cancel
hotel
cancel
car
1. 2. 3.
5.6.
In case of failure
trigger compensations
book
trip

87. 2 alterntive approaches: choreography & orchestration

88. Event-driven choreography
Hotel
Flight
Car
Trip
Trip
booked
Flight
booked
Trip
requested
Hotel
booked
Car
booked
Request
trip

89. Event-driven choreography
Hotel
Flight
Car
Trip
Trip
failed
Trip
requested
Hotel
booked
Car
booked
Request
trip
Flight
failed
Car
canceled
Hotel
canceled
Perform undo
(cancel car booking)
Perform undo
(cancel hotel)

90. Implementing changes in the process
Hotel
Flight
Car
Trip
Trip
failed
Trip
requested
Hotel
booked
Car
booked
Request
trip
Flight
failed
Car
canceled
Hotel
canceled
We have a new basic agreement
with the car rental agency and
can cancel for free within 1 hour
– do that first!

91. Implementing changes in the process
Hotel
Flight
Car
Trip
Trip
failed
Trip
requested
Hotel
booked
Car
booked
Request
trip
Flight
failed
Car
canceled
Hotel
canceled
You have to adjust all services and redeploy at the same time!
We have a new basic agreement
with the car rental agency and
can cancel for free within 1 hour
– do that first!

92. What we wanted
Photo by Lijian Zhang, available under Creative Commons SA 2.0 License

93. Orchestration
Hotel
Flight
Car
Trip
Trip
booked
Request
trip
Book
hotel
Hotel
booked
Car
booked
Flight
booked
Book
car
Book
flight

94. Orchestration
Hotel
Flight
Car
Trip
Trip
booked
Request
trip
Book
hotel
Hotel
booked
Car
booked
Flight
booked
Book
car
Book
flight
We have a new basic agreement
with the car rental agency and
can cancel for free within 1 hour
– do that first!
You have to adjust one service and redeploy only this one!

95. Describe orchestration with BPMN
Trip
Trip
booked
Request
trip

96. The workflow is part of the service
Trip

97. The workflow is part of the service
Trip
Payment

98. Live hacking
https://github.com/flowing/flowing-retail/blob/master/payment-
rest/src/main/java/io/flowing/retail/payment/port/resthacks/PaymentRestHacksControllerV6.java

99. has to implement
Timeout, Retry,
Compensation
has to offer
Compensation
has to implement
Idempotency
Client Service Provider

100. has to implement
Timeout, Retry,
Compensation
has to offer
Compensation
has to implement
Idempotency
Client Service ProviderDon‘t forget
about state

101. # Be aware of complexity of distributed systems
# Know strategies and tools to handle it
e.g. Circuit breaker (Hystrix)
Workflow engine for stateful retry, waiting, timeout
and compensation (Camunda)

102. Thank you!