The ease to set up a cloud deployment--done with little more than a credit card and an internet connection-- has resulted in a decentralization of IT management for many enterprises. This booming “Shadow IT” environment creates a host of management and cost issues for enterprises and usually leaves IT departments holding the bag. Very quickly, IT departments find themselves without control or even insight into the services being consumed, or even under-utilized, within their own organization. As independent units deploy in the cloud, the management and maintenance of these systems can lapse and quickly costs spiral out of control.
A smart IT manager quickly realizes that he/she needs to be able to gain visibility to the entire corporate cloud environment, centralize the management of the systems and create methods to do more with streamlined resources in the cloud – or face a potentially very nasty bill for cloud services at the end of each month.
For today’s IT departments to offer the best resources for the enterprise, the move to the cloud is becoming a necessity. To do this without loss of control, business and technology decision makers need to be able to see, measure and be notified on cost and consumption violations from day one. With this in place, IT’s management of cloud spending can be on budget and dynamically managed for optimal cost or performance improvements.
This session will walk CIO/IT managers through the key areas to consider in developing a cloud computing insight & management system and reporting mechanisms. It will also address why it is critical for IT to play a role in cloud development and management and some methods to help avoid circumvention of IT management within a “shadow” cloud.
Shine a light on “Shadow IT”: Taking control of usage and spending in the cloud by Sharon Wagner, CEO and Founder of Cloudyn
1. 1
Shine a Light on Shadow IT
Effective Cloud Cost Management
Sharon Wagner, CloudSlam ‘12
2. 2
Outline
Introduction, Cloud in Shadow IT
Cloud cost management process
Customer case study and lesson learned
Summary, questions and answers
www.cloudyn.com | info@cloudyn.com
3. 3
About Me
Director of Product Management, Oblicore
Sr. Principal, Cloud Connected Enterprise, CA
Founder and CEO, Cloudyn
www.cloudyn.com | info@cloudyn.com
4. 4
Shadow IT is here to stay
“IT controls less than 50% of corporate technology
expenditures”, PwC , June 2011
“In less than three years, 35% of enterprise IT
expenditures will happen outside of the corporate IT
budget.”, Gartner, Dec 2011
Cloud vendors doing their best to increase adoption
Variety of services
Provisioning agility
Resources availability
Continuously decreasing cost
www.cloudyn.com | info@cloudyn.com
5. 6
Aligning IT to business demands
IT
Control
Cloud
Adoption
Business
Agility
www.cloudyn.com | info@cloudyn.com
6. 7
Both IT and the business are:
Losing cost
Control Unable to maximize
Investment
Can’t plan Over-provisioning
Capacity resources
www.cloudyn.com | info@cloudyn.com | Cloudyn Confidential and Proprietary
7. 8
www.cloudyn.com | info@cloudyn.com | Cloudyn Confidential and Proprietary
8. 9
If you can’t beat them, join them!
Leverage your domain expertise
Know all cloud options
Find the best business terms
Streamline your cost management process
www.cloudyn.com | info@cloudyn.com
9. 10
Effective cloud cost management
Based on feedback from
key enterprises with $1M+
cloud spend annually Discover & Assess
Act, Change,
Optimize Set Operational
Controls
Set Organizational Align to Corporate
Controls Goals
www.cloudyn.com | info@cloudyn.com
10. 11
Energy & Co case study and lesson learned
A World wide energy company
Jim is an IT director
Energy & Co has a significant IaaS footprint
Multiple business units
Development, testing and production
what
Over 100 cloud accounts, +2000 servers resources
Fluctuating environment belong to
which task?
How will I
explain this
bill to my
boss?
What
resources are
REALLY
needed?
www.cloudyn.com | info@cloudyn.com
11. 12
Understand your cloud resources consumption
• 32 Extra Large servers running idle for 2 months Discover &
Assess
Set
• 3852 snapshots of a single volume
Act, Change,
Optimize Operational
Controls
Set Align to
• 11 reservations with no associated instances Organization
al Controls
Corporate
Goals
• 3 non-production databases running in 4.3% CPU utilization
• 80% predicted growth in annual cost
Taking care of these reduces cost by 34%
www.cloudyn.com | info@cloudyn.com
12. 13
Control resource consumption in real-time
Discover &
First, set operational alerts: Act, Change,
Assess
Set
Optimize Operational
Controls
Daily growth in # of instances exceeds 5% Set Align to
Organization Corporate
al Controls Goals
Weekly growth in cost exceeds 5%
Unused resources
Underutilized servers, databases
Now at least there won’t be any surprises
for Jim!
www.cloudyn.com | info@cloudyn.com
13. 14
Define cost units and break-down the spending
Map cloud resources to business units Discover &
Assess
Act, Change, Set
Optimize Operational
Define business units Controls
Set Align to
Define resource mapping rules Organization Corporate
al Controls Goals
Define budget structure
Define owners
Set cost and consumption views
Spending became clearer:
Jim could see who spends what, where and when
www.cloudyn.com | info@cloudyn.com
14. 15
Have unit owners control their costs
Discover &
Assess
Set business metrics Act, Change, Set
Optimize Operational
Controls
Budget violation projection Set Align to
Organization Corporate
al Controls Goals
Abnormal consumption growth
Cloud usage inefficiency
Correlate cloud cost with business activities
Now control was delegated to the unit
owners, with Jim supervision
www.cloudyn.com | info@cloudyn.com
15. 16
Usage-based cost and capacity optimization
Rightsizing first: Discover &
Assess
Act, Change, Set
Constantly eliminate unused resources Optimize Operational
Controls
Downscale overgrown instances Set Align to
Organization corporate
Get rid of retired storage al Controls Goals
Price optimization to follow:
Optimally reserve instances based on usage
Reallocate resources to existing reservations
Eventually the bill dropped by ~35% !
www.cloudyn.com | info@cloudyn.com
16. 17
Summary
Enhance your domain expertise
Know your cloud options
Think small – avoid traditional capacity allocation
For fine-grained control
Over-provision to the right extent
Leverage usage trends to rightsize consumption
Your environment is dynamic
Your solution should be dynamic too
www.cloudyn.com | info@cloudyn.com
17. 18
Thank You
sharon@cloudyn.com
Cloud Economics
@cloudyn_buzz
blog.cloudyn.com
Editor's Notes
new approach to cloud cost managementLest start by outline the dynamics of cloud environments
Discover & Assess => Insight into Resources Consumption (Intelligently determine the most suitable vendor, configuration and economical model)Set Operational Controls => Set consumption and cost indicators.Align to corporate objectives => Insight into Cost Structure / Allocation models How much my projects / customers / business activities costin the Cloud? Am I on or off budget? Am I expected to be on or off budget?Set organizational Controls => Set thresholds and indicators for budget versus actual, resources allocation for various business units, etc.Act => Implement deployment optimization in terms of resources provisioning, pricing plans, and ??? (will there be different recommendations for the cost allocation alerts?)Rightsize your cloud cost and capacity based on actual use pattern
Meet Vincent, IT director of Antlers and Hooves Enterprise. The enterprise uses cloud for multiple business purposes: marketing campaigns, Hadoop processing of data and extension of corporate IT on holidays. For each business purpose, in addition to production, there are multiple development and testing environments. Vincent has dedicated accounts for each business unit but looking through management consoles at the deployments he has absolutely no idea what resources belong to which task, which of them are required at all and who’s responsible for what. At the end of each month the bill becomes higher and higher. Each month Vincent needs to put someone for several days to manually map the resources to their owners. Vincent needs help to get the situation under control. What should be done?
Vincent Found: 15 Extra Large instances running 2 months after the marketing campaign was over3852 snapshots of a single volume11 Reservations that did not have the associated instancesCloud cost growing every dayGet full real-time visibility: Easily map the resources to the business tasksCalculate the cost of each task, including the history and the trendsAutomatically track deviations from norm Do the root cause analysis: Find the main cost contributorsQuickly find the cause of the budget deviationList of unused and underutilized resources may include:Unused reservationsForgotten instancesLeast-utilized instancesUnattached VolumesOld SnapshotsOver-sized databases
Vincent defined alerts to enable control:When number of instances grows by more than 5% in one dayWhen daily cost increases by more than 5%When monthly cost-to-date is higher by 5% than the same day previous monthWeekly and monthly costNow at least there won’t be any surprises!Track consumption trends to avoid unanticipated cloud cost:Measure your cloud resources consumption as it happensIdentify trends and spikes in usage and costSet automatic alerts to notify the deviation from planned budget and resources provisioningForecast the cost of each task [is this here? Or should it be in the organization controls?]
Insight into cost structure, budget management (plan vs. actual) and resources allocation (spending on each project / customer / business activity in the Cloud)Share data with the business:Allow each unit and task owner a dedicated view into his cost
Protective Foresight => be notified when approaching budget limits (total, or for specific tasks & business units)Involve relevant parties and endorse ownership:Alert relevant people so that the deviations are immediately handledset different alert thresholds for different roles: Task owners get alerts when the spend gets close to the budget limitsBusiness unit owners get aggregated alerts on the unit spend and “red” alerts for specific tasks
Implement deployment optimization in terms of resources provisioning & pricing plans, globally and for specific tasks / business unitsEliminate unneeded resources.Forgotten instancesUnused volumesOld snapshots Right-size your cloud.Make sure the instances you are using are just large enough for your needs. Do not over-provision: it’s easy to launch larger instance if required, so think small.RDS storage is easily grown. Use what you need.Pick the best matching pricing scheme.If appropriate – use spots. (Most effective with Hadoop)Use reservations. Light Utilization starts paying off after 2.5 months!Use cheaper regions if possible: us-east-1 and us-west-2 are your best bets in US. The latency differences between us-west-1 and us-west-2 are negligible in most cases, but the latter is cheaper.
new approach to cloud cost managementLest start by outline the dynamics of cloud environments