Optimizing Alert Monitoring with Oracle Enterprise Manager
•Download as PPTX, PDF•
4 likes•9,171 views
Watch this webinar to find out how OEM Grid configuration using Datavail’s Alert Optimizer™ and custom templates helps eliminate unwanted alerts, while enriching actionable alerts, and improving the performance of the entire database system. These five areas help organize the tuning approach and define the major concerns beyond the architecture, setup, and data model. It also addresses how performance tuning becomes less of a mystery if it can be measured, documented, affected, and improved.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (9)
Similar to Optimizing Alert Monitoring with Oracle Enterprise Manager
Similar to Optimizing Alert Monitoring with Oracle Enterprise Manager (20)
More from Datavail
More from Datavail (20)
Recently uploaded
Recently uploaded (20)
Optimizing Alert Monitoring with Oracle Enterprise Manager
- 1. Using Oracle Enterprise Manager, Alert Optimizer for Enterprise Monitoring Presented by John Kaufling, Chet Szatkowski, and Patrick Gates, Datavail
- 2. 3/19/2014 2www.datavail.com Free Analysis with Datavail’s Alert Optimizer Just for attending this webinar you have a chance to get a free analysis of one month of your alert history using Datavail’s Alert Optimizer Tool 10 random winners will be chosen from those who go to http://info.datavail.com/OEMOffer and fill out the requested info Must have OEM 10g, 12c or 11 Datavail will need remote access to your OEM Repository Gain the insight you need, from a 10,000 foot skyscraper view to a 100 foot magnification Alert Optimizer Analyzes the volume and distribution of configured alerts generated by OEM powered by Datavail
- 3. 3/19/2014 3www.datavail.com Topics Implementing OEM out of the box OEM Monitoring Templates OEM Thresholds, Metrics, and Policies Maintaining OEM with Datavail OEM Alert Optimizer Case Study Q & A
- 5. 3/19/2014 5www.datavail.com Why OEM? Oracle developed product with no additional licensing cost unless you use the tuning pack, diagnostic packs, or EBS plugin. Detects situations which might have a negative impact on overall system performance and allows these situations to be corrected before the impact is felt by the user community. Monitor from the lowest level host server key- performance-metrics like CPU and memory consumption, to the Oracle middle-tier components such as Web Application Servers and Oracle Workflow processes, through the system response time of an Oracle web page can be configured for monitoring in real time.
- 6. 3/19/2014 6www.datavail.com Configuring and Deploying OEM Grid The OEM Management Agents are installed on the target hosts. The Oracle Management Server can then automatically detect and configure the Management Agents and discover the various targets (the host itself, databases, OC4Js, Web servers etc.,) running on any particular host on which the agent itself is running. Using these configured agents, the Management Server is configured via its Administrative Web interface to collect various metrics from hosts, Web Servers, and, using optional Oracle EBS plug-ins, key parameters of the EBS architecture itself. For all configured agents metrics, key performance metric thresholds can be set so that an “alert” is generated when the current value of the monitored metric.
- 7. 3/19/2014 7www.datavail.com Configuring and Deploying OEM OEM Grid provides detailed information on the current status of all configured targets . The status of the configured “alerts” for various targets and target-types are also cleanly summarized in a “dashboard” so that any issues present can be easily detected, analyzed, and addressed in a forward-looking manner. Groups of targets can be associated with each others as “Systems” to collectively show the health of an overall environment as well as its lower-level components/targets.
- 8. 3/19/2014 8www.datavail.com OEM Notifications and Alerts OEM Grid can be configured to have alerts trigger specific actions; including a “Corrective Action” , notifications via email / SMS, the creation of a service organization’s “Ticketing System” ticket, or any other similar action. “Corrective Actions” for an alert can be user defined and allow for an automatic response to a specific alert. For example -- an automatic re-compilation of database objects which have become “invalid” etc. Using these notifications allows for a seamless end-to-end tracking system to be deployed which documents the lifecycle of any performance event for historical analysis of the response time to an event, the details analysis performed, and the remedies which were ultimately actioned by the Datavail DBA teams. OEM keeps a history of the generated alerts so that, if properly queried from the repository using Datvail’s Alert Optimizer, the alerts can be reviewed on a regular basis allowing trends in the recurrence of specific alerts, or groups of alerts, from specific targets or systems to be visually detected and ultimately corrected from recurring in large volumes.
- 10. 3/19/2014 10www.datavail.com OEM Templates Many OEM implementations use the Oracle provided templates with very little customization. Datavail has found it is better to create customized templates suited to your business needs. • Production and non-production • Production, test, qa, staging, dev • Data Warehouse, OLTP • Application A, Application B, Application C Allows you to customize thresholds and metrics for an environment or application. Not all systems are created equally and should not be monitored the same. • Space thresholds that trigger at 90% for an OLTP system are usually not adequate for a data warehouse database. • At 90% used on a data warehouse, there could still be several terabytes of free space.
- 11. 3/19/2014 11www.datavail.com OEM Templates What are the critical database components to have in your templates? • Alert.log errrors • Database up/down • ASM database up/down • Listener up/down • Server up/down • Cluster component status • ASM space • Filesystem space • Tablespace space • Archive log dest space • Recovery file dest space
- 12. 3/19/2014 12www.datavail.com Getting more from OEM Using Oracle’s Enterprise Manager Grid Control in combination with PL/SQL scripts which were custom developed by Datavail for Oracle’s E- Business Suite, an EBS environment can also be monitored in real-time for significant performance-impacting events. The Complete EBS architecture, from the lowest level host server key- performance-metrics like CPU and memory consumption, to the EBS middle-tier components such as Web Application Servers and Oracle Workflow processes, all the way up to the system response time of an EBS module’s web page can be configured for monitoring in real time.
- 13. Maintaining OEM
- 14. 3/19/2014 14www.datavail.com Maintaining OEM After all this setup, OEM is ready for prime time but how do you keep it tuned and working efficiently? OEM is not a turn it on and forget it monitoring tool. • You need to work at it to keep it running efficiently. Analyzing patterns within hundreds of alerts per day, and thousands of alerts per week becomes difficult, if not impossible, without using some sort of data grouping, filtering, and sorting currently unavailable in Oracle’s OEM Grid Control software. That is where a tool like Alert Optimizer can help you reduce the noise.
- 15. 3/19/2014 15www.datavail.com Maintaining OEM By querying the OEM Grid repository directly for alert histories and properly formatting the returned data set, this information can be logically grouped and visually analyzed for alerting patterns. The Alert Optimizer allows the clustering of alert history to be arranged chronologically, by system, by target, and by metric-type, amongst many other parameters returned from the repository for each individual alert. Organizing the alerts by the categories below quickly allows us to see what is happening with alert volumes. • Volume by Day • Volume by Week • Volume by Target Name • Volume by Metric Name • Hourly Alert Volume
- 16. 3/19/2014 16www.datavail.com Maintaining OEM with Alert Optimizer The Datavail OEM Alert Optimizer allows for any level of drill- down, filtering, sorting, grouping, and graphing to further analyze alerting trends for significant performance events across entire systems and over any time period. As the core OEM Grid dataset is now embedded in MS Excel itself, this data can be statistically analyzed, exported, manipulated, trended and charted etc. completely independent of the OEM Grid repository itself.
- 17. 3/19/2014 17www.datavail.com Maintaining OEM with Alert Optimizer By aggregating the alert volume data over time, and grouping by the parent system object from which the alert target is associated yields visually obvious alert volume patterns. This ultimately allows DBA teams to quickly focus on reducing the worst offenders, by adjusting the metric alert thresholds to values which more appropriately match the targets operational baseline (alerting by exception only), eliminating useless alerts for a particular metric, or by eliminate a target from monitoring/alerting altogether.
- 18. 3/19/2014 18www.datavail.com Maintaining OEM with Alert Optimizer Aggregation by “Target Name” allows for the immediate visualization, independent of time distribution of the worst offending target objects. By then reducing the largest volume offenders, the overall reduced alert volume allows the team responsible for addressing the alert’s root causes to more properly gain contextual information for the remaining alerts thereby suggesting system tuning, metric threshold modification, or specific alert elimination.
- 19. 3/19/2014 19www.datavail.com Maintaining OEM with Alert Optimizer • Aggregation purely by alert volume vs. time shows the immediate load placed upon those who are monitoring and addressing these alerts. As shown below, trends in easily-filterable overall periods of alert volume can be assessed. Ideally alert volume trends would head in an either even, or negative direction indicating a stable, or increasingly well performing target environment.
- 20. 3/19/2014 20www.datavail.com Maintaining OEM • Weekly meetings to review OEM Grid alert trending and address the most critical and/or highest volume alerts from the previous week. • Diligent attention to alert patterns and proper resolution of their root cause allows the Applications DBA support team to focus on forward-looking environment tuning issues and database performance, or reassign the DV DBA resources to deploying code changes or executing other requests from the client’s internal teams. • After regular weekly meetings for 6 months, total alert volume has been reduced by more than 50%. Maintaining OEM with Alert Optimizer
- 21. 3/19/2014 21www.datavail.com Maintaining OEM (Monitoring Templates) • Datavail reviews Oracle delivered OEM Grid baseline templates to ensure that the template settings match the customer’s target environment baseline. If the seeded templates for a particular object, or object type are too “narrow”, excessive alerts will result when pushed to the target, if too “wide”, WARNING and CRITICAL alerts will not be generated in time to rectify the situation before the end users are ultimately impacted by the potentially avoidable root cause. • Base-lining an environment is critical to matching each “dial” in the template of individual metric threshold to the target on which it is deployed. Developing a standard template for a target which can be pushed to new targets and then dialed-in saves time for newly deployed targets and prevents unnecessary and useless alerts and their associated incidents. • The most commonly finely tuned metrics are host-level CPU usage, database instance wait session classes, blocking session timing logic, application response time beacons (available in the Oracle EBS AMP add-in.)
- 22. Case Study
- 23. 3/19/2014 23www.datavail.com Customer Overview $20 Billion in revenue; Fortune 150 company; 16,500 employees worldwide Datavail supports this EBS environment, which is one of the world’s largest Oracle EBS roll-outs & customization projects Environmental Specifics • 8 – 10k incidents per mo., 90% automatically created from OEM Grid. • 10 target environments with differing outage windows, 4 PROD, 6 Quality (PreProd) • 24x7 execution, roughly evenly distributed between all shifts • Continuous optimization of Grid control alerting with client via Datavail’s Alert Optimizer tool
- 25. Questions? Questions can also be sent to Kelley.Weir@Datavail.com
- 26. Alert Optimizer Special Offer Don’t forget to visit http://info.datavail.com/OEMOffer For your chance to win a free analysis of your alert history.
Editor's Notes
- Thank you for taking time out of your busy day to join us. We are going to discuss implementing OEM, how we setup different templates to create meaningful actionable alerts, and how we have developed the Alert optimizer tool to help us maintain and manage OEM alert volumes.First I will talk about implementing OEM out of the box.Next, John Kaufling will walk you through OEM Templates, Thresholds, Metrics and PoliciesAnd Chet Szatkowski will wrap everything up by taking us through how to maintain OEM with our Alert optimizer tool and walking you through a customer example wherewe use the Alert optimizer to each and every week to improve the information coming out of OEM as well as control the alert volumes
- Why do we use OEM to monitor oracle databases, database servers, application servers, storage, etc.It is an oracle developed product. What better tool to monitor an oracle database than a tool developed by oracle.OEM allows you to manage Oracle‘s complete software and hardware stack, from applications-to-disk.IT organizations are coping with more and more complexity than ever before. We are facing issues Issues ranging from growing data volumes, to composite applications, to virtualization and how best utilize the Cloud. Managing all these systems and services 24x7 is a giant challenge and constant battle for IT. Oracle Enterprise Manager provides an integrated console that gives IT departments business visibility and comprehensive IT management across the stack. In short, it integrates IT management with business applications and systems. This integration enables IT to gain unobstructed views of business transactions and the business-user experience. As a result, IT professionals can focus their efforts where they will have the greatest positive impact on the business.Allows for the detection of situations which might have a negative impact on overall system performance and allows these situations to be corrected before the impact causes the user community to be exposed to the system performance event.
- The OEM Management Agents are installed on the target hosts.The Oracle Management Server can then automatically detect and configure the Management Agents and discover the various targets (the host itself, databases, OC4Js, Web servers etc.,) running on any particular host on which the agent itself is running.Using these configured agents, the Management Server is configured via its Administrative Web interface to collect various metrics from hosts, Web Servers, and, using optional Oracle EBS plug-ins, key parameters of the EBS architecture itself.For all configured agents metrics, key performance metric thresholds can be set so that an “alert” is generated when the current value of the monitored metric.
- OEM Grid provides detailed information on the current status of all configured targets. The status of the configured “alerts” for various targets and target-types are also cleanly summarized in a “dashboard” so that any issues present can be easily detected, analyzed, and addressed in a forward-looking manner.Groups of targets can be associated with each others as “Systems” to collectively show the health of an overall environment as well as its lower-level components/targets.As you can see in this dashboard view, you are quickly able to see a red light green status of the targets along with the number of alerts that have been generated.
- As many of you know, just turning on OEM out of the box can very quickly overwhelm you if with alerts notifications especially if you have hundreds or thousands of targets in your environment Using these notifications in conjunction with a ticketing system allows for a seamless end-to-end tracking of any performance event for historical analysis of the response time to an event, the detailed analysis performed, and the remedies which were ultimately actioned by the DBA teams.Many times OEM is not implemented in conjunction with a ticketing system to track all alerts that a DBA or DBA team are addressing. One of the benefits of our Alert Optimizer tool is the ability to visual show how many alerts or notifications are coming from OEM. If OEM notifications are only going to a DBAs mailbox or to a mailbox monitored by a DBA team no one else will has visibilty to the alert volumetrics being generated by OEM. The history within the OEM repository is what allowed us to develop the Alert Optimizer for our customers. The alert history allows us to trend alerts over many different timelines and allows us to pinpoint where the “noise” is coming from in relation to targets, metrics, and policies Now John is going to walk you through what we have done with templates
- Many times OEM grid is implemented out of the box with very little customization to the templates. This can quickly lead to the “boy that cried wolf” situation because of the copious generation of non actionable alerts.Many DBAs may ask, what is the advantage of creating templates?Monitoring templates simplify the task of standardizing monitoring settings across your enterprise by allowing you to specify the monitoring settings once and apply them to your monitored targets. This makes it easy for you to apply specific monitoring settings to specific classes of targets throughout your enterprise. For example, you can define one monitoring template for test databases and another monitoring template for production databases.Taking the time to create the separate templates allows us to have to customize thresholds and metrics for an environment or applicationAs you know, not all systems are created equally and should not be monitored the same.
- Out of the box OEM templates may have thresholds, metrics, or polices that be set at levels the are too narrow or too wide. Customizing OEM templates across our customer base has given us unique in-site in what we should include in our templates. So much of the time we disable many metrics and policies to get us down to monitoring the basics that are critical to supporting an oracle database.The basics we encourage our customers to use are shown here.As mentioned earlier, some metric thresholds come predefined out-of-box. While these values are acceptable for most monitoring conditions, your environment may require that you customize threshold values to more accurately reflect the operational norms of your environment. Setting accurate threshold values, however, may be more challenging for certain categories of metrics such as performance metrics.Monitoring session wait events and response time events can generate hundreds to thousands of non actionable alerts. For example, what are appropriate warning and critical thresholds for the Response Time Per Transaction database metric?For such metrics, it might make more sense to be alerted when the monitored values for the performance metric deviates from normal behavior. Enterprise Manager provides features to enable you to capture normal performance behavior for a target and determine thresholds that are deviations from that performance norm.As Chet will show you in a few minutes, you really need a tool like Datavail’s Alert Optimizer to identify where the alerts are coming from.
- User-defined metrics allow you to extend the reach of Enterprise Manager's monitoring to conditions specific to particular environments via custom scripts or SQL queries and function calls. Once a user-defined metric is defined, it will be monitored, aggregated in the repository, and can trigger alerts like any other metric in Enterprise Manager. There are two types of user-defined metrics: Operating System and SQL.Operating System (OS) User-Defined Metrics: Accessed from Host target home pages, these user-defined metrics allow you to implement custom monitoring functions via OS scripts.SQL User-Defined Metrics: Accessed from the Database target home pages, these user-defined metrics allow you to implement custom database monitoring using SQL queries or function calls.Now
- Our experience gained as a result of our partnership with several large OEM Grid installations, some of which include thousands of individual target objects being monitored, Datavail has developed unique tools to analyze the volume and distribution of configured alerts generated by OEM.
- This ultimately allows DBA teams to quickly focus on reducing the worst offenders,
- Diligent attention to alert patterns and proper resolution of their root cause allows the Applications DBA support team to focus on forward-looking environment tuning issues and database performance