1. Accelerate your Exadata
deployment
With the skills you already know
Marc Fielding
September 2013

2. About me
• Senior Consultant with Pythian’s
Advanced Technology Group
• 12+ years Oracle production
systems experience starting with
Oracle 7
• Blogger and conference
presenter
pythian.com/news/author/fielding
• Occasionally on twitter: @mfild
2 © 2013 Pythian

3. About Pythian
• Recognized Leader:
– Global industry-leader in remote database administration services and consulting for Oracle,
Oracle Applications, MySQL and Microsoft SQL Server
– Work with over 250 multinational companies such as Forbes.com, Fox Sports, Nordion and
Western Union to help manage their complex IT deployments
• Expertise:
– Pythian’s data experts are the elite in their field. We have the highest concentration of Oracle
ACEs on staff—8 including 2 ACE Directors—and 2 Microsoft MVPs.
– Pythian holds 7 Specializations under Oracle Platinum Partner program, including Oracle
Exadata, Oracle GoldenGate & Oracle RAC
• Global Reach & Scalability:
– Around the clock global remote support for DBA and consulting, systems administration,
special projects or emergency response
© 2013 Pythian3

4. © 2013 Pythian4
Why this presentation

5. So you have Exadata. What next?
• Underpinning Exadata are well-known Oracle
technologies
• SQL*Plus works just like on any other Oracle system
• RAC and Grid Infrastructure, 11gR2 (or 12c!)
• ASM is there
• Services, SCAN, etc no different than other Oracle RAC
environments
© 2013 Pythian5

6. So what is different?
• It’s an engineered system: you run the same
hardware as other Exadata customers
• Oracle development has tested on the exact same
configuration
• It’s a good thing: much fewer obscure OS issues,
driver issues etc
• Single point of contact at Oracle support – hardware
and software
• And yeah, there are are a few new concepts…
© 2013 Pythian6

7. © 2013 Pythian7
Storage servers
Management switch
Compute servers
Empty space
InfiniBand switches
KVM

8. The bits you know already
• It’s running Oracle database software
• Database servers: Xeon-based
servers running Oracle Linux 5
• 2, 4, 8- or more-node 11gR2 RAC
cluster
• Clustered ASM instances
• Ordinary Ethernet connections
to the outside
© 2013 Pythian8

9. The secret sauce
• InfiniBand networking
– Gives amazing latency: forget 1ms, try 1µs
– Great capacity too: 40gbps
– Used for RAC interconnect and storage
access – replaces fiber channel
• Storage servers: also Linux servers
– But they run Exadata-specific software
– Take I/O requests from database servers and send to disk
– Also have understanding of database concepts
© 2013 Pythian9

10. Differences from SAN/NAS
• No single “storage head” – storage management is
handled by the ASM cluster instead
• Easy scaling: when you add storage capacity, you get
processing capacity too
• Switched network increases network capacity as you
scale
• Usually dedicated database storage, so often managed
by DBAs too
• Hot/cold storage: tier outer and inner tracks on a disk
© 2013 Pythian10

11. Smart Scans (1)
• Somewhat analogous to parallel query
• Multiple parallel processes
• Communicate with shadow processes with SQL-type
constructs – rows, columns etc
• Save communication capacity by only sending the data
you need, not entire data blocks
• select col1 from big_table;
• Sends only col1 data, for example
• Also more complex actions like predicate evaluation,
decryption, bloom filter joins…
© 2013 Pythian11

12. Smart Scans (2)
• Put processing closer to the storage
• Inherently parallel: processing happens on all storage
servers that might have matching data
• Only apply to bulk, direct path operations – usually full
table scans
• Excludes
– Index range scans
– Table access by index rowid
– Blocks with uncommitted changes
– LOBs
© 2013 Pythian12

13. • Used in conjunction with smart scans
• Can avoid doing I/O entirely
• Split storage up into 1m “storage regions”
• Track high/low values
• Work best with pre-sorted data
• Example: select * from tab where col1 < 20;
© 2013 Pythian13
Storage Indexes
2-99 4-57 33-60 90-120

14. Hybrid Columnar Compression (1)
• New storage type with superb compression ratios: 10x or more
• Group blocks into compression units
• Store columns together
instead of rows
• Typical databases have repeated
values: the infamous “Y”/”N”
column for example
• Then apply well-known
compression like bzip2 or gzip
on top
© 2013 Pythian14

15. Hybrid Columnar Compression (2)
• Much like smart scans, data must be loaded in bulk:
CREATE TABLE AS SELECT, INSERT
/*+APPEND*/, SQL*Loader direct path, etc
• Otherwise data will get OLTP compression instead
• CPU overhead to compress/decompress (though
smart scans can offload decompression)
• Locks affect entire compression units rather than
single rows
© 2013 Pythian15

16. Recommendations: HCC
• Best for data that is inserted in large batches (data
warehouses for example)
• Also useful with time-based partitions: re-compress
old, no-longer-modified data
• There are 4 levels; ARCHIVE HIGH has significant
CPU overhead so use for data that’s rarely
accessed
• Use DBMS_COMPRESSION to predict
compression ratios, and see results afterwards
© 2013 Pythian16

17. Flash cache
• Designed for OLTP
• Flash memory has 5x better read
latency than hard drives
• Transparent to applications
• Smart enough not to cache RMAN backup I/O, ASM
secondary extents, and other non-repeatable reads
• DBAs can use resource manager and SQL to
prioritize cache objects
© 2013 Pythian17

18. Flash cache: recommendations
• With new Exadata X3, you may fit entire DB in
flash. Enjoy!
• Entirely different from the similarly-named
Database Smart Flash Cache in 11gR2
• You can put ASM disks on flash, but ASM
redundancy reduces usable space
• You can use back-end commands to actually
see what’s being cached
© 2013 Pythian18

19. I/O Resource Management
• You can use DBRM in Oracle 11g to manage
CPU time, and manage runaway queries
• But what if your system is bottlenecked on I/O
• In Exadata, just turn on IORM and your CPU
priorities apply to I/O too
• Can also allocate I/O capacity across databases
© 2013 Pythian19

20. Management tools
• OEM 12c: put a lot of work into managing Exadata
• Can now manage
all Exadata
components (incl
switches etc)
• Consolidation
planning tools to
evaluate workloads
© 2013 Pythian20

21. Management tools (2)
• Command-line tools all there: SQL*Plus etc
• For storage servers, a new tool: cellcli to
manage storage servers
• cellcli can be combined with dcli to run a
command on multiple machines.
• cellcli changes to one storage server aren’t
visible to others
© 2013 Pythian21

22. Hardware management
• Like other enterprise hardware, Exadata has out-of-
band management
• ILOM lets you access hung servers without driving
to the datacenter
• Very useful when doing patches and reboots
• Monitors system health and can send alerts when
metrics are out of range
• Most configuration changes made thorough Exadata
cellcli
© 2013 Pythian22

23. Backups
• Backups run through RMAN – no snapshot support
• Similar RMAN configuration to non-Exadata
• Use high-speed InfiniBand network if you can
• Oracle ZFS appliance can be configured for
InfiniBand for example
• Use IORM to manage backup I/O to avoid
performance degradation at backup time
• Use multiple channels to keep the disks busy
© 2013 Pythian23

24. Disaster recovery
• No storage-level replication like SRDF
• Data guard is the overwhelming
favourite
• Active data guard works well with
Exadata
• Non-Exadata standby: watch HCC
• Most common approach: two
datacenters with identically configured
Exadata machines
© 2013 Pythian24

25. Data migration
• Like backups: no storage-level replication
• Physical copies
– Data guard physical standby
– Transportable tablespaces
– Cross-platform incrementals
• Logical transfers
– Data Pump
– Direct path over DB Link
– GoldenGate
– Data guard logical standby
– External tables / SQL*Loader
© 2013 Pythian25

26. Patching
• Exadata has many components
• All of which have periodic patches
• Major work has been done to improve patching
• Recommended quarterly full-stack patches include
patches for all equipment
• Do not apply CPU/PSU patches – Exadata bundle
patches include security fixed, plus high-priority bug fixes
• Different patching options involve different levels of
downtime, data-loss risk, and execution time
• Review the options
© 2013 Pythian26

27. Diagnostic tools
• exachk is developed by Oracle development to
compare your configuration with Oracle-
recommended values
• Large library of tests, and regularly being added
• Good idea to run before and after system
changes like patches or configuration updates
• Download from My Oracle Support
• Required root access for full information
© 2013 Pythian27

28. New Exadata install – what to expect
• PDF configuration questionnaire
– How you want your system built out
• Hardware requirements
– Exadata is very dense – look at power, weight, cooling etc
• Site survey
• Equipment delivered
• Hardware configuration
• Software configuration
• Installation report
© 2013 Pythian29

29. Decisions to make
• High performance / high capacity
• ASM normal / high redundancy
• Data/FRA storage split
• Hot/cold diskgroups?
© 2013 Pythian30

30. Working with Oracle Support
• Oracle Support manages all Exadata components:
database, hardware, OS, network
• Even hard disk failures must go through Oracle Support
for triage and troubleshooting
• Dedicated engineered systems team handles front-line
support
• Much better experience than ordinary support
• Ordinary support-management skills are doubly
important though: creating SRs properly, escalations, etc
© 2013 Pythian31

31. Organizational support models
• Exadata blurs the lines
• Oracle support procedures are new for admins used
to dealing with other vendors
• Generally most Exadata work is DBAs
• Most common support model is to create a team
with mostly DBAs to manage entire Exadata stack
• Organizational reasons may require a more
traditional model though
• For you as a DBA: reach out
© 2013 Pythian33

32. But overall
• As a DBA, you’ll still be doing much the same
work as you’re used to on Exadata
• Performance tuning, backups, code
deployments, disaster recovery all exist in
Exadata land
© 2013 Pythian34

33. Gaining Exadata skills
• Not many of us have datacenters in our
basements to try out new technologies
• Learn technologies like RAC, and “real”
ASM with redundancy, direct-attach disk
• But there are plenty of resources:
– Exadata courses: Oracle university workshop,
and others
– Exadata books, particularly Expert Oracle Exadata
– Conference presentations – even if you can’t
attend, try and download the slides
© 2013 Pythian35

34. A word about the IOUG
• The IOUG is bring you today’s user group day
• By Oracle users, for Oracle users
• Collaborate 2014: April 7-11, Las Vegas
• Master class events this fall: Milwaukee, Rochester, Chicago,
Atlanta
• IOUG Kiosk: Moscone West, 2nd floor
© 2013 Pythian36

35. Thank you and Q&A
© 2013 Pythian37
fielding@pythian.com
1-877-PYTHIAN
pythian.com/blog
http://is.gd/PythianFacebook
@mfild @pythian
http://linkedin.com/company/pythian
My upcoming sessions:
Database Private Clouds
with Oracle 12c
Tuesday, high noon
Moscone South 236
It’s a solid state world:
How Exadata X3 leverages
flash storage
Wednesday 3:30pm
Westin SF Metropolitan I