This document provides an overview of Flex Clusters and Flex ASM in Oracle Database 12c. It defines Flex Clusters as a scalable and dynamic architecture with hub and leaf nodes. Leaf nodes do not require direct access to shared storage. It describes how to configure a cluster as a Flex Cluster and change node roles. It also introduces Flex ASM, which allows ASM to run on fewer nodes while providing failover of client connections.
2. Maaz Anjum
• Marietta, Georgia
• Solutions Architect at BIAS Corporation
• EM12c
• Golden Gate
• Engineered Systems
• Member of IOUG, GOUG, RMOUG
RAC SIG, EM SIG,
BIG DATA SIG
• Using Oracle products since 2001
Blog: maazanjum.com
Email: maazanjum@gmail.com
Twitter: @maaz_anjum
About Me
2
3. !
• Flex Cluster
• Flex ASM
• Your Database on a Flex Architecture
• Conclusion
Agenda
3
4. Standard Cluster
4
• What is an Oracle Cluster today?
– Set of hosts that have direct access to
storage
– All (typically) run
• Grid Infrastructure
• ASM
• Listener(s)
• Database(s)
5. • What is a Flex Cluster?
• Is Oracle Grid Infrastructure installed in an Oracle Flex Cluster configuration
– Scalable
– Dynamic
– Robust network of node
• Offers a platform for a variety of applications including Real Application Clusters,
with large number of nodes
Flex Cluster
5
6. • All nodes in an Oracle Flex Cluster belong to a single Oracle Grid Infrastructure cluster.
!
• This architecture was designed to centralize
– Policy decisions for deployment of resources based on application needs
– Account for various service levels, loads, failure responses, and recovery
6
Flex Cluster
7. • Arranged in a Hub and Spoke Architecture
• The hub-and-spoke model is most
frequently compared to the point-to-point
transit model.
Flex Cluster
7
A
D
B
F
E
C
• For a network of n' nodes
• only n - 1 routes are necessary to connect
all nodes
• the upper bound is n - 1
• and the complexity is O(n).
A
D
B
F
E
C
8. • This compares favorably to the routes, or O(n2), that would be required to
connect each node to every other node in a point-to-point network.
!
• An airline example: in a system with 10 destinations, the spoke-hub system requires
only 9 routes to connect all destinations, while a true point-to-point system would
require 45 routes.
Flex Cluster
8
Point-to-Point
Hub-Spoke
9. • Contains two types of nodes
– Hub
• Hub Nodes are similar to Oracle Grid Infrastructure nodes in an Oracle
Clusterware standard Cluster configuration.
– Leaf
• Leaf Nodes are different from standard Oracle Grid Infrastructure nodes, in that
they do not require direct access to shared storage, but instead request data
through Hub Nodes.
Flex Cluster
9
11. • Will be the basis for more uses cases in future releases (for resource
management)
!
• If a Leaf-node has access to shared storage, it can perform a role change to a
HUB-node
!
• A Standard Cluster can be converted to a Flex Cluster at any time after the
installation
!
• Once a cluster is configured as a Flex Cluster this configuration is irreversible
without a re-configuration of the cluster.
Flex Cluster
11
12. • An Example Setup
Flex Cluster
12
Host: flex1
Role: Hub
Storage
Host: flex2
Role: Hub
• RACAttack.org/12c
17. • Node Roles
• Change Role
!
[root@flex2
~]#
./crsctl
get
node
role
config
Node
'flex2'
configured
role
is
'hub’
!
[root@flex2
~]#
./crsctl
set
node
role
leaf
CRS-‐4408:
Node
'flex2'
configured
role
successfully
changed;
restart
Oracle
High
Availability
Services
for
new
role
to
take
effect.
!
[root@flex2
~]#
./crsctl
stop
crs
!
[root@flex2
~]#
./crsctl
start
crs
-‐wait
17
Flex Cluster
18. • Node Roles
• Resources after role change
• Listener
Once the node is converted to a leaf node, it has a local listener.
ora.LISTENER_LEAF.lsnr
OFFLINE
OFFLINE
flex2
STABLE
• # of ASM Instances also reduces
ora.asm
1
ONLINE
ONLINE
flex1
STABLE
2
ONLINE
OFFLINE
STABLE
3
ONLINE
OFFLINE
STABLE
• VIP Fails over to available node
ora.flex2.vip
1
ONLINE
INTERMEDIATE
flex1
FAILED
OVER,STABLE
18
Flex Cluster
19. • Bits and Pieces
– ASM Network
• By default, it will share the interconnect NIC
• Can be changed during install, or later
– Grid Naming Services (GNS)
• If you didn’t like it before, you’d better start liking it now.
• You can install GI in Standard Mode and reconfigure it with GNS prior to conversion to Flex
Mode.
– Optionally, prompted to install Cluster Health Monitor (CHM) Database
• -MGMTDB
– The Cluster Health Monitor (CHM) detects and analyzes operating system and cluster resource-related degradation
and failures.
– Initially allocates and uses approximately 4GB
19
Flex Cluster
20. • Cluster Health Monitor (CHM)
– Consists of three services
• System Monitor Service (osysmond)
• Cluster Logger Service (OLOGGERD)
• GI Management Repository (MGMTDB)
– Another Database resource?
– Stores Real-Time Operating System Metrics collected by CHM
– It will only run on a single node in the cluster
– Communicates with other Services via the Private Network
– Data files are on the same disk group as the OCR and Voting files.
– Cluster Troubleshooting
• OCLUMON
– Query the CHM repository to display node-specific metrics for a specified time period
• Leverage CHM to produce reports
[root@flex1
tmp]#
/u01/app/12.1.0.1/grid/bin/diagcollection.pl
–collect
[root@flex1
tmp]#
/u01/app/12.1.0.1/grid/bin/oclumon
dumpnodeview
-‐n
flex1
flex2
last
"12:00:00”
• Manage
[root@flex1
tmp]#
oclumon
manage
-‐get
alllogger
-‐details
Logger
=
flex1
Nodes
=
flex1,flex2
20
Flex Cluster
21. • Adding a new node is no different than with previous releases
– Specify the kernel parameters
– Configure block devices for Oracle Clusterware devices
– Ensure that you have set the block device permissions correctly
– Use short, nondomain-qualified names for all of the names in the /etc/hosts file
– Test whether the interconnect interfaces are reachable using the ping command
– Verify that the VIP addresses are not active at the start of the cloning process by using the ping
command
– Run CVU to verify your hardware and operating system environment
Flex Cluster
21
22. • Consider Oracle’s strive towards consolidation
!
• Experienced technology sprawl from acquisitions
!
• Attempting to enable ease of resource management
!
• Flex Clusters encompass not only databases, but applications as well
Flex Cluster
22
23. !
• Flex Cluster
• Flex ASM
• Your Database on a Flex Architecture
• Conclusion
Agenda
23
24. • The Next generation Automatic Storage Management (ASM) has a new name.
!
• Traditionally, we know ASM as another type of instance in a RAC/Single Instance
configuration.
!
• One ASM instance per node in a cluster
Flex ASM
24
25. • Flex ASM is
• Your ASM configured to run on fewer nodes in a cluster
• Allows for seamless client connection failover
25
Flex ASM
27. • With Oracle Flex ASM you can
– Consolidate all the storage requirements into a single set of disk groups.
!
– All these disk groups are mounted by and managed by a small set of Oracle ASM instances running
in a single cluster.
!
– You can specify the number of Oracle ASM instances with a cardinality setting. The default is three
instances.
!
– When using Oracle Flex ASM, you can configure Oracle ASM clients with direct access to storage or
the I/Os can be sent through a pool of I/O servers.
27
Flex ASM
28. • Configurations
• Oracle Flex ASM enables Oracle ASM instances to run on a separate physical server
from the database servers.
• An Oracle ASM instance can operate in several configurations in Oracle Flex ASM
Flex ASM
28
29. • Configurations
1. Local Oracle ASM clients with direct access to Oracle ASM disks
2. Oracle Flex ASM clients with direct access to Oracle ASM disks
3. Oracle ACFS access through the Oracle ASM proxy instance
• An Oracle ASM proxy instance is an Oracle instance running on a Hub node with a direct Oracle ASM client. Oracle
Automatic Storage Management Cluster File System (Oracle ACFS) and Oracle ASM Dynamic Volume Manager (Oracle
ADVM) are supported with an Oracle ASM proxy instance.
• Names start with +APX[1-9]
• Runs with limited processes
29
Flex ASM
31. • Setup
– During the installation process with Oracle Universal Installer (OUI), you can choose the
type of the Oracle Clusterware that should be installed. The Oracle Clusterware can be an
Oracle Flex ASM deployment that manages its own storage or a regular Oracle ASM
cluster.
Flex ASM
31
32. • Setup
– If you choose to install an Oracle Flex Cluster, Oracle Flex ASM is enabled by default because
an Oracle Flex Cluster requires Oracle Flex ASM.
– Check current mode
[oracle@flex2
~]$
asmcmd
showclustermode
ASM
cluster
:
Flex
mode
enabled
32
Flex ASM
33. • To install an Oracle Flex ASM deployment, note the following:
– Categorize the networks and choose the list of networks for use as Oracle ASM
Networks.
!
!
!
!
!
!
!
!
!
!
33
Flex ASM
36. • Connectivity
• Connections from a database instance to an ASM instance are based on SQLnet
using listeners.
– The listener directs the connection to the least loaded ASM instance based on the load metric it maintains.
– The connection details are fetched from CSS global data
– The ASM instance to which the database instance connects to is listed in the database alert log:
• NOTE: ASMBconnectedtoASMinstance +ASM1 (Flex mode; client id 0x10004)
– The userid and password supplied are also managed automatically. They are supplied while establishing
the session, not while connecting.
Flex ASM
36
38. • Administration
– Client Connections
[oracle@flex2
~]$
asmcmd
lsct
data
DB_Name
Status
Software_Version
Compatible_version
Instance_Name
Disk_Group
+ASM
CONNECTED
12.1.0.1.0
12.1.0.1.0
+ASM2
DATA
_mgmtdb
CONNECTED
12.1.0.1.0
12.1.0.0.0
-‐MGMTDB
DATA
steel
CONNECTED
12.1.0.1.0
12.1.0.0.0
steel_1
DATA
– Relocate Clients: Clients are automatically relocated to another instance if an Oracle ASM instance fails. If
necessary, clients can be manually relocated using the ALTER SYSTEM RELOCATE CLIENT command. For
example:
SQL>
ALTER
SYSTEM
RELOCATE
CLIENT
'client-‐id';
38
Flex ASM
39. SQL>
@clients
!
INSTANCE_NAME
DB_NAME
STATUS
-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐
-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐
-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐
+ASM1
+ASM
CONNECTED
sport
sport
CONNECTED
+APX1
+APX
CONNECTED
+APX2
+APX
CONNECTED
-‐MGMTDB
_mgmtdb
CONNECTED
!
SQL>
alter
system
relocate
client
'sport:sport';
!
System
altered.
!
SQL>
@clients
!
INSTANCE_NAME
DB_NAME
STATUS
-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐
-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐
-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐
+ASM1
+ASM
CONNECTED
+APX1
+APX
CONNECTED
+APX2
+APX
CONNECTED
-‐MGMTDB
_mgmtdb
CONNECTED
39
Flex ASM
40. • Should the ASM instance die, it will be restarted automatically
SQL>
shutdown
abort
ASM
instance
shutdown
SQL>
startup
ORA-‐10997:
another
startup/shutdown
operation
of
this
instance
inprogress
ORA-‐09968:
unable
to
lock
file
Linux-‐x86_64
Error:
11:
Resource
temporarily
unavailable
Additional
information:
23943
SQL>
exit
Disconnected
from
Oracle
Database
12c
Enterprise
Edition
Release
12.1.0.1.0
-‐
64bit
Production
With
the
Real
Application
Clusters
and
Automatic
Storage
Management
options
[oracle@flex1
~]$
sqlplus
/
as
sysasm
!
SQL*Plus:
Release
12.1.0.1.0
Production
on
Tue
Feb
4
16:04:17
2014
!
Copyright
(c)
1982,
2013,
Oracle.
All
rights
reserved.
!
Connected
to:
Oracle
Database
12c
Enterprise
Edition
Release
12.1.0.1.0
-‐
64bit
Production
With
the
Real
Application
Clusters
and
Automatic
Storage
Management
options
!
SQL>
40
Flex ASM
41. • Flex ASM in Action
– If an ASM instance crashes, it will be automatically restarted if ASM COUNT is set to ALL
– If an ASM instance crashes, and ASM COUNT is not set to ALL, it will be restarted on the first available hub node
41
Flex ASM
45. • Proxy ASM Instance
!
– The INSTANCE_TYPE initialization parameter has an additional value ASMPROXY, in addition to ASM and
RDBMS, to identify Oracle ASM proxy instances. An Oracle ASM proxy instance has its parameter set to
ASMPROXY.
!
– An Oracle ASM proxy instance is an Oracle instance running on a Hub node with a direct Oracle ASM client.
Oracle Automatic Storage Management Cluster File System (Oracle ACFS) and Oracle ASM Dynamic Volume
Manager (Oracle ADVM) are supported with an Oracle ASM proxy instance.
45
Flex ASM
46. • Instance startup alert.log shows connection to local ASM1 Instance
Thu
Feb
06
11:13:29
2014
NOTE:
ASMB
registering
with
ASM
instance
as
client
0xffffffffffffffff
(reg:822507052)
NOTE:
ASMB
connected
to
ASM
instance
+ASM1
(Flex
mode;
client
id
0x10001)
• Instance startup alert.log shows connection to local ASM2 Instance since ASM1 is
down
Thu
Feb
06
20:18:53
2014
NOTE:
ASMB
registering
with
ASM
instance
as
client
0xffffffffffffffff
(reg:2675147253)
NOTE:
ASMB
connected
to
ASM
instance
+ASM2
(Flex
mode;
client
id
0x10004)
46
Flex ASM
47. • Instance startup alert.log shows connection to local ASM1 Instance
Thu
Feb
06
11:13:29
2014
NOTE:
ASMB
registering
with
ASM
instance
as
client
0xffffffffffffffff
(reg:822507052)
NOTE:
ASMB
connected
to
ASM
instance
+ASM1
(Flex
mode;
client
id
0x10001)
• Instance startup alert.log shows connection to local ASM2 Instance since ASM1 is
down
Thu
Feb
06
20:18:53
2014
NOTE:
ASMB
registering
with
ASM
instance
as
client
0xffffffffffffffff
(reg:2675147253)
NOTE:
ASMB
connected
to
ASM
instance
+ASM2
(Flex
mode;
client
id
0x10004)
• In the ASM2 alert.log, we can see the connection establishing
Thu
Feb
06
20:18:53
2014
NOTE:
Flex
client
id
0x0
[sport:sport]
attempting
to
connect
NOTE:
registered
owner
id
0x10004
for
sport:sport
NOTE:
Flex
client
sport:sport
registered,
osid
15584,
mbr
0x0
(reg:2675147253)
Thu
Feb
06
20:18:59
2014
Reconfiguration
started
(old
inc
6,
new
inc
8)
List
of
instances:
1
2
(myinst:
2)
Global
Resource
Directory
frozen
Communication
channels
reestablished
47
Flex ASM
48. • New Parameters for Flexing ASM?
– Nope
– The only parameters to consider for review are when performing an upgrade
– Parameter Documentation located here
+ASM1.__oracle_base='/u01/app/oracle'#ORACLE_BASE
set
from
in
memory
value
+ASM2.__oracle_base='/u01/app/oracle'#ORACLE_BASE
set
from
in
memory
value
*.asm_diskstring='/dev/asm*'
*.asm_power_limit=1
*.large_pool_size=12M
*.remote_login_passwordfile='EXCLUSIVE'
48
Flex ASM
49. • Monitoring
– My personal favorite tool is Enterprise Manager
– Allows for monitoring as well as administration of Flex ASM
49
Flex ASM
52. !
• Flex Cluster
• Flex ASM
• Your Database on a Flex Architecture
• Conclusion
Agenda
52
53. • The database is oblivious to the chaotic symphony underneath the
covers
– Clients are automatically relocated if an ASM Instance goes down
– Adding a new node to a cluster (which currently exists as a standalone) and
migration to ASM is easier
– Even though with previous releases, there is very little over-head of ASM
• Fewer resources on a Hub nodes without ASM to manage
– The database is not the super-star in the cluster!
Flex your Database
Flex
53
54. • Test Scenario #1
– Database“sport”is connected to“+ASM1”
– Database“sport”is running a transaction – a silly little loop to insert some rows
– Kill“+ASM1”SMON process
– “sport”database’s connection will be terminated and relocated to the first available ASM instance
• In some cases, you might receive an ORA-03114 or ORA-03113 error message
• In my testing, the fail-over was seamless
NOTE:
ASMB
registering
with
ASM
instance
as
client
0x10002
(reg:3110852683)
NOTE:
ASMB
connected
to
ASM
instance
+ASM2
(Flex
mode;
client
id
0x10002)
NOTE:
ASMB
rebuilding
ASM
server
state
NOTE:
ASMB
rebuilt
1
(of
1)
groups
NOTE:
ASMB
rebuilt
9
(of
9)
allocated
files
NOTE:
fetching
new
locked
extents
from
server
NOTE:
0
locks
established;
0
pending
writes
sent
to
server
SUCCESS:
ASMB
reconnected
&
completed
ASM
server
state
– No lost rows observed
– This fail-over feature doesn’t necessarily replace RAC Services
54
Flex your Database
56. • Test Scenario #2
– Database Instance“sport”running on hub node“flex1”
– “flex1”is converted to a“leaf”node
– Upon“sport”startup, we receive an error
SQL>
startup
ORA-‐01078:
failure
in
processing
system
parameters
ORA-‐01565:
error
in
identifying
file
'+DATA/sport/spfilesport.ora'
ORA-‐17503:
ksfdopn:2
Failed
to
open
file
+DATA/sport/spfilesport.ora
ORA-‐12514:
TNS:listener
does
not
currently
know
of
service
requested
in
connect
descriptor
ORA-‐15077:
could
not
locate
ASM
instance
serving
a
required
diskgroup
– Ah, but there is no local“+ASM1”instance anymore
– If“sport”originally had no dependency on“+ASM1”it would startup successfully
56
Flex your Database
57. • Node Maintenance
– Since ASM Clients can be relocated
– Database Instances may not need to be shut down
• Database Migrations from Non-ASM to ASM?
Use-Cases
Flex
57
58. !
• Flex Cluster
• Flex ASM
• Your Database on a Flex Architecture
• Conclusion
58
Agenda
59. • First phase of Flex Clusters is targeted for
– Better consolidation of resources
• Future phases of Flex Clusters are aimed to tackle resource management
– Next project would involve configuring a Hub-Leaf configuration with middle tier
• Maybe Enterprise Manager 12c?
• Flex ASM allows for
– Detached Storage Management
– Automatic ASM Instance Restarts
– ASM Proxy Instances allow for remote connections to Storage from Remote Databases
• Databases Running on Flex ASM are just as happy!
Conclusion
59
60. • Oracle Docs on Flex Cluster and Flex ASM
• Flex ASM Whitepaper
• RACAttack: 12c Setup Guide
• Martin Bach’s Blog
– Flex ASM in Action
• Ask Mcleans post on Flex Clusters
• Bertrand Drouvot’s Blog – Flex ASM Gotchas
– Findings 1 & 2
• My Noggin!
Mentionable References
60
Hub:They are tightly connected, and have direct access to shared storage. In an Oracle Flex Cluster configuration, shared storage can be provisioned to leaf nodes independent of the Oracle Grid Infrastructure.Leaf:Hub Nodes can run in an Oracle Flex Cluster configuration without having any Leaf Nodes as cluster member nodes, but Leaf Nodes must be members of a cluster that includes at least one Hub Node.
Imagine a system maintenance window when all activity needs to be offloaded to a different cluster.
Imagine a system maintenance window when all activity needs to be offloaded to a different cluster.You could effectively keep relocating a client. It will essentially kill the connection and force a failover to occur.
Imagine a system maintenance window when all activity needs to be offloaded to a different cluster.You could effectively keep relocating a client. It will essentially kill the connection and force a failover to occur.
Imagine a system maintenance window when all activity needs to be offloaded to a different cluster.You could effectively keep relocating a client. It will essentially kill the connection and force a failover to occur.