HOWTO (eBiz) - E-Business Suite R12 Apps DBA full implementation project

During 2007/2008 I had the chance to work on a full implementation of eBiz version R12. I was the main DBA in site and my scope was the delivery of the Production eBiz cluster and rest of support environments. I was involved in the architecture design, installation of the OS via PXE boot (process that I built), installation of R12 in cluster both on the database tier and application tier, initial configuration of eBiz, HA test and documentations, and the cloning process on a RAC and MULTINODE configuration.
The interesting setup of this eBiz is the utilisation of the Shared APPL_TOP on a READ-ONLY file system. This is possible thanks to the new R12 features of the INST_TOP which is the separation of the config and log files outside the APPL_TOP.

Another interesting setup was the way I configured Concurrent Managers in HA mode or Parallel Concurrent Processes, ICM and FNDSM are the key components of this setup, I had lots of problems with TNS session timeout during fail-over tests which I managed to fix by with the help a lot of older metalink notes. I used OCFS2 to shared key directory path locations needed by the MULTINODE setup. Among these locations were APPLCSF, APPLPTMP, APPLTMP, FNDREVIVER, BNE and the list keep growing, this is not very well documented by Oracle and it was difficult to troubleshoot.

These days OCFS2 has been certified to be used on shared APPL_TOP configuration and there is not need to use READ-ONLY file system implementation anymore, however this configuration works perfectly. Other solutions are to use an external NAS device over NFS to mount APPL_TOP (please do not use local NFS server in apps eBiz nodes since this will introduce a single point of failure, some people don't think about HA all the time, so please speak to your System Architect before taking any architectural decision).

Cloning 8 nodes eBiz was really a challenge, specially when you use RAC and MULTINODE setup on a Shared APPL_TOP. The process is not very well documented, and ADCLONE tool are not fully prepared to manage a non-standard setup. I decided to use a manual clone process, where I had to run FND_SETUP.CONC_CLEAN and adfcfclone dbTechStack and adcfgclone appsTechStack. When you use MULTINODE and RAC there are other things to consider like running ADCONFIG which very tricky. 
 
If you have 4 database node, you need to run adconfig in the follwing order node1 , node2, node3 and node4, at this stage people normally think adconfig was run in all the nodes and that's it, but that is not really true... If you have a look at the tnsname.ora for example on the node1, you will see there is no reference of the rest of the nodes in the “ADDRESS LIST”, Why? This is because of a limitation of ADCONFIG which grabs the information of the nodes from the FND tables which are populated as ADCONFIG runs. 
 
To workaround this problem, your need to re-run ADCONFIG twice. For example: run ADCONFIG in node1, node2, node3 and node4, at this stage node4 know about (node 1,2,3,4), node3 know about (1,2,3 but not 4), node2 knows about (1,2 but not 3,4) and node1 knows about (1 but not 2,3,4). So you need to re-run ADCONFIG in this order node3, node2 and node1 to fix the problem. After running ADCONFIG in the remaining nodes TWICE check tnsnames.ora and you will see “ADDRESS LIST” contain reference to all the cluster nodes. This workaround applies both for APP TIER and DB TIER. Hopefully Oracle will fix their documentation soon and properly document this process.

Hardware/Software Specs:

HP class 7000 blade chassis. Composed of 16 blades.
Citrix Netscaler Load Balancers

Redhat 4 EL x86_64
Oracle CRS/ASM/RDBMS V10.2.0.4 and EBS R12.0.4

Contributions:
Thanks to Neil and Dipak for reviewing this documentation and helping all the way through the project.

Full documentation:

This post has references to step-by-step build documentation for the E-Business Suite OS build, Database RAC and Shared APPL_TOP read-only implementation, Multinode setup using Parallel concurrent processing. These documents are very long, see the document index below to have an idea of the content.

Please post a comment including your email address and I will send you the password.

ebsr12_os_build.pdf

ebsr12_oracle_rac_build.pdf

ebsr12_oracle_multinode_build.pdf

ebsr12_clone.pdf

I am asking for a minimum donation via PayPal logo located on the right hand side of my blog (this can come as part of your department's training budget). Alternatively, you can engage my services directly, please contact me on my email.

Click link to download full documentation ebsr12_multinode_rac_full_cluster.tgz.gpg (password will be emailed to you when you Donate).

Powered by

http://www.gnupg.org/download

Hardware Setup, Linux Red Hat EL 4 Build using PXE Boot, Backup Configuration, Maintenance tasks and Procedures

1 Document Control 4

1.1 Signoff 4

1.2 Document Owner 4

1.3 Distribution 4

1.4 Change History 4

2 Purpose 5

2.1 Scope 5

3 Hardware Setup 6

3.1 Overview of HP Blade System c7000 Enclosure 6

3.2 HP Blade System c7000 Enclosure Configuration 8

3.3 ProLiant c-Class Server Blade architecture 11

3.4 Network/SAN Interconnect Bays 13

3.5 Power and Thermal Management 14

4 Linux Red Hat OS Build using PXE Boot process 16

4.1 PXE boot server configuration using Linux 17

4.2 PXE boot server configuration using Microsoft SMS 19

4.3 Configuring the NFS server to store Red Hat Source packages 21

4.4 Start the OS build process 23

5 Configure Kernel Huge Pages for Oracle SGA 26

5.1 HugePages Features 26

5.2 HugePages setup 26

6 Configure CUPS unix printing 29

6.1 Configure cups web access 29

6.2 Add a printer to unix 30

7 Account set up and password policies 31

7.1 Define UNIX password policies 31

7.2 Set up sudo access 32

7.3 Set ACLs on JAVA_TOP and DEE_TOP 32

8 Install and configure CommVault UNIX File System agent 34

8.1 Install the file system agent 34

9 Install and configure CommVault Oracle agent 36

9.1 Install the Oracle agent 36

9.2 Configure the Oracle dataset backup on tape 37

9.3 Perform a disaster recovery of the database from a tape backup 42

10 Setting up backup storage policy and schedules in CommVault 54

10.1 CommVault Backup schedules 54

10.2 CommVault Storage Policies 55

10.3 Current backup policy 56

11 Appendix 57

11.1 FILE: ks-linux-oracle.cfg 57

11.2 FILE: fix_ip.sh 63

11.3 FILE: /etc/sudoers 66

11.4 FILE: create_accounts.sh 68

11.5 FILE: pfile.uat 69

Oracle 10g Release 2 Real Application Clusters and Automatic Storage Management with Oracle E-Business Suite Release 12, Configuring Application Tier using READ-ONLY file system

1 Document Control 4

1.1 Signoff 4

1.2 Document Owner 4

1.3 Distribution 4

1.4 Change History 4

2 Purpose 5

2.1 Scope 5

3 Oracle 10g Release 2 Real Application Clusters and Automatic Storage Management with Oracle E-Business Suite Release 12 6

3.1 Network Configuration 6

3.2 Disk configuration 7

3.3 EBS-R12 installation (non-cluster) 8

3.4 Install Oracle Clusterware 10.2.0.1 30

3.5 Install Oracle Clusterware patchset 10.2.0.4 54

3.6 Configure ONS to support multiple accounts configuration 60

3.7 Install Oracle ASM software 10.2.0.1 62

3.8 Install Oracle ASM patchset 10.2.0.4 69

3.9 Clone Oracle Database home from existing DB Tech Stack home 74

3.10 Create ASM Instances and Disk Groups 78

3.11 Convert 10gR2 Database to Oracle RAC using rconfig 99

3.12 Enable AutoConfig on Database Tier 104

3.12.1 Enable AutoConfig in First DB TIER node 104

3.12.2 Enable AutoConfig on all other DB TIER nodes 111

3.12.3 AutoConfig Post actions on all DB TIER nodes 115

3.13 Establish Applications Environment for Oracle RAC 118

4 Configuring Application Tier cluster using Read-Only file system implementation 120

4.1 Prepare existing node 120

4.2 Make the Applications files accessible 120

4.3 Configure the node to be added 120

4.4 Finishing Tasks 122

4.4.1 DNS Load Balancing with Single Web Entry Point 122

4.4.2 Implement Parallel Concurrent Processing 122

5 APPENDIX 125

5.1 FILE: rac-prereqs.sh 125

5.2 FILE: asm_build.sh 129

5.3 FILE: UAT_ConvertToRAC.xml 131

5.4 FILE: tnsnames.ora (DB TIER Node1) 133

5.5 FILE: tnsnames.ora (DB TIER Node2) 136

5.6 FILE: listener.ora (DB TIER node1) 139

5.7 FILE: listener.ora (DB TIER node2) 141

5.8 FILE: rac-enabled-pfile.ora 143

5.9 FILE: spfile (for the ASM) 161

5.10 FILE: ons_config_db.tmp 162

5.11 FILE: ons_config – CRS_HOME 163

E-Business Suite R12 Multi-Node configuration sharing the Application Tier File System as Read-Only, Network Load Balancing configuration and Parallel Concurrent Processing

1 Document Control 3

1.1 Signoff 3

1.2 Document Owner 3

1.3 Distribution 3

1.4 Change History 3

2 Purpose 4

2.1 Scope 4

3 Multi-Node and shared application tier file system concepts 5

4 Network Load balancing concepts 6

5 Parallel Concurrent Processing Concepts 8

6 Maintaining a Shared Application Tier File System 9

7 Add a node to a Shared Application Tier file system (Read-Only Implementation) 10

7.1 Prepare the Source System 11

7.2 Make the application files accesible to the Secondary node 11

7.3 Configure the node to be added 12

8 Network Load balancing for the Web Entry Point services 17

8.1 Configure Load Balancers with Single Web Entry Point 19

8.2 Enable OC4J clustering on the application tier nodes 22

9 Configuring Parallel Concurrent Processing 24

9.1 Configure a shared file system for PCP 25

9.2 Enabling Concurrent Processing service on App Tier nodes 29

10 APPENDIX 37

10.1 FILE: igccmprf.sh 37

10.2 FILE: mod_oc4j_conf_1013.tmp 40

10.3 FILE: build_ocfs2.sh 41

E-Business Suite R12 environment cloning procedures

1 Document Control 3

1.1 Signoff 3

1.2 Document Owner 3

1.3 Distribution 3

1.4 Change History 3

2 Purpose 4

2.1 Scope 4

3 Cloning process in summary 5

4 Clone DBTIER from source system RAC to target system RAC 6

4.1 Prerequisites 7

4.2 Prepare the Source System 8

4.3 Perform clone steps in DB TIER Target System 9

4.3.1 Enable AutoConfig on Applications Database Tier 26

4.3.2 Enable AutoConfig on all other DB TIER nodes 31

4.3.3 AutoConfig Post actions on all DB TIER nodes 35

4.4 Perform clone steps in APP TIER Target System 38

4.5 Perform post task clone actions (all app tiers) 42

5 APPENDIX 43

5.1 FILE: rac-enabled-pfile.ora 43

5.2 FILE: tnsnames.ora (node2) 59

5.3 FILE: listener.ora (node2) 62

5.4 FILE: listener.ora (node1) 64

5.5 FILE: tnsnames.ora (node1) 66

5.6 FILE: ons_config_db.tmp 69

5.7 FILE: adlsnr10RAC.ora 70

5.8 FILE: /tmp/clone.patch 71

5.9 FILE: /staging/oracle/cmclean.sql 74

5.10 FILE: /staging/oracle/coins-ebs.sql 79