(HOWTO: E-Business Suite / Oracle Database / Middleware) Clone Automation products

Products Offering

Fast & Automated Cloning

• Very fast for P1 issues, Immediate Reporting or ETL runs.
• Immediate backup and Restore of DB and APP disk storage.
• Fully automated clone of DB Homes and Database.
• Fully automated run of AD tools for DB and MIDDLE Tiers.
• “BYOS” Bring your own script that you use for post­clone­steps.
• Clone a SOURCE to a TARGET by just pressing the “Return” key and does it all for you.

Automated Cloning (large number of environments)

• Very good when you need to refresh 5 to 20 environments during the month.
• Fully automated clone of DB Homes and Database.
• Fully automated run of AD tools for DB and MIDDLE Tiers.
• “BYOS” Bring your own script that you use for post­clone­steps.
• Uses backups to clone DB and APP tier.
• Clone a SOURCE to a TARGET by just pressing the “Return” key and does it all for you.

(HOWTO: Intelligent Home) Incredible ZeroShell - Restricting iPhone and iPads from Kids at home

Zeroshell is simple AWESOME!!!

Copyright (C) 2006-2015 Fulvio Ricciardi - Lecce (ITALY) 
Email Fulvio.Ricciardi@zeroshell.net 
Web site http://www.zeroshell.org 

My kids are constantly looking at their iPads all the time, this is putting School Homework on the side, and for parents is like impossible to take away that candy. Zeroshell is a great tool !!! 

It has a built in transparent proxy and a Captive Portal which when configured as a gateway at home it allows devices such as iPads, computers, laptops, TV, etc to be restricted by time or bytes consumed. You simply register a user for each kid and assign bucks to their accounts. You also need to setup an accounting policy, for example 1$ per hour which means that 2$ is equivalent to 2 hours of web surfing.

Once access internet you need to just simply open your browser and Zeroshell captive portal will appear, you then need to login with your username and password and after this you will be given access. A time based policy will deny access after all the credit has been consumed.

Once credit is consumed internet access will be cut in the related iPad or device. 

Your kids will then be begging Mum and Dad to give them more credits.... Here is when your negotiation skills are used at a maximum... 

Sometimes is even hard to negotiate with your kids than your boss. Eventually, after some wording and little bit of negotiation, school homework get done and everyone is happy !!! 

Thanks Fulvio !!!! Zeroshell is great !!!!

Notes: You need to change your wireless internet router DHCP settings and set the gateway to the IP of your zeroshell server. Also make sure to lock each iPad IP so that DHCP provides the same IP all the time. That should be it !!!

(HOWTO: Java) – BLOCKED Thread Locking Tree

Understanding which Thread is waiting and which Thread is blocking

Finding the root cause of a block in java may be difficult sometimes, as thread dumps may be difficult to follow, and is hard to understand which thread is waiting and which thread is locking it.

Unfortunately, I have not seen any tools that can represent clearly a lock tree of a Threads running in a java process. This is the reason I thought to build a tool that can help to determine BLOCKER and WAITER threads from a java thread dump.

The idea is to take a thread dump and store the output to a file, then process that file using a perl script which I called “thread_lock_tree.pl”. The script will represent a list of threads and its status grouping the threads that a BLOCKING together with the ones that are WAITING. All this is done in a visual way using indentation, so that everyone can understand and see.

You start by taking a thread dump, using your favorite tool, for Example jstack.

$JAVA_HOME/bin/jstack 8978 > thread_dump_pid8978.tdump

As you can see from the below, it is difficult to determine which threads are waiting for which thread to finish.

$ cat thread_dump_pid8978.tdump

"AJPRequestHandler-HTTPThreadGroup-2744" prio=10 tid=0x36f92000 nid=0x6206 waiting for monitor entry [0x18621000]

java.lang.Thread.State: BLOCKED (on object monitor)

- waiting to lock <0x84095160> (a oracle.jdbc.driver.T4CConnection)

- locked <0xe2447c99> (a java.lang.Byte)

"AJPRequestHandler-HTTPThreadGroup-2743" prio=10 tid=0x36f98c00 nid=0x6177 waiting for monitor entry [0x18255000]

java.lang.Thread.State: BLOCKED (on object monitor)

- waiting to lock <0x779ce6b8> (a java.lang.Byte)

- locked <0xe2447c59> (a java.lang.Byte)

"AJPRequestHandler-HTTPThreadGroup-2742" prio=10 tid=0x36f92800 nid=0x60d7 runnable [0x182f6000]

java.lang.Thread.State: RUNNABLE

- locked <0x84095160> (a oracle.jdbc.driver.T4CConnection)

- locked <0xe2447c58> (a java.lang.Byte)

- locked <0x779ce6b8> (a java.lang.Byte)

"AJPRequestHandler-HTTPThreadGroup-2749" prio=10 tid=0x36f92000 nid=0x6206 waiting for monitor entry [0x18621000]

java.lang.Thread.State: BLOCKED (on object monitor)

- waiting to lock <0xe2447c59> (a java.lang.Byte)

- waiting to lock <0xe2447c99> (a java.lang.Byte)

- locked <0xe2457c99> (a java.lang.Byte)

"AJPRequestHandler-HTTPThreadGroup-4749" prio=10 tid=0x36f92000 nid=0x6206 waiting for monitor entry [0x18621000]

java.lang.Thread.State: BLOCKED (on object monitor)

- waiting to lock <0xe2447c59> (a java.lang.Byte)

- waiting to lock <0xe2447c99> (a java.lang.Byte)

"AJPRequestHandler-HTTPThreadGroup-5749" prio=10 tid=0x36f92000 nid=0x6206 waiting for monitor entry [0x18621000]

java.lang.Thread.State: BLOCKED (on object monitor)

- waiting to lock <0xe2457c99> (a java.lang.Byte)

To understand the lock situation, you can parse “tdump” file using the script.
Usage: perl thread_lock_tree.pl <file.tdump>

$ perl thread_lock_tree.pl thread_dump_pid8978.tdump

~~ Analyzing thread lock tree in thread dump file: thread_dump_pid8978.tdump ~~

AJPRequestHandler-HTTPThreadGroup-2742(RUNNABLE) (2)

|_AJPRequestHandler-HTTPThreadGroup-2744(BLOCKED) (2)

  |_AJPRequestHandler-HTTPThreadGroup-2749(BLOCKED) (1)

    |_AJPRequestHandler-HTTPThreadGroup-5749(BLOCKED) (0)

  |_AJPRequestHandler-HTTPThreadGroup-4749(BLOCKED) (0)

|_AJPRequestHandler-HTTPThreadGroup-2743(BLOCKED) (2)

  |_AJPRequestHandler-HTTPThreadGroup-2749(BLOCKED) (1)

    |_AJPRequestHandler-HTTPThreadGroup-5749(BLOCKED) (0)

  |_AJPRequestHandler-HTTPThreadGroup-4749(BLOCKED) (0)

As you can see in the above output, you can easily see which threads are locking and which ones are waiting in a simple locking tree format.

Click link to download script  thread_lock_tree.pl 

(HOWTO Oracle DB) - Performance problems due Un-Balanced CPU RAC nodes

Background:
Have you seen your production RAC database using lots of CPU in 1 node and NONE on other nodes?  Node1 seems to be particularly busy comparing to Node2, LAVG reaches 40+ in Node1, comparing to LAVG 2+ in Node2.


Question is Why?

This is because the database does not consider CPU utilization at the time of connecting a session. As result we have seen unbalanced CPU utilization across nodes in the cluster. In order to fix this problem, Oracle provides the ability to change the RAC load balancing algorithm. This is done by configuring the default service to use two load balancing features “Load Balancing Advisory” (LBA) and “Connection Load Balancing” (CLB). This change can be done on the fly with using the API “dbms_service.modify_service”.

According to the Oracle documentation DBMS_SERVICE uses the following parameters to modify load balancing mechanism.

DBMS_SERVICE.MODIFY_SERVICE(
   service_name        IN VARCHAR2,
   goal                IN NUMBER DEFAULT NULL,
   dtp                 IN BOOLEAN DEFAULT NULL,
   aq_ha_notifications IN BOOLEAN DEFAULT NULL,
   failover_method     IN VARCHAR2 DEFAULT NULL,
   failover_type       IN VARCHAR2 DEFAULT NULL,
   failover_retries    IN NUMBER DEFAULT NULL,
   failover_delay      IN NUMBER DEFAULT NULL,
   clb_goal            IN NUMBER DEFAULT NULL);

GOAL_NONE = 0 (Disables Load Balancing Advisory)

GOAL_SERVICE_TIME = 1 (Load Balancing Advisory is based on elapsed time for work done in the service plus available bandwidth to the service)
 
GOAL_THROUGHPUT = 2 (Load Balancing Advisory is based on the rate that work is completed in the service plus available bandwidth to the service)

CLB_GOAL_SHORT = 1 (Connection load balancing uses Load Balancing Advisory, when Load Balancing Advisory is enabled (either goal_service_time or goal_throughput). When GOAL=NONE (no load balancing advisory), connection load balancing uses an abridged advice based on CPU utilizatio)

CLB_GOAL_LONG = 2 (Balances the number of connections per instance using session count per service. This setting is recommended for applications with long connections such as forms. This setting can be used with Load Balancing Advisory when the connection pool is sized to accommodate gravitation within the pool itself (without adding or removing connections). The latter is the most efficient design).

Recommendation:
 In order to use CPU load balancing default service need to be changes to consider CPU connection load balancing on CLB parameter (set value to CLB_GOAL_SHORT or 1), and also set the load balancing goal to throughput (set value to GOAL_THROUGHPUT or 2).

Current setup is not using CLB

SQL> col name for a25
SQL> select inst_id,name,goal,clb_goal from gv$services;

   INST_ID NAME         GOAL         CLB_G
---------- -------------------- ------------
     1 PROD             NONE         LONG
     1 PROD_PRIM        NONE         LONG
     1 SYS$BACKGROUND   NONE         SHORT
     1 SYS$USERS        NONE         SHORT
     2 PROD             NONE         LONG
     2 PROD_PRIM        NONE         LONG
     2 SYS$BACKGROUND   NONE         SHORT
     2 SYS$USERS        NONE         SHORT

In order to change CLB goal as statements as per below

SQL> EXEC DBMS_SERVICE.MODIFY_SERVICE(service_name=>'PROD',goal=>2,clb_goal=>1);
SQL> EXEC DBMS_SERVICE.MODIFY_SERVICE(service_name=>'PROD_PRIM',goal=>2,clb_goal=>1);

 

GOODNESS Indicates how attractive a given instance is with respect to processing the workload that is presented to the service. A lower number is better. This number is internally computed based on the GOAL (LONG or SHORT) that is specified for the particular service.

SQL> select end_time,inst_id,service_name,cpupercall,goodness from gv$servicemetric where service_name like 'PROD%' order by end_time desc,inst_id;
 
END_TIME           INST_ID SERVICE_NAME         CPUPERCALL   GOODNESS
--------------- ---------- -------------------- ---------- ----------
03-JUL-12                2 PROD              2647.74074        800
03-JUL-12                1 PROD              406.694915       2300
03-JUL-12                1 PROD              1882.94935       2300
03-JUL-12                2 PROD              2024.03252        800

Testing:

In order to test these changes I will be using SwingBench tool which can simulate  Sales Order Entries transactions with a large number of sessions. SwingBench tool is open source and run on JAVA.

Swingbench will be setup as follows:

• Two desktop running swingbench will be used as load generators.

• A database called EBSTEST will be used. This is a 2 node RAC 11gR1 database.

• First desktop load generator will be runnig swingbench with 500 sessions, each session will reconnect after 5000 transaction have been completed. All this sessions will be forced to connect EBSTEST1 (instance 1) of the RAC cluster. Doing this will simulate the unbalanced situation, where 500+ sessions are connect to node1 and less sessions connected to node2. Also CPU utilization will be unbalanced.

• Second desktop load generator will be using the service_name EBSTEST which can conect either node of the cluster. Connecting to this server will consider LBA and CLB balancing algorithm. 500 session will be connected and each session will re-connect every 5000 transactions are completed.Next step is to un-assing the plan from the window

Note: First load generator can be seen here running 500 session and putting load in first node.

 Note: Load can be seen here first node LAVG is 31 where second node LAVG is 1.65.

 Note: sessions number is also unbalanced. 642 in instance 1 and 140 in instance 2.

Note: service load balancing is changed on the fly to consider CPU utilization. As you can see all new sessions are connected straight to instance 2 as you can see instance 1 still hold 644 sessions where instance 2 have gone from 140 to 338 sessions.

Note: as you can see node 2 LAVG has incresed from 1.6 to 4.39, confirming new  load balancing goal is now considering CPU utilization.

(HOWTO: Oracle DB) - Ebiz database running slow due to RESOURCE MANAGER feature is enabled

Ebiz database running slow due to RESOURCE MANAGER feature is enabled

RESOURCE MANAGER feature is enable in the database by default when you install 11g. This feature limit the amount of resources a process can use. This impact all payroll processes and other batch processes and cause session to wait on an event “resmgr:cpu quntum”.

This will hopefully remove the CPU limit that is causing “resmgr:cpu quntum” waits in your production database affecting concurrent programs mainly running during business hours.

Check database parameters

SQL> show parameter resource;

NAME TYPE VALUE

------------------------------------ ----------- ------------------------------

resource_limit boolean FALSE

resource_manager_cpu_allocation integer 8

resource_manager_plan string SCHEDULER[0x7AFFC]:DEFAULT_MAINTENANCE_PLAN

Disable resource manager from spfile

SQL> alter system set resource_manager_plan='' scope=both;

System altered.

Then, check resource manager windows

SQL> select window_name, resource_plan from dba_scheduler_windows;

WINDOW_NAME RESOURCE_PLAN

------------------------------ ------------------------------

WEEKNIGHT_WINDOW DEFAULT_MAINTENANCE_PLAN

WEEKEND_WINDOW DEFAULT_MAINTENANCE_PLAN

MONDAY_WINDOW DEFAULT_MAINTENANCE_PLAN

TUESDAY_WINDOW DEFAULT_MAINTENANCE_PLAN

WEDNESDAY_WINDOW DEFAULT_MAINTENANCE_PLAN

THURSDAY_WINDOW DEFAULT_MAINTENANCE_PLAN

FRIDAY_WINDOW DEFAULT_MAINTENANCE_PLAN

SATURDAY_WINDOW DEFAULT_MAINTENANCE_PLAN

SUNDAY_WINDOW DEFAULT_MAINTENANCE_PLAN

Next step is to un-assign the plan from the window

SQL> select 'execute dbms_scheduler.set_attribute('''||window_name||''',''RESOURCE_PLAN'','''');' from DBA_SCHEDULER_WINDOWS;

'EXECUTEDBMS_SCHEDULER.SET_ATTRIBUTE('''||WINDOW_NAME||''',''RESOURCE_PLAN'','''

--------------------------------------------------------------------------------

execute dbms_scheduler.set_attribute('WEEKNIGHT_WINDOW','RESOURCE_PLAN','');

execute dbms_scheduler.set_attribute('WEEKEND_WINDOW','RESOURCE_PLAN','');

execute dbms_scheduler.set_attribute('MONDAY_WINDOW','RESOURCE_PLAN','');

execute dbms_scheduler.set_attribute('TUESDAY_WINDOW','RESOURCE_PLAN','');

execute dbms_scheduler.set_attribute('WEDNESDAY_WINDOW','RESOURCE_PLAN','');

execute dbms_scheduler.set_attribute('THURSDAY_WINDOW','RESOURCE_PLAN','');

execute dbms_scheduler.set_attribute('FRIDAY_WINDOW','RESOURCE_PLAN','');

execute dbms_scheduler.set_attribute('SATURDAY_WINDOW','RESOURCE_PLAN','');

execute dbms_scheduler.set_attribute('SUNDAY_WINDOW','RESOURCE_PLAN','');

9 rows selected.

SQL> execute dbms_scheduler.set_attribute('WEEKNIGHT_WINDOW','RESOURCE_PLAN','');

PL/SQL procedure successfully completed.

...

...

SQL> commit;

Finally, disableauto tasks

SQL> BEGIN

DBMS_AUTO_TASK_ADMIN.DISABLE(

client_name => 'sql tuning advisor',

operation => NULL,

window_name => NULL);

END;

/

2 3 4 5 6 7

PL/SQL procedure successfully completed.

SQL> SQL> commit;

All Done.

Reference: High "Resmgr:Cpu Quantum" Wait Events In 11g Even When Resource Manager Is Disabled [ID 949033.1] 

HOWTO (Oracle DB) - Oradebug to trace session, hanganalyze and statedump

ORADEBUG Session Trace

The following steps are required to trace a user session with oradebug:

Obtain the SPID from v$process

SQL> select username, spid from v$process;

Start the debug session with the SPID of the process that needs traced

SQL> oradebug setospid 2280

SQL> oradebug unlimit

·         Select the appropriate trace level. There are four different options when specifying a tracing level.

·         Level 1 – provides “base set” tracing information. Bind variables are displayed as variables (:b1).

·         Level 4 – provides Level 1 data and the actual data values of bind variables.

·         Level 8 – provides Level 1 data and information on wait events when the elapsed time is greater than the CPU time.

·         Level 12 – combines levels 1, 4 and 8 tracing information. A Level 12 trace contains base set, bind variable values and wait events.

The oradebug command below will enable the maximum tracing possible:

SQL> oradebug event 10046 trace name context forever, level 12

Turn tracing off

SQL> oradebug event 10046 trace name context off

Obtain the trace file name. The oradebug facility provides an easy way to obtain the file name

SQL> oradebug tracefile_name

c:\oracle9i\admin\ORCL92\udump\mooracle_ora_2280.trc

Format the trace file with tkprof (as described in the earlier section on tkprof)

The result will be a trace file that contains more trace file information. Viewing wait events and bind variable values can be critical to diagnosing performance issues.

System state dump

In RAC system this need to be executed on all the instance

# Logon to sqlplus as sysdba

SQL> oradebug setmypid

SQL> oradebug unlimit

SQL> oradebug dump systemstate 10

... Wait at least 1 min

SQL> oradebug dump systemstate 10

... Wait at lease 1 min

SQL> oradebug dump systemstate 10

Hanganalyze dump

Syntax hangaanalyze for single instance

# Logon to sqlplus as sysdba

SQL> oradebug setmypid;

SQL> oradebug unlimit;

SQL> oradebug hanganalyze 3;

Syntax for hangaanalyze for RAC

# Logon to sqlplus as sysdba

SQL> oradebug setmypid;

SQL> oradebug unlimit;

SQL> oradebug setinst all

SQL> oradebug -g def hanganalyze 3

Connecting to SGA in case sqlplus does not work

$ sqlplus -prelim '/as sysdba

SQL> oradbug ...

HOWTO (ebiz) - Howto to restart OPP without having to restart full Concurrent Managers

This post describes the steps to restart OPP processes manually without having to take a full outage on concurrent managers.

Sometimes OPP does not re-start normally when DBA tries to re-start OPP from the Form "Concurrent Manager -> Administer" form.

Login to APPS and navigate to Concurrent Managers/Administer form

Query for %Output Post Processor% manager.

You will see Actual=4 and Target=4 processes

Click on Restart Button. Make sure Oputput Post Processor is selected.

Wait for at least 2 minutes.

You will see Target=0 and Actual=0

If Actual is still 4, then proceed with steps below to manual kill OPP processes.

Identify OPP process in the UNIX
Login to APPTIER server and identify the OPP processes.

[appl@papp1 log]$ ps -ef |grep FNDOPP | grep -v grep
appl  6947  7203  2 10:31 ?        00:01:21 /u01/oracle/PROD/apps/tech_st/10.1.3/appsutil/jdk/bin/java -DCLIENT_PROCESSID=6947 -server -Xmx384m -XX:NewRatio=2 -XX:+UseSerialGC -Doracle.apps.fnd.common.Pool.leak.mode=stderr:off -verbose:gc -mx1024m -Ddbcfile=/u01/oracle/PROD/inst/apps/PROD_papp1/appl/fnd/12.0.0/secure/PROD.dbc -Dcpid=486574 -Dconc_queue_id=6269 -Dqueue_appl_id=0 -Dlogfile=/u01/oracle/PROD/apps/logs/appl/conc/log/FNDOPP486574.txt -DLONG_RUNNING_JVM=true -DOVERRIDE_DBC=true -DFND_JDBC_BUFFER_MIN=1 -DFND_JDBC_BUFFER_MAX=2 oracle.apps.fnd.cp.gsf.GSMServiceController
appl  6948  7203  3 10:31 ?        00:01:30 /u01/oracle/PROD/apps/tech_st/10.1.3/appsutil/jdk/bin/java -DCLIENT_PROCESSID=6948 -server -Xmx384m -XX:NewRatio=2 -XX:+UseSerialGC -Doracle.apps.fnd.common.Pool.leak.mode=stderr:off -verbose:gc -mx1024m -Ddbcfile=/u01/oracle/PROD/inst/apps/PROD_papp1/appl/fnd/12.0.0/secure/PROD.dbc -Dcpid=486575 -Dconc_queue_id=6269 -Dqueue_appl_id=0 -Dlogfile=/u01/oracle/PROD/apps/logs/appl/conc/log/FNDOPP486575.txt -DLONG_RUNNING_JVM=true -DOVERRIDE_DBC=true -DFND_JDBC_BUFFER_MIN=1 -DFND_JDBC_BUFFER_MAX=2 oracle.apps.fnd.cp.gsf.GSMServiceController
appl  6949  7203  2 10:31 ?        00:01:06 /u01/oracle/PROD/apps/tech_st/10.1.3/appsutil/jdk/bin/java -DCLIENT_PROCESSID=6949 -server -Xmx384m -XX:NewRatio=2 -XX:+UseSerialGC -Doracle.apps.fnd.common.Pool.leak.mode=stderr:off -verbose:gc -mx1024m -Ddbcfile=/u01/oracle/PROD/inst/apps/PROD_papp1/appl/fnd/12.0.0/secure/PROD.dbc -Dcpid=486576 -Dconc_queue_id=6269 -Dqueue_appl_id=0 -Dlogfile=/u01/oracle/PROD/apps/logs/appl/conc/log/FNDOPP486576.txt -DLONG_RUNNING_JVM=true -DOVERRIDE_DBC=true -DFND_JDBC_BUFFER_MIN=1 -DFND_JDBC_BUFFER_MAX=2 oracle.apps.fnd.cp.gsf.GSMServiceController
appl  6953  7203  0 10:31 ?        00:00:27 /u01/oracle/PROD/apps/tech_st/10.1.3/appsutil/jdk/bin/java -DCLIENT_PROCESSID=6953 -server -Xmx384m -XX:NewRatio=2 -XX:+UseSerialGC -Doracle.apps.fnd.common.Pool.leak.mode=stderr:off -verbose:gc -mx1024m -Ddbcfile=/u01/oracle/PROD/inst/apps/PROD_papp1/appl/fnd/12.0.0/secure/PROD.dbc -Dcpid=486577 -Dconc_queue_id=6269 -Dqueue_appl_id=0 -Dlogfile=/u01/oracle/PROD/apps/logs/appl/conc/log/FNDOPP486577.txt -DLONG_RUNNING_JVM=true -DOVERRIDE_DBC=true -DFND_JDBC_BUFFER_MIN=1 -DFND_JDBC_BUFFER_MAX=2 oracle.apps.fnd.cp.gsf.GSMServiceController

Kill OPP processes

By Killing the processes you are forcing OPP process to stop. ICM will detect all OPP processes were found dead and Manually DBA has to Activate OPP manager from the Administer (concurrent managers) form.

$ kill -9 6953  6949  6948  6947

Tail the Main concurrent Manager log and confirm OPP processes have been found dead

$ cd $APPLCSF/log/PROD_1017.mgr

NOTE 1017 is the month and date of the last time concurrent managers started. Output is is a follows. You need to confirm all four processes are found dead.

                   Process monitor session started : 10-NOV-2011 10:29:13
Found dead process: spid=(7307), cpid=(485567), Service Instance=(6269)
Found dead process: spid=(7295), cpid=(485566), Service Instance=(6269)
Found dead process: spid=(7278), cpid=(485565), Service Instance=(6269)
Found dead process: spid=(7268), cpid=(485564), Service Instance=(6269)

Confirm Actual and target processes count is 0

Navigate to Concurrent Managers/Administer form and query for %Output Post Processor% manager.

You will see Actual=0 and Target=0 processes

Click on Activate Button. Make sure Oputput Post Processor is selected.

Wait for at least 2 minutes.

You will see Target=4 and Actual=4

Tail the Main concurrent Manager log and confirm OPP processes have started

$ cd $APPLCSF/log/PROD_1017.mgr

NOTE 1017 is the month and date of the last time concurrent managers started. Output is a follows. You need to confirm all four processes have started.

                 Process monitor session started : 10-NOV-2011 10:31:50
Starting FNDCPOPP Concurrent Manager               : 10-NOV-2011 10:31:51
Starting FNDCPOPP Concurrent Manager               : 10-NOV-2011 10:31:51
Starting FNDCPOPP Concurrent Manager               : 10-NOV-2011 10:31:51
Starting FNDCPOPP Concurrent Manager               : 10-NOV-2011 10:31:52

Check OPP log files are processing concurrent requests post actions correctly

[appl@papp1 log]$ $ cd $APPLCSF/log/FNDOPP*

check timestamp, normally last four processes

[appl@papp1 log]$ ls -lrt FNDOPP*
-rw-rw-r-- 1 appl dba 2836639 Nov  9 20:10 FNDOPP485566.txt
-rw-rw-r-- 1 appl dba 5704431 Nov  9 20:10 FNDOPP485565.txt
-rw-rw-r-- 1 appl dba 1316142 Nov 10 09:09 FNDOPP485567.txt
-rw-rw-r-- 1 appl dba 1016488 Nov 10 10:21 FNDOPP485564.txt
-rw-rw-r-- 1 appl dba   43403 Nov 10 11:59 FNDOPP486577.txt
-rw-rw-r-- 1 appl dba   47016 Nov 10 11:59 FNDOPP486576.txt
-rw-rw-r-- 1 appl dba   37590 Nov 10 11:59 FNDOPP486575.txt
-rw-rw-r-- 1 appl dba   42614 Nov 10 11:59 FNDOPP486574.txt

[appl@papp1 log]$ /sbin/fuser FNDOPP486574.txt FNDOPP486575.txt FNDOPP486576.txt FNDOPP486577.txt
FNDOPP486574.txt:     6947
FNDOPP486575.txt:     6948
FNDOPP486576.txt:     6949
FNDOPP486577.txt:     6953

Output file FNDOPP486574.txt

[11/10/11 10:34:43 AM] [486574:RT10109319] Executing post-processing actions for request 10109319.
[11/10/11 10:34:43 AM] [486574:RT10109319] Starting XML Publisher post-processing action.
[11/10/11 10:34:43 AM] [486574:RT10109319]
Template code: X142R
Template app:  XC
Language:      en
Territory:     null
Output type:   PDF
...
...
...
[11/10/11 10:34:44 AM] [486574:RT10109319] XML Publisher post-processing action complete.
[11/10/11 10:34:44 AM] [486574:RT10109319] Completed post-processing actions for request 10109319.