garden1.jpg

Upgrading 11.5.9 to 11.5.10.2


This document outlines the steps involved applying 3480000 11.5.10.2 Maintenance patch to a 11.5.9
system, and might help you when applying 3480000.This is the log of final application of patch 3480000 to production system which includes all the fixes discovered during the first , second and third run of 3480000 to test systems. See 3480000 : 11.5.10.2 first run which outlines detail of all errors and fixes, verses this document meaning “3480000 : 11.5.10.2 Final run” which incorporates all the fixes before and during the application of 3480000.
Disclaimer: Apply to a TEST environment first. Use on production systems is at DBAs own risk.
Applying 3480000 : Oracle Applications Release 11.5.10.2 Maintenance

Reading Note: 316365.1

Today is 22-Nov-07
Instance: PROD.

Prerequisites:

4238286: –>Not there –>Applied to PROD on 21-Nov-07
4337683 – Minipack 11i.AD.I.2 –>Already there.
4104924 –>Already there.
4318672 Technology Stack Validation :–> Not there. Applied to PROD completed on 21-Nov-07
3180164 –>Not there –>Applied to PROD on 21-Nov-07
Prepare TCA for Human Resources update: Note:279449.1 says that if we are on Oracle HRMS Family
Pack G this data update process has already been
executed and will not run it again. According to script HRMS11i.sql (211747.1) We are
currently
on on HRMS Family Pack K so we are good here.
OS: UnZip version 5.41 (or later) –>We are on UnZip 5.42 on both node. (unzip -v)
3480000 came in 4 different pieces of zip files, unzip each, one at a time starting with 1
(p3480000_11i_HPUX11_1of4.zip)
then 2, 3 and piece 4, they will all unzip into a sub directory named 3480000
–>This took 30 minutes.

Do the following steps in this order:

To prevent some errors encountered during the first run of 3480000 do the following before starting the patch 3480000:

1)To resolve error FAILED: file egot016.odf Note:316127.1:

on Node1 identify the version of the odf file egot016.odf.
grep Header $EGO_TOP/patch/115/odf/egot016.odf
# $Header: egot016.odf 115.1 2004/10/31 23:55:50 svdubey noship $
grep Header $EGO_TOP/patch/115/odf/egov028.odf
# $Header: egov028.odf 115.2 2004/10/31 23:46:41 svdubey noship $
If the file version of egot016.odf is 115.1 and the file version of egov028.odf is 115.2, then
do as follow:
cd $EGO_TOP/patch/115/odf
cp egot016.odf egot016.odf.org
cp egov028.odf egov028.odf.org
vi egot016.odf
Replace:
# $Header: egot016.odf 115.1 2004/10/31 23:55:50 svdubey noship $

With:
START_OF_AD_ODF_FILE
2.1.3
Current system time is 04-FEB-2005 16:56:54
# $Header: egot016.odf 115.1.50400.2 2005/02/04 17:09:18 kdenisen noship $

vi egov028.odf
Replace:
# $Header: egov028.odf 115.2 2004/10/31 23:46:41 svdubey noship $

With:
START_OF_AD_ODF_FILE
2.1.3
Current system time is 04-FEB-2005 16:56:58
# $Header: egov028.odf 115.2.50400.2 2005/02/04 17:10:40 kdenisen noship $

2) To resolve error FAILED: file csctxsrs.pls Note:362344.1:

sqlplus apps
SQL> create table FND_ORACLE_USERID_112207
as select * from FND_ORACLE_USERID;

update FND_ORACLE_USERID
set INSTALL_GROUP_NUM=1
where ORACLE_USERNAME=’APPS’;

1 row updated.
SQL> commit;

3) Patch needs additional space in APPLSYSX

connect /as sysdba
ALTER TABLESPACE APPLSYSX
ADD
DATAFILE ‘/u04/oradata/proddata/applsysx06.dbf’ SIZE 100M
AUTOEXTEND
ON NEXT 100M MAXSIZE 2000M;

ALTER TABLESPACE APPLSYSD
ADD DATAFILE ‘/u02/oradata/proddata/applsysd14.dbf’ SIZE 100M
AUTOEXTEND
ON NEXT 100M MAXSIZE 2000M;

4) Applying 4656019 : EDR 11.5.10 Interoperability patch for AME

If you apply this patch as a preventive mesure before applying 3480000 then workers will failed.
But it seems to prevent the error I got the first time I applied 3480000.
So during the second and third run of 3480000 against TRAIN on July and August, I applied 4656019 to TRAIN, it failed within 5 minutes, I used adctrl and tolled all workers to quit then I started 3480000 which did not stop this time at “Error loading seed data for TRANSACTION_INPUT_VAR”.
–>This run for 15 minutes and workers failed in node1 but runs for 10 minutes in node2
no error. Using adctrl I stopped all workers in node1 then apply this to node2, then I proceed
with applying 3480000.
/u01/app/R11i/prodappl/admin/PROD/log/adpatch_u4656019.log
/u02/app/R11i/prodappl/admin/PROD/log/adpatch_u4656019.log
I will apply this after completion of 3480000. This made 3480000 not to error on “Error
loading seed data for TRANSACTION_INPUT_VAR”, 3480000 applied successfully without this error
this second run, since I got this the first time I applied 3480000 to TRAIN.

Now applying 3480000:

Stop Apps.
Enable Maintenance Mode.

Applied to PROD on 22-Nov-07 6 PM.
Apply u3480000.drv to Node1 then Node2
When prompted if you want to continue with the previous failed session (from 4656019) say no
then yes to start a new adpatch session.
On node1 it run for 1 hour and gave the following error:

Interruption 1 while applying 3480000:

adrelink is exiting with status 1

End of adrelink session
Date/time is Thu Nov 22 18:56:10 PST 2007

An error occurred while relinking application programs.
Continue as if it were successful [No] :
–>To this I answered Yes, since we do not use module WIP.

–>It would then run for close to 2 hours and prompt you to drop the FND_INSTALL_PROCESSES
table from the previous adpatch failed session (from 4656019): say yes.
–>around 42000 jobs….

It run for another 9 hours and gave the following error:

Interruption 2 while applying 3480000:

There are now 7700 jobs remaining (current phase=A132):
ATTENTION: All workers either have failed or are waiting:

FAILED: file XMLImporter.class on worker 1.

cd $APPL_TOP/admin/PROD/log
and look at the log for worker 1 adwork001.log:

Importing /oracle/apps/pqh/selfservice/common/webui/QuickSearchRN
Error: Unable to save package file “/oracle/apps/pqh/selfservice/common/webui/QuickSearchRN”
since a document or package of th
e same name already exists.
ORA-20101:
ORA-06512: at “APPS.JDR_MDS_INTERNAL”, line 1988
ORA-06512: at line 1

–>I searched Metalink and found Note:396855.1 which discribes a similar problem but with a
different
package, the notes give the following solution:
connect as apps

BEGIN
jdr_utils.DELETEPACKAGE(’/oracle/apps/pqh/selfservice/common/webui/QuickSearchRN’);
END;
/

BEGIN
jdr_utils.DELETEDOCUMENT(’/oracle/apps/pqh/selfservice/common/webui/QuickSearchRN’);
END;
/

–> you should get the following message for both: PL/SQL procedure successfully completed.

Restart the failed worker using adctrl

–>This sloution worked fine and adpatch restarted and progressing.

–> I restarted the patch on 23-Nov-07 at 4:40 AM.

–> On Node1 It run for 3 more hours without error and finished on Node1 at 7:20 AM:
Log node1: /u01/app/R11i/prodappl/admin/PROD/log/adpatch_u3480000.log
–>On Node1 it run for a total of 12 hours.
Node2: u3480000.drv
On 23-Nov-07 at 8 AM strarting the patch on Node2.

On Node2 it run for 3 and half hours and gave the following error:

Interruption 3 while applying 3480000:

There are now 8 jobs remaining (current phase=Done):

The following Oracle Forms objects did not generate successfully:

au resource PNSULOCN.pll
au resource PNTSPACE.pll

cd $APPL_TOP/admin/TRAIN/log
And look at the workers log but nothing there, I searched Metaling for PNSULOCN.pll and found
Note:332529.1
which tells me to ignore this message and once the 11.5.10 patch is completed appy Patch
4550280
So I answered yes to continue:
Continue as if it were successful [No] : Yes

–>On Node2 it started to run again on 23-Nov-07 at 11:30 AM.

–> On node2 it run for another 2 hours no errors. Patch completed.
–> On Node2 completed at 3:45 PM. Run out of space on archive location moved some archived
logs to /u16.
Logfiles:
Node1: /u01/app/R11i/prodappl/admin/PROD/log/adpatch_u3480000.log
Node2: /u02/app/R11i/prodappl/admin/PROD/log/adpatch_u3480000.log

3480000 post patches:
–>The remaining of this document post tasks took over 2 hours to complete in TEST, not
including the help patch………..

Apply 4656019 : see document: Applying 4656019.txt
–>Applied to PROD on 23-Nov-07

Apply Patch 4550280:
apply Patch 4550280 : PNSULOCN.PLL & PNTSPACE.PLL WILL NOT GENERATE IN 11.5.FIN_PF.G AFTER APPLYING FIN.CU2
No prerequisites.
Apply u4550280.drv first to Node1 then Node2:
On node1 it run for 2 minutes no errors, node2 1 minutes no errors:
Logfiles:
Node1: /u01/app/R11i/prodappl/admin/PROD/log/adpatch_u4550280.log
Node2: /u02/app/R11i/prodappl/admin/PROD/log/adpatch_u4550280.log
Adadmin –>Generate Forms. On node2 for PN module only. –>20 minutes on node2 for
module PN no errors.
–>Disable Maintenance Mode.

Successfully restarted all apps and run a test concurent program.
–>there are some stuff running in the scheduler that look like stuff started by the upgrade
process such as “WRKR(02) (Purchasing post upgrade ….). Do not disturbe these and let them complete.
You will see two different gather stats generated by the upgrade process to run for ALL schema
for 10%, you may cancel both of them otherwise the rest of the upgrade will be very slow, also
schedule one gather stats to run after the 11.5.10.2 upgrade during the night for ALL schemas
and for 30%. If you feel that concurrent managers are hung by seeing no IO or CPU at the OS
but seeing Gather stats in the queue running give it a 10 or 15 minutes and if it is still hung
then stop all and gracefull bounce of the db, do the invalid obj as shown in the rest of this
do and then restart apps.
–>Once you cancel the gather stats, all the rest runs for around 30 minutes to
completion.

Drop MRC Schema:
logon as applprod
cd $APPL_TOP/admin
sqlplus SYSTEM/<SYSTEM password> @addrpmrc.sql <APPLSYS USERNAME> SAFE
(sample: sqlplus SYSTEM/This email address is being protected from spambots. You need JavaScript enabled to view it. applsys SAFE )
I get the following error:
ORA-20000: MRC schema name not registered in FND_ORACLE_USERID
–>This just means that MRC is not registred in our system. No additional actions are required.
Workflow:
see Doc ID: 268085.1 –>Not done since we do not use workflow at the time when this
patch was not applied.

Post patch Payables:
Using Payables Manager resposibility verify that concurrent program: ” Mini-pack Upgrade Report
” has been successfuly run by the upgrade process.
–>This was run for one minute no error.

Post patch Receivables:
Using Receivables Manager resposibility verify that concurrent program: ” Profile Option
Migration Report ” has been successfuly run by the upgrade process.
–>This was run for one minute no error.

Post patch Human Resources:
This maintenance pack includes HRMS Family Pack I, which is not separately available from
OracleMetaLink.
If you installed a later HRMS Family Pack prior to applying this maintenance pack, your HR
legislative data is up to date,
and no further action is required.
–>According to acript HRMS11i.sql (211747.1) we are on HRMS Family Pack K which is >
that HRMS Family Pack I
so our HR legislative data is up to date, and no further action is required.

Post patch Payroll:
If you did not apply HRMS Family Pack H (or later) before you applied this maintenance pack,
follow the tasks outlined in this section.
–>According to acript HRMS11i.sql (211747.1) we are on HRMS Family Pack K which is >
that HRMS Family Pack H
so no further action is required.

Post patch Time and Labor:
We are on 11i.HXT.J and we were running 11.5.9 before the upgrade.
–>so no further action is required.

On Node2 there large files (close to 300 MB) in $COMMON_TOP/admin/log/PROD_http_node filling
/u03, stuff like Events10.log
Metalink Note:342869.1 :jtffmctl.sh Creates Events.Log Files Which Fill the Disk.
Solution:
On node2 logon as applprod
cd $COMMON_TOP/admin/scripts/PROD_http_node
cp jtffmctl.sh jtffmctl.sh.org
vi jtffmctl.sh
Remove or Comment out the references to the parameters
-Dengine.LogLevel=9
-Ddebug=full
Restart apps.
If they do not zero out, then reduce the large ones to zero doing something like the following:
> Events16.log

Resolving Invalid objects:
OWNER OBJECT_NAME OBJECT_TYPE CREATED STATUS
———- ———————————– —————— ——— ——-
APPS AP_WEB_CREDIT_CARD_WF PACKAGE BODY 12-OCT-00 INVALID
APPS CS_WF_ACTIVITIES_PKG PACKAGE BODY 01-MAY-00 INVALID
APPS PA_PROJECT_PARTIES_PVT PACKAGE BODY 21-DEC-00 INVALID
APPS PA_PROJECT_PARTIES_UTILS PACKAGE BODY 21-DEC-00 INVALID
PACKAGE BODY APPS AP_WEB_CREDIT_CARD_WF Found note 334096.1
which tells me to apply patch 4341684
PACKAGE BODY APPS CS_WF_ACTIVITIES_PKG This was invalid before
the patch. Found Note:375355.1 which tells me to apply patch 4619025
PACKAGE BODY APPS PA_PROJECT_PARTIES_PVT TAR# 17441864.6
PACKAGE BODY APPS PA_PROJECT_PARTIES_UTILS TAR# 17441864.6

Invalid objects AP_WEB_CREDIT_CARD_WF:
alter PACKAGE AP_WEB_CREDIT_CARD_WF compile BODY ;
Warning: Package Body altered with compilation errors.

SQL> show error
Errors for PACKAGE BODY AP_WEB_CREDIT_CARD_WF:

292/11 PLS-00323: subprogram or cursor ‘SENDDEACTIVATEDNOTIF’ is
declared in a package specification and must be defined in the
package body

–>Found note 334096.1 which tells me to apply patch 4341684
Applying 4341684 : 11I.OIE.J (11i.FIN_PF.G) Rollup #2
Using adadmin I applied u4341684.drv to node1 then node2 (hotpatch):
On each node it run for 30 minutes no errors.
Logfiles:
Node1: /u01/app/R11i/prodappl/admin/PROD/log/adpatch_u4341684.log
Node2: /u02/app/R11i/prodappl/admin/PROD/log/adpatch_u4341684.log
–>This did correct the problem with AP_WEB_CREDIT_CARD_WF which is now valid.

Invalid objects PA_PROJECT_PARTIES_:
TAR:
Oracle says to apply patch 4667949 to fix invalid objects PA_PROJECT_PARTIES_PVT and
PA_PROJECT_PARTIES_UTILS
Applying 4667949 : Oracle Projects Rollup Patch 1 for Oracle Projects Family Pack L10
(11i.PJ_PF.L10), or Oracle Applications Maintenance Pack 11.5.10
Using adpatch I applied u4667949.drv first to Node1 then node2
On each node it run for 35 minutes no errors.
Logfiles:
Node1: /u01/app/R11i/prodappl/admin/PROD/log/adpatch_u4667949.log
Node2: /u02/app/R11i/prodappl/admin/PROD/log/adpatch_u4667949.log
–>Patch 4667949 did correct the problem with PA_PROJECT_PARTIES_PVT and
PA_PROJECT_PARTIES_UTILS which are now valid.

Invalid objects CS_WF_ACTIVITIES_PKG:
Applying 4619025 : ATG11510RUP3D1INST:DB INVALID PACKAGE CS_WF_ACTIVITIES_PKG
Using adadmin I applied u4619025.drv first to Node1 then node2
On each node it run for 3 minutes no errors.
Logfiles:
Node1: /u01/app/R11i/prodappl/admin/PROD/log/adpatch_u4619025.log
Node2: /u02/app/R11i/prodappl/admin/PROD/log/adpatch_u4619025.log
–>Patch 4619025 did correct the problem with CS_WF_ACTIVITIES_PKG which is now valid.

Increasing the application idle time out:
increase the ICX:Session Timeout to 90 minutes from default 30 minutes.

Resolution of reported problems while testing 11.5.10.2 upgrade:
Case 1):

super user reported the following problem while testing TRAIN after the 11.5.10.2 patch:
Description of the problem:
Using Oracle Web ADI responsibility.
Navigate to any menu function and you get:
Web ADI Configuration Error.
BNE_UIX_PHYSICAL_DIRECTORY is not configured.
Note:430991.1
Solution:
To implement the solution, please execute the following steps:
1. Go into the responsibility: System Administrator
2. Navigate to Profile->System->Profile: BNE%Directory%
3. Remove any value in the profile ” BNE UIX PHYSICAL DIRECTORY ”
4. Bounce the web server and retest the issue.
5. Migrate the solution as appropriate to other environments.
–>This did fix the problem.
–>End Case 1.

Case 2):
Email from super user reporting a problem while testing the 11.5.10.2 patch on TRAIN:
After our 11.5.10 upgrade, we keep receiving a message to populate the project calendar. Once
the calendar is populated, then the message is no longer received for this project.
Reference Note:388742.1
The following soltion was provided to him by Oracle:

First logon as sysadmin and make sure system profile “PA: Default Calendar” does point to
Project Calendar
Then on node1:

cd $PA_TOP/patch/115/sql
sqlplus /nolog
connect apps
spool c1279530.sql.log
@c1279530.sql
–>This would run for less that 40 seconds and give you:
PL/SQL procedure successfully completed.

Commit complete.

Now if you run the following SQL all calendar_id shoud be set to calendar_id=1 :
select calendar_id,count(*)
from pa_projects_all
group by calendar_id;

CALENDAR_ID COUNT(*)
———– ———-
1 4431

–>End Case 2.

Help patches:
apply 3275245 : 11.5.10 ONLINE HELP CONSOLIDATED PATCH
No prerequisites no down time.
cd /u14/stage/patches/applied/11.5.10.2/prerequisites/3275245/3275245
adpatch options=hotpatch u3275245.drv to node1 the node2.
On Node1 it run for 12 hours no errors on one parallel process. On node2 it run for 10 minutes
no errors.

Logfiles:
Node1: /u01/app/R11i/prodappl/admin/PROD/log/adpatch_u3275245.log
Node2: /u02/app/R11i/prodappl/admin/PROD/log/adpatch_u3275245.log
–>You do not have to wait for the Help patch to complete, users may acess the system while
this is being applied.
Timestamp of $PAY_TOP/reports/US/PAYUSGAR.rdf after patch:
ll $PAY_TOP/reports/US/PAYUSGAR.rdf
-rw-rw-rw- 1 applprod oaa 327680 Aug 15 2006

/u01/app/R11i/prodappl/pay/11.5.0/reports/US/PAYUSGAR.rdf

–>Has not been changed.
In file $FND_TOP/reports/PSTPL.prt change the line
printer Pasta Portrait Letter”
to
printer “Pasta Portrait Letter”
i.e. insert double quote before ‘Pasta’.
–>Has not been changed.
Additional path for FORMS60_PATH
As target application owner, connect to the node where form server is running.
cd $APPL_TOP
cat APPSORA.env
vi $APPL_TOP/<SID>_host.env (sample: /u03/app/R11i/patchappl/PATCH_http_node.env)
and search for FORMS60_PATH and add the following right after the first entery where
FORMS60_PATH is defined:
FORMS60_PATH=$FORMS60_PATH:$AU_TOP/forms/US; export FORMS60_PATH
à form server needs to be bounced for this to take effect.
Invalid objects: No invalid objects owned by apps..

Update patch log spreadsheet. Done.
Completed on 23-Nov-07 at 7 PM.

Latest Blog Items

email

Thank you for your interest in idbasolutions.

Contact us using this email: contact.us@idbasolutions.com