garden3.jpg
Upgrading 11.5.9 to 11.5.10.2


This document outlines the steps involved applying 3480000 11.5.10.2 Maintenance to an 11.5.9
system, and might help you when applying 3480000.
This is the log of the first application of patch 3480000 to test system which includes detail errors (and fixes) encountered during the first run of 3480000 to test systems.
Disclaimer: Apply to a TEST environment first. Use on production systems is at DBAs own risk.
Applying 3480000 : Oracle Applications Release 11.5.10.2 Maintenance

Reading Note: 316365.1

Today is 24-Apr-07
TRAIN:

Prerequisites:

4238286: –>Not there –>Applied to TRAIN on 24-Apr-07
4337683 – Minipack 11i.AD.I.2 –>Already there.
4104924 –>Already there.
4318672 Technology Stack Validation :–> Not there. Applied to TRAIN completed on 01-May-07
3180164 –>Not there –>Applied to TRAIN on 01-May-07
Prepare TCA for Human Resources update: Note:279449.1 says that if we are on Oracle HRMS Family Pack G this data update process has already been executed and will not run it again. According to script HRMS11i.sql (211747.1) We are currently on on HRMS Family Pack K so we are good here.
OS: UnZip version 5.41 (or later) –>We are on UnZip 5.42 on both node.
3480000 came in 4 different pieces of zip files, unzip each, one at a time starting with 1 (p3480000_11i_HPUX11_1of4.zip)
then 2, 3 and piece 4, they will all unzip into a sub directory named 3480000
–>This took 30 minutes.

Now applying 3480000:

Stop Apps.
Enable Maintenance Mode.

Applied to TRAIN on 02-May-07.
Apply u3480000.drv to Node1 then Node2
On node1 it run for 1 hour and gave the following error:

Interruption 1 while applying 3480000:

Done with link of product ‘wip’ on Wed May 2 12:28:12 PDT 2007
adrelink is exiting with status 1

End of adrelink session
Date/time is Wed May 2 12:28:17 PDT 2007
An error occurred while relinking application programs.
Continue as if it were successful [No] :
–>To this I answered Yes, since we do not use module WIP.

On Node1 it run for another 2 hours and gave the following error:

Interruption 2 while applying 3480000:

FAILED: file egot016.odf on worker 3.
ATTENTION: Please fix the above failed worker(s) so the manager can continue.

–>I cd to $APPL_TOP/admin/TRAIN/log
I look at the log of worker 3 (adwork003.log) which has failed.
Workers log file shows:
AD Worker – adussfrt: INFO: Do not understand token START_OF_AD_ODF_FILE in ODF file. Line is:
START_OF_BUILDING_BLOCK

AD Worker – adussfrh: ERROR: ODF file does not begin with standard header information

AD Worker – adusrtrf: ERROR: Unable to read header information
from the ODF file

–> end of worker log.
I search Metalink and found Note:316127.1 which tells me to do the following:

Identify the version of the odf file egot016.odf.
grep Header $EGO_TOP/patch/115/odf/egot016.odf
# $Header: egot016.odf 115.1 2004/10/31 23:55:50 svdubey noship $

grep Header $EGO_TOP/patch/115/odf/egov028.odf
# $Header: egov028.odf 115.2 2004/10/31 23:46:41 svdubey noship $

If the file version of egot016.odf is 115.1 and the file version of egov028.odf is 115.2, then
do as follow:
cd $EGO_TOP/patch/115/odf
cp egot016.odf egot016.odf.org
cp egov028.odf egov028.odf.org
vi egot016.odf
Replace:
# $Header: egot016.odf 115.1 2004/10/31 23:55:50 svdubey noship $

With:
START_OF_AD_ODF_FILE
2.1.3
Current system time is 04-FEB-2005 16:56:54
# $Header: egot016.odf 115.1.50400.2 2005/02/04 17:09:18 kdenisen noship $

vi egov028.odf
Replace:
# $Header: egov028.odf 115.2 2004/10/31 23:46:41 svdubey noship $

With:
START_OF_AD_ODF_FILE
2.1.3
Current system time is 04-FEB-2005 16:56:58
# $Header: egov028.odf 115.2.50400.2 2005/02/04 17:10:40 kdenisen noship $

–> End of Note:316127.1

Now using adctrl restart the failed worker.
When prompted by adctrl answer as follow, leave blank APPLSYS and provide apps password:
Enter the ORACLE username of Application Object Library [APPLSYS] :
Enter the ORACLE password of Application Object Library [APPS] : train

–>The worker restarted fine on 02-May-07 at 3:50 PM and patching continued without error.
It run for another hour and half and gave the following error:

Interruption 3 while applying 3480000:

ATTENTION: All workers either have failed or are waiting:

sqlplus -s APPS/***** @/u06/app/R11i/trainappl/admin/TRAIN/out/p001invok.sql &un_apps

Package created.

declare
*
ERROR at line 1:
ORA-01403: no data found
ORA-06512: at “SYSTEM.AD_DDL”, line 2280
ORA-06512: at line 15

–>I searched Metalink and found Note:362344.1 which tells me to do the following:

sqlplus apps/<password>

SQL> create table FND_ORACLE_USERID_050207
as select * from FND_ORACLE_USERID;

update FND_ORACLE_USERID
set INSTALL_GROUP_NUM=1
where ORACLE_USERNAME=’APPS’;

1 row updated.

SQL> commit;
Restart the failed worker. Use “adctrl” utility, option 2.
–>This fixed the problem.

–>The worker restarted fine on 02-May-07 at 7 PM and patching continued without error.

It run for another 6 hours and gave the following error:

Interruption 4 while applying 3480000:

ATTENTION: All workers either have failed or are waiting:

FAILED: file okcview.ldt on worker 1.
FAILED: file ontview.ldt on worker 2.
FAILED: file oksview.ldt on worker 3.

cd $APPL_TOP/admin/TRAIN/log
and look at the log for worker 1 adwork001.log:
adwork001.log points me to l740315.req, looking at the content of l740315.req:
It is pointing to this error:
Error loading seed data for FND_VIEW_COLUMN
–>THis did not return any thing on Metalink.
Looking at l740296.req it looks like I have run out of space on applsysx:
Upload from stage tables
Error loading seed data for FND_TABLE: APPLICATION_SHORT_NAME = PA, BUILDING_BLOCK_NAME =
PA_PLAN_RES_DEFAULTS, TABLE_NAME = PA_PLAN_RES_DEFAULTS, ORA-01654: unable to extend index
APPLSYS.FND_TABLES_U2 by 1280 in tablespace APPLSYSX

I attempted to add a new datafile:
ALTER TABLESPACE APPLSYSX
ADD
DATAFILE ‘/u08/oradata/traindata/applsysx05.dbf’ SIZE 100M
AUTOEXTEND
ON NEXT 100M MAXSIZE 2000M;

ALTER TABLESPACE APPLSYSD
ADD DATAFILE ‘/u07/oradata/traindata/applsysd14.dbf’ SIZE 100M
AUTOEXTEND
ON NEXT 100M MAXSIZE 2000M;
But I got:
ORA-00059: maximum number of DB_FILES exceeded

I had to modify init parameter db_files in the init file from 512 to 812 and bounce the database.
After the bounce I was able to successfully add a new datafile.
As the result of the database bounce, current adpatch was interupted but once restarted adpatch when prompted to continue where it left of I answered yes and adpatch picked up where it left off. It runs for 10 minutes and goes right where it failed, now I have to run adctrl and tell all workers to restart, after a few minutes workers status shows as running and they pass the point where they failed.
–>After error resolution, adpatch restarted on node1 on 03-May-07 at 9:35 AM.

It run for 1 1/2 hours and gave this error:

Interruption 5 while applying 3480000:

ATTENTION: All workers either have failed or are waiting:

FAILED: file bomrtgcr.ldt on worker 1.
FAILED: file bombomup.ldt on worker 2.
FAILED: file bombomcr.ldt on worker 3.

cd $APPL_TOP/admin/TRAIN/log
and look at the log for worker 1 adwork001.log:
adwork001.log: –>points to l742818.req which points to error:
Error loading seed data for TRANSACTION_INPUT_VAR

Adpatch log on node1:
log: /u06/app/R11i/trainappl/admin/TRAIN/log/adpatch_u3480000.log

I serached Metalink and found Note: 365796.1 which says to apply 5148708
–>5148708 is superceeded by 5055050 which we have already applied.

There is also Note: 392215.1 which says to apply Patch 4656019

Opened a TAR:
adpatch has been running for over 16 hours to apply 3480000. If I apply
5148708 then I would loose these 16 hours and have to restart 3480000 over.
Please provide a fix to this that does not involve a patch and interruption of
3480000
–>Solution see Metalink Note:175485.1.
–>after applying patch 4656019 and restoring the adpatch (see Note:175485.1) to before it failed when applying 3480000 I restarted the adpatch for 3480000 and then once it got to the point it failed, I run adctrl and restarted the workers and things are running fine at this moment.

–>adpatch for 3480000 restarted on 03-Mat-07 at 9 PM.

It run for 2 1/2 hours and gave this error:

Interruption 6 while applying 3480000:

ATTENTION: All workers either have failed or are waiting.
cd $APPL_TOP/admin/TRAIN/log
and look at the log for worker 1 adwork001.log:

Importing /oracle/apps/pqh/selfservice/common/webui/QuickSearchRN
Error: Unable to save package file “/oracle/apps/pqh/selfservice/common/webui/QuickSearchRN” since a document or package of the same name already exists.
ORA-20101:
ORA-06512: at “APPS.JDR_MDS_INTERNAL”, line 1988
ORA-06512: at line 1

–>I searched Metalink and found Note:396855.1 which discribes a similar problem but with a different package, the notes give the following solution:

connect as apps

BEGIN
jdr_utils.DELETEPACKAGE(‘/oracle/apps/pqh/selfservice/common/webui/QuickSearchRN’);
END;

BEGIN
jdr_utils.DELETEDOCUMENT(‘/oracle/apps/pqh/selfservice/common/webui/QuickSearchRN’);
END;

–> you should get the following message for both: PL/SQL procedure successfully completed.

Restart the failed worker using adctrl

–>This sloution worked fine and adpatch restarted and progressing.

–> I restarted the patch on 04-May-07 at 7:10 AM.

Interruption 7 while applying 3480000:

–>I left this run for 4 hours and noticed that it was just setting there doing nothing and using no CPU.
The log said completed ahlsrlpm.ldt , csfsperm.ldt and csfpprop.ldt and 0 running.
I had to bounce the database to realease whatever
hung adpatch session and did a ps -aef | grep -i adpatch and killed it and the started the database.
I then restarted the adpatch on 04-May-07 at 11:30 AM on node1 telling it to continue from it’s previous session.
It is now running from wher it left off and progressing.

–> On Node1 It run for 3 more hours without error and finished on Node1:
Log node1: /u06/app/R11i/trainappl/admin/TRAIN/log/adpatch_u3480000.log
–>On Node1 it run for a total of 19 hours.
Node2:
On 04-May-07 at 3:30 PM strarting the patch on Node2.

On Node2 it run for 9 hours and gave the following error:

Interruption 8 while applying 3480000:

The following Oracle Forms objects did not generate successfully:

au resource PNSULOCN.pll
au resource PNTSPACE.pll

cd $APPL_TOP/admin/TRAIN/log
And look at the workers log but nothing there, I searched Metaling for PNSULOCN.pll and found Note:332529.1
which tells me to ignore this message and once the 11.5.10 patch is completed apply Patch 4550280
So I answered yes to continue:
Continue as if it were successful [No] : Yes

–>On Node2 it started to run again on 05-May-07 at 8:45 AM.

–> On node2 it run for another 9 hours no errors. Patch completed.
–> On Node2 it run for a total of 18 hours.
Logfiles:
Node1: /u06/app/R11i/trainappl/admin/TRAIN/log/adpatch_u3480000.log
Node2: /u02/app/R11i/trainappl/admin/TRAIN/log/adpatch_u3480000.log

3480000 post patches:

Apply Patch 4550280:

apply Patch 4550280 : PNSULOCN.PLL & PNTSPACE.PLL WILL NOT GENERATE IN 11.5.FIN_PF.G AFTER APPLYING FIN.CU2
No prerequisites.
Apply u4550280.drv first to Node1 then Node2:
On each node it run for 10 minutes no errors:
Logfiles:
Node1: /u06/app/R11i/trainappl/admin/TRAIN/log/adpatch_u4550280.log
Node2: /u02/app/R11i/trainappl/admin/TRAIN/log/adpatch_u4550280.log
Adadmin –>Generate Forms. On node2 for PN module only. –>20 minutes on node2 for module PN no errors.
Successfully restarted all apps and run a test concurent program.
–>there are some stuff running in the scheduler that look like stuff started by the upgrade process
su as “WRKR(02) (Purchasing post upgrade ….). Do not disturbe these and let them complete.

Drop MRC Schema:

logon as appltrn:
cd $APPL_TOP/admin
sqlplus SYSTEM/<SYSTEM password> @addrpmrc.sql <APPLSYS USERNAME> SAFE
(sample: sqlplus SYSTEM/trainman @addrpmrc.sql applsys SAFE )
I get the following error:
ORA-20000: MRC schema name not registered in FND_ORACLE_USERID
–>This just means that MRC is not registred in our system. No additional actions are required.
3480000 post Workflow:
see Doc ID: 268085.1 –>Not done since we do not use workflow at the time when this patch was applied.

3480000 post Payables:

Using Payables Manager resposibility verify that concurrent program: ” Mini-pack Upgrade Report ” has been successfuly run by the upgrade process.
–>This was run for one minute no error.

3480000 post Receivables:

Using Receivables Manager resposibility verify that concurrent program: ” Profile Option Migration Report ” has been successfuly run by the upgrade process.
–>This was run for one minute no error.

3480000 post Human Resources:

This maintenance pack includes HRMS Family Pack I, which is not separately available from OracleMetaLink.
If you installed a later HRMS Family Pack prior to applying this maintenance pack, your HR legislative data is up to date, and no further action is required.
–>According to acript HRMS11i.sql (211747.1) we are on HRMS Family Pack K which is > that HRMS Family Pack I
so our HR legislative data is up to date, and no further action is required.

3480000 post Payroll:

If you did not apply HRMS Family Pack H (or later) before you applied this maintenance pack, follow the tasks outlined in this section.
–>According to acript HRMS11i.sql (211747.1) we are on HRMS Family Pack K which is > that HRMS Family Pack H so no further action is required.

3480000 post Time and Labor:

We are on 11i.HXT.J and we were running 11.5.9 before the upgrade.
–>so no further action is required.

On Node2 there large files (close to 300 MB) in $COMMON_TOP/admin/log/TRAIN_http_node filling /u03, stuff like Events10.log
Metalink Note:342869.1 :jtffmctl.sh Creates Events.Log Files Which Fill the Disk.
Solution:
cd $COMMON_TOP/admin/scripts/TRAIN_http_node
cp jtffmctl.sh jtffmctl.sh.org
vi jtffmctl.sh
Remove or Comment out the references to the parameters
-Dengine.LogLevel=9
-Ddebug=full
Restart apps.
If they do not zero out, then reduce the large ones to zero doing something like the following:
> Events16.log

Resolving Invalid objects:

PACKAGE BODY APPS AP_WEB_CREDIT_CARD_WF Found note 334096.1 which tells me to apply patch 4341684
PACKAGE BODY APPS CS_WF_ACTIVITIES_PKG This was invalid before the patch. Found Note:375355.1 which tells me to apply patch 4619025
PACKAGE BODY APPS PA_PROJECT_PARTIES_PVT TAR# 17441864.6
PACKAGE BODY APPS PA_PROJECT_PARTIES_UTILS TAR# 17441864.6

Invalid objects AP_WEB_CREDIT_CARD_WF:

alter PACKAGE AP_WEB_CREDIT_CARD_WF compile BODY ;
Warning: Package Body altered with compilation errors.

SQL> show error
Errors for PACKAGE BODY AP_WEB_CREDIT_CARD_WF:

292/11 PLS-00323: subprogram or cursor ‘SENDDEACTIVATEDNOTIF’ is
declared in a package specification and must be defined in the
package body

–>Found note 334096.1 which tells me to apply patch 4341684
Applying 4341684 : 11I.OIE.J (11i.FIN_PF.G) Rollup #2
Using adadmin I applied u4341684.drv to node1 then node2:
On each node it run for 30 minutes no errors.
Logfiles:
Node1: /u06/app/R11i/trainappl/admin/TRAIN/log/adpatch_u4341684.log
Node2: /u02/app/R11i/trainappl/admin/TRAIN/log/adpatch_u4341684.log
–>This did correct the problem with AP_WEB_CREDIT_CARD_WF which is now valid.

Invalid objects PA_PROJECT_PARTIES_:

TAR# 17441864.6:
Oracle says to apply patch 4667949 to fix invalid objects PA_PROJECT_PARTIES_PVT and PA_PROJECT_PARTIES_UTILS
Applying 4667949 : Oracle Projects Rollup Patch 1 for Oracle Projects Family Pack L10 (11i.PJ_PF.L10), or Oracle Applications Maintenance Pack 11.5.10
Using adadmin I applied u4667949.drv first to Node1 then node2
On each node it run for 35 minutes no errors.
Logfiles:
Node1: /u06/app/R11i/trainappl/admin/TRAIN/log/adpatch_u4667949.log
Node2: /u02/app/R11i/trainappl/admin/TRAIN/log/adpatch_u4667949.log
–>Patch 4667949 did correct the problem with PA_PROJECT_PARTIES_PVT and PA_PROJECT_PARTIES_UTILS which are now valid.

Invalid objects CS_WF_ACTIVITIES_PKG:

Applying 4619025 : ATG11510RUP3D1INST:DB INVALID PACKAGE CS_WF_ACTIVITIES_PKG
Using adadmin I applied u4619025.drv first to Node1 then node2
On each node it run for 3 minutes no errors.
Logfiles:
Node1: /u06/app/R11i/trainappl/admin/TRAIN/log/adpatch_u4619025.log
Node2: /u02/app/R11i/trainappl/admin/TRAIN/log/adpatch_u4619025.log
–>Patch 4619025 did correct the problem with CS_WF_ACTIVITIES_PKG which is now valid.

Increasing the application idle time out:
increase the ICX:Session Timeout to 90 minutes from default 30 minutes.

Resolution of reported problems while testing 11.5.10.2 upgrade:
Case 1):

Payroll super user reported the following problem while testing TRAIN after the 11.5.10.2 patch:
Using Oracle Web ADI responsibility.
Navigate to any menu function and you get:
Web ADI Configuration Error.
BNE_UIX_PHYSICAL_DIRECTORY is not configured.
Note:430991.1
Solution
To implement the solution, please execute the following steps:
1. Go into the responsibility: System Administrator
2. Navigate to Profile->System->Profile: BNE%Directory%
3. Remove any value in the profile BNE_UIX_PHYSICAL_DIRECTORY
4. Bounce the web server and retest the issue.
5. Migrate the solution as appropriate to other environments.
–>This did fix the problem.
–>End Case 1.

Case 2):

Email from Payroll super user reporting a problem while testing the 11.5.10.2 patch on TRAIN:
After our 11.5.10 upgrade, we keep receiving a message to populate the project calendar. Once the calendar is populated, then the message
is no longer received for this project.
Reference Note:388742.1
The following soltion was provided to Payroll super user by Oracle:

First logon as sysadmin and make sure system profile “PA: Default Calendar” does point to Project Calendar
Then:
cd $PA_TOP/patch/115/sql
sqlplus /nolog
connect apps
spool c1279530.sql.log
@c1279530.sql
–>This would run for less that 40 seconds and give you:
PL/SQL procedure successfully completed.

Commit complete.

Now if you run the following SQL all calendar_id shoud be set to calendar_id=1 :
select calendar_id,count(*)
from pa_projects_all
group by calendar_id;

CALENDAR_ID COUNT(*)
———– ———-
1 4078

–>End Case 2.

Help Patches:

apply 3275245 : 11.5.10 ONLINE HELP CONSOLIDATED PATCH
No prerequisites no down time.
adpatch options=hotpatch u3275245.drv to node1 the node2.
On Node1 it run for 12 hours no errors. On node2 it run for 30 minutes no errors.
Logfiles:
Node1: /u06/app/R11i/trainappl/admin/TRAIN/log/adpatch_u3275245.log
Node2: /u02/app/R11i/trainappl/admin/TRAIN/log/adpatch_u3275245.log
–>You do not have to wait for the Help patch to complete, users may acess the system while this is being applied.

Timestamp of $PAY_TOP/reports/US/PAYUSGAR.rdf after patch:
appltrn-/u14/stage/patches/applied/11.5.10.2/4619025/4619025> ll $PAY_TOP/reports/US/PAYUSGAR.rdf
-rw-rw-rw- 1 appltrn oaa 327680 Aug 15 2006 /u06/app/R11i/trainappl/pay/11.5.0/reports/US/PAYUSGAR.rdf
–>Has not been changed.
In file $FND_TOP/reports/PSTPL.prt change the line
printer Pasta Portrait Letter”

to
printer “Pasta Portrait Letter”
i.e. insert double quote before ‘Pasta’.
–>Has not been changed.

Invalid objects: No invalid objects owned by apps..

Update patch log spreadsheet. Done.
Completed on 20-Jan-07 at 9 PM.

Latest Blog Items

email

Thank you for your interest in idbasolutions.

Contact us using this email: contact.us@idbasolutions.com