User:Jamiepham/sandbox

<!-- EDIT BELOW THIS LINE -- =HP-UX General Update Patch Plan for 11.11 -2014 =

HP-UX General Update Patch Plan for 11.211 - 2014
This is a generic plan, it may need to be modified to suit individual systems or clients.

PURPOSE OF WORK
General plan for HP-UX patching for 11.11 systems for 2014


 * Schedule: G:\SOSS_USG2\Patching\2014\Patching2014

FIRMWARE

 * Firmware Plan RP7420 FC LAN Combo- 2014 - ssisapp2
 * Seniors decided that as ssisapp2 is internal disk boot, we will not update FC firmware for ssisapp2

Bundles
The last bundle released for 11.11 was December 2009 which has already been applied to these hosts. So all that remains is to install updates and patches as suggested by SWA and any outstanding Auscert advisories.

SWA
[____]Run Software Assistant (swa) to identify any additional patches or product upgrades which may be required in addition to the OE update.
 * SWA is called via a wrapper script,'run_swa'
 * Latest version is C.02.90 SWA B6834AA for HP-UX 11.11

# swlist SWA


 * [___] Check that search for the SHLIB_PATH is enabled

# chatr /opt/swa/bin/swa |grep "SHLIB_PATH"

# chatr +s enable /opt/swa/bin/swa # chatr /opt/swa/bin/swa |grep "SHLIB_PATH"
 * If not, set it:


 * Verify you have a current catalog

# ls -la /citec/3rd_party/var/lib/patch_mgmt/swa_catalog.xml


 * Catalog can be obtained via citecue5 (but it should be automatically updated once a week)

citecue5# /opt/java7/bin/java -jar /opt/swa/lbin/swaFetch.jar -x hp_id= \ -x hp_pw= -x file=/export/patches/swa_catalog.xml \ -x proxy=http://patchproxy.citec.com.au:3128

citecue5# scp /citec/3rd_party/var/lib/patch_mgmt/swa_catalog.xml \ yourhost:/citec/3rd_party/var/lib/patch_mgmt/swa_catalog.xml


 * [____] Run SWA for the host

# /citec/toolset/bin/run_swa | mailx -s "swa for `uname -n`" citecxxx@citec.qld.gov.au


 * If run_swa fails wanting itrc username and password, update the script with "-x prompt=false" to use the local catalog

SWA Patches
Patch ID    Date        Description --- PHCO_33205+ 2005-08-29  mountall, Dev IDs enabler, iSCSI support PHKL_30398+ 2004-03-31  KI FSS ID and KI_rfscall PHKL_34926+ 2006-09-15  Buffer cache performance improvement patch PHNE_43577  2013-06-26  ONC/NFS General Release/Performance Patch PHCO_43917  2014-04-25  m4(1) cumulative patch

Security and Additional Patches

 * NFS - PHNE_43577
 * INC269078
 * ESB-2014.0322 - [HP-UX] HP-UX: Denial of service - Remote/unauth NFS rpc.lockd
 * Patch has been uploaded to citecue2-mgt:/depot/11.11
 * Reboot required


 * OPENSSL - INC292409 - DoS fix
 * OpenSSL_A.00.09.08zb.001_HP-UX_B.11.11

Copy additional patches to depots

 * If there are any required patches that are not in the depot citecue2-mgt:/depot/11.11, then Logon to the ITRC Patch Database

citecue2-mgt # swcopy -p -x enforce_dependencies=false -s \ /depot/tmp/.depot \* @ /depot/11.11

If preview has no errors, then

citecue2-mgt # swcopy -x enforce_dependencies=false -s \ /depot/tmp/.depot \* @ /depot/11.11

Software Inventory

 * HP Analysis recommended the following patches for 11.11 systems

PHNE_43577 ONC/NFS PHCO_33205 11.11 mountall PHKL_30398 KI FSS ID PHKL_34926 Buffer cache performance improvment

Software Depots
citecue2-mgt:/depot/11.11

Manual Actions for Installed Software
As reported by SWA :

PRIMARIES: Please check for additional software that your host[s] may need via running Software Analysis and checking Category D. Download and add to the software depot citecue2-mgt, and include in the swinstall statements in this plan below. This should only be done for client development systems, do not introduce new patches to production.

Software
For ICE, QSS, DoH, TMR.. on hp-ux, our applications no longer depend on any hp-ux provided java or apache versions.
 * Check for any updated software versions, verify with support teams that no issues in upgrading
 * CHT038561:

Peter Baartz

Ignite

 * NOTE: HP-UX systems have been upgraded to Ignite-UX C.7.18.63 (21/05/2014)

Ignite-UX            C.7.16.283     HP-UX System Installation Services
 * Current:

IGNITE               C.7.18.63      HP-UX Installation Utilities (Ignite-UX)
 * New:


 * Do not swremove Ignite-UX before updating to a new version.
 * Doing so will cause some files to be reset,including the INDEX file, thus you will lose any customizations.
 * Ignite server citecue2-mgt will need to be upgraded to latest version of Ignite - DONE

# swlist IGNITE

Operations Agent - OVO
TC097CA        11.11.000      HP Operations Agent
 * Current Version

TC097EA        11.20.000      HP Operations Agent
 * Latest
 * CHT038523 sent to OSSSYS - when applying to first test host, contact OSSSYS (Yong Lee) to run tests for compatibility.

Java
Java15JDK                            1.5.0.28.00    Java 1.5 JDK for HP-UX Java15JRE                            1.5.0.28.00    Java 1.5 JRE for HP-UX Java60JDK                            1.6.0.21.00    Java 6.0 JDK for HP-UX Java60JRE                            1.6.0.19.00    Java 6.0 JRE for HP-UX Jdk15 	1.5.0.29.00 Jre15 	1.5.0.29.00 Jdk60 	1.6.0.22.00 Jre60 	1.6.0.22.00
 * Current:
 * New:
 * Send Task to SBT to check not being utilised
 * CHT038561

Apache
hpuxwsApache   B.2.0.64.04    HP-UX Apache-based Web Server
 * Current:

hpuxwsApache   B.2.0.64.05    HP-UX Apache-based Web Server
 * New:

Perl5
perl            E.5.8.8.L     perl            E.5.8.8.M
 * Current on ssisapp2:
 * Need to check with client if they want to update.
 * New:

Administration Tasks
[__] Raise a Change and Submit for Approval

[__] Arrange for downtime through Service Management

[__] Send Change Task to the affected support teams (SBT, OSS, Oracle DBA's etc)

Current Patch State
[__] Check the state of all current software on host (ensure none in an "installed" state)

# /usr/sbin/swlist -l fileset -a state | egrep -v 'configured|^#'

Patch Cleanup
[__] Run a cleanup to commit patches superseded at least once # /usr/sbin/cleanup -c 1

# /usr/contrib/bin/check_patches
 * Verify the system
 * If there are problems, review /tmp/check_patches.report

Software Preparation Tasks
[__] Notify client that the software will be updated based on SWA Analysis


 * Raise Change task to the support team or Service Manager
 * Advise them of the software update and version
 * Ensure the application is compatible with the new version
 * Should the client have any special pre/post tasks they need to be added and documented in the host specific plan
 * For Java, check for any additional patches required HPUXJAVAPATCHES
 * For ssisapp2 check for PERL
 * If there is as reason why the software cannot be upgraded (incompatible etc) make sure this is advised in writing by the client and updated in the Change and documented in the host specific plan

Preview - 11.11
[__] Preview the patching, check disk space is adequate

Bring the MGT LAN interface up   # checklan -c # checklan -u

Put the following into a script because the command line is too long for the shell. -p is to preview only add -i for interactive TUI if you wish add/remove any patches/products as per client instructions

# vi /var/tmp/patchcmd_2014.sh

swinstall -p -x autoreboot=true -s citecue2-mgt:/depot/11.11 \ PHNE_43577 PHKL_34926 PHKL_30398 PHCO_33205 PHCO_43917 \ DNSUPGRADE,r=C.9.3.2.13.0 hpuxwsApache,r=B.2.0.64.05 \ Jdk60,r=1.6.0.22.00 Jre60,r=1.6.0.22.00 \ Jdk15,r=1.5.0.29.00 Jre15,r=1.5.0.29.00 \ OpenSSL,r=A.00.09.08zb.001 \ perl,r=E.5.8.8.M              -> for ssisapp2 if required

Run the preview and check the results

# sh /var/tmp/patchcmd_2014.sh

DO NOT INSTALL the software at this stage
 * Ensure enough Disk Space is available
 * If more space is needed, extend file systems if able or raise request to DSM (with approval) for more disk.
 * You may possibly need an ignite install to extend file systems

Bring the MGT LAN interface down

# checklan -d # rm /var/tmp/patchcmd_2014.sh

Take an Ingite Backup
[___] Run Ignite Backup for each host prior to work

# crontab -l | grep run_make /citec/toolset/bin/run_make_net_recovery.sh -s citecue2-mgt -l lanXXX

Nickel
[___] Collect the system information before the patching on the host being patched The latest version of nickel is in /usr/contrib/bin. This is updated bu cfengine.

# cd /var/tmp # sh /usr/contrib/bin/nickel

This will output a gzip file in /tmp: nickel. .tar.gz Copy this file to nubgate2 before the actual patching.

Backups
[__] Reschedule all backups scheduled to run during the downtime window

Dbverify
[__] Reschedule dbverify if it is scheduled to run during the downtime window It is scheduled in cron.

IMPLEMENTATION
Patch Start Time ________________am|pm

Console
[__] Log on to the console Console Access

Backups
[__] Check the required backups were completed successfully

# /citec/nsr/log # /var/log

[__] Run archive backups if required for Oracle/SAP systems

# bdf | grep arch # /citec/nsr/etc/sap.archive.bk SID #/citec/3rd_party/bin/autoarchive -thresh 0 -log /var/log/autoarchive SID

Monitoring
[__] Turn off monitoring Put the hosts into Maintenance Mode in OVO:

NODE -> Start -> Outage Management -> Start Outage

Remove {hostname} from circle on nubgate2

# vi /home/circle/etc/usg2/hosts.conf [fullfm] DISABLED

Check auto boot
[___]Make sure the host has autoboot set.

# setboot # setboot -b on

Stop Applications
[__] Stop the applications as per the Shutdown|Startup instructions for the host


 * Ensure the application[s] start-up is disabled.

Stop CSS
[__] It may take several minutes to clear processes

# ps -ef | grep css # . /citec/toolset/etc/env # service css stop # ps -ef | grep css

NFS
[__] Unmount any nfs filesystems Before un-mounting the nfs file systems, make a note of them for the post tasks.

# bdf -t nfs # umount /filesystem # bdf -t nfs

Management LAN
[__] Enable the management LAN so you can access the software depot

# checlan -c # checklan -u # netstat -in

Pre-Installation Reboot
As recommended by HP

[___] Reboot the System
 * Ensure clean boot prior to patching

# cd /; shutdown -ry now


 * Examine log files

# view /etc/rc.log # view /var/adm/syslog/syslog.log # dmesg

[___] Kernel Rebuild Ensure Kernel Rebuild is successful - 11.11

# cd /stand/build # /usr/lbin/sysadm/system_prep -s system # mk_kernel -s /stand/build/system

Apply Patches and Software
Upgrade Open Source Software - N/A for 11.11

[___]Install the updates and patches on the host

Bring the MGT LAN interface up

# checklan -c # checklan -u

Put the following into a script because the command line is too long for the shell. add -i for interactive TUI if you wish swinstall -i -x autoreboot=true -s citecue2-mgt:/depot/11.11 \ PHNE_43577 PHKL_34926 PHKL_30398 PHCO_33205 PHCO_43917 \ DNSUPGRADE,r=C.9.3.2.13.0 hpuxwsApache,r=B.2.0.64.05 \ Jdk60,r=1.6.0.22.00 Jre60,r=1.6.0.22.00 \ Jdk15,r=1.5.0.29.00 Jre15,r=1.5.0.29.00 \ OpenSSL,r=A.00.09.08zb.001 \ perl,r=E.5.8.8.M    -> for ssisapp2 if required
 * 1) vi /var/tmp/patchcmd_2014.sh

Run the install and check the results

# sh /var/tmp/patchcmd_2014.sh

Host will reboot when finished Install can take 30 to 60 mins depending on the host [_______________] am/pm Shut Down TimeM [_______________] am/pm OS Start Up Time [_______________] am/pm Application Start Up Time

POST TASKS
[___]Verify Software

# swverify \* # /usr/sbin/swlist -l fileset -a state | egrep -v 'configured|^#' # /usr/contrib/bin/check_patches

If any patches or software are not configured:

# swconfig [patch|software]

Check /etc/rc.log for any errors.

# less /etc/rc.log

Make sure swconfig did not fail on startup. If it did investigate.

example error message. "/sbin/rc2.d/S120swconfig start" FAILED

[__] Ensure that there are no errors in the log files after the reboot

# dmesg # view /var/adm/syslog/syslog.log [__] Make sure the system can see all the hardware. # ioscan -fn

[__] Check that the host is patched to the correct level


 * 1) swlist -l patch PHNE_43577 PHKL_34926 PHKL_30398 PHCO_33205 PHCO_43917

swlist -l product DNSUPGRADE hpuxwsApache Java60JDK Java60JRE OpenSSL perl Java15JDK Java15JRE perl

Want to see :

DNSUPGRADE   C.9.3.2.13.0 hpuxwsApache B.2.0.64.05 Java60JDK    1.6.0.22.00 Java60JRE    1.6.0.22.00 Java15JDK    1.5.0.29.00 Java15JRE    1.5.0.29.00 OpenSSL      A.00.09.08zb.001 perl         E.5.8.8.M        -> for ssisapp2 if required

[___] Upgrade OVOA - problems with version - leave for OVOA fix from HP  # swinstall -p -s citecue2-mgt:/depot/dcoe/11.31.2014_03 TC097EA # swinstall -s citecue2-mgt:/depot/dcoe/11.31.2014_03 TC097EA # swlist TC097EA # perfstat -v Operations Agent 11.13.007 # opcagt -status

[__] Check SWA

# rm ~/.swa/cache/swa_inventory_*.xml # run_swa

NFS
[__] Re-mount any NFS file systems

# bdf -t nfs # mount -Qae # bdf -t nfs

Start Applications
[__] Start/Verify the applications as per the Shutdown|Startup instructions for the host

Ensure the application[s] start-up is enabled.

Check OVO Monitoring
[___] Verify running, start if required

# opcagt -status              (If not started # opcagt -start) # mwa status                  (If not started # mwa start)

Send test messages, check the OVO browser: # opcmsg s=minor o=opcmsg a=setalarm msg_g=usg2 msg_t="TESTING OVOA 11.20" # echo "TEST MIG" | setalarm -a SYS -l NOTICE -d "TESTING OVOA 11.20" -D -

Sendmail
[___] Verify mail

# echo TESTING | mailx -s "Test mail for `hostname` after HP-UX 11.11 Patching 2014" usg2@citec.com.au citec-hp@citec.qld.gov.au  # tail /var/adm/syslog/mail.log

Enable Monitoring
[__] Turn on monitoring Take the hosts out of Maintenance Mode in OVO:

NODE -> Start -> Outage Management -> End Outage

Add {hostname} back into circle on nubgate2 Remove the DISABLED tag

# vi /home/circle/etc/usg2/hosts.conf [fullfm]

Backups
[__] Enable any backups that may have been disabled during the outage

Downtime
[__] Create dt for the host on nubgate2 with the system and application down times

Management LAN
[__] Take down the management LAN

# checklan -c # checklan -d # netstat -in | grep 131.242.232

VERIFICATION
[___] Raise Task with Support Team to verify the applications after patching.

COMPLETION

 * After verification perform following tasks.

[___] Inform affected support teams and Service Management that the system and application(s) are back up and running. [___] Run and ignite backup post the patching [___] Update and close the change [___] Update the plan with any issues, corrections or additional information which may be helpful [___] Update G:\SOSS_USG2\Patching\Patching2014

REVERSION
swremove any problem patches or software

# swremove -i -s [PATCH]

Document the issues and fixes, log call with HP if required

If require full recovery from Ignite then;
 * Escalate to Service Management
 * Recover OS with Ignite Make Net Recovery