How to setup Verbosegc (Grabage Collection) and run Oracle Thread dumps for WebLogic performance Monitor: When an Oracle/BEA ticket is needed to troubleshoot WebLogic performance issues the Garbage Collection and Oracle Thread dumps are required by Oracle/BEA. The following steps will outline how to setup the verbosegc collection and create Oracle thread dumps to send to Oracle/BEA for analysis. The following topics will be covered in this document 1. Add the вЂ“verbosegc to the Java options in the Managed Server startup 2. Script to extract all the GC entries in the log and create Excel graph displaying the WebLogic memory 3. Run the Oracle Thread dumps if required Telcordia Technologies, Inc. Confidential вЂ” Restricted Access This document and the confidential information it contains shall be distributed, routed or made available solely to authorized persons having a need to know within Telcordia, except with written permission of Telcordia. Page 1 1. Add the вЂ“verbosegc parameter. п‚· п‚· п‚· Locate the file setXngEnv.sh - typically found in the bin directory (i.e.вЂ”/app/bea921/user_projects/domains/granite/bin) Copy the file before modification - cp setXngEnv.sh setXngEnv.sh .orig If running in multiple environments add a section for the managed server you wish to add the verbose parameter. File: setXngEnv.sh #!/bin/sh echo Setting XngParameters for ${SERVER_NAME} case "${SERVER_NAME}" in "admin") echo NO CHANGE;; "gateway") echo NO CHANGE;; *) echo CHANGE CLASSPATH=$LONG_DOMAIN_HOME/Xng_config_${SERVER_NAME}$CLASSPATH;; esac # If Specific Server environment changes are necessary, they can be made much like the following # if [ "$SERVER_NAME" = "admin" ] ; then #MEM_ARGS="" MEM_ARGS="-Xms256m -Xmx256m -XX:CompileThreshold=8000 -XX:PermSize=48m -XX:MaxPermSize=128m" echo MEM_ARGS fi if [ "$SERVER_NAME" = "xng650_00" ] ; then MEM_ARGS="-Xms512m -Xmx512m -XX:CompileThreshold=8000 -XX:PermSize=128m -XX:MaxPermSize=256m" JAVA_OPTIONS="-XX:CompileCommand=exclude,com/granite/asi/dto/AbstractDataObject,hashCode $JAVA_OPTIONS" fi Added lines Verbose Parameter if [ "$SERVER_NAME" = "xng650_02" ] ; then MEM_ARGS="-Xms512m -Xmx512m -XX:CompileThreshold=8000 -XX:PermSize=128m -XX:MaxPermSize=256m" JAVA_OPTIONS="-verbosegc -XX:CompileCommand=exclude,com/granite/asi/dto/AbstractDataObject,hashCode $JAVA_OPTIONS" fi #MEM_ARGS="-Xms1024m -Xmx1024m -XX:CompileThreshold=8000 -XX:PermSize=128m -XX:MaxPermSize=256m" #export MEM_ARGS Telcordia Technologies, Inc. Confidential вЂ” Restricted Access This document and the confidential information it contains shall be distributed, routed or made available solely to authorized persons having a need to know within Telcordia, except with written permission of Telcordia. Page 2 п‚· Stop and Start the Managed Server that has the verbose parameter. п‚· View the WLSM.LOG fileвЂ”usually found in the directoryвЂ” /app/bea921/user_projects/domains/granite/xng650_02 <Feb 1, 2010 11:55:49 AM EST> <Info> <Management> <BEA-141107> <Version: WebLogic Server 9.2 MP1 Sun Jan 7 00:56:31 EST 2007 883308 > <Feb 1, 2010 11:55:53 AM EST> <Info> <WebLogicServer> <BEA-000215> <Loaded License : /app/bea921/license.bea> <Feb 1, 2010 11:55:53 AM EST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING> <Feb 1, 2010 11:55:53 AM EST> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool> <Feb 1, 2010 11:55:54 AM EST> <Notice> <Log Management> <BEA-170019> <The server log file /app/bea921/user_projects/domains/granite/servers/ xng650_02/logs/xng650_02.log is opened. All server s ide log events will be written to this file.> <Feb 1, 2010 11:55:54 AM EST> <Error> <Socket> <BEA-000438> <Unable to load performance pack. Using Java I/O instead. Please ensure that a native performance library is in: '/opt/java1.5/jre/l ib/IA64N:/opt/java1.5/jre/lib/IA64N/server:/opt/java1.5/jre/../lib/IA64N:/app/bea921/patch_weblogic921/profiles/default/native:/lib::/app/bea921/weblogic92/ server/native/hpux11/IPF32:/app/bea9 21/weblogic92/server/native/hpux11/IPF32/oci920_8:/usr/lib' [GC 131072K->15821K(502464K), 0.0617879 secs] <Feb 1, 2010 11:55:58 AM EST> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.> <Feb 1, 2010 11:56:02 AM EST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STANDBY> <Feb 1, 2010 11:56:02 AM EST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING> [GC 146893K->27467K(502464K), 0.1018594 secs] <Feb 1, 2010 11:56:13 AM EST> <Warning> <EJB> <BEA-010100> <This server is not part of a cluster. Hence "InMemory replication feature" for StatefulSession EJB "WorkflowEJB" will have no effect on this server.> <Feb 1, 2010 11:56:15 AM EST> <Warning> <HTTP> <BEA-101369> <weblogic.servlet.internal.WebAppServletContext@f72499 - appName: 'template-tool6.5.0.02', name: 'template-tool-6.5.war', context-p Telcordia Technologies, Inc. Confidential вЂ” Restricted Access This document and the confidential information it contains shall be distributed, routed or made available solely to authorized persons having a need to know within Telcordia, except with written permission of Telcordia. Page 3 2. Script to extract all GC entries in file that can be opened in Excel Check if file gcstats.csv already exist remove or rename to another file name if found Create the column names as the 1st line of the file: п‚· п‚· echo "Memory Before, Memory After, Total JVM" > gcstats.csv (creates the gcstats.csv file) Run the following GREP command to extract the GC entries into gcstats.csv file п‚· grep "K)" wlsm.log | grep -v Full | cut -c 5-27 | sed 's/K->/,/g' | sed 's/K(/,/g' | sed 's/K//g' >> gcstats.csv п‚· FTP the gcstats.csv file using ASCII mode back to your PC. п‚· Open gcstats.csv file in Excel, should look like image below: Telcordia Technologies, Inc. Confidential вЂ” Restricted Access This document and the confidential information it contains shall be distributed, routed or made available solely to authorized persons having a need to know within Telcordia, except with written permission of Telcordia. Page 4 п‚· The memory items can be graphed to display the GC release of memory. Highlight the A, B and C columns Click the Insert tab, then choose Line chart, choose 2-D Line Telcordia Technologies, Inc. Confidential вЂ” Restricted Access This document and the confidential information it contains shall be distributed, routed or made available solely to authorized persons having a need to know within Telcordia, except with written permission of Telcordia. Page 5 п‚· Chart should display as follows Telcordia Technologies, Inc. Confidential вЂ” Restricted Access This document and the confidential information it contains shall be distributed, routed or made available solely to authorized persons having a need to know within Telcordia, except with written permission of Telcordia. Page 6 3. Collection of Thread Dumps for Oracle/BEA support ticket. To collect thread dumps the Unix Process ID must be determinedвЂ” run the script below to determine ps вЂ“ef | grep bea921 (bea921 is the WebLogic User ID) bea921: ps -ef |grep bea921 bea921 18530 18529 0 Feb 1 pts/tc 0:00 -ksh bea921 28035 28033 0 Jan 15 ? 0:00 /bin/sh /app/bea921/user_projects/domains/granite/bin/startWebLogic.sh nodebug nopointbase noiterativedev notestconsole bea921 26569 26552 0 Jan 15 ? 153:40 /opt/java1.5/bin/IA64N/java -server -Xms256m -Xmx256m -XX:CompileThreshold=8000 -XX:PermSize=48m XX:MaxPermSize=128m -da -Dpla bea921 18980 18978 0 Feb 1 pts/tc 0:00 /bin/sh /app/bea921/user_projects/domains/granite/bin/startWebLogic.sh nodebug nopointbase noiterativedev notestconsole bea921 24704 18530 0 10:02:18 pts/tc 0:00 grep bea921 bea921 28051 28035 0 Jan 15 ? 121:04 /opt/java1.5/bin/IA64N/java -server -Xms512m -Xmx512m -XX:CompileThreshold=8000 -XX:PermSize=128m -XX:MaxPermSize= 256m -da -Dpl bea921 18978 1 0 Feb 1 pts/tc 0:00 /bin/sh ./bin/startManagedWebLogic.sh xng650_02 qabhp1:7003 bea921 18996 18980 0 Feb 1 pts/tc 13:47 /opt/java1.5/bin/IA64N/java -server -Xms512m -Xmx512m -XX:CompileThreshold=8000 XX:PermSize=128m -XX:MaxPermSize=256m -verbose bea921 24703 18530 1 10:02:18 pts/tc 0:00 ps -ef Locate the process that shows the verbose option, in this example the process ID is 18996 Run the script Kill вЂ“3 18996 ( 3-4 times at 20 second intervals, thread dump info will go to the wlsm.log file.) bea921: kill -3 18996 bea921: kill -3 18996 Telcordia Technologies, Inc. Confidential вЂ” Restricted Access This document and the confidential information it contains shall be distributed, routed or made available solely to authorized persons having a need to know within Telcordia, except with written permission of Telcordia. Page 7 п‚· Entries in the wlsm.log file will look like these below: [] [04 Feb 2010 10:14:25,105] [com.granite.util.license.oracle.OracleLicenseService] [License Listener] [DEBUG] [License 6664 is valid.] Full thread dump [Thu Feb 04 10:14:42 EST 2010] (Java HotSpot(TM) Server VM 1.5.0.04 jinteg:07.27.06-16:18 IA64 mixed mode): "Thread-9" daemon prio=8 tid=00d43ed0 nid=43 lwp_id=909600 waiting on condition [40300000..40300c50] at java.lang.Thread.sleep(Native Method) at com.granite.middle.util.MTaccess.bgCheck(MTaccess.java:1167) at com.granite.middle.util.MTaccess.run(MTaccess.java:1013) "[STANDBY] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=00d431a0 nid=42 lwp_id=903905 in Object.wait() [403c0000..403c0cd0] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:474) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165) - locked <579057c0> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:186) Telcordia Technologies, Inc. Confidential вЂ” Restricted Access This document and the confidential information it contains shall be distributed, routed or made available solely to authorized persons having a need to know within Telcordia, except with written permission of Telcordia. Page 8