AIX: Page 2

[frs-278] How to get a thread dump from the JVM


Background

To assist in debugging potential issues it is sometimes necessary to generate a thread dump. This is a list of all threads in the JVM and their current stack of processing code.

Generating a thread dump

Preferred method: If you have access to FusionReactor, you can goto the "Resource->List All Threads" page in the left-hand menu, then click the "Stack Trace ALL" button at the top-right of the page.

Other ways to generate a thread dump differ depending on the platform:

Windows systems

  • Press Ctrl-Break in the command console you used to start your application.
    • If a command console isn't an option (eg when started as a service) you can use a tool like StackTrace to get a stack trace. At the time of writing, the licence allows free use when started with the JNLP Web Start option.
  • If you start your application with the com.sun.management.jmxremote option you should be able to attach jconsole and get a thread dump (JDK 5.0 or higher.)
  • There is a little tool called SendSignal which uses a clever trick to call the ctrl-break signal handler of any process.
  • JDK 5 & 6 offer tools for monitoring, management, and troubleshooting (jconsole, jps, jstat, jstatd, jinfo, jmap, jstack.)

Mac OS X

  • Press *Ctrl-* in the terminal console you used to start your application.
  • You can also generate a thread dump by sending the QUIT signal to the Java VM running your application kill -QUIT process_id where process_id is the process number of the respective java process.
    • Note: This method outputs the stacktrace to the standard output stream which may be redirected to a log file.
  • Thread dump by using gdb.
    • Attach to the target process with gdb and run the following command:
    • (gdb) call (void)pss()
  • The latest Apple 1.5 JVMs offer all the monitoring tools available for Linux and Solaris (jconsole, jps, jstat, jstatd, jinfo, jmap, jstack.)

Unix systems

  • Press Ctrl- in the terminal console you used to start your application.
  • You can also generate a thread dump by sending the QUIT signal to the Java VM running your application kill -QUIT process_id where process_id is the process number of the respective java process.
    • Note: This method outputs the stacktrace to the standard output stream which may be redirected to a log file.
  • JDK 5 & 6 offer tools for monitoring, management, and troubleshooting (jconsole, jps, jstat, jstatd, jinfo, jmap, jstack.)

Issue Details

Type: Technote
Issue Number: FRS-278
Components: Logging
Environment:
Resolution: Fixed
Last Updated: 08/Nov/11 3:01 PM
Affects Version:
Fixed Version: 4.0.0
Server:
Platform: Windows XP, Windows 2000, Windows 2003, Linux, MacOS, Solaris, Windows Vista, Windows x64, AIX, Windows 7, Windows 2008
Related Issues:

[frs-277] “java.io.IOException: Cannot run program “ln”: error=12, Not enough space” error when installing FusionReactor from a NAS or other file-system with “root squashing” enabled


Symptom

Installation may fail if attempting to install from an NFS share with root squash enabled.

Analysis

This is because by default, the installer extracts temporary files into the parent directory of the installer location. As this would not typically be accessible, the installation is likely to fail with a message such as "java.io.IOException: Cannot run program "ln": error=12, Not enough space".

Resolution

To resolve this, first copy the installer to a local, writeable file-system (eg /var/tmp)

Documentation links

Console Installation on Linux
Console Installation on Solaris

Issue Details

Type: Technote
Issue Number: FRS-277
Components: Installer
Environment:
Resolution: Fixed
Last Updated: 02/Nov/11 11:22 AM
Affects Version:
Fixed Version: 4.0.0
Server:
Platform: Linux, MacOS, Solaris, AIX
Related Issues:

[frs-245] Hotfix FR-2069: JDBC Prepared Statements throw ‘NullPointerException’ when called with null binding values.


Hotfix 2069 for FusionReactor 3.5.5 Description

This technote describes a fix for a condition under which FusionReactor's JDBC Driver Wrapper may throw NullPointerException when used with PreparedStatements whose binding values are null

Symptoms

When using the JDBC Driver Wrapper, passing a null to a setX method of PreparedStatement throws NullPointerException.

Analysis

FusionReactor's JDBC driver expects passed binding values to be non-null. The expected method of setting null for a given parameter is the setNull(...) method, however the JDBC API does not specifically prohibit null as a binding value being passed directly to the setX methods of PreparedStatement. Internally, the wrapper does some marshaling of this value in order to log it and display it in FusionReactor, and this operation was not null-guarded.

We confirm this to be a bug in the JDBC Driver Wrapper shipped up to FusionReactor 3.5.5.

Resolution

Customers with FusionReactor 3.5.5

Customers with FusionReactor 3.5.5 only installed should apply the attached Hotfix 2069, which resolves this issue.

Instructions for applying the hotfix are supplied in instructions.txt within the hotfix zip file.

Hotfixes are cumulative: any future hotfixes on the FusionReactor 3.5.5 stream, up to but not including the next point release, will contain this hotfix (and all earlier hotfixes since the last point release).

Customers with prior versions (< 3.5.5)

Customers with versions earlier than FusionReactor 3.5.5 should update to that version using the published installer/updater prior to applying this hotfix.

Included Hotfixes

  • None. This hotfix is the first to be issued on the 3.5.5 stream.

Files

hotfix-FR-2069.zip

Taxonomy

After installation, FusionReactor's "About" page should identify itself as *"
Revision: 3.5.5, Build: FusionReactor 3.5.x Maintenance Branch.17.19877"*

~Hotfix FR-2069 (B/P FR-2070) – obf. (Stream: FusionReactor 3.5.x Maintenance Branch, build 17, SVN 19877 JRH)

Issue Details

Type: Technote
Issue Number: FRS-245
Components: JDBC
Environment:
Resolution: Fixed
Last Updated: 02/Mar/11 2:27 PM
Affects Version: 3.5.5
Fixed Version: Pending
Server: ColdFusion 6, ColdFusion 7, ColdFusion 8, ColdFusion 9, Flex Data Services, JBoss, Jetty, JRun 4, LiveCycle Data Services, Railo, Resin, ServletExec, Tomcat, WebSphere, WebLogic
Platform: Windows XP, Windows 2000, Windows 2003, Linux, MacOS, Solaris, Windows Vista, Windows x64, AIX, Windows 7, Windows 2008
Related Issues:

[frs-238] Getting Started With FusionReactor


Guide to Getting Started With FusionReactor

New to FusionReactor? Use this guide to help you Get Started.

This document refers to FusionReactor 3.0.1 and JDBC Wrap/Un-Wrap tool version 0.8 and is applicable to FusionReactor 3.5.1.

Getting Started with FusionReactor

Useful Links

Download FusionReactor
Install Guide (pdf)
JDBC Wrapper Tool

About the Author

Ajas Mohammed is a Sr. Software Engineer / Sr. ColdFusion Programmer at Absentys LLC with both a Bachelor's and Master's in Computer Science. Ajas has worked in the Information Technology field for over 10 Years.

Ajas Mohammed's Blog

Issue Details

Type: DevNet
Issue Number: FRS-238
Components: Compression, Content Filters, CPU + Memory, Crash Protection, Enterprise Dashboard, FR Enterprise Dashboard Desktop Application, FusionReactor Settings, Installer, JDBC, License + Activation, Logging, Metrics, Request Managment, Thread Management
Environment:
Resolution: Fixed
Last Updated: 20/Nov/09 1:15 PM
Affects Version: 3.0.1, 3.5
Fixed Version: 3.0.1, 3.5
Server:
Platform: Windows XP, Windows 2000, Windows 2003, Linux, MacOS, Solaris, Windows Vista, Windows x64, AIX, Windows 7, Windows 2008
Related Issues:

[frs-237] Creating Daily FusionReactor Log Files


Creating Daily Log Files in FusionReactor

FusionReactor captures a lot of very useful data into log files. These log files can be used to analyze a server in great detail. FusionReactor's logging mechanmism allows you to decide how many log files you want to keep and how big each log file should be but it doesn't allow you to rotate the log files every day.

It would be great to rotate the log files every day so that we can keep a record for as long as we want instead of losing data when the log files rotate automatically.

Using FRAPI (FusionReactor's API) this is easy to achieve.

Requirements

FusionReactor 3.0.1+

The Article

FusionReactor captures data about Requests, JDBC Requests, Resources, CrashProtection actions and details about itself into log files. These log files are typically placed under a log folder that can for each instance of FusionReactor. On a Windows installation the log folder might by for example:

   C:FusionReactorinstancecoldfusion.cfmx9.INT000Blog

About FRAPI

FRAPI is a powerful and easy to use API that allows you to take control of FusionReactor programatically. Using FRAPI you can change the running configuration of FusionReactor.

Here's how to instantiate the FRAPI interface in CFML. Once we have FRAPI available we are able to read this FusionReactor's configuration:

        <cftry>
            <cfset frapi = createObject("java", "com.intergral.fusionreactor.api.FRAPI").getInstance()>
            <cfset config = frapi.getRunningConfiguration()>
            <cfdump var="#config#">
            <cfcatch>
                <!--- FRAPI isn't available so FusionReactor probably isn't installed --->
            </cfcatch>
        </cftry>

Note that we always wrap FRAPI calls in a TRY/CATCH block. The reason for this is that FusionReactor may not be installed on every computer that you run the code on. The TRY/CATCH blocks will make sure that FRAPI calls will have no influence on your code even if FusionReactor isn't installed.

To try FRAPI, create a "frapiconfig.cfm" file with the CFML code above and run the page on a system that has FusionReactor installed. After the page has completed you should see a dump of FusionReactors configuration for this instance.

Once we have the configuration, we can change it and tell FusionReactor to update very easily like this:

            <cfset config.setValue("logfile", "#install_location#/instance/#instance_name#/log/#datetime#/reactor.log")>
            <cfset frapi.setRunningConfiguration(config)>

FRAPI.CFM

        <cftry>
            <cfset frapi = createObject("java", "com.intergral.fusionreactor.api.FRAPI").getInstance()>
            <cfsavecontent variable="debugData"><cfinclude template="classic.cfm"></cfsavecontent>
            <cfset frapi.trace( debugData )>
            <cfcatch>
                <!--- FRAPI isn't available so FusionReactor probably isn't installed --->
            </cfcatch>
        </cftry>

Notes

  1. Log File Sizes
    Some of FusionReactor's log files can fill very quickly. This is especially true of the JDBC log files. You need to take this into consideration when you set the log file size(s). As a counter point you have to be aware that using very large log files sizes and rotating the log files every day can very quickly use up lots of disk space. Please take care when setting the log file sizes and consider archiving or removing log files based upon there age.
  1. Log File Count
    FusionReactor lets you set the number of log files you want to keep for each log type. It's tempting to set this to a large number so that FusionReactor doesn't lose any information but you should consider that the more log files FusionReactor has to manage and rotate, the slower the log file rotate will be. I would not advise you to set the log file count higher than 20 log files. It is more advisable to balance the number of log files and the log file size. For example, of you set the log file count to 10 and the log file size to 20480 (log file size is set in KBytes) then you would create a maximum of 200MBytes of log data (for the specific log).

Fast Track

  1. Copy the frapi.rotatelogdirectoy.cfm file linked to this article to a folder where you run ColdFusion (for example): <coldfusion folder>/wwwroot/fusionreactor
  2. Create a scheduled task to run this page once per day at 00:00:00 midnight

Summary

By rotating FusionReactor's log files daily you can keep log files for as long as you want and can prevent you from losing this valuable data when FusionReactor rotates it's log files automatically when the size of the file has been reached.

Issue Details

Type: DevNet
Issue Number: FRS-237
Components: Logging
Environment:
Resolution: Fixed
Last Updated: 06/Nov/09 6:51 PM
Affects Version: 2.0, 2.0.3, 2.0.4, 3.0, 3.0.1, 3.5
Fixed Version: 3.0.1, 3.5
Server:
Platform: Windows XP, Windows 2000, Windows 2003, Linux, MacOS, Solaris, Windows Vista, Windows x64, AIX, Windows 7, Windows 2008
Related Issues:

FRS-232: Capturing ColdFusion’s Debug Output in FusionReactor

[frs-227] Using FusionReactor on the AIX operating system


Symptoms

CPU metrics are not available in FusionReactor on AIX operating systems. This is most visible from the System Metrics charts.

Analysis

FusionReactor uses a native code plugin to retrieve these system level metrics directly from the operating system.

Resolution

We have built a native plugin library to call the "libperfstat" library for direct integration into FusionReactor.
Simply follow the instructions from instructions.txt in the attached AIX-NativePlugin-301.zip to add CPU metrics on IBM AIX platforms.


Requirements

  • FusionReactor 3.0.1 or later
    • You can browse to "About" under the "FusionReactor" tab of the FusionReactor web interface to verify which version of FusionReactor you are running.
  • Performance Statistics Library 5.3.9.0 or later
    • You can check for its presence with the following command:
      	$lslpp -l bos.perf.libperfstat bos.perf.perfstat
      
    • Which should result in output simiilar to:
        Fileset                      Level  State      Description        
        ----------------------------------------------------------------------------
      Path: /usr/lib/objrepos
        bos.perf.libperfstat       5.3.9.0  COMMITTED  Performance Statistics Library
                                                       Interface
        bos.perf.perfstat          5.3.9.0  COMMITTED  Performance Statistics
                                                       Interface
       
      Path: /etc/objrepos
        bos.perf.libperfstat      5.3.0.10  COMMITTED  Performance Statistics Library
                                                       Interface
        bos.perf.perfstat          5.3.9.0  COMMITTED  Performance Statistics
                                                       Interface
      

Installation

  1. Put the library file (libFusionReactor.aix.so) into the java.library.path or LD_LIBRARY_PATH. The easiest way to do this for your entire machine is to copy the attached library into "/usr/lib" making it available to all JVM instances.
  2. Restart your instance.
  3. Verify CPU metrics are available by browsing to "System Metrics" under the "Metrics" tab of the FusionReactor web interface.

Issue Details

Type: Technote
Issue Number: FRS-227
Components: CPU + Memory, Metrics
Environment:
Resolution: Fixed
Last Updated: 20/Aug/09 5:08 PM
Affects Version: 3.0.1
Fixed Version: Pending
Server:
Platform: AIX
Related Issues: