Blog

[frs-271] Understanding FusionReactor’s Timeout Protection

Introduction

We're sometimes asked about the timings for FusionReactor's Timeout Protection, part of the Crash Protection system. The idea behind this crash protection is that it places a hard upper limit on the length of time requests can run. You might find, however, that requests take a few seconds longer to be killed than you originally thought. Here's why.

Kill Strategies

After the time limit is reached, three strategies are employed by the Request Assassin to kill the request:

  • Soft Kill
    If the request makes any output to the page whatsoever, it will be killed. This is done by intercepting the call which writes data back to the network. The Request Assassin waits for 2 seconds for the request to write output, before trying the next strategy.
  • Interrupt
    We interrupt the thread in which the request runs. If the request is performing an interruptible operation (a call which can throw InterruptedException; sleeping, for instance), the exception will be raised. The Request Assassin waits for 1 second for the thread to process the interrupt – if it does so at all – before trying the final strategy.
  • Thread Stop
    The request is forcefully stopped by Java's thread stop mechanism. Although this isn't completely reliable – threads performing native operations will not be stopped, for instance – it works most of the time, and in fact there is no other way to kill a thread.

The Request Assassin

In addition to these delays, the Request Assassin doesn't run all the time: it wakes up every second, and checks whether there are any threads it should examine. If there are any, it starts to process them, employing the strategies above. If a strategy succeeds, it moves on to the next candidate. If not, it tries the next strategy, until the list of strategies is exhausted.

Because FusionReactor attempts to place as little load on the target system as possible, the candidates for kill are queued up. This is an additional source of delay – previous kill operations may have to be processed before the Request Assassin gets to the request of interest.

Worked Example

Let's look at a worked example, with Timeout Protection set to 1 second.

In the diagram, you can see (starting from the top timeline):

  • The flow of a request (blue).
  • The best possible timing of a crash protection (red).
  • The worst possible timing of a crash protection (red).

Best Possible Scenario

  • The Request Assassin wakes up at time T1 – the exact instant the request becomes eligible for kill. It immediately sets the Soft-Kill flag, and starts the soft kill delay, 2 seconds.
  • At time T3, the soft-kill delay expires, and the request is still alive, having not performed output. The thread is interrupted, and the Request Assassin waits for a further second.
  • At time T4, the request is still alive, so the Request Assassin performs a thread kill.

The request then dies, marked by the first red "X" in its timeline. If the request was performing native operations (i.e. using part of the Java language that's written in C), it will be killed as soon as it returns. One common native operation is blocking socket communication: this is performed in Java by a C library.

Worst Possible Scenario

The second timeline is much the same as the first, except the Request Assassin ran at time T0.9r, i.e. an instant before the request became eligible for kill at T1. The next time the Request Assassin runs is at time T2 (actually an instant before that), meaning the whole timeline starts 1 second later, at time T2, and completing with the kill at time T5.

Conclusion

We showed in this small article where the (up to) 3-second overhead in Timeout Protection comes from: a combination of soft-kill and interrupt delays, which allow us to try to stop a request without resorting to a hard kill.

Issue Details

Type: DevNet
Issue Number: FRS-271
Components: Crash Protection
Environment:
Resolution: Fixed
Last Updated: 22/Sep/11 3:36 PM
Affects Version:
Fixed Version: 4.0.0
Server:
Platform:
Related Issues:

[frs-232] Capturing ColdFusion’s Debug Output in FusionReactor

Capturing ColdFusion's Debug Output in FusionReactor

It would be great if you could capture the debug data from every page run but hide it away from the user so that they don't even see it. It's easy to do this with FusionReactor.

ColdFusion's powerful debug data output feature that lets you see lots of useful debugging information about the page run. Unfortunately the data is lost as soon as you run the next page in the browser window/tab and is only visible to the user that ran the page.

Requirements

FusionReactor
FusionDebug

The Article

We're all familiar with ColdFusion's debug output at the end of a CFML page. Many CFML programmers use it as the primary way of debugging CFML code.

There are a few problems with the way that debug data is made available to us however such as the fact that the only user that can see the debug output is the user that ran the page. This is even more of a problem if you want to see the debug data but don't want to scare the user that ran the page. Of course it's possible in CF to restrict the IP addresses that generate debug output but sometimes you want to be able to see the debug output from other users without them seeing it. It's easy to do with FRAPI.

About FRAPI

FRAPI is a powerful and easy to use API that allows you to take control of FusionReactor programatically. Using FRAPI you can change the running configuration of FusionReactor or add extra debugging information to requests using just a few lines of code.

Here's how to instantiate the FRAPI interface in CFML. Once we have FRAPI available we add extra information to a request data recorded in FusionReactor using FRAPI's trace() function:

      	<cftry>
            <cfset frapi = createObject("java", "com.intergral.fusionreactor.api.FRAPI").getInstance()>
            <cfset frapi.trace( "Hello FRAPI!")>
            <cfcatch>
                <!--- FRAPI isn't available so FusionReactor probably isn't installed --->
            </cfcatch>
        </cftry>

Note that we always wrap FRAPI calls in a TRY/CATCH block. The reason for this is that FusionReactor may not be installed on every computer that you run the code on. The TRY/CATCH blocks will make sure that FRAPI calls will have no influence on your code even if FusionReactor isn't installed.

To try FRAPI, create a "frapitest.cfm" file with the CFML code above and run the page on a system that has FusionReactor installed. After the page has completed go into FusionReactor's Request History, select the Request Details for the request and click on the Markers tab to see the "Hello FRAPI!" message. Cool no? FRAPI tracks messages on the request as soon as trace() is called so it's possible to see the trace() statements in the Markers tab updating as a page is being run!

So how would we capture CF's debug output? We don't want to add FRAPI code to every page that we write that's for sure. We need to understand how ColdFusion's debug output is done to know how to capture it.

When enabled, CF Debug output is appended at the end of the page output by rCF running a "Debug Output Formatting" template. This is simply a piece of CFML code to that formats the debugging information into HTML. What's even better is the fact that the you can write your own and tell ColdFusion to use your template instead of the default one. So what if we wrote a CF Debug Template that simply calls FRAPI and adds that debug data to the FusionReactor Request using trace() calls? This means that the information would be available in FusionReactor but the user wouldn't even know that debug output was enabled because the FRAPI trace() calls don't output anything. It's that easy! Ok, there is always a but…

…but I don't know how to access all of this CF Debug Data and for sure it's complex, right?

The CF Debug data structures are fairly complex and very powerful but this is reason that few people ever write there own CF Debug Formatting template. The good news is we don't need to know anything about it due to a powerful CFML tag called <CFSAVECONTENT>. I know that the advanced CFML programmers out there just went… "OK, I get it…", but for the rest of us let's go into a little more detail on how we can do this.

<CFSAVECONTENT> is a tag that allows the programmer to capture output into a variable instead of sending it back to the user. Using <CFSAVECONTENT> we can capture the output from the classic CF debugging template into a variable and then simply call FRAPI's trace() function passing that variable as the information to record! Easy!

FRAPI.CFM

        <cftry>
            <cfset frapi = createObject("java", "com.intergral.fusionreactor.api.FRAPI").getInstance()>
            <cfsavecontent variable="debugData"><cfinclude template="classic.cfm"></cfsavecontent>
            <cfset frapi.trace( debugData )>
            <cfcatch>
                <!--- FRAPI isn't available so FusionReactor probably isn't installed --->
            </cfcatch>
        </cftry>

All we have to do now is add our FRAPI Debug Template (frapi.cfm) to the CF debug templates by simply copying it folder where debug templates are placed, typically <coldfusion folder>/wwwroot/WEB-INF/debug.

Finally we have to Enable Request Debugging Output and select our new ("frapi.cfm") template as the Debugging Output Format in the ColdFusion Administrator -> Debug Output Settings page.

The next time you (or anyone) runs a CF page FusionReactor will capture the Debug data for the page. To see the data simple go to the FusionReactor->Request History, select the Request Detail for the request and then click on the Markers tab to see the CF Debugging Data for the page.

Notes

  1. Performance
    ColdFusion pages run slower when debugging is enabled and on some applications enabling CF Debug Output can seriously impact the performance of the server. You should test this on your applications before enabling debugging a production server.
  2. Memory
    FusionReactor will record the debug output on each request that it keeps in it's request history. This will use some memory; for example, if you have a Request History size of 100 requests and the average debug output data size is 20Kbytes then FusionReactor will use 2Mbytes of memory to
    record the debug data.
  3. Data
    The amount of debug data recorded is controlled via the settings on the Debug Output Settings page and the classic.cfm debug formatting output template. The data will not be added to FusionReactor until the page completes because this is when CF calls the Debug Output Formatting page. The
    debug data captured will honor any Debugging IP Address restrictions that you configure in the CF Administrator.

Fast Track

  1. Copy the frapi.cfm file linked to this article to your ColdFusion Debug Formatting Template folder (typically): <coldfusion folder>/wwwroot/WEB-INF/debug
  2. Select Debugging Output Format: "frapi.cfm" in the ColdFusion Administrator -> Debug Output.
  3. Enable Request Debugging Output in the ColdFusion Administrator->Debug Output Settings Settings
  4. Run a CF page.
  5. In FusionReactor->Request History select the Request Detail for the request and then click on the Markers tab to see the CF Debugging Data from the page.

Summary

FRAPI is a powerful way to capture extra debugging information on requests and makes FusionReactor a powerful platform for debugging on both development and production servers.

Issue Details

Type: DevNet
Issue Number: FRS-232
Components: Compression, Content Filters, CPU + Memory, Crash Protection, Enterprise Dashboard, FR Enterprise Dashboard Desktop Application, FusionReactor Settings, Installer, JDBC, License + Activation, Logging, Metrics, Request Managment, Thread Management
Environment:
Resolution: Fixed
Last Updated: 03/Nov/11 3:36 PM
Affects Version: 2.0, 2.0.3, 2.0.4, 3.0, 3.0.1, 3.5
Fixed Version: 2.0.3, 2.0.4, 3.0, 3.0.1, 3.5
Server:
Platform: Windows XP, Windows 2000, Windows 2003, Linux, MacOS, Solaris, Windows Vista, Windows x64, AIX, Windows 7, Windows 2008
Related Issues:

FRS-237: Creating Daily FusionReactor Log Files

FRS-280: Setting VM Options via FRAPI

[frs-280] Setting VM Options via FRAPI

Setting VM Options via FRAPI

The Sun Java VM (JVM) offers many options to allow the user to control how the VM behaves. The HeapDumpOnOutOfMemoryError option for example is very useful if your server is suffering from heap memory problems and crashing.

By turning on HeapDumpOnOutOfMemoryError the JVM will create a dump file when java.lang.OutOfMemoryError is thrown that can be used in Memory Analysis tools such as MAT to track down memory problems.

A list of VM manageable runtime options includes:

Option Description
-XX:-HeapDumpOnOutOfMemoryError Dump heap to file when java.lang.OutOfMemoryError is thrown
-XX:-PrintClassHistogram Print a histogram of class instances on Ctrl-Break
-XX:-PrintConcurrentLocks Print java.util.concurrent locks in Ctrl-Break thread dump
-XX:-PrintGC Print messages at garbage collection
-XX:-PrintGCDetails Print more details at garbage collection
-XX:-PrintGCTimeStamps Print timestamps at garbage collection

Typically these options are added to the command line that starts the VM in the form of -XX:<optionname>.

There are however major problems with this:

  • Typically you have to edit the file that sets the command line options (e.g. jvm.conf) to change the settings.
  • If the -XX command line option wasn't added to the command line options at start-up then you can't set the option without restarting the server

Using FRAPI

It would be great if you control these VM options using code instead of the command line. FRAPI is a powerful and easy to use API that allows you to take control of FusionReactor programatically. Using FRAPI you can change the running configuration of FusionReactor or add extra debugging information to requests using just a few lines of code. We can also use FRAPI to set any "Managable" VM Option on the Sun JVM. (take a look at the following VM Options for details of all of the VM Options and look for "Manageable").

CFML
<cfset frapi = createObject("java", "com.intergral.fusionreactor.api.FRAPI").getInstance()>
<cfset frapi.setVMOption("HeapDumpOnOutOfMemoryError","true")>
<cfset frapi.setVMOption("PrintConcurrentLocks","true")>
<cfset frapi.setVMOption("PrintGCDetails","true")>   
JSP
<%@ page import="com.intergral.fusionreactor.api.*" %>
<%
    FRAPI frapi = FRAPI.getInstance();
    frapi.setVMOption("HeapDumpOnOutOfMemoryError","true");
    frapi.setVMOption("PrintConcurrentLocks","true");
    frapi.setVMOption("PrintGCDetails","true");   
%>

Requirements

You must be using a version of FRAPI that includes the setVMOption method (introduced in FRAPI 4.0) and a Sun JVM 1.5 or higher.

Notes

  • Due to a Bug in Java the HeapDumpPath VM Option may throw a NullPointerException and typically cannot be used.
  • Only VM Options that are known as "Manageble" can be set/changed at runtime. Manageable means that the option is dynamically writeable through the JDK management interface.

Summary

FRAPI is a easy way to set powerful VM Options while the server is running, without having to restart the server or edit configuration files.

Issue Details

Type: DevNet
Issue Number: FRS-280
Components: FRAPI
Environment:
Resolution: Fixed
Last Updated: 03/Nov/11 5:35 PM
Affects Version: 4.0.0
Fixed Version: 4.0.0
Server:
Platform: Windows XP, Windows 2000, Windows 2003, Linux, MacOS, Solaris, Windows Vista, Windows x64, AIX, Windows 7, Windows 2008
Related Issues:

FRS-232: Capturing ColdFusion’s Debug Output in FusionReactor

[frs-236] OnDemand Seminar: FusionReactor Server Monitor Introduction (with Railo)

Introduction to FusionReactor (with Railo)

This recorded webinar demonstrates how to install FusionReactor on your Railo servers as well as discusses the many useful features FusionReactor offers.

Webinar Outline


  • Installing FusionReactor with Railo
  • Configuring the FusionReactor JDBC Wrapper (in Railo)
  • FusionReactor Feature Run-Down



    • First look: What can I see?
    • System Level Metrics
    • Request Level Metrics
    • JDBC Requests
    • Crash Protection
    • Miscellaneous FusionReactor Goodies
    • Enterprise Monitoring



  • Summary

Watch Webinar

FusionReactor Introduction (with Railo) – Part 1 of 4
FusionReactor Introduction (with Railo) – Part 2 of 4
FusionReactor Introduction (with Railo) – Part 3 of 4
FusionReactor Introduction (with Railo) – Part 4 of 4

Get Started Monitoring

Download a free trial of FusionReactor Enterprise

About FusionReactor

FusionReactor is a generic Java application server monitoring tool that can help you to find, fix and prevent server problems.

Issue Details

Type: DevNet
Issue Number: FRS-236
Components: Installer
Environment:
Resolution: Fixed
Last Updated: 17/May/10 11:36 AM
Affects Version: 3.0, 3.0.1, 3.5
Fixed Version: 3.5
Server:
Platform:
Related Issues:

[frs-238] Getting Started With FusionReactor

Guide to Getting Started With FusionReactor

New to FusionReactor? Use this guide to help you Get Started.

This document refers to FusionReactor 3.0.1 and JDBC Wrap/Un-Wrap tool version 0.8 and is applicable to FusionReactor 3.5.1.

Getting Started with FusionReactor

Useful Links

Download FusionReactor
Install Guide (pdf)
JDBC Wrapper Tool

About the Author

Ajas Mohammed is a Sr. Software Engineer / Sr. ColdFusion Programmer at Absentys LLC with both a Bachelor's and Master's in Computer Science. Ajas has worked in the Information Technology field for over 10 Years.

Ajas Mohammed's Blog

Issue Details

Type: DevNet
Issue Number: FRS-238
Components: Compression, Content Filters, CPU + Memory, Crash Protection, Enterprise Dashboard, FR Enterprise Dashboard Desktop Application, FusionReactor Settings, Installer, JDBC, License + Activation, Logging, Metrics, Request Managment, Thread Management
Environment:
Resolution: Fixed
Last Updated: 20/Nov/09 1:15 PM
Affects Version: 3.0.1, 3.5
Fixed Version: 3.0.1, 3.5
Server:
Platform: Windows XP, Windows 2000, Windows 2003, Linux, MacOS, Solaris, Windows Vista, Windows x64, AIX, Windows 7, Windows 2008
Related Issues:

[frs-237] Creating Daily FusionReactor Log Files

Creating Daily Log Files in FusionReactor

FusionReactor captures a lot of very useful data into log files. These log files can be used to analyze a server in great detail. FusionReactor's logging mechanmism allows you to decide how many log files you want to keep and how big each log file should be but it doesn't allow you to rotate the log files every day.

It would be great to rotate the log files every day so that we can keep a record for as long as we want instead of losing data when the log files rotate automatically.

Using FRAPI (FusionReactor's API) this is easy to achieve.

Requirements

FusionReactor 3.0.1+

The Article

FusionReactor captures data about Requests, JDBC Requests, Resources, CrashProtection actions and details about itself into log files. These log files are typically placed under a log folder that can for each instance of FusionReactor. On a Windows installation the log folder might by for example:

   C:FusionReactorinstancecoldfusion.cfmx9.INT000Blog

About FRAPI

FRAPI is a powerful and easy to use API that allows you to take control of FusionReactor programatically. Using FRAPI you can change the running configuration of FusionReactor.

Here's how to instantiate the FRAPI interface in CFML. Once we have FRAPI available we are able to read this FusionReactor's configuration:

        <cftry>
            <cfset frapi = createObject("java", "com.intergral.fusionreactor.api.FRAPI").getInstance()>
            <cfset config = frapi.getRunningConfiguration()>
            <cfdump var="#config#">
            <cfcatch>
                <!--- FRAPI isn't available so FusionReactor probably isn't installed --->
            </cfcatch>
        </cftry>

Note that we always wrap FRAPI calls in a TRY/CATCH block. The reason for this is that FusionReactor may not be installed on every computer that you run the code on. The TRY/CATCH blocks will make sure that FRAPI calls will have no influence on your code even if FusionReactor isn't installed.

To try FRAPI, create a "frapiconfig.cfm" file with the CFML code above and run the page on a system that has FusionReactor installed. After the page has completed you should see a dump of FusionReactors configuration for this instance.

Once we have the configuration, we can change it and tell FusionReactor to update very easily like this:

            <cfset config.setValue("logfile", "#install_location#/instance/#instance_name#/log/#datetime#/reactor.log")>
            <cfset frapi.setRunningConfiguration(config)>

FRAPI.CFM

        <cftry>
            <cfset frapi = createObject("java", "com.intergral.fusionreactor.api.FRAPI").getInstance()>
            <cfsavecontent variable="debugData"><cfinclude template="classic.cfm"></cfsavecontent>
            <cfset frapi.trace( debugData )>
            <cfcatch>
                <!--- FRAPI isn't available so FusionReactor probably isn't installed --->
            </cfcatch>
        </cftry>

Notes

  1. Log File Sizes
    Some of FusionReactor's log files can fill very quickly. This is especially true of the JDBC log files. You need to take this into consideration when you set the log file size(s). As a counter point you have to be aware that using very large log files sizes and rotating the log files every day can very quickly use up lots of disk space. Please take care when setting the log file sizes and consider archiving or removing log files based upon there age.
  1. Log File Count
    FusionReactor lets you set the number of log files you want to keep for each log type. It's tempting to set this to a large number so that FusionReactor doesn't lose any information but you should consider that the more log files FusionReactor has to manage and rotate, the slower the log file rotate will be. I would not advise you to set the log file count higher than 20 log files. It is more advisable to balance the number of log files and the log file size. For example, of you set the log file count to 10 and the log file size to 20480 (log file size is set in KBytes) then you would create a maximum of 200MBytes of log data (for the specific log).

Fast Track

  1. Copy the frapi.rotatelogdirectoy.cfm file linked to this article to a folder where you run ColdFusion (for example): <coldfusion folder>/wwwroot/fusionreactor
  2. Create a scheduled task to run this page once per day at 00:00:00 midnight

Summary

By rotating FusionReactor's log files daily you can keep log files for as long as you want and can prevent you from losing this valuable data when FusionReactor rotates it's log files automatically when the size of the file has been reached.

Issue Details

Type: DevNet
Issue Number: FRS-237
Components: Logging
Environment:
Resolution: Fixed
Last Updated: 06/Nov/09 6:51 PM
Affects Version: 2.0, 2.0.3, 2.0.4, 3.0, 3.0.1, 3.5
Fixed Version: 3.0.1, 3.5
Server:
Platform: Windows XP, Windows 2000, Windows 2003, Linux, MacOS, Solaris, Windows Vista, Windows x64, AIX, Windows 7, Windows 2008
Related Issues:

FRS-232: Capturing ColdFusion’s Debug Output in FusionReactor