FusionReactor Blog - News and expert analysis for Java APM users.
Continuous Profiling
Fig 1. Continuous Profiler alerting configeration
Software development
Continuous Profiling in Production
Profiling is the process of monitoring various parameters such as Method Execution, Thread Execution, Object Creation and Garbage Collection. All of which helps you understand your application system and solve any issue that may arise. It is simply the analysis of your application’s performance by measuring and comparing the frequency and duration of your function calls. Hence, profiling helps you with a detailed observation of your target application execution and its resource utilization.
” In software engineering, profiling is a form of dynamic program analysis that measures, for example, the space or time complexity of a program, the usage of particular instructions, or the frequency and duration of function calls. Most commonly, profiling information serves to aid program optimization.”
Wikipedia
There are a number of other approaches to profiling. However, if you are in a production environment, a low overhead sampling profiler seems to be a better choice. And given that continuous profiling is an approach that identifies and improves the faulty portion of your code on the fly, makes it the best option.
Why Continuous Profiling?
Because ‘continuous profiling’ is the collection of line-level profiling performance data from a production environment and making such data available to developers and operations team for rapid analysis. However, this is in contrast to an Ad Hoc production profiling – which involves the connection of a profiler to a production environment on an intermittent on-demand basis. Continuous profiling is a mechanism with significantly low overhead, that allows its user unlimited access (in real-time) to insights into code performance issues during development, staging and in production environments.
Fig 1. Continuous Profiling in a Software Development Life Cycle (SDLC).
Notably, debugging and profiling are two essential phases in software development. And as such, these stages could impact a severe strain on developers, software analysts, and the entire team if not well implemented. More so, with a Continuous Profiling system in place, developers can get a line-level understanding of the performance of their code – including but not limited to the consumption rate of such resources as CPU time, wall-clock time, memory and disk I/O which are naturally in short supply. Because these resources could lead to thread deadlocks and/or application bottlenecking when not evenly allocated.
Therefore, we need continuous profiling for the following reasons;
Evaluate overall application performance.
Memory affects everything; therefore, it is critical to be able to pinpoint memory issues throughout an application development life cycle. One of the most useful functionalities of continuous profiling is that it can show us how much memory was allocated for a specific transaction or web request. With continuous profiling, you can actively track memory and subsequently optimize memory usage. This allows developers to get an instant insight into the heap in their production java applications with low overhead memory. Also, having a wider overview of how specific threads are running is essential to understanding performance syntax. A continuous profiler provides an interface where users can instrument individual threads in order to get to the root cause of the stability of performance-related issues.
Fig 2. Continuous Profiler – Thread profiling Interface.
Reduce Server cost and Find Bottlenecks
Another notable feature of continuous profiling is its ability to profile code and spot performance bottlenecks on the go. Continuous profilers come equipped with a low overhead performance analysis feature, which is perfect for your production environment. Configured to take snapshots at regular intervals, continuous profiling provides excellent real-time insight so you will never miss an issue. Since profiling involves the process of measuring what part of your application is consuming a particular resource, a continuous profiler is able to track not just memory but also CPU usage on individual threads. As well as find and tune inefficient processes running on your application server.
Fig 3. Continuous Profiler – CPU Snapshots.
Determine performance issues in a particular method.
The heap memory is the runtime data area where the Java Virtual Machine (JVM) allocates memory for all class instances, methods
Fig 4. Continuous Profiler – Heap Histogram.
What is the best tool for profiling?
Although profiling in a development environment is easy, it is hardly enough. More so, to say that pinpointing performance issues in a production environment is easy is far from true. Hence selecting the best tool for continuous profiling becomes really necessary. However, it is pertinent to consider the following conditions when choosing one, such as;
- Choosing the right sampler for your programming language (e.g Java) with the least overhead and an optimal runtime.
- Selecting the perfect database for storing the data from your profilers.
- having a seamless means to generate reports from this data
That said, the aforementioned tasks would normally require the intuition of an expert. Nonetheless, with such sampling profilers like FusionReactor, there would be no need for a specialized software analyst during production-level profiling.
Would you like to know more about the FusionReactor profiler? Click here to start a free trial or book a demo.
FusionReactor Database Monitoring software
FusionReactor database monitoring software enables the monitoring and tracking of a database’s performance. This allows users to identify and solve any potential performance issues as well as track changes in the database’s function. FusionReactor analyzes and captures all the data related to your SQL statements so you can focus on improving performance and reducing bottlenecks. See right down to which SQL statements were run, the number of rows returned and the time spent on the query.
Database monitoring tools are used by database administrators to help maintain database performance and pinpoint potential issues to
Number 1 database monitoring software for small business
Best by customer satisfaction
FusionReactor is ranked number one Database Monitoring Software for small businesses, ranked by customer satisfaction. This means that our customers put us here, and our support and product teams work very hard in maintaining our position.
Why did our customers choose FR?
Ben N.
The most powerful feature that we’ve used so far is the automatic profiling of long-running requests. This has proved to be
enormously value for debugging bottlenecks as we can see exactly where the requests are being blocked by processing. With this feature, we were able to, within a single day, vastly improve the responsiveness and stability of our service.
Jan J. Managing Director, Pixl8 GmbH
Finding performance issues is so much easier when using FusionReactor. Typically and especially in complex web applications (think MVC, layered architecture, etc.) sometimes
those nasty db -query-in-loop are tough to find. Easy as pie with FR.
Forrest H. Solutions Architect, Auto Europe
Fusion Reactor has been key in helping us work through issues in deploying our new platform
Dave L. The Big Kahuna and CEO, Angry Sam Productions, Inc.
If you’re having issues with slow CFML requests, I definitely recommend giving FusionReactor a
try
Brad W. Senior Application Architect, Ortus Solutions
I’ve used it to find production errors, memory leaks, and performance issues as well.
noah b. Software Engineer, SYNNEX
FusionReactor APM is without a doubt one of my best tools for real-time monitoring of my applications,
Tony B. Software Developer and Operations, Trialsmith, Inc
Bottom line, FusionReactor saves time and development costs.
Jennifer H. Senior ColdFusion Developer, Webauthor.com
We value the wide
range realtime and logged metrics that are readily available. It is a tool that is used by entire IT team – from developers, to DBAs to NetOps
Read our reviews
Leave us a review
Dynamically Instrumenting ColdFusion Component Methods With FusionReactor Tracked Transactions In Lucee CFML
Originally
One of the really fun features of ColdFusion is its highly dynamic nature. Whether yonMissingMethod()getFunctionCalledName()
DISCLAIMER: Just because ColdFusion is a highly dynamic language, it doesn’t necessarily mean that you should be using all of these language features. Often times, the most clever code becomes the code that is hardest to maintain in the long-run. In reality, you should strive for boring code that everyone can understand.
To explore this idea, I created a very silly ColdFusion component that has a variety of public and private methods. Within these various methods, I am including nested method calls that are executing both with and without explicit scoping. I added all of this complexity to make sure that my “proxy” logic handles the various ways in which a developer may have wired things together:
component
output = false
hint = "I provide a sample component on which to try annotating methods."
{
public any function init( required any javaAgentHelper ) {
// This component is going to ask the JavaAgentHelper to add instrumentation to
// all of the Public and Private methods. This will wrap them in "tracked
// transactions", which I'm calling "Segments" (a hold-over from New Relic).
javaAgentHelper.annotateMethods( variables );
}
// ---
// PUBLIC METHODS.
// ---
public numeric function test() {
sleep( randRange( 10, 50 ) );
// Testing with and without scoping.
this.publicMethodA();
variables.privateMethodA();
publicMethodB();
privateMethodB();
return( getTickCount() );
}
public void function publicMethodA() {
sleep( randRange( 10, 50 ) );
publicMethodC();
}
public void function publicMethodB() {
sleep( randRange( 10, 50 ) );
this.publicMethodD();
}
public void function publicMethodC() {
sleep( randRange( 10, 50 ) );
}
public void function publicMethodD() {
sleep( randRange( 10, 50 ) );
variables.privateMethodE();
privateMethodF();
}
// ---
// PRIVATE METHODS.
// ---
private void function privateMethodA() {
sleep( randRange( 10, 50 ) );
// Testing with scoping.
variables.privateMethodC();
}
private void function privateMethodB() {
sleep( randRange( 10, 50 ) );
// Testing without scoping.
privateMethodD();
}
private void function privateMethodC() {
sleep( randRange( 10, 50 ) );
}
private void function privateMethodD() {
sleep( randRange( 10, 50 ) );
}
private void function privateMethodE() {
sleep( randRange( 10, 50 ) );
}
private void function privateMethodF() {
sleep( randRange( 10, 50 ) );
}
}
view rawMyService.cfc hosted with ❤ by GitHub
As you can see, this ColdFusion component is nothing more than a set of stubbed-out method calls that demonstrate simulated latency. The only point of interest to note is that the component is receiving an instance of JavaAgentHelper.cfc when it is instantiated. It is then asking the JavaAgentHelper.cfc component to add instrumentation to its own instance:
javaAgentHelper.annotateMethods( variables );
Now, before we dive into the details of what JavaAgentHelper.cfc is doing, let’s try to instantiate and consume the MyService.cfc ColdFusion component to see what happens in FusionReactor:
<cfscript> // MyService is going to use the JavaAgentHelper to "wrap" each method call so that // all methods calls on MyService, whether PUBLIC or PRIVATE, will be instrumented // with a FusionReactor "Tracked Transaction". service = new MyService( new JavaAgentHelper() ); dump( service.test() ); </cfscript>
view rawtest.cfm hosted with ❤ by GitHub
If we run this page and then look in FusionReactor’s dashboard, we get the following data:
As you can see, under the Relations tab, we get the full breakdown of all the public and private method calls made within MyService.cfc. The Gantt chart only shows a few levels; but, if you look at the full Transaction History, you can see all the nested method calls.
We can also see the same data in FusionReactor’s Cloud dashboard under the Tracing tab:
The Cloud dashboard shows all the same Transactions; but, is a bit more colorful.
Ok, now that we see what the automatic method instrumentation is doing for us, let’s look at the JavaAgentHelper.cfc to see how it works. Internally, the .annotateMethods() call is iterating over each method in the target component and is swapping out every given method with a proxy method that calls the original method, wrapped in a “Tracked Transaction”:
component
output = false
hint = "I help interoperate with the Java Agent that is instrumenting the ColdFusion application (which is provided by FusionReactor)."
{
// I initialize the java agent helper.
public any function init() {
// The FusionReactor Agent is not available in all contexts. As such, we have to
// be careful about trying to load the Java Class; and then, be cautious of its
// existence when we try to consume it. The TYPE OF THIS VARIABLE will be used
// when determining whether or not the FusionReactor API should be consumed. This
// approach allows us to use the same code in the calling context without having
// to worry if the FusionReactor agent is installed.
try {
// NOTE: The FRAPI was on Version 8.2.3 at the time of this writing.
variables.FRAPIClass = createObject( "java", "com.intergral.fusionreactor.api.FRAPI" );
} catch ( any error ) {
variables.FRAPIClass = "";
}
}
// ---
// PUBLIC METHODS.
// ---
/**
* I wrap all of the methods defined in the given Component Scope (VARIABLES) with
* PROXY methods that will automatically create a FusionReactor "tracked transaction"
* that records the timing of each invocation.
*
* @privateScope I am the VARIABLES scope of the component being instrumented
* @annotatePrivateMethods I determine if private methods should be instrumented.
*/
public void function annotateMethods(
required struct privateScope,
boolean annotatePrivateMethods = true
) {
// In order to make sure the proxy methods can create FusionReactor segments,
// let's store a reference to the JavaAgentHelper in the private scope. This will
// then be accessible on the VARIABLES scope.
privateScope.__javaAgentHelper__ = this;
// -- START: Proxy method. -- //
// Every relevant method in the given Component Scope is going to be replaced
// with this PROXY method, which wraps the underlying call to the original method
// in a FusionReactor Segment.
// --
// CAUTION: We need to use a FUNCTION DECLARATION here, not a CLOSURE, because
// this Function needs to execute in the CONTEXT of the ORIGINAL component (ie,
// it has to have all the correct Public and Private scope bindings).
function instrumentedProxy() {
var key = getFunctionCalledName();
var proxiedKey = ( "__" & key & "__" );
var segment = variables.__javaAgentHelper__.segmentStart( key );
try {
// NOTE: In a Lucee CFML component, both PUBLIC and PRIVATE methods can
// be accessed on the VARIABLES scope. As such, we are able to invoke the
// given method on the private component scope regardless of whether or
// not the proxied method is public or private.
return( invoke( variables, proxiedKey, arguments ) );
} finally {
variables.__javaAgentHelper__.segmentEnd( segment );
}
}
// -- END: Proxy method. -- //
// Replace each Function in the target component with a PROXY function.
// --
// NOTE: Both Public and Private methods show up in the private scope of the
// component. As such, we only need to iterate over the private scope when
// looking for methods to instrument.
for ( var key in structKeyArray( privateScope ) ) {
// Skip if not a defined, custom method.
if (
( key == "init" ) ||
! structKeyExists( privateScope, key ) ||
! isCustomFunction( privateScope[ key ] )
) {
continue;
}
// Skip if we're only annotating PUBLIC methods, and this key isn't aliased
// in the PUBLIC scope.
if (
! annotatePrivateMethods &&
! structKeyExists( privateScope.this, key )
) {
continue;
}
var proxiedKey = ( "__" & key & "__" );
// Regardless of whether or not we're dealing with a PUBLIC method, we always
// want to create a proxy in the PRIVATE scope - remember, all methods, both
// PUBLIC and PRIVATE, are accessible on the private Component scope.
privateScope[ proxiedKey ] = privateScope[ key ];
privateScope[ key ] = instrumentedProxy;
// However, if the original method is PUBLIC, we ALSO want to alias the given
// method on the PUBLIC scope so that we can allow for explicitly-scope calls
// (ie, this.method).
if ( structKeyExists( privateScope.this, key ) ) {
privateScope.this[ key ] = privateScope[ key ];
}
}
}
/**
* I end the segment and associate the resultant sub-transaction with the current
* parent transaction.
*
* @segment I am the OPAQUE TOKEN of the segment being ended and timed.
*/
public void function segmentEnd( required any segment ) {
if ( shouldUseFusionReactorApi() ) {
// In the case where the segment is not available (because the FusionReactor
// agent has not been installed), it will be represented as an empty string.
// In such cases, just ignore the request.
if ( isSimpleValue( segment ) ) {
return;
}
segment.close();
}
}
/**
* I start and return a new Segment to be associated with the current request
* transaction. The returned Segment should be considered an OPAQUE TOKEN and should
* not be consumed directly. Instead, it should be passed to the .segmentEnd() method.
* Segments will show up in the Transaction Breakdown table, as well as in the
* "Relations" tab in the Standalone dashboard and the "Traces" tab in the Cloud
* dashboard.
*
* @name I am the name of the segment being started.
*/
public any function segmentStart( required string name ) {
if ( shouldUseFusionReactorApi() ) {
return( FRAPIClass.getInstance().createTrackedTransaction( javaCast( "string", name ) ) );
}
// If the FusionReactor API feature is not enabled, we still need to return
// something as the OPAQUE SEGMENT TOKEN so that the calling logic can be handled
// uniformly within the application code.
return( "" );
}
// ---
// PRIVATE METHODS.
// ---
/**
* I check to see if this machine should consume the FusionReactor static API as part
* of the Java Agent Helper class (this is to allow the methods to exist in the
* calling context without a lot of conditional consumption logic).
*/
private boolean function shouldUseFusionReactorApi() {
// If we were UNABLE TO LOAD THE FRAPI CLASS, there's no API to consume.
if ( isSimpleValue( FRAPIClass ) ) {
return( false );
}
// Even if the FRAPI class is loaded, the underlying FusionReactor instance may
// not yet be ready for interaction. We have to wait until .getInstance() returns
// a non-null value.
if ( isNull( FRAPIClass.getInstance() ) ) {
return( false );
}
return( true );
}
}
view rawJavaAgentHelper.cfc hosted with ❤ by GitHub
There’s a lot of fun, dynamic stuff going on in this code: we’re declaring a function inside of another function (that is not a closure), we’re injecting methods into a component, we’re dynamically checking the name of an invoked method, we’re messing with Public and Private scopes.
It’s just hella exciting!
Dynamically Instrumenting ColdFusion Component Methods using FusionReactor Cloud
More information about FusionReactor Cloud or to start a free trial or request a free demo
Sending FusionReactor Tracked Transaction Metrics To The Cloud Dashboard With Lucee CFML 5.2.9.40
Originally posted by Ben Nadel on February 1, 2020, reproduced by kind permission.
One of the nice features of FusionReactor is that when you create a sub-Transaction with the FRAPI, you can graph that Transaction performance against the server’s CPU and Heap profile. This helps identify correlations, bottlenecks, and performance opportunities. This works out-of-the-box with the Standalone dashboard. However, at the time of this writing, FusionReactor does not send sub-Transaction metrics to the Cloud dashboard automatically. In order to graph sub-Transaction metrics in the Cloud dashboard, you have to explicitly enable them in your ColdFusion code. This was not obvious to me; so, I wanted to demonstrate how this works in Lucee CFML 5.2.9.40.
ASIDE: I want to give a special shout-out to Michael Flewitt, a Support
engineer at Integral (makers of FusionReactor), who spent no less than 3-hours working with me, helping me to figure how this code works (and why I wasn’t seeing the results that I expected to see). He is a true champion!
When you create a tracked-Transaction in your ColdFusion code, FusionReactor is implicitly logging six metrics about that Transaction’s performance. So, for example, when you create a Transaction called demo-segment:
frapi.createTrackedTransaction( "demo-segment" )
… FusionReactor implicitly logs the following numeric-aggregate metrics:
/transit/txntracker/demo-segment/active/activity/transit/txntracker/demo-segment/active/time/transit/txntracker/demo-segment/history/activity/transit/txntracker/demo-segment/history/time/transit/txntracker/demo-segment/error/activity/transit/txntracker/demo-segment/error/time
Because of these metrics, we can graph the sub-Transaction, demo-segment, in the Standalone dashboard:
In order to do the same thing with the Cloud dashboard, we have to explicitly enable the aforementioned metrics to be streamed to the Cloud. To see this in action, I’ve created a simple CFML page that creates a tracked-Transaction and then calls, .enableCloudMetric(), on 4-of-the-6 implicitly-created metrics:
<cfscript>
// Get the running FusionReactor API (FRAPI) instance from the FRAPI factory class.
// --
// Java Docs: https://www.fusion-reactor.com/frapi/8_0_0/com/intergral/fusionreactor/api/FRAPI.html
frapi = createObject( "java", "com.intergral.fusionreactor.api.FRAPI" )
.getInstance()
;
// ------------------------------------------------------------------------------- //
// ------------------------------------------------------------------------------- //
// By default, FusionReactor will use the name of the application as defined in the
// Application.cfc ColdFusion framework component. However, we can set the name
// programmatically.
frapi.setTransactionApplicationName( "FRAPI-Testing" );
// By default, FusionReactor will calculate the transaction name based on the request
// context. It actually "understands" the fact that we're using Framework One (FW/1)
// in production and uses the "action" value as the transaction name. That's the
// beauty of using an APM product that is embedded within the ColdFusion and CFML
// community. That said, we can set the transaction name programmatically.
// --
// See Framework Support: https://www.fusion-reactor.com/support/kb/frs-431/
frapi.setTransactionName( "testing-cloud-transaction-metrics" );
// ------------------------------------------------------------------------------- //
// ------------------------------------------------------------------------------- //
try {
// Let's explicitly wrap a segment of our code in a custom, tracked transaction.
// This way, we can see how this code executes in the context of parent request.
subtransaction = frapi.createTrackedTransaction( "demo-segment" );
// When a custom Transaction is explicitly created in the ColdFusion code,
// FusionReactor sends the Transaction data to the CLOUD dashboard; however, by
// default, it doesn't send the METRICS about that Transaction to the CLOUD
// dashboard. This means that we can see the Transaction in the Tracing and the
// data-tables; but, we can't graph it in our custom graphs. In order to do this,
// we have to explicitly set the Transaction-related metrics to be cloud-enabled.
frapi.enableCloudMetric( "/transit/txntracker/demo-segment/active/activity" );
frapi.enableCloudMetric( "/transit/txntracker/demo-segment/active/time" );
frapi.enableCloudMetric( "/transit/txntracker/demo-segment/history/activity" );
frapi.enableCloudMetric( "/transit/txntracker/demo-segment/history/time" );
// frapi.enableCloudMetric( "/transit/txntracker/demo-segment/error/activity" );
// frapi.enableCloudMetric( "/transit/txntracker/demo-segment/error/time" );
sleep( randRange( 500, 1500 ) );
} finally {
subtransaction.close();
}
</cfscript>
<!--- ------------------------------------------------------------------------------ --->
<!--- ------------------------------------------------------------------------------ --->
<script>
// Simulate regular throughput / traffic to this endpoint by refreshing.
setTimeout(
function() {
window.location.reload();
},
1000
);
</script>
view rawcloud-transaction.cfm hosted with ❤ by GitHub
Every 60-seconds, metrics in the local FusionReactor instance are aggregated and sent to the Cloud. So, once this demo-page has been running for a while, and the metrics have been sent to the Cloud, and have had time to get processed by the metrics-ingress, we should be able to find the following metrics in the custom-Graph tooling:
NOTE: I’m including the “error” metrics below, even though I didn’t enable them for the Cloud in my demo code. I’m including them for documentation / completeness purposes.
/custom/transit/txntracker/demo-segment/active/activity/custom/transit/txntracker/demo-segment/active/time/custom/transit/txntracker/demo-segment/history/activity/custom/transit/txntracker/demo-segment/history/time/custom/transit/txntracker/demo-segment/error/activity/custom/transit/txntracker/demo-segment/error/time
Notice that each of the metrics has been automatically prefixed with, /custom/:
NOTE: Part of the reason that I spent 3-hours with Michael Flewitt is because not all of the metrics were consistently showing up for me. And, in fact, even as I write this, I don’t see all of the metrics being logged in this demo. This appears to be either a bug in the rendering code; or, a timing issue with the metrics-ingress processing.
At this point, I can graph my sub-Transactions metrics in the Metrics section of the Cloud dashboard; but, I can’t add it to any of my Server performance graphs. In order to do this, I have to add Filtering to the query and then save the graph as a Server Template using the literal_or function:
Once I do this (and add the graph to my metric’s “Profile”), I can then find the Demo Segments graph in the Graphs section of my server performance monitoring:
Hopefully this is somewhat helpful for anyone else who might be using FusionReactor with their Lucee CFML code; and, is using the Cloud dashboard, not the Standalone dashboard.
Cloud Metrics is a Bit of an Uphill Battle
I am loving FusionReactor’s out-of-the-box functionality; but, to be honest, working with custom metrics has been an uphill battle. I am sure that a lot of this is my unfamiliarity with the tooling. But, some of the battle revolves around the stability of the platform. Some points of friction:
- The “Metrics” dropdown menu in the Query configuration fails to load like 90% of the time. Which means that creating a simple graph involves several minutes of page-refreshing in an attempt to get the “Metrics” dropdown to load.
- The custom metrics which I am enabling in my code often don’t show up in the “Metrics” dropdown. Which means, even when the dropdown finally loads (see point above), my metrics are not there.
- The “Filter” configuration only seems to load if the “Metrics” also loaded. And, since the Metrics fail to load most of the time (see above point), I can’t add Filtering to my queries.
- The Filtering functionality is confusing. For example, why do I even have to add filtering for “transaction related” metrics? If a Transaction can only ever be created as part of a running application (on a server), why do I have to explicitly identify the metric as a “Server template”? It would be great if all transaction-related metrics were defaulted as server templates.
- The difference between a “Graph” and a “Profile” took me a while to understand. I think this is more of a user-interface (UI) problem than anything else. Since Graphs and Profiles are created and updated in the same place, the optionally-hierarchical relationship between the Profile and the graphs that the Profile renders is not immediately obvious. Maybe I’ll make a demo of how this now that I think I finally get it.
I’ve discussed some of these issues with the FusionReactor support team and they are looking into it. For example, in the future, I won’t have to explicitly enable “cloud metrics” for sub-Transactions – that will just automatically happen.
While I’ve had some friction with the Cloud Metrics, I do want to be clear that I am loving FusionReactor. I’ve been living in it for the past week and have already identified a multitude of performance issues in my
See more about FusionReactor Cloud which has a free 14 day trial
Configuring FusionReactor in CommandBox
CommandBox is a tool that allows you to deploy your CFML applications through an easy-to-use command-line interface.
Instead of deploying a tomcat-based installer version of ColdFusion or Lucee, CommandBox utilizes an Undertow servlet and deploys a war file for the CFML server. This allows you to switch between a Lucee and ColdFusion server with the same application and configuration.
In terms of configuration, rather than having a multitude of small files, you can control everything from a single JSON file containing all settings for the Undertow servlet, application server as well as any installed modules.
Commandbox-fusionreactor module
To install FusionReactor in CommandBox, we recommend that the commandbox-fusionreactor module is used. This is a module designed and maintained by Ortus (makers of CommandBox).
The module, along with the FusionReactor module ensuring your FusionReactor instance is the latest version is stored in ForgeBox. This makes installation simple as you can run a single command to load the module.
box install commandbox-fusionreactor
Licensing FusionReactor
With the commandbox-fusionreactor module installed, you have access to the fr command namespace.
You can run commands such as ‘fr open’ to open FusionReactor in the browser.
To make licensing FusionReactor simple, you can run ‘fr register “myLicenseKey”‘, this automatically applies your license key to each running instance.
Passing in configuration
Any modified settings in FusionReactor are stored in the reactor.conf file of each FusionReactor instance. With CommandBox you can set this reactor.conf file to be passed into each running instance by running:
server set fusionreactor.reactorconfFile=path/reactor.conf
There are also several values you can set for FusionReactor directly through the server set command, see the full list here: https://commandbox.ortusbooks.com/embedded-server/fusionreactor#additional-jvm-args
Setting a fixed Application name
The default behaviour of FusionReactor automatically detects the name of the running application and applies this to transactions.
If you would like to disable this, you can do so by running:
server set fusionreactor.autoApplicationNaming=false server set fusionreactor.defaultApplicationName=myApp
Setting the Instance name
The instance name of FusionReactor will either be set to the name of the directory you are running box from, or to the name of the CommandBox server.
For example, if I have no server name set and run CommandBox from a folder called test, my instance is called test.
You can override this value via the server name, which is a value defined in the server.json config file. You can set this value by running:
//Within CommandBox server set name="myName" //Outside CommandBox using environment variables box server set name = "$var1+$var2+myName"
Removing FusionReactor
When removing the FusionReactor module, it is important to ensure that the –system flag is set on the uninstall command, i.e:
box uninstall commandbox-fusionreactor --system
If the system flag is not specified, CommandBox will try to uninstall from the current package, not from the CommandBox system packages.
Running ‘box restart’ after performing the uninstallation ensures that the module is not stored in memory and reloaded when a CommandBox server is restarted.
Running in Linux
When running in a Linux Desktop, we have seen that CommandBox can crash without warning. This is due to an issue with CommandBox interacting with the system tray.
If you are running Ubuntu 18.04 or greater, you will be required to install the libappindicator-dev package to allow CommandBox to use the system tray.
Alternatively, you can disable the CommandBox system tray element. To do this, run the following commands:
Local: server set trayEnable=false Global: config set server.defaults.trayEnable=false
What’s new in FusionReactor 8.3.0
Improved alerting, Event Snapshots, improved Cloud UI and more
FusionReactor 8.3 has new CPU alerts in crash protection. We have redeveloped Event Snapshot for ColdFusion users which means that it no longer courses server issues. FusionReactor Cloud now lets you choose your theme and gives you better warning notifications. As well as a host of other smaller improvements and bug fixes.
FusionReactor 8.3 is now available for download now!
CPU alerting in Crash Protection
This feature has been requested a lot. If you have a long-running request, large GC issues or problems with background tasks or threads then FusionReactor 8.3 Crash Protection can send you an alert.
It is easy to use and works in a similar way to Memory Protection. You simply set the required threshold, the minimum duration and your alerting strategy.
When the alert threshold is hit, you will have 3 available alerting strategies:
- Send an email containing the details of the stack, running requests and system metrics into your inbox
- Queue requests entering the application server until the CPU is below the threshold
- Reject requests entering the application server until the CPI is below the threshold
Event Snapshot for ColdFusion in FusionReactor 8.3
Every error now has an Event Snapshot, historically if you were a CF user then this might have had a negative effect on your server; this was down to CF error handling.
We have now redeveloped this feature so that it no longer impacts your server and enabled Event Snapshots by default.
Any recurring errors tracked by FusionReactor will now automatically generate an Event Snapshot, in the snapshot you will see the Exception, source code, stack frames and variables.
Note that for snapshots generated for Adobe ColdFusion servers, variables will only be available for the top stack frame.
Improvements to the Cloud UI
We have made a number of smaller improvements to the FusionReactor Cloud that make a large difference to the usability.
Themes
Everyone has a preference to whether they use a light theme or a dark theme so Cloud will now let you choose. Simply go into account settings or use the keyboard shortcut Q and you can decide on a light or a dark theme. Cloud trails are free – even if you are an existing customer.
Better warning messages
We have improved our messaging for offline servers and for when data is not available. You will now be prompted to change the timeframe to and adjust set filters to find the relevant data.
Other Improvements
- Brand new documentation to be released soon
- Middle click support in local FR UI
- Support for upcoming CommandBox 4.9
- Fix for ColdBox tracking
- Support for Arm 64 / Aarch 64 architectures
- Support for Wildfly 14 – 19
- Tracking for RMI calls in applications (Java 1.7 – 1.9)
- Kubernetes detection
Want to know more about FusionReactor 8.3 or have any questions?
Our support team is holding a Live Stream Q&A tomorrow at 11 am PST.
What’s new in FusionReactor 8.3.0 – Live Stream Support
On February 11th at 7PM UTC (11AM PST; see your local time), we are running our first live demo and Q&A session on the FusionReactor YouTube channel, covering what’s new in FusionReactor 8.3.0, which is releasing very soon.
This session will be hosted by Michael Flewitt, a technical support engineer for FusionReactor and the FusionReactor Cloud and features Charlie Arehart, a ColdFusion consultant and all-around expert as a guest.
Our intention is to host a regular demo and Q&A sessions covering everything related to the FusionReactor product, CFML/Java development and other exciting projects we are working on at Intergral.
During these sessions, you will have the opportunity to ask us any questions you may have and get direct advice from our support team.
Our first session will cover what’s new in FusionReactor 8.3.0!
We have been working hard to give you powerful new features and an improved user experience with FusionReactor including;
- Enhanced Crash Protection
- Superior error detection in ColdFusion
- Interactive and intuitive self-service support
- Significant user experience improvements in the Cloud
- A load more cool features
Subscribe to the FusionReactor Youtube channel and set a reminder here!
The demo should take around 30 minutes, at which point we will have time to answer all your questions.
We look forward to chatting with you soon!
Understanding StackTraces in Java
By guest author Thorben Janssen
The StackTrace is one of the key concepts in Java. It’s a call stack for the thread and lists all method calls since the start of the thread. You have probably seen its textual representation in your log file or console output. It gets printed to System.out whenever an exception is thrown and not handled by your application. The following snippet shows a typical example of such an output.
java.lang.NumberFormatException: For input string: "123a45" at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:68) at java.base/java.lang.Long.parseLong(Long.java:699) at java.base/java.lang.Long.valueOf(Long.java:1151) at org.thoughts.on.java.TestStackTrace.testStackTrace(TestStackTrace.java:17) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:567) ...
Using monitoring tools, like FusionReactor, or by calling the getStackTrace() method on the current thread, you can access the StackTrace for all active threads in your JVM. But there are other ways to examine and work with a StackTrace.
In most cases, you will not look at a StackTrace until you need to analyze an Exception. The StackTrace is part of an Exception object, and it shows all method calls that happened until the exception was thrown. That shows you where the exception occurred and how you reached that specific part of the code.
In the next step, you can then analyze your code and find out what caused the exception. But that’s a topic for a different article. In this one, I want to tell you more about exceptions with their StackTraces and all the information they provide, so that you have a better understanding of StackTraces in Java.
Exceptions in Java
An exception gets thrown whenever an error happens within a Java application. It gets represented by an object of the java.lang.Exception class or one of its subclasses. The JDK provides you with a huge set of different Exception classes. If you want, you can also implement your own business exceptions.
It’s a general best practice to use the most specific exception class for each error. A typical example for that is the valueOf method of the java.lang.Long class. You can call it with a java.lang.String and it throws a java.lang.NumberFormatException if the String has a format that can’t be parsed to a Long. The NumberFormatException is a subclass of the IllegalArgumentException, which indicates that an invalid argument value was passed to a method. As you can see, the IllegalArgumentException would describe the error situation, but the NumberFormatException is more specific and should be preferred.
private Long parseToLong(String s) {
return Long.valueOf(s);
}
Long l;
try {
l = parseToLong(s);
} catch (NullPointerException npe) {
// handle NullPointerException
log.error("No value provided. Using 0 as default.", npe);
l = 0L;
}
log.info(l);
Using the most specific exception class makes your code easier to read, and it enables you to implement a different catch clause for each exception class. This allows you to handle each error situation differently.
You could, for example, decide to throw a NullPointerException if the provided String is null and throw a NumberFormatException if it doesn’t have the correct format.
private Long parseToLong(String s) {
if (s == null) {
throw new NullPointerException("String can't be null");
}
return Long.valueOf(s);
}
In the code that calls this method, you can then implement 2 separate catch blocks that handle the NullPointerException and the NumberFormatExceptions in different ways. I did that in the following code snippet to provide different error messages for both situations. But you could, of course, use the same approach to implement a more complex error handling or to provide a fallback to default values.
Long l;
try {
l = parseToLong(s);
} catch (NullPointerException npe) {
// handle NullPointerException
log.error("No value provided. Using 0 as default.", npe);
l = 0L;
} catch (NumberFormatException nfe) {
// handle NullPointerException
log.error("Provided value was invalid. Using 0 as default.", nfe);
l = 0L;
}
log.info(l);
The structure of a StackTrace
In the previous code snippet, I wrote log messages that contained the caught exception objects. The following snippet shows an example of such a message in the log file. Your application writes a similar message for all unhandled exceptions to your console.
15:28:34,694 ERROR TestStackTrace:26 - Provided value was invalid. Using 0 as default. java.lang.NumberFormatException: For input string: "123a45" at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:68) at java.base/java.lang.Long.parseLong(Long.java:699) at java.base/java.lang.Long.valueOf(Long.java:1151) at org.thoughts.on.java.TestStackTrace.parseToLong(TestStackTrace.java:39) TestStackTrace.java:39 at org.thoughts.on.java.TestStackTrace.testStackTrace(TestStackTrace.java:18) TestStackTrace.java:18 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:567)
As you can see, the log message contains a long list of class and method names. This is the textual representation of the StackTrace of the exception. Whenever a new method gets called, it gets added to the top of the stack, and after it got executed, it gets removed from the stack. Based on this approach, the last method that got called before the exception occurred is at the top of the StackTrace and logged first. The following elements in the StackTrace and lines in the log file show which methods were previously called to reach the part of the code that caused the exception.
Using StackTraces to analyze incidents
If you take another look at the previously shown StackTrace, you can see that the exception was created in the forInputString method of the NumberFormatException class. The actual problem occurred in the parseLong method of the Long class, which was called by the valueOf method of the same class, which was called by parseToLong method of my TestStackTrace class. As you can see, the StackTrace provided by the NumberFormatException clearly shows where the exception happened and which chain of method calls lead to it.
This information is a good start to analyze an exception and to find the actual cause of it. But quite often, you will need more information to understand the issue. The Exception object and its StackTrace only describe which kind of error occurred and where it happened. But they don’t provide you with any additional information, like the values of certain variables. This can make it very hard to reproduce the error in a test case.
How FusionReactor can help
FusionReactor can provide you more information about the error situation. If you want, you can even debug the issues on your live system when it occurs next time.
The only thing you need to do is to log into the web interface of the FusionReactor instance that monitors your application, select the exception from the “Error History” and go to the “Error Details” tab. There you can activate the debugger for this specific exception.
After you’ve done that, FusionReactor will send you an email when the exception occurs again and pause the thread for a configured amount of time. The email contains the stack trace and information about the variable context. As long as the thread is paused, you can also use FusionReactor’s Production Debugger to debug the error in a similar way as you would in your IDE without affecting any of the other threads of your application.
Our move from Confluence to mkdocs
For many years the FusionReactor product documentation has been ran on a confluence server. We maintained our own server for many years and currently use the cloud version, but its never really been ticking all the boxes for our product documentation.
For each major release of FusionReactor we have a separate confluence space on the confluence cloud server. This allows users to see the documentation for the each version (excluding minor / bug fix releases) that the customer is running to ensure the screenshots are accurate.
The problem with having many copies of the documentation, for different version of the product, is that users don’t realize that they are reading documentation for the wrong version. This is not something which can be blamed on confluence but how we have been managing this system in the past. Google returns the search results which are most popular and as our 6.2.8 release was the newest release for the longest time, there has been a trend of people finding and using the docs from 6.2.8. Now its difficult for new releases of the documentation to be returned higher in googles search results.
This can be shown by googling for site:docs.fusion-reactor.com crash protection
you get the following:
The image above shows that the actual page for FusionReactor Crash Protection hit doesn’t appear and the first FR 8 crash protection related page is the 7th search result and this is an 8.0.0 hit not 8.2.0. (results will vary from region to region). When google returns so many old versions before the newest FR version it causes customer confusion and impacts our support team.
Another major issue we had when updating documentation for a new release, was that we found it very difficult to update images. We had no idea if there was a screen shot of some UI component on a specific page, so we had to check every page we could find to update the screenshot based UI change.
We have been using markdown and mkdocs for some time in other areas of documentation (like the FR Cloud docs) so we knew that this worked but moving from confluence was not going to be automatic and we had not done this before.
First we needed to get the space content out of Confluence. You can do this by going to the confluence space, selecting “Space Settings”. Then under “Content Tools” there is an “Export” tab, which shows :
If you select “HTML” then “Normal Export” to get a HTML file per space page. Once you press “Export” the task will run and give you a zip file of the content to download.
We then converted all the html files to markdown
for i in *.html ; do echo "$i" && pandoc -f html -t markdown_strict -s $i -o $i.md ; done
We then use rename and mmv to rename the files to get them from filenames with a name like Weekly-Report_245548143.html.md to Weekly-Report.md.
Using following commands :
rename 's/.html.md$/.md/' * mmv '*_[0-9]*\.md' '#1\.md'
The following image shows how the html files were converted to markdown files. We can now delete the html files if we want.
The renaming of files on disk also needs to be reflected in the markdown content. This was relatively simply using replace all with the same regex as we used with the mmv and rename commands.
We then setup the mkdocs and followed this mkdocs-material setup guide and made some customisation to use the FusionReactor icons and social links.
We managed to quickly get the docs running inside mkdocs as shown above.
The next problem we have going forward, before mkdocs actually replaces the current docs, is to update the documentation and fix it.
On our confluence versions of the docs we had lots of copy and paste content which was updated in one place and not another and we had a lot of broken links and out of date images.
We will continue to repair and improve our mkdocs over the next few weeks and months and then change the DNS entry.
Benefits of mkdocs
Below is a list of the benefits for mkdocs (as we see it) compared to our confluence documentation system.
- Ability to run automated tools against the docs.
- Allows checking of dead links.
- Automate spell checking and readability tools.
- Easy to find duplicate content and use markdown-include
- Ability to find and update images in a simply way.
Signing, Notarizing and Stapling on macOS
The Gatekeeper system has protected macOS users against malicious software since its introduction in OS X 10.7.3 (Lion). This system assures users that software comes from a trusted source and doesn’t contain malicious content. But how does it work?
Introduction
The Mac software ecosystem has historically been fairly untroubled by malicious viruses and software. This was due in part to the comparatively small user base and partly because the system — which is based on Unix — is naturally partitioned into users and groups, making it difficult for malicious code to obtain administrative privileges.
But the platform is increasing in popularity. Apple’s desktop operating system accounted for approximately 10% of desktop market share in 2019. The same kernel — Apple XNU, a modified Mach microkernel — is also used by the company’s IOS devices: iPhone and Apple Watch. This increasing install base makes the platform an attractive target for malicious software.
Apple has two main strategies in place to protect its users. We’ll look at each stage of this protection regime in the next sections, but broadly they comprise:
- Code Signing: this ensures that code comes from a known, trusted source, and hasn’t been altered since it was signed.
- Notarization: the code is inspected by Apple to ensure it falls within its safety guidelines; the resulting receipt can be attached to the software permanently, in a process known as “stapling.”
Code Signing
Signing ensures the code belongs to us and can’t be changed after it’s signed. This is done by the codesign tool, which:
- Creates a secure hash of the code itself (this hash will change if the code is tampered-with after the fact)
- Signs the hash and the code with our Developer Certificate. This puts our name and details on the code. Apple has checked our credentials and has also signed our Developer Certificate to say that we are a valid, trusted developer.
- Stamps the resulting signature with the current time. This ensures that if our certificate expires, you can continue to use this software.
Here’s how we test-sign the FusionReactor macOS Native Library, which is used by the FusionReactor Production Debugger:
xcrun codesign --verbose --strict --keychain /Users/jhawksley/Library/Keychains/login.keychain -s CERT_ID_HERE --timestamp target/libfrjvmti_x64.dylib
Notarization and Stapling
The second stage is to have Apple actually check our code.
The Notarization command looks like this:
xcrun altool --notarize-app --username "our_apple_id@intergral.com" --password "our_password" --primary-bundle-id "com.intergral.bundleid" --file bundle.zip
Before we ship the library to Apple for Notarization, we have to sign it using codesign, and we have to zip it up to minimize the transfer size. The username and password are those of a valid Apple Developer Account.
Notarization is an automated service, which — while not providing usability or design feedback, like App Review — does checks code is correctly signed, and doesn’t contain malicious content.
The result of this is a Notarization Ticket, which is a piece of secure data that Apple sends back to us and also publishes online in the Gatekeeper Code Directory Catalog.
Some types of software — for instance the native library we showed in Code Signing above — don’t have space in their structure for a ticket, so they can’t be stapled. Other types of software, like the FusionReactor macOS Installer, do have space, and the Notarization Ticket obtained above can be stapled to them.
When you run our software on your machine, Gatekeeper automatically checks to see if the software is valid. If there’s a network connection available, Gatekeeper uses the online Code Directory to look up the ticket and checks it against the software. Should no network is available, Gatekeeper uses the stapled ticket.
If a valid ticket is located, Gatekeeper knows that Apple has checked this software and that it meets their standards — and can run.
Why Bother?
Apple has been gradually tightening up the conditions under which unsigned software can run. In macOS Catalina, this is still possible (you have to turn Gatekeeper off using Terminal commands) although in the future even that may no longer be possible.
When macOS tries to use or install unsigned content on macOS Catalina, you’ll see the following dialog (which can’t be bypassed) — and the content is not opened.
When content has been correctly signed, Gatekeeper tells you where it came from and lets you decide whether to open it. Here’s what we see when open our (signed, notarized, stapled) installer from our build server.
Trust, but Verify
If you want to check a signature, this is easy to do. Open the Terminal app, and use the codesign command to retrieve the certificate:
codesign -d --verbose=4 ~/Downloads/FusionReactor_macos_8_3_0-SNAPSHOT.dmg
This spits out the following (excerpted for clarity):
Identifier=FusionReactor_macos_8_3_0-SNAPSHOT
CodeDirectory v=20100 size=179 flags=0x0(none) hashes=1+2 location=embedded
CandidateCDHashFull sha256=f684fe6584f8249c3bfb60c188dd18c614adc29e6539490094947e1e09bbb6c8
Authority=Developer ID Application: Intergral Information Solutions GmbH (R3VQ6KXHEL)
Authority=Developer ID Certification Authority
Authority=Apple Root CA
Timestamp=Dec 5, 2019 at 14:52:15
The Identifier is the file name (minus extension) we signed, and the Identifier and CandidateCDHashFull values tell Gatekeeper how to perform an online lookup of our Notarization Ticket.
The three Authority lines are the chain of trust: they show that Apple has trusted us by checking our details and issued a certificate, whose serial number is R3VQ6KXHEL.
Finally, the Timestamp shows when we actually signed this software. If you try to use it in future, perhaps when our certificate has expired (hopefully there’ll be many new versions of FusionReactor before then!), the Timestamp assures Gatekeeper that the software was signed within the certificate validity period, and should be treated as if the certificate was still valid. Gatekeeper should then continue to open this software indefinitely.
Automation
It’s possible to automate the signing, notarization and stapling process, but it’s not exactly straightforward.
Apple’s development tool – the venerable Xcode – handles signing, notarization and stapling seamlessly as part of its user interface. Apple does provide command-line tools to perform these tasks (codesign, altool and stapler) but these all make some assumptions that the user running them is logged in.
The exact mechanics of automated signing of Apple binaries are rather beyond the scope of this article. However, we can give you some hints:
- A Jenkins node running on a Mac Mini is used. Apple doesn’t allow virtualization of its hardware, and Docker on Mac is a linux environment, so the node must be a bare-metal Apple environment. Mac Minis are excellent little machines for this.
- The user under which the Jenkins node is running:
- Must be logged in to the desktop. This creates an Aqua desktop session for the user — which is valid even if the Jenkins node is launched as that user using
ssh. The Aqua session is required to use the code signing tools. The login can be done on the Mac itself, or using Apple Screen Sharing (which is a customized VNC) but the session should be ended by closing the window, not logging the user out. - Must have the Apple Developer ID code signing certificate, along with the corresponding private key, installed into its Login Keychain (the default keychain).
- Must have the keychain unlocked prior to code signing using the command
security unlock-keychain
- Must be logged in to the desktop. This creates an Aqua desktop session for the user — which is valid even if the Jenkins node is launched as that user using
Another wrinkle in the automation of this procedure is that the notarization command (xcrun altool --notarize-app) is asynchronous. Once the code is submitted (which itself can take a couple of minutes), the Apple Notarization Service returns some XML containing a RequestUUID. You have to then poll the Service using this UUID until it returns either success or failure. This can take up to 10 minutes, in our experience. If you don’t parallelize this part of your build, it will be impose a long delay.
