Introducing FusionReactor’s Production Java Profiler


If you’re trying to root out application performance bottlenecks on your production JVM environment – then you are faced with some significant challenges.

Knowing where and what to tune up within your applications can be a major headache. APM tools can help and generally they provide metrics which will highlight slow running queries or poorly performing requests. However, what if you’re looking for performance issues within your actual code – you know it’s performing poorly – but you want to understand exactly how slow and then figure out why!

For performance issues, (unless you’re already using FusionReactor!) you may think of using a profiler, such as JProfiler or jvisualvm. However, it’s not easy to attach such tools to a specific process (especially in production) and it’s common knowledge that such tools impose a heavy performance burden themselves which essentially will render them useless in a production environment.

We now have an alternative – FusionReactor’s Production Java Profiler.

FusionReactor 6 Ultimate Edition now includes a Production Profiler for Java, which is designed to help you identify bottlenecks in your production environment. The Production Profiler is VERY low overhead, so you gain maximum benefit, without impacting your applications. The profiler may be run on demand, but is switched on by default, so that it will automatically profile your application as soon as it detects a long running request or transaction. You can also limit the number of profile samples which are taken, which will enable you to control the level of accuracy of the sample. Our tests have shown that even with limited samples, the insight you gain into performance bottlenecks is invaluable.

What can you profile?

Any requests or transactions which are running on the JVM – e.g. Tomcat, JBoss, WildFly, Glassfish, Jetty etc. ColdFusion applications can also be profiled– as well as any other applications which are currently being monitored by FusionReactor 6.1 or higher.

What do you have to do to start profiling?

Install FusionReactor 6.1 or above – activate it using either a TRIAL license, or your own FR Ultimate license – and that’s it. The Production Profiler is automatically configured to immediately profile any request or transaction which runs longer than 3 seconds. The default profiling sample is 200ms.

You can also prProfiler Iconsofile individual requests by simply clicking on the Profile now button – indicated by the Clock icon – which you can see next to any running request.

As requests are profiled, they will be listed in the Profiled requests or transactions screen – as in the example below.

Profiled Transactions WebRequest

Clicking on the Profiler link, will take you to the Profiler details page – which shows a breakdown– represented in time and percent of how long was spent in specific sections/areas of your application code. The profile taken is basically an inverted view of the call stack.

Profiler

Bottlenecks are generally easy to spot. In our opinion, optimizing every part of your code is not worth the effort. Efforts should be focused on code segments where returns are higher, or rather highest.

Features and Benefits

  • When FusionReactor Ultimate is running, then profiling will start automatically for any request/transaction which runs for longer than 3 seconds. Note that this threshold can be easily altered to suit your needs.
  • Profiling can also be started on demand.
  • Find the root cause of performance bottlenecks in your application by viewing the profiled call stack in a top to bottom view.
  • Performance is shown in terms of time spent within that method – shown in seconds. The time will also be shown as a percentage.
  • Profile anything that runs on the JVM – without having to make any Java command line changes or restarting any processes – everything is built into FusionReactor Ultimate Edition.

If you would like to evaluate FusionReactor Ultimate – please download a trial license here

Transaction Memory Values Explained


One of the metrics gathered by FusionReactor during the execution of a transaction (request) is the memory usage of the transaction. This is displayed on the Main tab in the transaction details page. Since FusionReactor 6 we have added the Transaction Allocated Memory value to these details. All of these numbers can be a little confusing so below we have explained what all the memory values mean.

Explaining FusionReactor Transaction/Request Memory Values

The first few numbers are quite straight forward. These are simply a snap shot of the JVM state at either the start of the transaction (for transactions that are still running), or at the end of the transaction (for transactions that are finished).

Used Memory

This is the amount of memory that is currently used by the JVM, shown as both a absolute byte value and as a percentage of the maximum memory.

Max Memory

This is the maximum amount of memory that the JVM is allowed to use. This can be defined using the -Xmx Java argument.

Total Free Memory

This is the actual amount of memory that is not used by the JVM, this includes memory that has been allocated but not used yet. i.e. this is max memory minus used memory.

Allocated Memory

This is the current amount of memory that has been allocated by the JVM.

Free Memory

This is the amount of memory that has not been allocated by JVM. i.e. max memory minus allocated memory.

Transaction Allocated Memory

This value needs some deeper explanation in order to avoid confusion. This number is the total number of bytes that have been allocated during the life time of the transaction. Every time an object is created a portion of the JVM memory is used to hold this object, of course in java the garbage collector (GC) will automatically free any memory that is no longer needed.

This means that you can the reuse the same memory space again. So if we run the bit of code below, we will allocate memory space the string 10 times. Resulting in the transaction allocated memory value being 10 times the size of the single string.

for(int i = 0; i < 10; i++)
{

String str = “this is a string”;

}

This would mean that even if you have a max heap of 8 gigabyte the transaction allocated memory value can exceed this value. This would indicate that JVM is allocating and then freeing lots of object. If we look at the image below you can see that a single transaction can allocate more than the maximum heap, as long as it does not do it all in one go.

Transaction Memory Vales 2

Announcing FusionReactor 6 – the wait is over !


After approximately 10 years of engineering, 100 meetings, 1,000 pizzas and 10,000 cups of coffee we’re now ready with FusionReactor 6 – it’s taken a while – but it’s worth it and FR6 is by far the best, fastest and most feature rich version of FusionReactor ever launched. Oh – and as everything seems to be the power of 10 – FusionReactor is also exactly 10 years old this month 🙂

What’s different…? Well, FR6 includes tons of new features and improvements, but one feature really stands out – which is the Production Debugger available in the new ULTIMATE Edition of FusionReactor. We’ve always believed that monitoring is really a means to an end. The end goal is about your company being able to deliver your service, such that your customers are satisfied and your business makes money. So your applications need to be free of defects and always ON. Unfortunately, all software has defects, but we believe that the measure of a great monitor, is the speed at which the monitor can highlight and direct you to the defect, so that you can actually get it fixed and restore service.

FusionReactor has always been about giving you accurate, real-time insight into what’s happening on your server/application right now and the Production Debugger available in the ULTIMATE Edition takes that goal to a whole new level.

How to get FusionReactor Ultimate Edition

FusionReactor ULTIMATE can be purchased separately and it’s also possible to upgrade from either Standard or Enterprise Editions. Please contact sales for more details.

Here are some of the highlights which FR6 has to offer:

  • Memory Usage metrics
  • Application Filter view
  • Application Session Tracking
  • Enhanced Garbage Collection Analysis
  • SQL Query Planner (Explain)
  • ColdFusion Extended Metrics visualized
  • On the fly code de-compilation
  • Improved Notification Center
  • Improved Java Frameworks – Spring, Struts
  • Support for Fusion SaaS (CLOUD) – note Fusion SaaS will be released separately to FR6

The whole team is very proud of what we’ve achieved with FusionReactor 6 and we very much hope that you enjoy using it and that it continues to be your first choice tool to monitor, detect issues and protect your applications and servers.

David Tattersall
CEO Intergral – makers of FusionReactor, FusionAnalytics and FusionDebug

FusionReactor 6 sneak peek – Re-writing the definition of what Application Performance Monitoring is all about!


APM tools have emerged as a category to help businesses address issues when applications break or systems are slow. Unfortunately, even though many of those tools measure metrics and provide some insight into what’s going on, they don’t do anything to minimize downtime and really accelerate time to fix. To be honest, traditional APM tools simply don’t do enough.
FusionReactor 6 takes a different approach – we not only provide the basics and core APM expectations that every user has for server monitoring, FusionReactor 6 introduces the unique and patented capability that allows you to do real-time Production Debugging, a game changer in accelerating the time needed to IDENTIFY and FIX issues – PLUS FR’s unique Crash Protection, which will alert you and guard your servers against crashes, so you can take action BEFORE systems go down – resulting in high application and server resilience, improved quality, less downtime and happier customers !
FusionReactor 6 is breaking the mold of traditional APM tools and will enable you keep your production systems online longer and with better results.

Here are some of the highlights which you can you expect with FR6:

  • Production Debugging / Alerting
  • Enhanced Crash Protection / Scripting
  • System Monitor – extensive metrics
  • Daily + Service Level Reports + weekly, monthly summaries
  • Enhanced Stack Trace Analysis (Trace Assist)
  • Memory Usage metrics
  • Application Filter view
  • Application Session Tracking
  • Enhanced Garbage Collection Analysis
  • SQL Query Planner (Explain)
  • ColdFusion Extended Metrics visualized
  • On the fly code de-compilation
  • Improved Notification Center
  • Improved Support for Java Frameworks
  • Support for Fusion SaaS (CLOUD) – note Fusion SaaS will be released separately to FR6

Production Debugging/Alerting

Did you ever hear someone say, “We can’t reproduce that problem, it ONLY ever happens in production” – well, with the introduction of FR6’s new Production Debugging (PD) capability, this will become a thing of the past.
Production Debug gives you all the control you need to simply and securely interact with your production environment as issues unfold. The insight it provides will revolutionize how you identify problems.   Unlike a typical debugger, you can setup conditional breakpoints and be alerted by email when they fire.   The email will contain a link to the debug session, as well as stack trace and variable details at the instant the break point fired.   It has NEVER been easier and faster to get to the root of issues.

Production Debug

The Production Debug feature will only be available in the new FusionReactor Ultimate Edition.

Key features include:

  • Interactive debugger within FusionReactor’s interface
  • Conditional breakpoint (BP) and break on exception handling – allowing full control of debug sessions in your production environment
  • Usual debug features e.g. alter variable, step into, out etc.
  • Email breakpoint notification – when a breakpoint fires, FusionReactor will hold the thread and send you a link to the debugger at that point. Restricting this to a single thread ensures that your production environment is not impacted by the Production Debugger
  • Time controlled handling mechanism, allowing you to easily control how your production system handles break points / exceptions / debug sessions
  • Exception Alert Email – including Stack trace + Stack context – will enable rapid identification of issues
  • Visual indication (in the FR banner) that a Production Debug trigger has occurred
    Bug Notification

System Monitor

Over the years, customers have often asked for other system related metrics which were usually only available in more traditional system monitoring tools. We have now extended FR6 to include a wide range of new metrics and system specific information. Access to this information is via the System Resources icon in the main FusionReactor interface.

The new metrics being tracked, include:

  • System Environment details
  • Java Environment details
  • Currently logged in users
  • Network Details
  • Memory Swap Information
  • System Console Access
  • Running processes page; showing:
    • CPU usage
    • Memory usage
    • File descriptors
    • Threads
  • Process details page; showing:
    • Memory details
    • Environment variables
    • Modules
    • Process arguments
  • Disk IO / Per disk
  • Network IO / Per network adaptor
  • Disk Usage & capacity – broken down per disk
  • CPU usage per core
  • CPU usage break down (nice/wait/system/user)

System Metrics

Another important feature is the ability to set disk space thresholds, so now FusionReactor can alert you when disk space reaches a defined limit.

New FusionReactor Reports

Daily Reports

Feedback to the original FR Daily Report was really positive and we wanted to build on that with our new Daily Report, which adds additional information and draws your attention to potential issues and/or improvements in performance and stability. The new layout includes both previous values, and targets for those values. We think that this will allow for a better overview of how your server is doing on a day to day basis.

We have also added some more statistics to the report including number of sessions and number of 404 errors. In addition to these metrics we have added a breakdown of the recorded outage, with estimated start and end dates and times, durations and the reason for the outage. We also now have a breakdown of all response codes for the reported period, with average runtime, count and its percentage out of all requests.

Along with the Service Level report, we are introducing a Weekly report, which will produce a summary of your server for the week, as well as a monthly report, which will produce a summary of the calendar month.

ColdFusion Metrics Page

FR5 already collected a number of specific ColdFusion metrics. Till now, these were stored in various log files and were only available visually via FusionAnalytics. FR6 has 2 new screens available which break down the CF metrics and display them in graphical form. Some of the metrics shown, include : Active Sessions, # of current sessions, Template Requests, Web Service requests, App Scope Size, hit counts (breakdown of hits per request type) etc.

ColdFusion Metrics

SQL – Query Plan (Explain)

Since SQL is declarative language, there are typically a large number of alternative ways to execute a given query. These alternatives can provide widely varying performance. JDBC is typically one of the main areas which we find at the root of application related performance issues.

FusionReactor 6 has the capability to attach an SQL Query Plan to the given JDBC transaction in the event that the SQL statement execution is longer than a given time. This is made available on the transaction details page. This feature will provide database users and administrators the capability to manually examine the plans produced by the optimizer to spot potential performance issues. FusionReactor will run the query plan immediately after the actual query is executed.

FusionReactor 6 Query Plan

Session Tracking and Visualization per Application

In the stateless HTTP web world, Session’s play an important role for maintaining client state. Sessions are generally used to store user data – which can quickly eat up memory if not kept in check. Sessions are tracked (and identified) using a session ID. FR6 is now tracking and visualizing sessions. All created, destroyed and active sessions are tracked per application.

Session Planning

All about FusionReactor’s Daily Reports



Something you will hear over and over when looking at the APM space, is reports. Every APM solution has reports, and FusionReactor is no different.

With the release of FusionReactor 5.0.0, we included the Daily Report, which is a summary report which is sent as an email to the configured address(s) at the given time.  This plugin is very simple to use and just like other features in FusionReactor will work without a restart.   The idea behind this report is that it can be used to provide users with a quick indication of server health without having to check the instance directly via the FR interface.

Background

A quick overview of what the daily report consists of before we get started in making it work for us.

You can see from the image that the daily report contains a break down of the up time, web request activity and the core resource usage. The report is intended to quickly put over the basic state of your server over the last 24 hours; and in my opinion does it quite well.

Uptime is measured as a percentage of that day that the server was online, effectively it is (1 day – downtime / 1day) * 100.

 

Daily Report Screenshot

Setup

I feel that the best thing about this feature is that there is very little to do, as is often the case when setting up FusionReactor. All you need to do to receive this report is to enter the mail server settings within the FusionReactor -> Settings page. I will not go through all the settings as for the most part they are fairly simple. There are also a few other things in this email settings which can be helpful to know. For any further help please refer to the docs here.

From Address

The “From Address” is actually not needed to send an email, and is usually just used as a method to identify the sender. What i like to do for the FusionReactor instances is to use this to indicate the instance/server that the email has come from. So for instance i have the current released version of FusionReactor running on the testfr server. I can then use the email address release@ testfr.com, so now i can apply filters to the email account that this mail is sent to, allowing to be have different folders in the mail client for the different instances/server.

Using this approach means i can just look at the new mails per folder to know it there is any missing.

Host Name

This field is quite often overlooked. As FusionReactor’s internal web server, and discovered host name, is usually hidden from the outside world. The links in the email will not work unless you are inside the correct network. This field allows you to specify the host name and port that should be used in the links in the email. This has no affect on the email server used or and functionality, it is simply a way to set the host name that this instance of FusionReactor is available on.

As you can see from this example you can also include port numbers.

Send Test Email

This last part of the form is quite useful, if not a little annoying in its implementation. Basically this little button will send a test mail using the entered settings, unfortunately you do need to save the settings prior to clicking this button. This hopefully will be fixed in FR 6.

If you have set up the email settings for your mail server correctly the you will receive an email report from FusionReactor at 00:05 (server time). If you want to change the time the report is sent or the recipients read on.

Advanced Settings

There are a few more settings that can be changed if you want to have the email sent at a different time, or have the report sent to recipients other than those in the default mail settings for the instance. These settings are available in plugin configuration screen, go to FusionReactor -> Plugins -> Active Plugins. Then click on the Configuration button for the FusionReactor Daily Report Plugin, there are not many plugins on this page so should be easy to find.

Firstly, send time. If you want the report to be sent at another time, simply change the “Report Time” field to the time that you want the report sent. Note: this time should be in 24hour format. You should also be aware that this time also indicates the time that the report takes as the end of the day. So if you change the time to be 12:00, the report will be for the time from 12:00 – 12:00.

Next we have recipients, this is fairly straight forward. A comma separated list of email address that the email should be sent to, these recipients are only used for this report. Also note that a blank field indicates that it will use the To Address in the FusionReactor -> Settings page.

No need for the Daily Report?

If you do not want to receive reports from FusionReactor then you can either remove the settings in FusionReactor -> Settings (which will prevent all emails from FusionReactor), or you can disable the plugin from the plugin configuration.

FR6 Reporting Sneak Peak

We received lots of positive feedback to the Daily Report in FusionReactor and we decided to further extend the reporting in the upcoming FusionReactor 6 release.  FR6 will include an improved new Daily Report, as well as a brand new Service Level Report.   Here is a sneak peak of some of the features you can expect.

  • Session activity
  • Improved source details
  • 404 count
  • Outage details Start – end times
  • Durations
  • Reason
  • HTTP response code break down
  • Trend indicators
  • Improved configurations
  • Daily AND weekly summaries !
Preview of FR6 Daily Reports

Getting custom instrumentation and metrics using FusionReactor: Examples with ColdFusion



FusionReactor will simply work out-of-the-box to give you masses of metric data and insight into requests, transactions, queries and resources without adding any additional code to your application.   However, if you want to get even more insight into specific areas of your application, then it’s easy to customize FusionReactor.   This level of customization will enable you to instrument specific parts of your code that FusionReactor (normally) does not track e.g. you can track method calls, web service calls – anything you like.   All this can be achieved by simply adding a few lines of code to instruct FusionReactor to track metrics, add properties to transactions or even create new child transactions around parts of your code.

In this post I will show you how to instrument your ColdFusion code to:

  • Create graphed series
  • Add properties to transactions
  • Create new Transactions

Creating a Graphed Series

The first step in creating a tracked series in FusionReactor is to create an instance of the FusionReactor API, playfully named FRAPI. This requires the use of the ColdFusion createObject call, see below.

Create FRAPI object
<cfset frapiObj = createObject("java", "com.intergral.fusionreactor.api.FRAPI")>
<cfset frapi = frapiObj.getInstance()>

From here all we need do is call the method on the frapi variable to post the metric, see below.

 

Post Float value
<cfset frapi.postNumericAggregateMetricFloat("/series/name", 10.0)>
 As ColdFusion can have some problems with distinguishing float and long primitive values there is also a method that takes a long, see below.
Post long value
 <cfset frapi.postNumericAggregateMetricLong("/series/name", 10)>
 Now both of these methods will take the value and post it to the given series name, in the event that that series does not exist it will be created automatically. These two methods will also create the standard numeric aggregates that are available in FusionReactor (1 Hour, 1 Day and 1 Week), if you do not want this much data from the series then you can use the alternative methods to post data without creating the aggregates.
Non Aggregated metrics
<cfset frapi.postNumericMetricLong("/series/name", 10)>
<cfset frapi.postNumericMetricFloat("/series/name", 10.0)>

The first call to post will create the series and aggregates, so if you first call is to postNumericAggregateMetric.. then the series will have the aggregates.  Similarly if you create the series using the postNumeric. then the series will not have the aggregates, even if a later post is to postNumericAggregate.

 

A Simple Example

Simulate Revenue being generated via sales and display the revenue in a graph within FusionReactor.

<cfloop from=1 to=120 index="j">
   <cfscript>
   frapi.postNumericAggregateMetric("/Revenue", RandRange(100,300));
   Thread.sleep(JavaCast("long",1000));
   </cfscript>
   <cfflush>
</cfloop>

Once you have posted the values to FRAPI you can see the graphs in FusionReactor on the Custom Metrics page. (Metrics -> Custom Metrics) and will be able to access the /Revenue values from the drop down menu.

FRAPI customization in FusionReactor

 

Add Properties to a Transaction

FusionReactor will automatically track lots of ColdFusion metrics, one of the most important is Web Requests. Along with tracking requests, FusionReactor is able to attach properties to requests so you can better diagnose problems later on. To attach a property to a transaction using ColdFusion see the code snippet below.

 

Adding properties to WebRequest
<cfset transaction = frapi.getActiveTransaction()>
<cfset transaction.setProperty("/property/name","StringValue")>

The setProperty method is effectively a map put, and is able to take any object as a value. However the FusionReactor UI is only able to display the value as a string, so it is a good idea to ensure the result of the toString method of the given object is a user friendly string.

It is even possible to use the FusionReactor as a kind of private session store, that is only available for the life time of that request. So similar as above, but by using the getActiveMasterTransaction you can always get the top level transaction on this request. The difference between getActiveMasterTransaction and getActiveTransaction is explained in the next section. With this top level transaction you can put and get properties from it just like a session, but this is only available for this request.

 

Get property from Transaction
<cfset transaction = frapi.getActiveMasterTransaction()>
<cfset prop = transaction.getProperty("/property/name")>

If you do put properties into the request it is important to remember that FusionReactor will keep these requests in memory for some time, depending on settings, so it is NOT ADVISABLE to put large objects into the request properties !

 

Creating new Transactions

As mentioned above, FusionReactor will automatically create many transactions to track WebRequests, JDBC requests, CFHTTP among many more.  Let’s imagine you have a function or section of code that you want to track the performance of.   Using FusionReactor you can wrap this section of code is a custom transaction to visualize the performance inside FusionReactor.   Doing this is very simple, image we have the below CFFunction code.

 

Example CFFunction
<cffunction name="newRandomNumber" access="public" returnType="string" output="false">
<cfset rndObj = createObject("java", "java.util.Random")>
<cfset rnd = rndObj.init()>
<cfset area = rnd.nextInt(99999)>
<cfreturn area>
</cffunction>
 So from this example if we want to track all calls to this function all we need to is add a create and end transaction call.
Instrumented CFFunction
<cffunction name="newRandomNumber" access="public" returnType="string" output="false">
  <cfset frapiObj = createObject("java","com.intergral.fusionreactor.api.FRAPI")>
  <cfset frapi = frapiObj.getInstance()>
  <cfset trans = frapi.createTransaction("newRandomNumber")>
  <cfset rndObj = createObject("java", "java.util.Random")>
  <cfset rnd = rndObj.init()>
  <cfset area = rnd.nextInt(99999)>
  <cfset trans.close()>
  <cfreturn area>
</cffunction>

This code will now create a transaction with the flavour “newRandomNumber”, which will be added as a child transaction to the web request and visible in the FusionReactor UI. Alternatively there is a createTrackedTransaction method that will do the same as the createTransaction but will also create a periodic sampler that will produce a activity and duration graph for the given transaction flavour.

Summary

These three features of FRAPI are a small select sample of the functions available, and offer some unique and powerful options to developers that are looking for ways to track performance and debug code. There are many other methods available via the FRAPI interface many of which are helpful for both tracking performance and debugging errors. For more information on the available methods please review the Java docs.

Keeping it real – excluding the effect of long running requests in FusionReactor




Something that comes up when speaking to FusionReactor customers is the need to handle long running requests.  In FusionReactor as with many APM solutions there is a continuous running aggregate of the average run-time of the web requests.

Unfortunately if you have a single request that has to run for a long time it can cause this number to become skewed; essentially the average run-time of requests can end up being longer than it really is just because of this one request e.g. often a maintenance job run by the system admins, which is not so important when you are interested in real user experience.

Scenario 1 – Excluding a single long running request which is skewing performance metrics

Details

Let us assume you have a running web application, in this instance running on a tomcat server.  Now let’s assume that you have a request that is run via this application as some routine job.   This job typically runs over a minute and has a negative affect on the average run-time of your server.    We would like this request to be ignored by FusionReactor and not included in the metrics calculated, but every other request should continue to be tracked.

Goal

The goal is to eliminate this request from the metrics, and get a better picture of what the overall performance of the server is.

Using this example URL for the job: /JTA/performance/jdbc/jdbc30secondrating.jsp

Steps

FusionReactor has this feature right out of the box, FR has the ability to restrict requests from almost every feature that it offers.  There are a few different ways to go about adding this restriction, the simplest (but not always possible solution) is to simply click the ‘Stop Tracking’ button, next to the request you want to ignore, on the Request -> Activity screen.   This has the disadvantage of requiring the request to be running before it can be removed from the statistics. It also will not affect any statistics that have already been calculated from this request. This approach will only affect that single request, and as such is of limited usefulness when applying it to a routine job.   I mention it here as it is useful when running a job manually that you do not want to be tracked.

The better approach is to simply enter the request details prior to it entering the system.

  1. Go to FusionReactor -> Restrictions
  2. Enter the details of the request that you want to ignore from metrics. (help with settings)

We have some options here, if the request always comes from the same host, we can also add the host name check to the restrictions, this means FusionReactor will only ignore the request if it comes from the matching host name.

Another approach, which is quite a neat feature, is to filter the request based on the query parameters. So if this job is in fact the same end point as some other request, but contains specific query parameters, then we can filter on those, and we can actually add an extra query parameter to the request when it is the job.

For example, if we have the endpoint /JTA/performance/report.jsp, but this request by itself is part of the usual use model of the application. We can actually alter the request to be /JTA/performance/report.jsp?__fr_ignore=true, this should not have any effect on your request, unless you happen to use the query parameter __fr_ignore; which seems unlikely.   So now by changing the “Request” field of the restriction form to include this parameter, and set the “Parameter” field to “Check”, you can make FusionReactor ignore this request when the parameter __fr_ignore=true is sent, and continue to track it when it is not.

I should point out at this point that if you use the “Exact Match” strategy then you will also have to include any other parameters that are part of the request.   Another alternative is to use the “Regular Expression” strategy, covered later.

So now that the restriction is configured, clicking save will add this restriction, visible below the form, in real time, and any new request that match this pattern will be ignored from FusionReactor.

long_req_blog_1

Outcome – we’re keeping it real 🙂

At this point FusionReactor will now be ignoring the request that you have added to the restrictions, and will no longer be included it in the metrics for the server.    This will hopefully improve the rating of the server among your peers as it will now seem that by magic the server is performing better.

 

Scenario 2 – Using regex to exclude multiple tasks affecting your server performance rating

Similar to the previous scenario, but this time we have a whole sub section of the application that is only used by the system administrators to affect changes on the database, or perform clean up tasks.

Details

Lets us imagine that we have a number of administrative tasks, such as account merge, account removal etc.  These are not performed that often, but never the less do take some time.   These requests can then affect the overall statistics of the server, making it seem busier or less efficient than it actually is.

Goal

The goal here is to identify these administrative tasks and remove them from the request tracking.   We can identify these requests as they have the string /system/admin/ in the URL.   This being the root of all the system administrator tasks that we want to ignore.    Once approach would be to enter all the URLs one by one in the same way we did above.  The disadvantage is that if new jobs would be entered, we would need to remember to include the individual URL.  A much better approach is to use the “Regular Expression” strategy to ignore all the requests by using a single rule.

Steps

The way to add this rule is basically the same as the single rule, we went through above. The only real difference here is the use of the “Regular Expression” strategy and the Request actually being the regex and not the URL as it was above.

  1. Again we go to FusionReactor -> Restrictions
  2. Now we set the “Match Strategy” to “Regular Expression”
  3. Enter the regex .*/system/admin/.* into the “Request” field
  4. now simply save the form

You will immediately see the restriction enter the list below the form, and FusionReactor will immediately start ignoring any new request that contain the pattern /system/admin/.

Outcome

We have now, in these very simple steps, removed all system administrator request from being tracked into the FusionReactor statistics for your application.   You will now get a much clearer view of how your application is really performing.

long_req_blog_2

Additional

Regex, or Regular Expressions, are a very powerful tool for any language. However just like many, many things in programming the implementation of regex can vary from language to language.  FusionReactor uses the Java regex engine.    Plenty of documentation on the Java regex exists, here are a few links: FR quick list, Oracle Docs, online Java regex evaluator

How to keep your application / server responding when you’re running out of memory – FusionReactor Survival Strategy



FusionReactor has many great features one of the most overlooked and possibly one of the most powerful is EnGuard Crash Protection (CP).   This feature is able to alert you as well as prevent your servers from falling over by limiting the load based on a variety of criteria.

In this post I will walk you through how to setup the various settings for CP and the benefits of using these features.

Scenario – Application Server is running low on memory

Let us imagine the situation where, for whatever reason, the application server that you are running on is consistently running out of memory. By using FusionReactor’s Crash Protection features we can be alerted to the situation and take steps to keep the system / application operational.   Of course you will be investigating the cause of the memory issue, but in the mean time FusionReactor can buy you the time to ensure that service levels are maintained and you have minimized impact to your customers/business.   By using the FusionReactor’s “WebRequest Memory Protection” feature, you can limit the traffic on your server by either queuing or rejecting requests based on the current JVM memory usage.   FusionReactor can also invoke system garbage collections if we exceed the threshold values, to help reduce the over all memory usage of the application.

Goal

In this situation the goal is to reduce the concurrent requests running on the application when the memory usage exceeds a specific value. What we want to do is make FusionReactor queue requests when the memory usage exceeds 80% for more that 15 seconds.

Steps

Our first action here is to enable the Crash Protection features. This does not require a restart and will take effect immediately.

  • Go to Protection -> Settings
  • Click on “WebRequest Memory Protection”
    1. Change the “WebRequest Memory Protection Threshold” to 80%
    2. Change the “Minimum condition Duration” to 15
    3. Change the “Memory Protection Strategy” to “Queue and Email Notification” or “Queue”
    4. Enable “Automatic Garbage Collection Trigger”
  • Click “Save FusionReactor Settings”
  • CP Blog

If you are using the “Queue and Email Notifications” you will need to enable email notifications under Protection -> Settings -> Email Settings, you will also need valid Email settings in the FusionReactor -> Settings -> Email Settings page.

Results

When the system is setup correctly you will begin to see FusionReactor start queuing requests as they come in. These requests will be queued for 60 seconds, (configurable in the protection settings), before they timeout.    This request queue is nicely displayed in the Request -> Activity screen, clearly showing the time in the queue and the time until the request is rejected.    The request will be served as normal if the memory usage falls below 80% before the request times out.

The requests that are rejected based on the queue timeout, are tracked into a separate screen, Requests -> Rejections, and a summary is available on the WebMetrics page. Additionally the garbage collection metrics are tracked on the Resources -> Garbage Collection page (from FusionReactor 5 and later)

So hopefully at this stage you will have saved your system from collapsing from an out of memory error, kept the server running and bought yourself more time to diagnose the initial cause of the memory issues on the system.     FusionReactor saves the day (AGAIN !).

Advanced Option

In addition to the setting described above the following options are available to improve the effectiveness of FusionReactor Crash Protection.

Garbage Collection

When using the memory protection settings feature, you might have noticed the “Automatic Garbage Collection Trigger”. This feature is a very useful tool if you are experiencing memory issues on your application server. By using this setting you can configure FusionReactor to perform a garbage collection (GC) when the request starts.  As you may be aware performing a garbage collection can be an expensive operation, as such, FusionReactor comes with a rate limiting option. The “Minimum Garbage Collection Interval” setting allows you to set the minimum time that must elapse before FusionReactor will invoke another garbage collection.

FusionReactor v5 Java Server Monitor – A feature tour


A 20 minute feature tour of FusionReactor v5 visiting:

  • Enterprise Dashboard
  • Web Metrics
  • Real-Time Running Request List
  • Request History
  • Request Details
  • CPU Thread Time
  • TCP/IP Stream Metrics
  • HTTP Headers
  • JDBC Transactions
  • HTTPClient/WebService Transactions
  • Longest Requests (since server started)
  • Slowest Requests (recent pages longer than threshold)
  • HTTP 500 Server Error Details
  • User Experience Monitoring (UEM) (e.g. client network & browser rendering time)
  • Custom Transactions (e.g. tracking custom API calls)
  • Custom Metrics (tracking custom business metrics – e.g. revenue)
  • Custom Dashboard (e.g. revenue vs page execution time chart)
  • Metric History (e.g. CPU last hour/day/week; pan/zoom; etc)
  • Both Heap & Non-Heap Memory Spaces (e.g. PermGen)
  • Garbage Collection Timing
  • Thread State Charts (e.g. identifying locking contention)
  • EnGuard Crash Protection (e.g. alerts before failure or prevents server from failing)
  • Daily Report Email (e.g. managers don’t need to login for overview metrics)
  • Log rotation & archiving (including JVM logs)
  • Info/Warn/Error Notification system
  • Desktop, iOS, Android dashboard application
  • Stack trace running request (e.g. see what line of code is currently being executed / where a slow page is stuck)