Keeping it real – excluding the effect of long running requests in FusionReactor

How to keep your application / server responding when you’re running out of memory – FusionReactor Survival Strategy
Getting custom instrumentation and metrics using FusionReactor: Examples with ColdFusion



Something that comes up when speaking to FusionReactor customers is the need to handle long running requests.  In FusionReactor as with many APM solutions there is a continuous running aggregate of the average run-time of the web requests.

Unfortunately if you have a single request that has to run for a long time it can cause this number to become skewed; essentially the average run-time of requests can end up being longer than it really is just because of this one request e.g. often a maintenance job run by the system admins, which is not so important when you are interested in real user experience.

Scenario 1 – Excluding a single long running request which is skewing performance metrics

Details

Let us assume you have a running web application, in this instance running on a tomcat server.  Now let’s assume that you have a request that is run via this application as some routine job.   This job typically runs over a minute and has a negative affect on the average run-time of your server.    We would like this request to be ignored by FusionReactor and not included in the metrics calculated, but every other request should continue to be tracked.

Goal

The goal is to eliminate this request from the metrics, and get a better picture of what the overall performance of the server is.

Using this example URL for the job: /JTA/performance/jdbc/jdbc30secondrating.jsp

Steps

FusionReactor has this feature right out of the box, FR has the ability to restrict requests from almost every feature that it offers.  There are a few different ways to go about adding this restriction, the simplest (but not always possible solution) is to simply click the ‘Stop Tracking’ button, next to the request you want to ignore, on the Request -> Activity screen.   This has the disadvantage of requiring the request to be running before it can be removed from the statistics. It also will not affect any statistics that have already been calculated from this request. This approach will only affect that single request, and as such is of limited usefulness when applying it to a routine job.   I mention it here as it is useful when running a job manually that you do not want to be tracked.

The better approach is to simply enter the request details prior to it entering the system.

  1. Go to FusionReactor -> Restrictions
  2. Enter the details of the request that you want to ignore from metrics. (help with settings)

We have some options here, if the request always comes from the same host, we can also add the host name check to the restrictions, this means FusionReactor will only ignore the request if it comes from the matching host name.

Another approach, which is quite a neat feature, is to filter the request based on the query parameters. So if this job is in fact the same end point as some other request, but contains specific query parameters, then we can filter on those, and we can actually add an extra query parameter to the request when it is the job.

For example, if we have the endpoint /JTA/performance/report.jsp, but this request by itself is part of the usual use model of the application. We can actually alter the request to be /JTA/performance/report.jsp?__fr_ignore=true, this should not have any effect on your request, unless you happen to use the query parameter __fr_ignore; which seems unlikely.   So now by changing the “Request” field of the restriction form to include this parameter, and set the “Parameter” field to “Check”, you can make FusionReactor ignore this request when the parameter __fr_ignore=true is sent, and continue to track it when it is not.

I should point out at this point that if you use the “Exact Match” strategy then you will also have to include any other parameters that are part of the request.   Another alternative is to use the “Regular Expression” strategy, covered later.

So now that the restriction is configured, clicking save will add this restriction, visible below the form, in real time, and any new request that match this pattern will be ignored from FusionReactor.

Keeping it real – excluding the effect of long running requests in FusionReactor, FusionReactor

Outcome – we’re keeping it real 🙂

At this point FusionReactor will now be ignoring the request that you have added to the restrictions, and will no longer be included it in the metrics for the server.    This will hopefully improve the rating of the server among your peers as it will now seem that by magic the server is performing better.

 

Scenario 2 – Using regex to exclude multiple tasks affecting your server performance rating

Similar to the previous scenario, but this time we have a whole sub section of the application that is only used by the system administrators to affect changes on the database, or perform clean up tasks.

Details

Lets us imagine that we have a number of administrative tasks, such as account merge, account removal etc.  These are not performed that often, but never the less do take some time.   These requests can then affect the overall statistics of the server, making it seem busier or less efficient than it actually is.

Goal

The goal here is to identify these administrative tasks and remove them from the request tracking.   We can identify these requests as they have the string /system/admin/ in the URL.   This being the root of all the system administrator tasks that we want to ignore.    Once approach would be to enter all the URLs one by one in the same way we did above.  The disadvantage is that if new jobs would be entered, we would need to remember to include the individual URL.  A much better approach is to use the “Regular Expression” strategy to ignore all the requests by using a single rule.

Steps

The way to add this rule is basically the same as the single rule, we went through above. The only real difference here is the use of the “Regular Expression” strategy and the Request actually being the regex and not the URL as it was above.

  1. Again we go to FusionReactor -> Restrictions
  2. Now we set the “Match Strategy” to “Regular Expression”
  3. Enter the regex .*/system/admin/.* into the “Request” field
  4. now simply save the form

You will immediately see the restriction enter the list below the form, and FusionReactor will immediately start ignoring any new request that contain the pattern /system/admin/.

Outcome

We have now, in these very simple steps, removed all system administrator request from being tracked into the FusionReactor statistics for your application.   You will now get a much clearer view of how your application is really performing.

Keeping it real – excluding the effect of long running requests in FusionReactor, FusionReactor

Additional

Regex, or Regular Expressions, are a very powerful tool for any language. However just like many, many things in programming the implementation of regex can vary from language to language.  FusionReactor uses the Java regex engine.    Plenty of documentation on the Java regex exists, here are a few links: FR quick list, Oracle Docs, online Java regex evaluator