FusionReactor Blog - News and expert analysis for Java APM users.

“Fantastic insight!”

What do you like best?

FusionReactor provides amazing insight into the server health, specifically in regards to the way ColdFusion is operating. The reporting and visualization into the server are fantastic!

What do you dislike?

Really not much I can say here. We rarely have issues with FR as an application, but if we do the support group helps resolve it quickly.

Recommendations to others considering the product:

Where you’re finding issues with code and resources that aren’t easily identifiable or prompt resolution is critical, FusionReactor can really make a significant difference in you debugging and daily operations for overall application management!

What problems are you solving with the product? What benefits have you realized?

We can zero in on issues MUCH faster with FR and solve problems in a fraction of the time. When we have a lot of programmers providing code sometimes the most expensive part in debugging is finding the issue – with FR we can really hone in on the issue MUCH FASTER. The reporting and tools are second to none.

See review on G2

Start free trial

In FusionReactor 8.1 we introduced the Ephemeral Data Service (EDS). This service allows you to register your transient / short-lived instances to the Enterprise Dashboard so they can all be monitored from a single location.

Using EDS, you see each instance in your dashboard and have the ability to group these instances together to quickly see the health of either single instance or group of instances.

Specifically, with EDS you also gain access to secure tunnelling, this means when your instance spins up and registered with your Enterprise dashboard, a secured tunnel is created between the dashboard.

This allows data to flow between the instance and the dashboard to power metrics, but more importantly, it allows the Dashboard to view the user interface of the connected instance via a proxy.

Using the proxy means you can access the user interface of the running instance without having to expose a port to access FusionReactor, meaning your application can be locked down and secure without limiting access to your APM data.

How to Configure the EDS?

Below is a full guide for running a dashboard and application instances via Docker.

Configuring the EDS requires you to set a few java arguments on both the instance running the Enterprise dashboard and the application instances.

We will be using Docker and Docker-compose to create our ephemeral instances to run the EDS setup.

Java Arguments

Passing Java arguments into your JVM can be done in a number of different ways and changes depending on the application server you are running. For a list of all the common locations see this doc.

The full list of available arguments for the EDS can be found here.

For our example we will be using only the required arguments:

• -Dfr.ed.ds.enable=true – This argument allows the dashboard to use EDS connections
• -Xmx512m – This argument increases the memory of the dashboard instance, as the secured tunnels and metrics are held in memory
• -Dfr.ed.ds.target=fram:2106 – This argument tells the application instance which dashboard to register to

We will also be using the following arguments listed here to configure our instance settings automatically. These arguments are optional:

• -Dfrlicense=X – This argument sets the license key used by the instance
• -Dfradminpassword – This argument sets the administrator password for the instance

Docker Images

In our example, we created 2 simple docker images and a docker-compose file to run our EDS setup. All the files used in our example is available in our GitHub repository

Our FRAM instance uses an OpenJDK base image and the FusionReactor NoJRE installer. We are using a FRAM instance as it does not use an additional server seat so we do not need to pay for an extra license to run it.

Our Dockerfile contains the following:

FROM openjdk:12

ADD https://intergral-dl.s3.amazonaws.com/FR/Latest/FusionReactor_unix_nojre.tar.gz /opt

RUN tar -xvf /opt/FusionReactor_unix_nojre.tar.gz -C /opt

ADD start.sh /opt/fusionreactor/start.sh

EXPOSE 8087

WORKDIR /opt/fusionreactor/

CMD /opt/fusionreactor/start.sh

This docker image downloads the latest NoJRE installer, adds our custom start script with the required Java arguments and exposes the FRAM port 8087.

In the injected start.sh file we have:

#!/bin/bash

dir=`pwd`
LICENSE=${LICENSE:='XXXX-XXXX-XXXX-XXXX-XXXX'}
PASS=${PASS:='admin'}
java -javaagent:"$dir/instance/FRAM/fusionreactor.jar"=name=FRAM,address=:8087 -Dfrnopointcuts -Dfrstartupdelay=0 -Djava.awt.headless=true -Xmx512m -Dfr.ed.ds.enable=true -Dfrlicense=${LICENSE} -Dfradminpassword=${PASS} -cp "$dir/instance/FRAM/fusionreactor.jar" com.intergral.fusionreactor.agent.service.Service

In this bash file, we set the instance password and license key based on the environment variables LICENSE and PASS, and have our EDS server arguments in place.

Our Tomcat instance uses the tomcat base image. The Dockerfile contains:

FROM tomcat

RUN mkdir -p /opt/fusionreactor/instance/tomcat

ADD https://intergral-dl.s3.amazonaws.com/FR/Latest/fusionreactor.jar /opt/fusionreactor/instance/tomcat

ADD https://s3-us-west-1.amazonaws.com/intergral-dl/FR/Latest/libfrjvmti_x64.so /opt/fusionreactor/instance/tomcat

ADD runTomcat.sh /opt

RUN chmod =x /opt/runTomcat.sh

CMD /opt/runTomcat.sh

This file takes the base tomcat image, downloads the 2 FusionReactor files and adds a custom tomcat start script that allows us to set dynamic java arguments. The installation of FusionReactor is based off a manual instllation.

The tomcat start script looks like:

#!/bin/bash

LICENSE=${LICENSE:='XXXX-XXXX-XXXX-XXXX-XXXX'}
PASS=${PASS:='admin'}
TARGET=${TARGET:='fram'}
GROUPS=${GROUPS:='group1'}
export JAVA_OPTS="$JAVA_OPTS -javaagent:/opt/fusionreactor/instance/tomcat/fusionreactor.jar=address=8088,name=tomcat -agentpath:/opt/fusionreactor/instance/tomcat/libfrjvmti_x64.so -Dfrlicense=${LICENSE} -Dfr.ed.ds.target=${TARGET}:2106 -Dfradminpassword=${PASS} -Dfr.ed.ds.groups=${GROUPS}"
/usr/local/tomcat/bin/catalina.sh run

This file passes in the FusionReactor start arguments, the license key, admin password and the EDS target and groups are based off the environment variables LICENSE, PASS, TARGET and GROUPS.

Running the EDS setup

To run the EDS setup, we are using docker-compose. This allows us to configure services with environment variables and exposed ports, without needing to add them to our docker run commands each time.

Our docker-compose yml file is:

version: '3'
services:
    fram:
      image: fram
      container_name: fram
      ports:
        - "8187:8087"
      environment:
        - LICENSE=YOUR_LICENSE_HERE
        - PASS=admin1
    tomcat:
      image: edtomcat
      depends_on:
        - fram
      environment:
        - LICENSE=YOUR_LICENSE_HERE
        - PASS=admin2
        - TARGET=fram
        - GROUPS=production,tomcats

In this file we have our license key, passwords, target and groups specified and a dependency ensuring the FRAM dashboard is running before the tomcat instances deploy.

To run our EDS setup we can run docker-compose up –scale tomcat=X to run as many copies of the application as we require. Provided the memory option of -Xmx=512M has been set in the FRAM instance, it is possible for the dashboard to monitor around 10,000 instances.

Features of the EDS Enterprise Dashboard

With the EDS dashboard running, as instances start, stop or restart they will be removed and re-added to the dashboard as soon as they are running. This means as your instances scale up and down you can always get an accurate picture of the health of your infrastructure.

Group and Instance Health

The grouped cubes colour will change as your group health transitions from Error > Warning > OK.

The colour of the instance cube will indicate the health of the instance, you see the used heap memory, used CPU, number of active requests and number of active database calls with a quick glance.

The cubes will transition between OK, warning and error states based on configurable limits.

For the instance you can see an overview of the web metrics, without having to navigate to the instance to view the data.

Instance User Interface with Proxy

For each instance there is a link that will use the tunnel proxy to give you access to the instances user interface, without any ports of the application being exposed to the internet.

To access the instance through the tunnel, you can click the Arrow icon on the instance.

Using the tunnel applications could be completely locked down behind a firewall, load balancer or AWS security group and you can still access the FusionReactor data to diagnose any issues you are facing.

Since FusionReactor 8.2, users with Enterprise licenses (or better) can use Microsoft Active Directory to log in to FusionReactor. In this blog post, we’ll covere how to set this up.

Introduction

In FusionReactor 8.2, we changed the way we handle logging you in and out behind the scenes. We still have the Role/Password system, and it’s still the basis of the authentication system, but we’re now able to provide other ways to authenticate you.

The first new authentication scheme we added in 8.2 is the Lightweight Directory Access Protocol, or LDAP. LDAP is used by large organizations to enable their users to log in to different systems, without having to set these users up on each and every system. There are several Directory Service products that are LDAP-enabled – and can therefore be used by FusionReactor – and Microsoft’s Active Directory system is an important and popular one.

Setting up Active Directory itself is a long and quite complicated process, and isn’t covered by this post. We’ll assume you have a working Active Directory environment already.

Prerequisites and Required Information

Here’s what you’ll need to have before you start configuration FusionReactor. Your Active Directory administrator will be able to supply this information.

Prerequisites

• Your Active Directory server must have the features Lightweight Directory Services, AD DS Tools and AD DS and LDS Tools installed.
• If your Active Directory server is firewalled, the LDAP port (usually 389) must be accessible to your FusionReactor instance.

Required Information

You’ll need to find out:

• Whether your server uses/requires TLS encryption.
• The IP address and port of your Active Directory server.
• The X.500 Search Base – this what LDAP uses to limit queries against Active Directory.
  You can find this by running the following command in Command Prompt on your Active Directory server:
  dsquery * -scope base
• The X.500 Distinguished Name (DN) address of a user able to read the directory. FusionReactor will connect as this user to use the directory.
  You can use the following command in Command Prompt to list all users:
  dsquery user
• … and you’ll need the password of this user.

You’ll additionally need to have a strategy for mapping your Active Directory users to FusionReactor roles: Administrator, Manager, and Observer. Any roles you don’t have mappings for will be simply disabled – users won’t be able to log in to these roles.

One common strategy is to set up a new Active Directory User Group for each of the FusionReactor roles. You’ll need the Distinguished Name (DN) of the groups you’re going to use for these users. You can get this information in Command Prompt using the command dsquery group.

There are some additional prerequisites and rules for mapping users, and these are detailed in our User Manual.

Here are the details we’re using for our example:

• No TLS encryption.
• IP: 1.2.3.4 / 389
• Search base: DC=testdomain,DC=com
• User: CN=FusionReactor LDAP Reader,CN=Users,DC=testdomain,DC=com
• Password: fusion4u
• Active Directory group for FusionReactor Administrators: fradmins.

Now we can go ahead and configure FusionReactor.

Configuration FusionReactor

Navigate to FusionReactor > Manage Logins > FusionReactor X.500/LDAP Login to access the LDAP configuration section.

UI Elements

Many user interface elements can be configured for this login mechanism. We’ve set up some example data so you can see how it maps to the actual login screen. Here’s the configuration:

… and here’s the rendered login screen:

Active Directory Server Details

Here’s the configuration for the remaining details:

The Test Login button can be used to check these details are right. FusionReactor will try to log in to Active Directory using these details.

Now we’ll enter the filter for the Administrator user. Don’t worry about the syntax yet, we’ll cover that next. Any users matched by this filter in Active Directory will be able to log in as FusionReactor Administrator.

The Validate Filters button checks the syntax of the filters.

Finally we hit Save Login Settings to save this form.

To make LDAP the preferred login provider, go to FusionReactor > Manage Logins > Global Settings and set FusionReactor X.500/LDAP Login as the Preferred Identity Provider.

Now let’s have a look at the filter.

Crafting LDAP Filters

The filter we used to identify Administrators is as follows (we’ve formatted it so it’s easier to read).

(&
   (objectClass=user)
   (sAMAccountName={{USERID}})
   (memberof=CN=fradmins,CN=Users,DC=testdomain,DC=com)
)

LDAP filters are very powerful (more information). Here’s a breakdown of this one:

• The ampersand means that all the conditions in this block must be true to match a record in the directory.
• objectClass=user ensures that the filter only matches users. Active Directory can also store information about computers, printers and other resources. We are only interested in users.
• sAMAccountName={{USERID}} means that whatever the user provides in the upper login input box (which we labelled Active Directory Username earlier) will be used to select an Active Directory record with that value as the Windows Logon ID (what Active Directory calls the SAM Account Name).
• memberof=CN=fradmins,CN=Users,DC=testdomain,DC=com means that the user must be part of an Active Directory group called fradmins.

Conclusion

The ability to log in to any LDAP-enabled service opens up FusionReactor authentication to a whole new realm of Enterprise possibility. Active Directory is perhaps the most popular service for handling thousands of users, and FusionReactor’s LDAP interface allows these users to access FusionReactor without having to share a role password.

“Best root cause analysis Java tool”

What do you like best?

The installation is simple – I typically go with the manual JAR install because it’s super simple… drop a JAR on the filesystem and add a JVM argument.

Once running, the UI is easy to navigate – but not overly simplistic. You can drill down from high-level overview charts down to profiling stack-traces within a few clicks.

Easily the best tool I’ve used for root causes analysis on the JVM – and one of the cheapest too.

What do you dislike?

FusionReactor can do long-term trending but it’s not it’s strongest feature. In my experience, FusionReactor is much better at solving acute problems – sometime automatically through its crash protection features.

The cloud offering is improving all the time and does have better long-term trending (e.g. capacity planning)

Recommendations to others considering the product:

Install and give it a go – there’s a 14day free trial.

Automated installer is great if you have a GUI. If you’re in a container or CLI only, manual install is super-simple.

Try the profiling features!

What problems are you solving with the product? What benefits have you realized?

Root cause analysis (in production) of acute failures. When there’s a problem in production sometimes it’s not so simple to replicate – perhaps your data tier doesn’t have the same volume or differentiating set of data; or perhaps the network topology, redundancy or traffic levels are different. FusionReactor adds no noticable overhead and gives a wealth of insight – it’s always running and ready for us if we have a problem.

See review on G2

Start free trial

These days the development of software is a rapidly changing process. The introduction of cloud hosting and containerization has made big applications running in your server room a thing of the past.

Where before you would maintain one or two big projects with a team of engineers dedicated to a single project. Now you have hundreds of small services split into bite-sized chunks with developers working on several projects.

In the perfect world, every software project will have a full suite of unit, integration and system tests in place. Even in these rare cases, it is still nearly impossible to avoid an escalation or critical performance issue when you are running at scale in production.

In my role as a technical support engineer, I have seen all manner of escalations. 

They can come in the form of;

•  A performance bug in the product
• A corner case issue you wouldn’t consider a user hitting
• A service being offline that the product depends on
• A third party library the product depends on being upgraded
• Many more!

From a support engineers perspective, here are a few tips I have learned over the years on how to approach an escalation.

Tip 1: Reassure the customer

The initial customer contact in support is crucial. Letting the customer know their issue is under investigation reassures the customer and takes the pressure off the support engineer.

The initial response from support can often be automated, these days using a bot for first-line support is often standard practice.

A simple example of an email would be:

 “Dear {{ticket.requester.name}},

Thank you for reaching out to the {{team.name}}, a ticket has been created for you with the Ticket ID – {{ticket.id}}.

A support engineer will get back to you as soon as possible and work with you to resolve your issue.

Thank you for your patience.

Sincerely,

{{engineer.name}}”

This simple email lets the customer know their issue is assigned to an engineer and the team is willing to help. You could enhance this email by adding elements such as:

• Common issues / solutions section
• A personal sign off specific to one engineer
• A phone number or chat to use if the situation is critical
• A link to a support portal so they can see their ticket status

If the support case comes in via phone or chat, this is not so much of an issue. The customer is getting live responses from the support engineer.

Tip 2: Gather all the information you need

Investigating an issue can be simple, unfortunately this is often not the case. If the product you support is complicated, escalations could be caused by a wide variety of factors.

The goal of the support engineer is to find the route cause of the issue quickly, with as little time to set up a reproduction case as possible. 

When the customer sees an issue, getting their logs, screenshots and environment data will often show the root cause of the problem. You can get this information through email or chat, but a call is usually the quickest way to access this information.

Having a template email or call script to get this information in place will ensure a new support engineer can gather all the essential data. They can use this data to consult with the other engineers if they need assistance.

With this information, an engineer will be able to report the issue without needing to reproduce the issue themselves, drastically reducing the time taken to resolve the issue.

Tip 3: Where possible process the issue into an email

When you have all the information, you may need from the customer; It is often best to halt direct communication when you are investigating the issue.

If you are on a call or chat with the customer, filtering through log files and attempting to reproduce the issue can be a demanding process. Often it is better to convert the call or chat with the customer into an email. The customer will typically understand if you tell them you will get back to them when you know more about the issue.

Having to only focus on finding the issue removes any distractions. Even if finding the issue is your sole task, it may take hours or in extreme cases days to find the exact cause.

Tip 4: When reporting the issue, be clear on its priority

So now you have your issue, this process could have taken you five minutes or five days. At this point, the development team must be notified.

How the team is notified will change based on the process within the engineering team. Often the two conventional approaches are to raise an alert in a chat room or create an engineering ticket.

For a severe/blocking issue such as a service being unavailable, raising an alert in the office chat will alert the engineers, and they can restart the services as appropriate. It may be they already know there is an issue and are investigating.

For an issue in the product, like missing data or a page in the UI erroring; A ticket to fix the product must be created. The priority of an issue is often not something you as a support engineer can control, but you can advise.

The priority is based on several factors, such as:

• How many customers it affects
• What it prevents a user from doing
• If the issue can be worked around
• Who reported the escalation

For example;

• If the issue has been reported into support five times in the last 24 hours, this is something hitting a lot of customers. Not fixing the issue right away will cause the number of reports for the problem to build up in support.
• If the issue prevents the user from logging in, or from seeing any data in the software, this has a higher priority than if it prevents them from clicking a link to the documentation.
• If a user has to uninstall your product to continue their task, this has a higher impact. If they have to refresh every 6 hours to get the UI working, it should be fixed, but it can wait.
• If the issue is reported by a customer who generates 60% of the revenue for your company, it must be treated as a higher priority than a customer who is on a free trial.

You should consider these factors when creating a ticket and your rationale for your suggested priority should be documented on the ticket. This will give the engineers some context for when they will start work on the issue.

Tip 5: Keeping the customer up to date on the issue

As soon as the issue has been reported, it becomes something you as a support engineer has no control over. Until the developers resolve the issue, all you can do is keep the customer updated on changes.

It is essential to let the customer know the issue has been reported. If you have a ticket ID or public issue board providing them with the link to this means they can check the status themselves whenever they need.

Promising hard deadlines on when the issue will be resolved can be risky. If an issue is more complicated than it initially appears the fix may be delayed which could potentially annoy the customer. Being vague on a fix date is often the best solution, using “as soon as possible”, “in the next release” or “we hope to fix the issue within the week” are good alternatives to utilise.

Having a specific inbox or state for a ticket waiting on the development team gives you an easy 

If it has been a while since the ticket was created, it is often beneficial to let the customer know they have not been forgotten.

When the developers have resolved the issue, you can finally notify the customer and resolve the ticket. This entire process could take less than an hour, or it could take months. However long it takes you to get the satisfying feeling of making the customer happy.

“The best tool for Diagnosting issues with CF/Tomcat”

What do you like best?

Lets you dig into running apps and stack traces and find causes for slow running pages and hangs and kill stuck threads.

What do you dislike?

Can be very daunting and complex to get to grips with.

What problems are you solving with the product? What benefits have you realized?

We used FusionReactor with ColdFusion and Lucee application servers.

Read review on G2

Start free trial

Just as useful as a development tool as it is for production debugging

What do you like best?

FR is very polished with tons of data. Automatic JDBC wrapping takes zero setup and nearly every screen or dashboard is customizable. I use FR to debug queries that have run, how long they took, and even to fish out long running requests. I always use it when performance testing, and it has excellent reporting on the most hit pages and DB calls in my app. The profiler is the best tool I’ve never seen to tell me exactly what a request did after it’s done executing by pinpointing the exact lines of code in the stack traces that spent the most time.

What do you dislike?

FR is not free like other tools, but it also offers immensely more features once you get to know it. Now that they offer a cheap developer license, it is much more accessible.

Recommendations to others considering the product:

There is a 2 week trial you can use for free. The simplest installation method is via the commandbox-fusionreactor module for CommandBox CLI.

What problems are you solving with the product? What benefits have you realized?

I use it for general web development to track the processing of Ajax calls and REST APIs that can’t output debug information easily to the browser. I use it as a companion tool to JMeter when performance testing to track web request and JDBC throughput as well as response times, status codes, and errors. I’ve used it to find production errors, memory leaks, and performance issues as well. It also can notify me of long running requests so I have a heads up immediately if my site has slowed down. FR Cloud also serves as a consolidated place for metrics from my Docker swarm.

See review on G2

Start a Free FusionReactor trial

What do you like best?

The ability to be able to see instantly exactly what is going on with my applications, in real time, and to be able to quickly and efficiently pinpoint and debug issues all in one package.

What do you dislike?

There is very little not to like. The only thing that would make it better would be if it was free!

What problems are you solving with the product? What benefits have you realized?

Application bottlenecks, garbage collection tuning, badly performing database queries, and even the odd bug in the application server have personally all been successfully solved using FusionReactor.

See review on G2

Get a Free FusionReactor trial

“enlightening!”

What do you like best?

ability to quickly identify issues quickly

historical logs and trends using powerful graphing tools

What do you dislike?

having to install and manage agent updates (although it is relatively easy)

What problems are you solving with the product? What benefits have you realized?

finding errors in applications

find slow performing applications

Start your free trail on up to 3 servers

See full review on G2

What do you like best?

Being able to see all of the request details, e.g. activity, history, long running requests.

The profiler is a killer feature.

What do you dislike?

Not really much to dislike, maybe the profiler UX could be improved (a lot of scrolling).

What problems are you solving with the product? What benefits have you realized?

Several performance issues in production environments under high load. Finding long running queries.

Concurrency tests in combination with Gatling.

Start a Free FusionReactor trial

See review on G2