The FusionReactor team are delighted to announce that we are sponsoring the O’Reilly Velocity conference 2019 in San Jose.
We are offering a 25% discount on all event tickets – you can receive your discount, using the following coupon – FusionReactor25. To use the discount code visit the O’Reilly Velocity registration page.
We will be demonstrating FusionReactor 8 at the event and introducing our latest features including our root cause analysis tool.
This year Velocity is focussing on Performance Resilience and Security. To get ahead today, your organization needs to be cloud-native. The 2019 Velocity program will cover everything from Kubernetes and site reliability engineering to observability and performance to give you a comprehensive understanding of applications and services—and stay on top of the rapidly changing cloud landscape. Learn new skills, approaches, and technologies for building and managing large-scale, cloud-native systems, and connect with peers to share insights and ideas.
Get expert insights and essential training on:
Don’t forget to use the FusionReactor25 discount code at registration and get a 25% discount.
In my last article I described how I wanted to help introduce FusionReactor to ColdFusion developers with a special focus on helping solve practical problems and navigating terminology that may not be terribly familiar with if you haven’t used Java before. In that first article I focused on using the FusionReactor portal to find and diagnose pages that were throwing errors. In this follow up I’m going to highlight another great use of FusionReactor – finding slow pages.
Slow pages can be difficult to find. A page that runs fast on your local server could run much slower in production. A page with a complex set of inputs, like an advanced search field with numerous filters, can only run slow when a particular unique set of choices are made. Sometimes your code can be absolutely perfect, but an external factor is the culprit. Perhaps you’re integrating with a database that you have no control over, or maybe you’re using a third party API that has performance issues of its own.
In this article I’m going to highlight multiple examples of slow pages and how FusionReactor can help you find, and hopefully fix, each of them! Ready?
Before we begin writing some horribly slow code (I’m a pro at this!), let’s look at where FusionReactor displays these requests. In your left hand menu, under Requests, select Slow Requests:
You’ll notice the display is split between currently running slow requests and requests that have already finished but were considered slow. Which begs the question – what’s “slow”?
Under Requests, go to Settings, and then select WebRequest History. Here you will find multiple settings you can configure including what the threshold is for something being considered slow. In my FusionReactor 8 instance this was set to 8 seconds.
And just to be re-iterate, this is important even if your ColdFusion page isn’t returning markup. If you are using ColdFusion to power a client-side application with JSON data, then you still need to ensure you’re returning a speedy response.
So yes, this is very, very important stuff. I’d go ahead and change that number from 8 down to 1 or 2. See what works best for you.
Another view of your slow requests is the Longest Requests view, also found under the Requests menu:
This is a sorted view of requests with the slowest being displayed first. This is not filtered by the Slow Request threshold but covers everything. Also note it includes non-ColdFusion files as well!
Let’s start off by demonstrating two slow pages. One is always going to be slow and one uses an external resource that will also always slow. Here’s the first one:
<h1>Slow Testing</h1> Go to sleep...<br/> <cfflush> <cfset sleep(30000)> And done.
It simply uses the
sleep function to pause the execution for 30 seconds. If you’ve never seen
cfflush before, it tells the ColdFusion server to send out existing output to the browser. As a user, you would see the initial HTML, the browser continue to load, and then the rest of the results.
This will definitely cause a slow request to be logged. Unfortunately, unlike the error conditions we saw in the previous article, it’s a bit more difficult to diagnose. Let’s see why. First, here’s the details page for this particular request.
There’s a few things to note here. First, the slow time is nice, bold, and red. You can’t miss that and that’s a good thing. Now make note of JDBC. Don’t know what that means? JDBC is the acronym for Java Database Connectivity. Basically this is how ColdFusion speaks to databases. I started ColdFusion development way back in 96 or so, before ColdFusion ran on Java, but even back then the biggest culprit for slow requests was issues at the database layer. We aren’t doing any database requests on this page but nine times out of ten you want to turn your focus here. Every single one of those metrics is crucial and can be a flag for an underlying issue.
As a real example of this, I was working with a client and discovered they had about 1000+ database queries being run in a request. How did this happen? They had queries in an Application.cfm file, queries in a header file, queries in a file loaded by a header file, and so on and so on. 90% of these queries were the exact same query run multiple times. They weren’t stupid – they just didn’t realize everything going on in one particular request. This can happen as a web application grows larger and more complex, and is yet another reason to consider a tool like FusionReactor!
Alright, but this template isn’t slow because of a query. What do you do next? Click on the Profiler tab next. (And note, back in the main list of slow requests, every slow request has a direct link to this particular view.)
So in this case, you can’t really get to a particular line of your template or ColdFusion method. But even if you’ve never used Java before, seeing 100% correlated to
java.lang.Thread.sleep should give you a clue as to which particular tag was the issue.
Let’s consider another example, this time the ColdFusion template is using a third party API built on a serverless platform:
<h1>Slow Testing 2</h1> Call API...<br/> <cfflush> <cfhttp url="https://wt-c2bde7d7dfc8623f121b0eb5a7102930-0.sandbox.auth0-extend.com/slow"> <cfdump var="#cfhttp#">
In this case, the serverless function I’m hitting is using
setTimeout to delay the response for 3001 milliseconds. This page is going to be slow, but it’s something out of ColdFusion’s hands. (But to be clear, this is still something you could address. Can you cache the result? Can you fire it via a scheduled task and store the result locally? You’ve got options!) So how is this reported in the profile?
In this example, you can see two things that stand out – first
Http.java and then
Dump.cfc. The later clearly refers to the
cfdump tag which typically wouldn’t be run in production. But
Http.java should definitely be enough of a clue to tell you where the issue is. Of course, you may run into an issue if you have multiple HTTP calls in your ColdFusion page. In that case you’ll need to do further testing to see if you can determine the culprit. The important thing to note though is how FusionReactor got you nearly 100% of the way there!
In this article I simply introduced the Slow Request views in FusionReactor and used it to find a few problem scripts that should be fixed. In my next article I’m going to turn my attention to database specific issues and show you how FusionReactor helps there. As always, leave me a comment below if you have any questions!
As a ColdFusion developer, you may know that it’s running as a J2EE server but also may have zero to no idea what that means in a practical sense. ColdFusion has always been easy to use, especially for developers from a non-traditional background, and this sometimes means there’s aspects of the platform that aren’t quite as easy to understand as others. A great example of this are the things that are more Java-centric. FusionReactor integrates with your ColdFusion server from a Java-perspective, which means it maybe uses terms that may be unfamiliar to the developer who only knows CFML.
And yes, you can, and should, consider learning more about Java, but at the same time, we don’t always have the opportunity to pick up a new language! You may need to get things working now and what I’d like to do in this article is help you, the CFML developer, better understand how Fusion Reactor reports issues and get you to a point where you can quickly identify (and fix!) your problems.
My assumption is that you’ve already installed FusionReactor. If not, head over to the downloads and grab a copy. You can get your trial key there and use it to test and evaluate the product. As I’m writing this for CFML developers who aren’t familiar with Java, I strongly urge you to use the “Automated Installers”. I’m testing on a Windows machine with Lucee but
There’s a heck of lot to FusionReactor but in this article I’m going to focus on just one particular aspect – errors. Luckily, I’m a born error creator. One of the best. I should probably get endorsed on LinkedIn for writing buggy code. Thankfully that’s made me something of an expert in debugging a file and figuring out what went wrong. That’s rather easy while I’m writing and testing code. It isn’t necessarily helpful when the code is in production and being used by the public.
Let’s consider a simple template that seems to be bug free.
<h1>Debug Testing</h1> url.name = #url.name# len is #len(url.name)#
This script simply outputs the value of a variable passed in the query string, `name`, and then reports the length of the value. Given that the file is named `ray.cfm`, you can test this like so:
<p> http://127.0.0.1:8888/ray.cfm?name=raymond </p>
Which gives you:
Ok, astute readers will recognize the issue with this code. What happens when you don’t pass the `name` value in the query string?
In this case it’s probably obvious what you did wrong. This is easy enough to fix by either setting a default or adding logic to redirect when the value isn’t defined, but what if you miss this and deploy the code to production?
Assuming you’ve done that and got a report from your users (or even better, your manager at 2AM on Saturday), let’s quickly see how FusionReactor can help you find this issue.
Alright, so you’ve got a report about something going wrong. In a perfect world, that report includes the URL, query string, what the user was doing, the phase of the moon, and more. Also know that ColdFusion has excellent built-in error handling that can send you detailed reports… if you added that to your project.
But let’s pretend you have nothing, or something as useless as this:
“Hey, the web site is broke.”
Let’s begin by looking at the history of requests in the FusionReactor instance. In the left hand menu, mouse over Requests and select History:
There’s a lot to this table, but let’s focus on a few things in a somewhat descending order of importance:
* Status Code: When a request is made over the web, a server returns a code that represents how it was handled. `200` represents a good response. `500` is a bad response. (And note how FusionReactor is flagging that already!)
* URL: This tells you the URL of the request and normally, but not always, will give you an idea of the file that was requested. Many people will use URL Rewriting to “hide” the fact that ColdFusion is running. In this
That was the general request view, but most likely you want to focus in on just the errors. In that same left-hand nav, you can do so by selecting: Requests, Error History:
This is showing the same information, just filtered to requests with errors.
Alright, so you’ve found a request with an error, how do we diagnose it? On the right hand side is a “book” icon which will load details. Let’s do that and see what we get.
Overwhelming, right? Everything there is useful, but there’s a lot you don’t need right now in order to get to the issue at hand. Begin by clicking “Error Details” under “Transaction Details”
The first two blocks give you a good idea of what’s going on. `key [NAME] doesn’t exist` is a clue that your code tried to use a variable that didn’t exist. If the `key` aspect doesn’t make sense, it may be helpful to remember that ColdFusion variables are scopes, or structs, and when you access `url.something`, you’re trying to use a particular key in that scope.
Alright, so we’re closer. Now our particular ColdFusion file is very short but imagine a much larger template, how would we find where the error was thrown?
The bottom part of the template is the Java Stack Trace…
The stack trace can be overwhelming. In this case, you can slowly read line by line looking for a line mentioning a CFM file:
In this case, `ray.cfm` is the file name (pretty obvious) and `5` is the line number:
url.name = #url.name#
In theory, you’re done, but FusionReactor can help you a bit more. In the left hand navigation, look for the Debug menu and select Sources.
Out of the box, FusionReactor doesn’t have any defined sources, so simply click the “Add” button and specify your web root. (Note that you can have more than one, and most likely you’ll have one root directory per application.)
In the screenshot above I’ve simply specified my Lucee root.
When you’ve done that go back to the Requests view, find that broken request, and get to the stack trace again.
Now if you click on the method call, which is `ray_cfm$cf.call`, FusionReactor will actually be able to show you the file itself!
Notice how FusionReactor even highlights the line in the file!
FusionReactor contains a huge set of tools, but in this article we focused in specifically on helping diagnose an error in production. In the next one we’ll start expanding our use of FusionReactor and look for more performance related issues.
Thanks to our special guest author Raymond Camdem for writing this article for us.
Blog : www.raymondcamden.com
FusionReactor 8 (FR8) has been designed to make a developers life easier and dramatically reduce the time required to figure out what’s gone wrong in a production environment. The game-changing “Events Snapshot” capability takes the hard work out of debugging by automatically tracking all latency and thread errors and presenting the developer with insight immediately after the error has occurred.
For decades, developers have had to struggle with analyzing log files to figure out why production software is broken – the capability in FusionReactor dramatically revolutionizes this process, by automatically capturing detailed, structured information at the point that errors occur. FR8 will capture variable state across the whole scope of the stack, logged information and even instantly decompiles your code to show you the full context of where and why something is broken.
“In FR6 we introduced a Java debugger, which can be safely and securely used in production environments, FR8 takes this a step further and automates the process so that issues can be instantly identified allowing developers to spend less time looking for defects and more time creating new code” David Tattersall CEO
FusionReactor will now automatically capture any information which has been written to logs using one of the standard logging API’s, such as cflog, log4j 2, logback, SLF4J and Apache Commons Logging.
FusionReactor is proud and delighted to have been awarded the “Gold Trusted Service Award” for the third consecutive year.
Feefo is a reviews and customer insights technology company that provides businesses with the tools to collect real, purchase-verified reviews and insights.
Created by Feefo, Trusted Service is awarded only to those businesses that use Feefo to collect genuine reviews and insights. Those that meet the high standard, based on the number of reviews they have collected, and their average rating, receive the award. A badge of honour, this accreditation remains unique, as it is based purely on the interactions with real customers. As all reviews are verified as genuine, the accreditation is a true reflection of a business’ commitment to outstanding service.
To win the award you have to achieve a Feefo service rating of 4.5 for the whole of 2018 – we smashed this with a Feefo score of 4.9.
“We would like to thank the team at FusionReactor for their continued commitment to outstanding customer service and to our customers who left us fantastic reviews.”
David Tattersall CEO
We ensure that all your reviews are read and that your comments are fed back to our developers, support and sales teams.
The team at FusionReactor is aware of the changing developer landscape and strive to continually evolve our product with new developer features and support for new technologies. We are looking forward to an exciting 2019.
You can read our Customer reviews here.
Previously I blogged, in earlier posts, about installing and deploying FusionReactor Cloud to monitor CFML applications on Docker Swarm. The next step is to to configure its updated alerting system to let you know if anything is amiss.
Alerting is one of the primary three tabs within the FusionReactor Cloud portal, with sub-tabs for Alerts, Checks, and Subscriptions.
So, taking them one at a time, in a little more depth:
As you’ll see, FusionReactor Cloud offers far more granular control than the on-premise edition – this comes at the cost of significantly more configuration necessary to get it up and running.
In order to set up a Subscription, you’ll first need to enable one of the many available integrations. An integration is a way of contacting someone; options include Slack, PagerDuty, custom webhooks, and more. We’ll just stick with Email, as it’s the easiest and most obvious choice.
Now, I’m of the opinion that the Email integration should be enabled by default, but it’s not. So, once you’re logged into your account, click the Account button in the upper right, and select the Configuration option.
Click the Configure button for Email, then all you need to do is choose the Save option… and the Email integration is enabled.
On to the actual Subscription setup. Proceed to the Alerting tab and the Subscriptions sub-menu. In the upper right, the +Subscription will open a tab for actually setting up who gets alerts (and when it happens).
Here, you start to see the level of control that Alerting in FusionReactor Cloud provides. You can restrict this Subscription to certain days of the week, or hours of the day. This may be helpful, for example, if you have a separate team for after-hours. Or, you can have separate Subscriptions for Warning vs. Error states – the latter being set to higher priority, or sent via a different integration.
We’re setting up a basic email subscription, so we’ll name it “Web Team Email Alerts”, or something comparable, and choose the Email Service.
The default subject provided, “FusionReactor alert”, is fine; in the actual alert email this will be followed by the name of the Check that was triggered. Finally, we’ll add the email address we want to send the alert to, and click “Save”.
Your Subscription is now saved; you’ll see its details listed in the main Subscriptions panel, with the ability to duplicate/edit/delete, as well as test it. It hasn’t been linked to any Checks yet, so let’s do that next.
In the Alerting section, within the Checks sub-menu, click +Check in the upper right to open the tab for building out a new Check.
Within this panel there’s a lot going on. I’ll walk through the options one step at time, but we’ll save actually creating a Check for the next blog post.
Note that the Name entered here for the Check will be included in Subscription notifications that get sent, so it should be descriptive and clear. Optionally, you can also fill in a Description, for internal reference.
server.json. I’ve found Groups very helpful when deploying to Swarm; they make it easy to monitor multiple container replicas as a single entity.
Application: The name says it; your applications are listed here. Even if the application is replicated across nodes, you can apply a Check to it.
After clicking Save to add the Check, you’ll see its details listed in the main Checks panel. As with Subscriptions, from here you can duplicate, edit, or delete it. You can also temporarily disable Checks from this page.
Finally, we’ll move on to Alerts; fortunately, there’s nothing we need to configure here. The Alerts panel provides logged reporting of your Checks; any time that a Check changes status, it’s recorded here.
Basic logging here includes when Checks are added, paused, or deleted. These changes are recorded, but do not trigger Subscriptions. When a Check moves between the OK, WARNING, and ERROR statuses, the Subscriptions that are triggered will also be recorded here. The “View” option will open a panel with more information about the Alert/Check/Subscription.
If you want to see the power of alerting in FusionReactor for docker swarm for yourself you can get a trial of FusionReactor Cloud for free, just create your account and download the software.
For more information about setting up Alerting in FusionReactor for Docker Swarm contact the FusionReactor Cloud team.
Moving a standard ColdFusion installation to Docker Swarm requires rethinking – and frequently rewiring – portions of our infrastructure. Today, that means reconfiguring how we deploy FusionReactor to monitor our applications.
If you’re already using FusionReactor, you know its value (and if you’re not using it, you probably should be).1 Moving to Swarm means that you can’t continue to use the traditional on-premise installation of FusionReactor; it’s not suited to the distributed, dynamic environment of containers. Fortunately, there’s an alternative: FusionReactor Cloud.
In this post, I’ll outline one approach to deploying your applications with FusionReactor Cloud.
Two notes; first, this post assumes a degree of experience with Docker Swarm and CommandBox – learning to configure FusionReactor to monitor CFML apps on Docker Swarm isn’t of much use unless you’ve already worked out the process of deploying those apps in the first place. Second, there are a lot of moving pieces here, so if I gloss over details you think are important, please let me know and I’d be happy to clarify.
Cloud is an extension of the standard FusionReactor installation.2 While its interface differs considerably from the on-premise version, the functionality tracks pretty closely. I’ll review some of the differences in a future post. For our purposes, its most significant feature is that FusionReactor Cloud supports dynamic server monitoring. That is, you can use it for monitoring CFML apps running in Docker containers.
Before we get into how to use it, let’s address how to get it, and what it costs.
There’s no reference to Docker or containers in the Cloud licensing section of the FusionReactor pricing page, which is confusing. Here’s what I found: FusionReactor Cloud’s current fair usage model allows/supports monitoring up to 5 container instances per license.
Because the ratio of nodes-to-containers can vary considerably from one company to the next, this model may or may not work for your Swarm setup. If that’s the case, I’ll pass along the advice that I was given: Reach out to the FusionReactor team to discuss your situation and Docker configuration. They’re a developer friendly organization, willing to help users transition to the cloud.
The rest of the steps here assume that you have a Cloud license available. If you don’t, you can register for a free trial.
We’ll be using the CommandBox Docker image; it makes configuration and deployment very simple. If you’re not familiar with CommandBox, I encourage you to take a look; it’s a remarkable tool for CFML development. Also, the rest of this post won’t make sense if you’re not familiar with how CommandBox can be used to spin up ColdFusion servers.
There’s a FusionReactor module on ForgeBox. When added as a dependency in an application’s
box.json, it installs FusionReactor for the server.
There are two approaches to ensuring that this module is installed and ready when your containerized CommandBox server starts:
truein your Docker stack file, which triggers the
box installcommand to be run before the server is started.
RUN box installprior to the server warmup and start, to ensure the dependencies are installed.
The FusionReactor module and its settings have a dedicated page within the CommandBox documentation. Thanks to CommandBox, we can configure the module within our
server.json, like so:
Here’s a breakdown of the settings used above, beneath the
false, FusionReactor functionality is disabled. This might be something you’d want to do for testing or in different environments.
frlicenseservice.deactivateOnShutdown. When set to
true, the instance being monitored will deactivate its license on shutdown. Obviously, that’s what you want for a containerized server.
fr.cloud.groupJava property. These group names are really helpful in organizing your reporting and alerts within FusionReactor Cloud. They’re effectively tags for organizing your servers.
Assuming that the rest of your CFML stack is in order, with your
server.json configured, you’re ready to deploy to Swarm. The resulting CFML containers will be monitored within the FusionReactor Cloud portal and their licenses will automatically be deactivated as the containers are replaced.
Getting FusionReactor installed in your containers and deployed to Swarm is half the process; the second half is working through the differences between the on-premise installation and FusionReactor Cloud. In my next post, I plan on examining some of those differences and discussing the process of setting up FR Cloud alerts that mirror the on-premise functionality. Until then, good luck Dockering
If you would like to know more about FusionReactor Cloud for Docker Swarm contact our Cloud team.
Debugging in production is often considered to be the holy grail of root-cause analysis – just imagine the benefit of being able to introspect variable scope at production runtime! However, the performance degradation and breakpoint impact implications associated with traditional debuggers have rendered them useless in production environments. Intergral GmbH, makers of FusionReactor have developed an extremely low overhead Java debugger (Patent Pending) that can be safely and securely used across the whole Software Development Lifecycle (SDLC), including production.
Intergral GmbH, makers of FusionReactor are once again sponsoring The O’Reilly Velocity conference in London, 2018. The FusionReactor team will be located at booth #109 and will be on hand to demonstrate how it’s possible to debug and profile Java applications at production runtime…
“Debugging a Java application which is running in production is not something most people think of doing. There is typically a performance overhead as well as a number of technical challenges which make this impracticable using existing debugging technologies”, said David Tattersall, CEO of Intergral GmbH, makers of FusionReactor “Intergral have addressed these issues and developed a browser-based Java debugger, which can be used in a production environment and which we believe is a game changer in accelerating the time needed to isolate production problems and performance bottlenecks”.
FusionReactor is used by 1000’s of developers and DevOps to cut issue isolation time down from days to minutes. FusionReactor combines the core APM features of metric instrumentation and alerting with low-level profiling and debugging – it is the only product of its kind which is designed for use across the whole Software Development Life Cycle.
FusionReactor is a hybrid APM solution, available as on-premise and/or SaaS (Cloud-based). Four editions are on offer, Ultimate, Enterprise, Standard and Developer. The Ultimate and Developer Editions include the debugger and low-level profiling tools, Enterprise Edition allows monitoring of multiple instances and servers from a single Dashboard and Standard Edition includes all the core monitoring features. FusionReactor may be purchased as a monthly/annual subscription or as a one-time license. Standard Edition pricing starts at just US$ 39 per month, per server.
FusionReactor Cloud provides extended data retention, focus on “issue related transactions” and includes advanced alerting capability with seamless integration with tools such as Slack, PageDuty and Twilio etc.
About Intergral GmbH:
Intergral is a leading application intelligence company. Our flagship product, FusionReactor provides real-time visibility and alerting of application performance issues and can be effectively used across the whole Software Development Life Cycle. Thousands of customers trust FusionReactor to monitor their applications, enabling them to identify and respond faster to performance and stability problems. FusionReactor’s aim is to ensure applications run at peak operational and business performance. Learn more at www.Fusion-Reactor.com.
This year, the FusionReactor team are once again proud sponsors of the O’Reilly Velocity conference, which is being held in London between 30th October and 2nd November 2018.
We are offering a 25% discount on all event tickets – you can receive your discount, using the following coupon – FusionReactor25
We will be demonstrating FusionReactor at the event and showing you that when applications break or fail to perform, developers and DevOPS teams must go deeper than simply analyzing resource usage or business transaction fail rates – they need real-time insight and transparency to get a clear picture of why an application is actually failing or slowing down as these issues are happening in production. Please come and see us on booth #212
Under the motto of “Build Systems that Drive Business” – the Velocity conference is an amazing opportunity for Software Engineers and DevOps to build skills to meet the pressing demands of your work, hear from your peers and meet with industry leaders.
The conference will deal with aspects such as building secure systems, monitoring / observability / performance and microservices to name a few. Through expert-led presentations (from the likes of The University of Cambridge, Google and Microsoft), you’ll learn to build high-performance, resilient, and secure systems.
Read more about the London Velocity conference here – https://conferences.oreilly.com/velocity/vl-eu
Register here and use code FusionReactor25 to receive a 25% discount.
A customer recently wrote into the FusionReactor Support Team regarding their ColdFusion server, which was crashing on average once per week. We have used FusionReactor and specifically the archive viewer, to take the customers archived log data and diagnose the root cause of the issue.
This blog will explain the process of how it is possible to diagnose the root cause of the crash with the Archive viewer, a feature introduced in FusionReactor 7.2.0 that allows users to display and graph historical log data in a similar fashion to viewing data in the “live or recent” monitored application server.
Starting by viewing the resource log it was possible to see that the total memory allocated by the JVM was exceeding the maximum memory configured in the ColdFusion server:
From there it was then obvious that there was clearly an issue with memory usage in the server, looking in the memory summary logs it was then possible to see that the actual memory usage was only 20% of the usable memory, however, the committed memory was high.
Memory Summary Usage
The fact that the heap usage was only 20%, yet the committed memory was high points to the fact that a particular memory space was attempting to use more memory than it had available. At this point, it was then possible to log at the logs for each available memory space. From this, it was possible to see that the old gen memory space allocated had committed 14GB of its available space:
OldGen Memory Usage
The Old Gen memory space is used for long living objects that are committed to the JVM for longer periods of time, this includes objects such as the ColdFusion scopes and classes that are loaded into memory, this space is only cleared when major garbage collections occur within the JVM.
In a typical ColdFusion server, you will see considerably more minor garbage collections than major garbage collections, however using the gc (garbage collection) logs it was possible to see that large garbage collections were occurring on average every 4 minutes and taking between 750 ms and 1.5 seconds each time:
GC Marksweep times
Time spend on major garbage collections in Java will consume large amounts of CPU and reduce the performance of the application server.
Using the classes log file it was then possible to see that there was a very large number of classes loaded into the JVM, which would explain why 14GB of OldGen memory was committed to memory and the JVM was no longer able to allocate the required memory needed to run the application:
Count of loaded classes
It is important to note that the fact that the number of loaded classes is high is not the only cause of the Application server running out of OldGen Memory, but it is indeed a factor. In the Case of this ColdFusion server, the OldGen will contain the session scops, user session tracking, all loaded classes and many other objects that will need to exist for a longer period of time.
However, the fact that classes are being loaded periodically in this manner is highly irregular and would be cause for concern. If I take another CF server that is healthy I would see something similar to:
Example of the count of loaded classes for a similar ColdFusion server that is in a healthy state
From this it is possible to see that there is a much lower class count and most importantly the graph is flat, this is what I would expect as classes committed to the OldGen space should not be cleared and reloaded into memory this often, Only if Java believes the classes loaded are too old and there is a potential that the bytecode has changed.
From this information, it was then possible to recommend increasing the maximum heap space available on the JVM and potentially the available resource for OldGen memory, trimming down the running CFML applications would also be a possible solution to reduce the Old Gen and heap usage of the running JVM.