FusionReactor Java Performance Monitor: Blog

This year, the FusionReactor team are once again proud sponsors of the O’Reilly Velocity conference, which is being held in London between 30th October and 2nd November 2018.

We are offering a 25% discount on all event tickets – you can receive your discount, using the following coupon – FusionReactor25

We will be demonstrating FusionReactor at the event and showing you that when applications break or fail to perform, developers and DevOPS teams must go deeper than simply analyzing resource usage or business transaction fail rates – they need real-time insight and transparency to get a clear picture of why an application is actually failing or slowing down as these issues are happening in production.   Please come and see us on booth #212

Under the motto of “Build Systems that Drive Business” – the Velocity conference is an amazing opportunity for Software Engineers and DevOps to build skills to meet the pressing demands of your work, hear from your peers and meet with industry leaders.
The conference will deal with aspects such as building secure systems, monitoring / observability / performance and microservices to name a few. Through expert-led presentations (from the likes of The University of Cambridge, Google and Microsoft), you’ll learn to build high-performance, resilient, and secure systems.

Read more about the London Velocity conference here – https://conferences.oreilly.com/velocity/vl-eu

Register here and use code FusionReactor25 to receive a 25% discount.

A customer recently wrote into the FusionReactor Support Team regarding their ColdFusion server, which was crashing on average once per week.  We have used FusionReactor and specifically the archive viewer, to take the customers archived log data and diagnose the root cause of the issue.

This blog will explain the process of how it is possible to diagnose the root cause of the crash with the Archive viewer, a feature introduced in FusionReactor 7.2.0 that allows users to display and graph historical log data in a similar fashion to viewing data in the “live or recent” monitored application server.

Starting by viewing the resource log it was possible to see that the total memory allocated by the JVM was exceeding the maximum memory configured in the ColdFusion server:

Resource.log

From there it was then obvious that there was clearly an issue with memory usage in the server, looking in the memory summary logs it was then possible to see that the actual memory usage was only 20% of the usable memory, however, the committed memory was high.

Memory Summary Usage

The fact that the heap usage was only 20%, yet the committed memory was high points to the fact that a particular memory space was attempting to use more memory than it had available. At this point, it was then possible to log at the logs for each available memory space. From this, it was possible to see that the old gen memory space allocated had committed 14GB of its available space:

OldGen Memory Usage

The Old Gen memory space is used for long living objects that are committed to the JVM for longer periods of time, this includes objects such as the ColdFusion scopes and classes that are loaded into memory, this space is only cleared when major garbage collections occur within the JVM.

In a typical ColdFusion server, you will see considerably more minor garbage collections than major garbage collections, however using the gc (garbage collection) logs it was possible to see that large garbage collections were occurring on average every 4 minutes and taking between 750 ms and 1.5 seconds each time:

GC Marksweep times

Time spend on major garbage collections in Java will consume large amounts of CPU and reduce the performance of the application server.

Using the classes log file it was then possible to see that there was a very large number of classes loaded into the JVM, which would explain why 14GB of OldGen memory was committed to memory and the JVM was no longer able to allocate the required memory needed to run the application:

Count of loaded classes

It is important to note that the fact that the number of loaded classes is high is not the only cause of the Application server running out of OldGen Memory, but it is indeed a factor. In the Case of this ColdFusion server, the OldGen will contain the session scops, user session tracking, all loaded classes and many other objects that will need to exist for a longer period of time.

However, the fact that classes are being loaded periodically in this manner is highly irregular and would be cause for concern. If I take another CF server that is healthy I would see something similar to:

Example of the count of loaded classes for a similar ColdFusion server that is in a healthy state

From this it is possible to see that there is a much lower class count and most importantly the graph is flat, this is what I would expect as classes committed to the OldGen space should not be cleared and reloaded into memory this often, Only if Java believes the classes loaded are too old and there is a potential that the bytecode has changed.

From this information, it was then possible to recommend increasing the maximum heap space available on the JVM and potentially the available resource for OldGen memory, trimming down the running CFML applications would also be a possible solution to reduce the Old Gen and heap usage of the running JVM.

With the release of FusionReactor 7.4.1 we added support to track async requests in ColdFusion 2018, these async functions will be tracked as a child transaction of the web request.

This addition will allow you to track whether any asynchronous call you are making in your CFC/CFM files is slowing down your request, in previous versions of FusionReactor the only way this would be possible would have been to use the debugger or profiler.

Each async call will be tracked as a child request of the CFC/CFM script it was executed inside, so it will be visible in the relations tab of the requests details, due to the nature of the asynchronous call it is only possible to link a transaction back to the CFC/CFM script after the async function is complete.

Below is an example of a series of async calls run as both named and anonymous functions.

Example CFML

<cfscript>
getAccountBalance = function () {
var balance = 120000;
return balance;
}
function payCreditCardBill(accountBalance) {
var ccBill = 1890;
return accountBalance – ccBill;
}
payEMIs = function (accountBalance) {
var mortgageEMI = 1000;
var carLeaseEMI = 750;
var healthInsuranceEMI = 250;
return accountBalance – (mortgageEMI + carLeaseEMI + healthInsuranceEMI);
}
miscellenousExpenses = function (accountBalance) {
var shopping = 1500;
var clubExpense = 1000;
var casinoExpense = 2000;
return accountBalance – (shopping + clubExpense + casinoExpense);
}
function checkBalanceaccountBalance) {
while (accountBalance > 5000) {
accountBalance = miscellenousExpenses(accountBalance);
writeOutput(“checkBalance = ” & accountBalance & “<br/>”);
} if (accountBalance < 5000)
throw(message = “Account balance below threshold!!!”, type = “info”);
}
errorHandler = function (error) {
if (error.message contains “Account balance below threshold!”) {
return “You have reached your spending limit!”;
}
}

future = runAsync(getAccountBalance).then(payCreditCardBill).then(payEMIs).then(miscellenousExpenses).then(checkBalance).error(errorHandler);
writeOutput(future.get());
</cfscript>

Request tracking in FusionReactor

In FusionReactor we can see each async request is tracked under the master request syncBankExample.cfm. The flavour of each request will be CFUDF the tag used by ColdFusion to execute these functions.

An important thing to note is that how you declare your function to run asynchronously will change the accuracy of how well FusionReactor will track the async call. The methods payCreditCardBill and checkBalance can be referenced by name in the description of the runAsync method, however, every other method cannot. This is because payCreditCardBill and checkBalance are declared with the following syntax:

function functionName() { … )

The other asynchronous functions use the syntax:

functionName = function () { … }

For this reason, we highly recommend that anyone using the asynchronous calls uses the syntax “function name() { … }”

With the release of FusionReactor 7.4.0 we’ve added some new support for a number of ColdFusion tags (we already had support for CFHTTP and CFQUERY). This now means in both ColdFusion and Lucee the following tags will now be tracked as a child transaction of the web request:

• CFLDAP
• CFFTP
• CFMAIL
• CFIMAP
• CFPOP

Providing this support will allow you to spot if one of these tags would be slowing down a CFC / CFM call made within your CFML application, where before you would be able to see that a CFC / CFM call was slow, but not explicitly identify what was causing this without using the production debugger and profiler.

For each tag, each attribute is tracked as a transaction property allowing you to see exactly what variables are being used by the tag. The only exception to this is the password, which will not be tracked in FusionReactor.

Each tag will be tracked as a child request of the CFC/CFM script it was executed inside, so it will be visible in the relations tab of the request details. The child request will be generated in FusionReactor as soon as it begins executing, so will be visible under the Requests activity page as a relation.

Below are examples for each tag running on ColdFusion 2018, for Lucee the tag attributes will need to be modified.

CFLDAP

Example script

This code snippet will attempt to authenticate against an LDAP server with a given user and set a variable of isAuthenticated based on the output of the tag, when we run this snippet inside a CFC we see:

You can see here the query performed on the LDAP server and that no error occurred, so we can assume there was no issue authenticating with the server.

When viewing the properties tab of the CFLDAP transaction you are able to view all the attributes used in the tag:

CFFTP

Example script

This code snippet will establish a connection to the FTP server, retrieve a list of the directories and then terminate the connection, running this code in a CFC will result in:

Here it is possible to see the 3 distinct CFFTP calls and the time taken for each, each call will have a unique set of attributes and these can be seen in the properties tab of the transaction details.

CFMAIL

Example script

This example code will send a simple test email, running this code inside a CFC file will result in the following:

You can see a CFMAIL tag with the action send tracked in FusionReactor.

CFIMAP

Example script

This script will establish a connection to the IMAP server then close this connection, running this code snippet in a CFC file will result in:

You can see 1 request per CFIMAP call, so we see the connection open and close tags as separate requests. Like the CFMAIL tag attributes used in each tag are tracked as properties of the transaction

CFPOP

Example script

This script will query the mail server for the size of the user’s inbox and print this in the browser, running this in a CFC will result in:

You can see here the CFIMAP call took 4.9 seconds to execute, with an older version of FusionReactor it would be difficult to track down why CFM script was slow to execute without debugging server but now it is easy to see.

FusionsReactor 7.3, the latest release in the ColdFusion and Java monitoring tool has tons of new features and improvements. This article will be an overview of some of the most valuable aspects which affect the bottom line and ultimately guide on how FusionReactor can work for you.

You can find them listed also in the FR “release notes” technote (https://www.fusion-reactor.com/support/kb/frs-431/). Please note, if you scroll further you can see how the APM has developed throughout the installments.

Keeping up to date with the developments in platforms

There are a few FusionsReactor 7.3 enhancements which are specific to particular supported platforms.   Support is now included for Java 10, ColdFusion 2018 as well as Solr.

More developments include additions in application servers to the “System Information” page, with ColdFusion, Lucee, Tomcat, etc. now comes JBoss information as well. Server detection for a Commandbox run instance is now part and parcel.

Tomcat log files will also be captured alongside the other FR logs in the same timeframe as before (per hour), all set as standard.

FusionReactor has long done the same sort of excerpting of ColdFusion’s console log, the ColdFusion-out.log, which appears in the FR logs as fr-coldfusion-log.log. In the case of Tomcat, the log catalina and localhost file names will show up in the FR logs with the names juli-catalina.log and juli-localhost.log, respectively. Finally, the Tomcat stderr.log is archived by its original name.

Having these logs for the application server available while reviewing FR logs can be very helpful, especially for post-crash troubleshooting.

These capabilities allow FR 7.3 to keep up to date with the changing world, important when considering Moor’s Law. The furthered log file competences provide a richer data collection, giving you further insight into what’s going on with the environment. Ultimately, this affects more informed decision making which provides quality as well as faster resolution of issues.

GDPR compliant with further obfuscation

Progressions have been made with regards to the obfuscation of data, something very important in the modern age, which is not going away with the resent General Data Protection Regulation (GDPR) requirements. Versions from 6 have provided options for obfuscating data, including SQL and for MongoDB obfuscation, to FR 6.2’s optional obfuscation of request data, to FR7’s obfuscation of credit card data and more.

FR 7.3 enables you to obfuscate IP addresses.  This and the others aid in meeting GDPR and other security/privacy requirements. This way you can make sure data is safe and save money on any potential penalties as a result of GDPR enforcements.

In the email settings page, changes to the “to” address can be saved while testing

One of the most used features within the FR set up is the ability to receive messages when something has occurred with your application (this could be daily reports, or more importantly crash protection). Upon testing the email system, in previous versions you would have noticed that you could change the email address for the test purposes, however it would not have changed in the overall settings.

In FR 7.3, you can test your email all the same, adapting the “To address” value(s). Furthermore, one will be able to click the “save settings” button, which will save the input, until when changed again.

Realistically this saves time and ultimately provides an opportunity to customize FR 7.3 to your specification at a higher rate, meaning you can move on to more important tasks

Making the most out of FusionReactor Cloud

FusionsReactor 7.3 has been one of our main focal points in terms of developing more capability for the FR range. For those of you who were not aware, FR Cloud aims to expand the capability provided by the on-premise version of FR by providing access to historical data, combining instrumentation data across server instances to provide a single consolidated view and very focused access to specific issues, such as slow running requests or transactions which threw errors.    FusionReactor Cloud also includes a sophisticated alerting engine, which provides seamless integration with several tools, such as Slack, PagerDuty, HipChat, VictorOps etc.    You can read about the features here.

As part of the FR 7.3 release enhancements,  FR Cloud users will now be able to be see an indication of an active profile for a running request that is itself being profiled.

Another feature is the immediate request API which ultimately allows for a more instant connection between the cloud and on premise solution ensuring faster “real-time” performance.

Another benefit of the improvement to communication in FusionsReactor 7.3 is FR now being able to send more profile entries to FR Cloud by default. There’s also improved error handling and feedback when using the feature to start/stop Profiling of requests from FR Cloud. And thread dumps taken in FR Cloud now better match on-prem FR thread dumps (including more thread meta data), and sub-transaction information for recent and running requests as well as JDBC transactions are also improved.

More log functionality has also been updated. Users can now monitor their log file creation on FusionsReactor 7.3 on-premise and decide whether or not to send the data to the cloud, ultimately saving time, later on, and memory space. Although there is no direct capability for that on cloud, the brilliant thing, is when it comes (soon) it will be automatically uploaded to any using FR Cloud already.

FR Cloud is all about data, how to use it and the processing of said data. The ability to view multiple indications of profiles allows for efficiency of work, alongside the API feature which will speed up processes. Time saved through speed ultimately saves money, as well as allowing for employees to spend their time more effectively. The log and thread capabilities provide a cohesion between the on-site and cloud FR implementations, using synchronization alongside the enhanced data displays to insightfully inform decision making while allowing for improved timing between viewing real time data (on-site) or historical data (in the cloud). These ensure quality decision making for furthering business interests, in a timely manner which saves cost and enhances the availability of resources.

Conclusion; getting the update

So although not comprehensive, we have covered some of the salient FusionsReactor 7.3 updates. Furthermore there are some bug fixes which can be seen on FR 7 updates technote. Ultimately all these changes are aimed at quality, time saving, cost saving and ensuring customer satisfaction by improving the back end processes. All of which should improve in company profitability and sustainability.

If you already have a previous version of FusionsReactor 7.3 java monitoring tool, note that there is a “check for updates” link on the FR “about” page. If you are new to the product, download the latest version by clicking on FR downloads page.

NOTE – much of this blog text has come from the original, written by Charlie Arehart – which you can find here

This year we are back at Velocity Conference San Jose – You can visit us at booth #812 we will be showing off the latest FusionReactor developments and features to help developers fix problems faster! To celebrate this we are giving away a free ticket for Velocity Conference, San Jose, California.

The Prize is for one Bronze 2-Day Pass which Includes:
– All sessions (Wed & Thu)
– All keynote and plenary presentations
– Sponsor Pavilion and all events held there– All on-site networking events
– Lunch (Wed & Thu)

To enter the draw:
1) Start a trial of FusionReactor and send us a screenshot of your activation Menu / About Page. Send the screenshot to sales@fusion-reactor.com, the version must be FusionReactor 7.2.3 or higher.

We will be announcing the winner on, Friday 1st June, this will give the winner enough time to organize flights and accommodation.

If you do not win, please do not worry you can still save 25% on the entry price by using our Discount Code: FusionReactor25

Why debugging in production is so tempting?

In one of my first jobs, one of the tasks I had was to fix a bug which used to occur from time to time in a very complex production system. That is easy! – I thought. I will reproduce the same situation in my development environment, find the broken line, implement a quick fix and it’s done! However, it turned out that reproducing exactly the same scenario which occurred in production, was not possible – so after some time, I gave up this idea. I was forced to spend days analyzing logs and trying to correlate many different events, to come up with an idea of what might have happened.   Soon, I realized that it’s as tedious as looking for a needle in a haystack.  A couple of fruitless days later, I came to the conclusion that I would need to add more logging here and there and wait a couple of days or even months to see if the bug occurred again. Then I thought that hunting bugs in production is somehow crude, compared to the sophisticated tools we have when developing an application. You’re implementing a new feature and seeing that the result of what your service returned is not what you had expected? You just put a few breakpoints in and click the Debug button! A few moments later, you know exactly what happened.  Wouldn’t it be awesome to do the same in a production environment?

Why debugging in production is so hard?

Wait a second! – you might have thought. But don’t we have the remote debugging features in most of the modern IDEs? Couldn’t we just connect to the running production application and debug it as we do from our local environment?   While it’s possible, another problem arises: most of our business applications handle many requests per second. There is no easy way to control breakpoints firing everywhere when your application is being remotely debugged.   As you can imagine, we don’t want to block all of our users from using our application when we decide to debug it. More often than not, we also can’t just force our application to reproduce the bug which happened yesterday – sometimes the only way to do it is to wait until it occurs again to one of our users. Thus, keeping a remote debug session in production, without a strict control of how breakpoints fire, is like putting landmines in the forest and inviting our users to run through it.

A better and above all – safer way

FusionReactor is an Application Performance Monitor, which comes with many advanced capabilities which you wouldn’t normally expect to find in monitoring solution.   One of these, is the production debugger, designed to allow you to get low-level debug information from your production runtime environment – without affecting performance and without the issues and obstacles you would have with a traditional debugger.

One of the main issues, you would be faced with, using a traditional debugger – is that, once a breakpoint is set, it would fire for any thread which crosses that point in the code.   FusionReactor overcomes this, by having a range of techniques to control the way a breakpoint should fire.   For example, it can limit the number of times (threads) that a given breakpoint will trigger – which solves the problem of impacting too many users.   Need more ways to control it ? We can even configure a breakpoint to fire for a user from a specific IP address (session), or when a specific variable matches a value or when a specific exception takes place !   However, what if a breakpoint triggers at night when nobody from our team is watching?  The folks at FusionReactor have thought of this and the product allows you to define thread pause timeouts so if you would not intercept a paused thread within a specific time then the debugger will release the lock and allow thread execution to continue. When used with the thread limits this reduces the possible impact to one thread only – and only for n seconds, minutes or hours…

Another great benefit, is that FusionReactor can send out an email with the stack-trace and variables at the point that the trigger fires. This gives you a very flexible and unobtrusive way to get notified with plenty of information to make debugging easier than ever before.  It’s a LOT better and simpler than adding debug/log file code into your production applications!   For one thing – you don’t have to remember to remove the debug code in your production application !

If that’s not enough to convince you that debugging in production doesn’t have to be cumbersome, FusionReactor is shipped with a fully integrated IDE-style debugger which runs directly in your browser – no need to install additional fat clients to start remote debugging.   Everything is built in and ready to go.

Summary

What I described here is not a vision of the future it’s available now and it’s safe, secure and simple to use.   The alternative is (of course) good old log files, which we all know have their limitations.   I don’t see any reason to spend countless hours digging through log files hoping to find the needle in the haystack – using logs are “archaic techniques” and I think maybe it’s time to finally start using tools for the 21st century 😉

INTERGRAL SOFTWARE END USER LICENSE AGREEMENT (“EULA”)

THE FOLLOWING EULA APPLIES TO FUSIONREACTOR VERSION 7 AND ITS MINOR/MICRO UPDATES ONLY

IMPORTANT: CAREFULLY READ THE FOLLOWING LICENSE AGREEMENT. THIS
END USER LICENSE AGREEMENT (“EULA”) IS A LEGAL AGREEMENT BETWEEN
YOU (EITHER AN INDIVIDUAL OR, IF PURCHASED OR OTHERWISE ACQUIRED
BY OR FOR AN ENTITY, AN ENTITY) AND INTERGRAL. YOU ACCEPT AND
AGREE TO BE BOUND BY THE TERMS OF THIS AGREEMENT BY SELECTING THE
“ACCEPT” OPTION OR DOWNLOADING THE SOFTWARE OR BY INSTALLING,
USING, OR COPYING THE SOFTWARE. IF YOU DO NOT AGREE TO BE BOUND
BY THESE TERMS THEN DO NOT INSTALL, COPY, DOWNLOAD OR OTHERWISE
USE THE SOFTWARE. THIS EULA SHALL APPLY ONLY TO THE SOFTWARE
SUPPLIED BY INTERGRAL HEREWITH REGARDLESS OF WHETHER OTHER
SOFTWARE IS REFERRED TO OR DESCRIBED HEREIN.

Definitions

“INTERGRAL” means INTERGRAL Information Solutions GmbH and its
licensors, if any.

“Developer Version” means a version or edition of the Software to
be used only for design, development and evaluation purposes.

“Free Version” means a version or edition of the Software that
may have limited features, and may stop operating after a given
amount of time.

“Trial Version” means a version or edition of the Software to be
used only for review, demonstration and evaluation purposes. The
Trial Version may have limited features, and may stop operating
after a given amount of time.

“Full Version” means a version or edition of the Software that is
not a Developer Version nor a Trial Version of the Software nor a
Free Version of the Software.

“Accessible Code” means source code that is unprotected and
accessible.

“Protected Code” means any source code that is protected against
access by INTERGRAL or a third party and is not accessible under
this EULA.

“Fees” mean all fees and expenses payable by the Licensee to
INTERGRAL in acquiring the Software and as applicable any
Subscription or User Licenses.

“Software” means only the INTERGRAL software and third party
software programs, in each case, supplied by INTERGRAL together
with this EULA, including its Accessible Code and Protected Code
and any corresponding documentation, online or electronic
documentation, printed materials, and associated media. Any
updates to such Software that you are entitled to receive and
that have been provided to you by INTERGRAL shall also mean
Software for purposes of this Agreement.

License Grant
The Software is subject to the terms and conditions of this
Agreement. INTERGRAL hereby grants, and you accept, the right and
license to install and use the Software provided that you do not
use, copy, or install the Software on more than the number of
computers permitted by license, or permit the use, copying, or
installation by more users on more computers than the number
permitted by the license.

You may make one copy of the Software in machine-readable form
solely for backup purposes. You must reproduce on any such copy
all copyright notices and any other proprietary legends on the
original copy of the Software.

The Software is protected by copyright laws and international
copyright treaties, as well as other intellectual property laws
and treaties. INTERGRAL reserves all intellectual property
rights, including copyrights and trademark rights. Your license
rights under this EULA are non-exclusive.

License Restrictions

Restrictions on Use
Other than as expressly set forth above, you may not make or
distribute copies of the Software, or electronically transfer the
Software from one computer to another or over a network.

You may not decompile, “reverse-engineer”, disassemble, or
otherwise attempt to derive the source code for the Software.

You may not use the database portion of the Software in
connection with any software other than the Software.

You shall not (A) in the aggregate, install or use more than one
copy of the Trial Version of the Software, (B) download the Trial
Version of the Software under more than one username, (C) alter
the contents of a hard drive or computer system to enable the use
of the Trial Version of the Software for an aggregate period in
excess of the trial period for one license to such Trial Version,
(D) disclose the results of software performance benchmarks
obtained using the Trial Version to any third party without
INTERGRAL’s prior written consent, (E) use the Trial Version for
any application deployment or ultimate production purpose, or (F)
use the Trial Version of the Software for a purpose other than
the sole purpose of determining whether to purchase a license to
a Full Version or a Developer Version of the software; provided,
however, notwithstanding the foregoing, you are strictly
prohibited from installing or using the Trial Version of the
Software for any commercial training purpose.

You shall not use the Developer Version for any application
deployment in a live or stand-by production environment or
staging environment, in each case, including, without limitation,
in any environment accessed by application end- users, including,
but not limited to, servers, workstations, kiosks, and mobile
computers.

You shall not use the Software to develop any application having
the same primary function as the Software.

Subject to the terms and conditions of this Agreement you must at
all times ensure that the Software is not used for rental,
timesharing, subscription service, hosting, outsourcing, or as
part of a service or consulting practice to a third-party,
without first obtaining the express written consent of INTERGRAL.

Restrictions on Alteration
You may not alter, merge, adapt, translate or modify the Software
or create any derivative work of the Software or its accompanying
documentation. Derivative works include but are not limited to
translations. You may not alter any files or libraries in any
portion of the Software. You may not reproduce the database
portion or create any tables or reports relating to the database
portion.

Restrictions on Copying
You may not copy any part of the Software except to the extent
that licensed use inherently demands the creation of a temporary
copy stored in computer memory and not permanently affixed on
storage medium. You may make one archival copy which must be
stored on a medium other than a computer hard drive.

Restrictions on Transfer
Without first obtaining the express written consent of INTERGRAL,
you may not assign your rights and obligations under this
Agreement, or redistribute, encumber, sell, rent, lease,
sublicense, or otherwise transfer your rights to the Software.
You may not sell or transfer any Software purchased under a
volume discount. You may not sell or transfer any Trial Version
or Free Version of the Software.

Permitted Fixes
Notwithstanding anything else in this EULA but subject to the
terms and conditions contained herein, the Licensee is permitted
to modify the Accessible Code in the Software to develop bug
fixes, customizations or additional features solely for their
internal purposes of using the Software.

Upgrades
If this copy of the Software is an upgrade from an earlier
version of the Software, it is provided to you on a license
exchange basis. You agree by your installation and use of this
copy of the Software to voluntarily terminate your EULA with
respect to such prior license to the Software and that you will
not continue to install or use such prior license of the Software
or transfer it to another person or entity.

Ownership
The foregoing grants of rights give you limited license to use
the Software. Except as expressly provided in this Agreement,
INTERGRAL remain the owner of all right, title and interest in
the Software. All rights not specifically granted in this EULA
are reserved by INTERGRAL.

Fees
The Licensee must pay all Fees by their due date. Failure to pay
Fees by the due date will result in the immediate termination of
the licenses granted under this EULA.

You are responsible for reviewing the Software’s website for
changes in the fees, including, but not limited to, subscription
fees and payment terms.

TERMINATION
Without prejudice to any other rights and in addition to any
other termination rights in this EULA, INTERGRAL may terminate
this EULA if the Licensee fails to comply with the terms and
conditions of this EULA. Immediately upon termination of a
license granted under this EULA, the Licensee must at its own
cost remove all copies of the Software including all Accessible
Code from its computer systems and provide INTERGRAL with written
certification that it has destroyed all copies of the Software
including all Accessible Code in its possession, custody or
control.

LIMITED WARRANTY AND DISCLAIMER
TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW IN THE
JURISDICTION IN WHICH THE SOFTWARE IS PROVIDED, INTERGRAL
PROVIDES THE SOFTWARE AS IS AND WITH ALL FAULTS, AND EXCEPT
OTHERWISE EXPRESSLY CONTAINED IN THE EULA, HEREBY DISCLAIM ALL
OTHER WARRANTIES AND CONDITIONS, WHETHER EXPRESS, IMPLIED OR
STATUTORY.

INTERGRAL MAKES NO WARRANTY THAT THE SOFTWARE WILL MEET YOUR
REQUIREMENTS OR OPERATE UNDER YOUR SPECIFIC CONDITIONS OF USE.
INTERGRAL MAKES NO WARRANTY THAT OPERATION OF THE SOFTWARE WILL
BE SECURE, ERROR FREE, OR FREE FROM INTERRUPTION. YOU MUST
DETERMINE WHETHER THE SOFTWARE SUFFICIENTLY MEETS YOUR
REQUIREMENTS FOR SECURITY AND UNINTERRUPTABILITY. YOU BEAR SOLE
RESPONSIBILITY AND ALL LIABILITY FOR ANY LOSS INCURRED DUE TO
FAILURE OF THE SOFTWARE TO MEET YOUR REQUIREMENTS. INTERGRAL WILL
NOT, UNDER ANY CIRCUMSTANCES, BE RESPONSIBLE OR LIABLE FOR THE
LOSS OF DATA ON ANY COMPUTER OR INFORMATION STORAGE DEVICE.
INTERGRAL DISCLAIM ALL OTHER WARRANTIES AND REPRESENTATIONS,
WHETHER EXPRESS, IMPLIED, OR OTHERWISE, INCLUDING THE WARRANTIES
OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. THE
SOFTWARE IS NOT DESIGNED, INTENDED OR LICENSED FOR USE IN
HAZARDOUS ENVIRONMENTS REQUIRING FAIL- SAFE CONTROLS, INCLUDING
WITHOUT LIMITATION, THE DESIGN, CONSTRUCTION, MAINTENANCE OR
OPERATION OF NUCLEAR FACILITIES, AIRCRAFT NAVIGATION OR
COMMUNICATION SYSTEMS, AIR TRAFFIC CONTROL, AND LIFE SUPPORT OR
WEAPONS SYSTEMS. INTERGRAL SPECIFICALLY DISCLAIMS ANY EXPRESS OR
IMPLIED WARRANTY OF FITNESS FOR SUCH PURPOSES.

INTERGRAL PROVIDES NO REMEDIES OR WARRANTIES, WHETHER EXPRESS OR
IMPLIED, FOR THE TRIAL VERSION AND THE FREE VERSION OF THE
SOFTWARE. THE TRIAL VERSION AND THE FREE VERSION OF THE SOFTWARE
ARE PROVIDED “AS IS” AND WITH ALL FAULTS AND HEREBY DISCLAIM ALL
OTHER WARRANTIES AND CONDITIONS, WHETHER EXPRESS, IMPLIED OR
STATUTORY.

UNLESS OTHERWISE EXPLICITLY AGREED TO IN WRITING BY INTERGRAL,
INTERGRAL MAKES NO OTHER WARRANTIES, EXPRESS OR IMPLIED, IN FACT
OR IN LAW, INCLUDING, BUT NOT LIMITED TO, ANY IMPLIED WARRANTIES
OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE OTHER THAN
AS SET FORTH IN THIS AGREEMENT OR IN THE LIMITED WARRANTY
DOCUMENTS PROVIDED WITH THE SOFTWARE.

IF APPLICABLE LAW REQUIRES ANY WARRANTIES WITH RESPECT TO THE
SOFTWARE, ALL SUCH WARRANTIES ARE LIMITED IN DURATION TO THIRTY
(30) DAYS FROM THE DATE OF DELIVERY.

NO ORAL OR WRITTEN INFORMATION OR ADVICE GIVEN BY INTERGRAL, ITS
DEALERS, DISTRIBUTORS, AGENTS OR EMPLOYEES SHALL CREATE A
WARRANTY OR IN ANY WAY INCREASE THE SCOPE OF ANY WARRANTY
PROVIDED HEREIN.

LIMITED REMEDY
YOUR REMEDY FOR A BREACH OF THIS AGREEMENT OR OF ANY WARRANTY
INCLUDED IN THIS AGREEMENT IS THE CORRECTION OR REPLACEMENT OF
THE SOFTWARE. SELECTION OF WHETHER TO CORRECT OR REPLACE SHALL BE
SOLELY AT THE DISCRETION OF INTERGRAL. INTERGRAL RESERVES THE
RIGHT TO SUBSTITUTE A FUNCTIONALLY EQUIVALENT COPY OF THE
SOFTWARE AS A REPLACEMENT. IF INTERGRAL IS UNABLE TO PROVIDE A
REPLACEMENT OR SUBSTITUTE SOFTWARE OR CORRECTIONS TO THE
SOFTWARE, YOUR SOLE ALTERNATE REMEDY SHALL BE A REFUND OF THE
PURCHASE PRICE FOR THE SOFTWARE EXCLUSIVE OF ANY COSTS FOR
SHIPPING AND HANDLING.

ANY CLAIM MUST BE MADE WITHIN THE APPLICABLE WARRANTY PERIOD. ALL
WARRANTIES COVER ONLY DEFECTS ARISING UNDER NORMAL USE AND DO NOT
INCLUDE MALFUNCTIONS OR FAILURE RESULTING FROM MISUSE, ABUSE,
NEGLECT, ALTERATION, MISAPPLICATION, PROBLEMS WITH DATA NETWORKS,
PROBLEMS WITH ELECTRICAL POWER, ACTS OF NATURE, UNUSUAL
TEMPERATURES OR HUMIDITY, IMPROPER INSTALLATION, DAMAGE TO MEDIA
OR DAMAGE DETERMINED BY INTERGRAL TO HAVE BEEN CAUSED BY YOU. ALL
LIMITED WARRANTIES ON THE SOFTWARE ARE GRANTED ONLY TO YOU AND
ARE NON- TRANSFERABLE.

YOU AGREE TO INDEMNIFY AND HOLD INTERGRAL HARMLESS FROM ALL
CLAIMS, JUDGMENTS, LIABILITIES, EXPENSES, OR COSTS ARISING FROM
YOUR BREACH OF THIS AGREEMENT AND/OR ACTS OR OMISSIONS. THIS
REMEDY IS THE SOLE AND EXCLUSIVE REMEDY AVAILABLE TO YOU FOR
BREACH OF EXPRESS OR IMPLIED WARRANTIES WITH RESPECT TO THE
SOFTWARE AND RELATED DOCUMENTATION.

LIMITATION OF LIABILITY
UNDER NO CIRCUMSTANCES SHALL INTERGRAL, ITS DIRECTORS, OFFICERS,
EMPLOYEES AND AGENTS BE LIABLE TO YOU OR ANY OTHER PARTY FOR
INDIRECT, CONSEQUENTIAL, SPECIAL, INCIDENTAL, PUNITIVE, OR
EXEMPLARY DAMAGES OF ANY KIND (INCLUDING LOST REVENUES OR PROFITS
OR LOSS OF BUSINESS) RESULTING FROM THIS AGREEMENT, OR FROM THE
FURNISHING, PERFORMANCE, INSTALLATION, OR USE OF THE SOFTWARE,
WHETHER DUE TO A BREACH OF CONTRACT, BREACH OF WARRANTY, OR THE
NEGLIGENCE OF INTERGRAL OR ANY OTHER PARTY, EVEN IF INTERGRAL IS
ADVISED BEFOREHAND OF THE POSSIBILITY OF SUCH DAMAGES. TO THE
EXTENT THAT THE APPLICABLE JURISDICTION LIMITS INTERGRAL’S
ABILITY TO DISCLAIM ANY IMPLIED WARRANTIES, THIS DISCLAIMER SHALL
BE EFFECTIVE TO THE MAXIMUM EXTENT PERMITTED. INTERGRAL’S
LIABILITY UNDER THIS AGREEMENT WILL NOT, IN ANY EVENT, EXCEED THE
LICENSE FEES YOU PAID FOR THE SOFTWARE, IF ANY.

Indemnification
To the maximum extent permitted by law, you agree to defend,
indemnify and hold harmless INTERGRAL, its directors, officers,
employees and agents from and against any and all claims,
actions, suits or proceedings, as well as any and all losses,
liabilities, damages, costs and expenses (including reasonable
attorneys fees) arising out of or accruing from (a) your use of
the Software, (b) any application you develop on the Software
that infringes any copyright, trademark, trade secret, trade
dress, patent or other intellectual property right of any person
or defames any person or violates their rights of publicity or
privacy, and (c) any non- compliance by you with this License
Agreement.

Confidentiality
The Software contains trade secrets and proprietary know-how that
belong to INTERGRAL and it is being made available to you in
strict confidence. ANY USE OR DISCLOSURE OF THE SOFTWARE, OR OF
ITS ALGORITHMS, PROTOCOLS OR INTERFACES, OTHER THAN IN STRICT
ACCORDANCE WITH THIS LICENSE AGREEMENT, MAY BE ACTIONABLE AS A
VIOLATION OF OUR TRADE SECRET RIGHTS.

Publicity Rights
Licensee grants INTERGRAL the right to include Licensee as a
customer in Software promotional material. Licensee can deny
INTERGRAL this right at any time by submitting a written request
via email to SALES@INTERGRAL.COM, requesting to be excluded from
Software promotional material. Requests made after purchasing may
take thirty (30) calendar days to process.

Use of Decompiler
The Software includes decompiling functionality (“Decompiler”)
that enables reproducing source code from the original binary
code. You hereby acknowledge that the binary code and source code
may be protected by copyright, trademark and other laws
which may prohibit you from decompiling them and/or using
Decompiler. Before using Decompiler, you should make sure that
the decompilation is not prohibited by any applicable license
agreement of the application and/or original binary code
(except to the extent that you may be expressly permitted
under applicable law) or that you have obtained permission to
decompile the code from the copyright owner. Using Decompiler is
entirely optional. INTERGRAL does neither encourage nor condone
the use of the Decompiler and disclaims any liability for your
use of Decompiler in violation of applicable laws.

Use of Debugger
The Software includes debugging functionality (“Debugger”)
that enables debugging programs that may contain binary code,
source code and intellectual property protected by copyright,
trademark, patent and other laws which may prohibit you
from debugging them and/or using Debugger. Before using Debugger,
you should make sure that debugging is not prohibited by any
applicable license agreement of the program and/or original
binary code (except to the extent that you may be expressly
permitted under applicable law), or by other protecting laws,
or that you have obtained permission to debug the program from all
legal protection owner(s). Using Debugger is entirely optional.
INTERGRAL does neither encourage nor condone the use of the
Debugger and disclaims any liability for your use of Debugger
in violation of applicable laws.

FusionReactor Cloud Service
Use of the FusionReactor Cloud Service is governed by the terms
and conditions that accompanies or is included with the
FusionReactor Cloud Service. By accepting this EULA, you are also
accepting the additional terms and conditions when using the
Software with the FusionReactor Cloud Service.

Third Party Software
Any software provided along with the Software that is associated
with a separate license agreement is licensed to you under the
terms of that license agreement. This license does not apply to
those portions of the Software. Copies of these third party
licenses are included in all copies of the Software. By accepting
this EULA, you are also accepting the additional terms.

RESTRICTIONS
United States: If the Software is acquired by the Licensee in the
United States, the Licensee acknowledges: (a) the Software is
subject to U.S. export jurisdiction and agrees to comply with all
applicable international and national laws that apply to the
Software, including the U.S. Export Administration Regulations,
as well as end-user, end- use, and destination restrictions
issued by U.S. and other governments and notwithstanding the
above; and (b) the provisions of the USA Uniform Computer
Information Transaction Act do not apply to this EULA.

CONSENT TO COLLECT AND USE OF PERSONAL INFORMATION
You acknowledge and agree that we may collect, transfer, process
and store certain information (“Information”) collected by the
Software, including but not limited to:

(i) IP addresses of the system on which Software is installed;
(ii) MAC Addresses of the system on which Software is installed;
(iii) Email addresses configured for use with the Software;
(iv) Information about the system on which Software is installed,
for example: OS version; processor architecture;
operational status;
(v) Information about the application software on which Software
is installed, for example: application software version;
application software configuration; JVM configuration;
operational status;
(vi) Information regarding how you are using the Software,
for example: version of the Software; statistical information
about the Software; configuration of the Software.

INTERGRAL and its Data Processors may use Information subject to
applicable laws in order to license and bill for the Software and
services, improve its products and services or to provide
products or services (if any) to you. Such uses include, but are
not limited to:

(a) licensing and billing for the Software and services usage;
(b) administering the functionality of the Software and services;
(c) to improve, update or upgrade the Software and services;
(d) improving, developing and enhancing the products and
services of INTERGRAL;
(e) provide product support, products and services to you;
(f) complying with applicable laws or regulations.

INTERGRAL will not intentionally use Information to personally
identify the owner or user of the Software without your knowledge
or consent.

Any use of Information will be in accordance with the privacy
policies of INTERGRAL.

Information may be processed, stored or transferred to INTERGRAL
and its Data Processors which may be located in countries
outside of your country of residence.

INTERGRAL will use reasonable efforts to take appropriate
technical and organizational measures to prevent unauthorized
access to or disclosure of Information, but does not warrant it
will eliminate all risk of misuse of Information.

General: The export of the Software from the country of original
purchase may be subject to control or restriction by applicable
local law. Licensee is solely responsible for determining the
existence and application of any such law to any proposed export
and for obtaining any needed authorization. Licensee agrees not
to export the Software from any country in violation of
applicable legal restrictions on such export.

Governing Law, Jurisdiction and Costs
This EULA is governed by the laws of Baden-Wuerttemberg, Germany
without regard to Baden-Wuerttemberg’s conflict or choice of law
provisions. Exclusive jurisdiction and place of performance is
Boeblingen, Germany, as long as permitted by applicable law. The
United Nations Convention for the International Sale of Goods
shall not apply.

Changes to the License Agreement
INTERGRAL may make changes to the License Agreement as it
distributes new versions of the Software. When these changes are
made, INTERGRAL will make a new version of the License Agreement
available on the website where the Software is made available.

Entire Agreement and Severability
This EULA is the entire agreement between INTERGRAL and you, and
supersedes all prior or contemporaneous agreements or
understandings, whether oral or written any other communications
or advertising with respect to the Software; this EULA may be
modified only by written agreement signed by authorized
representatives of both you and INTERGRAL. No INTERGRAL dealer or
agent is authorized to make any amendment to this EULA.

If any provision of this EULA shall be held to be invalid or
unenforceable, the remainder of this EULA shall remain in full
force and effect and an enforceable term will be substituted
reflecting our intent as closely as possible. All rights not
expressly granted in this agreement are retained by INTERGRAL.
To the extent any express or implied restrictions are not
permitted by applicable laws, these express or implied
restrictions shall remain in force and effect to the maximum
extent permitted by such applicable laws. The failure or delay of
INTERGRAL to exercise any of its rights under this EULA or upon
any breach of this EULA shall not be deemed a waiver of those
rights or of the breach. You agree that any varying or additional
terms contained in any purchase order or other written notification
or document issued by you in relation to the Software licensed
hereunder shall be of no effect.

INTERGRAL and other trademarks contained in the Software are
trademarks or registered trademarks of INTERGRAL Information
Solutions GmbH. Third party trademarks, trade names, product
names and logos may be the trademarks or registered trademarks of
their respective owners. You may not remove or alter any
trademark, trade names, product names, logo, copyright or other
proprietary notices, legends, symbols or labels in the Software.
This EULA does not authorize you to use INTERGRAL’s or its
licensors’ names or any of their respective trademarks.

The buffer pool space is located outside of the garbage collector-managed memory. It’s a way to allocate native off-heap memory.  What’s the benefit of using buffer pools? To answer this question, let’s firstly learn what byte buffers are.

Byte Buffer

Non-Direct Buffer

java.nio package comes with the Bytebuffer class. It allows us to allocate both direct and non-direct byte buffers. Their is nothing special about non-direct byte buffers – they are an implementation of HeapByteBuffer created by ByteBuffer.allocate() and ByteBuffer.wrap() factory methods. As the name of the class suggests, these are on-heap byte buffers. Wouldn’t it be easier to allocate all the buffers on the Java heap space then?   Why would anyone need to allocate something in a native memory? To answer this question, we need to understand how operating systems perform I/O operations.   Any read or write instructions are executed on memory areas which are contiguous sequence of bytes. So does byte[] occupy a contiguous space on the heap? While technically it makes sense, the JVM specification does not have such guarantees. What’s more interesting, the specification doesn’t even guarantee that heap space will be contiguous itself! Although it seems to be rather unlikely that JVM will place a one-dimensional array of primitives in different places in memory, byte array from Java heap space cannot be used in native I/O operations directly. It has to be copied to a native memory before every I/O, which of course, leads to obvious inefficiencies. For this reason, a direct buffer was introduced.

Direct Buffer

A direct buffer is a chunk of native memory shared with Java from which you can perform a direct read.

An instance of DirectByteBuffer can be created using the ByteBuffer.allocateDirect() factory method. Byte buffers are the most efficient way to perform I/O operations and thus, they are used in many libraries and frameworks – for example in Netty.

Memory Mapped Buffer

A direct byte buffer may also be created by mapping a region of a file directly into memory. In other words, we can load a region of a file to a particular native memory region that can be accessed later.  As you can imagine, it can give a significant performance boost if we have the requirement to read the content of a file multiple times. Thanks to memory mapped files, subsequent reads will use the content of the file from the memory, instead of loading the data from the disc every time it’s needed. MappedByteBuffer can be created via the FileChannel.map() method.

An additional advantage of memory mapped files, is that the OS can flush the buffer directly to the disk when the system is shutting down. Moreover, the OS can lock a mapped portion of the file from other processes on the machine.

Allocation is Expensive

One of the problems with direct buffers is that it’s expensive to allocate them. Regardless of the size of the buffer, calling Buffer.allocateDirect() is a relatively slow operation. It is therefore more efficient to either use direct buffers for large and long-lived buffers or create one large buffer, slice off portions on demand, and return them to be re-used when they are no longer needed.   A potential problem with slicing may occur when slices are not always the same size. The initial large byte buffer can become fragmented when allocating and freeing objects of different size.  Unlike Java heap, direct byte buffer cannot be compacted, because it’s not a target for the garbage collector.

Monitoring the Usage of Buffer Pools

If you’re interested in the amount of direct or mapped byte buffers used by your application, then you can easily monitor them using FusionReactor.   FusionReactor provides a break-down of all the different memory spaces.   Simply navigate to Resources and then Direct – Buffer Pools.

By default, the Direct Buffer Pool graph is displayed. You can switch to the Mapped Buffer Pool by clicking on a drop-down in the top right corner. Java will grow those pools as required so the fact that Direct Memory Used covers Direct Capacity on the graph above, means that all buffer memory allocated so far is in use.

Please note – you can limit the amount of direct byte buffer space that an application can allocate, by using -XX:MaxDirectMemorySize=N flag.   Although this is possible, you would need a very good reason to do so.

If you ever had some serious issues with the performance of your Java application, most probably you know how valuable thread profiling can be. But do you know which profiler you should use?

There are two basic techniques used by profilers – sampling and instrumentation.

Sampling Profilers

A sampling profiler involves periodically asking the JVM for the current point of execution of all currently alive (active) threads. This type of profiler carries the least amount of overhead. This is important because introducing heavy measurement/instrumentation into the application can change the performance characteristics significantly.  Using a sampling technique, we get a snapshot of the next stack trace when the timer fires. So the profiler looks at each thread and determines which method the thread is executing at that moment. As there are gaps between consecutive measurements, sampling profiler achieves a trade-off between the level of accuracy obtained vs the overhead involved in actually taking the measurement, This is illustrated in the following figure:

As you can see, the thread spent most of its time in save method and a little bit in read method. If the sampling happens only when the thread is in a save method (more probable as this method dominates), the profiler will report that the thread spent 100% of its time in save method, which is of course not accurate.

A rather logical way to minimize this sampling error is to reduce the time interval between sampling and increase the profiling time. However, as we discussed earlier, this solution might impact the performance characteristics of the application, so a balance is the key.

Instrumented Profilers

Instrumented profilers introduce a much larger performance overhead into the application. This method usually involves injecting bytecode into the classes for the purpose of profiling.  This approach involves a higher performance impact, but generates a more accurate measurement when compared to the result from the sampling profiler. Another problem which may arise as a result of the way an instrumented profiler modifies the bytecode is the following; as you may know, JIT compiler inlines small methods. Because the instrumentation introduced by the profiler, some small methods might not be eligible to be in-lined anymore. It can have a serious impact on application performance. If you decide to use instrumented profilers, make sure that you instrument only a small section of the code.

Production Code Profiler in FusionReactor

Profiling in a development environment is easy. However, it might not be enough. When dealing with production data, we are exposed to a different scale and thus, we might observe different bottlenecks in our application. That’s why profiling in production is so important. As already mentioned, both sampling and instrumented profilers have their pros and cons. If you want to profile in a production environment, a low overhead sampling profiler seems to be a better choice. FusionReactor’s Production Code Profiler will help you identify bottlenecks in your production environment with a very low sampling overhead. The really cool thing about this profiler is that it can be configured in a way that it will automatically profile your application if it detects a long-running request or thread. What is a long-running request? It’s up to you to define, but 3 seconds is the default value. If you monitor some sort of latency-sensitive application, then you might want to decrease this value. Similarly, if your application performs some time-consuming computations, then most probably you don’t want to be notified all the time and increasing Minimum Transaction Time will be necessary. Taking a stacktrace snapshot takes as little as 1 ms per sample. With a default sampling rate of 200 ms, you can usually get some meaningful results from just 5 samples.

FusionReactor’s thread profiling output looks like this:

On the left-hand side, you can see the percentage of time the particular method took. On the right -hand side, time in seconds is shown. As you can see in the output, logging in lowCostSearchEngineClient class can be a potential bottleneck in this application. Thanks to the bundled de-compiler, you can instantly check the method by simply clicking on it.

It’s not always easy to pinpoint a performance issue in a running application, but profilers are usually good estimators. Sampling profilers such as FusionReactor’s Profiler, are perfect tools to use them in a production environment.