FusionReactor Blog - News and expert analysis for Java APM users.

In this video Brad Wood of Ortus Solutions introduces you to FusionReactor Cloud and Docker Swarm

“We’ve made a new screencast that shows an introduction to FusionReactor Cloud, a new way to monitor any number of servers in a consolidated interface that can scale up and down with your infrastructure. Not only do new servers automatically register and deregister themselves with the Cloud dashboard, but it supports a pay-for-what-you-use model that prevents you from locking into a specific number of licenses.

This is a great fit for deployments like Docker Swarm where you can scale your service up and down from day to day. In this demo, I’ll show you how to add FusionReactor cloud easily into any CommandBox-based server and show how to deploy it to a Docker Swarm on our Ortus Docker image and then play around with it.” Brad Wood, Ortus Solutions (Follow him on Twitter)

The FusionReactor Blog was selected as one of the Top 30 ColdFusion Blogs by FeedSpot. The whole FusionReactor team are proud to be part of this list.

Have a look at some of the other top ColdFusion blogs featured in the list.

We also got this cool badge for being featured in the top 30 :

VisualVM Buffer pool – what is the buffer pool?

The buffer pool space is located outside of the garbage collector-managed memory. It’s a way to allocate native off-heap memory.  What’s the benefit of using buffer pools? To answer this question, let’s firstly learn what byte buffers are.

Byte Buffer

Non-Direct Buffer

java.nio package comes with the Bytebuffer class. It allows us to allocate both direct and non-direct byte buffers. There is nothing special about non-direct byte buffers – they are an implementation of HeapByteBuffer created by ByteBuffer.allocate() and ByteBuffer.wrap() factory methods. As the name of the class suggests, these are on-heap byte buffers. Wouldn’t it be easier to allocate all the buffers on the Java heap space then?   Why would anyone need to allocate something in a native memory? To answer this question, we need to understand how operating systems perform I/O operations.   Any read or write instructions are executed on memory areas which are the contiguous sequence of bytes. So does byte[] occupy a contiguous space on the heap? While technically it makes sense, the JVM specification does not have such guarantees. What’s more interesting, the specification doesn’t even guarantee that heap space will be contiguous itself! Although it seems to be rather unlikely that JVM will place a one-dimensional array of primitives in different places in memory, byte array from Java heap space cannot be used in native I/O operations directly. It has to be copied to a native memory before every I/O, which of course, leads to obvious inefficiencies. For this reason, a direct buffer was introduced.

Direct Buffer

A direct buffer is a chunk of native memory shared with Java from which you can perform a direct read.

An instance of DirectByteBuffer can be created using the ByteBuffer.allocateDirect() factory method. Byte buffers are the most efficient way to perform I/O operations and thus, they are used in many libraries and frameworks – for example in Netty.

Memory Mapped Buffer

A direct byte buffer may also be created by mapping a region of a file directly into memory. In other words, we can load a region of a file to a particular native memory region that can be accessed later.  As you can imagine, it can give a significant performance boost if we have the requirement to read the content of a file multiple times. Thanks to memory mapped files, subsequent reads will use the content of the file from the memory, instead of loading the data from the disc every time it’s needed. MappedByteBuffer can be created via the FileChannel.map() method.

An additional advantage of memory mapped files is that the OS can flush the buffer directly to the disk when the system is shutting down. Moreover, the OS can lock a mapped portion of the file from other processes on the machine.

Allocation is Expensive

One of the problems with direct buffers is that it’s expensive to allocate them. Regardless of the size of the buffer, calling Buffer.allocateDirect() is a relatively slow operation. It is, therefore, more efficient to either use direct buffers for large and long-lived buffers or create one large buffer, slice off portions on demand, and return them to be re-used when they are no longer needed.   A potential problem with slicing may occur when slices are not always the same size. The initial large byte buffer can become fragmented when allocating and freeing objects of different size.  Unlike Java heap, direct byte buffer cannot be compacted, because it’s not a target for the garbage collector.

Monitoring Buffer Pools

If you’re interested in the amount of direct or mapped byte buffers used by your application, then you can easily monitor them using FusionReactor.   FusionReactor provides a break-down of all the different memory spaces.   Simply navigate to Resources and then Direct – Buffer Pools.

VisualVM Buffer pool

By default, the Direct Buffer Pool graph is displayed. You can switch to the Mapped Buffer Pool by clicking on a drop-down in the top right corner. Java will grow those pools as required so the fact that Direct Memory Used covers Direct Capacity on the graph above, means that all buffer memory allocated so far is in use.

Please note – you can limit the amount of direct byte buffer space that an application can allocate, by using -XX:MaxDirectMemorySize=N flag.   Although this is possible, you would need a very good reason to do so.

Try FusionReactor for VisualVM Buffer pool tools

Start Free Trial Compare Editions Pricing

Monitoring Buffer Pools – what is the buffer pool?

The buffer pool space is located outside of the garbage collector-managed memory. It’s a way to allocate native off-heap memory.  What’s the benefit of using buffer pools? To answer this question, let’s firstly learn what byte buffers are.

Byte Buffer

Non-Direct Buffer

java.nio package comes with the Bytebuffer class. It allows us to allocate both direct and non-direct byte buffers. There is nothing special about non-direct byte buffers – they are an implementation of HeapByteBuffer created by ByteBuffer.allocate() and ByteBuffer.wrap() factory methods. As the name of the class suggests, these are on-heap byte buffers. Wouldn’t it be easier to allocate all the buffers on the Java heap space then?   Why would anyone need to allocate something in a native memory? To answer this question, we need to understand how operating systems perform I/O operations.   Any read or write instructions are executed on memory areas which are the contiguous sequence of bytes. So does byte[] occupy a contiguous space on the heap? While technically it makes sense, the JVM specification does not have such guarantees. What’s more interesting, the specification doesn’t even guarantee that heap space will be contiguous itself! Although it seems to be rather unlikely that JVM will place a one-dimensional array of primitives in different places in memory, byte array from Java heap space cannot be used in native I/O operations directly. It has to be copied to a native memory before every I/O, which of course, leads to obvious inefficiencies. For this reason, a direct buffer was introduced.

Direct Buffer

A direct buffer is a chunk of native memory shared with Java from which you can perform a direct read.

An instance of DirectByteBuffer can be created using the ByteBuffer.allocateDirect() factory method. Byte buffers are the most efficient way to perform I/O operations and thus, they are used in many libraries and frameworks – for example in Netty.

Memory Mapped Buffer

A direct byte buffer may also be created by mapping a region of a file directly into memory. In other words, we can load a region of a file to a particular native memory region that can be accessed later.  As you can imagine, it can give a significant performance boost if we have the requirement to read the content of a file multiple times. Thanks to memory mapped files, subsequent reads will use the content of the file from the memory, instead of loading the data from the disc every time it’s needed. MappedByteBuffer can be created via the FileChannel.map() method.

An additional advantage of memory mapped files is that the OS can flush the buffer directly to the disk when the system is shutting down. Moreover, the OS can lock a mapped portion of the file from other processes on the machine.

Allocation is Expensive

One of the problems with direct buffers is that it’s expensive to allocate them. Regardless of the size of the buffer, calling Buffer.allocateDirect() is a relatively slow operation. It is, therefore, more efficient to either use direct buffers for large and long-lived buffers or create one large buffer, slice off portions on demand, and return them to be re-used when they are no longer needed.   A potential problem with slicing may occur when slices are not always the same size. The initial large byte buffer can become fragmented when allocating and freeing objects of different size.  Unlike Java heap, direct byte buffer cannot be compacted, because it’s not a target for the garbage collector.

Monitoring Buffer Pools

If you’re interested in the amount of direct or mapped byte buffers used by your application, then you can easily monitor them using FusionReactor.   FusionReactor provides a break-down of all the different memory spaces.   Simply navigate to Resources and then Direct – Buffer Pools.

By default, the Direct Buffer Pool graph is displayed. You can switch to the Mapped Buffer Pool by clicking on a drop-down in the top right corner. Java will grow those pools as required so the fact that Direct Memory Used covers Direct Capacity on the graph above, means that all buffer memory allocated so far is in use.

Please note – you can limit the amount of direct byte buffer space that an application can allocate, by using -XX:MaxDirectMemorySize=N flag.   Although this is possible, you would need a very good reason to do so.

Try FusionReactor for Monitoring Buffer Pools

Start Free Trial Compare Editions Pricing

The FusionTeam will once again be PLATINUM sponsors of this great event – make sure you join us and the Adobe ColdFusion Team in Las Vegas!

We will be showing off the new major release of FusionReactor, which is FR 7 as well as FusionReactor CLOUD.

To celebrate this, we are giving away a free ticket for the Adobe ColdFusion Summit.

The Prize is for one ColdFusion Summit registration which Includes:
– Admission to all keynote and breakout sessions
– Sponsor Pavilion.
– Attendee appreciation event
– Conference meals.

There are Two ways to enter the draw:

1) ON TWITTER : Retweet this tweet and follow us it is that simple! If you are already following us then just retweet this.

OR (You can do both to double your chances of winning).

2) ON FACEBOOK : Just Like us on Facebook – and share this post with your friends

The winner will be announced on : Friday 27th October

If you do not win, or you just want to sign up right away, you can get a reduced price ticket for just $249 by using our registration code: CF17OFFER249 when you purchase a ticket here

Understanding the Memory Mapped Buffer

To understand the memory mapped buffer we first need to understand the Java buffer pool. The Java buffer pool space is located outside of the garbage collector-managed memory. It’s a way to allocate native off-heap memory.  What’s the benefit of using a Java buffer pool? To answer this question, let’s firstly learn what byte buffers are.

Java Byte Buffer

Non-Direct Buffer

java.nio package comes with the Bytebuffer class. It allows us to allocate both direct and non-direct byte buffers. Their is nothing special about non-direct byte buffers – they are an implementation of HeapByteBuffer created by ByteBuffer.allocate() and ByteBuffer.wrap() factory methods. As the name of the class suggests, these are on-heap byte buffers. Wouldn’t it be easier to allocate all the buffers on the Java heap space then?   Why would anyone need to allocate something in a native memory? To answer this question, we need to understand how operating systems perform I/O operations.   Any read or write instructions are executed on memory areas which are contiguous sequence of bytes. So does byte[] occupy a contiguous space on the heap? While technically it makes sense, the JVM specification does not have such guarantees. What’s more interesting, the specification doesn’t even guarantee that heap space will be contiguous itself! Although it seems to be rather unlikely that JVM will place a one-dimensional array of primitives in different places in memory, byte array from Java heap space cannot be used in native I/O operations directly. It has to be copied to a native memory before every I/O, which of course, leads to obvious inefficiencies. For this reason, a direct buffer was introduced.

Java Direct Buffer

A direct buffer is a chunk of native memory shared with Java from which you can perform a direct read.

An instance of DirectByteBuffer can be created using the ByteBuffer.allocateDirect() factory method. Byte buffers are the most efficient way to perform I/O operations and thus, they are used in many libraries and frameworks – for example in Netty.

Memory Mapped Buffer

A direct byte buffer may also be created by mapping a region of a file directly into memory. In other words, we can load a region of a file to a particular native memory region that can be accessed later.  As you can imagine, it can give a significant performance boost if we have the requirement to read the content of a file multiple times. Thanks to memory mapped files, subsequent reads will use the content of the file from the memory, instead of loading the data from the disc every time it’s needed. MappedByteBuffer can be created via the FileChannel.map() method.

An additional advantage of memory mapped files, is that the OS can flush the buffer directly to the disk when the system is shutting down. Moreover, the OS can lock a mapped portion of the file from other processes on the machine.

Allocation is Expensive

One of the problems with direct buffers is that it’s expensive to allocate them. Regardless of the size of the buffer, calling Buffer.allocateDirect() is a relatively slow operation. It is therefore more efficient to either use direct buffers for large and long-lived buffers or create one large buffer, slice off portions on demand, and return them to be re-used when they are no longer needed.   A potential problem with slicing may occur when slices are not always the same size. The initial large Java byte buffer can become fragmented when allocating and freeing objects of different size.  Unlike Java heap, direct byte buffer cannot be compacted, because it’s not a target for the garbage collector-managed memory.

Monitoring the Usage of a Java Buffer Pool

If you’re interested in the amount of direct or mapped byte buffers used by your application, then you can easily monitor them using FusionReactor.   FusionReactor provides a break-down of all the different memory spaces.   Simply navigate to Resources and then Direct – Buffer Pools.

By default, the Java Direct Buffer Pool graph is displayed. You can switch to the Mapped Buffer Pool by clicking on a drop-down in the top right corner. Java will grow those pools as required so the fact that Direct Memory Used covers Direct Capacity on the graph above, means that all buffer memory allocated so far is in use.

Please note – you can limit the amount of direct Java byte buffer space that an application can allocate, by using -XX:MaxDirectMemorySize=N flag.   Although this is possible, you would need a very good reason to do so.

Understanding  Java Non-Direct Buffer

The Java buffer pool space is located outside of the garbage collector-managed memory. It’s a way to allocate native off-heap memory.  What’s the benefit of using a Java buffer pool? To answer this question, let’s firstly learn what byte buffers are.

Byte Buffer

Non-Direct Buffer

java.nio package comes with the Bytebuffer class. It allows us to allocate both direct and non-direct byte buffers. Their is nothing special about non-direct byte buffers – they are an implementation of HeapByteBuffer created by ByteBuffer.allocate() and ByteBuffer.wrap() factory methods. As the name of the class suggests, these are on-heap byte buffers. Wouldn’t it be easier to allocate all the buffers on the Java heap space then?   Why would anyone need to allocate something in a native memory? To answer this question, we need to understand how operating systems perform I/O operations.   Any read or write instructions are executed on memory areas which are contiguous sequence of bytes. So does byte[] occupy a contiguous space on the heap? While technically it makes sense, the JVM specification does not have such guarantees. What’s more interesting, the specification doesn’t even guarantee that heap space will be contiguous itself! Although it seems to be rather unlikely that JVM will place a one-dimensional array of primitives in different places in memory, byte array from Java heap space cannot be used in native I/O operations directly. It has to be copied to a native memory before every I/O, which of course, leads to obvious inefficiencies. For this reason, a direct buffer was introduced.

Direct Buffer

A direct buffer is a chunk of native memory shared with Java from which you can perform a direct read.

An instance of DirectByteBuffer can be created using the ByteBuffer.allocateDirect() factory method. Byte buffers are the most efficient way to perform I/O operations and thus, they are used in many libraries and frameworks – for example in Netty.

Memory Mapped Buffer

A direct byte buffer may also be created by mapping a region of a file directly into memory. In other words, we can load a region of a file to a particular native memory region that can be accessed later.  As you can imagine, it can give a significant performance boost if we have the requirement to read the content of a file multiple times. Thanks to memory mapped files, subsequent reads will use the content of the file from the memory, instead of loading the data from the disc every time it’s needed. MappedByteBuffer can be created via the FileChannel.map() method.

An additional advantage of memory mapped files, is that the OS can flush the buffer directly to the disk when the system is shutting down. Moreover, the OS can lock a mapped portion of the file from other processes on the machine.

Allocation is Expensive

One of the problems with direct buffers is that it’s expensive to allocate them. Regardless of the size of the buffer, calling Buffer.allocateDirect() is a relatively slow operation. It is therefore more efficient to either use direct buffers for large and long-lived buffers or create one large buffer, slice off portions on demand, and return them to be re-used when they are no longer needed.   A potential problem with slicing may occur when slices are not always the same size. The initial large byte buffer can become fragmented when allocating and freeing objects of different size.  Unlike Java heap, direct byte buffer cannot be compacted, because it’s not a target for the garbage collector-managed memory.

Monitoring the Usage of a Java Buffer Pool

If you’re interested in the amount of direct or mapped byte buffers used by your application, then you can easily monitor them using FusionReactor.   FusionReactor provides a break-down of all the different memory spaces.   Simply navigate to Resources and then Direct – Buffer Pools.

By default, the Direct Buffer Pool graph is displayed. You can switch to the Mapped Buffer Pool by clicking on a drop-down in the top right corner. Java will grow those pools as required so the fact that Direct Memory Used covers Direct Capacity on the graph above, means that all buffer memory allocated so far is in use.

Please note – you can limit the amount of direct byte buffer space that an application can allocate, by using -XX:MaxDirectMemorySize=N flag.   Although this is possible, you would need a very good reason to do so.

Understanding garbage collector-managed memory

The Java buffer pool space is located outside of the garbage collector-managed memory. It’s a way to allocate native off-heap memory.  What’s the benefit of using a Java buffer pool? To answer this question, let’s firstly learn what byte buffers are.

Byte Buffer

Non-Direct Buffer

java.nio package comes with the Bytebuffer class. It allows us to allocate both direct and non-direct byte buffers. Their is nothing special about non-direct byte buffers – they are an implementation of HeapByteBuffer created by ByteBuffer.allocate() and ByteBuffer.wrap() factory methods. As the name of the class suggests, these are on-heap byte buffers. Wouldn’t it be easier to allocate all the buffers on the Java heap space then?   Why would anyone need to allocate something in a native memory? To answer this question, we need to understand how operating systems perform I/O operations.   Any read or write instructions are executed on memory areas which are contiguous sequence of bytes. So does byte[] occupy a contiguous space on the heap? While technically it makes sense, the JVM specification does not have such guarantees. What’s more interesting, the specification doesn’t even guarantee that heap space will be contiguous itself! Although it seems to be rather unlikely that JVM will place a one-dimensional array of primitives in different places in memory, byte array from Java heap space cannot be used in native I/O operations directly. It has to be copied to a native memory before every I/O, which of course, leads to obvious inefficiencies. For this reason, a direct buffer was introduced.

Direct Buffer

A direct buffer is a chunk of native memory shared with Java from which you can perform a direct read.

An instance of DirectByteBuffer can be created using the ByteBuffer.allocateDirect() factory method. Byte buffers are the most efficient way to perform I/O operations and thus, they are used in many libraries and frameworks – for example in Netty.

Memory Mapped Buffer

A direct byte buffer may also be created by mapping a region of a file directly into memory. In other words, we can load a region of a file to a particular native memory region that can be accessed later.  As you can imagine, it can give a significant performance boost if we have the requirement to read the content of a file multiple times. Thanks to memory mapped files, subsequent reads will use the content of the file from the memory, instead of loading the data from the disc every time it’s needed. MappedByteBuffer can be created via the FileChannel.map() method.

An additional advantage of memory mapped files, is that the OS can flush the buffer directly to the disk when the system is shutting down. Moreover, the OS can lock a mapped portion of the file from other processes on the machine.

Allocation is Expensive

One of the problems with direct buffers is that it’s expensive to allocate them. Regardless of the size of the buffer, calling Buffer.allocateDirect() is a relatively slow operation. It is therefore more efficient to either use direct buffers for large and long-lived buffers or create one large buffer, slice off portions on demand, and return them to be re-used when they are no longer needed.   A potential problem with slicing may occur when slices are not always the same size. The initial large byte buffer can become fragmented when allocating and freeing objects of different size.  Unlike Java heap, direct byte buffer cannot be compacted, because it’s not a target for the garbage collector-managed memory.

Monitoring the Usage of a Java Buffer Pool

If you’re interested in the amount of direct or mapped byte buffers used by your application, then you can easily monitor them using FusionReactor.   FusionReactor provides a break-down of all the different memory spaces.   Simply navigate to Resources and then Direct – Buffer Pools.

By default, the Direct Buffer Pool graph is displayed. You can switch to the Mapped Buffer Pool by clicking on a drop-down in the top right corner. Java will grow those pools as required so the fact that Direct Memory Used covers Direct Capacity on the graph above, means that all buffer memory allocated so far is in use.

Please note – you can limit the amount of direct byte buffer space that an application can allocate, by using -XX:MaxDirectMemorySize=N flag.   Although this is possible, you would need a very good reason to do so.

FR7 represents a massive step forward for FusionReactor, includes 20 completely new core features and around 100 major improvements and bug fixes. Our primary goals for this release have been to enhance the array of system, application and JVM metrics available, as well as build upon and extend the core functionality within FR Ultimate Edition.

Our mission is to provide developers, DevOps and I.T. professionals unparalleled depth of insight, transparency and interpretation into what applications are doing at the point where things are going wrong! Why? So, that they can isolate issues and performance problems faster than with any other tool.

FR7 has significantly increased the array of supported metrics available. Ranging from complete JMX MBean metrics, to enhanced framework (ColdFusion and Java) support to NoSQL data stores and current streaming platforms, such as Apache Kafka™.

The other major focus of this release has been to enhance FR Ultimate Edition – which adds a completely new definition to the meaning of application monitoring. FR Ultimate combines the most useful components and features from the most used error detection and performance analysis tools available to developers: the code debugger, application profiler and with the launch of FR7, the memory profiler. These tools can all be used safely, securely and most importantly with MINIMAL OVERHEAD in your production environment.

The complexity of today’s distributed, service based, containerized, ephemeral environments, combined with unprecedented quantities of data means that re-producing production issues is virtually impossible. The error detection features built into FR Ultimate completely change how issues would be investigated, removing the need to try and reproduce defects in a test environment or scouring through log files in the hope of finding a clue. FR Ultimate enables you to interact with issues as they unfold, directly in production. It is the only product on the market which offers such capabilities.

The whole team is extremely proud of FusionReactor 7 and we hope that you enjoy using it and that it continues to be your #1 tool to monitor, detect issues and protect your applications and servers.

David Tattersall

CEO Intergral – makers of FusionReactor

New in FusionReactor 7.0.0, we added the ability to monitor your applications usage of Apache Kafka.

Firstly, what is Apache Kafka. Apache Kafka is a publish-subscribe service that allows for multiple system to publish to a given topic and for multiple system to create from a topic. It is commonly used to build real time data pipelines because it is reliable, fast and scalable. To find out more about Apache Kafka visit there website: https://kafka.apache.org/

There are two major aspects of using Kafka the consumers and producers. In this post we are going to talk about how FusionReactor can monitor consumer applications.

Consumer Transactions

When you start your application with FusionReactor (see Install Guide) we will automatically detect and start tracking calls to the Kafka brokers. These calls are then tracked into transactions and displayed in the Transaction History page.

These transaction track the topic that is being polled and the partition that was polled, as well as the number of records returned and the consumer group used. From these transactions we also graph the activity and execution time into the normal transaction graphs in FusionReactor.

In addition to the standard graphs in FusionReactor if you are using an Enterprise or Ultimate license you will also have access to the ‘Framework Source’ page this page will aggregate the Kafka metrics into a selection of graphs to show how the Kafka usage breaks down by, Broker, Topic and Partition.

Consumer Metrics

When you have a Java application connected to Kafka, there are various metrics available from the driver library. FusionReactor is able to track these metrics and display them over time in a collection of graphs. FusionReactor provides two pages of these graphs Kafka Metrics and Kafka Node Metrics, each page is then displayed based on the selected consumer.

Kafka Metrics

The Kafka Metrics page is intended to give a general overview of the state of the applications use of Kafka, so on this page you will see metrics including:

• Records consumed
• Incoming/Outgoing Bytes
• Commit Rate
• Heartbeat Rate
• And many more..

Kafka Node Metrics

The Kafka Node Metrics include metrics for each consumer that is running in the JVM, split down to each node that is providing the data. These metrics include:

• Incoming/Outgoing Bytes
• Latency
• Request Size
• And many more..

For more information about what these metrics are please visit the Apache Kafka documentation