FusionReactor Blog - News and expert analysis for Java APM users.

Memory Mapped Buffer

Understanding the Memory Mapped Buffer

To understand the memory mapped buffer we first need to understand the Java buffer pool. The Java buffer pool space is located outside of the garbage collector-managed memory. It’s a way to allocate native off-heap memory.  What’s the benefit of using a Java buffer pool? To answer this question, let’s firstly learn what byte buffers are.

Java Byte Buffer

Non-Direct Buffer

java.nio package comes with the Bytebuffer class. It allows us to allocate both direct and non-direct byte buffers. Their is nothing special about non-direct byte buffers – they are an implementation of HeapByteBuffer created by ByteBuffer.allocate() and ByteBuffer.wrap() factory methods. As the name of the class suggests, these are on-heap byte buffers. Wouldn’t it be easier to allocate all the buffers on the Java heap space then?   Why would anyone need to allocate something in a native memory? To answer this question, we need to understand how operating systems perform I/O operations.   Any read or write instructions are executed on memory areas which are contiguous sequence of bytes. So does byte[] occupy a contiguous space on the heap? While technically it makes sense, the JVM specification does not have such guarantees. What’s more interesting, the specification doesn’t even guarantee that heap space will be contiguous itself! Although it seems to be rather unlikely that JVM will place a one-dimensional array of primitives in different places in memory, byte array from Java heap space cannot be used in native I/O operations directly. It has to be copied to a native memory before every I/O, which of course, leads to obvious inefficiencies. For this reason, a direct buffer was introduced.

Java Direct Buffer

A direct buffer is a chunk of native memory shared with Java from which you can perform a direct read.

An instance of DirectByteBuffer can be created using the ByteBuffer.allocateDirect() factory method. Byte buffers are the most efficient way to perform I/O operations and thus, they are used in many libraries and frameworks – for example in Netty.

Memory Mapped Buffer

A direct byte buffer may also be created by mapping a region of a file directly into memory. In other words, we can load a region of a file to a particular native memory region that can be accessed later.  As you can imagine, it can give a significant performance boost if we have the requirement to read the content of a file multiple times. Thanks to memory mapped files, subsequent reads will use the content of the file from the memory, instead of loading the data from the disc every time it’s needed. MappedByteBuffer can be created via the FileChannel.map() method.

An additional advantage of memory mapped files, is that the OS can flush the buffer directly to the disk when the system is shutting down. Moreover, the OS can lock a mapped portion of the file from other processes on the machine.

Allocation is Expensive

One of the problems with direct buffers is that it’s expensive to allocate them. Regardless of the size of the buffer, calling Buffer.allocateDirect() is a relatively slow operation. It is therefore more efficient to either use direct buffers for large and long-lived buffers or create one large buffer, slice off portions on demand, and return them to be re-used when they are no longer needed.   A potential problem with slicing may occur when slices are not always the same size. The initial large Java byte buffer can become fragmented when allocating and freeing objects of different size.  Unlike Java heap, direct byte buffer cannot be compacted, because it’s not a target for the garbage collector-managed memory.

Monitoring the Usage of a Java Buffer Pool

If you’re interested in the amount of direct or mapped byte buffers used by your application, then you can easily monitor them using FusionReactor.   FusionReactor provides a break-down of all the different memory spaces.   Simply navigate to Resources and then Direct – Buffer Pools.

Java buffer pool, garbage collector-managed memory

By default, the Java Direct Buffer Pool graph is displayed. You can switch to the Mapped Buffer Pool by clicking on a drop-down in the top right corner. Java will grow those pools as required so the fact that Direct Memory Used covers Direct Capacity on the graph above, means that all buffer memory allocated so far is in use.

Please note – you can limit the amount of direct Java byte buffer space that an application can allocate, by using -XX:MaxDirectMemorySize=N flag.   Although this is possible, you would need a very good reason to do so.

Java Non-Direct Buffer

Understanding  Java Non-Direct Buffer

The Java buffer pool space is located outside of the garbage collector-managed memory. It’s a way to allocate native off-heap memory.  What’s the benefit of using a Java buffer pool? To answer this question, let’s firstly learn what byte buffers are.

Byte Buffer

Non-Direct Buffer

java.nio package comes with the Bytebuffer class. It allows us to allocate both direct and non-direct byte buffers. Their is nothing special about non-direct byte buffers – they are an implementation of HeapByteBuffer created by ByteBuffer.allocate() and ByteBuffer.wrap() factory methods. As the name of the class suggests, these are on-heap byte buffers. Wouldn’t it be easier to allocate all the buffers on the Java heap space then?   Why would anyone need to allocate something in a native memory? To answer this question, we need to understand how operating systems perform I/O operations.   Any read or write instructions are executed on memory areas which are contiguous sequence of bytes. So does byte[] occupy a contiguous space on the heap? While technically it makes sense, the JVM specification does not have such guarantees. What’s more interesting, the specification doesn’t even guarantee that heap space will be contiguous itself! Although it seems to be rather unlikely that JVM will place a one-dimensional array of primitives in different places in memory, byte array from Java heap space cannot be used in native I/O operations directly. It has to be copied to a native memory before every I/O, which of course, leads to obvious inefficiencies. For this reason, a direct buffer was introduced.

Direct Buffer

A direct buffer is a chunk of native memory shared with Java from which you can perform a direct read.

An instance of DirectByteBuffer can be created using the ByteBuffer.allocateDirect() factory method. Byte buffers are the most efficient way to perform I/O operations and thus, they are used in many libraries and frameworks – for example in Netty.

Memory Mapped Buffer

A direct byte buffer may also be created by mapping a region of a file directly into memory. In other words, we can load a region of a file to a particular native memory region that can be accessed later.  As you can imagine, it can give a significant performance boost if we have the requirement to read the content of a file multiple times. Thanks to memory mapped files, subsequent reads will use the content of the file from the memory, instead of loading the data from the disc every time it’s needed. MappedByteBuffer can be created via the FileChannel.map() method.

An additional advantage of memory mapped files, is that the OS can flush the buffer directly to the disk when the system is shutting down. Moreover, the OS can lock a mapped portion of the file from other processes on the machine.

Allocation is Expensive

One of the problems with direct buffers is that it’s expensive to allocate them. Regardless of the size of the buffer, calling Buffer.allocateDirect() is a relatively slow operation. It is therefore more efficient to either use direct buffers for large and long-lived buffers or create one large buffer, slice off portions on demand, and return them to be re-used when they are no longer needed.   A potential problem with slicing may occur when slices are not always the same size. The initial large byte buffer can become fragmented when allocating and freeing objects of different size.  Unlike Java heap, direct byte buffer cannot be compacted, because it’s not a target for the garbage collector-managed memory.

Monitoring the Usage of a Java Buffer Pool

If you’re interested in the amount of direct or mapped byte buffers used by your application, then you can easily monitor them using FusionReactor.   FusionReactor provides a break-down of all the different memory spaces.   Simply navigate to Resources and then Direct – Buffer Pools.

Java buffer pool, garbage collector-managed memory

By default, the Direct Buffer Pool graph is displayed. You can switch to the Mapped Buffer Pool by clicking on a drop-down in the top right corner. Java will grow those pools as required so the fact that Direct Memory Used covers Direct Capacity on the graph above, means that all buffer memory allocated so far is in use.

Please note – you can limit the amount of direct byte buffer space that an application can allocate, by using -XX:MaxDirectMemorySize=N flag.   Although this is possible, you would need a very good reason to do so.

garbage collector-managed memory

Understanding garbage collector-managed memory

The Java buffer pool space is located outside of the garbage collector-managed memory. It’s a way to allocate native off-heap memory.  What’s the benefit of using a Java buffer pool? To answer this question, let’s firstly learn what byte buffers are.

Byte Buffer

Non-Direct Buffer

java.nio package comes with the Bytebuffer class. It allows us to allocate both direct and non-direct byte buffers. Their is nothing special about non-direct byte buffers – they are an implementation of HeapByteBuffer created by ByteBuffer.allocate() and ByteBuffer.wrap() factory methods. As the name of the class suggests, these are on-heap byte buffers. Wouldn’t it be easier to allocate all the buffers on the Java heap space then?   Why would anyone need to allocate something in a native memory? To answer this question, we need to understand how operating systems perform I/O operations.   Any read or write instructions are executed on memory areas which are contiguous sequence of bytes. So does byte[] occupy a contiguous space on the heap? While technically it makes sense, the JVM specification does not have such guarantees. What’s more interesting, the specification doesn’t even guarantee that heap space will be contiguous itself! Although it seems to be rather unlikely that JVM will place a one-dimensional array of primitives in different places in memory, byte array from Java heap space cannot be used in native I/O operations directly. It has to be copied to a native memory before every I/O, which of course, leads to obvious inefficiencies. For this reason, a direct buffer was introduced.

Direct Buffer

A direct buffer is a chunk of native memory shared with Java from which you can perform a direct read.

An instance of DirectByteBuffer can be created using the ByteBuffer.allocateDirect() factory method. Byte buffers are the most efficient way to perform I/O operations and thus, they are used in many libraries and frameworks – for example in Netty.

Memory Mapped Buffer

A direct byte buffer may also be created by mapping a region of a file directly into memory. In other words, we can load a region of a file to a particular native memory region that can be accessed later.  As you can imagine, it can give a significant performance boost if we have the requirement to read the content of a file multiple times. Thanks to memory mapped files, subsequent reads will use the content of the file from the memory, instead of loading the data from the disc every time it’s needed. MappedByteBuffer can be created via the FileChannel.map() method.

An additional advantage of memory mapped files, is that the OS can flush the buffer directly to the disk when the system is shutting down. Moreover, the OS can lock a mapped portion of the file from other processes on the machine.

Allocation is Expensive

One of the problems with direct buffers is that it’s expensive to allocate them. Regardless of the size of the buffer, calling Buffer.allocateDirect() is a relatively slow operation. It is therefore more efficient to either use direct buffers for large and long-lived buffers or create one large buffer, slice off portions on demand, and return them to be re-used when they are no longer needed.   A potential problem with slicing may occur when slices are not always the same size. The initial large byte buffer can become fragmented when allocating and freeing objects of different size.  Unlike Java heap, direct byte buffer cannot be compacted, because it’s not a target for the garbage collector-managed memory.

Monitoring the Usage of a Java Buffer Pool

If you’re interested in the amount of direct or mapped byte buffers used by your application, then you can easily monitor them using FusionReactor.   FusionReactor provides a break-down of all the different memory spaces.   Simply navigate to Resources and then Direct – Buffer Pools.

Java buffer pool, garbage collector-managed memory

By default, the Direct Buffer Pool graph is displayed. You can switch to the Mapped Buffer Pool by clicking on a drop-down in the top right corner. Java will grow those pools as required so the fact that Direct Memory Used covers Direct Capacity on the graph above, means that all buffer memory allocated so far is in use.

Please note – you can limit the amount of direct byte buffer space that an application can allocate, by using -XX:MaxDirectMemorySize=N flag.   Although this is possible, you would need a very good reason to do so.

Announcing FusionReactor 7 – tons more metrics and features = tons more insight & faster issue identification

FR7 represents a massive step forward for FusionReactor, includes 20 completely new core features and around 100 major improvements and bug fixes. Our primary goals for this release have been to enhance the array of system, application and JVM metrics available, as well as build upon and extend the core functionality within FR Ultimate Edition.

Our mission is to provide developers, DevOps and I.T. professionals unparalleled depth of insight, transparency and interpretation into what applications are doing at the point where things are going wrong! Why? So, that they can isolate issues and performance problems faster than with any other tool.

FR7 has significantly increased the array of supported metrics available. Ranging from complete JMX MBean metrics, to enhanced framework (ColdFusion and Java) support to NoSQL data stores and current streaming platforms, such as Apache Kafka™.

FusionReactor JMX MBean Metrics

The other major focus of this release has been to enhance FR Ultimate Edition – which adds a completely new definition to the meaning of application monitoring. FR Ultimate combines the most useful components and features from the most used error detection and performance analysis tools available to developers: the code debugger, application profiler and with the launch of FR7, the memory profiler. These tools can all be used safely, securely and most importantly with MINIMAL OVERHEAD in your production environment.

FusionReactor Memory / Heap Profiler

The complexity of today’s distributed, service based, containerized, ephemeral environments, combined with unprecedented quantities of data means that re-producing production issues is virtually impossible. The error detection features built into FR Ultimate completely change how issues would be investigated, removing the need to try and reproduce defects in a test environment or scouring through log files in the hope of finding a clue. FR Ultimate enables you to interact with issues as they unfold, directly in production. It is the only product on the market which offers such capabilities.

The whole team is extremely proud of FusionReactor 7 and we hope that you enjoy using it and that it continues to be your #1 tool to monitor, detect issues and protect your applications and servers.

David Tattersall

CEO Intergral – makers of FusionReactor

Monitoring Kafka Consumer Applications

New in FusionReactor 7.0.0, we added the ability to monitor your applications usage of Apache Kafka.

Firstly, what is Apache Kafka. Apache Kafka is a publish-subscribe service that allows for multiple system to publish to a given topic and for multiple system to create from a topic. It is commonly used to build real time data pipelines because it is reliable, fast and scalable. To find out more about Apache Kafka visit there website: https://kafka.apache.org/

There are two major aspects of using Kafka the consumers and producers. In this post we are going to talk about how FusionReactor can monitor consumer applications.

Consumer Transactions

When you start your application with FusionReactor (see Install Guide) we will automatically detect and start tracking calls to the Kafka brokers. These calls are then tracked into transactions and displayed in the Transaction History page.

These transaction track the topic that is being polled and the partition that was polled, as well as the number of records returned and the consumer group used. From these transactions we also graph the activity and execution time into the normal transaction graphs in FusionReactor.

In addition to the standard graphs in FusionReactor if you are using an Enterprise or Ultimate license you will also have access to the ‘Framework Source’ page this page will aggregate the Kafka metrics into a selection of graphs to show how the Kafka usage breaks down by, Broker, Topic and Partition.

Consumer Metrics

When you have a Java application connected to Kafka, there are various metrics available from the driver library. FusionReactor is able to track these metrics and display them over time in a collection of graphs. FusionReactor provides two pages of these graphs Kafka Metrics and Kafka Node Metrics, each page is then displayed based on the selected consumer.

Kafka Metrics

The Kafka Metrics page is intended to give a general overview of the state of the applications use of Kafka, so on this page you will see metrics including:

  • Records consumed
  • Incoming/Outgoing Bytes
  • Commit Rate
  • Heartbeat Rate
  • And many more..

Kafka Node Metrics

The Kafka Node Metrics include metrics for each consumer that is running in the JVM, split down to each node that is providing the data. These metrics include:

 

  • Incoming/Outgoing Bytes
  • Latency
  • Request Size
  • And many more..

For more information about what these metrics are please visit the Apache Kafka documentation

 

FusionReactor features in the DZone Guide to Java: Development and Evolution

DZone have just published the 2017 Guide to Java: Development and Evolution and FusionReactor are featured in it. The 2017 Guide to Java explores upcoming features of Java 9, how to make your apps backwards-compatible, a look into whether Microservices are right for you, and using the Futures API in Java.

You can download it for free from the Dzone website.

We are Sponsoring Velocity Conference, San Jose,CA 2017: Build & Maintain Complex Distributed Systems

We are Sponsoring Velocity Conference, San Jose,CA 2017: Build & Maintain Complex Distributed Systems. Come and meet us at our booth #718. We will be showing off the latest FusionReactor developments and features to help Java developers fix problems faster! You can read more about our event attendance here.

If you cannot wait to meet us get started with a free trial of FusionReactor Java application Monitor or visit our webinars page to see FusionReactor in action. With both new webinars to sign up to and prerecorded webinars to watch.

PODCAST: Why FusionReactor is Different than Other APM Tools and What’s New in Version 7 & the CLOUD, with David Tattersall

David Tattersall the CEO of Intergral (makers of FusionReactor) talks on the ColdFusion Alive podcast with host Michael Smith. On Why It is Different to other APM tools and what is new in version 7 & the CLOUD

See the full podcast here: http://teratech.com/podcast/fusionreactor-application-performance-monitor-why-its-different-than-other-apm-tools-and-whats-new-in-version-7-the-cloud-with-david-tattersal/

Win a FREE Ticket for Velocity Conference San Jose, California 2017

This year we are back at Velocity Conference San Jose – You can visit us at booth #718 we will be showing off the latest FusionReactor developments and features to help developers fix problems faster! To celebrate this we are giving away a free ticket for Velocity Conference, San Jose, California.

The Prize is for one Bronze 2-Day Pass which Includes:
– All sessions (Wed & Thu)
– All keynote and plenary presentations
– Sponsor Pavilion and all events held there
– All on-site networking events
– Lunch (Wed & Thu)

There are Two ways to enter the draw:
1) ON TWITTER : Follow us and retweet this tweet it is that simple! If you are already following us then just retweet this.

OR (You can do both to double your chances of winning).

2) ON FACEBOOK : Just Like us on Facebook – and leave us a comment – “I want to go to Velocity”
We will be announcing the winner on, Friday 9th June, this will give the winner enough time to organise flights and accommodation.

If you do not win, please do not worry you can still save 25% on the entry price by using our Discount Code: FusionReactor25

Why Every Developer Needs FusionReactors Production Debugger

The situation is a common one when you’re a Java developer: you know there’s an application issue; you know it’s degrading performance and negatively impacting your customers; but you’re having trouble finding the root cause and you’re using valuable time that should be focused on the next deliverable. Today’s distributed application environments and architectures are more complex than ever and this only adds to the difficulty when trying to pinpoint software problems. In the recently published DZONE Guide to Performance & Monitoring, it was found that the most time consuming part of fixing performance issues is finding the root cause followed closely by being able to reproduce the performance issue.

The first tool most developers reach for when investigating problems are the application and database logs. Logs are an invaluable resource when looking for the key to an issue but searching through them takes time. They also might not give the developer a clear picture of what is causing the software to run slowly or throw exceptions. Many developers do not have access to production environments and often these logs are their only view into how the application is running.  However, the logs may not contain the information they need or the logs may not even be available. Metrics are also useful but in many cases they don’t provide the kind of detailed information a developer can use to narrow in on the most likely cause of application slowness. It also takes time to collect and inspect all the data that might provide clues as to what is going on within the application. Is it a poorly written SQL query? A memory leak? A null pointer exception? It could be one or more of these issues and it may take hours or days to fully understand what is causing the problem.

Tweaking and adjusting the code for applications to make them run as efficiently as possible is important, but what do you do when something breaks without warning? Defects in code can be harder to locate than a performance issue because they occur much more infrequently and can be difficult to reproduce. Logs may show that an error occurred but if you don’t know exactly what the application was doing at that specific time under those specific conditions you may not be able to figure out what happened.

At this point you may be nodding your head because you’ve dealt with similar situations and have used the same procedures to address performance issues or come up with an emergency break-fix solution to get your application back up and running. Traditional application performance management (APM) tools provide some information and can be set to alert when something is wrong but they don’t give you the whole picture. This is where FusionReactor comes in with a whole new way to monitor your code and how it functions as it’s running in production. Instead of grepping through your logs, diving into the heap, tallying object instances, running stack traces over and over, guessing at some breakpoints, or including debug data into your code after an issue has arisen, FusionReactor allows you to interact and debug your production application as it is running. FusionReactor was designed to give developers the information they need along with powerful production-safe tools like the Production Debugger and the low-impact Profiler. This means that instead of spending time collecting data and attempting to reproduce the problem, you can look at your code as it executes to see where the problem starts and learn exactly what is causing the issue so that you can start to work on a solution as fast as possible.

Instantly identify issues and bottlenecks

Developers have long been taught that debugging code while it’s running in production is a bad idea for many reasons including high overhead, having to spend time watching and waiting on the debugger while it runs, and the potential for a breakpoint to halt the entire application.

The FusionReactor uses JVMTI to bind directly into the built in debugger in the JVM, thus requiring no protocol overhead. The debug features are only enabled when the user wants to run them. The normal impact of having FusionReactor’s Debugger installed is zero (unless you use java 1.6.0._21 or earlier due to a bug in Java) apart from the memory used to load the library into memory.

Typically, you would set a breakpoint then step through your code and wait for a breakpoint to fire. The point being, you need to actually attend to the debugger as it’s running, which is very time consuming.  FusionReactor does away with this need, by sending out an email as soon as the breakpoint is hit. The email response can be configured one of two ways. In the first case, the breakpoint will fire, the stack trace and variable information is captured, and an email is sent to the developer. This is a non-invasive approach that allows the developer time to look at the data without any impact to the code running in production. In the second case, the breakpoint fires, stack trace and variable information is captured, and an email is sent however the thread where the breakpoint fired remains open for either a defined period of time or indefinitely, depending on how it has been configured. The developer can click on a link in the email which brings up the debugger at the point in the application where the breakpoint fired.  So when some problem occurs at 3:00 am, FusionReactor will capture the state, capture the thread and essentially freeze it, until you come into the office and start debugging it.

Instantly identify issues and bottlenecks

FusionReactor Debugger was designed with enhanced control mechanisms in mind to allow for safe production debugging:

  • Set how many times a breakpoint fires so that only those threads are affected.
  • Set a limit on the number of threads that can be paused in the debugger at any one time or how many threads a single breakpoint can pause.
  • Set a parameter on how long a thread can be paused. Once the time limit has been reached the thread will be released to allow the end user to continue.
  • Have a breakpoint send an email with the stack and variable information while it holds the thread for the predetermined time period.

It goes without saying that any production debugging should be done with caution so as not to affect other threads or the overall application, however FusionReactor’s Debugger allows you to target only what is needed to quickly investigate the issue.

The low-impact Java Profiler offers the ability to watch your code as it executes, enabling you to pinpoint bottlenecks in production. You can profile any request or transaction running on the JVM such as Tomcat, Glassfish, or Jboss. The Profiler can be run manually or configured to start when a particular condition has been set. This is a huge advantage in that you can leave the Profiler ‘on’ and set it to activate automatically when a request takes longer than expected instead of having to watch and wait for the event to happen. You can also limit the number of profile samples to control the accuracy of the sample. This information is then stored so that later you can drill down into the details to see how your code is running. The Profiler shows how much time is spent in each section of code as well as the overall percentage of it that makes up the transaction or request.

Performance tuning and troubleshooting will always be a part of development but hours or days of time spent searching through log files, running stack traces over and over, and trying to reproduce a production issue can be a thing of the past with FusionReactor. Being able to quickly solve issues results in less downtime, more stability, faster applications, happier customers, and less stressed developers!

 

To get start with a FREE trial of FusionReactor go here to Download FusionReactor and Start your trial.