[FRS-322] Performance considerations for FusionReactor v5


Note for v5.0.9 or newer

The below technote refers to v5.0.0 through v5.0.8. Although the information is still valid and valuable for users of newer releases, it's important to read this article in conjunction with other, newer technotes.

For v5.0.9 specifically, check both the v5.0.9 Special Release Notes and the general v5 Release Notes.

Background

FusionReactor is a light-weight Java application server monitoring tool. It's designed to run 24×7 in a production environment with minimal performance impact. This technote discusses some of the key facts regarding performance and the low overhead of FusionReactor.

Users upgrading from v4 and earlier

Users upgrading should first read these key articles:

Automatic JDBC Wrapping

In earlier versions of FusionReactor it was necessary to manually wrap your JDBC datasources in order to capture metrics for their use. This is no longer necessary in FusionReactor v5. However, it is arguably one of the highest overhead areas of FusionReactor. It's important to understand that if you did not wrap your datasources with v4.5 or earlier, you will see an additional overhead in v5 as we are now capturing substantially more data. Typically the overhead is still far less than 1% but it's important to be aware of this. If you have a poorly optimized application which (for example) runs hundreds of JDBC calls on every page there will be a relatively large volume of overhead.

Areas of overhead and related configuration options

JDBC "Query Location"

  • The JDBC "Query Location" feature takes a stack trace at the point of executing each JDBC query. On a poorly optimized system running lots of queries per request or a very high throughput system, this could have noticable overhead. To reduce it's effect you could (in order of preference):
    • Only track queries taking longer than X milliseconds.
      • Typically we are only interested in queries taking a long time to run. Therefore, by tracking only slow running queries we can significantly reduce the overall overhead but maintain usefulness.
      • By default all queries are tracked. The value to set depends on your environment but a good starting point could be 500 ms.
      • Documentation link: http://docs.intergral.com/display/FR50/JDBC+Settings
    • Disable the query location feature
      • If you are not frequently interested in the file & line of code where a JDBC query originated this feature could be disabled.
      • If you have an alternative method to easily find the SQL code location (eg source code grep), disabling may have minimal impact on your operations.
      • Documentation link: http://docs.intergral.com/display/FR50/JDBC+Settings
    • Disable JDBC tracking for a datasource
      • If you have a high throughput datasource that you do not wish to monitor, you can easily disable it with the __fusionreactor_exclude=true JDBC parameter.
      • This is most useful if you have other DB monitoring systems in place for a particular datasource
      • Documentation link: http://docs.intergral.com/display/FR50/JDBC+Monitoring+Options
    • Disable JDBC tracking for a JDBC driver class
      • If you have a third-party driver which does not work well with FusionReactor it is possible to disable interaction with that driver completely.
      • At the time of writing, there are only two (rarely used) drivers which appear to exhibit this problem – com.inet.tds. and jstels.jdbc.mdb.
      • Documentation link: https://groups.google.com/d/msg/fusionreactor/eY9ZpIrc4Fg/-uCxSOKBDbUJ

JDBC Log Files

  • By default, every JDBC request generates an entry in the JDBC log file. On a system with poor I/O performance, extremely high throughput or a poorly optimized application; it's possible this generates a high volume of I/O overhead.
  • Using the methods listed above, tracking only certain JDBC requests (eg over X ms) could reduce the I/O overhead.
  • In addition to the above methods, it's possible to disable JDBC logging to disk completely. It's recommended to avoid this as the log files are very useful for root cause analysis.
  • Documentation link: http://docs.intergral.com/display/FR50/JDBC+Settings
  • Documentation link: http://docs.intergral.com/pages/viewpage.action?pageId=22478957

Request Log Files

  • By default, each request writes two entries to the request log file (EXECUTING and COMPLETED)
  • On a system with poor I/O performance, extremely high throughput or a poorly optimized application; it's possible this generates a high volume of I/O overhead.
  • It's possible to disable request logging to disk completely. It's recommended to avoid this as the log files are very useful for root cause analysis.
  • Documentation link: http://docs.intergral.com/pages/viewpage.action?pageId=22478919

Other considerations

On *nix systems you may find you have the default security setting allowing each process to have only a minimal number of concurrent open file handles. Often on these systems both file-system files and TCP connections count as file handles. Therefore, even 1024 or 2048 may be a low number for the max file handle count.

Refer to your operating system vendor and documentation for further ulimit / security limit configuration details.

Issue Details

Type: Technote
Issue Number: FRS-322
Components: Metrics
Environment:
Resolution: Fixed
Last Updated: 18/Nov/13 3:34 PM
Affects Version: 5.0.0
Fixed Version: 5.0.0
Server:
Platform:
Related Issues: