We're sometimes asked about the timings for FusionReactor's Timeout Protection, part of the Crash Protection system. The idea behind this crash protection is that it places a hard upper limit on the length of time requests can run. You might find, however, that requests take a few seconds longer to be killed than you originally thought. Here's why.
After the time limit is reached, three strategies are employed by the Request Assassin to kill the request:
In addition to these delays, the Request Assassin doesn't run all the time: it wakes up every second, and checks whether there are any threads it should examine. If there are any, it starts to process them, employing the strategies above. If a strategy succeeds, it moves on to the next candidate. If not, it tries the next strategy, until the list of strategies is exhausted.
Because FusionReactor attempts to place as little load on the target system as possible, the candidates for kill are queued up. This is an additional source of delay – previous kill operations may have to be processed before the Request Assassin gets to the request of interest.
Let's look at a worked example, with Timeout Protection set to 1 second.
In the diagram, you can see (starting from the top timeline):
The request then dies, marked by the first red "X" in its timeline. If the request was performing native operations (i.e. using part of the Java language that's written in C), it will be killed as soon as it returns. One common native operation is blocking socket communication: this is performed in Java by a C library.
The second timeline is much the same as the first, except the Request Assassin ran at time T0.9r, i.e. an instant before the request became eligible for kill at T1. The next time the Request Assassin runs is at time T2 (actually an instant before that), meaning the whole timeline starts 1 second later, at time T2, and completing with the kill at time T5.
We showed in this small article where the (up to) 3-second overhead in Timeout Protection comes from: a combination of soft-kill and interrupt delays, which allow us to try to stop a request without resorting to a hard kill.
|Last Updated:||22/Sep/11 3:36 PM|
Comments are closed.