I couldn't wait to share this...

I recently had a funny email conversation with my system administrator the other day. I've reproduced the conversation here, removed names and proprietary info, etc. In our defense, it was early in the morning and both of us were busy and bit punchy:

- begin -
System Administrator (SA): I think there's something wrong with gate4. I checked the queue and it hangs when it checks data on 4. Is there a way to check the queue depth on the gate itself? Records in the database for today so far is a little over 300,000, which is about half of what it should be.

Me: If you can access the web-console on gate 4 then you can drill down the JMX tree to those components. What's the jboss-server.log saying on gate 4?

SA: I see the following over and over:
20060119 012937 WARN [TP-Processor25415] Unexpected error processing request: "System overload: Maximum number of concurrent requests (100) exceeded" [Source IP: ...

Me: Can you send me the last 1000 log entries from that file please?

SA: There. [A log file was attached to this email.]

Me: It's empty.

SA: You know why I can't check it from the app box?

Me: I give up, why?

SA: I don't know, that's why I'm asking you. You think I should restart jboss on gate4?

Me: Oh - I thought you were setting up a joke. :-/ Can you ssh into gate4? If so, can you get me the last 1000 lines from the jboss-server log? [Notice I ask for the same data again b/c my earlier response that the emailed file was empty went into the bit bucket.] Don't bounce it - I'd take the Apache instance on gate 4 down. Leave JBoss up so we can diagnose, k?

SA: Hmmm... too early in the morning for a joke. Anyway I gave you the log file.

Me: Yes, but it was empty. [*sigh*]
- end-

SA ended up sending me the log data I asked for, but then someone else told him to bounce JBoss so we lost the sick patient.

We ended up not doing any research into the root cause of the JBoss crash. It's just a beta, dood!