[Deploying Sakai] Running out of heap (hibernate, threadlocals, scheduler tool).

[Matthew Buckett]

We've currently been seeing some issue in production (2.8.x) where
tomcat basically runs out of heap (3G heap at the moment) and just
sits there doing GC after GC. Looking at a heap dump the first thing
that stood out was that we had some resultset objects which were
100MB+. Digging in a little further it turned out that these were from
the quartz scheduler tool which loads all the data from
scheduler_trigger_events and as we didn't have a job clearing out this
table the table had hundreds of thousands of events. We've now cleaned
out this table.

But the underlying concern was that when the heap dump was taken
nobody was using the job scheduler tool and hadn't done so in a while.
Looking at the heap dump it seems that the resultset hadn't been GCed
because it was the last query done in a hibernate session and
hibernate keeps the last one just in case. Now normally things get
tidied up at the end of a request, but the hibernate session is being
stored as a thread local and so escapes. This doesn't normally matter
as the next request comes in and overwrites the old hibernate session
(or re-uses it), but for sessions that don't get re-used they just eat
up memory.

Really the hibernate session management should be stored in a thread
local managed by ThreadLocalManager which cleans all threadlocals at
the end of a request. Does that make sense?

-- 
  Matthew Buckett, VLE Developer, IT Services, University of Oxford