[Deploying Sakai] Running out of heap (hibernate, threadlocals, scheduler tool).
azeckoski at unicon.net
Mon Nov 12 04:19:46 PST 2012
If it has to be stored somewhere and that somewhere has to be a TL
then I would say that it should definitely be one which is cleared out
by the TLM.
On Mon, Nov 12, 2012 at 6:56 AM, Matthew Buckett
<matthew.buckett at it.ox.ac.uk> wrote:
> We've currently been seeing some issue in production (2.8.x) where
> tomcat basically runs out of heap (3G heap at the moment) and just
> sits there doing GC after GC. Looking at a heap dump the first thing
> that stood out was that we had some resultset objects which were
> 100MB+. Digging in a little further it turned out that these were from
> the quartz scheduler tool which loads all the data from
> scheduler_trigger_events and as we didn't have a job clearing out this
> table the table had hundreds of thousands of events. We've now cleaned
> out this table.
> But the underlying concern was that when the heap dump was taken
> nobody was using the job scheduler tool and hadn't done so in a while.
> Looking at the heap dump it seems that the resultset hadn't been GCed
> because it was the last query done in a hibernate session and
> hibernate keeps the last one just in case. Now normally things get
> tidied up at the end of a request, but the hibernate session is being
> stored as a thread local and so escapes. This doesn't normally matter
> as the next request comes in and overwrites the old hibernate session
> (or re-uses it), but for sessions that don't get re-used they just eat
> up memory.
> Really the hibernate session management should be stored in a thread
> local managed by ThreadLocalManager which cleans all threadlocals at
> the end of a request. Does that make sense?
> Matthew Buckett, VLE Developer, IT Services, University of Oxford
> production mailing list
> production at collab.sakaiproject.org
> TO UNSUBSCRIBE: send email to production-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
Aaron Zeckoski - Software Architect - http://tinyurl.com/azprofile
More information about the production