[Deploying Sakai] Running out of heap (hibernate, threadlocals, scheduler tool).

[Aaron Zeckoski]

If it has to be stored somewhere and that somewhere has to be a TL
then I would say that it should definitely be one which is cleared out
by the TLM.
-AZ


On Mon, Nov 12, 2012 at 6:56 AM, Matthew Buckett
<matthew.buckett at it.ox.ac.uk> wrote:
> We've currently been seeing some issue in production (2.8.x) where
> tomcat basically runs out of heap (3G heap at the moment) and just
> sits there doing GC after GC. Looking at a heap dump the first thing
> that stood out was that we had some resultset objects which were
> 100MB+. Digging in a little further it turned out that these were from
> the quartz scheduler tool which loads all the data from
> scheduler_trigger_events and as we didn't have a job clearing out this
> table the table had hundreds of thousands of events. We've now cleaned
> out this table.
>
> But the underlying concern was that when the heap dump was taken
> nobody was using the job scheduler tool and hadn't done so in a while.
> Looking at the heap dump it seems that the resultset hadn't been GCed
> because it was the last query done in a hibernate session and
> hibernate keeps the last one just in case. Now normally things get
> tidied up at the end of a request, but the hibernate session is being
> stored as a thread local and so escapes. This doesn't normally matter
> as the next request comes in and overwrites the old hibernate session
> (or re-uses it), but for sessions that don't get re-used they just eat
> up memory.
>
> Really the hibernate session management should be stored in a thread
> local managed by ThreadLocalManager which cleans all threadlocals at
> the end of a request. Does that make sense?
>
> --
>   Matthew Buckett, VLE Developer, IT Services, University of Oxford
> _______________________________________________
> production mailing list
> production at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/production
>
> TO UNSUBSCRIBE: send email to production-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"



-- 
Aaron Zeckoski - Software Architect - http://tinyurl.com/azprofile