[Deploying Sakai] [Building Sakai] ClientAbortException: java.net.SocketException: Broken pipe

[Warwick Chapman]

Noah, thank you for this.  Much appreciated.  Let me try the 4G heap on the
VMs with 8G of memory.  The third VM has 6GB and I cannot increase it much
more than that.

The proviso of 25% for the system would then also fit into that VM with Xmx
4G (67%) and system 2G (33%).

I'm going to give it a crack.  If it goes down, there's be no heap
analysis, I'll have find somewhere to hide.

Aspirant members of parliament writing 'candidate tests' - the people take
becoming an MP *very* seriously, and don't take lightly to the server
falling over mid-test.  It happened on Saturday last week - during the 24
hours period the exam ran for, I lost application servers times for 250
applicants.  At that stage I had an Xmx of 1024m, and 2 vCPUs (and then 4
during a running reconfigure) per appserver.

Now I have an extra appserver, Xmx of 4G and 4 vCPUs per appserver.

-- Warwick Bruce Chapman | +27 83 7797 094 | http://warwickchapman.com


On Fri, Nov 1, 2013 at 6:34 PM, Noah Botimer <botimer at umich.edu> wrote:

> It sounds like you were running at 1GB heap, which is quite likely too low
> for an instance with lots of modules/webapps deployed. Increasing Xmx
> should do the trick, but in case you still have problems...
>
> You can flip the switch to dump heap on crash (and once it falls over),
> you could analyze the dump:
>
>
> http://stackoverflow.com/questions/542979/using-heapdumponoutofmemoryerror-parameter-for-heap-dump-for-jboss
>
> I've had good success with YourKit for examining heaps, but jhat and
> VisualVM can do it too. You would be looking for a particular class or set
> of classes that are disproportionately allocated (say 50% or more of
> total), indicating a leak (often a reference from an item to a
> container/observer that persists but no longer tracks the child).
>
> If you observe that a JVM (via jstat or other) has a growing heap but is
> not yet to crashing, you can dump the heap with jmap.
>
> I think it's pretty common these days to run with 4 and 6GB heaps on
> 64-bit JVMs (with Aaron's proviso of ~25% spare for system).
>
> Thanks,
> -Noah
>
>
> On Nov 1, 2013, at 11:51 AM, Aaron Zeckoski wrote:
>
> I would say 2g is adequate but I am adding the production list to this
> to see if anyone else has any comment here. Generally a single
> instance would be more than adequate for 60 users. We certainly are
> running with many more users per instance in many cases without issue.
>
> -AZ
>
>
> On Fri, Nov 1, 2013 at 11:36 AM, Warwick Chapman
> <warwickchapman at gmail.com> wrote:
>
> Ok, please give me your best guess because I don't have any opportunity
> left
>
> to tune.  On Sunday I've got to ensure this thing doesn't crash because of
>
> java.lang.OutOfMemoryError: Java heap space as I've been getting now ...
>
>
> I cannot believe I am working so hard to get Sakai stable for no more than
>
> 60 concurrent users.
>
>
> I have 3 OpenVZ on 3 separate pieces of tin.
>
>
> Two have been increased to 8192 and the third to 6144.
>
>
> Their JAVA_OPTS are now, respectively:
>
>
> 1. export JAVA_OPTS='-server -d64 -Xmx4g -XX:MaxPermSize=512m
>
> -Djava.awt.headless=true -Dhttp.agent=Sakai'
>
> 2. export JAVA_OPTS='-server -d64 -Xmx4g -XX:MaxPermSize=512m
>
> -Djava.awt.headless=true -Dhttp.agent=Sakai'
>
> 3. export JAVA_OPTS='-server -d64 -Xmx3g -XX:MaxPermSize=512m
>
> -Djava.awt.headless=true -Dhttp.agent=Sakai'
>
>
> I am worried that keeping them at 4192 and doubling up from the original
>
> 1024 will not be sufficient.  But I will be guided by you.
>
>
>
> -- Warwick Bruce Chapman | +27 83 7797 094 | http://warwickchapman.com
>
>
>
> On Fri, Nov 1, 2013 at 5:28 PM, Aaron Zeckoski <azeckoski at unicon.net>
> wrote:
>
>
> The "No" was for your "OK" question since you are missing MaxPermSize.
>
> You can set it to something higher but if it were me I would not do
>
> it. Larger memory space means it takes the JVM longer to do garbage
>
> collections. I would suggest you start with something smaller and tune
>
> it until things are working well for your users and load. Like I said,
>
> there is no single magic or "right" number. There are lots of wrong
>
> ones though.
>
>
> -AZ
>
>
>
> On Fri, Nov 1, 2013 at 11:20 AM, Warwick Chapman
>
> <warwickchapman at gmail.com> wrote:
>
> Even if I increase the VM memory and swap to 8G each?
>
>
> -- Warwick Bruce Chapman | +27 83 7797 094 | http://warwickchapman.com
>
>
>
> On Fri, Nov 1, 2013 at 5:17 PM, Aaron Zeckoski <azeckoski at unicon.net>
>
> wrote:
>
>
> No, you need MaxPermSize and you will definitely run into using swap
>
> since your OS uses up memory as well. You better go with 2g and 512M
>
> for Xmx and MaxPerm for now and you can tune from there.
>
> -AZ
>
>
>
> On Fri, Nov 1, 2013 at 11:12 AM, Warwick Chapman
>
> <warwickchapman at gmail.com> wrote:
>
> Aaron, if I set the VMs to 8G MEM and SWAP, will the following be OK:
>
>
> export JAVA_OPTS='-server -d64 -Xmx4g -Djava.awt.headless=true
>
> -Dhttp.agent=Sakai'
>
>
> -- Warwick Bruce Chapman | +27 83 7797 094 |
>
> http://warwickchapman.com
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/production/attachments/20131101/575d471b/attachment.html