[Deploying Sakai] [Building Sakai] ClientAbortException: java.net.SocketException: Broken pipe

Sun Nov 3 23:55:58 PST 2013

Solid as a rock.  Thank you very much.

-- Warwick Bruce Chapman | +27 83 7797 094 | http://warwickchapman.com

On Sun, Nov 3, 2013 at 11:23 AM, Warwick Chapman
<warwickchapman at gmail.com>wrote:

> Thank you all.  So far, so good today.  We're nearly half way through the
> tests and have 71 sessions at the moment, with the three appservers now at
> a LA of 0.00-0.08 each (though they do occasionally jump up to 0.40 or so
> and then drop back down.
>
> Memory utilisation is at 35% and 39% on the two appservers with 8.0G
> memory (Xmx4g), and on the 6.0G server (Xmx4g) is 56%.
>
> I am hopeful that it will remain solid for the rest of the day.
>
> Regardless, thank you to all of you who assisted me to get this issue
> resolved.  I appreciate it a great deal.
>
> -- Warwick Bruce Chapman | +27 83 7797 094 | http://warwickchapman.com
>
>
> On Fri, Nov 1, 2013 at 9:02 PM, Warwick Chapman <warwickchapman at gmail.com>wrote:
>
>> Noah, thank you for this.  Much appreciated.  Let me try the 4G heap on
>> the VMs with 8G of memory.  The third VM has 6GB and I cannot increase it
>> much more than that.
>>
>> The proviso of 25% for the system would then also fit into that VM with
>> Xmx 4G (67%) and system 2G (33%).
>>
>> I'm going to give it a crack.  If it goes down, there's be no heap
>> analysis, I'll have find somewhere to hide.
>>
>> Aspirant members of parliament writing 'candidate tests' - the people
>> take becoming an MP *very* seriously, and don't take lightly to the server
>> falling over mid-test.  It happened on Saturday last week - during the 24
>> hours period the exam ran for, I lost application servers times for 250
>> applicants.  At that stage I had an Xmx of 1024m, and 2 vCPUs (and then 4
>> during a running reconfigure) per appserver.
>>
>> Now I have an extra appserver, Xmx of 4G and 4 vCPUs per appserver.
>>
>> -- Warwick Bruce Chapman | +27 83 7797 094 | http://warwickchapman.com
>>
>>
>> On Fri, Nov 1, 2013 at 6:34 PM, Noah Botimer <botimer at umich.edu> wrote:
>>
>>> It sounds like you were running at 1GB heap, which is quite likely too
>>> low for an instance with lots of modules/webapps deployed. Increasing Xmx
>>> should do the trick, but in case you still have problems...
>>>
>>> You can flip the switch to dump heap on crash (and once it falls over),
>>> you could analyze the dump:
>>>
>>>
>>> http://stackoverflow.com/questions/542979/using-heapdumponoutofmemoryerror-parameter-for-heap-dump-for-jboss
>>>
>>> I've had good success with YourKit for examining heaps, but jhat and
>>> VisualVM can do it too. You would be looking for a particular class or set
>>> of classes that are disproportionately allocated (say 50% or more of
>>> total), indicating a leak (often a reference from an item to a
>>> container/observer that persists but no longer tracks the child).
>>>
>>> If you observe that a JVM (via jstat or other) has a growing heap but is
>>> not yet to crashing, you can dump the heap with jmap.
>>>
>>> I think it's pretty common these days to run with 4 and 6GB heaps on
>>> 64-bit JVMs (with Aaron's proviso of ~25% spare for system).
>>>
>>> Thanks,
>>> -Noah
>>>
>>>
>>> On Nov 1, 2013, at 11:51 AM, Aaron Zeckoski wrote:
>>>
>>> I would say 2g is adequate but I am adding the production list to this
>>> to see if anyone else has any comment here. Generally a single
>>> instance would be more than adequate for 60 users. We certainly are
>>> running with many more users per instance in many cases without issue.
>>>
>>> -AZ
>>>
>>>
>>> On Fri, Nov 1, 2013 at 11:36 AM, Warwick Chapman
>>> <warwickchapman at gmail.com> wrote:
>>>
>>> Ok, please give me your best guess because I don't have any opportunity
>>> left
>>>
>>> to tune.  On Sunday I've got to ensure this thing doesn't crash because
>>> of
>>>
>>> java.lang.OutOfMemoryError: Java heap space as I've been getting now ...
>>>
>>>
>>> I cannot believe I am working so hard to get Sakai stable for no more
>>> than
>>>
>>> 60 concurrent users.
>>>
>>>
>>> I have 3 OpenVZ on 3 separate pieces of tin.
>>>
>>>
>>> Two have been increased to 8192 and the third to 6144.
>>>
>>>
>>> Their JAVA_OPTS are now, respectively:
>>>
>>>
>>> 1. export JAVA_OPTS='-server -d64 -Xmx4g -XX:MaxPermSize=512m
>>>
>>> -Djava.awt.headless=true -Dhttp.agent=Sakai'
>>>
>>> 2. export JAVA_OPTS='-server -d64 -Xmx4g -XX:MaxPermSize=512m
>>>
>>> -Djava.awt.headless=true -Dhttp.agent=Sakai'
>>>
>>> 3. export JAVA_OPTS='-server -d64 -Xmx3g -XX:MaxPermSize=512m
>>>
>>> -Djava.awt.headless=true -Dhttp.agent=Sakai'
>>>
>>>
>>> I am worried that keeping them at 4192 and doubling up from the original
>>>
>>> 1024 will not be sufficient.  But I will be guided by you.
>>>
>>>
>>>
>>> -- Warwick Bruce Chapman | +27 83 7797 094 | http://warwickchapman.com
>>>
>>>
>>>
>>> On Fri, Nov 1, 2013 at 5:28 PM, Aaron Zeckoski <azeckoski at unicon.net>
>>> wrote:
>>>
>>>
>>> The "No" was for your "OK" question since you are missing MaxPermSize.
>>>
>>> You can set it to something higher but if it were me I would not do
>>>
>>> it. Larger memory space means it takes the JVM longer to do garbage
>>>
>>> collections. I would suggest you start with something smaller and tune
>>>
>>> it until things are working well for your users and load. Like I said,
>>>
>>> there is no single magic or "right" number. There are lots of wrong
>>>
>>> ones though.
>>>
>>>
>>> -AZ
>>>
>>>
>>>
>>> On Fri, Nov 1, 2013 at 11:20 AM, Warwick Chapman
>>>
>>> <warwickchapman at gmail.com> wrote:
>>>
>>> Even if I increase the VM memory and swap to 8G each?
>>>
>>>
>>> -- Warwick Bruce Chapman | +27 83 7797 094 | http://warwickchapman.com
>>>
>>>
>>>
>>> On Fri, Nov 1, 2013 at 5:17 PM, Aaron Zeckoski <azeckoski at unicon.net>
>>>
>>> wrote:
>>>
>>>
>>> No, you need MaxPermSize and you will definitely run into using swap
>>>
>>> since your OS uses up memory as well. You better go with 2g and 512M
>>>
>>> for Xmx and MaxPerm for now and you can tune from there.
>>>
>>> -AZ
>>>
>>>
>>>
>>> On Fri, Nov 1, 2013 at 11:12 AM, Warwick Chapman
>>>
>>> <warwickchapman at gmail.com> wrote:
>>>
>>> Aaron, if I set the VMs to 8G MEM and SWAP, will the following be OK:
>>>
>>>
>>> export JAVA_OPTS='-server -d64 -Xmx4g -Djava.awt.headless=true
>>>
>>> -Dhttp.agent=Sakai'
>>>
>>>
>>> -- Warwick Bruce Chapman | +27 83 7797 094 |
>>>
>>> http://warwickchapman.com
>>>
>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/production/attachments/20131104/db2e37bc/attachment.html 