[Deploying Sakai] [Building Sakai] ClientAbortException: java.net.SocketException: Broken pipe
Mike DeSimone
mdesimone at anisakai.com
Fri Nov 1 09:21:49 PDT 2013
I would agree with Aaron about the single instance; if you can, run one
10gb VM, set max heap to 6gb, max perm to 600m, will be fine. At least 2
vcpu, 4 is better. 6gb heap is if you want to be extra-sure to avoid any
OOM. You could tune the Xmx down to 4500m and likely be ok, but Sakai
loves heap, so go big there if you can.
Here are my suggestions (i am excluding the d64, headless, agent, etc. and
other non-memory related, but useful settings that you should retain). We
regularly run over 500 concurrent users with settings like this:
-Xmx6g -XX:PermSize=600m -XX:MaxPermSize=600m -XX:MaxNewSize=1500m
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
-XX:+UseCMSInitiatingOccupancyOnly
Thanks,
*********************************************
Mike DeSimone
Director of Enterprise Operations
Asahi Net International, Inc. (*ANI*)
O: +1 (602) 490-0473
W: anisakai.com
[image: Inline image 1]
<hq8t-hgc at asahi-net.or.jp>
**********************************************
On Fri, Nov 1, 2013 at 10:51 AM, Aaron Zeckoski <azeckoski at unicon.net>wrote:
> I would say 2g is adequate but I am adding the production list to this
> to see if anyone else has any comment here. Generally a single
> instance would be more than adequate for 60 users. We certainly are
> running with many more users per instance in many cases without issue.
>
> -AZ
>
>
> On Fri, Nov 1, 2013 at 11:36 AM, Warwick Chapman
> <warwickchapman at gmail.com> wrote:
> > Ok, please give me your best guess because I don't have any opportunity
> left
> > to tune. On Sunday I've got to ensure this thing doesn't crash because
> of
> > java.lang.OutOfMemoryError: Java heap space as I've been getting now ...
> >
> > I cannot believe I am working so hard to get Sakai stable for no more
> than
> > 60 concurrent users.
> >
> > I have 3 OpenVZ on 3 separate pieces of tin.
> >
> > Two have been increased to 8192 and the third to 6144.
> >
> > Their JAVA_OPTS are now, respectively:
> >
> > 1. export JAVA_OPTS='-server -d64 -Xmx4g -XX:MaxPermSize=512m
> > -Djava.awt.headless=true -Dhttp.agent=Sakai'
> > 2. export JAVA_OPTS='-server -d64 -Xmx4g -XX:MaxPermSize=512m
> > -Djava.awt.headless=true -Dhttp.agent=Sakai'
> > 3. export JAVA_OPTS='-server -d64 -Xmx3g -XX:MaxPermSize=512m
> > -Djava.awt.headless=true -Dhttp.agent=Sakai'
> >
> > I am worried that keeping them at 4192 and doubling up from the original
> > 1024 will not be sufficient. But I will be guided by you.
> >
> >
> > -- Warwick Bruce Chapman | +27 83 7797 094 | http://warwickchapman.com
> >
> >
> > On Fri, Nov 1, 2013 at 5:28 PM, Aaron Zeckoski <azeckoski at unicon.net>
> wrote:
> >>
> >> The "No" was for your "OK" question since you are missing MaxPermSize.
> >> You can set it to something higher but if it were me I would not do
> >> it. Larger memory space means it takes the JVM longer to do garbage
> >> collections. I would suggest you start with something smaller and tune
> >> it until things are working well for your users and load. Like I said,
> >> there is no single magic or "right" number. There are lots of wrong
> >> ones though.
> >>
> >> -AZ
> >>
> >>
> >> On Fri, Nov 1, 2013 at 11:20 AM, Warwick Chapman
> >> <warwickchapman at gmail.com> wrote:
> >> > Even if I increase the VM memory and swap to 8G each?
> >> >
> >> > -- Warwick Bruce Chapman | +27 83 7797 094 |
> http://warwickchapman.com
> >> >
> >> >
> >> > On Fri, Nov 1, 2013 at 5:17 PM, Aaron Zeckoski <azeckoski at unicon.net>
> >> > wrote:
> >> >>
> >> >> No, you need MaxPermSize and you will definitely run into using swap
> >> >> since your OS uses up memory as well. You better go with 2g and 512M
> >> >> for Xmx and MaxPerm for now and you can tune from there.
> >> >> -AZ
> >> >>
> >> >>
> >> >> On Fri, Nov 1, 2013 at 11:12 AM, Warwick Chapman
> >> >> <warwickchapman at gmail.com> wrote:
> >> >> > Aaron, if I set the VMs to 8G MEM and SWAP, will the following be
> OK:
> >> >> >
> >> >> > export JAVA_OPTS='-server -d64 -Xmx4g -Djava.awt.headless=true
> >> >> > -Dhttp.agent=Sakai'
> >> >> >
> >> >> > -- Warwick Bruce Chapman | +27 83 7797 094 |
> >> >> > http://warwickchapman.com
> >> >> >
> >> >> >
> >> >> > On Fri, Nov 1, 2013 at 5:09 PM, Aaron Zeckoski <
> azeckoski at unicon.net>
> >> >> > wrote:
> >> >> >>
> >> >> >> There is no magic number but you would definitely not want to
> exceed
> >> >> >> 3G for Xmx because if you allow the JVM to start using swap then
> it
> >> >> >> will die very quickly. Garbage collecting swap space causes the
> JVM
> >> >> >> to
> >> >> >> basically become unresponsive. 2G is probably safe assuming you
> >> >> >> don't
> >> >> >> have a ton of other things running on there. The key is to not set
> >> >> >> Xmx
> >> >> >> any higher than the amount of free ram you have (minus the
> MaxPerm).
> >> >> >>
> >> >> >> -AZ
> >> >> >>
> >> >> >> On Fri, Nov 1, 2013 at 11:03 AM, Warwick Chapman
> >> >> >> <warwickchapman at gmail.com> wrote:
> >> >> >> > Aaron, if my VM had 4096M memory and 4096 swap, what should Xmx
> >> >> >> > be?
> >> >> >> >
> >> >> >> > -- Warwick Bruce Chapman | +27 83 7797 094 |
> >> >> >> > http://warwickchapman.com
> >> >> >> >
> >> >> >> >
> >> >> >> > On Fri, Nov 1, 2013 at 1:30 PM, Aaron Zeckoski
> >> >> >> > <azeckoski at unicon.net>
> >> >> >> > wrote:
> >> >> >> >>
> >> >> >> >> Looks like MaxPermSize is still in there and you will need to
> >> >> >> >> define
> >> >> >> >> Xmx or the JVM will use the default for that which is much too
> >> >> >> >> small.
> >> >> >> >>
> >> >> >> >> Also, be aware that the GC printing will slow things down a bit
> >> >> >> >> so
> >> >> >> >> you
> >> >> >> >> don't want to run live with that on. If you need GC stats then
> >> >> >> >> use a
> >> >> >> >> profiler or jstat to grab a quick snapshot of the memory
> >> >> >> >> situation
> >> >> >> >> every once in awhile.
> >> >> >> >>
> >> >> >> >> -AZ
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> On Fri, Nov 1, 2013 at 7:21 AM, Warwick Chapman
> >> >> >> >> <warwickchapman at gmail.com> wrote:
> >> >> >> >> > Ok, so nothing serious then.
> >> >> >> >> >
> >> >> >> >> > On another, note, as per my previous thread this morning, you
> >> >> >> >> > previously
> >> >> >> >> > suggested a less specific set of JAVA_OPTS in order to allow
> >> >> >> >> > the
> >> >> >> >> > JVM
> >> >> >> >> > to
> >> >> >> >> > autotune (ergonomics). I have taken all but
> >> >> >> >> > -XX:MaxPermSize=512m
> >> >> >> >> > out
> >> >> >> >> > and am
> >> >> >> >> > giving it a bash. Is this ok?
> >> >> >> >> >
> >> >> >> >> > JAVA_OPTS='-server -d64 -XX:MaxPermSize=512m
> >> >> >> >> > -Djava.awt.headless=true
> >> >> >> >> > -Dhttp.agent=Sakai -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
> >> >> >> >> > -verbose:gc'
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > -- Warwick Bruce Chapman | +27 83 7797 094 |
> >> >> >> >> > http://warwickchapman.com
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > On Fri, Nov 1, 2013 at 1:06 PM, Aaron Zeckoski
> >> >> >> >> > <azeckoski at unicon.net>
> >> >> >> >> > wrote:
> >> >> >> >> >>
> >> >> >> >> >> Warwick,
> >> >> >> >> >> This is typically what that error indicates:
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> http://stackoverflow.com/questions/839314/clientabortexception-java-net-socketexception-connection-reset-by-peer-socket
> >> >> >> >> >>
> >> >> >> >> >> -AZ
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >> On Fri, Nov 1, 2013 at 6:55 AM, Warwick Chapman
> >> >> >> >> >> <warwickchapman at gmail.com> wrote:
> >> >> >> >> >> > Hi All
> >> >> >> >> >> >
> >> >> >> >> >> > Any idea what the following error is all about?
> >> >> >> >> >> >
> >> >> >> >> >> > 2013-11-01 12:51:49,585 WARN http-bio-8080-exec-17
> >> >> >> >> >> > org.sakaiproject.util.RequestFilter -
> >> >> >> >> >> > ClientAbortException: java.net.SocketException: Broken
> pipe
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:413)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:366)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:438)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:426)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:91)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.catalina.servlets.DefaultServlet.copy(DefaultServlet.java:1861)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.catalina.servlets.DefaultServlet.serveResource(DefaultServlet.java:961)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.catalina.servlets.DefaultServlet.doGet(DefaultServlet.java:411)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:695)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> >> >> >> >> >> > at java.lang.Thread.run(Thread.java:662)
> >> >> >> >> >> > Caused by: java.net.SocketException: Broken pipe
> >> >> >> >> >> > at java.net.SocketOutputStream.socketWrite0(Native
> >> >> >> >> >> > Method)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> java.net.SocketOutputStream.write(SocketOutputStream.java:136)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:371)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:240)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:84)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:192)
> >> >> >> >> >> > at
> >> >> >> >> >> > org.apache.coyote.Response.doWrite(Response.java:517)
> >> >> >> >> >> > at
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:408)
> >> >> >> >> >> > ... 29 more
> >> >> >> >> >> >
> >> >> >> >> >> > -- Warwick Bruce Chapman | +27 83 7797 094 |
> >> >> >> >> >> > http://warwickchapman.com
> >> >> >> >> >> >
> >> >> >> >> >> > _______________________________________________
> >> >> >> >> >> > sakai-dev mailing list
> >> >> >> >> >> > sakai-dev at collab.sakaiproject.org
> >> >> >> >> >> > http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> >> >> >> >> >> >
> >> >> >> >> >> > TO UNSUBSCRIBE: send email to
> >> >> >> >> >> > sakai-dev-unsubscribe at collab.sakaiproject.org
> >> >> >> >> >> > with a subject of "unsubscribe"
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >> --
> >> >> >> >> >> Aaron Zeckoski - Software Architect -
> >> >> >> >> >> http://tinyurl.com/azprofile
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> --
> >> >> >> >> Aaron Zeckoski - Software Architect -
> >> >> >> >> http://tinyurl.com/azprofile
> >> >> >> >
> >> >> >> >
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> Aaron Zeckoski - Software Architect -
> http://tinyurl.com/azprofile
> >> >> >
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Aaron Zeckoski - Software Architect - http://tinyurl.com/azprofile
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Aaron Zeckoski - Software Architect - http://tinyurl.com/azprofile
> >
> >
>
>
>
> --
> Aaron Zeckoski - Software Architect - http://tinyurl.com/azprofile
> _______________________________________________
> production mailing list
> production at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/production
>
> TO UNSUBSCRIBE: send email to
> production-unsubscribe at collab.sakaiproject.org with a subject of
> "unsubscribe"
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/production/attachments/20131101/2a8fdeee/attachment-0001.html
More information about the production
mailing list