[Deploying Sakai] Sakai Instances 100% CPU Usage

[Jon Gorrono]

My Linux box, at my desk, had CPU peak problems when I came in Monday. I
had to reboot. It was not debilitating, I just happened to noticed the
numbers.

Linux psl-6 2.6.43.8-1.fc15.x86_64 #1 SMP Mon Jun 4 20:33:44 UTC 2012
x86_64 x86_64 x86_64 GNU/Linux

I just checked all our Sakai nodes and only one has mildly inflated
numbers: 35-85% ... load ave's are less than 1.0 but 6-8X higher than the
other nodes.

We are running with

2.6.18-274.18.1.el5 #1 SMP Thu Feb 9 12:45:44 EST 2012 x86_64 x86_64 x86_64
GNU/Linux

On Tue, Jul 3, 2012 at 8:22 AM, Beth Kirschner <bkirschn at umich.edu> wrote:

> There was a leap second this weekend (June 30, 8pm EDT) which we believe
> caused all our servers to hit 100% CPU usage simultaneously. Has anyone
> else in the Sakai community seen this? We're running:
>  Linux 2.6.32-220.17.1.el6.x86_64
> java version "1.6.0_22"
>
> Thanks,
> - Beth
>
> Begin forwarded message:
>
>
>
>
> URL = http://wiki.os-ux.com/java-leap-second-bug-30-june-1-july-2012-fix/
> java leap second bug – 30 June / 1 July 2012 – fix
> ------------------------------
>
> If your java applications suddenly started to use 100% CPU, you’ve hit
> java leap second bug (actually it’s a bug in Linux kernel, just java
> programs seem to be severely affected).
>
> First, you may check if you have the following in your dmesg:
>
>
> [10703552.860274] Clock: inserting leap second 23:59:60 UTC
>
> Fortunately the fix is straightforward:
>
> /etc/init.d/ntpd stop
> date -s "$(date)"
>
> /etc/init.d/ntpd start
>
> (the other suggested way around the web would be date `date
> +"%m%d%H%M%C%y.%S"`, having the same effect).
>
> You don’t have to restart your java applications (tomcat, solr, wowza, or
> whatever using java); it should be enough to run the above commands.
>
> You may want to enable ntp the next day (there were some reports that
> enabling it makes java misbehave on some installations).
>
> Apparently this issue has knocked down lots of Linux servers running
> Debian, Ubuntu, CentOS and the like around the globe!
>
>  read more <https://access.redhat.com/knowledge/articles/15145>
>
> (via wpkg.org<http://blog.wpkg.org/2012/07/01/java-leap-second-bug-30-june-1-july-2012-fix/>
> )
> Author: Vucomir Ianculov on July 3, 2012
> Category: Linux / Unix <http://wiki.os-ux.com/category/linux-unix/>
> Tags: date <http://wiki.os-ux.com/tag/date/>, java<http://wiki.os-ux.com/tag/java/>
> , kernel <http://wiki.os-ux.com/tag/kernel/>, linux<http://wiki.os-ux.com/tag/linux-2/>
> , ntpd <http://wiki.os-ux.com/tag/ntpd/>, Zimbra<http://wiki.os-ux.com/tag/zimbra/>
> [image: Protected by Copyscape DMCA Copyright Detector]<http://www.copyscape.com/dmca-copyright-protection/>
>
>
>
>
> _______________________________________________
> production mailing list
> production at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/production
>
> TO UNSUBSCRIBE: send email to
> production-unsubscribe at collab.sakaiproject.org with a subject of
> "unsubscribe"
>



-- 
Jon Gorrono
PGP Key: 0x5434509D - http{
pgp.mit.edu:11371/pks/lookup?search=0x5434509D&op=index}
GSWoT Introducer - {GSWoT:US75 5434509D Jon P. Gorrono <jpgorrono -
www.gswot.org>}
http{middleware.ucdavis.edu}
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/production/attachments/20120703/4782044f/attachment.html