[Building Sakai] sakai mysql db down

Tue Dec 17 15:17:35 PST 2013

Whats the cause of the communication link failures bertween Sakai and your
database? If something isn't properly transactional, then interrupting it
in the middle might cause some sort of corruption, but I would say this is
not normal at all. I would be checking the root cause of these issues.

On Wed, Dec 18, 2013 at 10:10 AM, Sanghyun Jeon <euksa99 at gmail.com> wrote:

> Oh. We successfully recovered the table already. We are just wondering
> whether SAKAI_PREFERENCES table or Preference tool might have something
> causing this problem. We are currently having a lot of db related bug
> reports such as "caused by:
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link failure" and mysql db problem is often followed by our sakai crashes.
> Last week we have had two major crashes (four app nodes shut down) after db
> shut down and one is related with SAKAI_PREFERENCES table corruption. This
> week we have system crashes twice already (not related with
> SAKAI_PREFERENCES table, though but one is clearly related with db shut
> down -- it says mysqld normal shutdown but nobody shut down mysql at that
> time) but we don't have a clear clue why.
>
> Any advice and suggestion will be greatly appreciated.
>
> Thanks.
>
> S
>
>
> On Tue, Dec 17, 2013 at 2:28 PM, John Bush <jbush at anisakai.com> wrote:
>
>> Doesn't sound like a Sakai specific issue, try googling around for how
>> to recover mysql.  For example,
>>
>> http://rivenlinux.info/how-to-recover-innodb-corruption-for-mysql/
>>
>> On Tue, Dec 17, 2013 at 2:47 PM, Sanghyun Jeon <euksa99 at gmail.com> wrote:
>> > mysql log said so and please see the lines below highlighted on red.
>> >
>> > nnoDB: Page number (if stored to page already) 23,
>> >
>> > InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 171
>> >
>> > InnoDB: Page may be an index page where index id is 0 492
>> >
>> > InnoDB: (index "SAKAI_PREFERENCES_INDEX" of table
>> > "sakai"."SAKAI_PREFERENCES")
>> >
>> > InnoDB: Database page corruption on disk or a failed
>> >
>> > InnoDB: file read of page 187.
>> >
>> > InnoDB: You may have to recover from a backup.
>> >
>> > InnoDB: It is also possible that your operating
>> >
>> > InnoDB: system has corrupted its own file cache
>> >
>> > InnoDB: and rebooting your computer removes the
>> >
>> > InnoDB: error.
>> >
>> > InnoDB: If the corrupt page is an index page
>> >
>> > InnoDB: you can also try to fix the corruption
>> >
>> > InnoDB: by dumping, dropping, and reimporting
>> >
>> > InnoDB: the corrupt table. You can use CHECK
>> >
>> > InnoDB: TABLE to scan your table for corruption.
>> >
>> > InnoDB: See also
>> > http://dev.mysql.com/doc/refman/5.1/en/forcing-innodb-recovery.html
>> >
>> > InnoDB: about forcing recovery.
>> >
>> > InnoDB: Ending processing because of a corrupt database page.
>> >
>> > 131120 13:28:37 mysqld_safe Number of processes running now: 0
>> >
>> > 131120 13:28:37 mysqld_safe mysqld restarted
>> >
>> >
>> >
>> > On Mon, Dec 16, 2013 at 3:52 PM, Steve Swinsburg <
>> steve.swinsburg at gmail.com>
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> Do you have any logs for what happened? When you say 'because of
>> >> sakai_preferences table corruption' - what do you mean?
>> >>
>> >> cheers,
>> >> Steve
>> >>
>> >>
>> >> On Tue, Dec 17, 2013 at 5:05 AM, Sanghyun Jeon <euksa99 at gmail.com>
>> wrote:
>> >>>
>> >>> Hello all,
>> >>>
>> >>> We have a system crash last week because of sakai_preferences table
>> >>> corruption and we recovered the table successfully but the sakai db
>> shut
>> >>> down again this morning without reason.
>> >>>
>> >>> Version: '5.1.69-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306
>> >>> Source distribution
>> >>> 131216  8:42:56 [Note] /usr/libexec/mysqld: Normal shutdown
>> >>>
>> >>> 131216  8:42:56 [Note] Event Scheduler: Purging the queue. 0 events
>> >>> 131216  8:42:59  InnoDB: Starting shutdown...
>> >>> 131216  8:43:23  InnoDB: Shutdown completed; log sequence number 12
>> >>> 4183647936
>> >>> 131216  8:43:23 [Note] /usr/libexec/mysqld: Shutdown complete
>> >>>
>> >>> 131216 08:43:23 mysqld_safe mysqld from pid file
>> >>> /var/run/mysqld/mysqld.pid ended
>> >>> 131216 08:55:17 mysqld_safe Starting mysqld daemon with databases
>> from
>> >>> /opt/mysql
>> >>> 131216  8:55:17  InnoDB: Initializing buffer pool, size = 5.9G
>> >>> 131216  8:55:18  InnoDB: Completed initialization of buffer pool
>> >>> 131216  8:55:18  InnoDB: Started; log sequence number 12 4183647936
>> >>> InnoDB: !!! innodb_force_recovery is set to 4 !!!
>> >>>
>> >>>
>> >>> We are using Sakai 2.8.x and Tomcat 6 and mysql 5.1.69
>> >>> I am wondering whether any other school is having the similar
>> problem? I
>> >>> am also wondering anybody can shed some light on how I can prevent
>> this and
>> >>> we are currently at final exam period.  Any suggestion or advice will
>> be
>> >>> greatly appreciated.
>> >>>
>> >>> S
>> >>>
>> >>> _______________________________________________
>> >>> sakai-dev mailing list
>> >>> sakai-dev at collab.sakaiproject.org
>> >>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>> >>>
>> >>> TO UNSUBSCRIBE: send email to
>> >>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
>> >>> "unsubscribe"
>> >>
>> >>
>> >
>> >
>> > _______________________________________________
>> > sakai-dev mailing list
>> > sakai-dev at collab.sakaiproject.org
>> > http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>> >
>> > TO UNSUBSCRIBE: send email to
>> sakai-dev-unsubscribe at collab.sakaiproject.org
>> > with a subject of "unsubscribe"
>>
>>
>>
>> --
>> John Bush
>> 602-490-0470
>>
>> ** This message is neither private nor confidential in fact the US
>> government is storing it in a warehouse located in Utah for future
>> data mining use cases should they arise. **
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20131218/dcc6c570/attachment.html 