[Building Sakai] HELP : sakai not starting up

[Jon Wheat]

Sakai crashed with a PerGenSpace at 4pm today, and typically a reboot brings it back up. 
When I start it up, I tail the log and it gets to one point and just stops.   

I've attached the log (debug mode), if anyone can take a look at it.  I'm at a loss.  Its the same place every time.   
For grins I enabled auto.dll=true  just to see if it would error out on the db calls, but it doesn't.  In fact there are no errors at all.  I let it sit for about 45 minutes when I packed up and came into the office, thinking maybe it was working it out, and it was still hung when I got here. 

its been down for almost 3 hours now and people are getting ... um ... upset 

Below is the last set of lines when I had auto.ddl enabled  (it is not enabled in the attached log file) - maybe that will help someone figure out where in the startup its getting wedged. 

-Jon 

2010-10-30 18:41:21,105 DEBUG main org.sakaiproject.db.impl.BasicSqlService - borrowConnection() 
2010-10-30 18:41:21,143 DEBUG main org.sakaiproject.db.impl.BasicSqlService - returnConnection(Connection jdbc:oracle:thin:@portaldb.messiah.edu:1521:SAKAIPROD, UserName=SAKAIMGR, Oracle JDBC driver) 
2010-10-30 18:41:21,348 DEBUG main org.sakaiproject.db.impl.BasicSqlService - ddl(ClassLoader org.apache.catalina.loader.StandardClassLoader at 51127a, String quartz) 
2010-10-30 18:41:21,349 DEBUG main org.sakaiproject.db.impl.BasicSqlService - dbWriteFailQuiet(Connection null, String  CREATE TABLE qrtz_job_details ( JOB_NAME  VARCHAR2(80) NOT NULL, JOB_GROUP VARCHAR2(80) NOT NULL, DESCRIPTION VARCHAR2(120) NULL, JOB_CLASS_NAME   VARCHAR2(128) NOT NULL, IS_DURABLE VARCHAR2(1) NOT NULL, IS_VOLATILE VARCHAR2(1) NOT NULL, IS_STATEFUL VARCHAR2(1) NOT NULL, REQUESTS_RECOVERY VARCHAR2(1) NOT NULL, JOB_DATA BLOB NULL, PRIMARY KEY (JOB_NAME,JOB_GROUP) ), Object[] null) 
2010-10-30 18:41:21,349 DEBUG main org.sakaiproject.db.impl.BasicSqlService - dbWrite(String  CREATE TABLE qrtz_job_details ( JOB_NAME  VARCHAR2(80) NOT NULL, JOB_GROUP VARCHAR2(80) NOT NULL, DESCRIPTION VARCHAR2(120) NULL, JOB_CLASS_NAME   VARCHAR2(128) NOT NULL, IS_DURABLE VARCHAR2(1) NOT NULL, IS_VOLATILE VARCHAR2(1) NOT NULL, IS_STATEFUL VARCHAR2(1) NOT NULL, REQUESTS_RECOVERY VARCHAR2(1) NOT NULL, JOB_DATA BLOB NULL, PRIMARY KEY (JOB_NAME,JOB_GROUP) ), Object[] null, String null, Connection null, boolean true) 
2010-10-30 18:41:21,349 DEBUG main org.sakaiproject.db.impl.BasicSqlService - Sql.dbWrite(): null 
 CREATE TABLE qrtz_job_details ( JOB_NAME  VARCHAR2(80) NOT NULL, JOB_GROUP VARCHAR2(80) NOT NULL, DESCRIPTION VARCHAR2(120) NULL, JOB_CLASS_NAME   VARCHAR2(128) NOT NULL, IS_DURABLE VARCHAR2(1) NOT NULL, IS_VOLATILE VARCHAR2(1) NOT NULL, IS_STATEFUL VARCHAR2(1) NOT NULL, REQUESTS_RECOVERY VARCHAR2(1) NOT NULL, JOB_DATA BLOB NULL, PRIMARY KEY (JOB_NAME,JOB_GROUP) ) 

2010-10-30 18:41:21,349 DEBUG main org.sakaiproject.db.impl.BasicSqlService - borrowConnection() 
2010-10-30 18:41:21,350 DEBUG main org.sakaiproject.db.impl.BasicSqlService - prepareStatement(PreparedStatement oracle.jdbc.driver.T4CPreparedStatement at 7aacc1, Object[] null) 
2010-10-30 18:41:21,477 DEBUG main org.sakaiproject.db.impl.BasicSqlService - returnConnection(Connection jdbc:oracle:thin:@portaldb.messiah.edu:1521:SAKAIPROD, UserName=SAKAIMGR, Oracle JDBC driver) 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20101030/9d0477fa/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: catalina.out
Type: application/octet-stream
Size: 18530 bytes
Desc: not available
Url : http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20101030/9d0477fa/attachment.obj