[Building Sakai] Major issues restart Sakai this morning due to Lesson builder and or hibernate

Aaron Zeckoski azeckoski at unicon.net
Sat Oct 12 06:36:59 PDT 2013 

Jeremy,
I think something else is at play here. I have run Sakai on planes and
in other situations where the networking was not available without an
issue. I also just now verified I could startup Sakai by turning off
my networking and starting it with no issues.

Maybe there is something about the server environment that is causing
DTDs to be validated (or perhaps there is some kind of caching issue)?

A couple pages that might be helpful:
http://stackoverflow.com/questions/2256289/prevent-503-error-from-w3-org-when-spring-tries-to-load-xml-files
http://stackoverflow.com/questions/10334610/hibernate-mapping-not-working-in-offline-mode

I also think this came up before and there was a JIRA about it but I
am not seeing it when I try to search.

-AZ

On Sat, Oct 12, 2013 at 8:26 AM, Kusnetz, Jeremy <JKusnetz at apus.edu> wrote:
> Assuming my issues really were because it couldn’t pull the hibernate
> mapping file.
>
>
>
> I noticed that the other tools are using
> http://hibernate.sourceforge.net/hibernate-mapping-3.0.dtd and not
> http://www.hibernate.org/dtd/hibernate-mapping-3.0.dtd
>
>
>
> Which would explain why only removing Lesson Builder allowed Sakai to start.
>
>
>
> That said, should this be something we pull locally instead of from
> sourceforge.net or hibernate.org?  If either are down, as we have seen,
> Sakai won’t start.
>
>
>
> From: sakai-dev-bounces at collab.sakaiproject.org
> [mailto:sakai-dev-bounces at collab.sakaiproject.org] On Behalf Of Kusnetz,
> Jeremy
> Sent: Saturday, October 12, 2013 8:06 AM
> To: sakai-dev at collab.sakaiproject.org
> Subject: [Building Sakai] Major issues restart Sakai this morning due to
> Lesson builder and or hibernate
>
>
>
> We are using Sakai 2.9.x which we pulled at a very late 2.9.3 RC.
>
>
>
> This morning we were doing our normal restarts of Sakai at 5am.  About half
> of our nodes started up normally and the other half couldn’t start.
>
>
>
> When the nodes couldn’t start, we tried starting other environments that
> were completely unrelated and they couldn’t start either.
>
>
>
> Eventually we found that Lesson builder was throwing all sorts of errors.
> We haven’t touched lesson builder at all, and in fact we currently do not
> use it.
>
>
>
> I was able to get Sakai to start by removing all the lesson builder files
> from webapp, shared/lib and components.
>
>
>
> After, in our dev environment I pulled the latest 1.4.x snapshot of lesson
> builder, built and deployed it and Sakai still couldn’t start up.  I went to
> do some more digging and tried to restart again around 7:30am eastern and
> all of a sudden sakai was starting with lesson builder.  I even went to a
> separate environment that had the same version of lesson builder as
> production that wasn’t touched and even though it couldn’t start earlier,
> was now able to restart on it’s own with no errors.
>
>
>
> Is lesson builder making some call outside of the network when Sakai starts?
> Could it have been trying to call something that was down?  That’s the only
> thing I can think of to explain why Sakai was able to restart on it’s own
> later, and initially half of the nodes were able to restart.
>
>
>
> Actually looking at the logs, something in hibernate is calling out?
>
>
>
> Caused by: org.dom4j.DocumentException: Server returned HTTP response code:
> 503 for URL: http://www.hibernate.org/dtd/hibernate-mapping-3.0.dtd Nested
> exception: Server returned HTTP response code: 503 for URL:
> http://www.hibernate.org/dtd/hibernate-mapping-3.0.dtd
>
>
>
> If it was hibernate, why did removing lesson builder allow Sakai to start?
> If this hibernate error above is correct, why are we depending on
> hibernate.org to be up?
>
>
>
> Here are some logs showing the errors:
>
>
>
> 2013-10-12 05:19:59,634 ERROR localhost-startStop-1
> org.sakaiproject.util.NoisierDefaultListableBeanFactory - Failed to
> preinstantiate the singleton named
> org.sakaiproject.accountvalidator.dao.impl.ValidationLogicDaoTarget.
> Destroying all Spring beans.
>
> org.springframework.beans.factory.BeanCreationException: Error creating bean
> with name
> 'org.sakaiproject.accountvalidator.dao.impl.ValidationLogicDaoTarget'
> defined in file
> [/SAKAI/apus/tomcat_servers/servers/apache-tomcat-7.0.42-community-2.9.x-100/components/accountvalidator-pack/WEB-INF/spring-hibernate.xml]:
> Cannot resolve reference to bean
> 'org.sakaiproject.springframework.orm.hibernate.GlobalSessionFactory' while
> setting bean property 'sessionFactory'; nested exception is
> org.springframework.beans.factory.BeanCreationException: Error creating bean
> with name
> 'org.sakaiproject.springframework.orm.hibernate.GlobalSessionFactory'
> defined in file
> [/SAKAI/apus/tomcat_servers/servers/apache-tomcat-7.0.42-community-2.9.x-100/components/sakai-kernel-component/WEB-INF/db-components.xml]:
> Invocation of init method failed; nested exception is
> org.hibernate.MappingException: Failed to load class path resource
> [org/sakaiproject/lessonbuildertool/simplepageproperty.hbm.xml]
>
>         at
> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:275)
>
>         at
> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:104)
>
>         at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1245)
>
>         at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1010)
>
>         at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:472)
>
>         at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)
>
>         at
> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)
>
>         at
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
>
>         at
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
>
>         at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)
>
>         at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
>
>         at
> org.sakaiproject.util.NoisierDefaultListableBeanFactory.getBean(NoisierDefaultListableBeanFactory.java:112)
>
>         at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
>
>         at
> org.sakaiproject.util.NoisierDefaultListableBeanFactory.preInstantiateSingletons(NoisierDefaultListableBeanFactory.java:73)
>
>         at
> org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:728)
>
>         at
> org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:380)
>
>         at
> org.sakaiproject.component.impl.SpringCompMgr.init(SpringCompMgr.java:160)
>
>         at
> org.sakaiproject.component.cover.ComponentManager.getInstance(ComponentManager.java:98)
>
>         at
> org.sakaiproject.component.cover.ComponentManager.get(ComponentManager.java:107)
>
>         at
> org.sakaiproject.component.cover.ServerConfigurationService.getInstance(ServerConfigurationService.java:52)
>
>         at
> org.sakaiproject.component.cover.ServerConfigurationService.getSakaiHomePath(ServerConfigurationService.java:173)
>
>         at
> org.sakaiproject.util.ToolListener.contextInitialized(ToolListener.java:66)
>
>         at
> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4939)
>
>         at
> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5434)
>
>         at
> org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
>
>         at
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
>
>         at
> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
>
>         at
> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:633)
>
>         at
> org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:976)
>
>         at
> org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1653)
>
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
>
>         at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>
>         at java.lang.Thread.run(Thread.java:662)
>
> Caused by: org.springframework.beans.factory.BeanCreationException: Error
> creating bean with name
> 'org.sakaiproject.springframework.orm.hibernate.GlobalSessionFactory'
> defined in file
> [/SAKAI/apus/tomcat_servers/servers/apache-tomcat-7.0.42-community-2.9.x-100/components/sakai-kernel-component/WEB-INF/db-components.xml]:
> Invocation of init method failed; nested exception is
> org.hibernate.MappingException: Failed to load class path resource
> [org/sakaiproject/lessonbuildertool/simplepageproperty.hbm.xml]
>
>         at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1338)
>
>         at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:473)
>
>         at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:473)
>
>         at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)
>
>         at
> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)
>
>         at
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
>
>         at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)
>
>         at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
>
>         at
> org.sakaiproject.util.NoisierDefaultListableBeanFactory.getBean(NoisierDefaultListableBeanFactory.java:112)
>
>         at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
>
>         at
> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:269)
>
>         ... 36 more
>
> Caused by: org.hibernate.MappingException: Failed to load class path
> resource [org/sakaiproject/lessonbuildertool/simplepageproperty.hbm.xml]
>
>        at
> org.sakaiproject.springframework.orm.hibernate.impl.AdditionalHibernateMappingsImpl.processConfig(AdditionalHibernateMappingsImpl.java:65)
>
>         at
> org.sakaiproject.springframework.orm.hibernate.AddableSessionFactoryBean.postProcessConfiguration(AddableSessionFactoryBean.java:78)
>
>         at
> org.springframework.orm.hibernate3.LocalSessionFactoryBean.buildSessionFactory(LocalSessionFactoryBean.java:727)
>
>         at
> org.springframework.orm.hibernate3.AbstractSessionFactoryBean.afterPropertiesSet(AbstractSessionFactoryBean.java:211)
>
>         at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1369)
>
>         at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1335)
>
>         ... 47 more
>
> Caused by: org.hibernate.InvalidMappingException: Could not parse mapping
> document from input stream
>
>         at
> org.hibernate.cfg.Configuration.addInputStream(Configuration.java:527)
>
>         at
> org.sakaiproject.springframework.orm.hibernate.impl.AdditionalHibernateMappingsImpl.processConfig(AdditionalHibernateMappingsImpl.java:63)
>
>         ... 52 more
>
> Caused by: org.dom4j.DocumentException: Server returned HTTP response code:
> 503 for URL: http://www.hibernate.org/dtd/hibernate-mapping-3.0.dtd Nested
> exception: Server returned HTTP response code: 503 for URL:
> http://www.hibernate.org/dtd/hibernate-mapping-3.0.dtd
>
>         at org.dom4j.io.SAXReader.read(SAXReader.java:484)
>
>         at
> org.hibernate.cfg.Configuration.addInputStream(Configuration.java:518)
>
>         ... 53 more
>
> java.lang.Exception: traceback
>
>         at
> org.sakaiproject.util.NoisierDefaultListableBeanFactory.destroySingletons(NoisierDefaultListableBeanFactory.java:94)
>
>         at
> org.sakaiproject.util.NoisierDefaultListableBeanFactory.preInstantiateSingletons(NoisierDefaultListableBeanFactory.java:83)
>
>         at
> org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:728)
>
>         at
> org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:380)
>
>         at
> org.sakaiproject.component.impl.SpringCompMgr.init(SpringCompMgr.java:160)
>
>         at
> org.sakaiproject.component.cover.ComponentManager.getInstance(ComponentManager.java:98)
>
>         at
> org.sakaiproject.component.cover.ComponentManager.get(ComponentManager.java:107)
>
>         at
> org.sakaiproject.component.cover.ServerConfigurationService.getInstance(ServerConfigurationService.java:52)
>
>         at
> org.sakaiproject.component.cover.ServerConfigurationService.getSakaiHomePath(ServerConfigurationService.java:173)
>
>         at
> org.sakaiproject.util.ToolListener.contextInitialized(ToolListener.java:66)
>
>         at
> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4939)
>
>         at
> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5434)
>
>         at
> org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
>
>         at
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
>
>         at
> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
>
>        at
> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:633)
>
>         at
> org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:976)
>
>         at
> org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1653)
>
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
>
>         at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>
>         at java.lang.Thread.run(Thread.java:662)
>
> This message is private and confidential. If you have received it in error,
> please notify the sender and remove it from your system.
>
> This message is private and confidential. If you have received it in error,
> please notify the sender and remove it from your system.
>
>
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org
> with a subject of "unsubscribe"



-- 
Aaron Zeckoski - Software Architect - http://tinyurl.com/azprofile

More information about the sakai-dev mailing list