[Building Sakai] Major issues restart Sakai this morning due to Lesson builder and or hibernate
Kusnetz, Jeremy
JKusnetz at APUS.EDU
Sat Oct 12 09:04:14 PDT 2013
Reading this thread. If I'm understanding correctly since doctype was different in the simplepageproperty.hbm.xml than what is in the hibernate libraries which is set to point to sourceforge, it had to go out and pull the file instead of finding it in the classpath.
https://forum.hibernate.org/viewtopic.php?t=943281
"In your mapping files, you must have the exactly SAME doctype as found in the mapping DTD's.
Then and only then you'll see that the dtd's found in hibernate3.jar can be found through the classpath, and running behind a firewall, stand-alone, etc. will be no problem at all.
No local dtd's in your projects to solve this issue, no code to intercept. :-)"
From: sakai-dev-bounces at collab.sakaiproject.org [mailto:sakai-dev-bounces at collab.sakaiproject.org] On Behalf Of Kusnetz, Jeremy
Sent: Saturday, October 12, 2013 10:31 AM
To: 'Niebel, William (wdn5e)'; sakai-dev at collab.sakaiproject.org; azeckoski at unicon.net
Subject: Re: [Building Sakai] Major issues restart Sakai this morning due to Lesson builder and or hibernate
Is it possible that because the network was up and it was able to connect to hibernate.org but was receiving the 503 that we had problems?
This is the file causing the problems:
https://source.sakaiproject.org/svn/lessonbuilder/branches/lessonbuilder-1.4.x/hbm/src/java/org/sakaiproject/lessonbuildertool/simplepageproperty.hbm.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE hibernate-mapping PUBLIC "-//Hibernate/Hibernate Mapping DTD 3.0//EN"
"http://www.hibernate.org/dtd/hibernate-mapping-3.0.dtd">
<hibernate-mapping>
<class name="org.sakaiproject.lessonbuildertool.SimplePagePropertyImpl" table="lesson_builder_properties">
<id name="id" type="long">
<generator class="native"/>
</id>
<property name="attribute" type="string" length="255" not-null="true" unique="true"/>
<property name="value" type="text" />
</class>
</hibernate-mapping>
From: Niebel, William (wdn5e) [mailto:wdn5e at eservices.virginia.edu]
Sent: Saturday, October 12, 2013 9:46 AM
To: Kusnetz, Jeremy; sakai-dev at collab.sakaiproject.org<mailto:sakai-dev at collab.sakaiproject.org>
Subject: RE: [Building Sakai] Major issues restart Sakai this morning due to Lesson builder and or hibernate
In case it helps you sort this out, here's a quick comparison against UVa's files:
All our 1.4.1 lessonbuilder source *.hbm.xml files have the dtd sourced from hibernate.sourceforge.net like you found for *other* tools. (Also I see a simplepage.hbm.xml but no simplepageproperty.hbm.xml) I don't see any evidence that we patched that locally to replace the www.hibernate.org<http://www.hibernate.org> which you found for lessonbuilder. To be clear, I don't see "hibernate.org" in any dtd references in any hbm.xml files. We have only a few local, seemingly unrelated, patches for lessonbuilder. I do see that we patched the pom.xml files to change versioning from 1.4.1-SNAPSHOT to 1.4.1
________________________________
From: sakai-dev-bounces at collab.sakaiproject.org<mailto:sakai-dev-bounces at collab.sakaiproject.org> [sakai-dev-bounces at collab.sakaiproject.org] on behalf of Kusnetz, Jeremy [JKusnetz at APUS.EDU]
Sent: Saturday, October 12, 2013 8:26 AM
To: Kusnetz, Jeremy; sakai-dev at collab.sakaiproject.org<mailto:sakai-dev at collab.sakaiproject.org>
Subject: Re: [Building Sakai] Major issues restart Sakai this morning due to Lesson builder and or hibernate
Assuming my issues really were because it couldn't pull the hibernate mapping file.
I noticed that the other tools are using http://hibernate.sourceforge.net/hibernate-mapping-3.0.dtd and not http://www.hibernate.org/dtd/hibernate-mapping-3.0.dtd
Which would explain why only removing Lesson Builder allowed Sakai to start.
That said, should this be something we pull locally instead of from sourceforge.net or hibernate.org? If either are down, as we have seen, Sakai won't start.
From: sakai-dev-bounces at collab.sakaiproject.org<mailto:sakai-dev-bounces at collab.sakaiproject.org> [mailto:sakai-dev-bounces at collab.sakaiproject.org] On Behalf Of Kusnetz, Jeremy
Sent: Saturday, October 12, 2013 8:06 AM
To: sakai-dev at collab.sakaiproject.org<mailto:sakai-dev at collab.sakaiproject.org>
Subject: [Building Sakai] Major issues restart Sakai this morning due to Lesson builder and or hibernate
We are using Sakai 2.9.x which we pulled at a very late 2.9.3 RC.
This morning we were doing our normal restarts of Sakai at 5am. About half of our nodes started up normally and the other half couldn't start.
When the nodes couldn't start, we tried starting other environments that were completely unrelated and they couldn't start either.
Eventually we found that Lesson builder was throwing all sorts of errors. We haven't touched lesson builder at all, and in fact we currently do not use it.
I was able to get Sakai to start by removing all the lesson builder files from webapp, shared/lib and components.
After, in our dev environment I pulled the latest 1.4.x snapshot of lesson builder, built and deployed it and Sakai still couldn't start up. I went to do some more digging and tried to restart again around 7:30am eastern and all of a sudden sakai was starting with lesson builder. I even went to a separate environment that had the same version of lesson builder as production that wasn't touched and even though it couldn't start earlier, was now able to restart on it's own with no errors.
Is lesson builder making some call outside of the network when Sakai starts? Could it have been trying to call something that was down? That's the only thing I can think of to explain why Sakai was able to restart on it's own later, and initially half of the nodes were able to restart.
Actually looking at the logs, something in hibernate is calling out?
Caused by: org.dom4j.DocumentException: Server returned HTTP response code: 503 for URL: http://www.hibernate.org/dtd/hibernate-mapping-3.0.dtd Nested exception: Server returned HTTP response code: 503 for URL: http://www.hibernate.org/dtd/hibernate-mapping-3.0.dtd
If it was hibernate, why did removing lesson builder allow Sakai to start? If this hibernate error above is correct, why are we depending on hibernate.org to be up?
Here are some logs showing the errors:
2013-10-12 05:19:59,634 ERROR localhost-startStop-1 org.sakaiproject.util.NoisierDefaultListableBeanFactory - Failed to preinstantiate the singleton named org.sakaiproject.accountvalidator.dao.impl.ValidationLogicDaoTarget. Destroying all Spring beans.
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.sakaiproject.accountvalidator.dao.impl.ValidationLogicDaoTarget' defined in file [/SAKAI/apus/tomcat_servers/servers/apache-tomcat-7.0.42-community-2.9.x-100/components/accountvalidator-pack/WEB-INF/spring-hibernate.xml]: Cannot resolve reference to bean 'org.sakaiproject.springframework.orm.hibernate.GlobalSessionFactory' while setting bean property 'sessionFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.sakaiproject.springframework.orm.hibernate.GlobalSessionFactory' defined in file [/SAKAI/apus/tomcat_servers/servers/apache-tomcat-7.0.42-community-2.9.x-100/components/sakai-kernel-component/WEB-INF/db-components.xml]: Invocation of init method failed; nested exception is org.hibernate.MappingException: Failed to load class path resource [org/sakaiproject/lessonbuildertool/simplepageproperty.hbm.xml]
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:275)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:104)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1245)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1010)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:472)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
at java.security.AccessController.doPrivileged(Native Method)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
at org.sakaiproject.util.NoisierDefaultListableBeanFactory.getBean(NoisierDefaultListableBeanFactory.java:112)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
at org.sakaiproject.util.NoisierDefaultListableBeanFactory.preInstantiateSingletons(NoisierDefaultListableBeanFactory.java:73)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:728)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:380)
at org.sakaiproject.component.impl.SpringCompMgr.init(SpringCompMgr.java:160)
at org.sakaiproject.component.cover.ComponentManager.getInstance(ComponentManager.java:98)
at org.sakaiproject.component.cover.ComponentManager.get(ComponentManager.java:107)
at org.sakaiproject.component.cover.ServerConfigurationService.getInstance(ServerConfigurationService.java:52)
at org.sakaiproject.component.cover.ServerConfigurationService.getSakaiHomePath(ServerConfigurationService.java:173)
at org.sakaiproject.util.ToolListener.contextInitialized(ToolListener.java:66)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4939)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5434)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:633)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:976)
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1653)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.sakaiproject.springframework.orm.hibernate.GlobalSessionFactory' defined in file [/SAKAI/apus/tomcat_servers/servers/apache-tomcat-7.0.42-community-2.9.x-100/components/sakai-kernel-component/WEB-INF/db-components.xml]: Invocation of init method failed; nested exception is org.hibernate.MappingException: Failed to load class path resource [org/sakaiproject/lessonbuildertool/simplepageproperty.hbm.xml]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1338)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:473)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:473)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
at java.security.AccessController.doPrivileged(Native Method)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
at org.sakaiproject.util.NoisierDefaultListableBeanFactory.getBean(NoisierDefaultListableBeanFactory.java:112)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:269)
... 36 more
Caused by: org.hibernate.MappingException: Failed to load class path resource [org/sakaiproject/lessonbuildertool/simplepageproperty.hbm.xml]
at org.sakaiproject.springframework.orm.hibernate.impl.AdditionalHibernateMappingsImpl.processConfig(AdditionalHibernateMappingsImpl.java:65)
at org.sakaiproject.springframework.orm.hibernate.AddableSessionFactoryBean.postProcessConfiguration(AddableSessionFactoryBean.java:78)
at org.springframework.orm.hibernate3.LocalSessionFactoryBean.buildSessionFactory(LocalSessionFactoryBean.java:727)
at org.springframework.orm.hibernate3.AbstractSessionFactoryBean.afterPropertiesSet(AbstractSessionFactoryBean.java:211)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1369)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1335)
... 47 more
Caused by: org.hibernate.InvalidMappingException: Could not parse mapping document from input stream
at org.hibernate.cfg.Configuration.addInputStream(Configuration.java:527)
at org.sakaiproject.springframework.orm.hibernate.impl.AdditionalHibernateMappingsImpl.processConfig(AdditionalHibernateMappingsImpl.java:63)
... 52 more
Caused by: org.dom4j.DocumentException: Server returned HTTP response code: 503 for URL: http://www.hibernate.org/dtd/hibernate-mapping-3.0.dtd Nested exception: Server returned HTTP response code: 503 for URL: http://www.hibernate.org/dtd/hibernate-mapping-3.0.dtd
at org.dom4j.io.SAXReader.read(SAXReader.java:484)
at org.hibernate.cfg.Configuration.addInputStream(Configuration.java:518)
... 53 more
java.lang.Exception: traceback
at org.sakaiproject.util.NoisierDefaultListableBeanFactory.destroySingletons(NoisierDefaultListableBeanFactory.java:94)
at org.sakaiproject.util.NoisierDefaultListableBeanFactory.preInstantiateSingletons(NoisierDefaultListableBeanFactory.java:83)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:728)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:380)
at org.sakaiproject.component.impl.SpringCompMgr.init(SpringCompMgr.java:160)
at org.sakaiproject.component.cover.ComponentManager.getInstance(ComponentManager.java:98)
at org.sakaiproject.component.cover.ComponentManager.get(ComponentManager.java:107)
at org.sakaiproject.component.cover.ServerConfigurationService.getInstance(ServerConfigurationService.java:52)
at org.sakaiproject.component.cover.ServerConfigurationService.getSakaiHomePath(ServerConfigurationService.java:173)
at org.sakaiproject.util.ToolListener.contextInitialized(ToolListener.java:66)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4939)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5434)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:633)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:976)
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1653)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
This message is private and confidential. If you have received it in error, please notify the sender and remove it from your system.
This message is private and confidential. If you have received it in error, please notify the sender and remove it from your system.
This message is private and confidential. If you have received it in error, please notify the sender and remove it from your system.
This message is private and confidential. If you have received it in error, please notify the sender and remove it from your system.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20131012/221669ae/attachment.html
More information about the sakai-dev
mailing list