[WG: Sakai QA] [Building Sakai] serious blocker probelm with convertToFile: "null xml"? (2.6.x)

Jim Eng jimeng at umich.edu
Tue Apr 21 20:28:01 PDT 2009 

I'm still hoping to get information from Will about the sizes of his  
tables.  If it's possible for Will to go back to the 2.5.x database  
before the conversion to BINARY_ENTITY serialization and do the  
conversion to filesystem storage of content bodies first, that might  
solve the first immediate problem -- dealing with someone who already  
has 2.6.x resources and no way to convert to filesystem storage of  
resource bodies.

If that's not possible, we may need a way to help Will make the db- 
 >filesystem conversion in 2.6.x.

Once Will's problem is dealt with, it might be good to ask whether  
sakai should still support database storage of content bodies.   
Database storage of resource bodies is still around because it is said  
to be the easiest way to get sakai up and running for demo's out of  
the box.  But it is not useful for production, is it?  Doesn't it  
force people to do a conversion before they can really get started?

If we must continue to support database storage of content bodies,  
then we must supply a special method to convert to filesystem storage  
for resources with BINARY_ENTITY serialization. Someone would need to  
decide whether that is a blocker for 2.6.0 or if it could be supplied  
as part of 2.6.1.

Just my 2 cents.

Jim




On Apr 21, 2009, at 10:07 PM, Jim Eng wrote:

> The method convertToFile() in DbContentService has an SqlReader to  
> deserialize resources during the conversion from database storage to  
> filesystem storage of resource bodies.  The SqlReader was never  
> rewritten to deal with BINARY_ENTITY serialization.  It attempts to  
> read the XML and gets nulls, so it fails.
>
> Jim
>
>
> On Apr 21, 2009, at 9:48 PM, Jim Eng wrote:
>
>> Hi Will,
>>
>> Do you know from your logs or from queries of the CONTENT_RESOURCE
>> table in your database whether ContentHostingService is using XML or
>> BINARY_ENTITY serialization before the conversion?
>>
>> I am guessing that nobody has done db to filesystem conversion in
>> 2.6.x.  Were you running 2.5 or 2.4 and converted to 2.6?
>>
>> How many rows are in the CONTENT_RESOURCE table?
>>
>> How many rows are in the CONTENT_RESOURCE_BODY_BINARY table?
>>
>> Thanks.
>>
>> Jim
>>
>>
>>
>> On Apr 21, 2009, at 9:25 PM, will trillich wrote:
>>
>>> We're stuck without some pointers here... We're having one heck of a
>>> time finding the problem converting from resources-in-sql to
>>> resources-in-filesystem -- any suggestions are welcome!
>>>
>>>
>>>
>>> Short version:
>>>
>>> We're trying to migrate 2.6.x resources from SQL storage to  
>>> filesystem
>>> storage -- and three XSL files (formFiew-related files) get  
>>> converted,
>>> nothing else does. Lots of "DbContentService - convertToFile: null
>>> xml" log lines in catalina.out.
>>>
>>> Once Sakai is up, when trying to access any of the items from
>>> resources, the item isn't found of course (error reported in the
>>> browser). We can then upload new files and they're stored in the
>>> filesystem just fine; these newly uploaded items can be
>>> viewed/downloaded just fine.
>>>
>>> We can comment out bodyPath and bodyVolumes, restart sakai, and the
>>> original resources are healthy (viewable/downloadable) again.
>>>
>>> What are we missing?
>>>
>>>
>>>
>>> Long version:
>>>
>>> http://bugs.sakaiproject.org/confluence/display/DOC/Configuring+Content+Hosting
>>> , right?
>>>
>>> # in sakai.properties:
>>> bodyPath at org.sakaiproject.content.api.ContentHostingService=/ 
>>> Volumes/
>>> mongo/sakai
>>> bodyVolumes
>>> @org
>>> .sakaiproject
>>> .content
>>> .api
>>> .ContentHostingService
>>> =a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x,y,z
>>> convertToFile 
>>> @org.sakaiproject.content.api.ContentHostingService=true
>>>
>>> After starting sakai with convertToFile=true, there are exactly  
>>> three
>>> items in the bodyPath tree (there should be dozens even for a fresh
>>> sakai install):
>>>
>>> bash-3.2$ cd /Volumes/mongo/sakai/
>>> bash-3.2$ find . -type f
>>> ./c/2009/112/00/b51c365b-f67d-4ff1-9f13-6a7cfb07fa06
>>> ./d/2009/112/00/ad68e45b-ef6b-4d4e-9a01-1575260c2c9c
>>> ./s/2009/112/00/4a935949-0437-4024-a486-7a108072393f
>>>
>>> The three resources above are formView XSL files. Those are the only
>>> ones that get converted.
>>>
>>> ./x/2009/111/22/124ef7d4-36c2-4170-b027-853e704c5d07
>>>
>>> Thereafter, uploaded files (one shown above), as a test, worked just
>>> fine. (That is, conversion doesn't seem to be working at all, but  
>>> new
>>> uploads are just fine.)
>>>
>>>
>>> # from catalina.out:
>>> <snip>
>>> 2009-04-21 20:41:16,502  INFO main
>>> org.sakaiproject.event.impl.BaseNotificationService -
>>> org.sakaiproject.event.impl.DbNotificationService$$EnhancerByCGLIB$$346b4059 at ed5990.init
>>> ()
>>> 2009-04-21 20:41:16,503  INFO main
>>> org.sakaiproject.event.impl.DbNotificationService - .init(): table:
>>> SAKAI_NOTIFICATION locks-in-db: false
>>> 2009-04-21 20:41:16,546  INFO main
>>> org.sakaiproject.content.impl.DbResourceTypeRegistry - init()
>>> 2009-04-21 20:41:16,550  INFO main
>>> org.sakaiproject.content.impl.ResourceTypeRegistryImpl - init()
>>> 2009-04-21 20:41:16,719  INFO main
>>> org.sakaiproject.content.impl.DbContentService -
>>> ====================================================
>>> ====================================================
>>> Conversion of the ContentHostingService database tables is complete.
>>> Using new filesize column
>>> ====================================================
>>> ====================================================
>>>
>>> 2009-04-21 20:41:16,747  INFO main
>>> org.sakaiproject.content.impl.DbContentService - Migration of data  
>>> to
>>> the Binary format will be performed by this node
>>> 2009-04-21 20:41:16,814  INFO main
>>> org.sakaiproject.content.impl.BaseContentService - Loaded Storage as
>>> org.sakaiproject.content.impl.DbContentService$DbStorage at 9f750 for
>>> org.sakaiproject.content.impl.DbContentService at 4ba63b
>>> 2009-04-21 20:41:16,849  INFO main
>>> org.sakaiproject.content.impl.BaseContentService - init(): site  
>>> quota:
>>> 1048576 body path: /Volumes/wikiblogs3/sakai volumes: a, b, c, d, e,
>>> f, g, h, i, j, k, l, m, n, o, p, q, r, s, t, u, v, w, x, y, z,
>>> 2009-04-21 20:41:16,977  INFO main
>>> org.sakaiproject.content.impl.DbContentService - convertToFile
>>> 2009-04-21 20:41:16,995  WARN main
>>> org.sakaiproject.content.impl.DbContentService - convertToFile: null
>>> xml : /user/ce1a4481-91d8-455b-a2b9-33f5a26a3a77/Principios y
>>> reflexiones/Rubricas/rubricasola_porta_e_revisin_p3.pdf
>>> 2009-04-21 20:41:16,995  WARN main
>>> org.sakaiproject.content.impl.DbContentService - convertToFile: null
>>> xml : /group/PortfolioAdmin/system/contentOverText.xml
>>> 2009-04-21 20:41:16,995  WARN main
>>> org.sakaiproject.content.impl.DbContentService - convertToFile: null
>>> xml : /group/PortfolioAdmin/system/simpleRichText.xml
>>> 2009-04-21 20:41:16,995  WARN main
>>> org.sakaiproject.content.impl.DbContentService - convertToFile: null
>>> xml : /user/ce1a4481-91d8-455b-a2b9-33f5a26a3a77/Principios y
>>> reflexiones/Rubricas/rubricasola_porta_e_revisin_p2.pdf
>>> 2009-04-21 20:41:16,995  WARN main
>>> org.sakaiproject.content.impl.DbContentService - convertToFile: null
>>> xml : /group/PortfolioAdmin/system/freeFormRenderer.xml
>>>
>>> </snip>
>>>
>>> 2009-04-21 20:41:17,178  WARN main
>>> org.sakaiproject.content.impl.DbContentService - convertToFile: null
>>> xml : /attachment/fdfb4c6e-64f5-4ea1-ac16-e1cb99109263/Matrices/
>>> 77b08a12-efef-42ac-93ad-5ebfbb8aa03c/Reflexion
>>> Principio 2.docx
>>> 2009-04-21 20:41:17,178  WARN main
>>> org.sakaiproject.content.impl.DbContentService - convertToFile: null
>>> xml : /group/PortfolioAdmin/system/2column.jpg
>>> 2009-04-21 20:41:17,179  INFO main
>>> org.sakaiproject.content.impl.DbContentService - convertToFile:
>>> converted resources: 0
>>> 2009-04-21 20:41:17,179  INFO main
>>> org.sakaiproject.content.impl.DbContentService - convertToFile: done
>>> 2009-04-21 20:41:17,180  INFO main
>>> org.sakaiproject.content.impl.DbContentService - init(): tables:
>>> CONTENT_COLLECTION CONTENT_RESOURCE CONTENT_RESOURCE_BODY_BINARY
>>> CONTENT_ENTITY_GROUPS locks-in-db: false bodyPath:
>>> /Volumes/wikiblogs3/sakai storage:
>>> org.sakaiproject.content.impl.DbContentService$DbStorage at 9f750
>>> 2009-04-21 20:41:17,181  INFO main
>>> org
>>> .sakaiproject
>>> .component.app.messageforums.MessageForumsMessageManagerImpl
>>> - init()
>>>
>>> </snip>
>>>
>>>
>>>
>>> # uname -a
>>> Darwin vstream.uprrp.edu 9.6.0 Darwin Kernel Version 9.6.0: Mon  
>>> Nov 24
>>> 17:39:01 PST 2008; root:xnu-1228.9.59~1/RELEASE_PPC Power Macintosh
>>>
>>> # sakai source code version (from svn info):
>>> URL: https://source.sakaiproject.org/svn/sakai/branches/sakai_2-6-x
>>> Repository Root: https://source.sakaiproject.org/svn
>>> Repository UUID: 66ffb92e-73f9-0310-93c1-f5514f145a0a
>>> Revision: 60519
>>> Node Kind: directory
>>> Schedule: normal
>>> Last Changed Author: arwhyte at umich.edu
>>> Last Changed Rev: 60121
>>> Last Changed Date: 2009-04-15 18:01:01 -0400 (Wed, 15 Apr 2009)
>>>
>>> $ $JAVA_HOME/bin/java -version
>>> Shared archive: sharing disabled for server vm
>>> java version "1.5.0_16"
>>> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-
>>> b06-284)
>>> Java HotSpot(TM) Client VM (build 1.5.0_16-133, mixed mode)
>>>
>>> $ tomcat/bin/catalina.sh version
>>> Using CATALINA_BASE:   /opt/tomcat
>>> Using CATALINA_HOME:   /opt/tomcat
>>> Using CATALINA_TMPDIR: /opt/tomcat/temp
>>> Using JRE_HOME:       /Library/Java/Home
>>> Shared archive: sharing disabled for server vm
>>> Server version: Apache Tomcat/5.5.26
>>> Server built:   Jan 28 2008 01:35:23
>>> Server number:  5.5.26.0
>>> OS Name:        Mac OS X
>>> OS Version:     10.5.6
>>> Architecture:   ppc
>>> JVM Version:    1.5.0_16-b06-284
>>> JVM Vendor:     Apple Inc.
>>>
>>> -- 
>>> will trillich
>>> "If you do not change direction, you may end up where you are
>>> heading." -- Lao Tzu
>>> _______________________________________________
>>> sakai-dev mailing list
>>> sakai-dev at collab.sakaiproject.org
>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>
>>> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org
>>> with a subject of "unsubscribe"
>>>
>>>
>>
>> _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>
>> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org 
>>  with a subject of "unsubscribe"
>>
>>
>
>
>

More information about the sakai-qa mailing list