[Building Sakai] Course Management : CmSynchronizer JOB : File Size

[Ray Davis]

Groovy sounds like a great choice.

If you'd like to be able to run the script without going through a real 
web server's request/response layers, or if you can't deploy the SGS 
tool (which is very cool in its own right), you might also consider 
using the test harness's component manager emulator. For our scalability 
tests' Groovy-scripted data load, we trigger the job from Maven on the 
command line. Not too tough (although still very time-consuming) -- you 
can find the code and documentation here:

https://source.sakaiproject.org/svn/msub/berkeley.edu/stress-test/sakai_2-5-x
https://source.sakaiproject.org/svn/msub/berkeley.edu/stress-test/sakai_2-6-x

Best,
Ray

On 5/7/2009 8:49 AM, Thomas Amsler wrote:
> We ended up, at least for evaluating this process, using the Sakai
> Groovy Shell (SGS), which works great for such tasks.
> 
> Thank you for all the feedback.
> 
> Best,
> -- Thomas
> 
> 
> On Wed, May 6, 2009 at 11:16 AM, Kevin P. Foote <kpfoote at iup.edu> wrote:
>> Wow.. 3G xml .. bet that takes a while to load in a browser ha.. ;-)
>>
>> I load 3k+ courses with the CmSynchronizer but my xml is only ~ 20-30M
>>
>> I create the xml file in about 40 sec with perl but the Synchronizer
>> takes about 1 hour.
>>
>> If your dealing with 3G .. you'd probabbly be better off writing
>> something customized like others do as Casey mentioned.
>>
>> ------
>> thanks
>>  kevin.foote
>>
>> On Wed, 6 May 2009, Casey Dunn wrote:
>>
>> -> On Wed, May 6, 2009 at 10:57 AM, Thomas Amsler <tpamsler at ucdavis.edu> wrote:
>> ->
>> -> > I am experimenting with loading CM data via the CmSynchronizer Job. Is
>> -> > anybody using this appraoch to load their CM data into the Sakai CM
>> -> > tables? How are folks handling large data set that result int large
>> -> > CMData.xml file. I am dealing with about ~3GB of data.
>> -> >
>> -> > Source Code:
>> -> >
>> -> > https://source.sakaiproject.org/svn/course-management/branches/sakai_2-5-x/cm-impl/hibernate-impl/impl/src/java/org/sakaiproject/coursemanagement/impl/job/
>> -> >
>> -> >
>> -> > Best,
>> -> > -- Thomas
>> ->
>> ->
>> -> Stanford avoids that problem - the XML is split down to the Course level,
>> -> containing Offerings and their Sections. ( the example CmSynchronizer served
>> -> as an example; there are other Stanford specific tweaks to the feed )
>> ->
>> -> A 3Gig XML file must really be expensive to deal with.
>> ->
>> -> why not knock it apart, Thomas?
>> ->
>>
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> 
> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
>