[Building Sakai] Course Management : CmSynchronizer JOB : File Size

[Thomas Amsler]

Ray,

Thank you for sharing this. Also, Ray's code shows how to use the
groovy maven plugin to build groovy classes along with java ones. Good
stuff!!!


Best,
-- Thomas




On Thu, May 7, 2009 at 9:02 AM, Ray Davis <ray at media.berkeley.edu> wrote:
> Groovy sounds like a great choice.
>
> If you'd like to be able to run the script without going through a real web
> server's request/response layers, or if you can't deploy the SGS tool (which
> is very cool in its own right), you might also consider using the test
> harness's component manager emulator. For our scalability tests'
> Groovy-scripted data load, we trigger the job from Maven on the command
> line. Not too tough (although still very time-consuming) -- you can find the
> code and documentation here:
>
> https://source.sakaiproject.org/svn/msub/berkeley.edu/stress-test/sakai_2-5-x
> https://source.sakaiproject.org/svn/msub/berkeley.edu/stress-test/sakai_2-6-x
>
> Best,
> Ray
>
> On 5/7/2009 8:49 AM, Thomas Amsler wrote:
>>
>> We ended up, at least for evaluating this process, using the Sakai
>> Groovy Shell (SGS), which works great for such tasks.
>>
>> Thank you for all the feedback.
>>
>> Best,
>> -- Thomas
>>
>>
>> On Wed, May 6, 2009 at 11:16 AM, Kevin P. Foote <kpfoote at iup.edu> wrote:
>>>
>>> Wow.. 3G xml .. bet that takes a while to load in a browser ha.. ;-)
>>>
>>> I load 3k+ courses with the CmSynchronizer but my xml is only ~ 20-30M
>>>
>>> I create the xml file in about 40 sec with perl but the Synchronizer
>>> takes about 1 hour.
>>>
>>> If your dealing with 3G .. you'd probabbly be better off writing
>>> something customized like others do as Casey mentioned.
>>>
>>> ------
>>> thanks
>>>  kevin.foote
>>>
>>> On Wed, 6 May 2009, Casey Dunn wrote:
>>>
>>> -> On Wed, May 6, 2009 at 10:57 AM, Thomas Amsler <tpamsler at ucdavis.edu>
>>> wrote:
>>> ->
>>> -> > I am experimenting with loading CM data via the CmSynchronizer Job.
>>> Is
>>> -> > anybody using this appraoch to load their CM data into the Sakai CM
>>> -> > tables? How are folks handling large data set that result int large
>>> -> > CMData.xml file. I am dealing with about ~3GB of data.
>>> -> >
>>> -> > Source Code:
>>> -> >
>>> -> >
>>> https://source.sakaiproject.org/svn/course-management/branches/sakai_2-5-x/cm-impl/hibernate-impl/impl/src/java/org/sakaiproject/coursemanagement/impl/job/
>>> -> >
>>> -> >
>>> -> > Best,
>>> -> > -- Thomas
>>> ->
>>> ->
>>> -> Stanford avoids that problem - the XML is split down to the Course
>>> level,
>>> -> containing Offerings and their Sections. ( the example CmSynchronizer
>>> served
>>> -> as an example; there are other Stanford specific tweaks to the feed )
>>> ->
>>> -> A 3Gig XML file must really be expensive to deal with.
>>> ->
>>> -> why not knock it apart, Thomas?
>>> ->
>>>
>> _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>
>> TO UNSUBSCRIBE: send email to
>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
>> "unsubscribe"
>>
>
>