[Building Sakai] Course Management : CmSynchronizer JOB : File Size

[Thomas Amsler]

We ended up, at least for evaluating this process, using the Sakai
Groovy Shell (SGS), which works great for such tasks.

Thank you for all the feedback.

Best,
-- Thomas




On Wed, May 6, 2009 at 11:16 AM, Kevin P. Foote <kpfoote at iup.edu> wrote:
>
> Wow.. 3G xml .. bet that takes a while to load in a browser ha.. ;-)
>
> I load 3k+ courses with the CmSynchronizer but my xml is only ~ 20-30M
>
> I create the xml file in about 40 sec with perl but the Synchronizer
> takes about 1 hour.
>
> If your dealing with 3G .. you'd probabbly be better off writing
> something customized like others do as Casey mentioned.
>
> ------
> thanks
>  kevin.foote
>
> On Wed, 6 May 2009, Casey Dunn wrote:
>
> -> On Wed, May 6, 2009 at 10:57 AM, Thomas Amsler <tpamsler at ucdavis.edu> wrote:
> ->
> -> > I am experimenting with loading CM data via the CmSynchronizer Job. Is
> -> > anybody using this appraoch to load their CM data into the Sakai CM
> -> > tables? How are folks handling large data set that result int large
> -> > CMData.xml file. I am dealing with about ~3GB of data.
> -> >
> -> > Source Code:
> -> >
> -> > https://source.sakaiproject.org/svn/course-management/branches/sakai_2-5-x/cm-impl/hibernate-impl/impl/src/java/org/sakaiproject/coursemanagement/impl/job/
> -> >
> -> >
> -> > Best,
> -> > -- Thomas
> ->
> ->
> -> Stanford avoids that problem - the XML is split down to the Course level,
> -> containing Offerings and their Sections. ( the example CmSynchronizer served
> -> as an example; there are other Stanford specific tweaks to the feed )
> ->
> -> A 3Gig XML file must really be expensive to deal with.
> ->
> -> why not knock it apart, Thomas?
> ->
>