[Building Sakai] [Deploying Sakai] Sakora-csv incremental upload?

Wed May 22 07:52:38 PDT 2013

Hi Aaron,

Hope you are doing well! We have some questions regarding the latest changes in sakora-csv.

1. Our understanding is that sakora-csv is not designed to handle deletion in batch mode, at least not yet. But we believe it would be a great feature, especially to avoid time consuming full load just for deletion purpose. After doing some analysis, we think it is possible to do deletion with ignore flags in place and of course some code modifications. Here is our plan: 

·         Modify CsvUploadServlet.java to accept a new Boolean parameter - deletionMode. If deletionMode is true in the HTTP request, turn on ignoreMissingSessions and ignoreMembershipRemovals as well.

·         Modify CsvCommonHandlerService.java to take the overridden deletionMode parameter and add a deletionMode() method to get current SyncVar.

·         Modify readInputLine() method in each Csv***Handler.java impl class. If deletionMode is turned on, call corresponding remove method in Sakai courseManagement API and skip the rest of the processes.

·         Previous batch minus the current batch is one way to produce deletion csv files.

We tried this method with Section Membership and the deletion appeared to be working just fine. Before going too far, does the above sound right? Any concerns?

2. Not sure if this is a bug or just because we were not supplying the parameters in the right way. Say, for example, run a Curl command with [-F "ignoreMissingSessions=true" -F "ignoreMembershipRemovals=true"], then the following command with parameter [-F "userRemovalMode=ignore"] only. In this scenario, the ignoreMissingSessions=true and ignoreMembershipRemovals=true would be carried over to the second request. To prevent this from happening, we think the existing parameters should be cleared first before checking new ones. So we modified CsvUploadServlet.java (line 235 - 244) as follow,

                // remove any override

                jd.getJobDataMap().remove(OVERRIDE_IGNORE_MISSING_SESSIONS);

                jd.getJobDataMap().remove(OVERRIDE_IGNORE_MEMBERSHIP_REMOVALS);

                jd.getJobDataMap().remove(OVERRIDE_USER_REMOVAL_MODE);

                if (jobOverrides != null && !jobOverrides.isEmpty()) {

                                if (log.isDebugEnabled()) log.debug("SakoraCSV inserting "+jobOverrides.size()+" job overrides into job data map: "+jobOverrides);

                                jd.getJobDataMap().putAll(jobOverrides);

                }

We truly appreciate your time and look forward to your suggestions. If we can be of any help, please let us know too.

Best Regards,

Yi Zhu

Analyst/Programmer, Senior

IS - Applications Development

Wake Forest University

1834 Wake Forest Rd

zhuy at wfu.edu

336-758-2577

-----Original Message-----
From: Yi Zhu [mailto:zhuy at wfu.edu] 
Sent: Thursday, May 09, 2013 5:50 PM
To: 'Aaron Zeckoski'
Cc: sakai-dev at collab.sakaiproject.org; production at collab.sakaiproject.org
Subject: RE: [Deploying Sakai] Sakora-csv incremental upload?

Hi Aaron,

That's great news. Thank you for the great work! 

We look forward to experimenting this feature and will be more than happy to provide feedback!

Best Regards,

Yi

-----Original Message-----

From:  <mailto:azeckoski at gmail.com> azeckoski at gmail.com [ <mailto:azeckoski at gmail.com> mailto:azeckoski at gmail.com] On Behalf Of Aaron Zeckoski

Sent: Thursday, May 09, 2013 5:24 PM

To: Yi Zhu

Cc:  <mailto:sakai-dev at collab.sakaiproject.org> sakai-dev at collab.sakaiproject.org;  <mailto:production at collab.sakaiproject.org> production at collab.sakaiproject.org

Subject: Re: [Deploying Sakai] Sakora-csv incremental upload?

I did that work and the development work is complete but has not been proven in production yet. The early testing is complete (and

successful) though.

You are welcome to use it and provide feedback. The options are all documented in the README and on the wiki page.

-AZ

On Thu, May 9, 2013 at 5:14 PM, Yi Zhu < <mailto:zhuy at wfu.edu> zhuy at wfu.edu> wrote:

> Dear All,

> 

> 

> 

> Many of you probably are already familiar with Simple Sakai SIS 

> services

> (sakora-csv) and are aware of the slow performance of any full batch upload.

> We are currently investigating a way to improve the process and 

> potentially make multiple loads during normal business hours. In our 

> research, we realized that some recent changes in sakora-csv may make 

> the incremental update possible. For example, CsvUploadServlet.java 

> now accepts additional HTTP request parameters They are 

> userRemovalMode=true|false, ignoreMembershipRemovals=true|false, and ignoreMissingSessions=true|false.

> 

> 

> 

> Given the fact that the last update in SVN trunk was Feb 20, 2013, 

> does anyone know whether the development work is complete?

> 

> 

> 

> Are following Curl parameters all we need to submit an “incremental” 

> batch upload?

> 

> -F "ignoreMissingSessions=true" -F "ignoreMembershipRemovals=true" -F 

> "userRemovalMode=ignore"

> 

> 

> 

> Any help would be really appreciated!

> 

> 

> 

> PS, link to sakora-csv ---

>  <https://confluence.sakaiproject.org/display/IMSES/Simple+Sakai+SIS+Ser> https://confluence.sakaiproject.org/display/IMSES/Simple+Sakai+SIS+Ser

> vices#SimpleSakaiSISServices-Installation

> 

> 

> 

> 

> 

> Best Regards,

> 

> Yi Zhu

> 

> Analyst/Programmer, Senior

> 

> IS - Applications Development

> 

> Wake Forest University

> 

> 1834 Wake Forest Rd

> 

>  <mailto:zhuy at wfu.edu> zhuy at wfu.edu

> 

> 

> 

> 

> 

> 

> _______________________________________________

> production mailing list

>  <mailto:production at collab.sakaiproject.org> production at collab.sakaiproject.org

>  <http://collab.sakaiproject.org/mailman/listinfo/production> http://collab.sakaiproject.org/mailman/listinfo/production

> 

> TO UNSUBSCRIBE: send email to

>  <mailto:production-unsubscribe at collab.sakaiproject.org> production-unsubscribe at collab.sakaiproject.org

> with a subject of "unsubscribe"

--

Aaron Zeckoski - Software Architect -  <http://tinyurl.com/azprofile> http://tinyurl.com/azprofile

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20130522/8e9948e7/attachment.html 