[Building Sakai] [Deploying Sakai] Sakora-csv incremental upload?

[Yi Zhu]

Aaron,

1. You are exactly right. The idea is to only remove things contained in the files from Sakai. We are thinking about using sakora-csv to load initial data, then to add/update (with ignore flags), and delete(delete mode plus ignore flags). This diagram may better explain what we are trying to accomplish.
https://www.lucidchart.com/documents/view/435b-1550-519a592f-ac39-04ec0a00913e

2. This problem is not in the jobOverrides map, but due to the fact that JobDetail object is being reused across posts. Hence, the overrides are being retained in the same JobDataMap map. Though they get cleared when there is no override in the request. Because of that, the state of the running job could poetically be changed by a new post as well. Are we expected to only post new jobs when no CSV Loader job is running? I can see this could get tricky when we use different parameters between different loads.

Best Regards,
Yi



-----Original Message-----
From: azeckoski at gmail.com [mailto:azeckoski at gmail.com] On Behalf Of Aaron Zeckoski
Sent: Wednesday, May 22, 2013 5:08 PM
To: Yi Zhu
Cc: sakai-dev at collab.sakaiproject.org
Subject: Re: [Deploying Sakai] Sakora-csv incremental upload?

> 1. Our understanding is that sakora-csv is not designed to handle 
> deletion in batch mode, at least not yet. But we believe it would be a 
> great feature, especially to avoid time consuming full load just for deletion purpose.
> After doing some analysis, we think it is possible to do deletion with 
> ignore flags in place and of course some code modifications. Here is 
> our

Is the idea here to basically put the entire processor into a "delete"
mode so that things are removed instead of being added?

If not, I am not sure how this is different from the usual processing that happens without the flags to disable deletions.

Can you explain your goal here in more detail?


> 2. Not sure if this is a bug or just because we were not supplying the 
> parameters in the right way. Say, for example, run a Curl command with 
> [-F "ignoreMissingSessions=true" -F "ignoreMembershipRemovals=true"], 
> then the following command with parameter [-F 
> "userRemovalMode=ignore"] only. In this scenario, the 
> ignoreMissingSessions=true and ignoreMembershipRemovals=true would be 
> carried over to the second request. To prevent this from happening, we 
> think the existing parameters should be cleared first before checking new ones. So we modified CsvUploadServlet.java (line 235 - 244) as follow,
>                 // remove any override
>                 
> jd.getJobDataMap().remove(OVERRIDE_IGNORE_MISSING_SESSIONS);
> ..........


Are you saying the state of those values is maintained across requests? Each post should reset the state so that should not be happening.

Those values come from the jobOverrides map which is reset on each request so the state should be cleared each time a post is sent.

If you are running the originally posted job over and over then the state will be maintained for all those future runs. I don't think that is a bug. If the state is being maintained across posts then something is very strange.

-AZ


--
Aaron Zeckoski - Software Architect - http://tinyurl.com/azprofile