[Contrib: Evaluation System] Provider sync failing

[Joseph Delaney]

I don't have enough technical background to know the difference between "quartz" and "scheduledinvocationmanager".  Also our immediate issue is resolved, though I think this is still worth clarifying since it seems likely to happen again.

As I understand it, an evaluation syncs group membership when the group is assigned or updated.  https://jira.sakaiproject.org/browse/EVALSYS-1031 added the configuration option to have it sync on state change (along with the "quick sync" and scheduled sync options).  I think this is where the option to set a specific node to do the sync also came from. 

Since we're not using site memberships - we use an external provider and load evaluations (probably foolishly) by inserting directly into the database tables - that didn't work for us. So we patched EVALSYS-1031 into evalsys 1.3 a couple of years ago, and set an overnight sync. That's the method we continued to use with 1.4, and that's what failed. We only used the "quick sync" to try to kickstart it, and that also failed. 

Our problem seems to have been caused when some of the sakai nodes were restarted, apparently shifting the quartz processing to something other than what was specified in the evalsys settings, so the roster synchronization was skipped.  We resolved it by sequentially changing the evalsys settings until we hit the node that works. When the nodes get restarted again, the problem will presumably come back.

Here's more from Charles Hedrick:
> I believe I know how to do a real fix, but I'd like to see if someone will tell us what the current apparently off the wall code was intended to do. Maybe there's something going on that I don't understand.
> 
> (The real fix would be to ignore the test for what server it should run on, since Quartz will make sure it only runs on one.)



Joseph Delaney
Center for the Teaching Advancement and Assessment Research
Rutgers University
http://ctaar.rutgers.edu/

(848) 932-7466 
(Please make note of the new area code)


On Oct 10, 2013, at 12:55 PM, Nicola Monat-Jacobs <nicola at longsight.com> wrote:

> Just so we're all on the same page, normally an evaluation will "sync" its evalgroup membership with the course sites memberships when the state of the evaluation changes - like when it goes from in-queue to active. 
> 
> The quick sync is designed to sync these evalgroups with the course sites memberships if the curse sites membership has changed and needs to be updated when the eval state is not expected to change - like, for example, a new student is added to the course when the evaluation is already active.
> 
> Silly question: when you do a sync, you usually specify what 'state' you want to target for update. The evals you're looking to update are in that state right?
> 
> Does the sync job actually create a real quartz job? Or does it just piggyback off scheduledinvocationmanager? There is a patch that will allow you turn off quartz processing on nodes: https://jira.sakaiproject.org/browse/SAK-13776
> 
> So, you could have quartz off on all nodes except the one you wanted to have things run on. But that would be for all of the jobs, not just this one.
> 
> Also, from reading the ticket, its unclear to me if scheduledinvocationmanager even respects this setting.
> 
> This jira is also has some interesting comments: https://jira.sakaiproject.org/browse/SAK-23415
> 
> 
> 
> On Wed, Oct 9, 2013 at 1:42 PM, Joseph Delaney <joe.delaney at ctaar.rutgers.edu> wrote:
> We've suddenly run into a problem with evalsys where the provider no longer syncs the evaluation with the rosters. When we use "Quick Sync", the quartz job appears in the job list, disappears at the appropriate time, but no sync happens. We're running evalsys 1.4 with sakai 2.9.1, running on a cluster of 6 machines. Everything had been working normally since we updated to this configuration, about 6 weeks,  but it stopped last Thursday for no apparent reason.
> 
> This is the best we've been able to figure out - from Charles Hedrick:
>> The code in GroupMembershipSyncImpl.java will only do anything if the job runs in a specific server, which is set in evalsettings. But as far as I know, quartz will run it on a random server. It looks to me like there’s only a one in 6 chance that it will run on the right server.
> 
> 
> Is this correct? Does anyone have any suggestions on what we can do to get this functioning again?
> 
> Thanks,
> 
> 
> Joseph Delaney
> Center for the Teaching Advancement and Assessment Research
> Rutgers University
> http://ctaar.rutgers.edu/
> 
> (848) 932-7466 
> (Please make note of the new area code)
> 
> 
> 
> _______________________________________________
> evaluation mailing list
> evaluation at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/evaluation
> 
> TO UNSUBSCRIBE: send email to evaluation-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/evaluation/attachments/20131011/9eae64ea/attachment.html