[Contrib: Evaluation System] EvalSys: Use cases exploring roles/permissions and web services

[Sean Horner]

Hello,

I am exploring two use cases (or clusters of use cases) for EvalSys at
Pacific Lutheran University (PLU), both of which seem to lie partly outside
EvalSys' scope (at least for 1.3.x). In sharing these use cases, I hope to:

   - Share what we at PLU are doing and or thinking about doing with
   EvalSys, elaborating on contributions from my predecessor, Bryan Bakotich.
   - Share my line of thinking with others to check the validity of my
   assumptions, especially since I am relatively new to EvalSys and Sakai,
   especially its underlying architecture and code.
   - Suggest features or enhancements to EvalSys that could meet not only
   the use cases I am exploring but perhaps others' as well.
   - Fish for insights on how to create, configure, and initiate
   evaluations (EvalSys objects) via web services, if feasible or relatively
   straightforward. (See the second use case below.)
   - Glimpse future directions for EvalSys' design (whether with CLE or
   OAE) pertaining to EvalSys permissions and roles. *(Pertaining to this
   last item, I plan on attending the upcoming Sakai conference in Atlanta,
   including "Online Evaluations at Berkeley" presentation.) *

*PLU's Sakai/EvalSys Deployment*

We are currently using EvalSys 1.3.x. with Sakai CLE 2.7.x, though very
soon we will be upgrading to Sakai CLE 2.8.x.

*Key Assumption about EvalSys Permissions and Roles*

I assume that EvalSys permissions are designed to be divided between two
kinds of roles in a mutually exclusive manner:

"Instructor" or "maintain" roles (understood as evaluatee) are to enable
only:

   - eval.take.evaluation

"Student" or "access" roles (understood as evaluator) are to enable only:

   - eval.assign.evaluation
   - eval.be.evaluated
   - eval.write.template

As will hopefully become evident in the use cases I explore below, some
roles and permissions deviate from this norm.


*1. Use Case for Administrator Evaluations *

Instead of course/instructor evaluations, distributed to students and
managed by the instructor, PLU's primary use of EvalSys has been to conduct
annual administrator (personnel) evaluations. These administrator
evaluations are constructed to fit the normative role/permissions scheme
above, and they are constructed with the following steps:

   - For each administrator or supervisor (whether a faculty chair, a dean,
   the provost, etc.) for whom an evaluation will be conducted, create an
   unpublished project site in Sakai containing Site Info and EvalSys tools.
   This project site functions merely as a grouping mechanism to associate the
   Evaluation with persons in particular roles with respect to that evaluation
   (e.g., evaluation writer/assigner [maintain] and evaluators [access]; see
   below).
   - Add to each project site one or two persons in the "maintain" role
   whose job is to create, initiate, and close 40 to 50 administrator
   evaluations for the given year. (All these evaluations are based on a
   single, predefined template with multiple choice and open ended questions.)
   - Add to each appropriate project site persons in the "access" role
   designated to evaluate the particular chair, dean, etc.

However, this process bears the following constraint. At the conclusion of
the evaluation period, persons in the "maintain" role must serve as
mediators for the evaluation report recipients, evaluatees and their
respective supervisors, neither roles of which are defined in the above
scheme. (The normative scheme combines the role of evaluatee and evaluation
writer/assigner, which our administror evaluation scheme does not follow.)
Continuing with this example, the persons with the "maintain" role (the
evaluation writers/assigners) manually export reports and distribute them
to the appropriate evaluatees and corresponding supervisors. While this
process of manually exporting and distributing reports is relatively
painless for a few evaluations, it does become cumbersome for 40+
evaluations.

A move toward streamlining this already exists without any modification to
EvalSys code. I can add the evaluatee as a participant in the designated
project site with a role distinct from "maintain" and "access". This role,
which for the sake of this example I will call "evaluatee", is intended to
permit the evaluatee to read the evaluation report directly in Sakai once
the evaluation has closed. The only permission to be enabled for this
"evaluatee" role is eval.be.evaluated; the other three permissions are
disabled. I've tested this and it appears to work well.

However, this if I also want to grant the evaluatee's supervisor(s) direct
access to these reports in Sakai, I can do so, but with the following flaw.
I can give the evaluatee's supervisor(s) an eval.be.evaluated permission to
access the report directly (i.e., add them as participants with role
"evaluatee"), but doing so obviously yields an artifact in the evaluatee's
supervisor(s)' Evaluations dashboard. The evaluatee's supervisor(s) are
(mis)informed that they too are being evaluated, listed as a member of
"Groups I can be evaluated in:" for the project site (group) designated for
only one evaluatee (i.e., not the evaluatee).

While bearing in mind the complex issues of backwards-compatibility when
thinking about modifying changes to EvalSys roles and permissions, this use
case suggests value in decoupling a permission like "eval.read.report" from
"eval.be.evaluated". The conceptual decoupling of the role of "evaluatee"
from the "Instructor/maintain" role of evaluation writer/assigner is
already supported by EvalSys code. The role of "evaluatee's supervisor who
is also not an evaluator" is not supported currently.

*2. Use Case for One-on-One Performance Evaluations*

Recently I have received a request about the possibility of using EvalSys
to conduct annual one-on-one performance evaluations in Sakai for each
employee for a particular division, synchronised to the anniversaries of
each employee's start dates. I divide this request into two sets of
requirements.

*2a. Roles and Permissions*

The first set of requirements for this use case pertain to roles and
permissions, much like the previous use case. The evaluators in this case
would include both an employee (performing a self-evaluation) and the
employee's supervisor. Also, the employee and supervisor would ideally have
access to the evaluation report, not mediated by a "maintain" role.
Furthermore, a subset of supervisors up the management chain would also
ideally be granted direct access to this report, though they would not be
taking the evaluation themselves.

I recall amidst my own black-box testing that the current set of EvalSys
permissions do not permit users to function as both evaluators and
evaluatees. In other words defining a role to have the permissions
eval.take.evaluation and eval.be.evaluated seemed to not permit such users
to either take the evaluation or access the report. Enabling both
permissions cancelled one another out, and I presume this is by design.
To circumvent this, I think I would need to create two project sites, each
with their own Evaluation objects. Furthermore, I would need to settle with
the constraint of two distinct evaluation reports instead of one joint
report.

This use case suggests value in permitting a role to both take an
evaluation and to access the resulting report.

*2b. Manipulating EvalSys Entities via CLE WebServices*

The second set of requirements for this use case entails writing code to
glue together the following:

   - PLU's enterprise system for managing employee data (such as employee
   start dates) and
   - the management of performance evaluations in Sakai.

Assuming that I use the EvalSys web browser interface (or tool) to manage
these performance evaluations, I imagine that I would need to manually
create a project site and evaluation (or perhaps two each-- see section 2a
above) for each employee, once per year, with dates synchronised to the
anniversary of each employee's start date. Thus, I imagine this process to
be very time consuming and prone to serious errors (e.g., inadvertently
sending an evaluation report to the wrong recipients).

Therefore, I am curious about the feasibility (and/or relative ease) of
creating, configuring, and initiating evaluations (EvalSys objects) via web
services. I've not yet familiarised myself with EvalSys' design and APIs
(nor Sakai's Entity / EntitiyManager / EntityBroker / EntityProducer
scheme), though I presume a service layer exists for manipulating EvalSys
objects. Assuming this is feasible to implement, I'm curious if anyone else
has attempted it. For what it's worth, I have some experience writing web
service methods deployed within
$CATALINA_HOME/webapps/sakai-axis/SakaiScript.jws.

Finally, since I hope to also begin exploring Sakai OAE soon, I'm curious
if any extant EvalSys plans for Sakai OAE might shed light on some or any
of these issues I've described.

Thanks for taking the time to read this admittedly long note.

Sean

-- 
Sean Horner
Web Application Developer
Pacific Lutheran University
hornersa at plu.edu
253-536-5021
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/evaluation/attachments/20120517/dd0b53b3/attachment.html