[Portfolio] A manifesto for Grading and Rating in Sakai

John Norman john at caret.cam.ac.uk
Fri Oct 16 03:02:19 PDT 2009

I have collected my thoughts around grading and rating in Sakai. I  
offer them now partly because I feel ready, partly because there are  
open questions about Gradebook in Sakai 3 and partly because we have  
just had a discussion in which I suggest it is hard to break things  
out of a coherent Sakai 3 project. If accepted as is, this represents  
a logical area of activity than can readily be envisioned as a  
standalone activity - maybe even a separate product.

First of all I'd like to suggest that grading is a subset of a general  
rating and feedback activity. Many artifacts can be rated, from  
instructor performance during a course (course evaluation), through  
quality of a teaching asset or exercise (rating) to assessing the  
quality of a student portfolio (feedback) and assessing the  
performance of a student on an assignment or test (grading). The  
common pattern is: an artifact is produced by one individual (or  
group) and some value judgement is recorded by one or more other people.

The process by which an artifact is judged can be simple or complex.  
Complex processes include multi-stage workflows where raw scores are  
obtained by one process and raw scores moderated to a final grade by  
another process. I see plagiarism detection as one particular wrinkle  
in such a workflow.

I suggest that (nearly) everything in Sakai should be ratable/ 
gradable. I will refer to the ratable/gradable elements as "artifacts"  
to indicate that they may not be 'technical elements' but some  
aggregation of technical elements that makes sense for rating/grading  
purposes. Moreover, we should not forget that some of the artifacts  
that are rated/graded may not be electronic and the 'artifact' may be  
a proxy for some real world activity or output that cannot be captured  

The activity of rating/grading is essentially a human judgement. Tests  
and quizzes represent a subset of this situation where the human  
codifies their judgement into rules applied by the testing engine and  
the test engine automates the application of scores. The Quiz with the  
student answers represents the artifact and the raw scores and/or  
processed grade represents the judgement. The people involved in  
rating/grading can be anyone: students, teachers, peers.

The artifact to be rated or graded may not be stable over time, in  
which case a 'snapshot' of some kind is desirable for audit purposes.  
An example might be the state of my personal portfolio pages on the  
first day of May, when they are declared to be assessed. I may wish to  
continue maintaining the pages after the assessment, but their status  
at the time of assessing is worth recording. A different example might  
be my performance in a piece of drama. I have no idea how this would  
be recorded in the real world, but I imagine that the grader might  
write down some critique/commentary and then assign a grade. The  
critique/commentary would become the recorded artifact (in some places  
there might be a video recording but I don't assume that) and  
separately there would be a grade/score/rating. Teacher performance in  
class evaluated by students is not far from this model. The questions  
in the evaluation form might be considered the rubric for the teachers  

In this world, we would want a flexible reporting platform that allows  
grade information (including an archive of artifact snapshots) to be  
collected and analysed (and sometimes further processed). I suggest we  
think of using something like BIRT to create this flexible reporting  
environment and then consider certain predefined views of the data and  
derived reports from the data as the essence of "GradeBook"  
functionality. i.e. "GradeBook" is a subset of functionality from a  
powerful reporting environment. Ultimately "the official record" will  
need to be updated.

I think it is really important to anticipate that some of the  
artifacts to be graded may come from outside Sakai and Sakai needs to  
be able to accept artifacts for grading and also to accept graded  
artifacts for inclusion in reporting. I see two main implementation  
options for Sakai
1. A Sakai service with published external entry points (Moodle/Mahara  
integration would be an example)
2. A new Sakai 'product' which would be an institutional grading/ 
rating service that receives artifacts from a number of places  
(including the Sakai Course Management System) and manages the grading/ 
rating workflow into a flexible reporting system that creates a  
complete record for an individual and allows this information to be  
displayed in a number of places (including Sakai CMS)

A strong attraction of the second model is that it fits with the idea  
that assessing performance is a core competence of the institution  
that preceded and will survive the CMS, but which is unlikely to be  
developed for us by the commercial world. It could also represent a  
shared service with a student information system.

Having set out my manifesto, it is interesting to consider what the  
product council might do with it. From my personal perspective it  
would be great if we adopted it as the Sakai manifesto (following  
review/revision) and called for developments to align with it, but  
there is an open question regarding the value of 'adoption' of the  
manifesto if nobody is interested in developing products/code that  
address the manifesto.


PS I have forwarded this message that I saw as I came in this morning  
because in my mind it illustrates an early step in the direction of my  
manifesto, although I have taken it much further (perhaps  

Begin forwarded message:

> From: David Horwitz <david.horwitz at uct.ac.za>
> Date: 16 October 2009 09:29:58 BST
> To: sakai-dev <sakai-dev at collab.sakaiproject.org>,  production at collab.sakaiproject.org 
> , announcements at collab.sakaiproject.org
> Subject: [Announcements] 2.7 Framework: commons and edu-servise  
> 1.0.0-beta01 released
> Hi All,
> We're proud to announce the first of 2 framework releases in support  
> of the upcoming 2.7 release. The creation of these bundles aims to  
> rationalize our dependency tree and enable a more modular approach  
> to Sakai releases.
> Commons 1.0.0-beta01
> The commons package contains common services depended on by a number  
> of Sakai tools, but outside the scope of the Kernel. The services  
> included are:
> SakaiPerson Service (profile data)
> Type Service
> privacy service
> archive service
> import service
> The project site can be viewed at:
> http://source.sakaiproject.org/release/common/1.0.0-beta01/
> (Note experimental site no Sakai skins etc.)
> Edu-Services 1.0.0-beta01
> Edu-services contain core shared services that support teaching and  
> learning functionality in Sakai. It contains:
> Course management service
> Gradebook service
> Sections service
> The project site can be viewed at:
> http://source.sakaiproject.org/release/edu-services/1.0.0-beta01/
> _______________________________________________
> announcements mailing list
> announcements at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/announcements
> TO UNSUBSCRIBE: send email to announcements-unsubscribe at collab.sakaiproject.org 
>  with a subject of "unsubscribe"

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/portfolio/attachments/20091016/800cf403/attachment.html 

More information about the portfolio mailing list