[Portfolio] [DG: Teaching & Learning] A manifesto for Grading and Rating in Sakai

Mark Norton markjnorton at earthlink.net
Fri Oct 16 05:41:47 PDT 2009

In general, I agree with what you've written here, John.  I also think 
that grading is part of a larger class of rating activities.  That said, 
think that people have very specific expectations of a grading activity 
vs. a more general rating one.  The protocols for rating teacher 
performance seem to be quite different than grading a test.  I grant you 
that this is just a matter of workflow from a systems perspective, but 
we also need to consider this from the user's perspective.  Instructors 
are looking for very specific functionality to maintain grades over an 
academic term.  Sakai GB-1 provided basic capability, and GB-2 has 
carried it much further.  A Sakai 3 rating/grading initiative would need 
to take these things into account very early on, I believe.

- Mark Norton

John Norman wrote:
> I have collected my thoughts around grading and rating in Sakai. I 
> offer them now partly because I feel ready, partly because there are 
> open questions about Gradebook in Sakai 3 and partly because we have 
> just had a discussion in which I suggest it is hard to break things 
> out of a coherent Sakai 3 project. If accepted as is, this represents 
> a logical area of activity than can readily be envisioned as a 
> standalone activity - maybe even a separate product.
> First of all I'd like to suggest that grading is a subset of a general 
> rating and feedback activity. Many artifacts can be rated, from 
> instructor performance during a course (course evaluation), through 
> quality of a teaching asset or exercise (rating) to assessing the 
> quality of a student portfolio (feedback) and assessing the 
> performance of a student on an assignment or test (grading). The 
> common pattern is: an artifact is produced by one individual (or 
> group) and some value judgement is recorded by one or more other people. 
> The process by which an artifact is judged can be simple or complex. 
> Complex processes include multi-stage workflows where raw scores are 
> obtained by one process and raw scores moderated to a final grade by 
> another process. I see plagiarism detection as one particular wrinkle 
> in such a workflow.
> I suggest that (nearly) everything in Sakai should be 
> ratable/gradable. I will refer to the ratable/gradable elements as 
> "artifacts" to indicate that they may not be 'technical elements' but 
> some aggregation of technical elements that makes sense for 
> rating/grading purposes. Moreover, we should not forget that some of 
> the artifacts that are rated/graded may not be electronic and the 
> 'artifact' may be a proxy for some real world activity or output that 
> cannot be captured electronically.
> The activity of rating/grading is essentially a human judgement. Tests 
> and quizzes represent a subset of this situation where the human 
> codifies their judgement into rules applied by the testing engine and 
> the test engine automates the application of scores. The Quiz with the 
> student answers represents the artifact and the raw scores and/or 
> processed grade represents the judgement. The people involved in 
> rating/grading can be anyone: students, teachers, peers.
> The artifact to be rated or graded may not be stable over time, in 
> which case a 'snapshot' of some kind is desirable for audit purposes. 
> An example might be the state of my personal portfolio pages on the 
> first day of May, when they are declared to be assessed. I may wish to 
> continue maintaining the pages after the assessment, but their status 
> at the time of assessing is worth recording. A different example might 
> be my performance in a piece of drama. I have no idea how this would 
> be recorded in the real world, but I imagine that the grader might 
> write down some critique/commentary and then assign a grade. The 
> critique/commentary would become the recorded artifact (in some places 
> there might be a video recording but I don't assume that) and 
> separately there would be a grade/score/rating. Teacher performance in 
> class evaluated by students is not far from this model. The questions 
> in the evaluation form might be considered the rubric for the teachers 
> performance.
> In this world, we would want a flexible reporting platform that allows 
> grade information (including an archive of artifact snapshots) to be 
> collected and analysed (and sometimes further processed). I suggest we 
> think of using something like BIRT to create this flexible reporting 
> environment and then consider certain predefined views of the data and 
> derived reports from the data as the essence of "GradeBook" 
> functionality. i.e. "GradeBook" is a subset of functionality from a 
> powerful reporting environment. Ultimately "the official record" will 
> need to be updated. 
> I think it is really important to anticipate that some of the 
> artifacts to be graded may come from outside Sakai and Sakai needs to 
> be able to accept artifacts for grading and also to accept graded 
> artifacts for inclusion in reporting. I see two main implementation 
> options for Sakai
> 1. A Sakai service with published external entry points (Moodle/Mahara 
> integration would be an example)
> 2. A new Sakai 'product' which would be an institutional 
> grading/rating service that receives artifacts from a number of places 
> (including the Sakai Course Management System) and manages the 
> grading/rating workflow into a flexible reporting system that creates 
> a complete record for an individual and allows this information to be 
> displayed in a number of places (including Sakai CMS)
> A strong attraction of the second model is that it fits with the idea 
> that assessing performance is a core competence of the institution 
> that preceded and will survive the CMS, but which is unlikely to be 
> developed for us by the commercial world. It could also represent a 
> shared service with a student information system.
> Having set out my manifesto, it is interesting to consider what the 
> product council might do with it. From my personal perspective it 
> would be great if we adopted it as the Sakai manifesto (following 
> review/revision) and called for developments to align with it, but 
> there is an open question regarding the value of 'adoption' of the 
> manifesto if nobody is interested in developing products/code that 
> address the manifesto.
> John
> PS I have forwarded this message that I saw as I came in this morning 
> because in my mind it illustrates an early step in the direction of my 
> manifesto, although I have taken it much further (perhaps unrecognisably).
> Begin forwarded message:
>> *From: *David Horwitz <david.horwitz at uct.ac.za 
>> <mailto:david.horwitz at uct.ac.za>>
>> *Date: *16 October 2009 09:29:58 BST
>> *To: *sakai-dev <sakai-dev at collab.sakaiproject.org 
>> <mailto:sakai-dev at collab.sakaiproject.org>>,  
>> production at collab.sakaiproject.org 
>> <mailto:production at collab.sakaiproject.org>, 
>> announcements at collab.sakaiproject.org 
>> <mailto:announcements at collab.sakaiproject.org>
>> *Subject: **[Announcements] 2.7 Framework: commons and edu-servise 
>> 1.0.0-beta01 released*
>> Hi All,
>> We're proud to announce the first of 2 framework releases in support 
>> of the upcoming 2.7 release. The creation of these bundles aims to 
>> rationalize our dependency tree and enable a more modular approach to 
>> Sakai releases.
>> *Commons 1.0.0-beta01
>> *The commons package contains common services depended on by a number 
>> of Sakai tools, but outside the scope of the Kernel. The services 
>> included are:
>> SakaiPerson Service (profile data)
>> Type Service
>> privacy service
>> archive service
>> import service
>> The project site can be viewed at:
>> http://source.sakaiproject.org/release/common/1.0.0-beta01/
>> (Note experimental site no Sakai skins etc.)
>> *Edu-Services 1.0.0-beta01
>> *Edu-services contain core shared services that support teaching and 
>> learning functionality in Sakai. It contains:
>> Course management service
>> Gradebook service
>> Sections service
>> The project site can be viewed at:
>> http://source.sakaiproject.org/release/edu-services/1.0.0-beta01/
>> _______________________________________________
>> announcements mailing list
>> announcements at collab.sakaiproject.org 
>> <mailto:announcements at collab.sakaiproject.org>
>> http://collab.sakaiproject.org/mailman/listinfo/announcements
>> TO UNSUBSCRIBE: send email to 
>> announcements-unsubscribe at collab.sakaiproject.org with a subject of 
>> "unsubscribe"
> ------------------------------------------------------------------------
> _______________________________________________
> pedagogy mailing list
> pedagogy at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/pedagogy
> TO UNSUBSCRIBE: send email to pedagogy-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"

More information about the portfolio mailing list