[Portfolio] [Management] A manifesto for Grading and Rating in Sakai

Noah Botimer botimer at umich.edu
Fri Oct 16 08:07:55 PDT 2009

Hello John,

Thank you for the rather comprehensive narrative. I believe that  
these are important for the archives as we change our ideas and  
software over time. They leave a better historical record of our  
state of mind at any given point than a pile of JIRA tickets. Our  
successive approximation is better validated when we have a record of  
these richer "data" points.

Now, more on task...

This is a fair account from my perspective, and is especially  
important in that it carves out a first-class place for two things  
that have been historical weaknesses:

  1. The ability to treat various artifacts individually and in  
collections, consistently, across types of "stuff" and activity  
(e.g., reflection vs. feedback vs. grading)

  2. The ability to retrieve meaningful performance (or other) data  
in detail and aggregate, consistently, and without extensive one-off  

Interestingly enough, these two areas are what I've spent four years  
working on -- so I suppose it's not surprising that I call them out.  
I mention them as weaknesses from my experience. It has been  
difficult to combine assignment information with student-crafted  
presentation. It has been difficult to combine course-based  
(assignment, quiz, etc.) data and program-based activity (annual  
review, capstone, student teaching performance) and map them to  
curricular goals and reports...

Please do not take my comments as complaints of where we are. What is  
more important is that I see this narrative as recognizing these  
activities not, as we have, as things that can be bolted on post- 
construction but, rather, as shaping the core provisions of a  
meaningful academic and collaborative platform. We are, as a  
community, much more aware of our successes and shortfalls. This, I  
feel, is very healthy and inspiring.

I believe this discussion is going in the right direction and  
sincerely hope that we can find the energy to support it.


On Oct 16, 2009, at 6:02 AM, John Norman wrote:

> I have collected my thoughts around grading and rating in Sakai. I  
> offer them now partly because I feel ready, partly because there  
> are open questions about Gradebook in Sakai 3 and partly because we  
> have just had a discussion in which I suggest it is hard to break  
> things out of a coherent Sakai 3 project. If accepted as is, this  
> represents a logical area of activity than can readily be  
> envisioned as a standalone activity - maybe even a separate product.
> First of all I'd like to suggest that grading is a subset of a  
> general rating and feedback activity. Many artifacts can be rated,  
> from instructor performance during a course (course evaluation),  
> through quality of a teaching asset or exercise (rating) to  
> assessing the quality of a student portfolio (feedback) and  
> assessing the performance of a student on an assignment or test  
> (grading). The common pattern is: an artifact is produced by one  
> individual (or group) and some value judgement is recorded by one  
> or more other people.
> The process by which an artifact is judged can be simple or  
> complex. Complex processes include multi-stage workflows where raw  
> scores are obtained by one process and raw scores moderated to a  
> final grade by another process. I see plagiarism detection as one  
> particular wrinkle in such a workflow.
> I suggest that (nearly) everything in Sakai should be ratable/ 
> gradable. I will refer to the ratable/gradable elements as  
> "artifacts" to indicate that they may not be 'technical elements'  
> but some aggregation of technical elements that makes sense for  
> rating/grading purposes. Moreover, we should not forget that some  
> of the artifacts that are rated/graded may not be electronic and  
> the 'artifact' may be a proxy for some real world activity or  
> output that cannot be captured electronically.
> The activity of rating/grading is essentially a human judgement.  
> Tests and quizzes represent a subset of this situation where the  
> human codifies their judgement into rules applied by the testing  
> engine and the test engine automates the application of scores. The  
> Quiz with the student answers represents the artifact and the raw  
> scores and/or processed grade represents the judgement. The people  
> involved in rating/grading can be anyone: students, teachers, peers.
> The artifact to be rated or graded may not be stable over time, in  
> which case a 'snapshot' of some kind is desirable for audit  
> purposes. An example might be the state of my personal portfolio  
> pages on the first day of May, when they are declared to be  
> assessed. I may wish to continue maintaining the pages after the  
> assessment, but their status at the time of assessing is worth  
> recording. A different example might be my performance in a piece  
> of drama. I have no idea how this would be recorded in the real  
> world, but I imagine that the grader might write down some critique/ 
> commentary and then assign a grade. The critique/commentary would  
> become the recorded artifact (in some places there might be a video  
> recording but I don't assume that) and separately there would be a  
> grade/score/rating. Teacher performance in class evaluated by  
> students is not far from this model. The questions in the  
> evaluation form might be considered the rubric for the teachers  
> performance.
> In this world, we would want a flexible reporting platform that  
> allows grade information (including an archive of artifact  
> snapshots) to be collected and analysed (and sometimes further  
> processed). I suggest we think of using something like BIRT to  
> create this flexible reporting environment and then consider  
> certain predefined views of the data and derived reports from the  
> data as the essence of "GradeBook" functionality. i.e. "GradeBook"  
> is a subset of functionality from a powerful reporting environment.  
> Ultimately "the official record" will need to be updated.
> I think it is really important to anticipate that some of the  
> artifacts to be graded may come from outside Sakai and Sakai needs  
> to be able to accept artifacts for grading and also to accept  
> graded artifacts for inclusion in reporting. I see two main  
> implementation options for Sakai
> 1. A Sakai service with published external entry points (Moodle/ 
> Mahara integration would be an example)
> 2. A new Sakai 'product' which would be an institutional grading/ 
> rating service that receives artifacts from a number of places  
> (including the Sakai Course Management System) and manages the  
> grading/rating workflow into a flexible reporting system that  
> creates a complete record for an individual and allows this  
> information to be displayed in a number of places (including Sakai  
> CMS)
> A strong attraction of the second model is that it fits with the  
> idea that assessing performance is a core competence of the  
> institution that preceded and will survive the CMS, but which is  
> unlikely to be developed for us by the commercial world. It could  
> also represent a shared service with a student information system.
> Having set out my manifesto, it is interesting to consider what the  
> product council might do with it. From my personal perspective it  
> would be great if we adopted it as the Sakai manifesto (following  
> review/revision) and called for developments to align with it, but  
> there is an open question regarding the value of 'adoption' of the  
> manifesto if nobody is interested in developing products/code that  
> address the manifesto.
> John
> PS I have forwarded this message that I saw as I came in this  
> morning because in my mind it illustrates an early step in the  
> direction of my manifesto, although I have taken it much further  
> (perhaps unrecognisably).
> Begin forwarded message:
>> From: David Horwitz <david.horwitz at uct.ac.za>
>> Date: 16 October 2009 09:29:58 BST
>> To: sakai-dev <sakai-dev at collab.sakaiproject.org>,   
>> production at collab.sakaiproject.org,  
>> announcements at collab.sakaiproject.org
>> Subject: [Announcements] 2.7 Framework: commons and edu-servise  
>> 1.0.0-beta01 released
>> Hi All,
>> We're proud to announce the first of 2 framework releases in  
>> support of the upcoming 2.7 release. The creation of these bundles  
>> aims to rationalize our dependency tree and enable a more modular  
>> approach to Sakai releases.
>> Commons 1.0.0-beta01
>> The commons package contains common services depended on by a  
>> number of Sakai tools, but outside the scope of the Kernel. The  
>> services included are:
>> SakaiPerson Service (profile data)
>> Type Service
>> privacy service
>> archive service
>> import service
>> The project site can be viewed at:
>> http://source.sakaiproject.org/release/common/1.0.0-beta01/
>> (Note experimental site no Sakai skins etc.)
>> Edu-Services 1.0.0-beta01
>> Edu-services contain core shared services that support teaching  
>> and learning functionality in Sakai. It contains:
>> Course management service
>> Gradebook service
>> Sections service
>> The project site can be viewed at:
>> http://source.sakaiproject.org/release/edu-services/1.0.0-beta01/
>> _______________________________________________
>> announcements mailing list
>> announcements at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/announcements
>> TO UNSUBSCRIBE: send email to announcements- 
>> unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
> _______________________________________________
> management mailing list
> management at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/management
> TO UNSUBSCRIBE: send email to management- 
> unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/portfolio/attachments/20091016/d21280b2/attachment-0001.html 

More information about the portfolio mailing list