[Portfolio] A manifesto for Grading and Rating in Sakai
John Norman
john at caret.cam.ac.uk
Fri Oct 16 03:02:19 PDT 2009
I have collected my thoughts around grading and rating in Sakai. I
offer them now partly because I feel ready, partly because there are
open questions about Gradebook in Sakai 3 and partly because we have
just had a discussion in which I suggest it is hard to break things
out of a coherent Sakai 3 project. If accepted as is, this represents
a logical area of activity than can readily be envisioned as a
standalone activity - maybe even a separate product.
First of all I'd like to suggest that grading is a subset of a general
rating and feedback activity. Many artifacts can be rated, from
instructor performance during a course (course evaluation), through
quality of a teaching asset or exercise (rating) to assessing the
quality of a student portfolio (feedback) and assessing the
performance of a student on an assignment or test (grading). The
common pattern is: an artifact is produced by one individual (or
group) and some value judgement is recorded by one or more other people.
The process by which an artifact is judged can be simple or complex.
Complex processes include multi-stage workflows where raw scores are
obtained by one process and raw scores moderated to a final grade by
another process. I see plagiarism detection as one particular wrinkle
in such a workflow.
I suggest that (nearly) everything in Sakai should be ratable/
gradable. I will refer to the ratable/gradable elements as "artifacts"
to indicate that they may not be 'technical elements' but some
aggregation of technical elements that makes sense for rating/grading
purposes. Moreover, we should not forget that some of the artifacts
that are rated/graded may not be electronic and the 'artifact' may be
a proxy for some real world activity or output that cannot be captured
electronically.
The activity of rating/grading is essentially a human judgement. Tests
and quizzes represent a subset of this situation where the human
codifies their judgement into rules applied by the testing engine and
the test engine automates the application of scores. The Quiz with the
student answers represents the artifact and the raw scores and/or
processed grade represents the judgement. The people involved in
rating/grading can be anyone: students, teachers, peers.
The artifact to be rated or graded may not be stable over time, in
which case a 'snapshot' of some kind is desirable for audit purposes.
An example might be the state of my personal portfolio pages on the
first day of May, when they are declared to be assessed. I may wish to
continue maintaining the pages after the assessment, but their status
at the time of assessing is worth recording. A different example might
be my performance in a piece of drama. I have no idea how this would
be recorded in the real world, but I imagine that the grader might
write down some critique/commentary and then assign a grade. The
critique/commentary would become the recorded artifact (in some places
there might be a video recording but I don't assume that) and
separately there would be a grade/score/rating. Teacher performance in
class evaluated by students is not far from this model. The questions
in the evaluation form might be considered the rubric for the teachers
performance.
In this world, we would want a flexible reporting platform that allows
grade information (including an archive of artifact snapshots) to be
collected and analysed (and sometimes further processed). I suggest we
think of using something like BIRT to create this flexible reporting
environment and then consider certain predefined views of the data and
derived reports from the data as the essence of "GradeBook"
functionality. i.e. "GradeBook" is a subset of functionality from a
powerful reporting environment. Ultimately "the official record" will
need to be updated.
I think it is really important to anticipate that some of the
artifacts to be graded may come from outside Sakai and Sakai needs to
be able to accept artifacts for grading and also to accept graded
artifacts for inclusion in reporting. I see two main implementation
options for Sakai
1. A Sakai service with published external entry points (Moodle/Mahara
integration would be an example)
2. A new Sakai 'product' which would be an institutional grading/
rating service that receives artifacts from a number of places
(including the Sakai Course Management System) and manages the grading/
rating workflow into a flexible reporting system that creates a
complete record for an individual and allows this information to be
displayed in a number of places (including Sakai CMS)
A strong attraction of the second model is that it fits with the idea
that assessing performance is a core competence of the institution
that preceded and will survive the CMS, but which is unlikely to be
developed for us by the commercial world. It could also represent a
shared service with a student information system.
Having set out my manifesto, it is interesting to consider what the
product council might do with it. From my personal perspective it
would be great if we adopted it as the Sakai manifesto (following
review/revision) and called for developments to align with it, but
there is an open question regarding the value of 'adoption' of the
manifesto if nobody is interested in developing products/code that
address the manifesto.
John
PS I have forwarded this message that I saw as I came in this morning
because in my mind it illustrates an early step in the direction of my
manifesto, although I have taken it much further (perhaps
unrecognisably).
Begin forwarded message:
> From: David Horwitz <david.horwitz at uct.ac.za>
> Date: 16 October 2009 09:29:58 BST
> To: sakai-dev <sakai-dev at collab.sakaiproject.org>, production at collab.sakaiproject.org
> , announcements at collab.sakaiproject.org
> Subject: [Announcements] 2.7 Framework: commons and edu-servise
> 1.0.0-beta01 released
>
> Hi All,
>
> We're proud to announce the first of 2 framework releases in support
> of the upcoming 2.7 release. The creation of these bundles aims to
> rationalize our dependency tree and enable a more modular approach
> to Sakai releases.
>
> Commons 1.0.0-beta01
> The commons package contains common services depended on by a number
> of Sakai tools, but outside the scope of the Kernel. The services
> included are:
>
> SakaiPerson Service (profile data)
> Type Service
> privacy service
> archive service
> import service
>
> The project site can be viewed at:
> http://source.sakaiproject.org/release/common/1.0.0-beta01/
> (Note experimental site no Sakai skins etc.)
>
> Edu-Services 1.0.0-beta01
> Edu-services contain core shared services that support teaching and
> learning functionality in Sakai. It contains:
>
> Course management service
> Gradebook service
> Sections service
>
> The project site can be viewed at:
> http://source.sakaiproject.org/release/edu-services/1.0.0-beta01/
>
>
> _______________________________________________
> announcements mailing list
> announcements at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/announcements
>
> TO UNSUBSCRIBE: send email to announcements-unsubscribe at collab.sakaiproject.org
> with a subject of "unsubscribe"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/portfolio/attachments/20091016/800cf403/attachment.html
More information about the portfolio
mailing list