[Portfolio] [DG: Teaching & Learning] A manifesto for Grading and Rating in Sakai
Mark Norton
markjnorton at earthlink.net
Fri Oct 16 05:41:47 PDT 2009
In general, I agree with what you've written here, John. I also think
that grading is part of a larger class of rating activities. That said,
think that people have very specific expectations of a grading activity
vs. a more general rating one. The protocols for rating teacher
performance seem to be quite different than grading a test. I grant you
that this is just a matter of workflow from a systems perspective, but
we also need to consider this from the user's perspective. Instructors
are looking for very specific functionality to maintain grades over an
academic term. Sakai GB-1 provided basic capability, and GB-2 has
carried it much further. A Sakai 3 rating/grading initiative would need
to take these things into account very early on, I believe.
- Mark Norton
John Norman wrote:
> I have collected my thoughts around grading and rating in Sakai. I
> offer them now partly because I feel ready, partly because there are
> open questions about Gradebook in Sakai 3 and partly because we have
> just had a discussion in which I suggest it is hard to break things
> out of a coherent Sakai 3 project. If accepted as is, this represents
> a logical area of activity than can readily be envisioned as a
> standalone activity - maybe even a separate product.
>
> First of all I'd like to suggest that grading is a subset of a general
> rating and feedback activity. Many artifacts can be rated, from
> instructor performance during a course (course evaluation), through
> quality of a teaching asset or exercise (rating) to assessing the
> quality of a student portfolio (feedback) and assessing the
> performance of a student on an assignment or test (grading). The
> common pattern is: an artifact is produced by one individual (or
> group) and some value judgement is recorded by one or more other people.
>
> The process by which an artifact is judged can be simple or complex.
> Complex processes include multi-stage workflows where raw scores are
> obtained by one process and raw scores moderated to a final grade by
> another process. I see plagiarism detection as one particular wrinkle
> in such a workflow.
>
> I suggest that (nearly) everything in Sakai should be
> ratable/gradable. I will refer to the ratable/gradable elements as
> "artifacts" to indicate that they may not be 'technical elements' but
> some aggregation of technical elements that makes sense for
> rating/grading purposes. Moreover, we should not forget that some of
> the artifacts that are rated/graded may not be electronic and the
> 'artifact' may be a proxy for some real world activity or output that
> cannot be captured electronically.
>
> The activity of rating/grading is essentially a human judgement. Tests
> and quizzes represent a subset of this situation where the human
> codifies their judgement into rules applied by the testing engine and
> the test engine automates the application of scores. The Quiz with the
> student answers represents the artifact and the raw scores and/or
> processed grade represents the judgement. The people involved in
> rating/grading can be anyone: students, teachers, peers.
>
> The artifact to be rated or graded may not be stable over time, in
> which case a 'snapshot' of some kind is desirable for audit purposes.
> An example might be the state of my personal portfolio pages on the
> first day of May, when they are declared to be assessed. I may wish to
> continue maintaining the pages after the assessment, but their status
> at the time of assessing is worth recording. A different example might
> be my performance in a piece of drama. I have no idea how this would
> be recorded in the real world, but I imagine that the grader might
> write down some critique/commentary and then assign a grade. The
> critique/commentary would become the recorded artifact (in some places
> there might be a video recording but I don't assume that) and
> separately there would be a grade/score/rating. Teacher performance in
> class evaluated by students is not far from this model. The questions
> in the evaluation form might be considered the rubric for the teachers
> performance.
>
> In this world, we would want a flexible reporting platform that allows
> grade information (including an archive of artifact snapshots) to be
> collected and analysed (and sometimes further processed). I suggest we
> think of using something like BIRT to create this flexible reporting
> environment and then consider certain predefined views of the data and
> derived reports from the data as the essence of "GradeBook"
> functionality. i.e. "GradeBook" is a subset of functionality from a
> powerful reporting environment. Ultimately "the official record" will
> need to be updated.
>
> I think it is really important to anticipate that some of the
> artifacts to be graded may come from outside Sakai and Sakai needs to
> be able to accept artifacts for grading and also to accept graded
> artifacts for inclusion in reporting. I see two main implementation
> options for Sakai
> 1. A Sakai service with published external entry points (Moodle/Mahara
> integration would be an example)
> 2. A new Sakai 'product' which would be an institutional
> grading/rating service that receives artifacts from a number of places
> (including the Sakai Course Management System) and manages the
> grading/rating workflow into a flexible reporting system that creates
> a complete record for an individual and allows this information to be
> displayed in a number of places (including Sakai CMS)
>
> A strong attraction of the second model is that it fits with the idea
> that assessing performance is a core competence of the institution
> that preceded and will survive the CMS, but which is unlikely to be
> developed for us by the commercial world. It could also represent a
> shared service with a student information system.
>
> Having set out my manifesto, it is interesting to consider what the
> product council might do with it. From my personal perspective it
> would be great if we adopted it as the Sakai manifesto (following
> review/revision) and called for developments to align with it, but
> there is an open question regarding the value of 'adoption' of the
> manifesto if nobody is interested in developing products/code that
> address the manifesto.
>
> John
>
> PS I have forwarded this message that I saw as I came in this morning
> because in my mind it illustrates an early step in the direction of my
> manifesto, although I have taken it much further (perhaps unrecognisably).
>
> Begin forwarded message:
>
>> *From: *David Horwitz <david.horwitz at uct.ac.za
>> <mailto:david.horwitz at uct.ac.za>>
>> *Date: *16 October 2009 09:29:58 BST
>> *To: *sakai-dev <sakai-dev at collab.sakaiproject.org
>> <mailto:sakai-dev at collab.sakaiproject.org>>,
>> production at collab.sakaiproject.org
>> <mailto:production at collab.sakaiproject.org>,
>> announcements at collab.sakaiproject.org
>> <mailto:announcements at collab.sakaiproject.org>
>> *Subject: **[Announcements] 2.7 Framework: commons and edu-servise
>> 1.0.0-beta01 released*
>>
>> Hi All,
>>
>> We're proud to announce the first of 2 framework releases in support
>> of the upcoming 2.7 release. The creation of these bundles aims to
>> rationalize our dependency tree and enable a more modular approach to
>> Sakai releases.
>>
>> *Commons 1.0.0-beta01
>> *The commons package contains common services depended on by a number
>> of Sakai tools, but outside the scope of the Kernel. The services
>> included are:
>>
>> SakaiPerson Service (profile data)
>> Type Service
>> privacy service
>> archive service
>> import service
>>
>> The project site can be viewed at:
>> http://source.sakaiproject.org/release/common/1.0.0-beta01/
>> (Note experimental site no Sakai skins etc.)
>>
>> *Edu-Services 1.0.0-beta01
>> *Edu-services contain core shared services that support teaching and
>> learning functionality in Sakai. It contains:
>>
>> Course management service
>> Gradebook service
>> Sections service
>>
>> The project site can be viewed at:
>> http://source.sakaiproject.org/release/edu-services/1.0.0-beta01/
>>
>>
>> _______________________________________________
>> announcements mailing list
>> announcements at collab.sakaiproject.org
>> <mailto:announcements at collab.sakaiproject.org>
>> http://collab.sakaiproject.org/mailman/listinfo/announcements
>>
>> TO UNSUBSCRIBE: send email to
>> announcements-unsubscribe at collab.sakaiproject.org with a subject of
>> "unsubscribe"
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> pedagogy mailing list
> pedagogy at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/pedagogy
>
> TO UNSUBSCRIBE: send email to pedagogy-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
More information about the portfolio
mailing list