[DG: Teaching & Learning] [Management] A manifesto for Grading and Rating in Sakai

Fri Oct 16 11:16:16 PDT 2009

Bravo John, Mark and Noah!

Not only do your comments address critical features of grading and
feedback, your appeal to the community, and, more importantly, to the
Product Council, strikes at the heart of deciding the way forward.  While
I do agree that the community should decide what goes into Sakai 3, I also
think that it is crucial that *someone* give some direction about the
general structure.  It should be possible to add grading/feedback to just
about anything in Sakai, and even things that aren't in Sakai, but if the
hooks are not in the right places, the whole concept breaks down.  For
example, if the discussion widget does not have hooks for grading/feedback
on a comment, then commenting can't be part of an assignment.  There has
been a really good beginning to a discussion of this issue in the weekly
Teaching and Learning conference call, and it is conceivable that
eventually those ideas would filter back to anyone working on some new
part of Sakai3.  However, it is also clear that those development efforts
on a new discussion widget are ongoing right now, even before John's
manifesto has been widely adopted.  Maybe there is another channel for
these interconnectivity issues to be addressed, but it seems like the
Product Council would be the most efficient way to do this.  And I would
hope that some sort of broad manifesto, similar to John's but reaching
*all* aspects of the new product, would be a pretty high priority, so
everyone hoping to contribute would not be wasting time building widgets
that don't fit.  The effort to analyze tests and quizzes through
contextual inquiry and goal directed design (as in the Sakai 3.0
capabilities for learning activities project) is an important part of
this, but again, it would seem that the Product Council would be in a good
position to coordinate the work and the results.

Just my 2cents,

Thank you,

Ken Romeo

[http://kenro.web.stanford.edu]

Academic Technology Specialist [http://ats.stanford.edu]

Stanford Language Center [http://language.stanford.edu]

From: pedagogy-bounces at collab.sakaiproject.org
[mailto:pedagogy-bounces at collab.sakaiproject.org] On Behalf Of Noah
Botimer
Sent: Friday, October 16, 2009 8:08 AM
To: John Norman
Cc: management at collab.sakaiproject.org; portfolio at collab.sakaiproject.org;
pedagogy at collab.sakaiproject.org Learning
Subject: Re: [DG: Teaching & Learning] [Management] A manifesto for
Grading and Rating in Sakai

Hello John,

Thank you for the rather comprehensive narrative. I believe that these are
important for the archives as we change our ideas and software over time.
They leave a better historical record of our state of mind at any given
point than a pile of JIRA tickets. Our successive approximation is better
validated when we have a record of these richer "data" points.

Now, more on task...

This is a fair account from my perspective, and is especially important in
that it carves out a first-class place for two things that have been
historical weaknesses:

 1. The ability to treat various artifacts individually and in
collections, consistently, across types of "stuff" and activity (e.g.,
reflection vs. feedback vs. grading)

 2. The ability to retrieve meaningful performance (or other) data in
detail and aggregate, consistently, and without extensive one-off
programming

Interestingly enough, these two areas are what I've spent four years
working on -- so I suppose it's not surprising that I call them out. I
mention them as weaknesses from my experience. It has been difficult to
combine assignment information with student-crafted presentation. It has
been difficult to combine course-based (assignment, quiz, etc.) data and
program-based activity (annual review, capstone, student teaching
performance) and map them to curricular goals and reports...

Please do not take my comments as complaints of where we are. What is more
important is that I see this narrative as recognizing these activities
not, as we have, as things that can be bolted on post-construction but,
rather, as shaping the core provisions of a meaningful academic and
collaborative platform. We are, as a community, much more aware of our
successes and shortfalls. This, I feel, is very healthy and inspiring.

I believe this discussion is going in the right direction and sincerely
hope that we can find the energy to support it.

Thanks,

-Noah

On Oct 16, 2009, at 6:02 AM, John Norman wrote:

I have collected my thoughts around grading and rating in Sakai. I offer
them now partly because I feel ready, partly because there are open
questions about Gradebook in Sakai 3 and partly because we have just had a
discussion in which I suggest it is hard to break things out of a coherent
Sakai 3 project. If accepted as is, this represents a logical area of
activity than can readily be envisioned as a standalone activity - maybe
even a separate product.

First of all I'd like to suggest that grading is a subset of a general
rating and feedback activity. Many artifacts can be rated, from instructor
performance during a course (course evaluation), through quality of a
teaching asset or exercise (rating) to assessing the quality of a student
portfolio (feedback) and assessing the performance of a student on an
assignment or test (grading). The common pattern is: an artifact is
produced by one individual (or group) and some value judgement is recorded
by one or more other people. 

The process by which an artifact is judged can be simple or complex.
Complex processes include multi-stage workflows where raw scores are
obtained by one process and raw scores moderated to a final grade by
another process. I see plagiarism detection as one particular wrinkle in
such a workflow.

I suggest that (nearly) everything in Sakai should be ratable/gradable. I
will refer to the ratable/gradable elements as "artifacts" to indicate
that they may not be 'technical elements' but some aggregation of
technical elements that makes sense for rating/grading purposes. Moreover,
we should not forget that some of the artifacts that are rated/graded may
not be electronic and the 'artifact' may be a proxy for some real world
activity or output that cannot be captured electronically.

The activity of rating/grading is essentially a human judgement. Tests and
quizzes represent a subset of this situation where the human codifies
their judgement into rules applied by the testing engine and the test
engine automates the application of scores. The Quiz with the student
answers represents the artifact and the raw scores and/or processed grade
represents the judgement. The people involved in rating/grading can be
anyone: students, teachers, peers.

The artifact to be rated or graded may not be stable over time, in which
case a 'snapshot' of some kind is desirable for audit purposes. An example
might be the state of my personal portfolio pages on the first day of May,
when they are declared to be assessed. I may wish to continue maintaining
the pages after the assessment, but their status at the time of assessing
is worth recording. A different example might be my performance in a piece
of drama. I have no idea how this would be recorded in the real world, but
I imagine that the grader might write down some critique/commentary and
then assign a grade. The critique/commentary would become the recorded
artifact (in some places there might be a video recording but I don't
assume that) and separately there would be a grade/score/rating. Teacher
performance in class evaluated by students is not far from this model. The
questions in the evaluation form might be considered the rubric for the
teachers performance.

In this world, we would want a flexible reporting platform that allows
grade information (including an archive of artifact snapshots) to be
collected and analysed (and sometimes further processed). I suggest we
think of using something like BIRT to create this flexible reporting
environment and then consider certain predefined views of the data and
derived reports from the data as the essence of "GradeBook" functionality.
i.e. "GradeBook" is a subset of functionality from a powerful reporting
environment. Ultimately "the official record" will need to be updated. 

I think it is really important to anticipate that some of the artifacts to
be graded may come from outside Sakai and Sakai needs to be able to accept
artifacts for grading and also to accept graded artifacts for inclusion in
reporting. I see two main implementation options for Sakai

1. A Sakai service with published external entry points (Moodle/Mahara
integration would be an example)

2. A new Sakai 'product' which would be an institutional grading/rating
service that receives artifacts from a number of places (including the
Sakai Course Management System) and manages the grading/rating workflow
into a flexible reporting system that creates a complete record for an
individual and allows this information to be displayed in a number of
places (including Sakai CMS)

A strong attraction of the second model is that it fits with the idea that
assessing performance is a core competence of the institution that
preceded and will survive the CMS, but which is unlikely to be developed
for us by the commercial world. It could also represent a shared service
with a student information system.

Having set out my manifesto, it is interesting to consider what the
product council might do with it. From my personal perspective it would be
great if we adopted it as the Sakai manifesto (following review/revision)
and called for developments to align with it, but there is an open
question regarding the value of 'adoption' of the manifesto if nobody is
interested in developing products/code that address the manifesto.

John

PS I have forwarded this message that I saw as I came in this morning
because in my mind it illustrates an early step in the direction of my
manifesto, although I have taken it much further (perhaps unrecognisably).

Begin forwarded message:

From: David Horwitz <david.horwitz at uct.ac.za>

Date: 16 October 2009 09:29:58 BST

To: sakai-dev <sakai-dev at collab.sakaiproject.org>,
production at collab.sakaiproject.org, announcements at collab.sakaiproject.org

Subject: [Announcements] 2.7 Framework: commons and edu-servise
1.0.0-beta01 released

Hi All,

We're proud to announce the first of 2 framework releases in support of
the upcoming 2.7 release. The creation of these bundles aims to
rationalize our dependency tree and enable a more modular approach to
Sakai releases.

Commons 1.0.0-beta01
The commons package contains common services depended on by a number of
Sakai tools, but outside the scope of the Kernel. The services included
are:

SakaiPerson Service (profile data)
Type Service
privacy service
archive service
import service

The project site can be viewed at:
http://source.sakaiproject.org/release/common/1.0.0-beta01/
(Note experimental site no Sakai skins etc.)

Edu-Services 1.0.0-beta01
Edu-services contain core shared services that support teaching and
learning functionality in Sakai. It contains:

Course management service
Gradebook service
Sections service

The project site can be viewed at:
http://source.sakaiproject.org/release/edu-services/1.0.0-beta01/

_______________________________________________
announcements mailing list
announcements at collab.sakaiproject.org
http://collab.sakaiproject.org/mailman/listinfo/announcements

TO UNSUBSCRIBE: send email to
announcements-unsubscribe at collab.sakaiproject.org with a subject of
"unsubscribe"

_______________________________________________

management mailing list

management at collab.sakaiproject.org

http://collab.sakaiproject.org/mailman/listinfo/management

TO UNSUBSCRIBE: send email to
management-unsubscribe at collab.sakaiproject.org with a subject of
"unsubscribe"

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/pedagogy/attachments/20091016/223367b2/attachment-0001.html 