[DG: Teaching & Learning] [Portfolio] [Management] A manifesto for Grading and Rating in Sakai
Kenneth Robert Romeo
kenro at stanford.edu
Fri Oct 23 10:38:16 PDT 2009
Hello everyone,
I have mentioned it before, but … I think that while tracking is important
to the process of instruction, we should not lose sight of the fact that we
need to be able to show that an LMS in general and Sakai specifically is “a
difference that makes a difference.” We had a program evaluation a couple
years ago and I really needed to show how students and instructors were
spending more time on Sakai (adoption), and how instructors were coming up
with more complex and more effective activities (innovation), but we just
didn’t have that data. It would be good if we could show differences in
time on task for programs where curriculum is centrally managed vs.
instructor-produced. And on and on. Teachers would like to know this, but
we need to be able to show it to *decision-makers* to justify the investment
of time and money.
This kind of logging needs to be central to the design of Sakai3, not an
extra module that causes performance issues.
Ken
From: pedagogy-bounces at collab.sakaiproject.org
[mailto:pedagogy-bounces at collab.sakaiproject.org] On Behalf Of Josh Baron
Sent: Thursday, October 22, 2009 4:41 PM
To: David Goodrum
Cc: management at collab.sakaiproject.org; portfolio at collab.sakaiproject.org;
portfolio-bounces at collab.sakaiproject.org;
pedagogy at collab.sakaiproject.orgLearning
Subject: Re: [DG: Teaching & Learning] [Portfolio] [Management] A manifesto
for Grading and Rating in Sakai
David,
I've not had time to reply to the larger "manifesto" but wanted to comment
briefly on your idea of tracking.
I think it would be a very valuable component in the larger assessment
process. Taking this a step further, there are some key metrics that
research has started to show are indicators of students becoming at risk for
failing a course, this is particularly true in online courses. For example,
it is not uncommon for a student in an online course to complete the first
week or so of work but then begin to "fad" out and disappear from active
participation in the course. It would very powerful if Sakai could be
monitoring these metrics (e.g. number of visits, postings, etc.) and alert
instructors or others that the student may need some outreach. This might
be another use of the type of tracking you're discussing.
I would put forth one inherent risk that I've seen in using this type of
data. Just looking at these numbers does not always reflect the reality of
what is happening in the learning process. For example, we have had
situations in which a faculty member has noted that one of his online
students only signs in to his site twice per week for 10 minutes each
time...the instructor concluded that the student could not possibly be
reading all of the weekly materials and that he must be cheating as he was
doing well. As it turned out, the student preferred to print out all of the
materials each week and read them offline. My point is that if we give
instructors tools that allow them to automatically incorporate tracking data
into their grading process there may be unintended implications that need to
be considered.
Josh
-----------------------------
Joshua Baron
Director, Academic Technology and eLearning
Marist College
Poughkeepsie, New York 12601
(845) 575-3623 (work)
Twitter: JoshBaron
From:
David Goodrum <davidgoodrum at rocketmail.com>
To:
Sean Keesler <sean.keesler at threecanoes.com>, Noah Botimer
<botimer at umich.edu>
Cc:
management at collab.sakaiproject.org, portfolio at collab.sakaiproject.org,
"pedagogy at collab.sakaiproject.orgLearning"
<pedagogy at collab.sakaiproject.org>
Date:
10/22/2009 06:27 PM
Subject:
Re: [Portfolio] [Management] A manifesto for Grading and Rating in
Sakai
Sent by:
portfolio-bounces at collab.sakaiproject.org
_____
Hi everyone (sorry for the repost to the management list),
I really like what John has done here in pulling together several themes
around feedback, such as rating and grading; and I appreciate the issues
that Mark, Noah, Ken and Sean have raised in other replies.
With this narrative approach John has done a nice job of greatly extending
and weaving together more telegraphic comments made elsewhere by myself and
others (see <http://tinyurl.com/yzzh2kc> http://tinyurl.com/yzzh2kc --
recently put up in a Google doc by Clay) about how any Sakai 3 object (or
placeholder for an external object) or collections of such things (e.g., a
portfolio) could receive rich-text comment, a grade, or structured (e.g.,
rubric) or un-structured feedback; feedback could come from the instructor,
or from peers, from oneself (e.g., reflection), or from a reviewer outside
of the class or even outside of the institution; and in a particular
learning space/environment various kinds of data can get congregated (unlike
the current Gradebook tool that is either all points or all percentages) so
that, for example, attendance might be a checkmark, a collection of quizzes
might be points, papers with a letter grade, a project or portfolio with a
rubric, and so on.
Briefly, I’d like to add another related theme of tracking. At a prior time
in response to some of the limitations of the single Gradebook approach with
a single grade type, some of us at IU conceptualized a Tracking feature/tool
which would have the general purpose of providing instructors an
easy-to-manage method of tracking low-stakes, but high-in-quantity
activities for such items as class participation, session attendance, or
daily micro-assessments (where the instructor is basically checking off that
something was accomplished). The summary of an item could be linked
to/recorded in the Gradebook. Extending the notion, one might want to
extract a site stat and turn that into a tracked item (e.g., the number of
posts by an individual per week.)
I’m not sure that explains it well, but am merely trying to suggest that
tracking might be another feedback related activity in addition to rating
and grading.
Regards - David
_____
From: Sean Keesler <sean.keesler at threecanoes.com>
To: Noah Botimer <botimer at umich.edu>
Cc: management at collab.sakaiproject.org; portfolio at collab.sakaiproject.org;
"pedagogy at collab.sakaiproject.org Learning"
<pedagogy at collab.sakaiproject.org>
Sent: Tue, October 20, 2009 10:01:14 AM
Subject: Re: [Portfolio] [Management] A manifesto for Grading and Rating in
Sakai
One of the things that are crucial to making meaning out of the
assessment process (and the grades/ratings that are the record of that
process) are sets of rubrics that document HOW rating/grading should
be done or has been done.
How you do or DO NOT manage and/or mandate the application of rubrics
to the assessment of student work is a local decision that may vary
within and amongst faculty members, departments to entire colleges
with the university, but the capability to author, share, modify and
find rubrics suitable for a any one application would seem to me to be
a missing piece of John's manifesto and one that would make the idea
of assessment a core piece of Sakai 3.
It has a lot of impact on the deployment of ePortfolios where multiple
faculty (perhaps from different departments or colleges) could be
asked to blindly assess a collection of student work through their
lens of specialization. Providing guidance for these faculty to HOW to
grade a portfolio gives the entire process more validity.
Rubrics are also a vehicle for a university to articulate how it
differentiates it's standards for excellence from other colleges or
for showing that program X complies with Association Y's expectations.
A while ago I jotted down some different ways that rubrics might be
managed in an LMS. I believe that issues like the Spellings
Commission Report and the No Child Left Behind fiasco (K12) and so
they may be receiving more attention here than elsewhere. It may be
interesting to see what patterns exist in the community around the
application and use of rubrics.
1. Managed assessments:
Some rubrics are rather specific to (and must be tied to) a particular
assessment item and must be approved by an "assessment coordinator"
for educational QA purposes as part of a larger assessment system
strategy. Changing the assessment/rubric in this case involves more
than just the teacher.
2. Generally reusable (but unchangeable) rubrics
Some rubrics may be general purpose rubrics that are NOT tied to an
assessment, but the dissemination of these approved rubrics may be a
strategy of an institution to push forward an agenda of best practice
for assessment by providing a handy reference library of general
purpose writing, mathematics and science rubrics (for example). While
the choice whether or not to use one of these "off the shelf" rubrics
(and which one) is left to the teacher, providing some information to
the teacher about the schools expectations of its students at
different stages (and perhaps suggesting an appropriate rubric for
this grade level/stage of development) would make this service more
valuable.
3. Reusable rubric templates:
Similar to the above, but the library of "off the shelf" rubrics are
merely starting points. There is not a priority to ensure that
everyone is doing assessment the exact same way. When a teacher uses
one of these rubrics, they can easily edit the performance indicators
to suit their needs and create a new rubric, just for their new
assignment. (Rubristar approach)
4. Sharing of rubrics:
This is a bottom up approach to establishing "best practice". As the
teachers create their own rubrics against goals, they have the
opportunity to publish them as part of the "reusable" library so other
teachers can use/edit/republish them. (Someone?)
Sean Keesler
130 Academy Street
Manlius, New York 13104 USA
315-663-7756
<mailto:sean.keesler at threecanoes.com> sean.keesler at threecanoes.com
On Fri, Oct 16, 2009 at 11:07 AM, Noah Botimer < <mailto:botimer at umich.edu>
botimer at umich.edu> wrote:
> Hello John,
> Thank you for the rather comprehensive narrative. I believe that these are
> important for the archives as we change our ideas and software over time.
> They leave a better historical record of our state of mind at any given
> point than a pile of JIRA tickets. Our successive approximation is better
> validated when we have a record of these richer "data" points.
> Now, more on task...
> This is a fair account from my perspective, and is especially important in
> that it carves out a first-class place for two things that have been
> historical weaknesses:
> 1. The ability to treat various artifacts individually and in
> collections,
> consistently, across types of "stuff" and activity (e.g., reflection vs.
> feedback vs. grading)
> 2. The ability to retrieve meaningful performance (or other) data in
> detail
> and aggregate, consistently, and without extensive one-off programming
>
> Interestingly enough, these two areas are what I've spent four years
> working
> on -- so I suppose it's not surprising that I call them out. I mention
> them
> as weaknesses from my experience. It has been difficult to combine
> assignment information with student-crafted presentation. It has been
> difficult to combine course-based (assignment, quiz, etc.) data and
> program-based activity (annual review, capstone, student teaching
> performance) and map them to curricular goals and reports...
> Please do not take my comments as complaints of where we are. What is more
> important is that I see this narrative as recognizing these activities
> not,
> as we have, as things that can be bolted on post-construction but, rather,
> as shaping the core provisions of a meaningful academic and collaborative
> platform. We are, as a community, much more aware of our successes and
> shortfalls. This, I feel, is very healthy and inspiring.
> I believe this discussion is going in the right direction and sincerely
> hope
> that we can find the energy to support it.
> Thanks,
> -Noah
> On Oct 16, 2009, at 6:02 AM, John Norman wrote:
>
> I have collected my thoughts around grading and rating in Sakai. I offer
> them now partly because I feel ready, partly because there are open
> questions about Gradebook in Sakai 3 and partly because we have just had a
> discussion in which I suggest it is hard to break things out of a coherent
> Sakai 3 project. If accepted as is, this represents a logical area of
> activity than can readily be envisioned as a standalone activity - maybe
> even a separate product.
> First of all I'd like to suggest that grading is a subset of a general
> rating and feedback activity. Many artifacts can be rated, from instructor
> performance during a course (course evaluation), through quality of a
> teaching asset or exercise (rating) to assessing the quality of a student
> portfolio (feedback) and assessing the performance of a student on an
> assignment or test (grading). The common pattern is: an artifact is
> produced
> by one individual (or group) and some value judgement is recorded by one
> or
> more other people.
> The process by which an artifact is judged can be simple or complex.
> Complex
> processes include multi-stage workflows where raw scores are obtained by
> one
> process and raw scores moderated to a final grade by another process. I
> see
> plagiarism detection as one particular wrinkle in such a workflow.
> I suggest that (nearly) everything in Sakai should be ratable/gradable. I
> will refer to the ratable/gradable elements as "artifacts" to indicate
> that
> they may not be 'technical elements' but some aggregation of technical
> elements that makes sense for rating/grading purposes. Moreover, we should
> not forget that some of the artifacts that are rated/graded may not be
> electronic and the 'artifact' may be a proxy for some real world activity
> or
> output that cannot be captured electronically.
> The activity of rating/grading is essentially a human judgement. Tests and
> quizzes represent a subset of this situation where the human codifies
> their
> judgement into rules applied by the testing engine and the test engine
> automates the application of scores. The Quiz with the student answers
> represents the artifact and the raw scores and/or processed grade
> represents
> the judgement. The people involved in rating/grading can be anyone:
> students, teachers, peers.
> The artifact to be rated or graded may not be stable over time, in which
> case a 'snapshot' of some kind is desirable for audit purposes. An example
> might be the state of my personal portfolio pages on the first day of May,
> when they are declared to be assessed. I may wish to continue maintaining
> the pages after the assessment, but their status at the time of assessing
> is
> worth recording. A different example might be my performance in a piece of
> drama. I have no idea how this would be recorded in the real world, but I
> imagine that the grader might write down some critique/commentary and then
> assign a grade. The critique/commentary would become the recorded artifact
> (in some places there might be a video recording but I don't assume that)
> and separately there would be a grade/score/rating. Teacher performance in
> class evaluated by students is not far from this model. The questions in
> the
> evaluation form might be considered the rubric for the teachers
> performance.
> In this world, we would want a flexible reporting platform that allows
> grade
> information (including an archive of artifact snapshots) to be collected
> and
> analysed (and sometimes further processed). I suggest we think of using
> something like BIRT to create this flexible reporting environment and then
> consider certain predefined views of the data and derived reports from the
> data as the essence of "GradeBook" functionality. i.e. "GradeBook" is a
> subset of functionality from a powerful reporting environment. Ultimately
> "the official record" will need to be updated.
> I think it is really important to anticipate that some of the artifacts to
> be graded may come from outside Sakai and Sakai needs to be able to accept
> artifacts for grading and also to accept graded artifacts for inclusion in
> reporting. I see two main implementation options for Sakai
> 1. A Sakai service with published external entry points (Moodle/Mahara
> integration would be an example)
> 2. A new Sakai 'product' which would be an institutional grading/rating
> service that receives artifacts from a number of places (including the
> Sakai
> Course Management System) and manages the grading/rating workflow into a
> flexible reporting system that creates a complete record for an individual
> and allows this information to be displayed in a number of places
> (including
> Sakai CMS)
> A strong attraction of the second model is that it fits with the idea that
> assessing performance is a core competence of the institution that
> preceded
> and will survive the CMS, but which is unlikely to be developed for us by
> the commercial world. It could also represent a shared service with a
> student information system.
> Having set out my manifesto, it is interesting to consider what the
> product
> council might do with it. From my personal perspective it would be great
> if
> we adopted it as the Sakai manifesto (following review/revision) and
> called
> for developments to align with it, but there is an open question regarding
> the value of 'adoption' of the manifesto if nobody is interested in
> developing products/code that address the manifesto.
> John
> PS I have forwarded this message that I saw as I came in this morning
> because in my mind it illustrates an early step in the direction of my
> manifesto, although I have taken it much further (perhaps unrecognisably).
> Begin forwarded message:
>
> From: David Horwitz < <mailto:david.horwitz at uct.ac.za>
> david.horwitz at uct.ac.za>
> Date: 16 October 2009 09:29:58 BST
> To: sakai-dev < <mailto:sakai-dev at collab.sakaiproject.org>
> sakai-dev at collab.sakaiproject.org>,
> <mailto:production at collab.sakaiproject.org>
> production at collab.sakaiproject.org,
> <mailto:announcements at collab.sakaiproject.org>
> announcements at collab.sakaiproject.org
> Subject: [Announcements] 2.7 Framework: commons and edu-servise
> 1.0.0-beta01
> released
> Hi All,
>
> We're proud to announce the first of 2 framework releases in support of
> the
> upcoming 2.7 release. The creation of these bundles aims to rationalize
> our
> dependency tree and enable a more modular approach to Sakai releases.
>
> Commons 1.0.0-beta01
> The commons package contains common services depended on by a number of
> Sakai tools, but outside the scope of the Kernel. The services included
> are:
>
> SakaiPerson Service (profile data)
> Type Service
> privacy service
> archive service
> import service
>
> The project site can be viewed at:
> <http://source.sakaiproject.org/release/common/1.0.0-beta01/>
> http://source.sakaiproject.org/release/common/1.0.0-beta01/
> (Note experimental site no Sakai skins etc.)
>
> Edu-Services 1.0.0-beta01
> Edu-services contain core shared services that support teaching and
> learning
> functionality in Sakai. It contains:
>
> Course management service
> Gradebook service
> Sections service
>
> The project site can be viewed at:
> <http://source.sakaiproject.org/release/edu-services/1.0.0-beta01/>
> http://source.sakaiproject.org/release/edu-services/1.0.0-beta01/
>
>
> _______________________________________________
> announcements mailing list
> <mailto:announcements at collab.sakaiproject.org>
> announcements at collab.sakaiproject.org
> <http://collab.sakaiproject.org/mailman/listinfo/announcements>
> http://collab.sakaiproject.org/mailman/listinfo/announcements
>
> TO UNSUBSCRIBE: send email to
> <mailto:announcements-unsubscribe at collab.sakaiproject.org>
> announcements-unsubscribe at collab.sakaiproject.org with a subject of
> "unsubscribe"
>
> _______________________________________________
> management mailing list
> <mailto:management at collab.sakaiproject.org>
> management at collab.sakaiproject.org
> <http://collab.sakaiproject.org/mailman/listinfo/management>
> http://collab.sakaiproject.org/mailman/listinfo/management
> TO UNSUBSCRIBE: send email to
> <mailto:management-unsubscribe at collab.sakaiproject.org>
> management-unsubscribe at collab.sakaiproject.org
> with a subject of "unsubscribe"
>
> _______________________________________________
> portfolio mailing list
> <mailto:portfolio at collab.sakaiproject.org>
> portfolio at collab.sakaiproject.org
> <http://collab.sakaiproject.org/mailman/listinfo/portfolio>
> http://collab.sakaiproject.org/mailman/listinfo/portfolio
>
> TO UNSUBSCRIBE: send email to
> <mailto:portfolio-unsubscribe at collab.sakaiproject.org>
> portfolio-unsubscribe at collab.sakaiproject.org
> with a subject of "unsubscribe"
>
_______________________________________________
portfolio mailing list
<mailto:portfolio at collab.sakaiproject.org>
portfolio at collab.sakaiproject.org
<http://collab.sakaiproject.org/mailman/listinfo/portfolio>
http://collab.sakaiproject.org/mailman/listinfo/portfolio
TO UNSUBSCRIBE: send email to
<mailto:portfolio-unsubscribe at collab.sakaiproject.org>
portfolio-unsubscribe at collab.sakaiproject.org with a subject of
"unsubscribe"
_______________________________________________
portfolio mailing list
portfolio at collab.sakaiproject.org
<http://collab.sakaiproject.org/mailman/listinfo/portfolio>
http://collab.sakaiproject.org/mailman/listinfo/portfolio
TO UNSUBSCRIBE: send email to portfolio-unsubscribe at collab.sakaiproject.org
with a subject of "unsubscribe"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/pedagogy/attachments/20091023/38c7a02b/attachment-0001.html
More information about the pedagogy
mailing list