[Portfolio] Rubric forms in OSP - weighted averages

Sean Keesler sean.keesler at threecanoes.com
Wed Jan 13 20:48:28 PST 2010


I hear ya, Noah....

One of the very special uses of form in OSP is to create evaluation forms.
To some of us that means a score and a comment.

Some of us wish for something more sophisticated.

A rubric is an attempt at ensuring that all evaluators evaluate the
same. This might mean a bunch of things:

 - all of the evaluators are required to evaluate the same criteria
 - all evaluators know what level of performance warrants a score of a
"4" for a particular criteria
 - all of the evaluators weight the various criteria the same way when
they come up with a final rating as the result of a applying a rubric

My hope was that I could incorporate the last piece (weighting) into a
schema. It currently isn't supported.

Many of you probably already know that when you fill out a form you
end up with a piece of XML in your workspace that includes the schema
for the form and the data that your types into the form.
It's sort of self documenting....

For example, I can discover (by looking at the schema of the form)
what the descriptor was used for awarding a "3" when an evaluation was
performed.
If a schema provided clues about the relative weight of one criteria,
that ALSO could be stored each time a evaluation was performed.

A custom renderer could go some distance to enforcing uniform behavior
in awarding weighted rating scores. It would end up being very
implementation specific, but (as Noah indicates) valuable with a bit
of overhead in maintenance.

I'm commenting my stuff quite heavily.
I'll contribute it to the community lib and attempt to write something
about my use of "unusual" documentation elements as best I can.

This will be a "0.1" release of the AAC&U VALUE rubrics. I hope that
they are useful to the community.


Sean Keesler
130 Academy Street
Manlius, New York 13104 USA
315-663-7756
sean.keesler at threecanoes.com



On Wed, Jan 13, 2010 at 7:59 PM, Noah Botimer <botimer at umich.edu> wrote:
> For the sake of lazy documentation, I want to mention a conversation I had
> with Sean about this.
>
> Michigan has taken advantage of precisely this mechanism. We have indicated
> a couple of things about special fields in the schema, then used custom XSL
> to treat them differently both in creation and viewing.
>
> Though the annotation is straightforward and this constitutes a powerful
> technique, it is somewhat obtuse in the XSL. The base templates weren't
> really designed with extending them in mind, so some seemingly simple things
> turn out to be a lot of complex or duplicated code.
>
> An example of this might be multiple choice items, which have some
> intelligence about when to use checkboxes, radio buttons, or select tags.
> Any adjustment here require pretty intimate familiarity with what's going on
> and copying or recreating a bigger piece of functionality than you might
> expect.
>
> None of this is to discourage innovative uses but I do want to mention that
> it is likely to introduce ongoing maintenance if used widely. Because there
> aren't clear places to plug in system-wide extensions, things like multiple
> sites using the functionality can be a bit of a chore to track. You might
> consider these somewhere close to the realm of local source modifications
> when evaluating investment and payoff.
>
> Specific to Sean's scenario, I suggested that he add the annotation but be
> clear in the code that this is custom behavior. Because it's a bit of an
> exercise getting to know what schema pieces result in what user interface,
> some clarity that forms won't get a rubric feature without the corresponding
> XSL is in order.
>
> If folks are doing interesting things with this technique, it would be nice
> to hear about them. I can imagine a sort of cookbook in Confluence
> somewhere.
>
> Maybe Sean could even write up the first recipe.
>
> Thanks,
> -Noah
>
> On Jan 11, 2010, at 4:24 PM, Sean Keesler <sean.keesler at threecanoes.com>
> wrote:
>
>> I'm working on a custom renderer for a bunch of rubric forms for the
>> community library.
>> One of the ideas that Janice and I were discussing was to have the
>> form automatically calculate and store a final "score" based upon a
>> weighted average of the ratings for each criteria in the rubric... ie:
>> (weight1 * value1) + (weight2 * value2)  +....+ (weightn * valuen) =
>> final rating
>>
>> I'd really like to express the "weight" of each "field" in the
>> schema...that way a record of the weight could be stored along with
>> the data when the form is filled out.
>> To do something like that, I could create a new "ospi" or "sakai"
>> documentation piece....something like:
>>
>> <xs:documentation source="ospi.rubricWeight">.2</xs:documentation>
>>
>> My custom renderer could read these weights and make use of them, but
>> other renderers wouldn't use them at all (and someone would have to
>> actually manually figure out the final score....
>> Has anyone else built "extensions" like that? Can you think of any
>> good reason NOT to do something like that?
>>
>>
>> Sean Keesler
>> 130 Academy Street
>> Manlius, New York 13104 USA
>> 315-663-7756
>> sean.keesler at threecanoes.com
>> _______________________________________________
>> portfolio mailing list
>> portfolio at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/portfolio
>>
>> TO UNSUBSCRIBE: send email to
>> portfolio-unsubscribe at collab.sakaiproject.org with a subject of
>> "unsubscribe"
>>
>>
>


More information about the portfolio mailing list