[samigo-team] Floating point precision

[Ward, Lynn E.]

At IU, we experienced floating point problems similar to those described in SAM-1562 when we first deployed Samigo and decided to round to the second decimal place before sending scores to the GB.  So precision to 4 places would be an improvement over our current approach.

Lynn

==========================
Lynn Ward, Principal Systems Analyst
Instructional Technology Systems and Services
University Information Technology Services<http://uits.iu.edu/>
Indiana University
Information Technology and Communications Complex (IT 342J)
535 West Michigan Street, Indianapolis, IN 46202
Phone: 317-278-5713  E-mail: leward at iupui.edu<mailto:leward at iupui.edu>

From: David Horwitz <david.horwitz at uct.ac.za<mailto:david.horwitz at uct.ac.za>>
Date: Mon, 2 Apr 2012 13:44:32 +0200
To: Samigo Team <samigo-team at collab.sakaiproject.org<mailto:samigo-team at collab.sakaiproject.org>>
Subject: [samigo-team] Floating point precision

Hi All,

Findbugs points to several places where floats are compared using ==,
due to the way floats are represented in Binary these could well fail
as they might be slightly different at a high level of precision.

I also see a similar issues reported by Sam
(https://jira.sakaiproject.org/browse/SAM-1562) to do with a scoring
error that seems to also be related to precision.

The question in looking at a fix seems to be - what order of precision
do we really need? It would seem that 4 decimal places or so would be
about our actual limit.

Thoughts?

D
_______________________________________________ samigo-team mailing list samigo-team at collab.sakaiproject.org<mailto:samigo-team at collab.sakaiproject.org> http://collab.sakaiproject.org/mailman/listinfo/samigo-team
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/samigo-team/attachments/20120402/41501580/attachment-0001.html