[samigo-team] Floating point precision

[David Horwitz]

We may want different precision between internal comparison (so as not 
to loose data) and output (e.g to the gradebook)

D


On 04/02/2012 02:06 PM, Ward, Lynn E. wrote:
> At IU, we experienced floating point problems similar to those 
> described in SAM-1562 when we first deployed Samigo and decided to 
> round to the second decimal place before sending scores to the GB.  So 
> precision to 4 places would be an improvement over our current approach.
>
> Lynn
>
> ==========================
>
> Lynn Ward, Principal Systems Analyst
>
> Instructional Technology Systems and Services
>
> University Information Technology Services <http://uits.iu.edu/>
> Indiana University
> Information Technology and Communications Complex (IT 342J)
> 535 West Michigan Street, Indianapolis, IN 46202
> Phone: 317-278-5713  E-mail: leward at iupui.edu <mailto:leward at iupui.edu>
>
>
> From: David Horwitz <david.horwitz at uct.ac.za 
> <mailto:david.horwitz at uct.ac.za>>
> Date: Mon, 2 Apr 2012 13:44:32 +0200
> To: Samigo Team <samigo-team at collab.sakaiproject.org 
> <mailto:samigo-team at collab.sakaiproject.org>>
> Subject: [samigo-team] Floating point precision
>
> Hi All,
>
> Findbugs points to several places where floats are compared using ==,
> due to the way floats are represented in Binary these could well fail
> as they might be slightly different at a high level of precision.
>
> I also see a similar issues reported by Sam
> (https://jira.sakaiproject.org/browse/SAM-1562 
> <https://jira.sakaiproject.org/browse/SAM-1562>) to do with a scoring
> error that seems to also be related to precision.
>
> The question in looking at a fix seems to be - what order of precision
> do we really need? It would seem that 4 decimal places or so would be
> about our actual limit.
>
> Thoughts?
>
> D
> _______________________________________________ samigo-team mailing 
> listsamigo-team at collab.sakaiproject.org 
> <mailto:samigo-team at collab.sakaiproject.org>http://collab.sakaiproject.org/mailman/listinfo/samigo-team 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/samigo-team/attachments/20120402/1c9097c0/attachment.html