[samigo-team] Floating point precision

[Karen Tsao]

I think we round to second decimal place for displaying purpose. But we
have no objection on changing it to 4.

Thanks,
Karen

On Mon, Apr 2, 2012 at 5:18 AM, David Horwitz <david.horwitz at uct.ac.za>wrote:

> **
> We may want different precision between internal comparison (so as not to
> loose data) and output (e.g to the gradebook)
>
> D
>
>
>
> On 04/02/2012 02:06 PM, Ward, Lynn E. wrote:
>
>  At IU, we experienced floating point problems similar to those described
> in SAM-1562 when we first deployed Samigo and decided to round to the
> second decimal place before sending scores to the GB.  So precision to 4
> places would be an improvement over our current approach.
>
>  Lynn
>
>   ==========================
>
> ****
>
> Lynn Ward, Principal Systems Analyst
>
> Instructional Technology Systems and Services
>
> University Information Technology Services <http://uits.iu.edu/>
> Indiana University
> Information Technology and Communications Complex (IT 342J)
> 535 West Michigan Street, Indianapolis, IN 46202
> Phone: 317-278-5713  E-mail: leward at iupui.edu
>
>   From: David Horwitz <david.horwitz at uct.ac.za>
> Date: Mon, 2 Apr 2012 13:44:32 +0200
> To: Samigo Team <samigo-team at collab.sakaiproject.org>
> Subject: [samigo-team] Floating point precision
>
>  Hi All,
>
> Findbugs points to several places where floats are compared using ==,
> due to the way floats are represented in Binary these could well fail
> as they might be slightly different at a high level of precision.
>
> I also see a similar issues reported by Sam
> (https://jira.sakaiproject.**org/browse/SAM-1562<https://jira.sakaiproject.org/browse/SAM-1562>)
> to do with a scoring
> error that seems to also be related to precision.
>
> The question in looking at a fix seems to be - what order of precision
> do we really need? It would seem that 4 decimal places or so would be
> about our actual limit.
>
> Thoughts?
>
> D
> _______________________________________________ samigo-team mailing list
> samigo-team at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/samigo-team
>
>
> _______________________________________________
> samigo-team mailing list
> samigo-team at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/samigo-team
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/samigo-team/attachments/20120402/f89a8e62/attachment.html