[samigo-team] Floating point precision

[David Horwitz]

Hi All,

Findbugs points to several places where floats are compared using ==,
due to the way floats are represented in Binary these could well fail
as they might be slightly different at a high level of precision.

I also see a similar issues reported by Sam
(https://jira.sakaiproject.org/browse/SAM-1562 
<https://jira.sakaiproject.org/browse/SAM-1562>) to do with a scoring
error that seems to also be related to precision.

The question in looking at a fix seems to be - what order of precision
do we really need? It would seem that 4 decimal places or so would be
about our actual limit.

Thoughts?

D
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/samigo-team/attachments/20120402/4141d982/attachment.html