# [LON-CAPA-users] FW: Rounding Issue

Peter Riegler p.riegler at ostfalia.de
Sun Sep 22 08:33:11 EDT 2019

```Hi Gene,

most likely the problem is one of number representation (which
contributes to rounding errors).

The binary representation of \$Av+0.05 is (by which I mean "the result of
binary representation of \$Av plus binary respresentation of 0.05 as a
binary")

10.101100110011001100110011001100110011001100110011001

whereas the binary representation of 2.7 is

10.101100110011001100110011001100110011001100110011010

Both binary numbers differ only in the last two digits and the second
one is larger. That explains why 2.7 is not accepted.

I have done the computations on  64-bit machine which likely matches

I hope this is of help.

Peter

Am 22.09.2019 um 02:51 schrieb Harding, Gene L:
>
> Hi everyone,
>
> I have a quiz problem in LC that seems to be doing some odd things
> with the rounding. I have successfully recreated the problem reported
> by one of my students, but cannot figure out why it happens.
>
> The answer to the problem (part c) is 2.65 on the dot. The problem
> requests an answer with one place after the decimal, and the tolerance
> is +/- 0.05. In this case, since the value computes to exactly 2.65,
> LC should take anything from 2.6 to 2.7. In fact, in author view LC
> indicates it will take anything from 2.6 to 2.7:
>
> The trouble is that 2.6 works, 2.65 works, but 2.7 does not.
>
> This is the Perl script calculation (Rf=33, Ri=20):
>
> \$Av         = 1+\$Rf/\$Ri;           # Gain
>
> This is the response code:
>
> <numericalresponse unit="" format="1f" answer="\$Av" id="3">
>
>         <responseparam name="tol" type="tolerance" default="0.05"
> description="Numerical Tolerance" />
>
>
> </numericalresponse>
>
> Any idea what the problem is? I am baffled.
>
> Best regards,
>
> Gene L. Harding, PE
>
> Associate Professor of ECET
>
> Purdue University
>
> 574-520-4190
>
> https://polytechnic.purdue.edu/south-bend/
>
>
> _______________________________________________
> LON-CAPA-users mailing list
> LON-CAPA-users at mail.lon-capa.org
> http://mail.lon-capa.org/mailman/listinfo/lon-capa-users

--
Peter Riegler
Fakultät Informatik
Ostfalia Hochschule für angewandte Wissenschaften
- Hochschule Braunschweig/Wolfenbüttel
Salzdahlumer Str. 46/48
38302 Wolfenbüttel

Tel 05331 939 31540
Fax 05331 939 31542
Web www.ostfalia.de/pws/riegler

-------------- next part --------------
An HTML attachment was scrubbed...