HP Forums

I just noticed a strange behaviour of the RMDR and MOD functions. For large arguments they may return a "Domain Error". Some further investigation showed that these functions seem to work for y < 1E820 · x, and as soon as y gets larger the error occurs:


Since the arguments are out of range in single precision mode, all this happens only after switching to double precision.
Any idea what's going on here?

Dieter

The decNumber mod function isn't efficient and requires enough digits to represent the source numbers and the result (i.e. 2*384 + 16). I increase the number of carried digits for these operations to a little above this so that single precision never fails. I didn't go as far as supporting double precision -- there isn't sufficient RAM to do this directly.

Making double precision inputs work across the entire range will require a different algorithm and since we don't guarantee any results in double precision I didn't bother


- Pauli

(10-21-2015 09:39 PM)Paul Dale Wrote: [ -> ]The decNumber mod function isn't efficient and requires enough digits to represent the source numbers and the result (i.e. 2*384 + 16). I increase the number of carried digits for these operations to a little above this so that single precision never fails.

Wait... does this mean that RMDR and MOD internally use 400+ digits?

(10-21-2015 09:39 PM)Paul Dale Wrote: [ -> ]Making double precision inputs work across the entire range will require a different algorithm and since we don't guarantee any results in double precision I didn't bother

OK. Maybe a table with such restrictions would be helpful. I am also thinking of the distributions, Gamma and Beta with extreme cases of their parameters.

Dieter

Mod uses 450 digits internally.
Modulo reduction by two pi for the trig functions uses 820.

So in double precision, neither is adequate to support the entire range.


Pauli