Undocummented (?) limits for the HP50g

[3298]

(08-02-2021 02:15 PM)ttw Wrote:  There can be some problems with the FLOOR and other functions. I found an ordering that doesn't cause CAS to convert internally. I have a really big fraction nnnnnn/dddddd which I wish to multiply by a big number and take the quotient. I split the formula into Top*Number then result/Bottom then FLOOR. IQUOT and IREMAINDER fail (I think).

None of FLOOR, IQUOT and IREMAINDER accept binary integers, so you have to convert them anyway before calling these, which means we aren't even talking about binary integers anymore. (By the way, comparison and sorting works fine on binary integers over here.) Conversion through strings as I outlined sidesteps the reals entirely, so you don't need to chop up anything. I tested it and had no problems converting \(2^{63}+1\) that way. After that, all you need to worry about is staying with infinite-length integers for all calculations.
Also, your insistence on \(2^{38}\) is misleading: reals are decimal, \(10^{12}\) is the relevant threshold. And a well-documented one at that.

FLOOR is a command for real numbers which predates the CAS, so by calling it you're implicitly converting to real with all the 12-digit floating point implications. If you want to stay exact, IQUOT is exactly the right command for the job. It's a CAS command, and I checked its implementation with Nosy: the dispatch does use the dispatcher that would auto-convert to real, but before the "two real numbers" pattern it has a "two symbolic-class objects" pattern (symbolic class contains actual symbolic objects, but also real numbers). That means it won't get a chance to downgrade CAS integers. I checked with 2 128 ^ 2 64 ^ IQUOT, and the result was equal to 2 64 ^, as expected. (If there was a conversion to reals somewhere, it wouldn't have been accurate because the floating point format is decimal, not binary.)
That said, it's possible to accidentally bypass the symbolic-class case of the dispatch if you supply it with one integer and one real, because reals are not symbolic-class. Check your code, you might have some reals lurking in there if IQUOT fails to produce correct results.