03-29-2016, 05:20 PM (This post was last modified: 03-29-2016 07:31 PM by Visitor.)
Post: #246
 Visitor Junior Member Posts: 3 Joined: Mar 2015
Hello Claudio,

in dectranscen.c many constants (e.g. decconst_PI() ) are given back with twice the precision as set
due to shift right of only 2
Code:
 int nwords=253-((Context.precdigits+7)>>2);

is that intended?

currently a side effect will be that with the default settings (32 digits) if I enter pi0 i get back pi with almost 70 digits (instead of 32)

shouldn't the constant being rounded before pushing it to the display?

if I multiply this longer number by 1 it will be rounded to the system precision.

BTW I'm admiring your work and the time you are putting into this project. I just came recently across this project and am very excited of it.

regards

Martin
 « Next Oldest | Next Newest »