Some confusion in math result

Some confusion in math result

Postby hua » Thu Jan 07, 2010 3:25 am

Using windows calculator, 2957.45 / 9 = 328.60555555555555555555555555556. In my program I got 328.61 not the expected result of 328.605555...

What am I missing here? Using xHarbour+BCC55

TIA

p.s. Just tested in xbscript.exe by simple means of "? 2957.45/9". Got 328.61 here too
Last edited by hua on Thu Jan 07, 2010 3:44 am, edited 1 time in total.
FWH 11.08/FWH 19.12
BCC5.82/BCC7.3
xHarbour/Harbour
hua
 
Posts: 1070
Joined: Fri Oct 28, 2005 2:27 am

Re: Some confusion in math result

Postby nageswaragunupudi » Thu Jan 07, 2010 3:41 am

Internal value has full accuracy. What is displayed may be rounded off to 2 decimal places.

msginfo( 2957.45 / 9 ) shows 328.605556

We can use the result for further processing without losing accuracy.
Regards

G. N. Rao.
Hyderabad, India
User avatar
nageswaragunupudi
 
Posts: 10482
Joined: Sun Nov 19, 2006 5:22 am
Location: India

Re: Some confusion in math result

Postby hua » Thu Jan 07, 2010 3:53 am

Thanks for the reply Rao.

I think age is catching up to me :oops:

Resolved the issue easily by just temporarily increasing SET DECIMALS value
FWH 11.08/FWH 19.12
BCC5.82/BCC7.3
xHarbour/Harbour
hua
 
Posts: 1070
Joined: Fri Oct 28, 2005 2:27 am


Return to FiveWin for Harbour/xHarbour

Who is online

Users browsing this forum: No registered users and 29 guests