Stange substraction operation

[rekam]

Hello !

I have a very strange problem. Look at this :

 

Dim a, b, c as double
a = 1
b = 0.96

c = a - b
msgbox(c)

 

Why does the msgbox display 0.0399999999999999999 ?????? The result of this simple operation shouldn't be just 0.04 ?

 

Really, I don't understand. And the worst : when b < 0.95, c is correct :confused: :confused:

 

Well, has anybody already had such a problem ?

Thanks!

[BlueOysterCult]

Double has something to do with a 64 bit floating point

Pi as Double is 3.1415926535

 

I think maybe use Single?

[rekam]

It's strange, but it's the same problem...I think I'll cope that with Math.Round or something...even if it's not, let's say, "professional" :D

[Mehyar]

I tried your example and my msgbox displayed 0.04, although if you debug the debugger will show 0.0400000000036.

 

I agree its weird, but I do not know why your msgbox didnt display 0.04 ....

[PlausiblyDamp]

If you want exact numerics use the Decimal data type. Double and Single are both floating point numbers and as such are prone to rounding errors.

[donnacha]

Yeah, Decimal.Add/Subtract/Multiply combined with Decimal.Todouble should help.

 

e.g.

Dim a,b,c as double

 

b = 1234.555

c = 0234.554

a = Decimal.ToDouble(Decimal.Subtract(b,c))

 

should give a double of 1000.001