Rounding errors and number crunching

M

mjsnow

Newcomer
Joined
Sep 27, 2003
Messages
13
Hi All,

I am working on conversions between different geographical coordinate systems that require a very high degree of precision of at least 8 decimal places. Of course, at that level, rounding errors are a major factor. For example, 1.00000000 * 3600 converts to 0.00027778 and then back to 1.00000000. Of course, the conversion back to 1 never really equals one due to rounding errors – it equals 1.000008. There are some algorithms that deal with this problem such as the Kahn Summation Algorithm but I was wondering if .net (VB in particular) has any methods to deal with this problem directly? Maybe there is some technique that I am not aware of such as setting some parameter for the double or decimal type or something?

I tried using the decimal type which gives me the same answer as double which includes the 0.000008 and played around with the decimal.round function to no avail. Any suggestions?
 
have you considered adding some extra 0's on the end? i don't have my software near but that might get it to round at a different point in the number.
when i get upstairs i'll look for some stuff

brandon
 
Hi bpayne111,

Yep, I tried to round at 8, 12, and 24 (largest for the decimal type).

I am going to start working on the summation algorithm today and see what happens.
 
Hi folks,
isin't this annoying, you would think the system would be smart enough to avoid this. I had the same problem in doing additions. I wanted to check againt 100% and all the individual bits added up to 100 but the result was 100.000000000000001 so my comparison did not work.

I have found a solution which I find irritating to have to do, but you do what you need to do to get the thing to work even if it is daft.

The solution uses Decimal as a middle man to get around the rounding (pardon the pun). e.g.

dim res, a,b,c as double

a = 75.25
b = 24.74999
c = 00.00001

res = Decimal.ToDouble(Decimal.Add(a,Decimal.Add(b,c)))

The result should be 100.0

You should be able to solve yier problem with the Decimal.Subtract or Decimal.Multiply etc...

Lets hope next year is less .NET bug ridden.....
 
Its not that .net is bug ridden as it is the way computers do math. The problem is that in pure math, there is always a number between two other numbers making numbers, of course, infinite. In a computer, numbers are limited to a finite set of numbers. This means that they will always be rounding problems. In fact, some people have made whole careers developing algorithms to do accurate math. If life would only be so simple that we could just copy mathematical formulas from a textbook into a computer program or add up huge numbers with lots of decimal places.

Mike
 
Indeed, the same principle that creates the 100.0000001 is the same principle behind 1/3 = 0.333333333

mjsnow: You might want to consider making your own numerical datatype and handling all operations that you may want to do on it. You can implement it as an array of longs/decimals, maybe? :)
 
That is some good advice. In fact, what I did was use decimals and strings. .net has some really nice parsing and string manipulations that worked. I just did the math in decimal, converted it to a string, hacked off the end and converted it back to a decimal. However, I think these types of solutions are just a work around and not to efficient. I need to think it out a bit more when I have a little more time. Maybe, as you suggested, a custom number class or wrapper class of some type would be handy to have in the tool box.

Mike
 
Be carefull decimal.tostring(xx) does not always give the answer you would expect, you may need to add decimal.tostring(xx,NumberFormatInfo) to get the correct result.
 
Back
Top