Ruby rounding and formatting of decimal numbers

advertisements

I have looked through: Why can't decimal numbers be represented exactly in binary? and Why Are Floating Point Numbers Inaccurate?

My question is when I round and format floating point numbers using ruby, the results are still different than the result of doing the math buy hand. Below is an example of this:

Edit 2

[28] pry(main)> #####################################
[29] pry(main)> # test one
[30] pry(main)> #####################################
[31] pry(main)> foo = (6.0135 * (650000 / 1000))
             => 3908.7749999999996
[32] pry(main)> foo = '%.2f' % foo.round(2)
             => "3908.77"
[33] pry(main)> # should be 3908.78

[36] pry(main)> #####################################
[37] pry(main)> # test two
[38] pry(main)> #####################################
[39] pry(main)> foo = 650000 / 1000
             => 650
[40] pry(main)> foo = foo * 6.0135
             => 3908.7749999999996
[41] pry(main)> foo = '%.2f' % foo.round(2) # should be 3908.78
             => "3908.77"

[44] pry(main)> #####################################
[45] pry(main)> # test three
[46] pry(main)> #####################################
[47] pry(main)> foo = 650000 / 1000
             => 650
[48] pry(main)> foo = foo * 6.0135
             => 3908.7749999999996
[49] pry(main)> foo = foo.round(2) # should be 3908.78
=> 3908.77

[52] pry(main)> #####################################
[53] pry(main)> # test four - The result of test four is expected
[54] pry(main)> #####################################
[55] pry(main)> foo = 650000 / 1000
             => 650
[56] pry(main)> foo = foo * 6.0135
             => 3908.7749999999996
[57] pry(main)> foo = '%.2f' % foo
             => "3908.77"

[58] pry(main)> #####################################
[59] pry(main)> # test five
[60] pry(main)> #####################################
[61] pry(main)> foo = 650000 / 1000
             => 650
[62] pry(main)> foo = foo * 6.0135
             => 3908.7749999999996
[63] pry(main)> foo = foo.round(5)
             => 3908.775

Test 1: This is the normal formula I use in my library that is giving me the issue.

Test 2: My thought here was that maybe doing both operations in one assignment was causing some problems

Test 3: Well since '%.2f' % has been known to cause some rounding issues where it truncates things, maybe this is the issues.

Test 4: Since '%.2f' % wasn't the problem maybe the .round(2) is causing the problem.

Edit 2: Test 5: If I expand the digits that round is looking at I get a number that I could work with (ie. round(2) that number again). But this seems like a wonky solution that works for a limited case and not in general

I don't really care that the floating point is inaccurate. My question is how can I round this number correctly so that I get the correct answer (the answer you would get if you performed the operation by hand).

Also in general is there a best practice for rounding to correct for dosing point errors? I know it is to use two integers to represent floating point numbers, but that seems like a hassle.

Thank you.

Edit 1

If you use .round(5) instead of .round(2) you would get a more reasonable answer when comparing it to what the answer should be. Is there any drawbacks to increasing the number inside of the round?


round is working as expected. You get the wrong result, because your input is already flawed. The floating point number 6.0135 is actually:

6.01349999999999962341235004714690148830413818359375

Multiplying this number by 650 makes the error worse. You get a result that is closer to 3908.77 than to 3908.78:

foo = 6.0135 * 650
#=> 3908.7749999999996

(foo - 3908.77).abs
#=> 0.004999999999654392

(foo - 3908.78).abs
#=> 0.005000000000563887

To get the correct result, you could use something like this:

foo = (6.0135 * 10000).round  * 0.065
#=> 3908.775

foo.round(2)
#=> 3908.78

Or you could use BigDecimal to avoid floating point errors in the first place:

require 'bigdecimal'
foo = BigDecimal('6.0135') * 650
foo.to_s('F')
#=> "3908.775"

foo.round(2).to_s('F')
#=> "3908.78"