While trying to round a fraction to 9 digits, I came across the following: ? (format t "~9$" 763700091/112000148) 6.818742300 Nil But the rounding is only accurate to 5 digits. I expected format would take care of this automatically; ie, convert

I have a string string(5) "388.0". I simply want to get exactly float number with one zero or two zero or whatever numbers comes after decimal. How can I achieve it? Simply (float)$val gives me only float(388) without zero.Have you tried the bui

In java, is there a difference between negating a variable vs multiplying a variable by a float minus one, if the variable is also a float?In JLS §15.15.4 "Unary Minus Operator -", we find that For floating-point values, negation is not the same

I am using the following python code to read numbers out of the csv file: with open('file.csv') as p: rows = csv.reader(p) for row in rows: numbers = float(row[3]) I understand that although the csv file contains numbers, they are read as strings. In

I have an array of floats. Some of these floats are VERY close to integers, but they're still floats. I need to filter these values. For example: array_n = [n]; //an array of floats k = 160. In the if loop, n is an element in array_n. I need to pick

When I run my program, which just calculates a sine wave: for(i = 0; i < ADS1299_SIGNAL_WINDOW; i++){ TEST[i] = (float32_t)(10.0f * (float32_t)(arm_sin_f32((float32_t)(3.14f * i/ADS1299_SIGNAL_WINDOW)))); } The compiler generates the following line,

For the simple and efficient implementation of fast math functions with reasonable accuracy, polynomial minimax approximations are often the method of choice. Minimax approximations are typically generated with a variant of the Remez algorithm. Vario

In Python, should I use negative exponents or division? Example: >>> num = 2.0**-1 or >>> num = 1/2.0 I ran a little test, and it looks like the difference comes down to BINARY_POWER vs. BINARY_DIVIDE: import dis def exp_test(x): return

I'm working on a user defined function in MySQL that takes an integer (mediumint) and divides it by 100 (i.e. 10785 / 100 = 107.85). The supplied integer is a parameter called time. I've tried DECLARE dtime FLOAT; SET dtime = time / 100; DECLARE ftim

When calling Double.IsNaN() with Double.PositiveInfinity as argument, the result is false. This is against my intuition since infinity is not a number. Apparently "NaN" only exists in terms of a constant in .NET, is this described by the IEEE st

I hate having to ask this because I assume the answer must be simple, but I cannot for the life of me seem to track down the source. While trying to rewrite a function I ran across this problem: a = -j x = real(a) y = imag(a) y/x Which spits out Inf,

This question already has an answer here: Is floating point math broken? 22 answers I find the result very strange. Why not 0.3? Can somebody tell me why this result? Is it possible to fix this. ?- X is 5.3-5. X = 0.2999999999999998. ?- My second que

I know that floating point variables loose precision while casting. But what I don't understand is, why a cast from a smaller primitive to a bigger one is unprecise but vice versa not. I would understand if it happens from double to float but it's th

I have an expression that is used to estimate percentiles by interpolating between two values. windowMin + (currentPercentile - lastPercentile) * (windowMax - windowMin) / (percentile - lastPercentile) This has given me very good real-world results.

I am a little confused about rounding to nearest in floating point arithmetic. Let a, b, and c be normalized double precision floating point numbers. Is it true that a+b=b+a where + is correctly rounded to nearest floating point addition? My initial

According to this page on the GCC Wiki: http://gcc.gnu.org/wiki/Software_floating_point the "soft-fp" soft float implementation gcc provides has optional support for exceptions and rounding modes. Is there any way to configure GCC (libgcc) when

Does MATLAB support float16 operations? If so, how to convert a double matrix to float16? I am doing an arithmetic operation on a large matrix where 16-bit floating representation is sufficient for my representation. Representing by a double datatype

Possible Duplicate: Mysterious calculation error when multiply by 100 Here is a strange problem that i am facing. I am i store a value .6 to a float variable. When i multiply it with 100 I am getting wrong answer as 60.000004.Where does this .000004

I still have troubles with numbers in Vim: p.e. let a = 1.02 | let b = '10000000' | let total = a*b | echo total --> 1.02e7 I would like to avoid Exponentials. How can I have the output 10200000 ? let a = 4000000000 | let b = '1' | let total = a+b |

I am trying to calculate an aspect ratio for an image: Log.d("img.width:", String.valueOf(img.getIntrinsicWidth())); Log.d("img.height:", String.valueOf(img.getIntrinsicHeight())); float aspect = ((float)img.getIntrinsicWidth()) / ((fl

Some people say that machine epsilon for double precision floating point numbers is 2^-53 and other (more commonly) say its 2^-52. I have messed around estimating machine precision using integers besides 1 and aproaching from above and below (in matl

I'm making a game in C++ where is a tank that can moves on a stage. The tank has an angle (float, in degrees, and i'm supposing the tank is at 0º when his cannon points to the right), a speed (float), and there is a time constant called "deltaT"

Is it true that the more the floating point number is big (either positive or negative) the less we have bits to encode the decimal digits? Can we encode more decimal digits between 21 and 22 than between 216 and 232? Is there the same count of value

Possible Duplicate: Floating Point Limitations Hi there, maybe this has a very simple answer. I was playing around with the Python interpreter. >>> 1 1 >>> 1.1 1.1000000000000001 >>> 1.2 1.2 Why 1.1 was interpreted as 1.10000000

How do you write an NaN floating-point literal in C?In C99's <math.h> 7.12 Mathematics <math.h> [#5] The macro NAN is defined if and only if the implementation supports quiet NaNs for the float type. It expands to a constant expression of type

If have the following C function, used to determine if one number is a multiple of another to an arbirary tolerance #include <math.h> #define TOLERANCE 0.0001 int IsMultipleOf(double x,double mod) { return(fabs(fmod(x, mod)) < TOLERANCE); } It wo

I am coding up an implementation of Interpolation Search in C. The question is actually rather simple, I need to use the floating operations to do linear interpolation to find the correct index which will eventually be an integer result. In particula

I need to format float value and I only need 2 numbers after point and should be rounded value float first = 7, Second = 3,result; result = first / Second; // result contain 2.33333325 since I need like 2.33 Thanks ?5/3 1.6666666666666667 ?String.For

I have a program written in C# and some parts are writing in native C/C++. I use doubles to calculate some values and sometimes the result is wrong because of too small precision. After some investigation i figured out that someone is setting the flo

Im currently reading Code Complete by Steve McConnell, specifically page 295 on floating-point numbers. When i ran the following code: double nominal = 1.0; double sum = 0.0; for (int i = 0; i < 10; i++) { sum += 0.1; Console.WriteLine("sum: "