I have this number :
b = 1.324567890123456789
and the question was ask how was it stored in lua? Now when I type print(b), it shows that the end digits are
...1235
Now the question gave me the options of
...12345
or
...12346
or the option of none on the list -- and i was just wondering if anyone could help me solve this?
By default, Lua stores real numbers as double precision floating point values.
print calls tostring, which converts doubles to strings using "%.14g". [1]
Use string.format("%.17g",b) if you want more decimals.
[1] https://www.lua.org/source/5.3/luaconf.h.html#LUA_NUMBER_FMT
Related
I'm trying to get a json string but I noticed that when I pass some double numbers to my query the integer part is separated by a coma from the floating part. I need them to be separated by a point. Is this connected to the Language of the os?
You can use a simple regex to fix this issue:
\d+,\d+ to select the offending bits of JSON, and a string-based replace on the results. If you already have the numbers separated out, use the replace function (something like val.replace(",",".")) on the value you have, and cast it to a float (float(val) in Python).
I have problem with comparison of two variables of "Real" type. One is a result of mathematical operation, stored in a dataset, second one is a value of an edit field in a form, converted by StrToFloat and stored to "Real" variable. The problem is this:
As you can see, the program is trying to tell me, that 121,97 is not equal to 121,97... I have read
this topic, and I am not copletely sure, that it is the same problem. If it was, wouldn't be both the numbers stored in the variables as an exactly same closest representable number, which for 121.97 is 121.96999 99999 99998 86313 16227 83839 70260 62011 71875 ?
Now let's say that they are not stored as the same closest representable number. How do I find how exactly are they stored? When I look in the "CPU" debugging window, I am completely lost. I see the adresses, where those values should be, but nothing even similar to some binary, hexadecimal or whatever representation of the actual number... I admit, that advanced debugging is unknown universe to me...
Edit:
those two values really are slightly different.
OK, I don't need to understand everything. Although I am not dealing with money, there will be maximum 3 decimal places, so "currency" is the way out
BTW: The calculation is:
DATA[i].Meta.UnUsedAmount := DATA[i].AMOUNT - ObjQuery.FieldByName('USED').AsFloat;
In this case it is 3695 - 3573.03
For reasons unknown, you cannot view a float value (single/double or real48) as hexadecimal in the watch list.
However, you can still view the hexadecimal representation by viewing it as a memory dump.
Here's how:
Add the variable to the watch list.
Right click on the watch -> Edit Watch...
View it as memory dump
Now you can compare the two values in the debugger.
Never use floats for monetary amounts
You do know of course that you should not use floats to count money.
You'll get into all sorts of trouble with rounding and comparisons will not work the way you want them too.
If you want to work with money use the currency type instead. It does not have these problems, supports 4 decimal places and can be compared using the = operator with no rounding issues.
In your database you use the money or currency datatype.
I have a function that returns a float value like this:
1.31584870815277
I need a function that returns TRUE comparing the value and the two numbers after the dot.
Example:
if 1.31584870815277 = 1.31 then ShowMessage('same');
Sorry for my english.
Can someone help me? Thanks
Your problem specification is a little vague. For instance, you state that you want to compare the values after the decimal point. In which case that would imply that you wish 1.31 to be considered equal to 2.31.
On top of this, you will need to specify how many decimal places to consider. A number like 1.31 is not representable exactly in binary floating point. Depending on the type you use, the closest representable value could be less than or greater than 1.31.
My guess is that what you wish to do is to use round to nearest, to a specific number of decimal places. You can use the SameValue function from the Math unit for this purpose. In your case you would write:
SameValue(x, y, 0.01)
to test for equality up to a tolerance of 0.01.
This may not be precisely what you are looking for, but then it's clear from your question that you don't yet know exactly what you are looking for. If your needs are specifically related to decimal representation of the values then consider using a decimal type rather than a binary type. In Delphi that would be Currency.
If speed isn't the highest priority, you can use string conversion:
if Copy(1.31584870815277.ToString, 1, 4) = '1.31' then ShowMessage('same');
I have a string which I want to convert to a decimal.
eg.
tax_rate = "0.07"
tax_rate.to_d
In most cases it converts to 0.07 but sometimes it converts it to 0.07000000000000001. Why?
If it helps, when I look at the log of when I insert this value into the DB, this is the relevant part:
["tax_rate", #<BigDecimal:7f9b6221d7c0,'0.7000000000 000001E-1',18(45)>]
Hopefully that makes sense to someone.
Disclaimer:
I have a feeling someone is going to ask why I'm doing this.
I've created a simple settings model, where a user can update some global settings.
The Setting model has name, value and var_type columns.
The value column is a string. I use a helper method to retrieve the value in the appropriate format depending on the value of the var_type column.
I cannot explain why but there is a chance I can tell you how to avoid having this kind of trouble when dealing with numbers: use rationals.
Here is the documentation: http://ruby-doc.org/core-1.9.3/Rational.html
As stated, a rational will always give you the exact number you want and thus will avoid rounding errors.
From the doc:
10.times.inject(0){|t,| t + 0.1} #=> 0.9999999999999999
10.times.inject(0){|t,| t + Rational('0.1')} #=> (1/1)
Let me know if this solves your problem. : )
I have stringgrid on delphi form and i am trying to divide values of one cell with value of another cell in another column.
But the problem is, stringgrid cells are populated with different types of numbers, so I am getting ConvertErrors.
For example the numbers in cells can look like
0.37 or 34 or 0.0013 or 0.00 or 0.35 or 30.65 or 45.9108 or 0.0307 or 6854.93.
In another words I never know is it going to be real, float, integer or any other kind of type in those cells.
I have looked everywhere on internet but no luck. Anyone any ideas. By the way I am not exactly Delphi expert. Thanks.
For each string, convert it first to a float value using StrToFloat function in SysUtils.pas . This should allow for any numerical type to be dealt with (unless you have something unusual like complex numbers). As you have some zero values in your list above you should also ensure that you check for divide by zero conditions as this will also potentially throw an exception.
SysUtils has many functions such as TryStrToFloat, TryStrToInt, TryStrToInt64 etc for this purpose. These functions accept a reference parameter (var parameter) for returning the converted value and function itself returns true if the conversion is successful.
If you are sure that the string has a valid number then you can check the input string to see if it has a decimal point before deciding which function to use.
Treat all the numbers as float. Use StrToFloat, divide the numbers, and then convert the result back to string with FloatToStr. If the result is an integer, no decimal point would be produced.