I've spent many hours researching this and am pretty stuck: my question is - has the internal format of a Delphi TDateTime changed between Delphi 7 (released in 2002 or so) and today?
Scenario: I'm reading a binary logfile created by a Delphi 7 app, and the vendor tells me it's a TDateTime in the record, but decoding the bits shows it's clearly not standard IEEE 754 floating point even though the TDateTime produced by modern Delphi is.
But it's some kind of floating point with around 15 bits of exponent and 45 bits of significand (as opposed to 11 and 53 bits in IEE754), and the leading bit is a 1 (which in IEE754 indicates a negative number) for numbers that are clearly not negative, such as the current date/time.
Hints in old documentation suggested that TDateTime "read as" a double but wasn't necessarily represented internally as one, which means that the internal format would be mostly invisible except where these TDateTimes were written out in binary form.
My suspicion is that the change occurred with Delphi 8, which added .NET support, but I simply can't find any references to this anywhere. I have perl code (!) that picks apart these types mostly working, but I'd love to find a formal spec so I can do it properly.
Any old-timers run into this?
~~~ Steve
Nothing has changed since Delphi 7. In Delphi 7, and in fact previous versions, TDateTime is IEEE754, measuring the number of days since the Delphi epoch.
You are going to need to get in touch with the software vendor and try to work out what this data's format really is. It would be surprising if the format was a non-IEEE754 floating point data type. Are you quite sure that it is floating point?
As for BCB3, BCB6 and D4, it's exactly the IEEE 754 Double-precision floating-point format, in the VCL source file system.pas (as included in BCB6) it's defined by thus:
TDateTime = type Double;
Related
Why such code is used in some applications instead of a MOVE?
add 16 to ZERO giving SOME-RESULT
I spotted this in professionally written code at several spots.
Sorce is on this page
Why such code is used in some applications instead of a MOVE?
add 16 to ZERO giving SOME-RESULT
Without seeing more of the code, it appears that it could be a translation of IBM Assembler to COBOL. In particular, the ZAP (Zero and Add Packed) instruction may be literally translated to the above instruction, particularly if SOME-RESULT is COMP-3. Thus, someone checking the translation could see that the ZAP instruction was faithfully translated.
Or, it could be an assembler programmer's idea of a joke.
Having seen the code, I also note the use of
subtract some-data-item from some-data-item
which is used instead of
move zero to some-data-item
This is consistent with operations used with packed decimal fields in IBM Assembly, where there are no other instructions to accomplish "flexible" moves. By flexible, I mean that the packed decimal instructions contain a length field so that specific size MVC instructions need not be used.
This particular style, being unusual, may be related to catching copyright violations.
From my experience, I'm pretty sure I know the reason why the programmer would have done this. It has something to do with the binary representation of the number.
I bet SOME-RESULT is a packed-decimal (or COMP-3) format number. Let's assume the field is defined like this
05 SOME-RESULT PIC S9(5) COMP-3.
This results in a 3-byte field with a hex representation like this
x'00016C'
The decimal number is encoded as a binary encoded decimal (BCD, one decimal digit per half-byte), and the last half-byte holds the sign.
Let's take a look at how the sign is defined:
if it is one of x'C', x'A', x'F', x'E' (café), then the number is positive
if it is one of x'B', x'D', then the number is negative
any of x'0'..'x'9' are not valid signs, so we can distinguish signed packed-decimals from unsigned.
However, a zoned number (PIC 9(5) DISPLAY) - as in the source code - looks like this:
x'F0F0F0F1F6'
As you can see, each decimal digit is an EBCDIC character with the 'zone' part (the first half-byte) always being x'F'.
Now we get closer to your question!
What happens when we use
MOVE 16 TO SOME-RESULT
If you just MOVE a number to such a field, this results in being compiled into a PACK instruction on the machine code level.
PACK SOME-RESULT,=C'16'
A pack instruction takes a zoned number and packs it by picking only the second half-byte of each byte and storing it in the half-bytes of the packed number - with one exception! When it comes to the last byte, it simply flips the two half-bytes and stores them in the last half-byte of the decimal.
This means that the zone of the last byte of the zoned decimal becomes the sign in the packed decimal:
x'00016F'
So now we have an x'F' as the sign – which is a valid positive sign.
However, what happens if use this Cobol instruction instead
ADD 16 TO ZERO GIVING SOME-RESULT
This compiles into multiple machine level instructions
PACK SOME_RESULT,=C'0'
PACK TEMP,=C'16'
AP SOME_RESULT,TEMP
(or similar - the key point is that is needs an AP somewhere)
This makes a slight difference in the result, because the AP (add packed) instruction always sets the resulting sign to either x'C' for a positive or x'D' for a negative result.
So the difference lies in the sign
x'00016C'
Finally, the question is why would one make this difference? After all, both x'F' and x'C' are valid positive signs. So why care?
There is one situation when this slight difference can cause big problems: When the packed decimal is part of an index key, then we would not get a match, even though the numbers are semantically identical!
Because this situation occurred quite often in older databases like VSAM and DL/I (later: IMS/DB), it became good practice to "normalize" packed decimals if they were part of an index key.
However, some programmers adopted the practice without knowing why, so you may come across code that uses this "normalization" even though the data are not used for index keys.
You might also wonder why a compiler does not optimize out the ADD 16 TO ZERO. I'm pretty sure it once did, but that broke a lot of applications, so this specific optimization was removed again or at least made a non-default option with warnings.
Additional useful info
Note that at least the Enterprise Cobol for z/OS compiler allows you to see exactly the machine code that is produced from your source code if use the LIST compile option (see this example output). I recommend to always compile with options LIST, MAP, OFFSET, XREF because these options enable you find the exact problem in your Cobol source even when you only have a program dump from an abend.
Anyway, good programming practice is not to care about the compiler or the machine code, but about the other programmers who will have to maintain, and thus read and understand the code. Good practice would be to always prefer simple and readable instructions, and to document the reasons (right in the code) when deviating from this rule.
Some programmers like to do things "just because they can". I have a feeling that is what you are seeing here. It makes about as much sense as doing
a := 0 + b
would in go.
I'm getting a rounding oddity in Delphi 2010, where some numbers are rounding down in roundto, but up in formatfloat.
I'm entirely aware of binary representation of decimal numbers sometimes giving misleading results, but in that case I would expect formatfloat and roundto to give the same result.
I've also seen advice that this is the sort of thing "Currency" should be used for, but as you can see below, Currency and Double give the same results.
program testrounding;
{$APPTYPE CONSOLE}
{$R *.res}
uses
System.SysUtils,Math;
var d:Double;
c:Currency;
begin
d:=534.50;
c:=534.50;
writeln('Format: ' +formatfloat('0',d));
writeln('Roundto: '+formatfloat('0',roundto(d,0)));
writeln('C Format: ' +formatfloat('0',c));
writeln('C Roundto: '+formatfloat('0',roundto(c,0)));
readln;
end.
The results are as follows:
Format: 535
Roundto: 534
C Format: 535
C Roundto: 534
I've looked at Why is the result of RoundTo(87.285, -2) => 87.28 and the suggested remedies do not seem to apply.
First of all, we can remove Currency from the question, because the two functions that you use don't have Currency overloads. The value is converted to an IEEE754 floating point value and then follows the same path as your Double code.
Let's look at RoundTo first of all. It is quick to check, using the debugger, or an additional Writeln that RoundTo(d,0) = 534. Why is that?
Well, the documentation for RoundTo says:
Rounds a floating-point value to a specified digit or power of ten using "Banker's rounding".
Indeed in the implementation of RoundTo we see that the rounding mode is temporarily switched to TRoundingMode.rmNearest before being restored to its original value. The rounding mode only applies when the value is exactly half way between two integers. Which is precisely the case we have here.
So Banker's rounding applies. Which means that when the value is exactly half way between two integers, the rounding algorithm chooses the adjacent even integer.
So it makes sense that RoundTo(534.5,0) = 534, and equally you can check that RoundTo(535.5,0) = 536.
Understanding FormatFloat is quite a different matter. Quite frankly its behaviour is somewhat opaque. It performs an ad hoc rounding in code that differs for different platforms. For instance it is assembler on 32 bit Windows, but Pascal on 64 bit Windows. The overall approach appears to be to take the mantissa of the floating point value, convert it to an integer, convert that to text digits, and then perform the rounding based on those text digits. No respect is paid to the current rounding mode when the rounding is performed, and the algorithm appears to implement the round half away from zero policy. However, even that is not implemented robustly for all possible floating point values. It works correctly for your value, but for values with more digits in the mantissa the algorithm breaks down.
In fact it is fairly well known that the Delphi RTL routines for converting between floating point values and text are fundamentally broken by design. There are no routines in the Delphi RTL that can correctly convert from text to float, or from float to text. In fact, I have recently implemented my own conversion routines, that do this correctly, based on existing open source code used by other language runtimes. One of these days I will get around to publishing this code for use by others.
I'm not sure what your exact needs are, but if you are wishing to exert some control over rounding, then you can do so if you take charge of the rounding. Whilst RoundTo always uses Banker's rounding, you can instead use Round which uses the current rounding mode. This will allow you to perform the round using the rounding algorithm of your choice (by calling SetRoundMode), and then you can convert the rounded value to text. That's the key. Keep the value in an arithmetic type, perform the rounding, and only convert to text at the very last moment, after the correct rounding has been applied.
In this case, the value 534.5 is exactly representable in Double precision.
Looking into source code, reveals that the FormatFloat function rounds upwards if the last pending digit is 5 or more.
RoundTo uses the Banker's rounding, and rounds to nearest even number (534) in this case.
I am currently trying to work around with Lua 5.1 bytecode. I've gotten pretty far, and understand a lot. However, I am stuck with a question on instructions and numbers. I understand that the size of the instruction and number are located and defined in the header, but I am not sure how to get the actual number from the 4 bytes (or whatever size is specified in the header).
I've looked at output from ChunkSpy and I don't really understand how it went from those bytes to the number. I'd look in the source but I don't want to just copy it, I want to understand it. If anyone could tell me a bit about it or even point me in the right direction I'd be very grateful.
Thank you!
From A No-Frills Introduction to Lua 5.1 VM Instructions, numbers are stored in the constants pool.
The first byte is 3=LUA_TNUMBER.
The next bytes are the number, with the length as given in the header. Interpretation is based on the length, byte order and the integral flag as given in the header.
Typically, non-integral with 8 bytes means IEEE 754 64-bit double.
Deserializing bytes to double involves extracting the bits for the mantissa and exponent, and combining them with arithmetic operations. Perhaps you want that as a challenge and to start from a description of the standard: What Every Computer Scientist Should Know About Floating-Point Arithmetic, "Formats and Operations" section.
This question already has answers here:
Closed 11 years ago.
Possible Duplicate:
Large numbers in Pascal (Delphi)
I am trying to convert a 66bit value to decimal.
I note that the largest data type in delphi in int64 which can only allow 64bit data.
example of delphi code for such conversion is
result := strtoInt64('FFFFABCDEFF123456');
Please advice how to use delphi to this without returning out of range error.
Muda
Decimal in Delphi is called currency it uses 8 bytes = 64 bits.
You'll have to create your own type, see this article: http://www.delphi3000.com/articles/article_3772.asp
It describes how to create a 128bit integer.
Here's a bignum lib for Delphi: http://cc.embarcadero.com/item/27789
See also this question: Large numbers in Pascal (Delphi)
If you can afford having some rounding errors, you can use the Extended type. That's an easy solution. I remember reading somewhere that it won't be supported in 64bit Delphi anymore though, so I personally wouldn't use it if it can be avoided.
Anyway, do you really want to do calculations on your number? Are you sure you don't just want to have an array (of bytes for example)? If that is the case, you should look at HexToBin().
Documentation: http://docwiki.embarcadero.com/VCL/en/Classes.HexToBin
Example where you can see it in use: http://docwiki.embarcadero.com/CodeExamples/en/HexEncoding_(Delphi)
I'm dealing with timestamps in Lua showing the number of microseconds since the Epoch (e.g. "1247687475123456").
I would really like to be able to print that number in all its terrible glory, but Lua insists on printing it in scientific notation. I've scoured the available documentation about printing a formatted string, but the only available commands are "Print in scientific notation (%e/%E)" and "Automatically print in scientific notation if the number is very long (%g)". No options seem to be available to print the number in its normal form.
I realize that I could write a function that will take the original number, do some dividing by 10 and print the digits in a loop but that seems like an inelegant hassle. Surely there's some way of doing this that's built in to the language?
> print(string.format("%18.0f",1247687475123456))
1247687475123456
Lua as usually configured uses your platform's usual double-precision floating point format to store all numbers. For most desktop platforms today, that will be the 64-bit IEEE-754 format. The conventional wisdom is that integers in the range -1E15 to +1E15 can be safely assumed to be represented exactly.
In any case, the string.format() function passes its arguments through (with some minor tweaks) to the platform's implementation of printf(). The format string understood by printf() includes %e and %E to force "scientific" notation, and %f to force plain decimal notation. In addition, %g and %G choose the shortest notation.
For example:
E:\...>lua
Lua 5.1.4 Copyright (C) 1994-2008 Lua.org, PUC-Rio
> a = 1e17/3
> print(string.format("%f",a))
33333333333333332.000000
> print(string.format("%e",a))
3.333333e+016
> print(string.format("%.0f",a))
33333333333333332
Note that if the value fits within a 32-bit signed integer range, you can also use the %d format. However, results are not well defined if the value exceeds that range. System timestamps in microseconds are likely to exceed the 32-bit range.
If 16 decimal digits is not enough precision, there are several choices available for increased precision.
First, it would not be difficult to package a true 64-bit integer in a userdata along with a suitable set of arithmetic metamethods. This gets discussed occasionally on the Lua mailing list, but I don't recall seeing a completed module released by anyone.
Second, one of the Lua authors has released two modules supporting arbitrary precision arithmetic: lbc and lmapm. Both are found at that page.
Third, casual searching in Google readily turns up several other math library wrappers.