I am trying to parse .cds delphi database file. Simple int values and strings are easy to parse. But the only one i cannot understand is a DateTime format.
I found 6 bytes that affecting DateTime Value
I am using python and the following code:
data = '\x00\x00' + '\xBC\xCE\x6F\xEC\xE7\xCC'
data_long = struct.unpack('Q', data)[0]
But struct.unpack doesnt have 6 byte type values, so i added \x00 \x00 to make 8 byte long value ('Q' option)
Here is small sample .cds file with one row https://yadi.sk/d/PkZKy50YgCmqE
DateTimeIssl value = "16.04.2015 9:25:47"
I found 6 hex values but cant unpack it properly.
Can anyone tell me how to read it, or maybe give me a link to some documentation about .cds file structure?
Update:
OK! Thanks to Deltics for guide me how to read TDateTime. I found some test values on internet and i wrote decode function that converts it to Python datetime object.
data = '\x2E\xD8\x82\x2D\xCE\x47\xE3\x40'
data_double = struct.unpack('d', data)[0]
double_split = str(data_double).split('.')
SECONDS_IN_DAY = 60*60*24
time_from_starting_date = timedelta(days=int(double_split[0]), seconds=int(SECONDS_IN_DAY * (float(double_split[1]) * pow(0.1, len(double_split[1])))))
starting_date = datetime(1899, 12, 30)
result_date = starting_date + time_from_starting_date
print time_from_starting_date
print result_date
For 2E D8 82 2D CE 47 E3 40 it will be 08.02.2008 10:38:00.
Works fine.
But i still cannot found valid 8-bytes for field DateTimeIssl in file that i've linked above. Maybe there a different datetime format?
A Delphi date/time (TDateTime) is a Double precision floating point. This is an 8-byte value. You should not need to add any packing or null bytes. If you are having to do this then you have not identified the double value correctly in the file.
Looking at the sample CDS you linked to, each value that could sensibly be interpreted as a date/time (e.g. DateRoshd, DateTimeIssl) is followed by 8 bytes of data.
After reading the double value, the whole number part of this value indicates the date as the number of days since 30 Dec 1899. The decimal part is the time of day on that date.
e.g.
1.0 = 31 Dec 1899, 00:00 (midnight)
2.5 = 1 Jan 1900, 12:00 (midday)
More information on the Delphi TDateTime data type can be found here.
Responding to myself. Maybe for someone it will be useful.
In binary format, TClientDataSet DateTime contains INTEGER value of milliseconds since 02.01.0001, but stores as a 8-byte DOUBLE
So you have to read 8-bytes, unpack it as a double, then convert value to integer. Here is Python code that worked for me:
data = '\x00\xBC\xCE\x6F\xEC\xE7\xCC\x42' # Time: 2015-04-16 09:25:47
data_double = struct.unpack('d', data)[0]
time_from_starting_date = timedelta(days=-2, milliseconds=long(data_double))
starting_date = datetime(0001, 01, 02)
result_date = starting_date + time_from_starting_date
print "Time:", result_date
Related
How can I convert a DataFrame column of strings (in dd/mm/yyyy format) to datetime dtype?
The easiest way is to use to_datetime:
df['col'] = pd.to_datetime(df['col'])
It also offers a dayfirst argument for European times (but beware this isn't strict).
Here it is in action:
In [11]: pd.to_datetime(pd.Series(['05/23/2005']))
Out[11]:
0 2005-05-23 00:00:00
dtype: datetime64[ns]
You can pass a specific format:
In [12]: pd.to_datetime(pd.Series(['05/23/2005']), format="%m/%d/%Y")
Out[12]:
0 2005-05-23
dtype: datetime64[ns]
If your date column is a string of the format '2017-01-01'
you can use pandas astype to convert it to datetime.
df['date'] = df['date'].astype('datetime64[ns]')
or use datetime64[D] if you want Day precision and not nanoseconds
print(type(df_launath['date'].iloc[0]))
yields
<class 'pandas._libs.tslib.Timestamp'>
the same as when you use pandas.to_datetime
You can try it with other formats then '%Y-%m-%d' but at least this works.
You can use the following if you want to specify tricky formats:
df['date_col'] = pd.to_datetime(df['date_col'], format='%d/%m/%Y')
More details on format here:
Python 2 https://docs.python.org/2/library/datetime.html#strftime-strptime-behavior
Python 3 https://docs.python.org/3.7/library/datetime.html#strftime-strptime-behavior
If you have a mixture of formats in your date, don't forget to set infer_datetime_format=True to make life easier.
df['date'] = pd.to_datetime(df['date'], infer_datetime_format=True)
Source: pd.to_datetime
or if you want a customized approach:
def autoconvert_datetime(value):
formats = ['%m/%d/%Y', '%m-%d-%y'] # formats to try
result_format = '%d-%m-%Y' # output format
for dt_format in formats:
try:
dt_obj = datetime.strptime(value, dt_format)
return dt_obj.strftime(result_format)
except Exception as e: # throws exception when format doesn't match
pass
return value # let it be if it doesn't match
df['date'] = df['date'].apply(autoconvert_datetime)
Try this solution:
Change '2022–12–31 00:00:00' to '2022–12–31 00:00:01'
Then run this code: pandas.to_datetime(pandas.Series(['2022–12–31 00:00:01']))
Output: 2022–12–31 00:00:01
Multiple datetime columns
If you want to convert multiple string columns to datetime, then using apply() would be useful.
df[['date1', 'date2']] = df[['date1', 'date2']].apply(pd.to_datetime)
You can pass parameters to to_datetime as kwargs.
df[['start_date', 'end_date']] = df[['start_date', 'end_date']].apply(pd.to_datetime, format="%m/%d/%Y")
Use format= to speed up
If the column contains a time component and you know the format of the datetime/time, then passing the format explicitly would significantly speed up the conversion. There's barely any difference if the column is only date, though. In my project, for a column with 5 millions rows, the difference was huge: ~2.5 min vs 6s.
It turns out explicitly specifying the format is about 25x faster. The following runtime plot shows that there's a huge gap in performance depending on whether you passed format or not.
The code used to produce the plot:
import perfplot
import random
mdYHM = range(1, 13), range(1, 29), range(2000, 2024), range(24), range(60)
perfplot.show(
kernels=[lambda x: pd.to_datetime(x), lambda x: pd.to_datetime(x, format='%m/%d/%Y %H:%M')],
labels=['pd.to_datetime(x)', "pd.to_datetime(x, format='%m/%d/%Y %H:%M')"],
n_range=[2**k for k in range(19)],
setup=lambda n: pd.Series([f"{m}/{d}/{Y} {H}:{M}"
for m,d,Y,H,M in zip(*[random.choices(e, k=n) for e in mdYHM])]),
equality_check=pd.Series.equals,
xlabel='len(df)'
)
I have a numeric variable (year of birth) in SPSS and i would like to take the last for digits out of it. Most values are like 1988, 2001, 1948 etc. But about 250 respondents entered their year of birth like 30-2-1947, or 2-9-1984 etc. That means not all values have the same length. By taking the last 4 digits into a new variable I could create an age category for all the respondents.
How can I do that?
I tried by converting the variable to a string and using substr to get a part of the value, but I always had to choose a starting point. I want to start from the last digit and then move backwards.
Instead of using SUBSTR() you can try using RIGHT() to grab the last four digits.
* convert yob to string variable.
ALTER TYPE yob (A10).
EXE .
* use RIGHT to extract the last 4 digits and convert to numeric.
COMPUTE n_yob = NUMBER(RIGHT(yob, 4, F4)) .
EXE .
You can now use n_yob to calcuate age (ex: COMPUTE age = 2022-n_yob .). You can also use ALTER_TYPE again if you want to convert yob back to it's original type.
I have a 4 byte hexadecimal value that I have a script to print out, But I want to now take that value then subtract the value C8 from it 37 times and save them as different variables each time, But the problem is I don't know how to do hexadecimal calculations in lua, If anyone can link me to any documentation on how to do this then that would be much appreciated.
You can make a hexadecimal literal in Lua by prefixing it with 0x, as stated in the reference manual. I found this by googling "lua hex"; such searches usually get good results.
"Hexadecimal numbers" aren't anything special, hexadecimal is just a way to represent numbers, same as decimal or binary. You can do 1000-0xC8 and you'll get the decimal number 800.
Code to convert:
function convertHex()
local decValue = readInteger(0x123456);
hexValue = decValue
end
function hexSubtract()
for i = 1,37 do
local value = 0xC8
hexValue = hexValue - 0xC8
result = hexValue
if i == 37 then
print(result) --Prints dec value
print(string.format('%X',result)); --Prints hex value
end
end
end
Replace 0x123456 with your address, use those functions like this convertHex(),hexSubtract()
Ok, I really need OLE Automation date in lua.
From here:
public double ToOADate()
Return Value Type: System.Double A double-precision floating-point
number that contains an OLE Automation date equivalent to the value of
this instance.
So in C# this:
Console.Write("DateTime.Now.ToOADate() = " + DateTime.Now.ToOADate());
gives me this:
DateTime.Now.ToOADate() = 42146,4748270602
What is the best way to get simular value in Lua?
Some more details, based on EgorSkriptunoff answer.
So, that Lua code works just fine for me to get OLE Automation date in lua:
-- number of days between December, 30 1899 and January, 1 1970
local magicnumber = 25569
-- don't forget about time zone (UTC+3 for my case)
local utcshift = 3*3600
-- calc and print for test
local oleadate = magicnumber + ((os.time()+utcshift)/(3600*24))
print(oleadate)
Output:
42146.575740741
I'm new to Mongoid. In my model file, I've created a field with data type BigDecimal. I want to store time stamp in it. Below is the model that I'm using:
class Test
include Mongoid::Document
field :time_stamp, type: BigDecimal
end
And Below is the code that I'm using to create a document:
aTime = "Wed Apr 24 09:48:38 +0000 2013"
timest = aTime.to_time.to_i
Test.create({time_stamp: timest})
I see that the time_stamp is stored as String in the database. Can anybody direct me to store the timestamp as number in DB so that I could perform some operations on it. Thanks in advance.
According to this answer, the numeric types supported by MongoDB are:
MongoDB stores data in a binary format called BSON which supports these numeric data types:
int32 - 4 bytes (32-bit signed integer)
int64 - 8 bytes (64-bit signed integer)
double - 8 bytes (64-bit IEEE 754 floating point)
Reinforced by this statement in the Mongoid documentation:
Types that are not supported as dynamic attributes since they cannot be cast are:
BigDecimal
Date
DateTime
Range
I don't know about the things you want to with the field, but if you really want it stored as a number, you have to use a different numeric type that is supported by the MongoDB (BSON), probably Float or Integer.