sometimes toString(datetime()) return the milliseconds without the leading zeros to reach the length of 3 (yyyy-MM-dd'T'HH:mm:ss.SSSXXX). Is it a bug or normal behavior?
For example:
2019-11-21T15:59:22.53Z -> it should be 2019-11-21T15:59:22.053Z
2019-11-21T15:59:21.216Z -> OK
2019-11-21T15:30:09.042Z -> OK
This behavior causes an issue when I try to convert the string into a date.
Thank you
Try using the apoc.temporal.format function, specifying the iso_instant type conversion.
For example:
RETURN apoc.temporal.format(datetime("2019-11-21T22:04:19.13Z"), 'iso_instant');
will return:
"2019-11-21T22:04:19.130Z"
[UPDATE]
Since the TOSTRING() function is not documented to return any particular ISO 8601 string format for a datetime, one should not depend on it returning a specific format -- or even returning the same string for the same datetime across versions.
However, if you want a non-APOC approach that works with recent versions of neo4j (like, 3.5.12, on which this was tested), here is an example of one way to modify the current TOSTRING() output string to always have a 3-digit millisecond value:
// Generate a string.
// You can play with the number of digits after ".", and
// even eliminate the "." and any following digits.
WITH TOSTRING(datetime("2019-11-21T22:04:10.1Z")) AS d
// Always return a 3-digit millisecond time in result
WITH d, LENGTH(d) AS lth
RETURN d, CASE WHEN lth < 24
THEN SUBSTRING(d, 0, lth-1) + SUBSTRING('.000Z', lth - 20)
ELSE d END AS result
Related
I am currently working on a plugin for grandMA2 lighting control using Lua. I need the current time. The only way to get the current time is the following function:
gma.show.getvar('TIME')
which always returns the current system time, which I then store in a variable. An example return value is "12h54m47.517s".
How can I separate the hours, minutes and seconds into 3 variables?
If os.date is available (and matches gma.show.getvar('TIME')), this is trivial:
If format starts with '!', then the date is formatted in Coordinated Universal Time. After this optional character, if format is the string "*t", then date returns a table with the following fields: year, month (1–12), day (1–31), hour (0–23), min (0–59), sec (0–61, due to leap seconds), wday (weekday, 1–7, Sunday is 1), yday (day of the year, 1–366), and isdst (daylight saving flag, a boolean). This last field may be absent if the information is not available.
local time = os.date('*t')
local hour, min, sec = time.hour, time.min, time.sec
This does not provide you with a sub-second precision though.
Otherwise, parsing the time string is a typical task for tostring and string.match:
local hour, min, sec = gma.show.getvar('TIME'):match('^(%d+)h(%d+)m(%d*%.?%d*)s$')
-- This is usually not needed as Lua will just coerce strings to numbers
-- as soon as you start doing arithmetic on them;
-- it still is good practice to convert the variables to the proper type though
-- (and starts being relevant when you compare them, use them as table keys or call strict functions that check their argument types on them)
hour, min, sec = tonumber(hour), tonumber(min), tonumber(sec)
Pattern explanation:
^ and $ pattern anchors: Match the full string (and not just part of it), making the match fail if the string does not have the right format.
(%d)+h: Capture hours: One or more digits followed by a literal h
(%d)+m: Capture minutes: One or more digits followed by a literal m
(%d*%.?%d*)s: Capture seconds: Zero or more digits followed by an optional dot followed by again zero or more digits, finally ending with a literal s. I do not know the specifics of the format and whether something like .1s, 1.s or 1s is occasionally emitted, but Lua's tonumber supports all of these so there should be no issue. Note that this is slightly overly permissive: It will also match . (just a dot) and an s without any leading digits. You might want (%d+%.?%d+)s instead to force digits appearing before & after the dot.
Lets do it with string method gsub()
local ts = gma.show.getvar('TIME')
local hours = ts:gsub('h.*', '')
local mins = ts:gsub('.*%f[^h]', ''):gsub('%f[m].*', '')
local secs = ts:gsub('.*%f[^m]', ''):gsub('%f[s].*', '')
To make a Timestring i suggest string method format()
-- secs as float
timestring = ('[%s:%s:%.3f]'):format(hours, mins, secs)
-- secs not as float
timestring = ('[%s:%s:%.f]'):format(hours, mins, secs)
How can I convert a DataFrame column of strings (in dd/mm/yyyy format) to datetime dtype?
The easiest way is to use to_datetime:
df['col'] = pd.to_datetime(df['col'])
It also offers a dayfirst argument for European times (but beware this isn't strict).
Here it is in action:
In [11]: pd.to_datetime(pd.Series(['05/23/2005']))
Out[11]:
0 2005-05-23 00:00:00
dtype: datetime64[ns]
You can pass a specific format:
In [12]: pd.to_datetime(pd.Series(['05/23/2005']), format="%m/%d/%Y")
Out[12]:
0 2005-05-23
dtype: datetime64[ns]
If your date column is a string of the format '2017-01-01'
you can use pandas astype to convert it to datetime.
df['date'] = df['date'].astype('datetime64[ns]')
or use datetime64[D] if you want Day precision and not nanoseconds
print(type(df_launath['date'].iloc[0]))
yields
<class 'pandas._libs.tslib.Timestamp'>
the same as when you use pandas.to_datetime
You can try it with other formats then '%Y-%m-%d' but at least this works.
You can use the following if you want to specify tricky formats:
df['date_col'] = pd.to_datetime(df['date_col'], format='%d/%m/%Y')
More details on format here:
Python 2 https://docs.python.org/2/library/datetime.html#strftime-strptime-behavior
Python 3 https://docs.python.org/3.7/library/datetime.html#strftime-strptime-behavior
If you have a mixture of formats in your date, don't forget to set infer_datetime_format=True to make life easier.
df['date'] = pd.to_datetime(df['date'], infer_datetime_format=True)
Source: pd.to_datetime
or if you want a customized approach:
def autoconvert_datetime(value):
formats = ['%m/%d/%Y', '%m-%d-%y'] # formats to try
result_format = '%d-%m-%Y' # output format
for dt_format in formats:
try:
dt_obj = datetime.strptime(value, dt_format)
return dt_obj.strftime(result_format)
except Exception as e: # throws exception when format doesn't match
pass
return value # let it be if it doesn't match
df['date'] = df['date'].apply(autoconvert_datetime)
Try this solution:
Change '2022–12–31 00:00:00' to '2022–12–31 00:00:01'
Then run this code: pandas.to_datetime(pandas.Series(['2022–12–31 00:00:01']))
Output: 2022–12–31 00:00:01
Multiple datetime columns
If you want to convert multiple string columns to datetime, then using apply() would be useful.
df[['date1', 'date2']] = df[['date1', 'date2']].apply(pd.to_datetime)
You can pass parameters to to_datetime as kwargs.
df[['start_date', 'end_date']] = df[['start_date', 'end_date']].apply(pd.to_datetime, format="%m/%d/%Y")
Use format= to speed up
If the column contains a time component and you know the format of the datetime/time, then passing the format explicitly would significantly speed up the conversion. There's barely any difference if the column is only date, though. In my project, for a column with 5 millions rows, the difference was huge: ~2.5 min vs 6s.
It turns out explicitly specifying the format is about 25x faster. The following runtime plot shows that there's a huge gap in performance depending on whether you passed format or not.
The code used to produce the plot:
import perfplot
import random
mdYHM = range(1, 13), range(1, 29), range(2000, 2024), range(24), range(60)
perfplot.show(
kernels=[lambda x: pd.to_datetime(x), lambda x: pd.to_datetime(x, format='%m/%d/%Y %H:%M')],
labels=['pd.to_datetime(x)', "pd.to_datetime(x, format='%m/%d/%Y %H:%M')"],
n_range=[2**k for k in range(19)],
setup=lambda n: pd.Series([f"{m}/{d}/{Y} {H}:{M}"
for m,d,Y,H,M in zip(*[random.choices(e, k=n) for e in mdYHM])]),
equality_check=pd.Series.equals,
xlabel='len(df)'
)
See the following code:
void main()
{
String test = '2017-10-11T12:03:46.351363-04:00';
DateTime testDate = DateTime.parse(test);
print(testDate.microsecond);
print(testDate.toString());
}
When running this code I lose the microseconds in the string that I parse. Why is this and is there anyway to solve this?
The Dart docs say that the parse method supports microseconds.
'2017-10-11T12:03:46.351363-04:00' is not a format Dart's DateTime can handle.
DateTime only supports Z for GMT or no Z for local time.
Just print the value from a created DateTime to see what format it can parse.
print(DateTime.now());
print(DateTime.now().toUtc())
DateTime has also 2 constructors fromMicrosecondsSinceEpoch and fromMillisecondsSinceEpoch to create an instance from an integer value.
There's an issue here
https://github.com/dart-lang/co19/issues/17
As the test show it only 6 decimal places
https://github.com/dart-lang/co19/commit/8465825f60c9580d82ae01ffc040f3b589aaf667#diff-02c526d1dcb5aa2dcdab3500c14ede87R40
You can parse format 2018-12-11T12:00:32.304272Z
but cannot parse 2018-12-11T12:00:32.304272001Z
I found an issue for dart-lang/sdk.
https://github.com/dart-lang/sdk/issues/44876
The web Date object doesn't support microseconds. It's implemented
using the JavaScript Date object which only supports millisecond
precision. So, working as well as possible.
I am writing a log file parser, and have written some test code to parse this in C.
The string to be parsed looks as follows:
s := `10.0.0.1 Jan 11 2014 10:00:00 hello`
In C, parsing this in place is quite easy. First I find the pointer to the date within the string and then just consume as much as possible using strptime(). This is possible as strptime() will return the position in the string after the call.
Eventually I decided to go with Go instead of C, but while porting the code over I have some issues. As far as I can tell, time.Parse() does not give me any option to parse from within an existing string (though this can be solved with slices) or indication about how much of the original string it have consumed when parsing the date from within the string.
Is there any elegant way in Go I can parse the date/time right out of the string without having to first extract the datetime into an exact slice e.g. by returning the number of characters extracted after parsing?
Unfortunately, the time.Parse method can't tell you how many characters it parsed, so we will need to investigate other elegant solutions. In your example of parsing log statements, the use of regular expressions, as #rob74 suggested, is a reasonably elegant strategy. The example below ignores errors for brevity:
var r = regexp.MustCompile(`^((?:\d{1,3}\.){3}\d{1,3}) ([a-zA-Z]{3} \d{1,2} \d{4} \d{1,2}:\d{2}:\d{2}) (.*)`)
const longForm = "Jan 02 2006 15:04:05"
func parseRegex(s string) (ip, msg string, t time.Time) {
m := r.FindStringSubmatch(s)
t, _ = time.Parse(longForm, m[2])
ip, msg = m[1], m[3]
return ip, msg, t
}
Benchmarks show the above regular expression to be about two times more efficient than #rob74's example on my machine, parsing about a 100,000 lines per second:
BenchmarkParseRegex 100000 17130 ns/op
BenchmarkParseRegexRob74 50000 32788 ns/op
We can, however, keep the solution short and more efficient if we use strings.SplitN instead. For example:
func parseSplit(s string) (ip, msg string, t time.Time) {
parts := strings.SplitN(s, " ", 6)
t, _ = time.Parse(longForm, strings.Join(parts[1:5], " "))
ip, msg = parts[0], parts[5]
return ip, msg, t
}
This splits the string on the first 5 spaces and puts the remaining string (the message part) inside the final parts slice element. This is not very elegant, since we rely on the number of spaces in the date format, but we could count the spaces in the date format string programmatically for a more general solution. Let's see how this compares to our regular expression solution:
BenchmarkParseRegex 100000 17130 ns/op
BenchmarkParseSplit 500000 3557 ns/op
It compares quite favorably, as it turns out. Using SplitN is about five times faster than using regular expressions, and still results in concise and readable code. It does this at the cost of using slightly more memory for the slice allocation.
Maybe you should consider using a regular expression to split the log line, e.g.:
package main
import "fmt"
import "time"
import "regexp"
func main() {
s := "10.0.0.1 Jan 11 2014 10:00:00 hello"
r := regexp.MustCompile("^([^/w]+) ([a-zA-Z]+ [0-9]{1,2} [0-9]{4} [0-9]{1,2}:[0-9]{2}:[0-9]{2}) (.*)")
m := r.FindStringSubmatch(s)
if len(m) >= 4 {
fmt.Println("IP:", m[1])
fmt.Println("Timestamp:", m[2])
fmt.Println("Message:", m[3])
t, err := time.Parse("Jan 02 2006 15:04:05", m[2])
if err != nil {
fmt.Println(err.Error())
} else {
fmt.Println("Parsed Time:",t)
}
} else {
fmt.Println("Regexp mismatch!")
}
}
http://play.golang.org/p/EP-waAPGB4
I'm trying to parse a full ISO8601 datetime from JSON data in Lua.
I'm having trouble with the match pattern.
So far, this is what I have:
-- Example datetime string 2011-10-25T00:29:55.503-04:00
local datetime = "2011-10-25T00:29:55.503-04:00"
local pattern = "(%d+)%-(%d+)%-(%d+)T(%d+):(%d+):(%d+)%.(%d+)"
local xyear, xmonth, xday, xhour, xminute,
xseconds, xmillies, xoffset = datetime:match(pattern)
local convertedTimestamp = os.time({year = xyear, month = xmonth,
day = xday, hour = xhour, min = xminute, sec = xseconds})
I'm stuck at how to deal with the timezone on the pattern because there is no logical or that will handle the - or + or none.
Although I know lua doesn't support the timezone in the os.time function, at least I would know how it needed to be adjusted.
I've considered stripping off everything after the "." (milliseconds and timezone), but then i really wouldn't have a valid datetime. Milliseconds is not all that important and i wouldn't mind losing it, but the timezone changes things.
Note: Somebody may have some much better code for doing this and I'm not married to it, I just need to get something useful out of the datetime string :)
The full ISO 8601 format can't be done with a single pattern match. There is too much variation.
Some examples from the wikipedia page:
There is a "compressed" format that doesn't separate numbers: YYYYMMDD vs YYYY-MM-DD
The day can be omited: YYYY-MM-DD and YYYY-MM are both valid dates
The ordinal date is also valid: YYYY-DDD, where DDD is the day of the year (1-365/6)
When representing the time, the minutes and seconds can be ommited: hh:mm:ss, hh:mm and hh are all valid times
Moreover, time also has a compressed version: hhmmss, hhmm
And on top of that, time accepts fractions, using both the dot or the comma to denote fractions of the lower time element in the time section. 14:30,5, 1430,5, 14:30.5, or 1430.5 all represent 14 hours, 30 seconds and a half.
Finally, the timezone section is optional. When present, it can be either the letter Z, ±hh:mm, ±hh or ±hhmm.
So, there are lots of possible exceptions to take into account, if you are going to parse according to the full spec. In that case, your initial code might look like this:
function parseDateTime(str)
local Y,M,D = parseDate(str)
local h,m,s = parseTime(str)
local oh,om = parseOffset(str)
return os.time({year=Y, month=M, day=D, hour=(h+oh), min=(m+om), sec=s})
end
And then you would have to create parseDate, parseTime and parseOffset. The later should return the time offsets from UTC, while the first two would have to take into account things like compressed formats, time fractions, comma or dot separators, and the like.
parseDate will likely use the "^" character at the beginning of its pattern matches, since the date has to be at the beginning of the string. parseTime's patterns will likely start with "T". And parseOffset's will end with "$", since the time offsets, when they exist, are at the end.
A "full ISO" parseOffset function might look similar to this:
function parseOffset(str)
if str:sub(-1)=="Z" then return 0,0 end -- ends with Z, Zulu time
-- matches ±hh:mm, ±hhmm or ±hh; else returns nils
local sign, oh, om = str:match("([-+])(%d%d):?(%d?%d?)$")
sign, oh, om = sign or "+", oh or "00", om or "00"
return tonumber(sign .. oh), tonumber(sign .. om)
end
By the way, I'm assuming that your computer is working in UTC time. If that's not the case, you will have to include an additional offset on your hours/minutes to account for that.
function parseDateTime(str)
local Y,M,D = parseDate(str)
local h,m,s = parseTime(str)
local oh,om = parseOffset(str)
local loh,lom = getLocalUTCOffset()
return os.time({year=Y, month=M, day=D, hour=(h+oh-loh), min=(m+om-lom), sec=s})
end
To get your local offset you might want to look at http://lua-users.org/wiki/TimeZone .
I hope this helps. Regards!
There is also the luadate package, which supports iso8601. (You probably want the patched version)
Here is a simple parseDate function for ISO dates. Note that I'm using "now" as a fallback. This may or may not work for you. YMMV 😉.
--[[
Parse date given in any of supported forms.
Note! For unrecognised format will return now.
#param str ISO date. Formats:
Y-m-d
Y-m -- this will assume January
Y -- this will assume 1st January
]]
function parseDate(str)
local y, m, d = str:match("(%d%d%d%d)-?(%d?%d?)-?(%d?%d?)$")
-- fallback to now
if y == nil then
return os.time()
end
-- defaults
if m == '' then
m = 1
end
if d == '' then
d = 1
end
-- create time
return os.time{year=y, month=m, day=d, hour=0}
end
--[[
--Tests:
print( os.date( "%Y-%m-%d", parseDate("2019-12-28") ) )
print( os.date( "%Y-%m-%d", parseDate("2019-12") ) )
print( os.date( "%Y-%m-%d", parseDate("2019") ) )
]]