I'm using CsvProvider to parse a CSV file i receive from an external source. The date format of the file is dd/MM/yy. But CsvProvider infers it as MM/dd/yy and get all dates wrong.
Is there any way I can pass the dateformat to CsvProvider?
I can read the dates as string and then convert them using Seq.Map to Datetime (which is two steps).
I'm trying to find out if i can do in a single step
Example
csv file sample.csv is having data like below
11/01/90
12/01/90
13/01/90
14/01/90
15/01/90
I'm using the below F# code
type MyCsvProvider = CsvProvider<Sample= "sample.csv", HasHeaders=false, Schema="Date (date)">
MyCsvProvider.Load("sample.csv")
The CsvProvider type provider does not have a way of explicitly specifying a format for parsing dates, so if you have some completely non-standard format, you'll just have to read it as string (which is what CSV provider infers if it cannot parse dates automatically) and then parse the date values explicitly.
That said, you can specify the Culture parameter, which makes it possible to parse dates in format that is common in countries outside of the US. For example, your date format would work fine with the en-GB culture (in the UK, you first write day, then month and then year).
In the following small example, the Test property is inferred as DateTime:
type A = CsvProvider<"""Test
11/01/90
12/01/90
13/01/90
14/01/90
15/01/90""",Culture="en-GB">
let r = A.GetSample().Rows |> Seq.head
r.Test
Related
I have a G1 table (Gramex) in UI which has columns with date and time but as string.
When the user sorts the respective column with the help of inbuilt sort option, it sorts the data as string not as date.
Approach 1
The easiest way would be if the date and time were formatted like YYYY-MM-DD HH:MM:SS. For example, '2022-09-04 03:02:01'. This is a string that can be sorted consistently with the datetime.
You can create such strings in Python with the datetime.isoformat() function.
Approach 2
If your dates are stored in a different format, e.g. 04-09-2022, etc., then IF the data is loaded from a file, not a database, then you can modify the data with the function: transform to convert the column to the right data format.
For example:
url:
continent:
pattern: /data
handler: FormHandler
kwargs:
url: data.csv
function: data.assign(date_col=pd.to_datetime(data[date_col]).strftime('%Y-%m-%dT%H:%M:%S'))
This converts the date_col (replace this with your date column name) from any date format Pandas recognizes into a YYYY-MM-DD HH:MM:SS type format that is sortable.
If the data is loaded from a database, then you DO need to convert it into a sortable format first.
Hi I want to filter my data which is between two Persian date.
Note: Persian date is a string data type like : "1400/02/23"
when I want to filter my table in SQL Server I simply write like the code bellow:
SELECT * FROM Table
WHERE Date >="1400/01/02" AND Date <="1400/05/10"
but in C# syntax I do not know how to fetch date between two string to use in my filter back-end code. If I simply compare the error raise that string type data can not use comparison operator.
I would be glad and grateful if somebody help me
First, convert Persian Date to Gregorian date and then do the comparison.
I am getting data from rest API in JSON forma and have a scenario where a column can have multiple date format. The current date format could be either 2011-02-12T01:00:00 or 2020-04-15T20:44:57.38or could be null or something else also.
I want to parse it through expression and trying to capture the full date string. The following expression seems to be working fine however it is truncating the millisecond part and returning value upto second only.
iif(isnull(%date_fields%),'\N',
to_char(To_date(to_char(%date_fields%),'MM/DD/YYYY HH24:MI:SS'),'YYYY-MM-DD HH24:MI:SS'))
But when I tried it with millisecond usinf below expression:
iif(isnull(%date_fields%),'\N',
to_char(To_date(to_char(%date_fields%),'MM/DD/YYYY HH24:MI:SS.MS'),'YYYY-MM-DD HH24:MI:SS.MS'))
It is throwing error:
TT_11132 Transformation [Expression3] had an error evaluating output column [JobStartDate_out].
Error message is [<<Expression Error>> [TO_DATE]: invalid string for converting to Date
... t:TO_DATE(u:TO_CHAR(t:<02/12/2011 01:00:00>,u:'MM/DD/YYYY HH24:MI:SS'),u:'MM/DD/YYYY HH24:MI:SS.MS')].
I searched few option using below but getting parsing error.
DECODE (TRUE,
iff(isnull(%date_milli%),
'\N',
is_date(To_date(to_char(%date_milli%),'MM/DD/YYYY HH24:MI:SS'),'YYYY-MM-DD HH24:MI:SS'),
is_date(To_date(to_char(%date_milli%),'MM/DD/YYYY HH24:MI:SS.MS'),'YYYY-MM-DD HH24:MI:SS.MS'),
ERROR('NOT A VALID DATE')))
What could be the possible resolution to handle the multiple date format in Informatica? Here JSON date format is string and I am mapping it to date/time type and using Output Marco Fields to combine multiple similar column together.
Why dont you check both options - with and without milliseconds?
You can use below iif condition. Also i think your expression has some issues.
I assumed date_milli is a character type. If its a date, then you can change below expressions accordingly.
iff(isnull(%date_milli%),null,
iif( is_date(to_char(%date_milli%),'MM/DD/YYYY HH24:MI:SS'), to_date(to_char(%date_milli%),'MM/DD/YYYY HH24:MI:SS'),
iif( is_date(to_char(%date_milli%),'MM/DD/YYYY HH24:MI:SS.MS'), to_date(to_char(%date_milli%),'MM/DD/YYYY HH24:MI:SS.MS')
))
)
I am trying to use blew expression in Google SpreadSheet.
=min(GoogleFinance("HKDUSD","price",date(A2)-1,date(A2)))
But since A2 data is in mm/dd/year format, we need to convert it to yy,mm,dd format as parameter of date().Does Google SpeadSheet has some API to do such convert?
It actually works if you just add the dates as they are without the date function wrapped around them
=min(GoogleFinance("HKDUSD","price",L1-1,L1))
Edited: To answer your follow up question about the date format - a formulaic workaround you do uses regex replace and basically just swaps the two capture groups in month and day with this regex "(\w+)/(\w+)":
=min(GoogleFinance("HKDUSD","price",REGEXREPLACE(L1,"(\w+)/(\w+)","$2/$1")-1,REGEXREPLACE(L1,"(\w+)/(\w+)","$2/$1")))
using Delphi 2010 (Firebird [testing], MS Sql Server, Oracle [production])
The following is my SQL
SELECT p.script_no, MIN(p.start_Time) as startTime, MAX(p.end_Time) as endTime,
SUM(p.duration) as TotalDuration
FROM phase_times p
WHERE (p.script_no=:scriptNo) AND (Trunc(p.start_time) >= :beginDateRange) AND (Trunc(p.start_time) <= :endDateRange)
GROUP BY p.script_no
ParamByName('beginDateRange').AsDate:= Date - 30;
ParamByName('endDateRange').AsDate:= Date;
I am getting a "conversion error from string - 10-25-2012" and i am not sure why, since my datetime fields are in the "10/25/2012 9:20:49 AM" format in the database.
If i change it to the following : ParamByName('beginDateRange').AsString := formatDateTime('mm/dd/yyyy',Date - 30).....i get the error "conversion error from string - 10/25/2012"
reserching this error has provided me no new avenues, do you have any ideas?
According to the Interbase 6.0 manual, Embedded SQL guide, chapter 7, Firebird supports conversion from YYYY-MM-DD and YYYY-MM-DD HH:MM:SS.qqq. I also believe it supports American style shorthand dates (eg 1-JAN-2012) for conversion.
It may be there are some locale dependent conversion supported, but in general: use an actual date/timestamp type instead of a string.
UPDATE
I initially did not spot the TRUNC in your query: it is the cause of the conversion error as this function only works on numbers, see the manual entry for TRUNC.
Given your comment (and the respons of ain) I assume you are only interested in the date part, and want to ignore the time. If so, rewrite your use of TRUNC to:
CAST(<your-timestamp-field> AS DATE)
And the condition (Trunc(p.start_time) >= :beginDateRange) AND (Trunc(p.start_time) <= :endDateRange) to:
CAST(p.start_time AS DATE) BETWEEN :beginDateRange AND :endDateRange
Firebird doesn't support conversion from string to date and time value if string is in 12 hour format. Use 'dd.mm.yyyy hh:mm:ss' or 'mm/dd/yyyy hh:mm:ss' formats.
String to date/time conversions usually use user's locale.
So if you feed a date/time conversion function with string that does not match your date part of the locale - you will get similar errors.
Also specifying date/time values like "10/25/2012" is Locale dependent. So if you execute your program on a computer with different than US Locale (like Mine) - it's likely to fail if using "10/25/2012".
To be Locale independent I suggest two options:
Use StrToDateTime, specifying TFormatSettings
Use ISO 8601 for specifying date/time strings (but I don't think Delphi supports that...)
BTW programs like MS Sql, Excel etc. accept dates in ISO 8601. But you have to check this for FB.
Regarding this:
...since my datetime fields are in the "10/25/2012 9:20:49 AM" format in the database...
The internal storage of date/time fields varies between different DB Engines. What you see in your DB Management Software ("10/25/2012 9:20:49 AM" in your case) is the string representation of the data field, usually formatted (again) according your user Locale
if you connected with DB from Firebird 1.0 under the server Firebird 2.1 (for example) you need todo backup and restore under Firebird 2.1