I've hit a problem during parsing of a CSV file where I get the following error:
CSV::MalformedCSVError: Illegal quoting on line 3.
RAILS code in question:
csv = CSV.read(args.local_file_path, col_sep: "\t", headers: true)
Line 3 in the CSV file is:
A-067067 VO VIA CE 0 8 8 SWCH Ter 4, Loc Is Here, Mne, Per Fl Auia/Sey IMAC NEK_HW 2011-03-09 09:47:44 2011-03-09 11:50:26 2011-01-13 10:49:17 2011-02-14 14:02:43 2011-02-14 14:02:44 0 0 771 771 46273 "[O/H 15/02] B270 W31 ""TEXT TEXT 2 X TEXT SWITC" SOME_TEXT SOME_TEXT N/A Name Here RESOLVED_CLOSED RESOLVED_CLOSED
UPDATE: Tabs don't appear to have come across above. See pastebin RAW TEXT: http://pastebin.com/4gj7iUpP
I've read numerous threads all over StackOverflow and Google about why this is and I understand that. But the CSV row above has perfectly legal quoting does it not?
The CSV is tab delimited and there is only a tab followed by the quote on either side of the column in question. There is 1 quote in that field and it is double quoted to escape it. So what gives? I can't work it out. :(
Assuming I've got something wrong here, I'd like the solution to include a way to work around the issue as I don't have control over how the CSV is constructed.
This part of your CSV is at fault:
46273 "[O/H 15/02] B270 W31 ""TEXT TEXT 2 X TEXT SWITC" SOME_TEXT
At least one of these parts has a stray space:
46273 "
" SOME_TEXT
I'd guess that the "3" and the double are supposed to be separated by one or more tabs but there is a space before the quote. Or, there is a space after the quote on the other end when there are only supposed to be tabs between the closing quote and the "S".
CSV escapes double quotes by double them so this:
"[O/H 15/02] B270 W31 ""TEXT TEXT 2 X TEXT SWITC"
is supposed to be a single filed that contains an embedded quote:
[O/H 15/02] B270 W31 "TEXT TEXT 2 X TEXT SWITC
If you have a space before the first quote or after the last quote then, since your fields are tab delimited, you have an unescaped double quote inside a field and that's where your "illegal quoting" error comes from.
Try sending your CSV file through cat -t (which should represent tabs as ^I) to find where the stray space is.
Related
Hello guys I want to convert my non delimited file into a delimited file
Example of the file is as follows.
Name. CIF Address line 1 State Phn Address line 2 Country Billing Address line 3
Alex. 44A. Biston NJ 25478163 4th,floor XY USA 55/2018 kenning
And so on all the data are in this format.
First three lines are metadata and then the data.
How can I make it delimited in proper format using logic.
There are two parts in the problem:
how to find the column widths
how to split each line into fields and output a new line with delimiters
I could not propose an automated solution for the first one, because (not knowing anything about the metadata format), there is no clear way to find where one column ends and the next one begins. Some of the column headings contain multiple space-separated words and space is also used as a separator between the headings (and apparently one cannot use the rule "more than one space means the end of a heading name" because there's only one space between "Address line 2" and "Country" - and they're clearly separate columns. Clearly, finding the correct column widths requires understanding English and this is not something that you can write a program for.
For the second problem, things are much easier - once you have the column positions. If you figure the column positions manually (or programmatically, if you know something about the metadata that I don't - and you have a simple method for finding what's a column heading), then a program written in AWK can do this, for example:
cols="8,15,32,40,53,66,83,105"
awk_prog='BEGIN {
nt=split(cols,tabs,",")
delim=","
ORS=""
}
{ o=1 ;
for (i in tabs) { t=tabs[i] ; f=substr($0,o,t-o); sub(" *$","",f) ; print f
delim ; o=t } ;
print substr($0, o) "\n"
}'
awk -v cols="$cols" "$awk_prog" input_file
NOTE that the above program does not deal correctly with the case when the separator character (e.g. ",") appears inside the data. If you decide to use this as-is, be sure to use a separator that is not present in the input data. It may be better to modify the code to escape any separator characters found in the input data (there are different ways to do this - depends on what you plan to feed the output file to).
I need to be able to print Hebrew characters on my Epson TM-T20ii. I am trying to get my printer to switch to character code page 36(PC862) using
ESC t36
for some reason the printer is switching to code page 3 and then printing the number 6.
Is there a way to let the printer know that the 6 is part of my command?
If you know of a different workaround please comment below.
Thanks
You are making a mistake, you aren't meant to replace n with an actual number.
The proper syntax in your case would be ←t$
Explanation: the manual says "ESC t n", n referring to the page sheet, however you don't replace n with a number rather with the ASCII character n, so in your example 36 = $ because $ is the 36th character on the ASCII table.
I'm attempting to construct a regex string in Swift 4 that gets characters at the start of a line where some are known and others aren't.
Let's say I've got a text file with line breaks for each word that reads as follows:
pucker
tuckered
duckerdinger
sucker punch
I'd like to get every word that contains "cker" in it that's 1 to 8 characters long.
I'm attempting to use this statement ^..cker..{1,8} as my RegEx string. All I'm getting is a partial match in Patterns (a Mac App), but Regex101.com's saying no match, and most importantly, Xcode says I'm using an invalid regex. I've also tried ^(..cker..) and a bazillion other variations.
What am I screwing up and how do I fix it? What I'm trying to do seems like it would be super simple, but I've wasted more time than I care to admit fiddling with it.
Update:
This has been the best I've been able to get so far...
"\\b..cker..", but I'm only able to get words that are exactly 8 characters long. I'd like to capture words that contain "cker" that are the 3rd, 4th, 5th, and 6th letters while capturing words up to 8 characters long.
Try this regex:
\b(?=.*cker)[a-zA-Z]{1,8}\b
Click for Demo
Explanation:
\b - matches a word boundary
(?=.*cker) - Positive Lookahead to make sure our string should contain the character sequence cker
[a-zA-Z]{1,8} - Matches 1 to 8 occurrences of a letter
\b - matches a word boundary
I am having issues with the special CSV interpreter (no idea what its called) on iPad mobile browser.
iPad appears to reserve the character " as reserved or special. When this character appears the string is treated as a literal instead of seperated as a CSV.
INPUT:
1111,64-1111-11,Some Tool 12", 112233
Give the input above, the CSV mobile-safari display shows ([] represents a column)
[1111] [64-1111-11] [Some Tool 12, 112233]
Note that the " is missing. Also note that 112233 is not in its own column like it should be.
Question 2:
How can I get the CSV display tool in safari to not treat a six digit number as a phone number?
1234567
Shows up as a hyperlink and asks to "Add Contact" when I click it. I do not want the hyperlink.
UPDATE
iPad is ignoring the escape character (or backslash is not the escape character) for double quotes in CSV files. I am looking at the hex version of the file and I have
\" or 5C 22 (in hex with UTF-8 encoding).
Unfortuntely, the iPad displays the backslash and still treats " as a special character, thereby corrupting my data formatting. Anybody know how I can possibly use " on iPad CSV?
With regards the quotes, have you tried escaping them in the output?
EDIT: conventional escaping doesn't work for CSV files, my apologies. Most specifications state the following:
Fields that contain a special character (comma, newline, or double quote), must be enclosed in double quotes.
So, testing this on your CSV snippet, a file formatted like this:
1111,64-1111-11,"Some Tool 12""", 112233
or even like this:
1111,64-1111-11,Some Tool 12"""", 112233
… opens in Mobile Safari OK. How good or bad that looks in Excel you'd need to check.
Moving to the second issue, to prevent Mobile Safari from presenting numbers as phone numbers, add this to your page's head element:
<meta name="format-detection" content="telephone=no" />
I've been given a large file with a funny CSV format to parse into a database.
The separator character is a semicolon (;). If one of the fields contains a semicolon it is "escaped" by wrapping it in doublequotes, like this ";".
I have been assured that there will never be two adjacent fields with trailing/ leading doublequotes, so this format should technically be ok.
Now, for parsing it in VBScript I was thinking of
Replacing each instance of ";" with a GUID,
Splitting the line into an array by semicolon,
Running back through the array, replacing the GUIDs with ";"
It seems to be the quickest way. Is there a better way? I guess I could use substrings but this method seems to be acceptable...
Your method sounds fine with the caveat that there's absolutely no possibility that your GUID will occur in the text itself.
On approach I've used for this type of data before is to just split on the semi-colons regardless then, if two adjacent fields end and start with a quote, combine them.
For example:
Pax;is;a;good;guy";" so;says;his;wife.
becomes:
0 Pax
1 is
2 a
3 good
4 guy"
5 " so
6 says
7 his
8 wife.
Then, when you discover that fields 4 and 5 end and start (respectively) with a quote, you combine them by replacing the field 4 closing quote with a semicolon and removing the field 5 opening quote (and joining them of course).
0 Pax
1 is
2 a
3 good
4 guy; so
5 says
6 his
7 wife.
In pseudo-code, given:
input: A string, first character is input[0]; last
character is input[length]. Further, assume one dummy
character, input[length+1]. It can be anything except
; and ". This string is one line of the "CSV" file.
length: positive integer, number of characters in input
Do this:
set start = 0
if input[0] = ';':
you have a blank field in the beginning; do whatever with it
set start = 2
endif
for each c between 1 and length:
next iteration unless string[c] = ';'
if input[c-1] ≠ '"' or input[c+1] ≠ '"': // test for escape sequence ";"
found field consting of half-open range [start,c); do whatever
with it. Note that in the case of empty fields, start≥c, leaving
an empty range
set start = c+1
endif
end foreach
Untested, of course. Debugging code like this is always fun….
The special case of input[0] is to make sure we don't ever look at input[-1]. If you can make input[-1] safe, then you can get rid of that special case. You can also put a dummy character in input[0] and then start your data—and your parsing—from input[1].
One option would be to find instances of the regex:
[^"];[^"]
and then break the string apart with substring:
List<string> ret = new List<string>();
Regex r = new Regex(#"[^""];[^""]");
Match m;
while((m = r.Match(line)).Success)
{
ret.Add(line.Substring(0,m.Index + 1);
line = line.Substring(m.Index + 2);
}
(Sorry about the C#, I don't known VBScript)
Using quotes is normal for .csv files. If you have quotes in the field then you may see opening and closing and the embedded quote all strung together two or three in a row.
If you're using SQL Server you could try using T-SQL to handle everything for you.
SELECT * INTO MyTable FROM OPENDATASOURCE('Microsoft.JET.OLEDB.4.0',
'Data Source=F:\MyDirectory;Extended Properties="text;HDR=No"')...
[MyCsvFile#csv]
That will create and populate "MyTable". Read more on this subject here on SO.
I would recommend using RegEx to break up the strings.
Find every ';' that is not a part of
";" and change it to something else
that does not appear in your fields.
Then go through and replace ";" with ;
Now you have your fields with the correct data.
Most importers can swap out separator characters pretty easily.
This is basically your GUID idea. Just make sure the GUID is unique to your file before you start and you will be fine. I tend to start using 'Z'. After enough 'Z's, you will be unique (sometimes as few as 1-3 will do).
Jacob