For some reason the code below breaks in psql as supplied with Greenplum at the \copy stage:
\set tmp1 public.tmp1
DROP TABLE IF EXISTS :tmp1;
CREATE TABLE :tmp1 (new_id varchar(255), old_id BIGINT) DISTRIBUTED BY (old_id);
\echo :tmp1
\copy :tmp1 FROM 'file1.csv' WITH DELIMITER '|' HEADER CSV;
ERROR: syntax error at or near ":"
LINE 1: COPY :tmp1 FROM STDIN WITH DELIMITER '|' HEADER CSV;
How can you use a variable table name with the copy command in psql?
I don't think this has to do with greenplum or an old PostgreSQL version. The \copy meta command does not expand variables as documented here:
Unlike most other meta-commands, the entire remainder of the line is always taken to be the arguments of \copy, and neither variable interpolation nor backquote expansion are performed in the arguments.
To workaround you can build, store and execute the command in multiple steps. Replace your \copy ... line with
\set cmd '\\copy ' :tmp1 ' FROM ''file1.csv'' WITH DELIMITER ''|'' HEADER CSV'
:cmd
With this, you need to double (escape) the \ and ' within the embedded meta command. Keep in mind that \set concatenates all further arguments to the second one, so you need to quote spaces between the arguments.
Related
I have to replicate a do file of a colleague who used a macro for his file names. The problem is that the pathname contains a parenthesis, which causes problems:
*setting directory
cd "D:/Dropbox (Center for Child Well-being and Development)/2020/Playground"
*setup
sysuse auto
save "/Dropbox (Center for Child Well-being and Development)/example", replace
*problem
global path "/Dropbox (Center for Child Well-being and Development)"
local file "/example.dta"
global data "$path`file'""
disp "$data"
use $data
I get the following output
. disp "$data"
/Dropbox (Center for Child Well-being and Development)/example.dta
. use $data
invalid '('
r(198);
I know that calling the macro within quotations as use "$data" would do the job, but as it is not my do file I would like to try to avoid changing every occurrence where the macro is used.
I tried to escape the parenthesis with \( and add various numbers of quotations at any position I could imagine while constructing the global. Also I tried to add escaped quotations \" which did work neither.
From this, MySQL load data infile command works well with hexadecimal delimiter like X'01' or X'1e' in my case. But the same command can't be run with same command load data infile on MemSQL.
I tried specifying various forms of of the same delimiter \x1e like:
'0x1e' or 0x1e
X'1e'
'\x1e' or 'x1e'
All the above don't work and throw either syntax error or other error like this:
This is like the delimiter can't be resolved correctly:
mysql> load data local infile '/container/data/sf10/region.tbl.hex' into table REGION CHARACTER SET utf8 fields terminated by '\x1e' lines terminated by '\n';
ERROR 1261 (01000): Row 1 doesn't contain data for all columns
This is syntax error:
mysql> load data local infile '/container/data/sf10/region.tbl.hex' into table REGION CHARACTER SET utf8 fields terminated by 0x1e lines terminated by '\n';
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '0x1e lines terminated by '\n'' at line 1
mysql>
The data is actually delimited by non-printable hexadecimal character of \x1e and line terminated by regular \n. Use cat -A can see the delimited characters as ^^. So the delimiter should be correct.
$ cat -A region.tbl.hex
0^^AFRICA^^lar deposits. blithely final packages cajole. regular waters are final requests. regular accounts are according to $
1^^AMERICA^^hs use ironic, even requests. s$
Are there a correct way to use hex values as delimiter? I can't find such information in documentation.
For the purpose of comparison, hex delimiter (0x1e) can work well on MySQL:
mysql> load data local infile '/tmp/region.tbl.hex' into table region CHARACTER SET utf8 fields terminated by 0x1e lines terminated by '\n';
Query OK, 5 rows affected (0.01 sec)
Records: 5 Deleted: 0 Skipped: 0 Warnings: 0
MemSQL supported hex delimiters as of 6.7, of the form in the last code block in your question. Prior to that, you would need the literal quoted 0x1e character in your sql string, which is annoying to do from a CLI. If youre on an older version you may need to upgrade.
I have a problem trying to execute shell scripts from apple script. I do a "grep", but as soon as it contains special characters it doesn't work as intended.
(The script reads a list list ob subfolders in a directory and checks if any of the subfolders appear in a file.)
Here is my script:
set searchFile to "/tmp/output.txt"
set theCommand to "/usr/local/bin/pdftotext -enc UTF-8 some.pdf" & space & searchFile
do shell script theCommand
tell application "Finder"
set companies to get name of folders of folder ("/path/" as POSIX file)
end tell
repeat with company in companies
set theCommand to "grep -c " & quoted form of company & space & quoted form of searchFile
try
do shell script theCommand
set CompanyName to company as string
return CompanyName
on error
end try
end repeat
return false
The problem is e.g. with strings with umlauts. "theCommand" is somehow differently encoded that when I do it on the CLI directly.
$ grep -c 'Württemberg' '/tmp/output.txt' --> typed on command line
3
$ grep -c 'Württemberg' '/tmp/output.txt' --> copy & pasted from AppleScript
0
$ grep -c 'rttemberg' '/tmp/output.txt' --> no umlauts, no problems
3
The "ü" from the first and the second line are different; a echo 'Württemberg' | openssl base64 shows this.
I tried several encoding tricks at different places, basically everything I could find or think of.
Does anyone have any idea? How can I check which encoding a string has?
Thanks in advance!
Sebastian
Overview
This can work by escaping each character that has an accent in each company name before they are used in the grep command.
So, you'll need to escape each one of those characters (i.e. those which have an accent) with double backslashes (i.e. \\). For example:
The ü in Württemberg will need to become \\ü
The ö in Königsberg will need to become \\ö
The ß in Einbahnstraße will need to become \\ß
Why is this necessary:
These accented characters, such as a u with diaeresis, are certainly getting encoded differently. Which type of encoding they receive is difficult to ascertain. My assumption is that the encoding pattern used begins with a backslash - hence why escaping those characters with backslashes fixes the issue. Consider the u with diaeresis in the previous link, it shows that for the C/C++ language the ü is encoded as \u00FC.
Solution
In the complete script below you'll notice the following:
set accentedChars to {"ü", "ö", "ß", "á", "ė"} has been added to hold a list of all characters that will need to be escaped. You'll need to explicitly state each one as there doesn't seem to be a way to infer whether the character has an accent.
Before assigning the grepcommand to theCommand variable we firstly escape the necessary characters via the line reading:
set company to escapeChars(company, accentedChars)
As you can see here we are passing two arguments to the escapeChars sub-routine, (i.e. the non-escaped company variable and the list of accented characters).
In the escapeChars sub-routine we iterate over each char in the accentedChars list and invoke the findAndReplace sub-routine. This will escape any instances of those characters with backslashes found in the company variable.
Complete script:
set searchFile to "/tmp/output.txt"
set accentedChars to {"ü", "ö", "ß", "á", "ė"}
set theCommand to "/usr/local/bin/pdftotext -enc UTF-8 some.pdf" & ¬
space & searchFile
do shell script theCommand
tell application "Finder"
set companies to get name of folders of folder ("/path/" as POSIX file)
end tell
repeat with company in companies
set company to escapeChars(company, accentedChars)
set theCommand to "grep -c " & quoted form of company & ¬
space & quoted form of searchFile
try
do shell script theCommand
set CompanyName to company as string
return CompanyName
on error
end try
end repeat
return false
(**
* Checks each character of a given word. If any characters of the word
* match a character in the given list of characters they will be escapd.
*
* #param {text} searchWord - The word to check the characters of.
* #param {text} charactersList - List of characters to be escaped.
* #returns {text} The new text with the item(s) replaced.
*)
on escapeChars(searchWord, charactersList)
repeat with char in charactersList
set searchWord to findAndReplace(char, ("\\" & char), searchWord)
end repeat
return searchWord
end escapeChars
(**
* Replaces all occurances of findString with replaceString
*
* #param {text} findString - The text string to find.
* #param {text} replaceString - The replacement text string.
* #param {text} searchInString - Text string to search.
* #returns {text} The new text with the item(s) replaced.
*)
on findAndReplace(findString, replaceString, searchInString)
set oldTIDs to text item delimiters of AppleScript
set text item delimiters of AppleScript to findString
set searchInString to text items of searchInString
set text item delimiters of AppleScript to replaceString
set searchInString to "" & searchInString
set text item delimiters of AppleScript to oldTIDs
return searchInString
end findAndReplace
Note about current counts:
Currently your grep pattern only reports the number of lines that the word was found on. Not how many instances of the word were found.
If you want the actual number of instances of the word then use the -o option with grep to output each occurrence. Then pipe that to wc with the -l option to count the number of lines. For example:
grep -o 'Württemberg' /tmp/output.txt | wc -l
and in your AppleScript that would be:
set theCommand to "grep -o " & quoted form of company & space & ¬
quoted form of searchFile & "| wc -l"
Tip: If your want to remove the leading spaces in the count/number that gets logged then pipe it to sed to strip the spaces: For example via your script:
set theCommand to "grep -o " & quoted form of company & space & ¬
quoted form of searchFile & "| wc -l | sed -e 's/ //g'"
and the equivalent via the command line:
grep -o 'Württemberg' /tmp/output.txt | wc -l | sed -e 's/ //g'
I have a script that is a command line parser , it calls other scripts. One script that it calls is count.s. count.s takes 1 arg and counts words lines chars, it will accept a *.txt and works fine. In the command line parser argv[1] is *.c When I call it from my command line parser , and pass into count.s argv[1], it expands *.c and count.s only reads the first file it enounters. I think I need to take argv[1] , which is a *.whatever and take it as a literal.
How can I do this? I think I need to turn off the filename expansion in the command line parser and pass into count.s the actual *.whatever , instead of the expansion that occurs within the command line parser.
I can post my code if it helps , but I dont think it is really necessary
Basically, you can't.
The argument *.c is expanded by the shell before your script ever sees it.
Suppose the current directory contains foo.c and bar.c. If you type
myscript.csh *.c
then myscript.c will see bar.c and foo.c as two separate arguments -- exactly as if you had typed
myscript.csh bar.c foo.c
at the command line.
If you want your script to see the string *.c, then you have to pass that as an argument, which you can do by quoting the argument:
myscript.csh '*.c'
But your script has no control over how and whether wildcards are expanded.
What are you actually trying to accomplish?
I'm trying to pass a byte array from inside my rails app into another ruby script (still inside my rails app), for example:
`./app/animations/fade.sh "\x01\x01\x04\x00" &`
Yields ArgumentError (string contains null byte)
I suppose I'm stumped with how I can form this string and than pass it to my script, which will use it in this sort of fashion:
#sp.write ["#{ARGV[0]}", "f", "\x12"]
I'd like to form the string (on my rails app) like this if possible:
led = "\x01#{led.id}\x04\x00"
But I keep getting ArgumentError (string contains null byte) error. Is there a way I can form this string from elements in my rails app, then pass it to my external script?
You should just pass the data in through standard input, not the command line. You can use IO.popen for this purpose:
IO.popen("./app/animations/fade.sh", "w+") do |f|
f.write "\x01\x01\x04\x00"
end
And on the reading side:
input = $stdin.read
#sp.write [input, "f", "\x12"]
(By the way, it's more common to name Ruby scripts .rb instead of .sh; if fade.sh is meant to be a Ruby script, as I assume from the syntax you used in its example contents, you might want to name it fade.rb)
you could use base64 to pass the bytestring around
$ cat > test.sh
echo $1 | base64 -d
$ chmod a+x test.sh
and then from ruby:
irb
>> require 'base64'
=> true
>> `./test.sh "#{Base64.encode64 "\x01\x01\x04\x00"}"`
=> "\x01\x01\x04\x00"
Can your script accept input from STDIN instead? Perhaps using read.
If you can't do this, you could encode your null and escape your encoding.
E.G. 48656c6c6f0020576f726c64 could be encoded as 48656c6c6f200102020576f726c64
which in turn would be decoded again if both sides agree 2020=20 and 2001=00
Update I think encoding is what you'll have to do because I tried using read and it turns out to be a little too difficult. There's probably another option, but I don't see it yet.
Here's my script and two test runs:
dlamblin$ cat test.sh
echo "reading two lines of input, first line is length of second."
read len
read ans
echo "C string length of second line is:" ${#ans}
for ((c=0; c<$len; c++))
do
/bin/echo -n "${ans:$c:1},"
done
echo ' '
exit
dlamblin$ echo -e '12\0012Hello \0040World' | sh test.sh
reading two lines of input, first line is length of second.
C string length of second line is: 12
H,e,l,l,o, , ,W,o,r,l,d,
dlamblin$ echo -e '12\0012Hello \0000World' | sh test.sh
reading two lines of input, first line is length of second.
C string length of second line is: 5
H,e,l,l,o,,,,,,,,
#Octals \0000 \0012 \0040 are NUL NL and SP respectively