I am having issues executing the following query. Is there any way to maintain the upper case letters with the psql query? I have tried quotes which does not work and I have tried single quotes, in which case I get a syntax error. NOTE: It appears the columns were create using quotes, fine, but how do I reference quotes from within the command then?)
psql bash CLI
psql -h $DB_HOST -p $DB_PORT -U $DB_USER -d $DB_NAME -c "\copy table_name(fileName, time) FROM $OUTPUT_FILE WITH DELIMITER AS ',' NULL 'null'"
Result:
ERROR: column "filename" of relation "table_name" does not exist
PostgreSQL treats an identifier such as a table name as lowercase unless you quote it.
You said though that double quotes didn't work. That's probably because you didn't get the quoting right. Quoting in the shell is hard. You have to end your double quote string surrounding the entire query just to start a single quote string that contains the double quote:
psql -h $DB_HOST -p $DB_PORT -U $DB_USER -d $DB_NAME -c "\copy table_name("'"fileName"'", time) FROM $OUTPUT_FILE WITH DELIMITER AS ',' NULL 'null'"
You can leverage the fact that psql also accepts the command through standard input, which allows you to write it as follows without worrying about the quotes:
psql -h $DB_HOST -p $DB_PORT -U $DB_USER -d $DB_NAME <<SQL
\copy table_name("fileName", time) FROM $OUTPUT_FILE WITH DELIMITER AS ',' NULL 'null'
SQL
Related
PGPASSWORD=$PPASSWORD psql -h $PG_HOST -U $PUSERNAME -p $PG_PORT -d $PG_DB << EOF
\i /dbf/appworx/pgsql/$1.sql $2;
it giving error like \i: extra argument "232;" ignored
I am following https://docs.docker.com/get-started/06_bind_mounts/#start-a-dev-mode-container on a Windows PC and am stuck here:
Run the following command. We’ll explain what’s going on afterwards:
docker run -dp 3000:3000 \
-w /app -v "$(pwd):/app" \
node:12-alpine \
sh -c "yarn install && yarn run dev"
If you are using PowerShell then use this command:
docker run -dp 3000:3000 `
-w /app -v "$(pwd):/app" `
node:12-alpine `
sh -c "yarn install && yarn run dev"
When using Command Prompt, I get errors (tried multiple variations as shown below), and when using PowerShell, I don't appear to get errors but am not running anything as showed when executing docker ps.
Note that I would rather use Command Prompt and not PowerShell as I could use Linux commands with ComandPrompt on my PC.
What is the significance of backslashes when using Dockers with Command Prompt (and tick marks with PowerShell for that matter)?
I have since found that docker run -dp 3000:3000 -w /app -v "%cd%:/app" node:12-alpine sh -c "yarn install && yarn run dev" works without errors (got rid of backslashes, put on one line, and used %cd% instead of $(pwd)), but would still like to know why using the exact script in the example results in errors.
Using Command Prompt
C:\Users\michael\Documents\Docker\app>docker run -dp 3000:3000 \
docker: invalid reference format.
See 'docker run --help'.
C:\Users\michael\Documents\Docker\app> -w /app -v "$(pwd):/app" \
'-w' is not recognized as an internal or external command,
operable program or batch file.
C:\Users\michael\Documents\Docker\app> node:12-alpine \
The filename, directory name, or volume label syntax is incorrect.
C:\Users\michael\Documents\Docker\app> sh -c "yarn install && yarn run dev"
sh: yarn: command not found
C:\Users\michael\Documents\Docker\app>docker run -dp 3000:3000 \ -w /app -v "$(pwd):/app" \ node:12-alpine \ sh -c "yarn install && yarn run dev"
docker: invalid reference format.
See 'docker run --help'.
C:\Users\michael\Documents\Docker\app>docker run -dp 3000:3000 -w /app -v "$(pwd):/app" node:12-alpine sh -c "yarn install && yarn run dev"
docker: Error response from daemon: create $(pwd): "$(pwd)" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path.
See 'docker run --help'.
C:\Users\michael\Documents\Docker\app>
Using PowerShell
PS C:\Users\michael\Documents\Docker> docker run -dp 3000:3000 `
>> -w /app -v "$(pwd):/app" `
>> node:12-alpine `
>> sh -c "yarn install && yarn run dev"
849af42e78d4ab09242fdd6c3d03bcf1b6b58de984c4485a441a2e2c88603767
PS C:\Users\michael\Documents\Docker> docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
PS C:\Users\michael\Documents\Docker>
would still like to know why using the exact script in the example results in errors.
Because the command with the line-ending \ characters is meant for POSIX-compatible shells such as bash, not for cmd.exe
POSIX-compatible shells (sh, bash, dash, ksh, zsh):
use \ for line-continuation (continuing a command on the following line) and escaping in general.
use $varName to reference both environment and shell-only variables.
support $(...) for embedding the output from a command (...) in command lines (command substitution).
support both double-quoted ("...", interpolating) and single-quoted ('...', verbatim) strings; use '\'' to - in effect - include a ' inside '...'.
(Additionally, in bash, ksh, and zsh, there are the rarely used ANSI C-quoted strings, $'...', and, in bash and ksh, perhaps even more rarely, localizable strings, $"...").
cmd.exe:
uses ^ for line-continuation and escaping in general (in unquoted arguments only).
uses %varName% to reference environment variables (the only variable type supported).
doesn't support command substitutions at all.
supports only "..." strings (interpolating).
PowerShell:
uses ` (the backtick) for line-continuation and escaping in general.
uses $env:varName to reference environment variables, $varName to reference shell-only variables.
supports $(...), called subexpressions, the equivalent of command substitutions (outside of double-quoted strings, (...) is usually sufficient).
supports both double-quoted ("...", interpolating) and single-quoted ('...', verbatim) strings; use '' to embed a ' inside '...'.
Note: A common pitfall is that PowerShell has more metacharacters compared to both POSIX-compatible shells and cmd.exe, notably including # { } , ;, which therefore require individual `-escaping in unquoted arguments or embedding in quoted strings - see this answer.
Potential line-continuation pitfall: in all of the shells discussed, the escape character must be the very last character on the line - not even trailing (intra-line) whitespace is allowed (because the escape character would then apply to it rather than to the newline).
The information above is summarized in the following table:
Feature
POSIX shells _
cmd.exe _
PowerShell _
Line-continuation / escape character
Backslash (\)
Caret (^)
Backtick (`)
Double-quoted strings (interpolating)
✅
✅
✅
Single-quoted strings (verbatim)
✅
❌
✅
Get / set environment variables
$varName /export varName=...
%varName% /set varName=...
$env:varName /$env:varName = ...
Get / set shell-only variables
$varName/varName=...
❌ (no such variables exist, but you can limit the scope of env. vars. with setlocal)
$varName/$varName = ...
Command substitutions, subexpressions
$(...)
❌
(...) / $(...), esp. in strings
Note re setting variables with respect to whitespace on either side of the = symbol:
In POSIX-like shells, there must not be whitespace around =.
In cmd.exe, such whitespace is significant and becomes part of the variable / value name, and is therefore usually to be avoided.
In PowerShell, such whitespace is optional - you may use it to enhance readability; any string value to be assigned requires quoting (e.g., $var = 'hi!')
See also:
https://hyperpolyglot.org/shell for a much more comprehensive juxtaposition of these shells, though note that - as of this writing - the information about PowerShell is incomplete.
Sage Pourpre's helpful answer for links to the line-continuation documentation of the respective shells.
This is character escaping.
The X Character (\ for Bash, backtick for Powershell and ^ for Windows terminal )are used to remove any specific meanings to the next characters.
When used at the end of a line, this mean that the next character (The newline character) is completely ignored.
This keep the command essentially a one-line command from the point of view of the interpreter, but allow you to break it on multiple lines for better readability.
References
Powershell - About special characters
Escape sequences begin with the backtick character [`], known as the grave
accent (ASCII 96), and are case-sensitive. The backtick character can
also be referred to as the escape character.
Bash manual
3.1.2.1 Escape Character
A non-quoted backslash \ is the Bash escape character. It preserves the literal value of the next character that
follows, with the exception of newline. If a \newline pair appears,
and the backslash itself is not quoted, the \newline is treated as a
line continuation (that is, it is removed from the input stream and
effectively ignored).
How-to: Escape Characters, Delimiters and Quotes at the Windows command line
Escaping CR/LF line endings. The ^ escape character can be used to
make long commands more readable by splitting them into multiple lines
and escaping the Carriage Return + Line Feed (CR/LF) at the end of a
line:
ROBOCOPY \\FileServ1\e$\users ^ \\FileServ2\e$\BackupUsers ^ /COPYALL /B /SEC /MIR ^ /R:0 /W:0 /LOG:MyLogfile.txt /NFL /NDL
[...]
A couple of things to be aware of:
A stray space at the end of a line (after the ^) will break the
command, this can be hard to spot unless you have a text editor that
displays spaces and tab characters. If you want comment something out
with REM, then EVERY line needs to be prefixed with REM. Alternatively
if you use a double colon :: as a REM comment, that will still parse
the caret at the end of a line, so in the example above changing the
first line to :: ROBOCOPY… will comment out the whole multi-line
command.
version: '3.7'
services:
pgdump:
image: postgres:alpine
command: pg_dump -f "backup-`date -u -Iseconds`.pg_restore" $DATABASE_URL
This produces a file named
backup-`date -u -Iseconds`.pg_restore
instead of the desired
backup-2021-04-14T16:42:54+00:00.pg_restore.
I also tried:
command: pg_dump -f backup-`date -u -Iseconds`.pg_restore $DATABASE_URL
command: pg_dump -f "backup-${date -u -Iseconds}.pg_restore" $DATABASE_URL
command: pg_dump -f backup-${date -u -Iseconds}.pg_restore $DATABASE_URL
All of them yield different errors.
As of April 2021 command substitution is not supported by docker-compose according to this GitHub issue.
As a workaround in my use case, one could either use native docker run commands, where substitution works or use an .env file.
Current command
The date command itself is incorrect. Try running it on its own
date -u -Iseconds
echo `date -u -Iseconds`
From your command, I presume you want date in UTC seconds since epoch? Epoch by itself is UTC. So you just need seconds since Epoch. No need for -u parameter.
Solution
Here's the correct command in two forms:
A.
command: pg_dump -f "backup-`date +'%s'`.pg_restore" $DATABASE_URL
B.
command: pg_dump -f "backup-$(date +'%s').pg_restore" $DATABASE_URL
Explanation
There are multiple things to watch out for in the command you provided:
Notice the double quotes around the file name? This means you cannot nest another double-quote within the original outer pair without escaping the inner ones with \. Another alternative option is to use as many single-quote pairs you want within a pair of double-quotes. See this answer and this excerpt about 2.2.2 Single-Quotes and 2.2.3 Double-Quotes.
For string interpolation, you can use either $() or `` notation. But NOT within single-quotes as I said.
As a dry-run test, create a file directly with said notations:
vi "backup-`date +'%s'`.txt"
vi "backup-$(date +'%s').txt"
As for date format. Both GNU/date BSD/date accept %s to represent seconds since Epoch. Find "%s" in ss64 or man7 or cyberciti.
Docker-related, watch out what command does. Source:
command overrides the the default command declared by the container image (i.e. by Dockerfile's CMD).
You can create the filename and store it as a variable with shell command before doing the pg_dump:
version: '3.7'
services:
pgdump:
image: postgres:alpine
entrypoint: ["/bin/sh","-c"]
command: >
"FILENAME=backup-`date -u -Iseconds`.pg_restore
&& pg_dump -f $$FILENAME $$DATABASE_URL"
Successfully tested against Docker image for postgres 13.6.
I do the following on my server:
pg_dump -O -c register_production > register.sql
Then, after copying register.sql to my local environment, I try:
psql register_development < register.sql
This appears to work, but when I try to launch the Rails site locally, I get this:
PG::UndefinedTable: ERROR: relation "list_items" does not exist at character 28
How can I restore everything (including relations) from the server db to my local dev db?
I use this command to save my database:
pg_dump -F c -v -U postgres -h localhost <database_name> -f /tmp/<filename>.psql
And this to restore it:
pg_restore -c -C -F c -v -U postgres /tmp/<filename>.psql
This dumps the database in Postgres' custom format (-F c) which is compressed by default and allows for reordering of its contents. -C -c will drop the database if it exists already and then recreate it, helpful in your case. And -v specifies verbose so you can see exactly what's happening when this goes on.
Does the register_development database exist before you run the psql command? Because that form will not create it for you.
See http://www.postgresql.org/docs/8.1/static/backup.html#BACKUP-DUMP-RESTORE for more information.
The following command gives me a list of matching expressions:
grep -f /tmp/list Filename* > /tmp/output
The list file is then parsed and used to search Filename* for the parsed string. The results are then saved to output.
How would I output the parsed string from list in the case where there is no match in Filename*?
Contents of the list file could be:
ABC
BLA
ZZZ
HJK
Example Files:
Filename1:5,ABC,123
Filename2:5,ZZZ,342
Result of Running Command:
BLA
HJK
Stack overflow question 2480584 looks like it may be relevant, through the use of an if statement. However I'm not sure how to output the parsed string to the output file. Would require some type of read line?
TIA,
Mic
Obviously, grep -f list Filename* gives all matches of patterns from the file list in the files specified by Filename*, i.e.,
Filename1:5,ABC,123
Filename2:5,ZZZ,342
in your example.
By adding the -o (only print matching expression) and -h (do not print filename) flags, we can turn this into:
ABC
ZZZ
Now you want all patterns from list that are not contained in this list, which can be achieved by
grep -f list Filename* -o -h | grep -f /dev/stdin -v list
where the second grep takes it's patterns from the output of the first and by using the -v flag gives all the lines of file list that do not match those patterns.
This makes it:
$ grep -v "$(cat Filename* | cut -d, -f2)" /tmp/list
BLA
HJK
Explanation
$ cat Filename* | cut -d, -f2
ABC
ZZZ
And then grep -v looks for the inverse matching.