I am following https://docs.docker.com/get-started/06_bind_mounts/#start-a-dev-mode-container on a Windows PC and am stuck here:
Run the following command. We’ll explain what’s going on afterwards:
docker run -dp 3000:3000 \
-w /app -v "$(pwd):/app" \
node:12-alpine \
sh -c "yarn install && yarn run dev"
If you are using PowerShell then use this command:
docker run -dp 3000:3000 `
-w /app -v "$(pwd):/app" `
node:12-alpine `
sh -c "yarn install && yarn run dev"
When using Command Prompt, I get errors (tried multiple variations as shown below), and when using PowerShell, I don't appear to get errors but am not running anything as showed when executing docker ps.
Note that I would rather use Command Prompt and not PowerShell as I could use Linux commands with ComandPrompt on my PC.
What is the significance of backslashes when using Dockers with Command Prompt (and tick marks with PowerShell for that matter)?
I have since found that docker run -dp 3000:3000 -w /app -v "%cd%:/app" node:12-alpine sh -c "yarn install && yarn run dev" works without errors (got rid of backslashes, put on one line, and used %cd% instead of $(pwd)), but would still like to know why using the exact script in the example results in errors.
Using Command Prompt
C:\Users\michael\Documents\Docker\app>docker run -dp 3000:3000 \
docker: invalid reference format.
See 'docker run --help'.
C:\Users\michael\Documents\Docker\app> -w /app -v "$(pwd):/app" \
'-w' is not recognized as an internal or external command,
operable program or batch file.
C:\Users\michael\Documents\Docker\app> node:12-alpine \
The filename, directory name, or volume label syntax is incorrect.
C:\Users\michael\Documents\Docker\app> sh -c "yarn install && yarn run dev"
sh: yarn: command not found
C:\Users\michael\Documents\Docker\app>docker run -dp 3000:3000 \ -w /app -v "$(pwd):/app" \ node:12-alpine \ sh -c "yarn install && yarn run dev"
docker: invalid reference format.
See 'docker run --help'.
C:\Users\michael\Documents\Docker\app>docker run -dp 3000:3000 -w /app -v "$(pwd):/app" node:12-alpine sh -c "yarn install && yarn run dev"
docker: Error response from daemon: create $(pwd): "$(pwd)" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path.
See 'docker run --help'.
C:\Users\michael\Documents\Docker\app>
Using PowerShell
PS C:\Users\michael\Documents\Docker> docker run -dp 3000:3000 `
>> -w /app -v "$(pwd):/app" `
>> node:12-alpine `
>> sh -c "yarn install && yarn run dev"
849af42e78d4ab09242fdd6c3d03bcf1b6b58de984c4485a441a2e2c88603767
PS C:\Users\michael\Documents\Docker> docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
PS C:\Users\michael\Documents\Docker>
would still like to know why using the exact script in the example results in errors.
Because the command with the line-ending \ characters is meant for POSIX-compatible shells such as bash, not for cmd.exe
POSIX-compatible shells (sh, bash, dash, ksh, zsh):
use \ for line-continuation (continuing a command on the following line) and escaping in general.
use $varName to reference both environment and shell-only variables.
support $(...) for embedding the output from a command (...) in command lines (command substitution).
support both double-quoted ("...", interpolating) and single-quoted ('...', verbatim) strings; use '\'' to - in effect - include a ' inside '...'.
(Additionally, in bash, ksh, and zsh, there are the rarely used ANSI C-quoted strings, $'...', and, in bash and ksh, perhaps even more rarely, localizable strings, $"...").
cmd.exe:
uses ^ for line-continuation and escaping in general (in unquoted arguments only).
uses %varName% to reference environment variables (the only variable type supported).
doesn't support command substitutions at all.
supports only "..." strings (interpolating).
PowerShell:
uses ` (the backtick) for line-continuation and escaping in general.
uses $env:varName to reference environment variables, $varName to reference shell-only variables.
supports $(...), called subexpressions, the equivalent of command substitutions (outside of double-quoted strings, (...) is usually sufficient).
supports both double-quoted ("...", interpolating) and single-quoted ('...', verbatim) strings; use '' to embed a ' inside '...'.
Note: A common pitfall is that PowerShell has more metacharacters compared to both POSIX-compatible shells and cmd.exe, notably including # { } , ;, which therefore require individual `-escaping in unquoted arguments or embedding in quoted strings - see this answer.
Potential line-continuation pitfall: in all of the shells discussed, the escape character must be the very last character on the line - not even trailing (intra-line) whitespace is allowed (because the escape character would then apply to it rather than to the newline).
The information above is summarized in the following table:
Feature
POSIX shells _
cmd.exe _
PowerShell _
Line-continuation / escape character
Backslash (\)
Caret (^)
Backtick (`)
Double-quoted strings (interpolating)
✅
✅
✅
Single-quoted strings (verbatim)
✅
❌
✅
Get / set environment variables
$varName /export varName=...
%varName% /set varName=...
$env:varName /$env:varName = ...
Get / set shell-only variables
$varName/varName=...
❌ (no such variables exist, but you can limit the scope of env. vars. with setlocal)
$varName/$varName = ...
Command substitutions, subexpressions
$(...)
❌
(...) / $(...), esp. in strings
Note re setting variables with respect to whitespace on either side of the = symbol:
In POSIX-like shells, there must not be whitespace around =.
In cmd.exe, such whitespace is significant and becomes part of the variable / value name, and is therefore usually to be avoided.
In PowerShell, such whitespace is optional - you may use it to enhance readability; any string value to be assigned requires quoting (e.g., $var = 'hi!')
See also:
https://hyperpolyglot.org/shell for a much more comprehensive juxtaposition of these shells, though note that - as of this writing - the information about PowerShell is incomplete.
Sage Pourpre's helpful answer for links to the line-continuation documentation of the respective shells.
This is character escaping.
The X Character (\ for Bash, backtick for Powershell and ^ for Windows terminal )are used to remove any specific meanings to the next characters.
When used at the end of a line, this mean that the next character (The newline character) is completely ignored.
This keep the command essentially a one-line command from the point of view of the interpreter, but allow you to break it on multiple lines for better readability.
References
Powershell - About special characters
Escape sequences begin with the backtick character [`], known as the grave
accent (ASCII 96), and are case-sensitive. The backtick character can
also be referred to as the escape character.
Bash manual
3.1.2.1 Escape Character
A non-quoted backslash \ is the Bash escape character. It preserves the literal value of the next character that
follows, with the exception of newline. If a \newline pair appears,
and the backslash itself is not quoted, the \newline is treated as a
line continuation (that is, it is removed from the input stream and
effectively ignored).
How-to: Escape Characters, Delimiters and Quotes at the Windows command line
Escaping CR/LF line endings. The ^ escape character can be used to
make long commands more readable by splitting them into multiple lines
and escaping the Carriage Return + Line Feed (CR/LF) at the end of a
line:
ROBOCOPY \\FileServ1\e$\users ^ \\FileServ2\e$\BackupUsers ^ /COPYALL /B /SEC /MIR ^ /R:0 /W:0 /LOG:MyLogfile.txt /NFL /NDL
[...]
A couple of things to be aware of:
A stray space at the end of a line (after the ^) will break the
command, this can be hard to spot unless you have a text editor that
displays spaces and tab characters. If you want comment something out
with REM, then EVERY line needs to be prefixed with REM. Alternatively
if you use a double colon :: as a REM comment, that will still parse
the caret at the end of a line, so in the example above changing the
first line to :: ROBOCOPY… will comment out the whole multi-line
command.
Related
I am creating a Dockerfile that needs to source a script before a shell is run.
ENTRYPOINT ["/bin/bash", "-rcfile","<(echo '. ./mydir/scripttosource.sh')"]
However, the script isn't sourced as expected.
Combining these parameters on a command line (normal Linux instance, outside of any Docker container), it works properly, for example:
$ /bin/bash -rcfile <(echo '. ./mydir/scripttosource.sh')
So I took a look at what was actually used by the container when it was run.
$ docker ps --format "table {{.ID}} \t {{.Names}} \t {{.Command}}" --no-trunc
CONTAINER ID NAMES COMMAND
70a5f846787075bd9bd55432dc17366268c33c1ab06fb36b23a50f5c3aef19bb happy_cray "/bin/bash -rcfile '<(echo '. ./mydir/scripttosource.sh')'"
Besides the fact that it properly identified the emotional state of Cray computers, Docker seems to be sneaking in undesired single quotes into the third parameter to ENTRYPOINT.
'<(echo '. ./mydir/scripttosource.sh')'
Thus the command actually being executed is:
$ /bin/bash -rcfile '<(echo '. ./mydir/scripttosource.sh')'
Which doesn't work...
Now I realize there are more ways to skin this cat, and I could make this work a different way, I am curious about the insertion of single quotes to the third argument to ENTRYPOINT. Is there a way to avoid this?
Thank you,
At a super low level, the Unix execve(2) function launches a process by taking a sequence of words, where the first word is the actual command to run and the remaining words are its arguments. When you run a command interactively, the shell breaks it into words, usually at spaces, and then calls an exec-type function to run it. The shell also does other processing like replacing $VARIABLE references or the bash-specific <(subprocess) construct; all of these are at layers above simply "run a process".
The Dockerfile ENTRYPOINT (and also CMD, and less frequently RUN) has two forms. You're using the JSON-array exec form. If you do this, you're telling Docker that you want to run the main container command with exactly these three literal strings as arguments. In particular the <(...) string is passed as a literal argument to bash --rcfile, and nothing actually executes it.
The obvious answer here is to use the string-syntax shell form instead
ENTRYPOINT /bin/bash -rcfile <(echo '. ./mydir/scripttosource.sh')
Docker wraps this in an invocation of sh -c (or the Dockerfile SHELL). That causes a shell to preprocess the command string, break it into words, and execute it. Assuming the SHELL is bash and not a pure POSIX shell, this will handle the substitution.
However, there are some downsides to this, most notably that the sh -c invocation "eats" all of the arguments that might be passed in the CMD. If you want your main container process to be anything other than an interactive shell, this won't work.
This brings you to the point of trying to find simpler alternatives to doing this. One specific observation is that the substitution here isn't doing anything; <(echo something) will always produce the fixed string something and you can do it without the substitution. If you can avoid the substitution then you don't need the shell either:
ENTRYPOINT ["/bin/bash", "--rcfile", "./mydir/scripttosource.sh"]
Another sensible approach here is to use an entrypoint wrapper script. This uses the ENTRYPOINT to run a shell script that does whatever initialization is needed, then exec "$#" to run the main container command. In particular, if you use the shell . command to set environment variables (equivalent to the bash-specific source) those will "stick" for the main container process.
#!/bin/sh
# entrypoint.sh
# read the file that sets variables
. ./mydir/scripttosource.sh
# run the main container command
exec "$#"
# Dockerfile
COPY entrypoint.sh ./ # may be part of some other COPY
ENTRYPOINT ["./entrypoint.sh"] # must be JSON-array syntax
CMD ???
This should have the same net effect. If you get a debugging shell with docker run --rm -it your-image bash, it will run under the entrypoint wrapper and see the environment variables. You can do other setup in the wrapper script if required. This particular setup also doesn't use any bash-specific options, and might run better under minimal Alpine-based images.
insertion of single quotes can be avoided by using escape characters in the third argument to ENTRYPOINT.
ENTRYPOINT ["/bin/bash", "-rcfile","$(echo '. ./mydir/scripttosource.sh')"]
Some Dockerfiles have
RUN bash -c "apt-get update -qq && ... \"
while others write without quotes like
RUN apt-get update -qq && ... \
What is the difference between these variants? Is one of them more preferable over another one?
You should just write RUN apt-get update ... without manually inserting a sh -c wrapper.
The RUN, CMD, and ENTRYPOINT directives all share the same syntax. It's best documented for ENTRYPOINT but all three commands work the same way. There are two ways to write commands for them: you can either provide a specific set of command words as a JSON array (exec form), or you can write a string and Docker will automatically wrap it sh -c (shell form). For example:
# Create a directory with a space in its name
RUN mkdir "a directory"
# JSON-array form: each array element is a shell word
RUN ["ls", "-ld", "a directory"]
# String form: Docker provides a shell, so these two are equivalent
RUN ls -ld 'a directory'
RUN ["/bin/sh", "-c", "ls -ld 'a directory'"]
This makes your first form redundant: if you RUN bash -c '...', it's a string, and Docker automatically wraps it in sh -c. So you get in effect
# RUN bash -c '...'
RUN ["/bin/sh", "-c", "bash -c '...'"]
GNU bash has a number of extensions that are not POSIX-standard syntax and it's possible to run into trouble with these, particularly on an Alpine-based image where /bin/sh is a minimal shell from the BusyBox toolset. I could see this as an attempt to force a shell command to run using bash rather than the default shell. For most things that appear in Dockerfiles, they won't usually be so complex that they can't be easily rewritten in standard syntax.
# Needs bash for the non-standard `source` syntax
RUN bash -c 'source ./venv/bin/activate && pip list'
# But you can use the standard `.` instead
RUN . ./venv/bin/activate && pip list
If you must have bash interpreting RUN lines then I'd suggest using the SHELL directive to change the command that's used to interpret bare strings.
Style-wise, I also occasionally see JSON-array syntax that begins with an explicit CMD ["/bin/sh", "-c", "..."]. There's no reason to write this out; it's shorter and no less clear to use the string form.
# cat Dockerfile
FROM golang:1.10
WORKDIR /go/src/app
COPY source .
RUN go install -v
ENTRYPOINT [“app”,”-single=true”,”-port=8080"]
# docker run -p 8080:8080 valkyrie-app:v0.0.1
/bin/sh: 1: Syntax error: Unterminated quoted string
Getting this error while running docker run but docker build was successfull, can you please suggest on this error?
#docker build -t valkyrie-app:v0.0.1 .
.....
Successfully built d9ad881d0278
Successfully tagged valkyrie-app:v0.0.1
“ and ” are not valid JSON quotes. Only " is a legitimate quote, in either JSON or POSIX sh.
Thus, your RUN command is not valid JSON, so it's being parsed as a shell command (with only one valid double-quote character instead of matched pairs, hence the specific error seen).
If you're on MacOS, see How to Disable Smart Quotes on Ask Different.
I have a Dockerfile with an ENV declaration for set of paths to be searched that over time has become somewhat comically long:
ENV SPECIAL_PATHS=/foo/bar:/yada/dada:{... ~20 more .. }:/the/end
I cannot see what is the idiomatic way to break in the documentation. I could, of course, define pieces in multiple ENV lines and then combine, but I'd rather not add yet more layers.
You can use \ to break it up over multiple lines.
FROM alpine:3.8
ENV SPECIAL_PATHS \
/foo/bar:\
/yada/yada:\
/the/end
Here's the env in container run from the resulting image.
$ docker container run --rm env-test env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=2fae9abd1eea
SPECIAL_PATHS=/foo/bar:/yada/yada:/the/end
HOME=/root
I would use the backslash character (\) to escape the new line.
I'm trying to do this:
run "echo -n 'foo' > bar.txt"
and the contents of bar.txt ends up being:
-n foo \n
(With \n representing an actual newline)
I use run for other commands like rm -rf and, to my knowledge, it works fine.
I just found this in man echo:
Some shells may provide a builtin echo command which is similar or identical to this utility. Most notably, the builtin echo in sh(1) does not accept the -n option. Consult the builtin(1) manual page.
My version of bash has an echo builtin but seems to be respecting the -n flag. It looks like the shell on your deployment machine doesn't, in which case using the full path to the echo binary might do what you want here:
run "/bin/echo -n 'foo' > bar.txt"
It appears as though the -n flag isn't being interpreted as a flag by the shell. If, from the command line, one executes echo -Y hi, the output will be -Y hi.