I am trying to configure a Splash server with Docker in order to render javascript for scrapy.
I downloaded and installed Docker Toolbox (The latest version of Docker does not install on my Macbook pro 2009 due to lack of CPU MMU support).
I runned "docker run -p 5023:5023 -p 8050:8050 -p 8051:8051 scrapinghub/splash" in Docker Quickstart Terminal. The command performs but after "Starting factory..." I can not get my shell prompt back.
I tried to access http://localhost:8050/ in my browser and to ping localhost:8050 but this does not work: "This site can’t be reached". I would appreciate any help to understand what is the problem. Thank you!
## .
## ## ## ==
## ## ## ## ## ===
/"""""""""""""""""\___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ / ===- ~~~
\______ o __/
\ \ __/
\____\_______/
docker is configured to use the default machine with IP 192.168.99.100
For help getting started, check out the docs at https://docs.docker.com
Ordinateur:~ jb$ docker run -p 5023:5023 -p 8050:8050 -p 8051:8051 scrapinghub/splash
Unable to find image 'scrapinghub/splash:latest' locally
latest: Pulling from scrapinghub/splash
75c416ea735c: Pull complete
c6ff40b6d658: Pull complete
a7050fc1f338: Pull complete
f0ffb5cf6ba9: Pull complete
be232718519c: Pull complete
de1c9f88de34: Pull complete
0b8cdf21020f: Pull complete
cf35f91dcdb5: Pull complete
bfbc20b51f2f: Pull complete
f434ae0e9eb2: Pull complete
Digest: sha256:0a91eb2de229289b0c83f39d8bd079d2464a954001869417a5297aa60820af5b
Status: Downloaded newer image for scrapinghub/splash:latest
2017-07-26 10:45:59+0000 [-] Log opened.
2017-07-26 10:45:59.513146 [-] Splash version: 3.0
2017-07-26 10:45:59.515698 [-] Qt 5.9.1, PyQt 5.9, WebKit 602.1, sip 4.19.3, Twisted 16.1.1, Lua 5.2
2017-07-26 10:45:59.516027 [-] Python 3.5.2 (default, Nov 17 2016, 17:05:23) [GCC 5.4.0 20160609]
2017-07-26 10:45:59.516214 [-] Open files limit: 1048576
2017-07-26 10:45:59.516349 [-] Can't bump open files limit
2017-07-26 10:45:59.657299 [-] Xvfb is started: ['Xvfb', ':806019943', '-screen', '0', '1024x768x24', '-nolisten', 'tcp']
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
2017-07-26 10:45:59.992921 [-] proxy profiles support is enabled, proxy profiles path: /etc/splash/proxy-profiles
2017-07-26 10:46:00.273110 [-] verbosity=1
2017-07-26 10:46:00.273490 [-] slots=50
2017-07-26 10:46:00.275082 [-] argument_cache_max_entries=500
2017-07-26 10:46:00.276541 [-] Web UI: enabled, Lua: enabled (sandbox: enabled)
2017-07-26 10:46:00.278098 [-] Site starting on 8050
2017-07-26 10:46:00.278647 [-] Starting factory <twisted.web.server.Site object at 0x7f751d78ccf8>
If I restart docker and run "docker ps" I can see that the container is running:
Ordinateur:~ jb$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e2c23b576b88 scrapinghub/splash "python3 /app/bin/..." 2 hours ago Up 2 hours 0.0.0.0:5023->5023/tcp, 0.0.0.0:8050-8051->8050-8051/tcp gallant_feynman
Docker is available at address other than localhost. From your logs:
docker is configured to use the default machine with IP 192.168.99.100
Try accessing http://192.168.99.100:8050 instead of localhost. If you're using docker-machine, you can also get an ip address using docker-machine ip default command.
The server was running normally, I just had to replace localhost by my IP. Thank you!
Related
I have a cap rover instance in my digital ocean instance that I created. I want to use teh caprover instance to run cap rover sample apps.
I opened the digital ocean droplet web console in order to run a caprover isntance.
I ran the following lines of code:
ufw allow 80,443,3000,996,7946,4789,2377/tcp; ufw allow 7946,4789,2377/udp;
and got this:
Skipping adding existing rule
Skipping adding existing rule (v6)
Skipping adding existing rule
Skipping adding existing rule (v6)
I then ran this:
docker run -p 80:80 -p 443:443 -p 3000:3000 -v /var/run/docker.sock:/var/run/docker.sock -v /captain:/captain caprover/caprover
I got this:
Unable to find image 'caprover/caprover:latest' locally
latest: Pulling from caprover/caprover
Digest: sha256:39c3f188a8f425775cfbcdc4125706cdf614cd38415244ccf967cd1a4e692b4f
Status: Downloaded newer image for caprover/caprover:latest
docker: Error response from daemon: driver failed programming external connectivity on endpoint priceless_sammet (9da9028cfc4873818f113458237ebd00f9c64fa648b853730a60b10bea39c720): Bind for 0.0.0.0:3000 failed: port is already allocated.
I tried changing the ports to:
docker run -p 81:81 -p 444:444 -p 3321:3321 -v /var/run/docker.sock:/var/run/docker.sock -v /captain:/captain caprover/caprover
and got this:
Captain Starting ...
Installing Captain Service ...
Installation of CapRover is starting...
For troubleshooting, please see: https://caprover.com/docs/troubleshooting.html
>>> Checking System Compatibility <<<
Docker Version passed.
Ubuntu detected.
X86 CPU detected.
Total RAM 1033 MB
Are your trying to run CapRover on a local machine or a machine without public IP?
In that case, you need to add this to your installation command:
-e MAIN_NODE_IP_ADDRESS='127.0.0.1'
Otherwise, if you are running CapRover on a VPS with public IP:
Your firewall may have been blocking an in-use port: 80
A simple solution on Ubuntu systems is to run "ufw disable" (security risk)
Or [recommended] just allowing necessary ports:
ufw allow 80,443,3000,996,7946,4789,2377/tcp; ufw allow 7946,4789,2377/udp;
See docs for more details on how to fix firewall issues
Finally, if you are an advanced user, and you want to bypass this check (NOT RECOMMENDED),
you can append the docker command with an addition flag: -e BY_PASS_PROXY_CHECK='TRUE'
Installation failed.
Error: Port seems to be closed: 80
at Request._callback (/usr/src/app/built/utils/CaptainInstaller.js:149:24)
at Request.self.callback (/usr/src/app/node_modules/request/request.js:185:22)
at Request.emit (events.js:400:28)
at Request.<anonymous> (/usr/src/app/node_modules/request/request.js:1154:10)
at Request.emit (events.js:400:28)
at IncomingMessage.<anonymous> (/usr/src/app/node_modules/request/request.js:1076:12)
at Object.onceWrapper (events.js:519:28)
at IncomingMessage.emit (events.js:412:35)
at endReadableNT (internal/streams/readable.js:1334:12)
at processTicksAndRejections (internal/process/task_queues.js:82:21)
How can I open port 80, 443, and 3000 so that I can run the cap rover instance
Before upgrading my system, I was able to successfully connect to mongo running in a docker container using published ports. After upgrading, as shown in Case #1 connecting via published ports no longer work for me.
Case #1
~ docker run --rm -d -p 27017:27017 mongo:3.6
2594b7e5cbf481526589d221361c853338ff55ecb32d9e02eae17383960e971a
~ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2594b7e5cbf4 mongo:3.6 "docker-entrypoint.s…" 4 seconds ago Up 3 seconds 0.0.0.0:27017->27017/tcp dazzling_fermat
Robo3T Logs
Cannot connect to the MongoDB at localhost:27017.
Error:
Network is unreachable. Reason: network error while attempting to run command 'isMaster' on host 'localhost:27017'
~ sudo lsof -i -P -n | grep LISTEN
...
docker-pr 263637 root 4u IPv4 3723123 0t0 TCP *:27017 (LISTEN)
✘ ~ sudo ufw status
Status: inactive
Now I can only connect using the host networking stack.
Case #2
~ docker run --rm -d --network=host mongo:3.6
39929a8d50cc8554d256f7516d039621cd22ed8be86680ac0e1400809464b619
~ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
39929a8d50cc mongo:3.6 "docker-entrypoint.s…" 5 seconds ago Up 4 seconds admiring_grothendieck
Robo3T Logs
4:13:20 PM Info: Connecting to localhost:27017...
4:13:20 PM Info: Establish connection successful. Connection: localhost
Pre-upgrade:
Linux Mint 19 - Tricia,
Docker version was 19.xx something I believe.
Post Upgrade:
~ lsb_release -a
No LSB modules are available.
Distributor ID: Linuxmint
Description: Linux Mint 20
Release: 20
Codename: ulyana
~ docker --version
Docker version 20.10.7, build 20.10.7-0ubuntu1~20.04.1
I verified there are no running firewalls (UFD, etc), I can connect from container to container when specifying a private docker network for both the server and client. What am I missing? How can I connect using published ports again? Thanks in advance.
Docker on Linux generally uses the host's DNS and modifies your iptables to provide the connectivity between the host and container. If there's a problem with connectivity, in your case the most likely culprits are (in order of likelihood):
DNS entry missing for localhost or wrong IP version target. Try using 127.0.0.1 or ::1 as the hostname instead.
iptables rules are missing. Check the earlier link in my response for remediations and flags that can affect this.
The container might actually have issues starting up. Check the output of docker log <container_id> for errors after you start it. I would say this option is unlikely as things work under host network but don't discount this possibility too quickly.
## .
## ## ## ==
## ## ## ## ## ===
/"""""""""""""""""\___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ / ===- ~~~
\______ o __/
\ \ __/
\____\_______/
docker is configured to use the default machine with IP 192.168.99.100
For help getting started, check out the docs at https://docs.docker.com
Start interactive shell
laptop#LAPTOP-MDLBUKKN MINGW64 /c/Program Files/Docker Toolbox
$ docker pull opensecurity/mobile-security-framework-mobsf
Using default tag: latest
latest: Pulling from opensecurity/mobile-security-framework-mobsf
d51af753c3d3: Pull complete fc878cd0a91c: Pull complete 6154df8ff988: Pull complete fee5db0ff82f: Pull complete 2147c66bc825: Pull complete 0adbcd19651f: Pull complete c941f22acae2: Pull complete 2b48d720aadd: Pull complete c88c099643b2: Pull complete 00718adcba2f: Pull complete 9ec8cb716172: Pull complete 98b4a4bffe15: Pull complete 3d492495d95b: Pull complete ab0f85fef3b0: Pull complete 0df7809d4cce: Pull complete 8aa79076efbd: Pull complete Digest: sha256:7be8e3b29f2f3ce92077078ac3b053c3bdb33a627ea07256730e92a0fe20d3cb
Status: Downloaded newer image for opensecurity/mobile-security-framework-mobsf:latest
docker.io/opensecurity/mobile-security-framework-mobsf:latest
laptop#LAPTOP-MDLBUKKN MINGW64 /c/Program Files/Docker Toolbox
$ # Static Analysis Only
laptop#LAPTOP-MDLBUKKN MINGW64 /c/Program Files/Docker Toolbox
$ docker run -it -p 8000:8000 opensecurity/mobile-security-framework-mobsf:latest
[2020-05-01 03:06:02 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2020-05-01 03:06:02 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
[2020-05-01 03:06:02 +0000] [1] [INFO] Using worker: threads
[2020-05-01 03:06:02 +0000] [7] [INFO] Booting worker with pid: 7
How to open mobsf using docker , I have typed some commands , and any mistakes I have made? I still can't open ,dunno know how to open it ..please tell me the solution of the question , thanks !
You can run docker instance of mobsf via the prebuilt image using below commands
docker pull opensecurity/mobile-security-framework-mobsf
docker run -it --rm -p 8000:8000 opensecurity/mobile-security-framework-mobsf:latest
For more options for running mobsf docker, please view official documentation here
You have probably found a solution to this, but I will recommend not using the docker version. If you need help setting it up local this post describes it:
How to run Mobile Security Framework (MobSF)
By running it through docker you will lose the opportunity to use Dynamic analysis (often referred to as "DAST")
I am trying to follow this tutorial RSelenium and scraping as it seems to be just what I am trying to do. Use RSelenium to scrape a Javascript website.
I have installed Docker and all seems good, but I run into problems with the following command:
remDr <- remoteDriver(remoteServerAddr = "localhost", port = 4445L, browserName = "firefox'")
which I think is intended to be:
remDr <- remoteDriver(remoteServerAddr = "localhost", port = 4445L, browserName = "firefox")
However, both versions return this error:
bash: syntax error near unexpected token `('
What am I doing wrong?
Here is the full log:
Starting "default"...
(default) Check network to re-create if needed...
(default) Windows might ask for the permission to configure a dhcp server. Sometimes, such confirmation window is minimized in the taskbar.
(default) Waiting for an IP...
Machine "default" was started.
Waiting for SSH to be available...
Detecting the provisioner...
Started machines may have new IP addresses. You may need to re-run the `docker-machine env` command.
Regenerate TLS machine certs? Warning: this is irreversible. (y/n): Regenerating TLS certificates
Waiting for SSH to be available...
Detecting the provisioner...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
## .
## ## ## ==
## ## ## ## ## ===
/"""""""""""""""""\___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ / ===- ~~~
\______ o __/
\ \ __/
\____\_______/
docker is configured to use the default machine with IP 192.168.99.100
For help getting started, check out the docs at https://docs.docker.com
Start interactive shell
graha#LAPTOP-SWIFT MINGW64 /c/Program Files/Docker Toolbox
$ docker pull selenium/standalone-chrome Using default tag: latest
latest: Pulling from selenium/standalone-chrome
Digest: sha256:d46e05c47bad20ec4ad675368fa1b7addb6c9529e8fdc23f5eb55629235b8e14
Status: Image is up to date for selenium/standalone-chrome:latest
graha#LAPTOP-SWIFT MINGW64 /c/Program Files/Docker Toolbox
$ docker run -d -p 4445:4444 selenium/standalone-chrome fbb8c9145e92789f6941cc04fb74d216d43aed178825c1698ede2644589c715f
graha#LAPTOP-SWIFT MINGW64 /c/Program Files/Docker Toolbox
$ remDr <- remoteDriver(remoteServerAddr = "localhost", port = 4445L, browserName = "chrome")
bash: syntax error near unexpected token `('
graha#LAPTOP-SWIFT MINGW64 /c/Program Files/Docker Toolbox
$
Thanks.
Your basic problem is that you're trying to type an R command at the bash shell prompt, and that's never going to work. I think you may be missing the point of the tutorial you're following:
The Docker image you're using just provides Selenium, a tool for remotely controlling web browsers. It is expected that you are running R on your host, and pointing the RSelenium library at the selenium service offered by the Docker container.
You will need to install R on your host in order to follow that tutorial.
I'm getting started with docker using docker-toolbox on win 10 home. I'm experimenting with scrapy splash project (https://github.com/scrapy-plugins/scrapy-splash). I've installed the toolbox on a flash drive (e:)
If I understand correctly I have installed docker using the docker toolbox. When I click on the docker quickstart terminal, I get the screenshot.
I ran:
$ docker-machine start
but when I :
$ docker run -it scrapinghub/splash
2019-05-25 22:33:53+0000 [-] Log opened.
2019-05-25 22:33:53.053632 [-] Splash version: 3.3.1
2019-05-25 22:33:53.054895 [-] Qt 5.9.1, PyQt 5.9.2, WebKit 602.1, sip 4.19.4, Twisted 18.9.0, Lua 5.2
2019-05-25 22:33:53.055695 [-] Python 3.5.2 (default, Nov 12 2018, 13:43:14) [GCC 5.4.0 20160609]
2019-05-25 22:33:53.056773 [-] Open files limit: 1048576
2019-05-25 22:33:53.057319 [-] Can't bump open files limit
2019-05-25 22:33:53.165435 [-] Xvfb is started: ['Xvfb', ':1788299128', '-screen', '0', '1024x768x24', '-nolisten', 'tcp']
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
2019-05-25 22:33:53.257636 [-] proxy profiles support is enabled, proxy profiles path: /etc/splash/proxy-profiles
2019-05-25 22:33:53.258827 [-] memory cache: enabled, private mode: enabled, js cross-domain access: disabled
2019-05-25 22:33:53.422507 [-] verbosity=1, slots=20, argument_cache_max_entries=500, max-timeout=90.0
2019-05-25 22:33:53.424799 [-] Web UI: enabled, Lua: enabled (sandbox: enabled)
2019-05-25 22:33:53.426021 [-] Site starting on 8050
2019-05-25 22:33:53.426778 [-] Starting factory <twisted.web.server.Site object at 0x7efcd8d8dcc0>
2019-05-25 22:33:53.427649 [-] Server listening on http://0.0.0.0:8050
I tried to open the browser at
http://0.0.0.0:8050
But get the error in the title. What am I doing wrong?
edit:
I had to remove prior container by :
docker container ls
docker rm -f <container-name>
Then it worked at :
http://192.168.99.100:8050/
You missed the publishing part.
docker run -it -p 8050:8050 scrapinghub/splash