Docker container Xdebug (php-fpm) - There was a problem sending 179 bytes on socket 6: Broken pipe - docker

I am trying to set up Xdebug 3 in a docker container and capture a break point in Visual Studio Code.
My xdebug log:
[7] Log opened at 2021-02-16 16:12:12.027862
[7] [Step Debug] INFO: Checking remote connect back address.
[7] [Step Debug] INFO: Checking header 'HTTP_X_FORWARDED_FOR'.
[7] [Step Debug] INFO: Checking header 'REMOTE_ADDR'.
[7] [Step Debug] INFO: Client host discovered through HTTP header, connecting to 172.26.0.1:9099.
[7] [Step Debug] INFO: Connected to debugging client: 172.26.0.1:9099 (from REMOTE_ADDR HTTP header). :-)
[7] [Step Debug] -> <init xmlns="urn:debugger_protocol_v1" xmlns:xdebug="https://xdebug.org/dbgp/xdebug" fileuri="file:///var/www/public/index.php" language="PHP" xdebug:language_version="7.3.27" protocol_version="1.0" appid="7" idekey="VSCODE"><engine version="3.0.2"><![CDATA[Xdebug]]></engine><author><![CDATA[Derick Rethans]]></author><url><![CDATA[https://xdebug.org]]></url><copyright><![CDATA[Copyright (c) 2002-2021 by Derick Rethans]]></copyright></init>
[7] [Step Debug] -> <response xmlns="urn:debugger_protocol_v1" xmlns:xdebug="https://xdebug.org/dbgp/xdebug" status="stopping" reason="ok"></response>
[7] [Step Debug] WARN: 2021-02-16 16:12:12.039287: There was a problem sending 179 bytes on socket 6: Broken pipe.
xdebug.ini configuration:
[xdebug]
zend_extension=/usr/local/lib/php/extensions/no-debug-non-zts-20180731/xdebug.so
xdebug.client_port = 9099
xdebug.client_host = host.docker.internal
xdebug.idekey = VSCODE
xdebug.mode = debug
xdebug.start_with_request = yes
xdebug.discover_client_host = true
xdebug.log = /var/tmp/xdebug.log
Visual Studio Code configuration (json.config)
{
"version": "0.2.0",
"configurations": [
{
"name": "Listen for XDebug",
"type": "php",
"request": "launch",
"port": 9099,
"log": true,
"externalConsole": false,
"pathMappings": {
"/var/www/public": "${workspaceFolder}",
},
"ignore": [
"**/vendor/**/*.php"
]
},
]
}
I will be grateful for your help, I have run out of ideas why it doesn't work.

host.docker.internal - mac only
172.17.0.1/16 brd 172.17.255.255 scope global docker0
probably
zend_extension=/usr/local/lib/php/extensions/no-debug-non-zts-20180731/xdebug.so
xdebug.mode=debug
xdebug.client_port = 9003
xdebug.client_host=172.17.0.1
xdebug.start_with_request=yes
xdebug.extended_info=1
xdebug.remote_handler="dbgp"
xdebug.remote_connect_back=0
xdebug.idekey = "VSCODE"

Related

PHP Xdebug, VSCode, Docker & MacOS - debugger not disconnecting

I've encountered a weird problem that seems to have either started appearing when the Xdebug extension was updated or Visual Studio Code received an update.
When the debugger is active in VSCode the website works fine, when stopping the debugger, the Xdebug extension within the docker container seems to be unable to disconnect. And the website is stuck in a loading state.
I receive the following message in the xdebug.log when trying to access the site when the debugger is stopped in VSCode.
[318] Log opened at 2022-04-08 12:19:15.819009
[318] [Step Debug] INFO: Connecting to configured address/port: host.docker.internal:9001.
[318] [Step Debug] INFO: Connected to debugging client: host.docker.internal:9001 (through xdebug.client_host/xdebug.client_port). :-)
[318] [Step Debug] -> <init xmlns="urn:debugger_protocol_v1" xmlns:xdebug="https://xdebug.org/dbgp/xdebug" fileuri="file:///var/www/html/index.php" language="PHP" xdebug:language_version="7.4.0" protocol_version="1.0" appid="318" idekey="VSCODE"><engine version="3.1.4"><![CDATA[Xdebug]]></engine><author><![CDATA[Derick Rethans]]></author><url><![CDATA[https://xdebug.org]]></url><copyright><![CDATA[Copyright (c) 2002-2022 by Derick Rethans]]></copyright></init>
This seems like the Xdebug extension still tries to connect (successfully) to the listening port at 9001.
But the port is not listening on the host, when checking with lsof -i :9001.
PHP Xdebug settings:
zend_extension=xdebug.so
xdebug.mode=debug
xdebug.remote_handler=dbgp
xdebug.idekey=VSCODE
xdebug.start_with_request=yes
xdebug.log=/tmp/xdebug.log
xdebug.client_host=host.docker.internal
xdebug.client_port=9001
xdebug.discover_client_host=0
launch.json settings:
{
"version": "0.2.0",
"configurations": [{
"name": "Listen for XDebug",
"type": "php",
"request": "launch",
"port": 9001,
"log": false,
"externalConsole": false,
"pathMappings": {
"/var/www/html": "${workspaceFolder}",
},
"ignore": [
"shared/vendor/*"
]
},
]
}
I've also tried with a different client_host setting.
Added an alias to the en0 interface, to reach the host IP from Xdebug.
sudo ifconfig en0 alias 10.128.128.128 255.255.255.0
No variation of settings seems to eliminate this problem, anyone has any ideas?
The 4.8.0 build solves this issue.
The Docker for mac ticket: https://github.com/docker/for-mac/issues/6235
The builds that resolves the issue:
Intel: https://desktop-stage.docker.com/mac/main/amd64/78146/Docker.dmg
Apple silicon: https://desktop-stage.docker.com/mac/main/arm64/78146/Docker.dmg

How to fix the "Incoming connection from Xdebug" in PhpStorm

I have Xdebug 2.9.8 installed in a Docker container and the xdebug.ini settings are:
xdebug.default_enable=1
xdebug.remote_enable=on
xdebug.profiler_enable = off
xdebug.profiler_enable_trigger = off
xdebug.profiler_output_name = cachegrind.out.%t.%p
xdebug.profiler_output_dir = "/tmp/php7-xdebug.log"
xdebug.remote_log="/tmp/xdebug.log"
xdebug.remote_autostart=1
xdebug.remote_connect_back=1
xdebug.max_nesting_level = 1000
xdebug.remote_port=9000
xdebug.remote_handler = dbgp
xdebug.remote_mode = req
xdebug.remote_host=host.docker.internal
xdebug.idekey=PHPSTORM
xdebug.remote_handler=dbgp
xdebug.max_nesting_level=1500T
xdebug.var_display_max_children = 512
xdebug.var_display_max_data = 512
xdebug.var_display_max_depth = 12
But I keep getting the pop-up from PhpStorm:
I don't know if it has any relations with this post:
PhpStorm keeps asking for configuring "Incoming Connection From Xdebug" while I already have a server config
I also have the logs from /tmp/xdebug.log:
[23] Log opened at 2021-10-21 12:01:06
[23] I: Checking remote connect back address.
[23] I: Checking header 'HTTP_X_FORWARDED_FOR'.
[23] I: Checking header 'REMOTE_ADDR'.
[23] I: Remote address found, connecting to 172.24.0.1:9000.
[23] I: Connected to client. :-)
[23] -> <init xmlns="urn:debugger_protocol_v1" xmlns:xdebug="https://xdebug.org/dbgp/xdebug" fileuri="file:///var/www/html/domestico2.local/pub/index.php" language="PHP" xdebug:language_version="7.4.24" protocol_version="1.0" appid="23" idekey="PHPSTORM"><engine version="2.9.8"><![CDATA[Xdebug]]></engine><author><![CDATA[Derick Rethans]]></author><url><![CDATA[https://xdebug.org]]></url><copyright><![CDATA[Copyright (c) 2002-2020 by Derick Rethans]]></copyright></init>
[23] <- feature_set -i 1 -n show_hidden -v 1
[23] -> <response xmlns="urn:debugger_protocol_v1" xmlns:xdebug="https://xdebug.org/dbgp/xdebug" command="feature_set" transaction_id="1" feature="show_hidden" success="1"></response>
[23] <- feature_set -i 2 -n max_depth -v 1
[23] -> <response xmlns="urn:debugger_protocol_v1" xmlns:xdebug="https://xdebug.org/dbgp/xdebug" command="feature_set" transaction_id="2" feature="max_depth" success="1"></response>
[23] <- feature_set -i 3 -n max_children -v 100
[23] -> <response xmlns="urn:debugger_protocol_v1" xmlns:xdebug="https://xdebug.org/dbgp/xdebug" command="feature_set" transaction_id="3" feature="max_children" success="1"></response>
[23] <- feature_set -i 4 -n extended_properties -v 1
[23] -> <response xmlns="urn:debugger_protocol_v1" xmlns:xdebug="https://xdebug.org/dbgp/xdebug" command="feature_set" transaction_id="4" feature="extended_properties" success="1"></response>
[23] <- feature_set -i 5 -n notify_ok -v 1
[23] -> <response xmlns="urn:debugger_protocol_v1" xmlns:xdebug="https://xdebug.org/dbgp/xdebug" command="feature_set" transaction_id="5" feature="notify_ok" success="1"></response>
[23] <- feature_set -i 6 -n resolved_breakpoints -v 1
[23] -> <response xmlns="urn:debugger_protocol_v1" xmlns:xdebug="https://xdebug.org/dbgp/xdebug" command="feature_set" transaction_id="6" feature="resolved_breakpoints" success="1"></response>
[23] <- stdout -i 7 -c 1
[23] -> <response xmlns="urn:debugger_protocol_v1" xmlns:xdebug="https://xdebug.org/dbgp/xdebug" command="stdout" transaction_id="7" success="1"></response>
[23] <- status -i 8
[23] -> <response xmlns="urn:debugger_protocol_v1" xmlns:xdebug="https://xdebug.org/dbgp/xdebug" command="status" transaction_id="8" status="starting" reason="ok"></response>
[23] <- step_into -i 9
[23] -> <response xmlns="urn:debugger_protocol_v1" xmlns:xdebug="https://xdebug.org/dbgp/xdebug" command="step_into" transaction_id="9" status="break" reason="ok"><xdebug:message filename="file:///var/www/html/project.local/pub/index.php" lineno="12"></xdebug:message></response>
[23] <- eval -i 10 -- aXNzZXQoJF9TRVJWRVJbJ1BIUF9JREVfQ09ORklHJ10p
[23] -> <response xmlns="urn:debugger_protocol_v1" xmlns:xdebug="https://xdebug.org/dbgp/xdebug" command="eval" transaction_id="10"><property type="bool"><![CDATA[0]]></property></response>
[23] <- eval -i 11 -- aXNzZXQoJF9TRVJWRVJbJ1NFUlZFUl9OQU1FJ10p
[23] -> <response xmlns="urn:debugger_protocol_v1" xmlns:xdebug="https://xdebug.org/dbgp/xdebug" command="eval" transaction_id="11"><property type="bool"><![CDATA[1]]></property></response>
[23] <- eval -i 12 -- KHN0cmluZykoJF9TRVJWRVJbJ1NFUlZFUl9OQU1FJ10p
[23] -> <response xmlns="urn:debugger_protocol_v1" xmlns:xdebug="https://xdebug.org/dbgp/xdebug" command="eval" transaction_id="12"><property type="string" size="16" encoding="base64"><![CDATA[ZG9tZXN0aWNvMi5sb2NhbA==]]></property></response>
[23] <- eval -i 13 -- KHN0cmluZykoJF9TRVJWRVJbJ1NFUlZFUl9QT1JUJ10p
[23] -> <response xmlns="urn:debugger_protocol_v1" xmlns:xdebug="https://xdebug.org/dbgp/xdebug" command="eval" transaction_id="13"><property type="string" size="2" encoding="base64"><![CDATA[ODA=]]></property></response>
[23] <- eval -i 14 -- KHN0cmluZykoJF9TRVJWRVJbJ1JFUVVFU1RfVVJJJ10p
[23] -> <response xmlns="urn:debugger_protocol_v1" xmlns:xdebug="https://xdebug.org/dbgp/xdebug" command="eval" transaction_id="14"><property type="string" size="12" encoding="base64"><![CDATA[L2Zhdmljb24uaWNv]]></property></response>
Help me StackOverflow, you're my only hope!
P.S.
The path mapping config:
According to your info (screenshots of popup + PHP | Servers settings as well as Xdebug log file) you need to edit your Server entry a bit:
Change the Host from localhost to be the actual domain name: domestico2.local. It should work now.
Local test that I did:
URL: http://idetest.local.test/test.php
Server entry (it's all local, no path mappings needed in my case):
The debug works just fine.
If I now change the Host to some different name (e.g. idetest22.local.test) and try to debug the same URL I will see the similar popup:
If I click on "Accept" button PhpStorm will make a new Server entry for that domain for me:

webdriver: Request failed with status 500 due to unknown error: An unknown server-side error occurred while processing the command

capabilities:[{
appiumVersion: '1.18.0',
platformName: 'Android',
platformVersion: '10.0',
deviceOrientation: 'portrait ',
deviceName: 'Pixel_3a_API_29_portal',
app: "/Users/sandeep/Downloads/filename.apk",
udid:"emulator-5554",
appPackage: "appPackageString",
appActivity: "appActivityString",
autoAcceptAlerts: true,
autoGrantPermissions: true,
automationName: 'UiAutomator2',
}]
port: 4723,
hostname: '0.0.0.0',
path: "/wd/hub",
runner: 'local',
services: ['appium'],
I have used above configuration in config.js, but it is throwing error as below::
webdriver: Request failed with status 500 due to unknown error: An unknown server-side error occurred while processing the command. Original error: Process didn't end after 10000ms (cmd: '/usr/local/bin/adb -P 5037 -s emulator-5554 logcat -v threadtime')
Use Control+c to gracefully shut down the port which is already somewhere in use.
Or else use these commands in your terminal
netstat -ano|findstr "PID :4723" : This will tell you about the port which is in use
Example:
C:\Users>netstat -ano|findstr "PID :4723"
Proto Local Address Foreign Address State PID
TCP 0.0.0.0:4723 0.0.0.0:0 LISTENING 9100
Then use
2. taskkill /pid 9100 /f : This will kill the session gracefully

fluentbit fails to communicate with fluentd

I am trying a simple fluentbit / fluentd test with ipv6, but it is not working.
Configuration from fluentbit side:
[SERVICE]
Flush 5
Daemon off
[INPUT]
Name cpu
Tag fluent_bit
[OUTPUT]
Name forward
Match *
Host fd00:7fff:0:2:9c43:9bff:fe00:bb
Port 24000
Configuration from fluentd side:
<source>
type forward
bind ::
port 24000
</source>
~
<match fluent_bit>
type stdout
</match>
I start up fluentd with command: /usr/sbin/td-agent -c test.conf
Then, I start up fluentbit with command: /opt/td-agent-bit/bin/td-agent-bit -c test.conf
The output shows that there is a problem with communication:
Fluent Bit v1.6.6
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
[2020/11/29 01:53:49] [ info] [engine] started (pid=142)
[2020/11/29 01:53:49] [ info] [storage] version=1.0.6, initializing...
[2020/11/29 01:53:49] [ info] [storage] in-memory
[2020/11/29 01:53:49] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/11/29 01:53:49] [ info] [sp] stream processor started
[2020/11/29 01:53:53] [error] [io] connection #27 failed to: fd00:7fff:0:2:9c43:9bff:fe00:bb:24000
[2020/11/29 01:53:53] [error] [output:forward:forward.0] no upstream connections available
[2020/11/29 01:53:53] [ warn] [engine] failed to flush chunk '142-1606614829.871139401.flb', retry in 8 seconds: task_id=0, input=cpu.0 > output=forward.0
[2020/11/29 01:53:58] [error] [io] connection #28 failed to: fd00:7fff:0:2:9c43:9bff:fe00:bb:24000
[2020/11/29 01:53:58] [error] [output:forward:forward.0] no upstream connections available
[2020/11/29 01:53:58] [ warn] [engine] failed to flush chunk '142-1606614833.871418916.flb', retry in 6 seconds: task_id=1, input=cpu.0 > output=forward.0
[2020/11/29 01:54:01] [error] [io] connection #29 failed to: fd00:7fff:0:2:9c43:9bff:fe00:bb:24000
[2020/11/29 01:54:01] [error] [output:forward:forward.0] no upstream connections available
[2020/11/29 01:54:01] [ warn] [engine] chunk '142-1606614829.871139401.flb' cannot be retried: task_id=0, input=cpu.0 > output=forward.0
However when I run the command without using the config file but passing the parameters, it works:
# /opt/td-agent-bit/bin/td-agent-bit -i cpu -t fluent_bit -o forward://[fd00:7fff:0:2:9c43:9bff:fe00:bb]:24000 -v
Fluent Bit v1.6.6
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
[2020/11/29 01:56:53] [ info] Configuration:
[2020/11/29 01:56:53] [ info] flush time | 5.000000 seconds
[2020/11/29 01:56:53] [ info] grace | 5 seconds
[2020/11/29 01:56:53] [ info] daemon | 0
[2020/11/29 01:56:53] [ info] ___________
[2020/11/29 01:56:53] [ info] inputs:
[2020/11/29 01:56:53] [ info] cpu
[2020/11/29 01:56:53] [ info] ___________
[2020/11/29 01:56:53] [ info] filters:
[2020/11/29 01:56:53] [ info] ___________
[2020/11/29 01:56:53] [ info] outputs:
[2020/11/29 01:56:53] [ info] forward.0
[2020/11/29 01:56:53] [ info] ___________
[2020/11/29 01:56:53] [ info] collectors:
[2020/11/29 01:56:53] [ info] [engine] started (pid=151)
[2020/11/29 01:56:53] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2020/11/29 01:56:53] [debug] [storage] [cio stream] new stream registered: cpu.0
[2020/11/29 01:56:53] [ info] [storage] version=1.0.6, initializing...
[2020/11/29 01:56:53] [ info] [storage] in-memory
[2020/11/29 01:56:53] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/11/29 01:56:53] [debug] [forward:forward.0] created event channels: read=20 write=21
[2020/11/29 01:56:53] [debug] [router] default match rule cpu.0:forward.0
[2020/11/29 01:56:53] [ info] [sp] stream processor started
[2020/11/29 01:56:57] [debug] [task] created task=0x7f9b4e8580a0 id=0 OK
[2020/11/29 01:56:57] [debug] [output:forward:forward.0] request 5525 bytes to flush
[2020/11/29 01:56:57] [debug] [upstream] KA connection #27 to fd00:7fff:0:2:9c43:9bff:fe00:bb:24000 is now available
[2020/11/29 01:56:57] [debug] [task] destroy task=0x7f9b4e8580a0 (task_id=0)
[2020/11/29 01:57:02] [debug] [task] created task=0x7f9b4e8580a0 id=0 OK
[2020/11/29 01:57:02] [debug] [output:forward:forward.0] request 4420 bytes to flush
Does anyone understand what the difference is and how i can rectify this problem?
I resolved this issue by taking the source from the latest fluent-bit project and compiling it. Then I replaced /usr/sbin/td-agent-bit with the newly built fluent-bit (as td-agent).

How to solve 504 HTTP Code in Scrapy-Splash

I use scrapy for a long time but now I need to use scrapy-splash for some reason.
I ran
docker run -it -p 8050:8050 --rm --name spider -v /etc/splash/proxy-profiles:/etc/splash/proxy-profiles scrapinghub/splash --max-timeout 3600
In /etc/splash/proxy-profiles/default.ini I write corporate proxy
Log:
[user#vir ~]$ sudo docker run -it -p 8050:8050 --rm --name spider -v /etc/splash/proxy-profiles:/etc/splash/proxy-profiles scrapinghub/splash --max-timeout 3600
2020-10-30 10:36:19+0000 [-] Log opened.
2020-10-30 10:36:19.324303 [-] Xvfb is started: ['Xvfb', ':1087332563', '-screen', '0', '1024x768x24', '-nolisten', 'tcp']
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-splash'
2020-10-30 10:36:19.496380 [-] Splash version: 3.5
2020-10-30 10:36:19.593826 [-] Qt 5.14.1, PyQt 5.14.2, WebKit 602.1, Chromium 77.0.3865.129, sip 4.19.22, Twisted 19.7.0, Lua 5.2
2020-10-30 10:36:19.594291 [-] Python 3.6.9 (default, Jul 17 2020, 12:50:27) [GCC 8.4.0]
2020-10-30 10:36:19.594536 [-] Open files limit: 1048576
2020-10-30 10:36:19.594911 [-] Can't bump open files limit
2020-10-30 10:36:19.627214 [-] proxy profiles support is enabled, proxy profiles path: /etc/splash/proxy-profiles
2020-10-30 10:36:19.627645 [-] memory cache: enabled, private mode: enabled, js cross-domain access: disabled
2020-10-30 10:36:19.880659 [-] verbosity=1, slots=20, argument_cache_max_entries=500, max-timeout=3600.0
2020-10-30 10:36:19.881199 [-] Web UI: enabled, Lua: enabled (sandbox: enabled), Webkit: enabled, Chromium: enabled
2020-10-30 10:36:19.882106 [-] Site starting on 8050
2020-10-30 10:36:19.882406 [-] Starting factory <twisted.web.server.Site object at 0x7fda701815f8>
2020-10-30 10:36:19.883106 [-] Server listening on http://0.0.0.0:8050
To find out my docker ip: ip addr show docker0 | grep -Po 'inet \K[\d.]+'
It is 172.17.0.1
I try
curl 'http://172.17.0.1:8050/render.html?url=https://www.google.com/' --noproxy "*"
This --noproxy "*" is due to a company works behind a proxy.
And I get the 200 response in Docker Log. It says that proxy I set in /etc/splash/proxy-profiles/default.ini works correct.
2020-10-30 13:15:25.586864 [-] "172.17.0.1" - - [30/Oct/2020:13:15:25 +0000] "GET /render.html?url=https://www.google.ru HTTP/1.1" 200 199991 "-" "curl/7.29.0"
2020-10-30 13:18:22.103537 [events] {"path": "/render.html", "rendertime": 0.7088587284088135, "maxrss": 214808, "load": [0.02, 0.13, 0.14], "fds": 57, "active": 0, "qsize": 0, "_id": 140576160225768, "method": "GET", "timestamp": 1604063902, "user-agent": "curl/7.29.0", "args": {"url": "https://www.google.ru", "uid": 140576160225768}, "status_code": 200, "client_ip": "172.17.0.1"}
2020-10-30 13:18:22.103956 [-] "172.17.0.1" - - [30/Oct/2020:13:18:21 +0000] "GET /render.html?url=https://www.google.ru HTTP/1.1" 200 199128 "-" "curl/7.29.0"
Great! That's what I wanted
But when I try to implement ScrapyRequest and run Scrapy with code below
yield SplashRequest(
args={
'wait': 0.5,
'timeout': 10,
'proxy': '<hidden for security reasons>',
},
splash_url="http://172.17.0.1:8050",
url="http://www.google.ru/",
endpoint='render.html',
method="GET",
callback=self.parse_splash,
headers=headers,
dont_filter=True,
)
I got error
2020-10-30 16:45:37 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-10-30 16:45:37 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.google.ru/ via http://172.17.0.1:8050/render.html> (failed 1 times): 504 Gateway Time-out
2020-10-30 16:46:37 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-10-30 16:46:39 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.google.ru/ via http://172.17.0.1:8050/render.html> (failed 2 times): 504 Gateway Time-out
2020-10-30 16:47:27 [scrapy.crawler] INFO: Received SIGINT, shutting down gracefully. Send again to force
2020-10-30 16:47:27 [scrapy.core.engine] INFO: Closing spider (shutdown)
2020-10-30 16:47:37 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-10-30 16:47:40 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET http://www.google.ru/ via http://172.17.0.1:8050/render.html> (failed 3 times): 504 Gateway Time-out
2020-10-30 16:47:40 [scrapy.core.engine] DEBUG: Crawled (504) <GET http://www.google.ru/ via http://172.17.0.1:8050/render.html> (referer: None)
Important note!
When I use just requests - it works!
When I use Scrapy requests - it works!
When I use ScrapySplash requests - it doesnt work!
Here is overriden settings of my Scrapy crawler:
custom_settings = {
'ITEM_PIPELINES': {
'spidermon.pipelines.DotcomPipeline': 400
},
'DUPEFILTER_DEBUG': True,
'HTTPERROR_ALLOWED_CODES': [503, 504],
'LOG_ENABLED': True,
'DOWNLOAD_DELAY': 0.5,
'DOWNLOADER_MIDDLEWARES': {
'scrapy_splash.SplashCookiesMiddleware': 723,
'scrapy_splash.SplashMiddleware': 725,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware': None,
},
'SPIDER_MIDDLEWARES': {
'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
},
'SPLASH_URL': "http://172.17.0.1:8050",
'DUPEFILTER_CLASS': 'scrapy_splash.SplashAwareDupeFilter',
'HTTPCACHE_STORAGE': 'scrapy_splash.SplashAwareFSCacheStorage',
}

Resources