I'm trying to get source code of a url but it gives me an error. Could you help me please?
curl -v http://www.segundamano.es/anuncios-madrid/ -m 10* About to connect() to www.segundamano.es port 80 (#0)
* Trying 195.77.179.69...
* Connected to www.segundamano.es (195.77.179.69) port 80 (#0)
> GET /anuncios-madrid/ HTTP/1.1
> User-Agent: curl/7.29.0
> Host: www.segundamano.es
> Accept: */*
>
* Empty reply from server
* Connection #0 to host www.segundamano.es left intact
curl: (52) Empty reply from server
Many thanks and sorry for my english!
It looks like this domain is actively blocking curl (and wget) requests, if you pass a browser's UserAgent it appears that you can get around this (curl and wget use the same command line arguments for user agent). For example:
This doesn't work:
C:\>wget http://www.segundamano.es/anuncios-madrid/
SYSTEM_WGETRC = c:/progra~1/wget/etc/wgetrc
syswgetrc = C:\Program Files (x86)\GnuWin32/etc/wgetrc
--2013-10-16 10:06:13-- http://www.segundamano.es/anuncios-madrid/
Resolving www.segundamano.es... 195.77.179.69, 213.4.96.70
Connecting to www.segundamano.es|195.77.179.69|:80... connected.
HTTP request sent, awaiting response... 502 Bad Gateway
2013-10-16 10:06:15 ERROR 502: Bad Gateway.
But this does:
C:\>wget --user-agent="Mozilla/5.0 (Windows NT 5.2; rv:2.0.1) Gecko/20100101 Firefox/4.0.1" http://www.segundamano.es/anuncios-madrid/
SYSTEM_WGETRC = c:/progra~1/wget/etc/wgetrc
syswgetrc = C:\Program Files (x86)\GnuWin32/etc/wgetrc
--2013-10-16 10:06:29-- http://www.segundamano.es/anuncios-madrid/
Resolving www.segundamano.es... 195.77.179.69, 213.4.96.70
Connecting to www.segundamano.es|195.77.179.69|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `index.html'
[<=>] 178,588 267K/s in 0.7s
2013-10-16 10:06:33 (267 KB/s) - `index.html' saved [178588]
Related
I have a Deluge client (in a docker container - that's likely irrelevant).
I want to be able to connect to the daemon from the outside world while having it behind a reverse proxy.
I don't necessarily need TLS, but I suspect http2 may require it.
What works:
connecting locally on the network to the Deluge RPC with a Deluge desktop, Android and WebUI client works well.
sending requests to the nginx server is OK (I can see logs as I hit nginx)
All the networking around (firewalls, port forwardings, DNS are fine)
What doesn't work:
Deluge client can't connect to the http server
nginx config:
server {
server_name deluge.example.com;
listen 58850;
location / {
proxy_pass grpc://localhost:58846;
}
ssl_certificate /etc/ssl/nginx/example.com.pem;
ssl_certificate_key /etc/ssl/nginx/example.com.key;
proxy_request_buffering off;
gzip off;
charset utf-8;
error_log /var/log/nginx/nginx_deluge.log debug;
}
Major edit:
As it turns out, I believed the JSON RPC and gRPC are more similar than just the "RPC" in the name. Hence my "original" issue "nginx deluge rpc doesn't work", is no longer relevant.
Unfortunately, the "same" issue still prevails. I still can't connect to the proxy even when using a regular HTTP proxy while I can make HTTP requests locally.
I will surely post an update or even an answer should I figure it out in the next days...
When I try to connect with the Deluge client, I get this error message in the log file:--
2022/06/14 16:59:55 [info] 1332115#1332115: *7 client sent invalid method while reading client request line, client: <REDACTED IPv4>, server: deluge.example.com, request: " Fu�Uq���U����a(wU=��_`. a��¹�(���O����f�"
2022/06/14 16:59:55 [debug] 1332115#1332115: *7 http finalize request: 400, "?" a:1, c:1
2022/06/14 16:59:55 [debug] 1332115#1332115: *7 event timer del: 17: 243303738
2022/06/14 16:59:55 [debug] 1332115#1332115: *7 http special response: 400, "?"
2022/06/14 16:59:55 [debug] 1332115#1332115: *7 http set discard body
2022/06/14 16:59:55 [debug] 1332115#1332115: *7 HTTP/1.1 400 Bad Request
Server: nginx/1.22.0
Date: Tue, 14 Jun 2022 16:59:55 GMT
Content-Type: text/html
Content-Length: 157
Connection: close
When I change the line listen 58850; to listen 58850 http2;, as I probably should, I get the following error: (log verbosity set to "debug")
2022/06/14 15:04:00 [info] 1007882#1007882: *3654 client sent invalid method while reading
client request line, client: <REDACTED IPv4>,
server: deluge.example.com, request: "x�;x��;%157?O/3/-�#�D��"
The gibberish there is seemingly identical when trying to connect from a different network from a different device. It was Dx�;x��;%157?O/3/-�#�E�, (there is a D as first character now) but all other attempts are again without the leading D.
or this error: (log verbosity set to "info")
2022/06/14 17:09:13 [info] 1348282#1348282: *14 invalid connection preface while processing HTTP/2 connection, client: <REDACTED IPv4>, server: 0.0.0.0:58850
I tried decoding the gibberish between various encodings, in hoping it would be just bad encoding of a better error message or a lead to a solution.
I looked through the first two pages of Google hoping the error messages were pointing me to a solution someone else has had to my problem.
environment:
Docker version 20.10.17, build 100c70180f
nginx version: nginx/1.22.0
deluged 2.0.5
libtorrent: 2.0.6.0
we have some concurrency issues, that I'd like to reproduce on my machine. On production we have a passenger instance running. To get concurrent request, I tried to start the rails app (Rails 3.2) with thin and a threaded option like this:
bundle exec thin --threaded -p 3000 --threadpool-size 50 start -e production
I also ran RAILS_ENV=production bundle exec rake assets:precompile to get everything like in production.
However, when I access localhost:3000 in my browser, I get part of the HTML (it displays), but then the browser runs into a timeout with GET http://localhost:3000/ net::ERR_EMPTY_RESPONSE and loading of the page stops.
When I stop thin with Ctrl-C, I get the following message multiple times:
Unexpected error while processing request: Attempt to unlock a mutex which is not locked
Anybody an idea, why the browser gets a timeout, or how to get concurrent requests working on a local machine with thin? It would also be no problem to try another server like puma, but a whole apache / passenger installation would be too much.
Update
I tried it without the browser, and just do a curl --verbose http://localhost:3000. The first time, I get the HTML back:
* Rebuilt URL to: http://localhost:3000/
* Trying ::1...
* connect to ::1 port 3000 failed: Connection refused
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 3000 (#0)
> GET / HTTP/1.1
> Host: localhost:3000
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: text/html; charset=utf-8
< Content-Length: 88748
< X-UA-Compatible: IE=Edge,chrome=1
< ETag: "d16fa9f8e279774f09c2172988a6d5a6"
< Cache-Control: max-age=0, private, must-revalidate
< Set-Cookie: _session_id=c23af3cc254b66789b635edfefd4a120; path=/; HttpOnly
< X-Request-Id: 539b87c3916cf6733fc9e91a05c4f8e9
< X-Runtime: 0.488556
< Date: Mon, 07 Dec 2015 13:33:26 GMT
< X-Rack-Cache: miss
< Connection: keep-alive
< Server: thin
<
<!DOCTYPE html>
... (more html)
But the second time I run the curl command, it times out:
* Rebuilt URL to: http://localhost:3000/
* Trying ::1...
* connect to ::1 port 3000 failed: Connection refused
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 3000 (#0)
> GET / HTTP/1.1
> Host: localhost:3000
> User-Agent: curl/7.43.0
> Accept: */*
>
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server
Unfortunately, no errors are written to the thin's output or log/production.log.
Completely wild speculative guess... but do you have partial caching enabled in production, and not able to reach the cache server (e.g., memcache or redis) from your local environment? It might (???) explain why the HTML only partly renders.
Did you update your config so that they point to all your local resources instead of trying to reach the production resources?
What I can think of happening here is that, one of your resource is probably un-reachable. Also try to reduce the log level to debug in config/environments/production.rb
I have a Rails app on Heroku using Expedited SSL (in case any of that matters).
Now I have permanent forwarding from example.com to https://www.example.com. Occasionally, and I only noticed it on Chrome, when I request the page http://example.com, I get a ERR_INSECURE_RESPONSE error page, then after few seconds, it takes me to the https://www.example.com page and loads just fine.
I ran the inspector for http insecured warnings in console, but nothing. I tried several times to do curl, and on one occasion I got an error.
Bashar:example bashar$ curl -v http://example.com
* Rebuilt URL to: http://example.com/
* Hostname was NOT found in DNS cache
* Trying xx.xx.xx.xx...
* Connected to example.com (xx.xx.xx.xx) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.37.0
> Host: example.com
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer
Bashar:example bashar$ curl -v http://example.com
* Rebuilt URL to: http://example.com/
* Hostname was NOT found in DNS cache
* Trying xx.xx.xx.xx...
* Connected to example.com (xx.xx.xx.xx) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.37.0
> Host: example.com
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Cache-Control: max-age=900
< Content-Type: text/html
< Location: https://www.example.com
* Server Microsoft-IIS/7.5 is not blacklisted
< Server: Microsoft-IIS/7.5
< X-AspNet-Version: 4.0.30319
< X-Powered-By: ASP.NET
< Date: Tue, 11 Aug 2015 09:15:46 GMT
< Content-Length: 0
< Age: 1
< Connection: keep-alive
<
* Connection #0 to host example.com left intact
I've read several discussions about the topic, but didn't find one exactly like this. Any idea?
I'm not sure if there is another, or better solution. However I found this answer by Brian Heroku SSL on root domain
to move from GoDaddy to DNSimple to do the trick. Can't see the problem anymore, and even better, the site works with and without www, and with http and https. Seems t
I look after www.lesscss.org. The source is here https://github.com/cloudhead/lesscss.org.
This is a thin web application and runs on heroku. Accessing the website in a browser is fine.
We have had a bug that curl -I lesscss.org gives a 400 request - https://github.com/cloudhead/less.js/issues/1232
and it does
$ curl -I lesscss.org
HTTP/1.1 400 Bad Request
Server: nginx
Date: Tue, 19 Mar 2013 01:15:50 GMT
Connection: close
I've done alot of searching and haven't found a reason why or how to rectify the above.. shouldn't it be a 302 redirect or a 200 ok?
[Edit]
with verbose option
$ curl -Iv http://www.lesscss.org
* About to connect() to www.lesscss.org port 80 (#0)
* Trying 107.21.106.77...
* 0x8001f140 is at send pipe head!
* STATE: CONNECT => WAITCONNECT handle 0x80057408; line 1032 (connection #0)
* Connected to www.lesscss.org (107.21.106.77) port 80 (#0)
* STATE: WAITCONNECT => DO handle 0x80057408; line 1151 (connection #0)
> HEAD / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: www.lesscss.org
> Accept: */*
>
* STATE: DO => DO_DONE handle 0x80057408; line 1236 (connection #0)
* STATE: DO_DONE => WAITPERFORM handle 0x80057408; line 1352 (connection #0)
* STATE: WAITPERFORM => PERFORM handle 0x80057408; line 1363 (connection #0)
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 400 Bad Request
HTTP/1.1 400 Bad Request
< Server: nginx
Server: nginx
< Date: Wed, 20 Mar 2013 09:34:37 GMT
Date: Wed, 20 Mar 2013 09:34:37 GMT
< Connection: keep-alive
Connection: keep-alive
<
* STATE: PERFORM => DONE handle 0x80057408; line 1533 (connection #0)
* Connection #0 to host www.lesscss.org left intact
* Expire cleared
The toto engine that you're using has this line in it:
return [400, {}, []] unless #request.get?
So anything that is not a get will result in a 400. Toto could probably afford to allow head requests through (and combine with Rack::Head to drop request bodies on the floor if needed)
With -I you're making HEAD requests and not GET. Remove the -I flag and it will work. You can see how it works with the -v flag:
curl -Iv lesscss.org
It's really tricky in our production environment( Red Hat Enterprise Linux Server release 5.4 (Tikanga) and curl 7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5) that the POST request become to GET request automatically. Below is the relative log provide by nginx.
cache: [GET /login] miss
url->http://localhost:8080/login
About to connect() to localhost port 8080
Expire at 1339680839 / 265363 (300000ms)
Trying 127.0.0.1... * connected
Connected to localhost (127.0.0.1) port 8080 /usr/local/lib/ruby/gems/1.8/gems/typhoeus-0.4.0/lib/typhoeus/multi.rb:141: [BUG] Segmentation fault ruby 1.8.7 (2012-02-08 patchlevel 358) [x86_64-linux]
2012/06/14 21:28:59 [error] 29829#0: *6031 upstream prematurely closed connection while reading response header from upstream, client: 127.0.0.1, server: localhost, request: "POST /login HTTP/1.1", upstream: "passenger:unix:/passenger_helper_server:", host: "127.0.0.1:8081", referrer: "http://127.0.0.1:8081/login"
In fact, we send a POST request to login, but it convert to GET actually.
In our development environment( which is Ubuntu12.04 and curl 7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3), everything goes ok.
Who can explain it?
After I upgraded the curl from 7.15 to 7.22. The problem has been solved.