I am trying to make a request to a website with a proxy using httparty like so:
def self.fetch_page_with_html_response(url, proxy_id)
proxy = Proxy.find(proxy_id)
request_options = {
http_proxyaddr: proxy.url, http_proxyport: proxy.port, http_proxyuser: proxy.username, http_proxypass: proxy.password,
headers: {"User-Agent" => proxy.user_agent}
}
response = HTTParty.get(url, request_options)
response
end
On certain websites my requests either hangs or returns an error page where the website is blocking me from fetching the page.
When i use these same proxy settings in my Chrome browser using an extension like SwitchyOmega the requests goes through fine and the page loads.
Is there any reason why the request would be getting blocked from my web server but not through my browser?
I even tested using the same user agent and providing the same exact headers my browser is sending.
There could be some reasons in your case.
Can you check if proxy works correctly?
Please send get request to https://api.ipify.org/ using your proxy in code.
If it returns correct ip address, then the proxy works.
Please disable javascript run in chrome settings. Then browse the website via proxy.
And check if website load correctly.
Because some websites render html and css using javascript.
Please feel free to reply me if you still need help.
Related
From the Electron renderer, I am accessing a local GraphQL endpoint served by a Django instance on my computer, which I'd like to do over HTTP, not HTTPS. But Electron's Chromium seems to intercept my fetch request and preemptively return a 307 redirect.
So if my fetch request is POST to http://local.myapp.com:3000/v1/graphql, then Chromium returns a 307 and forces a redirect to https://local.myapp.com:3000/v1/graphql, which fails because my server is listening on port 3000 and for my use case I can't do a local cert for local.myapp.com.
Theoretically the first insecure request should be hitting an nginx docker container listening on port 3000 without any SSL requirement. And nginx is proxying the request to a Hasura container. But I'm not even seeing the requests in the nginx access logs, so I'm pretty sure the request is being intercepted by Chromium.
I believe this StackOverflow comment summarizes well why this is happening: https://stackoverflow.com/a/34213531
Although I don't recall ever returning a Strict-Transport-Security header from my GraphQL endpoint or Django server.
I have tried the following code without success to turn off this Chromium behavior within my Electron app:
import { app, } from 'electron'
app.commandLine.appendSwitch('ignore-certificate-errors',)
app.commandLine.appendSwitch('allow-insecure-localhost', )
app.commandLine.appendSwitch('ignore-urlfetcher-cert-requests', )
app.commandLine.appendSwitch('allow-running-insecure-content', )
I have also tried setting the fetch options to include {redirect: 'manual'} and {redirect: 'error'}. I can prevent the redirect but that doesn't do me any good because I need to make a successful request to the endpoint to get my data.
I tried replacing the native fetch with electron-fetch (link) and cross-fetch (link) but there seems to be no change in behavior when I swap either of those out.
Edit: Also, making the request to my GraphQL outside of Electron with the exact same header and body info works fine (via Insomnia).
So I have a couple of questions:
Is there a way to programmatically view/clear the list of HSTS domains that is being used by Chromium within Electron?
Is there a better way to accomplish what I'm trying to do?
I think the issue might be from the server, most servers don't allow HTTP in any possible way, they'll drop the data transfer and redirect you to HTTPS and there's a clear reason why they would do that.
Imagine you have an app that connects through HTTPS to send your API in return for some data, if someone just changed the https:// to http:// that'd mean the data will be sent un-encrypted and no matter what you do with your API key, it'll be exposed, that's why the servers don't ever allow any HTTP request, they don't accept even a single bit of data.
I could think of two solutions.
Chromium is not the reason for the redirect, our Django instance might be configured as production or with HTTPS listeners.
Nginx might be the one who's doing the redirecting (having a little bit of SSL def on the configuration)
Last but not least, just generate a cert with OpenSSL (on host http://local.myapp.com:3000/) note: include the port and use that on your Django instance. You can trust the certificate so that it could work everywhere on your computer.
I have successfully set everything up but unfortunately my GraphQL endpoint is not at the same location as the website that serves the client side.
I know this because in the error console of the browser it says :
http://localhost:3000/graphql Failed to load resource: the server responded with a status of 404 (Not Found)
three times then give up.
The page that I am using Relay is indeed at http://localhost:3000/ but my GraphQL endpoint is at http://localhost:5000/graphql. Looks like it uses the current URL then automatically append /graphql to it. How can I instruct Relay to get data from other place?
Ok, I found it. (https://facebook.github.io/relay/docs/guides-network-layer.html)
Relay.injectNetworkLayer(
new Relay.DefaultNetworkLayer('http://example.com/graphql')
);
And just in case you are running this on localhost it is still subjected to CORS because it is on different port. In my case I am using an Express server for GraphQL endpoint so I used cors middleware to whitelist my other page.
Using fiddler I can intercept an ajax post request and with request composer I can resend the same request, causing the server to respond it normally. It doesnt matter if protocol is http or https (fiddler deciphers HTTPS traffic), with tools like Fiddler it is just possible.
On web applications side is it possible to understand or prevent such requests? How?
No. There's no way to prevent this. URLs are meant to be accessed. If it shouldn't be accessed, don't put it online or require some method of authentication. Some clients may pass an identifiable User-Agent header that can be restricted, and Fiddler probably does as well. However, the whole point of tools like Fiddler is to be able to make any type of custom request, which includes this User-Agent string. So, even if you block the default User-Agent, there's nothing stopping the user of Fiddler from changing the User-Agent to something that won't be blocked.
I am trying to check the incomming request to my server. Another server which has hosted MVC application. An action method is sending some data to my server. I am using Fiddler. But somehow it is not showing the incoming request.
Below mentioned are my settings in Fiddler Custom Rules..
static function OnBeforeRequest(oSession: Session) {
if (oSession.host.toLowerCase() == "IP Address:8888")
oSession.host = "IP Address:82";
}
Below mentioned are my Fiddler Options.
Am I missing anything ?
It sounds like you're trying to use Fiddler as a reverse proxy. You should read the steps at http://www.fiddler2.com/r/?reverseproxy. The biggest thing to understand is that when running as a reverse proxy, you only see traffic in Fiddler if the client's URL is changed to point at Fiddler.
If it is ssl connection then you need to enable option 'capture https connection' from 'https' tab. Did you try to invoke request from other browser like chrome ? Does fiddler capture anything?
You don't need custom rule for this case. It should work if you enable these settings. I have faced only some problems in other browsers like FF.
I'm not sure I can answer your question fully without knowing a few additional pieces of information.
If the request being made is not a HTTP request, Fiddler will not be able to handle it.
Also, if you're using the loopback address localhost then Fiddler may not be able to find it.
So I have a controller action that renders json.. I can visit the url in the browser and see the json data, verifying that the route is working properly...
Yet, if I do:
uri = URI("#{request.protocol}#{request.host_with_port}/my_controller/action")
Net::HTTP.get(uri)
I get "Timeout::Error: Timeout::Error"
... ?
You're using a single-threaded HTTP server, i.e. Webrick. This means that it will only be able to serve one request at a time. You're attempting to make a request to the webserver from within the webserver itself. It won't be able to complete this action because of now hopefully obvious reasons.
Use a different web server, such as Thin, that would allow for this, or choose a different way to do this.