I'm creating a program to download web pages using the Http module of FSharp.Data. However, the module doesn't support setting http proxy server. In C# there is
_httpWebRequest.Proxy =
new System.Net.WebProxy("http://proxy.myCompany.com:80", true);
I tried to download the file from https://github.com/fsharp/FSharp.Data/blob/master/src/Library/Http.fs and use it directly in my F# project. However, the type of response changed from string to HttpResponse after I call the Http.Request from the downloaded file.
let response =
Http.Request (
url,
query=["userid", user; "password", password; "login", "Sign+On"],
meth="POST",
cookieContainer = cc)
What's the best way to extend the Http module with proxy support?
In FSharp.Data 2.0, you can pass the parameter customizeHttpRequest of type HttpWebRequest->HttpWebRequest to set the proxy like this:
Http.Request (
url,
query=["userid", user; "password", password; "login", "Sign+On"],
meth="POST",
cookieContainer = cc,
customizeHttpRequest = (fun req -> req.Proxy <- WebProxy("http://proxy.myCompany.com:80", true); req))
In the new version (upcoming release), we are renaming the current Http.Request to Http.RequestString and the current Http.RequestDetailed to Http.Request. This is a breaking change, but we thing it makes much more sense (and fits better with standard .NET naming). If you just want to copy the old file, you can always get the older version of the code from the appropriate branch on GitHub (e.g. Http.fs # tag v1.1.10).
However, I think that supporting HTTP proxies would be a great addition to the library. So the best thing to do would be to fork the project to your GitHub, add the feature and submit a pull request! I think that just adding an optional ?proxy parameter to the two methods and propagating the information to the underlying HttpWebRequestwould be the best way to do this.
The only tricky thing is that Http.Request should work on multiple versions of .NET (including Windows Phone, Silverlight, etc.) so you may need to check which of them actually support specifying the proxy.
If you do not have the time for helping out, then please submit a GitHub issue.
Have you tried overriding the proxy globally with WebRequest.DefaultWebProxy = new System.Net.WebProxy("http://proxy.myCompany.com:80", true)?
Related
When attempting to perform an HTTP post via Swagger (using the Swagger Type Provider) I get a 401 Unauthorized.
I suspect the answer to this might be UseDefaultCredentials which isn't exposed when inheriting from the SwaggerProvider.Internal.ProvidedSwaggerBaseType. So I was thinking that an upcast :> might help but posts I've seen elsewhere indicate naïveté.
Also, updating the config for fsi.exe to include the following proved wishful:
<system.net>
<defaultProxy enabled="true" useDefaultCredentials="true">
<proxy usesystemdefault="True" />
</defaultProxy>
</system.net>
Might anyone have a simple answer?
Thanks
Schema access
SwaggerProvider assumes that Swagger schema is easy-accessible and can be downloaded using simple Http request without authentication
type PetStore = SwaggerProvider<"http://petstore.swagger.io/v2/swagger.json">
user also can add any HTTP header to schema-request like this
type PetStore = SwaggerProvider<"http://petstore.swagger.io/v2/swagger.json", "Content-Type=application/json">
It can be Authorization: Basic QWxhZGRpbjpPcGVuU2VzYW1l header, but this literal string will be hard-coded in source code.
If schema-request requires more complex authentication, will be easier to download it and put it near source code
[<Literal>]
let schemaPath = __SOURCE_DIRECTORY__ + "/PetStore.Swagger.json"
type PetStore = SwaggerProvider<schemaPath>
Requests control
When schema are in place you have a full control over all HTTP request to the server using CustomizeHttpRequest parameter in provided type.
For example, if you want to use default credentials:
let store =
PetStore(
CustomizeHttpRequest=
fun (req:System.Net.HttpWebRequest) ->
req.UseDefaultCredentials <- true
req)
You are free to modify web request as you need:
Use default credentials
Specify runtime Credentials
Add headers to HTTP request
Add cookies to HTTP request
Break request ;) and etc.
CustomizeHttpRequest will be called in the run-time for each request, after SwaggerProvider built it and before call to the server. So you can change anything you want.
I signed up for the yellow pages.com API program found here: https://publisher.yp.com/home.
I went to make a call like this and I am getting back JSON in the browser:
http://pubapi.atti.com/search-api/search/devapi/search?term=Jumbo+China+5&searchloc=6108+Falls+Of+Neuse+Rd+27609&format=json&key=ZZZZZZZZ
When I take the json results and put it into Json2CSharp, it renders fine. When I try and load it into a type provider:
type RestaurantListingJson = JsonProvider< #"http://pubapi.atti.com/search-api/search/devapi/search?term=Jumbo+China+5&searchloc=6108+Falls+Of+Neuse+Rd+27609&format=json&key=ZZZZZZZ">
I get a 400
Looking at fiddler, I see
"User agent is a required field"
Has anyone run into this before? How do I add a user agent to a type provider?
Thanks in advance
I have not created an account, so I could not try this - but if the error message says "user agent is a required field", then I guess that the service requires setting the User-Agent header of the HTTP request.
This is not supported in static parameters of the JsonProvider, so the best way to get this to work would be to download the sample JSON, save it to a local file (say yp.json) and then use that to create the type provider:
type Yp = JsonProvider<"yp.json">
To actually download some data (when you want to make a request), you can use Http.RequestString which takes headers - there you can specify any required headers including User-Agent:
let response =
Http.RequestString("http://httpbin.org/user-agent", headers=["user-agent", "test"])
Then you can load the data using Yp.Parse(response) (rather than using the Load method directly to request a URL which would not let you specify the header).
The latest version of F# Data now always sends user agent and accept headers, so this should now work directly:
type RestaurantListingJson = JsonProvider<"http://pubapi.atti.com/search-api/search/devapi/search?term=Jumbo+China+5&searchloc=6108+Falls+Of+Neuse+Rd+27609&format=json&key=ZZZZZZZ">
Whatever I do, I always get a
Symfony\Component\HttpKernel\Exception\NotFoundHttpException: "No
route found for [...]"
in $crawler->text(), when I try to request an external URL with $crawler = $client->request('GET', 'http://anotherdomain.com');.
I want to do that because I'm using another virtualHost to render some pages with Symfony 1.2 and some others with Symfony 2.3.
I also tried to
$client = static::createClient(array(), array('HTTP_HOST' => 'anotherdomain.com'));
$client->followRedirects(true);
But it's always trying to render it whithin Symfony 2.
It's not possible, because $client actually doesn't send any http request (you may notice that when you try run your "functional" test with www server disabled - they still should work). Instead of that it simulates http request and run normal Symfony's dispatching.
How do I invoke a Service Operation in WCF from iOS?
I have a Service Operation defined in my WCF Data Service (tied to a stored procedure in my DB schema) that I need to invoke from iOS. Say I've got the following declaration in my .svc.cs file:
[WebInvoke(RequestFormat = WebMessageFormat.Json, ResponseFormat = WebMessageFormat.Json, BodyStyle = WebMessageBodyStyle.WrappedRequest)]
public IQueryable<Foo> GetFoos(int param1, DateTime param2, string param3)
{
return CurrentDataSource.GetFoos(param1, param2, param3).AsQueryable();
}
And I've got it set up with the proper rights in InitializeService:
config.SetServiceOperationAccessRule("GetFoos", ServiceOperationRights.AllRead);
When I try to invoke this via HTTP POST from iOS, I get back an error wrapped in JSON:
Bad Request - Error in query syntax.
It seems like it doesn't like how I'm passing my parameters. I'm passing them JSON-encoded (using NSJSONSerialization to turn an NSDictionary into a JSON string) in the request body of a POST request. The same method works on another web service (.svc) not connected to WCF that has operations annotated the same way.
An answer to another question of mine in a similar vein suggests that data formats can be negotiated between client and server, and I've read that dates are a pain to format, so maybe it's my DateTime parameter that's a problem. But I've tried both the JSON format (\/Date(836438400000)\/ and /Date(836438400000)/) and the JSON Light format (1996-07-16T00:00:00) to no avail.
So my question is this: what is the proper way to invoke this operation? If I need to have my app tell the server what format to expect, how do I do that?
Update: I tried using the format datetime'1996-07-16T00:00:00' as mentioned in this question. Same error.
Update 2: The MSDN page for Service Operations seems to suggest that nothing besides Method = "POST" is supported when annotating the WebInvoke for a Service Operation. I tried removing everything from what is quoted in the above code and setting the method to POST. Same error.
Update 3: On Pawel's suggestion, I made a new Service Operation on my Data Service just like this:
[WebInvoke(Method = "POST")]
public IQueryable<string> GetFoos()
{
List<string> foos = new List<string>();
foos.Add("bar");
return foos.AsQueryable();
}
I was able to make it work in Fiddler's Composer pane by setting the method to POST, adding accept:application/json;charset=utf-8 and Content-Length:0 to the headers. Then I added a single int parameter to the operation (called param1). I set the body of my request in Fiddler to {"param1":"1"} and ran it (and Fiddler automatically updated my content-length header), and got the same error. I changed the type of my parameter to string and ran my request again and it worked. So my problem seems to be non-string types.
You need to send parameters in the Url and not in the request body.
I want to create a proxy controller in grails, something that just takes whatever is passed in based on a url mapping, records what was asked for, sends the request to another server, records the response, and send the response back to the browser.
I'm having trouble with when the request has an odd file extension (.gif) or no file extension (/xxx?sdcscd)
My url mapping is:
"/proxy/$target**"
and I've attempted (per an answer to another question):
def targetURL = params.target
if (!FilenameUtils.getExtension(targetURL) && request.format) {
targetURL += ".${response.format}"
}
but this usually appends .html and never the .gif or ?csdcsd
Not sure what to do as I might just write the thing in straight Java
Actually, the real answer was sitting in the post you linked to previously all along, by Peter Ledbrook:
Disable file extension truncation by adding this line to grails-app/conf/Config.groovy:
grails.mime.file.extensions = false
This will disable the usage of file extensions for format, but will leave the file extension on params.target. You can completely ignore response.format!