Hi I have browsed some data and captured the traces in Wiresshark.
Is there anyway or filter that we can get all the get requests and there responses only in Wireshark.
You can filter Wireshark's output using many criterias, like the protocols used, source and destination address/ports, and so on. It really depends on what kind of requests/responses you want to filter. For further information, take a look at
https://wiki.wireshark.org/DisplayFilters.
HTTP
Display filter:
http.request || http.response
Related
First you must know I'm a total beginner, I'm trying to learn so I almost don't know anything.
On the basic page of the API, there is a curl command used as an example to show us how to make requests.
I'm using Ruby on Rails so I used "curl-to-ruby" website to translate it, but it did not work as expected.
I wanted it to show me this :
uri = URI.parse("REQUEST_URL")
response = JSON.parse(Net::HTTP.get(uri))
Instead I got this :
uri = URI.parse("REQUEST_URL")
response = Net:HTTP.get_response(uri)
I don't understand any of this, I thought I wouldn't need to and just use "curl-to-ruby", but apparently I really need to get this.
Would you please try to explain me ?
Or give me links ?
Or matters to read (curl, API, http) ?
Thank you very much, have a nice day.
It's because that command doesn't return just the content, it returns the whole HTTP response object including headers and body. You need to extract the response body and parse that using JSON.parse(), e.g.
JSON.parse(response.body)
See documentation here: https://docs.ruby-lang.org/en/2.0.0/Net/HTTP.html#method-c-get_response
(Also, there is nothing in the cURL command which would hint to the converter that the content-type of the response was expected to be JSON (e.g. perhaps an "accepts" header or something), so even if it were able to produce extra code adding the JSON.parse part, it has no way of knowing that it would be appropriate to do so in this case.)
I am using two different HTTP POST utilities (poster out of Firefox as well as Python requests API) to post a simple SPARQL insert to Virtuoso.
My URL is: http://localhost:8890/sparql
My request parameters are:
default-graph-uri: <MY_GRAPH>
should-sponge: soft
debug: on
timeout:
format: application/xml
save: display
fname:
I put the actual SPARQL (INSERT DATA { GRAPH...) in the content of the message.
I tried different content types, none of which worked. I do get 200 but the response is in HTML even though the above parameter set specifies application/xml, however, no data is inserted. When I try content type of text/turtle, I get 409 Invalid Path, which is also referenced in this post.
I can successfully do HTTP GET, however, that has a payload length limitation which I would like to exceed for performance reasons. The only difference with the GET is that the SPARQL goes in the URL under query parameter and the POST should enable a much larger payload in the message content, by including multiple triples in the same request, not just one (I have 100s of 1000s of inserts). I was trying to follow this documentation page.
I stopped by this question days ago trying to achieve the same with curl. Since it is a powerful (and far more convenient) alternative to browser extensions, here is the formulation that eventually proved successful:
curl -X POST \
-H "Content-Type:application/sparql-update" \
-H "Accept:text/html" \
--data "select distinct ?Concept where {[] a ?Concept} LIMIT 100" http://localhost:8890/sparql
More details on the headers in this thread.
If you are using python, I would avoid using the requests library. There are some dedicated libraries for RDF which abstract the process and make your life easier.
Try:
SPARQLWrapper
RDFLib
They are both form the same family of packages from rdflib
Based on experience, I find the SPARQLWrapper significantly simpler and easier to use for your use case. It's an abstracted version of RDFLib. The docs suggest something like this could work:
from SPARQLWrapper import SPARQLWrapper, POST
sparql = SPARQLWrapper("https://example.org/sparql")
sparql.setCredentials("some-login", "some-password") # if required
sparql.setMethod(POST) # this is the crucial option
sparql.setQuery("""
<QUERY GOES HERE>
""".format(PARSE SOME VARS INTO THE QUERY HERE IF YOU WANT)
)
results = sparql.query()
print results.response.read()
Make sure you add the option for POST. You should be doing bulk I/O in no time :).
There are many aspects to this "question" making it difficult to provide a simple answer, suitable to this site. This is one of the reasons I suggested the mailing list, which is better suited to conversational and/or multi-facet assistance.
Have you tried using curl as most of our examples do?
Looking at the Poster page on Mozilla Add-Ons, I see that you may need to manually add a ? to the end of your target URI -- so http://localhost:8890/sparql? rather than http://localhost:8890/sparql -- and it's not clear whether you've done that in your testing. On the project page, I also note its last commit was in 2012, and there are a great many open issues.
I'm not at all familiar with Python, so I've not dug in there.
Have you tried setting an Accept: header? This can have significant impact on the content returned by the server.
If I understand your described efforts correctly, your format: query parameter should be output-format:, and its value should not be application/xml but one of the supported formats listed in the documentation.
Neither the virtuoso-users post you referenced nor this question have enough detail to analyze the cause of the 409 Invalid Path error. Explicit details that allow us to reproduce this result would be helpful, optimally in a distinct thread.
This seems to be a Virtuoso specific issue. You can only post a query by using content type "application/sparql-update" instead of "application/sparql-query" which is common.
The request is done as follows with Python:
headers = {
'Content-Type': 'application/sparql-update',
'Accept': 'application/json'
}
s = Session();
s.mount(server_url, HTTPAdapter(max_retries=1))
response: Response = s.post(server_url, data=<sparql_string>, headers=headers, timeout=100)
return response.json();
Is there any way to log the raw http requests/responses in rails?
that is, the plain text http message, including the body, even if the body is binary
It depends a bit on what exactly you want to log. But I believe raw_post is what you are looking for. This will return the raw POST body. You can call this method in any of your methods in the controller.
A similar method exist for getting the GET params: raw_params
If that isn't good enough then a middleware layer might help.
I am trying to pass multiple values for a parameter in a POST request body as follows
var1=1&var2=2&var34=3,4&var5=5
I've tried several ways to pass var34 as a string of values 3 and 4 but still not working. Need some help.
Thanks!!!
This isn't really a question about Fiddler, so it's not clear what you're asking specifically.
The server interprets POST data according to its own rules, and there's no standard for handling duplicate name/values in urlencoded data.
Some servers would accept var1=1&var2=2&var34=3,4&var5=5 as you've used, while some would prefer var1=1&var2=2&var34=3&var34=4&var5=5.
What error or problem are you encountering?
It turns out that it's the server (I use R) side that I need to adjust the codes to accommodate the POST request. It has nothing to do with the request. Thank you so much for the suggestion!
I am using "/reader/api/0/stream/items/ids" API to get the item ids for sources that I want.
I have quite a number of sources, so I repeated "s=" parameter to include in the api url.
However, google has given me an error of "URL is too long".
So the question is that How can I solve it so that I just use one time api call to get item ids for that many sources?
Thanks
It seems that /reader/api/0/stream/items/ids path supports a POST method. This means the amount of data you could pass by using POST verb is much more than by using a query string and a GET method.
So use https://www.google.com/reader/api/0/stream/items/ids URL for the post, and pass your query string as a post data. Don't forget to include an action token(T) which is required for POST requests.