Telegraf plugin:
HTTP Input Plugin
I'm trying to use telegraf to collect data from an vendor API.
test.conf file looks like this:
[[inputs.http]] urls = ["https://10.10.10.10"] method = "POST" body = '{"F_":"LOGIN","DATA":{"ID":"user","PWD":"password"}}'
When debugging i can see this is the error i get:
[inputs.http] Error in plugin: [url=https://10.10.10.10]: received status code 411 (Length Required), expected any value out of [200]
The API documentation for my vendor API states that the field "Content-Length" and "Host" are mandatory, but I can find now way to enable that in the plugin.
I have also tried using the http_response plugin, and I am able to get the JSON reply, but unfortuately i have not been able to find a good way to get the response_body_field JSON appended to the influxdb.
Does anybody know if either:
I can enable the two headers dynamicly(The POST lenght will vary)
or:
use the response_body_field from http_response and smoothly parse it to the influxdb?
Related
I am trying to create a proof on concept using the TICK stack for monitoring. I have the helloworld stack running and showing CPU/Docker metrics.
I am trying to use the telegraf http input plugin to pull from an http endpoint:
From the docs i have simply configured the URL, GET and type (Set to json)
[[inputs.http]]
## One or more URLs from which to read formatted metrics
urls = [
"http://localhost:500/Queues"
]
method = "GET"
data_format = "json"
However nothing appears in Influx/Chronograf.
I can modify the endpoint to suit any changes there, but what am i doing wrong in telegraf config ?
I think I had the same struggle. For me the following conf worked:
[[inputs.http]]
name_override ="restservice_health"
urls = [
"https://localhost:5001/health"
]
method = "GET"
data_format = "value"
data_type = "string"
In this way, it appeared in Influxdb under the name "restservice_health" (allthough this option is not important for the example, so you could leave it out).
First, you would have to look at the result of the http://localhost:500/Queues request to make sure that it's a valid JSON object.
Then, depending on what is returned from that endpoint, you may have to configure the JSON parser, for example by setting json_query to a GJSON query to navigate the JSON response to the data you need.
I am getting following error whenever Jmeter gets a | (pipe) symbol in the URL since the "pipe" symbol is not allowed in URL. Is there any way to convert the |(pipes) in URL to %7C automatically?
Response code: Non HTTP response code: java.net.URISyntaxException
Assuming the pipe is in an existing parameter try the following:
Check "Encode?" box in "Send Parameters with the Request" input of HTTP Request
Wrap your variable in __urlEncode function as ${__urlencode(queryTerm)}
Use Beanshell Pre Processor as a child of HTTP Request with the following code:
vars.put("queryTerm", URLEncoder.encode(vars.get("queryTerm")));
This should convert the pipes automatically.
Got the instructions from here: Jmeter - Configure Keywords with spaces using CSV File Configuration
First - my question:
When accessing the Quickbooks API, v3 (as has been forced on me as of this weekend by Intuit) I am trying to access Journal Entries (but the following problem persists across any other query) and I'm trying to use the prescribed query?query=SELECT * FROM JournalEntry (what?).
https://qb.sbfinance.intuit.com/v3/company/<id>/query?query=SELECT * FROM JournalEntry
I get as result:
{"Fault":{"Error":[{"Message":"message=Exception authenticating OAuth; errorCode=003200; statusCode=401","code":"3200"}],"type":"AUTHENTICATION"},"requestId":"6f5e5f14af7d4867ad0d8f639ade7d04","time":"2013-11-12T16:10:44.724Z"}
Which, yes, tells me that there was an error with authentication. However, when I access a URL that doesn't include this ridiculous query syntax, everything works fine:
https://qb.sbfinance.intuit.com/v3/company/<id>/journalentry/<id>
I had a similar error when accessing the v2 API, and that was bad formatting on my part, but I don't see what's wrong with my query.
And because my code for generating the authentication tokens etc is identical for both types of request, I doubt that the problem is with how I'm authenticating. Similarly "exception" tells me that there's something going wrong that the API isn't identifying. Probably a formatting of the URL that is going wrong.
I've tried replacing the query URL spaces with both a '+' and a '%20', which returns the same error.
I'm using python and rauth. The code works fine for v2 (but that was deprecated over the weekend without warning, and now is no longer documented).
As a bonus, and because apparently this is Intuit's primary mode of communication with their clients: I'm shocked that Intuit no longer has private support tickets available on their website, and that they rely on a community environment like SO to provide support. The least they could do is provide their own support. Especially if we're paying for use of the API. This is absolutely shocking.
On top of that, the API returns inconsistent responses (the same request will return an error or a valid result, depending on... no change at all). An error I have reported through their support tickets, and they have duly ignored.
Oh, and the documentation says to use
https://quickbooks.api.intuit.com/v3/v3/company/companyID/query?query=selectStmt
while the API Explorer uses:
https://qb.sbfinance.intuit.com/v3/company/<id>/query?query=SELECT * FROM JournalEntry
Anyone know which one I should actually use?
Edit
For the response that is failing, my request headers are:
{
'Content-Length': u'62',
'Accept-Encoding': 'gzip,
deflate,
compress',
'accept': 'application/json',
'User-Agent': 'python-requests/1.2.3CPython/2.7.5Darwin/13.0.0',
'Content-Type': 'application/x-www-form-urlencoded',
'authorization': 'OAuthrealm="<companyId>",
oauth_nonce="3ad98c5f71bc9f102cc31ac9815cb6d08994454e",
oauth_timestamp="1384280420",
oauth_consumer_key="<consumerKey>",
oauth_signature_method="HMAC-SHA1",
oauth_version="1.0",
oauth_token="<oauthToken>",
oauth_signature="<oauthSignature"'
}
My url is:
https://quickbooks.api.intuit.com/v3/company/<id>/query?query=SELECT+*+FROM+JournalEntry&
And my response headers are:
{'content-length': '227', 'server': 'Apache/2.2.22 (Unix)', 'connection': 'close', 'date': 'Tue, 12 Nov 2013 18:20:20 GMT', 'content-type': 'application/json;charset=ISO-8859-1', 'www-authenticate': 'OAuth oauth_problem="signature_invalid"'}
My signature hashing function is correct. It's the standard function used by Rauth, and works fine for more standard API calls (that don't have spaces or SQL select queries in them).
Pass the URL to your HTTP call without encoding:
URL = https://quickbooks.api.intuit.com/v3/company/123456789/query?query="Select * from Customer"
But to build the signature, separate the parameters from the URL, then encode separately, you should get:
"GET" + "&" +
URLEncode(https://quickbooks.api.intuit.com/v3/company/123456789/query) + "&" +
URLEncode(query=Select%20%2A%20from%20Customer), where Select%20%2A%20from%20Customer is the encoding of Select * from Customer
Note the SQL gets encoded a second time, when generating the signature.
Et voila ! I spent a week on this, I know what I'm talking about.
(notations are from VBA language, so replace as appropriate)
It turns out that the actual problem is that the Quickbooks documentation is wrong as of this writing (2013/11/14).
The documentation says that the query URL expects a GET request, which is not the case. This works when submitting SELECT statement as part of the body of a POST request.
See here for more details: https://intuitpartnerplatform.lc.intuit.com/questions/786661-python-script-to-integrate-with-quickbook
I had tried this API call using Java devkit.
JournalEntry je = GenerateQuery.createQueryEntity(JournalEntry.class);
String jeQuery = select($(je)).generate();
System.out.println("Query - " + jeQuery);
QueryResult JournalEntryRes = service.executeQuery(jeQuery);
Request URI : https://quickbooks.api.intuit.com/v3/company/688779980/query?query=SELECT+*+FROM+JournalEntry&
Response XML
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<IntuitResponse xmlns="http://schema.intuit.com/finance/v3" time="2013-11-12T09:50:39.836-08:00">
<QueryResponse startPosition="1" maxResults="1" totalCount="1">
<JournalEntry domain="QBO" sparse="false">
<Id>22734</Id>
<SyncToken>0</SyncToken>
<MetaData>
<CreateTime>2013-10-15T08:42:12-07:00</CreateTime>
<LastUpdatedTime>2013-10-15T08:42:12-07:00</LastUpdatedTime>
</MetaData>
<TxnDate>2013-10-15</TxnDate>
<Line>
<Id>0</Id>
<Amount>100.00</Amount>
<DetailType>JournalEntryLineDetail</DetailType>
<JournalEntryLineDetail>
<PostingType>Debit</PostingType>
<AccountRef name="Advertising">9</AccountRef>
</JournalEntryLineDetail>
</Line>
<Line>
<Id>1</Id>
<Amount>100.00</Amount>
<DetailType>JournalEntryLineDetail</DetailType>
<JournalEntryLineDetail>
<PostingType>Credit</PostingType>
<AccountRef name="Advertising">9</AccountRef>
</JournalEntryLineDetail>
</Line>
<Adjustment>false</Adjustment>
</JournalEntry>
</QueryResponse>
</IntuitResponse>
You can try this call from V3 QBO ApiExplorer as well.
Query - SELECT * FROM JournalEntry
Thanks
you need to encode the query, but not the whole url
https://quickbooks.api.intuit.com/v3/company/123456789/query?query=" & URLEncode("Select * from Customer")
see sample explained here :
https://developer.intuit.com/docs/0100_quickbooks_online/0300_references/0000_programming_guide/0050_data_queries
I'm trying out http requests to download a pdf file from google docs using google document list API and OAuth 1.0. I'm not using any external api for oauth or google docs.
Following the documentation, I obtained download URL for the pdf which works fine when placed in a browser.
According to documentation I should send a request that looks like this:
GET https://doc-04-20-docs.googleusercontent.com/docs/secure/m7an0emtau/WJm12345/YzI2Y2ExYWVm?h=16655626&e=download&gd=true
However, the download URL has something funny going on with the paremeters, it looks like this:
https://doc-00-00-docs.googleusercontent.com/docs/securesc/5ud8e...tMzQ?h=15287211447292764666&\;e=download&\;gd=true
(in the url '&\;' is actually without '\' but I put it here in the post to avoid escaping it as '&').
So what is the case here; do I have 3 parameters h,e,gd or do I have one parameter h with value 15287211447292764666&ae=download&gd=true, or maybe I have the following 3 param-value pairs: h = 15287211447292764666, amp;e = download, amp;gd = true (which I think is the case and it seems like a bug)?
In order to form a proper http request I need to know exectly what are the parameters names and values, however the download URL I have is confusing. Moreover, if the params names are h,amp;e and amp;gd, is the request containing those params valid for obtaining file content (if not it seems like a bug).
I didn't have problems downloading and uploading documents (msword docs) and my scope for downloading a file is correct.
I experimented with different requests a lot. When I treat the 3 parameters (h,e,gd) separetaly I get Unauthorized 401. If I assume that I have only one parameter - h with value 15287211447292764666&ae=download&gd=true I get 500 Internal Server Error (google api states: 'An unexpected error has occurred in the API.','If the problem persists, please post in the forum.').
If I don't put any paremeters at all or I put 3 parameters -h,amp;e,amp;gd, I get 302 Found. I tried following the redirections sending more requests but I still couldn't get the actual pdf content. I also experimented in OAuth Playground and it seems it's not working as it's supposed to neither. Sending get request in OAuth with the download URL responds with 302 Found instead of responding with the PDF content.
What is going on here? How can I obtain the pdf content in a response? Please help.
I have experimented same issue with oAuth2 (error 401).
Solved by inserting the oAuth2 token in request header and not in URL.
I have replaced &access_token=<token> in the URL by setRequestHeader("Authorization", "Bearer <token>" )
I want to build a small script in python which needs to fetch an url. The server is a kind of crappy though and replies pure ASCII without any headers.
When I try:
import urllib.request
response = urllib.request.urlopen(url)
print(response.read())
I obtain a http.client.BadStatusLine: 100 error because this isn't a properly formatted HTTP response.
Is there another way to fetch an url and get the raw content, without trying to parse the response?
Thanks
It's difficult to answer your direct question without a bit more information; not knowing exactly how the (web) server in question is broken.
That said, you might try using something a bit lower-level, a socket for example. Here's one way (python2.x style, and untested):
#!/usr/bin/env python
import socket
from urlparse import urlparse
def geturl(url, timeout=10, receive_buffer=4096):
parsed = urlparse(url)
try:
host, port = parsed.netloc.split(':')
except ValueError:
host, port = parsed.netloc, 80
sock = socket.create_connection((host, port), timeout)
sock.sendall('GET %s HTTP/1.0\n\n' % parsed.path)
response = [sock.recv(receive_buffer)]
while response[-1]:
response.append(sock.recv(receive_buffer))
return ''.join(response)
print geturl('http://www.example.com/') #<- the trailing / is needed if no
other path element is present
And here's a stab at a python3.2 conversion (you may not need to decode from bytes, if writing the response to a file for example):
#!/usr/bin/env python
import socket
from urllib.parse import urlparse
ENCODING = 'ascii'
def geturl(url, timeout=10, receive_buffer=4096):
parsed = urlparse(url)
try:
host, port = parsed.netloc.split(':')
except ValueError:
host, port = parsed.netloc, 80
sock = socket.create_connection((host, port), timeout)
method = 'GET %s HTTP/1.0\n\n' % parsed.path
sock.sendall(bytes(method, ENCODING))
response = [sock.recv(receive_buffer)]
while response[-1]:
response.append(sock.recv(receive_buffer))
return ''.join(r.decode(ENCODING) for r in response)
print(geturl('http://www.example.com/'))
HTH!
Edit: You may need to adjust what you put in the request, depending on the web server in question. Guanidene's excellent answer provides several resources to guide you on that path.
What you need to do in this case is send a raw HTTP request using sockets.
You would need to do a bit of low level network programming using the socket python module in this case. (Network sockets actually return you all the information sent by the server as it as, so you can accordingly interpret the response as you wish. For example, the HTTP protocol interprets the response in terms of standard HTTP headers - GET, POST, HEAD, etc. The high-level module urllib hides this header information from you and just returns you the data.)
You also need to have some basic information about HTTP headers. For your case, you just need to know about the GET HTTP request. See its definition here - http://djce.org.uk/dumprequest, see an example of it here - http://en.wikipedia.org/wiki/HTTP#Example_session. (If you wish to capture live traces of HTTP requests sent from your browser, you would need a packet sniffing software like wireshark.)
Once you know basics about socket module and HTTP headers, you can go through this example - http://coding.debuntu.org/python-socket-simple-tcp-client which tells you how to send a HTTP request over a socket to a server and read its reply back. You can also refer to this unclear question on SO.
(You can google python socket http to get more examples.)
(Tip: I am not a Java fan, but still, if you don't find enough convincing examples on this topic under python, try finding it under Java, and then accordingly translate it to python.)
urllib.urlretrieve('http://google.com/abc.jpg', 'abc.jpg')