I programmed my ESP8266 to read the soil moisture. Depending on the moisture a water pump gets activated. Now I wanted the ESP to tweet different sentences, depending on the situation.
Therefore I connected my twitter account to thingspeak.com and followed this code
Connecting to the internet works fine.
Problems:
It does not tweet every time and if it tweets, only the first word from a sentence shows up at twitter.
According to the forum, where I found the code, I already tried to replace all the spaces between the words with "%20". However then nothing shows up at twitter at all. Also single words are not always posted to twitter.
This is the code I have problems with:
// if connection to thingspeak.com is successful, send your tweet!
if (client.connect("184.106.153.149", 80))
{
client.print("GET /apps/thingtweet/1/statuses/update?key=" + API + "&status=" + tweet + " HTTP/1.1\r\n");
client.print("Host: api.thingspeak.com\r\n");
client.print("Accept: */*\r\n");
client.print("User-Agent: Mozilla/4.0 (compatible; esp8266 Lua; Windows NT 5.1)\r\n");
client.print("\r\n");
Serial.println("tweeted " + tweet);
}
I don't get any error messages.
Maybe you could help me to make it visible if the tweet was really sent and how I manage to tweet a whole sentence.
I am using the Arduino IDE version 1.8.9 and I am uploading to this board
The rest of the code works fine. The only problem is the tweeting.
Update
I now tried a few different things:
Checking server response
Works and helps a lot. The results are:
Single words as String don't get any response at all
Same for Strings like "Test%20Tweet"
Strings with multiple words like "Test Tweet" get the following response and the first word of the String shows up as a tweet
HTTP/1.1 200 OK
Server: nginx/1.7.5
Date: Wed, 19 Jun 2019 18:44:22 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 1
Connection: keep-alive
Status: 200 OK
X-Frame-Options: SAMEORIGIN
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, PUT, OPTIONS, DELETE, PATCH
Access-Control-Allow-Headers: origin, content-type, X-Requested-With
Access-Control-Max-Age: 1800
ETag: W/"RANDOM_CHARS"
Cache-Control: max-age=0, private, must-revalidate
X-Request-Id: THE_ID
1
I think the Content-Length might be the problem?
But I don't know how to change it in this code.
Checking if the connection succeded
I implemented this into my code an it never shows up on the monitor. So I think i never have a problem with not connecting.
Use a hostname instead of IP address
I tried it and never got a bad request. On the other hand nothing shows up on twitter at all.
Check if your tweet variable contains any new-line characters (carriage return or line feed). For example, the following variable would cause problems
String tweet = "Tweet no. 1\r\n";
due to the new-line characters at the end. These characters will cause the first line of the HTTP request to be cut short. I.e., instead of
GET /apps/thingtweet/1/statuses/update?key=api_key&status=Tweet no. 1 HTTP/1.1\r\n
it would become
GET /apps/thingtweet/1/statuses/update?key=api_key&status=Tweet no. 1\r\n
and the server would reject it with a 400 (Bad request) error.
On the other hand
String tweet = "Tweet no. 1";
would be fine.
If your tweets may contain such characters, then try encoding them before passing them to client.print():
tweet.replace("\r", "%0D");
tweet.replace("\n", "%0A");
Use a hostname instead of IP address
According to https://uk.mathworks.com/help/thingspeak/writedata.html, the relevant hostname for the API you are using is api.thingspeak.com. Use that instead of the IP address. This is preferable because the IP address a hostname points to can change regularly. (The IP address you are using doesn't even seem to be correct - and may already be out of date.)
I.e., change
if (client.connect("184.106.153.149", 80)) {
to
if (client.connect("api.thingspeak.com", 80)) {
API endpoint
Are you sure you are using the correct API endpoint? According to the link above, it looks like the API endpoint you need is https://api.thingspeak.com/update.json - so you may need to change
client.print("GET /apps/thingtweet/1/statuses/update?key=" + API + "&status=" + tweet + " HTTP/1.1\r\n");
to
client.print("GET /update.json?api_key=" + API + "&status=" + tweet + " HTTP/1.1\r\n");
Check if the connection succeeded
Presently, your device sends the HTTP request if connects to the server successfully - but doesn't give any indication if the connection fails! So add an else block to handle that scenario and notify the user via the serial console.
if (client.connect("api.thingspeak.com", 80)) {
client.print("GET /apps/thingtweet/1/statuses/update?key=" + API + "&status=" + tweet + " HTTP/1.1\r\n");
// etc.
}
else {
Serial.println("Connection to the server failed!");
}
Checking server response
To check the response from the server, add the following block to your main loop - which will print the server response via the serial console.
delay(50);
while (client.available()) {
String response_line = client.readString();
Serial.println(response_line);
}
To clarify: that code should go inside your loop() function.
The response should include a status line - such as HTTP/1.1 200 OK if the request was successful, or HTTP/1.1 400 Bad Request if there was a problem.
In the case of a Bad request response, the full message will quite likely contain more information about the precise reason the request failed.
HTTP vs HTTPs
Lastly, are you sure that the API supports (plain, unencrypted) HTTP as well as HTTPs? If not, that may be your problem.
This is my first time here and thought of joining the forum because I am new to Lua programming and have almost given up on HTTP Post method.
I am trying my hands on IOT using ESP8266 (running on NodeMCU) and using ESPlore to send the Lua program to ESP8266.
So, my program's objective is to call an API and post few parameters using my Lua program running on ESP8266.
I tried the following approaches -
1. Using HTTP Post
conn=net.createConnection(net.TCP, 0)
conn:on("receive", display)
conn:connect(80,HOST)
conn:on("connection",function(obj)
local post_request = build_post_request(HOST,URI)
obj:send(post_request)
end
----function as below ----------------------------------------------------
function build_post_request(host, uri)
local data = ""
data = "param1=1¶m2=2"
request = "POST uri HTTP/1.1\r\n"..
"Host: example.com\r\n"..
"apiKey: e2sss3af-9ssd-43b0-bfdd-24a1dssssc46\r\n"..
"Cache-Control: no-cache\r\n"..
"Content-Type: application/x-www-form-urlencoded\r\n"..data
return request
end
----------------Response --------------------------------------
HTTP/1.1 400 Bad Request
Date: Sun, 11 Oct 2015 16:10:55 GMT
Server: Apache-Coyote/1.1
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 968
Connection: close
Apache Tomcat/7.0.54 - Error report
The request sent by the client was syntactically incorrect.
I don't understand what is wrong with it.
2. Using Luasocket
I have included following in my program -
local http = require"socket.http"
local ltn12 = require"ltn12"
and it throws following errors-
script1.lua:3: module 'socket.http' not found:
no field package.preload['socket.http']
no file 'socket/http.lc'
no file 'socket/http.lua'
I don't know how to get these libs and send to ESP8266 and not sure if that will suffice.
Question :
Which is the best method to post data to a server using an API.
a. If HTTP Post, what is the problem with my code.
b. If Luasocket then how do I send it to ESP8266 as I am not using a compiler on my laptop.
"Content-Type: application/x-www-form-urlencoded\r\n"..data
I don't understand what is wrong with it.
In HTTP,the headers are always delimited by \r\n\r\n. Without the second CR-LF pair, the following data with cause a header error as reported dy Tomcat.
Second you can't use the standard socket libraries on the ESP8266. You must use the net library which is a nodemcu wrapper around the Espressif SDK.
It is nasty to debug Amazon MWS Feed submissions because whatever you might be doing wrong Ama just returns the same senseless error message:
<Error>
<Type>Sender</Type>
<Code>InvalidParameterValue</Code>
<Message>Either Action or Operation query parameter must be present.</Message>
</Error>
I am 100% positive that I build StringToSign correctly as well as calculate HMAC-SHA256, etc. It took me a few weeks to collect and adopt a stack of useful functions for hashing, signing, base64ing, etc. MWS requests. Written in pure Pascal they were all tested on Orders and Products APIs.
Now when it comes to Feeds API I am stuck on the above error. All parameters are equal to those generated by MWS Scratchpad. I tested the submission StringToSign generated by MWS Scratchpad, but no luck.
What I noticed so far: there is a difference between the number/values of headers generated by MWS Scratchpad and my app.
The Scratchpad generates the following headers (it least they are displayed):
Host: mws.amazonservices.ca
x-amazon-user-agent: AmazonJavascriptScratchpad/1.0 (Language=Javascript)
Content-Type: text/xml
My app uses Indy (in XE4) TIdHTTP to make a request. When Amazon returns the above error, Request.RawHeaders.Text contains the following:
Content-Length: 251
x-amazon-user-agent: MyApp/1.1(Language=Delphi;Platform=Windows7)
Content-Type: text/xml
Host: mws.amazonservices.ca
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: identity
User-Agent: MyApp/1.1(Language=Delphi;Platform=Windows7)
Looks like the extra the headers are added to Request object by default. My first question is: can those extra headers theoretically influence on the behavior? I.e. can they be the showstoppers?
My second question: is there any option to mofidy the list of default heades in Indy HTTP.Request? To go on with the debugging I would rather exclude extra headers to see if the request works.
UPDATE: (SignedString)
AWSAccessKeyId=<AWSAccessKeyId>
&Action=GetFeedSubmissionList
&Merchant=<MerchantId>
&SignatureMethod=HmacSHA256
&SignatureVersion=2
&Timestamp=2015-07-26T09%3A04%3A59Z
&Version=2009-01-01
&Signature=1OI0PVgL3uh5sFXxjCzaaWEwGmW6h5e0dgLUFkPgoXg%3D
UPDATE: (Complete HTTP Request/Response provided by TIdLogFile)
Stat Connected.
Sent 28.07.2015 12:28:11:
POST / HTTP/1.1<EOL>
Content-Type: text/xml; charset=us-ascii<EOL>
Content-Length: 279<EOL>
x-amazon-user-agent: MyAppNameAndVer<EOL>
Host: mws.amazonservices.ca<EOL>
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8<EOL>
Accept-Encoding: identity<EOL>
User-Agent: Mozilla/3.0 (compatible; Indy Library)<EOL><EOL>
Sent 28.07.2015 12:28:11:
AWSAccessKeyId=<AWSAccessKeyId>
&Action=GetFeedSubmissionList
&MWSAuthToken=<AWSAuthToken>
&Merchant=<MerchantId>
&SignatureMethod=HmacSHA256
&SignatureVersion=2
&Timestamp=2015-07-28T10%3A28%3A09Z
&Version=2009-01-01
&Signature=I6euLIiVDzjZ8bbdtF840K0TJCkGh4NrUbQPtQtu78A%3D
Recv 28.07.2015 12:28:11:
HTTP/1.1 400 Bad Request<EOL>
Date: Tue, 28 Jul 2015 10:28:10 GMT<EOL>
Server: AmazonMWS<EOL>
x-mws-request-id: 7e63280d-1db5-4d10-af40-587747d032a8<EOL>
x-mws-timestamp: 2015-07-28T10:28:11.048Z<EOL>
x-mws-response-context: OmAlguEmW20QT07uIdb9d25xkX+JSS7uFr1rDvXIoqXMvbUFzUMt1b5Xl2WzDaJszbwr25N/J4c=<EOL>
Content-Type: text/xml<EOL>
Content-Length: 324<EOL>
Vary: User-Agent<EOL>
nnCoection: close<EOL><EOL>
<?xml version="1.0"?><LF><ErrorResponse xmlns="https://mws.amazonservices.com/"><LF> <Error><LF> <Type>Sender</Type><LF> <Code>InvalidParameterValue</Code><LF> <Message>Either Action or Operation query parameter must be present.</Message><LF> </Error><LF> <RequestID>7e63280d-1db5-4d10-af40-587747d032a8</RequestID><LF></ErrorResponse><LF>
Stat Disconnected.
Please note: I formatted the response for your convenience adding CRs after <EOL>
Confusing nnCoection header is explained here: Cneonction and nnCoection HTTP headers
My question is now resolved. Below is my way to thank people not breaking the Community standards ;-)
Days := 1;
repeat
IsSuccessful := DebugAmazonFeedSubmissionProc;
Inc(Days);
until IsSuccessful or (Days = 14);
if not IsSuccessful then
begin
AskTheGods(#StackOverflow);
Sleep(1000); // ...wake up and drink a cup'o'tea :)
Expert := TRemyLebeau.Create('delphi');
try
Expert.ReviewTheCode;
Expert.GetTheJobDone;
finally
// You may dispose the instance here,
// but the Class is one of the most valuable assets of the Community.
// Thank you, Remy!
end;
end;
Your request Content-Type header is wrong. You have it set to text/xml; charset=us-ascii, but you are not actually sending XML data to begin with. You need to set the Content-Type (via the TIdHTTP.Request.ContentType property) to application/x-www-form-urlencoded instead, per the MWS GetFeedSubmissionList documentation.
The TStrings version of TIdHTTP.Post() would set the ContentType for you, but you are using the TStream version instead so you have to set the ContentType manually. In this case, I do suggest you switch to the TStrings version, and let it handle encoding the form fields for you:
var
PostData: TStringList;
begin
PostData := TStringList.Create;
try
PostData.Add('AWSAccessKeyId=<AWSAccessKeyId>');
PostData.Add('Action=GetFeedSubmissionList');
PostData.Add('MWSAuthToken=<AWSAuthToken>');
PostData.Add('Merchant=<MerchantId>');
PostData.Add('SignatureMethod=HmacSHA256');
PostData.Add('SignatureVersion=2');
PostData.Add('Timestamp=2015-07-28T10:28:09Z'); // <-- NOT percent encoded yet!
PostData.Add('Version=2009-01-01');
PostData.Add('Signature=I6euLIiVDzjZ8bbdtF840K0TJCkGh4NrUbQPtQtu78A='); // <-- NOT percent encoded yet!
IdHTTP1.Request.CustomHeaders.Values['x-amazon-user-agent'] := 'MyAppNameAndVer';
IdHTTP1.Post('http://mws.amazonservices.ca', PostData);
finally
PostData.Free;
end;
end;
Docker API image creation / pull (/v1.6/images/create) apparently always return
HTTP/1.1 200 OK
Content-Type: application/json
no matter if the process is a success or a failure.
Furthermore, the payload is not valid json.
eg: /v1.6/images/create?fromImage=whatevertheflush
returns:
{"status":"Pulling repository whatevertheflush"}{"error":"Server error: 404 trying to fetch remote history for whatevertheflush","errorDetail":{"code":404,"message":"Server error: 404 trying to fetch remote history for whatevertheflush"}}
Not being valid json, and the HTTP error not being forwarded / used makes it awkward to handle errors for clients.
Indeed, docker-py just puke the payload (https://github.com/dotcloud/docker-py/blob/master/docker/client.py#L374). And DockerHTTPClient from openstack tries to return a value based on the http error code, which is always 200... (https://github.com/openstack/nova/blob/master/nova/virt/docker/client.py#L191)
Now, I understand the pull might take a long time, and that it somewhat make sense to start streaming an answer to the client, but I can't help thinking something is wrong here.
So, this is three fold:
am I missing something entirely here?
if not: if you are implementing a client application (say, in Python), how would you handle this (elegantly, if possible :))? try to detect valid json blocks, load them, and exit whenever we "think" something is wrong?
if not: is this going to change (for the better) in future docker versions?
This question is a bit old, but for the future reader who has landed on this page, I'd like to let you know you're not alone, we feel your pain. This API is indeed as terrible as it looks.
The TL;DR answer is "the /images/create response format is undocumented; discard the output and query /images/XXX/json after your create call completes."
I wrote some orchestration tools a few years ago, and I found the /images/create API to be extremely annoying. But let's dive in:
There is no documented schema of the 200 response; the v1.19 docs simply gave examples of a few records. The v1.37 (latest at the time I write this) docs don't even go that far, no details are provided at all of the response.
The response is sent as Transfer-Encoding: chunked, and each record sent is preceded by the byte count in hex. Here's a low-level exerpt (bypassing curl, so we can see what actually gets sent on the wire):
host-4:~ rg$ telnet localhost 2375
Trying ::1...
Connected to localhost.
Escape character is '^]'.
POST /images/create?fromImage=jenkins/jenkins:latest HTTP/1.1
Host: localhost:2375
User-Agent: foo/1.0
Accept: */*
HTTP/1.1 200 OK
Api-Version: 1.39
Content-Type: application/json
Docker-Experimental: true
Ostype: linux
Server: Docker/18.09.1 (linux)
Date: Wed, 06 Feb 2019 16:53:19 GMT
Transfer-Encoding: chunked
39
{"status":"Pulling from jenkins/jenkins","id":"latest"}
5e
{"status":"Digest: sha256:abd3e3f96fbc3445c420fda590f37e2bd3377f69affd47b63b3d826d084c5ddc"}
45
{"status":"Status: Image is up to date for jenkins/jenkins:latest"}
0
Yes, it streams the image download progress -- client libraries that don't give low-level access to the chunked records may just concatenate the data before it's provided to you. As you encountered, early versions of the API returned JSON records with the only delimiter being the chunked transfer encoding, so client code received a concatenated block of undelimited JSON and had to parse it by tracking curlies/quotes/escape chars! It has since been updated to now emit records delimited by newlines, but can we count on them always being there? Who knows! This behavior changed without ceremony, and was not preserved if you call older versions of the API on newer daemons.
It returns 200 OK immediately, which doesn't represent success or failure. (Given the nature of the call, I'd imagine it should probably return 202 Accepted instead. Ideally, we'd get a Location header pointing to a new URL that we could use to query the progress/status.)
The response data returned is huge, spammy, and just... silly. If you have a docker instance listening on TCP, try curl -Nv -X POST http://yourdocker:2375/images/create?fromImage=jenkins/jenkins:latest -o /tmp/omgwtf.txt. You'll be amazed. A ton of bandwidth is wasted transferring server-rendered ASCII bar graphs!!!. In fact, the records return each layer's progress three different ways, as numeric fields for current and total bytes, as a bar graph, and as a pretty-printed string with MB or GB units. Why isn't this just rendered on the client? Great question.
Instead, you need your client to parse kilobytes or megabytes of spam.
The bar graph has a randomly escaped unicode rep of the > character, despite being safely inside a JSON string. Someone was just throwing escape calls at the wall to see what stuck? ¯\_(ツ)_/¯
The records themselves are pretty arbitrary. There's an id field that changes what it references, and the only way to know what kind of record it is to parse the human-readable string. Pulling from XXX vs Pulling fs layer vs Downloading etc. As far as I can tell, the only real way to know if it's done is to track all the ids, and ensure you get a Pull complete for each at the time that the socket closes.
You might be able to look for Status: Downloaded newer image for XXX but I'm not sure if there are multiple possible responses for this.
As I mentioned at the start, you'll probably have the best luck requesting /images/XXX/json after /images/create claims to be complete. The combination of the two calls will give a pretty reliable indication of whether /images/create worked or not.
Here's a longer block of concatenated client response that shows a few different record types. Edited for brevity:
{"status":"Pulling from jenkins/jenkins","id":"latest"}
{"status":"Pulling fs layer","progressDetail":{},"id":"ab1fc7e4bf91"}
{"status":"Pulling fs layer","progressDetail":{},"id":"35fba333ff52"}
{"status":"Pulling fs layer","progressDetail":{},"id":"f0cb1fa13079"}
{"status":"Pulling fs layer","progressDetail":{},"id":"3d1dd648b5ad"}
{"status":"Pulling fs layer","progressDetail":{},"id":"a9f886e483d6"}
{"status":"Pulling fs layer","progressDetail":{},"id":"4346341d3c49"}
..
"status":"Waiting","progressDetail":{},"id":"3d1dd648b5ad"}
{"status":"Waiting","progressDetail":{},"id":"a9f886e483d6"}
{"status":"Waiting","progressDetail":{},"id":"4346341d3c49"}
{"status":"Waiting","progressDetail":{},"id":"006f2208d67a"}
{"status":"Waiting","progressDetail":{},"id":"fb85cf26717d"}
{"status":"Waiting","progressDetail":{},"id":"52ca068dbca7"}
{"status":"Waiting","progressDetail":{},"id":"82f4759b8d12"}
...
{"status":"Downloading","progressDetail":{"current":110118,"total":10780995},"progress":"[\u003e ] 110.1kB/10.78MB","id":"35fba333ff52"}
{"status":"Downloading","progressDetail":{"current":457415,"total":45344749},"progress":"[\u003e ] 457.4kB/45.34MB","id":"ab1fc7e4bf91"}
{"status":"Downloading","progressDetail":{"current":44427,"total":4340040},"progress":"[\u003e ] 44.43kB/4.34MB","id":"f0cb1fa13079"}
{"status":"Downloading","progressDetail":{"current":817890,"total":10780995},"progress":"[===\u003e ] 817.9kB/10.78MB","id":"35fba333ff52"}
{"status":"Downloading","progressDetail":{"current":1833671,"total":45344749},"progress":"[==\u003e ] 1.834MB/45.34MB","id":"ab1fc7e4bf91"}
{"status":"Downloading","progressDetail":{"current":531179,"total":4340040},"progress":"[======\u003e ] 531.2kB/4.34MB","id":"f0cb1fa13079"}
{"status":"Downloading","progressDetail":{"current":1719010,"total":10780995},"progress":"[=======\u003e ] 1.719MB/10.78MB","id":"35fba333ff52"}
{"status":"Downloading","progressDetail":{"current":3205831,"total":45344749},"progress":"[===\u003e ] 3.206MB/45.34MB","id":"ab1fc7e4bf91"}
{"status":"Downloading","progressDetail":{"current":1129195,"total":4340040},"progress":"[=============\u003e ] 1.129MB/4.34MB","id":"f0cb1fa13079"}
{"status":"Downloading","progressDetail":{"current":2640610,"total":10780995},"progress":"[============\u003e ] 2.641MB/10.78MB","id":"35fba333ff52"}
{"status":"Downloading","progressDetail":{"current":1719019,"total":4340040},"progress":"[===================\u003e ] 1.719MB/4.34MB","id":"f0cb1fa13079"}
{"status":"Downloading","progressDetail":{"current":4586183,"total":45344749},"progress":"[=====\u003e ] 4.586MB/45.34MB","id":"ab1fc7e4bf91"}
{"status":"Downloading","progressDetail":{"current":3549922,"total":10780995},"progress":"[================\u003e ] 3.55MB/10.78MB","id":"35fba333ff52"}
{"status":"Downloading","progressDetail":{"current":2513643,"total":4340040},"progress":"[============================\u003e ] 2.514M
...
{"status":"Pull complete","progressDetail":{},"id":"6d9b49fc8a28"}
{"status":"Extracting","progressDetail":{"current":380,"total":380},"progress":"[==================================================\u003e] 380B/380B","id":"6302e8b6563c"}
{"status":"Extracting","progressDetail":{"current":380,"total":380},"progress":"[==================================================\u003e] 380B/380B","id":"6302e8b6563c"}
{"status":"Pull complete","progressDetail":{},"id":"6302e8b6563c"}
{"status":"Extracting","progressDetail":{"current":1548,"total":1548},"progress":"[==================================================\u003e] 1.548kB/1.548kB","id":"7348f018cf93"}
{"status":"Extracting","progressDetail":{"current":1548,"total":1548},"progress":"[==================================================\u003e] 1.548kB/1.548kB","id":"7348f018cf93"}
{"status":"Pull complete","progressDetail":{},"id":"7348f018cf93"}
{"status":"Extracting","progressDetail":{"current":3083,"total":3083},"progress":"[==================================================\u003e] 3.083kB/3.083kB","id":"c651ee7bd59e"}
{"status":"Extracting","progressDetail":{"current":3083,"total":3083},"progress":"[==================================================\u003e] 3.083kB/3.083kB","id":"c651ee7bd59e"}
{"status":"Pull complete","progressDetail":{},"id":"c651ee7bd59e"}
{"status":"Digest: sha256:abd3e3f96fbc3445c420fda590f37e2bd3377f69affd47b63b3d826d084c5ddc"}
{"status":"Status: Downloaded newer image for jenkins/jenkins:latest"}
This code runs the Internet now. =8-O
This particular endpoint actually returns chunked encoding. An example via curl:
$ curl -v -X POST http://localhost:4243/images/create?fromImage=base
* About to connect() to localhost port 4243 (#0)
* Trying ::1...
* Connection refused
* Trying 127.0.0.1...
* connected
* Connected to localhost (127.0.0.1) port 4243 (#0)
> POST /images/create?fromImage=base HTTP/1.1
> User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8y zlib/1.2.5
> Host: localhost:4243
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Fri, 07 Feb 2014 04:21:59 GMT
< Transfer-Encoding: chunked
<
* Connection #0 to host localhost left intact
{"status":"Pulling repository base"}{"status":"Pulling image (ubuntu-quantl) from base","progressDetail":{},"id":"b750fe79269d"}{"status":"Pulling image (ubuntu-quantl) from base, endpoint: https://cdn-registry-1.docker.io/v1/","progressDetail":{},"id":"b750fe79269d"}{"status":"Pulling dependent layers","progressDetail":{},"id":"b750fe79269d"}{"status":"Download complete","progressDetail":{},"id":"27cf78414709"}{"status":"Download complete","progressDetail":{},"id":"b750fe79269d"}{"status":"Download complete","progressDetail":{},"id":"b750fe79269d"}* Closing connection #0
Now I'm not sure how you go about parsing this in Python, but in Ruby, I can use Yajl like so:
parts = []
Yajl::Parser.parse(body) { |o| parts << o }
puts parts
{"status"=>"Pulling repository base"}
{"status"=>"Pulling image (ubuntu-quantl) from base", "progressDetail"=>{}, "id"=>"b750fe79269d"}
{"status"=>"Pulling image (ubuntu-quantl) from base, endpoint: https://cdn-registry-1.docker.io/v1/", "progressDetail"=>{}, "id"=>"b750fe79269d"}
{"status"=>"Pulling dependent layers", "progressDetail"=>{}, "id"=>"b750fe79269d"}
{"status"=>"Download complete", "progressDetail"=>{}, "id"=>"27cf78414709"}
{"status"=>"Download complete", "progressDetail"=>{}, "id"=>"b750fe79269d"}
{"status"=>"Download complete", "progressDetail"=>{}, "id"=>"b750fe79269d"}
Using Docker v1.9 I still having this problem to deal with.
Also have found an issue on Docker Github repository: Docker uses invalid JSON format in some API functions #16925
Where some contributor suggests to use Content-Type HTTP header like this: application/json; boundary=NL
This not worked for me.
Then, while struggling with my custom parser, found this question StackOverflow: How to handle a huge stream of JSON dictionaries?
I am working on a IMAP, so just reading the body (body[header.fields (DATE FROM SUBJECT)]) I am passing this command.
But problem is like there some time my string return extra stuff excetp from my original string.
and some times I had getting limited part of the string means half part of the body.
so whenever I am passing second command it will accept as a first command and return result as the
first command pending resul.;
so the my concern is that I am not able to retrive proper data as of the part of the body.
as per my knowladge I think it's happen due to the internet datapacket tresfersize, but apart from this look at outlook or any other mail manager will work properly so what this is the mechanisam for
this data retriving.
or anything else to do fo my coding.....
Thanks..
Posting a sample response from the IMAP server that contains the "extra stuff" would help.
The problem you are most likely facing is with untagged server responses.
Heres what RFC3501 says:
Status responses can be tagged or untagged. Tagged status responses indicate the completion result (OK, NO, or BAD status) of a client command, and have a tag matching the command:
C: a002 NOOP
S: a002 OK NOOP completed
Some status responses, and all server data, are untagged. An
untagged response is indicated by the token "*" instead of a tag.
C: a047 NOOP
S: * 22 EXPUNGE
S: * 23 EXISTS
S: * 3 RECENT
S: * 14 FETCH (FLAGS (\Seen \Deleted))
S: a047 OK NOOP completed
So you need to distinguish between those 2 response types.
Please remember that checking if every received line starts from '*' character is not enough, as your email message may also have lines starting from star character:
C: a004 fetch 12 body[header]
S: * 12 FETCH (RFC822 {342}
S: Date: Wed, 17 Jul 1996 02:23:25 -0700 (PDT)
S: From: Terry Gray <gray#cac.washington.edu>
S: Subject: IMAP4rev1 WG mtg summary and minutes
S: MIME-Version: 1.0
S:
S: * This is email body containing start char
S: )
S: a004 OK FETCH completed
{342} is the exact number of bytes you are supposed to read.
The bottom line is don't reinvent the wheel use existing library.
You can check out mine IMAP component (not free).