AVPlayer won't play audio files from FFMPEG - ios

Before requesting audio data AVPlayer requests byte range 0-1 from FFMPEG.
FFMPEG gives a 200 response, but AVPlayer requires a 206 response.
This results in the request failing and audio can't be played.
Expected behavior:
Play tracks when streaming through ffmpeg
Current behavior: When trying to stream with ffmpeg we get "Operation Stopped"
Sample FFMPEG command:
ffmpeg -i "/path/to/audio/track.mp3" -vn -strict -2 -acodec pcm_u8 -f wav -listen 1 -seekable 1 http://localhost:8090/restream.wav
Player Log:
Error Domain=AVFoundationErrorDomain Code=-11850 "Operation Stopped" UserInfo={NSLocalizedFailureReason=The server is not correctly configured., NSLocalizedDescription=Operation Stopped, NSUnderlyingError=0x600003bcc4b0 {Error Domain=NSOSStatusErrorDomain Code=-12939 "(null)"}}
!av_interleaved_write_frame(): Broken pipe
!Connection to tcp://localhost:8090 failed: Connection refused
!Connection to tcp://localhost:8090 failed: Connection refused
!Connection to tcp://localhost:8090 failed: Connection refused
!Error writing trailer of http://localhost:8090/restream.wav: Broken pipe
This error is defined by Apple as:
+"The HTTP server sending the media resource is not configured as expected. This might mean that the server does not support byte range requests."
And summarised nicely in this StackOverflow post:
when AVPlayerItem receive a video URL , it do the following task:
Send a bytes request HTTP Request, and range = 0 -1
If the response code is 206 and return 1 bytes data, It do the 3th task, if not, AVErrorServerIncorrectlyConfigured error occurred.
continue send other HTTP Request, to download segment of All duration. and the response of VideoData code must be 206
In my situation , when send range[0-1] HTTP request, the server side give me a 200 OK response, So error occurred.
Network Log:
GET /file.wav HTTP/1.1
Host: localhost:1234
X-Playback-Session-Id: F72F1139-6F4C-4A22-B334-407672045A86
Range: bytes=0-1
Accept: */*
User-Agent: AppleCoreMedia/1.0.0.18C61 (iPhone; U; CPU OS 14_3 like Mac OS X; en_us)
Accept-Language: en-us
Accept-Encoding: identity
Connection: keep-alive
HTTP/1.1 200 OK
Content-Type: application/octet-stream
Transfer-Encoding: chunked
Reproduce using this sample app:
This can also be reproduced using standard ffmpeg and adding URL to local or remote ffmpeg URL
Can we solve this by making changes to FFMPEG or AVPlayer?

I asked this question on the ffmpeg email list and it seems it's not possible:
If I understand your request correctly, you are not asking for a
change of a return type (you could probably do that yourself) but for
the implementation of byte range requests. I don't think this is
possible at all with FFmpeg.
Also it's not possible to get AVPlayer to ignore the byte range request:
it is an apple standard that media providers need to support http 1.1
with the range header (check out the iTunes store guidelines for
podcasts for example), so I wouldn't expect it anytime soon
SO Q'n: Is there a way to stop the avplayer sending a range http header field

Related

The HTTP error code 502 returned frequently when calling Google EmbeddedAssistant API by Protocol Buffers over HTTP

I'm trying to call Google Assistant API using Protocol Buffers (protobuf) over HTTP. refer to:
https://googleapis.github.io/HowToRPC#grpc-fallback-experimental
My problem is that I frequently get HTTP error code 502 when sending request to the back-end service.
To test the problem, I wrote a python script to send (through HTTP POST) the pre-built protobuf binaries and check the response. The test results are:
32KB audio data (about 1 second length of audio), 20 times post, 0 times 502 error received / 32KB 20/0, failure rate: 0%
2*32KB, 20/0, failure rate: 0%
3*32KB, 20/3, failure rate: 15%
4*32KB, 20/10, failure rate: 50%
6*32KB, 20/19, failure rate: 95%
HTTP status code is 200 for successful requests, while 502 for failed cases.
where we can see, the larger audio length the greater failure rate.
The python code to post pre-built protobuf binaries is as below. while the content of file f1 is the just protobuf binaries.
def postData():
url = "https://embeddedassistant.googleapis.com/$rpc/google.assistant.embedded.v1alpha2.EmbeddedAssistant/Assist"
header = {"Content-type".encode('utf-8'): "application/x-protobuf".encode('utf-8'),"Accept".encode('utf-8'):"text/plain".encode('utf-8'), "Connection".encode('utf-8'):"keep-alive".encode('utf-8'), "Authorization".encode('utf-8'):repToken.encode('utf-8')}
with open(fl) as f:
r = requests.post(url, data=f, headers=header)
with open(fl + "_out", 'wb') as fd:
print(r.status_code)
fd.write(r.content)
f.close()
fd.close()
I also tried to post binary files which contain invalid protobuf, e.g a mp3 file,
and in this case, no matter the size of the file, the HTTP status code returned is always 400 with following message, which is just expected.
Invalid PROTO payload received. Invalid request data in stream body, unexpected message type: 7a
It seems the back-end service sets some kind of limitation for the latency of data transfer which makes it doesn’t work well with low bandwidth connections?

Not able to run Microsoft Bing Speech Recognition API on iOS device using iOS Client Sample provided by Microsoft

I was trying to explore Microsoft's Bing Speech Recognition API for iOS https://github.com/Microsoft/Cognitive-Speech-STT-iOS. I followed all the steps written in the read me. The app runs and it seems to be detecting the speech from microphone and sending it to the Microsoft server but it gives an error in the logs and the button disables itself without giving any text on the app.
Here are the logs that came in the console. Please help.
Application Name: com.Microsoft.SpeechRecognitionServerExample/1.0.1
Refreshing token /sts/v1.0/issueToken
Initializing Audio Services
Initializing Speech Services
No application id provided to controller
GetIdentityPropertyValue 3
Useragent Value iOS Assistant (iOS; 10.0.1;Mobile;ProcessName/AppName=com.Microsoft.SpeechRecognitionServerExample/1.0.1;DeviceType=Near;SpeechClient=1.0.160824)
Url: 'https://websockets.platform.bing.com/ws/speech/recognize'
Locale: 'en-us'
Application Id: ''
Version: 4.0.150429
UserAuthorizationToken:
ServerLoggingLevel: 1
Initiating websocket connection. m_connection=0x0 host=websockets.platform.bing.com port=443
Auth token status: 200
Authorization token hr 0 'Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzY29wZSI6Imh0dHBzOi8vc3BlZWNoLnBsYXRmb3JtLmJpbmcuY29tIiwic3Vic2NyaXB0aW9uLWlkIjoiZmNlYThiNTE3NGZmNDdkODk4ZWZiN2ZjYzNjZjM4MzMiLCJwcm9kdWN0LWlkIjoiQmluZy5TcGVlY2guUHJldmlldyIsImNvZ25pdGl2ZS1zZXJ2aWNlcy1lbmRwb2ludCI6Imh0dHBzOi8vYXBpLmNvZ25pdGl2ZS5taWNyb3NvZnQuY29tL2ludGVybmFsL3YxLjAvIiwiYXp1cmUtcmVzb3VyY2UtaWQiOiIiLCJpc3MiOiJ1cm46bXMuY29nbml0aXZlc2VydmljZXMiLCJhdWQiOiJ1cm46bXMuc3BlZWNoIiwiZXhwIjoxNDc1NDg3MDQ1fQ.4LF0gyhXU0T1iwDahlYlanKJ_wVjOOLhLyalFeDqIzA'
Successfully initialized client connection
Create ImpressionId: f43586374f8d8e455e48b090f2aaa5cd
Create ImpressionId: f627a603daa0cd4a16f59d9581118cdd
Reset
Create ImpressionId: dad2c09e82b8eaaacec3b20890de9bb8
ImpressionId: a9776c0ba8dc079c7d9658e6746a46ea
Adding requestId: 'bf17eb3310b2e39944d85dfe3d2868eb' for 'text/cu.client.context'
Subscribing request [bf17eb3310b2e39944d85dfe3d2868eb]
Waiting for connection/send completion.
Audio stream created
Adding requestId: 'dd3d699f39139326bedbb1dafd8816fb' for 'audio/x-wav'
Subscribing request [dd3d699f39139326bedbb1dafd8816fb]
Audio Stream Created
Creating transcoder 2
Microphone permissions: 0
Upgrade request returned with HTTP status code: 101.
Web socket handshake completed
CU Client connected
ConnectionStateChanged
Sent first chunk of audio stream, requestId='dd3d699f-3913-9326-bedb-b1dafd8816fb'
Received message: 'audio.stream.response'
Response request id: 'dd3d699f-3913-9326-bedb-b1dafd8816fb'
Response impression: 'a9776c0b-a8dc-079c-7d96-58e6746a46ea'
LanguageGeneration OK
Received message: 'audio.stream.response'
Response request id: 'dd3d699f-3913-9326-bedb-b1dafd8816fb'
Response impression: 'a9776c0b-a8dc-079c-7d96-58e6746a46ea'
LanguageGeneration OK
Received message: 'audio.stream.response'
Response request id: 'dd3d699f-3913-9326-bedb-b1dafd8816fb'
Response impression: 'a9776c0b-a8dc-079c-7d96-58e6746a46ea'
LanguageGeneration OK
Received message: 'audio.stream.response'
Response request id: 'dd3d699f-3913-9326-bedb-b1dafd8816fb'
Response impression: 'a9776c0ba8dc079c7d9658e6746a46ea'
Response Conversation: 'ab7082f3c4de8feb0d6210e8ec07dcb3'
LanguageGeneration OK
Sending audio stream endpoint, requestId='dd3d699f-3913-9326-bedb-b1dafd8816fb'
Sent audio stream endpoint, requestId='dd3d699f-3913-9326-bedb-b1dafd8816fb'
signaling OnAudioEvent(AUDIO_EVENT_RECORD_STOP)
originating error 0x80070057
Client UPL: 32880000 ticks
originating error 0x8000ffff
originating error 0x80070057
originating error 0x80004005
originating error 0x80004005
Failed to 'hresult', HR=80004005, WebSocket connection failed
No messages to retry, closing.
Closing web socket channel
CU Client connection dropped
ConnectionStateChanged
WebSocket closed unexpectedly, status: 0
Web socket channel already closed.

AVPlayer throwing : "Internal error: restarting too far ahead"

I am observing KVO AVPlayerItemNewErrorLogEntryNotification of the AVPlayer and I find the following error:
errorStatusCode:-12645 errorDomain :CoreMediaErrorDomain
errorComment:Internal error: restarting too far ahead (-1.4084s)
I am unable to understand what the below line really means.
"Internal error: restarting too far ahead"
Can anyone help me to find the root cause of this issue.
If long .ts video file respons with: -12645.
There is no documentation for error status codes returned by MPMediaPlayer in MPMovieErrorLogEvent class.
But some of them are:
HTTP status - errorStatusCode - errorDomain - errorComment
400 -12666 CoreMediaErrorDomain unrecognized http response 400
401 -12937 CoreMediaErrorDomain Authentication Error
402 -12666 CoreMediaErrorDomain unrecognized http response 402
403 -12660 CoreMediaErrorDomain HTTP 403: Forbidden
404 -12938 CoreMediaErrorDomain HTTP 404: File not found
405 -12666 CoreMediaErrorDomain unrecognized http response 405
406 -12666 CoreMediaErrorDomain unrecognized http response 406
407 -12937 CoreMediaErrorDomain Authentication Error
409 -12666 CoreMediaErrorDomain unrecognized http response 409
...
415 -12666 CoreMediaErrorDomain unrecognized http response 415
500 -12666 CoreMediaErrorDomain unrecognized http response 500
501 -12666 CoreMediaErrorDomain unrecognized http response 501
502 -12666 CoreMediaErrorDomain unrecognized http response 502
503 -12661 CoreMediaErrorDomain HTTP 503: Unavailable
504 -12666 CoreMediaErrorDomain unrecognized http response 504
505 -12666 CoreMediaErrorDomain unrecognized http response 505
if long .ts video file respons -12645 CoreMediaErrorDomain No response for media file in 10 s
video .ts file bitrate differ from m3u8 declaration -12318 CoreMediaErrorDomain Segment exceeds specified bandwidth for variant
for live stream.playlist m3u8 did not change too long -12642 CoreMediaErrorDomain Playlist File unchanged for 2 consecutive reads
if wrong host ip -1004 kCFErrorDomainCFNetwork -
if wrong dns host name -1003 kCFErrorDomainCFNetwork -
if bad formatted URL -1000 kCFErrorDomainCFNetwork -
if invalid https/ssl request -1202 kCFErrorDomainCFNetwork -
According to an Apple employee this error message means that:
The player is complaining that it was trying to restart a live stream,
but only had 1.4 seconds of material available. How many segments are
in your live playlist? These days we recommend that you always have at
least six segments in the playlist. I believe the only effect is to
delay palyback until it has enough data.
Source: https://forums.developer.apple.com/thread/40791
The cause of the error most probably originates from the m3u8 itself.
When streaming a live program (e.g. any TV channel) there is a delay between the availability of event data and the encoder creating media segments (.ts) that are then appended to the playlist.
I could imagine that these types of errors are generated if the playerItem playhead's position is at the very edge of the buffer and new media segments are not made available fast enough on the server.
Should these errors be prevented or handled on the client's side:
Since these errors are reported as internal and AVPlayer apparently recovers from it, it is safe to assume that they are being handled internally. They could therefore be understood as simple error reporting and one could choose to ignore them.
If one wanted to prevent them, the only way to do this might be to manually set the live buffer's edge back in time a few seconds.

NodeMCU - Lua - HTTP Post or luasocket - Need guidance

This is my first time here and thought of joining the forum because I am new to Lua programming and have almost given up on HTTP Post method.
I am trying my hands on IOT using ESP8266 (running on NodeMCU) and using ESPlore to send the Lua program to ESP8266.
So, my program's objective is to call an API and post few parameters using my Lua program running on ESP8266.
I tried the following approaches -
1. Using HTTP Post
conn=net.createConnection(net.TCP, 0)
conn:on("receive", display)
conn:connect(80,HOST)
conn:on("connection",function(obj)
local post_request = build_post_request(HOST,URI)
obj:send(post_request)
end
----function as below ----------------------------------------------------
function build_post_request(host, uri)
local data = ""
data = "param1=1&param2=2"
request = "POST uri HTTP/1.1\r\n"..
"Host: example.com\r\n"..
"apiKey: e2sss3af-9ssd-43b0-bfdd-24a1dssssc46\r\n"..
"Cache-Control: no-cache\r\n"..
"Content-Type: application/x-www-form-urlencoded\r\n"..data
return request
end
----------------Response --------------------------------------
HTTP/1.1 400 Bad Request
Date: Sun, 11 Oct 2015 16:10:55 GMT
Server: Apache-Coyote/1.1
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 968
Connection: close
Apache Tomcat/7.0.54 - Error report
The request sent by the client was syntactically incorrect.
I don't understand what is wrong with it.
2. Using Luasocket
I have included following in my program -
local http = require"socket.http"
local ltn12 = require"ltn12"
and it throws following errors-
script1.lua:3: module 'socket.http' not found:
no field package.preload['socket.http']
no file 'socket/http.lc'
no file 'socket/http.lua'
I don't know how to get these libs and send to ESP8266 and not sure if that will suffice.
Question :
Which is the best method to post data to a server using an API.
a. If HTTP Post, what is the problem with my code.
b. If Luasocket then how do I send it to ESP8266 as I am not using a compiler on my laptop.
"Content-Type: application/x-www-form-urlencoded\r\n"..data
I don't understand what is wrong with it.
In HTTP,the headers are always delimited by \r\n\r\n. Without the second CR-LF pair, the following data with cause a header error as reported dy Tomcat.
Second you can't use the standard socket libraries on the ESP8266. You must use the net library which is a nodemcu wrapper around the Espressif SDK.

How to handle docker API /images/create?

Docker API image creation / pull (/v1.6/images/create) apparently always return
HTTP/1.1 200 OK
Content-Type: application/json
no matter if the process is a success or a failure.
Furthermore, the payload is not valid json.
eg: /v1.6/images/create?fromImage=whatevertheflush
returns:
{"status":"Pulling repository whatevertheflush"}{"error":"Server error: 404 trying to fetch remote history for whatevertheflush","errorDetail":{"code":404,"message":"Server error: 404 trying to fetch remote history for whatevertheflush"}}
Not being valid json, and the HTTP error not being forwarded / used makes it awkward to handle errors for clients.
Indeed, docker-py just puke the payload (https://github.com/dotcloud/docker-py/blob/master/docker/client.py#L374). And DockerHTTPClient from openstack tries to return a value based on the http error code, which is always 200... (https://github.com/openstack/nova/blob/master/nova/virt/docker/client.py#L191)
Now, I understand the pull might take a long time, and that it somewhat make sense to start streaming an answer to the client, but I can't help thinking something is wrong here.
So, this is three fold:
am I missing something entirely here?
if not: if you are implementing a client application (say, in Python), how would you handle this (elegantly, if possible :))? try to detect valid json blocks, load them, and exit whenever we "think" something is wrong?
if not: is this going to change (for the better) in future docker versions?
This question is a bit old, but for the future reader who has landed on this page, I'd like to let you know you're not alone, we feel your pain. This API is indeed as terrible as it looks.
The TL;DR answer is "the /images/create response format is undocumented; discard the output and query /images/XXX/json after your create call completes."
I wrote some orchestration tools a few years ago, and I found the /images/create API to be extremely annoying. But let's dive in:
There is no documented schema of the 200 response; the v1.19 docs simply gave examples of a few records. The v1.37 (latest at the time I write this) docs don't even go that far, no details are provided at all of the response.
The response is sent as Transfer-Encoding: chunked, and each record sent is preceded by the byte count in hex. Here's a low-level exerpt (bypassing curl, so we can see what actually gets sent on the wire):
host-4:~ rg$ telnet localhost 2375
Trying ::1...
Connected to localhost.
Escape character is '^]'.
POST /images/create?fromImage=jenkins/jenkins:latest HTTP/1.1
Host: localhost:2375
User-Agent: foo/1.0
Accept: */*
HTTP/1.1 200 OK
Api-Version: 1.39
Content-Type: application/json
Docker-Experimental: true
Ostype: linux
Server: Docker/18.09.1 (linux)
Date: Wed, 06 Feb 2019 16:53:19 GMT
Transfer-Encoding: chunked
39
{"status":"Pulling from jenkins/jenkins","id":"latest"}
5e
{"status":"Digest: sha256:abd3e3f96fbc3445c420fda590f37e2bd3377f69affd47b63b3d826d084c5ddc"}
45
{"status":"Status: Image is up to date for jenkins/jenkins:latest"}
0
Yes, it streams the image download progress -- client libraries that don't give low-level access to the chunked records may just concatenate the data before it's provided to you. As you encountered, early versions of the API returned JSON records with the only delimiter being the chunked transfer encoding, so client code received a concatenated block of undelimited JSON and had to parse it by tracking curlies/quotes/escape chars! It has since been updated to now emit records delimited by newlines, but can we count on them always being there? Who knows! This behavior changed without ceremony, and was not preserved if you call older versions of the API on newer daemons.
It returns 200 OK immediately, which doesn't represent success or failure. (Given the nature of the call, I'd imagine it should probably return 202 Accepted instead. Ideally, we'd get a Location header pointing to a new URL that we could use to query the progress/status.)
The response data returned is huge, spammy, and just... silly. If you have a docker instance listening on TCP, try curl -Nv -X POST http://yourdocker:2375/images/create?fromImage=jenkins/jenkins:latest -o /tmp/omgwtf.txt. You'll be amazed. A ton of bandwidth is wasted transferring server-rendered ASCII bar graphs!!!. In fact, the records return each layer's progress three different ways, as numeric fields for current and total bytes, as a bar graph, and as a pretty-printed string with MB or GB units. Why isn't this just rendered on the client? Great question.
Instead, you need your client to parse kilobytes or megabytes of spam.
The bar graph has a randomly escaped unicode rep of the > character, despite being safely inside a JSON string. Someone was just throwing escape calls at the wall to see what stuck? ¯\_(ツ)_/¯
The records themselves are pretty arbitrary. There's an id field that changes what it references, and the only way to know what kind of record it is to parse the human-readable string. Pulling from XXX vs Pulling fs layer vs Downloading etc. As far as I can tell, the only real way to know if it's done is to track all the ids, and ensure you get a Pull complete for each at the time that the socket closes.
You might be able to look for Status: Downloaded newer image for XXX but I'm not sure if there are multiple possible responses for this.
As I mentioned at the start, you'll probably have the best luck requesting /images/XXX/json after /images/create claims to be complete. The combination of the two calls will give a pretty reliable indication of whether /images/create worked or not.
Here's a longer block of concatenated client response that shows a few different record types. Edited for brevity:
{"status":"Pulling from jenkins/jenkins","id":"latest"}
{"status":"Pulling fs layer","progressDetail":{},"id":"ab1fc7e4bf91"}
{"status":"Pulling fs layer","progressDetail":{},"id":"35fba333ff52"}
{"status":"Pulling fs layer","progressDetail":{},"id":"f0cb1fa13079"}
{"status":"Pulling fs layer","progressDetail":{},"id":"3d1dd648b5ad"}
{"status":"Pulling fs layer","progressDetail":{},"id":"a9f886e483d6"}
{"status":"Pulling fs layer","progressDetail":{},"id":"4346341d3c49"}
..
"status":"Waiting","progressDetail":{},"id":"3d1dd648b5ad"}
{"status":"Waiting","progressDetail":{},"id":"a9f886e483d6"}
{"status":"Waiting","progressDetail":{},"id":"4346341d3c49"}
{"status":"Waiting","progressDetail":{},"id":"006f2208d67a"}
{"status":"Waiting","progressDetail":{},"id":"fb85cf26717d"}
{"status":"Waiting","progressDetail":{},"id":"52ca068dbca7"}
{"status":"Waiting","progressDetail":{},"id":"82f4759b8d12"}
...
{"status":"Downloading","progressDetail":{"current":110118,"total":10780995},"progress":"[\u003e ] 110.1kB/10.78MB","id":"35fba333ff52"}
{"status":"Downloading","progressDetail":{"current":457415,"total":45344749},"progress":"[\u003e ] 457.4kB/45.34MB","id":"ab1fc7e4bf91"}
{"status":"Downloading","progressDetail":{"current":44427,"total":4340040},"progress":"[\u003e ] 44.43kB/4.34MB","id":"f0cb1fa13079"}
{"status":"Downloading","progressDetail":{"current":817890,"total":10780995},"progress":"[===\u003e ] 817.9kB/10.78MB","id":"35fba333ff52"}
{"status":"Downloading","progressDetail":{"current":1833671,"total":45344749},"progress":"[==\u003e ] 1.834MB/45.34MB","id":"ab1fc7e4bf91"}
{"status":"Downloading","progressDetail":{"current":531179,"total":4340040},"progress":"[======\u003e ] 531.2kB/4.34MB","id":"f0cb1fa13079"}
{"status":"Downloading","progressDetail":{"current":1719010,"total":10780995},"progress":"[=======\u003e ] 1.719MB/10.78MB","id":"35fba333ff52"}
{"status":"Downloading","progressDetail":{"current":3205831,"total":45344749},"progress":"[===\u003e ] 3.206MB/45.34MB","id":"ab1fc7e4bf91"}
{"status":"Downloading","progressDetail":{"current":1129195,"total":4340040},"progress":"[=============\u003e ] 1.129MB/4.34MB","id":"f0cb1fa13079"}
{"status":"Downloading","progressDetail":{"current":2640610,"total":10780995},"progress":"[============\u003e ] 2.641MB/10.78MB","id":"35fba333ff52"}
{"status":"Downloading","progressDetail":{"current":1719019,"total":4340040},"progress":"[===================\u003e ] 1.719MB/4.34MB","id":"f0cb1fa13079"}
{"status":"Downloading","progressDetail":{"current":4586183,"total":45344749},"progress":"[=====\u003e ] 4.586MB/45.34MB","id":"ab1fc7e4bf91"}
{"status":"Downloading","progressDetail":{"current":3549922,"total":10780995},"progress":"[================\u003e ] 3.55MB/10.78MB","id":"35fba333ff52"}
{"status":"Downloading","progressDetail":{"current":2513643,"total":4340040},"progress":"[============================\u003e ] 2.514M
...
{"status":"Pull complete","progressDetail":{},"id":"6d9b49fc8a28"}
{"status":"Extracting","progressDetail":{"current":380,"total":380},"progress":"[==================================================\u003e] 380B/380B","id":"6302e8b6563c"}
{"status":"Extracting","progressDetail":{"current":380,"total":380},"progress":"[==================================================\u003e] 380B/380B","id":"6302e8b6563c"}
{"status":"Pull complete","progressDetail":{},"id":"6302e8b6563c"}
{"status":"Extracting","progressDetail":{"current":1548,"total":1548},"progress":"[==================================================\u003e] 1.548kB/1.548kB","id":"7348f018cf93"}
{"status":"Extracting","progressDetail":{"current":1548,"total":1548},"progress":"[==================================================\u003e] 1.548kB/1.548kB","id":"7348f018cf93"}
{"status":"Pull complete","progressDetail":{},"id":"7348f018cf93"}
{"status":"Extracting","progressDetail":{"current":3083,"total":3083},"progress":"[==================================================\u003e] 3.083kB/3.083kB","id":"c651ee7bd59e"}
{"status":"Extracting","progressDetail":{"current":3083,"total":3083},"progress":"[==================================================\u003e] 3.083kB/3.083kB","id":"c651ee7bd59e"}
{"status":"Pull complete","progressDetail":{},"id":"c651ee7bd59e"}
{"status":"Digest: sha256:abd3e3f96fbc3445c420fda590f37e2bd3377f69affd47b63b3d826d084c5ddc"}
{"status":"Status: Downloaded newer image for jenkins/jenkins:latest"}
This code runs the Internet now. =8-O
This particular endpoint actually returns chunked encoding. An example via curl:
$ curl -v -X POST http://localhost:4243/images/create?fromImage=base
* About to connect() to localhost port 4243 (#0)
* Trying ::1...
* Connection refused
* Trying 127.0.0.1...
* connected
* Connected to localhost (127.0.0.1) port 4243 (#0)
> POST /images/create?fromImage=base HTTP/1.1
> User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8y zlib/1.2.5
> Host: localhost:4243
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Fri, 07 Feb 2014 04:21:59 GMT
< Transfer-Encoding: chunked
<
* Connection #0 to host localhost left intact
{"status":"Pulling repository base"}{"status":"Pulling image (ubuntu-quantl) from base","progressDetail":{},"id":"b750fe79269d"}{"status":"Pulling image (ubuntu-quantl) from base, endpoint: https://cdn-registry-1.docker.io/v1/","progressDetail":{},"id":"b750fe79269d"}{"status":"Pulling dependent layers","progressDetail":{},"id":"b750fe79269d"}{"status":"Download complete","progressDetail":{},"id":"27cf78414709"}{"status":"Download complete","progressDetail":{},"id":"b750fe79269d"}{"status":"Download complete","progressDetail":{},"id":"b750fe79269d"}* Closing connection #0
Now I'm not sure how you go about parsing this in Python, but in Ruby, I can use Yajl like so:
parts = []
Yajl::Parser.parse(body) { |o| parts << o }
puts parts
{"status"=>"Pulling repository base"}
{"status"=>"Pulling image (ubuntu-quantl) from base", "progressDetail"=>{}, "id"=>"b750fe79269d"}
{"status"=>"Pulling image (ubuntu-quantl) from base, endpoint: https://cdn-registry-1.docker.io/v1/", "progressDetail"=>{}, "id"=>"b750fe79269d"}
{"status"=>"Pulling dependent layers", "progressDetail"=>{}, "id"=>"b750fe79269d"}
{"status"=>"Download complete", "progressDetail"=>{}, "id"=>"27cf78414709"}
{"status"=>"Download complete", "progressDetail"=>{}, "id"=>"b750fe79269d"}
{"status"=>"Download complete", "progressDetail"=>{}, "id"=>"b750fe79269d"}
Using Docker v1.9 I still having this problem to deal with.
Also have found an issue on Docker Github repository: Docker uses invalid JSON format in some API functions #16925
Where some contributor suggests to use Content-Type HTTP header like this: application/json; boundary=NL
This not worked for me.
Then, while struggling with my custom parser, found this question StackOverflow: How to handle a huge stream of JSON dictionaries?

Resources