Decode JSON from stream of data Docker GO SDK - docker

I want to use Client.ContainerStats(ctx context.Context, containerID string, stream bool) method to get streaming stats of a container.
From what I understand, if I pass true to stream parameter, Docker will not close connection and periodically sends JSON containing stats of a container.
However, I don't know how decode JSON because I don't know where JSON data start and end.
What I'm using right now is that I don't use stream option and just fetch data periodically then decode it like this.
stats, err := dockerClient.ContainerStats(ctx, container.ContainerID, false)
msgBytes, _ := ioutil.ReadAll(stats.Body)
var containerStats ContainerStats
err = json.Unmarshal(msgBytes, &containerStats)
What I'm looking for is a function that block when I call it, then when it receives JSON data (I mean complete JSON data that can be decoded) it will return struct containing data that was decoded from JSON and then I can call that function again to get next stat without having to make a new request to Docker.

In your case, you have multiple options:
Map the result on a custom struct
Map the result on a map[string]interface{}
If you want to map by using a custom struct you can do something like this:
type myStruct struct {
Id string `json:"id"`
Read string `json:"read"`
Preread string `json:"preread"`
}
// perform actions to retrieve logs in stats
//...
var containerStats myStruct
json.NewDecoder(stats.Body).Decode(&containerStats)
fmt.Println(containerStats.Id)
With this solution, you have to decide which fields you want to map.
However, if you do not want to specify fields, you can perform something like this:
//Perform actions to retrieve logs in stats
//...
var containerStats map[string]interface{}
json.NewDecoder(stats.Body).Decode(&containerStats)
fmt.Println(containerStats["id"])
To conclude, if you have to manipulate your data, I recommend you to use the first solution by using custom structure.
EDITED: handle stream
By passing stream parameter to true, the docker api will return an io.ReadCloser which will be updated. Then, it's up to the caller to close the io.ReadCloser returned.
What you have to do is to perdiodically read the buffer value.
type myStruct struct {
Id string `json:"id"`
Read string `json:"read"`
Preread string `json:"preread"`
CpuStats cpu `json:"cpu_stats"`
}
type cpu struct {
Usage cpuUsage `json:"cpu_usage"`
}
type cpuUsage struct {
Total float64 `json:"total_usage"`
}
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 5 * time.Second)
cli, e := client.NewEnvClient()
if e != nil {
panic(e)
}
stats, e := cli.ContainerStats(ctx, "container_id", true)
if e != nil {
fmt.Errorf("%s", e.Error())
}
decoder := json.NewDecoder(stats.Body)
var containerStats myStruct
for {
select {
case <-ctx.Done():
stats.Body.Close()
fmt.Println("Stop logging")
return
default:
if err := decoder.Decode(&containerStats); err == io.EOF {
return
} else if err != nil {
cancel()
}
fmt.Println(containerStats.CpuStats.Usage.Total)
}
}
}
In this example, we are decoding the stats.Body ReadCloser when new data arrives, printing the total cpu usage, and closing the stream after 5 seconds.

Related

I can't get array and put in the another array swift from Firebase storage

I want get folders and images from Firebase storage. On this code work all except one moment. I cant append array self.collectionImages in array self.collectionImagesArray. I don't have error but array self.collectionImagesArray is empty
class CollectionViewModel: ObservableObject {
#Published var collectionImagesArray: [[String]] = [[]]
#Published var collectionImages = [""]
init() {
var db = Firestore.firestore()
let storageRef = Storage.storage().reference().child("img")
storageRef.listAll { (result, error) in
if error != nil {
print((error?.localizedDescription)!)
}
for prefixName in result.prefixes {
let storageLocation = String(describing: prefixName)
let storageRefImg = Storage.storage().reference(forURL: storageLocation)
storageRefImg.listAll { (result, error) in
if error != nil {
print((error?.localizedDescription)!)
}
for item in result.items {
// List storage reference
let storageLocation = String(describing: item)
let gsReference = Storage.storage().reference(forURL: storageLocation)
// Fetch the download URL
gsReference.downloadURL { url, error in
if let error = error {
// Handle any errors
print(error)
} else {
// Get the download URL for each item storage location
let img = "\(url?.absoluteString ?? "placeholder")"
self.collectionImages.append(img)
print("\(self.collectionImages)")
}
}
}
self.collectionImagesArray.append(self.collectionImages)
print("\(self.collectionImagesArray)")
}
//
self.collectionImagesArray.append(self.collectionImages)
}
}
}
If i put self.collectionImagesArray.append(self.collectionImages) in closure its works but its not what i want
The problem is caused by the fact that calling downloadURL is an asynchronous operation, since it requires a call to the server. While that call is happening, your main code continues so that the user can continue to use the app. Then when the server returns a value, your closure/completion handler is invoked, which adds the URL to the array. So your print("\(self.collectionImagesArray)") happens before the self.collectionImages.append(img) has ever been called.
You can also see this in the order that the print statements occur in your output. You'll see the full, empty array first, and only then see the print("\(self.collectionImages)") outputs.
The solution for this problem is always the same: you need to make sure you only use the array after all the URLs have been added to it. There are many ways to do this, but a simple one is to check whether your array of URLs is the same length as result.items inside the callback:
...
self.collectionImages.append(img)
if self.collectionImages.count == result.items.count {
self.collectionImagesArray.append(self.collectionImages)
print("\(self.collectionImagesArray)")
}
Also see:
How to wait till download from Firebase Storage is completed before executing a completion swift
Closure returning data before async work is done
Return image from asynchronous call
SwiftUI: View does not update after image changed asynchronous

Issue trying to work with NWConnection in Network.Framework

I am writing an application which sends commands and receives data via UDP network connection with a device.
I am trying to create an object which handles all the network related tasks.
I want the object to make a connection, send a command string to the connection, and receive data from the connection. I can get all the pieces to work but they happen out of sync with the running program. It is hard to explain but let me show you the code first then explain the issue.
import Foundation
import Network
var myConnection: NWConnection?
var backToString = "test"
class NetworkUDP: NSObject{
func makeConnection(){
let myPort = NWEndpoint.Port(rawValue: 50536)
let myHost = NWEndpoint.Host("192.168.7.239")
myConnection = NWConnection(host: myHost, port: myPort!, using: .udp)
myConnection?.start(queue: .main)
}
func send(myCommand: String) {
myConnection?.send(content: myCommand.data(using: String.Encoding.utf8), completion: NWConnection.SendCompletion.contentProcessed({(NWError) in print(NWError as Any)}))
print(myCommand)
}
func receive() {
myConnection?.receiveMessage { (data, context, isComplete, error) in
if (data != nil) {
backToString = String(decoding: data!, as : UTF8.self)
print(backToString)
} else {
print("Data = nil")
}
}
}
}
So if I instantiate this object and issue makeConnection(), send(myCommand: aCommand), receive() statements from a ViewController everything works but the functions run and return before the commands are actually sent and values returned.
In other words, I can't return the value of backToString as a return value from the function receive(). If I do it will always return "test" which is the initial value. It will eventually be replaced with the string value returned from the device but only after a delay and only after the function has already returned.
What I want to do is make a receive function of the form receive() -> String where String is the text string returned from the device that was sent a command. But I guess this isn't possible because the receive function returns before backToString actually receives any data. I am a bit confused by this. It seems like the function should stay halted until a value is received but it doesn't it just returns before the backToString variable receives the data from the UDP connection. However the print statement in the receive function does print the correct data but it only does so after the function receive has already returned.

Rx: How to modify a shared source observable within a retry

Top Level Question:
I want to know how, within a retry, I can modify its source observable if it is an observable shared between multiple subscribers (in this case a BehaviorSubject/Relay).
Solution(s) I have considered:
The suggestion of using defer from this post doesn't seem to naturally port over if the source observable needs to be shared.
Use case (to fully elaborate the question)
Say I have a server connection object that, when initialized, connects to an url. Once it is created, I can also use it to get a data stream for a particular input.
class ServerConnection {
var url: URL
init(url: URL)
func getDataStream(input: String) -> Observable<Data> // the observable also errors when the instance is destroyed.
}
However, one particular url or another may be broken or overloaded. So I may want to obtain the address of a mirror and generate a new ServerConnection object. Let's say I have such a function.
// At any point in time, gets the mirror of the url with the lowest load
func getLowestLoadMirror(url: URL) -> URL {}
Ideally, I want this "mirror url" switching should be an implementation detail. The user of my code may only care about the data they receive. So we would want to encapsulate this logic in a new class:
class ServerConnectionWithMirrors {
private var currentConnection: BehaviorRelay<ServerConnection>
init(startingURL: URL)
func dataStream(for inputParams: String) -> Observable<Data>
}
// usage
let connection = ServerConnectionWithMirrors(startingURL: "www.example.com")
connection.dataStream(for: "channel1")
.subscribe { channel1Data in
// do something with channel1Data
}.disposed(by: disposeBag)
connection.dataStream(for: "channel2")
.subscribe { channel2Data in
// do something with channel2Data
}.disposed(by: disposeBag)
How should I write the dataStream() function for ServerConnectionWithMirrors? I should be using retries, but I need to ensure that the retries, when faced with a particular error (ServerOverLoadedError) update the value on the behaviorRelay.
Here is code that I have so far that demonstrates the crux at what I am trying to do. One problem is that multiple subscribers to the behaviorRelay may all update it in rapid succession when they get an error, where only one update would do.
func dataStream(for inputParams: String) -> Observable<Data> {
self.currentConnection.asObservable()
.flatMapLatest { server in
return server.getDataStream(input: inputParams)
}
.retryWhen { errors in
errors.flatMapLatest { error in
if error is ServerOverLoadedError {
self.currentConnection.accept(ServerConnection(url: getLowestLoadURL()))
} else {
return Observable.error(error)
}
}
}
}
The answer to your top level question:
I want to know how, within a retry, I can modify its source observable if it is an observable shared between multiple subscribers (in this case a BehaviorSubject/Relay).
You cannot modify a retry's source observable from within the retry. (full stop) You cannot do this whether it is shared or not. What you can do is make the source observable in such a way that it naturally updates its data for every subscription.
That is what the question you referred to is trying to explain.
func getData(from initialRequest: URLRequest) -> Observable<Data> {
return Observable.deferred {
var correctRequest = initialRequest
let correctURL = getLowestLoadMirror(url: initialRequest.url!)
correctRequest.url = correctURL
return Observable.just(correctRequest)
}
.flatMapLatest {
getDataFromServer(request: $0)
}
.retryWhen { error in
error
.do(onNext: {
guard $0 is ServerOverloadedError else { throw $0 }
})
}
}
With the above code, every time deferred is retried, it will call its closure and every time its closure is called, the URL will the lowest load will be used.

RxSwift, how do I chain different observables

I am still a beginner in Reactive programming, and RxSwift in general.
I want to chain two different operation. In my case I simply want to download a zip file from a web server, and then unzip it locally.
I also want, at the same time to show the progress of the downloaded files.
So I started creating the first observable:
class func rx_download(req:URLRequestConvertible, testId:String) -> Observable<Float> {
let destination:Request.DownloadFileDestination = ...
let obs:Observable<Float> = Observable.create { observer in
let request = Alamofire.download(req, destination: destination)
request.progress { _, totalBytesWritten, totalBytesExpectedToWrite in
if totalBytesExpectedToWrite > 0 {
observer.onNext(Float(totalBytesWritten) / Float(totalBytesExpectedToWrite))
}
else {
observer.onNext(0)
}
}
request.response { _, response, _, error in
if let responseURL = response {
if responseURL.statusCode == 200 {
observer.onNext(1.0)
observer.onCompleted()
} else {
let error = NSError(domain: "error", code: responseURL.statusCode, userInfo: nil)
observer.onError(error)
}
} else {
let error = NSError(domain: "error", code: 500, userInfo: nil)
observer.onError(error)
}
}
return AnonymousDisposable () {
request.cancel()
}
}
return obs.retry(3)
}
After that, I create a similar function for the unzip
class func rx_unzip(testId:String) -> Observable<Float> {
return Observable.create { observer in
do {
try Zip.unzipFile(NSURL.archivePath(testId), destination: NSURL.resourceDirectory(testId), overwrite: true, password: nil)
{progress in
observer.onNext(Float(progress))
}
} catch let error {
observer.onError(error)
}
observer.onCompleted()
return NopDisposable.instance
}
}
For now I have this logic on the "View model layer", so I download-> subscribe on completed-> unzip
What I want is to combine the two Observable in one, in order to perform the download first, then on completed unzip the file. Is there any way to do this?
Concat operator requires the same data type
Indeed, the concat operator allows you to enforce the sequence of observables, however an issue you might encounter with using concat is that the concat operator requires that the Observables have the same generic type.
let numbers : Observable<Int> = Observable.from([1,2,3])
let moreNumbers : Observable<Int> = Observable.from([4,5,6])
let names : Observable<String> = Observable.from(["Jose Rizal", "Leonor Rivera"])
// This works
numbers.concat(moreNumbers)
// Compile error
numbers.concat(names)
FlatMap operator allows you to chain a sequence of Observables
Here's an example.
class Tag {
var tag: String = ""
init (tag: String) {
self.tag = tag
}
}
let getRequestReadHTML : Observable<String> = Observable
.just("<HTML><BODY>Hello world</BODY></HTML>")
func getTagsFromHtml(htmlBody: String) -> Observable<Tag> {
return Observable.create { obx in
// do parsing on htmlBody as necessary
obx.onNext(Tag(tag: "<HTML>"))
obx.onNext(Tag(tag: "<BODY>"))
obx.onNext(Tag(tag: "</BODY>"))
obx.onNext(Tag(tag: "</HTML>"))
obx.onCompleted()
return Disposables.create()
}
}
getRequestReadHTML
.flatMap{ getTagsFromHtml(htmlBody: $0) }
.subscribe (onNext: { e in
print(e.tag)
})
Notice how getRequestReadHTML is of type Observable<String> while the function getTagsFromHtml is of type Observable<Tag>.
Using multiple flatMaps can increase emission frequency
Be wary however, because the flatMap operator takes in an array (e.g. [1,2,3]) or a sequence (e.g. an Observable) and will emit all of the elements as an emission. This is why it is known to produce a transformation of 1...n.
If you defined an observable such as a network call and you are sure that there will only be one emission, you will not encounter any problems since its transformation is a 1...1 (i.e. one Observable to one NSData). Great!
However, if your Observable has multiple emissions, be very careful because chained flatMap operators will mean emissions will exponentially(?) increase.
A concrete example would be when the first observable emits 3 emissions, the flatMap operator transforms 1...n where n = 2, which means there are now a total of 6 emissions. Another flatMap operator could again transform 1...n where n = 2, which means there are now a total of 12 emissions. Double check if this is your expected behavior.
You can use concat operator to chain these two Observables. The resulting Observable will send next values from the first one, and when it completes, from the second one.
There is a caveat: you will get progress values ranging from 0.0 to 1.0 from rx_download and then again the progress from rx_unzip will start with 0.0. This might be confusing to the user if you want to show the progress on a single progress view.
A possible approach would be to show a label describing what is happening along with the progress view. You can map each Observable to a tuple containing the progress value and the description text and then use concat. It can look like that:
let mappedDownload = rx_download.map {
return ("Downloading", $0)
}
let mappedUnzip = rx_download.map {
return ("Unzipping", $0)
}
mapped1.concat(mapped2)
.subscribeNext({ (description, progress) in
//set progress and show description
})
Of course, there are many possible solutions, but this is more of a design problem than a coding one.

Using jwt-go Library - Key is invalid or invalid type

I am trying to pass in a token to the "Parse(token String, keyFunc Keyfunc)" GO routine defined in this GO-library (http://godoc.org/github.com/dgrijalva/jwt-go) for JWT-token parsing/validation.
When I pass the token to this function -
token, err := jwt.Parse(getToken, func(token *jwt.Token) (interface{}, error) {
return config.Config.Key, nil
})
I get an error which says "Key is invalid or invalid type".
My config struct looks like this in config.go file -
config struct {
Key string
}
Any suggestions to solve this problem? The token I am passing is a JWT token.
config struct {
Key string
}
Key needs to be a []byte
I am not sure if this can be an issue for someone else.
My problem was I was using Signing Method "SigningMethodES256" but "SigningMethodHS256" or Any with SigningMethodHS* works fine.
If someone knows why this is an issue please answer.
Other way is to do something like this -
token, err := jwt.Parse(getToken, func(token *jwt.Token) (interface{}, error) {
return []byte(config.Config.Key), nil
})
The whole idea being that the Parse function returns a slice of bytes.
Taking a look at the function signatures in the GoDoc for github.com/dgrijalva/jwt-go we see:
func Parse(tokenString string, keyFunc Keyfunc) (*Token, error)
type Keyfunc func(*Token) (interface{}, error)
Keyfunc requires you to return (interface{}, error). Given the mysterious interface{} type, you might expect to be fine returning a string; however, a peek under the hood reveals that Parse() tries to Verify(), which attempts the following type assertion with your interface{} value as the key:
keyBytes, ok := key.([]byte)
That will succeed for []byte types, but will fail for string types. When that fails, the result is the error message you are getting. Read more about type assertions in the Effective Go documentation to learn why it fails.
Example: https://play.golang.org/p/9KKNFLLQrm
package main
import "fmt"
func main() {
var a interface{}
var b interface{}
a = []byte("hello")
b = "hello"
key, ok := a.([]byte)
if !ok {
fmt.Println("a is an invalid type")
} else {
fmt.Println(key)
}
key, ok = b.([]byte)
if !ok {
fmt.Println("b is an invalid type")
} else {
fmt.Println(key)
}
}
[104 101 108 108 111]
b is an invalid type
This is working for me.
token.SignedString([]byte("mysecretkey"))
func GenerateJWT(email string, username string) (tokenString string, err error) {
expirationTime := time.Now().Add(1 * time.Hour)
claims := &JWTClime{
Email: email,
Username: username,
StandardClaims: jwt.StandardClaims{
ExpiresAt: expirationTime.Unix(),
},
}
token := jwt.NewWithClaims(jwt.SigningMethodHS256, claims)
tokenString, err = token.SignedString([]byte("mysecretkey"))
return
}

Resources