Hey Durable Functions people, I've just hit my first instance of having a large message url returned in the output property of a status query response. E.g.
http:\\xx.xx.xx.xx\devstoreaccount1\durablefunctionshub-largemessages\a2f52688638149d69dec9e49d0a6ce35/history-00000000000000ca-executioncompleted-result.json.gz
Is there a pattern / recommended library for downloading and deserializing these?
EDIT: I realize that there’s the GZipStream class but I’m wondering if there’s a new/little known method within the functions framework to take care of it for me?
Related
I'm writing a data pipeline using Reactor and Reactor Kafka and use spring's Message<> to save
the ReceiverOffset of ReceiverRecord in the headers, to be able to use ReciverOffset.acknowledge() when finish processing. I'm also using the out-of-order commit feature enabled.
When an event process fails I want to be able to log the error, write to another topic that represents all the failure events, and commit to the source topic. I'm currently solving that by returning Either<Message<Error>,Message<myPojo>> from each processing stage, that way the stream will not be stopped by exceptions and I'm able to save the original event headers and eventually commit the failed messages at the button of the pipeline.
The problem is that each step of the pipline gets Either<> as input and needs to filter the previous errors, apply the logic only on the Either.right and that could be cumbersome, especially when working with buffers and the operator get 'List<Either<>>' as input. So I would want to keep my business pipeline clean and get only Message<MyPojo> as input but also not missing errors that need to be handled.
I read that sending those message erros to other channel or stream is a soulution for that.
Spring Integration uses that pattern for error handling and I also read an article (link to article) that solves this problem in Akka Streams using 'divertTo()':
I couldn't find documentation or code examples of how to implement that in Reactor,
is there any way to use Spring Integration error channel with Reactor? or any other ideas to implement that?
Not familiar with reactor per se, but you can keep the stream linear. The trick, since Vavr's Either is right-biased is to use flatMap, which would take a function from Message<MyPojo> to Either<Message<Error>, Message<MyPojo>>. If the Either coming in is a right (i.e. a Message<MyPojo>, the function gets invoked and otherwise it just gets passed through.
// Apologies if the Java is atrocious... haven't written Java since pre-Java 8
incomingEither.flatMap(
myPojoMessage -> ... // compute a new Either
)
Presumably at some point you want to do something (publish to a dead-letter topic, tickle metrics, whatever) with the Message<Error> case, so for that, orElseRun will come in handy.
I have some mutations where (for reasons outside of my control), the result isn't immediately apparent in my data layer, so I can't read it from the graph after performing the operation. I would like to be able to show a notification/alert to the user to indicate that the operation was successful and also to provide some information to them that was generated during the mutation (in this case, it's a tracking code provided by a 3rd-party API). Outside of this example, i'm sure there'll be other cases where I want to show notifications of various types (success, info, warning, error) to the user.
Mutation responses only include requested information in the output fields, so the obvious strategy to me is have a TransientNotification type that's also returned in the output. But it feels strange to do it in the GraphQL schema because the data doesn't actually get stored anywhere.
So i'm looking for ideas for the best way to approach this. It's a fairly common UI pattern, and it currently feels like i'm having to jump through a lot of hoops to make it work.
In general, returning field in the mutation response containing the notification (TransientNotification) seems like a reasonable approach here. You can use mutation query fragments to ensure that this field is fetched by the component that triggers the mutation, and then the data will be available to display in the UI.
I don't have a current problem really, but rather a question that I can't seem to find a satisfying answer to.
What is good practice to use when handling callbacks and invokations of NSURLConnection in objective-c? Let's look at some examples.
Let's say I have 3 different ViewControllers, and I've created a class that does the network calls with NSURLConnection in order to obtain a JSON from some site. All 3 VC's will be doing different kinds of network calls. How would one go about doing this without having to repeat code or have the network handling all spread out? And what if I need to make more network calls depending on the result of a previous call?
I use the normal NSURLConnection delegates, such as didReceiveResponse, didReceiveData, connectionDidFinishLoading etc. So what I do is I store the statusCode in didReceiveResponse and in connectionDidFinishLoading I call a function I name handleResponse which checks the statusCode and if the statusCode states that the call was successful I translate the NSData* object that I've stored the data I received from didReceiveData delegate. This is all fine, but the data I translate I want to be sent to the correct viewController.
There are several options to do this:
I let the delegate for the NSURLConnection calls be the
ViewController itself. This sucks, since I have to duplicate the
NSURLConnection code in every VC I want to make a network call,
obviously.
Or I could handle all the data in the Network class that performs
the NSURLConnection calls. This sucks since I get logic code in a
network class, and I don't really want that? This also requires that the Network Class knows about classes and data structures that it probably shouldn't. It's a network class and should not do anything but networking stuff.
My final approach, which I try for the first time now, is to let the
ViewController tell the network class that it wants to make a
NSURLConnection call, and sets a NSNotification message which the
Network class should invoke in the handleResponse method I
mentioned earlier. The Network class sends the data it received
through that NSNotification and the logic for handling that data
falls on the VC that asked for instead. To me this is most logical
and feels like a somewhat good way of doing things.
The problem with solution 3 is that I have to have different Notification methods to handle different kinds of data, which could result in quite a few such methods. If different VC's also use the same kind of data handling as another NSNotification method, then I have to duplicate that code as well. I'm also fairly new to using NSNotification in general and not very knowledgeable about its drawbacks.
Final question, let's say I want to make 4 different NSURLConnection calls, one after the other has been confirmed finished and successful. How would you go about that? I've done this a few times, but it ends up as a counter in the form of an enum to keep track of which network call was just made and which one should be next, which results in switch-statements executing certain code depending on which was just performed. I'm thinking about maybe implementing some sort of queue which goes through all calls in a sequence instead.
I appreciate any kind of input!
Most if not all of these problems are solved by AFNetworking. Specially the problems concerning having to duplicate requests or connections.
For the final question, for example you can do it with blocks AFNetworking provides such a structure, so you can specifically coordinate multiple calls.
AFNetworking
There are a lot of tutorials out there and the brief documentation of the library github site provides short, concise and easy to understand examples. Getting to use it takes a few minutes.
I'm new to AFNetworking and I'm interested in using it to handle a few dozen JSON requests (for example, using a web service's API that responds with JSON) for my application, but I'm having some trouble understanding how I should do this.
Could anyone offer some insight on how I'd go about accomplishing this? Like I said, I'm new to the library so an explanation would be greatly appreciated if you explain with code.
For a more specific example as to what I'm trying to do, here's the Clear Read API I'm using, where you pass the URL as a parameter in the URL and are returned a JSON response (the API extracts the article from a URL, removing the other bloat).
Example URL: http://api.thequeue.org/v1/clear?url=http://blogs.balsamiq.com/product/2012/02/27/uxstackexchange/&format=json
I'll be taking a few dozen URLs and running them all through that service and wish to save the results.
I was previously doing this with NSURLConnection in a for loop, firing off several dozen NSURLConnections, which was causing my data to be quite messed up by the end, with timeouts and whatnot from so many going at once.
I understand that it would be better to do only a few at a time, and AFNetworking seems perfect for this kind of problem, but I'm really just confused how I'd use it/subclass it or whatever.
I'd recommend starting with their Getting Started guide.
There's not much too it, really: build an AFJSONRequestOperation for each call to the API you want to make, and in the success callback, handle the deserialized JSON appropriately. If you have a bunch of calls to make, use AFHTTPClient to a) simply some of the repetitive work of building those operations, and b) use the client's operation queue to batch them all up. You can then throttle the number of requests in flight at once with the queue's setMaxConcurrentOperationCount: method.
I really like the message passing primitives that D implements. I have only seen examples of message passing within a program though. Is there support for distributing the messages over e.g. a network?
The message passing functions are in std.concurrency, which only deals with threads. So, the type of message passing used to pass messages between threads is for threads only. There is no RMI or anything like that in Phobos. That's not to say that we'll never get something like that in Phobos (stuff is being added to Phobos all the time), but it doesn't exist right now.
There is, however, the std.socket module which deals with talking to sockets, which is obviously network-related. I haven't used it myself, but it looks like it sends and receives void[]. So, it's not as nice as sending immutable objects around like you do with std.concurrency, but it does allow you to do network communication via sockets and presumably in a much nicer manner than if you were using C calls directly.
Seems that this has been considered. From the Phobos documentation (found it through Jonathan M Davis answer)
This is a low-level messaging API upon
which more structured or restrictive
APIs may be built. The general idea is
that every messageable entity is
represented by a common handle type
(called a Cid in this implementation),
which allows messages to be sent to
in-process threads, on-host processes,
and foreign-host processes using the
same interface. This is an important
aspect of scalability because it
allows the components of a program to
be spread across available resources
with few to no changes to the actual
implementation.
Right now, only in-process threads are
supported and referenced by a more
specialized handle called a Tid. It is
effectively a subclass of Cid, with
additional features specific to
in-process messaging.