For monitoring purposes, I need to log and measure every grain call (already available through Grain Call Filters), but also every grain activation/deactivation to have Grain Type counters, also for auto-scaling purposes.
I haven't found any global hooks besides Incomming and Outgoing grain call filters
Is there any way to get into the lifecycle of Grains in a global manner, besides the Incomming and Outgoing call filters?
Yes, you can using a feature in orleans called life-cycle participation which is available for grains and silos as well.
You should check out the documentation site
Related
There is a couple of confusing points in the documentation that make me struggle to understand how exactly distribution across the cluster happens in Orleans. Hence, the questions.
Question #1
Orleans claims to have a built-in distribution capabilities to distribute across multiple servers. To me it sounds that Orleans can act as a load balancer itself and can scale out automatically. Thus, if I deploy Orleans app to several servers, then service discovery and load management should happen automatically, correct?
In this case, why some docs and articles suggest using other tools, like Ocelot or Consul, as a single entry point to Orleans cluster?
Question #2
I would like to use simple but distributed in-memory storage across several servers, like Redis or Apache Ignite, and I would like to know if it's possible to use a simple grain as this kind of a data storage?
Let's say, one grain will store a collection of restaurants and some other grain will keep track of the last 1000 visitors for selected restaurant. Can I activate these 2 grains only once as a singleton collection, add or remove records to each collection, and use these 2 grains as in-memory storage evenly available to all nodes in the cluster? Also, if answer is yes, do I need to add locks to these collections or each grain always exists in a single thread?
Service discovery and load management happen automatically indeed.
Consul is not a strong required. The only external requirement is a Membership table provider - something that is used internally by Orleans Clustering. There are many build in Membership table providers that come already built-in with Orleans. For example, Azure table storage. all you need is to configure Orleans to use it and of course have Azure storage account. Consul is another alternative to Membership table provider and there are more.
Another thing that does not come built-in is infrastructure scaling. If your service demand increases, something need to ask the infrastructure provider (Cloud Provider) to add more Servers. Once servers are added, Orleans will automatically adjust the workload and load balance across the new servers as well. But figuring out that more servers are needed and adding them is not done by Orleans itself (there likely some externally contributed tools to do that. maybe K8 can be configured to do that? I am not completely sure about that).
Yes, you can use those 2 grains as in-memory storage, just like you wrote. And no, you do not need to use locks. All grains are single threaded.
Given a microservice that owns a relational database and needs to scale horizontally, I see two approaches to provisioning of the database server:
provide each instance of the service with it's own DB server instance with a coupled process lifecycle
OR
have the instances connect to a shared (by identical instances of the same service) independent db server or cluster
With an event driven architecture and the former approach, each instance of the microservice would need to process each event and take the appropriate action to mutate its own isolated state. This seems inefficient.
Taking the latter approach, only one instance has to process the event to achieve the same effect but as a mutation of the shared state. One must ensure each event is processed by only one instance of the given microservice (is this trivial?) to avoid conflict.
Is there consensus on preferred approach here? What lessons has your experience taught you on this?
I would go with the first approach, a service local DB. Each instance has its own DB instance. This enables to change the persistence layer between versions of the service.
Changing the ER model otherwise would lead to conflicts. You would also be able to change to a NoSQL solution with this approach easily.
With the event driven design, I can recommend this book: Designing Event Driven Systems
As I see it, a service receives an request that leads to an Event. This Event is consumed by the other instances of the service, therefore the request doesn't need to be processed again, but the result has to be copied to the instances state.
I'm looking at the Apple networking guidelines that suggest that the user should try to work with the highest level of abstraction possible when dealing with networking.
I'm working on a client-server app, where the server is master, and an iOS device is slave. These communicate over HTTP, establishing a connection that lives for the lifetime of the app's usage session. The app and the server synchronize assets over this connection.
My question is - what level of abstraction is appropriate for implementing bi-directional sync over HTTP? Is it sockets, NSURLConnection, some AFNetworking subclass, input/output streams?
There are a lot of possible good answers to this. I think all I can do is offer one pattern which has worked well for me but it may not apply to your needs and use cases. To restate my comment above "whatever you do will be a tradeoff between responsiveness, power consumption, data consistency, and implementation cost."
The level of abstraction I aim for is a set of service objects which expose an interface in terms of the application's domain models. The rest of the app, primarily objects in the controller layer, should be able to communicate with these services by passing models to methods (e.g. "fetchUserWithId:userId" or "createUser:user") and without any awareness of the urls, paths, or HTTP verbs involved at the network layer.
Those service objects can map domain model operations into paths, HTTP verbs, and possibly request bodies or headers. In most cases I find that the services themselves can then share a lower level service which accepts those values and constructs the actual HTTP request. This provides a single location to configure host names, set global headers, and manage a request queue via NSURLRequest, NSURLSession, AFNetworking, or whatever library you prefer.
I'll include completion blocks on my service object methods so that controllers can be notified of success or failure but try not to use those blocks to pass models back up to the controller layer. Instead I prefer to have controllers monitor Core Data or some other persistence layer and react to changes. That way controllers remain flexible and respond to any update in the models they are concerned with and do not assume that they are aware of all possible sources of changes to those models.
So far none of this addresses how you should check for remote changes to your models. The best option may be to design a system which does not need to do so. What if your client obtained a set of recent changes only when posting data to the server, could it still provide a good user experience? Could the server use push notifications to occasionally notify clients of updates?
If you must check for changes sockets or long polling are usually more responsive than short polling but it may be hard for roaming mobile clients to keep those connections open. All of these approaches also tend to keep the client's radios active and consume lots of power in the process.
Without knowing more about the problem I'd default to short polling but try to design interactions which allow this to be as infrequent as possible (e.g. one check when the app resumes). I also use HTTP features (etags, if-modified-since, or custom content ranges) to limit the size of responses when there are no changes. If you have a good service layer managing network requests that also gives you a good place to introduce rate limiting. Allowing controllers to express interest to fetching up to date information but deferring to the services to throttle or batch requests based on what the rest of the app is doing (e.g. don't repeat the same request if those models were updated recently unless the user deliberately triggered the action).
I'm in the planning phase of our new site - it's an extension of some mobile apps we've built. We want to provide our users with a central point for communication and also provide features for users who don't want to/can't use the mobile apps. One of the features we're looking at adding is a reputation system similar in nature to the SO badge system. We're designing the system to use SOA.
I don't want to have to code all of this logic into the main app as discreet chunks. I'm thinking of creating a means to accomplish this which will allow us to define new thresholds and rules for gaining reputation and have them injected into some service. The two ways I've thought of doing this so far are:
To look for certain traits in a users actions and respond, this would mean having a service running that can run through the 'plugged in' award definitions and check for thresholds that have been met and respond appropriately.
To fire events when the user performs actions - listen out for those events and respond appropriately. Because the services which will be carrying out these actions are running in separate app domains potentially on separate servers the only way I can see having a central message bus to listen and respond to these events is by using something like MassTransit, nServiceBus or Rhino.Esb.
I know that using a service bus can very easily be inappropriately designed into an application that simply doesn't need it and most times - unless you're integrating disparate, heterogenous systems - you most likely won't need one when designing a new system but I'm a bit lost for options as to the best way to do this. I don't like the idea of having a service hammer the Db all the time in the background. But it does sound like it might be a lot simpler early on - later on - I dread to think!
Has anyone here designed a system like this? How did you accomplish this? We're designing for high throughput as we expect there will be times when the system will need to be able to cope with bursts of users.
I've designed a system that had similar requirements. To achieve this the key elements were:
Plugins
Event messaging - using Emesary
The basic concept is that the core is not aware of exactly which module will perform any given task.
The messages are defined and at points within the system they are dispatched. The sender is not aware if the message is required. This effectively decouples vast chunks of the system.
So to perform a job some code is plugged in, that registers with the event messaging bus and will receive messages. When it receives a message that it needs to process it will process it.
The Emesary code is extremely small and efficient in the first instance I've called it (Emesary and you're free to use it; or from Emesary CodePlex
As the system becomes more complex it is possible that there are lots of events flying about, if you get more than 20k a second it was always in my design to add filtering and routing (implemented by the recipient interface being extended to allow a recipient to specify messages it wants to receive during registration). I've never needed to add this filtering because Emesary is sufficiently efficient that it is the processing of the messages that takes the time.
I've build a version of Emesary which bridges two Notifiers across disparate systems using WCF, Corba and TCP/IP. I investigated using RabbitMQ and decided it was possible to use this underneath Emesary if needed.
Base Class Diagram
Scalable server.
This is a fairly complex example however it shows where Emesary fits in. In this diagram anything with a drop shadow can have multiple instances and this is managed outside of what I'm trying to explain here.
I am implementing some networking stuff in our project. It has been decided that the communication is very important and we want to do it synchronously. So the client sends something the server acknowledges.
Are there some general best practices for the interaction between the client and the server. For instance if there isn't an answer from the server should the client automatically retry? Should there be a timeout period before it retries? What happens if the acknowledgement fails? At what point do we break the connection and reconnect? Is there some material? I have done searches but nothing is really coming up.
I am looking for best practices in general. I am implementing this in c# (probably with sockets) so if there is anything .Net specific then please let me know too.
First rule of networking - you are sending messages, you are not calling functions.
If you approach networking that way, and don't pretend that you can call functions remotely or have "remote objects", you'll be fine. You never have an actual "thing" on the other side of the network connection - what you have is basically a picture of that thing.
Everything you get from the network is old data. You are never up to date. Because of this, you need to make sure that your messages carry the correct semantics - for instance, you may increment or decrement something by a value, you should not set its value to the current value plus or minus another (as the current value may change by the time your message gets there).
If both the client and the server are written in .NET/C# I would recommend WCF insted of raw sockets as it saves you a from a lot of plumbing code with serialization and deserialization, synchronization of messages etc.
That maybe doesn't really answer your question about best practices though ;-)
The first thing to do is to characterize your specific network in terms of speed, probability of lost messages, nominal and peak traffic, bottlenecks, client and server MTBF, ...
Then and only then you decide what you need for your protocol. In many cases you don't need sophisticated error-handling mechanisms and can reliably implement a service with plain UDP.
In few cases, you will need to build something much more robust in order to maintain a consistent global state among several machines connected through a network that you cannot trust.
The most important thing I found is that messages always should be stateless (read up on REST if this means nothing to you)
For example if your application monitors the number of shipments over a network do not send incremental updates (+x) but always the new total.
In a common think about network programming, I think you should learn about :
1. Socket (of course).
2. Fork and Threading.
3. Locking process (use mutex or semaphore or others).
Hope this help..