If reactive programming is avoiding threads and use instead asynchronous message passing between objects, how is managed the scalability?
How decide when a new instance is needed?
Is there some api that manages this transparently?
According to the Reactive Manifesto:
"Reactive Systems can react to changes in the input rate by increasing
or decreasing the resources allocated to service these inputs."
If the components are isolated from eachother and don't share resources, they can be replicated - i.e. you can run multiple instances of a component at once and split the inputs between them.
This is possible because the inputs take the form of asynchronous messages. From any one instance's perspective, it only has to deal with the message(s) it receives. It doesn't care about the ordering of a message in relation to other messages, nor about how other instances are dealing with their messages.
So you can have as many (scaled up) or as few (scaled down) instances of a component running at once as needed, and because they're all independently receiving different messages - i.e. different parts of the problem - they won't conflict, double-up on or compete with eachother. This property of upwards or downwards scalability is referred to as "elasticity".
There are a number of libraries, frameworks and APIs that can help you to implement a reactive system, such as scaling of components in response to workload, as long as you correctly configure them and use the constructs they give you. The biggest one at the moment seems to be the ReactiveX API, encompassing RxJS, RxJava and other platform-specific frameworks.
Note: Reactive programming isn't necessarily against threads - only against threads which compete for shared resources. If you decouple threads from eachother, for example, by splitting up a workload and distributing different parts of it to different threads, that's a form of replication, a technique endorsed by reactive programming.
Related
I have been working on a project which collects data from various third party data sources and mines into our data stores (DI). We have been using Pentaho for this.
I want to know if this can also be done with ESB (Camel or Mule) ?
And what other features does ESB brings which DI do not offers ?
I have read lots of articles on both ESB and DI but none of them were able to resolve this query. I have also read about mule data connectors for third party data sources.
DI (Data Integration not 'dependency-injection') or ETL approaches tend to be long running batch-style jobs to approach the solution of moving data from System A to System B. The ESB or lightweight integration approach is generally to break up the task into smaller pieces (blocks of data, or single event per data item) and allow for other systems to subscribe to the data stream-- generally over an Enterprise Messaging System-- without having to impact System A, System B or the existing code project. This also means that there is no human dependency requirement in the project plan. If System C comes along, they do not necessarily require resources from the System B team to access the data stream
There are suitable use cases to have both in any given environment. However, in my experience (Big Data/MDM best practices tend to agree) is that if you have an originating stream of data, some other system will want to access the data stream at some point as well. If the ability to access the data stream without having to change existing code, systems or other teams within your organization sounds useful in your use case, than it would be a good idea to design for that up front and go with the ESB approach. This allows new interested consumers to come in and not have to rewrite the process used by the existing systems. ESB/Lightweight integration systems tend to allow that design pattern more efficiently than DI/ETL tools.
Some random thoughts:
ESB's support that "one bad record problem" by allowing you to route that to an error queue to have a human look at it and then republish
ETL/DI tend to have a straight-line happy-path speed advantage
ETL/DI start getting complicated once you go past the simple point-to-point integration use case
IMHO: ESB's are better at supporting versioning of data sets, services and data models.
ETL/DI tend to have more mature UI's for non-technical users to perform data mapping tasks
ESB's are really strong at supporting runtime decoupling of systems. If System B is down, the data just sits in a queue until it comes back up. No long running blocking thread or risk of having to restart a job
ESB has a slightly higher ramp-up curve
ETL/DI generally leads to ESB eventually (most vendors offer both a DI and ESB product)
I'm facing difficulties with finding the best CEP product for our problem. We need a distributed CEP solution with shared memory. The main reason for distribution isn't speeding up the process, but having a fallback in case of hardware or software problems on nodes. Because of that, all nodes should keep their own copy of the event-history.
Some less important requirements to the CEP product are:
- Open source is a big pre.
- It should run on a Linux system.
- Running in a Java environment would be nice.
Which CEP products are recommended?
A number of commercial non-open source products employ a distributed data grid to store the stateful event processing data in a fault-tolerant manner. My personal experience is with TIBCO BusinessEvents, which internally uses TIBCO ActiveSpaces. Other products claim do similar things, e.g., Oracle Event Processing uses Oracle Coherence.
Open source solutions, I wouldn't be aware that any of them offers functionality like this out of the box. With the right skills you might be able to use them in conjunction with a data grid (I've seen people try to use Drools Fusion together with infinispan), but there are quite a number of complexities that you need think about that a pre-integrated product would take care of for you (transaction boundaries, data access, keeping track of changes, data modeling).
An alternative you might consider if performance doesn't dictate a distributed/load-balanced setup could be to just run a hot standby, i.e., two engines performing the same CEP logic, but only one engine (the active one) actually triggering outgoing actions. The hot-standby engine would be just evaluating the CEP logic to have the data in its memory ready to take over in case of failure but not trigger outgoing actions as long as the other engine is running.
I want to create a simple "multi-agent" system consisting of three agents. For each agent, there is a type created encapsulating the mailbox processor. There are attributes common to all the agents (position, id etc.) and functions (sendMessage, move) and agents differs from each other with the implementation of mailbox processors (how the messages are processed). In addition, they may differ by other functions specific to particular agents. Every agent should also contain (as one of its attribute) a list of other agents, to whom it will be sending the messages. This is a just very simple model based on which I plan to play with the mailbox processors in F#.
In OOP, this would mean creating an agent interface (or abstract class), and all the particular agents would be inherited from this interface with their own implementation.
I know OOP is possible in F#, however I would rather stick to pure functional design. However, it seems to me that OOP is the most suitable approach in this case. I would be glad if you could give me any idea with respect to functional (F#) design? Thank you.
First of all, the functional style and object-oriented style in F# are not really in conflict.
Functional style consists of using immutable types, pure functions without side-effects and F# data types such as discriminated unions, functions etc.
Object-oriented style is more focused on how you organize code (using classes and interfaces), but the code can still be purely functional without using any mutable state.
In agent-based systems, it makes a good sense to use functional style in the implementation of the agent, but to organize the agents using classes. I think this is probably the best practice in F# (see also this article on encapsulating F# agents on MSDN).
In your example, you're saying that an agent keeps a list of other agents that it sends messages to. There are a few alternatives worth considering (if you want to avoid interfaces):
Expose an F# event (Event<'T>). This way, the agent simply exposes a notification and does not have to explicitly manage a list of other agents (and this design also allows other types of subscribers).
Keep a list of functions. If you just need to send messages to other agents, then you essentially need just an interface with a single method. In that case, you could keep a list of functions such as
Message -> unit.
I generally prefer exposing events - this way, the system is less tightly coupled and you can more easily compose agents in various ways (they do not have to implement a specific interface to be composed). This article discusses agent-based architectures from a higher-level perspective, and may be useful too.
Why would you want to stick to a pure functional design ? F# allows a clean combination of functional and OOP principles and I would make use of both mechanisms and leverage the power of the language.
If you want to combine the functional and OOP aspects, I would start by making your objects immutable. Thus you're using objects, but in a functional paradigm.
You have come up with an OOD, is it surprising that an OOP solution seems the most natural?
If you rewrote the design description with processes, and data transformations in mind it would naturally fall out as a FP design, and sound really awkward in OO with lots of 'er classes. As it is there is almost zero description of the types of data or what transformations need to take place. At first glance I would say an agent is a function of three arguments a mailbox, a message handler(or list of message handlers), and a list of other agents' mailboxes to contact. If future dispatch is based on the message then the message handler is a function of two args. the message, and the list of mailboxes.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 3 years ago.
Improve this question
I am currently looking for a good middleware to build a solution to for a monitoring and maintenance system. We are tasked with the challenge to monitor, gather data from and maintain a distributed system consisting of up to 10,000 individual nodes.
The system is clustered into groups of 5-20 nodes. Each group produces data (as a team) by processing incoming sensor data. Each group has a dedicated node (blue boxes) acting as a facade/proxy for the group, exposing data and state from the group to the outside world. These clusters are geographically separated and may connect to the outside world over different networks (one may run over fiber, another over 3G/Satellite). It is likely we will experience both shorter (seconds/minutes) and longer (hours) outages. The data is persisted by each cluster locally.
This data needs to be collected (continuously and reliably) by external & centralized server(s) (green boxes) for further processing, analysis and viewing by various clients (orange boxes). Also, we need to monitor the state of all nodes through each groups proxy node. It is not required to monitor each node directly, even though it would be good if the middleware could support that (handle heartbeat/state messages from ~10,000 nodes). In case of proxy failure, other methods are available to pinpoint individual nodes.
Furthermore, we need to be able to interact with each node to tweak settings etc. but that seems to be more easily solved since that is mostly manually handled per-node when needed. Some batch tweaking may be needed, but all-in-all it looks like a standard RPC situation (Web Service or alike). Of course, if the middleware can handle this too, via some Request/Response mechanism that would be a plus.
Requirements:
1000+ nodes publishing/offering continuous data
Data needs to be reliably (in some way) and continuously gathered to one or more servers. This will likely be built on top of the middleware using some kind of explicit request/response to ask for lost data. If this could be handled automatically by the middleware this is of course a plus.
More than one server/subscriber needs to be able to be connected to the same data producer/publisher and receive the same data
Data rate is max in the range of 10-20 per second per group
Messages sizes range from maybe ~100 bytes to 4-5 kbytes
Nodes range from embedded constrained systems to normal COTS Linux/Windows boxes
Nodes generally use C/C++, servers and clients generally C++/C#
Nodes should (preferable) not need to install additional SW or servers, i.e. one dedicated broker or extra service per node is expensive
Security will be message-based, i.e. no transport security needed
We are looking for a solution that can handle the communication between primarily proxy nodes (blue) and servers (green) for the data publishing/polling/downloading and from clients (orange) to individual nodes (RPC style) for tweaking settings.
There seems to be a lot of discussions and recommendations for the reversed situation; distributing data from server(s) to many clients, but it has been harder to find information related to the described situation. The general solution seems to be to use SNMP, Nagios, Ganglia etc. to monitor and modify large number of nodes, but the tricky part for us is the data gathering.
We have briefly looked at solutions like DDS, ZeroMQ, RabbitMQ (broker needed on all nodes?), SNMP, various monitoring tools, Web Services (JSON-RPC, REST/Protocol Buffers) etc.
So, do you have any recommendations for an easy-to-use, robust, stable, light, cross-platform, cross-language middleware (or other) solution that would fit the bill? As simple as possible but not simpler.
Disclosure: I am a long-time DDS specialist/enthusiast and I work for one of the DDS vendors.
Good DDS implementations will provide you with what you are looking for. Collection of data and monitoring of nodes is a traditional use-case for DDS and should be its sweet spot. Interacting with nodes and tweaking them is possible as well, for example by using so-called content filters to send data to a particular node. This assumes that you have a means to uniquely identify each node in the system, for example by means of a string or integer ID.
Because of the hierarchical nature of the system and its sheer (potential) size, you will probably have to introduce some routing mechanisms to forward data between clusters. Some DDS implementations can provide generic services for that. Bridging to other technologies, like DBMS or web-interfaces, is often supported as well.
Especially if you have multicast at your disposal, discovery of all participants in the system can be done automatically and will require minimal configuration. This is not required though.
To me, it looks like your system is complicated enough to require customization. I do not believe that any solution will "fit the bill easily", especially if your system needs to be fault-tolerant and robust. Most of all, you need to be aware of your requirements. A few words about DDS in the context of the ones you have mentioned:
1000+ nodes publishing/offering continuous data
This is a big number, but should be possible, especially since you have the option to take advantage of the data-partitioning features supported by DDS.
Data needs to be reliably (in some way) and continuously gathered to
one or more servers. This will likely be built on top of the
middleware using some kind of explicit request/response to ask for
lost data. If this could be handled automatically by the middleware
this is of course a plus.
DDS supports a rich set of so-called Quality of Service (QoS) settings specifying how the infrastructure should treat that data it is distributing. These are name-value pairs set by the developer. Reliability and data-availability area among the supported QoS-es. This should take care of your requirement automatically.
More than one server/subscriber needs to be able to be connected to
the same data producer/publisher and receive the same data
One-to-many or many-to-many distribution is a common use-case.
Data rate is max in the range of 10-20 per second per group
Adding up to a total maximum of 20,000 messages per second is doable, especially if data-flows are partitioned.
Messages sizes range from maybe ~100 bytes to 4-5 kbytes
As long as messages do not get excessively large, the number of messages is typically more limiting than the total amount of kbytes transported over the wire -- unless large messages are of very complicated structure.
Nodes range from embedded constrained systems to normal COTS
Linux/Windows boxes
Some DDS implementations support a large range of OS/platform combinations, which can be mixed in a system.
Nodes generally use C/C++, servers and clients generally C++/C#
These are typically supported and can be mixed in a system.
Nodes should (preferable) not need to install additional SW or
servers, i.e. one dedicated broker or extra service per node is
expensive
Such options are available, but the need for extra services depends on the DDS implementation and the features you want to use.
Security will be message-based, i.e. no transport security needed
That certainly makes life easier for you -- but not so much for those who have to implement that protection at the message level. DDS Security is one of the newer standards in the DDS ecosystem that provides a comprehensive security model transparent to the application.
Seems ZeroMQ will fit the bill easily, with no central infrastructure to manage. Since your monitoring servers are fixed, it's really quite a simple problem to solve. This section in the 0MQ Guide may help:
http://zguide.zeromq.org/page:all#Distributed-Logging-and-Monitoring
You mention "reliability", but could you specify the actual set of failures you want to recover? If you are using TCP then the network is by definition "reliable" already.
I have read one of erlang's biggest adopters is the telecom industry. I'm assuming that they use it to send binary data between their nodes and provide for easy redundancy, efficiency, and parallelism.
Does erlang actually send just the binary to a central node?
Is it directly responsible for parsing the binary data into actual voice? Or is it fed to another language/program via ports?
Is responsible for the speed in a telephone call, speed as in the delay between me saying something and you hearing it.
Is it possible that erlang is solely used for the ease in parallel behavior and c++ or similar for processing speed in sequential functions?
I can only guess at how things are implemented in actual telecom switches, but I can recommend an approach to take:
First, you implement everything in Erlang, including much of the low-level stuff. This probably won't scale that much since signal processing is very costly. As a prototype however, it works and you can make calls and whatnot.
Second, you decide on what to do with the performance bottlenecks. You can push them to C(++) and get a factor of roughly 10 or you can push them to an FPGA and get a factor of roughly 100. Finally you can do CMOS work and get a factor of 1000. The price of the latter approach is also much steeper, so you decide what you need and go buy that.
Erlang remains in control of the control backplane in the sense of what happens when you push buttons the call setup and so on. But once a call has been allocated, we hand over the channel to the lower layer. ATM switching is easier here because once the connection is set you don't need to change it (ATM is connection-oriented, IP is packet-oriented).
Erlangs distribution features are there primarily for providing redundancy in the control backplane. That is, we synchronize tables of call setups and so on between multiple nodes to facilitate node takeover in case of hardware failure.
The trick is to use ports and NIFs post prototype to speed up the slower parts of the program.