I have an application developed with Erlang / MNesia and I am trying to implement redundancy for MNesia.
I want to add - remove nodes dynamically in runtime and handle synchronization of tables for every new joining node.
What is the best way to implement this using Erlang and MNesia?
Thanks.
You don't need to implement anything - mnesia already has these features. You can add and remove nodes from a mnesia cluster at runtime, add and remove table copies from nodes within the cluster, and mnesia:wait_for_tables/2 will let you cope with synchronization while adding nodes or table copies. Have a look at the mnesia documentation for more information.
Related
I am planning to use neo4j for an event management system. The involved entities are events, places, persons, organizations, so and so forth. But inorder to keep each org data separate I plan to create separate DB instance for each org. Becoz of this separation of Database instances, the 'Place' nodes are likely to get repeated in these mutliple db instances.
So, now, is it possible to aggregate for events based on Place nodes from all db instances? Or is it that I have to build my own custom aggregation like map-reduce?
Thanks in advance for helping me out on this..
In Neo4j 4.0, if you have a license for enterprise edition, you can leverage Neo4j Fabric, which should allow you to do exactly this: connect to a proxy instance, which must be configured to see other running db instances (which may be running on the same Neo4j dbms as the proxy instance, or which could instead be running on separate servers/clusters).
Then you can query across the graphs, aggregating and working with the result set across them as needed.
I am trying to setup a simple distributed application using erlang riak core framework. I read the documentation and it says I have to manually join the nodes via riak admin commands. I wanted to know what would happen if the entire cluster goes down.Do i need to have logic in my code to do a join on nodes every time cluster starts up or if there is a way to enlist all the nodes in a config?
When the nodes restart, they try to reconnect automatically, because when you joined some nodes with the command
riak_core:join("dev_1#127.0.0.1")
the data is saved locally to the node. If you want remove a node from cluster you should call
riak_core:leave()
Looking for some solutions to handle Erlang cluster partitions. Basically, whenever cluster participant is reachable again it should be added back to the cluster.The easiest solution is probably to use erlang node monitoring.
Are there any other / better solutions, maybe more dynamic which does not require fixed nodes list?
There are a few 3rd party libraries that don't have to be configured using a fixed node list. The two that I am familiar with are redgrid and erlang-redis_sd_epmd, there are probably others, but i'm just not familiar with them.
Both of these do have an external dependancy on redis which may or may not be desirable depending on what you decide is needed.
redgrid is the simpler implementation, but doesn't have a ton of features. Basically the erlang nodes connect to redis, and all erlang nodes connected to redis then establish connections to each other. You can associate meta-data with a node and retrieve it on another node.
erlang-redis_sd_epmd is a bit more complex, but allows a lot more configuration. For example instead of just automatically connecting all nodes, a node can publish services that it can perform, and a connecting node can look up nodes based on the services provided.
Not an off the shelf solution, but if you're already doing custom mods to ejabberd you can try integrating this code which resolves mnesia conflicts after cluster partitions.
https://github.com/uwiger/unsplit
What strategy does Mnesia use to define which nodes will store replicas of particular table?
Can I force Mnesia to use specific number of replicas for each table? Can this number be changed dynamically?
Are there any sources (besides the source code) with detailed (not just overview) description of Mnesia internal algorithms?
Manual. You're responsible for specifying what is replicated where.
Yes, as above, manually. This can be changed dynamically.
I'm afraid (though may be wrong) that none besides the source code.
In terms of documenation the whole Erlang distribution is hardly the leader
in the software world.
Mnesia does not automatically manage the number of replicas of a given table.
You are responsible for specifying each node that will store a table replica (hence their number). A replica may be then:
stored in memory,
stored on disk,
stored both in memory and on disk,
not stored on that node - in this case the table will be accessible but data will be fetched on demand from some other node(s).
It's possible to reconfigure the replication strategy when the system is running, though to do it dynamically (based on a node-down event for example) you would have to come up with the solution yourself.
The Mnesia system events could be used to discover a situation when a node goes down; given you know what tables were stored on that node you could check the number of their online replicas based on the nodes which were still online and then perform a replication if needed.
I'm not aware of any application/library which already manages this kind of stuff and it seems like a quite an advanced (from my point of view, at least) endeavor to make one.
However, Riak is a database which manages data distribution among it's nodes transparently from the user and is configurable with respect to the options you mentioned. That may be the way to go for you.
Can the Erlang ETS tables be shared among different processes? Thus, if I have two processes running on different Erlang running systems, can I somehow link them so that all the changes I do in one ETS table will be reflected in the other?
Within a single Erlang node, ETS tables can be fully shared by passing the public option to ets:new. (Beware that the table will be destroyed if its owner dies, though, unless you have set up an heir.)
If you need to share tables across several Erlang nodes, you need to use Mnesia.
You cannot "share" an ETS table between processes on different nodes, an ETS table is only accessible by processes on the node on which it was created. If you want to share ETS tables then you will need create a process on one node, the node with the table, and access the table from the other node through this process. It is not really that difficult.