I have two Aerospike servers cluster runnning with a replication factor of 2. Both servers have the same replicated objects count, which means all records are replacated. But still the monitoring panel shows incoming and outgoing migration going on.
This happened after I restarted one of the servers. Now de I/O rate in both servers are above it was before restarting.
Why is this happening?
When a node leaves the cluster, the partition id of any partition that node was a member of advances. When the node returns, they share their partition info with the cluster and migrations are required for any partition the returning node is a member of. This is done because while the node was down, the remaining node may have taken on writes.
For replication factor 2 with 2 nodes, both nodes are members of all partitions.
Related
I have 2 swarm nodes and I whish that in case one node shut down, the other one rearrange all services to itself.
Right now I have one leader(manager) and one worker, and it works perfectly if the worker goes down, because leader rearranges all services to itself.
My problem here is when leader goes down and no one assumes services within it.
I already tried with two managers, but didn't works.
So I am thinking about to let all my services in the worker node so if leader node goes down there is no problem at all and if worker node goes down, leader node would rearrange all services to itself.
I tried with
deploy:
placement:
constraints:
- "node.role!=manager"
But it also does not works, because it will never instance this service in a manager node.
So I would like to ask if there is any way to make those two nodes to rearrange all services to itself in case other goes down?!
or
There is an way to configure a service to "preferably" be deployed in one specific node if that node is available otherwise be deployed in any other node?
The rub of it is, you need 3 nodes, all managers. It is not a good idea, even with a 2 node swarm, to make 2 nodes managers as docker swarm uses the raft protocol for manager quorum, and this protocol requires a clear majority. With two manager nodes, if either node goes down, the remaining manager node only represents 50% of the swarm managers and so will not represent the swarm until qorum is restored.
Once you have 3 nodes - all managers - the swarm will tolerate any single nodes failure and move tasks to the other two nodes.
Don't bother with 4 manager nodes - they dont provide extra protection from single node failures, and don't protect from two node failures as, again, only 2 out 4 does not represet more than 50%, to survive 2 node failures you want 5 managers.
I'm a newbie to Erlang. In Erlang, node is represented by an atom, like 'name#host'. I want to ask how a node can communicate with the other nodes without increasing the number of atoms of it?
I want to build a very distributed storage system which may contain thousands of nodes.
For a specified node A, it can send/receive messages to/from any other nodes in the cluster, for example:
rpc:call(Node, Module, Method, [])
But with the node joining and leaving the cluster, node A may have communicated to thousands of nodes,
in this way, the number of atom of node A will keep increasing and finally reach the limit.
How to avoid this from happending?
If I use the Pid instead of Node to communicate, for example,
Pid ! Message
Will this way increase the number of atoms in node A? It is said that Pid contains the information of a remote node.
The maximum number of atoms is 1048576 and you can raise it with +t. see: erlang docs
There's no chance you hit the limit with erlang clustering. If you were to scale to the 10s of thousands to millions range, you are very likely to have several separate clusters.
Distributed Erlang keeps the cluster alive with tcp heartbeats between nodes. You probably don't really want a single cluster more than a few thousand nodes.
Related to Neo4j 1.9.3 HA clustering, our team had a few different interpretations on how master node(s) behave if a HA cluster 'splits' due to a network problem. We're trying to understand the impact on master nodes as well as the branched database that occurs if/when the master cannot deliver updates to the slaves.
Given a 5 node deployment, where a network failure splits things into two groups/sub-clusters:
Group[A] consists of 2 nodes
Group[B] consists of 3 nodes
Each member of a given group can communicate with each other, but [A] cannot communicate with [B]. Prior to the split, the original master node (of the 5-node cluster) lived within the new [A] or [B] groups.
Questions:
If the original master node lived within [A] (i.e. in a minority non-quorum group of cluster nodes), will it write lock its database (knowing that it is a branch at this point) until it can rejoin the entire cluster at which time it will honor the newly elected master from [B] (which was able to elect a new master because it had quorum)?
If the original master node lived within [B] (i.e. in a majority quorum group of cluster nodes), will it continue to allow write’s to its database, while [A] will be writelocked because it will not have a master? Or is a master elected in [A] even if it doesn’t have a quorum for the whole cluster?
Any help is much appreciated!
There will always be only one master in a Neo4j cluster. Since the cluster is aware of the number of members a master election requires to have a quorum of more than the half. In case of a split in the way you've described the following will happen:
original master lived in minority partition A: master will degrade to slave and serve only read requests but won't accept writes. Partition B has a quorum and will elect a new master. When partitioning is resolved the former master will continue to work as slave.
original master lived in majority partition B: master continues to be master and propagates transactions to other members of B. When partitioning is resolved the former members of A will catch up missing transactions.
In url: http://www.datastax.com/docs/datastax_enterprise3.2/solutions/about_hadoop
"Before starting an analytics/Hadoop node on a production cluster or data center, it is important to disable the virtual node configuration."
What will happen if I enable virtual node in an analytics/Hadoop node?
If you enable virtual nodes on hadoop node, it will lower performance of small Hadoop jobs by raising the number of mappers to at least the number of virtual nodes. E.g. if you use the default 256 vnodes / physical nodes setting, every Hadoop job will launch 257 mappers. Those mappers might have too little data to process and the server would spend most of the time managing those tasks instead of doing useful work.
On a decent hardware, a job with no data and 256 vnodes may take about 5-10 minutes, contrary to the same job requiring only about 20-40 seconds when configured without vnodes.
Something I not get, I have two mnesia nodes. One has ram and other has disc copy.
My question is:
You do create schema once? But schema is where you enter nodes.
I confused and found not good documentation on this
Let's start by clarifying the concepts. A mnesia cluster consists of nodes and tables; nodes may have copies of the tables. The type of the copy, which may be ram_copies, disc_copies, and disc_only_copies, applies to a given table on a given node. A node may have different types of copies of different tables, and a table may have different types of copies on different nodes. A special case is a node which doesn't have disc based copies at all; it is called a ram only node.
The schema is a special table that stores information about the nodes and tables. Each node must have a copy of this table in the cluster; ram only nodes obviously have a ram copy, other nodes have a disc copy. To be precise, a node must have a disc copy of the schema to have a disc-based copy of any other table.
When you call mnesia:create_schema, you are creating a disc copy of a schema without tables, to be loaded by mnesia when it is started (this function refuses to work if mnesia is already started). If your cluster contains multiple disc-based nodes, the schema is created on all these nodes simultaneously, and when mnesia is started on these nodes, they automatically connect to each other (the nodes know about each other from the schema).
When mnesia cannot load the schema from disk at startup, it creates an empty one for itself in ram (or refuses to start, depending on settings). After that, you can either turn it into a ram-only node by calling mnesia:change_config on a disc-based node of the cluster, in which case the empty schema will be replaced and the node will be synchronized with the rest of the cluster, or you can start creating tables and adding other ram only nodes (which still have an empty schema), building a ram-only cluster.
A ram only node can be turned into a disc node by calling mnesia:chang_table_copy_type on the table schema. This way you can build a complete disc-based cluster dynamically from scratch, without creating a disc-based schema beforehand. However if you have a fixed set of disc nodes, it's much easier to statically initialize the schema on them before starting the cluster the first time.