I've been using Redis as a queue to communicate between distributed python scripts. At any moment, some nodes push and some nodes pop values from a list.
I've run into a problem, however. At a certain point a LPUSH will make the server run out of memmory. As i understand the virtual memmory feature that used to exist in Redis until version 2.4 is considered deprecated (and thus advised against).
The problem i have is that a strategy that discards any key is not acceptable. As such, the server is configured with noeviction (values are not evicted and an error should be returned).
What I would need is a way to find out that the command failed from redis-py so I can make a particular node wait until there is space to push items into the list. I've looked through the code and redis-py itself throws no exceptions (it doesn't use exceptions as a design choice).
LPUSH itself returns the number of records in that particular list, however, since the list is accessed from different nodes, the value itself will tell me nothing.
Any ideas how I can achieve this?
Please tell me if any additional information on the nature of the problem would help clarify it.
Related
I have a Dockerized Django application which have a number of CRON-jobs that need to be executed.
Right now I'm running it with the package Supercronic (which is recommended for running cron-jobs inside containers). This will be deployed on a two servers for redunancy-purposes, i.e. If one goes down the other one need to take over and execute the cron-jobs.
However, the issue is that without any configuration this will result in duplicate cron-jobs being executed, one for each server. I've read that you can set up something called a "lease" for the cron-jobs to retrieve, to avoid duplicates from different servers, but I haven't found any instructions on how to set this up.
Can someone maybe point me in the right direction here?
If you are running Supercron in two different instance, Supercron doesn't know about whether the job gets triggered, Its up to the application to handle the consistency.
You can do it in many ways either controlling the state with File or DB entries or any better way where your docker application can check the status before it start executing the actual process.
Due to some cluster rules, it can happen that a process is killed for a given reason (e.g. being allocated in a node that has a high priority to another group of users)
Is there a way to make a task or dataset that was in that worker acquire the 'lost' tag such that the Dask/Distributed system re-runs that particular task as well as it's dependent graph?
Currently there is no way to retry a failed task. This is a reasonable request though, so I've opened a Github issue here. Future readers of this question may want to check on the status of that issue to see if things have changed.
Meanwhile, you can change a parameter that will avoid KilledWorker issues. The KilledWorker exception only occurs if a task was actively running on three workers while they unexpectedly died. This is meant to stop bad tasks from catastrophically taking down the entire cluster as they are moved from failed worker to failed worker. If your cluster is genuinely a bit unstable then you may want to increase this parameter three to a larger number like twenty. You should change the ~/.dask/config file on the machine that hosts your scheduler to include the following value :
allowed-failures: 20
Currently in dask/distributed master you can also assign this with the environment variable DASK_ALLOWED_FAILURES=20. This should be available in the release following 1.15.2.
It seems that I found the root of the problem:
If I'm not wrong dask/distributed uses the __sizeof__ attribute to plan the deployment of objects in a worker.
If this is true, as the specification of __sizeof__ states
"Only the memory consumption directly attributed to the object is accounted for, not the memory consumption of objects it refers to." (https://docs.python.org/3/library/sys.html#sys.getsizeof) distributed might be underestimating the size of the objects such as sparse matrices or third-party classes that might contain arrays or other large objects sent to a worker and getting my cluster scheduler (e.g. Torque) to kill the worker.
maybe the correct way of solving it would be to recommend adding a dask-specific method for computing the size instead of relying on sys.getsizeof?
I was wondering if it could be possible to recognize how a command is comformed with any services in Centreon? For example, which services contain the 'check_uptime' command?
Maybe its possible using some sql magic queries but I'ver done that.
Though your question reminded me of this week's icinga2 api development where dependency tracking for objects has been implemented - this is important in case someone deletes a checkcommand at runtime, with many hosts/services depending on it. By default the api will deny removal, but cascading deletes would cause the entire dependency tree being deleted.
A side-effect from that development is the output of such object dependencies inside the status query for these objects.
Check the screenshot over here: https://twitter.com/dnsmichi/status/637586226711764992
It may not help you now, but with 2.4 being released in November.
Since using the release of the latest build of Cloud Dataflow (0.4.150414) our jobs are running really slow when reading from cloud storage (GCS). After running for 20 minutes with 10 VMs we were only able to read in about 20 records when previously we could read in millions without issue.
It seems to be hanging, although no errors are being reported back to the console.
We received an email informing us that the latest build would be slower and that it could be countered by using more VMs but we got similar results with 50 VMs.
Here is the job id for reference: 2015-04-22_22_20_21-5463648738106751600
Instance: n1-standard-2
Region: us-central1-a
Your job seems to be using side inputs to a DoFn. Since there has been a recent change in how Cloud Dataflow SDK for Java handles side inputs, it is likely that your performance issue is related to that. I'm reposting my answer from a related question.
The evidence seems to indicate that there is an issue with how your pipeline handles side inputs. Specifically, it's quite likely that side inputs may be getting re-read from BigQuery again and again, for every element of the main input. This is completely orthogonal to the changes to the type of virtual machines used by Dataflow workers, described below.
This is closely related to the changes made in the Dataflow SDK for Java, version 0.3.150326. In that release, we changed the side input API to apply per window. Calls to sideInput() now return values only in the specific window corresponding to the window of the main input element, and not the whole side input PCollectionView. Consequently, sideInput() can no longer be called from startBundle and finishBundle of a DoFn because the window is not yet known.
For example, the following code snippet has an issue that would cause re-reading side input for every input element.
#Override
public void processElement(ProcessContext c) throws Exception {
Iterable<String> uniqueIds = c.sideInput(iterableView);
for (String item : uniqueIds) {
[...]
}
c.output([...]);
}
This code can be improved by caching the side input to a List member variable of the transform (assuming it fits into memory) during the first call to processElement, and use that cached List instead of the side input in subsequent calls.
This workaround should restore the performance you were seeing before, when side inputs could have been called from startBundle. Long-term, we will work on better caching for side inputs. (If this doesn't help fully resolve the issue, please reach out to us via email and share the relevant code snippets.)
Separately, there was, indeed, an update to the Cloud Dataflow Service around 4/9/15 that changed the default type of virtual machines used by Dataflow workers. Specifically, we reduced the default number of cores per worker because our benchmarks showed it as cost effective for typical jobs. This is not a slowdown in the Dataflow Service of any kind -- it just runs with less resources per worker, by default. Users are still given the options to override both the number of workers as well as the type of the virtual machine used by workers.
We had a similar issue. It is when the side-input is reading from a BigQuery table that has had its data streamed in, rather than bulk loaded. When we copy the table(s), and read from the copies instead everything works fine.
If your tables are streamed, try copying them and reading the copies instead. This is a workaround.
See: Dataflow performance issues
I was looking at the slave/pool modules and it seems similar to what I
want, but it also seems like I have a single point of failure in my
application (if the master node goes down).
The client has a list of gateways (for the sake of fallback - all do
the same thing) which accept connections, and one is chosen from
randomly by the client. When the client connects all nodes are
examined to see which has the least load and then the IP of the least-
loaded server is forwarded back to the client. The client then
connects to this server and everything is executed there.
In summary, I want all nodes to act as both gateways and to actually
process client requests. The load balancing is only done when the
client initially connects - all of the actual packets and processed on
the client's "home" node.
How would I do this?
I don't know if there is this modules implemented yet but what I can say, load balance is overrated. What I can argue is, random placing of jobs is best bet unless you know far more information how load will come in future and in most of cases you really doesn't. What you wrote:
When the client connects all nodes are examined to see which has the least load and then the IP of the least- loaded server is forwarded back to the client.
How you know that all those least loaded node will not be highest loaded just in next ms? How you know that all those high loaded nodes which you will not include in list will not drop load just in next ms? You really can't know it unless you have very rare case.
Just measure (or compute) your node's performance and set node's probability be chosen depend of it. Choose node randomly regardless of current load. Use this as initial approach. When you set it up, then you can try make up some more sophisticated algorithm. I bet that it will be very hard work to beat this initial approach. Trust me, very hard.
Edit: To be more clear in one subtle detail, I strongly argue that you can't predict future load from current and historical load but you should use knowledge about tasks durations probability and current decomposition of task's lifetime. This work is so hard to try achieve.
The purpose of a supervision tree is to manage the processes not necessarily forward requests. There is no reason you couldn't use different code to send requests directly to members of the list of available processes. See the pool:get_nodes or pool:get_node() functions for one way to get those lists.
You can let the pool module handle the management of the processes (restarting, monitoring, and killing processing) and use some other module to transparently redirect requests to the pool of processes. Maybe you were looking for distributed pools though? It'll be hard to get away from the master process in erlang whithout going to distributed nodes. The whole running system is pretty much one large supervision tree.
I recently remembered the pg module which allows you to setup process groups. messages sent to the group go to every process in the group. It might get you part way toward what you want. you would have to write the code to decide which process handles the request for real but you would get a pool without a master using it.