I have a similar problem to the one posed in this question: https://stackoverflow.com/a/73197921/2846161 but simpler I think. I'm having a hard time trying to find a way to just sync my containers in a FIFO manner (like a leader board). You connected first you get the first seat/place. That way, the first replica process the first chunk of data, the second replica the second chunk and so on. I was trying to work with redis structs (I'm using redsync already) but am quite lost frankly.
My idea was to do something like
mutex.lock()
getFirstFreeBucket() // buckets would be a struct where keys are the positions: "1", "2", etc
mutex.unlock()
but I don't know how to handle sudden disconnects with redis (panics, etc.)
Could anyone help me out here, please? thanks
Related
Background:
I have a language processing java app that requires about 16MB memory and takes about 40 seconds to initialise resources into that memory before exposing a webservice. I am new to containers and related technologies so apologies if my question is obvious...
Objective:
I want to make available several hundred instances of my app on-demand and in a pre-loaded/ pre-configured state. (eg I could make a call to AWS to stand-up 'n' instances of my app and they would be ready in <10seconds.)
Question:
I'm anticipating that I 'may' be able to create a docker image of the app, initialise it and pause hence be able to clone that on demand and 'un-pause' ? Could you advise whether what I am looking to do is possible and if so, how you would approach it.
AWS is my platform of choice so any AWS flavoured specifics would be super helpful.
I'd split your question in two, if you don't mind:
1. Spinning up N containers (or, more likely, scale on demand)
2. Preloading memory.
#1 is Kubernetes's bread and butter and you can find a ton of resources about it online, so allow me to focus on #2.
The real problem is that you're too focused on a possible solution to see the bigger picture:
You want to "preload memory" in order to speed up launch time (well, what do you think Java is doing in those 40s that the magick memory preloader wouldn't?).
A different approach would be to launch the container, let Java eat up resources for 40s, but not make that container available to the world during that time.
Kubernetes provides tools to achieve exactly that, see here:
https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/
Hope this helps!
I'm processing many Sidekiq jobs (tens of millions) which return 1 string and 2 integers as a result. There is a central computer which is hosting the Redis server/storing the results, and multiple computers acting as workers, fetching data from this central computer.
Inside the worker, I do the logic needed for processing and after the result is called, I just call:
Sidekiq::Client.push('class' => ResultsWorker, 'args' => [arg1, arg2, arg3])
and on the central computer, I just have another worker which just checks this queue and stores the results in a database.
I haven't seen such a pattern of "two-way communication" being mentioned anywhere in this Sidekiq community. So I was wondering, is there a reason for this? Is there some side-effect of doing this? I've tried putting a simple web app on the redis server comp which accepts the results and stores them, but I figured that for such small data, why not re-use the same Redis server with a different queue?
I haven't seen such a pattern of "two-way communication" being mentioned anywhere in this Sidekiq community.
This doesn't need any special mentioning or discussion, because there's nothing special or unusual about this. A job can indeed enqueue some other jobs. This happens left and right in big apps.
Although you should measure the overhead of this concrete situation in your case. It might turn out that persisting to the database directly from the first job will be much faster (no losses on making network requests, etc.)
I am getting through the online examples, and can already use mnesia ram copies and also connect them, but I am a bit confused on a couple of things.
1: Does the starter node (the one who creates the schema), only have the local schema? (for example, in root folder = Mnesia.name#ip)
I ask because on another node, I can simply start mnesia, and change_config(extra_db_nodes, [node]), and automatically get all the data that is on the starting node.
This seems weird to me, what happens if all nodes go down? This means starter node needs to be ran first before you can do anything.
2: There seems to be a lot of different ways to connect nodes, and to copy the tables ... Could I get a list of different ways to do this, and their impacts?
3: From the first question, after calling change_config, how can you know that its finished downloading all the data before you can start to use it? For example, if someone connects to the node, and you check if they are already online, they might be connected to another node and you dont get that data during the check.
4: After connecting to a node, are you automatically connected to all nodes? And does it automatically update your local ram copies without doing anything? How does it assure synchronization when reading, and writing? Do I have to do anything special?
And about question 1 again -- couldn't you have a node process running that holds the local schema, and use this node to connect all nodes together? And if possible could you forbid mnesia from copying ram copies to this node process?
I know this is a lot, so thank you for your time.
Not a direct answer to your questions, but you can check out Erlang Performance Lab which might help you understand how some operations in Mnesia works by visualizing the messages between different nodes.
I have an erlang application currently running on four nodes with a replicated mnesia db that stores minimal data regarding connected clients. The mnesia replication has been working seamlessly in the past (as far as I know anyway) but a client recently noticed that one of the nodes is missing some ids related to his application.
I'm not really sure how this happened. Our network may have had a hiccup at the time. Maybe? But, of more urgency at the moment is getting the data into a good state across all nodes. Is there a way to tell mnesia to replicate from a known-good node?
Mnesia is legendary about this issue. It's a huge PITA.
Looking at it from CAP theorem's point of view, most systems built with Mnesia end up being C-A (consistency-availability with no partition tolerance) systems. For most of the time you have (and heavily rely on) its hard consistency. Then a network partition happens...
It's still available for writes, but these writes destroy consistency. And later on, Mnesia has no mechanism for automatic data repair.
Everyone who uses Mnesia in a cluster should familiarize themselves with these tradeoffs. Your problem is a clear sign that using Mnesia was a poor choice. Double so if this data is critical to you.
I too use Mnesia in such a way (sometimes we all need speed you know). But I make sure to only use it to store data that I can easily reconstruct. In general, if you need it stored on disk, Mnesia is no good, except for toy projects.
I make sure to always have this function at hand:
reinit_mnesia_cluster() ->
rpc:multicall(mnesia, stop, []),
AllNodes = [node() | nodes()],
mnesia:delete_schema(AllNodes),
mnesia:create_schema(AllNodes),
rpc:multicall(mnesia, start, []).
Use it only after the network partition has been resolved and all nodes are reachable. This will erase all Mnesia replicas and start it anew. Again, if you can't live with what it does, then using Mnesia was a poor choice.
For important data that needs hard consistency, use SQL. For important data that needs availability, use Riak. For shared state that needs speed, use Redis. Mnesia is no replacement for these systems, although at first it does seem so.
Edit on 2014-11-16: Here is a much better article on the topic, explaining in detail what I said above https://medium.com/#jlouis666/mnesia-and-cap-d2673a92850
Honestly, I think the cleanest way to get an out-of-sync Mnesia to replicate from a known good node is to shut down the application on the bad node, and delete all its Mnesia database files, then do the following.
Write an escript that starts Mnesia up standalone using the "bad" node name and Mnesia directory, replicates the tables from a known good node, and shuts Mnesia down. Run that escript on the bad node.
The act of replicating the tables and shutting Mnesia down gracefully puts the node back in sync with the cluster. Then, when you start the application up on the bad node, it will join up and stay in sync with the cluster.
Of course, this description lacks precise details, but that's the gist of it. There are surely less brute force ways of doing this, but unless you have massive amounts of data to replicate, I think this way is the quickest and cleanest.
I wish to know is there any way that I can create the threads on other nodes without starting the process on the nodes.
For example :- lets say I have cluster of 5 nodes I am running an application on node1. Which creates 5 threads on I want the threads not to be created in the same system but across the cluster lets say 1 node 1 thread type.
Is there any way this can be done or is it more depends on the Load Scheduler and does openMP do something like that?
if there is any ambiguity in the question plz let me know I will clarify it.
Short answer - not simply. Threads share a processes' address space, and so therefore it is extremely difficult to relocate them across cluster nodes. And, if it is possible (systems do exist which support this) then getting them to maintain a consistent state introduces a lot of synchronization and communication overhead which impacts on performance.
In short, if you're distributing an application across a cluster, stick with multiple processes and choose a suitable communication mechanism.
generally, leave threads to vm or engine to avoid very inert locks, focus app or transport, if one, create time (200 hz=5ms heuristic), if 2, repaint, good pattern: eventdrive