In CouchDB there's huge BTree data structure and multiple processes (one for each request).
Erlang processes can't share state - so it seems that there should be dedicated process responsible for accessing BTree and communicating with other processes via messages. But it would be inefficient - because there only one process who can access data.
So how such cases are handled in Erland, and how it's handled in this specific case with CouchDB?
Good question this. If you want an authoritative answer the
best place to ask a question about couchdb internals is the couchdb mailing list they are very quick and one of the core devs can probably give you a better answer. I will try to answer this as best as I can just keep in mind that I may be wrong :)
The first clue is provided by the couchdb config file. Start couchdb in the shell mode
couchdb -i
point your browser to
http://localhost:5984/_utils/config.html
You will find that under the daemon section there is a key value pair
index_server {couch_index_server, start_link, []}
aha! so the index is served by a server. What kind of server? We will have to dive into the code:-
It is a gen_server. All the operations to the couchdb view are handled by this gen_server.
A gen_server is an erlang generic implementation of the client server model. It is concurrent by default. So your observation is correct. All the requests to the view are distinct process managed with the help of gen_server.
index_server defines three ets tables. You can verify this by typing
ets:i() in the erlang shell we started earlier and you should see:-
couchdb_indexes_by_db couchdb_indexes_by_db bag 1 320 couch_index_server
couchdb_indexes_by_pid couchdb_indexes_by_pid set 1 316 couch_index_server
couchdb_indexes_by_sig couchdb_indexes_by_sig set 1 316 couch_index_server
When the index_server gets a call to get_index it adds a list of Waiters to the ets couchdb_indexes_by_sig. Or if a process requests it it simply sends a reply with the location of the index.
When the index_server gets a call to async_open it simply iterates over the list of Waiters and sends a reply to them with the location of the index
Similarly there are calls to reset_indexes and other ops on indexes which again send a reply with the location of the index.
When the index is created for the first time couchdb calls async_open to serve the index to all the waiting processes. Afterwards every process is given access to the index.
An important point to note here is that the index server does not do anything special except for making the index available to other processes(for example to couch_mr_view_util.erl). In that respect it acts as a gateway.Index write operations are handled by couch_index.erl, couch-index_updater.erl and couch_index_compactor.erl which (unsurprisingly) are all gen_servers!
When a view is being created for the first time only one process can access it. The query_server process(couchjs by default). After the view has been built it can be read and updated in a concurrent manner. The actual querying of views is handled by couch_mr_view which is exposed to us as the familliar http api.
Related
I am using ReactiveMongoTemplate MongoDB ChangeStream to listen to updates to a MongoDB collection, perform some queries, and persist a document to another collection.
While it had been working fine locally, It started giving the below error after deployment to UAT which had lots of volume:
Too many operations are already waiting for a collection. Max number of operations (maxWaitQueueSize) of 500 has been exceeded.
What are the ways to resolve this?
I have the following in application.yml file
spring:
data:
mongodb:
uri: mongodb://host:port/db?authMechanism=<val1>&authSource=<val2>&authechanismProperties=<val3>
And this is how the simplified change streamer looks like:
#Autowired
ReactiveMongoTemplate reactiveMongoTemplate;
reactiveMongoTemplate
.changeStream(Sample.class)
.watchCollection("sample_collection")
.filter(
new Criteria.orOperator(
where("operationType").is("update"),
where("operationType").is("insert")
)
)
.listen()
.flatMap(r->processMessage(r)). // processMessage does some queries to collections including this collection being listened to and upserts to same mongodb in a different collection
.subscribeOn(Schedulers.boundedElastic())
.subscribe();
I understand I might need to add some connection pooling to be able to handle multiple connections? But how do I configure that with Reactive MongoDB?
I am new to reactive programming. Any pointers would be really helpful.
There are couple of things you can do here:
Check if there are some long blocking calls being made, causing the threads to be blocked, and leading to lots of connections being created, as the previous ones are still occupied performing the heavy task. Try checking for some optimisations to the code which is blocking those calls.
In Reactive Programming, you can check the presence of blocking code using BlockHound.
Increase the connections limit by specifying the waitQueueMultiple or maxPoolSize -
https://docs.mongodb.com/manual/reference/connection-string/#connection-pool-options
Before that you can check your mongo db stats to see the current and allowed connections using
db.serverStatus().connections
sir,
I am trying to create stateful proxy in opensips 2.4.
I just wanted a variable to hold received message information and process it.
So i checked "core variable" in opensips manual.it says, script variable are process wise. So i should not use to hold header value in script value like $var(Ruri)=$ru?? it will be overwritten by other call??
$var(userName)=$rU;
$var(removePlus) = '+';
# Search the string starting at 0 index
if($(var(userName){s.index, $var(removePlus)})==0){
$rU=$(var(userName){s.substr,1,0});
}
$var variables are process-local, meaning that you can't share them with other SIP workers even if you wanted to! In fact, they are so optimized that their starting value will often be what the same process left-behind during a previous SIP message processing (tip: you can prove this by running opensips with children = 1 and making two calls).
On the other hand, variables such as $avp are shared between processes, but not in a "dangerous" way that you have to worry about two INVITE retransmissions processing in parallel, each overwriting the other one's $avp, etc. No! That is taken care of under the hood. The "sharing" only means that, for example, during a 200 OK reply processed by a different process than the one which relayed the initial INVITE, you will still be able to read and write to the same $avp which you set during request processing.
Finally, your code seems correct, but it can be greatly simplified:
if ($rU =~ "^+")
strip(1);
We are looking to store Windows process tree information in a graph database.
We have an endpoint detection tool that collects process creation and deletion information. For example, the process id, the command that started the process, the parent process and so on.
The data will be loaded into the graph database in real-time.
Is there a way in Neo4j to trigger an event anytime we see "iexplore.exe" as the parent of processes such as "powershell.exe" or "cmd.exe"? In some cases, there may be several other processes between iexplore and powershell/cmd.
For example: the attacker exploits a vulnerability in IE, drops of a file on disk, executes it, and then uses powershell.exe.
Does graph databases provide us the ability to build in rules to capture more complex events like that?
You can use apoc triggers for set trigger and apoc.es.postRaw for external API call. For example:
CALL apoc.trigger.add(
'iexplorer',
'MATCH (P:Process {name:"iexplore.exe"})-[:parent_of*]->(C:Process)
WHERE C.name IN ["powershell.exe", "cmd.exe"]
WITH collect(distinct id(P)) as ids
CALL apoc.es.postRaw("http://localhost", "neo4jevent", ids) yield value
RETURN value',
'{phase:'after'}'
)
I'm not entirely sure the differences between the PID and Reference and when to use which.
If I were to spawn a new process with spawn/1 pid. I can kill it with the PID no? Why would I need a reference?
Likewise I see monitor/1 receiving a message with a ref and pid number.
Thanks!
Pid is process identifier. You can get one when you create new process with spawn, or you can get Pid of yourself with self(). It allows you to interact with given process. Especially send messages to it by Pid ! Message. And some other stuff, like killing it explicitly (should not do) or obtaining some process information with erlang:process_info.
And you can create relations between process with erlang:link(Pid) and erlang:monitor(process, Pid) (that's between Pid process, and process execution this function). In short, it gives you "notifications" when another process dies.
Reference is just almost unique value (of different type). One might say, that it gives you some reference to here and now, which you could recognize later. For example, if we are sending a message to another process, and we expect a response, we would like to make sure, that the message we will receive is associated to our request, and not just any message from someone else. The easiest way to do it is to tag the message with a unique value, and wait until a response with exactly the same tag.
Tag = make_ref(),
Pid ! {Tag, Message},
receive
{Tag, Response} ->
....
In this code, with use of pattern matching, we make sure that (we wait in receive until) Response is exactly for the Message we sent. No matter other messages from other processes. This is the most common use of reference you can encounter.
And now back to monitor. When calling Ref = monitor(process, Pid) we make this special connection with Pid process. Ref that is returned is just some unique reference, that we could use to demonitor this process. That is all.
One might ask, if we are able to create monitor with Pid, why do we need Ref for demonitoring? Couldn't we just use Pid again. In theory we could, but monitors are implemented in such a way, that multiple monitors could be established between two same processes. So when demonitoring, we have to remove only one of such connections. It is done in this way to make monitoring more transparent. If you have library of function that's creating and removing one monitor, you would not like to interfere with other libraries and functions and monitors they might be using.
According this page:
References are erlang objects with exactly two properties:
They can be created by a program (using make_ref/0), and,
They can be compared for equality.
You should use it ever you need to bind an unique identifier to some "object". Any time you could generate new one using erlang:make_ref/0. Documentation says:
make_ref() -> reference()
Returns an almost unique reference.
The returned reference will re-occur after approximately 2^82 calls;
therefore it is unique enough for practical purposes.
When you call erlang:monitor/2 function, it returns you reference to give you availability to cancel monitor (erlang:demonitor/1 function). This reference only identifies certain call of erlang:monitor/1. If you need operate with process (kill it, for example), you still have to use process pid.
Likewise I see monitor/1 receiving a message with a ref and pid number.
Yep, monitor sends messages like {'DOWN', Ref, process, Pid, Reason}. What to use (pid or ref) is only depends on your application logic, but (IMO) in most usual cases, there is no matter what to use.
First of all excuse me if I got some concept wrong, this a bit new to me. I have to retrieve a number of objects from a webdis server. The way it is being done at the moment is:
Get all the objects ids (serverUrl/ZRANGE/objects_index/-X/-1)
For each object, get attributes (serverUrl/GET/attributeY_objectIdX)
So if I have X objects with Y attributes I have to perform X * Y + 1 REST calls to get all he data, that seems highly inefficient.
From what I understand Multi is the command to perform a join but is not supported by webdis rest api (see Ideas, TODO on webdis page).
Is there a simpler solution that I am missing?
Should I reorganise the way the data is stored?
Can I use websockets to send a MULTI/EXEC command through json:
jsonSocket.send(JSON.stringify(["MULTI", "EXEC", "GET", "etc..."]));
First, instead of having one key per attribute, you should consider use hash objects, so you get one key per object, associated to several properties. The benefit is you can use the HGETALL command to retrieve all the properties of a given object at once. Instead of having X*Y+1 calls, you have only X+1.
Instead of:
SET user:1:name Didier
SET user:1:age 41
SET user:1:country FR
you could have:
HMSET user:1 name Didier age 41 country FR
Then, webdis supports HTTP 1.1 and websocket pipelining, and Redis server supports pipelining using its own protocol. So it should be possible to send several commands to webdis, wait for the results (which will be returned in the same order) while only paying for a single roundtrip.
For instance, the websocket example provided on webdis page actually performs a single roundtrip to execute two commands:
var jsonSocket = new WebSocket("ws://127.0.0.1:7379/.json");
jsonSocket.onopen = function() {
console.log("JSON socket connected!");
jsonSocket.send(JSON.stringify(["SET", "hello", "world"]));
jsonSocket.send(JSON.stringify(["GET", "hello"]));
};
jsonSocket.onmessage = function(messageEvent) {
console.log("JSON received:", messageEvent.data);
};
You could do something similar, and aggregate several HGETALL commands to retrieve the data by batch of n objects.
Please note that with Redis itself (i.e. without webdis), I would probably recommend the same strategy (pipelining HGETALL commands).