Graylog newbie here. I was wondering how I could check the flow of messages between dependent services in logs in Graylog. I'll give a small instance to make my self clear.
Let's there 4 services A,B,C and D.
How do i find this or trace this flow in the Graylog
A->B->C->D->A->C(crash).
Any answer would be appreciated as I am struggling to find answers for this. Thanks!
In order to tie these messages together, you need some sort of a key value. Some element that is present in all four log sources that lets you know they are related.
It could be a hostname, or an IP. Anything common between the four will do.
If you don’t have a key value, you will have to write a query that includes all four sources.
Something like “source:A OR B OR C OR D”. Then use timestamps to establish the sequence you are looking for.
Related
I am trying to implement logging on connected users in my vernemq client using erlang. From documentation, I found that this could be bad, due to the scalability of the client and the assumption that there might be a lot of clients connecting and disconnecting. This is not my case, I will just have a bunch of clients but a lot of messages. Anyway, to my question. Is it possible to change the log file when using error_logger? Or should I use a different module for logging? Log file can be in any location if it had to, but I need it separated from vernemqs console.log. A followup question would be, can I somehow get a floating window on logs? I don't need to keep logs from previous year and I don't want to manually clean them every day or week or something like that
Thanks for any responses
From OTP21 on, you should use logger instead of error_logger, although the error_logger API is kept for compatibility (it justs uses logger under the hood).
With logger, which you can configure with the system configuration, you can use file backends such as logger_std_h (check the example configurations).
In logger_std_h you can set file rotation.
One of the new features of MQTT 5 is the shared subscriptions feature, which allows client-side load balancing between multiple workers, so that multiple workers can be responsible for handling messages, but every message is only ever sent to a single server.
By default, this works with a round-robin approach, but I am in the need of a slightly more advanced scenario:
What I want is some kind of routing, so that one of the messages' properties gets used as some kind of routing key. I.e., I want multiple workers to be responsible for the messages, but all messages with value X in their routing key property should always go to the same worker, and all messages with Y should do as well. The workers for X and Y may be different, but all messages with X should always go to the same one.
Question 1: Is this even possible with MQTT 5? If so, what is the term I need to look for? I tried googling for this, but wasn't really successful (mainly, I guess, because I don't know exactly what to look for).
Now, supposed this is possible: How can I then handle cases where nodes join or leave? Then I still want only a single node to be responsible, so it would be great if the assignment was not statically, but could be adjusted dynamically (or even better, would adjust itself automatically). However, what I strictly need to avoid is that two messages with X ever go to different servers at the same time.
Question 2: Supposed, this is not possible – what alternatives do I have to MQTT 5?
You don't at a protocol level. That is the whole point of a shared subscription to distribute the incoming messages evenly across all the subscribers.
This also goes against the pub/sub paradigm, that messages are published to a topic not an individual subscriber.
If you want to route messages differently publish them to different topics. There is nothing to stop you republishing a message on a separate topic based on it's meta data once it's been received by a client if needed.
My company is considering using flume for some fairly high volume log processing. We believe that the log processing needs to be distributed, both for volume (scalability) and failover (reliability) reasons, and Flume seems the obvious choice.
However, we think we must be missing something obvious, because we don't see how Flume provides automatic scalability and failover.
I want to define a flow that says for each log line, do thing A, then pass it along and do thing B, then pass it along and do thing C, and so on, which seems to match well with Flume. However, I want to be able to define this flow in purely logical terms, and then basically say, "Hey Flume, here are the servers, here is the flow definition, go to work!". Servers will die, (and ops will restart them), we will add servers to the cluster, and retire others, and flume will just direct the work to whatever nodes have available capacity.
This description is how Hadoop map-reduce implements scalability and failover, and I assumed that Flume would be the same. However, the documentation sees to imply that I need to manually configure which physical servers each logical node runs on, and configure specific failover scenarios for each node.
Am I right, and Flume does not serve our purpose, or did I miss something?
Thanks for your help.
Depending on whether you are using multiple masters, you can code your configuration to follow a failover pattern.
This is fairly detailed in the guide: http://archive.cloudera.com/cdh/3/flume/UserGuide/index.html#_automatic_failover_chains
To answer your question, bluntly, Flume does not yet have an ability to figure out a failover scheme automatically.
I don't know if it sounds crazy, but here's the scenario -
I need to print a document over the internet. My pc ClientX initiates the process using the web browser to access a ServerY on the internet and the printer is connected to a ClientZ (may be yours).
1. The document is stored on ServerY.
2. ClientZ is purely a cliet; no IIS, no print server etc.
3. I have the specific details of ClientZ, IP, Port, etc.
4. It'll be completely a server side application (and no client-side on ClientZ) with ASP.NET & C#
- so, is it possible? If yes, please give some clue. Thanks advanced.
This is kind of to big of a question for SO but basically what you need to do is
upload files to the server -- trivial
do some stuff to figure out if they are allowed to print the document -- trivial to hard depending on scope
add items to a queue for printing and associate them with a user/session -- easy
render and print the document -- trivial to hard depending on scope
notify the user that the document has been printed
handling errors
the big unknowns here are scope, if this is for a school project you probably don't have to worry about billing or queue priority in step two. If its for a commercial product billing can be a significant subsystem in its self.
the difficulty in step 4 depends directly on what formats you are going to support as many formats are going to require document specific libraries or applications. There are also security considerations here if this is a commercial product since it isn't safe to try to render all types of files.
Notifications can be easy or hard depending on how you want to do it. You can post back to the html page, but depending on how long its going to take for a job to complete it might be nice to have an email option as well.
You also need to think about errors. What is going to happen when paper or toner runs out or when someone tries to print something on A4 paper? Someone has to be notified so that jobs don't just build up.
On the server I would run just the user interaction piece on the web and have a "print daemon" running as a service to manage getting the documents printed and monitoring their status. I would use WCF to do IPC between the two.
Within the print daemon you are going to need a set of components to print different kinds of documents. I would make one assembly per type (or cluster of types) and load them into your service as plugins using MEF.
sorry this is so general, but you are asking a pretty general and difficult to answer question.
I'm using MailboxProcessor classes in order to keep separate agents that do their own thing. Normally agents can communicate with one another in the same process, but I want agents to talk to one another when they are on separate processes or even different machines. What kind of mechanism is best for implementing communication between them? Is there some standard solution?
Please note that I'm using Ubuntu instances to run the agents.
I think you're going to have write your own routines to serialize messages, pass them accross the process boundaries and then dispatch them on the other side. This will also require a implementation of a ID system where each mailbox has an ID and processes can send messages to IDs instead of just Mailbox.Send. This is not easy, as local boxes will be able to access local memory, but remote mailboxes will not.
I would look at something like RPyC (http://rpyc.wikidot.com/) as it provides a protocol somewhat like you are looking for.
Basically the answer is 'no' there isn't really a good way to do this.