What is canary monitoring?
What does it mean?
Why canary?
Found a tool for canary monitoring here:
http://svn.apache.org/viewvc/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/Canary.java?view=markup
The symbolism is to the "canary in the coal mine"; miners in the old days, before electronic vapor detectors, would take a caged canary down into coal mine with them as an "early warning system" of dangerous gases. Contrary to popular rumor, the miners usually didn't wait for the canary to die; they were instead listening for the canary to stop chirping.
In this case, the program just sits there happily doing very little (basically polling HTableDescriptors which contain status data about Apache), until there's some failure, at which time it logs the nature of the failure (which can then be sent to a sysadmin via e-mail using some other system like event log sniffing).
A more "canary-like" system of monitoring is "heartbeat monitoring". This basically requires the system being monitored to send an "I'm still here and doing fine" message at set intervals. If the system misses a "heartbeat", the monitor knows there's something wrong. This is a little more akin to the canary chirping, as the monitor is just passively "listening" for these status messages, and the status messages are always a good thing (instead of the system used in your example which is actively watching the system being monitored and sends both good and bad messages).
An allusion to caged canaries mining workers would carry down into the
tunnels with them. If dangerous gases such as methane or carbon
monoxide leaked into the mine-shaft, the gases would kill the canary
before killing the miners.
http://en.wiktionary.org/wiki/canary_in_a_coal_mine
the miners usually didn't wait for the canary to die; they were instead listening for the canary to stop chirping.
Actually this is exactly what synthetic (or active) monitoring is all about [1]. There are lots of SaaS tools that does the job for you (https://www.site24x7.com/, https://www.pingdom.com), many for free. Some give API interfaces you can use to use to make customised version of your monitor (https://cula.io/).
[1] https://en.wikipedia.org/wiki/Synthetic_monitoring
Related
I have come across a lot of tools to monitor Linux servers which could generate alerts as well when the CPU usage goes alarmingly high, or the disk space goes very low etc.
However, in terms of Ejabberd I couldn’t find an exisiting module which could do something similar. I am particularly looking to receive alerts pertaining to mnesia getting overloaded, space availability etc. and other basic parameters worth monitoring.
Have a look at Exometer. It can report via SNMP. It doesn't come with the monitoring you're talking about out of the box but you should be able to configure it to report on whatever you need.
SNMP support comes standard with Erlang. You should have a look at Erlang/OTP os_mon. Depending on your needs, it may do what you want out of the box.
I need to check the disk space in a number of remote systems (Windows 2000) and have to sent a mail if the space is below a particular level. How can this be automated? What script can be used?
One option would be a system monitoring tool, such as Nagios.
Basically, what this will do is continually interrogate your systems, and alert you to anything you set it to (disk-space, CPU usage, page-faults, etc.). As a bonus you get a lovely web-based interface which lets you see your system status at a glance.
Note that you will need to enable remote WMI querying on each target system, or install an "agent" on each system.
Is there an application that is capable of monitoring AND logging information (to file) about another process (in particular IIS aspnet_wp.exe) like (in periods of time):
- memory usage of process
- cpu usage
Or maybe there is another way to monitor IIS process?
Thanks Pawel
You can check Process Monitor from Microsoft.
Process Monitor is an advanced
monitoring tool for Windows that shows
real-time file system, Registry and
process/thread activity. It combines
the features of two legacy
Sysinternals utilities, Filemon and
Regmon, and adds an extensive list of
enhancements including rich and
non-destructive filtering,
comprehensive event properties such
session IDs and user names, reliable
process information, full thread
stacks with integrated symbol support
for each operation, simultaneous
logging to a file, and much more.
One choice would be Process explorer http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx
You might want to look at the ANTS Performance and Memory Profilers from Red Gate. I'm using their memory profiler to track down some memory issues as I write this.
Pretty much any monitoring system or framework that allows custom checks is capable of this.
You write your check and just put an extra line in it to post/print/put something to a file of your choice.
I for example use sensu for which I wright custom checks in ruby.
So I can easily do something like
if %x[{ping]}
ok
%x[{ print ‘all is well’ > syslog]}
end
and similar to that you can do for most other monitoring systems like Nagios etc.
I'm using the UPS service to monitor the state of my UPS from an application -- the key at HKLM\SYSTEM\CCS\Services\UPS\Status has all the information you can get from the Power control panel. BUT -- I'd like to be able to tell the UPS to shut down from my app as well. I know that the service can tell the UPS to shut down -- for instance, after running a set number of minutes on battery -- and I'm wondering if there's some kind of command I can send to the service to initiate a shutdown manually.
I'm having trouble searching for this information -- people tend to misspell "Uninterruptible" (hrm, Firefox red-lined that but doesn't have an alternative) and "UPS" just gets hits for the shipping service. Maybe I can do something through System.ServiceController, or WMI?
CLARIFICATION: Yes, I am talking about powering down the physical UPS device. I know how to stop the service. I figured it would be a common problem -- I want my UPS to turn off with the PC. I had an idea I'm going to try, based on this page. You see, APC (and everybody else) has to supply a DLL for the UPS service to call, and since the function calls are well documented, there's no reason I shouldn't be able to P/Invoke them. I'll re-edit this once I know whether or not it worked.
Update: I tried invoking UPSInit, then UPSTurnOff, and nothing happens. I'll tinker with it some more, but the direct call to apcups.dll might be a dead end.
Check my comments to Herman, you want to shut the UPS down, not the UPS SERVICE, correct? I mean, you want that thing to shut off, kill the power, etc, right?
If so, you are looking it on a UPS by UPS model. I doubt two of them would work the same.
In your searches, instead of UPS, try "APC", or "battery". I think a lot of the code is what runs on laptops to deal with being on battery, etc...
Some place hidden in some dusty old files I have protocol information for APC UPS's, and the commands they respond to, and what they send to the PC etc. But this was WAY back in the day when we used to connect our UPS's to our computers with SERIAL cables... You could actually talk to a UPS with Qmodem or Hyperterm...
Learned it from talking to the guys at APC. They are very nice, and helpful. Now-a-days, I think you just post a URL coming from your Powerchute software, and it will talk directly to the UPS, and carry out your commands.
OK, I have the answer (tested!), but it's not pretty. My APC UPS communicates using the APC "Smart" protocol (more here). What you need in my case is a "soft shutdown", "S" command. But first you need to make sure it's in "Smart" mode ("Y"). Now, if you want to let the Windows UPS service monitor state, the service will have an iron grip on the COM port. So you can either a) let the Windows service turn the UPS off, or b) kill the service and turn the UPS off yourself.
The UPS itself has a "grace period" after it gets the "S" command, giving you time to shut down your OS. This means that to do (a) above, you have to:
Kill utility (mains) power
Wait for the Windows UPS Service timeout (default and minimum 2 minutes)
Wait for Windows to shut down -- right near the end, it will send the "S" command
Wait for the UPS grace period, after which it will actually turn itself off
I think we're going to opt for (a), just because (b) involves extra work killing the service and implementing the serial comms.
Please, tell in what language are you trying to do that... if you're using .NET you can do that with ServiceController class (read the docs).
For controlling services in Win32 API using C/C++, Service Functions (Windows).
For example to stop a service you can use ControlService function as follows (this is a quick and dirty example):
OpenService (hServMgr, TEXT("\\UPS_SERVICE_0"), SC_MANAGER_ALL_ACCESS);
SERVICE_STATUS stat;
ControlService (hUpsService, SERVICE_CONTROL_STOP, &stat)
Note that you need to provide a Service Manager handle in hServMgr and the \\UPS_SERVICE_0 name is the name that must match with your desired UPS service (either the Windows built-in or another).
Remember that to stop a service you need the proper security rights. This is not a problem with an Adminstration account, but keep in mind what happens when logging with a non-admin account.
Hope that helps.
About shutting down the physical UPS device, I remember back in WIn98 days I was able to poweroff the device talking with the UPS through the COM port, altough I don't remember the brand or how the programming interface was.
What, at a minimum, should an application health-monitoring system do for you (the developer) and/or your boss (the IT Manager) and/or the operations (on-call) staff?
What else should it do above the minimum requirements?
Is monitoring the 'infrastructure' applications (ms-exchange, apache, etc.) sufficient or do individual user applications, web sites, and databases also need to be monitored?
if the latter, what do you need to know about them?
ADDENDUM: thanks for the input, i was really looking for application-level monitoring not infrastructure monitoring, but it is good to know about both
Whether the application is running.
Unusual cpu/memory/network usage.
Report any unhandled exceptions.
Status of various modules (if applicable).
Status of external components (databases, webservices, fileservers, etc.)
Number of pending background tasks (if applicable).
Maybe track usage of the application and report statistics on most/less used functionalities so you know where optimizations are most beneficial.
The answer is 'it depends'. Why do you need to monitor? How large is your operations staff? Do you need reporting? What is the application environment? Who cares if the application fails? Who cares if an exception happens? Are any of the errors recoverable? I could ask questions like these for a long time.
Great question.
We've been looking for some application-level monitoring solution for our needs some time ago without any luck. Popular monitoring solution are mostly addressed to monitor infrastrcture and - in my opinion - they are too complicated for a requirements of most of small and mid-sized companies.
We required (mainly) following features:
alerts - we wanted to know about
incident as fast as possible
painless management - hosted service wouldbe
the best
visualizations - it's good to know what is going on and take some knowledge from the data
Because we didn't find suitable solution we started to write our own. Finally we've ended with up-and-running service called AlertGrid. (You can check it for free of course.)
The idea behind it is to provide an easy way to handle custom monitoring scenarios. Integration API is very simple (one function with two required parameters). At the momment we and others are using it for:
monitor scheduled tasks (cron jobs)
monitor entire application logic execution
alert on errors in applications
we are also working on examples of basic infrastructure monitoring using AlertGrid
This is such an open ended question, but I would start with physical measurements.
1. Are all the machines I think are hosting this site pingable?
2. Are all the machines which should be serving content actually serving some content? (Ideally this would be hit from an external network.)
3. Is each expected service on each machine running?
3a. Have those services run recently?
4. Does each machine have hard drive space left? (Don't forget the db)
5. Have these machines been backed up? When was the last time?
Once one lays out the physical monitoring of the systems, one can address those specific to a system?
1. Can an automated script log in? How long did it take?
2. How many users are live? Have there been a million fake accounts added?
...
These sorts of questions get more nebulous, and can be very system specific. They also usually can be derived reactively when responding to phsyical measurements. Hard drive fill up, maybe the web server logs got filled up because a bunch of agents created too many fake users. That kind of thing.
While plan A shouldn't necessarily be reactive, it is the way many a site setup a monitoring system.
Minimum: make sure it is running :)
However, some other stuff would be very useful. For example, the CPU load, RAM usage and (in multiuser systems) which user is running what. Also, for applications that access network, a list of network connections for each app. And (if you have access to client computer(s)) it would be cool to be able to see the 'window title' of the app - maybe check each 2-3 minutes if it changed and save it. Also, a list of files open by the application could be very useful, but it is not a must.
I think this is fairly simple - monitor so that you can be warned early enough before something goes wrong. That means monitor dependencies and the application itself.
It's really hard to provide specifics if you're not going to give details on the application you're monitoring, so I'd say use that as a general rule.
At a minimum you want to know that the system is healthy. This is subjective in what defines your system is healthy. Is it computers are up, the needed resources exist, the data is flowing through the system, the data is properly producing results, etc, etc.
In my project we do monitoring of most of this and then some. It really comes down to what is the highest level that you can use to analyze that everything is working. In our case we need to know down to the data output. If you just need to know down to the are these machines up it saves you on trying to show an inexperienced end user what is wrong.
There are also "off the shelf" tools that will do a lot of the hard work for you if you are just looking too hard into data results. I particularly liked Nagios when I was looking around but we needed more than it could easily show so I wrote our own monitoring system. Basically we also watch for "peculiarities" in the system, memory / cpu spikes, etc...
thanks everyone for the input, i was really looking for application-level monitoring not infrastructure monitoring, but it is good to know about both
the difference is:
infrastructure monitoring would be servers plus MS Exchange Server, Apache, IIS, and so forth
application monitoring would be user machines and the specific programs that they use to do their jobs, and/or servers plus the data-moving/backend applications that they run to keep the data flowing
sometimes it's hard to draw the line - an oversimplified definition might be "if your team wrote it, it's an application; if you bought it, it's infrastructure"
i think in practice it is best to monitor both
What you need to do is to break down the business process of the application and then have the software emit events at major business components. In addition, you'll need to create end to end synthetic transactions (eg. emulating end users clicking on a website). All that data would be fed into an monitoring tool. In the past, I've done JMX for applications of which flowed into Tivoli Monitoring's JMX Adapter and then I've done scripts that implement a "fake user" and then pipe in the results into Tivoli Monitoring's Script Adapter. Tivoli Monitoring takes the data and then creates application health and performance charts from that raw data.