I'm looking for some help and advice in terms of monitoring network connections to a L2TP / Ipse VPN platform Ubuntu 14.04.
My basic requirement is to get visibility of:
1) Who's logged in (over time).
2) How much data they have consumed.
A nice to have would be some configurable variables to show activity over a period of time (day, week, month).
Previously, I've been using PPTPD, and have used a great little script which is simple, but does the job. https://github.com/boukeversteegh/pptpd-monitor
I'm struggling to find anything for L2TP which has the same functionality. The only thing that comes close that I've found is 'bandwidthd', however, the available docs are pretty thin, and I'm struggling to get it configured and working.
Has anyone else managed to solve this problem in the past ?
Related
We’ve created some kind of python monitoring app that performs health-check of our system once in 10 minutes and sends text alarms to our engineers (via jabber/slack) if something went wrong.
Are there any best practices we can introduce to be sure monitoring works even if server it’s hosted on is down? Any good books/online materials covering stability topic? First idea was to use docker swarm and multiple servers (just because I know it exists and seems to solve the problem) but maybe there’re way better solutions I’m not aware of.
I would say the best practice would be to build your SRE stack out of off the shelf rather than home grown components.
prometheus, alertmanager and so on.
Then you want your actual alerting infrastructure to be cloud hosted - PagerDuty for example.
And use something like Pingdom as an external check that your crucial infrastructure is operating.
Does anyone know of an elasticsearch client library for iOS? Would be a bonus if it was written in swift as well.
The elastic search 'clients' section shows multiple libraries for a number of platforms but nothing for iOS, I feel like someone must have done this?
Cheers
I doubt that anyone has - last time I checked there were none and for good reasons. Keep in mind that in order to allow an IOS client (or Android for that matter) to use a client library to connect to Elasticsearch you'd have to open up your cluster for either an http or node access - which would allow anyone to do anything to your cluster.
Maybe you could proxy it to prevent deletions and insertions but even so it would open up your cluster's data and open you up to DoS attacks.
Generally a better idea is to create your own REST API that incorporates some type of authentication and authorization and does not open up your cluster to the world.
If you still feel strongly about moving forward you can always just hit the http interface of the REST API for ES. Or take a look at this project someone was working on a few years ago at least to give you a head start:
https://github.com/tallpsmith/ElasticSearchIOSHead
Some recent discussions on this topic:
http://elasticsearch-users.115913.n3.nabble.com/Objective-C-client-for-ElasticSearch-iphone-ipad-etc-td3911216.html
Running Elasticsearch server on a mobile device (android / iphone / ios)
This may be what you're looking for. ElasticSwift. Seems to be in active development but haven't looked deep into how far they've gone.
This is another iOS Swift client that was made back in 2018: Appbase-Swift. It's a lightweight ElasticSearch/appbase.io client. Doesn't seem like it's been updated in awhile though.
We tried a lot for elastic and finally concluded that it on most of the challenging side. another thing is a bit bulky to use on mobile, as per my understanding no room for the optimization. We can only use HTTP Web Service call for the same service. Also, the flavor of the offline search can not be implemented in the mobile elastic till date...
Our company provides web-based management software (servicedesk, helpdesk, timesheet, etc) for our clients.
One of them have been causing a great headache for some months complaining about the connection speed with our servers.
In our individual tests, the connection and response speeds are always great.
Some information about this specific client :
They have about 300 PC's on their local network, all using the same bandwith/server for internet access.
They dont allow us to ping their server, so we cant establish a trace route.
They claim every other site (google, blogs, news, etc) are always responding fast. We know for a fact they have no intention to mislead us and know this to be true.
They might have up to 100 PC's simulateneously logged in our software at any given time. They have a need to increase that amount up to 300 so this is a major issue.
They are helpfull and colaborative in this issue we are trying to resolve for a long time.
Some information about our server and software :
We have been able to allocate more then 400 users at a single time without major speed losses for other clients.
We have gone extensive lengths to make good use of data caching and opcode caching in the software itself, and we did notice the improvement (from fast to faster)
There are no database, CPU or memory bottlenecks or leaks. Other clients are able to access the server just fine.
We have little to no knowledge on how to do some analyzing on specific end-user problems (Apache running under Windows server), and this is where I could use a lot of help.
Anything that might be related to Apache configuration would also be helpfull.
While all signs points to it being an internal problem in this specific client network, we are dedicating this effort to solve that too, if that is the case, but do not have capable or instructed professionals to deal with network problems (they do, however, while their main argument is that 'all other sites are fast, only yours is slow')
you might want to have a look at the tools from google "page speed family": http://code.google.com/speed/page-speed/docs/overview.html
your customer should maybe run the page speed extension for you. maybe then you can find out what is the problem: http://code.google.com/speed/page-speed/docs/extension.html
From basic things likes page views per second to more advanced stuff like cpu or memory usage. Any ideas?
I think someone has asked the same type of question before here? Though I'm not too sure how helpful it is.
For CPU usage, etc, I would try RRDTool, or maybe something like Cacti.
Web service or web site? Since you mention page views: I believe you mean web site.
Google Analytics will probably give you everything you need to track usage statistics and best of all is free under most circumstances.
You might also want to monitor site up-time and have something to send email alerts if the site is down for some reason. We've used Nagios in the past and it works just fine.
I've been using monit (http://www.tildeslash.com/monit/) for years. It monitors CPU and memory usage as well as downtime for apache/mysql/etc... you can also configure it to notify you of outages and automatically restart services in real time.
I also use munin for reporting: http://munin.projects.linpro.no/
If you want reports on pageviews and whatnot, AWStats is the best I've used.
I use Nagios for general machine monitoring on Linux and I pretty much rely on Google Analytics for website reporting - I know that's not for everyone since some folks have privacy concerns about giving all their site data to Google.
Both are free and easy to install (Nagios is generally available through apt-get and Analytics is a pretty easy install on a site).
Nagios, however, can be a bear to configure.
I cast my vote for monit as well. The nice thing about is that it understands apache-status info and can notify/take actions when say 80% of max num of apache workers are in "busy" state.
but you need something else for hardware and general monitoring, something SNMP-aware, like zennos or zabbix
Munin and Cacti provide very nice interfaces and pre-built scripts for rrdtool. They can also monitor multiple servers and send out warnings and alerts through naigos.
What, at a minimum, should an application health-monitoring system do for you (the developer) and/or your boss (the IT Manager) and/or the operations (on-call) staff?
What else should it do above the minimum requirements?
Is monitoring the 'infrastructure' applications (ms-exchange, apache, etc.) sufficient or do individual user applications, web sites, and databases also need to be monitored?
if the latter, what do you need to know about them?
ADDENDUM: thanks for the input, i was really looking for application-level monitoring not infrastructure monitoring, but it is good to know about both
Whether the application is running.
Unusual cpu/memory/network usage.
Report any unhandled exceptions.
Status of various modules (if applicable).
Status of external components (databases, webservices, fileservers, etc.)
Number of pending background tasks (if applicable).
Maybe track usage of the application and report statistics on most/less used functionalities so you know where optimizations are most beneficial.
The answer is 'it depends'. Why do you need to monitor? How large is your operations staff? Do you need reporting? What is the application environment? Who cares if the application fails? Who cares if an exception happens? Are any of the errors recoverable? I could ask questions like these for a long time.
Great question.
We've been looking for some application-level monitoring solution for our needs some time ago without any luck. Popular monitoring solution are mostly addressed to monitor infrastrcture and - in my opinion - they are too complicated for a requirements of most of small and mid-sized companies.
We required (mainly) following features:
alerts - we wanted to know about
incident as fast as possible
painless management - hosted service wouldbe
the best
visualizations - it's good to know what is going on and take some knowledge from the data
Because we didn't find suitable solution we started to write our own. Finally we've ended with up-and-running service called AlertGrid. (You can check it for free of course.)
The idea behind it is to provide an easy way to handle custom monitoring scenarios. Integration API is very simple (one function with two required parameters). At the momment we and others are using it for:
monitor scheduled tasks (cron jobs)
monitor entire application logic execution
alert on errors in applications
we are also working on examples of basic infrastructure monitoring using AlertGrid
This is such an open ended question, but I would start with physical measurements.
1. Are all the machines I think are hosting this site pingable?
2. Are all the machines which should be serving content actually serving some content? (Ideally this would be hit from an external network.)
3. Is each expected service on each machine running?
3a. Have those services run recently?
4. Does each machine have hard drive space left? (Don't forget the db)
5. Have these machines been backed up? When was the last time?
Once one lays out the physical monitoring of the systems, one can address those specific to a system?
1. Can an automated script log in? How long did it take?
2. How many users are live? Have there been a million fake accounts added?
...
These sorts of questions get more nebulous, and can be very system specific. They also usually can be derived reactively when responding to phsyical measurements. Hard drive fill up, maybe the web server logs got filled up because a bunch of agents created too many fake users. That kind of thing.
While plan A shouldn't necessarily be reactive, it is the way many a site setup a monitoring system.
Minimum: make sure it is running :)
However, some other stuff would be very useful. For example, the CPU load, RAM usage and (in multiuser systems) which user is running what. Also, for applications that access network, a list of network connections for each app. And (if you have access to client computer(s)) it would be cool to be able to see the 'window title' of the app - maybe check each 2-3 minutes if it changed and save it. Also, a list of files open by the application could be very useful, but it is not a must.
I think this is fairly simple - monitor so that you can be warned early enough before something goes wrong. That means monitor dependencies and the application itself.
It's really hard to provide specifics if you're not going to give details on the application you're monitoring, so I'd say use that as a general rule.
At a minimum you want to know that the system is healthy. This is subjective in what defines your system is healthy. Is it computers are up, the needed resources exist, the data is flowing through the system, the data is properly producing results, etc, etc.
In my project we do monitoring of most of this and then some. It really comes down to what is the highest level that you can use to analyze that everything is working. In our case we need to know down to the data output. If you just need to know down to the are these machines up it saves you on trying to show an inexperienced end user what is wrong.
There are also "off the shelf" tools that will do a lot of the hard work for you if you are just looking too hard into data results. I particularly liked Nagios when I was looking around but we needed more than it could easily show so I wrote our own monitoring system. Basically we also watch for "peculiarities" in the system, memory / cpu spikes, etc...
thanks everyone for the input, i was really looking for application-level monitoring not infrastructure monitoring, but it is good to know about both
the difference is:
infrastructure monitoring would be servers plus MS Exchange Server, Apache, IIS, and so forth
application monitoring would be user machines and the specific programs that they use to do their jobs, and/or servers plus the data-moving/backend applications that they run to keep the data flowing
sometimes it's hard to draw the line - an oversimplified definition might be "if your team wrote it, it's an application; if you bought it, it's infrastructure"
i think in practice it is best to monitor both
What you need to do is to break down the business process of the application and then have the software emit events at major business components. In addition, you'll need to create end to end synthetic transactions (eg. emulating end users clicking on a website). All that data would be fed into an monitoring tool. In the past, I've done JMX for applications of which flowed into Tivoli Monitoring's JMX Adapter and then I've done scripts that implement a "fake user" and then pipe in the results into Tivoli Monitoring's Script Adapter. Tivoli Monitoring takes the data and then creates application health and performance charts from that raw data.