So, I've forwaded the auth logs of my web application to graylog, and now I want to implement some alerts for brutefroce.
Each record sent from the webapp to Graylog contains some information, among which username and hashed password.
I would like to find with a query for the alerts that count the unique combination of the fields login and hashed_password.
I know how to setup the alerts, but can't find a working query.
For example, I've this logs:
... login=foo hashed_password=XXXX ...
... login=foo hashed_password=XXXX ...
... login=foo hashed_password=XXXX ...
... login=foo hashed_password=YYYY ...
... login=foo hashed_password=ZZZZ ...
... login=foo hashed_password=AAAA ...
... login=foo hashed_password=AAAA ...
... login=foo hashed_password=BBBB ...
I would like to find a query print only one time each combination of this two fields. I've tried with many tries but without success.
From the log aboce, it should print only:
... login=foo hashed_password=XXXX ...
... login=foo hashed_password=YYYY ...
... login=foo hashed_password=ZZZZ ...
... login=foo hashed_password=AAAA ...
... login=foo hashed_password=BBBB ...
Some of the queries I've tried are the following:
auth:"error" AND distinct(login+hashed_password)
auth:"error" AND count(distinct(login+hashed_password))
auth:"error" AND count(login(hashed_password))
I'm running on a graylog 4.0 version and Elastic Oss 7.10, all the servers are Centos7
First, the bad news. Graylog does not support Elasticsearch v7.11. 7.10 is the latest supported version.
Second, more bad news. Graylog does not support any of the functions you are trying to use in queries. The supported syntax can be found here: https://docs.graylog.org/en/4.0/pages/searching/query_language.html?highlight=syntax#search-query-language
You can do some of what you want using dashboards. I would also suggest trying aggregation events (Alerts/Events) to alert you to when these conditions occur.
I found a solution to my problem:
I just grouped the result from the query by IP Source and logins, then I added this condition to create events:
card(hashed_password) > 15
Same things with similar kind allarms, where i put these conditions:
[count(login) > 100 AND [card(IP_Source) == 1 AND card(login) == 1 AND card(hashed_password) == 1]]
Where I grouped by the same voices.
I suggest using a GROK pattern to filter out only the messages with the informations you are looking for (login and pass).
GROK patterns allowed me to trim logs, deleting all the unhelpful informations and keeping only what I needed under a specific field name, which is extremely useful when you need to monitor suspicious log-ins.
For example: in your case, you could create a field named "login_failed_foo" and then set an alarm when the number of occurencies of this field reaches a certain point.
Related
I'm trying to use serilog's ExpressionTemplate and write the log level in the following way:
{ "level": "INF", ... }
I know how to alias #l to level - level: #l
and I know how to format the level to three upper-case letters - #l:u3 but I'm unable to combine the two
I have tried the following without success (fails when trying to format the template with an exception):
level: #l:u3
{'level': #l:u3}
At the time of writing this is a limitation in Serilog.Expressions, which will hopefully be addressed soon.
Update: version 3.4.0-dev-* now on NuGet supports ToString(#l, 'u3').
You can work around it with conditionals:
{'level': if #l = 'Information' then 'INF' else if #l = 'Warning' then 'WRN' else 'ERR'}
With a few more branches for remaining levels.
(Alternatively, you could write and plug in a user defined function along the lines of ToShortLevel(#l), and use that instead.)
I am using Elasticsearch 6.2.4 with my RoR application using elasticsearch-rails and elasticsearch-model.
My indexation is runninng without getting any errors. but when I try to perform a search from the application I am getting this error from Elasticsearch
<Elasticsearch::Transport::Transport::Errors::BadRequest: [400] {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"text is empty (possibly HTTP/0.9)"}],"type":"illegal_argument_exception","reason":"text is empty (possibly HTTP/0.9)"},"status":400}>
Everything was working normal prior to the upgrade of Elasticsearch from 1.5 to 6.2.4
I simplified my search query to try narrowing down the problem.
q = { "query" => { "match_all" => {} } }
But I still getting the same error. Probably I am not specifying a type in the query but wouldn't be unnecessary since I have a match_all condition ?
> {"query":{"match_all":{}}}
< {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"text is empty (possibly HTTP/0.9)"}],"type":"illegal_argument_exception","reason":"text is empty (possibly HTTP/0.9)"},"status":400}
I am brand new to Elasticsearch so excuse me in advance if there are some evident stuff that I am missing
Do you have any idea what is causing this error ? If you need more specific info just ask and I'll update this question.
Thanks.
Search request from application is resulting in HTTP 400 Bad Request. Are you able to perform a search request from outside the application i.e. invoking a curl command from your local etc ?
It looks like Base CRM has upgraded their API and replaced all of their endpoints/parameters.
Previously I was able to retrieve "Won" deals using this call:
session = BaseCrm::Session.new("<LEGACY_ACCESS_TOKEN>")
session.deals.all(stage: :won, sort_by: :last_activity, sort_order: :desc, page: 1)
This query recently started ignoring my parameters, yet it continued to respond with unfiltered data (that was fun when I realized that was happening).
The new syntax is:
client = BaseCRM::Client.new(access_token: "<YOUR_PERSONAL_ACCESS_TOKEN>")
client.deals.where(organization_id: google.id, hot: true)
yet this does not work:
client.deals.where(stage_name: :won)
client.deals.where(stage_name: "Won")
client.deals.where(stage_id: 8) # specified ID found in Base Docs for "Won"
etc.
I've looked into the most recent updates to the Base CRM Gem as well as the Base CRM API Docs but have not found a solution to searching by specific deal stage.
Has anyone had any luck with the new API and this kind of query?
Is there a way to use the legacy API?
I've left message with Base but I really need to fix this, you know, yesterday.
Thanks for your help!
ADDITIONAL INFO
The legacy API/gem responded with JSON where the v2 API/gem responds with a BaseCRM::Deal object:
$ session.deals.find(123456)
# <BaseCRM::Deal
dropbox_email="dropbox#67890.deals.futuresimple.com",
name="Cool Deal Name",
owner_id=54321,
creator_id=65432,
value=2500,
estimated_close_date=nil,
last_activity_at="2016-04-21T02:29:43Z",
tags=[],
stage_id=84588,
contact_id=098765432,
custom_fields={:"Event Location"=>"New York, NY", :Source=>"Friend"},
last_stage_change_at="2016-04-21T02:08:20Z",
last_stage_change_by_id=559951,
created_at="2016-04-18T22:16:35Z",
id=123456,
updated_at="2016-04-21T02:08:20Z",
organization_id=nil,
hot=false,
currency="USD",
source_id=1466480,
loss_reason_id=nil
>
Checkout stage_id. Is this a bug? According to the Docs stage_id should return an integer between 1 and 10.
I am trying to use flume to use the Twitter Stream API and index the tweet to my elasticsearch. I setup my flume.conf to use com.cloudera.flume.source.TwitterSource as twitter source (with my dev tokens) and I use the default elastisearch for the sink.
I am able to get the tweets (because I also save it into HDFS, and when I open the file I can see the tweets) but when i search into my elasticsearch, I get as response :
{
_index: twitter-2014-02-14
_type: tweet-rt
_id: ilL5ZrBRSlqrZcsVUbnO-g
_version: 1
_score: 1
_source: {
#message: org.elasticsearch.common.xcontent.XContentBuilder#12da4409
#timestamp: 2014-02-14T10:16:13.000Z
#fields: {
timestamp: 1392372973000
}
}
here example of my flume config.
# - ElasticSearch Sink
TwitterAgent.sinks.ES.type = elasticsearch
TwitterAgent.sinks.ES.channel = FileChannel
TwitterAgent.sinks.ES.hostNames = 192.168.10.100:9300
TwitterAgent.sinks.ES.indexName = twitter
TwitterAgent.sinks.ES.indexType = tweet-rt
TwitterAgent.sinks.ES.clusterName = testou
Do I have to add something else ? I dont understand why ES cannot deserialize my tweet.
Any ideas?
thankyou
This is weird. It's doing some form of identityHashCode on the XContentBuilder to get that message and it should not.
I think I'd recommend clearing out Flume and re-installing. I'd be concerned about classpath and JAR dependency issues.
What version of Flume?
For others who come across this error, this is a bug in flume elastic search sink which has been fixed now. See https://issues.apache.org/jira/browse/FLUME-2126
If you are on flume version earlier than 1.6 you may want to cherry pick and build one with this patch against your version.
I would like to monitor elasticsearch using nagios.
Basiclly, I want to know if elasticsearch is up.
I think I can use the elasticsearch Cluster Health API (see here)
and use the 'status' that I get back (green, yellow or red), but I still don't know how to use nagios for that matter ( nagios is on one server and elasticsearc is on another server ).
Is there another way to do that?
EDIT :
I just found that - check_http_json. I think I'll try it.
After a while - I've managed to monitor elasticsearch using the nrpe.
I wanted to use the elasticsearch Cluster Health API - but I couldn't use it from another machine - due to security issues...
So, in the monitoring server I created a new service - which the check_command is check_command check_nrpe!check_elastic. And now in the remote server, where the elasticsearch is, I've editted the nrpe.cfg file with the following:
command[check_elastic]=/usr/local/nagios/libexec/check_http -H localhost -u /_cluster/health -p 9200 -w 2 -c 3 -s green
Which is allowed, since this command is run from the remote server - so no security issues here...
It works!!!
I'll still try this check_http_json command that I posted in my qeustion - but for now, my solution is good enough.
After playing around with the suggestions in this post, I wrote a simple check_elasticsearch script. It returns the status as OK, WARNING, and CRITICAL corresponding to the "status" parameter in the cluster health response ("green", "yellow", and "red" respectively).
It also grabs all the other parameters from the health page and dumps them out in the standard Nagios format.
Enjoy!
Shameless plug: https://github.com/jersten/check-es
You can use it with ZenOSS/Nagios to monitor cluster health, data indices, and individual node heap usage.
You can use this cool Python script for monitoring your Elasticsearch cluster. This script check your IP:port for Elasticsearch status. This one and more Python script for monitoring Elasticsearch can be found here.
#!/usr/bin/python
from nagioscheck import NagiosCheck, UsageError
from nagioscheck import PerformanceMetric, Status
import urllib2
import optparse
try:
import json
except ImportError:
import simplejson as json
class ESClusterHealthCheck(NagiosCheck):
def __init__(self):
NagiosCheck.__init__(self)
self.add_option('H', 'host', 'host', 'The cluster to check')
self.add_option('P', 'port', 'port', 'The ES port - defaults to 9200')
def check(self, opts, args):
host = opts.host
port = int(opts.port or '9200')
try:
response = urllib2.urlopen(r'http://%s:%d/_cluster/health'
% (host, port))
except urllib2.HTTPError, e:
raise Status('unknown', ("API failure", None,
"API failure:\n\n%s" % str(e)))
except urllib2.URLError, e:
raise Status('critical', (e.reason))
response_body = response.read()
try:
es_cluster_health = json.loads(response_body)
except ValueError:
raise Status('unknown', ("API returned nonsense",))
cluster_status = es_cluster_health['status'].lower()
if cluster_status == 'red':
raise Status("CRITICAL", "Cluster status is currently reporting as "
"Red")
elif cluster_status == 'yellow':
raise Status("WARNING", "Cluster status is currently reporting as "
"Yellow")
else:
raise Status("OK",
"Cluster status is currently reporting as Green")
if __name__ == "__main__":
ESClusterHealthCheck().run()
I wrote this a million years ago, and it might still be useful: https://github.com/radu-gheorghe/check-es
But it really depends on what you want to monitor. The above measures:
if Elasticsearch responds to HTTP
if ingestion rate drops under the defined levels
if total number of documents drops the defined levels
But of course there's much more that might be interesting. From query time to JVM heap usage. We wrote a blog post about the most important ones here: https://sematext.com/blog/top-10-elasticsearch-metrics-to-watch/
Elasticsearch has APIs for all these, so you may be able to use a generic check_http_json to get the needed metrics. Alternatively, you may want to use something like Sematext Monitoring for Elasticsearch, which gets these metrics out of the box, then forward threshold/anomaly alerts to Nagios. (disclosure: I work for Sematext)