I'm using grep for analysing my log-files after an attack.
usually like that
grep -F "POST /xxxxx.php" ./access-log
Now someone attacked some of my websites but i don't know where the vulnerability, and also not, what the attackers ip address is. Now i want to find an ip-address, who sended a request to more than one of my websites, like that:
abcde.com-log:123.123.123.123 - - [12/Jan/2013:08:41:08 +0100] "POST /xxxxx.php HTTP/1.1" 200 1234 "-" "-"
wxyz.com-log:123.123.123.123 - - [12/Jan/2013:08:41:08 +0100] "POST /xxxxx.php HTTP/1.1" 200 1234 "-" "-"
but i don't know how i get grep or other unix tools to give me only that match, who matches is more than one log-file.
Assuming the IP address you want is the one that appears as the first field in each log file, try this:
awk '
/POST \/xxxxx\.php/ {
ip=$1
if ( !(ipFilePairs[ip,FILENAME]++) ) {
ipFileCnt[ip]++
ipFileList[ip] = ipFileList[ip] " " FILENAME
}
}
END {
for (ip in ipFileCnt)
if (ipFileCnt[ip] > 1)
print ip ":" ipFileList[ip]
}
' *.log
Related
We have a rails app(v5.2.6) running inside a unicorn.
we are bombarded with log msgs like the following:
24.33.33.243, 12.0.52.41 - - [26/Jun/2022:13:43:39 +0000] "GET /our_path HTTP/1.1" 200 - 0.0078
No idea where this log is coming from? its bombarding our log and I must find a way to filter it.
We have a custom Rails.logger, where we also filter some msg that applies to some regexs, the problem is that this msg gets to the log from another place
I'm using Graylog to manage my server logs.
I would like to filter the apache logs access to keep the logs with a http response code 4** and 5**
So I would like to use a regex:
If I search /HTTP/ I have the corresponding logs like:
[...] "HEAD /register HTTP/1.1" 301 460 "-" [...]
But if I search /HTTP\//, I have no message. I also tried with /HTTP\\// but same result.
I tried with the regex /HTTP(?:.*?)"\s[4|5](?:\d{2})/ but no message found.
How to search a simple pattern like HTTP/ with a regex in Graylog ?
Thx
Why don't you use an extractor to map http-status-codes to fields.
Then you can easily filter and group your logs to find those with special codes.
Please see the following links
Extractors
How to use a JSON extractor
I programmed my ESP8266 to read the soil moisture. Depending on the moisture a water pump gets activated. Now I wanted the ESP to tweet different sentences, depending on the situation.
Therefore I connected my twitter account to thingspeak.com and followed this code
Connecting to the internet works fine.
Problems:
It does not tweet every time and if it tweets, only the first word from a sentence shows up at twitter.
According to the forum, where I found the code, I already tried to replace all the spaces between the words with "%20". However then nothing shows up at twitter at all. Also single words are not always posted to twitter.
This is the code I have problems with:
// if connection to thingspeak.com is successful, send your tweet!
if (client.connect("184.106.153.149", 80))
{
client.print("GET /apps/thingtweet/1/statuses/update?key=" + API + "&status=" + tweet + " HTTP/1.1\r\n");
client.print("Host: api.thingspeak.com\r\n");
client.print("Accept: */*\r\n");
client.print("User-Agent: Mozilla/4.0 (compatible; esp8266 Lua; Windows NT 5.1)\r\n");
client.print("\r\n");
Serial.println("tweeted " + tweet);
}
I don't get any error messages.
Maybe you could help me to make it visible if the tweet was really sent and how I manage to tweet a whole sentence.
I am using the Arduino IDE version 1.8.9 and I am uploading to this board
The rest of the code works fine. The only problem is the tweeting.
Update
I now tried a few different things:
Checking server response
Works and helps a lot. The results are:
Single words as String don't get any response at all
Same for Strings like "Test%20Tweet"
Strings with multiple words like "Test Tweet" get the following response and the first word of the String shows up as a tweet
HTTP/1.1 200 OK
Server: nginx/1.7.5
Date: Wed, 19 Jun 2019 18:44:22 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 1
Connection: keep-alive
Status: 200 OK
X-Frame-Options: SAMEORIGIN
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, PUT, OPTIONS, DELETE, PATCH
Access-Control-Allow-Headers: origin, content-type, X-Requested-With
Access-Control-Max-Age: 1800
ETag: W/"RANDOM_CHARS"
Cache-Control: max-age=0, private, must-revalidate
X-Request-Id: THE_ID
1
I think the Content-Length might be the problem?
But I don't know how to change it in this code.
Checking if the connection succeded
I implemented this into my code an it never shows up on the monitor. So I think i never have a problem with not connecting.
Use a hostname instead of IP address
I tried it and never got a bad request. On the other hand nothing shows up on twitter at all.
Check if your tweet variable contains any new-line characters (carriage return or line feed). For example, the following variable would cause problems
String tweet = "Tweet no. 1\r\n";
due to the new-line characters at the end. These characters will cause the first line of the HTTP request to be cut short. I.e., instead of
GET /apps/thingtweet/1/statuses/update?key=api_key&status=Tweet no. 1 HTTP/1.1\r\n
it would become
GET /apps/thingtweet/1/statuses/update?key=api_key&status=Tweet no. 1\r\n
and the server would reject it with a 400 (Bad request) error.
On the other hand
String tweet = "Tweet no. 1";
would be fine.
If your tweets may contain such characters, then try encoding them before passing them to client.print():
tweet.replace("\r", "%0D");
tweet.replace("\n", "%0A");
Use a hostname instead of IP address
According to https://uk.mathworks.com/help/thingspeak/writedata.html, the relevant hostname for the API you are using is api.thingspeak.com. Use that instead of the IP address. This is preferable because the IP address a hostname points to can change regularly. (The IP address you are using doesn't even seem to be correct - and may already be out of date.)
I.e., change
if (client.connect("184.106.153.149", 80)) {
to
if (client.connect("api.thingspeak.com", 80)) {
API endpoint
Are you sure you are using the correct API endpoint? According to the link above, it looks like the API endpoint you need is https://api.thingspeak.com/update.json - so you may need to change
client.print("GET /apps/thingtweet/1/statuses/update?key=" + API + "&status=" + tweet + " HTTP/1.1\r\n");
to
client.print("GET /update.json?api_key=" + API + "&status=" + tweet + " HTTP/1.1\r\n");
Check if the connection succeeded
Presently, your device sends the HTTP request if connects to the server successfully - but doesn't give any indication if the connection fails! So add an else block to handle that scenario and notify the user via the serial console.
if (client.connect("api.thingspeak.com", 80)) {
client.print("GET /apps/thingtweet/1/statuses/update?key=" + API + "&status=" + tweet + " HTTP/1.1\r\n");
// etc.
}
else {
Serial.println("Connection to the server failed!");
}
Checking server response
To check the response from the server, add the following block to your main loop - which will print the server response via the serial console.
delay(50);
while (client.available()) {
String response_line = client.readString();
Serial.println(response_line);
}
To clarify: that code should go inside your loop() function.
The response should include a status line - such as HTTP/1.1 200 OK if the request was successful, or HTTP/1.1 400 Bad Request if there was a problem.
In the case of a Bad request response, the full message will quite likely contain more information about the precise reason the request failed.
HTTP vs HTTPs
Lastly, are you sure that the API supports (plain, unencrypted) HTTP as well as HTTPs? If not, that may be your problem.
I'm trying to write a lua Proto to parse our private protocol on http. But Wireshark didn't enter my dissector function when the media_type to "application/octet-stream". When the media_type is set to "text/html", everything looks fine.
Is there special handling for application/octet-stream?
I was working on it for almost a day, Could you help me out?
Thx a lot
My wireshark version is 1.10.2 on mac osx 10.8.5
Here's my code
do
local myproto= Proto("myprotoProtocol","myproto Protocol")
local f_version= ProtoField.uint32("Version","Version",base.DEC)
myproto.fields = {f_version}
local data_dis = Dissector.get("data")
local function myproto_dissector(tvb,pkt,root)
print("enter myproto_dissector, tvb.len:"..tostring(tvb:len()))
if tvb:len() < 17 then return false end
pkt.cols.protocol = "myproto"
local t =root:add(myproto,tvb)
t:add(f_version,tvb(0,2))
local version = tvb(0,2).uint()
print("version:"..tostring(version))
return true
end
function myproto.dissector(tvb,pkt,root)
print("enter myproto.dissector")
if not myproto_dissector(tvb,pkt,root) then
data_dis:call(tvb,pkt,root)
end
end
local tbl= DissectorTable.get("media_type")
tbl:add("application/octet-stream",myproto)
--tbl:add("text/html",myproto) --text/html looks fine
print("adding myproto into DissectorTable")
end
I use tshark to debugging
for application/octet-stream
$tshark -r test.pcapng |grep application/octet-stream
108 40.536817000 10.8.0.14 -> 10.130.142.72 HTTP 418 POST /protocol?uid=101225&uid=101225&_t=1382115502 HTTP/1.1 (application/octet-stream)
111 40.596037000 10.130.142.72 -> 10.8.0.14 HTTP 63 HTTP/1.1 200 OK (application/octet-stream)
120 40.657143000 10.8.0.14 -> 10.130.142.72 HTTP 445 POST /protocol?uid=101225&uid=101225&_t=1382115502 HTTP/1.1 (application/octet-stream)
124 40.729645000 10.130.142.72 -> 10.8.0.14 HTTP 63 HTTP/1.1 200 OK (application/octet-stream)
219 41.810493000 10.8.0.14 -> 10.130.142.72 HTTP 488 POST /protocol?uid=101225&uid=101225&_t=1382115503 HTTP/1.1 (application/octet-stream)
226 41.919401000 10.130.142.72 -> 10.8.0.14 HTTP 63 HTTP/1.1 200 OK (application/octet-stream)
$tshark -r test.pcapng -X lua_script:canon.lua | grep myproto
adding myproto into DissectorTable
for text/html
$tshark -r test.pcapng -X lua_script:canon.lua | grep myproto
adding myproto into DissectorTable
enter myproto.dissector
enter myproto_dissector, tvb.len:2
enter myproto.dissector
enter myproto_dissector, tvb.len:6
enter myproto.dissector
enter myproto_dissector, tvb.len:6
Perhaps it might be the bug of wireshark when the media_type is not list in the dissector table. 'application/octet-stream' is not listed on the table yet.
After I use Lua->evaluate in Wireshark, The dissector table shows my protocol like this, 'application/octet-stream' is in mess code.
When i use 'print(tbl:get_dissector("application/octet-stream"))' in tshark, it shows "MYPROTO". Looks the correct one.
Please submit a bug to Wireshark, at bugs.wireshark.org, either with or without your code changes, but preferably with a sample capture file showing the problem (even if it's only one or two packets that's fine).
(I would have made this a comment but I don't have enough points to do that apparently)
I've been checking my production.log today and there's a number of requests hitting my site that appear to be malicious, but I'm confused as to how they're even getting to us.
For example:
Processing PublicController#unknown_request (for 217.23.4.13 at 2009-11-09 09:15:52) [GET]
Parameters: {"anything"=>["results.aspx"], "action"=>"unknown_request", "first"=>"200", "controller"=>"public", "q"=>"\"bbs/cbbs.cgi?\" intitle:\"Book\" intext:\"2008\" site:.uz ", "count"=>"200", "FORM"=>"PERE"}
Completed in 16ms (View: 12, DB: 0) | 200 OK [http : // search . live .com /results.aspx?q=%22bbs/cbbs.cgi%3F%22%20intitle%3A%22Book%22%20intext%3A%222008%22%20site%3A.uz%20&count=200&first=200&FORM=PERE]
These are happening every 30 seconds or so. Obviously, PublicController/Unknown_request is my controller/action 404 error.
The access log shows these requests as:
217.23.4.13 - - [09/Nov/2009:09:57:25 +1000] "GET http://search.live.com/results.aspx?q=%22en-gb.html%22%20intitle%3A%22Home%22%20intext%3A%222006%22%20site%3A.mn%20&count=200&first=400&FORM=PERE HTTP/1.1" 200 3626 "-" "Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.1$
How are these requests even hitting my site? Does anyone have any ideas?
I think this might be the same problem you're having: http://penguinpetes.com/b2evo/index.php?p=567&more=1&c=1&tb=1&pb=1
Basically, live/bing are doing some sort of testing that involves going to your site looking like someone searched something completely irrelevant to the content you have.