I am working on a scanner with Nmap. I am expanding this scanner with NSE scripts.
I have a script that runs 'Nuclei', using Nmap. This script is made and used by someone else, and it has worked before. However, when I run it now, I get the error: sh: 1: nuclei: not found.
Nuclei is (of course) installed on the system, and it works as root and normal user. It looks like Nmap doesn't have access to Nuclei, but how to fix?
The NSE script:
local shortport = require "shortport"
local stdnse = require "stdnse"
portrule = function(host,port)
return true
end
action = function(host,port)
local handle = ""
local always = stdnse.get_script_args("nuclei.always")
local hostname = stdnse.get_hostname(host)
if port.number == 80 then
handle = io.popen("nuclei -u http://" .. hostname .. " -nc -silent -etags intrusive -rl 30 -rlm 1000 -bs 8 -c 8")
elseif port.number == 443 then
handle = io.popen("nuclei -u https://" .. hostname .. " -nc -silent -etags intrusive -rl 30 -rlm 1000 -bs 8 -c 8")
elseif always == "yes" then
handle = io.popen("nuclei -u " .. hostname .. " -nc -silent -etags intrusive -rl 30 -rlm 1000 -bs 8 -c 8")
end
local result = handle:read("*a")
handle:close()
return result
end
The Nmap command:
nmap -script=nuclei.nse -p80,443 -T2 IPADDRESS
Nmap is installed using Snap. It runs on Ubuntu.
The solution was quite simple:
Install nmap using apt, instead of snap did the job.
Related
I am trying to create an Airflow DAG from which I want to spin a Compute Engine instance with a docker image stored in Google Container Registry.
In other words, I wanted to replicate gcloud compute instances create-with-container with airflow dags using gcloud operators. I searched for airflow operators for such operations but couldn't find any way to make them work.
Possible references:
https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/compute.html
https://cloud.google.com/composer/docs/connect-gce-vm-sshoperator
A simple and clean solution to run a premade container using VMs with Airflow may consist in chaining the 3 steps below:
create a fresh new VM (through a BashOperator) with a startup script that pulls/runs the container and shut down the VM when the running is done;
use a PythonSensor to check when the VM is stopped (i.e. the docker finish running);
delete the VM (through a BashOperator) in order to repeat the previous steps when the airflow dag is triggered the next time.
All we need are the below bash commands:
bash_cmd = {
'active_account': \
'gcloud auth activate-service-account MYCLIENTEMAIL '
'--key-file=/PATH/TO/MY/JSON/SERVICEACCOUNT',
'set_project': \
'gcloud config set project MYPROJECTID',
'list_vm': \
'gcloud compute instances list',
'create_vm': \
'gcloud compute instances create-with-container VMNAME '
'--project=MYPROJECTID --zone=MYZONE --machine-type=e2-medium '
'--image=projects/cos-cloud/global/images/cos-stable-101-17162-40-5 '
'--boot-disk-size=10GB --boot-disk-type=pd-balanced '
'--boot-disk-device-name=VMNAME '
'--container-image=eu.gcr.io/MYPROJECTID/MYCONTAINER --container-restart-policy=always '
'--labels=container-vm=cos-stable-101-17162-40-5 --no-shielded-secure-boot '
'--shielded-vtpm --shielded-integrity-monitoring '
'--metadata startup-script="#!/bin/bash\n sleep 10\n sudo useradd -m bob\n sudo -u bob docker-credential-gcr configure-docker\n sudo usermod -aG docker bob\n sudo -u bob docker run eu.gcr.io/MYPROJECTID/MYCONTAINER\n sudo poweroff" ',
'delete_vm': \
'gcloud compute instances delete VMNAME --zone=MYZONE --delete-disks=boot',
}
active_account and set_project are used respectively to activate the service account and set the correct working project (where we want to run the VMs). This is required and needed when Airflow is running outside the GCP project where the VMs are instantiated. It's also important to have ComputeEngine privileges on the service account used. The container images to run must be located in the container registry of the same project where the VMs are instantiated.
list_vm returns the list of the existing VMs in the project with relative features and status (RUNNING/TERMINATED).
create_vm creates the VM attaching the docker to run from the container registry. The command to create the VM can be customized according to your needs. Important to note, you must add --metadata startup-script that includes the run of the docker and the VM power off when the docker finishes running. (to see how the startup script is generated see here).
delete_vm simply deletes the VM created by create_vm.
All these commands can be combined to work together in an Airflow DAG in this way:
import re
import os
import datetime
import subprocess
import airflow
from airflow.sensors.python import PythonSensor
from airflow.operators.bash_operator import BashOperator
def vm_run_check():
"function to list all the VMs and check their status"
finish_run = False
output = subprocess.check_output(
bash_cmd['active_account'] + " && " + \
bash_cmd['set_project'] + " && " + \
bash_cmd['list_vm'],
shell=True
)
output = output.decode("utf-8").split("\n")[:-1]
machines = []
for i in range(1,len(output)):
m = {}
for match in re.finditer(r"([A-Z_]+)( +)?", output[0]+" "*10):
span = match.span()
m[match.group().strip()] = output[i][span[0]:span[1]].strip()
machines.append(m)
machines = {m['NAME']:m for m in machines}
if VMNAME in machines:
if machines[VMNAME]['STATUS'] == 'TERMINATED':
finish_run = True
return finish_run
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'email': [''],
'email_on_failure': False,
'email_on_retry': False,
'retries': 0,
}
with models.DAG(
'MYDAGNAME',
catchup=False,
default_args=default_args,
start_date=datetime.datetime.now() - datetime.timedelta(days=3),
schedule_interval='0 4 * * *', # every day at 04:00 AM UTC
) as dag:
create_vm = BashOperator(
task_id="create_vm",
bash_command = bash_cmd['active_account'] + " && " + \
bash_cmd['set_project'] + " && " + \
bash_cmd['create_vm']
)
sensor_vm_run = PythonSensor(
task_id="sensor_vm_run"
python_callable=vm_run_check,
poke_interval=60*2, # check every 2 minutes
timeout=60*60, # check every 2 minutes for an hour
soft_fail=True,
mode="reschedule",
)
delete_vm = BashOperator(
task_id="delete_vm",
bash_command = bash_cmd['active_account'] + " && " + \
bash_cmd['set_project'] + " && " + \
bash_cmd['delete_vm']
)
create_vm >> sensor_vm_run >> delete_vm
i am not aware of lua script but i need some help.
Basically current lua script will receive structure.
in those structure has address parameter where will get two index parameters(ipv6 & ipv4) addresses.
lua script need to implement below case
ping ipv6 address and result will get store in local variable.
if local variable gets (ping success) will connect/call uv.tcp_connect for passed ipv6 address.
otherwise i will check the same for ipv4 address and try to connect/call uv.tcp_connect.
I am using online lua editor there its returning nil.
local results = load('ping -q -c1 -6 localhost 2>&1 >/dev/null && printf "IPv6: true" || (ping -q -c1 www.google.com 2>&1 >/dev/null && printf "IPv4 true" || printf "false")')
print(results)
output is:nil
and
if i am using in lua online editor ..
local handler = io.popen("ping -c 3 -i 0.5 www.google.com")-- wrong here.
local response = handler:read("*a")
print(response)
output error :
lua: main.lua:3: expected near '"ping -c 3 -i 0.5 www.google.com"'
kindly suggest me , am i missing something above.
To store output of system commands i suggest io.popen().
An example for conditional ping that tries first IPv6 and if fail IPv4...
> code.cmd
-- cmd(shell)
return function(shell)
return io.popen(shell, 'r'):read('a+')
end
> results={}
> results.ping=load(code.cmd)()('ping -q -c1 -6 localhost 2>&1 >/dev/null && printf "IPv6: true" || (ping -q -c1 localhost 2>&1 >/dev/null && printf "IPv4 true" || printf "false")')
> print(results.ping)
IPv6: true
...typed in a Lua console.
EDIT
Online Lua Environments dont support above code!
I have the following shell script that allows me to start my rails app, let's say it's called start-app.sh:
#!/bin/bash
cd /var/www/project/current
. /home/user/.rvm/environments/ruby-2.3.3
RAILS_SERVE_STATIC_FILES=true RAILS_ENV=production nohup bundle exec rails s -e production -p 4445 > /var/www/project/log/production.log 2>&1 &
the file above have permissions of:
-rwxr-xr-x 1 user user 410 Mar 21 10:00 start-app.sh*
if i want to check the process I do the following:
ps aux | grep -v grep | grep ":4445"
it'd give me the following output:
user 2960 0.0 7.0 975160 144408 ? Sl 10:37 0:07 puma 3.12.0 (tcp://0.0.0.0:4445) [20180809094218]
P.S: the reason i grep ":4445" is because i have few processes running on different ports. (for different projects)
now coming to monit, i used apt-get to install it, and the latest version from repo is 5.16, as i'm running on Ubuntu 16.04, also note that monit is running as root, that's why i specified the gid uid in the following. (because the start script is used to be executed from "user" and not "root")
Here's the configuration for monit:
set daemon 20 # check services at 20 seconds interval
set logfile /var/log/monit.log
set idfile /var/lib/monit/id
set statefile /var/lib/monit/state
set eventqueue
basedir /var/lib/monit/events # set the base directory where events will be stored
slots 100 # optionally limit the queue size
set mailserver xx.com port xxx
username "xx#xx.com" password "xxxxxx"
using tlsv12
with timeout 20 seconds
set alert xx#xx.com
set mail-format {
from: xx#xx.com
subject: monit alert -- $EVENT $SERVICE
message: $EVENT Service $SERVICE
Date: $DATE
Action: $ACTION
Host: $HOST
Description: $DESCRIPTION
}
set limits {
programOutput: 51200 B
sendExpectBuffer: 25600 B
fileContentBuffer: 51200 B
networktimeout: 10 s
}
check system $HOST
if loadavg (1min) > 4 then alert
if loadavg (5min) > 2 then alert
if cpu usage > 90% for 10 cycles then alert
if memory usage > 85% then alert
if swap usage > 35% then alert
check process nginx with pidfile /var/run/nginx.pid
start program = "/bin/systemctl start nginx"
stop program = "/bin/systemctl stop nginx"
check process redis
matching "redis"
start program = "/bin/systemctl start redis"
stop program = "/bin/systemctl stop redis"
check process myapp
matching ":4445"
start program = "/bin/bash -c '/home/user/start-app.sh'" as uid "user" and gid "user"
stop program = "/bin/bash -c /home/user/stop-app.sh" as uid "user" and gid "user"
include /etc/monit/conf.d/*
include /etc/monit/conf-enabled/*
Now monit, is detecting and alerting me when the process goes down (if i kill it manually) and when it's manually recovered, but it won't start that shell script automatically.. and according to /var/log/monit.log, it's showing the following:
[UTC Aug 13 10:16:41] info : Starting Monit 5.16 daemon
[UTC Aug 13 10:16:41] info : 'production-server' Monit 5.16 started
[UTC Aug 13 10:16:43] error : 'myapp' process is not running
[UTC Aug 13 10:16:46] info : 'myapp' trying to restart
[UTC Aug 13 10:16:46] info : 'myapp' start: /bin/bash
[UTC Aug 13 10:17:17] error : 'myapp' failed to start (exit status 0) -- no output
So far what I see when monit tries to execute the script is that it tries to load it (i can see it for less than 3 seconds using ps aux | grep -v grep | grep ":4445", but this output is different from the above output i showed up, it shows the content of the shell script being executed and specifically this one:
blablalba... nohup bundle exec rails s -e production -p 4445
and then it disappears. then it tries to re-execute the shell.. again and again...
What am I missing, and what is wrong with my configuration? note that I can't change anything in the start-app.sh because it's on production and working 100%. (i just want to monitor it)
Edit: To my understanding and experience, it seems to be a Environment Variable issue or path issue, but i'm not sure how to solve it, it doesn't make any sense to put the env variables inside monit .. what if someone else wanted to edit that shell script or add something new? i hope you get my point
As i expected, it was user-environment issue and i solved it by editing monit configuration as below:
Before (not working)
check process myapp
matching ":4445"
start program = "/bin/bash -c '/home/user/start-app.sh'" as uid "user" and gid "user"
stop program = "/bin/bash -c /home/user/stop-app.sh" as uid "user" and gid "user"
After (working)
check process myapp
matching ":4445"
start program = "/bin/su -s /bin/bash -c '/home/user/start-app.sh' user"
stop program = "/bin/su -s /bin/bash -c '/home/user/stop-app.sh' user"
Explanation: i removed (uid and gid) as "user" from monit because it will only execute the shell script in the name of "user" but it won't get/import/use user's env path, or env variables.
Telegraf v1.0.1
I'm not able to see telegraf[._] (tree) metric anymore after I enabled [[inputs.procstat]] plugin.
Telegraf is installed successfully. Process is running. I'm pretty much using the normal settings for inputs plugins and output plugin.
This is what I got:
ubuntu#jenkins:/tmp/giga_aks_testing/ansible$ grep -C 2 jenkins /etc/telegraf/telegraf.d/telegraf-custom-host-services-processes.conf; echo ; ps -eAf|grep jenkins; echo; pgrep -f jenkins; echo; cat -n /var/log/telegraf/telegraf.log; echo date; echo; ps -eAf|grep telegraf; echo ; sudo service telegraf status
[[inputs.procstat]]
exe = "jenkins"
prefix = "pgrep_serviceprocess"
root 2875 3685 0 2016 pts/3 00:00:00 sudo su jenkins
root 2876 2875 0 2016 pts/3 00:00:00 su jenkins
jenkins 2877 2876 0 2016 pts/3 00:00:00 bash
jenkins 11645 1 0 2016 ? 00:00:01 /usr/bin/daemon --name=jenkins --inherit --env=JENKINS_HOME=/var/lib/jenkins --output=/var/log/jenkins/jenkins.log --pidfile=/var/run/jenkins/jenkins.pid -- /usr/bin/java -Djava.awt.headless=true -jar /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8080
jenkins 11647 11645 0 2016 ? 05:33:22 /usr/bin/java -Djava.awt.headless=true -jar /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8080
ubuntu 21973 26885 0 06:57 pts/0 00:00:00 grep --color=auto jenkins
2875
2876
11645
11647
1 2017-01-07T06:54:00Z E! Error: procstat getting process, exe: [jenkins] pidfile: [] pattern: [] user: [] Failed to execute /usr/bin/pgrep. Error: 'exit status 1'
2 2017-01-07T06:55:00Z E! Error: procstat getting process, exe: [jenkins] pidfile: [] pattern: [] user: [] Failed to execute /usr/bin/pgrep. Error: 'exit status 1'
3 2017-01-07T06:56:00Z E! Error: procstat getting process, exe: [jenkins] pidfile: [] pattern: [] user: [] Failed to execute /usr/bin/pgrep. Error: 'exit status 1'
4 2017-01-07T06:57:00Z E! Error: procstat getting process, exe: [jenkins] pidfile: [] pattern: [] user: [] Failed to execute /usr/bin/pgrep. Error: 'exit status 1'
date
telegraf 19336 1 0 05:45 pts/0 00:00:04 /usr/bin/telegraf -pidfile /var/run/telegraf/telegraf.pid -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraftelegraf.d
ubuntu 21977 26885 0 06:57 pts/0 00:00:00 grep --color=auto telegraf
telegraf Process is running [ OK ]
ubuntu#jenkins:/tmp/giga_aks_testing/ansible$
Why, the log file is showing an error when the jenkins process is running and pgrep -f jenkins is returning valid result.
PS: [[inputs.procstat]] plugin uses pgrep -f <exe_value_pattern> for it's logic if pattern = method is used, and pgrep <executable> if exe = method is used.
The full /etc/telegraf/telegraf.d/telegraf-custom-host-services-processes.conf file is:
[[inputs.procstat]]
exe = "jenkins"
prefix = "pgrep_serviceprocess"
[[inputs.procstat]]
exe = "telegraf"
prefix = "pgrep_serviceprocess"
[[inputs.procstat]]
exe = "sshd"
prefix = "pgrep_serviceprocess"
OK. Seems like this is an OPEN bug.
Telegraf with [[inputs.procstat]] plugin entry won't barf if there's only one plugin in one file.
If you specify multiple entries, even if those exe = <executables_processes> are running, Telegraf will start spitting those errors out (PS: It won't stop Telegraf service from working though).
To fix the errors, this is what I did:
[[inputs.procstat]]
exe = "telegraf|.*"
prefix = "pgrep_serviceprocess"
Now, as pgrep is used for Telegraf's [[inputs.procstat]] plugin, it'll do this at OS level: pgrep "telegraf|.*".
Now, you can also just give exe = "." (simplest) or like exe = ".*" but practically those will not be easy to find out who actually is trying to do a grep on all processes running on the system.
NOTE: .* (will find every single processes running on the machine), so use it until we get a proper fix for this.
Related Source code Github file: https://github.com/influxdata/telegraf/blob/master/plugins/inputs/procstat/procstat.go
Related issue: https://github.com/influxdata/telegraf/issues/586
I still couldn't find, why "telegraf.x.x" metrics are not available after I enabled [[inputs.procstat]] input. Is that due to a separate file? I'm not sure. But, I can see procstat.x.x metric tree but telegraf.x.x metric tree is not visible now.
OR better,
One can also use:
[[inputs.procstat]]
pattern = "."
prefix = "pgrep_serviceprocess"
The above will do: pgrep -f "." where pattern is . (to catch everything aka every processs/cmd/service running on a machine).
OR (but the following is not scalable solution as you have to know for which user. In some boxes, Jenkins may be running using a user other than jenkins).
[[inputs.procstat]]
user = "jenkins"
prefix = "pgrep_serviceprocess"
The above will do: pgrep -u "jenkins" where user is jenkins (to catch everything aka every processs/cmd/service running on a machine).
To check whether jenkins is running or not or if enhanceio is running or not, you can use [[inputs.exec]] plugin as well. I simply used: [[inputs.filestat]] plugin and it worked when I looked for the pid file for both tools. https://github.com/influxdata/telegraf/tree/master/plugins/inputs/filestat
I started a docker container based on an image which has a file "run.sh" in it. Within a shell script, i use docker exec as shown below
docker exec <container-id> sh /test.sh
test.sh completes execution but docker exec does not return until i press ctrl+C. As a result, my shell script never ends. Any pointers to what might be causing this.
I could get it working with adding the -it parameters:
docker exec -it <container-id> sh /test.sh
Mine works like a charm with this command. Maybe you only forgot the path to the binary (/bin/sh)?
docker exec 7bd877d15c9b /bin/bash /test.sh
File location at
/test.sh
File Content:
#!/bin/bash
echo "Hi"
echo
echo "This works fine"
sleep 5
echo "5"
Output:
ArgonQQ#Terminal ~ docker exec 7bd877d15c9b /bin/bash /test.sh
Hi
This works fine
5
ArgonQQ#Terminal ~
My case is a script a.sh with content
like
php test.php &
if I execute it like
docker exec contianer1 a.sh
It also never returned.
After half a day googling and trying
changed a.sh to
php test.php >/tmp/test.log 2>&1 &
It works!
So it seems related with stdin/out/err.
>/tmp/test.log 2>&1
Please try.
And please note that my test.php is a dead loop script that monitors a specified process, if the process is down, it will restart it. So test.php will never exit.
As described here, this "hanging" behavior occurs when you have processes that keep stdout or stderr open.
To prevent this from happening, each long-running process should:
be executed in the background, and
close both stdout and stderr or redirect them to files or /dev/null.
I would therefore make sure that any processes already running in the container, as well as the script passed to docker exec, conform to the above.
OK, I got it.
docker stop a590382c2943
docker start a590382c2943
then will be ok.
docker exec -ti a590382c2943 echo "5"
will return immediately, while add -it or not, no use
actually, in my program, the deamon has the std input and std output, std err. so I change my python deamon like following, things work like a charm:
if __name__ == '__main__':
# do the UNIX double-fork magic, see Stevens' "Advanced
# Programming in the UNIX Environment" for details (ISBN 0201563177)
try:
pid = os.fork()
if pid > 0:
# exit first parent
os._exit(0)
except OSError, e:
print "fork #1 failed: %d (%s)" % (e.errno, e.strerror)
os._exit(0)
# decouple from parent environment
#os.chdir("/")
os.setsid()
os.umask(0)
#std in out err, redirect
si = file('/dev/null', 'r')
so = file('/dev/null', 'a+')
se = file('/dev/null', 'a+', 0)
os.dup2(si.fileno(), sys.stdin.fileno())
os.dup2(so.fileno(), sys.stdout.fileno())
os.dup2(se.fileno(), sys.stderr.fileno())
# do second fork
while(True):
try:
pid = os.fork()
if pid == 0:
serve()
if pid > 0:
print "Server PID %d, Daemon PID: %d" % (pid, os.getpid())
os.wait()
time.sleep(3)
except OSError, e:
#print "fork #2 failed: %d (%s)" % (e.errno, e.strerror)
os._exit(0)