I am trying to reduce the size of the message sent from my Windows event logs to graylog and I cannot for the life of me figuring out how tell it to drop certain fields
The only thing I can figure out is that i'm supposed to use delete() but how to use and where to place it in my config is very frustrating.
What I have so far is:
Exec $Message = delete($TargetLogonID);
But this results in:
Couldn't parse Exec block at C:\Program Files (x86)\nxlog\conf\nxlog.conf:67; couldn't parse statement at line 67, character 39 in C:\Program Files (x86)\nxlog\conf\nxlog.conf; function 'delete()' does not exist or takes different arguments
I think I figured it out. In my <Input eventlog>
I added
Exec delete($SubjectLogonId);
Exec delete($KeyLength);
Exec delete($Keywords);
Exec delete($SubjectUserSid);
Exec delete($ThreadID);
Exec delete($TransmittedServices);
Exec delete($Version);
Exec delete($LogonGuid);
Exec delete($LmPackageName);
Exec delete($ImpersonationLevel);
Exec delete($RecordNumber);
Exec delete($SourceModuleType);
Exec delete($AuthenticationPackageName);
Exec delete($OpcodeValue);
Exec delete($ProcessID);
Exec delete($ProcessName);
Exec delete($ProviderGuid);
Exec delete($TargetLogonId);```
Related
Im' trying to figure out how Whenever gem is working.
I would like to create a User instance every minute.
I put the following in my schedule.rb file :
set :output, "/log/cron_log.log"
set :environment, 'development'
every 1.minute do
rake "User.create!(nni: 'TEST0',
password: '123123',
password_confirmation: '123123',
nomprenom: 'user')"
end
Then I typed whenever --update-crontab and checked with crontab -l and get the following
# Begin Whenever generated tasks for: /home/harri/Documents/argia/config/schedule.rb at: 2018-07-10 17:19:18 +0200
* * * * * /bin/bash -l -c 'cd /home/harri/Documents/argia && RAILS_ENV=development bundle exec rake User.create!(nni: '\''TEST0'\'', password: '\''123123'\'', password_confirmation: '\''123123'\'', nomprenom: '\''user'\'') --silent >> /log/cron_log.log 2>&1'
# End Whenever generated tasks for: /home/harri/Documents/argia/config/schedule.rb at: 2018-07-10 17:19:18 +0200
Then I start my localhost and nothing happens, nothing in the log and it doesn't create any User instance.
I'm trying to figure out how Whenever gem is working.
It works by generating a crontab file with your commands. Then OS/crond take over.
Then I start my localhost and nothing happens, nothing in the log and it doesn't create any User instance.
Well, that's not a valid rake command you have there. Check cron logs, you'll see an error there. Did you mean to use runner?
I am doing a cron tab in AWS - Elastic Beanstalk with Ruby on Rails 3, but I don't know what is wrong.
I have this code in my .ebextensions/default.config
container_commands:
01remove_old_cron_jobs:
command: "crontab -r || exit 0"
02send_test_email:
command: crontab */2 * * * * rake send_email:test
leader_only: true
I receive this error:
Failed on instance with return code: 1 Output: Error occurred during build: Command 02send_test_email failed .
UPDATE 1
I tried next:
crontab.txt
*/2 * * * * rake send_email:test > /dev/null 2>&1
default.config
02_crontab:
command: "cat .ebextensions/crontab.txt | crontab"
leader_only: true
RESULT: No errors, but it does not work.
UPDATE 2
crontab.sh
crontab -l > /tmp/cronjob
#CRONJOB RULES
echo "*/2 * * * * /usr/bin/wget http://localhost/crontabs/send_test_email > /dev/null 2>&1" >> /tmp/cronjob
#echo "*/2 * * * * rake send_email:test > /dev/null 2>&1" >> /tmp/cronjob
crontab /tmp/cronjob
rm /tmp/cronjob
echo 'Script successful executed, crontab updated.'
default.config
02_crontab:
command: "/bin/bash .ebextensions/crontab.sh"
leader_only: true
RESULT: Works with url, but not with rake task.
Updated for 2018
In order to get this to work on the latest version of Elastic Beanstalk, I added the following to my .ebextensions:
.ebextensions/0005_cron.config
files:
"/etc/cron.d/mycron":
mode: "000644"
owner: root
group: root
content: |
56 11 * * * root . /opt/elasticbeanstalk/support/envvars && cd /var/app/current && /opt/rubies/ruby-2.3.4/bin/bundle exec /opt/rubies/ruby-2.3.4/bin/rake send_email:test >> /var/app/current/log/cron.log 2>&1
commands:
remove_old_cron:
command: "rm -f /etc/cron.d/*.bak"
How I got there:
There are four main issues to confront when trying to cron a rake task in AWS EB:
The first hurdle is making sure all of your EB and Rails environment variables are loaded. I beat my head against the wall a while on this one, but then I discovered this AWS forum post (login may be required). Running . /opt/elasticbeanstalk/support/envvars loads all of your environment variables.
Then we need to make sure we cd into the current app directory using cd /var/app/current.
Next we need to know where to find the bundle and rake executables. They are not installed in the normal bin directories, but are located in a directory specific to your ruby version. To find out where your executables are located, ssh into your EB server (eb ssh) and then type the following:
$ cd /var/app/current
$ which bundle
/opt/rubies/ruby-2.3.4/bin/bundle
$ which rake
/opt/rubies/ruby-2.3.4/bin/rake
You could probably guess the directory based on your ruby version, but the above commands will let you know for sure. Based on the above, your can build your rake command as:
/opt/rubies/ruby-2.3.4/bin/bundle exec /opt/rubies/ruby-2.3.4/bin/rake send_email:test
NOTE: If you update your ruby version, you will likely need to update your cron config as well. This is a little brittle. I'd recommend making a note in your README on this. Trust me, six months from now, you will forget.
The fourth thing to consider is logging. I'd recommend logging to the same location as your other rails logs. We do this by tacking on >> /var/app/current/log/cron.log 2>&1 to the end of our command string.
Putting all of this together leads to a cron command string of:
. /opt/elasticbeanstalk/support/envvars && cd /var/app/current && /opt/rubies/ruby-2.3.4/bin/bundle exec /opt/rubies/ruby-2.3.4/bin/rake send_email:test >> /var/app/current/log/cron.log 2>&1
Finally, I referenced the latest AWS documentation to build an .ebextensions config file for my cron command. The result was the .ebextensions/0005_cron.config file displayed at the top of this answer.
I am having the same issue. Though I figured out that the reason that rake task doesn't run correctly on eb is because of RACK_ENV, RAILS_ENV and BUNDLE_WITHOUT
Defaults of eb:
RACK_ENV: production
RAILS_ENV: production
BUNDLE_WITHOUT: test:development
When the cron runs rake task, it runs in development mode, and gives gem not found error, as gems grouped in development are not installed.
you can see this by changing your cron a bitfrom:
*/2 * * * * rake send_email:test > /dev/null 2>&1
to:
*/2 * * * * cd /var/app/current/ && /usr/bin/bundle exec /usr/bin/rake send_email:test > /tmp/cron_log 2>&1
and then checking the /tmp/cron_log file
To know the location of bundle and rake, run
which bundle
which rake
I tried setting RAILS_ENV in command in cron, but that didn't work aswell
One quick fix is to set
BUNDLE_WITHOUT to null
EDIT:
Finally I got it to work,
.ebextensions/.config
files:
"/tmp/cron_jobs" :
mode: "000777"
content: |
1 10 * * * cd /var/app/current/ && RACK_ENV=production rake some:task >> /var/app/current/log/cron_log 2>&1
encoding: plain
container_commands:
01_delete_cron_jobs:
command: "crontab -r -u webapp || exit 0"
02_add_cron_jobs:
command: "crontab /tmp/cron_jobs -u webapp"
leader_only: true
option_settings:
- option_name: RAILS_ENV
value: production
- option_name: RACK_ENV
value: production
Notice the '-u webapp' when removing and adding cron, this will run this cron under user webapp. The above will also run in production mode. And the output will be dumped in log/cron_log file.
If the above wont work then adding 'Bundle exec' before 'rake some:task' might work.
I've seen these used with separate files in .ebextensions, such as:
02send_test_email:
command: "cat .ebextensions/crontab | crontab"
leader_only: true
I haven't gotten around to it yet, but I took note of this along the way. Let us know if this works.
This stackoverflow post has much more information
After Update 1/2:
Cron doesn't know where rake is. Your application runs from /var/app/current, and you need to be running bundle exec rake from that directory.
Elastic beanstalk is horrible with logging errors, to get this right, ssh to the machine and experiment until you have the commands right, then put this back into your cron script. You can even try and re-run some of the eb scripts as found in the logs, then reverse that into your ebextensions files.
In my schedule.rb file I have the following lines:
set :output, '/log/cron_log.log'
every 5.minutes do
command 'echo "hello"'
end
I ran whenever -w as suggested in this question Rails, using whenever gem in development, and I assume the cronfile is written and running. (I tried restarting the Rails server as well.)
And when I run $ crontab -l I see the following:
0,5,10,15,20,25,30,35,40,45,50,55 * * * * /bin/bash
-l -c 'echo "hello" >> /log/cron_log.log 2>&1'
But I can't find the log file. I checked in rails_project/log, ~/log and /log to no avail. On OSX btw.
How can I get the log file to be written to the rails project log folder?
Where is your log?
You're putting the output file at the highest directory level:
$ cd /log
To see if the file exists and if it has data in it:
$ ls -la cron_log.log
To create the log file if needed:
$ touch cron_log.log
To open up permissions for your own local debugging (do NOT do this in production!)
$ chmod +rw cron_log.log
Is your command running?
To run the command manually to find out if it works as you expect:
$ /bin/bash -l -c 'echo "hello" >> /log/cron_log.log 2>&1'
To improve your security and protect your path, use full paths:
wrong: command 'echo "hello"'
right: command '/bin/echo "hello"'
To find the command full path:
$ which echo
To verify the cron is running as you expect:
$ sudo grep CRON /var/log/syslog
The grep result should have lines that something like this:
Jan 1 12:00:00 example.com CRON[123]: (root) CMD (... your command here ...)
Are you on a Mac?
If you're not seeing output in the syslog, and you're on a Mac, you may want to read about the Mac OSX switching from cron to launchd.
See the cron plist (/System/Library/LaunchDaemons/com.vix.cron.plist) and use a stdout/stderr path to debug cron itself. I don't recall if launchctl unloading and launchctl loading the plist is sufficient, or since it's a system daemon if you'd have to restart entirely. (see where is the cron log file in lion)
How to log relative to Rails?
To put the log relative to a Rails app, omit the leading slash (and typically call it cron.log)
set :output, "log/cron.log"
To put the log in a specific fully-qualified directory:
set :output, '/abc/def/ghi/log/cron.log'
The Whenever wiki has some good examples about redirecting output:
https://github.com/javan/whenever/wiki/Output-redirection-aka-logging-your-cron-jobs
Examples:
every 3.hours do
runner "MyModel.some_process", :output => 'cron.log'
rake "my:rake:task", :output => {:error => 'error.log', :standard => 'cron.log'}
command "/usr/bin/cmd"
end
I just started out with Rails, so excuse my fairly basic question. I am already noticing that the rake routes command takes a while to execute everytime I run it. I have about 20 routes for 3 controllers and it takes about 40 seconds to execute.
Is that normal? How could I speed this up?
P.S.: I am on Windows 7 with Rails 3.1.3 (set up with Rails Installer).
The rake routes task depends on the environment task which loads your Rails environment and requires thousands of Ruby files.
The startup time of a Rails environment and the corresponding rake routes execution time are very close (on my Linux on-steroids-laptop with a Rails application with ~ 50 routes):
$ time ruby -r./config/environment.rb -e ''
real 0m5.065s
user 0m4.552s
sys 0m0.456s
$ time rake routes
real 0m4.955s
user 0m4.580s
sys 0m0.344s
There is no easy way to decrease startup time as it relies on the way your interpreter requires script files : http://rhnh.net/2011/05/28/speeding-up-rails-startup-time
I came up with a solution to rake routes taking about 8 seconds to run every time. It's a simple file based cache that runs bundle exec rake routes, stores the output in a file under tmp. The filename is the md5 hash of config/routes.rb, so if you make a change and change it back, it will use the old cached file.
I put the following bash functions in an executable file I call fastroutes:
if [ ! -f config/routes.rb ]; then
echo "Not in root of rails app"
exit 1
fi
cached_routes_filename="tmp/cached_routes_$(md5 -q config/routes.rb).txt"
function cache_routes {
bundle exec rake routes > $cached_routes_filename
}
function clear_cache {
for old_file in $(ls tmp/cache_routes*.txt); do
rm $old_file
done
}
function show_cache {
cat $cached_routes_filename
}
function show_current_filename {
echo $cached_routes_filename
}
function main {
if [ ! -f $cached_routes_filename ]; then
cache_routes
fi
show_cache
}
if [[ "$1" == "-f" ]]
then
show_current_filename
elif [[ "$1" == "-r" ]]
then
rm $cached_routes_filename
cache_routes
else
main
fi
Here's a github link too.
This way, you only have to generate the routes once, and then fastroutes will used the cached values.
That seems a bit long, but do you really need to run rake routes that often? On my system, OSX Lion/Rails 3.2.0, rake routes takes ~10s.
In your Rakefile:
#Ouptut stored output of rake routes
task :fast_routes => 'tmp/routes_output' do |t|
sh 'cat', t.source
end
#Update tmp/routes_output if its older than 'config/routes.rb'
file 'tmp/routes_output' => 'config/routes.rb' do |t|
outputf = File.open(t.name, 'w')
begin
$stdout = outputf
Rake.application['routes'].invoke
ensure
outputf.close
$stdout = STDOUT
end
end
Rails environment takes a huge more amount of time to be loaded on Windows. I recommend you to give Unix a try, like Ubuntu, as Windows is the worst environment in which you can run and develop Ruby on Rails applications. But if you are just trying Rails, Windows is enough :)
When I execute %rake college:create[demo], I get the following error,
zsh: no matches found: college:create[demo]
Anybody has a solution for this?
when I execute rake -T, this is what I get when as one of the lines of the output:
rake college:create[config_name] # create a college profile
So, it is a valid command, but still zsh shows the error.
Try with:
rake college:create\[demo\]
You can also use noglob
noglob rake college:create[demo]
or just alias it in your .zshrc
alias rake='noglob rake'
zsh is trying to interpret your command as a wildcard file specication. college:create[demo] will expand to the list of existing files that match one of:
college:created
college:createe
college:createm
college:createo
This page shows some of the wildcarding that zsh performs, the specific example in this case being:
the [123] specifier, which indicates any of the characters 1, 2, or 3.
You need to escape the argument so that zsh doesn't think you're giving it a wildcard, such as with:
rake 'college:create[demo]'
The manpage for zshexpn shows all the expansions done on command lines in great detail. Search for Filename Generation for the xyzzy[demo] style generations.
If you are using rake via bundle exec or from bin/ dir add this to you .zshrc file:
alias bin/rake='noglob rake'
# or
alias rake="noglob bundled_rake