Not able to find curator.yml (elasticsearch-curator) in linux - devops

Official site of elasticsearch says the default config file exists in /home/username/.curator/curator.yml
https://www.elastic.co/guide/en/elasticsearch/client/curator/current/command-line.html
But there is no such folder.
Also, I tried creating curator.yml and give path using --config option. But, it throws me error
curator --config ./curator.yml
Error: no such option: --config
Installation was done using apt
sudo apt-get update && sudo apt-get install elasticsearch-curator
Help me create a config file as I want to delete my log-indexes

Please note that the documentation does not say it that file exists after creation, it says:
If --config and CONFIG.YML are not provided, Curator will look in ~/.curator/curator.yml for the configuration file.
The file must be created by the end user.
Also, if you installed via:
sudo apt-get update && sudo apt-get install elasticsearch-curator
but did not add the official Elastic repository for Curator, then you installed an older version. Please check which version you are running with:
$ curator --version
curator, version 5.4.1
If you do not see the current version (5.4.1 at the time this answer was added), then you do not have the appropriate repository installed.
The official documentation provides an example client configuration file here.
There are also many examples of action files in the examples

Yes, one needs to create both the curator.yml as well as action.yml files.
Since I am on centos 7, I happened to install curator from RPM, and in its default /opt/elastic-curator' I could follow up this good blog (but badly formatted!) : https://anchormen.nl/blog/big-data-services/monitoring-aws-redshift-with-elasticsearch/ to ge the files as follows(you may modify according to your needs) :
curator.yml
---
# Remember, leave a key empty if there is no value. None will be a string,
# not a Python "NoneType"
client:
hosts:
- <host1>
- <host2, likewise upto hostN >
port: 9200
url_prefix:
use_ssl: False
certificate:
client_cert:
client_key:
ssl_no_validate: False
http_auth:
timeout: 30
master_only: False
logging:
loglevel: INFO
logfile: /var/log/curator.log
logformat: default
blacklist: []
and an action.yml as follows :
---
# Remember, leave a key empty if there is no value. None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True. If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
1:
action: rollover
description: Rollover the index associated with index 'name', which should be in the form of prefix-000001 (or similar),or prefix-YYYY.MM.DD-1.
options:
disable_action: False
name: redshift_metrics_daily
conditions:
max_age: 1d
extra_settings:
index.number_of_shards: 2
index.number_of_replicas: 1
2:
action: rollover
description: Rollover the index associated with index 'name' , which should be in the form of prefix-000001 (or similar), or prefix-YYYY.MM.DD-1.
options:
disable_action: False
name: redshift_query_metadata_daily
conditions:
max_age: 1d
extra_settings:
index.number_of_shards: 2
index.number_of_replicas: 1

Related

spring-cloud-skipper : How to delete and re-deploy a package

I want to redeploy a package but I am getting an error:
skipper:>package install --package-name sg-cloud-MbakTestworld
--package-version 0.0.1 --release-name MbakTestworld --file E:\skipper\apps\MbakTestworld-upgrade-local.yml
Result:
Release with the name [] already exists and it is not deleted. Details
of the error have been omitted. You can use the stacktrace command to
print the full stacktrace.
My yml File:
spec:
applicationProperties:
server.port: 8029
spring.profiles.active: mbakCloud
deploymentProperties:
spring.cloud.deployer.memory: 512m
I found the answer:
skipper:>release delete --release-name MbakTestworld
MbakTestworld has been deleted.
Source: https://docs.spring.io/spring-cloud-skipper/docs/current/reference/htmlsingle/

How to properly setup docker-sync to exclude folders

I am trying to setup docker-sync to exclude my app/cache and app/logs folder but it is not working.
Things I've tried:
Using sync_excludes: ['.idea', 'app/cache/', 'app/logs/'] but it will be translated to something like this
command unison -ignore='Name .idea' -ignore='Name app/cache/*'
-ignore='Name app/logs/*'
So then I tried using sync_args and set like this:
sync_args:
- "-debug verbose"
- "-ignore='Path app/cache'"
- "-ignore='Path app/logs'"
command unison -ignore='Name .idea' -ignore='Name systems' \
-debug verbose -ignore='Path app/cache/*' -ignore='Path app/logs/*'
looking at the logs I can see this
[unox][DEBUG]: recvCmd: DIR
[unox][DEBUG]: sendCmd: OK
[fswatch+] >> OK
[pred] immutable 'app/cache' = false
[pred] ignore 'app/cache/prod' = true
[pred] ignorenot 'app/cache/prod' = false
[ignore] buildUpdateChildren: ignoring path app/cache/prod
But this still seeing the events being triggered and it is still syncing to my host machine.
[pred] ignore 'app/cache/prod/annotations/5702f47f407ddb07532bfd60d8ea2919489ef4bc#__construct.cache.php' = false
[pred] ignore 'app/cache/prod/annotations/f21b469a2214195ff16e2af43f249bdbfa245c25#findPublishedOr404.cache.php' = false
Anyone know what I am missing?
my last version look like this:
version: "2"
options:
# optional, activate this if you need to debug something, default is false
# IMPORTANT: do not run stable with this, it creates a memory leak, turn off verbose when you are done testing
verbose: true
syncs:
#IMPORTANT: ensure this name is unique
dt-akeneo-unison-sync:
notify_terminal: true
# which folder to watch / sync from - you can use tilde (~), it will get expanded. Be aware that the trailing slash makes a difference
# if you add them, only the inner parts of the folder gets synced, otherwise the parent folder will be synced as top-level folder
src: './'
# the files should be own by root in the target cointainer
sync_userid: 1000
sync_strategy: 'unison'
# optional, a list of regular expressions to exclude from the fswatch - see fswatch docs for details
watch_excludes: ['\.git', '\.gitignore', '.*\.md']
sync_args:
- "-debug verbose" #force Unison to choose the file with the later (earlier) modtime
- "-ignore='Path app/cache'"
- "-ignore='Path app/logs'"
- "-ignore='Path .git'"
- "-ignore='Path .git'"
- "-ignore='Path vendor'"
- "-ignore='Path upgrades'"
- "-ignore='Path systems'"
# optional: use this to switch to fswatch verbose mode
watch_args: '-v'
My Env:
{11:41}~ ➭ docker -v
Docker version 17.09.0-ce, build afdb6d4
{11:42}~ ➭ docker-sync -v
0.4.6
SO: OSx 10.11.6
Running my projects on a mounted Filesystem Case-Sensitive created using https://gist.github.com/scottsb/479bebe8b4b86bf17e2d
/dev/disk2s2 on /Users/neisantos/src (hfs, local, nodev, nosuid, journaled, noowners, nobrowse)
Don't know if it's too late.
I have almost the same setup:
OSx 10.12.6
Docker version 18.06.0-ce, build 0ffa825
docker-sync v 0.5.7
My project structure looks like this
./
build
changes
docker
etc
var
some-yml-files.yml
My docker-sync.yml looks like this
version: "2"
options:
verbose: true
syncs:
my-appcode-sync: # tip: add -sync and you keep consistent names as a convention
src: '.'
# sync_strategy: 'native_osx' # not needed, this is the default now
sync_excludes: ['docker/var', 'changes', '.git', '.idea']
I took this from https://github.com/EugenMayer/docker-sync/issues/421#issuecomment-309244156
Note the src: '.'. Initially I used src: './' like you did and ignoring folders didn't work either. After removing the / it worked for me.

How do I configure SaltStack to transfer a file (or install a package) for the first time?

I am running two instances of RedHat. I have SaltMaster installed on one machine and SaltMinion installed on another. I am using a free version of Salt. I want to test SaltStack to do a basic configuration management task. If it can transfer a file from SaltMaster to SaltMinion, that would be great. If it can install Apache web server on SaltMinion, that would be great. Either task will help me learn. My learning goal is semi-flexible.
I can use salt '*' test.ping. The response is True. I tried this command: salt '*' state.apply
I got this error:
> hostname.fqdn:
> Data failed to compile:
> ----------
> No matching salt environment for environment 'qa' found
> ----------
> No matching sls found for 'qa1' in env 'qa'
> ----------
> No matching sls found for 'base1' in env 'base'
> ----------
> No matching salt environment for environment 'dev' found
> ----------
> Specified SLS base1 in saltenv dev is not available on the salt master or through a configured fileserver
I modified the /etc/salt/master file. I uncommented these lines:
fileserver_backend:
- git
- roots
I tried this command again: salt '*' state.apply
I received this error:
> [ERROR ] Error parsing configuration file: /etc/salt/master -
> expected '<document start>', but found '<block mapping start>' in
> "<string>", line 547, column 1:
> fileserver_backend:
> ^ [ERROR ] Error parsing configuration file: /etc/salt/master - expected '<document start>', but found '<block mapping start>' in
> "<string>", line 547, column 1:
> fileserver_backend:
> ^
I have been following these directions here:
https://docs.saltstack.com/en/latest/topics/tutorials/states_pt1.html
I created a webserver.sls file.
I inserted these lines as the content:
apache: # ID declaration
pkg: # state declaration
- installed # function declaration
I do not see how the three lines in the directions above would be enough to configure SaltStack to work. Where would the apache installation media need to be? Where would the transfer happen from? Am I supposed to download the media to SaltMaster? I would assume so. But where would I put it? I have a satellite server for yum commands to work.
Alternatively, how do I get SaltStack to transfer a file from SaltMaster to SaltMinion?
The first error ([...]No matching sls found for 'qa1' in env 'qa'[...]) indicates that you have configured a lot of different environments (file_roots), which are not present on your master's filesystem. Your approach to solve this goes in the correct direction, but leads to this error:
[ERROR ] Error parsing configuration file: /etc/salt/master - expected '', but found '' in "", line 547, column 1: fileserver_backend: ^ [ERROR ] Error parsing configuration file: /etc/salt/master - expected '', but found '' in "", line 547, column 1: fileserver_backend: ^
You should no longer be able to test.ping your minion, as the salt master should not run anymore, does it? To solve it just read the error message. It tells you with which point in your salt master configuration file salt is unhappy.
The fileserver_backend configures which types of backend should be available. You should check the file_roots configuration to actually define which roots are available. Roots refer to salt states folders in your filesystem.
A very simple config might look like that:
file_roots:
base:
- /srv/salt
It assumes that /srv/salt is the root of your state tree - which effectively means, that your webserver.sls should be located in this folder.
Your webserver.sls looks promising - it should install apache2 on a minion, when you apply it.
Managing configuration files on the master and transferring them to the minions is something salt can easily achieve. A simple state might look like:
/etc/myawesomeconfigurationfile.conf:
file.managed:
source: salt://myawesomefile # refers to /srv/salt/myawesomefile
user: root
group: root
mode: 640
You also asked for media files that you want to manage. If you talk about application related data it is not a good idea to use salt to move them around. IMO other approaches like NFS, GlusterFS or anything else that decouples user content from your application would be a better approach.

SaltStack: dockerng is not available

I'm quite new to SaltStack. I've setup a salt-master and a salt-minion (via salt-cloud on my ESXi). It works fine so far. However, I cannot get dockerng to run any function on my minion. It always returns 'dockerng.xxxx' is not available:
# salt '*' test.ping
minion1:
True
$ salt '*' dockerng.version
minion1:
'dockerng.version' is not available.
However, When I call the same with salt-call directly on the minion:
$ salt-call dockerng.version
[INFO ] Determining pillar cache
local:
----------
ApiVersion:
1.23
Any hints/ideas?
Have you installed the python docker module on the minion itself? That's a requirement.
I just encountered exactly the same situation. Installing 'docker-py' on the salt-master worked for me. Of course, as suggested by Utah_Dave, docker-py would also be needed on any minion that would be targeted by dockerng.
I encountered this problem while using an image with docker pre-installed. The solution that works for me is to restart the salt-minion daemon:
salt-minion:
pkg:
- installed
- name: salt-minion
service.running:
- enable: True
- require:
- pkg: salt-minion
- service: docker
- pip: docker-py
- watch:
- pip: docker-py
taken from http://humankeyboard.com/saltstack/2013/how-to-restart-salt-minion.html
Unfortunately, the dockerng module doesn't work until the second run from the master. I'm still playing with watch and reload_modules trying to get this to work.
https://docs.saltstack.com/en/latest/ref/states/index.html#reloading-modules

Homebrew: Can't start elastic search

I'm in a big trouble, I can't start Elasticsearch and I need it for run my rails locally, please tell me what's going on. I installed Elasticsearch in the normal fashion then I did the following:
elasticsearch --config=/usr/local/opt/elasticsearch/config/elasticsearch.yml
But it shows the following error: [2015-11-01 20:36:50,574][INFO ][bootstrap] es.config is no longer supported. elasticsearch.yml must be placed in the config directory and cannot be renamed.
I tried several alternative ways of run it, like:
elasticsearch -f -D
But then I get the following error, and I can't find any useful for solve it, it seems to be related with file perms but not sure:
java.io.IOException: Resource not found: "org/joda/time/tz/data/ZoneInfoMap" ClassLoader: sun.misc.Launcher$AppClassLoader#33909752
at org.joda.time.tz.ZoneInfoProvider.openResource(ZoneInfoProvider.java:210)
at org.joda.time.tz.ZoneInfoProvider.<init>(ZoneInfoProvider.java:127)
at org.joda.time.tz.ZoneInfoProvider.<init>(ZoneInfoProvider.java:86)
at org.joda.time.DateTimeZone.getDefaultProvider(DateTimeZone.java:514)
at org.joda.time.DateTimeZone.getProvider(DateTimeZone.java:413)
at org.joda.time.DateTimeZone.forID(DateTimeZone.java:216)
at org.joda.time.DateTimeZone.getDefault(DateTimeZone.java:151)
at org.joda.time.chrono.ISOChronology.getInstance(ISOChronology.java:79)
at org.joda.time.DateTimeUtils.getChronology(DateTimeUtils.java:266)
at org.joda.time.format.DateTimeFormatter.selectChronology(DateTimeFormatter.java:968)
at org.joda.time.format.DateTimeFormatter.printTo(DateTimeFormatter.java:672)
at org.joda.time.format.DateTimeFormatter.printTo(DateTimeFormatter.java:560)
at org.joda.time.format.DateTimeFormatter.print(DateTimeFormatter.java:644)
at org.elasticsearch.Build.<clinit>(Build.java:51)
at org.elasticsearch.node.Node.<init>(Node.java:135)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:145)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:170)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
[2015-11-01 20:40:57,602][INFO ][node ] [Centurius] version[2.0.0], pid[22063], build[de54438/2015-10-22T08:09:48Z]
[2015-11-01 20:40:57,605][INFO ][node ] [Centurius] initializing ...
Exception in thread "main" java.lang.IllegalStateException: failed to load bundle [] due to jar hell
Likely root cause: java.security.AccessControlException: access denied ("java.io.FilePermission" "/usr/local/Cellar/elasticsearch/2.0.0/libexec/antlr-runtime-3.5.jar" "read")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
at java.security.AccessController.checkPermission(AccessController.java:884)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
at java.lang.SecurityManager.checkRead(SecurityManager.java:888)
at java.util.zip.ZipFile.<init>(ZipFile.java:210)
at java.util.zip.ZipFile.<init>(ZipFile.java:149)
at java.util.jar.JarFile.<init>(JarFile.java:166)
at java.util.jar.JarFile.<init>(JarFile.java:103)
at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:173)
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:340)
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:113)
at org.elasticsearch.node.Node.<init>(Node.java:144)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:145)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:170)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details.
Thanks for your help.
There are some changes with libexec with Elasticsearch/homebrew installation and that is why it is failing to start. There is a PR #45644 currently being worked on. Till the PR gets accepted, you can use the same formula to fix the installation of Elasticsearch.
First uninstall the earlier/older version. Then edit the formula of Elasticsearch:
$ brew edit elasticsearch
And use the formula from the PR.
Then do brew install elasticsearch, it should work fine.
To start Elasticsearch, just do:
$ elasticsearch
config option is no longer valid. For custom config, use path.config:
$ elasticsearch --path.conf=/usr/local/opt/elasticsearch/config

Resources