Related
I want to display nginx logs on kibana.
Elastic search and kibana are running fine.
Nginx logs are stored in /var/log/nginx/*.log
I installed filebeat and enbled the nginx service with it.
filebeat.yml
###################### Filebeat Configuration Example #########################
# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html
# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.
# ============================== Filebeat inputs ===============================
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
# filestream is an input for collecting log messages from files.
- type: filestream
# Change to true to enable this input configuration.
enabled: false
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /var/log/*.log
#- c:\programdata\elasticsearch\logs\*
# Exclude lines. A list of regular expressions to match. It drops the lines that are
# matching any regular expression from the list.
#exclude_lines: ['^DBG']
# Include lines. A list of regular expressions to match. It exports the lines that are
# matching any regular expression from the list.
#include_lines: ['^ERR', '^WARN']
# Exclude files. A list of regular expressions to match. Filebeat drops the files that
# are matching any regular expression from the list. By default, no files are dropped.
#prospector.scanner.exclude_files: ['.gz$']
# Optional additional fields. These fields can be freely picked
# to add additional information to the crawled log files for filtering
#fields:
# level: debug
# review: 1
# ============================== Filebeat modules ==============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
# ======================= Elasticsearch template setting =======================
setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false
# ================================== General ===================================
# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:
# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]
# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging
# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false
# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:
# =================================== Kibana ===================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: http://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
host: "localhost:3000"
# Kibana Space ID
# ID of the Kibana Space into which the dashboards should be loaded. By default,
# the Default Space will be used.
#space.id:
# =============================== Elastic Cloud ================================
# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).
# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:
###################### Filebeat Configuration Example #########################
# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html
# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.
# ============================== Filebeat inputs ===============================
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
# filestream is an input for collecting log messages from files.
- type: filestream
# Change to true to enable this input configuration.
enabled: false
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /var/log/*.log
#- c:\programdata\elasticsearch\logs\*
# Exclude lines. A list of regular expressions to match. It drops the lines that are
# matching any regular expression from the list.
#exclude_lines: ['^DBG']
# Include lines. A list of regular expressions to match. It exports the lines that are
# matching any regular expression from the list.
#include_lines: ['^ERR', '^WARN']
# Exclude files. A list of regular expressions to match. Filebeat drops the files that
# are matching any regular expression from the list. By default, no files are dropped.
#prospector.scanner.exclude_files: ['.gz$']
# Optional additional fields. These fields can be freely picked
# to add additional information to the crawled log files for filtering
#fields:
# level: debug
# review: 1
# ============================== Filebeat modules ==============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
# ======================= Elasticsearch template setting =======================
setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false
# ================================== General ===================================
# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:
# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]
# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging
# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false
# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:
# =================================== Kibana ===================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: http://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
host: "localhost:3000"
# Kibana Space ID
# ID of the Kibana Space into which the dashboards should be loaded. By default,
# the Default Space will be used.
#space.id:
# =============================== Elastic Cloud ================================
# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).
# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:
# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:
# ================================== Outputs ===================================
# Configure what output to use when sending the data collected by the beat.
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]
# Protocol - either `http` (default) or `https`.
#protocol: "https"
# Authentication credentials - either API key or username/password.
#api_key: "id:api_key"
#username: "temp"
#password: "temp#1234"
# ------------------------------ Logstash Output -------------------------------
#output.logstash:
# The Logstash hosts
#hosts: ["localhost:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
# ================================= Processors =================================
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
# ================================== Logging ===================================
# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug
# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
#logging.selectors: ["*"]
# ============================= X-Pack Monitoring ==============================
# Filebeat can export internal metrics to a central Elasticsearch monitoring
# cluster. This requires xpack monitoring to be enabled in Elasticsearch. The
# reporting is disabled by default.
# Set to true to enable the monitoring reporter.
#monitoring.enabled: false
# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Filebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:
# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:
# ============================== Instrumentation ===============================
# Instrumentation support for the filebeat.
#instrumentation:
# Set to true to enable instrumentation of filebeat.
#enabled: false
# Environment in which filebeat is running on (eg: staging, production, etc.)
#environment: ""
# APM Server hosts to report instrumentation results to.
#hosts:
# - http://localhost:8200
# API Key for the APM Server(s).
# If api_key is set then secret_token will be ignored.
#api_key:
# Secret token for the APM Server(s).
#secret_token:
# ================================= Migration ==================================
# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true
And /etc/filebeat/modules/nginx.yml file
# Module: nginx
# Module: nginx
access:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["var/logs/nginx/*access.log"]
# Error logs
error:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["var/logs/nginx/*error.log"]
# Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
ingress_controller:
enabled: false
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
after i run filebeat setup -e command I got the following output
2021-12-18T19:19:44.352+0530 INFO cfgfile/reload.go:262 Loading of config files completed.
2021-12-18T19:19:44.352+0530 INFO [load] cfgfile/list.go:129 Stopping 2 runners ...
Loaded Ingest pipelines
After this when i run systemctl restart filebeat
I got the following error
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Sat 2021-12-18 19:03:53 IST; 17min ago
Docs: https://www.elastic.co/beats/filebeat
Main PID: 49606 (code=exited, status=1/FAILURE)
Dec 18 19:03:53 ip-10-249-5-178.ap-south-1.compute.internal systemd[1]: Unit filebeat.service entered failed state.
Dec 18 19:03:53 ip-10-249-5-178.ap-south-1.compute.internal systemd[1]: filebeat.service failed.
Dec 18 19:03:53 ip-10-249-5-178.ap-south-1.compute.internal systemd[1]: filebeat.service holdoff time over, scheduling restart.
Dec 18 19:03:53 ip-10-249-5-178.ap-south-1.compute.internal systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
Dec 18 19:03:53 ip-10-249-5-178.ap-south-1.compute.internal systemd[1]: start request repeated too quickly for filebeat.service
Dec 18 19:03:53 ip-10-249-5-178.ap-south-1.compute.internal systemd[1]: Failed to start Filebeat sends log files to Logstash or directly to Elasticsearch..
Dec 18 19:03:53 ip-10-249-5-178.ap-south-1.compute.internal systemd[1]: Unit filebeat.service entered failed state.
Dec 18 19:03:53 ip-10-249-5-178.ap-south-1.compute.internal systemd[1]: filebeat.service failed.
Any help ??
Solved by deleting the file
rm -r /var/lib/filebeat/registry
Don't know the reason behind it but after this the service started successfully.
I'm trying to send logs from Winlogbeat to my ELK stack.
I installed my ELK stack with docker and configured TLS on it.
I did everything according to the official guide and it worked for my host.
However, when copying the same winlogbeat directory to my Event Collector server, it did not work (all files are the same including the yml file).
When trying to run the "winlogbeat.exe setup -e" I got the following error: 'error connecting to elasticsearch at "https://elastic-host:9200" Get "https://elastic-host:9200" Winlogbeat setup error: x509 certificate is valid for elastic-host ip, not elastic-host ip' (same ips). The CA is already added to the trusted root certificates. Everything is configured the same as on the host, on the host it works, on the server it doesn't. (the ELK server and the EVC are in the same segment so there shouldn't be any firewall drops)
My .yml (same file on host and EVC server):
on the host it works without the ssl as well and the traffic is still encrypted due to the TLS that I configured on the docker cluster. So I'm not sure the ssl configuration is needed (but I wanted to include them in case they are needed).
# This file is an example configuration file highlighting only the most common
# options. The winlogbeat.reference.yml file from the same directory contains
# all the supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/winlogbeat/index.html
# ======================== Winlogbeat specific options =========================
# event_logs specifies a list of event logs to monitor as well as any
# accompanying options. The YAML data type of event_logs is a list of
# dictionaries.
#
# The supported keys are name (required), tags, fields, fields_under_root,
# forwarded, ignore_older, level, event_id, provider, and include_xml. Please
# visit the documentation for the complete details of each option.
# https://go.es.io/WinlogbeatConfig
winlogbeat.event_logs:
- name: Application
ignore_older: 72h
- name: System
- name: Security
processors:
- script:
lang: javascript
id: security
file: ${path.home}/module/security/config/winlogbeat-security.js
- name: Microsoft-Windows-Sysmon/Operational
processors:
- script:
lang: javascript
id: sysmon
file: ${path.home}/module/sysmon/config/winlogbeat-sysmon.js
- name: Windows PowerShell
event_id: 400, 403, 600, 800
processors:
- script:
lang: javascript
id: powershell
file: ${path.home}/module/powershell/config/winlogbeat-powershell.js
- name: Microsoft-Windows-PowerShell/Operational
event_id: 4103, 4104, 4105, 4106
processors:
- script:
lang: javascript
id: powershell
file: ${path.home}/module/powershell/config/winlogbeat-powershell.js
- name: ForwardedEvents
tags: [forwarded]
processors:
- script:
when.equals.winlog.channel: Security
lang: javascript
id: security
file: ${path.home}/module/security/config/winlogbeat-security.js
- script:
when.equals.winlog.channel: Microsoft-Windows-Sysmon/Operational
lang: javascript
id: sysmon
file: ${path.home}/module/sysmon/config/winlogbeat-sysmon.js
- script:
when.equals.winlog.channel: Windows PowerShell
lang: javascript
id: powershell
file: ${path.home}/module/powershell/config/winlogbeat-powershell.js
- script:
when.equals.winlog.channel: Microsoft-Windows-PowerShell/Operational
lang: javascript
id: powershell
file: ${path.home}/module/powershell/config/winlogbeat-powershell.js
# ====================== Elasticsearch template settings =======================
setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false
# ================================== General ===================================
# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:
# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]
# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging
# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false
# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:
# =================================== Kibana ===================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: http://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
host: "192.168.101.129:5601"
protocol: https
username: "elastic"
password: "passwd"
setup.kibana.ssl.enabled: true
setup.kibana.ssl.certificate_authorities: ["C:\\Program Files\\Winlogbeat\\ca.crt"]
setup.kibana.ssl.certificate: "C:\\Program Files\\Winlogbeat\\winlogbeat.crt"
setup.kibana.ssl.key: "C:\\Program Files\\Winlogbeat\\winlogbeat.key"
# Kibana Space ID
# ID of the Kibana Space into which the dashboards should be loaded. By default,
# the Default Space will be used.
#space.id:
# =============================== Elastic Cloud ================================
# These settings simplify using Winlogbeat with the Elastic Cloud (https://cloud.elastic.co/).
# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:
# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:
# ================================== Outputs ===================================
# Configure what output to use when sending the data collected by the beat.
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["192.168.101.129:9200"]
username: "elastic"
password: "passwd"
# Protocol - either `http` (default) or `https`.
protocol: "https"
output.elasticsearch.ssl.certificate_authorities: ["C:\\Program Files\\Winlogbeat\\ca.crt"]
output.elasticsearch.ssl.certificate: "C:\\Program Files\\Winlogbeat\\winlogbeat.crt"
output.elasticsearch.ssl.key: "C:\\Program Files\\Winlogbeat\\winlogbeat.key"
# Authentication credentials - either API key or username/password.
#api_key: "id:api_key"
# ------------------------------ Logstash Output -------------------------------
#output.logstash:
# The Logstash hosts
#hosts: ["localhost:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
# ================================= Processors =================================
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
# ================================== Logging ===================================
# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug
# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]
# ============================= X-Pack Monitoring ==============================
# Winlogbeat can export internal metrics to a central Elasticsearch monitoring
# cluster. This requires xpack monitoring to be enabled in Elasticsearch. The
# reporting is disabled by default.
# Set to true to enable the monitoring reporter.
#monitoring.enabled: false
# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Winlogbeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:
# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:
# ============================== Instrumentation ===============================
# Instrumentation support for the winlogbeat.
#instrumentation:
# Set to true to enable instrumentation of winlogbeat.
#enabled: false
# Environment in which winlogbeat is running on (eg: staging, production, etc.)
#environment: ""
# APM Server hosts to report instrumentation results to.
#hosts:
# - http://localhost:8200
# API Key for the APM Server(s).
# If api_key is set then secret_token will be ignored.
#api_key:
# Secret token for the APM Server(s).
#secret_token:
# ================================= Migration ==================================
# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true
In your output, you need to specify ssl.verification_mode: certificate.
For your example, it looks like it is the Kibana output that has a certificate specified on it:
setup.kibana.ssl.enabled: true
setup.kibana.ssl.certificate_authorities: ["C:\\Program Files\\Winlogbeat\\ca.crt"]
setup.kibana.ssl.certificate: "C:\\Program Files\\Winlogbeat\\winlogbeat.crt"
setup.kibana.ssl.key: "C:\\Program Files\\Winlogbeat\\winlogbeat.key"
setup.kibana.ssl.verification_mode: certificate
Older versions of winlogbeat will need ssl.verification_mode: none instead.
See SSL/TLS configuration documentation at https://www.elastic.co/guide/en/beats/winlogbeat/current/configuration-ssl.html
I'm trying to build a IoT home network using MQTT but have fallen at the first hurdle. I'm going round in circies!
I'm running mosquitto on a Pi, but can't connect to it from an Arduino. I can connect from MQTT.fx running on a windows laptop though on the same LAN.
Arduino code below: I've run this with the test.mosquitto.org server address (85.119.83.194) and the local address of the Pi (192.168.1.200). Mosquitto connects but it fails to connect directly to the Pi.
Is the issue with my router??? Any suggestions really welcome!!!!
#include <SPI.h>
#include <Ethernet.h>
#include <PubSubClient.h>
// Update these with values suitable for your network.
byte mac[] = { 0xDE, 0xED, 0xBA, 0xFE, 0xFE, 0xED };
byte server[] = { 192, 168, 1, 200};
byte ip[] = { 192, 168, 1, 61 };
void callback(char* topic, byte* payload, unsigned int length) {
// handle message arrived
Serial.println("Message received");
}
EthernetClient ethClient;
PubSubClient client(server, 1883, callback, ethClient);
void setup()
{
Ethernet.begin(mac, ip);
Serial.begin(9600);
Serial.println("Ethernet Begin");
if (client.connect("arduinoClient")) {
Serial.println("Client connected");
client.subscribe("/notification/turnlighton");
}
else{
Serial.println("Client not connected");
}
}
void loop()
{
client.loop();
}
Conf file:
# Config file for mosquitto
#
# See mosquitto.conf(5) for more information.
#
# Default values are shown, uncomment to change.
#
# Use the # character to indicate a comment, but only if it is the
# very first character on the line.
# =================================================================
# General configuration
# =================================================================
# Time in seconds to wait before resending an outgoing QoS=1 or
# QoS=2 message.
#retry_interval 20
# Time in seconds between updates of the $SYS tree.
#sys_interval 10
# Time in seconds between cleaning the internal message store of
# unreferenced messages. Lower values will result in lower memory
# usage but more processor time, higher values will have the
# opposite effect.
# Setting a value of 0 means the unreferenced messages will be
# disposed of as quickly as possible.
#store_clean_interval 10
# Write process id to a file. Default is a blank string which means
# a pid file shouldn't be written.
# This should be set to /var/run/mosquitto.pid if mosquitto is
# being run automatically on boot with an init script and
# start-stop-daemon or similar.
#pid_file
# When run as root, drop privileges to this user and its primary
# group.
# Leave blank to stay as root, but this is not recommended.
# If run as a non-root user, this setting has no effect.
# Note that on Windows this has no effect and so mosquitto should
# be started by the user you wish it to run as.
#user mosquitto
# The maximum number of QoS 1 and 2 messages currently inflight per
# client.
# This includes messages that are partway through handshakes and
# those that are being retried. Defaults to 10. Set to 0 for no
# maximum. Setting to 1 will guarantee in-order delivery of QoS 1
# and 2 messages.
#max_inflight_messages 10
# The maximum number of QoS 1 and 2 messages to hold in a queue
# above those that are currently in-flight. Defaults to 100. Set
# to 0 for no maximum (not recommended).
#max_queued_messages 100
# =================================================================
# Default listener
# =================================================================
# IP address/hostname to bind the default listener to. If not
# given, the default listener will not be bound to a specific
# address and so will be accessible to all network interfaces.
# bind_address ip-address/host name
#bind_address
# Port to use for the default listener.
#port 1883
# The maximum number of client connections to allow. This is
# a per listener setting.
# Default is -1, which means unlimited connections.
# Note that other process limits mean that unlimited connections
# are not really possible. Typically the default maximum number of
# connections possible is around 1024.
#max_connections -1
# =================================================================
# Extra listeners
# =================================================================
# Listen on a port/ip address combination. By using this variable
# multiple times, mosquitto can listen on more than one port. If
# this variable is used and neither bind_address nor port given,
# then the default listener will not be started.
# The port number to listen on must be given. Optionally, an ip
# address or host name may be supplied as a second argument. In
# this case, mosquitto will attempt to bind the listener to that
# address and so restrict access to the associated network and
# interface. By default, mosquitto will listen on all interfaces.
# listener port-number [ip address/host name]
#listener
# The maximum number of client connections to allow. This is
# a per listener setting.
# Default is -1, which means unlimited connections.
# Note that other process limits mean that unlimited connections
# are not really possible. Typically the default maximum number of
# connections possible is around 1024.
#max_connections -1
# The listener can be restricted to operating within a topic hierarchy using
# the mount_point option. This is achieved be prefixing the mount_point string
# to all topics for any clients connected to this listener. This prefixing only
# happens internally to the broker; the client will not see the prefix.
#mount_point
# =================================================================
# Persistence
# =================================================================
# If persistence is enabled, save the in-memory database to disk
# every autosave_interval seconds. If set to 0, the persistence
# database will only be written when mosquitto exits.
# Note that writing of the persistence database can be forced by
# sending mosquitto a SIGUSR1 signal.
#autosave_interval 1800
# Save persistent message data to disk (true/false).
# This saves information about all messages, including
# subscriptions, currently in-flight messages and retained
# messages.
# retained_persistence is a synonym for this option.
persistence true
# The filename to use for the persistent database, not including
# the path.
#persistence_file mosquitto.db
# Location for persistent database. Must include trailing /
# Default is an empty string (current directory).
# Set to /var/lib/mosquitto/ if running as a proper service.
persistence_location /var/lib/mosquitto/
# =================================================================
# Logging
# =================================================================
# Places to log to. Use multiple log_dest lines for multiple
# logging destinations.
# Possible destinations are: stdout stderr syslog topic
# stdout and stderr log to the console on the named output.
# syslog uses the userspace syslog facility which usually ends up
# in /var/log/messages or similar.
# topic logs to the broker topic '$SYS/broker/log/<severity>',
# where severity is one of D, E, W, N, I which are debug, error,
# warning, notice and information.
# Use "log_dest none" if you wish to disable logging.
log_dest none
# Types of messages to log. Use multiple log_type lines for logging
# multiple types of messages.
# Possible types are: debug, error, warning, notice, information,
# none.
# Note that debug type messages are for decoding the incoming
# network packets.
# They are not logged in syslog.
#log_type error
#log_type warning
#log_type notice
#log_type information
# If set to true, client connection and disconnection messages will be included
# in the log.
#connection_messages true
# If set to true, add a timestamp value to each log message.
#log_timestamp true
# =================================================================
# Security
# =================================================================
# If set, only clients that have a matching prefix on their
# clientid will be allowed to connect to the broker. By default,
# all clients may connect.
# For example, setting "secure-" here would mean a client "secure-
# client" could connect but another with clientid "mqtt" couldn't.
#clientid_prefixes
# Boolean value that determines whether clients that connect
# without providing a username are allowed to connect. If set to
# false then a password file should be created (see the
# password_file option) to control authenticated client access.
# Defaults to true.
#allow_anonymous true
# Control access to the broker using a password file. The file is a
# text file # of lines in the format:
# username:password
# The password (and colon) may be omitted if desired, although this
# offers very little in the way of security.
#password_file
# Control access to topics on the broker using an access control list
# file. If this parameter is defined then only the topics listed will
# have access.
# Topic access is added with lines of the format:
#
# topic [read|write] <topic>
#
# The access type is controlled using "read" or "write". This parameter
# is optional - if not given then the access is read/write.
# <topic> can contain the + or # wildcards as in subscriptions.
#
# The first set of topics are applied to anonymous clients, assuming
# allow_anonymous is true. User specific topic ACLs are added after a
# user line as follows:
#
# user <username>
#
# The username referred to here is the same as in password_file. It is
# not the clientid.
#
#
# If is also possible to define ACLs based on pattern substitution within the
# topic. The patterns available for substition are:
#
# %c to match the client id of the client
# %u to match the username of the client
#
# The substitution pattern must be the only text for that level of hierarchy.
#
# The form is the same as for the topic keyword, but using pattern as the
# keyword.
# Pattern ACLs apply to all users even if the "user" keyword has previously
# been given.
#
# pattern [read|write] <topic>
#
# Example:
#
# pattern write sensor/%u/data
#
#acl_file
# =================================================================
# Bridges
# =================================================================
# Experimental support for connecting multiple MQTT brokers
# together.
# Specify multiple sets of connection, address and topic
# configurations.
# Each connection must have a unique name.
# Only a single address per configuration is currently supported,
# unlike in rsmb.
# The direction that the topic will be shared can be chosen by
# specifying out, in or both, where the default value is out.
# The QoS level of the bridged communication can be specified with the final
# topic option. The default QoS level is 2, to change the QoS the topic
# direction must also be given.
# Multiple topics can be specified per connection, but be careful
# not to create any loops.
#connection <name>
#address <host>[:<port>]
#topic <topic> [[out | in | both] qos-level]
# Set the client id for this bridge connection. If not defined,
# this defaults to 'name.hostname' where name is the connection
# name and hostname is the hostname of this computer.
#clientid
# Set the clean session variable for this bridge.
# When set to true, when the bridge disconnects for any reason, all
# messages and subscriptions will be cleaned up on the remote
# broker. Note that with cleansession set to true, there may be a
# significant amount of retained messages sent when the bridge
# reconnects after losing its connection.
# When set to false, the subscriptions and messages are kept on the
# remote broker, and delivered when the bridge reconnects.
#cleansession false
# If set to true, publish notification messages to the local and remote brokers
# giving information about the state of the bridge connection. Retained
# messages are published to the topic $SYS/bridge/connection/<clientid>/state.
# If the message is 1 then the connection is active, or 0 if the connection has
# failed.
#notifications true
# Set the keepalive interval for this bridge connection, in
# seconds.
#keepalive_interval 60
# Set the start type of the bridge. This controls how the bridge starts and
# can be one of three types: automatic, lazy and once. Note that RSMB provides
# a fourth start type "manual" which isn't currently supported by mosquitto.
#
# "automatic" is the default start type and means that the bridge connection
# will be started automatically when the broker starts and also restarted
# after a short delay (30 seconds) if the connection fails.
#
# Bridges using the "lazy" start type will be started automatically when the
# number of queued messages exceeds the number set with the "threshold"
# parameter. It will be stopped automatically after the time set by the
# "idle_timeout" parameter. Use this start type if you wish the connection to
# only be active when it is needed.
#
# A bridge using the "once" start type will be started automatically when the
# broker starts but will not be restarted if the connection fails.
#start_type automatic
# Set the amount of time a bridge using the lazy start type must be idle before
# it will be stopped. Defaults to 60 seconds.
#idle_timeout 60
# Set the number of messages that need to be queued for a bridge with lazy
# start type to be restarted. Defaults to 10 messages.
# Must be less than max_queued_messages.
#threshold 10
# Set the username to use when connecting to an MQTT v3.1 broker
# that requires authentication.
#username
# Set the password to use when connecting to an MQTT v3.1 broker
# that requires authentication. This option is only used if
# username is also set.
#password
# =================================================================
# External security checks
# =================================================================
# This section defines options for potential use with external
# databases.
# They are intended to make it easier to add external security
# checks along with WITH_EXT_SECURITY_CHECKS. If you haven't written
# code to support another database and compiled support in, you are
# unlikely to need to change anything here.
# Database hostname.
#db_host
# Database port.
#db_port
# Database name.
#db_name
# Database username.
#db_username
# Database password.
#db_password
# =================================================================
# Unsupported rsmb options - for the future
# =================================================================
#autosave_on_changes
#addresses
#notification_topic
#round_robin
# =================================================================
# rsmb options - unlikely to ever be supported
# =================================================================
#ffdc_output
#max_log_entries
#trace_level
#trace_output
#try_private
I'm using the Elasti Cloud as it appears to be the most suitable for quickly setting up application logging. I have 24 docker container running in different nodes, and some containers have no of replicas also. i want to export inside docker container logs to elk stack.. I don't want to install Filebeat on each of my containers because that seems like it goes directly against Docker's separation of duties mantra.
.... how do I get logs from my application containers to log stash server
You can send your syslog to Logstash by configuring rsyslogd like this
# /etc/rsyslog.d/99-ship-syslog.conf
*.*;syslog;auth,authpriv.none action(
type="omfwd"
Target="myremote.elk-server.net"
Port="5001"
Protocol="udp"
)
If you don't have rsyslog running yet, you can add it like so (alpine linux example):
# Dockerfile
FROM alpine:3.7
RUN apk update \
&& apk add rsyslog
COPY rsyslog.conf /etc/rsyslog.conf
EXPOSE 514 514/udp
VOLUME [ "/var/log", "/etc/rsyslog.d" ]
ENTRYPOINT [ "rsyslogd", "-n" ]
--
# rsyslogd.conf
#
# if you experience problems, check:
# http://www.rsyslog.com/troubleshoot
#### MODULES ####
module(load="imuxsock") # local system logging support (e.g. via logger command)
#module(load="imklog") # kernel logging support (previously done by rklogd)
module(load="immark") # --MARK-- message support
module(load="imudp") # UDP listener support
input(type="imudp" port="514")
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* action(type="omfile" file="/dev/console")
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none action(type="omfile" file="/var/log/messages")
# The authpriv file has restricted access.
authpriv.* action(type="omfile" file="/var/log/secure")
# Log all the mail messages in one place.
mail.* action(type="omfile" file="/var/log/maillog")
# Log cron stuff
cron.* action(type="omfile" file="/var/log/cron")
# Everybody gets emergency messages
*.emerg action(type="omusrmsg" users="*")
# Save news errors of level crit and higher in a special file.
uucp,news.crit action(type="omfile" file="/var/log/spooler")
# Save boot messages also to boot.log
local7.* action(type="omfile" file="/var/log/boot.log")
# log every host in its own directory
if $fromhost-ip then /var/log/$fromhost-ip/messages
# Include all .conf files in /etc/rsyslog.d
$IncludeConfig /etc/rsyslog.d/*.conf
$template GRAYLOGRFC5424,"<%PRI%>%PROTOCOL-VERSION% %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n"
*.info;mail.none;authpriv.none;cron.none;*.* ##graylog:514;GRAYLOGRFC5424 # forward everything to remote server
As you're running within a java-application, you can even send you logs directly to syslog. Here's a small configuration example with log4j
log4j.rootLogger=INFO, SYSLOG
log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender
log4j.appender.SYSLOG.syslogHost=myremote.elk-server.net
log4j.appender.SYSLOG.layout=org.apache.log4j.PatternLayout
log4j.appender.SYSLOG.layout.conversionPattern=%d{ISO8601} %-5p [%t] %c{2} %x - %m%n
log4j.appender.SYSLOG.Facility=LOCAL1
I have been having issues trying to deploy with rubber
in the Terminal
rubber vulcanize complete_passenger_postgresql
and in my rubber.yml
# REQUIRED: The name of your application
app_name: app-name
# REQUIRED: The system user to run your app servers as
app_user: app
# REQUIRED: Notification emails (e.g. monit) get sent to this address
#
admin_email: "root##{full_host}"
# OPTIONAL: If not set, you won't be able to access web_tools
# server (graphite, graylog, monit status, haproxy status, etc)
# web_tools_user: admin
# web_tools_password: sekret
# REQUIRED: The timezone the server should be in
timezone: US/Eastern
# REQUIRED: the domain all the instances should be associated with
#
domain: foo.com
# OPTIONAL: See rubber-dns.yml for dns configuration
# This lets rubber update a dynamic dns service with the instance alias
# and ip when they are created. It also allows setting up arbitrary
# dns records (CNAME, MX, Round Robin DNS, etc)
# OPTIONAL: Additional rubber file to pull config from if it exists. This file will
# also be pushed to remote host at Rubber.root/config/rubber/rubber-secret.yml
#
rubber_secret: "#{File.expand_path('~') + '/.ec2' + (Rubber.env == 'production' ? '' : '_dev') + '/rubber-secret.yml' rescue 'rubber-secret.yml'}"
# OPTIONAL: Encryption key that was used to obfuscate the contents of rubber-secret.yml with "rubber util:obfuscation"
# Not that much better when stored in here, but you could use a ruby snippet in here to fetch it from a key server or something
#
# rubber_secret_key: "XXXyyy=="
# REQUIRED All known cloud providers with the settings needed to configure them
# There's only one working cloud provider right now - Amazon Web Services
# To implement another, clone lib/rubber/cloud/aws.rb or make the fog provider
# work in a generic fashion
#
cloud_providers:
aws:
# REQUIRED The AWS region that you want to use.
#
# Options include
# ap-northeast-1 # Asia Pacific (Tokyo) Region
# ap-southeast-1 # Asia Pacific (Singapore) Region
# ap-southeast-2 # Asia Pacific (Sydney) Region
# eu-west-1 # EU (Ireland) Region
# sa-east-1 # South America (Sao Paulo) Region
# us-east-1 # US East (Northern Virginia) Region
# us-west-1 # US West (Northern California) Region
# us-west-2 # US West (Oregon) Region
#
region: us-east-1
# REQUIRED The amazon keys and account ID (digits only, no dashes) used to access the AWS API
#
access_key: XXX
secret_access_key: YYY
account: ZZZ #entered in
# REQUIRED: The name of the amazon keypair and location of its private key
#
# NOTE: for some reason Capistrano requires you to have both the public and
# the private key in the same folder, the public key should have the
# extension ".pub". The easiest way to get your hand on this is to create the
# public key from the private key: ssh-keygen -y -f gsg-keypair > gsg-keypair.pub
#
key_name: guy
key_file: "#{Dir[(File.expand_path('~') rescue '/root') + '/.ec2/*' + cloud_providers.aws.key_name].first}"
# OPTIONAL: Needed for bundling a running instance using rubber:bundle
#
# pk_file: "#{Dir[(File.expand_path('~') rescue '/root') + '/.ec2/pk-*'].first}"
# cert_file: "#{Dir[(File.expand_path('~') rescue '/root') + '/.ec2/cert-*'].first}"
# image_bucket: "#{app_name}-images"
# OPTIONAL: Needed for backing up database to s3
# backup_bucket: "#{app_name}-backups"
# REQUIRED: the ami and instance type for creating instances
# The Ubuntu images at http://alestic.com/ work well
# Ubuntu 14.04.1 Trusty instance-store 64-bit: ami-92f569fa
#
# m1.small or m1.large or m1.xlarge
image_type: m3.medium
image_id: ami-1ecae776
# OPTIONAL: Provide fog-specific options directly. This should only be used if you need a special setting that
# Rubber does not directly expose. Since these settings will be passed directly through to fog, we can't make any
# guarantee about how they work (if fog renames an attribute, e.g., your config will break). Please see the fog
# source code for the option names.
# fog_options:
# EBS I/O optimized instance
# EBS-optimized instances deliver dedicated throughput between Amazon EC2 and Amazon EBS, with options
# between 500 Mbps and 1000 Mbps depending on the instance type used.
# Read more and make sure that your image_type supports ebs_optimized function at: http://aws.amazon.com/ec2/instance-types/
# ebs_optimized: false
# OPTIONAL: EC2 spot instance request support.
#
# Enables the creation of spot instance requests. Rubber will wait synchronously until the request is fulfilled,
# at which point it will begin initializing the instance, unless spot_instance_request_timeout is set.
# spot_instance: true
#
# The maximum price you would like to pay for your spot instance.
# spot_price: "0.085"
#
# If a spot instance request can't be fulfilled in 3 minutes, fallback to on-demand instance creation. If not set,
# the default is infinite.
# spot_instance_request_timeout: 180
# digital_ocean:
# REQUIRED: The Digital Ocean region that you want to use.
#
# Options include
# New York 1
# Amsterdam 1
# San Francisco 1
# New York 2
# Amsterdam 2
# Singapore 1
#
# These change often. Check https://www.digitalocean.com/droplets/new for the most up to date options.
# Default to New York 2 since this is the only region that currently supports private networking
# region: New York 2
# REQUIRED: The image name and type for creating instances.
# image_id: Ubuntu 14.04 x64
# image_type: 512MB
# Optionally enable private networking for your instances.
# This is currently only supported in New York 2.
# private_networking: true
# Use an alternate cloud provider supported by fog. This doesn't fully work
# yet due to differences in providers within fog, but gives you a starting
# point for contributing a new provider to rubber. See rubber/lib/rubber/cloud(.rb)
# fog:
# credentials:
# provider: rackspace
# rackspace_api_key: 'XXX'
# rackspace_username: 'YYY'
# image_type: 123
# image_id: 123
# REQUIRED the cloud provider to use
#
cloud_provider: aws
# OPTIONAL: Where to store instance data.
#
# Allowed forms are:
# filesystem: "file:#{Rubber.root}/config/rubber/instance-#{Rubber.env}.yml"
# cloud storage (s3): "storage:#{cloud_providers.aws.backup_bucket}/RubberInstances_#{app_name}/instance-#{Rubber.env}.yml"
# cloud table (simpledb): "table:RubberInstances_#{app_name}_#{Rubber.env}"
#
# If you need to port between forms, load the rails console then:
# Rubber.instances.save(location)
# where location is one of the allowed forms for this variable
#
# instance_storage: "file:#{Rubber.root}/config/rubber/instance-#{Rubber.env}.yml"
# OPTIONAL: Where to store a backup of the instance data
#
# This is most useful when using a remote store in case you end up
# wiping the single copy of your instance data. When using the file
# store, the instance file is typically under version control with
# your project code, so that provides some safety.
#
# instance_storage_backup: "storage:#{cloud_providers.aws.backup_bucket}/RubberInstances_#{app_name}/instance-#{Rubber.env}-#{Time.now.strftime('%Y%m%d-%H%M%S')}.yml"
# OPTIONAL: Default ports for security groups
web_port: 80
web_ssl_port: 443
web_tools_port: 8080
web_tools_ssl_port: 8443
# OPTIONAL: Define security groups
# Each security group is a name associated with a sequence of maps where the
# keys are the parameters to the ec2 AuthorizeSecurityGroupIngress API
# source_security_group_name, source_security_group_owner_id
# ip_protocol, from_port, to_port, cidr_ip
# If you want to use a source_group outside of this project, add "external_group: true"
# to prevent group_isolation from mangling its name, e.g. to give access to graphite
# server to other projects
#
# security_groups:
# graphite_server:
# description: The graphite_server security group to allow projects to send graphite data
# rules:
# - source_group_name: yourappname_production_collectd
# source_group_account: 123456
# external_group: true
# protocol: tcp
# from_port: "#{graphite_server_port}"
# to_port: "#{graphite_server_port}"
#
security_groups:
default:
description: The default security group
rules:
- source_group_name: default
source_group_account: "#{cloud_providers.aws.account}"
- protocol: tcp
from_port: 22
to_port: 22
source_ips: [0.0.0.0/0]
web:
description: "To open up port #{web_port}/#{web_ssl_port} for http server on web role"
rules:
- protocol: tcp
from_port: "#{web_port}"
to_port: "#{web_port}"
source_ips: [0.0.0.0/0]
- protocol: tcp
from_port: "#{web_ssl_port}"
to_port: "#{web_ssl_port}"
source_ips: [0.0.0.0/0]
web_tools:
description: "To open up port #{web_tools_port}/#{web_tools_ssl_port} for internal/tools http server"
rules:
- protocol: tcp
from_port: "#{web_tools_port}"
to_port: "#{web_tools_port}"
source_ips: [0.0.0.0/0]
- protocol: tcp
from_port: "#{web_tools_ssl_port}"
to_port: "#{web_tools_ssl_port}"
source_ips: [0.0.0.0/0]
# OPTIONAL: The default security groups to create instances with
assigned_security_groups: [default]
roles:
web:
assigned_security_groups: [web]
web_tools:
assigned_security_groups: [web_tools]
# OPTIONAL: Automatically create security groups for each host and role
# EC2 Classic doesn't allow one to change what groups an instance belongs to after
# creation, so it's good to have some empty ones predefined. EC2 with VPC, however,
# does allow changing security groups after instance creation and allows far fewer
# security groups per instance, so you shouldn't enable this setting if using VPC.
auto_security_groups: false
# OPTIONAL: Automatically isolate security groups for each appname/environment
# by mangling their names to be appname_env_groupname
# This makes it safer to have staging and production coexist on the same EC2
# account, or even multiple apps. NB: due to the security group limits per instance
# in EC2 with VPCs, this option should only be enabled if you're using EC2 Classic.
isolate_security_groups: false
# OPTIONAL: Prompts one to sync security group rules when the ones in amazon
# differ from those in rubber
prompt_for_security_group_sync: false
# OPTIONAL: A list of CIDR address blocks that represent private networks for your cluster.
# Set this to open up wide access to hosts in your network. Consequently, setting the CIDR block
# to anything other than a private, unroutable block would be a massive security hole.
private_networks: [10.0.0.0/8]
# OPTIONAL: The packages to install on all instances
# You can install a specific version of a package by using a sub-array of pkg, version
# For example, packages: [[rake, 0.7.1], irb]
packages: [postfix, build-essential, git-core, libxslt-dev, ntp]
# OPTIONAL: The package manager mirror to use for installation of primary packages (i.e., those not explicitly
# sourced from a different repository). If not specified, whatever mirror configured by your server image
# will be used.
#
# Note that Ubuntu has a special URL that can be used to auto-select the mirror based upon geoip. To use
# it, specify 'mirror://mirrors.ubuntu.com/mirrors.txt' as the value.
# package_manager_mirror: 'mirror://mirrors.ubuntu.com/mirrors.txt'
# OPTIONAL: The command used to identify your particular OS version. This will be used for configurations
# in Rubber templates that are parameterized by OS version (e.g., package lists). If not specified, Ubuntu
# will be assumed.
os_version_cmd: 'lsb_release -sr'
# OPTIONAL: gem sources to setup for rubygems
# gemsources: ["https://rubygems.org"]
# OPTIONAL: The gems to install on all instances
# You can install a specific version of a gem by using a sub-array of gem, version
# For example, gem: [[rails, 2.2.2], open4, aws-s3]
gems: [open4, aws-s3, bundler, [rubber, "#{Rubber.version}"]]
# OPTIONAL: A string prepended to shell command strings that cause multi
# statement shell commands to fail fast. You may need to comment this out
# on some platforms, but it works for me on linux/osx with a bash shell
#
stop_on_error_cmd: "function error_exit { exit 99; }; trap error_exit ERR"
# OPTIONAL: The default set of roles to use when creating a staging instance
# with "cap rubber:create_staging". By default this uses all the known roles,
# excluding slave roles, but this is not always desired for staging, so you can
# specify a different set here
#
# staging_roles: "web,app,db:primary=true"
# Auto detect staging roles
staging_roles: "#{known_roles.reject {|r| r =~ /slave/ || r =~ /^db$/ }.join(',')}"
# OPTIONAL: Lets one assign amazon elastic IPs (static IPs) to your instances
# You should typically set this on the role/host level rather than
# globally , unless you really do want all instances to have a
# static IP
#
# use_static_ip: true
# OPTIONAL: Specifies an instance to be created in the given availability zone
# Availability zones are sepcified by amazon to be somewhat isolated
# from each other so that hardware failures in one zone shouldn't
# affect instances in another. As such, it is good to specify these
# for instances that need to be redundant to reduce your chance of
# downtime. You should typically set this on the role/host level
# rather than globally. Use cap rubber:describe_zones to see the list
# of zones
# availability_zone: us-east-1a
# OPTIONAL: If you want to use Elastic Block Store (EBS) persistent
# volumes, add them to host specific overrides and they will get created
# and assigned to the instance. On initial creation, the volume will get
# attached _and_ formatted, but if your host disappears and you recreate
# it, the volume will only get remounted thereby preserving your data
#
# hosts:
# production15:
# availability_zone: us-east-1b
# volumes:
# - size: 100 # size of vol in GBs
# zone: us-east-1b # zone to create volume in, needs to match host's zone
# device: /dev/sdh # OS device to attach volume to
# mount: /mnt/postgresql # The directory to mount this volume to
# filesystem: ext4 # the filesystem to create on volume
#
# # OPTIONAL: Provide fog-specific options directly. This should only be used if you need a special setting that
# # Rubber does not directly expose. Since these settings will be passed directly through to fog, we can't make any
# # guarantee about how they work (if fog renames an attribute, e.g., your config will break). Please see the fog
# # source code for the option names.
# fog_options:
# type: gp2 # type of volume, standard (EBS magnetic), io1 (provisioned IOPS - SSD), or gp2 (general purpose - SSD).
# iops: 500 # The number of I/O operations per second (IOPS) that the volume supports.
# # Required when the volume type is io1; not used with non-provisioned IOPS volumes.
# - size: 10
# zone: us-east-1a
# device: /dev/sdi
# mount: /mnt/logs
# filesystem: ext4
# fog_options:
# type: io1
# iops: 500
#
# # volumes without mount/filesystem can be used in raid arrays
#
# - size: 50
# zone: us-east-1a
# device: /dev/sdx
# fog_options:
# type: gp2
# iops: 500
# - size: 50
# zone: us-east-1a
# device: /dev/sdy
# fog_options:
# type: gp2
# iops: 500
#
# # Use some ephemeral volumes for raid array
# local_volumes:
# - partition_device: /dev/sdb
# zero: false # zeros out disk for improved performance
# - partition_device: /dev/sdc
# zero: false # zeros out disk for improved performance
#
# # for raid array, you'll need to add mdadm to packages. Likewise,
# # xfsprogs is needed for xfs filesystem support
# #
# packages: [xfsprogs, mdadm]
# raid_volumes:
# - device: /dev/md0 # OS device to to create raid array on
# mount: /mnt/fast # The directory to mount this array to
# mount_opts: 'nobootwait' # Recent Ubuntu versions require this flag or SSH will not start on reboot
# filesystem: xfs # the filesystem to create on array
# filesystem_opts: -f # the filesystem opts in mkfs
# raid_level: 0 # the raid level to use for the array
# # if you're using Ubuntu 11.x or later (Natty, Oneiric, Precise, etc)
# # you will want to specify the source devices in their /dev/xvd format
# # see https://bugs.launchpad.net/ubuntu/+source/linux/+bug/684875 for
# # more information.
# # NOTE: Only make this change for raid source_devices, NOT generic
# # volume commands above.
# source_devices: [/dev/sdx, /dev/sdy] # the source EBS devices we are creating raid array from (Ubuntu Lucid or older)
# source_devices: [/dev/xvdx, /dev/xvdy] # the source EBS devices we are creating raid array from (Ubuntu Natty or newer)
#
# # for LVM volumes, you'll need to add lvm2 to packages. Likewise,
# # xfsprogs is needed for xfs filesystem support
# packages: [xfsprogs, lvm2]
# lvm_volume_groups:
# - name: vg # The volume group name
# physical_volumes: [/dev/sdx, /dev/sdy] # Devices used for LVM group (you can use just one, but you can't stripe then)
# extent_size: 32 # Size of the volume extent in MB
# volumes:
# - name: lv # Name of the logical volume
# size: 999.9 # Size of volume in GB (slightly less than sum of all physical volumes because LVM reserves some space)
# stripes: 2 # Count of stripes for volume
# filesystem: xfs # The filesystem to create on the logical volume
# filesystem_opts: -f # the filesystem opts in mkfs
# mount: /mnt/large_work_dir # The directory to mount this LVM volume to
# OPTIONAL: You can also define your own variables here for use when
# transforming config files, and they will be available in your config
# templates as <%%= rubber_env.var_name %>
#
# var_name: var_value
# All variables can also be overridden on the role, environment and/or host level by creating
# a sub level to the config under roles, environments and hosts. The precedence is host, environment, role
# e.g. to install mysql only on db role, and awstats only on web01:
# OPTIONAL: Role specific overrides
# roles:
# somerole:
# packages: []
# somerole2:
# myconfig: someval
# OPTIONAL: Environment specific overrides
# environments:
# staging:
# myconfig: otherval
# production:
# myconfig: val
# OPTIONAL: Host specific overrides
# hosts:
# somehost:
# packages: []
And doing a
cap rubber:create
I get the error after:
* executing `rubber:setup_local_aliases'
/home/casekey/.rvm/gems/ruby-2.1.1/gems/rubber-2.16.0/lib/rubber/recipes/rubber/setup.rb:192:in `block (3 levels) in load': no implicit conversion of nil into String (TypeError)
setup.rb at line at 192:
local_hosts << ic.external_ip << ' ' << hosts_data.compact.join(' ') << "\n"
After attempting to debug this with binding.pry, the line 192 goes through without any error.
Any ideas are welcome.
I have also tried:
bundle exec rake rails:update:bin
as per Rails 4 Error with every command "`load': no implicit conversion of nil into String" (Mac OS X 10.9)