unable to start logstash - docker

this is my first time running logstash on container. Im running logstash on the same container elasticsearch + kibana. Its running on ubuntu.
i run my conf file by using
/usr/share/logstash/bin/logstash -f conf.d/logstash.conf
here is my logstash.conf :
input{
beats{
port=>5044
}
}
filter
{
grok {
match =>{
"message" => "%{TIMESTAMP_ISO8601:logtimestamp}\s%{DATA:S_IP}\s%{WORD:s_method}\s%{DATA:cs_uri_stem}\s%{DATA:cs_uri_query}\s%{DATA:s_port}\s%{GREEDYDATA:log_message}"
}
}
date{
match =>["logtimestamp","yyyy-MM-dd HH:mm:ss"]
target=>"#timestamp"
}
}
output{
stdout{codec=>rubydebug}
elasticsearch{
hosts=>"elastic#localhost:9200"
index=>"log_iis"
user =>"*****"
password=>"*****"
}
}
and it returning error as :
Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.8.0.jar) to field java.io.FileDescriptor.fd
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Thread.exclusive is deprecated, use Thread::Mutex
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2020-02-10 03:32:59.625 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2020-02-10 03:32:59.632 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.5.2"}
[INFO ] 2020-02-10 03:33:00.995 [Converge PipelineAction::Create<main>] Reflections - Reflections took 24 ms to scan 1 urls, producing 20 keys and 40 values
[ERROR] 2020-02-10 03:33:01.375 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: Illegal character in scheme name at index 7: elastic#localhost:9200", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:60)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1156)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1143)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:27)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:915)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77)", "org.jruby.runtime.Block.call(Block.java:129)", "org.jruby.RubyProc.call(RubyProc.java:295)", "org.jruby.RubyProc.call(RubyProc.java:274)", "org.jruby.RubyProc.call(RubyProc.java:270)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:830)"]}
warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`
create at org/logstash/execution/ConvergeResultExt.java:109
add at org/logstash/execution/ConvergeResultExt.java:37
converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:339
[ERROR] 2020-02-10 03:33:01.379 [Agent thread] agent - An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`", :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:339:in `block in converge_state'"]}
[FATAL] 2020-02-10 03:33:01.403 [LogStash::Runner] runner - An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:339:in `block in converge_state'"]}
[ERROR] 2020-02-10 03:33:01.432 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit)
and respond or explanation will be appreciated so much. thank you

The error log says that you get
Illegal character in scheme name at index 7: elastic#localhost:9200", which is the value of the hosts option.
I guess the problem is the at (#). Is that needed? Anyway, if you check the documentation of the Elasticsearch output plugin [1], it says that
Any special characters present in the URLs here MUST be URL escaped! This means # should be put in as %23 for instance.
[1] https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-hosts

Related

RabbitMQ cant start due to an error on log fomatter config

I'm using RabbitMQ 3.8.5-management with the following config:
log.file = rabbit.log
log.dir = /var/log/rabbitmq
log.file.level = info
log.file.formatter = json
log.file.rotation.date = $D0
I get the following error:
12:45:12.131 [error] You've tried to set log.file.formatter, but there is no setting with that name.
12:45:12.134 [error] Did you mean one of these?
12:45:12.182 [error] log.file.level
12:45:12.182 [error] log.file
12:45:12.182 [error] log.file.rotation.date
12:45:12.182 [error] Error preparing configuration in phase transform_datatypes:
12:45:12.183 [error] - Conf file attempted to set unknown variable: log.file.formatter
According to the documentation log.file.formatter should work - what is wrong?
checked documentation on RabbitMQ.
checked other SO posts.
entered the container and remove the config - it works without it.
Looks like JSON logging and the log.file.formatter setting was added with RabbitMQ 3.9.0 release.
Try upgrading if possible.

Upgrading docker artifactory pro 6.X to 7.X

compose artifactory pro v6.9.0 running
In my compose I have two services :
image: docker.bintray.io/jfrog/artifactory-pro:6.9.0
image: docker.io/library/postgres:9.6.11
I was able to upgrade to 6.23.13 without any problem just by changing the version of the image.
When I try the same thing with any 7.X version (after upgrading to at least 6.10 as the doc says), I have errors.
For example, trying 7.21.3, I have these warnings
2022-08-01T08:41:30.343L [tomct] [WARNING] [ ] [org.apache.catalina.startup.HostConfig] [org.apache.catalina.startup.HostConfig deployDescriptor] - A docBase [/opt/jfrog/artifactory/app/artifactory/tomcat/webapps/access.war] inside the host appBase has been specified, and will be ignored
2022-08-01T08:41:30.343L [tomct] [WARNING] [ ] [org.apache.catalina.startup.HostConfig] [org.apache.catalina.startup.HostConfig deployDescriptor] - A docBase [/opt/jfrog/artifactory/app/artifactory/tomcat/webapps/artifactory.war] inside the host appBase has been specified, and will be ignored
...
2022-08-01T08:41:37.344Z [jfrt ] [WARN ] [ce1b2553475da56b] [c.z.h.p.ProxyConnection:182 ] [ocalhost-startStop-2] - HikariCP Main - Connection org.apache.derby.impl.jdbc.EmbedConnection#1597179442 (XID = 24), (SESSIONID = 3), (DATABASE = {db.home}), (DRDAID = null) marked as broken because of SQLSTATE(0A000), ErrorCode(20000)
java.sql.SQLFeatureNotSupportedException: Feature not implemented: No details.
and these errors
08:41:34,803 |-ERROR in ch.qos.logback.core.joran.action.AppenderAction - Could not create an Appender of type [org.artifactory.usage.appender.UsageTrafficTimeBasedRollingFileAppender]. ch.qos.logback.core.util.DynamicClassLoadingException: Failed to instantiate type org.artifactory.usage.appender.UsageTrafficTimeBasedRollingFileAppender
...
2022-08-01T08:41:37.113Z [jfrt ] [ERROR] [ce1b2553475da56b] [d.d.l.DbDistributeLocksDao:506] [ocalhost-startStop-2] - Unable to detect database version Unable to get connection from unique lock data source
2022-08-01T08:41:37.353Z [jfrt ] [ERROR] [ce1b2553475da56b] [tifactoryHomeConfigListener:55] [ocalhost-startStop-2] - Failed initializing Home. Caught exception:
java.lang.IllegalStateException: Could not find database table: db_properties
After reading the docs I do not clearly understand if I have to download the new docker-compose package from jfrog. I've tried but once there, the config.sh ask for external database and no question about reusing existing image directory.
Thx for help
Not sure if this is still relevant for you, but I got the same problem. As far as I see there are new DB connection environment variables, that must be set. This error message is just a symptom, because without the new DB connection env, Artifactory will try to use an in-memory database (Apache Derby). Something like this is needed to fix it (as Docker Compose config of the Artifactory container):
environment:
- JF_SHARED_DATABASE_TYPE=postgresql
- JF_SHARED_DATABASE_USERNAME=${POSTGRESQL_USERNAME}
- JF_SHARED_DATABASE_PASSWORD=${POSTGRESQL_PASSWORD}
- JF_SHARED_DATABASE_URL=jdbc:postgresql://postgresql:5432/artifactory
- JF_SHARED_DATABASE_DRIVER=org.postgresql.Driver

getting this error while trying to run cube js

I'm getting this error while trying to run cube js with the default command in the getting started docs. I've started this in a folder and running it in docker.
Warning. There is no cube.js file. Continue with environment variables
πŸ”₯ Cube Store (0.28.31) is assigned to 3030 port.
Warning. Option apiSecret is required in dev mode. Cube.js has generated it as e3b8c5a35fe378f4d481ada777e5f3c4
πŸ”“ Authentication checks are disabled in developer mode. Please use NODE_ENV=production to enable it.
πŸ¦… Dev environment available at http://localhost:4000
πŸš€ Cube.js server (0.28.31) is listening on 4000
2021-09-03 15:06:01,512 INFO [cubestore::http::status] <pid:17> Serving status probes at 0.0.0.0:3031
2021-09-03 15:06:01,515 INFO [cubestore::metastore] <pid:17> Using existing metastore in /cube/conf/.cubestore/data/metastore
thread '
main
' panicked at '
called `Result::unwrap()` on an `Err` value: Error { message: "IO error: While fsync: a directory: Invalid argument" }
', /project/cubestore/src/metastore/mod.rs:1542:40
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Cube Store Start Error: undefined
I guess it’s corrupted metastore due to it was incorrectly shutdown for you locally. Could you please try to drop the .cubestore directory?

enable local cache on emqx

I see the documentation on https://docs.emqx.io/broker/v3/en/guide.html#emq-x-bridge-cache-configuration and it says that you can enable the cache on file if the network fails because emqx now is not doing this stuff.
When i set, for example the parameter on emqx 3.0.0.0 it fails on start and says in the lof file that is not declared:
You've tried to set bridge.xxx.queue.replayq_seg_bytes, but there is no setting with that name.
2020-03-03T19:43:22.777171+03:00 [error] Did you mean one of these?
2020-03-03T19:43:22.962094+03:00 [error] bridge.$name.mqueue_type
2020-03-03T19:43:22.962572+03:00 [error] bridge.$name.clean_start
2020-03-03T19:43:22.962760+03:00 [error] bridge.$name.start_type
2020-03-03T19:43:23.102793+03:00 [error] Error generating configuration in phase transform_datatypes
2020-03-03T19:43:23.103040+03:00 [error] Conf file attempted to set unknown variable: bridge.aps.queue.replayq_seg_bytes
You know if its problem of my version of emqx or is possible a problem with the sintax.
Thanks in advance
Greetings
It's sintax error.
bridge.xxx.queue.replayq_seg_bytes
It's means set the xxx bridge use queue.replayq_seg_bytes config.
bridge.mqtt.xxx.address = 127.0.0.1:1883
Is exists? By the way the EMQ X v4.0.6 is recommended.

Start RabbitMQ server failed

I start my rabbitmq server by command ./rabbitmq-server .
But on starting broker, it show "BOOT FAILED":
BOOT FAILED
Error description:
function_clause`enter code here`
Stack trace:
[{rabbit_networking,'-boot_tcp/0-lc$^0/1-0-',[5670],[]},
{rabbit_networking,boot_tcp,0,[]},
{rabbit_networking,boot,0,[]},
{rabbit,'-run_boot_step/1-lc$^1/1-1-',1,[]},
{rabbit,run_boot_step,1,[]},
{rabbit,'-start/2-lc$^0/1-0-',1,[]},
{rabbit,start,2,[]},
{application_master,start_it_old,4,
[{file,"application_master.erl"},{line,272}]}]
Your configuration is incorrect. The variable tcp_listeners in rabbitmq.config is set to an integer, 5670, but it needs to be set to a list of integers, such as [5670]. See the RabbitMQ configuration guide for more information.

Resources