For some reason my Flume 1.2 avro sink is being removed from the configuration as being empty. I've included the agent config and debug output below.
#Flume agent configuration
collector_agent.sources = tailSource1
collector_agent.sinks = avroSink1
collector_agent.channels = memChannel1
collector_agent.sources.tailSource1.type = exec
collector_agent.sources.tailSource1.command = tail -n0 -F /f/f2-traffic/logs/archive_epc.log
collector_agent.sources.tailSource1.channels = memChannel1
collector_agent.sources.tailSource1.batchSize = 5
collector_agent.sources.tailSource1.restart = true
collector_agent.sinks.avroSink1.type = avro
collector_agent.sinks.avroSink1.channels = memChannel1
collector_agent.sinks.avroSink1.hostname = 10.11.14.83
collector_agent.sinks.avroSink1.port = 4545
collector_agent.channels.memChannel1.type = memory
collector_agent.channels.memChannel1.transactionCapacity = 5
#Console log output:
bin/flume-ng agent --conf conf/ --conf-file ../test_epc_config.conf --name collector_agent -Dflume.root.logger=DEBUG,console
+ exec /usr/java/latest/bin/java -Xmx20m -Dflume.root.logger=DEBUG,console -cp '/root/flume_test/apache-flume-1.2.0/conf:/root/flume_test/apache-flume-1.2.0/lib/*' -Djava.library.path= org.apache.flume.node.Application --conf-file ../test_epc_config.conf --name collector_agent
2012-09-06 11:09:26,590 (main) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 1
2012-09-06 11:09:26,594 (main) [INFO - org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node starting - collector_agent
2012-09-06 11:09:26,597 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)] Node manager starting
2012-09-06 11:09:26,599 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)] Configuration provider starting
2012-09-06 11:09:26,599 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 10
2012-09-06 11:09:26,600 (lifecycleSupervisor-1-0) [DEBUG - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:191)] Node manager started
2012-09-06 11:09:26,601 (lifecycleSupervisor-1-2) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:86)] Configuration provider started
2012-09-06 11:09:26,601 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:188)] Checking file:../test_epc_config.conf for changes
2012-09-06 11:09:26,601 (conf-file-poller-0) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)] Reloading configuration file:../test_epc_config.conf
2012-09-06 11:09:26,606 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:avroSink1
2012-09-06 11:09:26,607 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:992)] Created context for avroSink1: type
2012-09-06 11:09:26,607 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:avroSink1
2012-09-06 11:09:26,607 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:902)] Added sinks: avroSink1 Agent: collector_agent
2012-09-06 11:09:26,607 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:avroSink1
2012-09-06 11:09:26,607 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:avroSink1
2012-09-06 11:09:26,608 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:295)] Starting validation of configuration for agent: collector_agent, initial-configuration: AgentConfiguration[collector_agent]
SOURCES: {tailSource1={ parameters:{restart=true, command=tail -n0 -F /f/f2-traffic/logs/archive_epc.log, batchSize=5, channels=memChannel1, type=exec} }}
CHANNELS: {memChannel1={ parameters:{transactionCapacity=5, type=memory} }}
SINKS: {avroSink1={ parameters:{port=4545, hostname=10.11.14.83, channels=memChannel1, type=avro} }}
2012-09-06 11:09:26,613 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateChannels(FlumeConfiguration.java:450)] Created channel memChannel1
2012-09-06 11:09:26,626 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:649)] Creating sink: avroSink1 using AVRO
2012-09-06 11:09:26,628 (conf-file-poller-0) [WARN - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:668)] Configuration empty for: avroSink1.Removed.
2012-09-06 11:09:26,629 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:353)] Post validation configuration for collector_agent
AgentConfiguration created without Configuration stubs for which only basic syntactical validation was performed[collector_agent]
SOURCES: {tailSource1={ parameters:{restart=true, command=tail -n0 -F /f/f2-traffic/logs/archive_epc.log, batchSize=5, channels=memChannel1, type=exec} }}
CHANNELS: {memChannel1={ parameters:{transactionCapacity=5, type=memory} }}
2012-09-06 11:09:26,629 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:117)] Channels:memChannel1
2012-09-06 11:09:26,629 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:118)] Sinks null
2012-09-06 11:09:26,629 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:119)] Sources tailSource1
2012-09-06 11:09:26,629 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:122)] Post-validation flume configuration contains configuration for agents: [collector_agent]
2012-09-06 11:09:26,629 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:249)] Creating channels
2012-09-06 11:09:26,629 (conf-file-poller-0) [DEBUG - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:68)] Creating instance of channel memChannel1 type memory
2012-09-06 11:09:26,668 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.<init>(MonitoredCounterGroup.java:68)] Monitoried counter group for type: CHANNEL, name: memChannel1, registered successfully.
2012-09-06 11:09:26,669 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:273)] created channel memChannel1
2012-09-06 11:09:26,669 (conf-file-poller-0) [DEBUG - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:74)] Creating instance of source tailSource1, type exec
2012-09-06 11:09:26,678 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:92)] Starting new configuration:{ sourceRunners:{tailSource1=EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource#100c62c8 }} sinkRunners:{} channels:{memChannel1=org.apache.flume.channel.MemoryChannel#1d2940b3} }
2012-09-06 11:09:26,679 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:99)] Starting Channel memChannel1
2012-09-06 11:09:26,679 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:114)] Waiting for channel: memChannel1 to start. Sleeping for 500 ms
2012-09-06 11:09:26,680 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:82)] Component type: CHANNEL, name: memChannel1 started
2012-09-06 11:09:27,181 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:138)] Starting Source tailSource1
2012-09-06 11:09:27,182 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.ExecSource.start(ExecSource.java:145)] Exec source starting with command:tail -n0 -F /f/f2-traffic/logs/archive_epc.log
2012-09-06 11:09:27,184 (lifecycleSupervisor-1-0) [DEBUG - org.apache.flume.source.ExecSource.start(ExecSource.java:163)] Exec source started
Typo based off of copy & paste from the Flume Users Guide on AvroSink. The
collector_agent.sinks.avroSink1.channels
should actually have been
collector_agent.sinks.avroSink1.channel
since a Sink can only have 1 channel.
As you mentioned, .channel is the right property for sinks.
We committed a patch today to fix all of the broken sink examples (they all contained the same error!): https://issues.apache.org/jira/browse/FLUME-1613
Related
I have this setup that has been working fine since last year December which suddenly refused to work.
I have this docker-compose yaml file like this:
version: "3.8"
services:
zookeeper1:
image: debezium/zookeeper:1.8
container_name: zookeeper1
ports:
- 2181:2181
networks:
- internalnet
kafka1:
image: debezium/kafka:1.8
container_name: kafka1
ports:
- 9092:9092
depends_on:
- zookeeper1
environment:
- KAFKA_BROKER_ID=100
- KAFKA_ZOOKEEPER_CONNECT=zookeeper1:2181
- KAFKA_ADVERTISED_HOST_NAME=kafka1
- KAFKA_LISTENERS=LISTENER_BOB://kafka1:29092,LISTENER_FRED://localhost:9092
- KAFKA_ADVERTISED_LISTENERS=LISTENER_BOB://kafka1:29092,LISTENER_FRED://localhost:9092
- KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=LISTENER_BOB:PLAINTEXT,LISTENER_FRED:PLAINTEXT
- KAFKA_INTER_BROKER_LISTENER_NAME=LISTENER_BOB
- KAFKA_ZOOKEEPER_CONNECTION_TIMEOUT_MS=60000
networks:
- internalnet
volumes:
- ./kafka/kafka1/kafka_data:/kafka/data
- ./kafka/kafka1/kafka_logs:/kafka/logs
networks:
internalnet:
driver: bridge
Zookeeper is running ok but kafka fails to run with the following log:
WARNING: Using default NODE_ID=1, which is valid only for non-clustered installations.
Starting in ZooKeeper mode using NODE_ID=1.
Using ZOOKEEPER_CONNECT=0.0.0.0:2181
Using configuration config/server.properties.
Using KAFKA_LISTENERS=LISTENER_BOB://kafka1:29092,LISTENER_FRED://localhost:9092 and KAFKA_ADVERTISED_LISTENERS=LISTENER_BOB://kafka1:29092,LISTENER_FRED://localhost:9092
2022-09-16 16:26:57,844 - INFO [main:Log4jControllerRegistration$#31] - Registered kafka:type=kafka.Log4jController MBean
2022-09-16 16:26:58,521 - INFO [main:X509Util#77] - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2022-09-16 16:26:58,667 - INFO [main:LoggingSignalHandler#72] - Registered signal handlers for TERM, INT, HUP
2022-09-16 16:26:58,674 - INFO [main:Logging#66] - starting
2022-09-16 16:26:58,678 - INFO [main:Logging#66] - Connecting to zookeeper on 0.0.0.0:2181
2022-09-16 16:26:58,719 - INFO [main:Logging#66] - [ZooKeeperClient Kafka server] Initializing a new session to 0.0.0.0:2181.
2022-09-16 16:26:58,733 - INFO [main:Environment#98] - Client environment:zookeeper.version=3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a, built on 04/08/2021 16:35 GMT
2022-09-16 16:26:58,734 - INFO [main:Environment#98] - Client environment:host.name=44841d8b6caa
2022-09-16 16:26:58,734 - INFO [main:Environment#98] - Client environment:java.version=11.0.14.1
2022-09-16 16:26:58,734 - INFO [main:Environment#98] - Client environment:java.vendor=Red Hat, Inc.
2022-09-16 16:26:58,734 - INFO [main:Environment#98] - Client environment:java.home=/usr/lib/jvm/java-11-openjdk-11.0.14.1.1-5.fc34.x86_64
2022-09-16 16:26:58,735 - INFO [main:Environment#98] - Client environment:java.class.path=/kafka/libs/activation-1.1.1.jar:/kafka/libs/aopalliance-repackaged-2.6.1.jar:/kafka/libs/argparse4j-0.7.0.jar:/kafka/libs/audience-annotations-0.5.0.jar:/kafka/libs/commons-cli-1.4.jar:/kafka/libs/commons-lang3-3.8.1.jar:/kafka/libs/connect-api-3.0.0.jar:/kafka/libs/connect-basic-auth-extension-3.0.0.jar:/kafka/libs/connect-file-3.0.0.jar:/kafka/libs/connect-json-3.0.0.jar:/kafka/libs/connect-mirror-3.0.0.jar:/kafka/libs/connect-mirror-client-3.0.0.jar:/kafka/libs/connect-runtime-3.0.0.jar:/kafka/libs/connect-transforms-3.0.0.jar:/kafka/libs/hk2-api-2.6.1.jar:/kafka/libs/hk2-locator-2.6.1.jar:/kafka/libs/hk2-utils-2.6.1.jar:/kafka/libs/jackson-annotations-2.12.3.jar:/kafka/libs/jackson-core-2.12.3.jar:/kafka/libs/jackson-databind-2.12.3.jar:/kafka/libs/jackson-dataformat-csv-2.12.3.jar:/kafka/libs/jackson-datatype-jdk8-2.12.3.jar:/kafka/libs/jackson-jaxrs-base-2.12.3.jar:/kafka/libs/jackson-jaxrs-json-provider-2.12.3.jar:/kafka/libs/jackson-module-jaxb-annotations-2.12.3.jar:/kafka/libs/jackson-module-scala_2.12-2.12.3.jar:/kafka/libs/jakarta.activation-api-1.2.1.jar:/kafka/libs/jakarta.annotation-api-1.3.5.jar:/kafka/libs/jakarta.inject-2.6.1.jar:/kafka/libs/jakarta.validation-api-2.0.2.jar:/kafka/libs/jakarta.ws.rs-api-2.1.6.jar:/kafka/libs/jakarta.xml.bind-api-2.3.2.jar:/kafka/libs/javassist-3.27.0-GA.jar:/kafka/libs/javax.servlet-api-3.1.0.jar:/kafka/libs/javax.ws.rs-api-2.1.1.jar:/kafka/libs/jaxb-api-2.3.0.jar:/kafka/libs/jersey-client-2.34.jar:/kafka/libs/jersey-common-2.34.jar:/kafka/libs/jersey-container-servlet-2.34.jar:/kafka/libs/jersey-container-servlet-core-2.34.jar:/kafka/libs/jersey-hk2-2.34.jar:/kafka/libs/jersey-server-2.34.jar:/kafka/libs/jetty-client-9.4.43.v20210629.jar:/kafka/libs/jetty-continuation-9.4.43.v20210629.jar:/kafka/libs/jetty-http-9.4.43.v20210629.jar:/kafka/libs/jetty-io-9.4.43.v20210629.jar:/kafka/libs/jetty-security-9.4.43.v20210629.jar:/kafka/libs/jetty-server-9.4.43.v20210629.jar:/kafka/libs/jetty-servlet-9.4.43.v20210629.jar:/kafka/libs/jetty-servlets-9.4.43.v20210629.jar:/kafka/libs/jetty-util-9.4.43.v20210629.jar:/kafka/libs/jetty-util-ajax-9.4.43.v20210629.jar:/kafka/libs/jline-3.12.1.jar:/kafka/libs/jopt-simple-5.0.4.jar:/kafka/libs/kafka-clients-3.0.0.jar:/kafka/libs/kafka-log4j-appender-3.0.0.jar:/kafka/libs/kafka-metadata-3.0.0.jar:/kafka/libs/kafka-raft-3.0.0.jar:/kafka/libs/kafka-server-common-3.0.0.jar:/kafka/libs/kafka-shell-3.0.0.jar:/kafka/libs/kafka-storage-3.0.0.jar:/kafka/libs/kafka-storage-api-3.0.0.jar:/kafka/libs/kafka-streams-3.0.0.jar:/kafka/libs/kafka-streams-examples-3.0.0.jar:/kafka/libs/kafka-streams-scala_2.12-3.0.0.jar:/kafka/libs/kafka-streams-test-utils-3.0.0.jar:/kafka/libs/kafka-tools-3.0.0.jar:/kafka/libs/kafka_2.12-3.0.0.jar:/kafka/libs/log4j-1.2.17.jar:/kafka/libs/lz4-java-1.7.1.jar:/kafka/libs/maven-artifact-3.8.1.jar:/kafka/libs/metrics-core-2.2.0.jar:/kafka/libs/metrics-core-4.1.12.1.jar:/kafka/libs/netty-buffer-4.1.62.Final.jar:/kafka/libs/netty-codec-4.1.62.Final.jar:/kafka/libs/netty-common-4.1.62.Final.jar:/kafka/libs/netty-handler-4.1.62.Final.jar:/kafka/libs/netty-resolver-4.1.62.Final.jar:/kafka/libs/netty-transport-4.1.62.Final.jar:/kafka/libs/netty-transport-native-epoll-4.1.62.Final.jar:/kafka/libs/netty-transport-native-unix-common-4.1.62.Final.jar:/kafka/libs/osgi-resource-locator-1.0.3.jar:/kafka/libs/paranamer-2.8.jar:/kafka/libs/plexus-utils-3.2.1.jar:/kafka/libs/reflections-0.9.12.jar:/kafka/libs/rocksdbjni-6.19.3.jar:/kafka/libs/scala-collection-compat_2.12-2.4.4.jar:/kafka/libs/scala-java8-compat_2.12-1.0.0.jar:/kafka/libs/scala-library-2.12.14.jar:/kafka/libs/scala-logging_2.12-3.9.3.jar:/kafka/libs/scala-reflect-2.12.14.jar:/kafka/libs/slf4j-api-1.7.30.jar:/kafka/libs/slf4j-log4j12-1.7.30.jar:/kafka/libs/snappy-java-1.1.8.1.jar:/kafka/libs/trogdor-3.0.0.jar:/kafka/libs/zookeeper-3.6.3.jar:/kafka/libs/zookeeper-jute-3.6.3.jar:/kafka/libs/zstd-jni-1.5.0-2.jar
2022-09-16 16:26:58,740 - INFO [main:Environment#98] - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
2022-09-16 16:26:58,745 - INFO [main:Environment#98] - Client environment:java.io.tmpdir=/tmp
2022-09-16 16:26:58,745 - INFO [main:Environment#98] - Client environment:java.compiler=<NA>
2022-09-16 16:26:58,745 - INFO [main:Environment#98] - Client environment:os.name=Linux
2022-09-16 16:26:58,748 - INFO [main:Environment#98] - Client environment:os.arch=amd64
2022-09-16 16:26:58,748 - INFO [main:Environment#98] - Client environment:os.version=5.10.16.3-microsoft-standard-WSL2
2022-09-16 16:26:58,748 - INFO [main:Environment#98] - Client environment:user.name=kafka
2022-09-16 16:26:58,748 - INFO [main:Environment#98] - Client environment:user.home=/kafka
2022-09-16 16:26:58,749 - INFO [main:Environment#98] - Client environment:user.dir=/kafka
2022-09-16 16:26:58,749 - INFO [main:Environment#98] - Client environment:os.memory.free=975MB
2022-09-16 16:26:58,749 - INFO [main:Environment#98] - Client environment:os.memory.max=1024MB
2022-09-16 16:26:58,749 - INFO [main:Environment#98] - Client environment:os.memory.total=1024MB
2022-09-16 16:26:58,754 - INFO [main:ZooKeeper#1006] - Initiating client connection, connectString=0.0.0.0:2181 sessionTimeout=18000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$#3fc79729
2022-09-16 16:26:58,782 - INFO [main:ClientCnxnSocket#239] - jute.maxbuffer value is 4194304 Bytes
2022-09-16 16:26:58,797 - INFO [main:ClientCnxn#1736] - zookeeper.request.timeout value is 0. feature enabled=false
2022-09-16 16:26:58,807 - INFO [main:Logging#66] - [ZooKeeperClient Kafka server] Waiting until connected.
2022-09-16 16:26:58,840 - INFO [main-SendThread(0.0.0.0:2181):ClientCnxn$SendThread#1181] - Opening socket connection to server 0.0.0.0/0.0.0.0:2181.
2022-09-16 16:26:58,842 - INFO [main-SendThread(0.0.0.0:2181):ClientCnxn$SendThread#1183] - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2022-09-16 16:26:58,861 - WARN [main-SendThread(0.0.0.0:2181):ClientCnxn$SendThread#1300] - Session 0x0 for sever 0.0.0.0/0.0.0.0:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290)
Also got the following log from kafka server.log
2022-09-16 14:26:52,915 - INFO [main:Log4jControllerRegistration$#31] - Registered kafka:type=kafka.Log4jController MBean
2022-09-16 14:26:54,942 - INFO [main:Logging#66] - starting
2022-09-16 14:26:54,965 - INFO [main:Logging#66] - Connecting to zookeeper on 0.0.0.0:2181
2022-09-16 14:26:55,082 - INFO [main:Logging#66] - [ZooKeeperClient Kafka server] Initializing a new session to 0.0.0.0:2181.
2022-09-16 14:26:55,341 - INFO [main:Logging#66] - [ZooKeeperClient Kafka server] Waiting until connected.
2022-09-16 14:27:55,367 - INFO [main:Logging#66] - [ZooKeeperClient Kafka server] Closing.
2022-09-16 14:27:55,758 - INFO [main:Logging#66] - [ZooKeeperClient Kafka server] Closed.
2022-09-16 14:27:55,797 - ERROR [main:MarkerIgnoringBase#159] - Fatal error during KafkaServer startup. Prepare to shutdown
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:254)
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:250)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:108)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1981)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:457)
at kafka.server.KafkaServer.startup(KafkaServer.scala:196)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
2022-09-16 14:27:55,813 - INFO [main:Logging#66] - shutting down
2022-09-16 14:27:55,858 - INFO [main:Logging#66] - shut down completed
2022-09-16 14:27:55,861 - ERROR [main:MarkerIgnoringBase#143] - Exiting Kafka.
2022-09-16 14:27:55,864 - INFO [kafka-shutdown-hook:Logging#66] - shutting down
2022-09-16 14:42:16,757 - INFO [main:Log4jControllerRegistration$#31] - Registered kafka:type=kafka.Log4jController MBean
2022-09-16 14:42:18,622 - INFO [main:Logging#66] - starting
2022-09-16 14:42:18,624 - INFO [main:Logging#66] - Connecting to zookeeper on 0.0.0.0:2181
2022-09-16 14:42:18,656 - INFO [main:Logging#66] - [ZooKeeperClient Kafka server] Initializing a new session to 0.0.0.0:2181.
2022-09-16 14:42:18,749 - INFO [main:Logging#66] - [ZooKeeperClient Kafka server] Waiting until connected.
2022-09-16 14:43:18,769 - INFO [main:Logging#66] - [ZooKeeperClient Kafka server] Closing.
2022-09-16 14:43:19,784 - INFO [main:Logging#66] - [ZooKeeperClient Kafka server] Closed.
2022-09-16 14:43:19,796 - ERROR [main:MarkerIgnoringBase#159] - Fatal error during KafkaServer startup. Prepare to shutdown
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:254)
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:250)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:108)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1981)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:457)
at kafka.server.KafkaServer.startup(KafkaServer.scala:196)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
2022-09-16 14:43:19,809 - INFO [main:Logging#66] - shutting down
2022-09-16 14:43:19,858 - INFO [main:Logging#66] - shut down completed
2022-09-16 14:43:19,870 - ERROR [main:MarkerIgnoringBase#143] - Exiting Kafka.
2022-09-16 14:43:19,876 - INFO [kafka-shutdown-hook:Logging#66] - shutting down
2022-09-16 14:53:57,029 - INFO [main:Log4jControllerRegistration$#31] - Registered kafka:type=kafka.Log4jController MBean
2022-09-16 14:53:59,011 - INFO [main:Logging#66] - starting
2022-09-16 14:53:59,017 - INFO [main:Logging#66] - Connecting to zookeeper on 0.0.0.0:2181
2022-09-16 14:53:59,115 - INFO [main:Logging#66] - [ZooKeeperClient Kafka server] Initializing a new session to 0.0.0.0:2181.
2022-09-16 14:53:59,247 - INFO [main:Logging#66] - [ZooKeeperClient Kafka server] Waiting until connected.
2022-09-16 14:54:59,256 - INFO [main:Logging#66] - [ZooKeeperClient Kafka server] Closing.
2022-09-16 14:55:00,389 - INFO [main:Logging#66] - [ZooKeeperClient Kafka server] Closed.
2022-09-16 14:55:00,397 - ERROR [main:MarkerIgnoringBase#159] - Fatal error during KafkaServer startup. Prepare to shutdown
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:254)
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:250)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:108)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1981)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:457)
at kafka.server.KafkaServer.startup(KafkaServer.scala:196)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
2022-09-16 14:55:00,400 - INFO [main:Logging#66] - shutting down
2022-09-16 14:55:00,491 - INFO [main:Logging#66] - shut down completed
2022-09-16 14:55:00,525 - ERROR [main:MarkerIgnoringBase#143] - Exiting Kafka.
2022-09-16 14:55:00,529 - INFO [kafka-shutdown-hook:Logging#66] - shutting down
I even went as far as uninstalling docker-desktop and re-installing, still the same issue.
Extra info: i am running docker-desktop (with wsl-2 on ubuntu distro) on windows 11.
It's possible Debezium did an update which broke your setup, so I suggest you grab a latest compose file, many of which exist at
https://github.com/debezium/debezium-examples
Look at the logs,
Using ZOOKEEPER_CONNECT=0.0.0.0:2181
It's not using KAFKA_ZOOKEEPER_CONNECT ... Remove the KAFKA_ prefix to set the appropriate value, then your logs should say something like ZOOKEEPER_CONNECT=zookeeper1:2181
I'm trying to send kubernetes' logs with Filebeat and Logstash. I do have some deployment on the same namespace.
I tried the suggested configuration for filebeat.yml from elastic in this [link].(https://raw.githubusercontent.com/elastic/beats/7.x/deploy/kubernetes/filebeat-kubernetes.yaml)
So, this is my overall configuration:
filebeat.yml
filebeat.inputs:
- type: container
paths:
- '/var/lib/docker/containers/*.log'
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
# To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
#filebeat.autodiscover:
# providers:
# - type: kubernetes
# node: ${NODE_NAME}
# hints.enabled: true
# hints.default_config:
# type: container
# paths:
# - /var/log/containers/*${data.kubernetes.container.id}.log
output.logstash:
hosts: ['logstash.default.svc.cluster.local:5044']
Logstash Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: logstash-deployment
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: logstash
template:
metadata:
labels:
app: logstash
spec:
containers:
- name: logstash
image: docker.elastic.co/logstash/logstash:7.15.0
ports:
- containerPort: 5044
volumeMounts:
- name: config-volume
mountPath: /usr/share/logstash/config
- name: logstash-pipeline-volume
mountPath: /usr/share/logstash/pipeline
volumes:
- name: config-volume
configMap:
name: logstash-configmap
items:
- key: logstash.yml
path: logstash.yml
- name: logstash-pipeline-volume
configMap:
name: logstash-configmap
items:
- key: logstash.conf
path: logstash.conf
Logstash Configmap
apiVersion: v1
kind: ConfigMap
metadata:
name: logstash-configmap
namespace: default
data:
logstash.yml: |
http.host: "0.0.0.0"
path.config: /usr/share/logstash/pipeline
logstash.conf: |
input {
beats {
port => 5044
}
}
output {
tcp {
mode => "client"
host => "10.184.0.4"
port => 5001
codec => "json_lines"
}
stdout {
codec => rubydebug
}
}
Logstash Service
kind: Service
apiVersion: v1
metadata:
name: logstash
namespace: default
spec:
selector:
app: logstash
ports:
- protocol: TCP
port: 5044
targetPort: 5044
Filebeat daemonset are running, also the Logstash deployment. Both of them kubectl logs shows:
Filebeat daemonset shows
2021-10-13T04:10:14.201Z INFO instance/beat.go:665 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2021-10-13T04:10:14.219Z INFO instance/beat.go:673 Beat ID: b90d1561-e989-4ed1-88f9-9b88045cee29
2021-10-13T04:10:14.220Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2021-10-13T04:10:14.220Z INFO [beat] instance/beat.go:1014 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "b90d1561-e989-4ed1-88f9-9b88045cee29"}}}
2021-10-13T04:10:14.220Z INFO [beat] instance/beat.go:1023 Build info {"system_info": {"build": {"commit": "9023152025ec6251bc6b6c38009b309157f10f17", "libbeat": "7.15.0", "time": "2021-09-16T03:16:09.000Z", "version": "7.15.0"}}}
2021-10-13T04:10:14.220Z INFO [beat] instance/beat.go:1026 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.16.6"}}}
2021-10-13T04:10:14.221Z INFO [beat] instance/beat.go:1030 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-10-06T19:41:55Z","containerized":true,"name":"filebeat-hvqx4","ip":["127.0.0.1/8","10.116.6.42/24"],"kernel_version":"5.4.120+","mac":["ae:ab:28:37:27:2a"],"os":{"type":"linux","family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":9,"patch":2009,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0,"id":"38c2fd0d69ba05ae64d8a4d4fc156791"}}}
2021-10-13T04:10:14.221Z INFO [beat] instance/beat.go:1059 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 8, "ppid": 1, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2021-10-13T04:10:12.819Z"}}}
2021-10-13T04:10:14.221Z INFO instance/beat.go:309 Setup Beat: filebeat; Version: 7.15.0
2021-10-13T04:10:14.222Z INFO [publisher] pipeline/module.go:113 Beat name: filebeat-hvqx4
2021-10-13T04:10:14.224Z WARN beater/filebeat.go:178 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-10-13T04:10:14.225Z INFO [monitoring] log/log.go:142 Starting metrics logging every 30s
2021-10-13T04:10:14.225Z INFO instance/beat.go:473 filebeat start running.
2021-10-13T04:10:14.227Z INFO memlog/store.go:119 Loading data file of '/usr/share/filebeat/data/registry/filebeat' succeeded. Active transaction id=0
2021-10-13T04:10:14.227Z INFO memlog/store.go:124 Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=0
2021-10-13T04:10:14.227Z WARN beater/filebeat.go:381 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-10-13T04:10:14.228Z INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 0
2021-10-13T04:10:14.228Z INFO [crawler] beater/crawler.go:71 Loading Inputs: 1
2021-10-13T04:10:14.228Z INFO beater/crawler.go:148 Stopping Crawler
2021-10-13T04:10:14.228Z INFO beater/crawler.go:158 Stopping 0 inputs
2021-10-13T04:10:14.228Z INFO beater/crawler.go:178 Crawler stopped
2021-10-13T04:10:14.228Z INFO [registrar] registrar/registrar.go:132 Stopping Registrar
2021-10-13T04:10:14.228Z INFO [registrar] registrar/registrar.go:166 Ending Registrar
2021-10-13T04:10:14.228Z INFO [registrar] registrar/registrar.go:137 Registrar stopped
2021-10-13T04:10:44.229Z INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000}},"id":"/"},"cpuacct":{"id":"/","total":{"ns":307409530}},"memory":{"id":"/","mem":{"limit":{"bytes":209715200},"usage":{"bytes":52973568}}}},"cpu":{"system":{"ticks":80,"time":{"ms":85}},"total":{"ticks":270,"time":{"ms":283},"value":270},"user":{"ticks":190,"time":{"ms":198}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"f5abb082-a094-4f99-a046-bc183d415455","uptime":{"ms":30208},"version":"7.15.0"},"memstats":{"gc_next":19502448,"memory_alloc":10052000,"memory_sys":75056136,"memory_total":55390312,"rss":112922624},"runtime":{"goroutines":12}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0},"queue":{"max_events":4096}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":2},"load":{"1":0.14,"15":0.28,"5":0.31,"norm":{"1":0.07,"15":0.14,"5":0.155}}}}}}
Logtash deployment logs shows:
Using bundled JDK: /usr/share/logstash/jdk
warning: no jvm.options file found
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2021-10-13 08:46:58.674 [main] runner - Starting Logstash {"logstash.version"=>"7.15.0", "jruby.version"=>"jruby 9.2.19.0 (2.5.8) 2021-06-15 55810c552b OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +jit [linux-x86_64]"}
[INFO ] 2021-10-13 08:46:58.698 [main] writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[INFO ] 2021-10-13 08:46:58.700 [main] writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[WARN ] 2021-10-13 08:46:59.077 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2021-10-13 08:46:59.097 [LogStash::Runner] agent - No persistent UUID file found. Generating new UUID {:uuid=>"7a0e5b89-70a1-4004-b38e-c31fadcd7251", :path=>"/usr/share/logstash/data/uuid"}
[INFO ] 2021-10-13 08:47:00.950 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2021-10-13 08:47:01.468 [Converge PipelineAction::Create<main>] Reflections - Reflections took 203 ms to scan 1 urls, producing 120 keys and 417 values
[WARN ] 2021-10-13 08:47:02.496 [Converge PipelineAction::Create<main>] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-10-13 08:47:02.526 [Converge PipelineAction::Create<main>] beats - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-10-13 08:47:02.664 [Converge PipelineAction::Create<main>] jsonlines - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2021-10-13 08:47:02.947 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x3b822f13#/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125 run>"}
[INFO ] 2021-10-13 08:47:05.467 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>2.52}
[INFO ] 2021-10-13 08:47:05.473 [[main]-pipeline-manager] beats - Starting input listener {:address=>"0.0.0.0:5044"}
[INFO ] 2021-10-13 08:47:05.555 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2021-10-13 08:47:05.588 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2021-10-13 08:47:05.907 [[main]<beats] Server - Starting server on port: 5044
So, my questions are:
Why is Filebeat not ingesting the logs from kubernetes?
Are there different ways to use hosts logstash on filebeat.yml? Because some examples are using DNS name just like my conf. when others are just using service names.
How to trigger/test logs to make sure my configuration is running
well?
My mistake, on filebeat environment I missed initiating the ENV node name. So, from the configuration above I just added
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
the filebeat running well now
I am trying to use my felipeogutierrez/explore-flink:1.11.1-scala_2.12 image available here into a kubernetes cluster configuration like it is saying here. I compile my project https://github.com/felipegutierrez/explore-flink with maven and I extend the default flink image flink:1.11.1-scala_2.12 with this Dockerfile:
FROM maven:3.6-jdk-8-slim AS builder
# get explore-flink job and compile it
COPY ./java/explore-flink /opt/explore-flink
WORKDIR /opt/explore-flink
RUN mvn clean install
FROM flink:1.11.1-scala_2.12
WORKDIR /opt/flink/usrlib
COPY --from=builder /opt/explore-flink/target/explore-flink.jar /opt/flink/usrlib/explore-flink.jar
ADD /opt/flink/usrlib/explore-flink.jar /opt/flink/usrlib/explore-flink.jar
#USER flink
then the tutorial 2 says to create the common cluster components:
kubectl create -f k8s/flink-configuration-configmap.yaml
kubectl create -f k8s/jobmanager-service.yaml
kubectl proxy
kubectl create -f k8s/jobmanager-rest-service.yaml
kubectl get svc flink-jobmanager-rest
and then create the jobmanager-job.yaml:
kubectl create -f k8s/jobmanager-job.yaml
I am getting a CrashLoopBackOff status error on the flink-jobmanager pod and the log says that it cannot find the class org.sense.flink.examples.stream.tpch.TPCHQuery03 in the flink-dist_2.12-1.11.1.jar:1.11.1 jar file. However, I want that kubernetes try also to look into the /opt/flink/usrlib/explore-flink.jar jar file. I am copying and adding this jar file on the Dockerfile of my image, but it seems that it is not working. What am I missing here? Below is my jobmanager-job.yaml file:
apiVersion: batch/v1
kind: Job
metadata:
name: flink-jobmanager
spec:
template:
metadata:
labels:
app: flink
component: jobmanager
spec:
restartPolicy: OnFailure
containers:
- name: jobmanager
image: felipeogutierrez/explore-flink:1.11.1-scala_2.12
imagePullPolicy: Always
env:
args: ["standalone-job", "--job-classname", "org.sense.flink.examples.stream.tpch.TPCHQuery03"]
ports:
- containerPort: 6123
name: rpc
- containerPort: 6124
name: blob-server
- containerPort: 8081
name: webui
livenessProbe:
tcpSocket:
port: 6123
initialDelaySeconds: 30
periodSeconds: 60
volumeMounts:
- name: flink-config-volume
mountPath: /opt/flink/conf
- name: job-artifacts-volume
mountPath: /opt/flink/usrlib
securityContext:
runAsUser: 9999 # refers to user _flink_ from official flink image, change if necessary
volumes:
- name: flink-config-volume
configMap:
name: flink-config
items:
- key: flink-conf.yaml
path: flink-conf.yaml
- key: log4j-console.properties
path: log4j-console.properties
- name: job-artifacts-volume
hostPath:
path: /host/path/to/job/artifacts
and my complete log file:
$ kubectl logs flink-jobmanager-qfkjl
Starting Job Manager
sed: couldn't open temporary file /opt/flink/conf/sedSg30ro: Read-only file system
sed: couldn't open temporary file /opt/flink/conf/sed1YrBco: Read-only file system
/docker-entrypoint.sh: 72: /docker-entrypoint.sh: cannot create /opt/flink/conf/flink-conf.yaml: Permission denied
/docker-entrypoint.sh: 91: /docker-entrypoint.sh: cannot create /opt/flink/conf/flink-conf.yaml.tmp: Read-only file system
Starting standalonejob as a console application on host flink-jobmanager-qfkjl.
2020-09-21 08:08:29,528 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - --------------------------------------------------------------------------------
2020-09-21 08:08:29,531 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Preconfiguration:
2020-09-21 08:08:29,532 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
JM_RESOURCE_PARAMS extraction logs:
jvm_params: -Xmx1073741824 -Xms1073741824 -XX:MaxMetaspaceSize=268435456
logs: INFO [] - Loading configuration property: jobmanager.rpc.address, flink-jobmanager
INFO [] - Loading configuration property: taskmanager.numberOfTaskSlots, 4
INFO [] - Loading configuration property: blob.server.port, 6124
INFO [] - Loading configuration property: jobmanager.rpc.port, 6123
INFO [] - Loading configuration property: taskmanager.rpc.port, 6122
INFO [] - Loading configuration property: queryable-state.proxy.ports, 6125
INFO [] - Loading configuration property: jobmanager.memory.process.size, 1600m
INFO [] - Loading configuration property: taskmanager.memory.process.size, 1728m
INFO [] - Loading configuration property: parallelism.default, 2
INFO [] - The derived from fraction jvm overhead memory (160.000mb (167772162 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
INFO [] - Final Master Memory configuration:
INFO [] - Total Process Memory: 1.563gb (1677721600 bytes)
INFO [] - Total Flink Memory: 1.125gb (1207959552 bytes)
INFO [] - JVM Heap: 1024.000mb (1073741824 bytes)
INFO [] - Off-heap: 128.000mb (134217728 bytes)
INFO [] - JVM Metaspace: 256.000mb (268435456 bytes)
INFO [] - JVM Overhead: 192.000mb (201326592 bytes)
2020-09-21 08:08:29,533 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - --------------------------------------------------------------------------------
2020-09-21 08:08:29,533 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting StandaloneApplicationClusterEntryPoint (Version: 1.11.1, Scala: 2.12, Rev:7eb514a, Date:2020-07-15T07:02:09+02:00)
2020-09-21 08:08:29,533 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - OS current user: flink
2020-09-21 08:08:29,533 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Current Hadoop/Kerberos user: <no hadoop dependency found>
2020-09-21 08:08:29,534 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.265-b01
2020-09-21 08:08:29,534 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Maximum heap size: 989 MiBytes
2020-09-21 08:08:29,534 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JAVA_HOME: /usr/local/openjdk-8
2020-09-21 08:08:29,534 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - No Hadoop Dependency available
2020-09-21 08:08:29,534 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM Options:
2020-09-21 08:08:29,534 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Xmx1073741824
2020-09-21 08:08:29,534 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Xms1073741824
2020-09-21 08:08:29,535 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -XX:MaxMetaspaceSize=268435456
2020-09-21 08:08:29,535 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlog.file=/opt/flink/log/flink--standalonejob-0-flink-jobmanager-qfkjl.log
2020-09-21 08:08:29,535 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties
2020-09-21 08:08:29,535 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties
2020-09-21 08:08:29,535 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml
2020-09-21 08:08:29,535 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Program Arguments:
2020-09-21 08:08:29,536 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - --configDir
2020-09-21 08:08:29,536 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - /opt/flink/conf
2020-09-21 08:08:29,536 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - --job-classname
2020-09-21 08:08:29,536 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - org.sense.flink.examples.stream.tpch.TPCHQuery03
2020-09-21 08:08:29,537 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Classpath: /opt/flink/lib/flink-csv-1.11.1.jar:/opt/flink/lib/flink-json-1.11.1.jar:/opt/flink/lib/flink-shaded-zookeeper-3.4.14.jar:/opt/flink/lib/flink-table-blink_2.12-1.11.1.jar:/opt/flink/lib/flink-table_2.12-1.11.1.jar:/opt/flink/lib/log4j-1.2-api-2.12.1.jar:/opt/flink/lib/log4j-api-2.12.1.jar:/opt/flink/lib/log4j-core-2.12.1.jar:/opt/flink/lib/log4j-slf4j-impl-2.12.1.jar:/opt/flink/lib/flink-dist_2.12-1.11.1.jar:::
2020-09-21 08:08:29,538 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - --------------------------------------------------------------------------------
2020-09-21 08:08:29,540 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Registered UNIX signal handlers for [TERM, HUP, INT]
2020-09-21 08:08:29,577 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Could not create application program.
org.apache.flink.util.FlinkException: Could not find the provided job class (org.sense.flink.examples.stream.tpch.TPCHQuery03) in the user lib directory (/opt/flink/usrlib).
at org.apache.flink.client.deployment.application.ClassPathPackagedProgramRetriever.getJobClassNameOrScanClassPath(ClassPathPackagedProgramRetriever.java:140) ~[flink-dist_2.12-1.11.1.jar:1.11.1]
at org.apache.flink.client.deployment.application.ClassPathPackagedProgramRetriever.getPackagedProgram(ClassPathPackagedProgramRetriever.java:123) ~[flink-dist_2.12-1.11.1.jar:1.11.1]
at org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.getPackagedProgram(StandaloneApplicationClusterEntryPoint.java:110) ~[flink-dist_2.12-1.11.1.jar:1.11.1]
at org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:78) [flink-dist_2.12-1.11.1.jar:1.11.1]
I had two problems with my configurations. First the Dockerfile was not copying the explore-flink.jar to the right location. Second I did not need to mount the volume job-artifacts-volume on the Kubernetes file jobmanager-job.yaml. Here is my Dockerfile:
FROM maven:3.6-jdk-8-slim AS builder
# get explore-flink job and compile it
COPY ./java/explore-flink /opt/explore-flink
WORKDIR /opt/explore-flink
RUN mvn clean install
FROM flink:1.11.1-scala_2.12
WORKDIR /opt/flink/lib
COPY --from=builder --chown=flink:flink /opt/explore-flink/target/explore-flink.jar /opt/flink/lib/explore-flink.jar
and the jobmanager-job.yaml file:
apiVersion: batch/v1
kind: Job
metadata:
name: flink-jobmanager
spec:
template:
metadata:
labels:
app: flink
component: jobmanager
spec:
restartPolicy: OnFailure
containers:
- name: jobmanager
image: felipeogutierrez/explore-flink:1.11.1-scala_2.12
imagePullPolicy: Always
env:
#command: ["ls"]
args: ["standalone-job", "--job-classname", "org.sense.flink.App", "-app", "36"] #, <optional arguments>, <job arguments>] # optional arguments: ["--job-id", "<job id>", "--fromSavepoint", "/path/to/savepoint", "--allowNonRestoredState"]
#args: ["standalone-job", "--job-classname", "org.sense.flink.examples.stream.tpch.TPCHQuery03"] #, <optional arguments>, <job arguments>] # optional arguments: ["--job-id", "<job id>", "--fromSavepoint", "/path/to/savepoint", "--allowNonRestoredState"]
ports:
- containerPort: 6123
name: rpc
- containerPort: 6124
name: blob-server
- containerPort: 8081
name: webui
livenessProbe:
tcpSocket:
port: 6123
initialDelaySeconds: 30
periodSeconds: 60
volumeMounts:
- name: flink-config-volume
mountPath: /opt/flink/conf
securityContext:
runAsUser: 9999 # refers to user _flink_ from official flink image, change if necessary
volumes:
- name: flink-config-volume
configMap:
name: flink-config
items:
- key: flink-conf.yaml
path: flink-conf.yaml
- key: log4j-console.properties
path: log4j-console.properties
I'm trying for the first time to set up a cluster where filebeat sends logs to ES which I can view in Kibana. All I'm trying to do is see the logs I write into the file /tmp/aaa.log in Kibana. I'm getting a little lost in all the configuration. Can someone tell me what I'm doing wrong based on the configuration files below?
Here's my docker-compose.yml:
---
version: '3.6'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:${TAG}
container_name: elasticsearch
ports: ['9200:9200']
networks: ['stack']
environment:
- xpack.security.enabled=false
volumes:
- 'es_data:/usr/share/elasticsearch/data'
kibana:
image: docker.elastic.co/kibana/kibana:${TAG}
container_name: kibana
ports: ['5601:5601']
networks: ['stack']
depends_on: ['elasticsearch']
environment:
- xpack.security.enabled=false
logstash:
image: docker.elastic.co/logstash/logstash:${TAG}
container_name: logstash
networks: ['stack']
depends_on: ['elasticsearch']
environment:
- xpack.security.enabled=false
filebeat:
image: docker.elastic.co/beats/filebeat:${TAG}
container_name: filebeat
volumes:
- /tmp/filebeat.yml:/usr/share/filebeat/filebeat.yml
networks: ['stack']
depends_on: ['elasticsearch', 'kibana']
networks: {stack: {}}
And here's filebeat.yml:
filebeat.prospectors:
- input_type: log
paths:
- /tmp/aaa.log
output.elasticsearch:
hosts: ['elasticsearch:9200']
And I run this with TAG=5.6.13 docker-compose up (I have to use ES version 5).
Here are the logs:
2018/11/27 16:20:57.165350 beat.go:297: INFO Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2018/11/27 16:20:57.165389 beat.go:192: INFO Setup Beat: filebeat; Version: 5.6.13
2018/11/27 16:20:57.165502 output.go:263: INFO Loading template enabled. Reading template file: /usr/share/filebeat/filebeat.template.json
2018/11/27 16:20:57.166247 output.go:274: INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /usr/share/filebeat/filebeat.template-es2x.json
2018/11/27 16:20:57.167063 output.go:286: INFO Loading template enabled for Elasticsearch 6.x. Reading template file: /usr/share/filebeat/filebeat.template-es6x.json
2018/11/27 16:20:57.167554 metrics.go:23: INFO Metrics logging every 30s
2018/11/27 16:20:57.167888 client.go:128: INFO Elasticsearch url: http://elasticsearch:9200
2018/11/27 16:20:57.167909 outputs.go:108: INFO Activated elasticsearch as output plugin.
2018/11/27 16:20:57.168015 publish.go:300: INFO Publisher name: 34df7198d027
2018/11/27 16:20:57.168185 async.go:63: INFO Flush Interval set to: 1s
2018/11/27 16:20:57.168194 async.go:64: INFO Max Bulk Size set to: 50
2018/11/27 16:20:57.168512 beat.go:233: INFO filebeat start running.
2018/11/27 16:20:57.168546 registrar.go:68: INFO No registry file found under: /usr/share/filebeat/data/registry. Creating a new registry file.
2018/11/27 16:20:57.174446 registrar.go:106: INFO Loading registrar data from /usr/share/filebeat/data/registry
2018/11/27 16:20:57.174491 registrar.go:123: INFO States Loaded from registrar: 0
2018/11/27 16:20:57.174515 crawler.go:38: INFO Loading Prospectors: 1
2018/11/27 16:20:57.174633 prospector_log.go:65: INFO Prospector with previous states loaded: 0
2018/11/27 16:20:57.174715 prospector.go:124: INFO Starting prospector of type: log; id: 16715230261889747
2018/11/27 16:20:57.174726 crawler.go:58: INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2018/11/27 16:20:57.174735 registrar.go:236: INFO Starting Registrar
2018/11/27 16:20:57.174754 sync.go:41: INFO Start sending events to output
2018/11/27 16:20:57.174788 spooler.go:63: INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2018/11/27 16:21:27.168018 metrics.go:39: INFO Non-zero metrics in the last 30s: registrar.writes=1
2018/11/27 16:21:57.167828 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/11/27 16:22:27.167772 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/11/27 16:22:57.167974 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/11/27 16:23:27.167752 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/11/27 16:23:57.167944 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/11/27 16:24:27.167943 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/11/27 16:24:32.039122 filebeat.go:267: INFO Stopping filebeat
2018/11/27 16:24:32.039158 crawler.go:90: INFO Stopping Crawler
2018/11/27 16:24:32.039166 crawler.go:100: INFO Stopping 1 prospectors
2018/11/27 16:24:32.039187 prospector.go:180: INFO Prospector ticker stopped
2018/11/27 16:24:32.039187 prospector.go:137: INFO Prospector channel stopped because beat is stopping.
2018/11/27 16:24:32.039198 prospector.go:232: INFO Stopping Prospector: 16715230261889747
2018/11/27 16:24:32.039215 crawler.go:112: INFO Crawler stopped
2018/11/27 16:24:32.039223 spooler.go:101: INFO Stopping spooler
2018/11/27 16:24:32.039249 registrar.go:291: INFO Stopping Registrar
2018/11/27 16:24:32.039264 registrar.go:248: INFO Ending Registrar
2018/11/27 16:24:32.041518 metrics.go:51: INFO Total non-zero values: registrar.writes=2
2018/11/27 16:24:32.041533 metrics.go:52: INFO Uptime: 3m34.878904973s
2018/11/27 16:24:32.041538 beat.go:237: INFO filebeat stopped.
2018/11/28 08:43:17.481376 beat.go:297: INFO Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2018/11/28 08:43:17.481411 beat.go:192: INFO Setup Beat: filebeat; Version: 5.6.13
2018/11/28 08:43:17.481500 output.go:263: INFO Loading template enabled. Reading template file: /usr/share/filebeat/filebeat.template.json
2018/11/28 08:43:17.482638 output.go:274: INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /usr/share/filebeat/filebeat.template-es2x.json
2018/11/28 08:43:17.483675 metrics.go:23: INFO Metrics logging every 30s
2018/11/28 08:43:17.483780 output.go:286: INFO Loading template enabled for Elasticsearch 6.x. Reading template file: /usr/share/filebeat/filebeat.template-es6x.json
2018/11/28 08:43:17.484701 client.go:128: INFO Elasticsearch url: http://elasticsearch:9200
2018/11/28 08:43:17.484745 outputs.go:108: INFO Activated elasticsearch as output plugin.
2018/11/28 08:43:17.484844 publish.go:300: INFO Publisher name: 34df7198d027
2018/11/28 08:43:17.484975 async.go:63: INFO Flush Interval set to: 1s
2018/11/28 08:43:17.484982 async.go:64: INFO Max Bulk Size set to: 50
2018/11/28 08:43:17.485563 beat.go:233: INFO filebeat start running.
2018/11/28 08:43:17.485607 registrar.go:85: INFO Registry file set to: /usr/share/filebeat/data/registry
2018/11/28 08:43:17.485630 registrar.go:106: INFO Loading registrar data from /usr/share/filebeat/data/registry
2018/11/28 08:43:17.485656 registrar.go:123: INFO States Loaded from registrar: 0
2018/11/28 08:43:17.485688 crawler.go:38: INFO Loading Prospectors: 1
2018/11/28 08:43:17.485758 prospector_log.go:65: INFO Prospector with previous states loaded: 0
2018/11/28 08:43:17.485840 prospector.go:124: INFO Starting prospector of type: log; id: 16715230261889747
2018/11/28 08:43:17.485848 crawler.go:58: INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2018/11/28 08:43:17.485881 sync.go:41: INFO Start sending events to output
2018/11/28 08:43:17.485898 registrar.go:236: INFO Starting Registrar
2018/11/28 08:43:17.485945 spooler.go:63: INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2018/11/28 08:43:47.483962 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/11/28 08:44:17.484051 metrics.go:34: INFO No non-zero metrics in the last 30s
My mistake. I stupidly forgot to map the log file in docker-compose.yml.
I'm trying to use Logstash with Docker Compose, the .yml file looks like this:
user-service:
image: images/user-service
ports:
- "2222:2222"
links:
- logstash
logstash:
image: images/logstash
command: logstash -e 'input{} output{}'
ports:
- "5045:5045"
And logstash starts and ends, as the console shows:
logstash_1 | Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
logstash_1 | 01:51:30.164 [[main]-pipeline-manager] INFO logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250}
logstash_1 | 01:51:30.246 [[main]-pipeline-manager] INFO logstash.pipeline - Pipeline main started
logstash_1 | 01:51:30.860 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
logstash_1 | 01:51:33.318 [LogStash::Runner] WARN logstash.agent - stopping pipeline {:id=>"main"}
docker_logstash_1 exited with code 0
What could be the problem?
I resolved it changing the input from the filter:
input{ tcp{
port=> 9600
host=>localhost
}
}