I have 10 files with some data varying in length.I would like to store corresponding data in same file and with same filename, but flume is splitting up the data and saving as FlumeData.timestamp. I am using the configuration as below:
a1.sources = r1
a1.sinks = k2
a1.channels = c1
a1.channels.c1.type = file
a1.channels.c1.checkpointDir = /mnt/flume/checkpoint
a1.channels.c1.dataDirs = /mnt/flume/data
a1.channels.c1.trackerDir = /mnt/flume/track
a1.channels.c1.transactionCapacity = 10000000
a1.channels.c1.capacity = 500000000
a1.channels.c1.maxFileSize = 10000000
a1.channels.c1.useDualCheckpoints = true
a1.channels.c1.backupCheckpointDir = /mnt/flume/backup
a1.channels.c1.checkpointInterval = 2000
a1.channels.c1.minimumRequiredSpace = 9000000
a1.sources.r1.channels = c1
a1.sources.r1.type = spooldir
a1.sources.r1.spoolDir = /usr/local/netlog/
a1.sources.r1.fileHeader = true
a1.sources.r1.bufferMaxLineLength = 500
a1.sources.r1.bufferMaxLines = 10000
a1.sources.r1.batchSize = 100000
#a1.sources.r1.deletePolicy = immediate
a1.sinks.k2.type = hdfs
a1.sinks.k2.channel = c1
a1.sinks.k2.hdfs.filePrefix = %{file}
a1.sinks.k2.hdfs.fileType = DataStream
a1.sinks.k2.hdfs.batchSize = 100000
a1.sinks.k2.hdfs.rollSize = 10000000
a1.sinks.k2.hdfs.rollInterval = 0
a1.sinks.k2.hdfs.rollSize = 0
a1.sinks.k2.hdfs.rollCount = 0
a1.sinks.k2.hdfs.idleTimeout = 0
a1.sinks.k2.hdfs.writeFormat = Text
a1.sinks.k2.hdfs.path = /user/flume
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k2.channel = c1
Kindly suggest how i can store same 10 files with same filename and data within it. File size can vary from 2 MB to 15 MB.
The error i see in logs is
lib/native org.apache.flume.node.Application --conf-file conf/flume-conf.properties --name a1
2014-12-03 20:49:47,545 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:61)] Configuration provider starting
2014-12-03 20:49:47,550 (lifecycleSupervisor-1-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:78)] Configuration provider started
2014-12-03 20:49:47,555 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume-conf.properties for changes
2014-12-03 20:49:47,555 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:133)] Reloading configuration file:conf/flume-conf.properties
2014-12-03 20:49:47,571 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:k2
2014-12-03 20:49:47,571 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1020)] Created context for k2: hdfs.batchSize
2014-12-03 20:49:47,572 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:k2
2014-12-03 20:49:47,572 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:k2
2014-12-03 20:49:47,572 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:k2
2014-12-03 20:49:47,572 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:k2
2014-12-03 20:49:47,572 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:k2
2014-12-03 20:49:47,573 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:k2
2014-12-03 20:49:47,573 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:k2
2014-12-03 20:49:47,573 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:930)] Added sinks: k2 Agent: a1
2014-12-03 20:49:47,573 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:k2
2014-12-03 20:49:47,573 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:k2
2014-12-03 20:49:47,575 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:k2
2014-12-03 20:49:47,576 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:313)] Starting validation of configuration for agent: a1, initial-configuration: AgentConfiguration[a1]
SOURCES: {r1={ parameters:{bufferMaxLineLength=500, channels=c1, spoolDir=/usr/local/netlog/, bufferMaxLines=10000, fileHeader=true, batchSize=100000, type=spooldir} }}
CHANNELS: {c1={ parameters:{trackerDir=/mnt/flume/track, maxFileSize=10000000, dataDirs=/mnt/flume/data, type=file, transactionCapacity=10000000, capacity=500000000, checkpointDir=/mnt/flume/checkpoint} }}
SINKS: {k2={ parameters:{hdfs.batchSize=100000, hdfs.idleTimeout=0, hdfs.filePrefix=%{file}, hdfs.path=/user/flume, hdfs.writeFormat=Text, hdfs.rollSize=0, hdfs.rollCount=0, channel=c1, hdfs.rollInterval=0, hdfs.fileType=DataStream, type=hdfs} }}
2014-12-03 20:49:47,583 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateChannels(FlumeConfiguration.java:468)] Created channel c1
2014-12-03 20:49:47,593 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:674)] Creating sink: k2 using HDFS
2014-12-03 20:49:47,596 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:371)] Post validation configuration for a1
AgentConfiguration created without Configuration stubs for which only basic syntactical validation was performed[a1]
SOURCES: {r1={ parameters:{bufferMaxLineLength=500, channels=c1, spoolDir=/usr/local/netlog/, bufferMaxLines=10000, fileHeader=true, batchSize=100000, type=spooldir} }}
CHANNELS: {c1={ parameters:{trackerDir=/mnt/flume/track, maxFileSize=10000000, dataDirs=/mnt/flume/data, type=file, transactionCapacity=10000000, capacity=500000000, checkpointDir=/mnt/flume/checkpoint} }}
SINKS: {k2={ parameters:{hdfs.batchSize=100000, hdfs.idleTimeout=0, hdfs.filePrefix=%{file}, hdfs.path=/user/flume, hdfs.writeFormat=Text, hdfs.rollSize=0, hdfs.rollCount=0, channel=c1, hdfs.rollInterval=0, hdfs.fileType=DataStream, type=hdfs} }}
2014-12-03 20:49:47,597 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:135)] Channels:c1
2014-12-03 20:49:47,597 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:136)] Sinks k2
2014-12-03 20:49:47,597 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:137)] Sources r1
2014-12-03 20:49:47,597 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)] Post-validation flume configuration contains configuration for agents: [a1]
2014-12-03 20:49:47,598 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:150)] Creating channels
2014-12-03 20:49:47,629 (conf-file-poller-0) [INFO - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:40)] Creating instance of channel c1 type file
2014-12-03 20:49:47,635 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:205)] Created channel c1
2014-12-03 20:49:47,636 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:39)] Creating instance of source r1, type spooldir
2014-12-03 20:49:47,654 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:40)] Creating instance of sink: k2, type: hdfs
2014-12-03 20:49:48,108 (conf-file-poller-0) [INFO - org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.java:555)] Hadoop Security enabled: false
2014-12-03 20:49:48,111 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:119)] Channel c1 connected to [r1, k2]
2014-12-03 20:49:48,125 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:138)] Starting new configuration:{ sourceRunners:{r1=EventDrivenSourceRunner: { source:Spool Directory source r1: { spoolDir: /usr/local/netlog/ } }} sinkRunners:{k2=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor#1f87c88 counterGroup:{ name:null counters:{} } }} channels:{c1=FileChannel c1 { dataDirs: [/mnt/flume/data] }} }
2014-12-03 20:49:48,130 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:145)] Starting Channel c1
2014-12-03 20:49:48,130 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.channel.file.FileChannel.start(FileChannel.java:259)] Starting FileChannel c1 { dataDirs: [/mnt/flume/data] }...
2014-12-03 20:49:48,147 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.channel.file.Log.<init>(Log.java:328)] Encryption is not enabled
2014-12-03 20:49:48,149 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.channel.file.Log.replay(Log.java:373)] Replay started
2014-12-03 20:49:48,150 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.channel.file.Log.replay(Log.java:385)] Found NextFileID 0, from []
2014-12-03 20:49:48,155 (lifecycleSupervisor-1-0) [ERROR - org.apache.flume.channel.file.Log.replay(Log.java:481)] Failed to initialize Log on [channel=c1]
java.io.EOFException
at java.io.RandomAccessFile.readInt(RandomAccessFile.java:786)
at java.io.RandomAccessFile.readLong(RandomAccessFile.java:819)
at org.apache.flume.channel.file.EventQueueBackingStoreFactory.get(EventQueueBackingStoreFactory.java:79)
at org.apache.flume.channel.file.Log.replay(Log.java:417)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:279)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2014-12-03 20:49:48,160 (lifecycleSupervisor-1-0) [ERROR - org.apache.flume.channel.file.FileChannel.start(FileChannel.java:290)] Failed to start the file channel [channel=c1]
java.io.EOFException
at java.io.RandomAccessFile.readInt(RandomAccessFile.java:786)
at java.io.RandomAccessFile.readLong(RandomAccessFile.java:819)
at org.apache.flume.channel.file.EventQueueBackingStoreFactory.get(EventQueueBackingStoreFactory.java:79)
at org.apache.flume.channel.file.Log.replay(Log.java:417)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:279)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2014-12-03 20:49:48,162 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:173)] Starting Sink k2
2014-12-03 20:49:48,163 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:184)] Starting Source r1
2014-12-03 20:49:48,163 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.source.SpoolDirectorySource.start(SpoolDirectorySource.java:77)] SpoolDirectorySource source starting with directory: /usr/local/netlog/
2014-12-03 20:49:48,185 (lifecycleSupervisor-1-3) [DEBUG - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.<init>(ReliableSpoolingFileEventReader.java:132)] Initializing ReliableSpoolingFileEventReader with directory=/usr/local/netlog, metaDir=.flumespool, deserializer=LINE
2014-12-03 20:49:48,204 (lifecycleSupervisor-1-3) [DEBUG - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.<init>(ReliableSpoolingFileEventReader.java:154)] Successfully created and deleted canary file: /usr/local/netlog/flume-spooldir-perm-check-5019906964160509405.canary
2014-12-03 20:49:48,218 (lifecycleSupervisor-1-3) [DEBUG - org.apache.flume.source.SpoolDirectorySource.start(SpoolDirectorySource.java:110)] SpoolDirectorySource source started
2014-12-03 20:49:48,343 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: SOURCE, name: r1: Successfully registered new MBean.
2014-12-03 20:49:48,343 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: SOURCE, name: r1 started
2014-12-03 20:49:48,344 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: SINK, name: k2: Successfully registered new MBean.
2014-12-03 20:49:48,347 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: SINK, name: k2 started
2014-12-03 20:49:48,356 (SinkRunner-PollingRunner-DefaultSinkProcessor) [DEBUG - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:143)] Polling sink runner starting
2014-12-03 20:49:48,357 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable to deliver event. Exception follows.
java.lang.IllegalStateException: Channel closed [channel=c1]. Due to java.io.EOFException: null
at org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:329)
at org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:122)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:376)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readInt(RandomAccessFile.java:786)
at java.io.RandomAccessFile.readLong(RandomAccessFile.java:819)
at org.apache.flume.channel.file.EventQueueBackingStoreFactory.get(EventQueueBackingStoreFactory.java:79)
at org.apache.flume.channel.file.Log.replay(Log.java:417)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:279)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
2014-12-03 20:49:48,659 (pool-4-thread-1) [ERROR - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:256)] FATAL: Spool Directory source r1: { spoolDir: /usr/local/netlog/ }: Uncaught exception in SpoolDirectorySource thread. Restart or reconfigure Flume to continue processing.
java.lang.IllegalStateException: Channel closed [channel=c1]. Due to java.io.EOFException: null
at org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:329)
at org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:122)
at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:181)
at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:235)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readInt(RandomAccessFile.java:786)
at java.io.RandomAccessFile.readLong(RandomAccessFile.java:819)
at org.apache.flume.channel.file.EventQueueBackingStoreFactory.get(EventQueueBackingStoreFactory.java:79)
at org.apache.flume.channel.file.Log.replay(Log.java:417)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:279)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
... 7 more
2014-12-03 20:49:53,359 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable to deliver event. Exception follows.
java.lang.IllegalStateException: Channel closed [channel=c1]. Due to java.io.EOFException: null
at org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:329)
at org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:122)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:376)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readInt(RandomAccessFile.java:786)
at java.io.RandomAccessFile.readLong(RandomAccessFile.java:819)
at org.apache.flume.channel.file.EventQueueBackingStoreFactory.get(EventQueueBackingStoreFactory.java:79)
at org.apache.flume.channel.file.Log.replay(Log.java:417)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:279)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
thanks in advance
ok, you need a few more props fro hdfs sink:
a1.sinks.k2.hdfs.filePrefix = [your prefix]
a1.sinks.k2.hdfs.fileSuffix = .[your suffix]
suffix would be .tsv or .csv for instance, while prefix can be anything - you can also use variables for date and time - this requires that you use the timestamp interceptor. You can also create your own interceptor and generate your own variables into your file name. If you omit this, flume will add its own sequence between prefix and suffix.
As an addition to our previous comments, the props to disable rollovers are the following:
a1.sinks.k2.rollInterval = 0
a1.sinks.k2.rollSize = 0
a1.sinks.k2.rollCount = 0
a1.sinks.k2.idleTimeout = 0
to access the file name of the original file from your source, append the following in your hdfs sink config:
a1.sinks.k2.hdfs.filePrefix = %{file}
to simplify your channel config, do the following:
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
Thanks much for your help Erik it helped to go close to the problem i found that flume was creating a file in the data source directory .flumespool which i had to delete along with /mnt/flume/* and also rename the data file to the same name as it was if it was changed to .COMPLETED
Related
I am facing log issue in maven application in which I am using a dependency jar of another non maven application both are using log4j 2.18.0 version. But, all the logs of main applications are also printing in dependency jar logs location file after the call goes to dependency jar method.
I have mentioned below both the log4j2.properties file content:
Main application log4j2.properties:
status = warn
appender.console.type = Console
appender.console.name = LogToConsole
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
# Rotate log file
appender.rolling.type = RollingFile
appender.rolling.name = LogToRollingFile
appender.rolling.fileName = C:/LOG/Main.log
appender.rolling.filePattern = C:/LOG/Main.log.%d{yyyy-MM-dd}
appender.rolling.layout.type = PatternLayout
appender.rolling.layout.pattern = %d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
appender.rolling.policies.type = Policies
appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling.strategy.type = DefaultRolloverStrategy
# Log to console and rolling file
logger.app.name = com.indra
logger.app.level = debug
logger.app.appenderRef.rolling.ref = LogToRollingFile
logger.app.appenderRef.console.ref = LogToConsole
logger.app.additivity=false
rootLogger.level = info
rootLogger.appenderRef.rolling.ref = LogToRollingFile
Dependency jar log4j2.properties:
status = warn
appender.console.type = Console
appender.console.name = LogToConsole
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %5p | %d | %t | %F | %L | %m%n
# Rotate log file
appender.rolling.type = RollingFile
appender.rolling.name = LogToRollingFile
appender.rolling.fileName = C:/LOG2/dependency.log
appender.rolling.filePattern = C:/LOG2/dependency.log.%d{yyyy-MM-dd}
appender.rolling.layout.type = PatternLayout
appender.rolling.layout.pattern = %5p | %d | %t | %F | %L | %m%n
appender.rolling.policies.type = Policies
appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling.strategy.type = DefaultRolloverStrategy
# Log to console and rolling file
logger.app.name = com.indra
logger.app.level = info
logger.app.appenderRef.rolling.ref = LogToRollingFile
logger.app.appenderRef.console.ref = LogToConsole
logger.app.additivity=false
rootLogger.level = debug
rootLogger.appenderRef.rolling.ref = LogToRollingFile
Please help me with this!
All the loggers of an application use the same logger context with a single configuration (cf. architecture).
Since logging configuration is up to user of the application, not the developer, library jars should not contain any Log4j2 configuration file nor should they depend upon log4j-core (they should only depend upon log4j-api, cf. API separation).
Your application can contain a Log4j2 configuration and log4j-core to provide a default configuration to the application user. However the configuration should not use paths specific to your system.
Conventionally a loggers name is equal to the fully qualified name of the class that uses it, which allows you to easily split your library's and main application logs. If all the packages in your library start with com.indra.dependency you can use a single configuration similar to this:
appender.1.type = RollingFile
appender.1.name = main
...
appender.2.type = RollingFile
appender.2.name = dependency
...
logger.1.name = com.indra
logger.1.appenderRef.1.ref = main
logger.2.name = com.indra.dependency
logger.2.appenderRef.1.ref = dependency
logger.2.additivity = false
I need to use Python Multiprocessing for some CPU Bound task. When I run my app and new process is created I see in kivy log that it imports a kivy.deps and some more stuff again. It takes a lot of time and it looks like window is created again. Why is that happening? I expected the app to just create new process and do only what MyProcess class is for.
LOG
[INFO ] [deps ] Successfully imported "kivy_deps.angle" 0.2.0
[INFO ] [deps ] Successfully imported "kivy_deps.glew" 0.2.0
[INFO ] [deps ] Successfully imported "kivy_deps.sdl2" 0.2.0
[INFO ] [Kivy ] v1.11.1
[INFO ] [Kivy ] Installed at "C:\Users\Patryk\PycharmProjects\pythonProject\Random\venv_python\lib\site-packages\kivy\__init__.py"
[INFO ] [Python ] v3.6.3 (v3.6.3:2c5fed8, Oct 3 2017, 17:26:49) [MSC v.1900 32 bit (Intel)]
[INFO ] [Python ] Interpreter at "C:\Users\Patryk\PycharmProjects\pythonProject\Random\venv_python\Scripts\python.exe"
[INFO ] [Factory ] 184 symbols loaded
[INFO ] [Image ] Providers: img_tex, img_dds, img_sdl2, img_pil, img_gif (img_ffpyplayer ignored)
[INFO ] [Window ] Provider: sdl2
[INFO ] [GL ] Using the "OpenGL" graphics system
[INFO ] [GL ] GLEW initialization succeeded
[INFO ] [GL ] Backend used <glew>
[INFO ] [GL ] OpenGL version <b'4.6.0 NVIDIA 441.66'>
[INFO ] [GL ] OpenGL vendor <b'NVIDIA Corporation'>
[INFO ] [GL ] OpenGL renderer <b'NVS 510/PCIe/SSE2'>
[INFO ] [GL ] OpenGL parsed version: 4, 6
[INFO ] [GL ] Shading version <b'4.60 NVIDIA'>
[INFO ] [GL ] Texture max size <16384>
[INFO ] [GL ] Texture max units <32>
[INFO ] [Window ] auto add sdl2 input provider
[INFO ] [Window ] virtual keyboard not allowed, single mode, not docked
[INFO ] [Base ] Start application main loop
[**IMPORTING DEPS AGAIN**]
[INFO ] [deps ] Successfully imported "kivy_deps.angle" 0.2.0
[INFO ] [deps ] Successfully imported "kivy_deps.glew" 0.2.0
[INFO ] [deps ] Successfully imported "kivy_deps.sdl2" 0.2.0
[INFO ] [Kivy ] v1.11.1
[INFO ] [Kivy ] Installed at "C:\Users\Patryk\PycharmProjects\pythonProject\Random\venv_python\lib\site-packages\kivy\__init__.py"
[INFO ] [Python ] v3.6.3 (v3.6.3:2c5fed8, Oct 3 2017, 17:26:49) [MSC v.1900 32 bit (Intel)]
[INFO ] [Python ] Interpreter at "C:\Users\Patryk\PycharmProjects\pythonProject\Random\venv_python\Scripts\python.exe"
[INFO ] [Factory ] 184 symbols loaded
[INFO ] [Image ] Providers: img_tex, img_dds, img_sdl2, img_pil, img_gif (img_ffpyplayer ignored)
MP RUN
[INFO ] [WindowSDL ] exiting mainloop and closing.
[INFO ] [Base ] Leaving application in progress...
Process finished with exit code 0
MY CODE
from kivy.app import App
from kivy.lang.builder import Builder
from kivy.uix.boxlayout import BoxLayout
from multiprocessing import Process
kv = '''
MyWindow:
'''
class MyProcess(Process):
def __init__(self):
super().__init__()
def run(self):
print('MP RUN')
class MyWindow(BoxLayout):
def __init__(self, *args, **kwargs):
super().__init__(**kwargs)
mp = MyProcess()
mp.start()
class MyApp(App):
def build(self):
return Builder.load_string(kv)
if __name__ == '__main__':
MyApp().run()
I'm not sure offhand exactly what the new process inherits, but in this case you're clearly using it to run Python code that includes your kivy imports. Maybe you should use a separate file that doesn't include the imports you don't want, but I'm not sure what is the intention. The multiprocessing documentation may say something about the best way to avoid it.
kafka2.12-2.4.0 confluent5.4.1
I am trying to use Confluent's Schema-register.
But when I start schema-register and connect-Distributed.
Connect logs did not report errors.
connect-avro-distributed.properties
key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://k2:8081
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://k2:8081
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
plugin.path=/usr/local/tools/confluent-5.4.1/share/java,/usr/local/tools/kafka/kafka_2.12-2.4.0/plugin
I have configured the confluent jar address so that connect can find the class. (plugin.path)
But when I POST the conector request.
{
"name": "dbz-mysql-avro-connector",
"config": {
"connector.class": "io.debezium.connector.mysql.MySqlConnector",
"tasks.max": "1",
"database.hostname": "xx.xx.xx.xx",
"database.port": "3306",
"database.user": "debezium",
"database.history.kafka.topic": "dbhistory.debezium.mysql.avro",
"database.password": "123456",
"database.server.id": "184124",
"database.server.name": "debezium",
"key.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"key.converter.schema.registry.url": "http://k2:8081",
"value.converter.schema.registry.url": "http://k2:8081",
"table.whitelist": "debeziumdb.hosttable",
"database.history.kafka.bootstrap.servers": "k1:9092,k2:9092,k3:9092"
}
}
Throw the Exception.
[2020-04-23 10:37:00,064] INFO Creating task dbz-mysql-avro-connector-0 (org.apache.kafka.connect.runtime.Worker:419)
[2020-04-23 10:37:00,065] INFO ConnectorConfig values:
config.action.reload = restart
connector.class = io.debezium.connector.mysql.MySqlConnector
errors.log.enable = false
errors.log.include.messages = false
errors.retry.delay.max.ms = 60000
errors.retry.timeout = 0
errors.tolerance = none
header.converter = null
key.converter = class io.confluent.connect.avro.AvroConverter
name = dbz-mysql-avro-connector
tasks.max = 1
transforms = []
value.converter = class io.confluent.connect.avro.AvroConverter
(org.apache.kafka.connect.runtime.ConnectorConfig:347)
[2020-04-23 10:37:00,065] INFO EnrichedConnectorConfig values:
config.action.reload = restart
connector.class = io.debezium.connector.mysql.MySqlConnector
errors.log.enable = false
errors.log.include.messages = false
errors.retry.delay.max.ms = 60000
errors.retry.timeout = 0
errors.tolerance = none
header.converter = null
key.converter = class io.confluent.connect.avro.AvroConverter
name = dbz-mysql-avro-connector
tasks.max = 1
transforms = []
value.converter = class io.confluent.connect.avro.AvroConverter
(org.apache.kafka.connect.runtime.ConnectorConfig$EnrichedConnectorConfig:347)
[2020-04-23 10:37:00,067] INFO TaskConfig values:
task.class = class io.debezium.connector.mysql.MySqlConnectorTask
(org.apache.kafka.connect.runtime.TaskConfig:347)
[2020-04-23 10:37:00,067] INFO Instantiated task dbz-mysql-avro-connector-0 with version 1.1.0.Final of type io.debezium.connector.mysql.MySqlConnectorTask (org.apache.kafka.connect.runtime.Worker:434)
[2020-04-23 10:37:00,067] ERROR Failed to start task dbz-mysql-avro-connector-0 (org.apache.kafka.connect.runtime.Worker:470)
java.lang.NoClassDefFoundError: io/confluent/connect/avro/AvroConverterConfig
at io.confluent.connect.avro.AvroConverter.configure(AvroConverter.java:61)
at org.apache.kafka.connect.runtime.isolation.Plugins.newConverter(Plugins.java:293)
at org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:440)
at org.apache.kafka.connect.runtime.distributed.DistributedHerder.startTask(DistributedHerder.java:1140)
at org.apache.kafka.connect.runtime.distributed.DistributedHerder.access$1700(DistributedHerder.java:125)
at org.apache.kafka.connect.runtime.distributed.DistributedHerder$13.call(DistributedHerder.java:1155)
at org.apache.kafka.connect.runtime.distributed.DistributedHerder$13.call(DistributedHerder.java:1151)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2020-04-23 10:37:00,071] INFO [Worker clientId=connect-1, groupId=connect-cluster] Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1125)
All jars are in this directory.
Now what can I do to allow the class to be introduced, or does the version of confluent not exist in the class?
Thanks.
I finally solved this exception.
I did not use the confluent platform, just installed the schema-registry component.
To be precise, I only installed the community version and only activated the schema-registry component.
Then I downloaded the Avro jar package on the official website, and finally put it in the plugin completely, and started the connect successfully.
Confluent Avro jar address
And, I executed the following statement so that it can be read.
export CLASSPATH=/usr/local/tools/kafka/kafka_2.12-2.4.0/plugin/*
Looks like your kafka-connect-avro-converter is not compatible with other confluent jars. your Q does not list the kafka-connect-avro-converter jar as well. Can you add the correct version jar of kafka-connect-avro-converter in your classpath.
I am getting the following error when trying to run even the simplest of Kivy apps:
C:\Users\kprice\Desktop>python test.py
[INFO ] [Logger ] Record log in C:\Users\kprice\AppData\Roaming\SPB_16.6\.kivy\logs\kivy_17-03-03_57.txt
[INFO ] [Kivy ] v1.9.2.dev0, git-9f2daa0, 20170302
[INFO ] [Python ] v3.6.0 (v3.6.0:41df79263a11, Dec 23 2016, 08:06:12) [MSC v.1900 64 bit (AMD64)]
[INFO ] [Factory ] 193 symbols loaded
[INFO ] [Image ] Providers: img_tex, img_dds, img_sdl2, img_gif (img_pil, img_ffpyplayer ignored)
[INFO ] [Text ] Provider: sdl2
[INFO ] [OSC ] using <thread> for socket
[INFO ] [Window ] Provider: sdl2
[INFO ] [GL ] Using the "OpenGL" graphics system
[INFO ] [GL ] GLEW initialization succeeded
[INFO ] [GL ] Backend used <glew>
[INFO ] [GL ] OpenGL version <b'4.3.0 - Build 10.18.15.4256'>
[INFO ] [GL ] OpenGL vendor <b'Intel'>
[INFO ] [GL ] OpenGL renderer <b'Intel(R) HD Graphics 4400'>
[INFO ] [GL ] OpenGL parsed version: 4, 3
[INFO ] [GL ] Shading version <b'4.30 - Build 10.18.15.4256'>
[INFO ] [GL ] Texture max size <16384>
[INFO ] [GL ] Texture max units <32>
[INFO ] [Shader ] fragment shader: <b"WARNING: 0:6: '' : #version directive missing">
[INFO ] [Shader ] vertex shader: <b"WARNING: 0:6: '' : #version directive missing">
[INFO ] [Window ] auto add sdl2 input provider
[INFO ] [Window ] virtual keyboard not allowed, single mode, not docked
Traceback (most recent call last):
File "test.py", line 13, in <module>
MyApp().run()
File "C:\Users\kprice\AppData\Local\Programs\Python\Python36\lib\site-packages\kivy\app.py", line 802, in run
root = self.build()
File "test.py", line 10, in build
return Label(text='Hello world')
File "C:\Users\kprice\AppData\Local\Programs\Python\Python36\lib\site-packages\kivy\uix\label.py", line 291, in __init__
self._trigger_texture()
File "kivy\_clock.pyx", line 75, in kivy._clock.ClockEvent.__call__ (kivy\_clock.c:2223)
RuntimeError: release unlocked lock
The code for the app is as follows:
from kivy.app import App
from kivy.uix.label import Label
class MyApp(App):
def build(self):
return Label(text='Hello world')
if __name__ == '__main__':
MyApp().run()
Any ideas on why even the most basic program seems to crash?
That was a typo for a change in the kivy.clock code. It basically called the method twice, which it shouldn't (no thread lock to release).
I'm not sure if it's because of Cython or because of Windows, but I believe that error should return a ThreadError. There is too small amount of RuntimeErrors in Kivy, so I'm not sure why does it return a wrong type of error.
I have a Spatialite database generated by a Django management command, which I'm embedding in an iOS app. The db opens perfectly, but I've found that querying the database on iOS only works if I'm running on a 64-bit device. Any other devices, simulated or otherwise, produce the output SQLITE_NOTADB on calling sqlite3_prepare_v2, with the error message file is encrypted or is not a database.
Obviously, the Mac that produced this database is a 64-bit machine, but SQLite databases should be bit-agnostic, so I don't see how this should be a problem. Perhaps this doesn't apply to Spatialite databases? Are there any flags I can use with any of the SQLite functions (maybe sqlite3_open or sqlite3_prepare_v2 or maybe a pragma command) to make it read the file in the same way as the 64-bit arch iOS does? Or perhaps there is a way to generate the Spatialite db from django in a more platform-compatible format? Any suggestions here are welcome.
Here's a snippet of my code, if anyone can find any obvious problems:
#implementation DataModel
#synthesize db;
- (id) init {
self = [super init];
if (self != nil) {
spatialite_init(1);
sqlite3 *newDbConnection;
if (sqlite3_open([[self spatialiteDbPath] UTF8String], &newDbConnection) == SQLITE_OK) {
NSLog(#"Database opened successfully");
db = newDbConnection;
} else {
NSLog(#"Error opening database");
db = NULL;
}
}
return self;
}
- (NSArray *) getLockupsForRegion: (MKCoordinateRegion) region {
NSMutableArray *lockups = [[NSMutableArray alloc] init];
NSString *query = [NSString stringWithFormat:#"\
SELECT name, X(location) as lat, Y(location) as lon, \
covered, type, capacity \
FROM lockups_lockup \
WHERE WITHIN(location, GeomFromText('POLYGON((%f %f, %f %f, %f %f, %f %f))'));",
regionCorners[0].latitude, regionCorners[0].longitude,
regionCorners[1].latitude, regionCorners[1].longitude,
regionCorners[2].latitude, regionCorners[2].longitude,
regionCorners[3].latitude, regionCorners[3].longitude];
sqlite3_stmt *statement;
if(db &&
sqlite3_prepare_v2(db, [query UTF8String], -1, &statement, NULL) == SQLITE_OK) {
while(sqlite3_step(statement) == SQLITE_ROW) {
NSLog(#"A row");
}
} else {
NSLog(#"Query failed for reason: %s", sqlite3_errmsg(db));
NSLog(#"Sqlite version: %s", sqlite3_version);
}
return lockups;
}
#end
Launching the app and running on a non-64bit iOS platform getLockupsForRegion: produces the log output:
SpatiaLite version ..: 4.1.1 Supported Extensions:
- 'VirtualShape' [direct Shapefile access]
- 'VirtualDbf' [direct DBF access]
- 'VirtualText' [direct CSV/TXT access]
- 'VirtualNetwork' [Dijkstra shortest path]
- 'RTree' [Spatial Index - R*Tree]
- 'MbrCache' [Spatial Index - MBR cache]
- 'VirtualSpatialIndex' [R*Tree metahandler]
- 'VirtualFDO' [FDO-OGR interoperability]
- 'SpatiaLite' [Spatial SQL - OGC]
PROJ.4 version ......: Rel. 4.8.0, 6 March 2012
GEOS version ........: 3.4.2-CAPI-1.8.2 r3921
2013-12-31 00:29:39.567 App[8320:70b] Database opened successfully
2013-12-31 00:29:48.128 App[8320:70b] Query failed for reason: file is encrypted or is not a database
2013-12-31 00:29:48.710 App[8320:70b] Sqlite version: 3.8.1
And the log output under arm64:
SpatiaLite version ..: 4.1.1 Supported Extensions:
- 'VirtualShape' [direct Shapefile access]
- 'VirtualDbf' [direct DBF access]
- 'VirtualText' [direct CSV/TXT access]
- 'VirtualNetwork' [Dijkstra shortest path]
- 'RTree' [Spatial Index - R*Tree]
- 'MbrCache' [Spatial Index - MBR cache]
- 'VirtualSpatialIndex' [R*Tree metahandler]
- 'VirtualFDO' [FDO-OGR interoperability]
- 'SpatiaLite' [Spatial SQL - OGC]
PROJ.4 version ......: Rel. 4.8.0, 6 March 2012
GEOS version ........: 3.4.2-CAPI-1.8.2 r3921
2013-12-31 01:10:34.491 App[8548:70b] Database opened successfully
2013-12-31 01:10:44.913 App[8548:70b] A row
2013-12-31 01:10:44.913 App[8548:70b] A row
2013-12-31 01:10:44.914 App[8548:70b] A row
2013-12-31 01:10:44.914 App[8548:70b] A row
2013-12-31 01:10:44.915 App[8548:70b] A row
2013-12-31 01:10:44.915 App[8548:70b] A row
2013-12-31 01:10:44.916 App[8548:70b] A row
2013-12-31 01:10:44.916 App[8548:70b] A row
Versions:
Python: 2.7.5
Django: 1.6
PySqlite: 2.6.3
Sqlite (on machine generating Spatialite db): 3.8.1
Spatialite (on machine generating Spatialite db): 4.1.1
Sqlite (on iOS): 3.8.1
Spatialite (on iOS): 4.1.1
Any suggestions on this would be much appreciated!