How to make Flume Spooling Directory Source work with the Sub Directories of a folder too.
My source folder have several other folders too, I want my flume agent to look into these sub directories too for a file to dump it into sink.
Is there any way to do it?
The Spooling Directory won't check any of subdirectories, unless you explicitly configure it to check those subdirectories, eg:
a1.channels = ch-1
a1.sources = src-1 src-sub-1 src-sub-2
a1.sources.src-1.type = spooldir
a1.sources.src-sub-1.type = spooldir
a1.sources.src-sub-2.type = spooldir
a1.sources.src-1.channels = ch-1
a1.sources.src-sub-2.channels = ch-1
a1.sources.src-sub-1.channels = ch-1
a1.sources.src-1.spoolDir = /var/log/apache/flumeSpool
a1.sources.src-sub-1.spoolDir = /var/log/apache/flumeSpool/subdir
a1.sources.src-sub-2.spoolDir = /var/log/apache/flumeSpool/secondSubdir
In the currently release version of Flume (1.6.0) there isn't a way to do so however there is an issue being worked on to do this:
https://issues.apache.org/jira/browse/FLUME-1899
There is a patch available in the issue - this may/may not help you depending on whether you're able to build a custom Flume deployable.
a1.sources.src-1.recursiveDirectorySearch=true
It will check all sub-directories present in directory.
Related
I inadvertently added my new console application to TFS (I didn't even check it in) and now it tries to connect to TFS when I open the solution.
What do I need to remove TFS references from my console application?
Open up your solution (.sln) file in notepad. Remove the section that has references to Scc like below
GlobalSection(TeamFoundationVersionControl) = preSolution
SccNumberOfProjects = 2
SccEnterpriseProvider = {4CA58AB2-18FA-4F8D-95D4-32DDF27D184C}
SccTeamFoundationServer = http://mytfs:8080/tfs/ieb
SccLocalPath0 = .
SccProjectUniqueName1 = ConsoleApp3\\ConsoleApp3.csproj
SccProjectName1 = ConsoleApp3
SccLocalPath1 = ConsoleApp3
EndGlobalSection
Also, remove the .vssscc file from the directory where your Solution (.sln) file directory.
When running my Jenkins build i need to update the contents of a file with a version number in this case. I have come across a plugin called text-file-operations but rather than write a whole new file I thought it would be better to update.
In this example I have a podspec file located in the root of the project which just needs a version number updated with a variable I have created earlier in the process.
spec.version = '13.4.0'
I just need to convert that to
spec.version = "${VERSION_NUMBER}"
Is there a way to do this ?
Is this what you want then?
Groovy + how to append text in file ( new line )
f = new File('<filename>')
f.append("spec.version = ${VERSION_NUMBER}\n")
how to use flume for data streaming between two directories?
spool_dir.sources = src-1
spool_dir.channels = channel-1
spool_dir.sinks = sink-1
# source
spool_dir.sources.src-1.type = spooldir
spool_dir.sources.src-1.channels = channel-1
spool_dir.sources.src-1.spoolDir = /usr/lib/flume/source
#sink
spool_dir.sinks.sink-1.type = spooldir
spool_dir.sinks.sink-1.channels = channel-1
spool_dir.sinks.sink-1.spoolDir = /usr/lib/flume/sink
# Bind the source and sink to the channel
spool_dir.sources.src-1.channels = channel-1
spool_dir.sinks.sink-1.channel = channel-1
What i understand your asking is you want to monitor files coming into one folder and copy them to say another folder, if thats the case then your source looks good but for sink use file_Roll, instead of spool dir.
a1.channels = c1
a1.sinks = k1
a1.sinks.k1.type = file_roll
a1.sinks.k1.channel = c1
a1.sinks.k1.sink.directory = /var/log/flume
Basically in Flume Source and Sink are different interfaces, so you have to choose spool as source saying i want to monitor particular directory, but then i want to write into file or hdfs using one of the sink, so there is no spool sink instead there is file_roll sink, it might or might not work for you. But choose one of the flume sink as target http://flume.apache.org/FlumeUserGuide.html#flume-sinks
I am trying to set up Weceem using the source from GitHub. It requires a physical path definition for the uploads directory, and for a directory for appears to be used for writing searchable indexes. The default setting for uploads is:
weceem.upload.dir = 'file:/var/www/weceem.org/uploads/'
I would like to define those using relative paths like WEB-INF/resources/uploads. I tried a methodology I have used previously for accessing directories with relative path like this:
File uploadDirectory = ApplicationHolder.application.parentContext.getResource("WEB-INF/resources/uploads").file
def absoluteUploadDirectory = uploadDirectory.absolutePath
weceem.upload.dir = 'file:'+absoluteUploadDirectory
However, 'parentContext' under ApplicationHolder.application is NULL. Can anyone offer a solution to this that would allow me to use relative paths?
look at your Config.groovy you should have (maybe it is commented)
// locations to search for config files that get merged into the main config
// config files can either be Java properties files or ConfigSlurper scripts
// "classpath:${appName}-config.properties", "classpath:${appName}-config.groovy",
grails.config.locations = [
"file:${userHome}/.grails/${appName}-config.properties",
"file:${userHome}/.grails/${appName}-config.groovy"
]
Create Conig file in deployment server
"${userHome}/.grails/${appName}-config.properties"
And define your prop (even not relative path) in that config file.
To add to Aram Arabyan's response, which is correct, but lacks an explanation:
Grails apps don't have a "local" directory, like a PHP app would have. They should be (for production) deployed in a servlet container. The location of that content is should not be considered writable, as it can get wiped out on the next deployment.
In short: think of your deployed application as a compiled binary.
Instead, choose a specific location somewhere on your server for the uploads to live, preferably outside the web server's path, so they can't be accessed directly. That's why Weceem defaults to a custom folder under /var/www/weceem.org/.
If you configure a path using the externalized configuration technique, you can then have a path specific to the server, and include a different path on your development machine.
In both cases, however, you should use absolute paths, or at least paths relative to known directories.
i.e.
String base = System.properties['base.dir']
println "config: ${base}/web-app/config/HookConfig.grooy"
String str = new File("${base}/web-app/config/HookConfig.groovy").text
return new ConfigSlurper().parse(str)
or
def grailsApplication
private getConfig() {
String str = grailsApplication.parentContext.getResource("config/HookConfig.groovy").file.text
return new ConfigSlurper().parse(str)
}
I have a file "settings.ini" which needs to reside next to the Qt executable.
I can add a custom build step for this in Qt Creator which calls something like this:
copy %{sourceDir}/settings.ini %{buildDir}/settings.ini
This works great so far, but I'd like to include this in the *.pro file so I can put this up in our SVN too.
How can I do this using qmake/.pro-files only?
To copy %{sourceDir}/settings.ini to the build directory without requiring to call make install use:
copydata.commands = $(COPY_DIR) $$PWD/settings.ini $$OUT_PWD
first.depends = $(first) copydata
export(first.depends)
export(copydata.commands)
QMAKE_EXTRA_TARGETS += first copydata
$$PWD is the path of current .pro file. If your settings.ini file is not located in the same directory than the project file, then use something like $$PWD/more_dirs_here/settings.ini
Note: I found this solution here. I recommend to read the whole article as it explains how it works.
You probably want to use the INSTALLS keyword in QMake. It will require you to run make install after your build, but it does work cross-platform.
install_it.path = %{buildDir}
install_it.files += %{sourceDir}/settings.ini
INSTALLS += install_it
for osx bundles you can handle it this way
see Resource files in OS X bundle
add this to you project file:
APP_QML_FILES.files = path/to/file1.qml path/to/file2.qml
APP_QML_FILES.path = Contents/Resources
QMAKE_BUNDLE_DATA += APP_QML_FILES
this example copies the files to Contents/Resources
Compatible with Windows and Mac OSX Dev environments:
Change {AppName} to respective application name
# Define mac/windows specific target dirs
TARGETDIR = ''
macx {
TARGETDIR += $$OUT_PWD/{AppName}.app/Contents/MacOS/
}
else {
TARGETDIR += $$OUT_PWD
}
# Directories do not exist for the first build
# Without mkdata, build is successful after 5 tries. To avoid, use mkdata
mkdata.commands = $(MKDIR) $${TARGETDIR}
copydata.commands = $(COPY_FILE) $$PWD/settings.ini $${TARGETDIR}
first.depends = $(first) mkdata copydata
export(first.depends)
export(mkdata.commands)
export(copydata.commands)
QMAKE_EXTRA_TARGETS += first mkdata copydata
Happy to add Unix support if someone posts Unix solution in the comments.