I have a Jenkins job that can process any number of files. However the File Parameter only lets the user select a single file.
Is there a way to select an arbitrary number of files or to select a directory? (I can work out the appriopriate files given a directory).
There seems to be no option for selecting a directory with a parameter.
Asking the user to enter the full path in a String parameter is very unfriendly.
I did consider using a File Parameter to select one file in the directory and then extracting the directory from this but in fact you just get the single file uploaded - there doesn't appear to be any directory information.
(Note: it isn't an option to run the job multiple times once for each file, the files need to be processed as a set.)
Any ideas?
Related
I need to read csv files inside a folder. New csv files are generated every time a user submits a form. I'm using the "file" input plugin to read the data and send it to Influxdb. These steps are working fine.
The problem is that the same file is read multiple times every data collection interval. I was thinking of a solution where I could move the file that was read to a different folder, but I couldn't do that with Telegraf's "exec" output plug.
ps: I can't change the way csv files are generated.
Any ideas on how to avoid reading the same csv file multiple times?
As you discovered file input plugin is used to read entire files at each collection interval.
My suggestion is for you to instead use the directory monitor input plugin. This will read files in a directory, monitor the directory for new files, and parse the ones that have not already been picked up yet. There are some configuration settings in that plugin that make it easier to time when new files are read as well.
Another option is to use the tail input plugin which will tail a file and only read new updates to that file as things come. However, I think the directory monitor is more likely something you are after for your scenario.
Thanks!
I currently need to be able to allow a user to upload a file to Jenkins for the job to then use.
I understand that the File parameter can allow me to do so, but how about if the user needs to add 100 files?
Is there a way to either:
Allow multiple files to be uploaded at once?
Upload a folder instead and then use my script to loop through the folder?
As a side note, the filenames need to have the same name as they had before they are uploaded too. When uploading one file via the File parameter, I could work around fixing that. With 100 files, maybe less so...
Environment:IIB9 broker on windows
SFTP server is on windows
We have requirements to process a batch of files generated by backend system in sequential order (i.e FIFO). A batch can have multiple files.
All the files are placed in the IIB source directory from where FileInputNode is polling using move command.
I Want to know if FileInputNode is capable of picking up files in the order they were created by backend system.
Thanks,
Below is the extract from help contents where in it says that FileInput node reads the file in the order they were created by default (oldest are picked up first).
How the broker reads a file at the start of a flow
The FileInput node processes messages that are read from files. The
FileInput node searches a specified input directory (in the file
system that is attached to the broker) for files that match specified
criteria. The node can also recursively search the input directory's
subdirectories.
Files that meet the criteria are processed in age order, that is the
oldest files are processed first regardless of where they appear in
the directory structure
.
https://www.ibm.com/support/knowledgecenter/en/SSMKHH_9.0.0/com.ibm.etools.mft.doc/ac55280_.htm
Yes you can loop in your messageflow and set the filepath of the file you want to load, one at a time. Feel free to ask for information if needed.
For the order, I have to check(you can still call a java node that will list the files in the order you want)
Regards
I am working on product testing automation. I am using Jenkins to create a job that will first browse some file in a directory. But I got a problem with Conditional Step in File Exist condition. It is not working when I search *.job file, it only works with specific file name I put. Sorry I don't have enough reputation to post picture :)
Anyone can help me solve this issue. Thank so much for reading
I don't think File Exists supports more than one file, you could try the "File match condition". Definitions below:
File exists Run if the file exists.
The file is expanded using the Token Macro Plugin. If the file is
relative, then it is relative to the Base directory. If the file is
absolute, then it will be tested on the Node that contains the Base
directory (the master for $JENKINS_HOME and Artifacts dir).
Files match Run if one or more files match the selectors.
Separate multiple includes or excludes patterns with a comma. e.g.
Includes "target/site/**/*.html,target/site/images/" Excludes
"**/*.gif" If no includes pattern is configured, then the pattern
"**" will be used, which matches all files in the directory and all
sub directories. The includes and excludes are Ant Patterns.
I have lots of files of same type like .xml in a folder. How can I select this folder from interface and iterate over each file and send this to appropriate tables of database.
Thanks
Sonu
Do you always put the files in the same directory? For example, if you generate these files in some other system, then just want the data imported into your application you could:
Create a job that runs every X minutes
It iterates over each file in the directory and parses the XML, creating and saving the objects to the database
Shifts or deletes the files when it has processed them
Jobs are a Grails concept/plugin: http://www.grails.org/Job+Scheduling+(Quartz)
Processing XML is easy in Groovy - you have many options, depends on your specific scenario - http://groovy.codehaus.org/Processing+XML
Processing files is also trivial - http://groovy.codehaus.org/groovy-jdk/java/io/File.html#eachFile(groovy.lang.Closure)
This is a high level overview. Hope it helps.