I have a folder path stored in the property nightly.basepath (here the value would be /path/to/my/nightly) :
/path/to/my/nightly
|- build_20100310
|- build_20100311
|- build_20100312
I want to get the folder of the lastly generated build in the nightly.basepath folder, and store the path to this last folder in the nightly.last property. (So here, the value of nightly.last should be /path/to/my/nightly/build_20100312.)
Assume the alphabetical order reflects chronological order.
What is the easiest way to do that in Ant ?
You could try the ant-contrib TimestampSelector. I haven't used that particular task but I have used other ones in ant-contrib and haven't had any issues with them.
Related
I am passing in an wilcard match string as gs://dev-test/dev_decisions-2018-11-13*/. And i am passing to TextIO as below.
p.apply(TextIO.read().from(options.getLocalDate()))
Now i want to read all folders from the bucket named dev-test and filter and only read files from the latest folder. Each folder has a name with timestamp appended to it.
I am new to dataflow and not sure how would I go about doing this.
Looking at the JavaDoc here it seems as though we can code:
String folder = // The GS path to the latest/desired folder.
PCollection<String> myPcollection = p.apply(TextIO.Read.from(folder+"/*")
The resulting PCollection will thus contain all the text lines from all the files in the specified folder.
Assuming you can have multiple folders in the same bucket with the same date prefix/suffix as for example "data-2018-12-18_part1", "data-2018-12-18_part2" etc, the following will work. Its a python example but it works for Java as well. You will just need to get the date formatted as per your folder name and construct the path accordingly.
# defining the input path pattern
input = 'gs://MYBUCKET/data-' + datetime.datetime.today().strftime('%Y-%m-%d') + '*\*'
(p
| 'ReadFile' >> beam.io.ReadFromText(input)
...
...
it will read all the files from all the folders matching the pattern
If you know that the most recent folder will always be today's date, you could use a literal string as in Tanveer's answer. If you don't know that and need to filter the actual folder names for the most recent date, I think you'll need to use FileIO.match to read file and directory names, and then collect them all to one node in order to do figure out which is the most recent folder, then pass that folder name into TextIO.read().from().
The filtering might look something like:
ReduceByKey.of(FileIO.match("mypath"))
.keyBy(e -> 1) // constant key to get everything to one node
.valueBy(e -> e)
.reduceBy(s -> ???) // your code for finding the newest folder goes here
.windowBy(new GlobalWindows())
.triggeredBy(AfterWatermark.pastEndOfWindow())
.discardingFiredPanes()
.output()
I have a directory with a list of property files for different environment(DEV/STG/QA etc.,)
I want to call an Ant target in a loop with each of this file. How do I do this. I downloaded ant-contrib and tried using the foreach but I couldn't find any example where I can read property files one at a time and call the target. Any suggestions?
I have been looking at a lot of samples on this site and online, nothing seem to match my requirement.
As per apacche documentation : https://ant.apache.org/manual/Tasks/property.html
<property file="foo.properties"/>
reads a set of properties from a file called "foo.properties".
so, if you have my.var=25 , then ${my.var} will get you its value 25. For your requirement you can iterate through the files based on 'env' name & do required tasks.
I am working with Ant in Groovy, and I want to operate on all files in a directory that match certain criteria. I have the following:
def fileSet = ant.fileset(dir: srcDir, id: "files", includes: pattern){
date(datetime: time when: "before", pattern: datePattern)
}
ant.zip(destfile: "$destDir/$newZipFileName"){
fileset(refid: "files")
}
This works great on the files in the source directory. But how can I get it to also archive the directories in the source location, while maintaining the directory structure in the archive?
"**/*" as my pattern got all the files, but flattened them into one directory
I've tried using a dirset instead of a fileset, but to no avail.
How can I zip everything up while mirroring the existing directory structure in the archive?
(To clarify, I define the fileset outside of my task because I have several tasks that might use the same fileset, and it gets passed around).
I have a couple of job in Jenkins that archive artifact from the source tree for another job (some unit tests or alike). I have the current situation :
top_dir
\scripts_dir
\some_files
\dir1
\dir2
\dir3
\other_dir
I would like to archive all that is in "top_dir" including the files in "scripts_dir", but not the subdirectories "dir1, dir2,...", which I do not know the name, that are in "scripts_dir". These subdirs are actually Windows directory joints that point to other places on the disk, and I do not want them to be copied.
How do I achieve this with the inculde/excludes pattern of Jenkins ?
I already tried, having include=top_dir/ , exclude=
**/scripts_dir/*/
**/scripts_dir/*/**
**/scripts_dir/**/*
but it always exculdes the whole "scripts_dir" folder.
Finally, by using brute force, I found that the following expression does exclude all the files in the subdirectories of scripts_dir (whatever symlink or not), then removing these subdirs, while keeping the files directly in scripts_dir :
**/scripts_dir/**/*/*/
Thanks for the help anyway.
Reading the ANT manual, there an followsymlinks attribute that defaults to true. You said those things you want to exclude are symlinks (although i am not sure if this will work with Windows joints). Try adding followsymlinks=false
Another solution: if all your files under scripts_dir have a set number of characters in the extension, you can put that into your include statement. This will only pickup files with extensions of 3 characters:
**/scripts_dir/*.???
More on this here
Is it possible to use Ant to copy all the files (not folders) from a hierarchy of folders into one destination folder?
For instance, I have a tree like this:
res
|-images
| |-fg.png
| +-bg.png
+-sounds
+-music.mp3
And I would like a result like this:
data
|-fg.png
|-bg.png
+-music.mp3
The way things are named, filename conflicts are not an issue.
Is this possible to do with Ant?
You can use the flatten attribute of the copy task to accomplish this. You can read more at http://ant.apache.org/manual/Tasks/copy.html.