I need to copy the files which are generated within Grails to Hadoop dynamically. How will I write code for this in Grails? Whenever a file is generated it should be copied into Hadoop. If the incoming file already exists, it should get updated in Hadoop.
I used shell script to connect grails and hadoop.
I had all the commands to run hadoop jobs in myjob.sh (Workflow Script)
And i added the code to execute shell script in my controller
def scriptCom="/folderlocation/shellscript.sh"
println "[[Running $scriptCom]]"
def proc = scriptCom.execute()
def oneMinute = 60000
proc.waitForOrKill(oneMinute)
if(proc.exitValue()!=0){
println "[[return code: ${proc.exitValue()}]]"
println "[[stderr: ${proc.err.text}]]"
return null
}else{
println "[[stdout:$revisionid]]"
return proc.in.text.readLines()
}
Related
I am trying to script a groovy script which copies a complete folder with all subfolder and jobs to the actual folder, where the script is executed.
Here you can see how my folderstructure looks like.
--> Templ
|-->Folder
|-->Folder
|-->Subfolder
|-->Subsubfolder
|-->Subfolder
|-->Folder
-->Execution 2020
|-->Copyscript
I tried with different Plug-Ins like Jobcopy Builder.
Finally I tried with groovy scrips but nothing seems to work.
the simplest way to use AntBuilder
def ant = new AntBuilder()
ant.copy(todir: myDir) {
fileset(dir: "src/test") {
include(name: "**/*.java")
}
}
example taken from here
http://docs.groovy-lang.org/latest/html/documentation/ant-builder.html
to see all parameters of ant copy command see documentation:
https://ant.apache.org/manual/Tasks/copy.html
I am preparing a Jenkins pipeline script in Groovy language. I would like to move all files and folders to another location. As Groovy supports Java so I used below java code to perform the operation.
pipeline{
agent any
stages{
stage('Organise Files'){
steps{
script{
File sourceFolder = new File("C:\\My-Source");
File destinationFolder = new File("C:\\My-Destination");
File[] listOfFiles = sourceFolder.listFiles();
echo "Files Total: " + listOfFiles.length;
for (File file : listOfFiles) {
if (file.isFile()) {
echo file.getName()
Files.copy(Paths.get(file.path), Paths.get("C:\\My-Destination"));
}
}
}
}
}
}
}
This code throws the bellow exception:
groovy.lang.MissingPropertyException: No such property: Files for
class: WorkflowScript
I tried with below code too, but it's not working either.
FileUtils.copyFile(file.path, "C:\\My-Destination");
Finally, I did try with java I/O Stream to perform the operation and the code is bellow:
def srcStream = new File("C:\\My-Source\\**\\*").newDataInputStream()
def dstStream = new File("C:\\My-Destination").newDataOutputStream()
dstStream << srcStream
srcStream.close()
dstStream.close()
But it's not working either and throws the below exception:
java.io.FileNotFoundException: C:\My-Source (Access is denied)
Can anyone suggest me how to solve the problem and please also let me know how can I delete the files from the source location after copy or move it? One more thing, during the copy can I filter some folder and files using wildcard? Please also let me know that.
Don't execute these I/O functions using plain Java/Groovy. Even if you get this running, this will always be executed on the master and not the build agents. Use pipeline steps also for this, for example:
bat("xcopy C:\\My-Source C:\\My-Destination /O /X /E /H /K")
or using the File Operations Plugin
fileOperations([fileCopyOperation(
excludes: '',
flattenFiles: false,
includes: 'C:\\My-Source\\**',
targetLocation: "C:\\My-Destination"
)]).
I assume I didn't hit the very right syntax for Windows paths here in my examples, but I hope you get the point.
I need to monitor what are the changes going with a job on jenkins(update the changes to a file). Need to list the env variables of a job. JOB_NAME,BUILD_NUMBER,BUILD_STATUS,GIT_URL for that build(all the builds of a job). I didn't find out a good example with the groovy. What is the best way to fetch all the info?
build.getEnvironment(listener) should get you what you need
Depending on what you would like to achieve there are at least several approaches to retrieve and save environment variables for:
current build
all past builds
Get environments variables for current build (from slave)
Execute Groovy script
// Get current environment variables and save as
// a file in $WORKSPACE.
new File(".",'env.txt').withWriter('utf-8') { writer ->
System.getenv().each { key, value ->
writer.writeLine("${key}:${value}")
}
}
Using Groovy Plug-in.
Get environment variables for current build (from master)
Execute system Groovy script
// Get current environment variables and save as
// a file in $WORKSPACE.
import hudson.FilePath
def path = "env-sys.txt"
def file = null
if (build.workspace.isRemote()) {
file = new FilePath(build.workspace.channel, build.workspace.toString() + "/" + path)
} else {
file = new FilePath(build.workspace.toString() + "/" + path)
}
def output = ""
build.getEnvironment(listener).each { key, value ->
output += "${key}:${value}\n"
}
file.write() << output
Using Groovy Plug-in.
Environment variables returned by Groovy scripts are kept in map. If you don't need all of them, you can access individual values using standard operators/methods.
Get environment variables for all past builds (from master)
This approach expecst that you have installed EnvInject Plug-in and have access to $JENKINS_HOME folder:
$ find . ${JENKINS_HOME}/jobs/[path-to-your-job] -name injectedEnvVars.txt
...
ps. I suspect that one could analyze EnvInject Plug-in API and find a way to extract this information directly from Java/Groovy code.
Using EnvInject Plug-in.
To look for only specific variables you can utilize find, grep and xargs tools .
You can use below script to get the Environment Variables
def thread = Thread.currentThread()
def build = thread.executable
// Get build parameters
def buildVariablesMap = build.buildVariables
// Get all environment variables for the build
def buildEnvVarsMap = build.envVars
String jobName = buildEnvVarsMap?.JOB_NAME // This is for JOB Name env variable.
Hope it helps!
I am using the grails-cdn-asset-pipline plugin. I've gone through the installation and configuration steps on GitHub and I reach the usage section which says
Add this command to your build process (usually before war generation and deployment).
// If all the settings are defined in your Config.groovy
grails asset-cdn-push
// Or
grails asset-cdn-push --provider=S3 --directory=my-bucket --gzip=true --storage-path=some-prefix --expires=365 --region=eu-west-1 --access-key=$MY_S3_ACCESS_KEY --secret-key=$MY_S3_SECRET_KEY
Where in my project do I put this command?
Is it something that I can do within the context of my project, or do I need to keep it separate in another build process and run it in an environment like Jenkins?
In _Events.groovy, I tried to invoke the script in the eventCreateWarStart, but I am having no luck there. (Code taken from this question)
eventCreateWarStart = { warName, stagingDir ->
def pluginManager = PluginManagerHolder.pluginManager
def plugin = pluginManager.getGrailsPlugin("cdn-asset-pipline")
def pluginDir = plugin.descriptor.file.parentFile
Map<String, String> env = System.getenv()
final processBuilder = new ProcessBuilder()
processBuilder.directory(new File("${cdnAssetPipelinePluginDir}/scripts"))
processBuilder.command([env['GRAILS_HOME']+"/bin/grails","cdn-asset-push"])
println processBuilder.directory()
Process proc = processBuilder.start()
proc.consumeProcessOutput(out, err)
proc.waitFor()
}
This link explains the run-script functionality which was merged into Grails 1.3.6. But I ran into the same problem of not knowing where to run it automatically.
Our company's Jenkins has master and two slave nodes. I am writing plugin for it. One of the things for plugin to do is to checkout some files from svn. This action cannot be extracted from plugin.
To do this I execute console command "svn checkout" from java code of my plugin. The problem is that files from svn are checked out to master, rather than to slave nodes.
How can I make files be checked out to slave?
First you have these objects, usually received as parameters for perform method:
Launcher launcher;
AbstractBuild<?, ?> build;
BuildListener listener;
Then you have created and added arguments to an argumentListBuilder, maybe something like:
ArgumentListBuilder command = new ArgumentListBuilder();
command.addTokenized("xcopy /?");
Then, do something like:
ProcStarter ps = launcher.new ProcStarter();
ps = ps.cmds(command).stdout(listener);
ps = ps.pwd(build.getWorkspace()).envs(build.getEnvironment(listener));
Proc proc = launcher.launch(ps);
int retcode = proc.join();
ProcStarter will run the command at the node specified by the launcher instance. But please at least glance over the javadocs of all above classes before using, above is not direct copy-paste from working code.
Here is code based on Hyde's answer, suitable for the Groovy script console (at /script)
def static Run(nodeName, runCommand)
{
def output = new java.io.ByteArrayOutputStream();
def listener = new hudson.util.StreamTaskListener(output);
def node = hudson.model.Hudson.instance.getNode(nodeName);
def launcher = node.createLauncher(listener);
def command = new hudson.util.ArgumentListBuilder();
command.addTokenized(runCommand);
def ps = launcher.launch();
ps = ps.cmds(command).stdout(listener);
// ps = ps.pwd(build.getWorkspace()).envs(build.getEnvironment(listener));
def proc = launcher.launch(ps);
int retcode = proc.join();
return [retcode, output.toString()]
}
// for (aSlave in hudson.model.Hudson.instance.slaves) {
(recode, output) = Run("build-slave9", "xcopy /?");
println output;
(Caveats: untested for programs that read stdin. Note the ByteArrayOutputStream, so don't run programs with very long output. Untested for non-ASCII output.)