SSIS , counting files in a folder using script task, looping the task untill number of files reaches 25 - ssis-2012

Need to develop a package which should read number of files from ftp/folder. If the count is less than 25 , keep looping, Go to the next task once the count reaches 25.
What i tried is:
I used a script task, created few variable and variable have file count(Successfully). I used expression in precedence constraint for checking if the number of files in a particular folder is 25. if not it wont go to another task. What i cant do is keep the script task looping until the file count becomes 25. I tried using for each loop but couldn't get trough. please suggest.
Please have a look at this. i have 2 script tasks. first one counts and display the number of files in a folder. here is the script for that.
enter code here
public void Main()
{
// TODO: Add your code here
String FolderPath =
Dts.Variables["User::FolderPath"].Value.ToString();
string Prefix =
Dts.Variables["User::prefix"].Value.ToString();
Int32 FileCnt = 0;
var directory = new DirectoryInfo(FolderPath);
FileInfo[] files = directory.GetFiles(Prefix + "*");
//Declare and initilize variables
//Get one Book(Excel file at a time)
foreach (FileInfo file in files)
{
FileCnt += 1;
MessageBox.Show(file.Name);
}
MessageBox.Show(FileCnt.ToString());
Dts.Variables["User::FileCnt"].Value =
Convert.ToInt32(FileCnt);
}
#region ScriptResults declaration
/// <summary>
/// This enum provides a convenient shorthand within the scope of
this class for setting the
/// result of the script.
///
/// This code was generated automatically.
/// </summary>
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
#endregion
}
}
then i edited the precedence constraint and wrote and expression #FileCnt == 25, which means next script task will only be executed when the number of files in the folder is 25. what i want now is that the first script task should be running in a loop until the folder gets 25 file. i think we need to use foreachloop container here. did you get what im looking for now??

Related

Groovy read file line by line and store it into list

I have a file called apps.txt which has three app names in it
frontendapp
authorizationservice
connectorservice*
In jenkins pipeline i want to perform some operation on them one by one so i am trying to get them in groovy list using this code -
list = readFile.readFileLineByLine("${workspace}/apps.txt").collect {it}
for (item in list) {
println "I need to perform some operations on files"
}
But getting groovy.lang.MissingPropertyException.
If i use file class like this - list = new File("${workspace}/apps.txt").collect {it} then it search for a file on Jenkins master node only and i get fileNotFoundException.
If i use list = readFile("${workspace}/apps.txt").collect {it} then list gets values character by character. How i can get app names from apps.txt inorder to perform operation on each app.
Your attempts are close, but mix things up.
Those are working ways:
def list = new File("${workspace}/apps.txt").text.readLines()
Note the .text call inbetween.
def list = readFile("${workspace}/apps.txt").readLines()
Or with the helper Jenkins provides.
Side note: .collect{it} is just .collect() - and usually is only
needed to copy a list. Since the read lines from the file are already
eagerly read, those are copies.
This work for me in a windows worker
script {
def list = readFile("filesChanged").readLines()
for(item in list){
print item
bat('type '+item)
}
}

Jedis/Redis SocketTimeout exception on Lua scripts

We are using lua scripts to perform batch deletes of data on updates to our DB. Jedis executes the lua script using a pipeline.
local result = redis.call('lrange',key,0,12470)
for i,k in ipairs(result) do
redis.call('del',k)
redis.call('ltrim',key,1,k)
end
try (Jedis jedis = jedisPool.getResource()) {
Pipeline pipeline = jedis.pipelined();
long len = jedis.llen(table);
String script = String.format(DELETE_LUA_SCRIPT, table, len);
LOGGER.info(script);
pipeline.eval(script);
pipeline.sync();
} catch (JedisConnectionException e) {
LOGGER.info(e.getMessage());
}
For large ranges we notice that the lua scripts slow down and we get SocketTimeOutExceptions.
running redis-cli slowlog displays only the lua scripts that have taken too long to execute.
Is there a better way to do this? is my lua script blocking?
When I use just pipeline to do the batch deletes, the slowlog also returns slow queries.
try (Jedis jedis = jedisPool.getResource()) {
Pipeline pipeline = jedis.pipelined();
long len = jedis.llen(table);
List<String> queriesContainingTable = jedis.lrange(table,0,len);
if(queriesContainingTable.size() > 0) {
for (String query: queriesContainingTable) {
pipeline.del(query);
pipeline.lrem(table,1,query);
}
pipeline.sync();
}
} catch (JedisConnectionException e) {
LOGGER.info("CACHE INVALIDATE FAIL:"+e.getMessage());
}
slowlog is capable of storing top 128 slowlogs alone (can be changed in redis.conf slowlog-max-len 128). So your 1st model of using LUA script is surely a blocking one.
If you delete such a number (12470) one by one it is surely a blocking one as it take more time to complete. Out of the 2 models 2nd one is fine for me (using pipeline), because you avoid the iteration all you do is hitting del query for n times.
You can use del of multiple keys for every 100 or 1000 (whichever you feel as optimal after a small testing). You can group them to a pipeline altogether.
Or if you can do the same without atomicity, you can delete every 100 or 1000 keys at once in a loop, so that it wouldn't be a blocking call.
Try out with different combinations take the metrics and go with the optimized one.

Output a text file from Ranorex to include just a pass/fail result and a number

I am trying to get Ranorex to output a text file which will look like the following:
Pass
74
The pass/fail result will be obtained based on whether the test running has passed or failed. The number will be hardcoded to all I need to do is store that in a variable and include it in the output.
I would have thought it would have been simple but I'm struggling to get any help from Ranorex. I though I might be able to use the reporting function, change the output file type and alter the report structure but that didn't work either.
Although I am used to Ranorex and writing my own user code, I am new to adapting it in this way.
All my user code is written in C#
Can anyone offer any assistance?
Thanks!
Edit: So I've now managed to get Ranorex to output a text file and I can put any text into it, including a string stored in a variable.
However I'm struggling to store the pass/fail result of my test in a string that I can output.
I've discovered a way to do this however it relies on the following:-
The user code must be in separate test
This separate test must exist in a sibling test case to the one your main test is in
Both this test case and the case containing your main test must both be part of a parent test case
For example:
Parent TC
.....-AddUser TC
.........-MAIN TEST
.....-AddUser FailCheck
.........-USER CODE
You can then set your AddUser TC to 'Continue with sibling on fail'
The user code is as follows:
public static void Output()
{
string result = "";
ITestCase iCase = TestSuite.Current.GetTestCase("Add_User_Test"); // The name of your Test Case
if(iCase.Status == Ranorex.Core.Reporting.ActivityStatus.Failed){
result = "Failed"; }
if(iCase.Status == Ranorex.Core.Reporting.ActivityStatus.Success){
result = "Passed"; }
int testrunID = 79;
using (StreamWriter writer =
new StreamWriter("testresult.txt"))
{
writer.WriteLine(testrunID);
writer.WriteLine(result);
}
}
This will take the testrunID (specific to each test case) and the result of the test and output it to a text file.
The idea is then to read in the file with a custom java application I've developed and push the data into a test case management program such as QA Complete which can mark tests as Passed/Failed automatically
You can run the test suite directly using the TestSuiteRunner.Run() method. This will allow you to look at the return value of that directly and output pass or failure based on the return value.
http://www.ranorex.com/Documentation/Ranorex/html/M_Ranorex_Core_Testing_TestSuiteRunner_Run.htm
if(TestSuiteRunner.Run(typeof({testSuiteclass}),{Command Line Arguments})==0)
{
File.WriteLine("success");
}
else
{
File.WriteLine("failure");
}

What URL will get the status code (result) of the last Jenkins job?

I am wondering if anyone knows what URL is required (as a GET or POST) that will get the status code (result) of the last Jenkins job (when the build# is not known by the client calling the GET request)? I just want to be able to detect if the result was RED or GREEN/BLUE .
I have this code sample, but I need to adjust it so that it works for Jenkins, for this purpose (as stated above):
public class Main {
public static void main(String[] args) throws Exception {
URL url = new URL("http://localhost/jenkins/api/xml");
Document dom = new SAXReader().read(url);
for( Element job : (List<Element>)dom.getRootElement().elements("job")) {
System.out.println(String.format("Name:%s\tStatus:%s",
job.elementText("name"), job.elementText("color")));
}
}
}
Once I figure out the answer, I will share a full example of how I used it. I want to create a job that collects information on a test suite of 20+ jobs and reports on all of them with an email.
You can use the symbolic descriptor lastBuild:
http://localhost/jenkins/job/<jobName>/lastBuild/api/xml
The result element contains a string describing the outcome of the build.

How can I build all build.xml (.java) files under a directory

We needed to automate testing that all of the Java samples we ship compile properly. We need it to build all files without our listing each one. Listing each one means if someone forgets to add a new one (which will happen someday), explicit calls will miss it. By walking all build.xml files, we always get everything.
Doing this is pretty easy:
Install the samples on a clean VM (that we revert back to the snapshot for each test run).
Create a build.xml file that calls all the build.xml files installed.
Use ant to run the generated build.xml
Step 2 requires a means to generate the build.xml file. Is there any way to tell ant to run all build.xml files under a sub-directory or to create a build.xml that calls all the underlying build.xml files?
It sounds like what you want to do is run the same build process for a number of sub-projects that (hopefully) follow a standard layout pattern.
If that's the case, you can create a single build.xml that knows how to compile those projects, and make a top-level build script which finds all the sub directories, then calls the common build script in each one. Subant was taylor-made for this, and doesn't require a magic C# program to generate scripts in each directory.
We couldn't find anything so we wrote a program that creates a build.xml that calls all build.xml files under a directory. Full solution is at Windward Wrocks (my blog).
The code is (yep, using C# to create a build file for Java):
using System;
using System.IO;
using System.Text;
using System.Xml;
using System.Xml.Linq;
namespace BuildJavaTestScript
{
public class Program
{
/// <summary>
/// Build build.xml for all build.xml files in sub-directories.
/// </summary>
/// <param name="args">Optional: build.xml root_folder</param>
public static void Main(string[] args)
{
string projFile = Path.GetFullPath(args.Length > 0 ? args[0] : "build.xml");
string rootDirectory = Path.GetFullPath(args.Length > 1 ? args[1] : Directory.GetCurrentDirectory());
Console.Out.WriteLine(string.Format("Creating build file {0}", projFile));
Console.Out.WriteLine(string.Format("Root directory {0}", rootDirectory));
XDocument xdoc = new XDocument();
XElement elementProject = new XElement("project");
xdoc.Add(elementProject);
elementProject.Add(new XAttribute("name", "BuildAll"));
elementProject.Add(new XAttribute("default", "compile"));
XElement elementTarget = new XElement("target");
elementProject.Add(elementTarget);
elementTarget.Add(new XAttribute("name", "compile"));
XElement elementEcho = new XElement("echo");
elementTarget.Add(elementEcho);
elementEcho.Add(new XAttribute("message", "Build All: jdk = ${java.home}, version = ${java.version}"));
// add .sln files - recursively
AddBuildXmlFiles(elementTarget, rootDirectory, rootDirectory);
Console.Out.WriteLine("writing build file to disk");
// no BOM
using (var writer = new XmlTextWriter(projFile, new UTF8Encoding(false)))
{
writer.Formatting = Formatting.Indented;
xdoc.Save(writer);
}
Console.Out.WriteLine("all done");
}
private static void AddBuildXmlFiles(XElement elementTarget, string rootDirectory, string folder)
{
// add build.xml files
foreach (string fileOn in Directory.GetFiles(folder, "build.xml"))
{
string filename = Path.GetFileName(fileOn);
string workingFolder;
if (folder.StartsWith(rootDirectory))
{
workingFolder = folder.Substring(rootDirectory.Length).Trim();
if ((workingFolder.Length > 0) && (workingFolder[0] == Path.DirectorySeparatorChar || workingFolder[0] == Path.AltDirectorySeparatorChar))
workingFolder = workingFolder.Substring(1);
}
else
workingFolder = folder;
if (workingFolder.Length == 0)
continue;
XElement elementExec = new XElement("ant");
elementExec.Add(new XAttribute("dir", workingFolder));
elementExec.Add(new XAttribute("antfile", filename));
elementTarget.Add(elementExec);
}
// look in sub-directories
foreach (string subDirectory in Directory.GetDirectories(folder))
AddBuildXmlFiles(elementTarget, rootDirectory, subDirectory);
}
}
}

Resources