I am trying to check Log files for Errors and write the name of each file into an Excel register file.
Basically if I have 60 log files and 30 of them have Error I want to have 30 rows added to one excel file. And each time I run this mapping new rows to be added to that excel.
How can I do that? So far I have tried a couple of things, but none have actually done anything for me:
I have tried to specify the excel file in Outputfilename box, I added Sheet1 and the workbook name, but nothing.
Related
I have 20 files (csv files) in the same bucket. I am able to read all the file in one go and load on to bigquery. But when there is some data type mismatches, im able to get that row into invalidDataTag where as i am unable to find the file name that has the error record.
inputFilePattern is gs://bucket-name/* this picks up all the files that are present under the bucket. and reading the files as below
PCollection<String> sourceData = pipeline.apply(Constants.READ_CSV_STAGE_NAME, TextIO.read().from(options.getInputFilePattern()));
Is there a way where i can find the file name that has the error row in it ?
My suggestion would be to add a column to the BigQuery table that indicates which file the record came from.
I have an SSIS package that looks through excel files with a date at the end of each file. I have watched many vidoes but can't find an answer. If I move the "original file" that I used with the connection manager(see link for screenshot) the package fails to complete with "Error: The GetEnumerator method of the ForEach Enumerator has failed with error 0x80004005 "Unspecified error". This occurs when the ForEach Enumerator cannot enumerate." If I then put the file back to the correct place it works again. My question is do I have to have the file there forever? Is there no way to do this and it not look for that file to exist?
You are using static file names. If you are OK with using static file names, then you will have to have some other process update that file periodically. However, if you want to be more dynamic where the process loops over multiple files, then in the ForEachLoop Enumerator, you just set the following settings:
Note: using *.xlsx is the method for getting any file with the extension of xlsx.
You then map the file path to a variable:
You then set your Connection Manager's ConnectionString property to the file path:
I am reading the filename from the excel spreadsheet and iterating over the files present in a specified folder. If the file name in the folder matches with the filename in the excel sheet the message "Yes" is entered in the second column of the excel sheet otherwise "No" is entered. But when the filenames are not equal "No" is not shown in the excel spreadsheet it is left as blank. I am also trying to copy the files whose file names are matched to another folder but I am getting some error.
First thing, it would be more efficient and simpler if you use If file exists command.
If file exists ($filepath$) Then
Set Cell B$Counter$ = Yes
Copy file command
Else
Set Cell B$Counter$ = No
The error happened because you are looping on files on a folder and the counter is for the file loop not for the excel.
I've tried googling for an answer but have had no luck.
What I need to do is write to a csv file that has a double header row. The columns I need as headers are in row 2 and I can set them using 'Header rows to skip'.
However, I need to start writing data at line 3 and this isn't currently happening.
What's happening is that the header in row 1 is being removed.
Everything else is fine with the package.
Any ideas ?
You can trick your Flat File Target into producing the desired output if you can do without the comfort of automatically creating the column header names from your pipeline metadata:
On your Flat File Connection, uncheck the option "Column names in the first data row"
Open your Flat File Target "Advanced Editor" and switch to the Component Properties tab.
Find the Header property under Custom Properties and edit the text to contain the desired header data ending with \r\n\r\n to produce the desired blank line between your head line and your data.
Setting Header to
column1;column2\r\n\r\n`
your resulting file should look something like this:
column1;column2CRLF
CRLF
val11;val12CRLF
val21,val22CRLF
Is there a way to read data from an external data sheet like excel, Text file etc. in eggPlant?
When running the same script for various set of Input parameters this would prove useful for me instead of hardcoding the values..
-Siva
Since this is the most viewed Eggplant question, I'll give it a more robust answer.
Yes! Using data from a data file is a fantastic way to parameterize your test without hardcoding!
Saving Data
To do so, you have to save your data in .csv or .txt format, within the Suite's Resources directory. This allows you to open and interact it from within Eggplant Functional.
Importing Data
In your script, you can reference these data files with just their filename, for example,
put ResourcePath("myData.txt") into FilePath
will save the entire file myData.txt from the Resources directory into a variable FilePath.
Accessing Data
You can then access each row of that file like any other file.
put line 1 of file FilePath into Name
put line 2 of file FilePath into DOB
If you save your data as a .csv, you can specify a row and column of a specific piece of data.
put item 2 in line 1 of file FilePath into Last_Name
Read more about reading files in the Eggplant Documentation!
For more complicated resource files, read this page in the Eggplant Documentation!
1. Enter the data in the excel sheet and save it as a CSV file.
2. Piece of code:
repeat with theData= each line of file "D:\TestData.csv"
log item 1 of theData
end repeat