Parsing robocopy log file to PSCustomObject - parsing

I'm trying to create a PSCustomObject from a robocopy log file. The first piece is pretty easy but I'm struggling with the $Footer part. I can't seem to find a good way to split up the values.
It would be nice if every entry has it's own Property, so it's possible to use for example $Total.Dirs or $Skipped.Dirs. I was thinking about Import-CSV, because that's just great on how it allows you to have column headers. But this doesn't seem to fit here. There's another solution I found here but it seems a bit of overkill.
Code:
Function ConvertFrom-RobocopyLog {
Param (
[Parameter(Mandatory=$true,ValueFromPipelineByPropertyName=$true,Position=0)]
[String]$LogFile
)
Process {
$Header = Get-Content $LogFile | select -First 10
$Footer = Get-Content $LogFile | select -Last 7
$Header | ForEach-Object {
if ($_ -like "*Source*") {$Source = (($_.Split(':'))[1]).trim()}
if ($_ -like "*Dest*") {$Destination = (($_.Split(':'))[1]).trim()}
}
$Footer | ForEach-Object {
if ($_ -like "*Dirs*") {$Dirs = (($_.Split(':'))[1]).trim()}
if ($_ -like "*Files*") {$Files = (($_.Split(':'))[1]).trim()}
if ($_ -like "*Times*") {$Times = (($_.Split(':'))[1]).trim()}
}
$Obj = [PSCustomObject]#{
'Source' = $Source
'Destination' = $Destination
'Dirs' = $Dirs
'Files' = $Files
'Times' = $Times
}
Write-Output $Obj
}
}
Log file:
-------------------------------------------------------------------------------
ROBOCOPY :: Robust File Copy for Windows
-------------------------------------------------------------------------------
Started : Wed Apr 01 14:28:11 2015
Source : \\SHARE\Source\
Dest : \\SHARE\Target\
Files : *.*
Options : *.* /S /E /COPY:DAT /PURGE /MIR /Z /NP /R:3 /W:3
------------------------------------------------------------------------------
0 Files...
0 More Folders and files...
------------------------------------------------------------------------------
Total Copied Skipped Mismatch FAILED Extras
Dirs : 2 0 2 0 0 0
Files : 203 0 203 0 0 0
Bytes : 0 0 0 0 0 0
Times : 0:00:00 0:00:00 0:00:00 0:00:00
Ended : Wed Apr 01 14:28:12 2015
Thank you for your help.

You can clean this up more but this is the basic approach I would take.
Function ConvertFrom-RobocopyLog {
Param (
[Parameter(Mandatory=$true,ValueFromPipelineByPropertyName=$true,Position=0)]
[String]$LogFile
)
Process {
$Header = Get-Content $LogFile | select -First 10
$Footer = Get-Content $LogFile | select -Last 7
$Header | ForEach-Object {
if ($_ -like "*Source*") {$Source = (($_.Split(':'))[1]).trim()}
if ($_ -like "*Dest*") {$Destination = (($_.Split(':'))[1]).trim()}
}
$Footer | ForEach-Object {
if ($_ -like "*Dirs*"){
$lineAsArray = (($_.Split(':')[1]).trim()) -split '\s+'
$Dirs = [pscustomobject][ordered]#{
Total = $lineAsArray[0]
Copied = $lineAsArray[1]
Skipped = $lineAsArray[2]
Mismatch = $lineAsArray[3]
FAILED = $lineAsArray[4]
Extras = $lineAsArray[5]
}
}
if ($_ -like "*Files*"){
$lineAsArray = ($_.Split(':')[1]).trim() -split '\s+'
$Files = [pscustomobject][ordered]#{
Total = $lineAsArray[0]
Copied = $lineAsArray[1]
Skipped = $lineAsArray[2]
Mismatch = $lineAsArray[3]
FAILED = $lineAsArray[4]
Extras = $lineAsArray[5]
}
}
if ($_ -like "*Times*"){
$lineAsArray = ($_.Split(':',2)[1]).trim() -split '\s+'
$Times = [pscustomobject][ordered]#{
Total = $lineAsArray[0]
Copied = $lineAsArray[1]
FAILED = $lineAsArray[2]
Extras = $lineAsArray[3]
}
}
}
$Obj = [PSCustomObject]#{
'Source' = $Source
'Destination' = $Destination
'Dirs' = $Dirs
'Files' = $Files
'Times' = $Times
}
Write-Output $Obj
}
}
I wanted to make a function to parse the footer lines but $Times is a special case since it does not have all the same columns of data.
With $Times the important difference is how we are doing the split. Since the string contains more than one colon we need to account for that. Using the other paramenter in .Split() we specify the number of elements to return.
$_.Split(':',2)[1]
Since these logs always have output and no blanks row elements we can do this assuming that the parsed elements of $lineAsArray will always have 6 elements.
Sample Output
Source : \\SHARE\Source\
Destination : \\SHARE\Target\
Dirs : #{Total=2; Copied=0; Skipped=2; Mismatch=0; FAILED=0; Extras=0}
Files : #{Total=203; Copied=0; Skipped=203; Mismatch=0; FAILED=0; Extras=0}
Times : #{Total=0:00:00; Copied=0:00:00; FAILED=0:00:00; Extras=0:00:00}
So if you wanted the total files copied you can now use dot notation.
(ConvertFrom-RobocopyLog C:\temp\log.log).Files.Total
203

Not that clear what you want to do, but this will go some way to showing you how to get the stats into an array of objects
$statsOut = #()
$stats = Get-Content $LogFile | select -Last 6 | select -first 4
$stats | % {
$s = $_ -split "\s+"
$o = new-object -type pscustomobject -property #{"Name"=$s[0];"Total"=$s[2];"Copied"=$s[3];"Skipped"=$s[4];"mismatch"=$s[5]};
$statsOut += ,$o
}
Gives:
[PS] > $statsOut | ft -Auto
mismatch Name Skipped Total Copied
-------- ---- ------- ----- ------
0 Dirs 2 2 0
0 Files 203 203 0
0 Bytes 0 0 0

Related

how to extent this code to accommodate D and E drives as well

I would like to extent the following code to accommodate all other drives on my machine.
Below code was written only on C drive..i am having difficulty modifying it, tried seperating values for driveletters, but the code is not displaying anything when run..
$ServerListFile = "D:\serverList.txt"
$ServerList = Get-Content $ServerListFile -ErrorAction SilentlyContinue
$Result = #()
ForEach($c`enter code here`omputername in $ServerList)
{
$AVGProc = Get-WmiObject -computername $computername win32_processor |
Measure-Object -property LoadPercentage -Average | Select Average
$OS = gwmi -Class win32_operatingsystem -computername $computername |
Select-Object #{Name = "MemoryUsage"; Expression = {“{0:N2}” -f ((($_.TotalVisibleMemorySize -
$_.FreePhysicalMemory)*100)/ $_.TotalVisibleMemorySize) }}
$vol = Get-WmiObject -Class win32_Volume -ComputerName $computername -Filter "DriveLetter = 'C:'" |
Select-object #{Name = "C PercentFree"; Expression = {“{0:N2}” -f (($_.FreeSpace / $_.Capacity)*100)
} }
$result += [PSCustomObject] #{
ServerName = "$computername"
CPULoad = "$($AVGProc.Average)%"
MemLoad = "$($OS.MemoryUsage)%"
CDrive = "$($vol.'C PercentFree')%"
}
$Outputreport = "<HTML><TITLE> Server Health Report </TITLE>
<BODY background-color:peachpuff>
<font color =""#99000"" face=""Microsoft Tai le"">
<H2> Server Health Report </H2></font>
<Table border=1 cellpadding=0 cellspacing=0>
<TR bgcolor=gray align=center>
<TD><B>Server Name</B></TD>
<TD><B>Avrg.CPU Utilization</B></TD>
<TD><B>Memory Utilization</B></TD>
<TD><B>Drive C Free Space</B></TD>
</TR>"
Foreach($Entry in $Result)
{
if(($Entry.CpuLoad) -or ($Entry.memload) -ge "80")
{
$Outputreport += "<TR bgcolor=white>"
}
else
{
$Outputreport += "<TR>"
}
$Outputreport += "<TD>$($Entry.Servername)</TD><TD align=center>$($Entry.CPULoad)</TD><TD
align=center>$($Entry.MemLoad)</TD><TD align=center>$($Entry.CDrive)</TD></TR>"
}
$Outputreport += "</Table></BODY></HTML>"
}
$Outputreport | out-file "D:\Result $(Get-Date -Format yyy-mm-dd-hhmm).htm"

Parse wikipedia {{Location map}} templates

I would like to parse the Wikipedia power plant lists, which contain the {{Location map}} template. In my example I'm using the German translation, but this shouldn't change the basic process.
How can I get out the label=, lat=, lon= and region= parameters from such code?
Probably this is nothing for a html parser like BeautifulSoup, but rather awk?
{{ Positionskarte+
| Tadschikistan
| maptype = relief
| width = 600
| float = right
| caption =
| places =
{{ Positionskarte~
| Tadschikistan
| label = <small>[[Talsperre Baipasa|Baipasa]]</small>
| marktarget =
| mark = Blue pog.svg
| position = right
| lat = 38.267584
| long = 69.123906
| region = TJ
| background = #FEFEE9
}}
{{ Positionskarte~
| Tadschikistan
| label = <small>[[Kraftwerk Duschanbe|Duschanbe]]</small>
| marktarget =
| mark = Red pog.svg
| position = left
| lat = 38.5565
| long = 68.776
| region = TJ
| background = #FEFEE9
}}
...
}}
Thanks in advance!
Just extract information with regular expressions.
For example like this (PHP)
$k = "{{ Positionskarte+
| Tadschikistan
| maptype = relief
| width = 600
| float = right
| caption =
| places =
{{ Positionskarte~
| Tadschikistan
| label = <small>[[Talsperre Baipasa|Baipasa]]</small>
| marktarget =
| mark = Blue pog.svg
| position = right
| lat = 38.267584
| long = 69.123906
| region = TJ
| background = #FEFEE9
}}
{{ Positionskarte~
| Tadschikistan
| label = <small>[[Kraftwerk Duschanbe|Duschanbe]]</small>
| marktarget =
| mark = Red pog.svg
| position = left
| lat = 38.5565
| long = 68.776
| region = TJ
| background = #FEFEE9
}}
}}";
$items = explode("Positionskarte~", $k);
$result = [];
foreach ($items as $item) {
$info = [];
$pattern1 = '/label\s+=\s+(.+)/';
preg_match($pattern1, $item, $matches);
if (!empty($matches)) {
$info['label'] = $matches[1];
}
$pattern2 = '/lat\s+=\s+(.+)/';
preg_match($pattern2, $item, $matches);
if (!empty($matches)) {
$info['lat'] = $matches[1];
}
$pattern3 = '/long\s+=\s+(.+)/';
preg_match($pattern3, $item, $matches);
if (!empty($matches)) {
$info['long'] = $matches[1];
}
$pattern4 = '/region\s+=\s+(.+)/';
preg_match($pattern4, $item, $matches);
if (!empty($matches)) {
$info['region'] = $matches[1];
}
if(!empty($info)) {
$result[] = $info;
}
}
var_dump($result);

Comparison of columns from two files and create new column

I want to compare first two columns from file1.txt and file2.txt, if match found add new columns (3rd and 4th) to file1.txt with values found from column 3 and 4 of file2.txt and "NA" to non-match.
file1.txt
ch1 100
ch1 200
ch3 100
ch4 200
file2.txt
ch1 100 0.5 0.6
ch1 200 0.1 1.2
ch3 400 0.2 0.9
ch4 200 1.0 3.0
outputfile.txt
ch1 100 0.5 0.6
ch1 200 0.1 1.2
ch3 100 NA NA
ch4 200 1.0 3.0
I tried join/awk commands but it is not giving the desired output.
The standard awk technique reads the whole of file1.txt into memory. If your files are too big to fit, then considerably more effort is required (but it can be done even so).
awk 'FNR == NR { k[$1,$2] = 1; next }
{ if (k[$1,$2] == 1) { print $0; k[$1,$2] = 2 } }
END { for (i in k) { if (k[i] == 1) { sub(SUBSEP, " ", i); print i, "NA", "NA" } } }' \
file1.txt file2.txt
The first line reads the first file and records the keys that are read. The second line does most of the processing. If the key of $1, $2 in the second file matches a record, then print $0, and record that the key was matched (by setting the value to 2 from 1). The third line (the END block) looks at all the keys in k and if the value is not 2, it was not matched so the key is printed with the two NA columns. The sub(SUBSEP, " ", i) part fixes the sub-separator between the two keys in i into a space.
Raw output from the awk:
ch1 100 0.5 0.6
ch1 200 0.1 1.2
ch4 200 1.0 3.0
ch3 100 NA NA
After passing through column -t (on my Mac):
ch1 100 0.5 0.6
ch1 200 0.1 1.2
ch4 200 1.0 3.0
ch3 100 NA NA
join -a1 -1 1 -2 1 -e "NA" -o 1.1,2.2,2.3 \
<(sed 's/ \+/_/' file1.txt | sort) <(sed 's/ \+/_/' file2.txt | sort) |
sed 's/_/ /' | column -t

separate 8th field

I could not separate my file:
chr2 215672546 rs6435862 G T 54.00 LowDP;sb DP=10;TI=NM_000465;GI=BARD1;FC=Silent ... ...
I would like to print first seven fields and from 8th field print just DP=10 and GI=BARD1. DP in GI info is always in 8th field. Fields are continue (...) so 8th field is not last.
I know how to extract 8th field :
awk '{print $8}' PLZ-10_S2.vcf | awk -F ";" '/DP/ {OFS="\t"} {print $1}'
of course how to extract first seven fields, but how to pipe it together? Between all fields is tab.
If DP= and GI= are always in the same position within $8:
$ awk 'BEGIN{FS=OFS="\t"} {split($8,a,/;/); $8=a[1]";"a[3]} 1' file
chr2 215672546 rs6435862 G T 54.00 LowDP;sb DP=10;GI=BARD1 ... ...
If not:
$ awk 'BEGIN{FS=OFS="\t"} {split($8,a,/;/); $8=""; for (i=1;i in a;i++) $8 = $8 (a[i] ~ /^(DP|GI)=/ ? ($8?";":"") a[i] : "")} 1' file
chr2 215672546 rs6435862 G T 54.00 LowDP;sb DP=10;GI=BARD1 ... ...
One way is to split() with semicolon the eight field and traverse all results to check which of them begin with DP or GI:
awk '
BEGIN { FS = OFS = "\t" }
{
split( $8, arr8, /;/ )
$8 = ""
for ( i = 1; i <= length(arr8); i++ ) {
if ( arr8[i] ~ /^(DP|GI)/ ) {
$8 = $8 arr8[i] ";"
}
}
$8 = substr( $8, 1, length($8) - 1 )
print $0
}
' infile
It yields:
chr2 215672546 rs6435862 G T 54.00 LowDP;sb DP=10;GI=BARD1 ... ...

Foreach command with a folder with many files using a template txt file

Using the following code to select help select 1 file from a list of many dated files and then copy to share. Using the following code very kindly provided by fellow Stacker
$pastdays = -7
$pastdate = [datetime]::Now.AddDays($pastdays)
$files = Get-ChildItem "V:\Capacity Manager Size reports\"
$filename = ($files | Where-Object {$_.Name -like "Database_Server1_Userdb1_" + $pastdate.Day + "_" + $pastdate.Month + "_" + $pastdate.Year+ "*.pdf"})
Copy-Item -Path $filename.FullName "\\Server50\report_Archive"
I need to expand this process to run for many files in the folder – each with slightly different names but following a common template to the file name
Original filenames examples:
Database_Server1_UserDB1_30_10_2012_00_20_51
Database_Server2_UserDB2_30_10_2012_01_20_51
Etc
Etc
But im not sure how to best do this in a foreach ?
I was thinking create a template file to load into a variable that holds all the examples of the files to examined before copying
So:
$Report_Template = Get-Content "C:\Powershell\FileTemplate_Name.txt";
FileTemplate_name.txt:
Database_Server1_tempdb_
Database_Server1_UserDB1_
Database_Server1_UserDB2_
Database_Server2_UserDB2_
Database_Server2_UserDB4_
$Report_Template = Get-Content "C:\Powershell\QCM_Template_Name.txt";
$pastdays = -7
$pastdate = [datetime]::Now.AddDays($pastdays)
$files = Get-ChildItem "V:\Capacity Manager Size reports\"
foreach($template in $Report_Template)
$filename = ($files | Where-Object {$_.Name -like $template + $pastdate.Day + "_" + $pastdate.Month + "_" + $pastdate.Year+ "*.pdf"})
Copy-Item -Path $filename.FullName "\\Server50\CapacityManagerReports_Archive\"
Try this:
$Report_Template = Get-Content "C:\Powershell\QCM_Template_Name.txt";
$pastdays = -7
$pastdate = [datetime]::Now.AddDays($pastdays)
$files = Get-ChildItem "V:\Capacity Manager Size reports\"
foreach($template in $Report_Template)
{
$filename = ($files | Where-Object {$_.Name -like $template + $pastdate.Day + "_" + $pastdate.Month + "_" + $pastdate.Year+ "*.pdf"})
Copy-Item -Path $filename.FullName "\\Server50\CapacityManagerReports_Archive\"
}
But do you really need template file? Can't you use file extension, or some part of the name, or just copy everything from this date? Maybe your inital folder with files has wrong structure and you should better switch to multiple folders, like separate folder for each Database_Server1?
Ive done the following which seems to work
$Report_Template = Get-Content "C:\Powershell\QCM_Template_Name.txt";
$pastdays = -3
$pastdate = [datetime]::Now.AddDays($pastdays)
$Results = Get-ChildItem "V:\Capacity Manager Size reports\"
Foreach ($Files in $Results)
{foreach($template in $Report_Template)
{If ($files.Name -like $($template + $pastdate.Day + "_" + $pastdate.Month + "_" + $pastdate.Year +"*.pdf"))
{Copy-Item -Path $files.FullName "\\Server50\CapacityManagerReports_Archive\"}
}
}

Resources