Scripting a URL validity check from a text file list - url

I am trying to modify the PowerShell script I have found on ElegantCode.Com. I want to change it to specify a large text file of HTTP links, rather than naming the links as a parameter individually.
Once the script parses the file, I want it to pipe or echo out only the links that are valid back to a new file.
I am falling at the first hurdle and can't even figure out how I pass the input file in as a parameter.
Direct link for the script is here
BEGIN {
}
PROCESS {
$url = $_;
$urlIsValid = $false
try
{
$request = [System.Net.WebRequest]::Create($url)
$request.Method = 'HEAD'
$response = $request.GetResponse()
$httpStatus = $response.StatusCode
$urlIsValid = ($httpStatus -eq 'OK')
$tryError = $null
$response.Close()
}
catch [System.Exception] {
$httpStatus = $null
$tryError = $_.Exception
$urlIsValid = $false;
}
$x = new-object Object | `
add-member -membertype NoteProperty -name IsValid -Value $urlIsvalid -PassThru | `
add-member -membertype NoteProperty -name Url -Value $_ -PassThru | `
add-member -membertype NoteProperty -name HttpStatus -Value $httpStatus -PassThru | `
add-member -membertype NoteProperty -name Error -Value $tryError -PassThru
$x
}
}
END {
}

It appears the script it expecting the url to be piped in. The variable $_ represents the current pipeline object. So if the text file contained on URL per line you could do something like this:
Get-Content Urls.txt | Where {$_ -notmatch '^\s*$'} | Check-Url
I put the where in the pipe to eliminate blank lines.

To pipe the valid urls to a file as requested (adding to Keith's answer):
$validUrls = ".\ValidUrls.txt"
if (Test-Path $validUrls) { Remove-Item -Force $validUrls }
$result = New-Item -Type File -Path $validUrls
Get-Content Urls.txt | Where {$_ -notmatch '^\s*$'} | Foreach-Object {
$url = ($_ | Check-Url)
if ($url.IsValid)
{
$url.Url | Out-File $result -Append
}
}

Related

Parse GUID out of string

I am running into an issue with this following using trim. It is removing some of the GUID. I would like to just get the GUID out of this. If anyone could help it would be appreciative.Thank you in advanced.
((Get-ADOrganizationalUnit -Filter {Name -eq "OUName"} -Properties LinkedGroupPolicyObjects,gplink) | ForEach-Object {if($_.GPlink){$_.GPlink.Split(",") | ForEach-Object {if($_ -like 'DC=DCNAME*'){if($_.length -gt 10){(((($_).trim('DC=DCNAME;0][LDAP://cn={')).trim('}')) ) }} }}})
My output is this:
754FF9F1-078A-4E05-913D-4F36572B2FC6
EDDAAB18-2BA6-42E6-A5EC-21B0227BE71A
7DF312DB-EB73-418E-8F64-3E391F4639B7
6E3512-4100-48A3-9A65-4DA17A0E2D87
72EF89D6-2C57-40AC-A116-2CAD89F453ED
2][LDAP://cn={31BB7749-F6DC-4098-8F10-9D8B4B0F0C0A
78528B0-F379-4E8F-A166-ACE1448AF9B2
I'm not sure how that DC=.. filter is helping you. The DC= part comes after the guid, so since you're splitting on ,, won't the first GUID be ignored by the time you get a match with -like 'DC=DCNAME*'? Ex.
[LDAP://cn={7BE35F55-E3DF-4D1C-8C3A-38F81F451D86},cn=policies, cn=system,DC=wingtiptoys,DC=local;2][LDAP://cn={7BE35F55-E3DF-4D1C-8111111},cn=......
If you only want to extract the guids, I would just extact {GUID}, like:
Get-ADOrganizationalUnit -Filter {Name -eq "OUName"} -Properties LinkedGroupPolicyObjects, gplink |
ForEach-Object {
if($_.GPlink) {
#Anything between { and }
[regex]::Matches($gplink,'(?<=\{).*?(?=\})') | ForEach-Object { $_.Value }
}
}
without your input i cannot be sure, but you might try something like this
Get-ADOrganizationalUnit -Filter {Name -eq 'OUName'} -Properties LinkedGroupPolicyObjects, gplink | ForEach-Object {
if ($_.GPlink) {
$_.GPlink.Split(',') | ForEach-Object {
if ($_ -like 'DC=DCNAME*' -and $_.length -gt 10) {
#$_ = $_ -replace 'DC=DCNAME;\d+\]\[LDAP:\/\/cn=\{'
#$_.trim('}')
$_ -replace '^.*([a-f0-9]{8}-(?:[a-f0-9]{4}-){3}[a-f0-9]{12}).*$', '$1'
}
}
}
}

Powershell 2.0 Get users for a local group

How can I get user list from a local group? I only have PS 2.0 and it does not have Get-ADGroup command.
I can get local groups:
$adsi = [ADSI]"WinNT://$env:COMPUTERNAME"
$groups = $adsi.Children | Where { $_.SchemaClassName -eq 'Group' }
$group | ft Name
What I need is to list all the members for each group.
You can try the following
$obj = [ADSI]"WinNT://$env:COMPUTERNAME"
$admingroup = $obj.Children | Where { $_.SchemaClassName -eq 'group'} | where {$_.name -eq 'Administrators'}
$admingroup.Invoke('Members') | % {$_.GetType().InvokeMember('Name', 'GetProperty', $null, $_, $null)}
$admingroup.Invoke('Members') | % {$_.GetType().InvokeMember('Name', 'GetProperty', $null, $_, $null)}
Here are the common properties
String :
Description, FullName, HomeDirectory, HomeDirDrive, Profile, LoginScript, ObjectSID
Integer :
UserFlags, PasswordExpired, PrimaryGroupID
Time :
PasswordAge
You'll find more in Microsoft documentation.
Try this
$computer = [ADSI]"WinNT://$env:COMPUTERNAME"
$computer.psbase.children | where { $_.psbase.schemaClassName -eq 'group' } | foreach {
write-host $_.name
write-host "------"
$group =[ADSI]$_.psbase.Path
$group.psbase.Invoke("Members") | foreach {$_.GetType().InvokeMember("Name", 'GetProperty', $null, $_, $null)}
write-host
}
This doesn't give the domain though, hence i had to look for other ways, like:
If you want to see members of a local group quickly:
PS C:\> net localgroup USERS
Alias name USERS
Comment Users are prevented from making accidental or intentional system-wide changes and can run most applications
Members
-------------------------------------------------------------------------------
NT AUTHORITY\Authenticated Users
NT AUTHORITY\INTERACTIVE
The command completed successfully.
Now you can manipulate this output a bit to get what you need:
$computer = [ADSI]"WinNT://$env:COMPUTERNAME"
$groups = $computer.psbase.children | where { $_.psbase.schemaClassName -eq 'group' } | select -ExpandProperty Name
Foreach($group in $groups)
{
write-host $group
write-host "------"
net localgroup $group | where {$_ -notmatch "command completed successfully"} | select -skip 6
Write-host
}

exception calling "Parse" with 1 argument: Input string was not in a correct format

I have a script that works properly when run as an administrator but gives a Parse error when run as a normal user. Any Ideas?;
SCRIPT
`NeverExpires = 9223372036854775807;
$ExpireMin = (Get-Date).AddDays(4);
$ExpireMax = (Get-Date).AddDays(9);
$Userlist = Get-ADUser -Filter * -Properties name, samaccountname, accountexpirationdate, enabled, distinguishedname, accountExpires | Where-object {($_.DistinguishedName -notlike "*OU=Terminated,OU=Users,OU=Home Office,DC=Domain,DC=com")} |
Where-Object {$_.accountExpires -ne $NeverExpires `
-and [datetime]::FromFileTime([int64]::Parse($_.accountExpires)) -lt $ExpireMax `
-and [datetime]::FromFileTime([int64]::Parse($_.accountExpires)) -gt $ExpireMin }
$Userlist | select name, samaccountname, accountexpirationdate, enabled, distinguishedname | export-csv $ReportName -notypeinformation
Send-MailMessage -To $To -From $From -Subject $Subject -Body $Body -SMTPServer $SMTPServer -Attachments $ReportName
Get-ADUser -Filter * -Properties accountExpires |
Where-Object {$_.accountExpires -ne $NeverExpires `
-and [datetime]::FromFileTime([int64]::Parse($_.accountExpires)) -lt $ExpireMax `
-and [datetime]::FromFileTime([int64]::Parse($_.accountExpires)) -gt $ExpireMin } | ForEach {
$account = $_
$manager = Get-ADUser -Identity $account -Properties EmailAddress,Manager | %{(Get-AdUser $_.Manager -Properties EmailAddress).EmailAddress}`
I would say that $_.accountExpires is null either because the property could not be retrieved or $_ is itself null. Powershell will quietly convert null to the empty string resulting an invalid format for parsing. Note that the Parse call is completely unnecessary because powershell will automatically try to coerce the string for you and will likely give you a much better error message. Although null will be coerced to 0 as a long.

How to parse PDF content to database with powershell

I have a pdf document that I would like to extract content out of. The issue I am having is this... I search for the IMEI keyword, and it finds it, but I need the actual IMEI value which is the next item in the loop.
In the PDF the value looks like this:
IMEI 90289393092
returning value via the below script:
-0.1 -8.8 9.8 -0.1 446.7 403.9 Tm (IMEI:) Tj
I only want to have the value:
90289393092
Script I am using:
Add-Type -Path .\itextsharp.dll
$reader = New-Object iTextSharp.text.pdf.pdfreader -ArgumentList "$pwd\PDF\DOC001.pdf"
for ($page = 1; $page -le $reader.NumberOfPages; $page++) {
$lines = [char[]]$reader.GetPageContent($page) -join "" -split "`n"
foreach ($line in $lines) {
if ($line -match "IMEI") {
$line = $line -replace "\\([\S])", $matches[1]
$line -replace "^\[\(|\)\]TJ$", "" -split "\)\-?\d+\.?\d*\(" -join ""
}
}
}
this is the way for using itextsharp.dll and read a pdf as plain text:
Add-Type -Path .\itextsharp.dll
$reader = New-Object iTextSharp.text.pdf.pdfreader -ArgumentList c:\ps\a.pdf
for ($page = 1; $page -le $reader.NumberOfPages; $page++)
{
$strategy = new-object 'iTextSharp.text.pdf.parser.SimpleTextExtractionStrategy'
$currentText = [iTextSharp.text.pdf.parser.PdfTextExtractor]::GetTextFromPage($reader, $page, $strategy);
[string[]]$Text += [system.text.Encoding]::UTF8.GetString([System.Text.ASCIIEncoding]::Convert( [system.text.encoding]::default , [system.text.encoding]::UTF8, [system.text.Encoding]::Default.GetBytes($currentText)));
}
$Reader.Close();
And this can be the regex you need but I haven't tested it
[regex]::matches( $text, '(?<=IMEI\s+)(\d+)(?=\s+)' ) | select -expa value

Using System.Drawing.Image to Export-Csv with $imageFile.Width and $_.Fullname

I wonder if there is a betterr way to write this script to gather image dimensions and filepaths. The script works great on small to medium size directories, but I'm not positive that 100,000+ files/folders is possible.
Measure-Command {
[Void][System.Reflection.Assembly]::LoadFile( "C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.Drawing.dll")
$path = "\\servername.corp.company.com\top_directory"
$data = Get-ChildItem -Recurse $path | % {
$imageFile = [System.Drawing.Image]::FromFile($_.FullName) ;
New-Object PSObject -Property #{
name = $_.Name
fullname = $_.Fullname
width = $imageFile.Width
height = $imageFile.Height
length = $_.Length
}
}
$data | Where-Object {$_.width -eq 500 -or $_.width -eq 250 -or $_.width -eq 1250 } |
Export-Csv \\servername.corp.company.com\top_directory\some_directory\log_file.csv -NoTypeInformation }
I don't actually use the Where-Object filter right now.
When running the above script on a remote directory with appx. 20,000 files + folders the script takes appx. 26 minutes, before creating a .csv.
I am running the script from Powershell V2 ISE on Windows 7 and I belive the remote server is on Windows Server 2003.
Would running the script directly from the remote server be faster?
Is the process of exoprting the csv slow since all data is collected in "cache" before being written to the csv?
If all I had to go through was 20,000 files, I'd wait the 26 minutes, but 500,000 files and folders is a long wait.
I'm testing out the below method since I think my real issue is not the speed, but storing too much data in the memory. Thanks to a post from George Howarth for this, and to PoSherLife for the top script -http://powershellcommunity.org/tabid/54/aft/4844/Default.aspx
Measure-Command {
[System.Reflection.Assembly]::LoadFile( "C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.Drawing.dll")
"Name|SizeInBytes|Width|Height|FullName" >> C:\Users\fcool\Documents\JPGInfo.txt
$path = "C:\Users\fcool\Documents"
$images = Get-ChildItem -Recurse $path -Include *.jpg
foreach ($image in $images)
{
$name = $image.Name
$length = $image.Length
$imageFile = [System.Drawing.Image]::FromFile($image.FullName)
$width = $imageFile.Width
$height = $imageFile.Height
$FullName = $image.Fullname
"$name|$length|$width|$height|$FullName" >> C:\Users\fcool\Documents\JPGInfo.txt
$imageFile.Dispose()
}
}
Is there any risk/loss of performance when running these scripts on non-image filetypes?
when I don't exclude non-images, I get this error:
Exception calling "FromFile" with "1" argument(s): "Out of memory."
At C:\scripts\directory_contents_IMAGE_DIMS_ALT_method.ps1:13 char:46
+ $imageFile = [System.Drawing.Image]::FromFile <<<< ($image.FullName)
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : DotNetMethodException
thanks for any advice!
and thanks again to George Howarth and to PoSherLife for the scripts!
Using -Filter with Get-ChildItem is much faster than -Include however you can only apply one filter string. So if you only want to match *.jpg you can use filter. In my testing using filter was close to 5 times faster than include.
Get-ChildItem -Recurse \\server\Photos -Filter *.jpg | % {
$image = [System.Drawing.Image]::FromFile($_.FullName)
if ($image.width -eq 500 -or $image.width -eq 250 -or $image.width -eq 1250) {
New-Object PSObject -Property #{
name = $_.Name
fullname = $_.Fullname
width = $image.Width
height = $image.Height
length = $_.Length
}
}
} | Export-Csv 'C:\log.csv' -NoTypeInformation

Resources