Filter by process/PID in Wireshark - wireshark

Is there a way to filter/follow a TCP/SSL stream based on a particular process ID using Wireshark?

Just in case you are looking for an alternate way and the environment you use is Windows, Microsoft's Network Monitor 3.3 is a good choice. It has the process name column. You easily add it to a filter using the context menu and apply the filter.. As usual the GUI is very intuitive...

I don't see how. The PID doesn't make it onto the wire (generally speaking), plus Wireshark allows you to look at what's on the wire - potentially all machines which are communicating over the wire. Process IDs aren't unique across different machines, anyway.

You could match the port numbers from wireshark up to port numbers from, say, netstat which will tell you the PID of a process listening on that port.

Use Microsoft Message Analyzer v1.4
Navigate to ProcessId from the field chooser.
Etw
-> EtwProviderMsg
--> EventRecord
---> Header
----> ProcessId
Right click and Add as Column

Use strace is more suitable for this situation.
strace -f -e trace=network -s 10000 -p <PID>;
options -f to also trace all forked processes, -e trace=netwrok to only filter network system-call and -s to display string length up to 10000 char.
You can also only trace certain calls like send,recv, read operations.
strace -f -e trace=send,recv,read -s 10000 -p <PID>;

If you want to follow an application that still has to be started then it's certainly possible:
Install docker (see https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/)
Open a terminal and run a tiny container: docker run -t -i ubuntu /bin/bash (change "ubuntu" to your favorite distro, this doesn't have to be the same as in your real system)
Install your application in the container using the same way that you would install it in a real system.
Start wireshark in your real system, go to capture > options . In the window that will open you'll see all your interfaces. Instead of choosing any, wlan0, eth0, ... choose the new virtual interface docker0 instead.
Start capturing
Start your application in the container
You might have some doubts about running your software in a container, so here are the answers to the questions you probably want to ask:
Will my application work inside a container ? Almost certainly yes, but you might need to learn a bit about docker to get it working
Won't my application run slow ? Negligible. If your program is something that runs heavy calculations for a week then it might now take a week and 3 seconds
What if my software or something else breaks in the container ? That's the nice thing about containers. Whatever is running inside can only break the current container and can't hurt the rest of the system.

On Windows there is an experimental build that does this, as described on the mailing list, Filter by local process name

This is an important thing to be able to do for monitoring where certain processes try to connect to, and it seems there isn't any convenient way to do this on Linux. However, several workarounds are possible, and so I feel it is worth mentioning them.
There is a program called nonet which allows running a program with no Internet access (I have most program launchers on my system set up with it). It uses setguid to run a process in group nonet and sets an iptables rule to refuse all connections from this group.
Update: by now I use an even simpler system, you can easily have a readable iptables configuration with ferm, and just use the program sg to run a program with a specific group. Iptables also alows you to reroute traffic so you can even route that to a separate interface or a local proxy on a port whith allows you to filter in wireshark or LOG the packets directly from iptables if you don't want to disable all internet while you are checking out traffic.
It's not very complicated to adapt it to run a program in a group and cut all other traffic with iptables for the execution lifetime and then you could capture traffic from this process only.
If I ever come round to writing it, I'll post a link here.
On another note, you can always run a process in a virtual machine and sniff the correct interface to isolate the connections it makes, but that would be quite an inferior solution...

I have a PowerShell script that might help in cases like that and made it a bit nicer to place it here. My tests with PowerShell Version 5.2 and 7.2 on Windows 10 were both successful, but atm i can't test it on other OS.
What it does:
It builds a Wireshark filter with IPs and ports a process had used in network statistics. You may watch the last two picture first, to understand it better.
The long story:
It gets network statistics for TCP (listener and connections) and UDP
(listener) multiple times until you want to proceed. You will want to
wait until you finished testing your process. After you choose to
continue, it shows the current processes with process ID from wich you
must select one or multiple processes. The processes are the first
filter you can apply - the case the OP would like to have should be
only one process. Then you must select what connections/ports you may
want in your filter - usually select all here. After that you must
select another type of filter wich also defines how the Wireshark
filter will look like. The filter will be displayed and automatically
copied to clipboard.
Depending on your selections and your process, the filter might get
long.
What it doesn't:
It can't monitor your processes and their network activities. It justs gets the data multiple times. Between the get commands you might miss some connections.
It also can't see any udp packet, so it does not get anything about the remote part for udp. But it will get the local UDP listening ports.
Other limitations are: Local listening on 0.0.0.0 will be translated to your local ip address. Listening on 127.0.0.1 will be skipped, as i had no need for local connection monitoring for now.
So here is the Code:
"Attention: This script can NOT make a filter for a process, but it can build it regarding some local to remote connections (TCP) and vice versa, and ports (UDP)."
"It works good for some cases, but not for all."
"In general it is designed to filter as less as possible."
"You may still see packets from some other processes depending on your selection"
""
"Press return to continue"
Read-Host | Out-Null
# Load Functions
function Out-WireSharkSyntax($data) {
$data = $data -replace "\)|\(| eq | or | and |==|!|{|}| not | in ",';$0;' -split ";"
foreach ($Line in $data) {
$color = switch ($Line) {
"(" {"blue"}
")" {"blue"}
"!" {"cyan"}
" eq " {"yellow"}
" or " {"cyan"}
" and " {"cyan"}
" not " {"cyan"}
" in " {"cyan"}
"==" {"yellow"}
"||" {"yellow"}
"{" {"darkred"}
"}" {"darkred"}
Default {"green"}
}
Write-Host -ForegroundColor $color -NoNewline -BackgroundColor Black $line}
}
$count=0
$sleepTimer=500 #in milliseconds to restart the query for used TCP ports and listening UDP ports
$QuitKey=81 #Character code for 'q' key.
$CurrentDateTime = Get-Date
#$LocalIPv4address = #(Get-NetIPAddress -AddressFamily IPv4 -InterfaceIndex $(Get-NetConnectionProfile | Select-Object -ExpandProperty InterfaceIndex) | Select-Object -ExpandProperty IPAddress)
$LocalIPv4address = (Get-NetIPAddress -AddressFamily IPv4 -AddressState Preferred -PrefixOrigin manual,dhcp).IPAddress
if ($LocalIPv4address.count -ne 1) {
"Could not detect exact one IPAddress. Enter the IPAddress to be used:`r`nYour local dectected addresses were:$($LocalIPv4address -join " OR ")"
$LocalIPv4address = Read-Host
}
"Retrieving network network statistics every $sleepTimer milliseconds..."
"(very short connections may not be captured with this script because of this!)"
$TcpAndUdpProperties = #{Name="NetStatEntryAsString";Expression={$_.LocalAddress + "--" + $_.LocalPort + "--" + $_.RemoteAddress + "--" + $_.RemotePort + "--" + $_.cimclass.cimclassname}},`
"LocalAddress","LocalPort","OwningProcess","RemoteAddress","RemotePort","CreationTime"
# Properties for both equal to get equal list header in all cases
$TcpAndUdpNetworkStatistic = #()
Write-Host "Press 'q' to stop collecting network statistics and to continue with the script."
Write-Host "Wireshark should now capture and you start what ever you would like to monitor now."
while($true)
{
if($host.UI.RawUI.KeyAvailable) {
$key = $host.ui.RawUI.ReadKey("NoEcho,IncludeKeyUp")
if($key.VirtualKeyCode -eq $QuitKey) {
#For Key Combination: eg., press 'LeftCtrl + q' to quit.
#Use condition: (($key.VirtualKeyCode -eq $Qkey) -and ($key.ControlKeyState -match "LeftCtrlPressed"))
Write-Host ("`r`n'q' is pressed! going on with the script now.")
break
}
}
# Temporary convertion to JSON ensures that not too much irrelevant data being bound to the new variable
$TcpAndUdpNetworkStatistic += `
(Get-NetTCPConnection | select -Property $($TcpAndUdpProperties + #{Name="Protocol";Expression={"TCP"}}) | ConvertTo-Json | ConvertFrom-Json) + `
(Get-NetUDPEndpoint | select -Property $($TcpAndUdpProperties + #{Name="Protocol";Expression={"UDP"}}) | ConvertTo-Json | ConvertFrom-Json)
# exclude IPv6 as it is not handled in this script, remove 127.0.0.1 connections and remove duplicates
$TcpAndUdpNetworkStatistic = $TcpAndUdpNetworkStatistic | where {$_.LocalAddress -notmatch ":" -and $_.LocalAddress -notlike "127.*"} | ConvertTo-Csv -NoTypeInformation | Sort-Object -Unique -Descending |ConvertFrom-Csv | sort Protocol,LocalAddress,LocalPort
$TcpAndUdpNetworkStatistic | where {$_.localaddress -eq "0.0.0.0"} | foreach {$_.localaddress = $LocalIPv4Address}
$count++
Write-Host ("`rChecked network statistics {0} time{1}. Collected {2} netstat entries" -f $count,$(("s"," ")[($count -eq "1")]),$TcpAndUdpNetworkStatistic.Count) -NoNewline
Start-Sleep -m $sleepTimer
}
$TcpAndUdpNetworkStatistic | where {$_.localaddress -eq "0.0.0.0"} | foreach {$_.localaddress = $LocalIPv4Address}
$ProcessIDToNetworkstatistic = $TcpAndUdpNetworkStatistic | Group-Object OwningProcess -AsHashTable -AsString
"Getting processlist..."
$processselection = "Id", "Name", #{Name="MainModuleName";Expression={$_.MainModule.ModuleName}}, "Company",
"Path", "Product", "Description", "FileVersion", "ProductVersion", "SessionID", "CPU", "Threads", "StartTime"
$GetNetListedProcesses = Get-Process | Where {$ProcessIDToNetworkstatistic.GetEnumerator().name -contains $_.ID} | Select -Property $processselection
"Output processlist to gridview... Read the gridview title and make selection there..."
$ProcessIDs = ($GetNetListedProcesses |Select #{Name="Port/Session Count";Expression={$ProcessIDToNetworkstatistic["$($_.id)"].count}},* | `
Out-GridView -Title "Select process to view network statistics related to process id" -Passthru).ID
"Output related network statistics to gridview... Read the gridview title and make selection there..."
$TcpAndUdpNetworkStatisticFilteredByProcessID = $TcpAndUdpNetworkStatistic | Where {$ProcessIDs -contains $_.OwningProcess} | `
Out-Gridview -Title "Select lines that contain data you may like to have in your Wireshark filter" -Passthru
# for statistic and later processing
$UDPLocalPorts = ($TcpAndUdpNetworkStatisticFilteredByProcessID | where {$_.Protocol -eq "UDP"}).LocalPort | foreach {[int]$_} | Sort-Object -Unique
$TCPConnections = $TcpAndUdpNetworkStatisticFilteredByProcessID | where {$_.Protocol -eq "TCP"}
$TCPLocalPorts = #(foreach ($Connection in $TCPConnections) { [int]$Connection.LocalPort }) | Sort-Object -unique
$TCPRemotePorts = #(foreach ($Connection in $TCPConnections) { [int]$Connection.RemotePort })| Sort-Object -unique | where {$_ -ne 0}
$UDPLocalEndpoints = $TcpAndUdpNetworkStatisticFilteredByProcessID | where {$_.Protocol -eq "UDP"}
$UDPLocalPorts = #(foreach ($Endpoint in $UDPLocalEndpoints) { [int]$Endpoint.LocalPort }) | Sort-Object -unique
$FilterOptionsDialog = "
You can choose between the following filters
[all] for UDP + TCP filter - including remote address where possible ( filterable: $(($TcpAndUdpNetworkStatisticFilteredByProcessID).count) )
[tall] for TCP with listening ports and connections including remote ports and addresses ( filterable: $(($TcpAndUdpNetworkStatisticFilteredByProcessID | where {$_.Protocol -eq "TCP"}).count) )
[tcon] for TCP without listening ports - only connections including remote ports and addresses ( filterable: $(($TcpAndUdpNetworkStatisticFilteredByProcessID | where {$_.Protocol -eq "TCP" -and [int]$_.RemotePort -eq 0}).count) )
[u] for UDP portfilter - only local listening port - no `"connections`" ( filterable: $(($TcpAndUdpNetworkStatisticFilteredByProcessID | where {$_.Protocol -eq "UDP"}).count) )
[p] for portfilter only by ports ( filterable: $($TCPLocalPorts.count) local TCP / $($TCPRemotePorts.count) remote TCP / $($UDPLocalPorts.count) UDP )
[ptl] for portfilter only by local TCP ports (no UDP) ( filterable: $($TCPLocalPorts.count) local TCP / $($TCPRemotePorts.count) remote TCP )
[pt] for portfilter only by TCP ports (remote port ignored and no UDP) ( filterable: $($TCPLocalPorts.count) local TCP )
[pu] for portfilter only by UDP ports (only listening ports - no information about used ports) ( filterable: $($UDPLocalPorts.count) )
Type your selection and press return"
$WiresharkFilter = ""
do {
$tmp = read-host $FilterOptionsDialog
} while ("all","u","tcon","tall","p","pt","ptl","pu" -notcontains $tmp)
switch ($tmp)
{
"all" {
# TCP connections with local and remote IP filter - both ports included - udp only listening are included
$ConnectionFilterResolved = "("
$ConnectionFilterResolved += $(foreach ($connection in $TcpAndUdpNetworkStatisticFilteredByProcessID) {
if ([int]$connection.remoteport -eq 0) {
$ConnectionFilter = "(ip.addr eq {0} and {2}.port eq {1})"
$ConnectionFilter -f $connection.LocalAddress,$connection.LocalPort,$connection.Protocol.ToLower()
} else {
$ConnectionFilter = "(ip.addr eq {0} and ip.addr eq {1}) and (tcp.port eq {2} and tcp.port eq {3})"
$ConnectionFilter -f $connection.LocalAddress,$connection.RemoteAddress, $connection.LocalPort, $connection.RemotePort
}
}) -join ") or ("
$ConnectionFilterResolved += ")"
$WiresharkFilter += $ConnectionFilterResolved
}
"u" {
# udp.port only - without remote IP filter
#Building the filter variable
$FilteredPortlist = $TcpAndUdpNetworkStatisticFilteredByProcessID | where {$_.Protocol -eq "UDP"} | foreach { "udp.port eq $($_.LocalPort)"} | sort | get-unique
if ($FilteredPortlist) {
$WiresharkFilter += "(" +
($FilteredPortlist -join ") or (") +
")"
}
}
"tall" {#tall
# TCP connections with local and remote IP filter - both ports included - only listening are included without remote data)
$tcpStatsFilterResolved = "("
$tcpStatsFilterResolved += $(foreach ($connection in ($TcpAndUdpNetworkStatisticFilteredByProcessID | where {$_.Protocol -eq "TCP"} )) {
if ([int]$connection.remoteport -eq 0) {
$TcpFilter = "(ip.addr eq {0} and tcp.port eq {1})"
$TcpFilter -f $connection.LocalAddress,$connection.LocalPort
} else {
$TcpFilter = "(ip.addr eq {0} and ip.addr eq {1}) and (tcp.port eq {2} and tcp.port eq {3})"
$TcpFilter -f $connection.LocalAddress,$connection.RemoteAddress, $connection.LocalPort, $connection.RemotePort
}
}) -join ") or ("
$tcpStatsFilterResolved += ")"
$WiresharkFilter += $tcpStatsFilterResolved
}
"tcon" {
# TCP connections only - listening only ports are not included)
$tcpStatsFilterResolved = "("
$tcpStatsFilterResolved += $(foreach ($connection in ($TcpAndUdpNetworkStatisticFilteredByProcessID | where {$_.Protocol -eq "TCP" -and [int]$_.RemotePort -eq 0} )) {
$TcpFilter = "(ip.addr eq {0} and ip.addr eq {1}) and (tcp.port eq {2} and tcp.port eq {3})"
$TcpFilter -f $connection.LocalAddress,$connection.RemoteAddress, $connection.LocalPort, $connection.RemotePort
}) -join ") or ("
$tcpStatsFilterResolved += ")"
$WiresharkFilter = $tcpStatsFilterResolved
}
"p" {
# Ports only - remote and local
$TCPWiresharkFilter = "tcp.port in {" + ( ($TCPLocalPorts + $TCPRemotePorts | Sort-Object -unique ) -join ", " ) + "}"
$UDPWiresharkFilter = "udp.port in {" + ( $UDPLocalPorts -join ", " ) + "}"
$Or = ( ""," or " )[$TCPConnections.count -gt 0 -and $UDPLocalEndpoints.count -gt 0]
$WiresharkFilter = "$TCPWiresharkFilter$Or$UDPWiresharkFilter"
}
"ptl" {
# Local tcp ports only - remote are excluded
$WiresharkFilter = "tcp.port in {" + ( $TCPLocalPorts -join ", " ) + "}"
}
"pt" {
# tcp ports only - remote and local ports
$WiresharkFilter = "tcp.port in {" + ( ($TCPLocalPorts + $TCPRemotePorts | Sort-Object -unique ) -join ", " ) + "}"
}
"pu" {
# udp ports only - no remote anyway
$WiresharkFilter = "udp.port in {" + ( $UDPLocalPorts -join ", " ) + "}"
}
}
if ($WiresharkFilter.toString().length -gt 5) {
# Output to clipboard
$WiresharkFilter | Set-Clipboard
"The following filter should be in your clipboard already"
""
""
Out-WireSharkSyntax $WiresharkFilter
""
""
"Attention: All filtering is done on network statistic data arrount that time `"$CurrentDateTime`" and the additional $(1 - $count) checks that were done."
"`tThis filter is not perfect, but it works for some cases or is a good template to be customized afterwards."
} else {
"Everything was filtered out by your selections - I got no data to create a filter"
}
""
"Press return to end script"
Read-Host | Out-Null
Here is what it might look like
Another possible result
You may optimize the code for your needs, but for me it is more than enough.
If someone has already found a better/builtin solution for Wireshark, please share your information.

In some cases you can not filter by process id. For example, in my case i needed to sniff traffic from one process. But I found in its config target machine IP-address, added filter ip.dst==someip and voila. It won't work in any case, but for some it's useful.

Get the port number using netstat:
netstat -b
And then use the Wireshark filter:
tcp.port == portnumber

You can check for port numbers with these command examples on wireshark:-
tcp.port==80
tcp.port==14220

Related

How do I know if I'm pulling from Docker Hub? [duplicate]

When doing a docker push or when pulling an image, how does Docker determine if there is a registry server in the image name or if it is a path/username on the default registry (e.g. Docker Hub)?
I'm seeing the following from the 1.1 image specification:
Tag
A tag serves to map a descriptive, user-given name to any single image
ID. Tag values are limited to the set of characters [a-zA-Z_0-9].
Repository
A collection of tags grouped under a common prefix (the name component
before :). For example, in an image tagged with the name my-app:3.1.4,
my-app is the Repository component of the name. A repository name is
made up of slash-separated name components, optionally prefixed by a
DNS hostname. The hostname must follow comply with standard DNS rules,
but may not contain _ characters. If a hostname is present, it may
optionally be followed by a port number in the format :8080. Name
components may contain lowercase characters, digits, and separators. A
separator is defined as a period, one or two underscores, or one or
more dashes. A name component may not start or end with a separator.
For the DNS host name, does it need to be fully qualified with dots, or is "my-local-server" a valid registry hostname? For the name components, I'm seeing periods as valid, which implies "team.user/appserver" is a valid image name. If the registry server is running on port 80, and therefore no port number is needed on the hostname in the image name, it seems like there would be ambiguity between the hostname and the path on the registry server. I'm curious how Docker resolves that ambiguity.
TL;DR: The hostname must contain a . dns separator, a : port separator, or the value "localhost" before the first /. Otherwise the code assumes you want the default registry, Docker Hub.
After some digging through the code, I came across distribution/distribution/reference/reference.go with the following:
// Grammar
//
// reference := name [ ":" tag ] [ "#" digest ]
// name := [hostname '/'] component ['/' component]*
// hostname := hostcomponent ['.' hostcomponent]* [':' port-number]
// hostcomponent := /([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9-]*[a-zA-Z0-9])/
// port-number := /[0-9]+/
// component := alpha-numeric [separator alpha-numeric]*
// alpha-numeric := /[a-z0-9]+/
// separator := /[_.]|__|[-]*/
//
// tag := /[\w][\w.-]{0,127}/
//
// digest := digest-algorithm ":" digest-hex
// digest-algorithm := digest-algorithm-component [ digest-algorithm-separator digest-algorithm-component ]
// digest-algorithm-separator := /[+.-_]/
// digest-algorithm-component := /[A-Za-z][A-Za-z0-9]*/
// digest-hex := /[0-9a-fA-F]{32,}/ ; At least 128 bit digest value
The actual implementation of that is via a regex in distribution/distribution/reference/regexp.go.
But with some digging and poking, I found that there's another check beyond that regex (e.g. you'll get errors with an uppercase hostname if you don't don't include a . or :). And I tracked down the actual split of the name to the following in distribution/distribution/reference/normalize.go:
// splitDockerDomain splits a repository name to domain and remotename string.
// If no valid domain is found, the default domain is used. Repository name
// needs to be already validated before.
func splitDockerDomain(name string) (domain, remainder string) {
i := strings.IndexRune(name, '/')
if i == -1 || (!strings.ContainsAny(name[:i], ".:") && name[:i] != "localhost") {
domain, remainder = defaultDomain, name
} else {
domain, remainder = name[:i], name[i+1:]
}
if domain == legacyDefaultDomain {
domain = defaultDomain
}
if domain == defaultDomain && !strings.ContainsRune(remainder, '/') {
remainder = officialRepoName + "/" + remainder
}
return
}
The important part of that for me is the check for the ., :, or the hostname localhost before the first / in the first if statement. With it, the hostname is split out from before the first /, and without it, the entire name is passed to the default registry hostname.
The image-spec at https://github.com/moby/moby/blob/master/image/spec/v1.1.md has now been updated to say that tags are limited to 128 characters.
The PR thread is here https://github.com/docker/distribution/issues/2248
Some Ruby code is here https://github.com/cyber-dojo/runner/blob/e98bc280c5349cb2919acecb0dfbfefa1ac4e5c3/src/docker/image_name.rb
Some Ruby tests are https://github.com/cyber-dojo/runner/blob/e98bc280c5349cb2919acecb0dfbfefa1ac4e5c3/test_server/image_name_test.rb
Note: Many URL parsing libraries aren't able to parse docker image references / tags, unless they conform to standardized URL format.
Example Ansible Snippet:
- debug: #(FAILS)
msg: "{{ 'docker.io/alpine' | urlsplit() }}"
# ^-- This will fail, because the image reference isn't in standard URL format
# If you can convert the docker image reference to standard URL format
# Then most URL parsing libraries will work correctly
- debug: #(WORKS)
msg: "{{ ('https://' + 'docker.io/alpine') | urlsplit() }}"
# ^-- Example: This becomes standard URL syntax, so it parses correctly
- debug: #(FAILS)
msg: "{{ ('http://' + 'busybox:1.34.1-glibc') | urlsplit('path') }}"
# ^-- Unfortunately, this trick won't work to turn 100% of images into
# Standard URL format for parsing. (This example fails as well)
Based on BMitch's answer I realized a simple if statement algorithmic logic could be used to convert arbitrary docker image references / tags into standardized URL format, which allows them to be parsed by most libraries.
Algorithm in human speak:
1. look for / in $TAG
2. If / not found
Then return ("https://docker.io/" + $TAG)
3. If / found, split $TAG into 2 parts by first /
and test text left of /, to look for ".", ":", or "localhost"
4. If (".", ":", or "localhost" found in text left of 1st /)
Then return (https://" + $TAG)
5. If (".", ":", or "localhost" not found in text left of 1st /)
Then return (https://docker.io/ + $TAG)
(This logic converts docker tags into standardized URL format
so they can be processed by URL parsing libraries.)
Algorithm in Bash:
vi docker_tag_to_standardized_url_format.sh
(Copy paste the following)
#!/bin/bash
#This standardizes the naming of docker images
#Basically busybox --------------------> https://docker.io/busybox
# myregistry.tld/myimage:tag -> https://myregistry.tld/myimage:tag
STDIN=$(cat -)
INPUT=$STDIN
OUTPUT=""
echo "$INPUT" | grep "/" > /dev/null
if [ $? -eq 0 ]; then
echo "$INPUT" | cut -d "/" -f1 | egrep "\.|:|localhost" > /dev/null
#Note: grep considers . as wildcard, \ is escape character to treat \. as .
if [ $? -eq 0 ]; then
OUTPUT="https://$INPUT"
else
OUTPUT="https://docker.io/$INPUT"
fi
else
OUTPUT="https://docker.io/$INPUT"
fi
echo $OUTPUT
Make it executable:
chmod +x ./docker_tag_to_standardized_url_format.sh
Usage Example:
# Test data, to verify against edge cases
A=docker.io/alpine
B=docker.io/rancher/system-upgrade-controller:v0.8.0
C=busybox:1.34.1-glibc
D=busybox
E=rancher/system-upgrade-controller:v0.8.0
F=localhost:5000/helloworld:latest
G=quay.io/go/go/gadget:arms
####################################
echo $A | ./docker_tag_to_standardized_url_format.sh
echo $B | ./docker_tag_to_standardized_url_format.sh
echo $C | ./docker_tag_to_standardized_url_format.sh
echo $D | ./docker_tag_to_standardized_url_format.sh
echo $E | ./docker_tag_to_standardized_url_format.sh
echo $F | ./docker_tag_to_standardized_url_format.sh
echo $G | ./docker_tag_to_standardized_url_format.sh

how to remove zero packets (empty streams) records in wireshark

I am very new to wireshark. in my day to day job i need to remove the packet bytes zero records from captured PCAP file. please help me in this process. attached image is for reference
wireshark packets zero.png
Since you have 47 TCP Streams and 28 that you want to remove, it might be a bit faster to filter for all the TCP streams that you do want to keep since there are only 19 of those.
For the 19 streams you want:
Right-click on the first TCP conversation and choose "Prepare a Filter -> Selected -> A<-->B".
For the next 17 TCP conversations, right-click on each one and choose "Prepare a Filter -> ... And Selected -> A<-->B".
Finally, for the last TCP stream, right-click on the TCP conversation and choose "Apply as Filter -> ... And Selected -> A<-->B".
You may wish to export the resulting filtered packets to a new file via "File -> Export Specified Packets... -> All packets:Displayed" so you won't have to keep filtering for those streams anymore.
If you have a large number of streams to filter, then you are better off scripting something. Here's a script you can use that seems to work well in my testing on my Linux machine. If you're using Windows, you will need to write an equivalent batch file, or you may be able to use it as is if you have Cygwin installed.
#!/bin/sh
# Check usage
if [ ${#} -lt 2 ] ; then
echo "Usage: $0 <infile> <outfile>"
exit 0
fi
infile=${1}
outfile=${2}
# TODO: Could also pass the filter on the command-line too.
filter="ip.dst eq 192.168.10.44 and tcp.len > 0"
stream_filter=
for stream in $(tshark -r ${infile} -Y "${filter}" -T fields -e tcp.stream | sort -u | tr -d '\r')
do
if [[ -z ${stream_filter} ]] ; then
stream_filter="tcp.stream eq ${stream}"
else
stream_filter+=" or tcp.stream eq ${stream}"
fi
done
tshark -r ${infile} -Y "${stream_filter}" -w ${outfile}
echo "Wrote ${outfile}"

How are Docker image names parsed?

When doing a docker push or when pulling an image, how does Docker determine if there is a registry server in the image name or if it is a path/username on the default registry (e.g. Docker Hub)?
I'm seeing the following from the 1.1 image specification:
Tag
A tag serves to map a descriptive, user-given name to any single image
ID. Tag values are limited to the set of characters [a-zA-Z_0-9].
Repository
A collection of tags grouped under a common prefix (the name component
before :). For example, in an image tagged with the name my-app:3.1.4,
my-app is the Repository component of the name. A repository name is
made up of slash-separated name components, optionally prefixed by a
DNS hostname. The hostname must follow comply with standard DNS rules,
but may not contain _ characters. If a hostname is present, it may
optionally be followed by a port number in the format :8080. Name
components may contain lowercase characters, digits, and separators. A
separator is defined as a period, one or two underscores, or one or
more dashes. A name component may not start or end with a separator.
For the DNS host name, does it need to be fully qualified with dots, or is "my-local-server" a valid registry hostname? For the name components, I'm seeing periods as valid, which implies "team.user/appserver" is a valid image name. If the registry server is running on port 80, and therefore no port number is needed on the hostname in the image name, it seems like there would be ambiguity between the hostname and the path on the registry server. I'm curious how Docker resolves that ambiguity.
TL;DR: The hostname must contain a . dns separator, a : port separator, or the value "localhost" before the first /. Otherwise the code assumes you want the default registry, Docker Hub.
After some digging through the code, I came across distribution/distribution/reference/reference.go with the following:
// Grammar
//
// reference := name [ ":" tag ] [ "#" digest ]
// name := [hostname '/'] component ['/' component]*
// hostname := hostcomponent ['.' hostcomponent]* [':' port-number]
// hostcomponent := /([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9-]*[a-zA-Z0-9])/
// port-number := /[0-9]+/
// component := alpha-numeric [separator alpha-numeric]*
// alpha-numeric := /[a-z0-9]+/
// separator := /[_.]|__|[-]*/
//
// tag := /[\w][\w.-]{0,127}/
//
// digest := digest-algorithm ":" digest-hex
// digest-algorithm := digest-algorithm-component [ digest-algorithm-separator digest-algorithm-component ]
// digest-algorithm-separator := /[+.-_]/
// digest-algorithm-component := /[A-Za-z][A-Za-z0-9]*/
// digest-hex := /[0-9a-fA-F]{32,}/ ; At least 128 bit digest value
The actual implementation of that is via a regex in distribution/distribution/reference/regexp.go.
But with some digging and poking, I found that there's another check beyond that regex (e.g. you'll get errors with an uppercase hostname if you don't don't include a . or :). And I tracked down the actual split of the name to the following in distribution/distribution/reference/normalize.go:
// splitDockerDomain splits a repository name to domain and remotename string.
// If no valid domain is found, the default domain is used. Repository name
// needs to be already validated before.
func splitDockerDomain(name string) (domain, remainder string) {
i := strings.IndexRune(name, '/')
if i == -1 || (!strings.ContainsAny(name[:i], ".:") && name[:i] != "localhost") {
domain, remainder = defaultDomain, name
} else {
domain, remainder = name[:i], name[i+1:]
}
if domain == legacyDefaultDomain {
domain = defaultDomain
}
if domain == defaultDomain && !strings.ContainsRune(remainder, '/') {
remainder = officialRepoName + "/" + remainder
}
return
}
The important part of that for me is the check for the ., :, or the hostname localhost before the first / in the first if statement. With it, the hostname is split out from before the first /, and without it, the entire name is passed to the default registry hostname.
The image-spec at https://github.com/moby/moby/blob/master/image/spec/v1.1.md has now been updated to say that tags are limited to 128 characters.
The PR thread is here https://github.com/docker/distribution/issues/2248
Some Ruby code is here https://github.com/cyber-dojo/runner/blob/e98bc280c5349cb2919acecb0dfbfefa1ac4e5c3/src/docker/image_name.rb
Some Ruby tests are https://github.com/cyber-dojo/runner/blob/e98bc280c5349cb2919acecb0dfbfefa1ac4e5c3/test_server/image_name_test.rb
Note: Many URL parsing libraries aren't able to parse docker image references / tags, unless they conform to standardized URL format.
Example Ansible Snippet:
- debug: #(FAILS)
msg: "{{ 'docker.io/alpine' | urlsplit() }}"
# ^-- This will fail, because the image reference isn't in standard URL format
# If you can convert the docker image reference to standard URL format
# Then most URL parsing libraries will work correctly
- debug: #(WORKS)
msg: "{{ ('https://' + 'docker.io/alpine') | urlsplit() }}"
# ^-- Example: This becomes standard URL syntax, so it parses correctly
- debug: #(FAILS)
msg: "{{ ('http://' + 'busybox:1.34.1-glibc') | urlsplit('path') }}"
# ^-- Unfortunately, this trick won't work to turn 100% of images into
# Standard URL format for parsing. (This example fails as well)
Based on BMitch's answer I realized a simple if statement algorithmic logic could be used to convert arbitrary docker image references / tags into standardized URL format, which allows them to be parsed by most libraries.
Algorithm in human speak:
1. look for / in $TAG
2. If / not found
Then return ("https://docker.io/" + $TAG)
3. If / found, split $TAG into 2 parts by first /
and test text left of /, to look for ".", ":", or "localhost"
4. If (".", ":", or "localhost" found in text left of 1st /)
Then return (https://" + $TAG)
5. If (".", ":", or "localhost" not found in text left of 1st /)
Then return (https://docker.io/ + $TAG)
(This logic converts docker tags into standardized URL format
so they can be processed by URL parsing libraries.)
Algorithm in Bash:
vi docker_tag_to_standardized_url_format.sh
(Copy paste the following)
#!/bin/bash
#This standardizes the naming of docker images
#Basically busybox --------------------> https://docker.io/busybox
# myregistry.tld/myimage:tag -> https://myregistry.tld/myimage:tag
STDIN=$(cat -)
INPUT=$STDIN
OUTPUT=""
echo "$INPUT" | grep "/" > /dev/null
if [ $? -eq 0 ]; then
echo "$INPUT" | cut -d "/" -f1 | egrep "\.|:|localhost" > /dev/null
#Note: grep considers . as wildcard, \ is escape character to treat \. as .
if [ $? -eq 0 ]; then
OUTPUT="https://$INPUT"
else
OUTPUT="https://docker.io/$INPUT"
fi
else
OUTPUT="https://docker.io/$INPUT"
fi
echo $OUTPUT
Make it executable:
chmod +x ./docker_tag_to_standardized_url_format.sh
Usage Example:
# Test data, to verify against edge cases
A=docker.io/alpine
B=docker.io/rancher/system-upgrade-controller:v0.8.0
C=busybox:1.34.1-glibc
D=busybox
E=rancher/system-upgrade-controller:v0.8.0
F=localhost:5000/helloworld:latest
G=quay.io/go/go/gadget:arms
####################################
echo $A | ./docker_tag_to_standardized_url_format.sh
echo $B | ./docker_tag_to_standardized_url_format.sh
echo $C | ./docker_tag_to_standardized_url_format.sh
echo $D | ./docker_tag_to_standardized_url_format.sh
echo $E | ./docker_tag_to_standardized_url_format.sh
echo $F | ./docker_tag_to_standardized_url_format.sh
echo $G | ./docker_tag_to_standardized_url_format.sh

output as a single row from dhcp output

I'm running dhcp query through powershell. I am getting the information I want to see but not in a format I need.
Sample:
data I'm working with the command:
netsh dhcp server \\dhcpserver scope 10.0.1.0 show clientsvq
gets output of
10.0.1.21 - 255.255.255.0 -00-23-7d-e9-45-58 -2/19/2016 5:13:50 PM -D -BUILD-01.example.com -NonQuarantined -INACTIVE -No
10.0.1.22 - 255.255.255.0 -00-23-7d-e9-54-38 -2/19/2016 5:13:55 PM -D -BUILD-02.example.com -NonQuarantined -INACTIVE -No
10.0.1.23 - 255.255.255.0 -00-23-7d-e8-cf-80 -6/11/2016 11:30:30 AM -D -BUILD-03.example.com -NonQuarantined -INACTIVE -No
What I want as an output is this:
10.0.1.21 00237de94558 BUILD-01.example.com
10.0.1.22 00237de95438 BUILD-02.example.com
10.0.1.23 00237de8cf80 BUILD-03.example.com
tried formatting the output by extracting the desired column and stripping off the special characters, command:
netsh dhcp server \\dhcpserver scope 10.0.1.0 show clientsvq | findstr example.com | foreach {"$(($_ -split '\s+',10)[0,3])" -replace "\-",""; "$(($_ -split '\s+',10)[8])" -replace "^-","";}
I get the following output format:
10.0.1.21 00237de94558
BUILD-01.example.com
10.0.1.22 00237de95438
BUILD-02.example.com
10.0.1.23 00237de8cf80
BUILD-03.example.com
Can someone point me out where my logic is flawed? I can't seemed to get to the much needed correct output. Thanks in advance.
Here's another way to parse the output and return rich objects instead of text:
netsh dhcp server \\dhcpserver scope 10.0.1.0 show clientsvq | foreach {
$values = ($_ -split '\s+-')[0,2,5]
if($values[0] -as [ipaddress])
{
New-Object PSObject -Property #{
IPAddress = [ipaddress]$values[0].Trim()
MAC = $values[1].Trim() -replace '-'
Name = $values[2].Trim()
}
}
}
Here's a version that seems to work. I simplified what you attempted by storing the array returned by the -split operator in a variable to avoid the unwinding that would happen if it was sent down the pipeline. This way we can access each column in a straightforward way. Note, to simplify, I just replaced your dhcp command with $dhcpcommand:
$dhcpcommand | select-string example.com | %{$cols = #($_ -split '\s+'); "$($cols[0]) $($cols[3] -replace '-','') $($cols[8])"}

How to capture count of particular packets with tcpdump?

I am interested in getting a count of all LDAP/Kerberos/DNS packets.
I tried the following, but this captures the full packet.
tcpdump -i any -Z root "tcp port 389 or tcp port 88 or udp port 53" -w ~/ldap_kerberos_dns.cap
Is there a way I can just capture how many ldap/Kerberos/DNS packets were exchanged
without actually capturing the full packet.
Expected output should be something like:
LDAP: 100
Kerberos: 200
UDP: 300
Take a look at tshark statistics:
$ tshark -r 04.pcap -q -z io,phs
===================================================================
Protocol Hierarchy Statistics
Filter:
eth frames:649 bytes:124780
ipv6 frames:605 bytes:116558
udp frames:212 bytes:33686
dhcpv6 frames:171 bytes:28044
dns frames:25 bytes:2914
ntp frames:10 bytes:1300
cldap frames:6 bytes:1428
icmpv6 frames:80 bytes:7008
tcp frames:313 bytes:75864
nbss frames:108 bytes:24063
smb frames:7 bytes:1554
smb2 frames:101 bytes:22509
tcp.segments frames:1 bytes:103
dcerpc frames:16 bytes:4264
epm frames:2 bytes:544
tcp.segments frames:1 bytes:214
drsuapi frames:8 bytes:2352
kerberos frames:16 bytes:9358
tcp.segments frames:8 bytes:2130
dcerpc.cn_deseg_req frames:1 bytes:1514
ldap frames:16 bytes:5945
tcp.segments frames:3 bytes:1101
ldap frames:1 bytes:803
ip frames:40 bytes:8018
udp frames:40 bytes:8018
nbdgm frames:30 bytes:7300
smb frames:30 bytes:7300
mailslot frames:30 bytes:7300
browser frames:30 bytes:7300
dns frames:10 bytes:718
arp frames:4 bytes:204
===================================================================
See the man-page for more information.
tshark approach
If you have Wireshark (based on question tags, not the actual question) then tshark along the lines of #joke's comment is one way to go, if you don't mind its verbose stats output:
tshark -i any -n -q -z 'io,stat,0,FRAMES()tcp.port==389,FRAMES()tcp.port==88,FRAMES()udp.port==53'
Capturing on Pseudo-device that captures on all interfaces
^C142 packets captured
=============================================
| IO Statistics |
| |
| Interval size: 4.319 secs (dur) |
| Col 1: FRAMES()tcp.port==389 |
| 2: FRAMES()tcp.port==88 |
| 3: FRAMES()udp.port==53 |
|-------------------------------------------|
| |1 |2 |3 |
| Interval
| Interval | FRAMES | FRAMES | FRAMES |
|-------------------------------------------|
| 0.000 <> 4.319 | 100 | 200 | 300 |
=============================================
Though the output from that is verbose, I don't think you can get closer with tshark alone. Another approach, still more verbose, would be:
tshark -q -z io,phs "tcp port 389 or tcp port 88 or udp port 53"
These two commands do not write a capture file. Use Ctrl+C when you're ready to quit, or see the comment below about -a and autostop conditions. One caveat is that those statistics rely on the protocol dissectors, not the source/destination ports, so there can be discrepancies (e.g. connections with no data, or content non-compliant with protocol), "malformed" packets will be reported instead. It also means you won't get reliable results if you optimize and only capture 40 bytes per packet ("short" TCP or UDP packets will be reported instead).
tcpdump approach
A simple, but inelegant way is to run multiple tcpdump instances (assume bash as a shell) --
for pp in "tcp port 88" "tcp port 389" "udp port 53"; do
tcpdump -i any -Z root $pp -w /dev/null 2> ${pp// /-}.stats &
done
Packets are not written to a capture file (discarded via /dev/null).
Then wait as required, kill the tcpdump processes (by PID as listed, or kill %1 %2 %3 if no other background jobs), and inspect the .stats files:
grep captured *.stats
tcp-port-389.stats:0 packets captured
tcp-port-88.stats:0 packets captured
udp-port-53.stats:4 packets captured
perl approach
Since this is stackoverflow, here's a quick and dirty perl/libpcap solution (just add error handling):
#!/usr/bin/perl
use Net::Pcap;
use NetPacket::Ethernet;
use NetPacket::IP;
use NetPacket::TCP;
use strict;
use warnings;
my ($dev,$err,$address, $netmask,$pcap,$filter,$cfilter,$fd,%stats);
my $bail=0;
my ($rin,$rout)=('','');
$SIG{INT} = sub { print "quit...\n"; $bail=1; };
$filter='(tcp port 88 or tcp port 389 or udp port 53)';
$dev=Net::Pcap::lookupdev(\$err);
Net::Pcap::lookupnet($dev, \$address, \$netmask, \$err);
$pcap = Net::Pcap::open_live($dev, 60, 1, 0, \$err);
Net::Pcap::pcap_setnonblock($pcap, 1, \$err);
$fd=Net::Pcap::pcap_get_selectable_fd($pcap);
vec($rin,$fd,1)=1;
Net::Pcap::compile( $pcap, \$cfilter, $filter, 0, $netmask);
Net::Pcap::setfilter($pcap, $cfilter);
while (Net::Pcap::dispatch($pcap,0, \&handlepackets, '')>=0) {
select($rout=$rin,undef,undef,0.250);
printf("."); $|=1;
$bail && last;
};
Net::Pcap::freecode($cfilter);
Net::Pcap::close($pcap);
sub handlepackets() {
my ($user_data, $header, $packet) = #_;
my $ether = NetPacket::Ethernet::strip($packet);
my $ip = NetPacket::IP->decode($ether);
my $tcp = NetPacket::TCP->decode($ip->{'data'});
if ($tcp->{'src_port'} < 1024) {
$stats{$tcp->{'src_port'}}++;
} elsif ($tcp->{'dest_port'} < 1024) {
$stats{$tcp->{'dest_port'}}++;
}
}
END {
if (keys %stats) {
for my $kk (sort keys %stats) {
my ($name, $aliases, $port, $prot)=getservbyport($kk,"tcp");
printf("%12s %4i\n",$name,$stats{$kk});
}
} else { printf("No stats...\n"); }
}

Resources