I have the following setup:
A FPGA sending out data on UART at a baudrate of 3Mbps. The data transmitted is a chunk of 1024 bytes sent at a variable periodicity ranging from 20ms to 200ms. (So even in the worst case, datarate is far from 3Msps)
A FTDI 232RG
A piece of python running on my computer (Windows), doing basically : opening a COM port with pyserial, 3Msps, polling the in_waiting until it reaches the size of a packet (1024 bytes), formatting the packet received and print it on the screen
The script works well for low repetition frequency, but I face issues with higher repetitions (typically 20ms). When the periodicity in 20ms I eventually end up getting a buffer overflow somewhere before the in_waiting. I checked the timing of my python loop and it takes about 4ms. So it looks like there is something upstream (in the FTDI or Windows) that feeds the pyserial buffer with more than one packet within the 4ms following a packet.
I tried changing the FTDI latency in the driver (from 16ms default down to a few ms) but it does not seem to help.
I am currently clueless about what is happening. Would you have any advice about how to understand better what is happening?
Thanks for your help!
You could create a "loop" between TX and RX and run the following code (tested with a FT2232H, so mostlikely you need to change the identifier string):
import time
import serial
import serial.tools.list_ports
print([(x[0],x[2]) for x in serial.tools.list_ports.comports()])
port = [x[0] for x in serial.tools.list_ports.comports() if "FT4Q1LJFB" in x[2]][0]
ser = serial.Serial(port,12000000)
while True:
t0 = time.time()
counter = 0
for i in range(1000):
ser.write([1]*3000)
recv = ser.read(ser.inWaiting())
delta_t = time.time() - t0
counter += len(recv)
print(counter / delta_t)
For me the following output is shown
[('COM7', 'USB VID:PID=0403:6010 SER=FT4Q1LJFA'), ('COM8', 'USB VID:PID=0403:6010 SER=FT4Q1LJFB')]
0.0
0.0
0.0
0.0
96787.81184093593
1201991.0268273412
1201197.0857713912
1201166.9350959768
1201445.4072856384
You will notice that it is 0.0 in the beginning. This is because I connected RX and TX after starting the program resulting in a ramping up of the received bytes. This is the "default" mode meaning 8 bits + 1 start bit + 1 stop bit = 10 bits per word which explains why "only" 1.2 Mbytes per second are transmitted.
I'm trying to downscale a bit of old data in influxdb. Basically, my measurements contain values from several different hosts, and I need to keep them separate. So, I'm trying to aggregate data to 5m intervals, but keep the host values separate.
I tried as follows, but that fails with a syntax error "mixing multiple selector functions with tags or fields is not supported":
CREATE CONTINUOUS QUERY cq_5m_net ON telegraf
BEGIN SELECT
max(bytes_recv) AS bytes_recv, max(bytes_sent) AS bytes_sent,
max(drop_in) AS drop_in, max(drop_out) AS drop_out,
max(err_in) AS err_in, max(err_out) AS err_out,
host INTO telegraf.rp_5m.net
FROM telegraf.autogen.net GROUP BY time(5m), *
END
Can this be accomplished some other way?
Also, raw dataset looks like this (a bit simplified):
time host bytes_recv bytes_sent drop_in drop_out
---- ---- ---------- ---------- ------- --------
14805418900000 web2.x.net 6358608058 5036008977 148414 0
14805419000000 web3.x.net 6358600106 5036501326 148414 0
14805419100000 web2.x.net 6358604711 5036770259 148414 0
14805419200000 web3.x.net 6358602310 5038123405 148414 0
14805419300000 web2.x.net 6358600274 5039360985 148414 0
14805419400000 web2.x.net 6358618730 5039651324 148414 0
14805419500000 web2.x.net 6358616379 5030898027 148414 0
14805419600000 web3.x.net 6358612454 5031543210 148414 0
14805419700000 web3.x.net 6358611562 5032444497 148414 0
14805419800000 web3.x.net 6358610357 5032699567 148414 0
That explains what kind of issue you have.
The downright generic answer to your question ("Can this be accomplished some other way?") is - yes, the other-way's name is Kapacitor.
For more detailed answer regarding the query, you provide more details first, please - how your measurement looks like?
But the main question here: why do you need that permanent downsampling at all?
Data storage is cheap (as data records seems to be pretty lean), time interval based aggregations on the fly are cheap and fast for properly shaped data - you just can do it any time you need it in a blink of an eye.
Can you justify why do you roll this way?
I want to create a CSV to import it on excel, containing all the packet details shown in wireshark.
Each row should correspond to a packet and the columns to the field details.
Using the following tshark command:
tshark -r mycapturefile.cap -E -V
I can show the information I need like:
Frame 1077: 42 bytes on wire (336 bits), 42 bytes captured (336 bits)
Encapsulation type: Ethernet (1)
Arrival Time: Aug 15, 2017 14:02:27.095521000 EDT
[Time shift for this packet: 0.000000000 seconds]
Epoch Time: 1502820147.095521000 seconds
and other packet details...
What I want is that information provided with -V, so the -T fields option in wireshark is discarded. Wireshark export options also don't provide the data I need, only the pdml format, but I think is more tedius to parse.
I have searched for a tool, a script or parser with no results. Since each packet is different, make a personal parser may be difficult/tedious and considering people can extract this information but provide no sources of how to do it, there must be a method or tool that can do it.
Do you know any tool, script or method that already do this?
Thanks in advance.
There is a ton of information coming down. You gotta use that -Y display filter to whittle it down. The resulting text can then be parsed.
Try -Y "frame.number == 1077" -V and then parse the text that is returned.
In my case I wanted certificate information.
Function GetCertsFromWireSharkPackets2 ($CERTTEXT){
foreach($Cert in($CERTTEXT|?{$_ -match "Source:.*\d{1,3}\.\d{1,3}\.\d{1,3}\.|Destination:.*\d{1,3}\.\d{1,3}\.\d{1,3}\.|Certificate:"} | %{$_.trim() -replace 'Source:','|Source:' -replace ":",'=' }) -join "`n"| %{$_.split('|')}|?{$_}) {
$Cert|%{$Props = [regex]::matches($_,"(?sim)(?<=^).*?(?=\=)").value ; $Dups = [regex]::matches($Props,"(?sim)\b(\w+)\s+\1\b").value.split(' ') ; $values = [regex]::matches($_,"(?sim)(?<=\=).*?(?=$)").value.trim()}
$PropsNoDups = ($Props -join "`n").replace(($Dups|select -first 1),'').split(10)|?{$_} ;
if(($PropsNoDups.count + $Dups.count) -ne $Props.count){$dups+=($dups|select -First 1)}
for($X=1;$X -lt $Dups.count;$X++){$dups[$X] +=$X}
$ValidProps = $PropsNoDups+$Dups ; $StitchCount = $Values.Count
$ValidP_V = For($x=0;$x -lt $StitchCount;$x++){ '"'+$ValidProps[$x] + '"="' + $Values[$x] +'"'} ;$ValidP_V =($ValidP_V -join "`n")|?{$_} ; $ExpText = "New-Object psobject -Property #{`n"+$ValidP_V+"`n}"
Invoke-Expression($ExpText)|select Source, Destination, Certificate, Certificate1, Certificate2, Certificate3
} }
#Click refresh on a few browser tabs to generate traffic.
$CERTTEXT = .\tshark.exe -i 'Wi-Fi' -Y "ssl.handshake.certificate" -V -a duration:30
GetCertsFromWireSharkPackets2 $CERTTEXT
Source : cybersandwich.com (107.170.193.139)
Destination : KirtCarson.com (222.168.3.118)
Certificate : 3082057e30820466a0030201020212030e2782075e8f90f5... (id-at-commonName=multi.zeall.us)
Certificate1 : 308204923082037aa00302010202100a0141420000015385... (id-at-commonName=Let's Encrypt Authority
X3,id-at-organizationName=Let's Encrypt,id-at-countryName=US)
Certificate2 :
Certificate3 :
I'm writing a program which uses ZeroMQ to communicate with other running programs on the same machine. I want to choose a port number at run time to avoid the possibility of collisions. Here is an example of a piece of code I wrote to accomplish this.
#!/usr/bin/perl -Tw
use strict;
use warnings;
my %in_use;
{
local $ENV{PATH} = '/bin:/usr/bin';
%in_use = map { $_ => 1 } split /\n/, qx(
netstat -aunt |\
awk '{print \$4}' |\
grep : |\
awk -F: '{print \$NF}'
);
}
my ($port) = grep { not $in_use{$_} } 50_000 .. 59_999;
print "$port is available\n";
The procedure is:
invoke netstat -aunt
parse the result
choose the first port on a fixed range which doesn't appear on netstat list.
Is there a system utility better suited to accomplishing this?
context = zmq.Context()
socket = context.socket(zmq.ROUTER)
port_selected = socket.bind_to_random_port('tcp://*', min_port=6001, max_port=6004, max_tries=100)
First of all, from your code it looks like you are trying to choose a port between 70000 and 79999. You do know that port numbers only go up to 65535, right? :-)
You can certainly do it this way, even though there are a couple of problems with the approach. The first problem is that netstat output differs between different operating systems so it's hard to do it portably. The second problem is that you still need to wrap the code in a loop which tries again to find a new port number in case it was not possible to bind to the chosen port number, because there's a race condition between ascertaining that the port is free and actually binding to it.
If the library you are using allows you to specify the port number as 0 and allows you to call getsockname() on the socket after it is bound, then you should just do that. Using 0 makes the system choose any free port number, and with getsockname() you can find out which port it chose.
Failing that, it would probably actually be more efficient to not bother calling netstat and just try to find to different port numbers in a loop. If you succeed, break from the loop. If you fail, increment the port number by 1, go back, and try again.
Okay, this is a Windows specific question.
I need to be able to access the ink levels of a printer connected to a computer. Possibly direct connection, or a network connection.
I recognize that it will likely be different for each printer (or printer company at least) but where can I find the information of how they reveal ink levels to a PC. Also, what is the best language to read this information in?
Okay, this is a OS agnostic answer... :-)
If the printer isn't a very cheapo model, it will have built-in support for SNMP (Simple Network Management Protocol). SNMP queries can return current values from the network devices stored in their MIBs (Management Information Bases).
For printers there's a standard defined called Printer MIB. The Printer MIB defines standard names and tree locations (OIDs == Object Identifiers in ASN.1 notation) for prtMarkerSuppliesLevel which in the case of ink marking printers map to ink levels.
Be aware that SNMP also allows private extensions to the standard MIBs. Most printer vendors do hide many additional pieces of information in their "private MIBs", though the standard info should always be available through the queries of the Printer MIB OIDs.
Practically every programming language has standard libraries which can help you to make specific SNMP queries from your own application.
One such implementation is Open Source, called Net-SNMP, which also comes with a few powerfull commandline tools to run SNMP queries.
I think the OID to query all levels for all inks is .1.3.6.1.2.1.43.11.1.1.9 (this webpage confirms my believe) but I cannot verify that right now, because I don't have a printer around in my LAN at the moment. So Net-SNMP's snmpget command to query ink levels should be something like:
snmpget \
-c public \
192.168.222.111 \
".1.3.6.1.2.1.43.11.1.1.9"
where public is the standard community string and 192.168.222.111 your printer's IP address.
I have an SNMP-capable HP 8600 pro N911a around to do some digging, so the following commands may help you a bit. Beware that this particular model has some firmware problems, you can't query "magenta" with snmpget, but you see a value with snmpwalk (which does some kind of recursive drill-down).
OLD: You can query the names and sequence of values, but I couldn't find the "max value" to calculate a clean percentage so far ;(. I'm guessing so far the values are relative to 255, so dividing by 2.55 yields a percentage.
Update: Marcelo's hint was great! From Registers .8.* you can read the max level per cartridge, and I was totally wrong assuming the max value can only be an 8-bit value. I have updated the sample script to read the max values and calculate c
There is also some discussion over there at Cacti forums.
One answer confirms that the ink levels are measured as percent (value 15 is "percent" in an enumeration):
# snmpwalk -v1 -c public 192.168.100.173 1.3.6.1.2.1.43.11.1.1.7
SNMPv2-SMI::mib-2.43.11.1.1.7.0.1 = INTEGER: 15
SNMPv2-SMI::mib-2.43.11.1.1.7.0.2 = INTEGER: 15
SNMPv2-SMI::mib-2.43.11.1.1.7.0.3 = INTEGER: 15
SNMPv2-SMI::mib-2.43.11.1.1.7.0.4 = INTEGER: 15
You need to install the net-snmp package. If you're not on Linux you might need some digging for SNMP command line tools for your preferred OS.
# snmpwalk -v1 -c public 192.168.100.173 1.3.6.1.2.1.43.11.1.1.6.0
SNMPv2-SMI::mib-2.43.11.1.1.6.0.1 = STRING: "black ink"
SNMPv2-SMI::mib-2.43.11.1.1.6.0.2 = STRING: "yellow ink"
SNMPv2-SMI::mib-2.43.11.1.1.6.0.3 = STRING: "cyan ink"
SNMPv2-SMI::mib-2.43.11.1.1.6.0.4 = STRING: "magenta ink"
# snmpwalk -v1 -c public 192.168.100.173 1.3.6.1.2.1.43.11.1.1.9.0
SNMPv2-SMI::mib-2.43.11.1.1.9.0.1 = INTEGER: 231
SNMPv2-SMI::mib-2.43.11.1.1.9.0.2 = INTEGER: 94
SNMPv2-SMI::mib-2.43.11.1.1.9.0.3 = INTEGER: 210
SNMPv2-SMI::mib-2.43.11.1.1.9.0.4 = INTEGER: 174
# snmpwalk -v1 -c praxis 192.168.100.173 1.3.6.1.2.1.43.11.1.1.8.0
SNMPv2-SMI::mib-2.43.11.1.1.8.0.1 = INTEGER: 674
SNMPv2-SMI::mib-2.43.11.1.1.8.0.2 = INTEGER: 240
SNMPv2-SMI::mib-2.43.11.1.1.8.0.3 = INTEGER: 226
SNMPv2-SMI::mib-2.43.11.1.1.8.0.4 = INTEGER: 241
On my Linux box I use the following script to do some pretty-printing:
#!/bin/sh
PATH=/opt/bin${PATH:+:$PATH}
# get current ink levels
eval $(snmpwalk -v1 -c praxis 192.168.100.173 1.3.6.1.2.1.43.11.1.1.6.0 |
perl -ne 'print "c[$1]=$2\n" if(m!SNMPv2-SMI::mib-2.43.11.1.1.6.0.(\d) = STRING:\s+"(\w+) ink"!i);')
# get max ink level per cartridge
eval $(snmpwalk -v1 -c praxis 192.168.100.173 1.3.6.1.2.1.43.11.1.1.8.0 |
perl -ne 'print "max[$1]=$2\n" if(m!SNMPv2-SMI::mib-2.43.11.1.1.8.0.(\d) = INTEGER:\s+(\d+)!i);')
snmpwalk -v1 -c praxis 192.168.100.173 1.3.6.1.2.1.43.11.1.1.9.0 |
perl -ne '
my #c=("","'${c[1]}'","'${c[2]}'","'${c[3]}'","'${c[4]}'");
my #max=("","'${max[1]}'","'${max[2]}'","'${max[3]}'","'${max[4]}'");
printf"# $c[$1]=$2 (%.0f)\n",$2/$max[$1]*100
if(m!SNMPv2-SMI::mib-2.43.11.1.1.9.0.(\d) = INTEGER:\s+(\d+)!i);'
An alternative approach could be using ipp. While most of the printers I tried support both, I found one which only worked with ipp and one that only worked for me with snmp.
Simple approach with ipptool:
Create file colors.ipp:
{
VERSION 2.0
OPERATION Get-Printer-Attributes
GROUP operation-attributes-tag
ATTR charset "attributes-charset" "utf-8"
ATTR naturalLanguage "attributes-natural-language" "en"
ATTR uri "printer-uri" $uri
ATTR name "requesting-user-name" "John Doe"
ATTR keyword "requested-attributes" "marker-colors","marker-high-levels","marker-levels","marker-low-levels","marker-names","marker-types"
}
Run:
ipptool -v -t ipp://192.168.2.126/ipp/print colors.ipp
The response:
"colors.ipp":
Get-Printer-Attributes:
attributes-charset (charset) = utf-8
attributes-natural-language (naturalLanguage) = en
printer-uri (uri) = ipp://192.168.2.126/ipp/print
requesting-user-name (nameWithoutLanguage) = John Doe
requested-attributes (1setOf keyword) = marker-colors,marker-high-levels,marker-levels,marker-low-levels,marker-names,marker-types
colors [PASS]
RECEIVED: 507 bytes in response
status-code = successful-ok (successful-ok)
attributes-charset (charset) = utf-8
attributes-natural-language (naturalLanguage) = en-us
marker-colors (1setOf nameWithoutLanguage) = #00FFFF,#FF00FF,#FFFF00,#000000,none
marker-high-levels (1setOf integer) = 100,100,100,100,100
marker-levels (1setOf integer) = 6,6,6,6,100
marker-low-levels (1setOf integer) = 5,5,5,5,5
marker-names (1setOf nameWithoutLanguage) = Cyan Toner,Magenta Toner,Yellow Toner,Black Toner,Waste Toner Box
marker-types (1setOf keyword) = toner,toner,toner,toner,waste-toner
marker-levels has current toner/ink levels, marker-high-levels are maximus (so far I've only seen 100s here), marker-names describe meaning of each field (tip: for colors you may want to strip everything after first space, many printers include cartridge types in this field).
Note: the above is with cups 2.3.1. With 2.2.1 I had to specify the keywords as one string instead ("marker-colors,marker-h....). Or it can be left altogether, then all keywords are returned.
More on available attributes (may differ between printers): https://www.cups.org/doc/spec-ipp.html
More on executing ipp calls (including python examples): https://www.pwg.org/ipp/ippguide.html
I really liked tseeling's approach!
Complementarily, I found out that the max value for the OID ... .9 is not 255 as guessed by him, but it actually varies per individual cartridge. The values can be obtained from OID .1.3.6.1.2.1.43.11.1.1.8 (the results obtained by dividing by these values match the ones obtained by running hp-inklevels command from hplip.
I wrote my own script that output CSVs like below (suppose printer IP addr is 192.168.1.20):
# ./hpink 192.168.1.20
black,73,366,19.9454
yellow,107,115,93.0435
cyan,100,108,92.5926
magenta,106,114,92.9825
values are in this order: <color_name>,<level>,<maxlevel>,<percentage>
The script source (one will notice I usually prefer awk over perl when the puzzle is simple enough):
#!/bin/sh
snmpwalk -v1 -c public $1 1.3.6.1.2.1.43.11.1.1 | awk '
/.*\.6\.0\./ {
sub(/.*\./,"");
split($0,TT,/[ "]*/);
color[TT[1]]=TT[4];
}
/.*\.8\.0\./ {
sub(/.*\./,"");
split($0,TT,/[ "]*/);
maxlevel[TT[1]]=TT[4];
}
/.*\.9\.0\./ {
sub(/.*\./,"");
split($0,TT,/[ "]*/);
print color[TT[1]] "," TT[4] "," maxlevel[TT[1]] "," TT[4] / maxlevel[TT[1]] * 100;
}'