How to use add_packet_field in a Wireshark Lua dissector? - lua

I am stumbling my way through writing a dissector for our custom protocol in Lua. While I have basic field extraction working, many of our fields have scale factors associated with them. I'd like to present the scaled value in addition to the raw extracted value.
It seems to me tree_item:add_packet_field is tailor-made for this purpose. Except I can't get it to work.
I found Mika's blog incredibly helpful, and followed his pattern for breaking my dissector into different files, etc. That's all working.
Given a packet type "my_packet", I have a 14-bit signed integer "AOA" that I can extract just fine
local pref = "my_packet"
local m = {
aoa = ProtoField.new("AOA", pref .. ".aoa", ftypes.INT16, nil, base.DEC, 0x3FFF, "angle of arrival measurement"),
}
local option=2
local aoa_scale = 0.1
function m.parse(tree_arg, buffer)
if option == 1 then
-- basic field extraction. This works just fine. The field is extracted and added to the tree
tree_arg:add(m.aoa, buffer)
elseif option == 2 then
-- This parses and runs. The item is decoded and added to the tree,
-- but the value of 'v' is always nil
local c,v = tree_arg:add_packet_field(m.aoa, buffer, ENC_BIG_ENDIAN)
-- this results in an error, doing arithmetic on 'nil'
c:append_text(" (scaled= " .. tostring(v*aoa_scale) .. ")")
end
end
(I use ProtoField.new instead of any of the type-specific variants for consistency in declaring my fields)
The documentation for add_packet_field says that the encoding argument is mandatory.
There is a README in the source code that says ENC_BIG_ENDIAN should be specified for network byte-order data (mine is). I know that section is for proto_tree_add_item, but I traced the code far enough to see that add_packet_field ends up passing the encoding to proto_tree_add_item.
Basically, at this point, I'm lost. I did find this post from 2014 that suggested limited support for add_packet_field but surely by now something as basic as an integer value is supported?
Also, I do know how to declare a Field and extract the value after tree:add does the parsing; worst case I'll fall back to that, but surely there is a more expedient way to access the just-parsed value added to the tree?
Wireshark Version
3.2.4 (v3.2.4-0-g893b5a5e1e3e)
Compiled (64-bit) with Qt 5.12.8, with WinPcap SDK (WpdPack) 4.1.2, with GLib
2.52.3, with zlib 1.2.11, with SMI 0.4.8, with c-ares 1.15.0, with Lua 5.2.4,
with GnuTLS 3.6.3 and PKCS #11 support, with Gcrypt 1.8.3, with MIT Kerberos,
with MaxMind DB resolver, with nghttp2 1.39.2, with brotli, with LZ4, with
Zstandard, with Snappy, with libxml2 2.9.9, with QtMultimedia, with automatic
updates using WinSparkle 0.5.7, with AirPcap, with SpeexDSP (using bundled
resampler), with SBC, with SpanDSP, with bcg729.
Running on 64-bit Windows 10 (1803), build 17134, with Intel(R) Xeon(R) CPU
E3-1505M v6 # 3.00GHz (with SSE4.2), with 32558 MB of physical memory, with
locale English_United States.1252, with light display mode, without HiDPI, with
Npcap version 0.9991, based on libpcap version 1.9.1, with GnuTLS 3.6.3, with
Gcrypt 1.8.3, with brotli 1.0.2, without AirPcap, binary plugins supported (19
loaded).
Built using Microsoft Visual Studio 2019 (VC++ 14.25, build 28614).

Looking at the try_add_packet_field() source code, only certain FT_ types are supported, namely:
FT_BYTES
FT_UINT_BYTES
FT_OID
FT_REL_OID
FT_SYSTEM_ID
FT_ABSOLUTE_TIME
FT_RELATIVE_TIME
None of the other FT_ types are supported [yet], including FT_UINT16, which is the one you're interested in here, i.e., anything else just needs to be done the old fashioned way.
If you'd like this to be implemented, I'd suggest filing a Wireshark enhancement bug request for this over at the Wireshark Bug Tracker.

Related

Ada + Machine Learning (Python Framework)

I'm trying to write a simple machine learning application in Ada, and also trying to find a good framework to use. My knowledge of one thing is extremely minimal, and of the other is somewhat minimal.
There are several nifty machine learning frameworks out there, and I'd like to leverage one for use with an Ada program, but I guess I'm just...at a loss. Can I use an existing framework written in Python, for instance and wrap (or I guess, bind?) the API calls in Ada? Should I just pass off the scripting capabilities? I'm trying to figure it out.
Case in point: Scikit (sklearn)
https://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html#
This does some neat stuff, and I'd like to be able to leverage this, but with an Ada program. Does anyone have advice from a similar experience?
I am just researching, so I have tried finding information.
http://www.inspirel.com/articles/Ada_Python_Binding.html
https://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html#
The inspirel solution is based on python2.7. If you're using anything from python3.5 onwards a few mods need to be made. On Linux, changing to say python 3.7, you'd just change
--for Default_Switches ("Ada") use ("-lpython2.7");
for Default_Switches ("Ada") use ("-lpython3.7");
but on windows, the libraries aren't dumped in a community lib so gnat doesn't know where to find them. All the packages are kept separately. The -L has to be added to tell the linker where to find the library. Alternatively, you can use for lib_dir. In my case, I did a non-admin install of python, so it looks something like
for Default_Switches ("Ada") use ("-L\Users\StdUser\AppData\Local\Programs\Python\Python37-32\libs", "-lpython37");
Note that on windows, the library is called python37: not python3.7. Use gprbuild instead of gnatmake -p, which has been deprecated. If you do all your mods correctly
gprbuild ada_main.gpr
should give you an executable in obj\ada_main.exe if it builds. If a later version of python is used, some edits need to be made
python_module.py
#print 'Hello from Python module'
print('Hello from Python module')
#print 'Python adding:', a, '+', b
print('Python adding:', a, '+', b)
ada_main.adb
-- Python.Execute_String("print 'Hello from Python!'");
Python.Execute_String("print('Hello from Python!')");
Some routines have been deprecated so the linkage has to change
python.adb
--pragma Import(C, PyInt_AsLong, "PyInt_AsLong");
pragma Import(C, PyInt_AsLong, "PyLong_AsLong");
--pragma Import(C, PyString_FromString, PyString_FromString");
pragma Import(C, PyString_FromString, "PyUnicode_FromString");
Running the build and executable should give
C:\Users\StdUser\My Documents\ada-python>gprbuild ada_main.gpr
Compile
[Ada] ada_main.adb
Bind
[gprbind] ada_main.bexch
[Ada] ada_main.ali
Link
[link] ada_main.adb
C:\Users\StdUser\My Documents\ada-python>obj\ada_main.exe
executing Python directly from Ada:
Hello from Python!
loading external Python module and calling functions from that module:
Hello from Python module!
asking Python to add two integers:
Python adding: 10 + 2
Ada got result from Python: 12
we can try other operations, too:
subtract: 8
multiply: 20
divide : 5
Remember to put the pythonxx.dll somewhere on your path otherwise it won't be able to find the library when it starts executing.

Pointcloud Visualization in Drake Visualizer in Python

I would like to visualize pointcloud in drake-visualizer using python binding.
I imitated how to publish images through lcm from here, and checked out these two issues (14985, 14991). The snippet is as follows :
point_cloud_to_lcm_point_cloud = builder.AddSystem(PointCloudToLcm())
point_cloud_to_lcm_point_cloud.set_name('pointcloud_converter')
builder.Connect(
station.GetOutputPort('camera0_point_cloud'),
point_cloud_to_lcm_point_cloud.get_input_port()
)
point_cloud_lcm_publisher = builder.AddSystem(
LcmPublisherSystem.Make(
channel="DRAKE_POINT_CLOUD_camera0",
lcm_type=lcmt_point_cloud,
lcm=None,
publish_period=0.2,
# use_cpp_serializer=True
)
)
point_cloud_lcm_publisher.set_name('point_cloud_publisher')
builder.Connect(
point_cloud_to_lcm_point_cloud.get_output_port(),
point_cloud_lcm_publisher.get_input_port()
)
However, I got the following runtime error:
RuntimeError: DiagramBuilder::Connect: Mismatched value types while connecting output port lcmt_point_cloud of System pointcloud_converter (type drake::lcmt_point_cloud) to input port lcm_message of System point_cloud_publisher (type drake::pydrake::Object)
When I set 'use_cpp_serializer=True', the error becomes
LcmPublisherSystem.Make(
File "/opt/drake/lib/python3.8/site-packages/pydrake/systems/_lcm_extra.py", line 71, in _make_lcm_publisher
serializer = _Serializer_[lcm_type]()
File "/opt/drake/lib/python3.8/site-packages/pydrake/common/cpp_template.py", line 90, in __getitem__
return self.get_instantiation(param)[0]
File "/opt/drake/lib/python3.8/site-packages/pydrake/common/cpp_template.py", line 159, in get_instantiation
raise RuntimeError("Invalid instantiation: {}".format(
RuntimeError: Invalid instantiation: _Serializer_[lcmt_point_cloud]
I saw the cpp example here, so maybe this issue is specific to python binding.
I also saw this python example, but thought using 'PointCloudToLcm' might be more convenient.
P.S.
I am aware of the development in recent commits on MeshcatVisualizerCpp and MeshcatPointCloudVisualizerCpp, but I am still on the drake-dev stable build 0.35.0-1 and want to stay on drake visualizer until the meshcat c++ is more mature.
The old version in pydrake.systems.meshcat_visualizer.MeshcatVisualizer is a bit too slow on my current use-case (multiple objects drop). I can visualize the pointcloud with this visualization setting, but it took too much machine resources.
Only the message types that are specifically bound in lcm_py_bind_cpp_serializers.cc can be used on an LCM message input/output port connection between C++ and Python. For all other LCM message types, the input/output port connection must be from a Python system to a Python system or a C++ System to a C++ System.
The lcmt_image_array is listed there, but not the lcmt_point_cloud.
If you're stuck using Drake's v0.35.0 capabilities, then I don't see any great solutions. Some options:
(1) Write your own PointCloudToLcm system in Python (by re-working the C++ code into Python, possibly with a narrower set of supported features / channels for simplicity).
(2) Write your own small C++ helper function MakePointCloudPublisherSystem(...) that calls LcmPublisherSystem::Make<lcmt_point_cloud> function in C++, and bind it into Python. Then your Python code can call MakePointCloudPublisherSystem() and successfully connect that to the existing C++ PointCloudToLcm.

Getting InvalidGeometry: LinearRing failed ring test after upgrading rgeo gem

After upgrading the rgeo gem from 0.6.0 to 2.1.1, we've started to get 'LinearRing failed ring test' on certain geometries that never caused us problems before. (The geometry data is generated from external sources outside of our control, and stored in PostGIS table.)
rgeo initializer:
GEO_FACTORY = RGeo::Geographic.simple_mercator_factory
PROJECTION_FACTORY = GEO_FACTORY.projection_factory
RGeo::ActiveRecord::SpatialFactoryStore.instance.tap do |config|
config.default = PROJECTION_FACTORY
end
Example code:
district = District.first
# Convert projected geometry to geographic geometry
geographic_geometry = GEO_FACTORY.unproject(district.geometry)
RGeo::Error::InvalidGeometry: LinearRing failed ring test
I also tried initializing the factory with the following options, but the errors persisted:
FACTORY = RGeo::Geographic.simple_mercator_factory(uses_lenient_assertions:true)
FACTORY = RGeo::Geographic.simple_mercator_factory(lenient_multi_polygon_assertions:true)
I've inspected some of these geometries where that are failing the LinearRing test, and it appears they are failing for a variety of reasons. Some possibly violate ring winding order, while with others I couldn't perceive the exact problem. But the point is, these are all geometries that we were able to work with before, and now are raising errors. I'd like to understand how we can return to the more lenient mode.
NOTES:
This is only happening on my Macbook (GEOS version 3.8.0), but not on our Linux production server (GEOS version 3.5.0)
REFERENCES:
https://github.com/rgeo/rgeo/issues/194
https://github.com/rgeo/rgeo-geojson/issues/33
This doesn't appear to be an issue with the rgeo gem but rather the underlying geos library on your mac.
There appears to be a difference in the way that geos 3.8.x (3.8.1 is the latest version in Homebrew circa 4/13/2020) and earlier versions of the geos library handle the is_simple calculation that determines Polygon validity. This affects any RGeo factory (most of them but not RGeo::Geographic.spherical_factory or RGeo::Cartesian.simple_factory) backed by the CAPI.
If you download version 3.5.x of geos from https://trac.osgeo.org/geos/ and build it with cmake (takes a while but installs cleanly) you should have consistent behavior between your production and local development environments.
If you were using the simple_mercator factory I doubt that differences in how spherical coordinates (implemented in pure ruby) are validated explain the differences in behavior that you see.
See this issue: https://github.com/rgeo/rgeo/issues/218
You most probably encountered this [1] bug. Reproducing needs Mac OS and shows the error you have:
RGeo::Error::InvalidGeometry (LinearRing failed ring test)
Issue is linked to pull request [2], which found the issue on Float vs BigDecimal difference. Somehow Floats are not so precise, when there are enough decimals included.
Side note: Pull request is only 26 days old, so it is still open, thus you'll need to build it from branch for the moment.
Source:
[1] https://github.com/rgeo/rgeo/issues/212
[2] https://github.com/rgeo/rgeo/pull/213

Get the operating system in maxima

Is it possible to get the operating system in maxima? I have some code that needs the unix / or windows \ for path names. How can I find out which operating system the code is running in?
To give some context, I have the following code:
windows: false$
divider: "/"$
if (windows) then divider: "\\"$
initfile: concat(maxima_userdir, divider, "maxima-init.mac");
load(operatingsystem)$
dir: getcurrentdirectory();
if (substring(dir, slength(dir)) # divider) then dir: concat(dir, divider)$
repo: concat(dir, "$$$.mac")$
live: concat(dir, "live_packages", divider, "$$$.mac")$
with_stdout(initfile, printf(true, ""))$
with_stdout(initfile, printf(true, concat("file_search_maxima: append (file_search_maxima, [
~s,
~s
]);"), repo, live))$
Take a look at the output of build_info, specifically the field host (i.e. foo#host where foo : build_info()). See ? build_info for more information.
On my (Linux) system I get: x86_64-unknown-linux-gnu I think on MS Windows you'll get a string containing windows or at least win or maybe win32.
There may be other ways to figure out the system type so let me know if that doesn't work for you. Also it is possible that there is a global variable floating around which tells the path separator; I would have to look for that.
If you're not adverse to writing a little bit of Lisp code, another approach is to use the file and directory functions in Common Lisp, which are more extensive than in Maxima. See the section on filenames in the Common Lisp Hyperspec. I think maybe MERGE-PATHNAMES and/or MAKE-PATHNAME might be relevant.

Imagemagick & 3Drotate

I recently had to move my site from one server to another and it appears Fred's 3Drotate script creates files that have Imagemagick settings in them instead of image data. When I first ran the script I received the following error:
expr: warning: unportable BRE: `^[0-9][0-9]*$': using `^' as the first character of the basic regular expression is not portable; it is being ignored
expr: warning: unportable BRE: `^[+-][0-9][0-9]*$': using `^' as the first character of the basic regular expression is not portable; it is being ignored
expr: warning: unportable BRE: `^[0-9]*[\\.][0-9]*$': using `^' as the first character of the basic regular expression is not portable; it is being ignored
expr: warning: unportable BRE: `^[+-][0-9]*[\\.][0-9]*$': using `^' as the first character of the basic regular expression is not portable; it is being ignored
I was able to resolve this by removing each instance of the '^' character. The script runs and it creates a file of a few hundred KB, but the contents appear to be a configuration such as:
id=MagickCache
quantum-depth=16
class=DirectClass colors=0 matte=False
columns=500 rows=500 depth=8
colorspace=sRGB
compression=JPEG quality=75
units=PixelsPerInch
resolution=72x72
page=500x500+0+0
rendering-intent=Perceptual
gamma=0.454545
red-primary=0.64,0.33 green-primary=0.3,0.6 blue-primary=0.15,0.06
white-point=0.3127,0.329
date:create=2012-08-10T20:44:21-07:00
date:modify=2012-08-10T20:44:21-07:00
jpeg:colorspace=2
jpeg:sampling-factor=2x2,1x1,1x1
Any ideas? I'm running imagemagick version 6.7.8-9 which is newer than what was on my original server, however I don't know which version that was.
Update:
I'm on a CentOS box using 3DRotate revised by Fred on 3/11/10. ImageMagick is version 6.7.8-9 whereas my old server, also CentOS was using version 6.7.6-0.
Fred's 3Drotate script still works fine for me. I have no reason to complain.
This is on Mac OS X Lion, with Bash version GNU bash, Version 4.2.37(2)-release....
You are strongly advised to re-download the script and try again. If you want to report a bug, you should give info about your OS, your Bash and your ImageMagick versions...
Oh, and see also this page, which states...
...you need ImageMagick v6.3.5.0 or higher for the script to work,
...gives some hints about troubleshooting and
...tells you the eMail address of the author so you can contact him to discuss your problems.
With recent changes to ImageMagick between IM 6.7.6.7 and IM 6.7.8.3 for colorspace changes and grayscale becoming linear, I have had to go through recently all my scripts and make appropriate changes. I am only part way through my scripts as of this date, but have fixed about 1/3 to 1/2, but 3Drotate did not need any changes since the last one on 11/26/2011. But your version is to old, so you will need to get an update. The problems you seem to have may or may not be related. But appear to be related to the unix utility expr. I would make sure that you have a current version of that installed. The following works perfectly fine for me on both IM 6.7.6.0 and 6.7.8.9 on my Mac OSX Snow Leopard
3Drotate pan=45 tilt=45 auto=zc mandril.jpg mandril_test.jpg
The first question I would ask is what was your exact command line. As you can see the arguments are a bit different from my other scripts as they include equal signs.
If you still have trouble, report them to me or on the ImageMagick Discourse forum at http://www.imagemagick.org/discourse-server/viewforum.php?f=1
Fred

Resources