What technology/API/technique to use for network monitoring? - driver

Okay so, i am building a utility to monitor filesystem, registry and network activity (process wise; only activity by selected processes). I have accomplished the filesystem and registry activity part by developing a minifilter driver. However, i am uncertain as to how should i do it for the network. What i want to do is much like what sysinternal's TCPView does, however i only want to monitor ESTABLISHED connections by the selected processes. Here is what i want to get in real time for each connection:
-protocol (TCP or UDP)
-source port
-remote IP and port
-[optional] amount of bytes transferred on a particular connection from the time that the monitoring started
What should i use? i heard about LSPs, but reading into it a bit further i realized that it is VERY difficult to write proper functioning LSPs, not to mention that there is hardly any material available for them to learn from scratch. Also that they are becoming obsolete. Problem is that i only have about 2-3 weeks of time to learn+write this module. Due to the time limit i certainly don't want to go for something like WFP, unless there is a very good tutorial for it out there, and i am not talking about the MSDN documentation. I don't know if this could be 'easily' done using NDIS etc.
Anyhow, so what should i do, where should i focus my efforts. Should i take the risk with learning about the LSPs, or would NDIS accomplish the task, or something else. I am kind of clueless right now. Help me out here!

Look at GetExtendedTcpTable and GetExtendedUdpTable. These APIs will get you most of what you need. But just to keep things interesting, the demo code I wrote uses GetTcp6Table2 instead.
#define WIN32_LEAN_AND_MEAN
#define UNICODE
#define _UNICODE
#include <windows.h>
#include <winsock2.h>
#include <Ws2tcpip.h>
#include <iphlpapi.h>
#include <Tcpestats.h>
#include <Tcpmib.h>
#include <Mstcpip.h>
#include <stdlib.h>
#include <stdio.h>
#pragma comment(lib, "iphlpapi.lib")
#pragma comment(lib, "ws2_32.lib")
PCWSTR
StringFromState(MIB_TCP_STATE State)
{
switch (State)
{
case MIB_TCP_STATE_CLOSED:
return L"CLOSED";
case MIB_TCP_STATE_LISTEN:
return L"LISTEN";
case MIB_TCP_STATE_SYN_SENT:
return L"SYN_SENT";
case MIB_TCP_STATE_SYN_RCVD:
return L"SYN_RCVD";
case MIB_TCP_STATE_ESTAB:
return L"ESTAB";
case MIB_TCP_STATE_FIN_WAIT1:
return L"FIN_WAIT1";
case MIB_TCP_STATE_FIN_WAIT2:
return L"FIN_WAIT2";
case MIB_TCP_STATE_CLOSE_WAIT:
return L"CLOSE_WAIT";
case MIB_TCP_STATE_CLOSING:
return L"CLOSING";
case MIB_TCP_STATE_LAST_ACK:
return L"LAST_ACK";
case MIB_TCP_STATE_TIME_WAIT:
return L"TIME_WAIT";
case MIB_TCP_STATE_DELETE_TCB:
return L"DELETE_TCB";
default:
return L"[Unknown]";
}
}
LPWSTR (NTAPI *pRtlIpv6AddressToStringW)(const IN6_ADDR *, LPWSTR);
int __cdecl main()
{
ULONG r;
// We need to load this dynamically, because ntdll.lib doesn't export it
HMODULE ntdll = LoadLibrary(L"ntdll");
pRtlIpv6AddressToStringW = (decltype(pRtlIpv6AddressToStringW))GetProcAddress(ntdll, "RtlIpv6AddressToStringW");
// Initial guess for the table size
ULONG cbTable = 100;
MIB_TCP6TABLE2 *table = nullptr;
while (true)
{
table = (MIB_TCP6TABLE2*)malloc(cbTable);
if (!table)
return 1;
r = GetTcp6Table2(table, &cbTable, FALSE);
if (ERROR_INSUFFICIENT_BUFFER == r)
{
// Try again with bigger buffer
free(table);
continue;
}
else if (ERROR_SUCCESS == r)
{
break;
}
else
{
free(table);
wprintf(L"GetTcp6Table2 = %u\n", r);
return 1;
}
}
// Print table heading
wprintf(L"%56s %56s %10s %6s\n", L"Local endpoint", L"Remote endpoint", L"State", L"PID");
for (ULONG i = 0; i < table->dwNumEntries; i++)
{
MIB_TCP6ROW2 const &entry = table->table[i];
WCHAR localAddr[46];
WCHAR remoteAddr[46];
pRtlIpv6AddressToStringW(&entry.LocalAddr, localAddr);
pRtlIpv6AddressToStringW(&entry.RemoteAddr, remoteAddr);
WCHAR localEndpoint[56];
WCHAR remoteEndpoint[56];
swprintf_s(localEndpoint, L"[%s]:%-5u", localAddr, ntohs(entry.dwLocalPort));
swprintf_s(remoteEndpoint, L"[%s]:%-5u", remoteAddr, ntohs(entry.dwRemotePort));
wprintf(L"%56s %56s %10s %6u\n",
localEndpoint, remoteEndpoint,
StringFromState(entry.State),
entry.dwOwningPid);
}
free(table);
return 0;
}
Example output (actual addresses anonymized):
C:\>test.exe
Local endpoint Remote endpoint State PID
[::]:80 [::]:0 LISTEN 4
[::]:135 [::]:0 LISTEN 980
[::]:445 [::]:0 LISTEN 4
[::]:1025 [::]:0 LISTEN 692
[2001:xxxx:x:xxx:x:xxxx:xxx.xx.xxx.xx]:6044 [xxxx:xxx:xxxx:xxxx::x]:443 ESTAB 3248
[2001:xxxx:x:xxx:x:xxxx:xxx.xx.xxx.xx]:6045 [xxxx:xxx:xxxx:xxxx::x]:443 ESTAB 3248
[2001:xxxx:xx:x:xxxx:xxxx:xxxx:xxxx]:53759 [2001:xxxx:xx:xxxx:xxx:xxxx:xxxx:xxxx]:135 TIME_WAIT 0

Sounds to me like you want something like winpcap which is what wireshark uses.
http://www.winpcap.org/
Also the netstat code from ReactOS might be interesting
http://doxygen.reactos.org/dd/d3f/netstat_8c_source.html

Related

cannot publish data to my local mqtt server

Please i wish somebody could help me with this. I've been struggling into it since a couple of weeks, i am so new to that.
I want to send data from ESP32 SIM800L to a mqtt broker.
The mqtt server is running on my local machine and the ESP32 SIM800 can perfectly connect to APN.
I saw many tutorials doing it with WIFI connection but not GPRS(what i am using).
I finally find this: tinyGSM and this :arduino mqtt mongodb
And i adapted it as follows, but still getting connection failed:
// Your GPRS credentials (leave empty, if not needed)
const char apn[] = "internet.tn"; // APN (example: internet.vodafone.pt) use https://wiki.apnchanger.org
const char gprsUser[] = ""; // GPRS User
const char gprsPass[] = ""; // GPRS Password
// SIM card PIN (leave empty, if not defined)
const char simPIN[] = "";
uint32_t lastReconnectAttempt = 0;
// TTGO T-Call pins
#define MODEM_RST 5
#define MODEM_PWKEY 4
#define MODEM_POWER_ON 23
#define MODEM_TX 27
#define MODEM_RX 26
#define I2C_SDA 21
#define I2C_SCL 22
// Set serial for debug console (to Serial Monitor, default speed 115200)
#define SerialMon Serial
// Set serial for AT commands (to SIM800 module)
#define SerialAT Serial1
// Configure TinyGSM library
#define TINY_GSM_MODEM_SIM800 // Modem is SIM800
#define TINY_GSM_RX_BUFFER 1024 // Set RX buffer to 1Kb
#include <Wire.h>
#include <TinyGsmClient.h>
#include <PubSubClient.h>
#ifdef DUMP_AT_COMMANDS
#include <StreamDebugger.h>
StreamDebugger debugger(SerialAT, SerialMon);
TinyGsm modem(debugger);
#else
TinyGsm modem(SerialAT);
#endif
// I2C for SIM800 (to keep it running when powered from battery)
TwoWire I2CPower = TwoWire(0);
const char* broker = "localhost";
const char* topicInit = "GsmClientTest/init";
// Function prototypes
void subscribeReceive(char* topic, byte* payload, unsigned int length);
// TinyGSM Client for Internet connection
// gsm and MQTT related objects
TinyGsmClient client(modem);
PubSubClient mqtt(client);
long mqtttimer = 0; // Timer for counting 5 seconds and retrying mqtt connection
byte mqtttarea = 1;
#define uS_TO_S_FACTOR 1000000 /* Conversion factor for micro seconds to seconds */
#define TIME_TO_SLEEP 3600 /* Time ESP32 will go to sleep (in seconds) 3600 seconds = 1 hour */
void mqttCallback(char* topic, byte* payload, unsigned int len) {
SerialMon.print("Message arrived [");
SerialMon.print(topic);
SerialMon.print("]: ");
SerialMon.write(payload, len);
SerialMon.println();}
boolean mqttConnect() {
SerialMon.print("Connecting to ");
SerialMon.print(broker);
// Connect to MQTT Broker
boolean status = mqtt.connect("GsmClientTest");
// Or, if you want to authenticate MQTT:
//boolean status = mqtt.connect("GsmClientName", "mqtt_user", "mqtt_pass");
if (status == false) {
SerialMon.println(" fail");
return false;
}
SerialMon.println(" success");
mqtt.publish(topicInit, "GsmClientTest started");
// mqtt.subscribe(topicLed);
return mqtt.connected();}
void setup() {
SerialMon.begin(9600);
// Start I2C communication
I2CPower.begin(I2C_SDA, I2C_SCL, 400000);
// Set modem reset, enable, power pins
pinMode(MODEM_PWKEY, OUTPUT);
pinMode(MODEM_RST, OUTPUT);
pinMode(MODEM_POWER_ON, OUTPUT);
digitalWrite(MODEM_PWKEY, LOW);
digitalWrite(MODEM_RST, HIGH);
digitalWrite(MODEM_POWER_ON, HIGH);
// Set GSM module baud rate and UART pins
SerialAT.begin(9600, SERIAL_8N1, MODEM_RX, MODEM_TX);
delay(3000);
// Restart SIM800 module, it takes quite some time
// To skip it, call init() instead of restart()
SerialMon.println("Initializing modem...");
modem.restart();
// use modem.init() if you don't need the complete restart
// Unlock your SIM card with a PIN if needed
if (strlen(simPIN) && modem.getSimStatus() != 3 ) {
modem.simUnlock(simPIN);
}
SerialMon.print("Connecting to APN: ");
SerialMon.print(apn);
if (!modem.gprsConnect(apn, gprsUser, gprsPass)) {
SerialMon.println(" fail");
}
else {
SerialMon.println(" OK");
}
// MQTT Broker setup
mqtt.setServer(broker, 1883);
mqtt.setCallback(mqttCallback);
esp_sleep_enable_timer_wakeup(TIME_TO_SLEEP * uS_TO_S_FACTOR);
}
void loop() {
// This is needed at the top of the loop!
if (!mqtt.connected()) {
SerialMon.println("=== MQTT NOT CONNECTED ===");
// Reconnect every 10 seconds
uint32_t t = millis();
if (t - lastReconnectAttempt > 10000L) {
lastReconnectAttempt = t;
if (mqttConnect()) {
lastReconnectAttempt = 0;
}
}
delay(100);
return;
}
mqtt.publish(topicInit, "Hello");
mqtt.loop();
}
You set the broker's name to localhost:
const char* broker = "localhost";
localhost and the IP address 127.0.0.1 mean "the host that this code is running on". When you're typing commands on the computer running the broker, localhost will mean that computer. There's no way it will work on the ESP32.
You need to name or IP address of the computer running the broker. How you find that will depend on the operating system you're running.
If that computer is on your local network it's probably using a private IP address like 10.0.1.x or 192.168.1.x. If that's the case you'll need to either use port forwarding in your router to forward packets to it (and then you'll use your router's IP address and not your broker's).
If you're using your router's IP address, that can change without warning, so you'll need to use something like Dynamic DNS to keep up to date with its current IP address.
You'll likely be better off running the broker outside of your network on a cloud-based virtual server or by using one of the several commercial MQTT services out there. Most of them have a free tier which will allow a reasonable amount of traffic per month.
Regardless, localhost will never work here. You need the real, public IP address or name of your broker.

L2CAP IOS + Linux (Bluez)

I'm trying to make a simple L2CAP Socket communication between IOS and a Linux PC.
I've been able to:
Create an L2CAP connection between two Linux machines (using example code from https://github.com/atwilc3000/sample/tree/master/Bluetooth)
Create an L2CAP connection between two Iphones (using example code from https://github.com/github-deden/iOS_L2Cap)
On that IOS example they are using some PSM advertise in order to chose the correct PSM for the L2CAP channel. On the integration, I've set a fixed PSM on both sides. The Iphone is connecting to the Linux machine fixed PSM. I've tried multiple PSM (0x1001, 0x25).
The problem is, I can't connect and can't get any information on what is happening on the air.
My question is, do I need to implement a dynamic/advertise PSM on the Linux application? Do I need to pick a specific PSM? Have you been able to make this work? Do you have any suggestions?
Thanks in advance!
Server code:
#include <stdio.h>
#include <unistd.h>
#include <string.h>
#include <stdlib.h>
#include <sys/socket.h>
#include <bluetooth/bluetooth.h>
#include <bluetooth/l2cap.h>
#include "l2cap_socket.h"
int main(int argc, char **argv)
{
struct sockaddr_l2 loc_addr = { 0 }, rem_addr = { 0 };
char buf[1024] = { 0 };
int server_socket, client_socket, bytes_read;
unsigned int opt = sizeof(rem_addr);
printf("Start Bluetooth L2CAP server...\n");
/* allocate socket */
server_socket = socket(AF_BLUETOOTH, SOCK_SEQPACKET, BTPROTO_L2CAP);
/* bind socket to the local bluetooth adapter */
loc_addr.l2_family = AF_BLUETOOTH; /* Addressing family, always AF_BLUETOOTH */
bacpy(&loc_addr.l2_bdaddr, BDADDR_ANY); /* Bluetooth address of local bluetooth adapter */
loc_addr.l2_psm = htobs(L2CAP_SERVER_PORT_NUM); /* port number of local bluetooth adapter */
printf("binding\n");
if(bind(server_socket, (struct sockaddr *)&loc_addr, sizeof(loc_addr)) < 0) {
perror("failed to bind");
exit(1);
}
printf("listening\n");
/* put socket into listening mode */
listen(server_socket, 1);
/* accept one connection */
client_socket = accept(server_socket, (struct sockaddr *)&rem_addr, &opt); /* return new socket for connection with a client */
ba2str( &rem_addr.l2_bdaddr, buf );
printf("connected from %s\n", buf);
/* read data from the client */
memset(buf, 0, sizeof(buf));
bytes_read = recv(client_socket, buf, sizeof(buf), 0);
if( bytes_read > 0 ) {
printf("received [%s]\n", buf);
}
/* close connection */
close(client_socket);
close(server_socket);
return 0;
}
Client is based on (from https://github.com/bluekitchen/CBL2CAPChannel-Demo).
I have now a working version based on https://github.com/bluekitchen/btstack
On the iOS side i have been using https://github.com/bluekitchen/CBL2CAPChannel-Demo
On the server side le_data_channel_server.

Writing to the serial monitor on the Sparkfun ESP8266 Thing

Below is my current code:
#include <Wire.h>
#include <ESP8266WiFi.h>
#include <BlynkSimpleEsp8266.h>
// You should get Auth Token in the Blynk App.
// Go to the Project Settings (nut icon).
char auth[] = "836addccd2ee4f05b96f0f3ad831249e"; // ***Type in your Blynk Token
// Your WiFi credentials.
// Set password to "" for open networks.
char ssid[] = "_Fast&Furious";// ***your wifi name
char pass[] = "Mclaren2018";// ***and password
const int MOTION_PIN = 4; // Pin connected to motion detector
WidgetLCD lcd(V1);
void setup()
{
Serial.begin(115200);
Blynk.begin(auth, ssid, pass);
pinMode(MOTION_PIN, INPUT_PULLUP);
Serial.println("SETUP");
}
void loop()
{
Blynk.run();
int proximity = digitalRead(MOTION_PIN);
if (proximity == LOW) // If the sensor's output goes low, motion is detected
{
Blynk.virtualWrite(5,1023);
lcd.clear();
lcd.print(0,0,"Motion detected");
Serial.println("Motion detected!");
}
else
{
Blynk.virtualWrite(5,0);
lcd.clear();
lcd.print(0,0,"Motion NOT detected");
Serial.println("Motion NOT detected!");
}
}
I am currently trying to simily write some text to the serial console. But when I upload my code it will just write a string of k's to the console. What am I doing wrong to produce such a strange output?
This is a link to the tutorial I have been following: http://designinformaticslab.github.io/productdesign_tutorial/2017/01/24/motion_sensor.html
Any help would be much appreciated!
It all looks good to me, are you sure you have the baud rate set correctly on the serial monitor? I would write a new program real quick that ONLY does serial output and get that working (this simplifies the problem to solve, and makes it more obvious if it is something like serial port speed), then go back to your more complete program and it should work.

arduino programming: not enough memory message

I am new to arduino programming (Arduino Pro Mini 3.3v version), i have some code like below. I am connecting 9DOF, OLED screen and a BLE breakout to arduino pro mini.
I already went through some of the memory optimization tips, but i still have some issue. Even with the following code, i only have 9 bytes left for dynamic memory. If i enable BTLEserial.begin();, it will kill the memory. Please any suggestions will be appreciated.
#include <Wire.h>
#include <SPI.h>
#include <SparkFunLSM9DS1.h>
#include "Adafruit_BLE_UART.h"
#include <Adafruit_GFX.h>
#include <Adafruit_SSD1306.h>
#define OLED_RESET 4
Adafruit_SSD1306 display(OLED_RESET);
LSM9DS1 imu;
#define LSM9DS1_M 0x1E // Would be 0x1C if SDO_M is LOW
#define LSM9DS1_AG 0x6B // Would be 0x6A if SDO_AG is LOW
#define ADAFRUITBLE_REQ 10
#define ADAFRUITBLE_RDY 2
#define ADAFRUITBLE_RST 9
Adafruit_BLE_UART BTLEserial = Adafruit_BLE_UART(ADAFRUITBLE_REQ, ADAFRUITBLE_RDY, ADAFRUITBLE_RST);
void setup(void) {
Serial.begin(9600);
display.begin(SSD1306_SWITCHCAPVCC, 0x3D); // initialize with the I2C addr 0x3D (for the 128x64)
display.display();
delay(2000);
display.clearDisplay();
display.drawPixel(10, 10, WHITE);
display.display();
delay(2000);
display.clearDisplay();
imu.settings.device.commInterface = IMU_MODE_I2C;
imu.settings.device.mAddress = LSM9DS1_M;
imu.settings.device.agAddress = LSM9DS1_AG;
if (!imu.begin())
{
while (1)
;
}
// BTLEserial.begin(); - if i uncomment this code, i will get a not enough memory error.
}
aci_evt_opcode_t laststatus = ACI_EVT_DISCONNECTED;
void loop() {
displayAllDOF();
}
void displayAllDOF(){
display.setTextSize(1);
display.setTextColor(WHITE);
imu.readGyro();
display.setCursor(0,0);
display.print("G:");
display.print(imu.calcGyro(imu.gx));
display.print(", ");
display.print(imu.calcGyro(imu.gy));
display.print(", ");
display.print(imu.calcGyro(imu.gz));
display.println(" ");
imu.readAccel();
display.print("A:");
display.print(imu.calcAccel(imu.ax));
display.print(", ");
display.print(imu.calcAccel(imu.ay));
display.print(", ");
display.print(imu.calcAccel(imu.az));
display.println(" ");
imu.readMag();
display.print("M:");
display.print(imu.calcMag(imu.mx));
display.print(", ");
display.print(imu.calcMag(imu.my));
display.print(", ");
display.print(imu.calcMag(imu.mz));
display.println(" ");
display.display();
display.clearDisplay();
}
To start, you'll need to figure out where your RAM is going - How much does each library take? Do you really need to run them all at the same time? You know that you can run the display library, and the IMU code in your current setup - Can you implement something that only enables the IMU code, pulls data, then disables it? And the same with the display and BTLE code? That way each library is only consuming RAM when it's needed, and frees it once it's operation is finished
Update 1
An example of what I mentioned above. I do not know if all the libraries implement the .end() function. They may have a similar method you can use.
// Simple data storage for the .gx and .gy values
typedef struct {
float x, y;
} GyroData_t;
GyroData_t getImuData() {
GyroData_t data;
// Create the IMU class, gather data from it, and then destroy it
LSM9DS1 *imu = new LSM9DS1();
imu->begin();
imu->readGyro();
data.x = imu.gx;
data.y = imu.gy;
imu->end();
// This will reclaim the RAM that was used by the IMU - We no longer need it
delete imu;
return data;
}
void displayAllDOF() {
// Gather the IMU data
GyroData_t data = getImuData();
// Create the display object, and print the data we received
Adafruit_SSD1306 *display = new Adafruit_SSD1306(OLED_RESET);
display->print(...);
....
display->end();
// Reclaim the display RAM used
delete display;
// Do any bluetooth operations now
doBluetoothStuff();
}
void doBluetoothStuff() {
Adafruit_BLE_UART *BTLEserial = new Adafruit_BLE_UART(ADAFRUITBLE_REQ, ADAFRUITBLE_RDY, ADAFRUITBLE_RST);
BTLESerial->begin();
...
BTLESerial->end();
delete BTLESerial;
}

how to use collect-view in Cooja in my own application

i have a question about collect-view. Can collect-view be used in other applications? i simulate some applications by cooja, and want to collect information about network,such as network map,ETX and so on. how can i use collect-view to collect information about my own applications?
thanks!
In the apps/collect-view or in your application you can add your own information to be sent by collect by redefining collect_view_arch_read_sensors, check for example how it is done for the Z1 mote in the collect-view-z1.c:
#include "collect-view.h"
#include "cc2420.h"
#include "dev/leds.h"
#include "dev/i2cmaster.h"
#include "dev/tmp102.h"
#include "collect-view-z1.h"
/*---------------------------------------------------------------------------*/
static uint16_t
get_temp()
{
/* XXX Fix me: check /examples/z1/test-tmp102.c for correct conversion */
return (uint16_t)tmp102_read_temp_raw();
}
/*---------------------------------------------------------------------------*/
void
collect_view_arch_read_sensors(struct collect_view_data_msg *msg)
{
static int initialized = 0;
if(!initialized) {
tmp102_init();
initialized = 1;
}
msg->sensors[BATTERY_VOLTAGE_SENSOR] = 0;
msg->sensors[BATTERY_INDICATOR] = 0;
msg->sensors[LIGHT1_SENSOR] = 0;
msg->sensors[LIGHT2_SENSOR] = 0;
msg->sensors[TEMP_SENSOR] = get_temp();
msg->sensors[HUMIDITY_SENSOR] = 0;
}
Notice that collect_view_arch_read_sensors(msg); is called by collect-view.c when you call collect_view_construct_message(...) from your application. This is how it is done in the examples/ipv6/rpl-collect/udp-sender.c example:
/* Collect structure */
struct {
uint8_t seqno;
uint8_t for_alignment;
struct collect_view_data_msg msg;
} msg;
/* Create the collect structure */
collect_view_construct_message(&msg.msg, &parent,
parent_etx, rtmetric,
num_neighbors, beacon_interval);
/* Send over UDP */
uip_udp_packet_sendto(client_conn, &msg, sizeof(msg),
&server_ipaddr, UIP_HTONS(UDP_SERVER_PORT));
When contiki calls the Call back function, for example the recv() function in rime-protocol, The sink node should use printf() to print out the sensor data. This output then will be read by Collect View GUI per-line.
take a look at
tools/collect-view/src/org/contikios/contiki/collect/CollectServer.java
The handleIncomingData() called by serialData(), then checks if it's a sensor data or not in SensorData.parseSensorData().
It will ignore if empty, incorrect format, comment, or annotation
sensor data will be displayed in Collect View GUI if meets some condition. for example, the data length calculated match with datalen implisit on its sensor data.

Resources