I want to write a program which Controls all the web browsing activities on PC.
i.e. Checking all the websites users go to, filtering some of them, ... .
But I have no idea how to capture all the packets, processing them, and even act to some (think of filtering unwanted sites).
Any help, sample code, open source program...?
There are different levels you can put yourself in the middle of the communication:
By implementing a proxy and having the browser connect to the proxy
By implementing a firewall/snooper and handling the raw packets
By implementing a network driver and handling the raw packets
IMHO, number 1 is easiest. Look at SQUID for an example. Number 2 is doable too, take a look at fiddler. You could take a look at the Click Modular Router for option number 3.
Depending on the browser, maybe a simple browser plugin could do?
Related
I sometimes need to use Wireshark to analyze communication issues with a particular protocol that my application uses. Wireshark already comes with a dissector for the protocol, and displays the communication in the best possible way I can imagine.
But I also need to view the actual communication together with events happening inside my application. The application is capable of generating various logs and traces. The information in them is actually more structured, but for the simplicity, let's say it is just a sequence of entries where each entry has a timestamp and a textual message.
Currently, I have to place the Wireshark and the logs alongside on the screen, and painfully correlate the timestamps in order to figure out how they belong together. In order to make my analysis much easier, I would like to view the information from my logs merged together with the communication protocol messages in Wireshark, properly sorted by their timestamps.
I found that Wireshark has a Merge capability, so this is where I am directing my investigation. I think that with some effort, I might be able to do the following:
1) Design my own "protocol", and generate PCAPNG file from my application, with the event timestamps and messages, and
2) Developer a Wireshark dissector for the above, so that I can view the events in Wireshark.
The first part of my question is whether my approach is the right one.
But I also wonder whether I cannot achieve what I want in some simpler way. Ideally, I would like to reuse something that already exists, and specifically, avoid developing a specialized dissector. Isn't there a protocol with identical features (just timestamps and textual messages), with a dissector that Wireshark already has, that I can use?
Maybe you could make use of syslog along with syslogd or rsyslogd?
One way to inject arbitrary messages into trace files without even having a syslog server is to make use of nc (netcat). For example:
echo -n "Hello World" | nc -w 0 -u 1.1.1.1 514
Wireshark will also dissect this message as syslog traffic. This can be useful when trying to insert "markers" into capture files near where an event of interest occurs.
In any case, making use of syslog facilities would save you from having to write your protocol.
I have a live video chat application and I use a TURN server which supports STUN/TURN and both UPD/TCP transmission.
Sometimes users can be connected to the network which blocks that much ports and protocols that WebRTC connection just cannot happen (usually those are corporate networks). I would like to check if a WebRTC connection is possible before users try to connect to each other (actually, perform a technical check).
How can I do it? Ideas I have in my head:
Try to download a hosted chunk of data (audio file, for example) via WebRTC - is it possible and would this be enough to make sure both inbound and outbound connections are open?
Use a TURN server as a host to make a connection to and see if it fails (have no idea if I can do it or not)
Use Flash to try to download/upload a chunk of data over specific ports and protocols. May be even using Cirrus. However, I am not sure this test will be accurate from WebRTC prospective.
Any other ideas?
Additional requirement: the checking technique must support Chrome, Opera and Firefox. Preferably also IE/Safari via Temasys plugin.
Edition 1 - gathering ICE candidates is a good idea, however, it is not 100% reliable. Once I checked logs in my application and it actually gathered relay ICE candidates, but video/audio transmission failed. Tested on Apprtc as well and got same results.
The best way to check is to connect with just a data channel first. Your users won't notice. If that works then audio and video are almost guaranteed to work. As a bonus, you can use the data channel for signaling for super-fast connecting when your users are ready.
the typical WebRTC approach to this is to create a peerconnection with STUN and TURN servers, call createOffer and setLocalDescription and watch the candidates gathered. See e.g. http://webrtc.github.io/samples/src/content/peerconnection/trickle-ice/
If you get srflx candidates, your stun server works (i.e. UDP is not blocked). More interesting is whether you get relay candidates. If you do, using TURN as a fallback will work. Quality might suffer if TURN/TCP is used. If you don't get relay candidates... calls are very unlikely to work.
I'm trying to find a way to take multiple mobile broadband connections, and bridge them together to expose one wifi access point.
I want to tether the phone's broadband, and multiplex the data.
How can this be achieved? Is there a router, that has multiple wifi's that can be used to connect to the phones?
Will I need to write a script to manage the data? Or is there something out there?
This is actually a little more complicated than it seems at first as you need to consider both ends of the connection.
You can combine multiple lower bandwidth connections as you suggest to provide an aggregated capacity at your WiFI access point, but you need to remember that the back end will still see multiple individual connections. So if you have a single server that you are trying to receive a single large stream from, it will see multiple smaller connections instead of a single large one.
You can get round this by adding some functionality on the server side to 'bind' or 'bond' the individual streams and make them appear as a single large connection to your server but you usually need a service from your ISP or some specialist 'bonding' provider to achieve this.
See the very good discussion and links in the answer below on Superuser for more information:
https://superuser.com/questions/660798/merging-two-incoming-broadband-lines-for-faster-internet
We intend to design a system with three "tiers".
HQ, with a single server
lots of "nodes" on a regional basis
users, with iPads.
HQ communicates 2-way with the nodes which communciate 2-way with the users. Users never communicate with HQ nor vice-versa.
The powers that be decree a Windows app from HQ (using Delphi) and a native desktop app for the users' iPads. They have no opinion on the nodes.
If there are compelling technical arguments, I might be able to beat them down from "decree" to "prefer" on the Windows program (and, for isntance, make it browser based). The nodes have no GUI, they just sit there playing middle-man.
What's the best way for these things to communicate (SOAP/HTTP/AJAX/jQuery/home-brewed-protocol-on-top-of-TCP/something-else?) Is it best to use the same protocol end to end, or different protocols for hq<-->node and node<-->iPad?
Both ends of each of those two interfaces might wish to initiate a transaction (which I can easily do if I roll my own protocol), so should I use push/pull/long-poll or what?
I hope that this description makes sense. Please ask questions if it does not. Thanks.
Update:
File size is typcially below 1MB with nothing likely to be above 10MB or even 5MB. No second file will be sent before a first file is acknowledged.
Files flow "downhill" from HQ to node to iPad. Files will never flow "uphill", but there will be some small packets of data (in addition to acks) which are initiated by user action on the iPad. These will go to the local node and then to the HQ. We are probably talking <128 bytes.
I suppose there will also be general control & maintenance traffic at a low rate, in all directions.
For push / pull (publish / subscribe or peer to peer communication), cross-platform message brokers could be used. I am not sure if there are (iOS) client libraries for Microsoft Message Queue (MSMQ), but I would also evaluate open source solutions like HornetQ, Apache ActiveMQ, Apollo, OpenMQ, Apache QPid or RabbitMQ.
All these solutions provide a reliable foundation for distributed messaging, like failover, clustering, persistence, with high performance and many clients attached. On this infrastructure message with any content type (JSON, binary, plain text) can be exchanged, and on top messages can contain routing and priority information. They also support transacted messaging.
There are Delphi and Free Pascal client libraries available for many enterprise quality open source messaging products. (I am am the author of some of them, supporting ActiveMQ, Apollo, HornetQ, OpenMQ and RabbitMQ)
Check out MessagePack: http://msgpack.org/
Also, here's more RPC discussion on SO:
RPC frameworks available?
MessagePack: fast cross-platform serializer and RPC - please share experience
ICE might be of interest to you: http://zeroc.com/index.html
They have an iOS layer: http://zeroc.com/icetouch/index.html
IMHO there are too little requisites to decide what technology to use. What data are exchanged, how often, what size? Are there request/response time constraints? etc. etc. Never start selecting a technology before you understand your needs deeply.
I need to use one logical PGM based multicast address in application while enable such application "seamlessly" running across several different geo-locations (i.e. think US/Europe/Australia).
Application is quite throughput (several million biz. messages a day) and latency demanding whith a lot of small but very frequently send messages. Classical Atom pub will not work here due some external limits of latencies.
I have come up with several options to connect those datacenters but can’t find the best one.
Options which I have considered are:
1) Forward multicast messages via VPN’s (can VPN handle such big load).
2) Translate all multicast messages to “wrapper messages” and forward them via AMQP.
3) Write specialized in-house gate which tunnels multicast messages via TCP to other two locations.
4) Any other solution
I would prefer option 1 as it does not need additional code writes from devs. but I’m afraid it will not be reliable connection.
Are there any rules to apply for such connectivity?
What the best network configuration with regard to the geographical configuration is for above constrains.
Just wanted to say hello :)
As for the topic, we have not much experience with multicasting over WAN, however, my feeling is that PGM + WAN + high volume of data would lead to retransmission storms. VPN won't make this problem disappear as all the Australian receivers would, when confronted with missing packets, send NACKS to Europe etc.
PGM specification does allow for tree structure of nodes for message delivery, so in theory you could place a single node on the receiving side that would in its turn re-multicast the data locally. However, I am not sure whether this kind of functionality is available with MS implementation of PGM. Optionally, you can place a Cisco router with PGM support on the receiving side that would handle this for you.
In any case, my preference would be to convert the data to TCP stream, pass it over the WAN and then convert it back to PGM on the other side. Some code has to be written, but no nasty surprises are to be expected.
Martin S.
at CohesiveFT we ran into a very similar problem when we designed our "VPN-Cubed" product for connecting multiple clouds up to servers behind our own firewall, in one VPN. We wanted to be able to run apps that talked to each other using multicast, but for example Amazon EC2 does not support multicast for reasons that should be fairly obvious if you consider the potential for network storms across a whole data center. We also wanted to route traffic across a wide area federation of nodes using the internet.
Without going into too much detail, the solution involved combining tunneling with standard routing protocols like BGP, and open technologies for VPNs. We used RabbitMQ AMQP to deliver messages in a pubsub style without needing physical multicast. This means you can fake multicast over wide area subnets, even across domains and firewalls, provided you are in the VPN-Cubed safe harbour. It works because it is a 'network overlay' as described in technical note here: http://blog.elasticserver.com/2008/12/vpn-cubed-technical-overview.html
I don't intend to actually offer you a specific solution, but I do hope this answer gives you confidence to try some of these approaches.
Cheers, alexis