We have a problem with our BizTalk applications with receive locations and send ports connecting to an oracle database. We run out of connections.
I don't know why, but original developers used both WCF-Custom and WCF-OracleDB, but I think both use ODP.NET as the ADO.NET provider.
Since, in ADO.NET and certainly in ODP.NET, connection pools are keyed on the connection string (exact string match, I think), a connection pool could logically be shared among send ports and receive locations. Since we don't have control of the connection string itself, we have to assume that connection strings in the adapters are consistently generated from one port to the other.
My questions are:
1- Am I right to assume that receive locations and send ports can share connection pools, as long as they run on the same host instance, and
2- Would it be a good idea to group similar ports and locations (the ones using the same connections strings) into one host instance?
Thank you,
Michel
Based on the below website, the connection pool is indeed determined via the uniqueness of the connection string:
http://www.connectionstrings.com/oracle-data-provider-for-net-odp-net/
(See "Specifying Pooling parameters")
One way to tackle this problem or at least give you a better insight, would be to enable ODP.NET tracing and performance counters. This will allow you a clear view on how many connections are being used in the pool(s).
For more information on how to enable those, see:http://blog.ilab8.com/2011/09/02/odp-net-pooling-and-connection-request-timed-out/
Set pooling=false on the connection string.
Related
I would like to understand networking services with a large user base a bit better so that I know how to approach a project I am busy with.
The following statements that I make may be incorrect but they still lead to the question that I want to ask...
Please consider Skype and TeamViewer clients. It seems that both keep persistent network connections open to their respective servers. They use these persistent connections to initiate additional connections. Some of these connections are created by means of Hole Punching if the clients are behind NATs. They are then used for direct Peer-to-Peer communications.
Now according to http://expandedramblings.com/index.php/skype-statistics/ there are 300 million users using Skype and 4.9 million daily active users. I would assume that most of that 4.9 million users will most probably have their client apps running most of the day. That is a lot of connections to the Skype servers that are open at any given time.
So to my question; Is this feasible or at least acceptable? I mean, wouldn't it be better to not have a network connection open while idle and aspecially when there are so many connections open to the servers at once? The only reason I can think is that it would be the only way to properly do Hole Punching. Techically, how is this achieved on the server side?
Is this feasible or at least acceptable?
Feasible it certainly is, you mention already two popular apps that do it, so it is very doable in practice.
As for acceptable, to start no internet authority (e.g. IETF) has ever said it is unacceptable to have long-lived connections even with low traffic.
Furthermore, the only components for which this matters are network elements that keep connection/flow state. These are for sure the endpoints and so-called middleboxes like NAT and firewalls. For the client this is only one connection, the server is usually fine tuned by the application developers (who made this choice) themselves, so for these it is acceptable. For middleboxes it's simple: they have no choice, they're designed to just work with all kind of flows, including long-lived persistent connections.
I mean, wouldn't it be better to not have a network connection open while idle and aspecially when there are so many connections open to the servers at once?
Not at all. First of all, that could be 'much' slower as you'd need to set up a full connection before each control-plane call. This is especially noticeable if your RTT is big or if the servers do some complicated connection proxying/redirection for load-balancing/localization purposes.
Next to that this would historically make incoming calls difficult for a huge amount of users. Many ISP's block/blocked unknown incoming connections from the internet by means of a firewall. Similar, if you are behind a NAT device that does not support UPnP or PCP you can't open a port to listen on for your public IP address. So you need it even aside from hole-punching.
The only reason I can think is that it would be the only way to
properly do Hole Punching. Techically, how is this achieved on the
server side?
Technically you can't do proper hole-punching as soon as the NAT devices maintain a full <src-ip,src-port,dest-ip,dest-port,protocol> (classical 5-tuple) flow match. Then the best you can do with 'hole punching' is set up a proxy between peers.
What hole-punching relies on is that the NAT flow lookup is only looking at <src-ip,src-port,protocol> upstream and <dest-ip,dest-port,protocol> downstream to do the translation. In that case both clients just set up a connection to the server, their ip and port gets translated and the server passes this to the other client. The other client can now start sending packets to that translated <ip,port> combination which should work because NAT ignores the server's ip/port. But even if the particular NAT would work like this, some security device (e.g. stateful firewall) might detect session hi-jacking and drop this anyway.
Nowadays you rather use UPnP to open up a port to listen on your public IP which is much easier if supported.
I've got an old connection log to a website which contains the IP addresses, the user agent string and the timestamp.
I want to know how many of them used 3G to access a website as I suspect there is a latency issue.
Now I know that I can use some code to have this info directly from the user.
But I still want to process this old log using only IP addresses even if it is only for one ISP.
I thought I could do that by finding a list of the subnets used for 3g.
Do you think it can be done?
Otherwise, what do you suggest?
Hope you can help.
Cheers,
One approach could be to lookup every IP using "whois" services, if the ISP listed is a phone carrier only, you can be certain the connection was made by a mobile connection.
This is not a total solution, but it might give you enough information.
Now, to find a service which will allow you to make all the requests required, might be another matter.
I'm building a chat server with .NET. I have tried opening about 2000 client connections and my Linksys WRT54GL router (with tomato firmware) drops dead each time. The same thing happens when I have several connections open on my Azureus bit-torrent client.
I have three questions:
Is there a limit on the number of open sockets I can have in Windows Server 2003?
Is the Linksys router the problem? If so is there better hardware recommended?
Is there a way to possibly share sockets so that I can handle more open client connections with fewer resources?
AS I've mentioned before, Raymond Chen has good advice on this sort of question: If you have to ask about OS limits, you're probably doing something wrong. The IP protocol only allows for a maximum of 65535 ports and many of these are reserved and not available for general use. I would suggest that your messaging protocols need to be thought out in more detail so that OS limits are not an issue. I'm sure there are many good resources describing such systems, and there are certainly people here that would have good ideas about it.
EDIT: I'm going to put some thoughts about implementing a scalable chat server.
First off, designate a single port on the server for clients to communicate through. Whenever a client needs to update the chat state (a new user message for example) do the following:
create message packet
open port to server
send packet
close port
The server then does the following:
connection request received
get packet
close connection
process packet
for each client that requires updating
open connection to clients
send update packet
close connection
When a new chat session is started, the client starting the session sends a 'new session' message to the server with the clients user details and IP address for responses. The server creates a new chat session and responds with the session ID. The client then sends packets containing the messages the user types, the server processes them and forwards the message to other clients in the same session. When a client leaves the chat, it sends a 'end session' message to the server. The server removes the client from the session and destroys the session when there are no more clients in the session.
Hope that gets you thinking.
i have found some answers to this that i feel i should share:
Windows 2003 server has a limit on the number of ports that may be used. but this is configurable via a registry tweak to change the MaxUSerPort setting from 5000 to say, 64k( max).
Exploring further, i realize that the 64k port restriction is actually per IP address, hence a single server can easily attain much more ports, and hence TCP connections by either installing multiple network cards, or binding more than one IP address to a network card. that way, you can scale your system to handle n x 64k ports.
Had for days a problem with the available sockets on my Window 7 machine. After reading some articles about socket leaks in Win 7, I applied a Windows patch - nothing changed.
Below there is an article describing windows connection problems in great detail:
http://technet.microsoft.com/en-us/magazine/2007.12.network.aspx
For me it worked the following:
Open Regedit
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\Tcpip\Parameters: Create TcpNumConnections, REG_DWORD, decimal value 500 (this can be set according to your needs); EnableConnectionRateLimiting, REG_DWORD, value 0;
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\Tcpip: Create MaxUserPort, REG_DWORD, decimal value 65534
Restart Windows
Hi
let me make my question clear. Two people using my app are connected to the internet. Both have each other's IP and they want to chat (like Y!messanger) with each other.
I think I need to use Indy components; right? Which component should I use?
Thanks in advance
Have you looked at any of the demos on Indy's website yet?
In general, you are looking to create a "Client/Server" type application. A quick Google search for "indy client server example" pulls up lots of results, including this one: http://www.devarticles.com/c/a/Delphi-Kylix/A-Real-World-Client-Server-Application-in-Delphi/
In reality, this gets a lot more complicated when you have firewalls and NATs with private IP addresses. You will have to consider how your application will either get around or through these types of technologies.
Similar to what Scott said, I think that your biggest problem is getting them talking to each other. My computers at home go through a router, which blocks all incoming connection requests (i.e. requests to start a conversation between two computers) from the Internet. My computers can send connection requests OUT, and start a conversation that way, but unless you modify the router (port forwarding) my computers can not receive connection requests.
You need a server somewhere to which both people will connect, that can then relay messages back and forth. To get really tricky, once the connection is made to the server the two computers can then be put into direct contact, but that involves UDP packets and some clever magic.
You don't have to use Indy components, you just need anything that will handle communications over the network. Any HTTP or sockets network stack will do. Indy is the defacto standard for Delphi Win32.
To do network communications, you will need to create a listener object or service on machine A and a sender object on machine B to send a network message from A to B. To send a message from B to A, you will need a reverse path as well - 4 objects total to perform bidirectional comms. Some object wrappers hide this detail internally. I don't recall offhand whether Indy hides this or not.
It would probably be easiest if you use a common TCP/IP protocol for your machine to machine communications, such as HTTP. This will make it easier to get your connections through firewalls and proxies that frequently exist between arbitrary users. To avoid conflicting with any HTTP web services that might be running on either machine, you should use a custom port number with the IP address: 192.168.1.10:12345, not the standard HTTP web server port 80. This is what most of the IM clients do.
So I have a connection pool setup. Which is great and all since I have an application that really needs it. However what I would like to know is if it is possible to share this connection pool with other J2SE apps? Would this even be worth it, as opposed to creating a connection pool based on each apps needs? If it would be prudent, how can I accomplish this?
It is not hard having connection pools in a single JVM doing multiple things - that is what applications servers do everyday (using JNDI to throw objects across classloaders)
The interesting part is when you have the connection pool in a separate JVM from the client code needing it, as this does not immediately allow simply asking for and getting a connection from the pool and returning it afterwards.
Basically you have two options:
Doing remote requests for all your JDBC commands over the network. This will most likely mean that the data will travel over the network twice, from the database to the connection pool, and then from the connection pool to your application. If the database connections are very expensive objects then this might be a viable solution.
Use RMI to get the connection object from the connection pool JVM to your own machine. This is a very expensive operation, but can as far as I know include the actual driver classes, allowing your connection pool to provide connections to databases not known to your application JVM. To me this would only make sense if the database connections were ridiculoulusly expensive or it was a requirement to be able to support additional databases after deployment without changing the original deployments.
Note that the primary reason for having connection pools at all is because connections are expensive to create, use shortly and then discard. Some databases more than others, e.g. MySQl is (or was when I tried) very cheap so it might be the simplest just to do that.
So. First of all: Measure what your connection pool buys you in time, and then consider if it is worth your while to centralize this further.