How I broke the TCP Stack
Today I spend a large part of the day tracking down various strange errors. Those errors would be of the following variety:
- The connection could not be made because the target machine actively refused it.
- A connect request was made on an already connected socket.
- The underlying connection was closed: A connection that was expected to be kept alive was closed by a server.
Now, I am not doing anything strange with TCP/IP, I am just making plain old web service calls (and no, not using my hand rolled SOAP stack).
I can't really figure out what is going on, but I know that the problem is no in the TCP/IP level, it is something in my code, that is breaking things. It doesn't happen always, but it happens often enough.
I have ensured to my satisfaction that the server it up, no firewall in the way, respond to calls, etc.
Flat out weird, and I am not looking forward to trying to figure it out tomorrow.
Any ideas?
Comments
I've seen that before.. unfortunately, I don't remember what the cause was. it may have been related to sql server or iis. perhaps it was thread issues with iis.. wish I could remember and save you some time figuring it out.
Does the back-end live on another server within the LAN?
Grimace,
Tested on both LAN and WAN
Ayende,
There are several options you may want to investifate, do you have the correct config file in your web.confing, if the webservice proxy cant find the config file it will use a default value
<DefaultSettingValue("http://localhost/GIT/WebServices/ERToolkit/ERToolKitWebService.asmx"), SpecialSetting(SpecialSetting.WebServiceUrl), DebuggerNonUserCode, ApplicationScopedSetting>
Which may be your problem.
Hope this helps.
I've seen the seemingly-random "The connection could not be made because the target machine actively refused it" error from time to time when working on SvnBridge (particularly when hosting in IIS). Haven't been able to figure it out. I hope you can.
I could be wrong, but it sounds like you're suffering from the "HTTP 1.1 is broken in .NET web services" problem. If so, the code below will solve it. (I have a piece of code somewhere that auto-generate these files, it's time-consuming to patch up every web service you ever use...)
using System;
using System.Net;
static class WebReferenceHelper
{
}
namespace YourWebServiceNameSpace {
public partial class YourWebService {
} } }
Do you open and close the TCP connection repeatedly? And how frequently?
You may exhaust the number of ephemeral ports on the server machine if you open-use-close TCP connections in a tight loop.
I believe the reason is in the TCP protocol for dealing with closing down sockets where there is timeout period where the ephemeral port number is reserved. And you have like 16K ephemeral port numbers only.
Stevens Unix Network Programming is the definite source for this.
Proxy settings?
What about setting tcpview against the issue and check if the connections are properly closed?
We also started running into these problems intermittently, and we were never able to track down any rhyme or reason as to why they were failing. We ended up doing something similar to Julian's solution, but we found that only setting KeepAlive = false was enough to keep the errors from occurring. So far we've been error free and haven't seen any effects on performance.
There are rather basic suggestions that you may have already tried, but here are my experiences with those errors:
You have an application listening on some TCP/IP port, but nothing listening on that particular port.
You attempted to register a listener on a port which already had a listener.
So, if your service shutdown unexpectedly and released the port, you'd get error #1. If it shutdown unexpectedly and failed to release the port, you'd get error #2.
If you're client (web service caller) is running on a Windows 2003 SP2 server, check the following: http://support.microsoft.com/kb/919797
and choose to disable the MinFileBytesPerSec setting. That helped our situation.
-bp
Julian,
Yes, that was the first thing I tried. But while it can cause the last issue,, it isn't causing the others.
Steffen,
I am making ~ 20 calls, not nearly enough to exhaust the open connections.
James,
The application is trying to contact IIS, I verified that IIS suffered no mischief through the entire thing.
And I am not registering a socket to make an HTTP call.
I ran into these types of issues for .NET Remoting while running as a service since it did not manage the requests.
Are your requests long running? If so have you tried to set your application pool to contain a web garden so it can start up multiple processes? This may not be a long term solution if the separate processes are undesirable but you might want to give it a try to see if the ThreadPool is possibly hindering it...
are you able to telnet into that port?
if you are, issue in your code
if you are not, firewall or something else on the way
As I mentioned, some calls succeed, and yes, I can get there.
I know it is an issue with my code, I just have no idea what can cause that
Run this: "netsh int ip set chimney ENABLED" on the server where the webservice is.
Do you see a whole bunch of sockets AWAITING_CLOSE if you run netstat on the server?
Nope :-)
Try Brian Pearson "solution", it worked for me iirc
Hi,
You need to monitor connections to see what is causing a problem. I am running into the same issue where a .NET application has to talk to a third party server via HTTP 1.1 using persistent connections.
One of the common suggestion is to turn off "keepalive." I believe that this is not right because you are going to do exactly the opposite of what HTTP 1.1 aims to provide -- permanent connections. What I suspect happens is the following:
The client establishes a connection with a server.
The connection becomes idle.
The server attempts to close a connection. The connection is not fully closed yet.
The client is sending data to the connection that is about to be closed.
The connection is closed on the server side before the client's data has reached the server.
Obviously, the server is closing a connection that for a fraction of time is seen as normal to the client. What you can do is to tell the client to close connections more often. You can do this by decreasing the value of 'System.Net.ServicePointManager.MaxServicePointIdleTime' property. The default value is 100,000 milliseconds (100 seconds). It means that the client will close an idle connection after 100 seconds of inactivity. You can disable "keepalive" to close these connections more frequently or reduce the idle time to something reasonable, like 5 seconds. By explicitly closing connections on the client side you may be able to beat the odds of the client using a connection that has been idle for a long time.
We've got the same problem with the server connection messages in a .net 2.0 site we're building. Plenty of possible fixes out there if you Google that exact message. If we implement a solution that works I'll be back...
XOXO
Rachel
I don't know if you've figured this one out yet.. But just in case it's still a problem, and the problem is in "the network", and not somewhere in your application stack, WireShark is always your friend! Install, run, learn and love! It can tell you everything you need to know about the TCP streams, and for once you'll know exactly what you send and recieve. :)
http://www.wireshark.org/ - free and beautiful!
I don't know if you've figured this one out yet.. But just in case it's still a problem, and the problem is in "the network", and not somewhere in your application stack, WireShark is always your friend! Install, run, learn and love! It can tell you everything you need to know about the TCP streams, and for once you'll know exactly what you send and recieve. :)
http://www.wireshark.org/ - free and beautiful!
Comment preview