High memory usage with WCF Discovery
Yes, I know that this is basically saying that select is broken, but I am seeing some very strange stuff here. The code in question is this:
for (int i = 0; i < 15; i++) { var discoveryClient = new DiscoveryClient(new UdpDiscoveryEndpoint()); var findCriteria = new FindCriteria(typeof(IDiscoverableService)) { Duration = TimeSpan.FromSeconds(1) }; discoveryClient.Find(findCriteria); discoveryClient.Close(); }
The full repro can be found here.
The problem is, put simply, that before this code, the working set is: Working Set: 57,640 kb, after this have executed, it is 90,288 kb.
I went over the code with a fine tooth comb, but I don’t really see where all of this memory have gone.
The actual memory reported by GC.GetTotalMemory does go down after the cleanup, so I guess it could be that .NET isn’t releasing the memory back to the OS, but it still worries me somewhat.
I tried it with higher numbers for the loop, and it seems like eventually it settles down on some number and doesn’t grow from there. My main problem is what happens when you start doing async stuff. Let us take a look here:
int count = 150; var countdown = new CountdownEvent(count); for (int i = 0; i < count; i++) { var discoveryClient = new DiscoveryClient(new UdpDiscoveryEndpoint()); var findCriteria = new FindCriteria(typeof(IDiscoverableService)) { }; discoveryClient.FindProgressChanged += (sender, eventArgs) => { }; // do nothing discoveryClient.FindCompleted += (sender, eventArgs) => { countdown.AddCount(); discoveryClient.Close(); PrintMemory("Complete: "); }; discoveryClient.FindAsync(findCriteria); } countdown.Wait();
At peek, I am seeing 600 MB (!) of memory used. Note that we are now using the default duration of 20 seconds, and it seems that DiscoveryClient is very heavy on memory.
If you know how, can you take a look at the code and tell me that I am crazy?
Comments
Fore some time ago I have a problem with objects that was using events and the original object (discoveryClient) cant be released and disposed. The problem this time was that the object was only referenced by event-handler and because of that you have a reference to the object and the GC was not able to discard the object. After remove the event-handler after the object was used fixed my memory-leak problem.
Patric, But I am not holding any reference to the DiscoveryClient in the event handlers, or anywhere else.
As an MS intern, I have an MS Q&A tech support card. Tell me if you need one :)
Funny enough I was looking at this problem last week about wcf service holding onto large amounts of memory it turned out to be a simple configuration change to the setting maxBufferPoolSize. When you look at a lot of examples about sending large files etc through wcf services everyone blindly sets all the settings (maxBufferSize, maxReceivedMessageSize etc) to huge values and ussually the maxBufferPoolSize get set to the same huge value. However, this value indicates the pool size not a buffer size. so you want to limit your buffer pool to a number of cached buffers and not let it run wild. Set the maxBufferPoolSize to 1 just to test this theory and your memory usage should drop dramtically (if it is this!).
I posted something on stackoverflow about this http://stackoverflow.com/questions/6713180/addressalreadyinuseexception-when-using-custombinding-but-not-when-using-nettcpbi/6720923#6720923
hope this helps!
Jon, Interesting observation, and I tried playing with those values, but I don't think that this is it, I am seeing the same numbers, pretty much, even after limiting the max pool size to just 512 (bytes!).
Vadim, Thanks, I wouldn't want you to "waste" this card unless you really don't have anything else to do with it.
I could be wrong but could you try using ( var discoveryClient = new DiscoveryClient(new UdpDiscoveryEndpoint()) ) { var findCriteria = new FindCriteria(typeof(IDiscoverableService)) { Duration = TimeSpan.FromSeconds(1) }; discoveryClient.Find(findCriteria); discoveryClient.Close(); }
Saj, Nope, I am calling Close there.
How about bringing out the big guns: procdump from sysinternals to make a dump, and windbg with the psscor4 extensions to show you where the memory has gone. ( with !dumpheap -stat )
Remco, I have done that with dotTrace, I can see that most of the memory is byte[], held by WCF, doesn't really help me figure out what is going on.
Really strange, the second example levels out at 210MB on my box (Vista 64bit), with no config of WCF whatsoever.
Remco, Even so, 200MB still seems to be very excessive amount, no?
This may be simplistic, but is the "discoveryClient.Find(findCriteria)" method disposing of its results after returning?
Kerry, I would hope so, I am disposing on it directly afterward, at any rate.
Maybe it's a debugger interaction. running the sample outside of the debugger got me these results with a debugger attached i got 209 for all the results.
Before collect:209,000 After collect:40,000 After finalizers:40,000
so select isn't broken. the debugger is?
full code:
It gets weirder, the results aren't consistent.
only if I add this bit of code before the console.readkey I get a consistent drop of memory usage to about 40 MB
So my guess there's something fishy going on in the unmanaged world.
Last spam of the day, just edited the full repro to include an extra sleep of 60 seconds. Verdict: Select isn't broken, it's garbage collector is slow ;-)
Remco, My problem isn't so much that this is a memory leak, my problem is that it is using so much memory.
Ah yes, well, can't help you there. apparently 640K is not enough for even 1 discoveryclient.
Maybe it behaves better memorywise if just 1 discovery client with multiple async discoveries in flight is used instead of a whole bunch of clients? I am unfamiliar with System.ServiceModel.Discovery so I'm just guessing here.
It's probably the Binding buffer configuration.
Using the below UdpDiscoveryEndpoint configuration for both service and client there is substantially less memory usage (from 170MB to 67MB maximum working set). Also, discovery is slower.
Remco, I am probably going to try that next, yes. But I think that the actual cost is per connected server, need to check that further
That is very interesting, because it looks like most of the memory cost is actually on the Service Host, not on the DiscoverClient.
Just FYI, the post looks all funny in Google Reader, apparently doubly-encoded.
Just looked through the System.ServiceModel.Discovery assembly using reflector and the Find method on DiscoveryClient instantiates a SyncOperationState object which internally contains ManualResetEvent.
SyncOperationState class is internal but not disposable, and I cannot see anywhere that the ManualResetEvent is closed or disposed.
I suppose the GC will eventually get around to collecting this but would definitely contribute to the excessive memory
Robert, I find it highly unlikely that a single ManualResetEvent would be that expensive.
Comment preview