What am I missing? MSMQ Perf Issue
I am getting some strange perf numbers from MSMQ, and I can’t quite figure out what is going on here.
The scenario is simple, I have a process reading from queue 1 and writing to queue 2. But performance isn’t anywhere near where I think it should be.
In my test scenario, I have queue 1 filled with 10,000 messages, each about 1.5 Kb in size. My test code does a no op move between the queues. Both queues are transactional.
Here is the code:
private static void CopyData()
{
var q1 = new MessageQueue(@".\private$\test_queue1");
var q2 = new MessageQueue(@".\private$\test_queue2");
q2.Purge();
var sp = Stopwatch.StartNew();
while (true)
{
using (var msmqTx = new MessageQueueTransaction())
{
msmqTx.Begin();
Message message;
try
{
message = q1.Receive(TimeSpan.FromMilliseconds(0), msmqTx);
}
catch (MessageQueueException e)
{
Console.WriteLine(e);
break;
}
q2.Send(message, msmqTx);
msmqTx.Commit();
}
}
Console.WriteLine("{0:#,#}", sp.ElapsedMilliseconds);
}
Using this code, it takes 236.8 seconds to move 10,000 messages. If I use System.Transactions, instead of MSMQ’s internal transactions, I get comparable speeds.
Just to give you an idea, this is about 40 messages a second, this number is ridiculously low.
Changing the code so each operation is a separate transaction, like this:
private static void CopyData()
{
var q1 = new MessageQueue(@".\private$\test_queue1");
var q2 = new MessageQueue(@".\private$\test_queue2");
q2.Purge();
var sp = Stopwatch.StartNew();
while (true)
{
Message message;
try
{
message = q1.Receive(TimeSpan.FromMilliseconds(0), MessageQueueTransactionType.Single);
}
catch (MessageQueueException e)
{
Console.WriteLine(e);
break;
}
q2.Send(message, MessageQueueTransactionType.Single);
}
Console.WriteLine("{0:#,#}", sp.ElapsedMilliseconds);
}
Means that it takes 16.3 seconds, or about 600 messages per second, which is far closer to what I would expect.
This is on a quad core machine 8 GB RAM (4 GB free), so I don’t think that it is the machine that is causing the problem. I can see similar results on other machines as well.
Am I missing something? Is there something in my code that is wrong?
Comments
Is there a similar speed difference if you have only one queue and you are only reading messages?
Tapio
Tapio,
If I am just reading with I get 2331 messages per second using msmq transactions and about 3,000 messages per second using Single
Correct me if I'm wrong but transactional queues pesist every message to disk before returning from Send, so therefore the amount of RAM has little affect and more likely disc IO could be an issue.
What's the performance like you just dump 10,000 1.5kb messages to disc directly?
The code just here push 10,000 messages in 5.32 seconds, about 1,800 messages/sec
for (int i = 0; i < 10000; i++)
{
}
It would nice to profile the first example and see where the time is actually spent. Maybe the MessageQueueTransactionType.Single does not create a "real" transaction. You can't even rollback 'em or can't you? The first case creates a transaction that includes 2 resources (queues) while the second one creates 2 "transactions" each having only one queue. Maybe that has something to do with it. What happens if you create a transactionscope and use MessageQueueTransactionType.Auto?
Tapio
Tapio,
Most of the time is spent on the Commit()
Same results for DTC
Hi
"If I use System.Transactions, instead of MSMQ’s internal transactions, I get comparable speeds."
You are trying to perform two independent transactional operations within the same transaction so I expect it is using MSDTC to coordinate them (should be able to see the MSDTC transaction count to verify).
A single MSMQ transaction only applies to one operation, not both, so you'll see a much faster speed.
Cheers
John Breakwell (MSFT)
John,
Just to get it straight, if I am making two operations with MSMQ in the same transaction, I can expect the speed to drop to a crawl?
That doesn't quite make sense.
Just to give you an idea, I can do thousands of transactions pre sec using SQL Server + DTC.
I don't understand why making two operations (on two queues on the same machine) will cause this major slow down.
For that matter, I just tested it making two operations on the SAME queue, and I experience the same slow down.
I would assume that making transactional operations is pretty important for MSMQ, and as far I as I can tell, behaviors like read from queue, send to queue in the same transaction are pretty common
I noticed something very similar Ayende.
With Windows Server 2003 using .Single I would easily get 2k messages a second with load tests. On Server 2008/w7 I get on the order of maybe 350 pt second. Running it under dotTrace, all of the time is spent in MSMQ (my total application touch time is less than 500ms compared to like 22 seconds in MSMQ).
I figure it had to be something to do with the transaction support on 2k8/vista/7 compared to 2003 server. But you're right, it's dog slow on the newer OS.
Chris,
We are seeing something similar on Win2003 Server as well, though
Just for comparison, on my machine, 32-bit Dual core, 3GB RAM it takes 162 seconds. Not superfast, but about a minute faster than your measurement.
This is on Vista...haven't used MSMQ for a while so I had to do a fresh MSMQ feature install. Installed without any addons like triggers HTTP support, etc. - just the core.
Reducing the commit load by always reading and then writing 10 messages in a single transaction literally slahed the time to 1/10th, showing that the commit indeed is by far the most costly operation in the code.
Wow, quite shocking. I'm using custom message queuing implementation based on a table in SQL server database and I'm getting about 150 messages per second (while enclosing message handler in a TransactionScope). And I was wondering if moving to msmq will allow me to handle thousands messages per second - looks like i wouldn't even be able to have current performance.
@Frank - the problem is that you rarely can send or receive messages in batches, usually each message is an individual transaction.
@Ayende - I suspect this also affects RSB performance as RSB is using distributed transactions?
Rafal,
Yes, this affects RSB, because RSB make use of MSMQ.
I still need to test how RQ behaves here.
I'm guessing RhinoMQ and RhinoTX will be in extended beta sometime tomorrow afternoon...
J Healy,
You are aware that there IS Rhino Queues project, right?
Ah, that's right - I've clearly been distracted over the summer and need to catch up with things a bit. As I [vaguely] recall you were going to get Rhino SB running on top of your Queues, did that happen?
J Healy,
Yes, RSB works with RQ
I think the trouble is that your transaction is guarding an operation against a queue that is also receiving messages. The premise is that if something fails, the record from the receive transaction will be "put back", and this might be causing your performance headache.
From your example I don't believe a transaction is required. If the order of the received messages is important then use Peek, attempt to process, and if successful (or determined to be poisoned) pull it off the queue (I.e. ReceiveByID). If the order of messages isn't important read it, attempt to process, then if it fails, decide whether to pop it back on to try again, or discard.
Transactions carry a pretty steep performance hit in MSMQ and are really geared towards coordinating multiple queues.
@Steve - you probably are right, there's no need to enclose everything in a distributed transaction if the only thing you have to do is to put message back into the queue. You could use 'local' msmq transaction for that purpose and a separate, distributed transaction for the message handler.
Steve,
What happens if my process crashes? A message is lost
And Peek will let two concurrent consumers read the same message (BAD!)
I am also trying to coordinate between two queues, so I want to make use of this.
Rafal,
In the test code, I AM using local MSMQ transaction
Ayende, thanks for pointing this out, it's worse than I thought. But there's a post on Udi Dahan's blog, regarding NServiceBus performance ( www.udidahan.com/.../nservicebus-performance/) and he says:
"OK, so using the default nServiceBus distribution using MSMQ, on servers where the queue files themselves were on separate SCSI RAID disks, we were pumping around 1000 durable, transactionally processed messages per second, per server."
Rafal,
Yes, that is why I am shocked by this.
Would it meet your requirements to create one transaction for all of the moves? I tried this by moving the "while (true)" statement down to after the msmqTx.Begin and moved the msmqTx.Commit after the ending "while" brace. This dramatically sped up the processing. I tested what an abort would do where the commit would normally happen after 10000 records were moved. It was very fast as well.
Kerry,
No, this is just test code.
I am trying to see why my app (which does something similar) acts this way
Yes. MSMQ transactions are EXTREMELY slow. I have seen similar behavior. To get decent numbers with transactions, you must use something like transaction batching. Of course, if you just switch to WCF and net.msmq binding, this is provided out of the box.
I've never had to try and use transactions in this manner. You're likely going to get nailed trying to commit a change to a Queue that is actively receiving messages. My suggestion around Peek was if the order was significant which would imply 1 listener.
As for losing messages, that is an exceptional circumstance. Could it be handled by the poster of the message if it's expecting a response? MSMQ exposes quite a bit of functionality such as PeekById and prioritization to help detect and re-send dropped messages. You may want to reconsider whether the cost of attempting to guarantee delivery is worth it, or just handle the exceptional case. (ala using UDP vs TCP)
I don't know if this will work, but have you tried moving the creation of the transaction instance itself outside of the loop? (Still do the Begin and Commit inside the loop for each message, but avoid recreating a new transaction instance each iteration.) Not sure if transactions can be re-used like this or not...
Steve,
Consider the simple case of:
tx.begin
order_msg = read_order_from_queue
if process_msg order_msg:
tx.commit
I have a lot of logic that does this. If I crash, I want things to recover properly.
If I need to manage stuff manually, I might as well drop transactions all together.
Jesse,
Can you give me a link to this, I am not familiar with the term
I remember similar perf issues a long time ago with MSMQ transactions that were caused by slow write caching of the RAID controller. Replacement of the RAID controller improved a lot. Maybe MSMQ transactions are rather sensitive to disk write latency.
Here's how to do it with WCF.
msdn.microsoft.com/en-us/library/aa395219.aspx
You can do the same type of thing manually with MSMQ. The big slowdown appears to be due to the transactions themselves, so if you can send multiple messages in a single transaction, you will get significantly better performance.
Yeah, in that scenario I would advise having the sender manage contingency for it's message not being received and processed rather than transactions. I don't seen anything with what you've done that might suggest why the transaction is more expensive than it should be. (Other than maybe disk latency as Gerke mentioned.)
Services where I use MSMQ always have their own ACK private queue available for acknowledgements / status updates of their respective messages that they pass the address along as part of the request message. If they don't get an ACK to their satisfaction, they ping the queue for their request (in case it's still pending) and re-send if they don't find it. It's simple to wrap in, and if the scenario only manifests itself <2% of the time, it's a lot better option than accepting a 15-50% (or larger) performance hit overall.
Of course if you're looking for "fire & forget" where the poster can't be assumed to be responsible for checking on its own requests, then either accept the performance hit of the transaciton, or consider using a database instead.
Where I would use transactions is if I needed to ensure a message makes it onto multiple queues successfully, or situations where messaging is used to transmit sequential messages or messages on demand between 2 or more points. Chain is satisfied, commit.
Ah well, good luck in any case. ;)
Steve,
Yes, I'll probably have to go this way, because of the horrible perf of tx in MSMQ.
Very disappointed, thanks for the ideas
I don't know if it's true with MSMQ 4.0, but with earlier versions, having journaling enabled significantly increases overhead when receiveing messages from a queue.
One very obvious thing:
With what serializer do you create your msmq messages? The default (xml) is by far the slowest one. You should set it explicitely to get most out of it.
Yours,
Alois Kraus
Alois,
This is using a stream to do stuff, not a serializer.
It's definitely the single-item transaction, Ayende. I copied your initial code and then modified it to perform two runs. The first run without transactions:
SendData: 269 milliseconds
CopyData: 554 milliseconds
I then wrapped the loops in a transaction, so I'm committing all 10,000 messages in a single transaction:
SendData: 481 milliseconds
CopyData: 1,944 milliseconds
Your original code that commits each item in a separate transaction:
SendData: 16,107 milliseconds
CopyData: 194,201 milliseconds
So, yeah, the transaction cost here is astronomical and seems to have some sort of exponential function.
Comment preview