Myrinet timing

I've spent a lot of time lately improving and trying to understand the myrinet performance.

Here I show round trip performance between vxworks nodes. First of all, I see a significantly better performance between the node "bdb" (used to be gb) and other vxworks nodes. The myrinet card on this node is newest board that does NOT have the Myrinet 2 functionality. (lanai 4 is very old sector broker boards, lanai 7 bdb and some of the l3 nodes, lanai 9 is the new nodes used by trigger and some of the newest detectors) The newer nodes are running in a "backward compatability mode". Also, the BDB node is different. I still don't know if the performance difference is due to the drivers, the electronics, or the processor itself.

Round Trip Messages

Here I show the round trip times (I don't divide by two). I don't see any latencies > 400 usec, although there is a tail with about 15 of 50000 messages with times greater than 200 usec. This is node dependent, and I don't see it on the trigger nodes. I don't know the source.

Most messages take about 60-70 usec round trip, but there is a striking peak (on a log plot) at ~150 usec. I beleive this is due to the way the myrinet control program on the NIC works. The physical layer of myrinet is NOT "reliable" (although it has low error rates). The way that they guarentee reliability is to send ack signals back to the sender. However, the ack signals themselves are not reliable, so they also put in a timeout. However, the timeout is handled in a strange way, every millisecond a timer goes off. All messages that have not been acknowledged are then assumed to be bad, and are queued for re-sending. Because the timer signals are out of sync with the messages this means that it is "normal" for messages to be interupted this way. If you assume message takes 20usec to send, and that when you send round trips you are sending messages 1/2 of the time, you would expect about 1 in 100 trips would be interrupted. This is about the ratio of messages in the second peak to the first.

Unidirectional messages

Timing while sending unidirectional messages, is a bit more confusing. The distribution of times depends very much on which node sends to which node. The tails on the distributions are much longer than for round-trip messages. The reasons for these observations are tied up with the internal buffering of the messages. This is not an issue with ping-pong messages, because there is never more than one message in the system at a time.

Before describing what is happening its usefull to know the rates for unidirectional sends. The sustained average rate depends almost completely on the sending node:

l2ana01 (linux lanai9) = 9.3 usec
bdb (vxworks lanai7) = 14 usec (user task high priority)
bdb (vxworks lanai7) = 17 usec (user task low priority)
l1/ctb (vxworks lanai9) = 20 usec
pmd (vxworks lanai9 slower cpu?) = 25 usec

You also need to know that to user myriLib you have at least two tasks. The first is the user task, which performs the message sends. The second is the receive task which receives all information coming from the network card. This information consists of the arriving messages, and also the confirmations that sending messages have been sent.

There is some competition for resources between these two tasks, so I do observe some effects of inverting the priorities for them. I assume, of course that the user task blocks when not in use, so if you are polling you don't have a choice.

Sending unidirectional messages I get better performance keeping the user task at high priority. This ensures that the card always has messages to send.

In this plot, you see spray messages sent from the slowest node (pmd) to a faster one (bdb). In the case where the user task has a high priority (uh), you see a nice peak at about 25usec (the sustained rate), and no additional structure. What happens is that the users buffers are filled up and one message is sent at a time.

However, when the priority of the user task is lower than the receive task what happens is that several messages get put into the queue which the NIC is processing the first one. However, as messages finish getting processed, the receive tasks starves the user task as it processes the "event sent" notifications. The result is that the time distribution has a sharp peak at ~10usec followed by another broad peak at 40-100 usec.

I hacked up the driver to have a counter of the number of messages currently in the system. Then I take the difference between each successive message send to see how many messages finished being processed during this send. Here is the result for a low priority user task. (If the user task has high priority, the number is always 1 after the first 29 sends fill up the buffers).

This feature is even more pronounced with a faster NIC doing the sending

Here, for the low priority user task, you see exactly the same situation, although the distribution is shifted to slower values because the more events can processed at a time.

You see in addition a new feature with the HIGH priority sender. In this case, the user buffers are always completely full, however I beleive the same queueing effect occurs with the buffers in the destination receiving node. Unfortunately, I don't have any way to put hooks in to demonstrate that this leads to the peak in the high priority messages.

Here I show the rates corrected by the number of sends that were processed during the period of the send. (t / nserved) Note that the rates are slightly inflated, because the time spend sending 0 messages is not in the data of this plot. However, you do see the point of buffing the messages in the first place -- the rate per message drops as the number of sends increases.

Finally, I show the timing distributions for the spray messages on the trigger nodes. Sending from l1 to ctb (same speed), I don't see much structure. Sending from l2 I see strong evidence of the queueing effect.

Spray messages with empty buffers

The situation in L1 and on the DSM clients is different. Here you need to do some processing that is independent of the myrinet card every event. The myrinet card is doing its own work in parallel.

I have mimicked this situation (poorly), by sending 5 messages at a time and then waiting for 10ms (the shortest wait in vxworks). This gives a good indication of how much of the latency is due to the host CPU.

We see that the times are much more deterministic, and very much shorter. Here we can also see the difference between the single message versions of myriMsgSend and the multiple message version. The multiple message version saves about 10% (8 usec / msg compared to 9 usec/msg. The reason for this is just that the multiple message version saves several semaphore operations.

Simplified trigger protocol

The final test I made was a much simplified version of the L2 protocol. Here I sent two messages from L1, first a message to CTB and then to L2. CTB forwards the message to L2. Finally L2 returns the message to L1. I show the round trip times.

The result is quite similiar to the results from ping-pong messages, exept the times are somewhat longer (more messages) and the distribution a bit broader as would be expected. I do, unfortunately see some very long events. They don't show up every time, because my test only runs 50000 cycles. My last plot is the same plot but for a run that had a long delay in it. It shows that the delayed event does not seem to be correlated with other delayed events. It also shows that the delayed events are not frequent enought to adversely effect the overall rate.

The rate of cycles with > 2ms is approximately 1 in 100,000 to 200,000 cycles.

Jeff Landgraf
Last modified: Tue Aug 19 18:25:12 EDT 2003