PCI message passing in the RoB-in context

G.J.Crone 11/6/1996

Introduction

This note describes a simple message passing protocol based on software FIFO-s using mailbox registers in the PCI bridge chip as the read and write indices. Accesses to mailbox registers are effectively local and do not require mastering of the bus on the other side of the bridge, however since in most (if not all) cases there are two bridges between a device on a PMC and a device on the motherboard, access to the mailboxes from one side still requires mastering of the PCI bus.

For each channel one block of shared RAM and two mailboxes are required. The RAM block is organised as a circular buffer with the indices of the last words read and written stored in the mailboxes. Since the indices are likely to require 16bits or less, a pair of mailbox registers could be used for more than one channel (in the same direction). Separate mailboxes are used for the read pointer (RP) and the write pointer (WP) to avoid having to implement any locking. Only the sender writes to the WP mailbox and only the receiver writes to the RP mailbox.

As a general principle, a PMC card should not access any resources on the motherboard unless invited to do so. This implies that the mailboxes used should be in the RoB-in's bridge chip.

The most likely candidates for the bridge on the RoB-in are the PLX9060xx and the AMCC. The AMCC chip has eight 32bit mailbox registers, four writable from each side of the bridge. The PLX chip has four bidirectional 32bit mailbox registers. 

                       Message direction --->
               Motherboard                     |   PMC
         +-----------------------+             |
         | message(n)    word(0) |             |
         | message(n)    word(1) |             | +----+
         |            .          |  readonly <-- | RP | <-- writeonly
         |            .          |             | +----+
         |            .          | writeonly --> | WP | --> readonly
         | message(n)    word(m) |             | +----+
  RP --> | message(n+1)  word(0) |             |
         |            .          |             |
         |            .          |             |
         |            .          |             |
  WP --> | message(n+1)  word(m) |             |


                        Message direction <---
               Motherboard                     |   PMC
         +-----------------------+             |
         | message(n)    word(0) |             |
         | message(n)    word(1) |             | +----+
         |            .          | writeonly --> | RP | --> readonly
         |            .          |             | +----+
         |            .          |  readonly <-- | WP | <-- writeonly
         | message(n)    word(m) |             | +----+
  RP --> | message(n+1)  word(0) |             |
         |            .          |             |
         |            .          |             |
         |            .          |             |
  WP --> | message(n+1)  word(m) |             |



  Sender can write when
     (WP > RP)  AND (((WP + message_length) modulo buffer_size) < RP)
    OR (WP < RP) AND (WP + message_length < RP)
  Assuming message_length < buffer_size of course!

  Receiver reads whenever WP <> RP

Sending messages

The sender can write a message, as long as there is room in the buffer, and store the index of the last word written in the write mailbox. In the case of the motherboard sending to the PMC this requires two accesses to the PCI bus, one to read RP and one to write WP (although it may only be necessary to read RP if the value last read would indicate insufficient space). In the case of the PMC sending to the motherboard access to the mailboxes is local but the data block itself is written across the PCI bus in a block transfer.

Receiving messages

The receiver can detect whether there is anything to read by comparing the contents of RP (or its own knowledge of what it last wrote to RP) with the contents of WP in a polling loop or it could be prompted to read the mailbox by "ringing" a doorbell register. In the case of the PMC, polling the mailboxes would not involve any access to the PCI bus as they would appear on the local side of the bridge. If the motherboard processor were to poll, it would involve PCI bus access.

Having calculated how much data is in the FIFO, the receiver can read all the outstanding messages in one block transfer (or two if the wrap point in the circular buffer is spanned). Alternatively, it may transfer just a single message in one (or two) block transfers (by reading the message length of the message from within the message). Another alternative option is to use fixed length messages and to pass the index of the last message slot used in the mailbox registers.

Data flowing to and from the RoB-in


            RoB controller                    RoB-in
             ---------->                   <---------
   RoIR  ~10kHz   ~32bytes 320kB/s  |  RoID  ~10khz   ~1kbyte 10MB/s
   L3R    ~1kHz   ~16bytes  16kB/s  |  L3D    ~1khz   ~1kbyte  1MB/S
   DR     ~5kHz  ~160bytes 800kB/s  |
   MonR   ~1kHz?  ~16bytes  16kB/s  |  MonD    1kHz?  ~1kbyte  1MB/s
   Control ~1Hz?   ~8bytes    8B/s  |
   L3done ~1kHz    ~8bytes   8kB/s  |
                                    |  ErrStat  1Hz?  ~2bytes   8B/s
         ======            =======           ======           ======
         ~18kHz            1.16MB/s          ~12kHz           12MB/s

   Total bandwidth  ~13MB/s
The RoIR/RoID, L3R/L3D and MonR/MonD records are definitely request/response pairs therefore the request could specify where the data should be written and bypass this protocol for the output (the output address could even be in the RoB-out rather than the RoB controller).

The data requests (RoIR, L3R & MonR) and decision records could all be combined on one channel. Control requests should have a dedicated channel.

PCI bus accesses


        Motherboard            ---->      PMC
         read RP                 |     read data
         write WP                |
         ----------------------  |     ----------------
         2 single word accesses  |     1 block transfer

        Motherboard            <----      PMC
         write RP                |     write data
         read WP                 |
         ----------------------  |     ----------------
         2 single word accesses  |     1 block transfer

   3 accesses * 18kHz  = 54kHz   |  3 accesses * 12kHz  = 36kHz

   Total PCI bus access rate = 87kHz
GJC (gjc@hep.ucl.ac.uk)