Draft Description of Proposed ROB-IN Module ------------------------------------------- (R.Cranfield 18-Jun-1996) (Last mod: 20-Jun-1996/RC) ROB-IN Discussion Group: ------------------------ CERN: R.McLaren EDINBURGH: O.Boyle RHUL: G.Boorman, B.Green, J.Strong UCL: R.Cranfield, G.Crone INTRODUCTION: For the purpose of more convenient development and prototyping work it has been proposed to split the ATLAS Readout Buffer (ROB) into three components referred to as the ROB-IN, the ROB-Controller and the ROB-OUT. The idea is that these components can be developed separately and used in configurations with one or other being replaced or emulated by alternative hardware or software. Each component would interact with some aspect of its environment according to the full LHC requirement, whilst the remaining communication (between components) could follow a more relaxed specification. Thus the ROB-IN would be built to receive front-end data at full speed, the ROB-OUT would be built to interface fully to a level-2 network, and the ROB-controller would be designed to receive messages in the proper format at the full rate. More specifically it seems useful to design the ROB-IN and ROB-OUT as PMC modules which plug on to a ROB-Controller motherboard. A convenient option for the motherboard would be a PowerPC SBC, preferably a 6U VME board, such as the CES RIO2 or the RADSTONE PPC60x. RoIRs Control | | | | | | V V --------------------------------- | | | --------------------------- | | | | | | | | | | | LVL1 | | -------- |P| | | =============>> | ROB-IN | |M| | | data | | -------- |C| | | | | | | | | | | | | | | | | | | | --------------------------- | | ROB-Controller | | --------------------------- | | | | | RoID | | | | | | <<============ |P| | | | | --------- |M| | | LVL3 | | | ROB-OUT | |C| | | <<============ --------- | | | | data | | | | | | | | | | | --------------------------- | | | --------------------------------- This document contains a draft description of a ROB-IN module proposed for development by a joint CERN/Edinburgh/RHUL/UCL team. OBJECTIVE: The objective of the proposed ROB-IN development is to design and produce a ROB-IN module which satisfies a set of user requirements (ROB-IN URD) derived from the ROB user requirements (ROB URD) to serve both as a basis for the eventual ATLAS ROB design and as a prototype for trigger/DAQ development work from the end of 1997. MAIN CONSTRAINTS: The most important constraints imposed by the ROB-IN URD are that the module be in single-width PMC format, communicating with the ROB-Controller and ROB-OUT across PCI, and capable of handling front-end data input on S-LINK at 100 MBytes/second. ARCHITECTURAL CONSIDERATIONS: It is proposed to base the design on the existing RHUL/UCL ROB, which has already been demonstrated to run at LHC rate, albeit with a simpler set of requirements. This design uses a multiplexed buffer memory with "pages" managed by an associated DSP microprocessor. The DSP buffer manager software provides the buffer hardware logic with a stream of "free" page addresses via a FIFO ("free-page FIFO") and reads back a stream of "used" pages on another FIFO ("used-page FIFO"). The buffer manager software also keeps track of which events are held in which pages so that it can supply data when required (by RoI Request or LVL2 accept) and free up pages when events are rejected. The essential strategy for the ROB-IN proposal is to add an S-LINK interface to the front-end, a PCI interface to the "back-end", upgrade the performance of the buffer-manager processor, and fit the whole lot on to a single-width PMC module! The ROB-IN would still act as a sort of ROB, but communicating directly to the ROB-Controller and ROB-OUT rather than the outside world. The main rationale for this approach is that the operation of the hardware buffer is intimately bound up with the software buffer manager process, which in turn directly involves the interpretation of RoI and data requests and LVL2 decisions. Whilst this buffer manager functionality cannot therefore be easily split across modules, it IS convenient to separate off both the network-specific functionality (in the ROB-OUT) and the interaction with other LVL2 and LVL3 components (in the ROB-Controller). The interfacing of RoI and data requests as well as decisions can be defined in an "internal" protocol, leaving it to the ROB-Controller and ROB-OUT to follow any externally imposed formats and conventions. DESIGN POINTS: 1) Size restrictions The size of the single-width PMC format is a major constraint on the ROB-IN design, which influences several of the design points below. It is conceivable, though not desirable, that the thickness of the board could exceed the spec, making it a double-width board, but it is important that the footprint stays at the single-width spec in order that both a ROB-IN and a ROB-OUT could be accommodated on a dual-site motherboard. 2) Power restrictions The limit for the power that can be supplied across the PMC interface is 15 watts. This constrains, in particular, the choice of on-board buffer manager processor. 3) Choice of ROB-IN processor Although the existing C40-based ROB can cope with more than the 100 kHz LHC event rate, the software on the ROB-IN is likely to be significantly more complex, and it is unlikely that a 33 MHz C40 DSP (or even a 40 MHz one) will be fast enough. At the same time none of the C40 special features are essential for the buffer manager job, although the twin busses and on-chip DMA are convenient. This opens the door to other processor choices and it is intended to make a survey of likely options. Plausible candidates include higher-speed members of the C40 family, SHARC DSP, PowerPC, Pentium, I960 and Alpha, though this last is probably ruled out by power restrictions. 4) Choice of PCI bridge There are also several options for the PCI bridge, including the possibility of custom logic. The current RHUL design for a C40/PMC interface, which is seen as a testbed for the ROB-IN PMC design, is based on the PLX chip, but it is intended to survey the functionality and convenience of other options. 5) Communication across PMC interface The preferred method for the communication of requests and data in and out of the ROB-IN across the PMC interface would have been to use FIFOs, addressable on one side from the PCI bus and on the other from the ROB-IN processor bus. However it is highly unlikely that such FIFOs could be physically accommodated within the small footprint of a PMC module. An alternative is to use memory on the PCI bus, OFF the ROB-IN board, in combination with mailbox registers in the PCI bridge. This combination can be used as a set of "software FIFOs", whose status can be polled from the ROB-IN without using the PCI bus. Indeed it is possible to follow the principle that the offboard memory (and PCI bus) is only ever accessed by the ROB-IN in response to direct requests from the PCI side (e.g. ROB-Controller). Such an approach is described in detail in the accompanying note by Gordon Crone. 6) Bus connections and DMA Some potential ROB-IN processors (e.g. C40 family) have dual busses and on-chip DMA. The former allows, for example, decoupling of buffer-memory access from PMC access, though it is not clear that this offers any performance advantage. Some sort of DMA IS likely to be necessary for adequate buffer-manager performance; this may be provided by the ROB-IN processor, the PCI bridge, or an external device. 7) TTC information It is still not clear that the ROB is the best place to do the kind of event-ID integrity checking (and possible resynchronisation) made possible by the distribution of event-ID/bunch-crossing information via the TTC system. The ROD seems a better place to do this, particularly since it will probably need to receive TTC signals anyway, whilst the addition of TTC connections to all the ROBs is likely to represent a significant overhead. The current ROB-IN design will include a TTC connection only if it is deemed important by the ATLAS trigger/DAQ coordinators. 8) Self-test facilities The ROB URD contains requirements for standalone testing in which the ROB should be able to act as the first, the last, or an intermediate link in a test-chain. In the case of the ROB-IN it is only the requirement to be first in the chain which is problematic, since acting as an intermediate link in the chain is just its natural mode, and it needs no special facilities to act as the last link as long as it is used in conjunction with a ROB-controller motherboard (i.e. the ROB-controller is really the last link with the ROB-IN acting as an intermediate link). It would indeed seem to be useful for the ROB-IN to act as its own data-source in certain test scenarios, without the need for an external front-end source. Ideally such a test source should feed in directly to the front-end of the ROB-IN in order to test out the paging hardware. However, board space is too limited in the PMC form factor for it to be realistic to incorporate the functionality of e.g. a SLATE. The possibilities of a downloadable memory or a simple pattern generator will be explored. 9) S-LINK communications It is important that the ROB-IN should be able to request interruption of the front-end data flow, for example when its buffer is nearly full. It is presumed that this will be via an XOFF signal on the S-LINK connection from the ROD. It is proposed that such an XOFF be generated by a programmable threshold on the free-page FIFO, so that the ROB-IN requests dataflow interruption whenever the number of free pages drops below a certain level. It is further proposed that the XOFF state be the default after power-up or reset so that the dataflow must be actively switched on by the ROB-IN processor. There is also a proposal that the XOFF signal could be used to provide a simple mechanism to "ping" the ROD to check whether it was still alive if no data had been received when expected.