Survey of SCT latency budget ============================ Email archive ------------- Email archive to support the latency budget table. There follows some edited emails etc. ..................................................................... Date: Wed, 14 Jul 1999 20:32:44 +0100 (BST) From: Martin Postranecky Subject: Re: SCT Latency Regarding the TIM latency, I agree with the values in your table : - The first BC covers delays between the TTCrx optoreceiver and the TIM output buffer flip flops. - The second BC is from the TIM output buffer to the ROD input buffers ( ie. it includes the ROD set-up time => it includes the "Step 1." of Dick's table below ) On Sun, 27 Jun 1999, John Lane wrote: > TIM > 1 BC translator, logic and buffer > 1 BC buffer flip-flop > The optimization of latency makes the TIM and TTCrx fine delay > settings interdependent. The ROD setup time is part of the latency > of the last flip-flop on the TIM. ..................................................................... Date: Tue, 13 Jul 1999 21:53:59 -0700 From: Richard Jared Subject: ROD latency I have included steps in the trigger latency that may be covered in other areas. The latency steps for the ROD and interface cards are detailed below: Step 1. One clock 40 tick. Time to transport the trigger from the TIM to latching in the ROD. Step 2. One clock 40 tick. Time to start the trigger command in the ROD. Step 3. One clock 40 tick. Time to register trigger pattern and adjust delay of trigger in clock ticks. Step 4. One clock 40 tick. Delay time to select the active links in the ROD for throttling. Step 5. One clock 40 tick. Time to transport the trigger from the ROD to latching in the BOC. Note, The three clock 40 ticks that are used to generate the trigger are assumed to be counted in the front end chips. ..................................................................... Date: Wed, 14 Jul 1999 11:02:59 +0200 From: Mark HATCH Subject: Re: SCT cable/fibre lengths I have looked at your web page and have a few comments. - ... Yesterday we had an IDET services meeting and Jan Troska of RAL presented cable length estimates and latency. Basically ATLAS TC agreed to investigate the routing of the fibres around the voussoirs in such a way that the total cable lengths between IDET/RODRACKS/TTCRACKS do not exceed 130m. - From an organisational point of view it would be best if you went through Andy Nichols of RAL for all matters relating to lengths and routing of cables. He is my official contact for ALL services of the IDET. In this way he is informed of all progress in this field. ..................................................................... Date: Wed, 14 Jul 1999 09:47:20 +0100 From: A.Nichols@rl.ac.uk As far as the fibre routes go, it was decided to keep to the 2 routes per quadrant, as we have now. ... Some people still want to do the phi rotation at the barrel radius and emerge at one azimuth, but I managed to deflect that idea, pointing out the difficulties... [eg not enough patch-panel space for all fibres to exit in one place] ..................................................................... Date: Tue, 13 Jul 1999 13:36:55 -0700 (PDT) From: Alex Grillo Subject: Re: Fibre Lengths I participated in the ID Services Meeting today via phone from California. Three fibre paths were discussed. The longest (~116m) is as Tony describes. The shortest (which I think is still 97m but I'm not sure) is the one many engineers don't like. The intermediate one is phi-symmetric inside the ID but saves length by somehow taking a short cut through the voussoirs. My understanding is from verbal descriptions since I had no drawings to view. The "voussoirs" route is purported to be 107m which when added to the new 27m from CTP rack to ROD rack would give us 134m and require us to use up about 1 of our BCs from contingency. I told them that was probably OK but I wanted to make sure there was sufficient conservatism to cover routing into racks and through patch panels. The response I believe is that this needs to be confirmed. ... In summary, we still don't have a final routing but everyone seems to be aware and accepting of the 130m limit (with maybe a slight increase to 134m) and the "voussoirs" path seems to be most likely to be accepted. ..................................................................... Date: Tue, 13 Jul 1999 12:01:03 +0100 (BST) Date: Mon, 5 Jul 1999 08:52:28 +0100 (BST) From: "Tony Weidberg (LHC)" Subject: Fibre Lengths Jan Troska and I have gone over the fibre lengths for the latency calculation including the effects of the proposed positions of the SCT racks in USA15. The positioning of the racks does help compared to the old model but not as much as I had hoped. Here is a summary of worst case fibre lengths (i.e. the ones you need for the latency calculation). TTCvi-ROD : 27m (cf 32m in your table) ROD-PPB1 : 116m PPB1-DORIC: 2m We need to add on some contingency for Patch Panels: 2m Cavern Cable Length Uncertainty: 5m which gives a total ROD-DORIC (worst case) distance of 125m. This is of course very bad and uses up too much of the contingency in the latency budget. We have also looked at routing by the Voussoirs and in which case the worst case length ROD-DORIC is 107m which would be acceptable. The 116m is assuming a phi symmetric routing. The shorter length is still coming out in 4 phi slots but is going via the voussoirs. The route with all fibres coming out at one phi slot is strongly disliked by the engineers. ..................................................................... Date: Mon, 14 Jun 1999 20:39:27 -0700 From: Richard Jared Subject: Minutes of the SCT June 2 meeting ... It was agreed that all free latency be placed in Alex's latency contingency list and that we apply for any latency that we need in the future. ... Latency is currently 128 clock ticks with 2.5 contingency, compared to a pipeline of 132 with 4 contingency required by Alex. ..................................................................... From: Mark HATCH [mailto:Mark.Hatch@cern.ch] Sent: 05 March 1999 16:37 Subject: SCT & Pixel - Routing of optical fibres Notes from the meeting at RAL on the 4/3/99 Present: M.Hatch CERN A.Nichols RAL 1. Routing of fibres from the IDET to PPB2 At MH's request AN has studied an alternative routing of the fibres from inside the IDET to PPB2 so that they can all exit on the USA15 side. This has two benefits: a) Reduction in the length of cables b) Grouping of the fibres so that, if necessary, they can be accommodated in a secure trunking for safety reasons. AN handed over a series of drawings to MH showing how the fibre ribbons could be routed inside the IDET volume. This arrangement is a great improvement on the previous scheme. It drastically cuts the length of cable allowing all the fibres to exit the ATLAS detector close to the wall of USA15. 2. Routing of fibres from PPB2 to the USA15 From each end of the IDET, and after PPB2, there will be two bunches of fibre cable. Each bunch will contain 32 cables of 9mm dia for the SCT and approx. 16 cables for the Pixel. Each cable will contain 96 fibres. It is proposed that these cables will be housed in a trunking, possibly of the screwed down type for safety reasons. MH will investigate trunking type and dimensions. 3. Cable lengths From the PPB1 to outside the magnet, point A = 22m From point A to the entrance of the cable gallery, point B = 13m From point B to the entry to USA15, point C = 12m From point C to the FURTHEST rack in USA15, point D = 40m Cabling inside rack = 3m Total = 90m Notes: - No contingency in these cable lengths. - Rack positions NOT known - Distance from DAQ rack to Trigger rack NOT known. 4. Allocation of racks in USA15 According to P.Farthouat there has been no official allocation of racks to anyone in the USA15. The procedure for how this is done needs to be defined within ATLAS. ..................................................................... Date: Sat, 13 Feb 1999 01:10:45 +0000 (GMT) From: John Lane Notes on latency phone meeting 11 February 1999 ----------------------------------------------- Present: Alex Grillo, John Lane, Andy Lankford, Steve Pier, Roy Wastie Alex said we need to sort out the latency issue in the next two weeks because we are close to submitting the next iteration of ABC(D). Roy said the latency budget for the BPM chip can now be reduced from 4 BC to 2 BC. It clocks in the data at 80 MHz (not 40 MHz). Steve said he had a few options for the ROD output to BOC in http://positron.ps.uci.edu/~pier/ROD/pdf/ROD_BOC_Interface0.PDF and the BPM chip and BOC engineers could choose which one meets their timing requirements. Alex said the ABC pipeline length was found to be 129 instead of 128 and then the RAL designers added 3 more columns to the pipeline to make 132. It could also be noted that the ABCD pipeline must be modulo 12 (hence 132) and so we increased the ABC to match it but we don't want to add any more because we would have to add another 12. The latency target for the SCT becomes 128 BC. The row for CAFE latency in the latency table should be absorbed into the ABC(D) row because ABCD contains its function. ABC(D) budget becomes 7 BC conservative by 1 BC (from 8-0.6=7.4 BC). ABC(D) latency includes one clock period to allow for adjusting the phase of the clock with respect to the input signal. Andy said we can reduce the latency by getting the L1A signal to the Link more directly, maybe even having TTCrx on BOC, if really necessary. We should understand the fibre lengths: by how much are they conservative? Roy would try to find out more about the fibre lengths in time for John's talk next Thursday. John said today we have saved 2.4 BC and the total becomes 129.6 BC. ..................................................................... ..................................................................... Date: Sat, 14 Nov 1998 03:30:30 -0800 (PST) From: Alex Grillo What we need to make sure is that when the L1A arrives at the pipeline of ABC(D), the data is still there. Therefore, I would like the table of latency budgets to include all the way up to the point of extracting data from the pipeline. Here are my estimates for the latency from the output of the DORIC onward. In the spirit of this being a budget table where each component takes a value to live within, I will provide over-estimates. Signal propagation DORIC->ABC(D) 4ns Receive command signal by ABC(D) 25ns ABC(D) command decode for L1A 75ns Logic time inside ABC(D) 50ns Data fetch from pipeline 50ns (This includes 1 clock to fetch pipeline column prior to BC of actual L1A.) Also, there are some times which add to the available latency time. These occur before the data is entered into the pipeline. Again in the interests of over-estimating any problems, I will set these to very short times or just take as zero. Particle flight time from IR to sensor 0ns Signal propagation in strips 0ns Signal process to comparotor (CAFE) 16ns Signal propagation to pipeline latch 0ns ..................................................................... Date: Tue, 24 Nov 1998 09:44:15 +0000 (GMT) Date: Fri, 27 Nov 1998 08:05:38 +0000 (GMT) Date: Tue, 1 Dec 1998 16:35:40 +0000 (GMT) Date: Thu, 21 Jan 1999 13:01:09 +0000 (GMT) From: "Tony Weidberg (LHC)" Propagation Delays for TTC Links -------------------------------- BPM ASIC: the worst case propagation delay (for the L1A signal in the BiPhase Mark chip when the channel delay is set to zero) is 4 clock cycles (100 ns) from i/p to o/p. The BPM i/p time is defined as the clock rising edge. Fibre: speed of propagation for two types of fibre is 5 ns/m. (measured propagation delay 4.92 +/- 0.03 ns/m) Total fibre length = 97 m. (90m ROD to PPB2/PPF2 plus 7m PPB2/PPF2 to furthest detector) Total propagation delay in fibre = 485 ns. The fibre length assumes the worst case position for the RODs in the trigger counting room. DORIC4: i/p of Biphase mark edge to o/p of data leading edge = 19.5 ns Total propagation delay from i/p of BPM to o/p of DORIC = 605 ns. ..................................................................... Date: Thu, 03 Dec 1998 10:23:29 -0800 From: Steve Pier There is a fair amount of contingency. The absolute minimum is 2 BC. See Khang's reply to my email below. The best way to think about it is 7 clock periods. We should be designing a synchronous system in which propagation delays are irrelevant so long they are << 25 ns and setup and hold times are met. If I said the ROD latency is the absolute minimum of 2 clocks, I would mean: cycle <------ 0 -----><------ 1 -----><------ 2 -----><------ 3 -----> CLK __--------________--------________--------________--------________ ^ ^ ^ ^ <-- t1 --> L1A ____________----------------____________________________________________ <-t2-> 1 1 0 XC ________________________________________--------------------------------__ One way to do the accounting: cycle 0 is part of the TIM latency (it is associated with the last flip-flop on the TIM), cycles 1 and 2 are ROD latency, cycle 3 is associated with the first flip-flop on the BOC. The three signals above are at the ROD slot. L1A is TTC0, (but shown active high for clarity). XC is the control stream from ROD to BOC. Carets mark the active edges of the clock. t1 is due to propagation delays on the TIM and backplane. (It is the unusual hold time mentioned in the ROD to/from TTC document.) t2 is due to propagation delays on the ROD. It might include such things as CLK translator/buffer delay and the clock-to-output delay of the last flip-flop on the ROD. Both t1 and t2 are irrelevant so long as they are << 25 ns and setup and hold times at the receiving flip-flop are met. The ROD number is from L1A to the first bit of the command, so it does not include the number of bits in the command itself. This is sensible, because not all commands are the same length. ----------------------------------------------------------------------- Date: Fri, 6 Nov 1998 15:05:31 -0800 (PST) From: Khang Dao To: pier@positron.ps.uci.edu (Steve Pier) > --------------------------------------------------------------------- > I assume fast command latency will not have an adjustment on the ROD. > Optional pipeline delays are indicated by parentheses. > > 1 buffer flip-flop external to FPGA > (1) flip-flop in FPGA I/O block (at input pin) > 1 loadable shift register (e.g., loaded with 110 when L1 received) > (1) flip-flop in FPGA I/O block (at output pin) > (1) buffer flip-flop external to FPGA > --------------------------------------------------------------------- > Unless I have forgotten something important, the absolute minimum ROD latency > would be two clocks and a probable maximum would be five clocks. I agree the absolute minimum is 2 but the likely latency is the 5 cycles you mention above. I think 7 cycles might be a bit safer, but we can go lower if really forced to. Date: Mon, 07 Dec 1998 10:40:45 -0800 From: Steve Pier There are two types of contributions to L1 delay. Perhaps it would be best to tabulate the two types separately for each component in the system. Type D (discrete) is digital pipeline delays (flip-flops or equivalents), which are always whole numbers. Type C (continuous) is everything else. Date: Tue, 05 Jan 1999 14:17:55 -0800 From: Steve Pier I think you could safely reduce the ROD latency number to 4 clock periods. I would prefer having a few clocks for contingency. ..................................................................... Date: Fri, 06 Nov 1998 09:27:39 +0100 Date: Tue, 10 Nov 1998 15:23:47 +0100 From: Philippe FARTHOUAT The latency breakdown is the following (all details in level-1 TDR page 452): - The CTP issues the L1A after 60.1 BC (1502 ns); - The signal has then to go to the TTCvi n * 5ns (with n= the length of the cable from the CTP to the TTCvi; this may be either close to the CTP or close to your electronics racks if you prefer); - The TTCvi, and TTC crate introduces 1 BC latency (25 ns); - Then you have to count the distribution to the TTCrx 5 ns * fiber length; - Then the TTCrx chip: 3 BC (75 ns); - Then in your case the protocol change and elec-opto conversion; - Then the fiber length; - Then the opto-elec and the DORIC. To summarise, there are 64.1 BC (1602.5 ns) introduced by the level-1 trigger and the electronics components of the TTC (TTCvi, TTC crate and TTCrx). The rest of the latency are cable and fibre delays plus your TTC electronics. Nick rightly pointed out that I forgot to mention that the L1A latency can be expanded by 500 ns (contingency). So the L1A can be issued by the CTP as late as 80.1 BC after the interaction (2002 ns). Date: Wed, 02 Dec 1998 08:40:31 +0100 From: Philippe FARTHOUAT I would like to try to clarify again the latency question. The L1A is available at the output of the CTP after 1502 ns to which a 500ns contingency must be added. Then the electronics part of the TTC requires 100 ns (TTCvi, modulator,electro-opt,opt-elec,TTCrx). You must then add the cable and fibre length to reach your ROD crates. This could be as high as 40 m or 200 ns. We are now at 2302 ns. To this number you must add the fibre length to the front-end. The 97 m seems a good approximation and gives 485 ns. This leads to 2787 ns to which you must add the time needed by the SCT electronics to encode the signal and to convert it to opto at the ROD level, and the opto-elec conversion+Doric+transmission to ABC(D). Assuming a pipeline length of 128-132 clocks, you have 413-513 ns to do it which seems to me largely enough (this is done in 100 ns in the TTC). If there is a big problem one can act on the distance between the CTP and your RODs. But do not take it as granted as the TRT must minimize the distance detector-ROD because of the copper links. ..................................................................... Date: Thu, 3 Dec 1998 14:23:12 +0100 (MET) From: Nick Ellis I have now discussed with Philippe and we agree that interpretation (2) in your mail is the correct one. According to the latency estimate in the LVL1 TDR, we have used up 55 ns of the 500 ns contingency. (I should point out that we still hope to recover this 55 ns by improvements to the design of the LVL1 trigger, using faster cables, etc.) On Wed, 2 Dec 1998, John Lane wrote: > I'm trying to understand the latest time the L1A > could be issued by the CTP after the interaction. > > So far I've thought of these two interpretations: > > 1. The latest time the L1A can be issued by the CTP is 80.1 BC > and the advertised 2500ns latency is now 2555ns (82.2 + 20 BC) ? > > 2. 2.2 BC of contingency has been used up and the latest time the > L1A can be issued by the CTP is 77.9 BC (60.1 + 20 - 2.2 BC) ? ..................................................................... ..................................................................... ATLAS Level-1 Trigger Technical Design Report, 24 June 1998 http://atlasinfo.cern.ch/Atlas/GROUPS/DAQTRIG/TDR/tdr.html From chapter 16.6, page 442: ... A worst-case length of 80 m has been taken. This corresponds to the maximum distance between any on-detector front-end electronics element and the furthest crate in the underground control room (USA15). From chapter 18, page 451: The maximum acceptable latency of the ATLAS LVL1 trigger is specified in a document [below] ... Any detector that wishes to have one or more of the following: ... * non-optimal (from the point of view of latency) cable routing between the TTC crates in the USA15 counting room and the front-end electronics; must be prepared to cope with latency beyond 2.5 us. ... ..................................................................... Trigger & DAQ Interfaces with Front-End Systems: Requirement Document (version 2.0), ATLAS note DAQ-NO-103, 9 June 1998 http://www.cern.ch/Atlas/GROUPS/FRONTEND/FEreq980310.ps From section 5.1.1, pages 22-23: * ... For sub-detectors using the standard TTC distribution system and the shortest cables possible for returning the L1A signal to the FE electronics, this gives a maximum latency 2.5 us. * The quoted latency assumes that the cables returning the L1A signal to the on-detector electronics follow the shortest possible path. It DOES NOT INCLUDE ANY EXTRA DELAYS INTRODUCED BY LONGER CABLING ROUTINGS DUE TO SPECIAL IMPLEMENTATIONS (E.G. RETURN PATH VIA REMOTELY-LOCATED RODs). ..................................................................... http://www.hep.ucl.ac.uk/~jbl/SCT/archive/SCT_latency_email.txt