Ph. Farthouat
G. Mornacchi								22 Jan 04

Run Checkpoint, run number and RODs

Introduction

After recalling the definition of a run checkpoint [1], we propose a mechanism to implement it, based on a special trigger type (Sweeper Event, also SE in the following) and on the tagging of the run number into the event at the ROD level.

This note does not cover the internals of the DAQ run control, viz. finite state machines, run control states etc. The proposed mechanism requires the run control to 

1) 	generate and maintain a data base of unique run numbers, according to a specified format, and
2)	perform a predefined sequence of operations as part of the process of moving from one run to another. The sequence of operations is defined in this note.

This note, also, is not concerned with implementation details in DAQ and HLT elements. In particular how the end of run is detected, namely [1]: by checking the run number or by means of a special trigger type.

Checkpoint

A transition from one run to another, triggered by a change in run conditions, made of two parts:

1)	A synchronous part, where the run control sequences those operations that ought to be executed in a defined order.
2)	An asynchronous part that is performed time-wise independently by any TDAQ elements downstream of the ROD. 

A checkpoint is intended to be used for transitions between runs that are "light" (for instance the transition does not necessitate the re-initialization of all TDAQ elements, in particular all level-2 and event filter processors) and such that the values of the changed conditions are not stored within the event data. This latter situation (the change in condition recorded in the event, as it may happen during certain types of calibration runs) is assumed not to need a transition to a new run and it is out of the scope of this note.

Implementation

TDAQ elements (that is to say components downstream of, but not including, the ROD) notice that a checkpoint has to happen when they recognize a special condition in an event. As indicated in [1], as regards what the special condition is, there are two different implementation options: 

1)	an event fragment contains a run number greater than the run number associated to the previous event fragment. While this may seem the simplest implementation, it has the consequence that some TDAQ elements, e.g. an HLT processor, may have to deal, concurrently and for a short time, with multiple runs.
2)	an event fragment contains a special trigger type indicating the last event of the run: the sweeper event (SE), see below. A clean solution with a drawback: the sweeper event must be broadcast to every HLT processor (but the SE can still be handled by a TDAQ element asynchronously with respect to the other TDAQ elements).

Which solution, 1) or 2) above, will be eventually implemented in TDAQ is not in the scope of this note. This note proposes a behavior for some components and the ROD such that either 1) or 2) could be implemented by TDAQ downstream of the ROD. That is to say we propose a mechanism to both insert the new run number into the event and to allow the SE to flow through the system.

The above considerations require that, for any event, the run number is stored in the header of each event fragment, in particular the fragment transmitted by the RODs. A mechanism to tag the run number into the ROD event fragment is suggested below: it essentially consists of  a special trigger type (SE) sent to the ROD, to signal the last event of the previous run; after this event the run number is incremented by one. The SE follows the usual data flow path: the ROD transmits the SE to the ROS.

The run number, itself a 32-bit number, is 
	1) generated by the run control, and
	2) partitioned in two fields: a run type identifier (8 bits) and a sequence number for that identifier (24 bits). Run type identifiers are associated to individual sub-detector partitions, physics run, other special runs.

Sequence of operations

Cold start of  run
During the cold start of a run, the 32-bit run number is generated by the run control (which in particular has the knowledge of what type of run it is) and it is sent to all the SBC controlling the ROD crates. These processors will write this run number in an incrementable register in all the RODs. 

Change in conditions (for example a change in the trigger type pattern)
The Run Control executes the following sequence: 

a)	Block level-1 triggers.
b)	Apply conditions change (for ex. change the trigger type pattern).
c)	Save conditions change to the conditions data base.
d)	Increments the current run number and stores it into a run number data base.
e)	Send a command to the CTP (or the LTP playing the role of the CTP for the current partition) to generate the SE.
f)	Re-enable level-1 triggers.

The CTP/LTP executes the following sequence:

a)	Generate a special event type upon receiving the command to produce a SE (SE). It means it generates a L1A together with the Trigger-Type 0x07 identifying the SE.
b)	LTP sends this L1A together with the associated Trigger-Type to the read-out electronics through the TTC system (usual way of working).
c) 	The CTP issues as many ECR as needed to set the value of the ROD ECR counter to 0 and the value of the L1ID to 0 (256 ECRs take ~ 150 microseconds).

The RODs execute the following sequence:

a)	Upon treatment of the SE (i.e when building the event fragment corresponding to this event) it increments its local run number register by one (the 24-bit part). The SE is tagged by the previous run number.
b) 	 The ROD raises the ROD busy signal: always, even when the ROD does not need to perform any end of run operations (this feature would be used for testing purposes, i.e. to check that the ROD has received and recognized the SE trigger type). The ROD busy  is released either immediately afterwards or when the ROD has completed the end of run operations.
c)	The ROD transmits the fragment to the ROS  following the usual path.
d)	The subsequent event fragments, corresponding to the following L1As, will have the new run number in their event fragment header.

The sequences a) through f) for the run control, a) through b) for the CTP/LTP and a) through d) for the ROD are executed by these (run control, CTP/LTP and ROD) components at every transition between runs: a checkpoint or a normal (i.e. fully synchronous) run transition. In particular the SE is generated at every end of run.

The SE flows throughout the HLT/DAQ system up to permanent storage.

Required modifications

a)	The CTP can receive a command to generate a SE (already existing facility)
b)	The CTP generates as many ECR as are needed to set the ROD ECR counters to 0 (the CTP knows the current value of the ECR counter) and the L1ID to 0.
c)	The LTP does not require any modification
d)	The RODs contain an incrementable 32-bit register accessible through VME
e)	The RODs recognize (i.e decode) the SE type, and increment the run number register upon treatment of this event
f)	The RODs store the run number in the event fragment header for each event
g)	The RODs transmits SE's normally as any other event type.
h)	The L2PU (level-2 processing unit) and EFPU (event filter processing unit) receive a SE and flag it as an accepted event.
i)	The behavior with respect to the SE of the other HLT/DAQ elements is unspecified at the moment. 
j)	The event header [2] is modified at all levels, i.e. whenever a header is generated, from, and including, the ROD downwards, the run number is set into the header.

Sweeper Event Trigger-Type
It is proposed to use the Trigger-Type 0x07 to identify the SE. 

 Run Number structure
The run number is a 32 bit quantity. The 8 highest bits are defined by the run control and identify the run type (256 run types), for example:

	- sub-detector calibration run (1 type per sub-detector)
	- physics run, 1 type
	- combined detectors run (several types)
	- other run types

 The other 24 bits represent the ordered sequence of runs within the type

 24 bits = 16 777 216 runs

 A year of LHC running is 10**7 seconds, that is ~ 16 years of 10-second long runs, per run type.

 References

[1] H. Burchkart et al., ATLAS TDAQ Global Issues Working group, Run and States, ATL-DAQ-2003-024.
[2] C. Bee et al., The raw event format in the ATLAS trigger and DAQ. ATL-DAQ-98-129.