Automatizing the online filter test management for a general-purpose particle detector

Abstract

This paper presents a software environment to automatically configure and run online triggering and dataflow farms for the ATLAS experiment at the Large Hadron Collider (LHC). It provides support for a broad set of users, with distinct knowledge about the online triggering system, ranging from casual testers to final system deployers. This level of automatization improves the overall ATLAS TDAQ work flow for software and hardware tests and speeds-up system modifications and deployment.

Introduction

The Large Hadron Collider (LHC) [1], at CERN, is the world's newest and most powerful tool for particle physics research. It is designed to collide 14 TeV proton beams at an unprecedented luminosity of ${10}^{34}{\text{cm}}^{-2}{\text{s}}^{-1}$. It can also collide heavy (Pb) ions with an energy of 2.8 TeV per nucleon at a peak luminosity of ${10}^{27}{\text{cm}}^{-2}{\text{s}}^{-1}$. During nominal operation, LHC will provide collisions at a 40 MHz rate. To analyze the sub-products resulting from collisions, state-of-art detectors are placed around the LHC collision points.

Among LHC detectors, ATLAS (A Toroidal LHC Apparatus) [2] is the largest. ATLAS comprises multiple, fine-granularity sub-detectors for tracking, calorimetry and muon detection (see Fig. 1). Due to ATLAS high-resolution sub-detectors, $\sim 1.5\text{MB}$ of information is expected per recorded event. Considering the machine clock, a data rate of $\sim 60\text{TB}/\text{s}$ will be produced, mostly consisting of known physics processes. To achieve a sizeable reduction in the recorded event rate, an online triggering system, responsible for retrieving only the most interesting physics channels, is put in place. This system is constructed using a three-level architecture [3]. The first-level trigger performs coarse granularity analysis using custom hardware. The next 2 levels (High Level Triggers – HLT) operate using full detector granularity and use complex data analysis techniques to further reduce the event rate [4].

The HLT software is developed using high-level programming languages and is executed using off-the-shelf workstations running Linux and interconnected through Gigabit Ethernet. Data are provided to HLT by specialized Data Acquisition (DAQ) modules known as Readout Systems (ROS). Both HLT and DAQ are referred to as Trigger and Data Acquisition (TDAQ). The TDAQ comprises thousands of devices, and has been designed to be highly configurable. To provide the run configuration of all TDAQ devices, a special database system was developed [5].

While there is a single production version of the TDAQ running at the ATLAS experiment, the TDAQ software itself can be run on many different testbeds for testing and evolution [6], [7]. Setting device and application parameters for a given TDAQ run is a complex task that requires very specialized knowledge. Misconfigurations may result in TDAQ operation errors, unnecessarily wasting project resources. Additionally, the TDAQ project involves more than 400 collaborators, with different testing interests. There are users who simply want to execute TDAQ with standard parameters, while developers might want to test special cases, requiring more specific adjustments.

To help all TDAQ users and final deployers, a scriptable environment was designed. Such environment was developed based on a multi-layer approach. Users with limited knowledge regarding TDAQ will feel more comfortable working at the highest layer, as it will demand only a few high-level parameters. On the other hand, experts might profit from the lower-level layers, since they allow the configuration of very specific parameters, for customized TDAQ operation.

Although the proposed environment automatizes the TDAQ configuration task, its execution still needs to be manually started, and running parameters (accept rate, number of processing cores running, etc.) observed from the screen. This can be a tedious task, as tests of new software releases or hardware upgrades must be performed on a regular basis for different running setups. Therefore, an automated TDAQ execution environment was also developed. Such an environment is capable of automatically setting up TDAQ, as well as, for a given amount of time, collect monitoring information about its execution. At the end, post-processing analysis can take place, and results (graphics and histograms) be generated for further analysis.

The remaining of this paper is organized as follows. Section 2 details the trigger and data acquisition modules. Section 3 describes the configuration database service used to provide configuration parameters to the TDAQ infrastructure. In Section 4, the developed environment for configuring the TDAQ infrastructure is shown. In sequence, Section 5 shows the environment created for automatically operate the TDAQ system. Conclusions are derived in Section 6.