Всё для Учёбы — студенческий файлообменник
1 монета

Студенческий документ № 052170 из НИЯУ МИФИ

Design of versatile ASIC and protocol tester for CBM readout system

This content has been downloaded from IOPscience. Please scroll down to see the full text.

View the table of contents for this issue, or go to the journal homepage for more

Download details:

IP Address:

This content was downloaded on 09/06/2017 at 06:48 Please note that terms and conditions apply.

You may also be interested in:

GBT based readout in the CBM experiment

J. Lehnert, A.P. Byszuk, D. Emschermann et al.

A versatile small form factor twisted-pair TFC FMC for MTCA AMCs

L. Meder, J. Lebedev and J. Becker

The Versatile Link Demo Board (VLDB)

R. Martin Lesma, F. Alessio, J. Barbosa et al.

System-level considerations for the front-end readout ASIC in the CBM experiment from the power supply perspective

K. Kasinski, P. Koczon, S. Ayet et al.

The high performance readout chain for the DSSC 1Megapixel detector, designed for high throughput during pulsed operation mode

M Kirchgessner, J Soldat, A Kugel et al.

Integration and testing of the DAQ system for the CMS Phase 1 pixel upgrade

B. Akgun The upgraded HADES trigger and data acquisition system

J Michel, M Bohmer, M Kajetanowicz et al.

A compact system for two-dimensional readout of Gas Electron Multiplier detectors

B Mindur, W Dbrowski, T Fiutowski et al.

The ARAGORN front-end-An FPGA based implementation of a Time-to-Digital Converter

M. Buchele, H. Fischer, F. Herrmann et al.

Received: November 13, 2016

Accepted: January 19, 2017

Published: February 16, 2017

Topical Workshop on Electronics for Particle Physics,

26-30 September 2016,

Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany

Design of versatile ASIC and protocol tester for CBM readout system

W.M. Zabolotny,a,1 A.P. Byszuk,a D. Emschermann,b M. Guminski,a B. Juszczyk,a

K. Kasinski,c G. Kasprowicz,a J. Lehnert,b W.F.J. Muller,b K. Pozniak,a R. Romaniuka and

R. Szczygielc aInstitute of Electronic Systems, Warsaw University of Technology,

ul. Nowowiejska 15/19, Warszawa, 00-665 Poland

bGSI-Helmholtzzentrum fur Schwerionenforschung GmbH,

Planckstra?e 1, Darmstadt, 64291 Germany cAGH University of Science and Technology, Department of Measurement and Electronics,

Av. Mickiewicza 30, Cracow, 30-059 Poland

E-mail: wzab@ise.pw.edu.pl

Abstract: Silicon Tracking System (STS), Muon Chamber (MUCH) and Transition Radiation Detector (TRD) subdetectors in the Compressed Baryonic Matter (CBM) detector system at Facility for Antiproton and Ion Research (FAIR) use the same innovative protocol ensuring reliable synchronization of the communication link between the controller and the front-end ASIC, transmission of time-deterministic commands to the ASIC and efficient readout of data. The paper describes the FPGA-based tester platform which can be used both for the verification of the protocol implementation in a front-end ASIC at the design stage, and for testing of the produced ASICs. Due to its modularity, the platform can be easily adapted for different integrated circuits and readout systems.

Keywords: Data acquisition circuits; Detector control systems (detector and experiment monitoring and slow-control systems, architecture, hardware, algorithms, databases); Front-end electronics for detector readout; Digital electronic circuits


1 Introduction 1


1.2 Communication via GBTX 2

1.3 Communication protocol 3

1.4 Motivation for tester design 4

2 Tester design 4

2.1 SMX2 model 4

2.2 DPB+ROB model 5

2.2.1 IPbus controller 5

2.2.2 E-Link black box model 6

2.2.3 SMX2 command transmitter and processor 6

2.2.4 SMX2 receiver 6

2.3 Tester software 7

3 Performed tests and results 7

4 Conclusions 8

1 Introduction

The Compressed Baryonic Matter (CBM) experiment is one of the experiments prepared at the Facility for Antiproton and Ion Research (FAIR ) in Darmstadt. Its main aim is the exploration of the QCD phase diagram in the region of high baryon densities during high-energy nucleus-nucleus collisions [1]. CBM will utilize various particle detectors: Micro Vertex Detector (MVD), Silicon Tracking System (STS), Ring Imaging Cherenkov Detector (RICH), Muon Chamber (MUCH),

Transition Radiation Detector (TRD), Time of Flight Detector (TOF), Projectile Spectator Detector (PSD). The detector-specific front-end boards (FEBs) are used to receive the signals produced by the detector and convert them into the digital form suitable for transmission to the data acquisition (DAQ) system. Because FEBs are usually located in the irradiated area, they are based on dedicated radiation-hard ASICs. To minimize the design and development effort, a common communication protocol (HCTSP) [2] was developed and implemented in readout ASICs for STS, MUCH and TRD [3] detectors.


The STS/MUCH-XYTER2 (SMX2) is designed to read out the double-sided silicon microstrip sensors (STS) and GEM sensors (MUCH). The chip performs self-triggered amplitude and time digitization in 128 channels. The front-end circuit has been optimized for low-noise [4, 5] operation with the average rate of 250kHit/s/channel. For each hit, it provides the 14-bit timestamp (with

Figure 1. Simplified diagram of the SMX2 chip.

3.125ns resolution) and 5-bit signal amplitude data. The digital back-end of the ASIC [6] provides the data path for high-throughput readout of the digitized hits and the control path for configuration and control of the data acquisition and diagnostics features of the chip. The digital information generated in each channel is written to the individual channel FIFO. All FIFOs are read by the hit sorter, and the received data is delivered to the five software maskable serializers and further to differential readout links. The ASIC also implements the multi-level data flow control (throttling) to accommodate possible beam intensity fluctuations. The chip implements multiple diagnostic and monitoring features including reporting of detected SEU events and link-related problems. The internal structure of the chip is shown in figure 1.

1.2 Communication via GBTX

Eight SMX2s chips located on a FEB board are interfaced to the GBTX-based Readout Boards (ROB) [7] via so-called E-Links. The GBTX chips [8]2 provide the synchronous, deterministiclatency, bidirectional3 communication between the FEBs (via E-Links) and the controlling system (Data Processing Boards - DPBs [10, 11]) via a high-speed 4.8Gb/s optical link.4 Each E-Link is a serial, SPI-like synchronous interface, that may be used with configurable clock frequency (40MHz, 80MHz, 160MHz or 320MHz) and bit rate (80Mb/s, 160Mb/s or 320Mb/s) depending on the number of E-Links used in a single GBTX. The configuration for the SMX2 uses 160MHz clock, 160Mb/s downlink-direction data and 320Mb/s uplink-direction data links. Readout of the double-sided sensors in the STS implies the necessity of using AC-coupled electrical links (see figure 2) due to the different ground potentials of FEBs attached to both sides of the sensor biased with high voltage [12] connected to the same ROB. The clock and downlink data signals are shared between the SMX2s located on a single FEB.

Figure 2. The STS detector readout system structure.

1.3 Communication protocol

The main aim of the communication protocol is to maximize the data throughput in the uplink direction (from ASIC to DPB) and to provide time-deterministic reliable transfer of the control commands in the downlink (from DPB to ASIC) direction.

The E-Link alone provides only a simple synchronous serial port providing the clock line, data inputs, and data outputs. It does not provide any means for byte or word synchronization, which must be handled by the protocol. Moreover, DC-balancing of the bitstream is required by the AC-coupled E-Links. Application of 8b/10b encoding solves both mentioned problems.

The link between the GBTX chip and the SMX2 introduces constant but unknown delays in clock line, data output line, and data input lines, that may differ for each target ASIC. These delays must be compensated using the phase-adjusted GBTX clock and adjustable delays on GBTX data inputs. Thepropersettingsarefoundduringthesynchronizationprocedure[2], controlledbyspecial DC-balanced 20-bit long sequences (Start-of-synchronization - SOS, and End-of-synchronization - EOS), which are easily distinguishable from 8b/10b encoded data and detectable even before the link is synchronized. The applied solution allows avoiding programmable delay lines in the SMX2, which results in simpler and more reliable design.

To ensure constant latency after the synchronization, the protocol uses constant-length frames (6-bytes or 60-bits after encoding in the downlink direction and 3 bytes or 30-bits after encoding in the uplink direction). The downlink frames encode operations related to access to the SMX2 registers located in the 15-bit address space. The downlink frames start with a comma character and are protected with a 15-bit CRC. Different types of uplink frames contain the hit data (or empty hit) with least significant bits of the time-stamp, the most significant bits of the time-stamp, the value read from the register, or the confirmation of the register write operation. The control-related uplink frames are protected with short (3-bit or 4-bit) CRC. Additionally, after each 216 frames, the SMX2 injects a special synchronization frame consisting of three K28.5 comma characters5 into the uplink stream. Those solutions allow keeping the link synchronized in the case of rare frame corruptions due to single-event upsets and provide detection of loss of synchronization.

The commands and responses related to access the registers are additionally labeled with the 4-bit sequence number. That allows retransmitting commands, which have not been acknowledged at the proper time.

The important feature of the protocol is that it allows reliable synchronization and resynchronization of the communication link between the readout controller and the SMX2 without clearing the internal state of the chip. That facilitates debugging of the control and readout system.

The resulting Hit and Control Transfer Synchronous Protocol (STS-HCTSP) is described in [2]. ForSMX2theachievablespeedofregisteraccessis2.6Mframes/sandspeedofhitdatatransmission is from 9.41MHit/s (for 1 uplink) to 47.05MHit/s (for 5 uplinks).

1.4 Motivation for tester design

Due to the innovative design of the STS-HCTSP protocol implemented in the SMX2, it was necessary to verify operation of its control part with the proposed communication protocol before ASIC tape-out. Simulations alone were not sufficient. Simulation including both analog effects and full synchronization procedure and data transmission would be too complex and time-consuming. Therefore it was necessary to create the hardware tester able to emulate the digital part of the SMX2 (SMX2 model) and the DPB board together with the ROB board (DPB+ROB model, see section 2.2).

2 Tester design The chosen hardware platform for the tester was the AFCK board [13], which is the development and prototyping platform for the CBM readout DPB boards [11].

To allow connection of long AC-coupled differential links, a dedicated multichannel bidirectional FMC/VHDCI adapter board was developed [14]. It is possible to test different link technologies and cabling solutions by simple replacement of the FMC boards.

The tester firmware implementing both the SMX2 model and the DPB+ROB model enables two testing configurations. The minimal, single-board setup allows performing functional tests of communication between the SMX2 model and the DPB+ROB model. However, the results may be affected by possible undesired couplings between both models.6 Therefore, the two-boards configuration may be used to ensure that, for example, the link synchronization is not affected by such couplings. When separate power supplies are used, it is also possible to test the link operation with controlled common mode voltage shift between the boards. Both configurations are shown in figure 3.

2.1 SMX2 model

The SystemVerilog RTL code of the SMX2 digital back-end developed at AGH University was used as the starting point. However, due to inherent differences between the FPGA and ASIC technologies, that code had to be slightly modified to enable implementation of the SMX2 model in an FPGA chip.

Figure 3. Two basic configurations for SMX2 protocol tester. The single board configuration (a) provides basic setup for testing of communication between the DPB+ROB and SMX2. The two board configuration (b) eliminates possible parasitic couplings, ensuring that the boards communicate only via the AC coupled LVDS link.

Figure 4. Block diagram of the firmware implemented in the tester AFCK. The design implements both the SMX2 model and the DPB+ROB model.

InASICthedividedclockmaybegeneratedinasequential, flip-flopbasedfrequencydivider. In FPGA such design results in high clock jitter and difficult to eliminate hold-time errors. The optimal solution should use the MMCM or PLL blocks [15] to generate divided clocks. Unfortunately, the phase adjusted clock delivered to the SMX2 model may be discontinuous (as described in section 2.2.2, it is produced by the DRP controlled PLLE2 blocks, where the output clock is switched off during the reconfiguration). Therefore the divided clocks have been emulated using the high-speed clock received from the link, and clock enable signals generated in sequential logic. Such solution works correctly even with the discontinuous clock.

The DDR outputs in the ASIC code may be handled by an HDL-described logic. In the FPGA they must be implemented with the dedicated ODDR blocks [16].

Special measures were taken to ensure coherency of FPGA and ASIC implementations.

2.2 DPB+ROB model

The block diagram of the tester firmware with the structure of the DPB+ROB model is presented in figure 4. The components of the model are described in the following sections.

2.2.1 IPbus controller

The DPB+ROB model is controlled via Ethernet, using the IPbus protocol [17]. The IPbus controller allows accessing the registers in the SMX2 command processor (SMX2 CMD Processor) and reading the SMX2 readout FIFO (SMX2 Readout FIFO). The dedicated IPbus controller block is built using the standard IPbus sources with small modifications [18, 19].

2.2.2 E-Link black box model

In the final system, the communication with the SMX2 will be provided by the GBTX chip, located on the ROB board and connected via an optical link to the DPB (see figure 2). In the tester, the DPB+ROB model directly accesses the LVDS lines connected to the SMX2 model. Therefore, the whole part of the system consisting of the E-Link block in the GBTX, the GBTX itself, the optical link and the optical link controller in the DPB is replaced with an "E-Link black-box model". That model behaves simply as an I/O block with one delay-controlled clock output, one data output, and five delay-controlled data inputs. The data inputs may be individually masked to simulate the situation where the particular SMX2 is connected by a smaller number of uplink connections [7] or to emulate a faulty data link. The delay-control functions, which in the real system will be provided by the GBTX ASIC, must be emulated in the tester. The Kintex 7 FPGAs offer the adjustable output delay blocks ODELAYE2 [16], but it is not available in the High Range (HR) ports, connected to the FMC connectors in the AFCK board. Therefore, the adjustable output clock delay was implemented using the DRP controlled PLLE2 [15] block. The phase shift may be adjusted with the resolution of 125ps, which is worse than the resolution offered by the GBTX (50ps) but is sufficient to test the link synchronization.

The adjustable data delays have been implemented using the IDELAYE2 blocks [16], that provide the delay resolution of 78ps, which is less, than the resolution of the GBTX input data phase-aligners, equal to 391ps for 320Mb/s data rate.

2.2.3 SMX2 command transmitter and processor

The60-bitSMX2controlframesaredeliveredbytheSMX2commandtransmitter(SMX2CMDTx). Depending on the selected operation mode, the transmitter may produce either the standard 8b/10b encoded, 6-byte sequence describing the register access command or the "no operation command", or 6 K28.1 synchronization characters, or three 20-bit special synchronization sequences - SOS or EOS (see section 1.3). In the case of register access commands, the SMX2 CMD Processor ensures their reliable delivery. It offers 5 "command slots" that can be written (via IPbus) with the command to be transmitted. The slots are scanned periodically, and each scheduled command is retransmitted until its confirmation is delivered by the SMX2 receiver block.

2.2.4 SMX2 receiver

Data delivered via the E-Links are analyzed by the SMX2 receiver (SMX2 Rx). During the normal operation, the acknowledgments and responses to the register access commands are extracted and delivered to the SMX2 CMD Processor. That allows marking the register write command as executed. In a case of the register read command, it delivers the read value (also confirming that the command was executed). Other frames are written to the SMX2 receive FIFO (SMX2 Readout FIFO), from where they can be received by IPbus. To ensure high throughput of data reception, IPbus block read command is used to read the FIFO [19]. However, the IPbus protocol does not support a method to end the block access earlier in the case if no more data are available. Therefore, the information about the FIFO status is encoded in the data. The received frames occupy only

Figure 5. Test setup for single-board [11] (on the left), and for two-boards (on the right) configurations.

lower 24 bits. The 31st bit is used to inform if the data was available, and the 30th bit informs, that some data were lost due to readout FIFO overflow.

The SMX2 Rx also contains special blocks detecting the K28.1 and K28.5 comma characters and EOS special frame in the continuous stream of bits, and SOS special frame even in a nonsynchronized link. Those blocks are used to perform the link synchronization procedure.

2.3 Tester software

The SMX2 tester software was written in Python, to enable fast prototyping and possibility of interactive work. Use of Python allows easy modification of the scripts by the user. It is also possible to rewrite time critical procedures in C or C++ after the algorithm is tested.

The developed software implements the object interface to IPbus-accessible registers and a class for writing and reading the SMX2 registers. Two link synchronization procedures are implemented - one for the "full synchronization", assuming that no correct time/phase relations between outgoing data and clock, and incoming data are known; and the second one for the "fast synchronization" which reuses the information acquired during the "full synchronization". The user can easily select which type of synchronization should be performed. The test software also allows defining which uplinks (from SMX2 to DPB) links are operational. That functionality was used to test if the implemented synchronization procedure is able to ensure proper communication as long, as at least one of the uplinks is working. The software also receives simulated hit data, and stores them in the file.

3 Performed tests and results

The tests were performed both in the single-board configuration and in the two-boards configuration, using the 3.75 meters long VHDCI cable (see figure 5). The same firmware was used in both cases, however in the two-boards configuration only the SMX2 model was connected in the 1st board, and only the ROB+DPB model was connected in the 2nd board. The tests included thorough testing of the link synchronization procedure. Both "full" and "standard" synchronization procedures were tested.

The initial tests revealed problems related to the recovery from the situation when all connected uplinks were accidentally masked due to SEU or software error. The problem was eliminated by the automaticunmaskingofallthelinkswhenSMX2enterssynchronizationstateafterreceptionofSOS orEOScommand. Afterthatchange, bothsynchronizationproceduresandrecoveryfrompotentially disastrous situations (including the described one) were successfully tested. Results of successive tests were used to improve the protocol specification [2] and related controllers implementations. After the link synchronization, access to SMX2 registers and reception of simulated hit data were also successfully tested. Successful access to internal registers allowed configuration of the internal test data generator and testing of the data reception (however the hit rate had to be limited due to low IPbus bandwidth).

Results obtained in both single-board, and two-boards configurations were the same. The tests were performed at nominal transfer speed.

4 Conclusions

ThecreatedtesterisausefultoolforearlyverificationofthedesignofdigitalpartofSMX2including the communication protocol. The system helped to identify and eliminate problems present in the initial versions of the protocol and ASIC back-end designs. The tests of the final versions performed at the nominal speed of the SMX2 links (160Mb/s downlink and 320Mb/s uplink) have proven correct operation of the digital part of the ASIC and the communication protocol.

The presented platform will be used to create a test system for the fabricated SMX2 ASICs.7

The DPB+ROB model may be used to test also other FEE systems using the GBTX based E-Links. Both firmware and software will also be the basis for the development of components of the CBM readout chain.

The created platform and associated methodology may also be reused in preproduction verification of other ASIC-based control and communication systems.


Junfeng Yang from GSI significantly contributed to debugging, extending and packaging the IPbus controller [20]. Work was supported by GSI. The AGH work was supported by Polish Ministry of Science and Education.

References [1] FAIR, CBM - The Compressed Baryonic Matter experiment, (2017) http://www.fair-center.eu/for-users/experiments/cbm.html.

[2] K. Kasinski, R. Szczygiel, W.M. Zabolotny, J. Lehnert, C.J. Schmidt and W.F.J. Muller, A protocol for hit and control synchronous transfer for the front-end electronics at the CBM experiment, Nucl.

Instrum. Meth. A 835 (2016) 66.

[3] P. Fischer and M. Krieger, Status of SPADIC development, talk given at the 27th CBM Collaboration Meeting, GSI, Darmstadt, Germany, 11-15 April 2016 http://spadic.uni-hd.de/publications/talks/2016/2016-04-12_spadic_cm.pdf.

[4] K. Kasinski and R. Kleczek, A flexible, low-noise charge-sensitive amplifier for particle tracking application, in proceedings of the 2016 MIXDES - 23rd International Conference Mixed Design of

Integrated Circuits and Systems, Lodz, Poland, 23-25 June 2016, pp. 124-129 [doi:10.1109/MIXDES.2016.7529715].

[5] K. Kasinski, R. Kleczek and R. Szczygiel, Front-end readout electronics considerations for Silicon Tracking System and Muon Chamber, 2016 JINST 11 C02024.

[6] K. Kasinski, R. Szczygiel and W.M. Zabolotny, Back-end and interface implementation of the STS-XYTER2 prototype ASIC for the CBM experiment, 2016 JINST 11 C11018.

[7] J. Lehnert et al., The GBT-based readout concept for the silicon tracking system of the CBM experiment, Proc. SPIE 9662 (2015) 96622S.

[8] P.R.S. Moreira, The radiation hard GBTX link interface chip, (2013) https://indico.cern.ch/event/267408/.

[9] J. Lehnert, A.P. Byszuk, D. Emschermann, K. Kasinski, W.F.J. Muller, C.J. Schmidt, R. Szczygiel and W.M. Zabolotny, GBT Based Readout in the CBM Experiment, 2017 JINST 12 C02061, in proceedings of Topical Workshop on Electronics for Particle Physics, Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany, 26-30 September 2016.

[10] W.M. Zabolotny and G. Kasprowicz, Data processing boards design for CBM experiment, Proc. SPIE 9290 (2014) 929023.

[11] W.M. Zabolotny et al., Versatile prototyping platform for Data Processing Boards for CBM experiment, 2016 JINST 11 C02031.

[12] C.J. Schmidt and P. Koczon, Low and high voltage powering concept for the CBM Silicon Tracking System, GSI-SR2014-MU-NQM-CBM-48 (2015) [doi:10.15120/GR-2015-1-MU-NQM-CBM-48].

[13] F. Switakowski, AMC FMC carrier kintex (AFCK), (2015) http://www.ohwr.org/projects/afck.

[14] G. Kasprowicz, FMC DIO 32ch LVDS, (2016) http://www.ohwr.org/projects/fmc-dio-32chlvdsa/wiki.

[15] Xilinx, 7 Series FPGAs Clocking Resources. User Guide. UG472 (v1.12), (2016) http://www.xilinx.com/support/documentation/user_guides/ug472_7Series_Clocking.pdf.

[16] Xilinx, 7 Series FPGAs SelectIO Resources. User Guide. UG471 (v1.8), (2016) http://www.xilinx.com/support/documentation/user_guides/ug471_7Series_SelectIO.pdf.

[17] R. Frazier, G. Iles, D. Newbold and A. Rose, Software and firmware for controlling CMS trigger and readout hardware via gigabit ethernet, Phys. Procedia 37 (2012) 1892.

[18] W.M. Zabolotny, Free Ethernet MAC for IPbus, (2015) https://svnweb.cern.ch/trac/cactus/ticket/897.

[19] W.M. Zabolotny, IPbus slave for reading FIFOs, (2016) https://svnweb.cern.ch/trac/cactus/ticket/1683.

[20] P.-A. Loizeau et al., The prototype readout chain for CBM using the AFCK board and its software components, Proc. SPIE 9662 (2015) 96622X.

1 Corresponding author.

c 2017 IOP Publishing Ltd and Sissa Medialab srl doi:10.1088/1748-0221/12/02/C02060

2 The usage of GBTX chips in the CBM control and readout systems is exhaustively described in [9].

3 The ROB board will be equipped with 3 GBTX chips. One of them will be used for bidirectional communication.

Two others will be used only for unidirectional uplink transmission.

4 Due to the protocol overhead, the effective user bandwidth is between 3.2Gb/s and 4.48Gb/s, depending on the mode used.

5 K28.5 and K28.1 are special comma characters used in 8b/10b encoding.

6 Such couplings may result from the optimizations introduced by the synthesis tools in the FPGA or may be a result of couplings between the tracks on the AFCK PCB.

7 In October 2016 first tests with the tester connected to the first manufactured SMX2 ASICs have been performed. The synchronization procedures (including recovery from non-functional link state), control (R/W) accesses to the internal registers, and readout of generated hits via the readout fifo were successfully verified.





- 2 -

- 2 -

Показать полностью… https://vk.com/doc98292923_446212756
2 Мб, 9 июня 2017 в 9:05 - Россия, Москва, НИЯУ МИФИ, 2017 г., pdf
Рекомендуемые документы в приложении