We have been investigating the data corruption problems seen in ETOW and ESMD this year. These are the same as seen last year, see 2003. The errors are first of all at DAQ, either missing or corrupted data, then if main header is ok we find nx256, corrupted tokens and crate IDs and nx256 (barrel calls them ladder events). Up to 30% of our data has been corrupted at times.
Recently Tonko took a run driving our TCD with software rather than tirgger hardware. He took the data at 2kHz. Below are two of Hal's error spectra for ETOW for this run and a normal data run. It is clear that all of the mismatched tokens and crate IDs as well as nx256 go away in this specially triggered run.
nx256 rate for software triggered run. No errors in 2500 events!
From run 5022003.
From a regular run (50190022) we see 101 out of 1000 errors
Now Hal's error spectrum for the same software triggered run.
and the same error spectrum for the normal run. Channels 4-7 correspond
to errors in FEE crate 1. ch 4 is crate ID and token corrupted at same
time. ch 5 would be the crate ID is not as expected (sequential 1-6)
and this error by itself. ch 6 is the token not matching the expected
for this crate header. ch 7 is empty. Then this pattern repeats
for the other 5 crates. From ch 30 starts a histogram of the number
of crate ID errors in a single event. ch 36 => all 6 crates have error
at same time. 107 events have errors in the crate ID for this spectrum.
Date: Tue, 27 Jan 2004 15:20:00 -0500
I took a series of successive runs under various conditions as Gerard
requested. I have now processed these for ETOW and ESMD errors using
the QA plots.
The 1st was a standard pedestal run for barrel and endcap. pedasphys ETOW ESMD BTOW and BSMD all in. Rate was 350 Hz determined by BSMD dead time. Sorry forgot to write down error rates from run monitor screen. R# 5027064. 25kevts A run like this is taken between every fill.
I find no crate ID errors in any of these runs. All errors for ETOW and ESMD are in the tokens.
18% nx256 in the towers, and 13.8% tower token errors, no ESMD token errors.
crate 2 spectra
Next I removed BSMD and got a rate of 1kHz. Again I forgot to write down errors on screen. R# 5027065 25kevts
8% nx256 in the towers, and 3.7% tower token errors, and 23.5% ESMD token errors.
crate 2 spectra
crate 92 (box 7S1) spectra
All rest of runs are BTOW ETOW and ESMD
Next I used the production min bias configuration. w/o beam this gives
4 Hz of pulser triggered data. (Tonko suggested this)
R# 5027067 had some beam in. Don't bother analyzing
r# 5027068 4 Hz 1kevts Again I forgot to write down the daq error
from the screen. But from all the above runs they did not seem to
vary much but were there at usual levels.
15% nx256 in the towers, and 6.4% tower token errors, and 0% ESMD token errors.
crate 2 spectra
crate 92 (box 7S1) spectra
Next I took pedasphys with 1 token Event rate was variable from 1-40 Hz
.
errors spurious 4kevts r# 5027071
ETOW .3% .2%
ESMD 1.7% .8%
BTOW 0% .1%
Note spurious = unexpected data was sent to daq
error = expected data did not arrive or is corrupted in event header.
crate 2 spectra
crate 92 (box 7S1) spectra
Repeat pedasphys with 112 tokens R# 5027072 same as 5027065
25kevts at 1kHz
ETOW .6% .1%
ESMD .4% .1%
BTOW 0% 0%
13% nx256 in the towers, and 7.4% tower token errors, and 24% ESMD token errors.
crate 2 spectra
crate 92 (box 7S1) spectra
Taking the software triggered TCD events is easy. Tonko showed me the trick. Take L1CTL out of the run. (this is level 1 control). I used pedasphys config. Then start the run. When the acknowledge box opens the 1st line (# of events) is irrelevant. The second line is the rate you want to trigger at. (normally enter ZDC coinc. rate here) Start the run. Nothing will happen. Click on issue triggers. You will be asked if you intend to run the softwared trigger. Answer yes. Next you will be asked how many events. Enter your desired number and then watch them come in. When it stops click stop run.
I took 3 runs in this mode
5027073 10 Hz 2.5kevts
5027074 1000 Hz selected but it ran at 700Hz 25kevts
5027075 300 Hz 25 kevts
None of these had any errors whatsoever on the run status page.
Curiously they had a small number of nx256 errors in the MAPMTs but no
other errors in spectra. From the spectra this appears to be chance
hits of the noise in that region on ch 256.
Run 73
crate 2 spectra
crate 92 (box 7S1) spectra
Run 74
crate 2 spectra
crate 92 (box 7S1) spectra
Run 75
crate 2 spectra
crate 92 (box 7S1) spectra
5027076 was a standard ped as phys run with BTOW ETOW and ESMD. I took it to make sure I left things back to normal. 12% nx256 in the towers, and 7.3% tower token errors, and 23% ESMD token errors.
crate 2 spectra
crate 92 (box 7S1) spectra
Here is a summary table of the statistics from the above.
| ETOW | ETOW | ESMD | ESMD | BTOW | BTOW | |||||||||||
| Run | evts | token errs | crate errs | nx256 | MAPMT errs | MA n256 | % Tn256 | %tkn/crate | %MAPM/box | DAQ err | DAQ spur | DAQ err | DAQ spur | DAQ err | DAQ spur | |
| 64 | Peds 4dets | 4125 | 2838 | 0 | 741 | 0 | 1 | 18.0% | 13.8% | 0.0% | ||||||
| 65 | Ped no BSMD | 6202 | 1151 | 0 | 514 | 7292 | 0 | 8.3% | 3.7% | 23.5% | ||||||
| 68 | 4Hz pulser minb | 259 | 83 | 0 | 38 | 0 | 0 | 14.7% | 6.4% | 0.0% | ||||||
| 71 | ped 1 token | 987 | 823 | 0 | 147 | 0 | 0 | 14.9% | 16.7% | 0.0% | 0.3% | 0.2% | 1.7% | 0.8% | 0.0% | 0.1% |
| 72 | Ped no BSMD | 6170 | 2277 | 0 | 827 | 7563 | 1 | 13.4% | 7.4% | 24.5% | 0.6% | 0.1% | 0.4% | 0.1% | 0.0% | 0.0% |
| 73 | soft trg 10Hz | 625 | 0 | 0 | 0 | 0 | 6 | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| 74 | soft trg 700Hz | 6250 | 0 | 0 | 0 | 0 | 33 | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| 75 | soft trg 300 Hz | 6250 | 0 | 0 | 2 | 0 | 43 | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| 76 | Ped no BSMD | 6174 | 2268 | 0 | 750 | 6986 | 39 | 12.1% | 7.3% | 22.6% | ||||||