For TOF there are 12 IOC's in total, 11 for the PL512 LV power supplies (1 for each PL512) and then 1 for the CAEN HV mainframe.
First, the main TOF window
the indicator colors might be different depending upon the current run state.
Second the light weight LV interface
If one or both interfaces are not present it will be explained shortly how to launch them.
There should also be the alarm handler, which has two components. The first is a small window (referred to as the runtime window) with a single rectangular button labeled TOF_ALARMS
It usually is in the top left corner of the display. If you don't see it and have checked all of the desktops then see the section on starting the Alarm Handler.
The second component to the alarm handler is the main window which lists all of the TOF alarms
You can launch this window by clicking on the single button in the runtime window of the alarm handler. From this main window you can also launch GUI interfaces and acknowledge alarms, more on that in a moment.
Alarm Handler
From the above example one can see there is a tree structure to the alarms. At the top is the TOF_ALARMS group which contains a HV and LV group (the MTD group is not covered in this manual). The HV group has a further sub-structure to have alarms associated with the power supply Boards and then individual Sectors (i.e. channels). As for the LV group, there is a THUB group and then again a group for each sector. If one digs deeply enough in this tree structure one eventually reaches the individual alarms.
To the left of each group is a button which can be used to acknowledge alarms - currently there aren't any alarms hence the empty state of the button.
On the right, there is a triangle to expand or collapse the tree structure.
For certain groups there also is a P button after this triangle, it launches an associated process which is relevant to this particular group of alarms - for TOF this means launching a GUI interface. Click on the button labeled P which is to the right of the TOF_ALARMS icon to launch the main TOF window. Clicking on an identical button by the LV will launch the light weight LV GUI.
In the angular brackets <> the current mask for the group is shown. There is a brief explanation at the bottom of the main window in the message area. I made it such that the alarms are disabled for channels which are off, hence the D mask element. The T refers to not having to acknowledge transient alarms. These are alarms which return to a non-alarm state before the operator can acknowledge them. If the T is present it means the alarm will automatically silence itself, else it would continue beeping until it is acknowledged.
As the name implies the purpose is to make the detector operator aware of any problems with the TOF HV or LV system. This naturally means it should always be running unless there is a TOF expert in the control room who turned it off.
There are several types of alarms each having a different color associated with them
| Type | Color | Runtime icon |
|---|---|---|
| Major | Red | 
|
| Minor | Yellow | 
|
| Error/Invalid | White | 
|
As seen above the runtime window button will change in color depending upon the highest alarm present (invalid/error, major and then minor). The button will also blink and an alarm klaxon (condition one :) will sound if there is an unacknowledged alarm.
To acknowledge (i.e. silence) alarms you will need to have the main alarm handler window open (click on the runtime window if need be).
In the above example all of the HV sectors have a red alarm present and also unacknowledged - the button to the left of each group which has an alarm is now labeled red and is also blinking. To acknowledge all of the alarms press the button to the left of the TOF_ALARMS icon. You can also acknowledge individual alarms by clicking the associated acknowledge button for the particular alarm.
Once you have acknowledged the alarms the klaxon and blinking will stop until the next change in an alarm state. However, as long as the alarms are still relevant the warning icon will remain next to each active alarm.
To the right of each alarm group there is now a row of numbers. The meaning is shown in the bottom message area, from left to right the numbers indicate: error alarm, invalid alarm, major, minor and no alarm count (i.e. in descending order of severity). In the above example the HV group has 60 major alarm counts. Digging into the Sectors group we see each sector has 6 major alarm counts. Choosing a particular sector (East_1 in this case) we have reached the actual alarms and see what each alarm is for. This information is relevant for the TOF expert to diagnose the problem and see if anything can be done about it.
In the bottom right there is a SilenceOneHour button which will mute all red and yellow alarms (error/invalid alarms are still audible), the screen will turn pink to remind you of the muted alarm state
TOF is a new system which is being used in this run for the first time, so we don't know what alarm conditions normally arise - there are a few which are listed in the trouble shooting section. During ramp up and ramp down of the HV the alarms will go off, this is normal and can't be avoided. Also, there are still firmware updates being done to the TOF trays which requires changing the LV which also causes alarms - ideally the shift leader is to be made aware of this.
Outside of these scenarios the detector operators will need to contact an expert each time an alarm is present. This will help in creating a list of troubleshooting steps and potentially lessen the number of insignificant alarms (i.e. caused by too tight alarm ranges). Feel free to contact me directly at bbiritz@ucla.edu, comments, complaints and suggestions are welcome.
Regular Operations
Under normal conditions the only operations the detector operator needs to perform are:
Naturally this is superseded by any instructions a TOF expert has given the shift leader.
HV
For normal operations and unless you are told otherwise, in order to bring the high voltage to operating voltage you just need to press the Full HV/Standby switch. This will bring up the voltage on all of the HV sectors from 5000 to 7000V.
It will take about 4 minutes for the operating voltage of 7000 V to be reached. During this process the sectors will go from brown to red to yellow to finally green -
To lower the voltage to standby level use again the "Full HV/Standby" button, it takes one minute to ramp down to 5000 V from the operating voltage.
If you run into any of the problems listed below the expert should be contacted - doubly so for ones which aren't listed! I have compiled these trouble shooting items for a couple of reasons:
then one or more of EPICS IOC's is not running. To resolve this do the following
See which IOC's are running
To check on this launch a terminal and type: runningIOCs.sh
As is shown above, there should be 5 East#IOC.cmd (1 through 5), a HVIOC.cmd, one THUBIOC.cmd and then 5 West#IOC.cmd (1 through 5) - since you are reading this one or more of them must be missing.
Starting the HV Input/Output Controller (IOC)
If the HV IOC is not running then open a terminal and type TOFHV_IOC
Give the alarm handler and also the main TOF GUI a moment to reestablish connection with the IOC. If there still are white error or invalid alarms or there still are white indicators then see the HV reads 0 or HV On/Off button missing item.
Starting the LV Input/Output Controller (IOC)
If none of the LV IOC's are running then there is a single command to launch all of them at once, TOFLV_IOC
In the case of at least one LV IOC running you will need to manually start each missing LV IOC, there is a command for each IOC (ignore the startx command)
For example, to launch the IOC for West sector 1, type in a terminal startWest1IOC.
It is due to the IOC having lost and then regained connection with the HV mainframe. To correct this situation you will first need to launch a terminal and type screenHVIOC
after you have pressed return you will be inside of a screen session containing the HV IOC. You should see at least one "lost connection & regained connection" message
To shut down the IOC type exit and press return
This is how the terminal should look after exiting
Next check to see if there are any orphaned semaphores by typing ipcs at the prompt
If there are, you will need to clear them one at a time by typing ipcrm -s and the particular number as shown above.
Once any semaphores are cleared you can relaunch the HV IOC using the usual TOFHV_IOC command. Use screenHVIOC to once more check on the status of the IOC and make sure it hasn't lost connection again - if so repeat the above steps.
To disconnect from the HV IOC you will need to press Ctrl+a, Ctrl+d, which will give you a "detached" message - else you just shut down the IOC once again!
If there are two or more versions of the same IOC running then a TOF expert needs to be contacted for them to safely deal with the situation.
There can be additional remote alarm handlers running (missing the -T option, as seen above), however if the output doesn't contain a line with a -T then you will need to launch the alarm handler by typing TOF_Alarm in a terminal - do not close this terminal window
