Authored by O. Tsai, 04/13/2006 (updated 03/13/2007)

You will be operating three detectors:

BTOW	Barrel Electromagnetic Calorimeter
BSMD	Barrel Shower Maximum Detector
BPSD	Barrel Preshower Detector

All three detectors are delicate instruments and require careful and precise operation.

It is critical to consult and follow the “STAR DETECTOR STATES FOR 2007 RUN”
and “Detector Readiness Checklist”  for instructions.

Rule 1: If you have a concern of what you are going to do with any of these detectors please don’t hesitate to ask people around you in the control room or call experts to get help or explanations.

This manual will tell you:

1. how to turn On/Off  low and high voltages for  all three detectors.
2. how to prepare BTOW for “Physics”.
3. how to recover from a PMT HV Trip.
4. how to deal with common problems.
   

First, familiarize yourself with the environment of the control room. This is a picture of the four terminal windows which you will be using to operate the BEMC systems.

 

(clicking on a link will take you directly to that section in the manual)

1 - (on emc02.starp.bnl.gov)
BEMC Main Control Window
BEMC PMT Low Voltage Power Supply Slow Control

2 - (on emcsc.starp.bnl.gov)
BTOW HV Control

3 - (on hoosier.starp.bnl.gov)
BSMD HV Control

4 - (on emc01.starp.bnl.gov)
BPSD HV Control
 
To login on any of these computers use the emc account with password (check the control room copy of the manual).

Terminal 1    “BEMC Main Control Window”

Usually this terminal is logged on to emc02.starp.bnl.gov
The list of tasks which you will be doing from this terminal is:

1. Prepare BEMC detectors for Physics.
2. Turning On/Off low voltages on FEEs.
3. Turning On/Off BEMC crates.
4. Resetting Radstone Boards (HDLC).
5. Explain to experts during phone calls what you see on some of the terminals.

The screenshot above shows how the display on emc02.starp.bnl.gov usually looks during the run. There are five windows open all the time. They are:

1. “Barrel EMC Status”  - green.
2. “BEMC MAIN CONTROL” – gray.
3. “BARREL EMC CANBUS” – blue.
4. Terminal on sc3.starp.bnl.gov  (referred to as the ‘sc3 window’)
5. Terminal from telnet scserv 9039 (referred to as ‘HDLC window’)
   

Prepare BEMC detectors for PHYSICS.

In normal operation this is a one click operation.



Click “Prepare for PHYSICS” button and in about 10 minutes “Barrel EMC Status” window will turn green and tell you that you are ready to run.  This window may look a little bit different from year to year depending trigger requirements for BTOW. 

However if this window turns red, then you will be requested to follow the suggested procedures which will popup on this window: simply click on these procedures to perform them. 

During “prepare for physics” you can monitor the messages on the sc3 window. This will tell you what the program is actually doing. For example, when you click  “Prepare for Physics” you will start a multi-step process which includes:

1. Turning OFF FEEs on all SMD/PSD crates
2. Programming TDC (Tower Data Collector, Crate 80)
3. Reprogramming FPGAs on all BTOW crates
4. Configuring all BTOW crates
5. Configuring all SMD/PSD FEEs
6. Loading pedestals on all BTOW crates
7. Loading LUTs on all BTOW crates  (only for pp running)
8. Checking BTOW crate configuration
9. Checking SMD/PSD configuration
10. Checking that pedestals were loaded correctly (optional)
11. Checking that LUTs were loaded correctly (only for pp running)

Usually, you will not be asked to use any other buttons shown on this window. 
 

BEMC MAIN CONTROL

You can initiate all steps outlined above manually from the BEMC MAIN CONTROL window shown below, and do much more with the BEMC system. However, during normal operation you will not be asked to do that, except in cases when an expert on the phone might ask you to open some additional window from this panel and read back some parameters to diagnose a problem. 



You might be asked to:

1. Turn OFF or ON  SMD/PSD FEE
2. Open SMD/PSD panel and read voltages and currents on different SMD/PSD FEEs
3. Open East or West panels to read voltages on some BTOW crates. (Click on Voltages)
   

 That will help experts to diagnose problems you are calling about.

BARREL EMC CANBUS

From this window you can turn Off and On BEMC crates, read parameters of VME crates. This screenshot shows you this window during normal operation with all BEMC crates being ON.

1. Crate 80 TDC  (Tower Data Collector and “Radstone boards”)
2. Thirty crates for BTOW
3. Eight crates for BSMD 
4. Four crates for BPSD
   

The BTOW crates are powered in groups of three or four from a single power supply (PS) units.  The fragment below explains what you see.




Tower crates 0x10, 0x11, 0x12, 0x13 are all powered from a single power supply: PS 16.
Thus, by clicking the On and Off buttons you will be switching On/Off all four crates and the communication with PMT boxes which are associated with them. (see details  in Tower HV Control GUI description).

Sc3 and HDLC windows.

Two other terminal windows on “Terminal 1” are the so-called sc3, and HDLC windows.
These need to be open at all times. To open the sc3 terminal you will need to login as sysuser on sc3.starp.bnl.gov  with password (check the control room copy of the manual).

From this sc3 terminal you run two programs. The first program is emc.tcl. If you need for some reason to restart “BEMC MAIN CONTROL” or “Barrel EMC Status” GUI you need to start emc.tcl: the alias for this is emc.  To kill this program use alias kill_emc.

To open “Barrel EMC Canbus” GUI use alias emc_canbus.
If you need to reboot canbus then:

1. open sc3 window
2. telnet scserv 9040
3. press “Ctrl” and “x” keys
4. wait while canbus reboots (~5 minutes or so)
5. press “Ctrl” and “]” keys
6. quit telnet session
7. close sc3 window
   

To open an HDLC window, first you need to open an sc3 window and then telnet scserv 9039.
To close telnet session you need to press “Ctrl” and “]”, and then quit from telnet.
You may be asked by experts on the phone to reset the radstone boards. This is why you need this window open. There are two radstone boards and to reset them type:
and

Terminal 1. “BEMC PMT Low Voltage Power Supply Slow Control”

There is a change in operation procedures for Run7 for PMT HV Low Voltage power supplies.
There are two low voltage power supply PL512 units which powers PMT bases.
PL512 with IP address 130.199.60.81 powers West side and PL512 with IP address 130.199.60.79 powers East side of the detector. A single power supply feeds thirty PMT boxes.  The GUI for both PL512 should be open all time on one of the workspace on Terminal1.  A screenshot below shows GUI at normal conditions. Both PL512 should be ON all the time, except the case when power cycling of the PMT bases is required.
There are two buttons to turn power On and Off, as usual, wait 30 sec. after turning power supply Off before you will turn it On again. To start GUI click on desktop icon PMT_LV.

Terminal 2.       “BTOW HV Control”

This is typical screen shot of the BTOW HV GUI during “Physics” running.

What is shown on this screen?

The top portion of the screen shows the status of the sixty BTOW PMT boxes.  In this color scheme green means OK, yellow means bad, gray means masked out.  The fragment below explains what the numbers on this screen means.



Each green dot represent a single PMT box.  Label 0x10 tells you that communication to PMT boxes 1 and 2 is done thru BTOW crate 0x10, boxes 3 and 4 thru crate 0x11, etc…
 

Subpanel “Main Control”

The HV on PMTs is usually ON during the Run.  The two buttons on top are for turning the HV On and OFF on all PMT boxes. The most frequently used button on this subpanel is “Re-Apply HV all PMT Boxes” which is usually used to recover after HV trip. Sometimes you will need to “Re-Apply HV current PMT box” if the system does not set the HV on all boxes cleanly.
The scale 0-60 shows you a progress report. The small icons below this scale tells you what PMT box and PMT were addressed or readout. 

Once you recover from a HV trip please pay attention to the small boxes labeled “Box”, “PMT”, and the speed at which the program reads the voltages on the PMTs.  This will tell you which box has “Timeout” problems and which power supply will need to be power cycled.

Subpanel “PMTs In a Selected Box”

This subpanel shows you the status of the PMTs in a given PMT box.

If you want to manually bring a single PMT box to the operational state by clicking on “Re-Apply HV current PMT box” on the Main Control subpanel you will need to specify which Box To Display on the panel first.

Subpanel “Alarm Control”

On the Alarm Control sub-panel the SOUND should be always ON, except for the case when you are recovering from a HV trip and wish to mute this annoying sound.

Shift personnel are asked to report HV trips in the shift log (type of trip, e.g. single box#, with or without timeout, massive trip, etc…)

Please don’t forget to switch this button back to the ON position after recovering from a trip.

The Main Alarm LED will switch color from green to red in case of an alarm.
 

HV trip with Timeout problem.

Typical situation – you hear a sound alarm indicating a HV trip. The auto-recovery program did not bring all PMT boxes to the operational state, e.g. some boxes will be shown in yellow.  First thing to check for the presence of a “Timeout” problem.


Look at the right upper corner of the GUI. If the field below “Timeouts” is blank then try to recover by re-applying  HV to all PMT boxes if the number of bad boxes more then two. If only one or two boxes is yellow then you can try to re-apply HV to the current PMT box. And then read HV all PMT boxes to be sure the system is back to operational mode.

However, if the field below “Timeouts” is filled with numbers (these are PMTs addresses) then you have a Timeout problem.  The procedure to recover is below:

1. notify shift leader about this problem and tell him that it will take at least 10 min. to bring back BEMC for Physics running.
2. Second, try to identify which PMT box has timeouts (usually it will be first box in yellow counting from 1 to 60). If you are not sure which box has the timeout problem then read all pmt boxes by clicking on corresponding button at the “Main Control” subpanel, and observe which box creates the problem. The box with timeout problem will be responding VERY slowly and will be clearly seen on the “Main Control” subpanel. At the same time, PMTs addresses will be appear in “Timeouts” white space to the right of the green control panel. 
3. As soon as you find the box with the timeout problem, click “Cancel” on the “Main Control” subpanel and then click “OFF” – you will need to turn HV OFF on all PMTs.
4. Wait till HV is shut OFF (all PMT Boxes).
5. From Terminal 1, power cycle correspondent PL512 (Off, wait 30 sec., On).
6. Now turn ON HV on PMT boxes from the “Main Control” subpanel. It will take about 2 to 3 minutes first to send desired voltages to all PMTs and then read them back – if the HV is set correctly then…
   

However, if during step (6) you still get a “Timeout” problem then you will need to:

1. Call experts
2. Turn Off HV
3. Power Cycle all PS and PL512
4. Turn On HV
5. If HV set correctly 
6. Do “Prepare for Physics”

Terminal 3.       “BSMD HV Control”

Your login name is emc, your password is __________________________

This screenshot shows how the window on the terminal3 will look when the HV is Off on the BSMD modules. There should be two open windows. One is a LabView GUI and another is a telnet session  SY1527 (CAEN HV Main Frame).

In normal operation it is a one click procedure to turn the HV On or OFF on the BSMD.

There is complete description of the BSMD HV program in a separate folder in this document.

Although, operation is very simple, attention should be made for audio alarms.
Do not mute the ALARM. Shift personnel are asked to report all BSMD trips in the shift log.

Easy Troubles:

BEMC Main Control seems to be frozen, e.g. program doesn’t respond to operators requests.

Probable reason: RadStone cards in a “funny state” and needs to be reset

From Terminal 1 try to:

1. kill_emc
2. soft reboot first “Ctrl” + “x” from HDLC terminal window.
3. start emc and see if problem solved

If the problem is still there then:

1. kill_emc
2. Power Cycle Crate 80
3. start emc and see if problem solved
   

If problem is still there call experts

PL512 Information (Run7 configuration)

There are two PL512 power supplies which provides power to the BEMC PMT boxes.  Both are located in the rack XXXX,  second floor.
The top unit (IP address 130.199.60.79) serves West side of the detector.  The bottom unit (IP address 130.199.60.81) serves East side of the detector.  The connection scheme is shown below

PMT Boxes
                      West Side                                East Side

1-8	9-16	17-22	23-30		32-39	40-47	48-53	54-31
U 0,4,8	1,5,9	3,7,11	2,6,10		0,4,8	1,5,9	3,7,11	2,6,10
U 0,1,2,3	+5 V
U 4,5,6,7	-5 V
U 8,9,10,11	+12 V

Note ,  BEMC PMT Low Voltage Power Supply Slow Control channels enumerated from 1 to 12.

Slow control for PL512 runs on EMC02,  login as emc, alias PMT_LV.
You will need to specify IP address.
Configuration (Experts Only)  password is ____________.
Log files will be created each time you will start PMT_LV in the directory
/home/emc/logs/
for example
/home/emc/logs/0703121259.txt  (March 12, 2007, 12:59)

There is a “bypass” how to operate PL512. Both power supply units connected thru USB to a VIDEO PC on the platform.  It is possible to run PC Anywhere and operate PL512 using standard WIENER software. Be sure that there is only one session is open on VIDEO PC, usually video controls opens on STAR-UTILITIES.star.bnl.gov.  Operation is tricky due to conflict of running two PL512 connected to the same PC, (you will be required to enable/disable USB ports on that PC) live it to experts.

Terminal 4.       “BPSD HV Control”

The BPSD (Barrel PreShower Detector) HV supplies are two Lecroy 1440 HV systems located on the second floor platform, racks 2C5.  Each 1440 is commanded by a 1445A controller which communicates via the telnet server on the second floor of the platform (SCSERV [130.199.60.167]).  The left supply uses port 9041 and the right supply uses port 9043.

On the upper right terminal (#4) there should be two open windows running telnet sessions which control the BPSD (Barrel PreShower Detector) HV.  They should be constantly monitoring the first 15 channels or so in each supply.  It is not necessary to do anything more than verify that these channels are set to -850 volts.

If the displays are not running:

Open 2 terminal windows on emc01 (terminal 4) and ssh to sc3.starp.bnl.gov (sysuser/d*):

In the first  window type:
In the next window type:

Most of the time, typing ‘enter’ a few times should re-establish the connection with the running system.

If connection is established but the system is not running:

Establish both connections as above.  If connection is refused then see below.

Type ‘enter’ several times to connect with the controller.
If the controller responds with a prompt 1>  or 2>

For 1> type:
For 2> type:

Each telnet session should now show the voltages rise steadily to -850 volts.If this is not the case, then type OFF in each window and call one of the Expert Contacts.

If the telnet connection is refused:

If you get a message that the telnet connection is refused, then you must reset ports 9041 and 9042 on the telnet server SCSERV on the second floor.  Here is the procedure:

Open a terminal and log on to:  scserv.starp.bnl.gov [130.199.60.167]  root/(password from Wayne Betts)

->menu  -> connections -> kill connection 

A copy of this manual as a Word document is available for download below.