Preliminary Version
FTPC procedures for Y2007

This procedure manual is updated whenever the procedures change. Consult the "FTPC detector operator home page" for the very latest information. (http://orion.star.bnl.gov/public/ftpc/operator.html)

Date of last update: March 27, 2007

Janet Seyboth - MPI Terry Tarnowsky - Purdue

PLEASE READ BEFORE OPERATING THE FTPC.

FTPC procedures for Y2006 html
FTPC procedures for Y2005 html
FTPC procedures for Y2005 in MS-Word format for printing DOC
FTPC procedures for Y2004 html
FTPC procedures for Y2003 html
FTPC procedures for Y2001 html

To Access the 2007 FTPC Electronic ShiftLog:

Go to FTPC Detector Operator homepage.
Select link for "Run Operations".
Under "Shift and Run Records" select "Electronic ShiftLog 2007".
Select "Start the editor".
Choose "FTPC" from the list of subsystems.
Make ShiftLog entry.

General introduction

The STAR FTPC (Forward Time Projection Chamber) is a cylindrical TPC with radial drift using an Ar-CO₂ 50%-50% gas mixture. Oxygen content in the gas has to be kept strictly below 10 ppm to avoid a significant absorption of the charge of the electron clouds during their drift to the anodes.

The drift field is generated by the HV electrode (cathode) at the inner radius of the cylinder and terminated by the field cages at the cylinder's ends.

A potential of -10 kV is applied to the cathode and distributed to the field cages in order to generate such a field.

The signals are generated by the collection of the charge induced by the drifting electron clouds, formed by the passage of charged particles in the gas, at the anodes located at the outer radius of the cylinder.

The amplification stage is accomplished by sense wires in proximity to the anodes, kept at a typical voltage of +1800V.

A gating grid is also placed in proximity to the anodes, to avoid charge accumulation in the drift region. The typical potential of the open gate is -76V.

Two such detectors are installed in STAR, covering the -4 < η < -2.5 (east) and 2.5 < η < 4 (west) pseudorapidity windows.

In summary, the two detectors are operated by the following apparatus:

- gas system (Ar-CO₂ 50%-50%, O₂ < 10 ppm);

- cooling system (to cool down the electronics)

- field cage (cathode) power supply (-10kV, 1 channel/ detector, Heinzinger);

- anode sense wires (anodes) power supply (+1800V, 6 channels/detector, Le Croy);

- gating grid power supply (-76V open, 4 channels/ detector, UCLA);

- electronic power supplies (10 units/detector, LBL).

The STAR-FTPC detector can be fully operated by remote control at the FTPC console (cassini.starp) in the STAR control room. The exception is the gas system located in the gas mixing room.

The FTPC is based on the EPICS infrastructure, as the other STAR control systems. It is a complex and delicate system, interfacing with many different hardware parts. Sometime it may seem slow, but the emphasis had to be put on safety of operation. The obvious general recommendation is therefore to be patient and mindful when operating the control system.

Once you are at the FTPC console you should see the FTPC control and monitor systems main page (main control window):

Figure 1: The Main Control window.

If this image is not on the screen you should verify that you have an open window on the slow control state machine (sc3.starp.bnl.gov). The sc3 username (sysuser) and password should be known by your shift leader. If not, contact a member of the FTPC or the Slow Control group. To start the FTPC control and monitor system just type "ftpc" on the sc3.starp.bnl.gov window.

If the "main control window" still does not appear, somebody else is using it. Please, check who else is working on the FTPC Slow Control. You can always force the window by deleting the FTPC lock-file from the sc3.starp.bnl.gov window:

cd epics/R3.12.2-LBL.4/FTPCtopApp

rm /star/sc/users/sysuser/epics/R3.12.2-LBL.4/FTPCtopApp/ftpc_lock

Preliminary checks

At the beginning of each shift the detector operator should check the FTPC operator home page (STAR home -> Experiment -> Subsystems -> FTPC -> Detector operator home page) and should go through a series of checks (a checklist will be made available by FTPC people on the web), to ascertain that all parts of the system are in a proper state and ready to operate. If this is not the case the operator should take action where possible or call the FTPC experts immediately and alert the shift leader.

Interlock status

Next the interlock status should be checked. The status of the interlocks affecting the FTPC is shown on the main control window. A green light signifies that the interlock is giving permission for operation.

If one or more interlocks failed (red light on) the detector cannot be operated.

The "Electronic permissive" and the "Detector leak" entries in our interlock frame reflect the status of the STAR General Interlock System (SGIS). In case one of these interlocks failed (red light on) nothing can be done by an FTPC operator, but the shift leader should be notified in case he/she is not yet aware of the situation.

If the "HV permissive from gas" interlock fails (red light) you should immediately notify one of the FTPC experts and the shift leader. In this case all the high voltages of the FTPCs are turned off automatically and the detector is not operational.

If the "LV permissive from cooling" interlock fails (red light) you should make sure that the cooling system and its circulation pump switches are set to their "On" status. After turning on the cooling system wait for the interlock to clear (it takes few seconds). If the "LV permissive from cooling" interlock is still triggered (red light) cycle the power on the cooling system once more. If the interlock condition persists, alert one of the FTPC experts immediately and notify the shift leader

Cooling system

The cooling system is controlled by the "Cooling System" frame on the main control window.

It is a leak less system maintained below atmospheric pressure. It uses the modified chilled water (MCW) supply to cool the FTPC cooling water in a secondary circuit.

Monitoring relevant quantities

Relevant quantities can be monitored by selecting the appropriate monitor window from the "Monitor" scroll down menu in the main control window (Figure 1). Detailed information is given in the following windows:

- Gas;

- Temperatures and Pressures;

- Readout Board Voltages;

- Anode Voltages and Currents;

- Experimental Hall Temperature and Humidity;

- Summary.

The "Summary" window should ALWAYS accompany the main control window on the computer screen.

To open this window, go to the "Monitor" --> and click on "Summary" option.

Figure 2: The "Summary" window in a typical "Physics" configuration.

In this window are shown the anode sense wire (anodes) measured voltages and currents, the HV electrode (cathode) measured voltages and currents, the pressures and temperatures of the cooling water, the gas temperature at the detectors, the gas flow and the O₂ content, the gating grid voltages, and the temperature and humidity in the experimental hall.

In the top left corner of the window is shown the status of the several parts of the detector. As a general rule numbers are presented in colors: yellow signifies a low severity alarm, red a high severity alarm and green the normal range of operation. A number in black color means that that specific parameter is not yet connected to an alarm condition.

The update of the values in the monitor windows takes few seconds. Also, some variables update faster than others.

Configuration

Normally the detector should stay "ON", and the detector operator only has to change its "configuration". The detector can be configured for Physics, Laser, Pedestal, Stand by, or Magnet ramp modes. The only difference between these configurations is the voltage of the anodes and the status (on/off) of the FEEs and the readout boards. When the detector is turned on the cathode and the gating grid voltages are on in every configuration.
To achieve the proper configuration select it from the scroll down menu on the left side of the main control window and then press the configure button. If the detector is ''ON'' the anode voltage will ramp to the proper value, otherwise the voltage set point will be stored and be ready to be loaded as soon as the anode voltages are turned on.

The low voltage power supplies are always on, unless the detector is configured in "magnet ramp" mode.
After the magnet ramp is completed the detector operator should change the configuration to "Stand by'' or to any pertinent value, but not leave it in the "Magnet ramp" mode. In this way the electronic will be turned on, temperatures and pressures will be readable and the detector will stabilize its temperature.
During RHIC filling and dumping or during short shut downs the detector should be configured in "stand by" mode.

Turning off the detector

This operation should be performed only when a long shut down of RHIC is foreseen (> 1 shift) or a possible danger to the detector is foreseen but is not immediate: an incoming thunderstorm associated with RHIC turning off, possibility of beam hitting the detector, before a system reboot if possible, or any other dangerous situation communicated by the shift leader or the main control room (read the "Emergency shut down" paragraph for procedures in case of an immediate danger).
Has to be emphasized that the standard status of the detector is "ON".

Also, the cooling system should always be left ON

The easy and preferred way to turn off the detector is to press the "OFF" button in the "Turn on/off" frame in the main control window. The operator should then observe the voltages to go to zero from the summary window and the messages scrolling on the main control window message frame to ascertain that the turn off process is being accomplished properly. The turn off process takes about 2-3 minutes. When the detector is turned off water pressures and temperatures are not readable and are set to zero.

Another way to turn off the detector is to access the "Quick on/off" window through the "Control" scroll down menu on the main control window. From this window all the FTPC subsystems can be switched on/off separately.

Figure 3: The "Quick on/off" window.

For this second method (Quick on/off) the order in which systems should be switched off is indicated by the direction of the red arrow, beginning at the bottom and proceeding upward. It is important to follow this order to avoid subjecting more delicate parts to unnecessary stress by the induced currents in the turn off process. When using this method to turn off the detector care should be taken to ascertain that, for each subsystem, the operation is completed before switching the next.

As a third option to turn off the detector all channels of all subsystems can be switched off separately by selecting the proper control window from the "Control" menu in the main control window. This option should be generally left to the detector experts.

Emergency shut down

The emergency shut down is NOT a healthy procedure for the detector, and should be avoided whenever is possible.

To perform an emergency shut down press the corresponding button on the main control window. The following window will appear:

Figure 4a: The emergency shut down confirmation window.

The main control window will disappear. To shut down press the "Yes" button. To go back to the main control window press the "Go-back" button. During this process the windows might get the wrong color map. To restore the proper colors they have to be slightly dragged. Situations that may require an emergency shut down include a sudden increase significantly beyond the standard operating range (alarms off and beeping) of all or most of the anode and/or cathode currents of the detectors.

Once the emergency shut down procedure has started all voltages will be zero within few seconds, and afterwards the control program will set all operating parameters to the standard "Off" status.

If the emergency shut down was required by a specific FTPC problem, or when in doubt, notify immediately, by phone or in person, the FTPC experts, otherwise just send them an e-mail.

If an emergency shut down is performed, an FTPC expert must be notified following the incident.

Turning on the detector

Make sure that the STAR magnet is not ramping!

Similarly to the turn off procedure this action can be performed through a single button operation (which is the preferred way to do it), pressing the "ON" button from the "Turn on/off" frame in the main control window. This procedure takes about 3-5 minutes to complete. After completion the detector will be in the state defined in the "Configuration" menu.

The other option is to open the "Quick on/off" window (Figure 3) from the "Control" scroll down menu in the main control window and to proceed turning on the subsystems along the green (ON) line. In this case care has to be taken to insure that one action is completed before stepping to the next. When going through this procedure using the "Quick on/off" panel the detector configuration is overwritten and all components will be turned on.

All channels of all subsystem can also be switched on individually by selecting the appropriate control window from the "Control" scroll down menu in the main control window. However this is an option that should be left to the detector experts. During the turn on procedure the operator should observe carefully the message frame in the main control window and the monitor summary window to verify that the procedure is being executed properly.

Computer system

The control system makes use of 4 computers:

- the MVME 167 control processor in the FTPC VME crate (#71) on the south platform;

- the slow control state machine (sc3.starp.bnl.gov)

- the FTPC WINDOW NT PC (bond.starp.bnl.gov) in the gas mixing room at the FTPC gas rack;

- the FTPC control console (cassini.starp.bnl.gov).

The MVME 167 is the processor where all the control and monitoring tasks are running. It sends all the parameter information to a parameter database that is managed by the slow control state machine.

Data coming from the gas system are read out by the FTPC WINDOW NT PC and delivered to the MVME 167 through the online computer. They are then processed by the MVMe 167 and the parameters are sent back to the parameters database on the slow control state machine.

Finally the Graphical User Interface of the control software is run on the slow control state machine from the FTPC control console.

Rebooting

The possibility of a computer crash cannot be excluded a priori. If the MVME 167 CPU crashes all parameter fields depending on it will be shown in white color on the graphical interface.

Figure 4: The main control window after a crash of the MVME167 processor.

In this case a reboot of the MVME 167 is urged. From the main control window press the "Reboot" button. The following window will appear:

Figure 4a: The reboot confirmation window.

The main control window will disappear. To reboot press the ''Yes'' button. To go back to the main control window press the ''Go-back'' button. During this process the windows might get the wrong color map. To restore the proper colors they have to be slightly dragged. If a window with the following message appears:

Mon Jun 18 13:51:00 EDT 2001

medmCAExceptionHandlerCb:

channel name: cu_vme71_control_boB2

message: Could not perform a database value put for that channel

... ignore it, close it and try the "Reboot" button again until the window does not appear anymore (although often it works straight away at times it may need five or more attempts). At this point the operator can be sure that the reboot is actually occurring. It takes about 3 minutes, but it might take more, depending on the traffic on the network. Another method to reboot the MVME 167 is to enter the "VME STATUS" screen which can be opened from the main control window:

Figure 4: VME status window.

and to turn the VME crate off (wait for the voltage indicators to go to zero), wait a few instants and then turn it on again.

The reboot is not an ordinary operation, and the FTPC experts should be notified by e-mail of any reboot.
After the reboot the system should be brought again to its original (on or off) state by operating the standard "turn on" or "turn off" procedure. This is only to make sure that all the parameters are in the proper settings for the present running conditions and were not altered by the reboot process.
If the FTPC console (cassini.starp.bnl.gov) crashes it can be rebooted in the standard way. The operator will then have to log in again using the username (operator) and password written on the computer.
If the gas parameters are not updated the status of the FTPC computer in the gas room should be checked. If it has crashed it should be rebooted. The operator should then log in using the username and password written on the computer. During the login the password for the cross mounting of the /online disk will be asked for. This will be found on the computer too.

Alarm handler

The alarm handler alerts the operator when a sensitive parameter goes out of the range of standard operation. A visual and an audible signal are released in this case. The detector operator should click on the alarm handler miniature window (Figure 5),

Figure 5: The alarm handler miniature window.

to open the main alarm handler window (Figure 6).

Figure 6: The main alarm handler window.

From this window, by pressing the uppermost left square, the alarm is acknowledged and silenced. The colored squares indicate the parameter in an alarm state. The color red stands for a high level alarm, yellow for a low level alarm, and white for the loss of communication. By pressing the square with the "G" some guidance is given.

When turning off the low voltages, or turning on the anode voltage, a transient alarm might go off. This will result in one or two beeps maximum and does not need to be acknowledged. The reason is that switching on/off devices may cause their current to deviate slightly above the alarm threshold.

If the Alarm Handler (Figure 5) is closed by mistake, all windows including "medm" should be closed and the FTPC control program restarted following the same procedure as mentioned earlier.

Known problems

FEEs and Temperatures

Magnet ramps
In case of a magnet ramp it is necessary to turn off the FEEs (default in the Magnet ramp configuration). In this mode we lose the FTPC temperature information. When the magnet ramp is finished and the FEEs are on again, there is a non-negligible possibility that the temperature information does not come up in the East detector. This information is VERY IMPORTANT for our physics results.

Please check that the temperature sensors are working by clicking on the "Monitor" button in the "main control window" (Figure 1). Then select the "Pressure and temperature" option from the pop-up menu. The "Temperature-Pressure" window (Figure 8) will appear. Normally the temperature readings should be in the 20s. Some of the readings will be lower than 10. This is due to a known RDOB problem (see below).

In case ALL COOLING PLATE AND BODY TEMPERATURE values for FTPC East are 0.000, the only solution currently possible is to try to reboot the ftpc slow control. with the "Reboot" button in the "main control window" (see chapter about rebooting procedure). This process will take 5-10 minutes and during this time the connection will be off and there will be no parameter values. After that, the detector should be on again. The rebooting is not a normal procedure so, please, consult an FTPC expert first.

Additional sensors seen in rows at the bottom of Figure 8 will always display temperatures, even when the FTPC FEEs are off. These are from an independent system.

Temperature 7 in FTPC W does not exist and will always display

0.000

Broken sensors
~~The "Output gas" thermistor on the west chamber is broken. The display always shows 0.0 in white color. The alarm handler is already set to ignore this broken sensor.~~ Corrected for 2005.

Figure 8: The Temperature and Pressure monitoring window.

Anode problems

In the case that one or more of the anode HV channels trip, the operator should first clear the trip. To do this, open the "Anode voltage" window (Figure 7) by clicking on the "Control" button in the main control window (Figure 1). The channel that has tripped will be marked by a red flag, while all the others should be marked green. Please make a note in the FTPC log book (and/or Electronic ShiftLog) as to which channel tripped. To clear the trip just press the "CLEAR" button in the bottom left corner of the window. After a short time the tripped channel(s) should be marked with a brown flag (disabled channel). To restore the voltage press the "ON" button on the anode voltage switch in the "Quick on/off" window (Figure 3). If the trip happens once more on the same channel(s) the operator should call a detector expert.

Changing the configuration from physics to laser, sometimes the anodes voltages in the West chamber do not go up to the nominal value (1200 V). This value should be forced from the "Control"-->"Anodes" window.

Figure 7: The anodes control window.

RDOB problems

Standard set points



Cathode voltage (2 channels)	-10kV±5V
Anode voltage (2x6 channels), Physics	+1765V±2V West, +1750V±2V East
Anode voltage (2x6 channels), laser	+1200V±2V West, +1200V±2V East
Anode voltage (2x6 channels), pedestals	+1000V±2V West, +1000V±2V East
Anode voltage (2x6 channels), stand by	0V West/East, (+~15V are displayed)
Gating grid (2x4 channels)	-76V±2V open, -76V±115V closed

Standard values



Water pressure in (West/East)	-400mbar -> -100mbar
Water temperature (West/East)	<31 C

O2(ppm) (West/East)	<10ppm
H2O (dp C) (West/East)	<-50 C dp
Ar flow = CO2 flow (West/East)	72 l/h->78 l/h;West=East

Cathode current	0.14mA
Anode current	-15nA < I < 15nA

Created: Wed Jun 13 14:25:19 EDT 2001 by Gaspare Lo Curto
Revised for Y2003: Jan 1, 2003 by Maria Mora
Last modified: Mar 27 18:00:00 GMT 2007 by Terry Tarnowsky

Preliminary Version FTPC procedures for Y2007