HVSYS ALARMS/RECOVERY:

---

1. Red/Yellow/"Blinking" Bases and the HV base error count

   1. The HVSys monitoring system serially (clockwise around the endcap) checks the status bits reported by the Cockroft Walton bases and updates the HI-VOLTAGE GUI. At this level, problem tower (outer radius of display) and MAPMT boxes (toward interior of display) take on the color code most often of a single tube in the box (use the detail button to see which one). Individual base readouts sometimes fluctuate; it is also true that during the run a small number of bases fail and will have to be replaced. In any case a red box (or two) is not a cause to stop running with the Endcap!

   2. "ErrorCount" box on EEMC top level GUI. This number indicates the total number of "bad" (unmasked) bases in the system. It will cause an alarm at some fraction of a box count of tubes/bases, indicating either a box failure or accrue of a significant number of individual bad bases.

   3. Action: refer to communications section below to make make sure everything is in order and working properly. If new box errors (e.g., red boxes on GUI) and associated error count increase has occurred, it is possible a base really has gone bad ... although sometimes they recover by themselves over period of hours or even days.

   4. Open individual box screens and note bad base(s), monitor situation and inform EEMC experts at next opportunity. (there is an alarm masking feature for permanently bad bases at box level GUI -- any masking should be done only in conjunction with expert consultation and log entry).

2. Controller and CW cell communication error problems
   
   
   1. HVSys GUI: if all controllers show big green dot = ok; reload voltages for sector(s) that appear affected/not green (e.g., hit "reload" on GUI sector display and wait for controller/display to update).

   2. HVSys GUI: if controller shows big black dot = failure (likely also 4 branch "enables" -- 4 small dots -- are off).
      1. Most direct way to recover is to cycle AC power for problem controller. To do this: from any eemc slow controls window, connect to the network power switch via "telnet eemc-pwrs1" ... username = [apc]; pswd = [see book]. From the funky keyboard instructions, navigate to the network controls for the outlets 1,2,3 (corresponding to controllers A,B,C). Use the controls to power off and then power back on the troubled controller -- it may take a few power cycles to get the controller connection (big green dot -- be patient after a power cycle for the slow controls to circle around and try and get the connection again).
      2. When connected, reset controller and reload voltages in affected sectors (use table below or observe GUI sector color). -- should eventually get ready state gray color for affected sectors.
      3. Turn "on" sectors at GUI (if not on already) -- this should enable corresponding branches -- but if not (or one is impatient) click branch enables on. Iterate reset/reload if necessary to get system back up.

3. Controls failure/hangup -- HVSys GUI "all white"

   If this occurs, likely other systems will be affected as well. See the Controls Recovery & Restart section for the details of the HVSys recovery. [Note: reference back to the sections directly above (and for completeness below) may be required if there are communication problems on restart.]

4. HVSys - High Voltage for tower & MAPMT CW bases
   

   Table of HV controllers, branches and detector sectors:
   --> 3 controllers (4 branches each)

   	Controller A	Controller B	Controller C
   Branch	tower subsectors	tower subsectors	tower subsectors
   0	11TD-1TC	3TD-5TC	7TD-9TC
   1	1TD-3TC	5TD-7TC	9TD-11TC
   Branch	MAPMT sectors	MAPMT sectors	MAPMT sectors
   2	sec 12,1	sec 4,5	sec 8,9
   3	sec 2,3	sec 6,7	sec 10,11