HVSYS ALARMS/RECOVERY:

---

1. Controller and CW cell communication errors
   
   
   1. HVSys GUI: if all controllers show big green dot = ok; reload voltages for sector(s) that appear affected/not green (e.g., hit "reload" on GUI sector display and wait for controller/display to update)

   2. HVSys GUI: if controller shows big black dot = failure (likely also 4 branch "enables" -- 4 small dots -- are off)
      1. Can attempt to regain communication by controller "reset" (followed by "on" may help though not required) -- Note: these "actions" take time so be patient (and may take several tries)
      2. (or) cycle AC power for problem controller. To do this: from any eemc-sc window "telnet eemc-pwrs1" and then power cycle with e.g., "/off n" command ...then "/on n" command, where n= 1,2,3 for controller A,B,C -- it may take several power cycles to get the controller connection (big green dot -- again be patient). Note: the power switch doesn't repeat your typing but does report its action.
      3. When connected again, reset controller and reload voltages in affected sectors (use table below or observe GUI sector color). -- should eventually get ready state gray color for affected sectors.
      4. Turn "on" sectors at GUI (if not on already) -- this should enable corresponding branches -- but if not (or one is impatient) click branch enables on.

2. Controls failure/hangup -- HVSys GUI "all white"

   If this occurs, likely other systems affected as well.

   1. HV values are still loaded in CW cells (could continue a bit if running during a store), but to gain slow control of HVSys one must reboot VxWorks -- this (at present) turns off HV and reloads values

   2. When ready -- reboot EEMC slow controls IOC

      1. From existing VxWorks monitor window (or get via "telnet eemc-serv 7007") type reboot at prompt, or do a "ctrl_x" to force a boot
      2. Alternatively, use CANBus connection to VME crate #90 and click on "SysReset"
      3. Follow reboot progress and messages to IOC monitor until startup script finishes

   3. Monitor HVSys GUI for progress

      1. Hopefully the 3 controllers will come back with big green (communication) dots (the corresponding branches may be enabled or not)
      2. Similarly, the body of the GUI should turn gray for all boxes indicating voltages are loaded and cells ready to ramp up
      3. If above 2 conditions are satisfied ... click "on" HV with button in GUI center and system should come up in a few minutes
      4. If not ... refer to "Controller and CW cell communication errors" section (above) and restore control and base loaded values ...
      5. Note: the VxWorks reboot will affect also the tower FEE and MAPMT systems (although they also will stay configured if continuously powered) ... see tower and MAPMT trouble sections -- or Recovery & Restart -- for control restore action

3. HV base error count -- "ErrorCount" box on EEMC top level GUI

   1. Refer to communications section above to make make sure everything is in order and working

   2. If still problems with error count (e.g., red boxes on GUI), it is possible a base really has gone bad ... although sometimes they recover by themselves over period of hours or even days.

   3. Open individual box screens and note bad base(s), monitor situation and inform EEMC experts at next opportunity. (there is an alarm masking feature for permanently bad bases at box level GUI -- any masking should be done only in conjunction with expert consultation and log entry)
   
   
4. HVSys - High Voltage for tower & MAPMT CW bases
   --> 3 controllers (4 branches each)

   Table of HV controllers, branches and detector sectors:

   	Controller A	Controller B	Controller C
   Branch	tower subsectors	tower subsectors	tower subsectors
   0	11TD-1TC	3TD-5TC	7TD-9TC
   1	1TD-3TC	5TD-7TC	9TD-11TC
   Branch	MAPMT sectors	MAPMT sectors	MAPMT sectors
   2	sec 12,1	sec 4,5	sec 8,9
   3	sec 2,3	sec 6,7	sec 10,11