RECOVERY & RESTART:


  1. Controls (VxWorks IOC) failure -- GUI "all white"

      In this state. HV values likely still loaded in CW cells and FEE's still configured, but to gain slow control of HVSys and resume monitoring functions of FEE one must reboot VxWorks -- this turns off HV and reloads values

    1. When ready -- reboot EEMC slow controls IOC

      1. From existing VxWorks monitor window (or get via "telnet eemc-serv 7007") type reboot at prompt, or do a "ctrl_x" to force a reboot
      2. Alternatively, use CANBus connection to VME crate #90 and click on "SysReset" (this reboots all processors in crate)
      3. Follow reboot progress and messages to IOC monitor until startup script finishes

    2. Monitor HVSys GUI for progress

      1. Hopfully the 3 controllers will come back with big green (communication) dots (the corresponding branches may be enabled or not)
      2. Similarly, the body of the GUI should turn gray for all boxes indicating voltages are loaded and cells ready to ramp up
      3. If above 2 conditions are satisfied ... click overall "on" HV with GUI button and system should come up in a few minutes
      4. If not ... refer to "Controller and CW cell errors" section communication and restore control and base loaded values ...

    3. Tower FEE GUI screens

      1. System may appear to come back up and monitor (set "installed" mode) without errors and overall green status for all crates
      2. However, re-configure the crates via "master reload"
      3. If problems, refer to previous tower guidance

    4. Monitor MAPMT FEE GUI screen

      1. MAPMT FEE boxes display may come back up, monitor and look ok (esp. at top level GUI)
      2. A "master reload" is required in order to gain control and be certain of the box status
      3. If problems, refer to previous MAPMT guidance
  2. Restart Controls -- e.g., hangup or other intractabilities

    1. Follow abve procedures for "GUI White" - reboot