CONTROLS RECOVERY & RESTART:

---

1. Controls (VxWorks/IOC) failure -- GUI "all white" -- (and not GUI display problem)

   If things were operating well previously, in a simple controls failure state the HV values are still loaded in the CW cells and the FEE's are still configured (hence one can continue to run [with caution] for a limited time in this way, monitoring J-evp plots to ensure things continue to operate well).

   BUT, to be sure of status and to gain slow control of HVSys and resume monitoring functions of the FEE's, one must reboot the VxWorks CPU -- at present this action turns off the HV and reloads the settings. Similarly the Tower and MAPMT FEE's have to be reconfigured to re-establish their GUI control/monitoring function.

   1. When ready (and if controls haven't "re-booted" by themselves) -- reboot EEMC slow controls VxWorks/IOC CPU by one of the following 2 methods:
      1. Reset the Controls crate (preferred). Use the CANBus connection (select "POWER" from the top GUI to bring up the EEMC VME crate GUI). Open the power supply display for VME crate #90 via the pink detail button; then click once on "SysReset" located at the right (this reboots all processors in the crate, e.g., the VxWorks CPU).

         It can be useful to monitor progress on the VxWorks/IOC monitor. Use an existing VxWorks monitor window, or get it via instructions in Overview & Operations Windows ("telnet eemc-serv.trg.bnl.local 7008" from e.g., ssh to daqman).

      2. Alternatively, use the VxWorks monitor window connection as described directly above, and type reboot at the prompt, or do a "ctrl_x" to force a reboot.

      3. In either case, follow the reboot progress and messages on the VxWorks/IOC monitor until the startup script finishes or just wait until one sees the controls returning as evidenced by the various control GUI's coming back to life (e.g., tower montioring starts to cycle through the crates).

         Then bring the HVSys, Tower FEE and MAPMT FEE systems back up as described in the next 3 steps. The appropriate actions can be taken in parallel to speed recovery [e.g., check step "5)" and get MAPMT FEE configuration going if it is not already doing so automatically; similarly get step "4)" Tower FEE master reload started; then check progress of step "3)" the HVSys system, etc.].

      4. Monitor HVSys GUI for progress
         1. Hopefully the 3 controllers will come back on the HV GUI with big green "communication" dots (corresponding branches may be enabled or not).
         2. Similarly, the body of the GUI should turn gray for all boxes indicating voltages are loaded and cells ready to ramp up.
         3. If the above 2 conditions are satisfied ... click overall "on" HV with the GUI button and system should come up in a few minutes
         4. If not ... refer to " Controller and CW cell errors" section (HV/troubleshoot) and restore control and base loaded values ... when ready turn on HV.

      5. Tower FEE GUI screens

         1. System will usually come back up with green power and green status indicators. The tower crate "ready" indicators will be red.
         2. If all power is on, Re-configure the crates via "master reload"
         3. If problems, refer to previous tower troubleshoot instructions; or similar tower guidance (from Normal Ops/Tower FEE issues).

      6. Monitor MAPMT FEE GUI screen

         1. MAPMT FEE display should come back up with all boxes (crates) indicating an unconfigured state on the GUI.
         2. A "master reload" is required in order to gain control and be certain of the box status
            1. Depending on exact history, controls may automatically launch the master reload ... if so, let it proceed.
            2. If not, manually "master reload" all sectors (abort/reset may be needed to get it started)
         3. If problems, refer to previous MAPMT troubleshoot instructions; or similar MAPMT guidance (from Normal Ops/MAPMT FEE issues).

   2. Restart Controls -- e.g., hangup or other intractabilities

      1. Follow above procedures for "GUI White" - reboot

         ---