TOWER FEE ALARMS/RECOVERY:


  1. Crate status/alarm

    1. Check power via CANBus screen and turn supplies on if found off:

      • dual supply #94 - tower crates 1&2
      • dual supply #92 - tower crates 3&4
      • dual supply #93 - tower crates 5&6

      Do "Master Reload" to restart everything correctly. If there are troubles with individual crates after a few tries, see next:

    2. On tower FEE individual screens (persistent trouble and/or not solved with "Master Reload"):

      1. Re-configure crates & load peds for each crate in question (e.g., via load, write, read on individual FEE crate screens) -- make sure FEE boards go back to "run" mode at end
      2. Similarly re-load LUT's to affected crates (power cycle reloads default LUT)
      3. For crates that don't want to configure the soft reset in the "resets" pop-up window can be used

  2. Temperature alarm -- these are readings from crate controller

    1. If temperature is high for all crates -- likely air blower or duct system failure

      1. Running without air cooling can damage FEE electronics (and in any case the temperature dependent electronic drifts will render the data unusable)
      2. Turn off power to crates and investigate with (experts) access ASAP

    2. Temperature is high for a (one) crate

      1. Either excessive power use by electronics (failure) or individual crate air duct problem
      2. As above ... turn off power to affected crate(s) and investigate with (experts) access ASAP
      3. or, if one crate affected it could be temperature increase in WAH coupled with tight crate controls level.

  3. Controls (VxWorks IOC) failure/hangup -- GUI "all white"

    1. Tower crate configurations still loaded (could continue running a bit), but to gain slow control/monitoring reboot VxWorks (impacts other controlled systems!)
    2. As described elsewhere (see HVSys section) reboot EEMC slow controls IOC

      1. From existing VxWorks monitor window (or get via "telnet eemc-serv 7007") type reboot at prompt, or do a "ctrl_x" to force a boot
      2. Alternatively, CANBus connect to crate #90 & "SysReset"
      3. Follow reboot progress on IOC monitor

    3. Monitor tower FEE GUI screens

      1. System may come back up and monitor (set "installed" mode) without errors and overall green status for all crates
      2. However, a re-configure of the crates ("load, read, write" all + LUT write -- e.g., do "Master Reload") won't hurt and likely is best procedure

  4. Note: when crate power is cycled or restored -- for whatever reason

    1. All affected crates (remember the supplies service 2 crate each) must be re-configured and LUT's downloaded ("Master Reload")