TOWER FEE ALARMS/RECOVERY:

  1. Crate status/alarm

    1. Check that power to the FEE crates is on as indicated in the left column of the TOWER GUI (or via the POWER/CANBus screen). Turn supplies back on if found off:

      • dual supply #94 - tower crates 1&2
      • dual supply #92 - tower crates 3&4
      • dual supply #93 - tower crates 5&6

      Individual crates affected by the power restore can be recovered with "load", "write" and "read" to check operation and refresh the GUI; starting a next run will configure all crates again and properly download the LUTs.

      For a massive failure, do "Master Reload" to restart everything correctly. If there are troubles with individual crates after a few tries, see next (or Normal Ops/tower trouble section).

  2. Individual tower crate problems

    1. On tower FEE individual screens (persistent trouble):

      1. Re-configure crates & load peds for each crate in question (e.g., via "load", "write", "read" on individual FEE crate screens) -- make sure FEE boards go to command mode and back to "run" mode at end.

        • a hard "reset" (popup resets window from individual tower crate GUI) reprograms all FPGA's to power up value and hence must be followed by complete reconfiguration (LUTs are downloaded upon run start) as per a power cycle.

      2. If a power cycle is employed to clear problems, or if any tower FEE crates have tripped off, a "Master Reload" (or its equivalent as described above at the crate level - recall there are 2 crates per supply!) is required after resetting the LV power.

    2. When in doubt, consult an EEMC expert!

  3. Temperature problems (or anomalous PL500 LV supply shutdown)

    1. Air flow sensors (detecting flow from the west WAH wall mounted blower) were installed before run 11 on all three tower crate pairs and interlocked to the corresponing PL500 power supplies. The sensors read in the range "11-13" under normal operation as indicated on the power supply GUI under the rubric "Air Flow" Transducer".

    2. If a sensor is reporting out of the normal range the corresponding supply will be shut down (preventing damage to the crate FEE cards).

    3. If the sensors are out of range for all crates -- likely the air blower or duct system is in failure. These type of events will need to be investigated by EEMC experts.

    4. There is also a software alarm on the "rate of change" of the crate controller temperature readouts. Its relevance is likely superceded by the air flow sensor interlock system.

  4. Controls (VxWorks IOC) failure/hangup -- GUI "all white"

    If this occurs, likely other systems will be affected as well. See the Controls Recovery & Restart section for the details of the Tower FEE crate recovery.

  5. Note: when crate power is cycled or restored -- for whatever reason

    1. All affected crates (remember the supplies service 2 crates each) must be re-configured and LUT's downloaded ("Master Reload")