MAPMT FEE ALARMS/RECOVERY:

    The monitoring feature of the MAPMT GUI has been updated to better reflect the current status of the MAPMT boxes and FEE electronics (serial sequence through boxes) and alarms have been added to warn of problems.

  1. Individual box status/alarm

      Note: the following actions interrupt DAQ function of box!

    1. if configuration or communication indicator fails (RED) for individual boxes:

      1. pull up individual screen and try to reset box by reloading FPGAs (lower right button) ... follow this by "load", "write" and "read". It may be necessary to repeat a few times.

      2. if errors continue (e.g., HDLC communication) it is likely the box has lost its initial configuration download and this will have to be downloaded again.

        1. one can check if this is really the case by typing at VxWorks monitor prompt: smdReConfig "channel","node" (channel and node are given for you at left/top of individual box GUI). If last line it spits out contains "DONE=0", box requires download.

        2. the download can we accomplished with the "mapmtDownload" buttons at the bottom of the individual MAPMT box screen. In general it is the rdo.bit file that needs downloading and this will be sufficient (a separate "mapmt_fee.bit" usually not required). The download takes several minutes (one can chart progress on the VxWorks monitor) ... at the GUI the indication the download is done is that the box monitoring process starts again.

        3. after download, go back to sequence 2.1 above: "reload FPGAs", "load", "write", "read". Box should now configure without errors and status will turn green.

  2. Multiple box status/alarm or power supply alarm

    1. Check power via CANBus screen and turn supplies on if found off:
      (these supplies each power 16 boxes/4sectors of SMD & Pre/Post FEE)

      Table of sector MAPMT boxes and associated dual power supplies:
      Supply #97 Supply #96 Supply #95
      sectors MAPMT boxes sectors MAPMT boxes sectors MAPMT boxes
      12 12S1-3; 12P1 4 4S1-3; 4P1 8 8S1-3; 8P1
      1 1S1-3; 1P1 5 5S1-3; 5P1 9 9S1-3; 9P1
      2 2S1-3; 2P1 6 6S1-3; 6P1 10 10S1-3; 10P1
      3 3S1-3; 3P1 7 7S1-3; 7P1 11 11S1-3; 11P1

    2. Reconfigure all boxes that have had power restored

      1. Reconfigure with sector specific "Master Reload" via the GUI for a 3rd of the MAPMT readout sectors at a time (see above table) -- see HDLC/config chatter on GUI and chart progress
      2. If all supplies power cycled, use a global MAPMT "Master Reload" -- (it will take ~ 9 minutes)
        -- see HDLC/config chatter on GUI and chart progress
      3. For boxes that give errors on configure -- see details in 1.1 above

  3. Temperature alarm -- readings of the individual box (water cooled) center plates

    1. If temperature is high for all boxes -- likely water chiller failure (either chiller has shut off or reservoir too low)

      1. Boxes have internal thermal interlock (trip temperature reached ~ 30-40 minutes after cooling loss) so in principle will shut down ... but
      2. (can check chiller reservoir level indicator via remote camera at upper right STAR magnet console -- west side, position 1)
      3. Best to turn off power to crates and investigate with (experts) access ASAP -- if a water leak has developed we want FEE powered down and chiller off

    2. Temperature is high for a (one) box

      1. Either abnormal electronics operation or blockage of individual box cooling
      2. Expert help will be likely needed to secure the affected box while keeping the others powered

  4. Controls (VxWorks IOC) failure/hangup -- GUI "all white"

    1. MAPMT box configurations still loaded (could continue running a bit), but to gain slow control/monitoring reboot VxWorks (impacts other systems!)

    2. As described (also) elsewhere reboot EEMC slow controls IOC

      1. From existing VxWorks monitor window (or get via "telnet eemc-serv 7007") type reboot at prompt, or do a "ctrl_x" to force a boot
      2. Alternatively, CANBus connect to crate #90 & "SysReset"
      3. Follow reboot progress on IOC monitor
    3. Monitor MAPMT FEE GUI screen
      1. Boxes will eventually display non-configured status even though unless power supplies have gone off boxes will still be configured
      2. A "Master Reload" will reconfigure all boxes properly ... see above in case of ensuing individual box errors

  5. Note: when MAPMT power supplies are cycled or restored -- for whatever reason

    1. All affected MAPMT boxes (remember the supplies service 16 boxes) must be re-configured -- follow instructions above in this section