STAR RCF Monitoring Shifts

This document was initially produced based on a version of a tutorial written for PHENIX by S. Mioduszewski and T. Chujo.
It was last modified on

IMPORTANT Note for RCF Monitoring: 

After diagnosing the problem, the shift person must decide whether an expert needs to be called during the night. If the problem does not interfere with data-taking, a CTS ticket should be filed. However, if the problem does affect data-taking, then the appropriate expert should be called.

(1) RCF Monitoring

• Who is on shift for RCF Monitoring?

During normal business hours (8 AM - 4 PM), a RCF staff member is always on shift. The next shift is the evening shift, 4 PM - 12 AM, which is the responsibility of an RCF operator. The owl shift is 12 AM - 8 AM and belongs to a PHENIX or STAR shift person (alternating weeks). On BNL lab holidays, PHENIX/STAR is responsible for shifts during the entire day. In the case of STAR, the online QA shift person is responsible for the RCF monitoring. Here is the schedule of RCF shifts. Please check first the RCF shift schedule assigned to STAR, when you start taking the offline shift. The hard copy of the shift schedule is also available at the counting house.  There are two things to be monitored, 1) RCF Network, 2) HPSS status. In the following sections, you can figure out how to monitor the RCF and what you should do when you find the RCF related troubles. The collection of links provided by RCF staff is found here.  

• What are the responsibilities of the person on shift?

The RCF monitoring shift person is required to carry a pager which can be found together with the RCF phone-list above onllinux2 in the STAR counting house. If anyone from the 4 experiments experiences a problem with RCF, he/she should call the appropriate RCF responsible person or page the person on shift. The shift person is not required (or qualified) to fix the problem. The main responsibility of the shift person is to diagnose the problem and decide whether to call the appropriate RCF expert or to file a CTS ticket. The shift person is also required to monitor the HPSS from the counting house. The counting house should always have a display of the HPSS monitoring screen.

1-1. Network Monitoring

• Status Monitor

The status of monitored machines can be checked from here. Figure 1 shows the snapshot of the status of monitored machines. You can check the current status of the ssh gateway servers, FTP servers, NFS servers, DNS/NIS servers and mail servers. In this status monitor, green signal indicates OK, yellow is the warning sign and  red is failure sing. Failure (red) on one of the RMINE (NFS server) machines does not warrant a call to the expert in the middle of the night. A CTS ticket should be filed. However, RNIS (NIS server) machines and ssh gateway machines (RSSH) are critical and DO warrant a late-night call during data-taking. Call the appropriate RCF responsible person (see below). The hard copy of list of RCF staff's home phone number ishould be available at the control room. If you cannot find this list (already provided 3 times since the beginning of the run), please send me (J.Lauret) a note ASAP and I will provide it again.

• On-Call RCF persons for RNIS and SSHGW problem

1. Maurice Askinazi  x2159, pager: 631-441-2595, home: (see the phone list at the counting house).
2. Shigeki Misawa     x2635, pager: NA,                  home: (see the phone list at the counting house).
   
   

  Figure 1. Status of monitored machines display.

• Check Ping status of Network Switches (SW1 though SW9)

Check the following two links to see the PING status of switches. 

1. rolly.rhic.bnl.gov (on .80 subnet, as viewed from OUTSIDE the RCF firewall)
2. ru10.rcf.bnl.gov (on .6 subnet, as viewed from INSIDE the RCF firewall)

If any of the SWITCHES (SW1 through SW9) are reported unreachable for 5 CONSECUTIVE minutes, this is a reportable network condition. Call the RCF networking responsible person.

• On-call person for Network problem

 Terry Healy   x4199,  home : (see the phone list at the counting house).

The following links are useful to monitor the network and diagnose problems. 

• DAQ Network Activity for all experiments ( STAR) - Check the data flow from Experiments to HPSS in RCF.
• RCF current Gigabit Snapshot  - Overall traffic within RCF and activity level of switches.
• Network diagram - Current configuration of the RCF Gigabit connections.
• RCF Network Monitoring Note - Detail instructions about network monitoring. 
• RCF Network Page - Collection of links to monitor the RCF network.
• RCF Ping Script List

1-2. HPSS Monitoring

• Check list for the offline shifter 

1. Check the clock in HPSS GUI is actually working. If not, HPSS is dead. Call RCF HPSS experts immediately.  
2. Check the "Servers" status is Normal (green). If you see the "Major Problem" , call RCF HPSS experts. (yellow is "Suspect" or "Minor", you do not have to call).
3. Check the "Space Thresholds" is Normal (green). If you see the "Major Problem" for more than 30 min. , call RCF HPSS experts. Otherwise, disks will fill up!
4. Check  "Alarms and Events" status by clicking "Monitor" menu on the top of the GUI. Watch this screen, but red does not necessarily mean that someone needs to be called. Only if there are continual red lines, then expert should be called.

  Figure 2. HPSS Health and Status monitor GUI.

• On-call person for HPSS problem

1. Rasvan Popescu   x5806, pager: 631-554-8747, home phone : (see the phone list at the counting house).
2. John Riordan       x7201, pager: 631-554-9011,  home phone : (see the phone list at the counting house).

• Here are a list of links toward several monitoring tools and interfaces.
• How to setup HPSS monitoring software at the STAR counting house. (not necessary to read, if HPSS GUI has already set up) 

1. Log on to onllinux2 in the STAR control room as starcrew. See instructions taped to the monitor.
5. Login as " STAR_op" on " hpss.rcf.bnl.gov" using ssh by typing "ssh hpss.rcf.bnl.gov -l STAR_op" and then enter the password. (ask J. PORTER <porter@bnl.gov>  or Jérôme LAURET <jlauret@bnl.gov> for the password.) 
6. The procedure starts automatically the HPSS control. Accept the default values for user and display. When a graphical window prompts you for username and password, use user "STAR_op" and the same password as for the hpss account. TYPE the username in the first field, THEN SWITCH TO THE SECOND FIELD (using a tab or the mouse), type the password and then press "Enter". The main control will show up! Use the menus to access details of the state of the system. Check the "Monitor" menu. It is important that the clock is running in the main control GUI.  
   
7. To logoff (IMPORTANT!!)
   To logoff:
   - Select "Session" --> "Quit Sammi" and confirm.
   - Don't use "Log off Session".
   
   The terminal window that initiated the session will not close since its process handles the X encrypted channel. Minimize it and ignore it. It will automatically close when the HPSS session will be terminated.

1-3. Paging RCF Personnel

• Who-is-on-Call List
• Page the Operator-on-Shift
• Send E-mail to the Operator-on-Shift

Instructions for CAS and CRS node monitor

• Status of CAS (Central Analysis Server) machines

  Status of the CAS machines (the analysis nodes) can be found here. The link under " CAS Operational Status" lists any CAS machines experiencing problems. Failure on one of the RCAS machines does not warrant a call to the expert in the middle of the night. A CTS ticket should be filled.

• Status of CRS (Central Reconstruction Server) machines

  Status of the CRS machines (the reconstruction farm) can be found here. Failure on one of the CRS machines does not warrant a call to the expert in the middle of the night. A CTS ticket should be filed. Reconstruction Farm latest staging failures is also available.

• Other links to CAS/CRS monitoring tools
  can be found here (CRS) and here (CAS)