STAR   Computing  GRID  Monitoring  MonaLisa

The STAR Web Repository

Introduction

The MonALISA monitoring framework provides a mechanism to create "pseudo-clients", a service which collects selected information from specific groups of ML monitoring services. A "pseudo client" subscribes to ML monitoring services through predicates and filter agents indicating the type of monitoring data it would like to receive. The selected information is stored locally in the Web Repository in a MySQL database and is accesible through a Web Service. Servlets and JSPs are used to present historical and real time values of the stored data. The Web Repository presents a global view of a set of distributed ML Services that is available to higher level services.
For a description of ML Repositories visit the MonALISA Repositories page.

The STAR Web Repository

The MonALISA developers have set up a Web Repository for the Star group of ML Services. The Repository is located at http://monalisa-starlight.cern.ch:8888/index.html and its purpose is to provide global views of the STAR MonALISA Services.
Any MonALISA Service may provide data to the Web Repository by joining the Star group of ML Services.
The Web Repository data are available as a Web Service:
http://monalisa-starlight.cern.ch:8888/axis/services/MLWebService

Configuration of the STAR Web Repository

pseudo-client Configuration

The functionality of the "pseudo-client" on monalisa-starlight.cern.ch as well as the storage configuration of the Web Repository is set in the configuration file ./MPrespository/JStoreClient/conf/App.properties. The parameter lia.Monitor.groups indicates which group (or groups, separated by comma) the pseudo-client will request data from and lia.Monitor.LUSs indicates the Look Up Services the "pseudo-client" will query (via unicast) to discover the ML monitoring Services that belong to the specified group (or groups): 
        lia.Monitor.LUSs = monalisa.cern.ch,monalisa.cacr.caltech.edu
        lia.Monitor.group = star
Upon discovering the available ML services the "pseudo-client" subscribes to these services and indicates the type of monitoring data it would like to collect by a set of predicates. The predicates are, basically, regular expressions (seperated by comma) with the following structure:
Farm/Cluster/Node/start_time/end_time/function_list
(multiple functions should be separated by |).

For example: 
lia.Monitor.JiniClient.Store.predicates=*/Master/*/-1/-1/Load5|%_IN|%_OUT,*/WAN/*
The local ML Monitoring Service starts a new thread to serve each pseudo-client that subscribes with it. The pseudo-client receives all the data that match the predicate, but what actually gets stored in the DB tables of the Repository is not the raw data.

Storage Configuration

The collected data will be stored in a MySQL database (mon_data). The tables are configured in the same config file (App.properties). The following parameters are set for each table:
Table Parameter Description
total_time The total time in secs that will keep data in a table. Data older than total_time will be removed from the table.
samples Number of samples in the table. total_time/samples gives the time interval between samples.
table_name Table name
descr Table description
write_mode
  • 0 (mediated table, default).
    Values will be stored in the table every total_time/samples time intervals averaging received values.
  • 1 (unmediated table).
    Raw data will be stored in the table. The samples parameter (if specified) is ignored.

There is a total of 9 tables in the mon_data database. monitor_1hour is the mediated table that stores data for the last one hour and is the most frequently accessed table. Data older than one hour are removed from the table. monitor_1hour_nm is the unmediated table that stores the received data as is. The rectime column in all tables is the time when the data were entered in the table, not when the data were collected. This is true even for the unmediated table in which data is entered as is. Under util/ShowStoreConfig there is a utility that reads the configuration file and displays the storage configuration: a list of tables with the values for their parameters.

Utilities are provided in the form of shell scripts under the util/SimpleDBShell directory to access locally the database tables. All scripts access data in the monitor_1hour table.

Accessing the Web Repository

Accessing the Web Repository through a Web Service Client

A number of ML Web Service Clients are provided under the MLrepository/WSClients/ directory to remotely access the data in the Web Repository. The provided Web Service Clients are the same as the Clients to the regular ML monitoring service. You can read more details about using the WS Clients in the section that that describes them in the ML monitoring service: here .
To access the STAR Web Repository, use the URL:
http://monalisa-starlight.cern.ch:8888/axis/services/MLWebService
The client requests are translated into SQL queries for the monitor_1hour table (mediated). The developers are currently looking into translating WS Client requests to SQL queries for the monitor_1hour_nm table (unmediated).

Accessing the Web Repository through a MySQL Client

You can access the MySQL tables of the Web Repository remotely using a MySQL client. Make sure you use the correct port number. The default port number (3306) is used for another project's Repository (Grid3). For the STAR Web Repository the port number is 3333.

I had no problems accessing the Grid3 MySQL service: 

[monalisa@lin003 bin]$ ./mysql -h monalisa-starlight.cern.ch  -u mon_user -p 
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1244 to server version: 4.0.18-standard

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql>
but I did have problems accesing the Star MySQL service: 
[monalisa@lin003 bin]$ ./mysql -h monalisa-starlight.cern.ch -P 3333 -u mon_user -p 
Enter password: 
ERROR 2003: Can't connect to MySQL server on 'monalisa-starlight.cern.ch' (111)
It might be a firewall/blocked port issue.

Testing

To test the remote access to the Repository and the integrity of the stored data, I have slightly modified the WS Client to connect to both the local ML WS and the ML WS for the Repository and compare the retrieved values. Since the repository values are retrieved from the mediated table monitor_1hour they do not exactly match the values for the same parameters retrieved from the local ML monitoring service. The values in the repository are averaged over a time interval (depending on the values of the parameters samples and total_time described in the table above) and then entered in the DB table.

If, however, we compare the values retrieved from the local service to those in the unmediated table monitor_1hour_nm then the values of the parameters are identical except of the rectime parameter. The rectime parameter is the time that the data were entered into the table and since we are looking at two different DBs (one in the repository and one in the local ML service), there is a difference in the recording time. The time difference is of the order of 2-3 seconds.

Also, there seems to be a delay in the data in the Web Repository DB tables. If, for example, I request the latest value of a parameter from the monitor_1hour_nm DB table and compare the rectime to the latest value of the local ML service, the Web repository will be about 1 minute behind. In other words, the latest value in the DB of the repository is about 1 minute old compared to the one in the local ML service.
According to the ML developers this 1 minute delay is caused because the monitoring data that the "pseudo-client" gets from the ML services is first cached into a buffer and then flushed onto the disk every 1 minute. The reason for buffering the data is that some filters need to be applied to the data before writing them to the DB. Since the Web Service Client gets the data from the DB the data will be about 1 minute old. The data for the history plots come from the DB also and thus they have the same delay. The data for the real-time plots, however, get plotted as soon as they are retrieved by the "pseudo-client". Thus, we should not get the 1 minute delay in the real-time plots.

The Web Front of the Repository

The Web Front of the Repository provides access to real-time and historical graphs of the collected monitoring data. It is based on Tomcat Servlet/JSP Java Technologies. Static html and JavaScript files are located in the REPOSITORY_HOME/tomcat/webapps/ROOT/ directory.

The Web Front for the STAR Web Repository can be accesed at: http://monalisa-starlight.cern.ch:8888/

Stratos Efstathiadis - page was last modified