STAR Computing GRID Scheduler Development

Setting Up STAR Unified Meta – Scheduler Statistics Collection

Introduction

Introduction:
The STAR Unified Meta Scheduler (SUMS) has the ability to record and display detailed information about its usage. SUMS will work perfectly normally without collecting statistics data. It is recommended that large sites set up this option, as it may prove useful.

The system works as follows; the scheduler has a built in database connector that allows it to insert data directly into a data base. To view this data, in a way that makes sense to a human, SUMS comes with a set of JSP’s (java server pages) that can be deployed on a web server. These pages access the database using queries to extrapolate and filter data. The JSPs are built on a powerful java plotting engine that renders the data into different types of graphs that are accessed in real time via the web.

Instructions for setting up SUMS statistics collection:

Note : These instructions for setting up SUMS statistics collection makes the assumption the user has some Unix, MySQL, Tomcat and SUMS skills.

Building the statistics tables:

First a data base must be set up for collecting the statistics. You may use a preexisting data base or set one up form scratch. SUMS requires MySQL 4.x or above. You will need to be able to Log in as the administrator. A script to create the required tables can be found eider in the SUMS precompiled build or in CVS under the path CVSROOT/scheduler/res/ext/ the name of the script is createStatDB.sql. Once the script is run the following tables should exist:

Job_03, Job_04, Job_05, Job_06, Job_07, Job_08, Request_03, Request_04, Request_05, Request_06, Request_07, Request_08, InputFile, Job_all, Request_all

The Job_yy (where yy is the year) is used for holding information about jobs. The scheduler will make one entry for every job created. It will look at the system clock on the server the scheduler is running on to determine the year and it will write into the table with the corresponding year. After the current year has passed the data for that year may be compressed to save space. Under MySQL (at the time of this writing) once a table is compressed it may still be queried, however it cannot be written to any more. The request tables work in the same way and the information applies as with the job table. One entry will be made every time a user runs a request in this table. The InputFile table gets an entry for every file in the users process. This table can get very big very fast. In the current version of the JSPs it is not used, so data in this table may be backed up and purged if it gets too big. The Job_all and Request_all are virtual tables. They are generated on the fly by concatenating all the years together. In other words the Request_all contains the sum of all the Request_yy tables together.

You may also what to create a separate account for the scheduler. That can both read and create entries in the tables Job_yy, Request_yy and InputFiles. It is best to limit the rights on this account as the username and password is stored in plane text.

Note: If you are using an older version of scheduler statistics please back up any tables with conflicting names in the same database. A script will be available for converting the old tables into the new format.

Configuring the scheduler:

Next the scheduler’s configuration file must be set up, so SUMS knows where to write its statistics information to. This is done in the scheduler’s java serialized XML configuration file. Open the configuration file (see note below if you are having difficulty finding it as there may be more then one) and find the statistics tag. If for some reason there is no statistics tag, just copy the example tag in the example below into the config file.

<void method="put">
      <string>statistics</string>
      <object class="java.util.Hashtable">
               <void method="put">
                     <string>URL</string>
                     <string>jdbc:mysql://jeremy.star.bnl.gov/stats_Scheduler</string>
               </void>
               <void method="put">
                     <string>password</string>
                     <string>!stats4sums@PDSF&BNL</string>
               </void>
               <void method="put">
                      <string>username</string>
                      <string>sums</string>
               </void>
        </object>
  </void>

Note (identifying the configuration file): It may sometimes be hard to identify the configuration file being used if there are multiple files present. The “star-submit” script that calls the scheduler will have as the first argument to the class the configuration file used. It is usually in the same directory (folder) as the scheduler.jar with the extension “.xml” and the first line after the generic xml version tag and comments should be <java version="1.4.2" class="java.beans.XMLDecoder"> if it is a valid configuration file.

Getting the WAR file:

Note: These instructions are for NetBeans IDE 3.6. later version may be different.

The web site for displaying the statistics comes as one war file that Tomcat can expand when a person requests to view a page on the site. The war file is not included in the stander scheduler build. It is recommended you request the file pre-built by contacting me directly (see e-mail link at bottom of page).

If you wish to build the war file your self, you must check it out of CVS as a project separate from the scheduler, see (put link here, for checking out of CVS).

The root of the project must be /data01/CVS/scheduler/statwww/. In this path there will be a file named WEB-INF, right click on this file. A dropdown menu will appear select “Export WAR File …”. A window will appear asking you where you want to save the file to. It may be saved anywhere however Tomcat must have access to it. The name of the WAR file can be anything you wish however for these examples we will be using the name “schedstat.war”. If you choose something different the name must be kept consistent in the Tomcat configuration file, to be described later.

Installing the war file in tomcat:

Note: The installation of Tomcat its self is not covered in the scope of this document.

These instruction where written with Apache Jakarta Tomcat version 5.5.4 (it has proved to be very stable). The syntax of the tomcat configuration file changes with every version so it is a good idea to use the recommended version.

After the tomcat server is installed. The war file will need to be installed.

Edit the file $CATALINA_HOME/conf/server.xml. In the <Host> tags add a new context for the war file. Note the JSPs also need database access, so the same data base information (URL, password, username) will need to be set here in the required format. In addition the path to the war file will need to be set.

Use this context for Apache Jakarta Tomcat version 5.5.4:

<Context path="/apps/schedStat" docBase="/var/tomcat5/webapps/schedstat.war" debug="5" reloadable="true" crossContext="true">
                        <Environment name="siteName" override="true" type="java.lang.String" value="BNL"/>
                        <Resource name="jdbc/SchedStatDB" auth="Container" type="javax.sql.DataSource" maxActive="100" maxIdle="30" maxWai
t="10000" username="sums" password="!stats4sums@PDSF&BNL" driverClassName="com.mysql.jdbc.Driver" url=" jdbc:mysql://jeremy.star.bnl.gov/stats_Scheduler?autoReconnect=true"/>
                </Context>

Use this context for Apache Jakarta Tomcat version 4:

<Context className="org.apache.catalina.core.StandardContext" cachingAllowed="true" charsetMapperClass="org.apache.catalina.util.CharsetMapper" cookies="true" crossContext="false" debug="0" docBase="/var/tomcat4/webapps/schedstat.war" mapperClass="org.apache.catalina.core.StandardContextMapper" path="/apps/schedStat" privileged="false" reloadable="true" swallowOutput="false" useNaming="true" wrapperClass="org.apache.catalina.core.StandardWrapper">
          <Environment name="siteName" override="true" type="java.lang.String" value="BNL"/>
          <Resource auth="Container" description="DB Connection" name="jdbc/SchedStatDB" scope="Shareable" type="javax.sql.DataSource"/>
          <ResourceParams name="jdbc/SchedStatDB">
            <parameter>
              <name>validationQuery</name>
              <value>show create table Request</value>
            </parameter>
            <parameter>
              <name>url</name>
              <value>jdbc:mysql://duvall.star.bnl.gov/Scheduler_bnl?autoReconnect=true</value>
            </parameter>
            <parameter>
              <name>password</name>
              <value>4stats!</value>
          </parameter>
            <parameter>
              <name>maxActive</name>
              <value>50</value>
            </parameter>
            <parameter>
              <name>maxWait</name>
              <value>5000</value>
            </parameter>
            <parameter>
              <name>driverClassName</name>
              <value>com.mysql.jdbc.Driver</value>
            </parameter>
            <parameter>
              <name>username</name>
              <value>scheduler</value>
           </parameter>
            <parameter>
              <name>maxIdle</name>
              <value>5</value>
            </parameter>
          </ResourceParams>
        </Context>

After the file has been configured save it, start Tomcat and verify that you can view the graphs in the web pages, if so then you have successfully completed the setup.

If the page can not be viewed at all, the cause is most likely related to the war file and Tomcat. Check that Tomcat as running and that the path to the war file is right. If you can see the page but the graphs d not render, the cause is most likely related to the database. Verify that the account information is correct.

Note: If you decide to deploy again don’t forget to purge the schedStat folder in $CATALINA_HOME/webapps/apps/ .

Levente Hajdu - page was last modified