Introduction
The STAR scheduler needs access to monitoring data for decision making
mechanisms. The average load per CPU per cluster is, for example, a
parameter that can be used to decide where jobs will be submitted.
We have currenlty integrated the ML WS Client into the scheduler. The
scheduler can pull data from individual ML monitoring sites that belong
to the star group and have been configured to provide access
to monitoring data as a Web Service.
We are
looking into having the integrated Client pull data from the Repository
rather than each individual site. This change should be trivial: all we
have to do is change to URL to that of the Repository and thus have
access to monitoring data from all sites instead of quering each site
individually.
We are looking into the idea of integrating a pseudo-client into the
scheduler. Real-time data do make it to the "pseudo-client" (and are
available for real-time plots). Thus, integrating the "pseudo-client"
with the scheduler should provide us with un-mediated, real-time data.
Also, it should be faster (we won't be accessing the repository over the
network, no
SQL queries to be executed) and more reliable (no single point of failure).
Using the ML Web Service Client
One method to provide to the Scheduler access to the ML monitoring data
is through the ML Web Service Client. The WS Client can access data
in ML monitoring services that have been started as a Web Service and in
Web Repositories.
The getValues function
This function translates the requestes
to SQL queries that pull the data from the monitor_1hour DB table
which provides mediated monitoring data. For more on this, read:
Accessing the Web Repository
The getValues function is available at the WS running at the
local ML monitoring services and the ML Web Repositories. But using this
WS function to access data from individual ML Services is not a scalable
solution. Every time a new site is added new adjustments need to be made
to include the new service in the loop. Even using this function to access
the data from the Web Repository does not seem to be a good solution.
The Client requests are translated into SQL queries for the
DB tables. So you are actually accessing averaged, historical
data rather than raw, real-time data. The retrieved data
may be delayed up to one minute due to the buffering and averaging of the data
retrieved from the individual monitoring sites. Also, accessing data from a
single point, may make this point a single point of failure: if the
repository becomes unavailable, no monitoring data will be available.
The getLastValues and getFilteredLastValues functions
The developers
have provided two more functions: getLastValues and
getFilteredLastValues. These two WS functions are available
at the Web Repository only. They both provide un-mediated data directly
from the data cache. So, no SQL queries are involved. The difference
between the two functions is the getLastValues function takes
no arguments, so all the data in the data cache of the Web Repository
are returned, while the getFilteredLastValues takes four String
arguments (the Farm, the Cluster, the node and the function). So, using the
getFilteredLastValues function tends to provide results faster.
Stratos Efstathiadis - page was last modified
