STAR  Computing  GRID  Scheduler  Development Overview of the project

The first thing one should have clear is that the scheduler is a client tool. That is, the user executes the scheduler giving the xml request, the scheduler produces the scripts, submits the job and then exits. We say client in the sense that there is no component of the scheduler that sits running and waits for requests. Future work will be done to bring the scheduler to the server, more specifically in the form of a web service.

A consequence of this is that, for now, the scheduler is only a submission tool. Job monitoring and other components require a server component.

The other component that we provide with the scheduler is the statistics, which is a server component based on a database and a web server.

Terminology

Request/Jobs and Job/Processes

There are a couple of issues with the terminology due to the evolution of the project. The main one has to do with Request/Job and Job/Process. The scheduler is going to get an XML request and is going to submit jobs to the batch system. This is the terminology used in the code and in the documentation. Historically, though, we hadn't clarified well the concept of request/job, and we used processes to refer to the jobs and job to refer to the request. Unfortunately this terminology is still present in the user interface, and in the XML itself. Don't make that confuse you! :-)

I should have eliminated the old terminology from most of the code and the documentation. If you find it, please, either make the maintainer aware or fix it right away.

Star-submit entry point

The star-submit command line tool is essentially the main method of the SchedulerCommandLine class. It will set the logging parameters, read the configuration file and build the scheduler components, assemble the scheduler and submit the file.

The scheduler configuration mechanism is located in the ComponentLibrary class. This is initialized with the XML, which is read with the Bean XML Deserialization, and it will create a Map (Hashtable) mapping names to objects. Some names, like "defaultDispatcher" or "defaultNames", are special, and identify key components. In the configuration one can specify extra components, with any name preferred. These components will be accessible at the command line through -policy and -dispatcher.

JobInitializer/Policy/Dispatcher

The submission process is divided into three steps. To each of these steps correspond an interface the implementation of which is defined at runtime.

A Scheduler is divided in three parts:

• JobInitializer - It receives the request, parses it and creates data objects describing the request. Few functionalities are included in this step: the wildcard resolution and the input filelists implementation are two examples. Some other include the validation of the request itself.
• Policy - The policy converts a request into a series of jobs. To do this, it might have to communicate to different middleware (i.e. file catalog, monitoring tools, ...), therefore the crucial part of the planning is done in this step.
• Dispatcher - The dispatcher takes the jobs, creates all scripts and file needed and submits them. In the dispatcher one is meant to hide all of the cluster/site specifics. It can be the batch system itself (i.e. LSF/Condor/Condor-G/...) or the details of the cluster (i.e. resource mapping specific to the cluster/site).

The main package basically includes the definition of these interfaces plus the definition of the data classes

Gabriele Carcassi - page was last modified