Proposed architecture

This is the architecture we use to wrap the Wayne scheduler. We will use the same architecture to wrap around the next chosen scheduling tool. The effort is to provide forward compatibility, and to save as much work as possible. The architecture tries to decouple aspects of the scheduling that we will want to change in the future.

Scheduling phases

The scheduling process is divided into stages. To each stage will correspond an interface (a pure abstract class).

  1. Job Initialization: the XML job description is read and validated. The XML is parsed, and a memory representation of the job is created. Validation may include whether the input files exists, the user name is allowed to scheduler or if the catalog query returns values.
    The proposed interface is simply:
    public interface JobInitializer {
    JobRequest[] analizeRequest(String xmlFileName);
  2. Policy: decides where and how the job will run. The policy decides how to split the job, and on which machine it will be executed. It will do so by checking the status of the farm, querying the file catalog and the like. The intention is to provide an easy and powerful way for the farm manager to modify the behaviour of the scheduler.
    The proposed interface is simply:
    public interface Policy {
    void assignTargetMachine(JobRequest job);
  3. Dispatcher: prepares the environment for the jobs and forwards the request to the underlying technology. We may divide this part into three parts:

    In the staging part, the dispatcher creates the environment for the job. That is, create the file list, the environment variables and copies the file, depending on the way the job will be executed.
    In the execution part, the underlying scheduler (LSF, Condor...) is called.
    In the de-staging part, the necessary job cleanup or copies are done. In some cases this part might not be necessary.
    The proposed interface is simply:
    public interface Dispatcher {
    void dispatch(JobRequest job);
    void retrieveOutput(JobRequest job);

The details of JobRequest class, and other classes that might be necessary, is still under development.

Gabriele Carcassi - page was last modified