gov.bnl.star.offline.scheduler.lsf
Class LSFResourceStrategy

java.lang.Object
  gov.bnl.star.offline.scheduler.lsf.LSFResourceStrategy

public class LSFResourceStrategy

extends java.lang.Object

Encapsulate the resource usage parameter (-R) for an LSF farm. Essentially, its a placeholder for the prepareResourceUsageSwitch() which will calculate the appriate string. The administrator should be able to create different subclasses for each farm as needed.

This class provides a scheleton for a typical resource usage for files residing on different NFS servers. It was designed to fit STAR needs at two sites. In case it doesn't fit the needs, the class can be subclassed to define new behaviour.

The resource switch is prepared in this way: Takes the list of input files and iterates over them.
Checks if each file needs a resource (resourceRequired()).
To determine that, checks if the file is on NFS, and looks for the typical data vault prefix (getDataVaultPrefix). For example, "/star/dataxx/..."; it extract the vaultNumber and it substitute it in the resourceSyntax
Increases the count of the resource (increaseResourceUsage)
Prepares the switch (prepareSwitch)

The resource value for each resource will the be given by the following formula:
min (base+nCount*fileFactor, max)

The subclass need to implement the abstract methods, which defines things that always change between different farms.

TODO Further work should be done to be able to configure the ResourceStrategy depending on the farm on which the job will be dispatched. This should be included in the GRID dispatchers.

Version:

July 24, 2003

Author:

Gabriele Carcassi, Jerome Lauret

Field Summary

protected int	base Base value for the resource value formula.
protected int	fileFactor FileFactor value for the resource value formula.
protected int	max Max parameter for the standard increaseResourceUsage implementation.

Constructor Summary

LSFResourceStrategy()

Method Summary

protected void	addResource(java.lang.String resName, int base) Addes a new resource, and initializes the value to base.
protected void	calculateLSFResources(Job job) Filles the resource table with all the resources.
protected java.lang.String	definedNames() A list of names, comma separated, of all the resources used.
protected boolean	determineResourceNotUsed() Determines whether any resources at all have been assigned.
int	getBase() Returns the base value for the resource value formula.
java.lang.String	getDataVaultPrefix() Returns the prefix with which a file on NFS identifies on which file server or data vault is residing.
int	getFileFactor() Returns the fileFactor value for the resource value formula.
int	getMax() Returns the max value for the resource value formula.
java.lang.String	getResourceNameSyntax() Returns the syntax to be used to convert a data vault number to the LSF resource.
java.lang.String	getSwitchSyntax() Returns the syntax that will be used to prepare the final switch
protected int	increaseResource(java.lang.String resName, int base, int amount) Increases the value associated to the given resource by a given amount.
protected int	increaseResourceUptoLimit(java.lang.String resName, int base, int amount, int limit) Increases the value associated to the given resource by a given amount.
protected void	increaseResourceUsage(java.lang.String resource) Increase the resource usage as part of the standard implementation.
protected void	initResources() Initializes the resource table.
boolean	isRemoveCharactersFromVaultNumber() If true removes all the characters from the vault number.
boolean	isTrimVaultNumberDecimal() Getter for property trimVaultNumberDecimal.
boolean	isTrimVaultNumberLeadingZeros() Returns true if the leading zeros of a vault number will be removed when creating the corresponding LSF resource.
protected java.lang.String	nameEqualValueColumnSeparated() Returns a string already formatted containing names and values column separated (i.e.
protected java.lang.String	nameEqualValueCommaSeparated() Returns a string already formatted containing names and values comma separated (i.e.
protected java.lang.String	prepareResourceName(java.lang.String vaultNumber) Given the vault id, returns the LSF resource name to be used in the -R parameter.
java.lang.String	prepareResourceUsageSwitch(Job job) Returns the -R parameter to be used to dispatch the given job.
protected int	prepareResourceValue(java.lang.String resName) Returns the value currently associated to the resource.
protected java.lang.String	prepareSwitch() Prepares the -R parameters to give to LSF at the end of the resource distribution.
protected java.lang.String	resourceRequired(PhysicalFile file) Determines which resource the job might need to access the file.
void	setBase(int base) Changes the base value for the resource value formula.
void	setDataVaultPrefix(java.lang.String dataVaultPrefix) Changes the prefix of a NFS file.
void	setFileFactor(int fileFactor) Changes the fileFactor value for the resource value formula.
void	setMax(int max) Changes the max value for the resource value formula.
void	setRemoveCharactersFromVaultNumber(boolean removeCharactersFromVaultNumber) Set to true removes all the characters from the vault number
void	setResourceNameSyntax(java.lang.String resourceNameSyntax) Changes the syntax to be used to convert a data vault number to the LSF resource.
void	setSwitchSyntax(java.lang.String switchSyntax) Changes the syntax that will be used to prepare the final switch.
void	setTrimVaultNumberDecimal(boolean trimVaultNumberDecimal) Setter for property trimVaultNumberDecimal.
void	setTrimVaultNumberLeadingZeros(boolean trimVaultNumberLeadingZeros) Set to true if the leading zeros of a vault number will be removed when creating the corresponding LSF resource.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

base

protected int base

Base value for the resource value formula.

fileFactor

protected int fileFactor

FileFactor value for the resource value formula.

max

protected int max

Max parameter for the standard increaseResourceUsage implementation.

Constructor Detail

LSFResourceStrategy

public LSFResourceStrategy()

Method Detail

getBase

public int getBase()

Returns the base value for the resource value formula.

setBase

public void setBase(int base)

Changes the base value for the resource value formula.

getFileFactor

public int getFileFactor()

Returns the fileFactor value for the resource value formula.

setFileFactor

public void setFileFactor(int fileFactor)

Changes the fileFactor value for the resource value formula.

getMax

public int getMax()

Returns the max value for the resource value formula.

setMax

public void setMax(int max)

Changes the max value for the resource value formula.

getDataVaultPrefix

public java.lang.String getDataVaultPrefix()

Returns the prefix with which a file on NFS identifies on which file server or data vault is residing. For example, "/star/data07/muDST/..." represents a file on data vault "07" (that is the data vault number or id), therefore the prefix is "/star/data".

Returns:

The prefix that identifies data vaults on NFS.

setDataVaultPrefix

public void setDataVaultPrefix(java.lang.String dataVaultPrefix)

Changes the prefix of a NFS file. The resource identifies the data vault number by looking at a prefix in the file name. For example "/star/data07/minbias/..." would have "/star/data" as a prefix.

getResourceNameSyntax

public java.lang.String getResourceNameSyntax()

Returns the syntax to be used to convert a data vault number to the LSF resource.

setResourceNameSyntax

public void setResourceNameSyntax(java.lang.String resourceNameSyntax)

Changes the syntax to be used to convert a data vault number to the LSF resource. It can be any string, and it should contain the $vaultNumber variable, which will be substituted with the particular vault number. For example, to have something like "dv7io", you should have "dv$vaultNumberio".

isTrimVaultNumberLeadingZeros

public boolean isTrimVaultNumberLeadingZeros()

Returns true if the leading zeros of a vault number will be removed when creating the corresponding LSF resource.

setTrimVaultNumberLeadingZeros

public void setTrimVaultNumberLeadingZeros(boolean trimVaultNumberLeadingZeros)

Set to true if the leading zeros of a vault number will be removed when creating the corresponding LSF resource. Basically, 07 will be changed to 7.

prepareResourceName

protected java.lang.String prepareResourceName(java.lang.String vaultNumber)

Given the vault id, returns the LSF resource name to be used in the -R parameter.

Parameters:

vaultNumber - the vault id, as identified in getDataVaultPrefix

Returns:

the LSF resource name for the vault

getSwitchSyntax

public java.lang.String getSwitchSyntax()

Returns the syntax that will be used to prepare the final switch

setSwitchSyntax

public void setSwitchSyntax(java.lang.String switchSyntax)

Changes the syntax that will be used to prepare the final switch. You can use the $definedNames, $nameEqualValueCommaSeparated or $nameEqualValueColumnSeparated variables in your switch. For example, the following switch: "\"select[$definedNames] rusage[$nameEqualValueCommaSeparated]\"" will produce something like: "select[defined(01)] rusage[01=50]"

prepareSwitch

protected java.lang.String prepareSwitch()

Prepares the -R parameters to give to LSF at the end of the resource distribution. It will use definedNames(), nameEqualValueCommaSeparated() or nameEqualValueColumnSeparated() to prepare it more easily.

Returns:

the -R parameter for the current job

increaseResourceUsage

protected void increaseResourceUsage(java.lang.String resource)

Increase the resource usage as part of the standard implementation. The standard formula starts counting from base, adds fileFactor for each usage until max is reached. In pseudo-code: if (nRes == 0) nRes = base; nRes += nRes; if (nRes > max) nRes = max;

One can modify the three protected parameters (base, fileFactor, max) to tune the usage.

Parameters:

resource - the resource for which to increase the counting.

addResource

protected void addResource(java.lang.String resName,
                           int base)

Addes a new resource, and initializes the value to base.

Parameters:

resName - the name of the resource

base - the initial value for the resource

increaseResource

protected int increaseResource(java.lang.String resName,
                               int base,
                               int amount)

Increases the value associated to the given resource by a given amount. If the resource doesn't exists, first it will be created and initialized.

Parameters:

resName - the name of the resource

base - the initial value if the resource has to be created

amount - the amount to increase the resource by

Returns:

the new value associated to the resource

increaseResourceUptoLimit

protected int increaseResourceUptoLimit(java.lang.String resName,
                                        int base,
                                        int amount,
                                        int limit)

Increases the value associated to the given resource by a given amount. If the resource doesn't exists, first it will be created and initialized. If the increased amount is over the limit, the value will be set to the limit.

Parameters:

limit - upper limit to which the resource value will be set

resName - the name of the resource

base - the initial value if the resource has to be created

amount - the amount to increase the resource by

Returns:

the new value associated to the resource

prepareResourceValue

protected int prepareResourceValue(java.lang.String resName)

Returns the value currently associated to the resource.

Parameters:

resName - resource name

Returns:

resource value

prepareResourceUsageSwitch

public java.lang.String prepareResourceUsageSwitch(Job job)

Returns the -R parameter to be used to dispatch the given job.

Parameters:

job - the job to be dispatched

Returns:

the -R parameter to use

initResources

protected void initResources()

Initializes the resource table.

nameEqualValueCommaSeparated

protected java.lang.String nameEqualValueCommaSeparated()

Returns a string already formatted containing names and values comma separated (i.e. "res1=34,res2=12").

Returns:

resource name/value pairs comma separated (i.e. "res1=34,res2=12")

nameEqualValueColumnSeparated

protected java.lang.String nameEqualValueColumnSeparated()

Returns a string already formatted containing names and values column separated (i.e. "res1=34:res2=12").

Returns:

resource name/value pairs column separated (i.e. "res1=34:res2=12")

definedNames

protected java.lang.String definedNames()

A list of names, comma separated, of all the resources used.

Returns:

(i.e. "res1,res2")

calculateLSFResources

protected void calculateLSFResources(Job job)

Filles the resource table with all the resources. It basically iterates over the input files and calls increaseResourceUsage for each file that map to a resource.

Parameters:

job - the job

resourceRequired

protected java.lang.String resourceRequired(PhysicalFile file)

Determines which resource the job might need to access the file.

Parameters:

file - input file the job will be using

Returns:

the resource needed

determineResourceNotUsed

protected boolean determineResourceNotUsed()

Determines whether any resources at all have been assigned.

Returns:

true if any resource is used.

isTrimVaultNumberDecimal

public boolean isTrimVaultNumberDecimal()

Getter for property trimVaultNumberDecimal.

Returns:

Value of property trimVaultNumberDecimal.

setTrimVaultNumberDecimal

public void setTrimVaultNumberDecimal(boolean trimVaultNumberDecimal)

Setter for property trimVaultNumberDecimal.

Parameters:

trimVaultNumberDecimal - New value of property trimVaultNumberDecimal.

isRemoveCharactersFromVaultNumber

public boolean isRemoveCharactersFromVaultNumber()

If true removes all the characters from the vault number.

Returns:

Value of property removeCharactersFromVaultNumber.

setRemoveCharactersFromVaultNumber

public void setRemoveCharactersFromVaultNumber(boolean removeCharactersFromVaultNumber)

Set to true removes all the characters from the vault number

Parameters:

removeCharactersFromVaultNumber - New value of property removeCharactersFromVaultNumber.