Reference

class obob_condor.JobCluster(required_ram='2G', adjust_mem=True, request_cpus=1, jobs_dir='jobs', inc_jobsdir=True, owner=None, python_bin=None, working_directory=None, singularity_image=None)

This is the main class, the controller of obob_condor. It collects all the jobs and takes care of submitting them to the cluster. It also contains information about how much RAM the jobs need, how many CPUs are requested etc.

Parameters

• required_ram (str, float, int, optional) – The amount of RAM required to run one Job in megabytes. A string like “2G” or “200M” will be converted accordingly.

• adjust_mem (bool, optional) – If True, the job will be restarted automatically if it gets killed by condor because it uses too much RAM.

• request_cpus (int, optional) – The number of CPUs requested

• jobs_dir (str, optional) – Folder to put all the jobs in. This one needs to be on the shared filesystem (so somewhere under /mnt/obob)

• inc_jobsdir (str, optional) – If this is set to True (default), jobs_dir is the parent folder for all the jobs folders. Each time a job is submitted, a new folder is created in the jobs_dir folder that contains all the necessary files and a folder called “log” containing the log files. If jobs_dir is set to False, the respective files are put directly under jobs_dir. In this case, jobs_dir must either be empty or not exist at all to avoid any side effects.

• owner (str, optional) – Username the job should run under. If you submit your jobs from one of the bombers, you do not need to set this. If you have set up your local machine to submit jobs and your local username is different from your username on the cluster, set owner to that username.

• python_bin (str, optional) – The path to the python interpreter that should run the jobs. If you do not set it, it gets chosen automatically. If the python interpreter you are using when submitting the jobs is on /mnt/obob/ that one will be used. If the interpreter you are using is not on /mnt/obob/ the default one at /mnt/obob/obob_mne will be used.

• working_directory (str, optional) – The working directory when the jobs run.

• singularity_image (str, optional) – Set this to a singularity image to have the jobs execute in it. Can be a link to a local file or to some online repository.

add_job(job, *args, **kwargs)

Add one job to the JobCluster. All further arguments will be passed on to the Job.

Parameters

• job (child of obob_condor.Job) – The job class to be added.

• *args – Variable length argument list.

• **kwargs – Arbitrary keyword arguments.

run_local()

Runs the added jobs locally.

submit(do_submit=True)

Runs the added jobs on the cluster.

Parameters

do_submit (bool, optional) – Set this to false to not actually submit but prepare all files.

class obob_condor.Job(*args, **kwargs)

Abstract class for Jobs. This means, in order to define you own jobs, they need to be a subclass of this one.

You must implement (i.e. define in your subclass) the run() method. The run method can take as many arguments as you like. Only the types of arguments are restricted because they need to be saved to disk. In general, strings, numbers, lists and dictionaries are fine.

You can implement shall_run(). This can be used to see whether some output file already exists and restrict job submission to missing files.

run(*args, **kwargs)

Implement this method to do the job.

shall_run(*args, **kwargs)

This is an optional method. It gets called with the same arguments as the run() method, before the job is submitted. If it returns True, the job is submitted, if it returns False, it is not.

class obob_condor.AutomaticFilenameJob(*args, **kwargs)

Bases: Job

Abstract class for Jobs providing automatic filename generation.

In order for this to work, you need to:

1. Set base_data_folder and job_data_folder as a class attribute.

2. If you use shall_run(), you need to do the super call.

This class then automatically creates the filename for each job using all the keyword arguments supplied.

Please take a look at Automatically generating filenames for your jobs for detailed examples.

Variables

• base_data_folder (str or pathlib.Path) – The base folder for the data. Is normally set once for all jobs of a project.

• job_data_folder (str or pathlib.Path) – The folder where the data for this job should be saved.

• exclude_kwargs_from_filename (list) – Normally, all keyword arguments are used to build the filename. if you want to exclude some of them, put the key in the list here.

• include_hash_in_fname (bool) – Include a hash of all arguments in the filename. This is helpful if you excluded some keyword arguments from filename creation but still need to get distinct filename.

• run_only_when_not_existing (bool) – If true, this job will only run if the file does not already exist.

• create_folder (bool) – If true, calling folders are created automatically

• data_file_suffix (str, optional) – The extension of the file. Defaults to .dat

classmethod get_full_data_folder()

Return the data folder for this job (i.e. base_data_folder plus job_data_folder).

shall_run(*args, **kwargs)

This is an optional method. It gets called with the same arguments as the run() method, before the job is submitted. If it returns True, the job is submitted, if it returns False, it is not.

property full_output_path

The full path to the output file.

Type

pathlib.Path

property output_filename

The filename for this subject.

Type

str

property output_folder

The output folder for this subject.

Type

pathlib.Path

class obob_condor.PermuteArgument(args)

This is a container for to-be-permuted arguments. See the example in the introductions for details.