How to use obob_condor

Submitting Jobs to the OBOB cluster is quite simple using the obob_condor package.

Make sure you are on your bomber

Accessing the cluster requires the code to be run on the bomber as only those are connected to the cluster.

All the code also needs to be on /mnt/obob because the nodes also need access to them.

You also need to be in a python environment that has obob_condor in its dependencies. If you do not now what this means, please take a look at How to create a python environment for you cluster jobs.

Define a simple job

The first thing you need to do is define what your job should do. You therefore write a class that derives from obob_condor.Job. The only things you have to do is to supply a run method.

import obob_condor

class MyJob(obob_condor.Job):
    def run(self):
        print('Hello World!')

The job class can be anywhere in your sourcecode tree. It can be defined in the script you are running, in a python module or package. As long as you can import it, it is ok.

Define a job that takes arguments

Your job can also take any kind of arguments:

import obob_condor

class JobWithArgs(obob_condor.Job):
    def run(self, normal_arg, key_arg='my_default'):
        print('normal_arg=%s\nkey_arg=%s' % (str(normal_arg), str(key_arg)))

Getting and configuring the JobCluster

In order to submit your job, you need to get an instance of obob_condor.JobCluster. The constructor of this class has a lot of keyword arguments. You can set none, some or all of them. They all have quite sensible defaults:

import obob_condor

my_jobs = obob_condor.JobCluster(required_ram='6G')

Now we have a JobCluster that asks for 6GB of RAM per Job.

Adding jobs to the JobCluster

In order to add the jobs, use the obob_condor.JobCluster.add_job() method:

my_jobs.add_job(MyJob)
my_jobs.add_job(JobWithArgs, 'this_is_the_normal_arg', key_arg='and this the key arg')

Adding multiple jobs with just one call

A common use case of job submission is that you want to run the same job on a number of different combinations of parameters.

Let’s consider a job like this:

class AverageData(obob_condor.Job):
    def run(self, subject_id, condition, lp_filter_freq)
        ...

And you have a list of subject_ids and conditions:

subject_ids = [
    '19800908igdb',
    '19990909klkl',
    '17560127anpr']

conditions = [
    'visual',
    'auditory']

We want to run the jobs for all combinations of subject_ids and conditions. This is what obob_condor.PermuteArgument is for:

from obob_condor import PermuteArgument

my_jobs.add_job(AverageData, PermuteArgument(subject_ids), PermuteArgument(conditions), 30)

This call adds 6 jobs, one for every combination of subject_ids and conditions.

This works for all kinds of arguments (normal ones and keyword arguments).

Submitting the Job

Now, all you need to do is to call submit:

my_jobs.submit()

For more advanced uses take a look at the Reference.