Using Ganga to submit grid ntupling jobs
This is a set of instructions for how to use ganga to submit ntupling jobs using AODtoLiverpoolNtuple. I did this at CERN, but there shouldn't really be any reason for this not to work at Liverpool (barring a few changes in the initial setup).
At the moment, the setup is rather complicated, and the submission script is pretty basic. In principle, the same thing could be done more simply by using a job template, or a script which takes arguments such as the DQ2 dataset to use as input.
Step 1: Set up the grid
For lxplus, see the ATLAS Workbook
. You need to set up the grid, DQ2 and ganga. Ganga heavily uses a directory called
. Before using ganga for the first time, you may want to get some scratch space and create this directory as a soft link to
To get scratch space (extra space which is not backed up), email atlas-support@cernREMOVETHIS.ch
and ask for it. Then, create your soft link:
ln -s ~/scratch0/gangadir ~/gangadir
Step 2: Avoiding CASTOR
Castor is basically dead, but ganga tries to write output there by default. One way around this (under test!) would be to use a different grid Storage Element. I've set up a directory at Manchester, with the following command:
The bit after the
is the srm directory, which I'll write as
for the rest of the page.
Step 3: Prepare ganga
The first time you run ganga, it creates the file
. This needs to be edited, most of these should just involve uncommenting a line, line numbers are approximate:
- Line 33
RUNTIME_PATH = GangaAtlas
- Line 227
VirtualOrganisation = atlas
- Line 275
LCGOutputLocation = srm://<SE>/<dir> NB: This doesn't seem to work for me, see below.
- Line 280
ATLAS_SOFTWARE = /afs/cern.ch/project/gd/apps/atlas/slc3/software
With the current version (4.2.X) of ganga, jobs will systematically fail at certain grid nodes (for example,
) with the error
curl: command not found
. This will be fixed, but for now lines such as
ExcludedCEs = ic\.ac\.uk
need to be added in the LCG section of
. I am still testing this, please let me know if it works for you.
Step 4: Customising the submission script
Now look at the attachments on this page. There are only three files, and this is all you need. They are:
- SubmitGanga.py Delete the
.txt from the filename. This is the script that is used to submit the job.
- AODtoLiverpoolNtuple_JobOptions.py Again, delete the
.txt. These are the jobOptions I use, or use the one in CVS.
- AODtoLiverpoolNtuple-00003.tar.gz This contains all the source code and is sent off to define the job. Although this is based on the job.prepare() method in ganga, it is not the same, it contains a small but necessary change to make it work at a remote site.
So, taking the
file, a few small changes have to be made:
- Lines 16 and 18: Point these towards wherever you have the tarball and jobOptions, respectively
- Line 25: Replace with the name of the DQ2 dataset you want to ntuple
- Line 12: Change as appropriate, depending on the number and size of the AODs...
- Line 34: Point this towards
srm://<SE>/<dir>/ which you chose in step 2.
- Line 37: This name must match the ntuple name given in the job options, or the output will not copy from the CE to the SE - if you are using the job options on this page, no change is needed. The name
Ntuple.root does not need to change from job to job - when copying the ntuples from the grid, you have to specify a full local filename anyway, and so you can easily call the file whatever you like at that stage.
Step 5: Submitting!
From within ganga:
Keep tabs on your jobs by typing
for more detail, where
is the job number. For information on an individual subjob, type
... You get the idea.
With any luck, the job will work, and output data will be deposited in the directory you set up in steps 2 and 3. See this page
for ways of seeing and copying this data back to Liverpool.
- 22 Mar 2007