Difference: AtlasGanga (1 vs. 6)

(a) Setup Grid environment

In order to use GANGA you should run the following commands in a clean shell.

Set up the grid environment and create a new proxy

• voms-proxy-init -voms atlas

• source /home/robinson/athena/DiJets/15.6.7/cmthome/setup.sh

(c) Setup GANGA

Now source the GANGA setup

• source /afs/cern.ch/sw/ganga/install/etc/setup-atlas.sh

Running GANGA

• execfile('')

Using GANGA with the LCG backend

• lcg-infosites --vo atlas

Using GANGA with the LCG backend

Using the Panda backend

The Panda backend has to be used for jobs sent to US sites. It requires a slightly different form of job submission script. A sample job script is shown here:

j = Job()
j.application = Athena()
j.name='PTResolution.LowPT.SmallEta.J5'
j.application.option_file=[ '/home/robinson/athena/15.6.0/PhysicsAnalysis/ForwardJets/run/jobOptions.PTResolution.LowPT.SmallEta.py' ]
j.application.athena_compile = True
j.application.atlas_release='15.6.0'
j.application.prepare()
j.inputdata=DQ2Dataset()
j.inputdata.dataset=[ 'mc08.105014.J5_pythia_jetjet.merge.AOD.e344_s479_s520_r809_r838/' ]
j.outputdata=DQ2OutputDataset()
j.splitter=DQ2JobSplitter()
j.splitter.numsubjobs=500
j.backend=Panda()
j.submit()

Using the NorduGrid backend

If submitting jobs to NorduGrid in the Netherlands there is only one cloud. Therefore, the your script needs to be of the following form, changing the backend to 'NG'.

j = Job()
j.application = Athena()
j.name='PTResolution.LowPT.SmallEta.J5'
j.application.option_file=[ '/home/robinson/athena/15.6.0/PhysicsAnalysis/ForwardJets/run/jobOptions.PTResolution.LowPT.SmallEta.py' ]
j.application.athena_compile = True
j.application.atlas_release='15.6.0'
j.application.prepare()
j.inputdata=DQ2Dataset()
j.inputdata.dataset=[ 'mc08.105014.J5_pythia_jetjet.merge.AOD.e344_s479_s520_r809_r838/' ]
j.outputdata=DQ2OutputDataset()
j.outputdata.outputdata=['PTResolution.root']
j.splitter=DQ2JobSplitter()
j.splitter.numsubjobs=500
j.backend=NG()
j.submit()

• force a job into a particular status: jobs(jobid).force_status("failed") The repository for input/output files for every job is located by default at: $HOME/gangadir/workspace/username/LocalAMGA

• dq2-ls -r "datasetname" (outside ganga)

• lcg-infosites --vo atlas ce tag -- JamesRobinson - 13 Nov 2009

How to setup GANGA

• source ~/cmthome/setup.sh -tag=15.6.0

• ganga

Using GANGA

The ganga command line is a python shell which can be used to submit jobs. A sample job script is shown here:

j = Job()
j.application = Athena()
j.name='PTResolution.LowPT.SmallEta.J5'
j.application.option_file=[ '/home/robinson/athena/15.6.0/PhysicsAnalysis/ForwardJets/run/jobOptions.PTResolution.LowPT.SmallEta.py' ]
j.application.athena_compile = True
j.application.atlas_release='15.6.0'
j.application.prepare()
j.inputdata=DQ2Dataset()
j.inputdata.dataset=[ 'mc08.105014.J5_pythia_jetjet.merge.AOD.e344_s479_s520_r809_r838/' ]
j.outputdata=DQ2OutputDataset()
j.outputdata.outputdata=['PTResolution.root']
j.splitter=DQ2JobSplitter()
j.splitter.numsubjobs=500
j.backend = LCG()
j.backend.requirements.cloud='UK'
j.submit()

The most important options here are

• j.application.option_file which contains your Athena jobOptions
• j.outputdata.outputdata which contains the output specified by your Athena jobOptions

To execute this script (which should be in the run directory from which you ran ganga), simple type

• execfile('scriptname')

Alternatively, the job can be submitted from outside the ganga shell by typing

• ganga scriptname

For more information about submitting your own jobs see the GANGA tutorial: https://twiki.cern.ch/twiki/bin/view/Atlas/FullGangaAtlasTutorial

*Useful GANGA python shell commands

• exit GANGA: ctrl-D
• get online help: help (exit help: ctrl-D)
• view job repository: jobs
• view subjobs with: jobs(jobid).subjobs
• to get info about specific jobs: jobs(jobid)
• to get the job status: jobs(jobid).status
• remove job: jobs(jobid).remove()
• view job output directory of finished jobs that is retrieved back to the job repository: jobs(jobid).peek()
• view stdout or stderr for debugging failed jobs: jobs(jobid).peek('stdout.gz','emacs')
• export job configuration to a file: export(jobs[jobid], '~/jobconf.py')
• force a job into a particular status: jobs(jobid).force_status("failed")

Common GANGA Problems

The datasets belonging to the container that you want to run on must all be present on the same cloud (although not necessarily at the same site). You can check where datasets are available by running:

• dq2-ls -r "datasetname" (outside ganga) The Athena version that you request must be present at all sites that your job is sent to. You can check which versions are available at which sites by running:
• lcg-infosites --vo atlas ce tag

Using the Panda backend

The Panda backend has to be used for jobs sent to US sites. It requires a slightly different form of job submission script. A sample job script is shown here:

j = Job()
j.application = Athena()
j.name='PTResolution.LowPT.SmallEta.J5'
j.application.option_file=[ '/home/robinson/athena/15.6.0/PhysicsAnalysis/ForwardJets/run/jobOptions.PTResolution.LowPT.SmallEta.py' ]
j.application.athena_compile = True
j.application.atlas_release='15.6.0'
j.application.prepare()
j.inputdata=DQ2Dataset()
j.inputdata.dataset=[ 'mc08.105014.J5_pythia_jetjet.merge.AOD.e344_s479_s520_r809_r838/' ]
j.outputdata=DQ2OutputDataset()
j.splitter=DQ2JobSplitter()
j.splitter.numsubjobs=500
j.backend=Panda()
j.submit()

-- JamesRobinson - 13 Nov 2009

OLD BUT MAY STILL BE RELEVANT

• Output Sandbox: * the output can be found by default in /gangadir/workspace/Local/jobid/output (j.outputdata.local_location='/home/bernius/outputGanga') is not working for me) * to specify which files you want to receise: j.outputsandbox=['*.dat','*.txt','*.root'] or j.outputsandbox=['*'] (to receive all)

• to get info about specific jobs: jobs(jobid)
• to get the job status: jobs(jobid).status
• remove job: jobs(jobid).remove()
• view job output directory of finished jobs that is retrieved back to the job repository: jobs(jobid).peek()

https://twiki.cern.ch/twiki/bin/view/Atlas/DAGangaFAQ

Although I haven't had a chance to try it all out properly yet -- AdamD - 23 Jul 2007

Running a Grid Job using GANGA

http://documentation.hepcg.org/res/ap3/w_301106.pdf

How to setup and use GANGA

(a) Setup Grid environment and GANGA

• source /afs/cern.ch/project/gd/LCG-share/current/etc/profile.d/grid_env.sh
• source /afs/cern.ch/sw/ganga/install/etc/setup-atlas.sh

(b) Setup Athena

* source ~/cmthome/setup.sh -tag=12.0.6

(c) Run GANGA

https://twiki.cern.ch/twiki/bin/view/Atlas/GangaTutorial427

GANGA on NorduGrid

https://twiki.cern.ch/twiki/bin/view/Atlas/GangaNGTutorial430

Some GANGA commands and things to know

• exit GANGA: ctrl-D
• get online help: help (exit help: ctrl-D)
• repository for input/output files for every job is located by default at: $HOME/gangadir/workspace/Local
• view job repository: jobs
• view subjobs with: subjobs
• to get info about specific jobs: jobs[jobid]
• remove job: jobs[jobid].remove()
• view job output directory of finished jobs that is retrieved back to the job repository: jobs[jobid].peek()
• export job configuration to a file: export(jobs[jobid], '~/jobconf.py')

Sandbox fun

• • GANGA keeps the input sandbox for all jobs in $HOME/gangadir/workspace so there might be quota problems
  • The size is by default 10MB -> Submission failes because "JobSizeException: Job Size exceeds limits." , look at tarfile in /gangadir/workspace/Local/jobid how big the file is
• Output Sandbox: * the output can be found by default in /gangadir/workspace/Local/jobid/output (j.outputdata.local_location='/home/bernius/outputGanga') is not working for me) * to specify which files you want to receise: j.outputsandbox=['*.dat','*.txt','*.root'] or j.outputsandbox=['*'] (to receive all)
• there are more options for the Input and Output Sandboxes, see https://twiki.cern.ch/twiki/bin/view/Atlas/GangaUpdates420

When you submit a job, GANGA will try to tar up your whole testarea to send with the job, which will inevitably be much larger than the 10MB limit for most sites. If it's only a little bit over then you can try and delete some things but a useful strategy is to create a separate testarea just for GANGA. The only things you need to run your job successfully are the job options and your testarea/InstallArea folder so if you just copy those into the fake testarea, your job should still run fine and fit in under the size limit.

More Information about GANGA can be found in the Links 1.-4.

Making your jobs actually work

Most likely when you first try to submit grid jobs you will encounter lots of problems with your job being sent to a site where the dataset you asked for is empty. In general this is related to the fact that the resource broker doesn't really understand the concept of incomplete datasets and handles them badly. On the LCG your best bet is to try and find out where your dataset is available and send the job there yourself. The first port of call here is AMI:

The ATLAS Metadata Interface (AMI): http://lpsc1168x.in2p3.fr:8080/opencms/opencms/AMI/www/index.html

Using AMI you can search for your dataset. When you've found the one you want, the DQ2 link next to it takes you to the PANDA page where you can browse around and try and figure out what sites actually have your data. Once you've done that you need to find a computing element (CE) which has access to the storage element (SE) which holds your dataset. Some information on this topic is available if you run:

lcg-infosites --vo atlas closeSE

Although it's a bit of black magic to work out what is actually associated with what.

What's easier than all this is to use NorduGrid instead which allows data to travel to the node where your job is (to some extent). On NorduGrid the splitter element of the job setup seems to be important and setting the number of subjobs to the number of files in the dataset seems to produce the best results (least failed subjobs).

Links

• Grid tutorial 1stFeb 2007: http://jcatmore.web.cern.ch/jcatmore/atlasDCtutorial.html

• 1. https://edms.cern.ch/file/454439/2/LCG-2-UserGuide.html

• 2. https://twiki.cern.ch/twiki/bin/view/Atlas/GangaTutorial427

• 3. http://jcatmore.web.cern.ch/jcatmore/atlasDCtutorial.html

• 4. https://hypernews.cern.ch/HyperNews/Atlas/get/GANGAUserDeveloper.html

-- CatrinBernius - 09 May 2007