AtlasDQ2 < Main

---+ DQ2 Things
 dq2 is handy software for copying files (things like AODs, ESDs, EVNTs, TAGs...) off the grid. To set it up, do this:

<verbatim>
source /afs/cern.ch/atlas/offline/external/GRID/ddm/DQ2Clients/setup.sh
voms-proxy-init -voms atlas
export DQ2_LOCAL_SITE_ID=UKI-LT2-UCL-HEP_LOCALGROUPDISK</verbatim>

Then there are two main commands you'll probably feel like using:

<verbatim>
dq2-ls -g <dataset name>
dq2-get -r -v <dataset name> <file in dataset>
</verbatim>

The dq2-ls lists files in a dataset. The dq2-get gets them. The -g tells it to list files in the global catalogue, -r says look remotely, -v says tell me what the hell you're doing.

Dataset names are either official things like mc12.005200.T1_McAtNlo_Jimmy.evgen.EVNT.v12000401 or your own job output like user.adamdavison.005667.ntuples.v6. You can also use "*" as a wildcard in dataset names.

If you want a certain number of files from the dataset without caring which ones, you can do
<verbatim>dq2-get -n<desired number> <dataset name></verbatim>

You also sometimes want a whole dataset rather than just one or two files for testing, then you just do:

<verbatim>dq2-get -r -v <dataset name>
</verbatim>

And hope you've got enough free disk space!

---+ Combining multiple datasets

I found it useful to be able to add all of the files from several datasets to one single large dataset. For example, the FDR data is split so that a single run is composed of several datasets in different locations. In order to run a job on a whole run's worth of data you would either need to set several Ganga jobs going or create your own dataset from the existing files. Since I could not find how to do that documented anywhere else I do so here:

Get access to some additional DQ2 tools:
<verbatim>
source /afs/cern.ch/atlas/offline/external/GRID/ddm/current/dq2.sh
</verbatim>

Create your new dataset (replace my grid user name with your own!):
<verbatim>
dq2-register-dataset users.jamesmonk.test
</verbatim>

You need to know the logical file names (lfn) and ids of the files you want to add by using the command
<verbatim>
dq2-list-files fdr08_run1.0003070.MinBias.recon.ESD.o1_r12
</verbatim>

and then register some of those files in the dataset you just created (you need the lfn and the id from the previous command as argument):
<verbatim>
dq2-register-files users.jamesmonk.test fdr08_run1.0003070.MinBias.recon.ESD.o1_r12._lb0007._0001.1 688D2582-09DA-DC11-BEB6-000423D992A8
</verbatim>

finally you will have to register a location at which those files are actually present
<verbatim>
dq2-register-location users.jamesmonk.test IN2P3-CC_DATADISK
</verbatim>

(you can find out where the original dataset was present by using
<verbatim>dq2-list-dataset-replicas</verbatim>)

Since that's quite a lot to do if you plan on using data from very many datasets (just one FDR run was over 700 files) I have a little script that creates a new combined dataset for you. You basically just create a file called users.$USERNAME.whatever.you.want and list all the datasets you'd like in it. Then run ./user_dataset.sh users.$USERNAME.whatever.you.want and it creates a new dataset with the same name and contains all of the data from the datasets in that file.

Note that one feature of this is that the new dataset probably will not be complete in any single location because the original files were scattered around the grid. This should not matter too much since Ganga's DQ2 job splitter seems now to be splitting the jobs up and sending them to different locations.

-- Main.JamesMonk - 23 Apr 2008
   * [[%ATTACHURL%/user_dataset.sh][user_dataset.sh]]: script to create new user dataset from multiple datasets
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
sh	user_dataset.sh	r1	manage	1.4 K	2008-04-23 - 16:37	JamesMonk	script to create new user dataset from multiple datasets
This topic: Main > HEPGroup > AtlasStuff > AtlasDQ2
Topic revision: r4 - 2010-05-06 - JamesRobinson