Software/EasyBuild - Revision history

MatthieuHentz: /* Show loaded modules */

2019-02-21T09:49:12Z

Show loaded modules

← Older revision		Revision as of 09:49, 21 February 2019
Line 175:		Line 175:

	=== Show loaded modules ===		=== Show loaded modules ===
	To see what's loaded, use		To see what's loaded use

	<pre>		<pre>
	module list		module list
	</pre>		</pre>

	~~which prints, for example,~~

	~~1) GCC/4.9.3 9) bzip2/1.0.6-GCC-4.9.3 17) Mesa/11.2.1-GCC-4.9.3~~
	~~2) expat/2.1.1-GCC-4.9.3 10) freetype/2.6.3-GCC-4.9.3 18) libGLU/9.0.0-GCC-4.9.3-Mesa-11.2.1~~
	~~3) CLHEP/2.3.3.0-GCC-4.9.3 11) fontconfig/2.11.95-GCC-4.9.3 19) libffi/3.2.1-GCC-4.9.3~~
	~~4) cURL/7.44.0-GCC-4.9.3 12) X11/20160819-GCC-4.9.3 20) gettext/0.19.6-GCC-4.9.3~~
	~~5) Xerces-C++/3.1.2-GCC-4.9.3 13) libdrm/2.4.68-GCC-4.9.3 21) PCRE/8.38-GCC-4.9.3~~
	~~6) zlib/1.2.8-GCC-4.9.3 14) ncurses/6.0-GCC-4.9.3 22) GLib/2.49.5-GCC-4.9.3~~
	~~7) libxml2/2.9.3-GCC-4.9.3 15) LLVM/3.8.0-GCC-4.9.3 23) Qt5/5.7.0-GCC-4.9.3~~
	~~8) libpng/1.6.21-GCC-4.9.3 16) eudev/3.1.5-GCC-4.9.3 24) Geant4/10.04.p02-GCC-4.9.3~~


	=== Remove loaded modules ===		=== Remove loaded modules ===

MatthieuHentz: /* Load modules */

2019-02-21T09:48:55Z

Load modules

← Older revision		Revision as of 09:48, 21 February 2019
Line 149:		Line 149:

	=== Load modules ===		=== Load modules ===
	Load a given module using		Load a given module using the command

			<pre>module load</pre>

			This will load the specified module along with all its dependencies. For instance,

	<pre>		<pre>
Line 155:		Line 159:
	</pre>		</pre>

	and ~~where required~~ source the relevant environment variables, e.g.		loads Geant4 and all of the following

			1) GCC/4.9.3 7) libxml2/2.9.3-GCC-4.9.3 13) libdrm/2.4.68-GCC-4.9.3 19) libffi/3.2.1-GCC-4.9.3
			2) expat/2.1.1-GCC-4.9.3 8) libpng/1.6.21-GCC-4.9.3 14) ncurses/6.0-GCC-4.9.3 20) gettext/0.19.6-GCC-4.9.3
			3) CLHEP/2.3.3.0-GCC-4.9.3 9) bzip2/1.0.6-GCC-4.9.3 15) LLVM/3.8.0-GCC-4.9.3 21) PCRE/8.38-GCC-4.9.3
			4) cURL/7.44.0-GCC-4.9.3 10) freetype/2.6.3-GCC-4.9.3 16) eudev/3.1.5-GCC-4.9.3 22) GLib/2.49.5-GCC-4.9.3
			5) Xerces-C++/3.1.2-GCC-4.9.3 11) fontconfig/2.11.95-GCC-4.9.3 17) Mesa/11.2.1-GCC-4.9.3 23) Qt5/5.7.0-GCC-4.9.3
			6) zlib/1.2.8-GCC-4.9.3 12) X11/20160819-GCC-4.9.3 18) libGLU/9.0.0-GCC-4.9.3-Mesa-11.2.1 24) Geant4/10.04.p02-GCC-4.9.3

			Don't forget to source the relevant environment variables where required, e.g.

	<pre>		<pre>

MatthieuHentz: /* Use modules */

2019-02-21T09:44:57Z

Use modules

← Older revision		Revision as of 09:44, 21 February 2019
Line 118:		Line 118:

	= Use modules =		= Use modules =
	~~As Simon says, I've made~~ modules ~~('environment~~ modules ~~for unix') on~~ the ~~system using EasyBuild. I've installed~~ the ~~latest Geant4 for BDSIM (our application) but I've also now put on~~ the ~~latest patch version~~ of ~~all recent Geant4 versions (was quite easy)~~. ~~You can access them from a CentOS7 node (I'm presuming Simon's here):~~		=== Set path to modules ===
			Before modules can be accessed, the module tool needs to be given the path to the desired set of modules. For example,

			<pre>
	module use /unix/pbt/software/eb/CentOS7/modules/*		module use /unix/pbt/software/eb/CentOS7/modules/*
	~~module avail Geant4~~		</pre>

	~~.. this prints out the following~~		=== List available modules ===
			You can then list all modules available with

	~~------------------------------------------------~~ /~~unix/pbt/software/eb/CentOS7/modules/all -------------------------------------------------~~		<pre>module avail</pre>
	~~Geant4/10.00.p04-GCC-4.9.3 Geant4/10.01.p03-GCC-4.9.3 Geant4/10.02.p03-GCC-4.9.3 Geant4/10.03.p03-GCC-4.9.3 Geant4/10.04.p02-GCC-4.9.3~~

			or check what versions of a given module are available using

			<pre>module avail [modulename]</pre>

			For example,

			<pre>module avail Geant4</pre>

			prints out the following

			<pre>
			-------------------------------------------------------------------- /unix/pbt/software/eb/CentOS7/modules/phys --------------------------------------------------------------------
			Geant4/10.00.p04-GCC-4.9.3 Geant4/10.01.p03-GCC-4.9.3 Geant4/10.02.p03-GCC-4.9.3 Geant4/10.03.p03-GCC-4.9.3 Geant4/10.04.p02-GCC-4.9.3 Geant4/10.04.p02-mt-GCC-4.9.3

			-------------------------------------------------------------------- /unix/pbt/software/eb/CentOS7/modules/all ---------------------------------------------------------------------
			Geant4/10.00.p04-GCC-4.9.3 Geant4/10.01.p03-GCC-4.9.3 Geant4/10.02.p03-GCC-4.9.3 Geant4/10.03.p03-GCC-4.9.3 Geant4/10.04.p02-GCC-4.9.3 Geant4/10.04.p02-mt-GCC-4.9.3
			</pre>

			=== Load modules ===
			Load a given module using

			<pre>
	module load Geant4/10.04.p02-GCC-4.9.3		module load Geant4/10.04.p02-GCC-4.9.3
			</pre>

			and where required source the relevant environment variables, e.g.

			<pre>
	source $EBROOTGEANT4/bin/geant4.sh		source $EBROOTGEANT4/bin/geant4.sh
			</pre>

	~~The last line sources the required environmental variable for Geant4 to be found. This will load the corresponding CLHEP version required (multiple available).~~		=== Show loaded modules ===
			To see what's loaded, use

	~~To see what's loaded:~~		<pre>
			module list
			</pre>

	~~module list~~		which prints, for example,

	~~Currently Loaded Modulefiles:~~
	1) GCC/4.9.3 9) bzip2/1.0.6-GCC-4.9.3 17) Mesa/11.2.1-GCC-4.9.3		1) GCC/4.9.3 9) bzip2/1.0.6-GCC-4.9.3 17) Mesa/11.2.1-GCC-4.9.3
	2) expat/2.1.1-GCC-4.9.3 10) freetype/2.6.3-GCC-4.9.3 18) libGLU/9.0.0-GCC-4.9.3-Mesa-11.2.1		2) expat/2.1.1-GCC-4.9.3 10) freetype/2.6.3-GCC-4.9.3 18) libGLU/9.0.0-GCC-4.9.3-Mesa-11.2.1
Line 147:		Line 179:
	8) libpng/1.6.21-GCC-4.9.3 16) eudev/3.1.5-GCC-4.9.3 24) Geant4/10.04.p02-GCC-4.9.3		8) libpng/1.6.21-GCC-4.9.3 16) eudev/3.1.5-GCC-4.9.3 24) Geant4/10.04.p02-GCC-4.9.3

	~~To get rid of these loaded modules:~~

			=== Remove loaded modules ===
			To get rid of all loaded modules:

			<pre>
	module purge		module purge
			</pre>
	~~To see all modules:~~

	~~module avail~~

	~~There's the latest ROOT 6 on there too as well as a few versions of CLHEP (note Geant4 versions use different ones).~~

	~~The module use and module load should be done in each new environment~~ / ~~terminal.~~

	~~I tried making an easybuild recipe for GATE, but it fails at the end (I even tried a patch to remove their hard-coded make files for gjm) but still no success.~~


	= Install modules =		= Install modules =

MatthieuHentz: /* Install modules */

2019-02-20T19:38:13Z

Install modules

← Older revision		Revision as of 19:38, 20 February 2019
Line 165:		Line 165:

	=== Installing modules ===		=== Installing modules ===
	~~Hello all,~~

	~~- BDSIM has to use the single threaded version -> "high throughput" only vs "high performance"~~		Example command to install module:
	- BDSIM requires external CLHEP so Geant4's internal one isn't mixed in -> otherwise, whilst BDSIM would work, the random number generator found is unclear and the results are not strongly reproducible.

	~~However, if you just want a multithreaded Geant4 build for your own application... yes you can make a separate easybuild file. In fact the full~~ command ~~I've used~~ to ~~construct a~~ module ~~is (for example)~~:

	eb Bdsim-1.2.1-GCC-4.9.3.eb --prefix=/unix/pbt/software/eb/CentOS7 -robot --robot-path=./: --modules-tool=EnvironmentModulesC --module-syntax=Tcl --allow-modules-tool-mismatch --optarch=GENERIC --buildpath=/tmp/easybuild/		eb Bdsim-1.2.1-GCC-4.9.3.eb --prefix=/unix/pbt/software/eb/CentOS7 -robot --robot-path=./: --modules-tool=EnvironmentModulesC --module-syntax=Tcl --allow-modules-tool-mismatch --optarch=GENERIC --buildpath=/tmp/easybuild/

	~~The trick to~~ get it to not replace a current version ~~is to rename~~ the module in the ~~easybuild~~ file. When ~~you make one~~, ~~easybuild~~ checks for ~~ones~~ of the same name and version and if one ~~with~~ the ~~same name and recipe is built it~~ won't ~~build it~~. If the options are updated, ~~I believe it will update~~ the build~~. Anyway, if you change the name it~~ will ~~create a new one. We used the name also as part of the source URL, so you should just hard code that~~. ~~I've made an example here:~~		Two types of the same version (i.e. one single-threaded and one multi-threaded) can be installed side-by-side. To get EasyBuild to not replace a current version, the module can be renamed in the EasyBuild file. When a module is built, EasyBuild checks for modules of the same name and version and if one already exists, the build won't complete. If the options are updated, the build will be updated.

	~~/unix/pbt/software/eb/accsoft-eb-nevay/Geant4-10.04.p02-mt-GCC-4.9.3.eb~~
	~~(copied from the regular geant4 one, turned multithreading on, changed version to have "-mt" and hardcoded source url)~~

	When you build a module, you should do the module use command so it can find the others already available (it checks the requirements of the one you're building) and also the EasyBuild module itself. You should have a clean environment with no other modules loaded.		When you build a module, you should do the `module use` command so it can find the others already available (it checks the requirements of the one you're building) and also the EasyBuild module itself. You should have a clean environment with no other modules loaded.

	/unix/pbt/software/eb/easybuildonly.sh		/unix/pbt/software/eb/easybuildonly.sh

RaffaellaRadogna: /* =New commands for centOS 7 */

2019-02-18T22:39:58Z

=New commands for centOS 7

← Older revision		Revision as of 22:39, 18 February 2019
Line 337:		Line 337:
	https://modules.readthedocs.io/en/latest/FAQ.html		https://modules.readthedocs.io/en/latest/FAQ.html

	===New commands for centOS 7 ==		===New commands for centOS 7 ===

	# Tell module tool where the modules are		# Tell module tool where the modules are

RaffaellaRadogna: /* EasyBuild */

2019-02-18T22:39:22Z

EasyBuild

← Older revision		Revision as of 22:39, 18 February 2019
Line 328:		Line 328:

	EasyBuild allows to easily reproduce software builds/installations and install multiple versions.		EasyBuild allows to easily reproduce software builds/installations and install multiple versions.

			==Environment Modules==
			===Reference===
			http://modules.sourceforge.net/man/module.html
			===Environment Modules===
			Typically users initialize their environment when they log in by setting environment information for every application they will reference during the session.
			The Environment Modules system is a tool to help users manage their Unix or Linux shell environment, by allowing groups of related environment-variable settings to be made or removed dynamically. The modules system is based on modulefiles,[6] which specify groups of environment settings that need to be made together. Modulefiles can be installed in a central location for general use, or in a user directory for personal use. Environment Modules modulefiles are written in the Tcl (Tool Command Language) and are interpreted by the modulecmd program via the module[7] user interface. The key advantage of Environment Modules is that it is shell independent and supports all major shells such as bash, ksh, zsh, sh, tcsh, and csh. The second key advantage is that it allows to use multiple versions of the program or package from the same account by just loading proper module. modulefiles are created on per application per version basis. They can be dynamically loaded, unloaded, or switched. Along with the capability of using multiple versions of the same software it also can be used to implement site policies regarding the access and use of applications.
			https://modules.readthedocs.io/en/latest/FAQ.html

			===New commands for centOS 7 ==

			# Tell module tool where the modules are
			module use /unix/pbt/software/eb/CentOS7/modules/*
			# Load Geant4
			# This loads all the necessary libraries like CLHEP, Qt5, etc.
			module load Geant4/10.04.p02-GCC-4.9.3
			source $EBROOTGEANT4/bin/geant4.sh
			module load CMake/3.5.2-GCC-4.9.3

			'''what does this mean'''
			* module use [-a\|--append] directory…
			Prepend one or more directories to the MODULEPATH environment variable.
			* load modulefile...
			Load modulefile(s) into the shell environment.

			when Geant4 is installed as a module
			To see which versions are available, execute: module avail geant4
			To then use a particular version, execute for example
			module load geant4/10.02.p03
			After you load the module, you can see where the software is installed by executing
			echo $EBROOTGEANT4

			Set GEANT environment variables
			source $EBROOTGEANT4/bin/geant4.sh

RaffaellaRadogna: /* Geant4 */

2019-02-18T22:34:21Z

Geant4

← Older revision		Revision as of 22:34, 18 February 2019
Line 299:		Line 299:
	** EXPAT_LIBRARY If CMake cannot locate your external EXPAT installation, set this to the full path to the EXPAT library, e.g. /usr/lib/libexpat.so		** EXPAT_LIBRARY If CMake cannot locate your external EXPAT installation, set this to the full path to the EXPAT library, e.g. /usr/lib/libexpat.so
	** EXPAT_INCLUDE_DIR If CMake cannot locate your external EXPAT installation, set this to the directory containing the EXPAT headers (e.g. if you have /foobar/expat.h, then set this to /foobar).		** EXPAT_INCLUDE_DIR If CMake cannot locate your external EXPAT installation, set this to the directory containing the EXPAT headers (e.g. if you have /foobar/expat.h, then set this to /foobar).

			== EasyBuild ==
			===Reference===
			https://media.readthedocs.org/pdf/caylo-easybuild/docfix/caylo-easybuild.pdf
			EasyBuild is a flexible software build and installation framework that allows you to manage (scientific) software on High Performance Computing (HPC) systems in an efficient way. Divert from the standard configure / make / make install with custom procedures. Allows for easily reproducing previous builds.

			===EasyBuild===
			EasyBuild consists of a collection of Python modules and packages that interact with each other, dynamically picking up additional Python modules as needed for building and installing a (stack of) software package(s) specified via simple specification files

			In EasyBuild terminology: the EasyBuild framework leverages easyblocks to automatically build and install software using particular compiler toolchains, as specified by one or multiple easyconfig files.

			The EasyBuild framework provides functionality commonly needed when installing scientific software on HPC systems. For example, it deals with downloading, unpacking and patching of sources, loading module files for dependencies, setting up the build environment, autonomously running (interactive) shell commands, creating module files that match the specification files, etc.
			The implementation of a particular software build and install procedure is done in a Python module, which is aptly referred to as an easyblock.
			For each software package being built, the EasyBuild framework will determine which easyblock should be used, based on the name of the software package or the value of the easyblock specification parameter (see Easyblock specification). Since EasyBuild v2.0, an easyblock must be specified in case no matching easyblock is found based on the software name (cfr. Automagic fallback to ConfigureMake).
			The specification files that are supplied to EasyBuild are referred to as easyconfig files (or simply easyconfigs .eb), which are basically plain text files containing (mostly) key-value assignments for build parameters supported by the framework, also referred to as easyconfig parameters (see Writing easyconfig files: the basics for more information).
			An easyconfig file serves as a build specification for EasyBuild.
			Easyconfigs typically follow a (fixed) strict naming scheme, i.e. <name>-<version>[-<toolchain>][<versionsuffix>
			A handful of easyconfig parameters are mandatory:
			* name, version: specify what software (version) to build
			* homepage, description: metadata (used for module help)
			* toolchain: specifies name and version of compiler toolchain to use
			* Using external modules as dependencies
			* Configure/build/install command options:
			** configopts: options for configure command
			** preconfigopts: options used as prefix for configure command
			** buildopts, prebuildopts: options for build command
			** installopts, preinstallopts: options for install

			EasyBuild allows to easily reproduce software builds/installations and install multiple versions.

RaffaellaRadogna: /* Geant4 */

2019-02-18T22:32:23Z

Geant4

← Older revision		Revision as of 22:32, 18 February 2019
Line 269:		Line 269:
	* create a directory in which to configure and run the build and store the build products /path/to/geant4.10.05-build		* create a directory in which to configure and run the build and store the build products /path/to/geant4.10.05-build
	* $ cmake -DCMAKE_INSTALL_PREFIX=/path/to/geant4.10.05-install /path/to/geant4.10.05		* $ cmake -DCMAKE_INSTALL_PREFIX=/path/to/geant4.10.05-install /path/to/geant4.10.05
	* CMake Variable CMAKE_INSTALL_PREFIX is used to set the install directory, the directory under which the Geant4 libraries, headers and support files will be installed. It must be supplied as an absolute path. The second argument to CMake is the path to the source directory.		* CMake Variable CMAKE_INSTALL_PREFIX is used to set the install directory, the directory under which the Geant4 libraries, headers and support files will be installed. It must be supplied as an absolute path. The second argument to CMake is the path to the source directory.
	* Additional arguments may be passed to CMake to activate optional components of Geant4, such as visualization drivers, or tune the build and install parameters.		* Additional arguments may be passed to CMake to activate optional components of Geant4, such as visualization drivers, or tune the build and install parameters.
	* If you run CMake and decide afterwards you want to activate additional options, simply rerun CMake in the build directory, passing it the extra options plus the build directory path.		* If you run CMake and decide afterwards you want to activate additional options, simply rerun CMake in the build directory, passing it the extra options plus the build directory path.
	* Datasets are not required to be present to build Geant4, but may be required to run your application, depending on the physics models you use. If you wish to download and install the datasets automatically as part of your build of Geant4, simply add the option-DGEANT4_INSTALL_DATA=ON to the arguments passed to CMake.		* Datasets are not required to be present to build Geant4, but may be required to run your application, depending on the physics models you use. If you wish to download and install the datasets automatically as part of your build of Geant4, simply add the option-DGEANT4_INSTALL_DATA=ON to the arguments passed to CMake.
	* After the configuration has run, CMake will have generated Unix Makefiles for building Geant4. To run the build, simply execute make in the build directory		* After the configuration has run, CMake will have generated Unix Makefiles for building Geant4. To run the build, simply execute make in the build directory
Line 293:		Line 293:
	* GEANT4_USE_SYSTEM_EXPAT (DEFAULT : ON) If set to ON, build Geant4 with an external install of Expat.		* GEANT4_USE_SYSTEM_EXPAT (DEFAULT : ON) If set to ON, build Geant4 with an external install of Expat.
	* GEANT4_USE_SYSTEM_ZLIB (DEFAULT : OFF) If set to ON, build Geant4 with an external install of zlib.		* GEANT4_USE_SYSTEM_ZLIB (DEFAULT : OFF) If set to ON, build Geant4 with an external install of zlib.


	===Advanced Options===		===Advanced Options===

RaffaellaRadogna: /* Geant4 */

2019-02-18T22:31:32Z

Geant4

← Older revision		Revision as of 22:31, 18 February 2019
Line 264:		Line 264:
	* Coin3D libraries and headers with SoXt(SoWin) graphics binding on Linux/macOS(Windows), compiled against the C++11 standard.		* Coin3D libraries and headers with SoXt(SoWin) graphics binding on Linux/macOS(Windows), compiled against the C++11 standard.
	* Motif User Interface and Visualization (Linux and macOS)		* Motif User Interface and Visualization (Linux and macOS)


	===Standard GEANT4 Building and Installation===		===Standard GEANT4 Building and Installation===

RaffaellaRadogna: /* Summary from Raffy about EasyBuild, Environment Modules and GEANT4 */

2019-02-18T22:30:56Z

Summary from Raffy about EasyBuild, Environment Modules and GEANT4

← Older revision		Revision as of 22:30, 18 February 2019
Line 244:		Line 244:

	= Summary from Raffy about EasyBuild, Environment Modules and GEANT4 =		= Summary from Raffy about EasyBuild, Environment Modules and GEANT4 =
			== Geant4 ==
	===Reference===		===Reference===
	http://geant4-userdoc.web.cern.ch/geant4-userdoc/UsersGuides/InstallationGuide/html/gettingstarted.html		http://geant4-userdoc.web.cern.ch/geant4-userdoc/UsersGuides/InstallationGuide/html/gettingstarted.html
Line 257:		Line 258:
	===User Interface and Visualization Drivers===		===User Interface and Visualization Drivers===
	* Qt User Interface and Visualization		* Qt User Interface and Visualization
	** Qt4 (>=4.6) or Qt5 headers and libraries		** Qt4 (>=4.6) or Qt5 headers and libraries
	* OpenGL or MesaGL headers and libraries.		* OpenGL or MesaGL headers and libraries.
	** X11 OpenGL Visualization (Linux and macOS)		** X11 OpenGL Visualization (Linux and macOS)
	* Open Inventor Visualization		* Open Inventor Visualization
	* Coin3D libraries and headers with SoXt(SoWin) graphics binding on Linux/macOS(Windows), compiled against the C++11 standard.		* Coin3D libraries and headers with SoXt(SoWin) graphics binding on Linux/macOS(Windows), compiled against the C++11 standard.
	* Motif User Interface and Visualization (Linux and macOS)		* Motif User Interface and Visualization (Linux and macOS)