# Annotation of /branches/dirac/doc/install/srcommon.tex

Revision 3335 - (hide annotations)
Wed Nov 3 05:02:57 2010 UTC (8 years, 10 months ago) by jfenwick
Original Path: trunk/doc/install/srcommon.tex
File MIME type: application/x-tex
File size: 12435 byte(s)
Fix the launcher so that it can give an environment even if buildvars is not present.
Environment now includes scons for standalone builds.
Fixes and cleanup for the install documentation.
scons TEMPLATES updated to define the optionfile version by default.

...mumble
finley no longer imports escript into the default namespace
...

 1 jfenwick 2289 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2 % 3 jfenwick 2881 % Copyright (c) 2003-2010 by University of Queensland 4 jfenwick 2289 % Earth Systems Science Computational Center (ESSCC) 5 6 % 7 % Primary Business: Queensland, Australia 8 % Licensed under the Open Software License version 3.0 9 10 % 11 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 12 13 % This file contains material common to all src distributions. 14 15 % The original version of this content came from the esscc twiki page maintained by ksteube 16 17 caltinay 2536 This chapter describes how to build \esfinley from source assuming that the dependencies are already installed (for example using precompiled packages for your OS). 18 jfenwick 3335 Section~\ref{sec:deps} describes the dependencies, while Section~\ref{sec:compilesrc} gives the compile instructions. 19 jfenwick 2529 20 jfenwick 3335 If you would prefer to build all the dependecies from source in the escript-support packages please see Chapter~\ref{chap:allsrc}. 21 jfenwick 2289 \esfinley is known to compile and run on the following systems: 22 \begin{itemize} 23 caltinay 2547 \item \linux using gcc\footnote{There are some problems with \openmp under gcc prior to version 4.3.2. Also do not link the gomp library with gcc 4.3.3 - it causes problems.} 24 jfenwick 2923 \item \linux using icc on SGI ICE 8200. (At this stage, we do not recommend building with intel-11)\footnote{There is a bug in icpc-11 related to exception handling and openmp. This results in binaries which crash.} 25 caltinay 2547 \item \macosx using gcc 26 caltinay 2536 \item \winxp using the Visual C compiler (we do not specifically discuss Windows builds in this guide). 27 jfenwick 2289 \end{itemize} 28 29 jfenwick 3335 If you have compiled a previous version of \esfinley please note that the format of the \file{..._options.py} file has changed so you will not be able to reuse old options file for this build. 30 31 32 jfenwick 2289 \section{External dependencies} 33 jfenwick 2529 \label{sec:deps} 34 jfenwick 2289 The following external packages are required in order to compile and run \esfinley. 35 Where version numbers are specified, more recent versions can probably be subsituted. 36 You can either try the standard/precompiled packages available for your operating system or you can download and build them from source. 37 caltinay 2536 The advantage of using existing packages is that they are more likely to work together properly. 38 jfenwick 2289 You must take greater care if downloading sources separately. 39 40 \begin{itemize} 41 caltinay 2547 \item python-2.5.1 (\url{http://python.org}) \\- 42 Python interpreter (you must compile with shared libraries.) 43 \item numpy 1.1.0 (\url{http://numpy.scipy.org}) \\- 44 jfenwick 3335 Arrays for Python 45 caltinay 2547 \item boost-1.35 (\url{http://www.boost.org}) \\- 46 Interface between C++ and Python 47 \item scons-0.989.5 (\url{http://www.scons.org/}) \\- 48 jfenwick 3335 Python-based alternative to \texttt{make}. 49 jfenwick 2289 \end{itemize} 50 51 caltinay 2547 The version numbers given here are not strict requirements, more recent (and in some cases older) versions are very likely to work. 52 jfenwick 2289 The following packages should be sufficient (but not necessarily minimal) for Debian 5.0 (Lenny''): 53 jfenwick 2583 python-dev, libboost-python1.35-dev, scons, python-numpy, g++. 54 jfenwick 2289 55 caltinay 2536 These packages may be required for some of the optional capabilities of the system: 56 jfenwick 2289 57 \begin{itemize} 58 \item netcdf-3.6.2 (\url{http://www.unidata.ucar.edu/software/netcdf}) \\- 59 caltinay 2547 Used to save data sets in binary form for checkpoint/restart (must be compiled with -fPIC) 60 jfenwick 3335 % \item vtk-5.0.4 (\url{http://www.vtk.org}) \\- 61 % Used to save VTK files for visualization 62 % \begin{itemize} 63 % \item cmake-2.4.6 (\url{http://www.cmake.org}) \\- 64 % Required to build VTK 65 % \item mesa-7.0.3 (\url{http://www.mesa3d.org})\\- 66 % Free OpenGL replacement used by VTK 67 % \end{itemize} 68 caltinay 2536 \item netpbm (\url{http://netpbm.sourceforge.com}) \\- 69 Tools for producing movies from images 70 \item parmetis-3.1 (\url{http://glaros.dtc.umn.edu/gkhome/metis/parmetis/overview}) \\- 71 caltinay 2547 Optimization of the stiffness matrix 72 caltinay 2536 \item MKL \\(\url{http://www.intel.com/cd/software/products/asmo-na/eng/307757.htm}) \\- 73 Intel's Math Kernel Library for use with their C compiler. 74 jfenwick 2780 \item Lapack - Available in various versions from various places. \\ 75 Currently only used to invert dense square matrices larger than 3x3. 76 jfenwick 2289 \end{itemize} 77 78 The following packages might be useful for mesh generation: 79 \begin{itemize} 80 \item gmsh-2.2.0 (\url{http://www.geuz.org/gmsh}) \\- 81 caltinay 2547 Mesh generation and viewing 82 jfenwick 2289 \begin{itemize} 83 caltinay 2536 \item fltk-1.1.9 (\url{http://www.fltk.org}) \\- 84 caltinay 2547 Required to build gmsh 85 caltinay 2536 \item gsl-1.10 (\url{http://www.gnu.org/software/gsl}) \\- 86 caltinay 2547 Required to build gmsh 87 caltinay 2536 \end{itemize} 88 caltinay 2547 \item triangle-1.6 (\url{http://www.cs.cmu.edu/~quake/triangle.html}) \\- 89 Two-dimensional mesh generator and Delaunay triangulator. 90 jfenwick 2289 \end{itemize} 91 92 Packages for visualization: 93 \begin{itemize} 94 \item mayavi-1.5 (\url{http://mayavi.sourceforge.net}) \\- 95 caltinay 2547 MayaVi is referenced in our User's Guide for viewing VTK files 96 \item visit-1.11.2 (\url{https://wci.llnl.gov/codes/visit/}) \\- 97 A featureful visualization system with movie-making capabilities. 98 jfenwick 2289 \end{itemize} 99 100 jfenwick 2324 \section{Compilation}\label{sec:compilesrc} 101 jfenwick 3322 Throughout this section we will assume that the source code is uncompressed in a directory called \file{escript.d}. 102 jfenwick 2289 You can call the directory anything you like, provided that you make the change before you compile. 103 104 You need to indicate where to find the external dependencies. 105 jfenwick 3322 To do this, create a file in the \file{escript.d/scons} directory called \file{x_options.py} where x'' is the name of your computer (output of the \texttt{hostname} command). 106 jfenwick 2602 Please note that if your hostname has non-alphanumeric characters in it (eg - ) you need to replace them with underscores. 107 jfenwick 3322 For example the options file for \texttt{bob-desktop} would be named \file{bob_desktop_options.py}. 108 jfenwick 2602 109 jfenwick 2568 From now on all paths will be relative to the top level of the source. 110 jfenwick 2780 As a starting point copy the contents one of the following files into your options file: 111 jfenwick 2289 \begin{itemize} 112 jfenwick 3322 \item \file{scons/TEMPLATE_linux.py} (\linux and \macosx desktop) 113 \item \file{scons/TEMPLATE_windows.py} (\winxp) 114 jfenwick 2289 \end{itemize} 115 116 jfenwick 3335 This options file controls which features and libraries your build of escript will attempt to use. 117 For example to use OpenMP or MPI you will need to enable it here. 118 119 If you want to try escript out without customising your build, then change directories to \file{escript.d} and 120 \begin{shellCode} 121 scons 122 \end{shellCode} 123 If this works you can skip to Section~\ref{sec:diff}. 124 If not, then you will need to make some modications to the the file. 125 Read on. 126 127 caltinay 3265 The template files contain all available options with a comment explaining the 128 purpose of each. 129 Check through the file and ensure that the relevant paths and names are correct 130 for your system and that you enable optional components that you wish to use. 131 For example, to use netCDF, find the netcdf-related lines, uncomment them 132 (i.e. remove the \# at the beginning of the lines) and change them according 133 to your installation: 134 jfenwick 2780 \begin{shellCode} 135 caltinay 3265 netcdf = True 136 netcdf_prefix = '/opt/netcdf4' 137 netcdf_libs = ['netcdf_c++', 'netcdf'] 138 jfenwick 2780 \end{shellCode} 139 140 caltinay 3265 In this example, netCDF \emph{header} files must be located in 141 jfenwick 3322 \file{/opt/netcdf4/include}\footnote{or \ldots/include32 or \ldots/include64 or \ldots/inc} 142 and the \emph{libraries} in \file{/opt/netcdf4/lib}\footnote{or \ldots/lib32 or \ldots/lib64}. 143 caltinay 3265 If this scheme does not apply to your installation then you may also specify 144 the include-path and library-path directly like so: 145 \begin{shellCode} 146 netcdf_prefix = ['/usr/local/include/netcdf', '/usr/local/lib'] 147 \end{shellCode} 148 The order is important: the first element in the list is the 149 \emph{include}-path, the second element is the \emph{library}-path and both 150 must be specified. 151 152 If a line in the options file is commented out and you do not require the 153 feature, then it can be ignored. 154 jfenwick 2780 155 jfenwick 2289 To actually compile (if you have $n$ processors, then you can use \texttt{scons -j$n$} instead): 156 157 \begin{shellCode} 158 jfenwick 2529 cd escript.d 159 jfenwick 2289 scons 160 \end{shellCode} 161 162 caltinay 3265 As part of its output, scons will tell you the name of the options file it used 163 as well as a list of features and whether they are enabled for your build. 164 If you enabled an optional dependency and the library or include files could 165 not be found you will be notified and the build will stop. 166 jfenwick 2289 167 caltinay 3265 Note, that you can override all settings from the options-file on the scons 168 command line. For example, if you usually build an optimized version but would 169 like to build a debug version into a separate directory without changing your 170 default settings, you can use: 171 jfenwick 2289 \begin{shellCode} 172 caltinay 3265 scons debug=1 prefix=debugbuild 173 jfenwick 2289 \end{shellCode} 174 caltinay 3265 This will install the binaries and libraries built in debug mode into 175 jfenwick 3322 directories underneath \file{./debugbuild}. 176 jfenwick 2289 177 caltinay 3265 To run the unit test suite that comes with the source code issue 178 jfenwick 2289 \begin{shellCode} 179 scons all_tests 180 \end{shellCode} 181 caltinay 2536 Grab a coffee or two while the tests compile and run. 182 caltinay 3265 An alternative method is available for running tests on \openmp and \mpi builds. 183 jfenwick 2289 184 \subsection{Compilation with \openmp} 185 caltinay 3265 \openmp is generally enabled by setting compiler and linker switches. For the 186 most common compilers these are automatically set by build system and all you 187 have to do is set the \texttt{openmp} option to True in your options file. If 188 this does not work or your compiler is different, then consult your compiler 189 documentation for the precise switches to use and modify the \texttt{omp_flags} 190 and \texttt{omp_ldflags} variables in your options file. 191 caltinay 2536 For example, for gcc compilers which support \openmp use: 192 jfenwick 2289 \begin{shellCode} 193 caltinay 3265 openmp = True 194 omp_flags = '-fopenmp' 195 omp_ldflags = '-fopenmp' 196 jfenwick 2289 \end{shellCode} 197 jfenwick 3335 (The two latter settings can also be left out as this is the default OpenMP on gcc.) 198 jfenwick 2289 199 caltinay 3265 You can test your \openmp-enabled build, e.g. using 4 threads by issuing 200 jfenwick 2289 \begin{shellCode} 201 gross 2363 export ESCRIPT_NUM_THREADS=4 202 scons all_tests 203 \end{shellCode} 204 205 caltinay 2536 \subsection{Compilation with \mpi} 206 caltinay 3265 You need to have \mpi preinstalled on your system. 207 There are a number of implementations so we do not provide any specific advice 208 here. 209 Set the following variables in your options file to according to your 210 installation: 211 jfenwick 2289 \begin{itemize} 212 caltinay 3265 \item \texttt{mpi} \\ 213 which \mpi implementation (flavour) is used. Valid values are 214 caltinay 2536 \begin{itemize} 215 caltinay 3265 \item[\texttt{none}] \mpi is disabled 216 caltinay 2536 \item[\texttt{MPT}] SGI MPI implementation \\ 217 \url{http://techpubs.sgi.com/library/manuals/3000/007-3687-010/pdf/007-3687-010.pdf} 218 caltinay 3265 \item[\texttt{MPICH}] Argonne's MPICH implementation \\ 219 \url{http://www.mcs.anl.gov/research/projects/mpi/mpich1/} 220 caltinay 2536 \item[\texttt{MPICH2}] Argonne's MPICH version 2 implementation \\ 221 \url{http://www.mcs.anl.gov/research/projects/mpi/mpich2/} 222 \item[\texttt{OPENMPI}] Open MPI \\ 223 \url{http://www.open-mpi.org/} 224 caltinay 3265 \item[\texttt{INTELMPI}] Intel MPI \\ 225 caltinay 2536 \url{http://software.intel.com/en-us/intel-mpi-library/} 226 \end{itemize} 227 caltinay 3265 \item \texttt{mpi_prefix} \\ 228 where to find \mpi headers and libraries (see netCDF example above) 229 caltinay 2536 \item \texttt{mpi_libs} \\ 230 caltinay 3265 which libraries to link to. 231 jfenwick 2289 \end{itemize} 232 233 caltinay 3265 To test your build using 6 processes enter: 234 jfenwick 2290 \begin{shellCode} 235 caltinay 3265 export ESCRIPT_NUM_PROCS=6 236 scons all_tests 237 jfenwick 2290 \end{shellCode} 238 jfenwick 3335 and on $2$ processes with $4$ threads each (provided \openmp is enabled)\footnote{Unless your system has $8$ cores expect this to be slow}: 239 gross 2363 \begin{shellCode} 240 export ESCRIPT_NUM_THREADS=4 241 jfenwick 3335 export ESCRIPT_NUM_PROCS=2 242 caltinay 3265 scons all_tests 243 gross 2363 \end{shellCode} 244 Alternatively, you can give a hostfile 245 \begin{shellCode} 246 export ESCRIPT_NUM_THREADS=4 247 export ESCRIPT_HOSTFILE=myhostfile 248 caltinay 3265 scons all_tests 249 gross 2363 \end{shellCode} 250 caltinay 3265 Note that depending on your \mpi flavour it may be required to start a daemon 251 before running the tests under \mpi. 252 gross 2363 253 jfenwick 3335 \subsection{Difficulties}\label{sec:diff} 254 jfenwick 2289 255 caltinay 3265 \subsubsection{Mismatch of runtime and build libraries} 256 Most external libraries used by \esfinley are linked dynamically. 257 This can lead to problems if after compiling \esfinley these libraries are 258 updated. 259 The same applies to the installed Python executable and libraries. 260 Whenever these dependencies change on your system you should recompile 261 \esfinley to avoid problems at runtime such as load errors or segmentation 262 faults. 263 jfenwick 2289 264 caltinay 2536 \subsubsection{\openmp builds segfault running examples} 265 jfenwick 3322 One known cause for this is linking the \file{gomp} library with escript built using gcc 4.3.3. 266 While you need the \texttt{-fopenmp} switch you should not need to link \file{gomp}. 267 caltinay 2536 268 jfenwick 3322

 ViewVC Help Powered by ViewVC 1.1.26