# Contents of /trunk/doc/install/source.tex

Revision 6530 - (show annotations)
Thu Mar 9 01:31:15 2017 UTC (23 months, 1 week ago) by jfenwick
File MIME type: application/x-tex
File size: 18691 byte(s)
Noting dependency on legacy-netcdf for debian styles

 1 %!TEX root = install.tex 2 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 3 % Copyright (c) 2012-2017 by The University of Queensland 4 5 % 6 % Primary Business: Queensland, Australia 7 % Licensed under the Apache License, version 2.0 8 9 % 10 % Development until 2012 by Earth Systems Science Computational Center (ESSCC) 11 % Development 2012-2013 by School of Earth Sciences 12 % Development from 2014 by Centre for Geoscience Computing (GeoComp) 13 % 14 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 15 16 % Notes about compilers 17 18 \chapter{Installing from Source}\label{chap:source} 19 20 This chapter assumes you are using a unix/posix like system (including MacOSX). 21 22 \section{Parallel Technologies}\label{sec:par} 23 It is likely that the computer you run \escript on, will have more than one processor core. 24 \escript can make use of multiple cores [in order to solve problems more quickly] if it is told to do so, 25 but this functionality must be enabled at compile time. 26 Section~\ref{sec:needpar} gives some rough guidelines to help you determine what you need. 27 28 There are two technologies which \escript can employ here. 29 \begin{itemize} 30 \item OpenMP -- the more efficient of the two [thread level parallelism]. 31 \item MPI -- Uses multiple processes (less efficient), needs less help from the compiler. 32 \end{itemize} 33 34 Escript is primarily tested on recent versions of the GNU and Intel suites (g++'' / icpc''). 35 However, it also passes our tests when compiled using clang++''. 36 Escript now requires compiler support for some features of the C++11 standard. 37 See Appendix~\ref{app:cxxfeatures} for a list. 38 39 40 Our current test compilers include: 41 \begin{itemize} 42 \item g++ (5 and 6) 43 \item clang++ (OSX 10.11 default) 44 \item intel icpc v15, v16 45 \end{itemize} 46 47 Note that: 48 \begin{itemize} 49 \item OpenMP will not function correctly for g++ $\leq$ 4.2.1 (and is not currently supported by clang). 50 \item icpc v11 has a subtle bug involving OpenMP and C++ exception handling, so this combination should not be used. 51 \end{itemize} 52 53 \subsection{What parallel technology do I need?}\label{sec:needpar} 54 If you are using any version of Linux released in the past few years, then your system compiler will support 55 \openmp with no extra work (and give better performance); so you should use it. 56 You will not need MPI unless your computer is some form of cluster. 57 58 If you are using BSD or MacOSX and you are just experimenting with \escript, then performance is 59 probably not a major issue for you at the moment so you don't need to use either \openmp or MPI. 60 This also applies if you write and polish your scripts on your computer and then send them to a cluster to execute. 61 If in the future you find escript useful and your scripts take significant time to run, then you may want to recompile 62 \escript with more options. 63 64 65 66 Note that even if your version of \escript has support for \openmp or MPI, you will still need to tell the system to 67 use it when you run your scripts. 68 If you are using the \texttt{run-escript} launcher, then this is controlled by 69 the \texttt{-t}, \texttt{-p}, and \texttt{-n} options. 70 If not, then consult the documentation for your MPI libraries (or the compiler documentation in the case of OpenMP 71 \footnote{It may be enough to set the \texttt{OMP\_NUM\_THREADS} environment variable.}). 72 73 If you are using MacOSX, then see the next section, if not, then skip to Section~\ref{sec:build}. 74 75 \section{MacOS} 76 This release of \escript has only been tested on OSX 10.11. 77 For this section we assume you are using either \texttt{homebrew} or \texttt{MacPorts} as a package 78 manager\footnote{Note that package managers will make changes to your computer based on programs configured by other people from 79 various places around the internet. It is important to satisfy yourself as to the security of those systems.}. 80 You can of course install prerequisite software in other ways. 81 For example, we have had \emph{some} success changing the default 82 compilers used by those systems. However this is more complicated and we do not provide a guide here. 83 Successful combinations of OSX and package managers are given in the table below. 84 85 \begin{center} 86 \begin{tabular}{|c|c|c|}\hline 87 & \texttt{homebrew} & \texttt{MacPorts} \\\hline 88 OSX 10.9 & Yes & No\\\hline 89 OSX 10.10& Yes & Yes\\\hline 90 \end{tabular} 91 \end{center} 92 93 \noindent Both of those systems require the XCode command line tools to be installed\footnote{As of OSX10.9, the 94 command \texttt{xcode-select --install} will allow you to download and install the commandline tools.}. 95 96 \section{Building}\label{sec:build} 97 98 To simplify things for people, we have prepared \texttt{_options.py} files for a number of 99 systems\footnote{These are correct a time of writing but later versions of those systems may require tweaks. 100 Also, these systems represent a cross section of possible platforms rather than meaning those systems get particular support.}. 101 The \texttt{_options.py} files are located in the \texttt{scons/templates} directory. We suggest that the file most relevant to your OS 102 be copied from the templates directory to the scons directory and renamed to the form XXXX_options.py where XXXX 103 should be replaced with your computer's (host-)name. 104 If your particular system is not in the list below, or if you want a more customised 105 build, 106 see Section~\ref{sec:othersrc} for instructions. 107 \begin{itemize} 108 \item Debian - \ref{sec:debsrc} 109 \item Ubuntu - \ref{sec:ubsrc} 110 \item OpenSuse - \ref{sec:susesrc} 111 \item Centos - \ref{sec:centossrc} 112 \item Fedora - \ref{sec:fedorasrc} 113 \item MacOS (macports) - \ref{sec:macportsrc} 114 \item MacOS (homebrew) - \ref{sec:homebrewsrc} 115 \item FreeBSD - \ref{sec:freebsdsrc} 116 \end{itemize} 117 118 Once these are done proceed to Section~\ref{sec:cleanup} for cleanup steps. 119 120 All of these instructions assume that you have obtained the \escript source (and uncompressed it if necessary). 121 \subsection{Debian}\label{sec:debsrc} 122 123 \begin{shellCode} 124 sudo aptitude install python-dev python-numpy libboost-python-dev libnetcdf-dev 125 sudo aptitude install libnetcdf-cxx-legacy-dev 126 sudo aptitude install scons lsb-release libboost-random-dev 127 sudo aptitude install python-sympy python-matplotlib python-scipy 128 sudo aptitude install python-pyproj python-gdal 129 \end{shellCode} 130 131 \noindent If you are running \textit{Wheezy}, you can use: 132 \begin{shellCode} 133 sudo aptitude install gmsh 134 \end{shellCode} 135 to add extra meshing functionality. 136 137 \begin{optionalstep} 138 If for some reason, you wish to rebuild the documentation, you would also need the following: 139 \begin{shellCode} 140 sudo aptitude install python-sphinx doxygen python-docutils texlive 141 sudo aptitude install zip texlive-latex-extra latex-xcolor 142 \end{shellCode} 143 \end{optionalstep} 144 145 \noindent In the source directory execute the following (substitute jessie for XXXX): 146 \begin{shellCode} 147 scons -j1 options_file=scons/templates/XXXX_options.py 148 \end{shellCode} 149 150 \noindent If you wish to test your build, you can use the following: 151 \begin{shellCode} 152 scons -j1 py_tests options_file=scons/templates/XXXX_options.py 153 \end{shellCode} 154 155 \subsection{Ubuntu}\label{sec:ubsrc} 156 157 If you have not installed \texttt{aptitude}, then substitute \texttt{apt-get} in the following. 158 \begin{shellCode} 159 sudo aptitude install python-dev python-numpy libboost-python-dev 160 sudo aptitude install libnetcdf-cxx-legacy-dev 161 sudo aptitude install libnetcdf-dev libboost-random-dev 162 sudo aptitude install scons lsb-release 163 sudo aptitude install python-sympy python-matplotlib python-scipy 164 sudo aptitude install python-pyproj python-gdal gmsh 165 \end{shellCode} 166 167 168 \begin{optionalstep} 169 If for some reason, you wish to rebuild the documentation, you would also need the following: 170 \begin{shellCode} 171 sudo aptitude install python-sphinx doxygen python-docutils texlive 172 sudo aptitude install zip texlive-latex-extra latex-xcolor 173 \end{shellCode} 174 \end{optionalstep} 175 176 \noindent In the source directory execute the following (substitute precise, quantal or raring as appropriate for XXXX): 177 \begin{shellCode} 178 scons -j1 options_file=scons/templates/XXXX_options.py 179 \end{shellCode} 180 181 \noindent If you wish to test your build, you can use the following: 182 \begin{shellCode} 183 scons -j1 py_tests options_file=scons/templates/XXXX_options.py 184 \end{shellCode} 185 186 187 188 \subsection{OpenSuse}\label{sec:susesrc} 189 These instructions were prepared using release $13.2$. 190 191 \noindent Install packages from the main distribution: 192 \begin{shellCode} 193 sudo zypper in libboost_python1_54_0 libboost_random1_54_0 194 sudo zypper in python-devel python-numpy libnetcdf_c++-devel 195 sudo zypper in python-scipy python-sympy python-matplotlib 196 sudo zypper in gcc gcc-c++ scons boost-devel netcdf-devel 197 \end{shellCode} 198 These will allow you to use most features except some parts of the \downunder inversion library. 199 If you wish to use those, you will need some additional packages [python-pyproj, python-gdal]. 200 This can be done now or after Escript installation. 201 202 \begin{shellCode} 203 sudo zypper addrepo \ 204 http://ftp.suse.de/pub/opensuse/repositories/Application:/Geo/openSUSE_13.2/ osgf 205 sudo zypper install python-pyproj python-gdal 206 \end{shellCode} 207 208 Now to build escript itself. 209 In the escript source directory: 210 \begin{shellCode} 211 scons -j1 options_file=scons/templates/opensuse13.1_options.py 212 \end{shellCode} 213 214 \noindent If you wish to test your build, you can use the following: 215 \begin{shellCode} 216 scons -j1 py_tests options_file=scons/templates/opensuse13.1_options.py 217 \end{shellCode} 218 219 \noindent Now go to Section~\ref{sec:cleanup} for cleanup. 220 221 \subsection{CentOS}\label{sec:centossrc} 222 These instructions were prepared using CentOS release $7.0$. 223 The core of escript works, however some functionality is not available because the default packages for some dependencies in CentOS are too old. 224 225 \noindent Add the \texttt{EPEL} repository. 226 \begin{shellCode} 227 yum install epel-release.noarch 228 \end{shellCode} 229 230 \noindent Install packages: 231 \begin{shellCode} 232 yum install netcdf-devel netcdf-cxx-devel gdal-python 233 yum install python-devel numpy scipy scons boost-devel 234 yum install python-matplotlib gcc gcc-c++ 235 yum install boost-python 236 \end{shellCode} 237 238 The above packages will allow you to use most features except 239 the \downunder inversion library. 240 If you wish to use those it, you will need to install some additional packages. 241 242 \noindent For some coordinate transformations, \downunder can also make use of the python interface to a tool called \texttt{proj}. 243 There does not seem to be an obvious CentOS repository for this though. 244 If it turns out to be necessary for your particular application, the source can be downloaded. 245 246 \noindent Now to build escript itself. 247 In the escript source directory: 248 \begin{shellCode} 249 scons -j1 options_file=scons/templates/centos7_0_options.py 250 \end{shellCode} 251 252 \noindent Now go to Section~\ref{sec:cleanup} for cleanup. 253 254 \subsection{Fedora}\label{sec:fedorasrc} 255 These instructions were prepared using release $21.5$. 256 257 \noindent Install packages 258 \begin{shellCode} 259 yum install netcdf-cxx-devel gcc-c++ scipy 260 yum install sympy scons pyproj gdal python-matplotlib 261 yum install boost-devel 262 \end{shellCode} 263 264 \noindent Now to build escript itself. 265 In the escript source directory: 266 \begin{shellCode} 267 scons -j1 options_file=scons/templates/fedora21_5_options.py 268 \end{shellCode} 269 270 \noindent If you wish to test your build, you can use the following: 271 \begin{shellCode} 272 scons -j1 py_tests options_file=scons/templates/fedora21_5_options.py 273 \end{shellCode} 274 275 \noindent Now go to Section~\ref{sec:cleanup} for cleanup. 276 277 \subsection{MacOS 10.10 (macports)}\label{sec:macportsrc} 278 279 The following will install the capabilities needed for the \texttt{macports_10.10_options.py} file. 280 281 \begin{shellCode} 282 sudo port install scons 283 sudo port select --set python python27 284 sudo port install boost 285 sudo port install py27-numpy 286 sudo port install py27-sympy 287 sudo port select --set py-sympy py27-sympy 288 sudo port install py27-scipy 289 sudo port install py27-pyproj 290 sudo port install py27-gdal 291 sudo port install netcdf-cxx 292 sudo port instal silo 293 \end{shellCode} 294 295 \begin{shellCode} 296 scons -j1 options_file=scons/templates/macports_10.10options.py 297 \end{shellCode} 298 299 300 \subsection{MacOS 10.9, 10.10 (homebrew)}\label{sec:homebrewsrc} 301 302 The following will install the capabilities needed for the \texttt{homebrew_10.10_options.py} file. 303 OSX 10.9 can use the same file. 304 305 \begin{shellCode} 306 brew install scons 307 brew install boost-python 308 brew install homebrew/science/netcdf --with-cxx-compat 309 \end{shellCode} 310 311 There do not appear to be formulae for \texttt{sympy} or \texttt{pyproj} so if you wish to use those features, then 312 you will need to install them separately. 313 314 315 \begin{shellCode} 316 scons -j1 options_file=scons/templates/homebrew_10.10_options.py 317 \end{shellCode} 318 319 320 \subsection{FreeBSD}\label{sec:freebsdsrc} 321 322 At time of writing, \texttt{numpy} does not install correctly on FreeBSD. 323 Since \texttt{numpy} is a critical dependency for \escript, we have been unable to test on FreeBSD. 324 325 \begin{comment} 326 \subsubsection{Release 10.0} 327 328 Install the following packages: 329 \begin{itemize} 330 \item python 331 \item scons 332 \item boost-python-libs 333 \item bash 334 \item netcdf 335 \item silo 336 \item py27-scipy 337 \item py27-gdal 338 \item py27-matplotlib 339 \item py27-pyproj 340 \item py27-sympy 341 \end{itemize} 342 343 \noindent Next choose (or create) your options file. 344 For the setup as above the escript source comes with a prepared file in 345 \texttt{scons/templates/freebsd10.0_options.py}. 346 Finally to build escript issue the following in the escript source directory 347 (replace the options file as required): 348 \begin{shellCode} 349 scons -j1 options_file=scons/templates/freebsd10.0_options.py 350 \end{shellCode} 351 352 \emph{Note:} Some packages installed above are built with gcc 4.7. Somewhere 353 in the toolchain a system-installed gcc library is pulled in which is 354 incompatible with the one from version 4.7 and would prevent escript from 355 executing successfully. As explained in the FreeBSD 356 documentation\footnote{see \url{http://www.freebsd.org/doc/en/articles/custom-gcc/article.html}} 357 this can be fixed by adding a line to \texttt{/etc/libmap.conf}: 358 \begin{shellCode} 359 libgcc_s.so.1 gcc47/libgcc_s.so.1 360 \end{shellCode} 361 362 \end{comment} 363 \subsection{Other Systems / Custom Builds}\label{sec:othersrc} 364 365 \escript has support for a number of optional packages. 366 Some, like \texttt{netcdf} need to be enabled at compile time, while others, such as \texttt{sympy} and the projection packages 367 used in \downunder are checked at run time. 368 For the second type, you can install them at any time (ensuring that python can find them) and they should work. 369 For the first type, you need to modify the options file and recompile with scons. 370 The rest of this section deals with this. 371 372 To avoid having to specify the options file each time you run scons, copy an existing \texttt{_options.py} file from the 373 \texttt{scons/} or \texttt{scons/templates/} directories. Put the file in the \texttt{scons} directory and name 374 it \textit{yourmachinename}\texttt{_options.py}.\footnote{If the name 375 has - or other non-alpha characters, they must be replaced with underscores in the filename}. 376 For example: on a machine named toybox, the file would be \texttt{scons/toybox_options.py}. 377 378 Individual lines can be enabled/disabled, by removing or adding \# (the python comment character) to the beginning of the line. 379 For example, to enable OpenMP, change the line 380 \begin{verbatim} 381 #openmp = True 382 \end{verbatim} 383 to 384 \begin{verbatim} 385 openmp = True 386 \end{verbatim} 387 388 If you are using libraries which are not installed in the standard places (or have different names) you will need to 389 change the relevant lines. 390 A common need for this would be using a more recent version of the boost::python library. 391 You can also change the compiler or the options passed to it by modifying the relevant lines. 392 393 \subsubsection*{MPI} 394 If you wish to enable or disable MPI, or if you wish to use a different implementation of MPI, you can use the \texttt{mpi} 395 configuration variable. 396 You will also need to ensure that the \texttt{mpi_prefix} and \texttt{mpi_libs} variables are uncommented and set correctly. 397 To disable MPI use, \verb|mpi = 'none'|. 398 399 \subsubsection{Python3} 400 \escript works with \texttt{python3} but until recently, many distributions have not distributed python3 versions of their packages. 401 You can try it out though by modifying or adding the following variables in your options file: 402 403 \begin{verbatim} 404 pythoncmd='python3' 405 \end{verbatim} 406 407 \begin{verbatim} 408 usepython3=True 409 \end{verbatim} 410 411 \begin{verbatim} 412 pythonlibname='whatever_your_python3_library_is_called' 413 \end{verbatim} 414 415 416 417 418 \subsubsection{Testing} 419 As indicated earlier, you can test your build using \texttt{scons py_tests}. 420 Note however, that some features like \texttt{netCDF} are optional for using \escript, the tests will report a failure if 421 they are missing. 422 423 \section{Cleaning up} 424 \label{sec:cleanup} 425 426 Once the build (and optional testing) is complete, you can remove everything except: 427 \begin{itemize} 428 \item bin 429 \item esys 430 \item lib 431 \item doc 432 \item CREDITS 433 \item LICENSE 434 \item README 435 \end{itemize} 436 The last three aren't strictly required for operation. 437 The \texttt{doc} directory is not required either but does contain examples of escript scripts. 438 439 You can run escript using \texttt{\textit{path_to_escript_files}/bin/run-escript}. 440 Where \texttt{\textit{path_to_escript_files}} is replaced with the real path. 441 442 \begin{optionalstep} 443 You can add the escript \texttt{bin} directory to your \texttt{PATH} variable. 444 The launcher will then take care of the rest of the environment. 445 \end{optionalstep} 446 447 \section{Optional Extras} 448 449 Some other packages which might be useful include: 450 \begin{itemize} 451 \item Lapack and UMFPACK --- direct solvers (install the relevant libraries and enable them in the options file). 452 \item support for the Silo file format (install the relevant libraries and enable them in the options file). 453 \item VisIt --- visualisation package. Can be used independently but our \texttt{weipa} library can make a Visit 454 plug-in to allow direct visualisation of escript simulations. 455 \item gmsh --- meshing software used by our \texttt{pycad} library. 456 \item Mayavi2 --- another visualisation tool. 457 \end{itemize} 458 459 460 \subsection{Trilinos} 461 \escript has now some support for Trilinos\footnote{\url{https://trilinos.org/}} 462 solvers and preconditioners. 463 The most significant limitation is that the current Trilinos release does not 464 support block matrices so \escript can only use Trilinos solvers for single 465 PDEs (i.e. no PDE systems). 466 467 If your distribution does not provide Trilinos packages you can build a working 468 version from source. 469 \ldots 470 %provide cmake command here 471