# Contents of /trunk/doc/install/source.tex

Revision 4555 - (show annotations)
Tue Dec 3 03:54:20 2013 UTC (5 years, 9 months ago) by sshaw
File MIME type: application/x-tex
File size: 22229 byte(s)
updated doc install instructions to match that required

 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2 % Copyright (c) 2012-2013 by University of Queensland 3 4 % 5 % Primary Business: Queensland, Australia 6 % Licensed under the Open Software License version 3.0 7 8 % 9 % Development until 2012 by Earth Systems Science Computational Center (ESSCC) 10 % Development since 2012 by School of Earth Sciences 11 % 12 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 13 14 % Notes about compilers 15 16 \chapter{Installing from Source}\label{chap:source} 17 18 This chapter assumes you are using a unix/posix like system. 19 20 \section{Intro} 21 22 Some initial questions: 23 \begin{enumerate} 24 \item Are you using MacOS? (If not, then skip to Section~\ref{sec:par}). 25 \item Which parallel technologies do you wish to use? 26 \item Which packaging system are you using? 27 \end{enumerate} 28 29 30 31 \subsection{MacOS and compilers} 32 \emph{This information is as accurate as far as we can tell at time of writing but things may change.} 33 34 In order to install \escript from source, you need a compiler. 35 This is a bit harder on MacOS than on other systems which either provide a compiler as part of the base install (BSD) or allow 36 people to download one as part of the install process (Linux). 37 In earlier releases of its operating system (Snow Leopard'' and earlier), Apple included an optional developer tools'' package (\texttt{XCode}) on the install media. 38 For more recent releases, \texttt{XCode} is available for free'' from Apple's application store. 39 Why free''? 40 The only way to access the application store is with an AppleID which requires either: 41 \begin{itemize} 42 \item purchasing an iTunes gift card. 43 \item giving Apple access to your credit card. 44 \item signing up as an Apple developer\footnote{If you do this you can download a command line tools'' package 45 which installs the relevant compilers without needing to install all of \texttt{XCode}.} and giving up personal information. 46 \end{itemize} 47 48 If you install \texttt{XCode}, you will need to download the command line tools'' optional package [see \texttt{XCode} documentation for details]. 49 50 There are also a number of projects on the net which aim to deliver compilers for MacOS. 51 Use at your own risk. 52 For example: 53 \begin{itemize} 54 \item \url{http://hpc.sourceforge.net} 55 \item \url{http://kennethreitz.org/experiments/xcode-gcc-and-homebrew} 56 \end{itemize} 57 58 59 \subsection{Parallel Technologies}\label{sec:par} 60 It is extremely likely that the computer you run \escript on, will have more than one processor core. 61 \escript can make use of multiple cores [in order to solve problems more quickly] if it is told to do so, 62 but this functionality must be enabled at compile time. 63 64 There are two technologies which \escript can employ here. 65 \begin{itemize} 66 \item OpenMP -- more efficient of the two [thread level parallelism]. 67 \item MPI -- Uses multiple processes (less efficient), needs less help from the compiler. 68 \end{itemize} 69 70 Escript is primarily tested on recent versions of the GNU or Intel suites (gcc, g++'' / icc, icpc''). 71 However, it also passes our tests when compiled using clang, clang++''. 72 The table below shows what methods are available with which compilers. 73 74 \begin{center} 75 \begin{tabular}{|l|c|c|c|}\hline 76 & Serial & OpenMP & MPI \\\hline 77 $\leq$ gcc-4.2.1 & \checkmark & \raisebox{-0.1cm}{\footnotemark}& \checkmark \\\hline 78 gcc (recent $\geq 4.3.2$) & \checkmark& \checkmark& \checkmark \\\hline 79 icc(10) & \checkmark& \checkmark& \checkmark \\\hline 80 icc(11) & \checkmark& \raisebox{-0.1cm}{\footnotemark} &\checkmark \\\hline 81 icc(12) & \checkmark& \checkmark&\checkmark \\\hline 82 clang & \checkmark& & \checkmark\\\hline 83 \end{tabular} 84 \end{center} 85 \addtocounter{footnote}{-1} 86 \footnotetext{The \openmp support in gcc-4.2.1 is buggy/non-functional.} 87 \addtocounter{footnote}{1} 88 \footnotetext{There is a subtle bug in icc-11 when \openmp and c++ exception handling 89 are combined.} 90 91 \noindent Where both \openmp and \mpi are marked, \escript can be compiled with either or both. 92 A \checkmark mark means that combination passes our tests. 93 94 95 96 \subsection{Packaging System} 97 The packaging system (also known as the package manager) is the tool you use to search for and install new open source software. 98 For Linux, there will be one set up by default: the apt tools on Debian and Ubuntu, yast on Suse, yum on the RedHat family. 99 On BSD systems this will be a combination of \texttt{pkg_add} and the \texttt{ports} tree. 100 101 For MacOS, this is a bit more tricky. 102 There are a number of possible systems including \texttt{macports} and \texttt{homebrew}\footnote{There is also \texttt{fink}, 103 but we have not experimented with that.}, but they do not come pre-installed so if you want to make use of one you will need 104 you will need to install it. 105 106 Packaging systems will make changes to your computer based on programs configured by other people from 107 various places around the internet. 108 It is important to satisfy yourself as to the security of those systems. 109 110 If you are using Linux or have decided that you don't want to use OpenMP skip to Section~\ref{sec:build}. 111 112 \subsubsection*{Changing Compilers on MacOS and BSD} 113 \emph{Not for the faint-hearted.} 114 115 In order to use OpenMP, your compiler must support it. 116 The compilers provided in Apple's package and BSD do not support OpenMP. 117 \texttt{Clang} has no support for it at all, while \texttt{GCC4.2.1} has some, 118 but it does not work very well\footnote{Rejects valid code, crashes etc}. 119 You can use the packaging system to install a more up to date version of \texttt{gcc}. 120 However, you will need to install a number of other dependencies in order for escript to operate. 121 If these dependencies end up fighting over which versions of libraries to use, there will be problems. 122 \footnote{For example: Trying to use two versions of \texttt{Python} in the same program 123 or two versions of the \texttt{c++} standard library.} 124 To avoid this, if you are intend to change the compiler or python versions, you should install the new compiler first, 125 then the new version of python. 126 After this, you can install the other dependencies. 127 \emph{Please note that none of the Mac based package managers nor BSD recommend changing the default $C$ compiler so 128 do so at your own risk. If you want OpenMP though, there does not seem to be a choice.} 129 130 The following steps seemed to work when we tested them but we cannot make guarantees. 131 132 \subsubsection*{Changing compiler on FreeBSD(9.1)} 133 The following sequence passes our unit tests under OpenMP. 134 \begin{itemize} 135 \item Install \texttt{gcc46} from ports. 136 \item Modify \texttt{/etc/make.conf} to set the default compiler to be \texttt{gcc46} 137 \item Install the remaining dependencies. 138 \item Configure \escript to build with \openmp. 139 \end{itemize} 140 141 We chose version $4.6$ rather than a later one because one of the optional dependencies (scipy) will try to install it anyway. 142 143 \subsection*{Changing compiler on MacPorts} 144 The following sequence passes our unit tests under OpenMP. 145 \begin{itemize} 146 \item \texttt{port install gcc47} 147 \item Set the default compiler for the command line with: \texttt{port select --set gcc mp-gcc47} 148 \item Set the default compiler for macports by adding the following line to \hfill~\linebreak[4] \file{/opt/local/etc/macports/macports.conf}: 149 \begin{shellCode} 150 #Added by user to coerce macports into using a newer compiler 151 default_compiler macports-gcc-4.7 152 \end{shellCode} 153 \item Now install the remainder of the dependencies using \texttt{port -s install X}. 154 This will build them from source rather than downloading precompiled versions. 155 (This will unfortunately mean larger downloads.). 156 In some cases, some dependencies will not build properly from source. 157 In those cases(where XYZ is the dependency that fails to build), \texttt{port clean XYZ} then \texttt{port install XYZ}. 158 Then \emph{try to install the rest from source}. 159 160 Installing from source via macports is necessary to convince macports to link against the libraries provided by 161 your new compiler rather than the default one. 162 \end{itemize} 163 164 165 166 167 \subsection*{Changing compiler on Homebrew} 168 There is no configuration file that needs to be changed in order to use a different compiler. 169 You do need to do things in the correct order and make sure that you have made any required changes to 170 your environment variables before moving on to the next step. 171 In particular, the compiler must be installed before anything else, followed by Python. 172 173 174 % make it clear that escript can be customised to use whatever you have 175 176 %Should include optional customisation 177 178 %Talk about options_files and the ability to specify them 179 180 %talk about -j1 and replacing it with more ops 181 %Talk about installation prefix 182 183 %also note that this doesn't build the doco but we do have downloads for that or you can install extra packages 184 185 \section{Building}\label{sec:build} 186 187 To simplify things for people, we have prepared \texttt{_options.py} files for a number of 188 systems\footnote{These are correct a time of writing but later versions of those systems may require tweaks. 189 Also, these systems represent a cross section of possible platforms rather than meaning those systems get particular support.}. 190 If your particular system is not in the above list, or if you want a more customised 191 build\footnote{for example, you want MPI functionality or you wish to use a different compiler}, 192 see Section~\ref{sec:othersrc} for instructions. 193 \begin{itemize} 194 \item Debian - \ref{sec:debsrc} 195 \item Ubuntu - \ref{sec:ubsrc} 196 \item OpenSuse - \ref{sec:susesrc} 197 \item Centos - \ref{sec:centossrc} 198 \item Fedora - \ref{sec:fedorasrc} 199 \item MacOS (macports) - \ref{sec:macportsrc} 200 \item MacOS (homebrew) - \ref{sec:homebrewsrc} 201 \item FreeBSD - \ref{sec:freebsdsrc} 202 \end{itemize} 203 204 Once these are done proceed to Section~\ref{sec:cleanup} for cleanup steps. 205 206 All of these instructions assume that you have obtained the source (uncompressed it if necessary). 207 \subsection{Debian}\label{sec:debsrc} 208 209 \begin{shellCode} 210 sudo aptitude install python-dev python-numpy libboost-python-dev libnetcdf-dev 211 sudo aptitude install scons lsb-release 212 sudo aptitude install python-sympy python-matplotlib python-scipy 213 sudo aptitude install python-pyproj python-gdal 214 \end{shellCode} 215 216 217 \begin{optionalstep} 218 If for some reason, you wish to rebuild the documentation, you would also need the following: 219 \begin{shellCode} 220 sudo aptitude install python-sphinx doxygen python-docutils texlive 221 sudo aptitude install zip texlive-latex-extra latex-xcolor 222 \end{shellCode} 223 \end{optionalstep} 224 225 \noindent In the source directory execute the following (substitute squeeze or wheezy as appropriate for XXXX): 226 \begin{shellCode} 227 scons -j1 options_file=scons/os/XXXX_options.py 228 \end{shellCode} 229 230 \noindent If you wish to test your build, you can use the following: 231 \begin{shellCode} 232 scons -j1 py_tests options_file=scons/os/XXXX_options.py 233 \end{shellCode} 234 235 \subsection{Ubuntu}\label{sec:ubsrc} 236 237 If you have not installed \texttt{aptitude}, then substitute \texttt{apt-get} in the following. 238 \begin{shellCode} 239 sudo aptitude install python-dev python-numpy libboost-python-dev libnetcdf-dev 240 sudo aptitude install scons lsb-release 241 sudo aptitude install python-sympy python-matplotlib python-scipy 242 sudo aptitude install python-pyproj python-gdal 243 \end{shellCode} 244 245 246 \begin{optionalstep} 247 If for some reason, you wish to rebuild the documentation, you would also need the following: 248 \begin{shellCode} 249 sudo aptitude install python-sphinx doxygen python-docutils texlive 250 sudo aptitude install zip texlive-latex-extra latex-xcolor 251 \end{shellCode} 252 \end{optionalstep} 253 254 \noindent In the source directory execute the following (substitute precise, quantal or raring as appropriate for XXXX): 255 \begin{shellCode} 256 scons -j1 options_file=scons/os/XXXX_options.py 257 \end{shellCode} 258 259 \noindent If you wish to test your build, you can use the following: 260 \begin{shellCode} 261 scons -j1 py_tests options_file=scons/os/XXXX_options.py 262 \end{shellCode} 263 264 265 266 \subsection{OpenSuse}\label{sec:susesrc} 267 These instructions were prepared using release $12.3$. 268 269 \noindent Install packages from the main distribution: 270 \begin{shellCode} 271 sudo yast2 --install libboost_python1_49_0 python-devel python-numpy 272 sudo yast2 --install python-scipy python-sympy python-matplotlib libnetcdf_c++-devel 273 sudo yast2 --install gcc-c++ scons boost-devel netcdf-devel 274 \end{shellCode} 275 These will allow you to use most features except some parts of the \downunder inversion library. 276 If you wish to use those, you will need some additional packages [python-pyproj, python-gdal]. 277 This can be done after Escript installation. 278 279 \begin{optionalstep} 280 Add \url{http://ftp.suse.de/pub/opensuse/repositories/Application:/Geo/openSUSE_12.3/} 281 to your repositories in \texttt{YaST}. 282 \begin{shellCode} 283 sudo yast2 --install python-pyproj, python-gdal 284 \end{shellCode} 285 \end{optionalstep} 286 287 Now to build escript itself. 288 In the escript source directory: 289 \begin{shellCode} 290 scons -j1 options_file=scons/os/opensuse12.3_options.py 291 \end{shellCode} 292 293 \noindent If you wish to test your build, you can use the following: 294 \begin{shellCode} 295 scons -j1 py_tests options_file=scons/os/opensuse12.3_options.py 296 \end{shellCode} 297 298 \noindent Now go to Section~\ref{sec:cleanup} for cleanup. 299 300 \subsection{Centos}\label{sec:centossrc} 301 These instructions were prepared using release $6.4$. 302 303 \noindent Install packages from the main distribution: 304 \begin{shellCode} 305 yum install python-devel numpy scipy scons boost-devel 306 yum install python-matplotlib gcc-g++ 307 yum install boost-python 308 \end{shellCode} 309 310 The above packages will allow you to use most features except saving and loading files in \texttt{netCDF} 311 format and the \downunder inversion library. 312 If you wish to use those features, you will need to install some additional packages. 313 NetCDF needs to be installed when you compile if you wish to use it. 314 \begin{optionalstep} 315 \noindent Add the \texttt{EPEL} repository. 316 \begin{shellCode} 317 rpm -U http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm 318 \end{shellCode} 319 320 \begin{shellCode} 321 yum install netcdf-devel sympy gdal-python 322 \end{shellCode} 323 \end{optionalstep} 324 325 \noindent For some coordinate transformations, \downunder can also make use of the python interface to a tool called \texttt{proj}. 326 There does not seem to be an obvious centos repository for this though. 327 If it turns out to be necessary for your particular application, the source can be downloaded. 328 329 \noindent Now to build escript itself. 330 In the escript source directory: 331 \begin{shellCode} 332 scons -j1 options_file=scons/os/centos6.4_options.py 333 \end{shellCode} 334 335 \noindent If you wish to test your build, you can use the following: 336 \begin{shellCode} 337 scons -j1 py_tests options_file=scons/os/centos6.4_options.py 338 \end{shellCode} 339 340 \noindent Now go to Section~\ref{sec:cleanup} for cleanup. 341 342 \subsection{Fedora}\label{sec:fedorasrc} 343 These instructions were prepared using release $18$. 344 345 \noindent Install packages 346 \begin{shellCode} 347 yum install netcdf-cxx-devel gcc-c++ scipy 348 yum install sympy scons pyproj gdal python-matplotlib 349 yum install boost-devel 350 \end{shellCode} 351 352 \noindent Now to build escript itself. 353 In the escript source directory: 354 \begin{shellCode} 355 scons -j1 options_file=scons/os/fedora18_options.py 356 \end{shellCode} 357 358 \noindent If you wish to test your build, you can use the following: 359 \begin{shellCode} 360 scons -j1 py_tests options_file=scons/os/fedora18_options.py 361 \end{shellCode} 362 363 \noindent Now go to Section~\ref{sec:cleanup} for cleanup. 364 365 \subsection{MacOS (macports)}\label{sec:macportsrc} 366 367 \begin{shellCode} 368 port install python27 369 port select --set python python27 370 port install scons 371 port install openmpi 372 port install py27-numpy 373 port install boost 374 port install py27-sympy 375 port select --set py-sympy py27-sympy 376 install py27-scipy 377 install py27-pyproj 378 install py27-gdal 379 install py27-netcdf4 380 install netcdf-cxx 381 \end{shellCode} 382 383 \begin{shellCode} 384 scons -j1 options_file=scons/os/macports_options.py 385 \end{shellCode} 386 387 388 \subsection{MacOS (homebrew)}\label{sec:homebrewsrc} 389 390 Note that these steps add non-official'' packages. 391 You will also want to make sure that the homebrew Python is executed in preference to the system 392 Python\footnote{Putting \texttt{/usr/local/bin} at the front of your PATH is one way to do this.}. 393 394 \begin{shellCode} 395 brew install scons 396 brew install python 397 brew install boost 398 brew tap samueljohn/python 399 brew tap homebrew/science 400 pip install nose 401 brew install gfortran 402 brew install samueljohn/python/numpy 403 brew install scipy 404 brew install gdal 405 brew install openmpi 406 brew install matplotlib 407 brew install netcdf --enable-cxx-compat 408 \end{shellCode} 409 410 There do not appear to be formulae for \texttt{sympy} or \texttt{pyproj} so if you wish to use those features, then 411 you will need to install them separately. 412 413 414 \begin{shellCode} 415 scons -j1 options_file=scons/os/homebrew_options.py 416 \end{shellCode} 417 418 419 \subsection{FreeBSD}\label{sec:freebsdsrc} 420 The following was tested on the $9.1$ release of FreeBSD. 421 It passes the majority of tests but there is an issue related to some features in the inversion library. 422 The following set of installations works'' but is not guaranteed to be minimal\footnote{Depending on your needs you might be able to 423 get by with a smaller set of packages.}. 424 425 Install the following packages: 426 \begin{itemize} 427 \item subversion 428 \item scons 429 \item boost-python-libs 430 \item bash 431 \end{itemize} 432 433 Now install the following ports: 434 \begin{itemize} 435 \item science/py-scipy 436 \item science/py-netCDF4 437 \item math/sympy 438 \item graphics/py-pyproj 439 \item graphics/py-gdal 440 \item net/openmpi 441 \end{itemize} 442 443 You will need to add \texttt{/usr/local/mpi/openmpi/bin} to your path if you wish to build with MPI. 444 445 Next choose (or create) your options file. 446 In this case we have three prepared in the \texttt{scons/os} directory: 447 \begin{itemize} 448 \item \texttt{freebsd91_options.py} 449 \item \texttt{freebsd91_mpi_options.py} If you would like to use MPI. 450 \item \texttt{freebsd91_gcc46_options.py} Use this if you have managed to change compilers to gcc4.6 (and would like to use OpenMP). 451 \end{itemize} 452 453 In the escript source directory (where ZZZ is your options file): 454 \begin{shellCode} 455 scons -j1 options_file=ZZZ 456 \end{shellCode} 457 458 459 460 \subsection{Other Systems / Custom Builds}\label{sec:othersrc} 461 462 \escript has support for a number of optional packages. 463 Some, like \texttt{netcdf} need to be enabled at compile time, while others, such as \texttt{sympy} and the projection packages 464 used in \downunder are checked at run time. 465 For the second type, you can install them at any time (ensuring that python can find them) and they should work. 466 For the first type, you need to modify the options file and recompile with scons. 467 The rest of this section deals with this. 468 469 To avoid having to specify the options file each time you run scons, copy an existing \texttt{_options.py} file from the 470 \texttt{scons/} or \texttt{scons/os/} directories. Put the file in the \texttt{scons} directory and name 471 it \textit{yourmachinename}\texttt{_options.py}.\footnote{If the name 472 has - or other non-alpha characters, they must be replaced with underscores in the filename}. 473 For example: on a machine named toybox, the file would be \texttt{scons/toybox_options.py}. 474 475 Individual lines can be enabled/disabled, by removing or adding \# (the python comment character) to the beginning of the line. 476 For example, to enable OpenMP, change the line 477 \begin{verbatim} 478 #openmp = True 479 \end{verbatim} 480 to 481 \begin{verbatim} 482 openmp = True 483 \end{verbatim}. 484 485 If you are using libraries which are not installed in the standard places (or have different names) you will need to 486 change the relevant lines. 487 A common need for this would be using a more recent version of the boostpython library. 488 489 You can also change the compiler or the options passed to it by modifying the relevant lines. 490 491 \subsubsection*{MPI} 492 If you wish to enable or disable MPI, or if you wish to use a different implementation of MPI, you can use the \texttt{mpi} 493 configuration variable. 494 To disable MPI use, \verb|mpi = 'none'|. 495 You will also need to ensure that the \texttt{mpi_prefix} and \texttt{mpi_libs} variables are uncommented and set correctly. 496 497 \subsubsection{Python3} 498 \escript works with \texttt{python3} but until recently, many distributions have not distributed python3 versions of their packages. 499 You can try it out though by modifying the following variables: 500 501 \begin{verbatim} 502 pythoncmd='python3' 503 \end{verbatim} 504 505 \begin{verbatim} 506 usepython3=True 507 \end{verbatim} 508 509 \begin{verbatim} 510 pythonlibname='whateveryourpython3libraryiscalled' 511 \end{verbatim} 512 513 514 515 516 \subsubsection{Testing} 517 As indicated earlier, you can test your build using \texttt{scons py_tests}. 518 Note however, that some features like texttt{netCDF} are optional for using \escript, the tests will report a failure if 519 they are missing. 520 521 \section{Cleaning up} 522 \label{sec:cleanup} 523 524 Once the build (and optional testing) is complete, you can remove everything except: 525 \begin{itemize} 526 \item bin 527 \item esys 528 \item lib 529 \item doc 530 \item CREDITS.TXT 531 \item README_LICENSE 532 \end{itemize} 533 The last two aren't strictly required for operation. 534 The \texttt{doc} directory is not required either but does contain examples of escript scripts. 535 536 You can run escript using \texttt{\textit{path_to_escript_files}/bin/run-escript}. 537 Where \texttt{\textit{path_to_escript_files}} is replaced with the real path. 538 539 \begin{optionalstep} 540 You can add the escript \texttt{bin} directory to your \texttt{PATH} variable. 541 The launcher will then take care of the rest of the environment. 542 \end{optionalstep} 543 544 \section{Optional Extras} 545 546 Some other packages which might be useful include: 547 \begin{itemize} 548 \item support for silo format (install the relevant libraries and enable them in the options file). 549 \item Visit --- visualisation package. Can be used independently but our \texttt{weipa} library can make a Visit 550 plug-in to allow direct visualisation of escript files. 551 \item gmsh --- meshing software used by our \texttt{pycad} library. 552 \item mayavi --- another visualisation tool. 553 \end{itemize} 554 555 556 %Need a better title but this is stuff like visit and silo (for non-debian distros) 557 %Perhaps - optional extras 558 559 560 561 562

 ViewVC Help Powered by ViewVC 1.1.26