ViewVC logotype

Contents of /trunk/doc/install/source.tex

Parent Directory Parent Directory | Revision Log Revision Log

Revision 4555 - (show annotations)
Tue Dec 3 03:54:20 2013 UTC (5 years, 9 months ago) by sshaw
File MIME type: application/x-tex
File size: 22229 byte(s)
updated doc install instructions to match that required
1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
2 % Copyright (c) 2012-2013 by University of Queensland
3 % http://www.uq.edu.au
4 %
5 % Primary Business: Queensland, Australia
6 % Licensed under the Open Software License version 3.0
7 % http://www.opensource.org/licenses/osl-3.0.php
8 %
9 % Development until 2012 by Earth Systems Science Computational Center (ESSCC)
10 % Development since 2012 by School of Earth Sciences
11 %
12 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
14 % Notes about compilers
16 \chapter{Installing from Source}\label{chap:source}
18 This chapter assumes you are using a unix/posix like system.
20 \section{Intro}
22 Some initial questions:
23 \begin{enumerate}
24 \item Are you using MacOS? (If not, then skip to Section~\ref{sec:par}).
25 \item Which parallel technologies do you wish to use?
26 \item Which packaging system are you using?
27 \end{enumerate}
31 \subsection{MacOS and compilers}
32 \emph{This information is as accurate as far as we can tell at time of writing but things may change.}
34 In order to install \escript from source, you need a compiler.
35 This is a bit harder on MacOS than on other systems which either provide a compiler as part of the base install (BSD) or allow
36 people to download one as part of the install process (Linux).
37 In earlier releases of its operating system (``Snow Leopard'' and earlier), Apple included an optional ``developer tools'' package (\texttt{XCode}) on the install media.
38 For more recent releases, \texttt{XCode} is available for ``free'' from Apple's application store.
39 Why ``free''?
40 The only way to access the application store is with an AppleID which requires either:
41 \begin{itemize}
42 \item purchasing an iTunes gift card.
43 \item giving Apple access to your credit card.
44 \item signing up as an Apple developer\footnote{If you do this you can download a ``command line tools'' package
45 which installs the relevant compilers without needing to install all of \texttt{XCode}.} and giving up personal information.
46 \end{itemize}
48 If you install \texttt{XCode}, you will need to download the ``command line tools'' optional package [see \texttt{XCode} documentation for details].
50 There are also a number of projects on the net which aim to deliver compilers for MacOS.
51 Use at your own risk.
52 For example:
53 \begin{itemize}
54 \item \url{http://hpc.sourceforge.net}
55 \item \url{http://kennethreitz.org/experiments/xcode-gcc-and-homebrew}
56 \end{itemize}
59 \subsection{Parallel Technologies}\label{sec:par}
60 It is extremely likely that the computer you run \escript on, will have more than one processor core.
61 \escript can make use of multiple cores [in order to solve problems more quickly] if it is told to do so,
62 but this functionality must be enabled at compile time.
64 There are two technologies which \escript can employ here.
65 \begin{itemize}
66 \item OpenMP -- more efficient of the two [thread level parallelism].
67 \item MPI -- Uses multiple processes (less efficient), needs less help from the compiler.
68 \end{itemize}
70 Escript is primarily tested on recent versions of the GNU or Intel suites (``gcc, g++'' / ``icc, icpc'').
71 However, it also passes our tests when compiled using ``clang, clang++''.
72 The table below shows what methods are available with which compilers.
74 \begin{center}
75 \begin{tabular}{|l|c|c|c|}\hline
76 & Serial & OpenMP & MPI \\\hline
77 $\leq$ gcc-4.2.1 & \checkmark & \raisebox{-0.1cm}{\footnotemark}& \checkmark \\\hline
78 gcc (recent $\geq 4.3.2$) & \checkmark& \checkmark& \checkmark \\\hline
79 icc(10) & \checkmark& \checkmark& \checkmark \\\hline
80 icc(11) & \checkmark& \raisebox{-0.1cm}{\footnotemark} &\checkmark \\\hline
81 icc(12) & \checkmark& \checkmark&\checkmark \\\hline
82 clang & \checkmark& & \checkmark\\\hline
83 \end{tabular}
84 \end{center}
85 \addtocounter{footnote}{-1}
86 \footnotetext{The \openmp support in gcc-4.2.1 is buggy/non-functional.}
87 \addtocounter{footnote}{1}
88 \footnotetext{There is a subtle bug in icc-11 when \openmp and c++ exception handling
89 are combined.}
91 \noindent Where both \openmp and \mpi are marked, \escript can be compiled with either or both.
92 A \checkmark mark means that combination passes our tests.
96 \subsection{Packaging System}
97 The packaging system (also known as the package manager) is the tool you use to search for and install new open source software.
98 For Linux, there will be one set up by default: the apt tools on Debian and Ubuntu, yast on Suse, yum on the RedHat family.
99 On BSD systems this will be a combination of \texttt{pkg_add} and the \texttt{ports} tree.
101 For MacOS, this is a bit more tricky.
102 There are a number of possible systems including \texttt{macports} and \texttt{homebrew}\footnote{There is also \texttt{fink},
103 but we have not experimented with that.}, but they do not come pre-installed so if you want to make use of one you will need
104 you will need to install it.
106 Packaging systems will make changes to your computer based on programs configured by other people from
107 various places around the internet.
108 It is important to satisfy yourself as to the security of those systems.
110 If you are using Linux or have decided that you don't want to use OpenMP skip to Section~\ref{sec:build}.
112 \subsubsection*{Changing Compilers on MacOS and BSD}
113 \emph{Not for the faint-hearted.}
115 In order to use OpenMP, your compiler must support it.
116 The compilers provided in Apple's package and BSD do not support OpenMP.
117 \texttt{Clang} has no support for it at all, while \texttt{GCC4.2.1} has some,
118 but it does not work very well\footnote{Rejects valid code, crashes etc}.
119 You can use the packaging system to install a more up to date version of \texttt{gcc}.
120 However, you will need to install a number of other dependencies in order for escript to operate.
121 If these dependencies end up fighting over which versions of libraries to use, there will be problems.
122 \footnote{For example: Trying to use two versions of \texttt{Python} in the same program
123 or two versions of the \texttt{c++} standard library.}
124 To avoid this, if you are intend to change the compiler or python versions, you should install the new compiler first,
125 then the new version of python.
126 After this, you can install the other dependencies.
127 \emph{Please note that none of the Mac based package managers nor BSD recommend changing the default $C$ compiler so
128 do so at your own risk. If you want OpenMP though, there does not seem to be a choice.}
130 The following steps seemed to work when we tested them but we cannot make guarantees.
132 \subsubsection*{Changing compiler on FreeBSD(9.1)}
133 The following sequence passes our unit tests under OpenMP.
134 \begin{itemize}
135 \item Install \texttt{gcc46} from ports.
136 \item Modify \texttt{/etc/make.conf} to set the default compiler to be \texttt{gcc46}
137 \item Install the remaining dependencies.
138 \item Configure \escript to build with \openmp.
139 \end{itemize}
141 We chose version $4.6$ rather than a later one because one of the optional dependencies (scipy) will try to install it anyway.
143 \subsection*{Changing compiler on MacPorts}
144 The following sequence passes our unit tests under OpenMP.
145 \begin{itemize}
146 \item \texttt{port install gcc47}
147 \item Set the default compiler for the command line with: \texttt{port select --set gcc mp-gcc47}
148 \item Set the default compiler for macports by adding the following line to \hfill~\linebreak[4] \file{/opt/local/etc/macports/macports.conf}:
149 \begin{shellCode}
150 #Added by user to coerce macports into using a newer compiler
151 default_compiler macports-gcc-4.7
152 \end{shellCode}
153 \item Now install the remainder of the dependencies using \texttt{port -s install X}.
154 This will build them from source rather than downloading precompiled versions.
155 (This will unfortunately mean larger downloads.).
156 In some cases, some dependencies will not build properly from source.
157 In those cases(where XYZ is the dependency that fails to build), \texttt{port clean XYZ} then \texttt{port install XYZ}.
158 Then \emph{try to install the rest from source}.
160 Installing from source via macports is necessary to convince macports to link against the libraries provided by
161 your new compiler rather than the default one.
162 \end{itemize}
167 \subsection*{Changing compiler on Homebrew}
168 There is no configuration file that needs to be changed in order to use a different compiler.
169 You do need to do things in the correct order and make sure that you have made any required changes to
170 your environment variables before moving on to the next step.
171 In particular, the compiler must be installed before anything else, followed by Python.
174 % make it clear that escript can be customised to use whatever you have
176 %Should include optional customisation
178 %Talk about options_files and the ability to specify them
180 %talk about -j1 and replacing it with more ops
181 %Talk about installation prefix
183 %also note that this doesn't build the doco but we do have downloads for that or you can install extra packages
185 \section{Building}\label{sec:build}
187 To simplify things for people, we have prepared \texttt{_options.py} files for a number of
188 systems\footnote{These are correct a time of writing but later versions of those systems may require tweaks.
189 Also, these systems represent a cross section of possible platforms rather than meaning those systems get particular support.}.
190 If your particular system is not in the above list, or if you want a more customised
191 build\footnote{for example, you want MPI functionality or you wish to use a different compiler},
192 see Section~\ref{sec:othersrc} for instructions.
193 \begin{itemize}
194 \item Debian - \ref{sec:debsrc}
195 \item Ubuntu - \ref{sec:ubsrc}
196 \item OpenSuse - \ref{sec:susesrc}
197 \item Centos - \ref{sec:centossrc}
198 \item Fedora - \ref{sec:fedorasrc}
199 \item MacOS (macports) - \ref{sec:macportsrc}
200 \item MacOS (homebrew) - \ref{sec:homebrewsrc}
201 \item FreeBSD - \ref{sec:freebsdsrc}
202 \end{itemize}
204 Once these are done proceed to Section~\ref{sec:cleanup} for cleanup steps.
206 All of these instructions assume that you have obtained the source (uncompressed it if necessary).
207 \subsection{Debian}\label{sec:debsrc}
209 \begin{shellCode}
210 sudo aptitude install python-dev python-numpy libboost-python-dev libnetcdf-dev
211 sudo aptitude install scons lsb-release
212 sudo aptitude install python-sympy python-matplotlib python-scipy
213 sudo aptitude install python-pyproj python-gdal
214 \end{shellCode}
217 \begin{optionalstep}
218 If for some reason, you wish to rebuild the documentation, you would also need the following:
219 \begin{shellCode}
220 sudo aptitude install python-sphinx doxygen python-docutils texlive
221 sudo aptitude install zip texlive-latex-extra latex-xcolor
222 \end{shellCode}
223 \end{optionalstep}
225 \noindent In the source directory execute the following (substitute squeeze or wheezy as appropriate for XXXX):
226 \begin{shellCode}
227 scons -j1 options_file=scons/os/XXXX_options.py
228 \end{shellCode}
230 \noindent If you wish to test your build, you can use the following:
231 \begin{shellCode}
232 scons -j1 py_tests options_file=scons/os/XXXX_options.py
233 \end{shellCode}
235 \subsection{Ubuntu}\label{sec:ubsrc}
237 If you have not installed \texttt{aptitude}, then substitute \texttt{apt-get} in the following.
238 \begin{shellCode}
239 sudo aptitude install python-dev python-numpy libboost-python-dev libnetcdf-dev
240 sudo aptitude install scons lsb-release
241 sudo aptitude install python-sympy python-matplotlib python-scipy
242 sudo aptitude install python-pyproj python-gdal
243 \end{shellCode}
246 \begin{optionalstep}
247 If for some reason, you wish to rebuild the documentation, you would also need the following:
248 \begin{shellCode}
249 sudo aptitude install python-sphinx doxygen python-docutils texlive
250 sudo aptitude install zip texlive-latex-extra latex-xcolor
251 \end{shellCode}
252 \end{optionalstep}
254 \noindent In the source directory execute the following (substitute precise, quantal or raring as appropriate for XXXX):
255 \begin{shellCode}
256 scons -j1 options_file=scons/os/XXXX_options.py
257 \end{shellCode}
259 \noindent If you wish to test your build, you can use the following:
260 \begin{shellCode}
261 scons -j1 py_tests options_file=scons/os/XXXX_options.py
262 \end{shellCode}
266 \subsection{OpenSuse}\label{sec:susesrc}
267 These instructions were prepared using release $12.3$.
269 \noindent Install packages from the main distribution:
270 \begin{shellCode}
271 sudo yast2 --install libboost_python1_49_0 python-devel python-numpy
272 sudo yast2 --install python-scipy python-sympy python-matplotlib libnetcdf_c++-devel
273 sudo yast2 --install gcc-c++ scons boost-devel netcdf-devel
274 \end{shellCode}
275 These will allow you to use most features except some parts of the \downunder inversion library.
276 If you wish to use those, you will need some additional packages [python-pyproj, python-gdal].
277 This can be done after Escript installation.
279 \begin{optionalstep}
280 Add \url{http://ftp.suse.de/pub/opensuse/repositories/Application:/Geo/openSUSE_12.3/}
281 to your repositories in \texttt{YaST}.
282 \begin{shellCode}
283 sudo yast2 --install python-pyproj, python-gdal
284 \end{shellCode}
285 \end{optionalstep}
287 Now to build escript itself.
288 In the escript source directory:
289 \begin{shellCode}
290 scons -j1 options_file=scons/os/opensuse12.3_options.py
291 \end{shellCode}
293 \noindent If you wish to test your build, you can use the following:
294 \begin{shellCode}
295 scons -j1 py_tests options_file=scons/os/opensuse12.3_options.py
296 \end{shellCode}
298 \noindent Now go to Section~\ref{sec:cleanup} for cleanup.
300 \subsection{Centos}\label{sec:centossrc}
301 These instructions were prepared using release $6.4$.
303 \noindent Install packages from the main distribution:
304 \begin{shellCode}
305 yum install python-devel numpy scipy scons boost-devel
306 yum install python-matplotlib gcc-g++
307 yum install boost-python
308 \end{shellCode}
310 The above packages will allow you to use most features except saving and loading files in \texttt{netCDF}
311 format and the \downunder inversion library.
312 If you wish to use those features, you will need to install some additional packages.
313 NetCDF needs to be installed when you compile if you wish to use it.
314 \begin{optionalstep}
315 \noindent Add the \texttt{EPEL} repository.
316 \begin{shellCode}
317 rpm -U http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
318 \end{shellCode}
320 \begin{shellCode}
321 yum install netcdf-devel sympy gdal-python
322 \end{shellCode}
323 \end{optionalstep}
325 \noindent For some coordinate transformations, \downunder can also make use of the python interface to a tool called \texttt{proj}.
326 There does not seem to be an obvious centos repository for this though.
327 If it turns out to be necessary for your particular application, the source can be downloaded.
329 \noindent Now to build escript itself.
330 In the escript source directory:
331 \begin{shellCode}
332 scons -j1 options_file=scons/os/centos6.4_options.py
333 \end{shellCode}
335 \noindent If you wish to test your build, you can use the following:
336 \begin{shellCode}
337 scons -j1 py_tests options_file=scons/os/centos6.4_options.py
338 \end{shellCode}
340 \noindent Now go to Section~\ref{sec:cleanup} for cleanup.
342 \subsection{Fedora}\label{sec:fedorasrc}
343 These instructions were prepared using release $18$.
345 \noindent Install packages
346 \begin{shellCode}
347 yum install netcdf-cxx-devel gcc-c++ scipy
348 yum install sympy scons pyproj gdal python-matplotlib
349 yum install boost-devel
350 \end{shellCode}
352 \noindent Now to build escript itself.
353 In the escript source directory:
354 \begin{shellCode}
355 scons -j1 options_file=scons/os/fedora18_options.py
356 \end{shellCode}
358 \noindent If you wish to test your build, you can use the following:
359 \begin{shellCode}
360 scons -j1 py_tests options_file=scons/os/fedora18_options.py
361 \end{shellCode}
363 \noindent Now go to Section~\ref{sec:cleanup} for cleanup.
365 \subsection{MacOS (macports)}\label{sec:macportsrc}
367 \begin{shellCode}
368 port install python27
369 port select --set python python27
370 port install scons
371 port install openmpi
372 port install py27-numpy
373 port install boost
374 port install py27-sympy
375 port select --set py-sympy py27-sympy
376 install py27-scipy
377 install py27-pyproj
378 install py27-gdal
379 install py27-netcdf4
380 install netcdf-cxx
381 \end{shellCode}
383 \begin{shellCode}
384 scons -j1 options_file=scons/os/macports_options.py
385 \end{shellCode}
388 \subsection{MacOS (homebrew)}\label{sec:homebrewsrc}
390 Note that these steps add ``non-official'' packages.
391 You will also want to make sure that the homebrew Python is executed in preference to the system
392 Python\footnote{Putting \texttt{/usr/local/bin} at the front of your PATH is one way to do this.}.
394 \begin{shellCode}
395 brew install scons
396 brew install python
397 brew install boost
398 brew tap samueljohn/python
399 brew tap homebrew/science
400 pip install nose
401 brew install gfortran
402 brew install samueljohn/python/numpy
403 brew install scipy
404 brew install gdal
405 brew install openmpi
406 brew install matplotlib
407 brew install netcdf --enable-cxx-compat
408 \end{shellCode}
410 There do not appear to be formulae for \texttt{sympy} or \texttt{pyproj} so if you wish to use those features, then
411 you will need to install them separately.
414 \begin{shellCode}
415 scons -j1 options_file=scons/os/homebrew_options.py
416 \end{shellCode}
419 \subsection{FreeBSD}\label{sec:freebsdsrc}
420 The following was tested on the $9.1$ release of FreeBSD.
421 It passes the majority of tests but there is an issue related to some features in the inversion library.
422 The following set of installations ``works'' but is not guaranteed to be minimal\footnote{Depending on your needs you might be able to
423 get by with a smaller set of packages.}.
425 Install the following packages:
426 \begin{itemize}
427 \item subversion
428 \item scons
429 \item boost-python-libs
430 \item bash
431 \end{itemize}
433 Now install the following ports:
434 \begin{itemize}
435 \item science/py-scipy
436 \item science/py-netCDF4
437 \item math/sympy
438 \item graphics/py-pyproj
439 \item graphics/py-gdal
440 \item net/openmpi
441 \end{itemize}
443 You will need to add \texttt{/usr/local/mpi/openmpi/bin} to your path if you wish to build with MPI.
445 Next choose (or create) your options file.
446 In this case we have three prepared in the \texttt{scons/os} directory:
447 \begin{itemize}
448 \item \texttt{freebsd91_options.py}
449 \item \texttt{freebsd91_mpi_options.py} If you would like to use MPI.
450 \item \texttt{freebsd91_gcc46_options.py} Use this if you have managed to change compilers to gcc4.6 (and would like to use OpenMP).
451 \end{itemize}
453 In the escript source directory (where ZZZ is your options file):
454 \begin{shellCode}
455 scons -j1 options_file=ZZZ
456 \end{shellCode}
460 \subsection{Other Systems / Custom Builds}\label{sec:othersrc}
462 \escript has support for a number of optional packages.
463 Some, like \texttt{netcdf} need to be enabled at compile time, while others, such as \texttt{sympy} and the projection packages
464 used in \downunder are checked at run time.
465 For the second type, you can install them at any time (ensuring that python can find them) and they should work.
466 For the first type, you need to modify the options file and recompile with scons.
467 The rest of this section deals with this.
469 To avoid having to specify the options file each time you run scons, copy an existing \texttt{_options.py} file from the
470 \texttt{scons/} or \texttt{scons/os/} directories. Put the file in the \texttt{scons} directory and name
471 it \textit{yourmachinename}\texttt{_options.py}.\footnote{If the name
472 has - or other non-alpha characters, they must be replaced with underscores in the filename}.
473 For example: on a machine named toybox, the file would be \texttt{scons/toybox_options.py}.
475 Individual lines can be enabled/disabled, by removing or adding \# (the python comment character) to the beginning of the line.
476 For example, to enable OpenMP, change the line
477 \begin{verbatim}
478 #openmp = True
479 \end{verbatim}
480 to
481 \begin{verbatim}
482 openmp = True
483 \end{verbatim}.
485 If you are using libraries which are not installed in the standard places (or have different names) you will need to
486 change the relevant lines.
487 A common need for this would be using a more recent version of the boostpython library.
489 You can also change the compiler or the options passed to it by modifying the relevant lines.
491 \subsubsection*{MPI}
492 If you wish to enable or disable MPI, or if you wish to use a different implementation of MPI, you can use the \texttt{mpi}
493 configuration variable.
494 To disable MPI use, \verb|mpi = 'none'|.
495 You will also need to ensure that the \texttt{mpi_prefix} and \texttt{mpi_libs} variables are uncommented and set correctly.
497 \subsubsection{Python3}
498 \escript works with \texttt{python3} but until recently, many distributions have not distributed python3 versions of their packages.
499 You can try it out though by modifying the following variables:
501 \begin{verbatim}
502 pythoncmd='python3'
503 \end{verbatim}
505 \begin{verbatim}
506 usepython3=True
507 \end{verbatim}
509 \begin{verbatim}
510 pythonlibname='whateveryourpython3libraryiscalled'
511 \end{verbatim}
516 \subsubsection{Testing}
517 As indicated earlier, you can test your build using \texttt{scons py_tests}.
518 Note however, that some features like texttt{netCDF} are optional for using \escript, the tests will report a failure if
519 they are missing.
521 \section{Cleaning up}
522 \label{sec:cleanup}
524 Once the build (and optional testing) is complete, you can remove everything except:
525 \begin{itemize}
526 \item bin
527 \item esys
528 \item lib
529 \item doc
530 \item CREDITS.TXT
532 \end{itemize}
533 The last two aren't strictly required for operation.
534 The \texttt{doc} directory is not required either but does contain examples of escript scripts.
536 You can run escript using \texttt{\textit{path_to_escript_files}/bin/run-escript}.
537 Where \texttt{\textit{path_to_escript_files}} is replaced with the real path.
539 \begin{optionalstep}
540 You can add the escript \texttt{bin} directory to your \texttt{PATH} variable.
541 The launcher will then take care of the rest of the environment.
542 \end{optionalstep}
544 \section{Optional Extras}
546 Some other packages which might be useful include:
547 \begin{itemize}
548 \item support for silo format (install the relevant libraries and enable them in the options file).
549 \item Visit --- visualisation package. Can be used independently but our \texttt{weipa} library can make a Visit
550 plug-in to allow direct visualisation of escript files.
551 \item gmsh --- meshing software used by our \texttt{pycad} library.
552 \item mayavi --- another visualisation tool.
553 \end{itemize}
556 %Need a better title but this is stuff like visit and silo (for non-debian distros)
557 %Perhaps - optional extras

  ViewVC Help
Powered by ViewVC 1.1.26