# Contents of /trunk/doc/install/source.tex

Revision 4439 - (show annotations)
Wed Jun 5 02:09:35 2013 UTC (6 years, 1 month ago) by jfenwick
File MIME type: application/x-tex
File size: 15071 byte(s)
some work on install doco
 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2 % Copyright (c) 2012-2013 by University of Queensland 3 4 % 5 % Primary Business: Queensland, Australia 6 % Licensed under the Open Software License version 3.0 7 8 % 9 % Development until 2012 by Earth Systems Science Computational Center (ESSCC) 10 % Development since 2012 by School of Earth Sciences 11 % 12 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 13 14 % Notes about compilers 15 16 \chapter{Installing from Source}\label{chap:source} 17 18 \section{Intro} 19 20 Some initial questions: 21 \begin{enumerate} 22 \item Are you using MacOS? 23 \item Which parallel technologies do you wish to use? 24 \item Which packaging system are you using? 25 \end{enumerate} 26 27 28 29 \subsection{MacOS and compilers} 30 \emph{This information is as accurate as far as we can tell at time of writing but things may change.} 31 32 In order to install \escript from source, you need a compiler. 33 This is a bit harder on MacOS than on other systems which either provide a compiler as part of the base install (BSD) or allow 34 people to download one as part of the install process (Linux). 35 In earlier releases of its operating system (Snow Leopard'' and earlier), Apple included an optional developer tools'' package (\texttt{XCode}) on the install media. 36 For more recent releases, \texttt{XCode} is available for free'' from Apple's application store. 37 Why free''? 38 The only way to access the application store is with an AppleID which requires either: 39 \begin{itemize} 40 \item purchasing an iTunes gift card. 41 \item giving Apple access to your credit card. 42 \item signing up as an Apple developer\footnote{If you do this you can download a command line tools'' package 43 which installs the relevant compilers without needing to install all of \texttt{XCode}.} and giving up personal information. 44 \end{itemize} 45 46 If you install \texttt{XCode}, you will need to download the command line tools'' optional package [see \texttt{XCode} documentation for details]. 47 48 There are a number of projects on the net which aim to deliver compilers for MacOS. 49 For example: 50 \begin{itemize} 51 \item \url{http://hpc.sourceforge.net} 52 \item \url{http://kennethreitz.org/experiments/xcode-gcc-and-homebrew} 53 \end{itemize} 54 Use at your own risk. 55 56 57 58 \subsection{Parallel Technologies} 59 It is extremely likely that the computer you run \escript on, will have more than one processor core. 60 \escript can make use of multiple cores [in order to solve problems more quickly] if it is told to do so, 61 but this functionality must be enabled at compile time. 62 63 There are two technologies which \escript can employ here. 64 \begin{itemize} 65 \item OpenMP -- more efficient of the two [thread level parallelism]. 66 \item MPI -- Uses multiple processes (less efficient), needs less help from the compiler. 67 \end{itemize} 68 69 Escript is primarily tested on recent versions of the GNU or Intel suites (gcc, g++'' / icc, icpc''). 70 However, it also passes our tests when compiled using clang, clang++''. 71 The table below shows what methods are available with which compilers. 72 73 \begin{center} 74 \begin{tabular}{|l|c|c|c|}\hline 75 & Serial & OpenMP & MPI \\\hline 76 $\leq$ gcc-4.2.1 & \checkmark & \raisebox{-0.1cm}{\footnotemark}& \checkmark \\\hline 77 gcc (recent $\geq 4.3.2$) & \checkmark& \checkmark& \checkmark \\\hline 78 icc(10) & \checkmark& \checkmark& \checkmark \\\hline 79 icc(11) & \checkmark& \raisebox{-0.1cm}{\footnotemark} &\checkmark \\\hline 80 icc(12) & \checkmark& \checkmark&\checkmark \\\hline 81 clang & \checkmark& & \checkmark\\\hline 82 \end{tabular} 83 \end{center} 84 \addtocounter{footnote}{-1} 85 \footnotetext{The \openmp support in gcc-4.2.1 is buggy/non-functional.} 86 \addtocounter{footnote}{1} 87 \footnotetext{There is a subtle bug in icc-11 when \openmp and c++ exception handling 88 are combined.} 89 90 \noindent Where both \openmp and \mpi are marked, \escript can be compiled with either or both. 91 A \checkmark mark means that combination passes our tests. 92 93 94 95 \subsection{Packaging System} 96 The packaging system (also known as the package manager) is the tool you use to search for and install new open source software. 97 For Linux, there will be one set up by default: the apt tools on Debian and Ubuntu, yast on Suse, yum on the RedHat family. 98 On BSD systems this will be a combination of \texttt{pkg_add} and the \texttt{ports} tree. 99 100 For MacOS, this is a bit more tricky. 101 There are a number of possible systems including \texttt{macports} and \texttt{homebrew}\footnote{There is also \texttt{fink}, but we have not experimented with that.}, but they do not come pre-installed so you will need to install one. 102 103 Packaging systems will make changes to your computer based on programs configured by other people from 104 various places around the internet. 105 It is important to satisfy yourself as to the security of those systems. 106 107 If you are using Linux or have decided that you don't want to use OpenMP skip to Section~\ref{} 108 109 \subsubsection{Changing Compilers on MacOS and BSD} 110 \emph{Not for the faint-hearted.} 111 112 In order to use OpenMP, your compiler must support it. 113 The compilers provided in Apple's package and BSD do not support OpenMP. 114 \texttt{Clang} has no support for it at all, while \texttt{GCC4.2.1} has some, but it does not work very well\footnote{Rejects valid code, crashes etc}. 115 You can use the packaging system to install a more up to date version of \texttt{gcc}. 116 However, you will need to install a number of other dependencies in order for escript to operate. 117 If these dependencies end up fighting over which versions of libraries to use, there will be problems 118 \footnote{For example: Trying to use two versions of \texttt{Python} in the same program or two versions of the \texttt{c++} standard library.} 119 To avoid this, if you are intend to change the compiler or python versions, you should install the new compiler first, 120 then the new version of python. 121 After this, you can install the other dependencies. 122 \emph{Please note that none of the Mac based packages nor BSD recommend changing the default $C$ compiler so 123 do so at your own risk. If you want OpenMP though, there does not seem to be a choice.} 124 125 \subsubsection*{Changing compiler on FreeBSD(9.1)} 126 The following sequence passes our unit tests under OpenMP. 127 \begin{itemize} 128 \item Install \texttt{gcc46} from ports. 129 \item Modify \texttt{/etc/make.conf} to set the default compiler to be \texttt{gcc46} 130 \item Install the remaining dependencies. 131 \item Configure \escript to build with \openmp. 132 \end{itemize} 133 134 We chose version $4.6$ rather than a later one because one of the optional dependencies (scipy) will try to install it anyway. 135 136 \subsection*{Changing compiler on MacPorts} 137 The following sequence passes our unit tests under OpenMP. 138 \begin{itemize} 139 \item \texttt{port install gcc47} 140 \item Set the default compiler for the command line with: \texttt{port select --set gcc mp-gcc47} 141 \item Set the default compiler for macports by adding the following line to \linebreak[2] \file{/opt/local/etc/macports/macports.conf}: 142 \begin{shellCode} 143 #Added by user to coerce macports into using a newer compiler 144 default_compiler macports-gcc-4.7 145 \end{shellCode} 146 \item Now install the remainder of the dependencies using \texttt{port -s install X}. 147 This will build them from source rather than downloading precompiled versions. 148 (This will unfortunately mean larger downloads.). 149 In some cases, some dependencies will not build properly from source. 150 In those cases(where XYZ is the dependency that fails to build), \texttt{port clean XYZ} then \texttt{port install XYZ}. 151 Then \emph{try to install the rest from source}. 152 153 Installing from source via macports is necessary to convince macports to link against the libraries provided by 154 your new compiler rather than the default one. 155 \end{itemize} 156 157 158 159 160 \subsection*{Changing compiler on Homebrew} 161 There is no configuration file that needs to be changed in order to use a different compiler. 162 You do need to do things in the correct order and make sure that you have made any required changes to 163 your environment variables before moving on to the next step. 164 In particular, the compiler must be installed before anything else and then python. 165 166 167 % make it clear that escript can be customised to use whatever you have 168 169 %Should include optional customisation 170 171 %Talk about options_files and the ability to specify them 172 173 %talk about -j1 and replacing it with more ops 174 %Talk about installation prefix 175 176 %also note that this doesn't build the doco but we do have downloads for that or you can install extra packages 177 178 \section{Building} 179 180 To simplify things for people, we have prepared \texttt{_options.py} files for a number of 181 systems\footnote{These are correct a time of writing but later versions of those systems may require tweaks. 182 Also, these systems represent a cross section of possible platforms rather than meaning those systems get particular support.}. 183 If your particular system is not in the above list, or if you want a more customised build, 184 see Section~\ref{sec:othersrc} for instructions. 185 \begin{itemize} 186 \item Debian - \ref{sec:debsrc} 187 \item Ubuntu - \ref{sec:ubsrc} 188 \item OpenSuse - \ref{sec:susesrc} 189 \item Centos - \ref{sec:centossrc} 190 \item Fedora - \ref{sec:fedorasrc} 191 \item MacOS (macports) - \ref{sec:macportsrc} 192 \item MacOS (homebrew) - \ref{sec:homebrewsrc} 193 \item FreeBSD - \ref{sec:freebsdsrc} 194 \end{itemize} 195 196 Once these are done proceed to Section~\ref{sec:cleanup} for cleanup steps. 197 198 All of these instructions assume that you have obtained the source (uncompressed it if necessary). 199 \subsection{Debian}\label{sec:debsrc} 200 201 \begin{shellCode} 202 sudo aptitude install python-dev python-numpy libboost-python-dev libnetcdf-dev 203 sudo aptitude scons lsb-release 204 sudo aptitude install python-sympy python-matplotlib python-scipy 205 sudo aptitude install python-pyproj python-gdal 206 \end{shellCode} 207 208 209 \begin{optionalstep} 210 If for some reason, you wish to rebuild the documentation, you would also need the following: 211 \begin{shellCode} 212 sudo aptitude install python-sphinx doxygen python-docutils texlive 213 sudo aptitude install ghostscript texlive-latex-extra latex-xcolor 214 \end{shellCode} 215 \end{optionalstep} 216 217 In the source directory execute the following (substitute squeeze or wheezy as appropriate for XXXX): 218 \begin{shellCode} 219 scons -j1 options_file=scons/os/XXXX_options.py 220 \end{shellCode} 221 222 If you wish to test your build, you can use the following: 223 \begin{shellCode} 224 scons -j1 py_tests options_file=scons/os/XXXX_options.py 225 \end{shellCode} 226 227 \subsection{Ubuntu}\label{sec:ubsrc} 228 229 \subsection{OpenSuse}\label{sec:susesrc} 230 These instructions were prepared using release $12.3$. 231 232 Install packages from the main distribution: 233 \begin{shellCode} 234 sudo yast2 --install libboost_python1_49_0 python-devel python-numpy 235 sudo yast2 --install python-scipy python-sympy python-matplotlib libnetcdf_c++-devel 236 sudo yast2 --install gcc-c++ scons boost-devel netcdf-devel 237 \end{shellCode} 238 These will allow you to use most features except some parts of the \downunder inversion library. 239 If you wish to use those, you will need some additional packages [python-pyproj, python-gdal]. 240 This can be done after Escript installation. 241 242 \begin{optionalstep} 243 Add \url{http://ftp.suse.de/pub/opensuse/repositories/Application:/Geo/openSUSE_12.3/} 244 to your repositories in \texttt{YaST}. 245 \begin{shellCode} 246 sudo yast2 --install python-pyproj, python-gdal 247 \end{shellCode} 248 \end{optionalstep} 249 250 Now to build escript itself. 251 In the escript source directory: 252 \begin{shellCode} 253 scons -j1 options_file=scons/os/opensuse12.3_options.py 254 \end{shellCode} 255 256 If you wish to test your build, you can use the following: 257 \begin{shellCode} 258 scons -j1 py_tests options_file=scons/os/opensuse12.3_options.py 259 \end{shellCode} 260 261 Now go to Section~\ref{sec:cleanup} for cleanup. 262 263 \subsection{Centos}\label{sec:centossrc} 264 These instructions were prepared using release $6.4$. 265 266 Install packages from the main distribution: 267 \begin{shellCode} 268 yum install python-devel numpy scipy scons boost-devel 269 yum install python-matplotlib gcc-g++ 270 yum install boost-python 271 \end{shellCode} 272 273 The above packages will allow you to use most features except saving and loading files in \texttt{netCDF} 274 format and the \downunder inversion library. 275 If you wish to use those features, you will need to install some additional packages. 276 NetCDF needs to be installed when you compile if you wish to use it. 277 \begin{optionalstep} 278 Add the \texttt{EPEL} repository. 279 \begin{shellCode} 280 rpm -U http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm 281 \end{shellCode} 282 283 \begin{shellCode} 284 yum install netcdf-devel sympy gdal-python 285 \end{shellCode} 286 287 For some coordinate transformations, \downunder can also make use of the python interface to a tool called \texttt{proj}. 288 There does not seem to be an obvious centos repository for this though. 289 If it turns out to be necessary for your particular application, the source can be downloaded. 290 \end{optionalstep} 291 292 Now to build escript itself. 293 In the escript source directory: 294 \begin{shellCode} 295 scons -j1 options_file=scons/os/centos6.4_options.py 296 \end{shellCode} 297 298 If you wish to test your build, you can use the following: 299 \begin{shellCode} 300 scons -j1 py_tests options_file=scons/os/centos6.4_options.py 301 \end{shellCode} 302 303 Now go to Section~\ref{sec:cleanup} for cleanup. 304 305 \subsection{Fedora}\label{sec:fedorasrc} 306 These instructions were prepared using release $18$. 307 308 Install packages 309 \begin{shellCode} 310 yum install netcdf-cxx-devel gcc-c++ scipy 311 yum install sympy scons pyproj gdal python-matplotlib 312 yum install boost-devel 313 \end{shellCode} 314 315 Now to build escript itself. 316 In the escript source directory: 317 \begin{shellCode} 318 scons -j1 options_file=scons/os/fedora18_options.py 319 \end{shellCode} 320 321 If you wish to test your build, you can use the following: 322 \begin{shellCode} 323 scons -j1 py_tests options_file=scons/os/fedora18_options.py 324 \end{shellCode} 325 326 Now go to Section~\ref{sec:cleanup} for cleanup. 327 328 \subsection{MacOS (macports)}\label{sec:macportsrc} 329 330 \subsection{MacOS (homebrew)}\label{sec:homebrewsrc} 331 332 \subsection{FreeBSD}\label{sec:freebsdsrc} 333 334 335 336 \subsection{Other Systems / Custom Builds}\label{sec:othersrc} 337 338 \section{Cleaning up} 339 \label{sec:cleanup} 340 341 Once the build (and optional testing) is complete, you can remove everything except: 342 \begin{itemize} 343 \item bin 344 \item esys 345 \item lib 346 \item doc 347 \item CREDITS.TXT 348 \item README_LICENSE 349 \end{itemize} 350 The last two aren't strictly required for operation. 351 The \texttt{doc} directory is not required either but does contain examples of escript scripts. 352 353 You can run escript using \texttt{\textit{path_to_escript_files}/bin/run-escript}. 354 Where \texttt{\textit{path_to_escript_files}} is replaced with the real path. 355 356 \begin{optionalstep} 357 You can add the escript \texttt{bin} directory to your \texttt{PATH} variable. 358 The launcher will then take care of the rest of the environment. 359 \end{optionalstep} 360 361 \section{Optional Extras} 362 %Need a better title but this is stuff like visit and silo (for non-debian distros) 363 %Perhaps - optional extras 364 365 366 367 368