PAnEn: Parallel Analog Ensemble

DOI License: MIT Codacy Badge Build Status codecov lifecycle Binder dockeri.co

About Parallel Analog Ensemble

Parallel Analog Ensemble (PAnEn) is a parallel implementation for the Analog Ensemble (AnEn) technique which generates uncertainty information for a deterministic predictive model. Analogs are generated by using the predictive model and the corresponding historical observations. An introduction to Analog Ensemble technique can be found in this post. It has been successfully applied to forecasting of several weather variables, for example, temperature, wind speed, and solar photovoltaic generation. Publications can be found in the reference section.

PAnEn is developed by the Geoinformatics and Earth Observation Laboratory at Penn State University. This website contains information for installing and using the PAnEn programs and libraries. R and C++ documentation is provided. Posts on various topics are included, and are organized by tags. If you have any questions, please submit a ticket here.

This package contains several programs and libraries:

Citation

Please consider citing this package. The bibtex entry can be found here. If you are using RAnEn, you can also see the citation by typing citation('RAnEn'). Or you can use the following formatted text to cite this package.

Weiming Hu, Guido Cervone, Laura Clemente-Harding, and Martina Calovi. (2019). Parallel Analog Ensemble. Zenodo. http://doi.org/10.5281/zenodo.3384321

Requirement and Dependencies

A list of dependencies are provided below. Note that you don’t necessarily have to install them all because some of them can be automatically installed or you simply are not installing some components of the program. For example, you won’t need R if you only need the C++ program, and you won’t need Boost C++ and NetCDF-C if you are only installing the R package RAnEn.

Dependency Description
CMake Required for the C++ program.
GCC/Clang Required for the C++ program.
NetCDF-C Optional for the C++ program. If it is not found, the project will try to build it.
Boost C++ Optional for the C++ program. It is recommended to let the project build it for the C++ program.
CppUnit Required for the C++ program when building tests.
R Required for the R library.
OpenMP Optional for both R and C++.

Installation

C++ PAnEn

First, make sure you have already installed the dependencies. Typically, GNU compilers with a version later than 4.9, netCDF-C, and CMake are required. If you are using MacOS, you probably need to install GNU compilers in order to have OpenMP multithreading available.

Then, please clone or download the repository here and create a build/ folder under the repository directory.

git clone https://github.com/Weiming-Hu/AnalogsEnsemble.git
cd AnalogsEnsemble
mkdir build
cd build
cmake ..

# If you would like to change the default compiler, specify the compilers like this
#
# CC=[full path to CC] CXX=[full path to CXX] cmake ..

# You can scoll down to explore more parameters for cmake
#
# cmake -DCMAKE_INSTALL_PREFIX=/some/folder/ -DBOOST_TYPE=SYSTEM [other parameters] ..

Read the output messages and make sure there are no errors. If you would like to change cmake parameters, please delete all files in the build/ folder and rerun the cmake command.

Then, please compile the programs and libraries.

make

# Or if you are using UNIX system, use the flag -j[number of cores] to parallelize the make process
#
# make -j4

# Build document if needed. The /html and the /man folders will be in your build directory
#
# make document

# If you want to install the program to your machine
#
# make install

After compilation, the programs and libraries should be in the folder AnalogsEnsemble/output. Please cd into the binary folder [Where your repository folder is]/AnalogsEnsemble/output/bin/ and run the following command to see help messages.

./analogGenerator 
# Analog Ensemble program --- Analog Generator
# Available options:
#  -h [ --help ]             Print help information for options.
#  -c [ --config ] arg       Set the configuration file path. Command line 
#                            options overwrite options in configuration file. 
# ... [subsequent texts ignored]

If you want to clean up the folder, please do the following.

make clean

RAnEn

The command is the same for RAnEn installation and update.

The R version should be later than or equal to 3.3.0. And the following R packages are needed:

If your operating system is Windows, please also install Rtools.

One-Line Solution

The following R command installs the latest version of RAnEn.

# Read more if OpenMP is not supported
#
install.packages("https://github.com/Weiming-Hu/AnalogsEnsemble/raw/master/RAnalogs/releases/RAnEn_latest.tar.gz", repos = NULL)
Solution for a Specific Version

If you want to install a specific version of RAnEn, you can go to the release folder, copy the full name of the tarball file, replace the following part [tarball name] (including the square bracket) with it, and run the command in R.

# Read more if OpenMP is not supported
#
install.packages("https://github.com/Weiming-Hu/AnalogsEnsemble/raw/master/RAnalogs/releases/[tarball name]", repos = NULL)

If Openmp multithreading is not supported, or if you simply want to use a different compiler, please create a Makevars file under ~/.R, with the following content.

CXX11=[C++11 compiler]

gribConverter

gribConverter provides the functionality to convert from a GRIB2 file format to NetCDF file format directly that is ready to be used by other Analog computation tools like similarityCalculator. By default, this tool is NOT built.

gribConverter uses Eccodes. Eccodes is not reliable when dealing with multi-field messages. Always remember to make sure the GRIB file is flat which means it does not include any multi-field messages. Check here for how to check this.

If you have already installed Eccodes on your system, to build gribConverter, you need to include the following parameters. The order of the parameters does not matter.

cmake -DBUILD_GRIBCONVERTER=ON <other arguments if you have any> ..

# If the eccodes library cannot be found at default locations,
# explicitly specify the root folder which contains folders like
# bin/, include/, and lib/.
#
cmake -DBUILD_GRIBCONVERTER=ON -CMAKE_PREFIX_PATH=<path to eccodes> <other arguments if you have any> ..

If you do not have Eccodes installed or you do not have the permission to install any libraries, the installation process is included in the cmake process. It will try to download and build the library for you under the project directory.

cmake -DBUILD_GRIBCONVERTER=ON -DECCODES_TYPE=BUILD <other arguments if you have any> ..

To build Eccodes, the following packages are needed:

Note that if you do not have either OpenJPEG or Jasper, the compilation would complete but the test would not be complete.

Mac OS

If you are using Mac OS, the recommended way to build gribConverter is to install Eccodes and then just link to it.

If you cannot install Eccodes, additional to that you need to make sure you have the required dependencies, you need check the followings:

$ # Check whether you have installed jasper. I have installed jasper because
$ # there is a mark after formulae jasper.
$ #
$ brew search jasper
==> Formulae
jasper ✔

==> Casks
jasper                                             tibco-jaspersoft-studio

$ # There is a good chance that Mac OS will not find the correct jasper if
$ # there are multiple installed. We need to manually specify the locaion.
$ #
$ brew info jasper
jasper: stable 2.0.16 (bottled)
Library for manipulating JPEG-2000 images
https://www.ece.uvic.ca/~frodo/jasper/
/usr/local/Cellar/jasper/2.0.16_1 (40 files, 1.4MB) *
  Poured from bottle on 2019-04-21 at 23:26:48
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/jasper.rb
==> Dependencies
Build: cmake ✔
Required: jpeg ✔
==> Analytics
install: 34,471 (30 days), 108,948 (90 days), 408,801 (365 days)
install_on_request: 377 (30 days), 1,105 (90 days), 7,674 (365 days)
build_error: 0 (30 days)

$ # Now we have the location japser. We need to pass this to cmake.
$ # Note the include folder at the very end.
$ #
$ cmake -DJASPER_INCLUDE_DIR=/usr/local/Cellar/jasper/2.0.16_1/include/ [additional arguments] ..

# Similarly, we check the installation of openjpeg.
$ brew search openjpeg
==> Formulae
openjpeg ✔

$ # I have the packages installed. Let's check the location.
$ brew info openjpeg
openjpeg: stable 2.3.1 (bottled), HEAD
Library for JPEG-2000 image manipulation
https://www.openjpeg.org/
/usr/local/Cellar/openjpeg/2.3.1 (516 files, 12.8MB) *
  Poured from bottle on 2019-10-06 at 22:23:45
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/openjpeg.rb
==> Dependencies
Build: cmake ✔, doxygen ✔
Required: libpng ✔, libtiff ✔, little-cms2 ✔
==> Options
--HEAD
	Install HEAD version
==> Analytics
install: 70,099 (30 days), 216,905 (90 days), 930,201 (365 days)
install_on_request: 480 (30 days), 1,645 (90 days), 10,600 (365 days)
build_error: 0 (30 days)

$ # Finally, we give the location to cmake as well.
$ # Note the include/openjpeg-2.3 folder at the end.
$ #
$ cmake -DOPENJPEG_INCLUDE_DIR=/usr/local/Cellar/openjpeg/2.3.1/include/openjpeg-2.3/ [additional arguments] ..

Then you can follow the same commands for compiling and installation.

make <-j 4>
make test

After compilation, the programs and libraries should be in the folder AnalogsEnsemble/output. Please cd into the binary folder [Where your repository folder is]/AnalogsEnsemble/output/bin/ and run the following command to see help messages.

~/github/AnalogsEnsemble/output/bin$ ./gribConverter 
Parallel Ensemble Forecasts --- GRIB Converter v 3.6.3
Copyright (c) 2018 Weiming Hu @ GEOlab
Available options:
  -h [ --help ]                 Print help information for options.
  -c [ --config ] arg           Set the configuration file path. Command line 
                                options overwrite options in configuration 
                                file.
  --output-type arg             Set the output type. Currently it supports 
                                'Forecasts' and 'Observations'.
  --folder arg                  Set the data folder.

# ... [subsequent texts ignored]

If you have specified an installation path, you program will reside under the bin/ folder of your installation path.

For more information on how to use the tool, please see an example here.

CMake Parameter Look-Up Table

Parameter Explanation Default
CC The C compiler to use. [System dependent]
CXX The C++ compiler to use. [System dependent]
INSTALL_RAnEn Build and install the RAnEn library. OFF
BOOST_TYPE BUILD for building Boost; SYSTEM for using the library on the system. BUILD
NETCDF_CXX4_TYPE BUILD for building Netcdf C++4; SYSTEM for using the library on the system. BUILD
CPPUNIT_TYPE BUILD for building CppUnit; SYSTEM for using the library on the system. BUILD
CMAKE_BUILD_TYPE Release for release mode; Debug for debug mode. Release
CMAKE_BUILD_TESTS Build tests. ON
CMAKE_PREFIX_PATH Which folder(s) should cmake search for packages besides the default. Paths are surrounded by double quotes and separated with semicolons. [Empty]
CMAKE_INSTALL_PREFIX The installation directory. [System dependent]
BUILD_NETCDF Build NetCDF library regardless of its existence. OFF
USE_NCCONFIG Use the nc_config program if found. This might cause problems if NetCDF is not properly setup. OFF
BUILD_HDF5 Build HDF5 library regardless of its existence. OFF
VERBOSE Print detailed messages during the compiling process. OFF
CODE_PROFILING Print time profiling information. OFF
ENABLE_MPI Build the MPI version of the CAnEnIO library for parallel I/O. This is option is not recommended. OFF
BUILD_GRIBCONVERTER Build the GRIB Converter program. Eccodes library is required. OFF
ECCODES_TYPE BUILD for building Eccodes; SYSTEM for using the library on the system. SYSTEM

References

Known Issues

Please see known issues in this post. If you could not find solutions to your issue, please submit an issue. Thank you.

Feedbacks

We appreciate collaborations and feedbacks from users. Please contact the maintainer Weiming Hu through weiming@psu.edu or submit tickets if you have any problems.

Thank you!

# "`-''-/").___..--''"`-._
#  (`6_ 6  )   `-.  (     ).`-.__.`)   WE ARE ...
#  (_Y_.)'  ._   )  `._ `. ``-..-'    PENN STATE!
#    _ ..`--'_..-_/  /--'_.' ,'
#  (il),-''  (li),'  ((!.-'
# 
# Authors: 
#     Weiming Hu <weiming@psu.edu>
#     Guido Cervone <cervone@psu.edu>
#     Laura Clemente-Harding <lec170@psu.edu>
#     Martina Calovi <mcalovi@psu.edu>
#
# Contributors: 
#     Luca Delle Monache
#         
# Geoinformatics and Earth Observation Laboratory (http://geolab.psu.edu)
# Department of Geography and Institute for CyberScience
# The Pennsylvania State University