Climate Indices in Python — climate_indices 1.0.10 documentation (2024)

This project contains Python implementations of various climate index algorithms which providea geographical and temporal picture of the severity of precipitation and temperature anomaliesuseful for climate monitoring and research.

The following indices are provided:

SPI,Standardized Precipitation Index, utilizing both gamma and Pearson Type III distributions
SPEI,Standardized Precipitation Evapotranspiration Index, utilizing both gamma and Pearson Type III distributions
PET,Potential Evapotranspiration, utilizing either Thornthwaiteor Hargreaves equations
PNP,Percentage of Normal Precipitation

The following are provided as experimental/development versions only, not fully vetted nor suitable for research purposes:

PDSI,Palmer Drought Severity Index
scPDSI,Self-calibrated Palmer Drought Severity Index
PHDI,Palmer Hydrological Drought Index
Z-Index,Palmer moisture anomaly index (Z-index)
PMDI, Palmer Modified Drought Index

This Python implementation of the above climate index algorithms is being developedwith the following goals in mind:

to provide an open source software package to compute a suite ofclimate indices commonly used for climate monitoring, with welldocumented code that is faithful to the relevant literature andwhich produces scientifically verifiable results
to provide a central, open location for participation and collaborationfor researchers, developers, and users of climate indices
to facilitate standardization and consensus on best-of-breedclimate index algorithms and corresponding compliant implementations in Python
to provide transparency into the operational code used for climatemonitoring activities at NCEI/NOAA, and consequent reproducibilityof published datasets computed from this package
to incorporate modern software engineering principles and scientific programmingbest practices

Getting started¶

The installation and configuration described below isperformed using a bash shell, either on Linux, Windows, or MacOS.

Windows users will need to install and configure a bash shell in orderto follow the usage shown below. We recommended eitherbabun or Cygwinfor this purpose.

Configure the Python environment¶

This project’s code is written in Python 3. It is recommended to useeither the Miniconda3 (minimal Anaconda) orAnaconda3 distribution. The below instructionswill be Anaconda specific (although relevant to any Python virtual environment), and assume the use of a bash shell.

A new Anaconda environment can be createdusing the conda environment management system that comespackaged with Anaconda. In the following examples, we’ll use an environment named indices_env(any environment name can be used instead of indices_env) which will be created andpopulated with all required dependencies through the use of the provided setup.py file.

First, create the Python environment:

$ conda create -n indices_env

The environment can now be ‘activated’:

$ source activate indices_env

Once the environment has been activated then subsequent Python commands will runin this environment where the package dependencies for this project are present.

Now the package can be added to the environment along with all required modules(dependencies) via pip:

$ python -m pip install climate-indices

For development of the package itself it pays to install the dependencies viathe requirements.txt file:

$ python -m pip install -r requirements.txt

NCO¶

NetCDF Operators is a requirement and must be installed for utilization of this package.Instructions for installation on various platforms is available here.If using an Anaconda environment as advised above then it’s as simple as runningthe following command within the activated conda environment:

$ conda install -c conda-forge nco

Indices Processing¶

The installation will provide an “entry point” script which interacts with the corecomputational package to compute one or more climate indices. This script isprocess_climate_indices and is used to compute indices corresponding to griddedNetCDF datasets as well as US climate division NetCDF datasets.

This Python entry point script is written to be run via bash shell command, i.e.

$ process_climate_indices <options>

The options for the entry point script are described below:

Option	Description
index	Which of the climate indices to compute.Valid values are ‘spi’, ‘spei’, ‘pnp’, ‘scaled’,‘pet’, and ‘palmers’. ‘scaled’ indicates allthree scaled indices (SPI, SPEI, and PNP) and‘palmers’ indicates all Palmer indices (PDSI,PHDI, PMDI, SCPDSI, and Z-Index).
periodicity	The periodicity of the input dataset files.Valid values are ‘monthly’ and ‘daily’. NOTE: Only SPI and PNP support daily inputs.
netcdf_precip	Input NetCDF file containing aprecipitation dataset, required for allindices except for PET. Requires the use ofvar_name_precip in conjunction so as toidentify the NetCDF’s precipitation variable.
var_name_precip	Name of the precipitation variable withinthe input precipitation NetCDF.
netcdf_temp	Input NetCDF file containing atemperature dataset, required for PET.If specified in conjunction with an indexspecification of SPEI or Palmers then PETwill be computed and written as a sideeffect, since these indices require PET.This option is mutually exclusive withnetcdf_pet/var_name_pet, as eithertemperature or PET is required as an input(but not both) when computing SPEI and/orPalmers. Requires the use ofvar_name_temp in conjunction so as toidentify the NetCDF’s temperature variable.
var_name_temp	Name of the temperature variable within theinput temperature NetCDF.
netcdf_pet	Input NetCDF file containing a PET dataset,required for SPEI and Palmers.This option is mutually exclusive withnetcdf_temp/var_name_temp, as eithertemperature or PET is required as an input(but not both) when computing SPEI and/orPalmers. Requires the use ofvar_name_pet in conjunction so as toidentify the NetCDF’s PET variable.
var_name_pet	Name of the PET variable within the input PETNetCDF.
netcdf_awc	Input NetCDF file containing available watercapacity, required for Palmers. Requires theuse of var_name_awc in conjunction so as toidentify the NetCDF’s AWC variable.
awc_var_name	Name of the available water capacity variablewithin the input AWC NetCDF.
output_file_base	Base file name for all output files. Each computed index will have a correspondingoutput file whose name will begin withthis base name plus the index’sabbreviation plus a month scale(if applicable), connected with underscores,plus the ‘.nc’ extension. For examplefor SPI at 3-month scalethe resulting output files will benamed <output_file_base>_spi_gamma_03.ncand <output_file_base>_spi_pearson_03.nc.
scales	Time step scales over which the PNP, SPI, andSPEI values are to be computed. Required whenthe index argument is ‘spi’, ‘spei’,‘pnp’, or ‘scaled’. The periodicityoption will infer whether the scales used aremonth or day scales. NOTE: When used for US climate divisionsprocessing this option specifies month scales
calibration_start_year	Initial year of the calibration period.
calibration_end_year	Final year of the calibration period(inclusive).
multiprocessing	Valid values are ‘all’ (uses all availableCPUs), ‘single’ (uses a single CPU), or‘all_but_one’ (uses all CPUs minus one).Default value is ‘all_but_one’.
save_params	Save distribution fitting variables to this filepath. The fittings NetCDF is to be used as inputwhen using the load_params option. [NOTE:only for use with the spi entrypoint for SPI.]
load_params	Load distribution fitting variables from thisfilepath. The fittings NetCDF file is one thatwas created by the save_params option. [NOTE:only for use with the spi entrypoint for SPI.]

Example Input and Output Datasets¶

Example NetCDF datasets that are valid input to the indices processing scriptsdescribed above are available from the associated projectexample_climate_indices.The input NetCDF files used in the examples below (nclimdiv.nc,nclimgrid_lowres_prcp.nc, etc.) can be fetched from this repository, as wellas associated output NetCDF datasets that can be used to validate result of thebelow examples.

Example Command Line Invocations¶

US Climate Divisions (all indices)¶

$ process_climate_indices --index all --periodicity monthly --scales 3 6--netcdf_precip /data/nclimdiv.nc--netcdf_temp /data/nclimdiv.nc--netcdf_awc /data/nclimdiv.nc--output_file_base /data/nclimdiv--var_name_precip prcp --var_name_temp tavg --var_name_awc awc--calibration_start_year 1951 --calibration_end_year 2010

The above command will compute all indices from an input NetCDF dataset containingprecipitation, temperature, and available water capacity variables (in this case,the US Climate Divisions NetCDF dataset provided in the example inputs directory).The input dataset is monthly data and the calibration period used will beJan. 1951 through Dec. 2010. The indices will be computed at 3-month and 6-month scales.Upon completion the individual NetCDF files will contain variables for all computed indices:/data/nclimdiv_pet.nc, /data/nclimdiv_pnp_03.nc, /data/nclimdiv_pnp_06.nc,/data/nclimdiv_spi_gamma_03.nc, /data/nclimdiv_spi_gamma_06.nc,/data/nclimdiv_spi_pearson_03.nc, /data/nclimdiv_spi_pearson_06.nc,/data/nclimdiv_spei_gamma_03.nc, /data/nclimdiv_spei_gamma_06.nc,/data/nclimdiv_spei_pearson_03.nc, /data/nclimdiv_spei_pearson_06.nc,/data/nclimdiv_pdsi.nc, /data/nclimdiv_phdi.nc, /data/nclimdiv_pmdi.nc,/data/nclimdiv_scpdsi.nc, and /data/nclimdiv_zindex.nc.Parallelization will occur utilizing all but one of the available CPUs(default since the –multiprocessing option is omitted).

PET monthly¶

$ process_climate_indices --index pet --periodicity monthly --netcdf_temp/data/nclimgrid_lowres_tavg.nc --var_name_temp tavg --output_file_base<out_dir>/nclimgrid_lowres --multiprocessing all_but_one

The above command will compute PET (potential evapotranspiration) using theThornthwaite method from an input temperature dataset (in this case, the reducedresolution nClimGrid temperature dataset provided in the example inputs directory).The input dataset is monthly data and the calibration period used will be Jan. 1951through Dec. 2010. The output file will be <out_dir>/nclimgrid_lowres_pet.nc.Parallelization will occur utilizing all but one of the available CPUs.

SPI daily¶

$ process_climate_indices --index spi --periodicity daily --netcdf_precip/data/cmorph_lowres_daily_conus_prcp.nc --var_name_precipprcp --output_file_base <out_dir>/cmorph_lowres_daily_conus --scales 30 90--calibration_start_year 1998 --calibration_end_year 2016--multiprocessing all

The above command will compute SPI (standardized precipitation index, both gammaand Pearson Type III distributions) from an input precipitation dataset (in this case,the reduced resolution CMORPH precipitation dataset provided in the example inputsdirectory). The input dataset is daily data and the calibration period used will beJan. 1st, 1998 through Dec. 31st, 2016. The index will be computed at 30-day and90-day timescales. The output files will be <out_dir>/cmorph_lowres_daily_conus_spi_gamma_30.nc,<out_dir>/cmorph_lowres_daily_conus_spi_gamma_90.nc,<out_dir>/cmorph_lowres_daily_conus_spi_pearson_30.nc, and<out_dir>/cmorph_lowres_daily_conus_spi_pearson_90.nc. Parallelization will occur utilizingall CPUs.

SPI monthly¶

$ process_climate_indices --index spi --periodicity monthly --netcdf_precip/data/nclimgrid_lowres_prcp.nc --var_name_precip prcp--output_file_base <out_dir>/nclimgrid_lowres --scales 6 12--calibration_start_year 1951 --calibration_end_year 2010--multiprocessing all

The above command will compute SPI (standardized precipitation index, both gamma andPearson Type III distributions) from an input precipitation dataset (in this case,the reduced resolution nClimGrid precipitation dataset provided in the example inputs directory).The input dataset is monthly data and the calibration period used will beJan. 1951 through Dec. 2010. The index will be computed at 6-month and 12-month timescales.The output files will be <out_dir>/nclimgrid_lowres_spi_gamma_06.nc,<out_dir>/nclimgrid_lowres_spi_gamma_12.nc, <out_dir>/nclimgrid_lowres_spi_pearson_06.nc,and <out_dir>/nclimgrid_lowres_spi_pearson_12.nc. Parallelization will occur utilizingall CPUs.

SPEI monthly¶

$ process_climate_indices --index spei --periodicity monthly --netcdf_precip/data/nclimgrid_lowres_prcp.nc --var_name_precip prcp --netcdf_pet/data/nclimgrid_lowres_pet.nc --var_name_pet pet --output_file_base<out_dir>/nclimgrid_lowres --scales 9 18 --calibration_start_year 1951--calibration_end_year 2010 --multiprocessing all

The above command will compute SPEI (standardized precipitation evapotranspiration index,both gamma and Pearson Type III distributions) from input precipitation and potential evapotranspiration datasets(in this case, the reduced resolution nClimGrid precipitation and PET datasets provided in the example inputs directory).The input datasets are monthly data and the calibration period used will be Jan. 1951 through Dec. 2010. The indexdatasets will be computed at 9-month and 18-month timescales. The output files will be<out_dir>/nclimgrid_lowres_spei_gamma_09.nc, <out_dir>/nclimgrid_lowres_spei_gamma_18.nc,<out_dir>/nclimgrid_lowres_spei_pearson_09.nc, and <out_dir>/nclimgrid_lowres_spei_pearson_18.nc.Parallelization will occur utilizing all CPUs.

Palmers monthly¶

$ process_climate_indices --index palmers --periodicity monthly --netcdf_precip/data/nclimgrid_lowres_prcp.nc --var_name_precip prcp --netcdf_pet/data/nclimgrid_lowres_pet.nc --var_name_pet pet --netcdf_awc/data/nclimgrid_lowres_soil.nc --var_name_awc awc --output_file_base<out_dir>/nclimgrid_lowres --calibration_start_year 1951 --calibration_end_year 2010--multiprocessing all

The above command will compute the Palmer drought indices: PDSI (original Palmer Drought Severity Index),PHDI (Palmer Hydrological Drought Index), PMDI (Palmer Modified Drought Index), Z-Index (Palmer Z-Index),and SCPDSI (Self-calibrated Palmer Drought Severity Index) from input precipitation, potentialevapotranspiration, and available water capacity datasets (in this case, the reduced resolution nClimGridprecipitation, PET, and AWC datasets provided in the example inputs directory). The input datasets are monthlydata and the calibration period used will be Jan. 1951 through Dec. 2010. The output files will be<out_dir>/nclimgrid_lowres_pdsi.nc, <out_dir>/nclimgrid_lowres_phdi.nc,<out_dir>/nclimgrid_lowres_pmdi.nc, <out_dir>/nclimgrid_lowres_scpdsi.nc, and <out_dir>/nclimgrid_lowres_zindex.nc.Parallelization will occur utilizing all CPUs.

Pre-compute SPI distribution fitting variables¶

In order to pre-compute fititng parameters for later use as inputs to subsequentSPI calculations we can save both gamma and Pearson distributinon fitting parametersto NetCDF, and later use this file as input for SPI calculations over the samecalibration period.

$ spi --periodicity monthly --scales 1 2 3 6 9 12 24 36 48 60 72--calibration_start_year 1998 --calibration_end_year 2016--netcdf_precip /data/nclimgrid/nclimgrid_prcp.nc --var_name_precip prcp--output_file_base /data/nclimgrid/nclimgrid --multiprocessing all--save_params /data/nclimgrid/nclimgrid_fitting.nc --overwrite

In the above example we demonstrate how distribution fitting parameters can be saved as NetCDF.This fittings NetCDF can then be used as pre-computed variables in subsequent SPI computations.Inital command computes both distribution fitting values and SPI for various month scales.The distribution fitting variables are written to the file specified by the –save_params option.The second command also computes SPI but instead of computing the distribution fitting valuesit loads the pre-computed fitting values from the NetCDF file specified by the –load_paramsoption.

For Developers¶

Download the code¶

Clone this repository:

$ git clone https://github.com/monocongo/climate_indices.git

Move into the source directory:

$ cd climate_indices

Within this directory, there are four subdirectories:

climate_indices: main computational package
tests: unit tests for the main package
notebooks: Jupyter Notebooks describing the internals of the computational modules
docs: documentation files

Testing¶

Initially, all tests should be run for validation:

$ tox

If you run the above from the main branch and get an error then pleasesend a report and/or add an issue, as all tests should pass.

Get involved¶

Please use, make suggestions, and contribute to this code. Withoutdiverse participation and community adoption this project will not reachits potential.

Are you aware of other indices that would be a good addition here? Canyou identify bottlenecks and help optimize performance? Can you suggest newways of comparing these implementations against others (or othercriteria) in order to determine best-of-breed? Please fork the code andhave at it, and/or contact us to see if we can help.

Read our contributingguidelines
File anissue, orsubmit a pull request
Send us an email

Copyright and licensing¶

This is a developmental version of code that is originally developed atNCEI/NOAA, official release version available ondrought.gov.This software is under BSD 3-Clause license, copyright James Adams, 2017.Please read more on our license page.

Citation¶

You can cite climate_indices in your projects and research papers via the BibTeXentry below.

@misc {climate_indices, author = "James Adams", title = "climate_indices, an open source Python library providing reference implementations of commonly used climate indices", url = "https://github.com/monocongo/climate_indices", month = "may", year = "2017--"}