Module documentation¶

prior_engine module¶

Prior Engine for MULTIPLY.

class multiply_prior_engine.prior_engine.PriorEngine(**kwargs)¶

Bases: object

Prior Engine for MULTIPLY.

holds prior initialization methods (e.g. config loading). calls specific submodules (soilmoisture_prior, vegetation_prior, ..)

_check()¶

initial check for passed values of - config - datestr - variables

Returns

Return type

_concat_priors(prior_dict)¶

Concatenate individual state vectors and covariance matrices for sm, veg, ..

Returns: dictionary with keys beeing superordinate prior name (sm, ..)
Return type: dictionary

_get_prior(var)¶

Called by get_priors for all variables to be inferred. For specific variable/prior (e.g. sm climatology) get prior info and calculate/provide prior.

Parameters

var – prior name (e.g. sm, lai, ..)

Returns

Return type

default_config = '/home/docs/checkouts/readthedocs.org/user_builds/multiply-prior-engine/checkouts/latest/docs/../multiply_prior_engine/sample_config_prior.yml'¶

get_priors()¶

Get prior data. calls _get_prior for all variables (e.g. sm, lai, ..) passed on to get_mean_state_vector method.

Returns: dictionary with prior names/prior types/filenames as {key/{key/values}}.
Return type: dictionary of dictionary

multiply_prior_engine.prior_engine._get_config(configfile)¶

Load config from self.configfile. writes to self.config.

Returns

multiply_prior_engine.prior_engine.get_mean_state_vector(datestr: str, variables: list, config: str = './sample_config_prior.yml') → dict¶

Return dictionary with variable dependent sub dictionary with prior type (key) and filenames of prior files (values).

Parameters

datestr – The date (time?) for which the prior needs to be derived
variables – A list of variables (sm, lai, roughness, ..)

for which priors need to be available

Returns: dictionary with keys being the variables and

values being a dictionary of prior type and filename of prior file.

prior module¶

Prior Class for MULTIPLY.

class multiply_prior_engine.prior_creator.PriorCreator(**kwargs)¶

Bases: object

_abc_cache = <_weakrefset.WeakSet object>¶

_abc_negative_cache = <_weakrefset.WeakSet object>¶

_abc_negative_cache_version = 211¶

_abc_registry = <_weakrefset.WeakSet object>¶

_check()¶

_create_datetime()¶

_create_time_vector()¶

Creates a time vector dependent on start & end time and time interval from config file. A vector containing datetime objects is written to self.time_vector. A vector containing months ids (1-12) for each timestep is written to self.time_vector_months.

Returns

Return type

abstract compute_prior_file() → str¶: Might perform some computation, then retrieves the path to a file containing the prior info :return:

abstract classmethod get_variable_names() → List[str]¶

Returns: A list of the variables that this prior creator is able to create priors for

soilmoisture_prior module¶

Soil Priors for Prior Engine in MULTIPLY.

class multiply_prior_engine.soilmoisture_prior_creator.MapPriorCreator(**kwargs)¶

Bases: multiply_prior_engine.prior_creator.PriorCreator

Not Implemented Prior which is based on a LC map and a LUT

_abc_cache = <_weakrefset.WeakSet object>¶

_abc_negative_cache = <_weakrefset.WeakSet object>¶

_abc_negative_cache_version = 211¶

_abc_registry = <_weakrefset.WeakSet object>¶

classmethod get_variable_names()¶

Returns: A list of the variables that this prior creator is able to create priors for

class multiply_prior_engine.soilmoisture_prior_creator.RoughnessPriorCreator(**kwargs)¶

Bases: multiply_prior_engine.soilmoisture_prior_creator.MapPriorCreator

Not Implemented Roughness Prior Creator which is based on a LC map and a LUT

_abc_cache = <_weakrefset.WeakSet object>¶

_abc_negative_cache = <_weakrefset.WeakSet object>¶

_abc_negative_cache_version = 211¶

_abc_registry = <_weakrefset.WeakSet object>¶

_map_lut()¶: should do the mapping of s, l, ACL type

_read_lc()¶

_read_lut()¶

calc()¶

compute_prior_file()¶: Might perform some computation, then retrieves the path to a file containing the prior info :return:

classmethod get_variable_names()¶

Returns: A list of the variables that this prior creator is able to create priors for

save()¶: save mapped roughness data to file

class multiply_prior_engine.soilmoisture_prior_creator.SoilMoisturePriorCreator(**kwargs)¶

Bases: multiply_prior_engine.prior_creator.PriorCreator

Soil moisture prior class. Calculation of climatological prior.

_abc_cache = <_weakrefset.WeakSet object>¶

_abc_negative_cache = <_weakrefset.WeakSet object>¶

_abc_negative_cache_version = 211¶

_abc_registry = <_weakrefset.WeakSet object>¶

_calc_climatological_prior()¶

Calculate climatological prior. Reads climatological file and extracts proper values for given timespan and -interval. Then converts the means and stds to state vector and covariance matrices.

Returns: state vector and covariance matrix
Return type: tuple

_check_gdal_compliance(fn)¶

_create_global_vrt(fn, local=True)¶

Create VRT file for file.

By default, the .vrt-file will be written to a local temporary directory. If local is set to False, the file is written to the directory the input file (fn) currently lives in.

Parameters

fn – file name
local – create temporary local vrt.

Returns

file name of created vrt, or initial file name if no success.

Return type

string

_extract_climatology()¶: Extract climatology values for ROI. Part of _clac_climatological_prior().

_get_climatology_file()¶: Load pre-processed climatology into self.clim_data. Part of prior._calc_climatological_prior().

_get_prior_file_from_dir(directory, return_vrt=True)¶

Get filename(s) of prior file(s) from directory. If multiple files are found self._merge_multiple_prior_files is called.

Currently, the following prior types are supported: - climatology (calculated from ESA CCI data, standard) - coarse (daily aggregated SMAP L4 data, standard) - soil_map (gloabal soil texture map data from soilgrids.org) - user prior, provided through user_prior_creator

Parameters: directory – directory containing the files (from config)
Returns: filename
Return type: string

_get_recent_sm_proxy()¶

_merge_multiple_prior_files(fn_list)¶

Merge files if more than one is available for current time step. should be obsolete.

Parameters: fn_list – file list to process
Returns: file name of merged file
Return type: string

_provide_prior_file()¶

Provide variable and prior type specific prior file name to Prior Engine.

Returns: absolute path to prior file for requested prior.

The file is gdal-compatible to be used in inference engine - either GeoTiff or VRT format. It includes 2 bands:

mean value raster

uncertainty raster

Return type: string

compute_prior_file()¶

Initialize prior specific (climatological, …) calculation.

Returns: filename of prior file
Return type: string

classmethod get_variable_names()¶

Returns: A list of the variables that this prior creator is able to create priors for

vegetation_prior module¶

class multiply_prior_engine.vegetation_prior_creator.VegetationPriorCreator(**kwargs)¶

Bases: multiply_prior_engine.prior_creator.PriorCreator

Description

AssignPFTTraits2Map(PFT, PFT_ids, varnames)¶

Create Vegetation trait Prior map, using the Trait-database and PFT distribution maps This function sets up a parallel processing chain around - processespercore: here the actual assignment of traits to PFT distributions is performed

Parameters

PFT – arrays containing global Maps of PFT distributions
PFT_ids – a list containing PFT ids
varnames – list of variables to be converted into global file

Returns

map of vegetation trait-averages per PFT id, map of vegetation trait-uncertainties per PFT id

Return type

Combine2PFT(LCC_map, CLM_map_i)¶: Create PFT maps using CCI Landcover and Koppen Climate zone information :param LCC_map: CCI Landcover map :param CLM_map_i: Regridded Koppen Climate Zone map :returns: PFT occurrence map, PFT classes, Number of PFTs, PFT ids :rtype:

CombineTiles2Virtualfile(variable, doystr, directory_data)¶

Combine all geotiff files into a virtual global file

Parameters

variable – variable to be converted into global file
doystr – string containing date&time ‘2007-12-31 04:23’ for which global file needs to be created

Returns

the filename of the global VRT file

Return type

CreateDummyDatabase()¶

create netcdf Database files to hold database values

Returns

Return type

CreateRealDatabase()¶

DownloadCrossWalkingTable()¶

Download Crosswalking table Here the Cross walking table is downloaded to create the CCI landcover map. At the moment this is simply a placeholder for future functionality.

Returns

Return type

DynamicProcessing(varnames, LCC_lon, LCC_lat, Prior_pbm_avg, Prior_pbm_unc, doystr, write_output=True)¶

Extending Peak Biomass (PBM) traits to seasonal Priors At this moment, this function is only a placeholder for the later implementations. The final implementation will be modelled using - covariances between traits and (seasonal) meteorological variables - phenological evolution (trained using plant growth models)

Parameters

varnames – list of variables to be converted into global file
LCC_lon – array with longitude values of (subsetted tile of) study area
LCC_lat – array with latitude values of (subsetted tile of) study area
Prior_pbm_avg – Vegetation Traits mean value at PBM
doystr – string containing date&time ‘2007-12-31 04:23’ for processing needs to be performed
Prior_pbm_unc – Vegetation Traits uncertainty value at PBM
write_output – Binary Value (TRUE/FALSE) controlling the writing of outputfiles

Returns

Return type

ExtractPFT4TryDatabaseEntries(Lat_, Lon_, Plantgroup_, Crop_, LeafType_, C3C4_, LeafPhen_)¶

OfflineProcessing()¶: Creation of LCC landcover map This :returns: :rtype:

PhenologicalEvolution(Prior_pbm_avg, Prior_pbm_unc, doystr, Meteo_map_i=None)¶

Model the Phenological Evolution of Vegetation traits This function is a placeholder to be used when the dynamic functionality is created.

Parameters

Prior_pbm_avg – Vegetation trait-averages at Peak Biomass
Prior_pbm_unc – Vegetation trait-uncertainty (@PBM)
doystr – string containing date&time ‘2007-12-31 04:23’ for processing needs to be performed
Meteo_map_i – Place_holder for meteorological data-files

Returns

Temporal Prior-averages, Temporal Prior-uncertainties

Return type

ProcessData(variables=None, state_mask=None, timestr='2005-05-05 05:55', logger=None, file_prior=None, file_lcc=None, file_biome=None, file_meteo=None)¶

Process Data Apriori Calculation of prior using Databases of Vegetation Traits. This function is split into two parts (which are run for all Tiles over the study are) - OfflineProcessing: This only has to be performed once (to make sure all the input data is available) - StaticProcessing: Creating Peak Biomass (PBM) Priors - DynamicProcessing: Extending PBM traits to seasonal priors (a placeholder for the later implementations)

Parameters

variables – list of variables to be converted into global file
state_mask – place holder for spatial mask (not implemented)
timestr – string containing date&time ‘2007-12-31 04:23’ for which global file needs to be created
logger – log-file for capturing message from the scripts
file_prior – place-holder for prior (TRY) database - filename (at the moment hardcoded)
file_lcc – place-holder for landcover data - filename (at the moment hardcoded)
file_biome – place-holder for biome data - filename (at the moment hardcoded)
file_meteo – place-holder for meteorological data - filename (at the moment hardcoded)

Returns

filenames to global VRT prior files

Return type

ReadClimate()¶

Read Climate Zone information A Climate Zone map (created on basis of the Koppen Climatic Zone classification)is read.

Returns: climate zone map, longitude, latitude, climate zone classes
Return type

ReadLCC()¶

Read Landcover information The Landcover map from the Climate Change Initiaive (CCI) is read.

Returns: landcover map, longitude, latitude, landcover class names
Return type

ReadMeteorologicalData(doystr)¶

Read Meteorological Variables This function is a placeholder to be used when the dynamic functionality is created.

Parameters: doystr – string containing date&time ‘2007-12-31 04:23’ for processing needs to be performed
Returns: Meteorological data (to be used for upscaling Peak Biomass traits to seasonal priors)
Return type

ReadTraitDatabase(varnames, pft_id=1)¶

Read Traits from Database A local (modified) version of the Try Database (containing vegetation traits) is read.

Parameters

varnames – list of variables to be converted into global file
pft_id – list of pft id numbers for which the traits needs to be read.

Returns

an array of Traits per PFT group

Returns

an array of Traits per PFT group

Return type

ReadTryDatabase()¶

ReadTryFile()¶

RescaleCLM(CLM_lon, CLM_lat, CLM_map, LCC_lon, LCC_lat)¶

Collocate Climate Zone map with landcover coordinates The Climate Zone map has a different resolution/grid than the Landcover map. This preprocessing is performed to collocate both (in order to facilitate the merging downstream.)

Parameters

CLM_lon – array containing the longitude values of the Climate Zone map
CLM_lat – array containing the latitude values of the Climate Zone map
CLM_map – array containing the Climate Zone map
LCC_lon – array containing the longitude values of the CCI Landcover map
LCC_lat – array containing the latitude values of the CCI Landcover map

Returns

array containing the Regridded Climate Zone map

Return type

RunCrossWalkingTable(Path2CWT_tool=None, Path2LC=None)¶

Creating CCI landcover maps (using crosswalking table).

please note that to run the crosswalking tool, the specific requirements for BEAM need to be met (java64bit + …)

Parameters

Path2CWT_tool –
Path2LC –

Returns

Return type

StaticProcessing(varnames, write_output=False)¶

Creating Peak Biomass (PBM) Priors Priors are created by upscaling vegetation traits obtained through the TRY database. Within the TRY database vegetation traits are provided per PFT group. In order to upscale these values, a global PFT map is required. This is created by merging a global Landcover map (from Climate Change Initiative, CCI) with a climate zone map (using the Koppen classification). This is accomplished by -ReadLCC: Reading the CCI Landcover map -ReadClimate: Reading the Koppen Climate zone map -RescaleCLM: Rescaling Climate zone map to collocate with Landcover CCI. -Combine2PFT: Combining Climate zone + Landcover maps into PFTs Using this global PFT map, the values from the TRY database are afterwards spatially distributed by -AssignPFTTraits2Map: assigning and aggregating traits to PFT maps.

Parameters

varnames – list of variables to be converted into global file
write_output – Binary value (TRUE/FALSE) controlling the writing of outputfiles

Returns

longitude, latitude, Prior_avg, Prior_unc

Return type

WriteGeoTiff(LCC_lon, LCC_lat, Prior_avg, Prior_unc, doystr='static')¶: Write Vegetation Prior data (mean/unc) to GEOTIFF outputfiles. :param LCC_lon: longitude of the Prior data (same as used Landcover map) :param LCC_lat: latitude of the Prior data (same as used Landcover map) :param Prior_avg: Vegetation prior average values :param Prior_unc: Vegetation prior uncertainty values :param doystr: string containing date&time ‘2007-12-31 04:23’ for data to be written :returns: -

WriteOutput(LCC_lon, LCC_lat, Prior_avg, Prior_unc, doystr='static')¶

Write Vegetation Prior data (mean/unc) to NETCDF outputfiles. This functionality is obsolete as all outputs are written to GeoTiff files

Parameters

LCC_lon – longitude of the Prior data (same as used Landcover map)
LCC_lat – latitude of the Prior data (same as used Landcover map)
Prior_avg – Vegetation prior average values
Prior_unc – Vegetation prior uncertainty values
doystr – string containing date&time ‘2007-12-31 04:23’ for data to be written

Returns

Return type

_abc_cache = <_weakrefset.WeakSet object>¶

_abc_negative_cache = <_weakrefset.WeakSet object>¶

_abc_negative_cache_version = 212¶

_abc_registry = <_weakrefset.WeakSet object>¶

compute_prior_file()¶

Combine Tiles into single Prior VRT file

Returns: filename of specific VRT file
Return type

classmethod get_variable_names()¶

Returns: A list of the variables that this prior creator is able to create priors for

multiply_prior_engine.vegetation_prior_creator._get_config(configfile)¶

Load config from self.configfile. writes to self.config.

Returns

multiply_prior_engine.vegetation_prior_creator.fun(f, q_in, q_out)¶

multiply_prior_engine.vegetation_prior_creator.parmap(f, X, nprocs=4)¶

Enable Parallel processing This code is created to enable parallel processing with python

Parameters

f – function to be called
X – input to the function
nprocs – number of cores to be used

Returns

output of function

Return type

multiply_prior_engine.vegetation_prior_creator.processespercore(varname, PFT, PFT_ids, VegetationPriorCreator)¶

Create Prior values from PFT distributions and Vegetation traits For each PFT the specific trait (according to varname) are read from the Trait-Database. These traits are then statistically analysed to produce the mean and standard deviations. These trait values are then evaluated against the PFT distribution (occurrence) map and joint together to create a single Prior (mean&uncertainty) estimate for each spatial location

Please note that: This function is encapsulated within the parmap method to run in parallel on different cores

Parameters

varname – variable to be processed
PFT – arrays containing global Maps of PFT distributions
PFT_ids – a list containing PFT ids
VegetationPriorCreator – class containing all the functionality to be run (per core)

Returns

Vegetation Prior average values, Vegetation Prior uncertainty values

Return type