axiom.drs package

Subpackages

• axiom.drs.processing package
  • Submodules
  • axiom.drs.processing.ccam module
  • Module contents

Submodules

axiom.drs.cli module

Command-line methods for the DRS subsystem.

axiom.drs.cli.drs_launch(path, jobscript, log_dir, batches=None, dry_run=True, interactive=False, unlock=False, **launch_context)

Method to launch a series of qsubs for DRS processing.

Args:

path (str): Globbable path of payload files. jobscript (str): Path to the job script. log_dir (str): Path to which to save the log files. batches (int): Number of batches to split variables into (for parallel processing). dry_run (bool): Print out the commands rather than executing. interactive (bool): Dump the interactive flag into the qsub command when dumping. unlock (bool): Unlock locked payloads prior to submission (for rerunning walltime overruns) **launch_context: Additional arguments that will be interpolated as launch context.

axiom.drs.cli.generate_payloads(payload_dst, input_files, output_dir, start_year, end_year, project, model, domain, variables=None, schema=None, output_frequencies='1H,6H,1D,1M', num_batches=1, extra=None)

axiom.drs.cli.get_parser(config=None, parent=None)

Parse arguments for command line utiltities.

Args:

config (dict) : Configuration dictionary. parent (obj) : Parent parser object (for integration into the main axiom CLI)

Returns:

argparse.Namespace : Arguments object.

axiom.drs.cli.get_parser_consume(config=None, parent=None)

A parser that can consume JSON payload files.

Args:

config ([type], optional): [description]. Defaults to None. parent ([type], optional): [description]. Defaults to None.

axiom.drs.cli.get_parser_generate_payloads(parent=None)

A parser to generate payloads.

Args:

parent (object, optional): Parent parser object. Defaults to None.

axiom.drs.cli.get_parser_generate_user_config(parent=None)

Get a parser for generating a set of user config files from the installation directory.

Args:

parent (object, optional): Parent parser. Defaults to None.

axiom.drs.cli.get_parser_launch(parent=None)

A parser that can launch DRS processing.

Args:

parent ([type], optional): [description]. Defaults to None.

axiom.drs.cli.get_parser_rerun_failures(parent=None)

Get a parser for rerunning payloads.

Args:

parent (object, optional): Parent parser. Defaults to None.

axiom.drs.cli.rerun_failures(input_dir)

Method to rerun payloads based on the .failed output files.

Args:

input_dir (str) : Path to the input directory containing both the failed files and the original payloads.

axiom.drs.cli.split_args(values)

Split an argument that is comma-separated.

Args:

values (str): Values

Returns:

list : List of split arguments.

axiom.drs.domain module

class axiom.drs.domain.Domain(name, dx, lat_min, lat_max, lon_min, lon_max)

Bases: object

Domain specification class.

Args:

name (str): Name of the domain. dx (float): Delta-x. lat_min (float): Minimum latitude. lat_max (float): Maximum latitude. lon_min (float): Minimum longitude. lon_max (float): Maximum longitude.

from_config(config)

Generate a domain object a section in a parsed configparser config object.

Args:

key (str): Key of the domain in the section. config (configparser.Config) : Configuration object.

from_dict()

Convert from a dictionary definition to a Domain class object.

Args:

domain_dict (dict) : Domain dictionary.

Returns:

Domain : Domain class.

from_directive()

Generate domain object from a directive.

Example: ‘Antarctica,1.0,-180.0,180.0-90.0,-50.0’

Args:

directive (str): Domain directive.

Returns:

Domain : Domain class.

subset_xarray(ds, drop=True)

Subset an xarray object with this domain object.

Args:

ds (xarray.Dataset or xarray.DataArray) drop (bool) : Drop the data outside of the domain. Defaults to True.

Returns:

xarray.Dataset or xarray.DataArray : Object subset with this domain.

to_dict()

Convert the internal representation into a dict object.

Returns:

dict : Dictionary.

to_directive()

Convert the internal representation into a directive string.

Returns:

str : Directive.

axiom.drs.payload module

Class to represent a JSON payload.

class axiom.drs.payload.Payload(input_files, output_directory, start_year, end_year, output_frequency, project, model, domain, variables=[], batch=1, **kwargs)

Bases: object

Class to represent a Payload object.

Args:

input_files (str): Globbable path to input files. output_directory (str): Output directory. start_year (int) : Start year. end_year (int): End year. output_frequency (str): Output frequency. project (str): Project key from projects.json. model (str): Model key from models.json. domain (str): Domain key from domains.json. variables (list, optional): List of variables to process. Defaults to []. **kwargs : Additional key/value pairs added as a metadata.

Attributes:

input_files (str): Globbable path to input files. output_directory (str): Destination output directory (DRS built from here). project (str): Project key from projects.json. model (str): Model key from models.json. domain (str): Domain key from domains.json. start_year (int): Start year. end_year (int): End year. variables (list): List of variable names to process. output_frequency (str): Output frequency/resolution of data. extra (dict): Additional metadata key/value pairs.

from_dict()

Create a Payload from a dictionary.

Args:

d (dict): Dictionary.

Returns:

axiom.drs.Payload: Payload object.

Examples:

>>> payload = Payload.from_dict(d)

from_json()

Instantiate a payload from a json filepath.

Args:

filepath (str): Path.

Returns:

axiom.drs.Payload: Payload.

Examples:

>>> payload = Payload.from_json('payload.json')

get_filename()

Generate a filename for this payload.

Returns:

str: Filename

Examples:

>>> filename = payload.get_filename()

to_dict()

Convert the Payload object to a dictionary.

Returns:

dict: Dictionary representation of the Payload.

Examples:

>>> payload.to_dict()

to_json(filepath=None)

Serialize the Payload to the filepath (save as JSON).

Args:

filepath (str, optional): Filepath to write. Defaults to None (Writes default filename to current directory).

Examples:

>>> payload.to_json('payload.json')

axiom.drs.payload.generate_payloads(input_files, output_directory, start_year, end_year, project, model, domain, variables=None, schema=None, output_frequencies=['1H', '6H', '1D', '1M'], num_batches=1, **extra)

Generate payload files.

Args:

input_files (str): Globbable path to input files. output_directory (str): Destination filepath, DRS will be built from here. start_year (int): Start year. end_year (int): End year. project (str): Project key from projects.json. model (str): Model key from models.json. domain (str): Domain key from domains.json. variables (list(str), optional): List of variables to project. Defaults to None (read all from schema). schema (str): Schema name or filepath. output_frequencies (list(str), optional): List of output frequencies. Defaults to [‘1H’, ‘6H’, ‘1D’, ‘1M’]. num_batches (int, optional): Number of batches to split processing into. Defaults to 1. **extra : Key/value pairs added as additional metadata.

Returns:

list : A list of payload objects.

axiom.drs.utilities module

CCAM DRS post-processing

axiom.drs.utilities.assemble_qsub_command(jobscript, directives, **context)

Assemble the qsub command.

Args:

jobscript (str) : Path to the jobscript. directives (list) : List of directives to interpolate. **context (dict) : Context dictionary to interpolate into the directives.

Returns:

str : The qsub command.

axiom.drs.utilities.assemble_qsub_vars(**kwargs)

Assemble variables into a qsub-compliant -v format, without the -v.

Args:

**kwargs (dict): Keyword arguments to convert into qsub variables.

Returns:

str : String of qsub variables.

axiom.drs.utilities.detect_input_frequency(ds)

Detect the (closest) input frequency of the data.

Args:

ds (xarray.Dataset): Input data.

Returns:

str : Pandas offset directive.

axiom.drs.utilities.detect_resolution(paths)

Attempt to detect the input resolution from a list of paths.

Args:

paths (list): List of file paths.

Raises:

ResolutionDetectionException: If there are too many possible resolutions in a path. ResolutionDetectionException: If there are inconsistent resolutions detected between paths.

Returns:

int: Resolution in km.

axiom.drs.utilities.filter_by_variable_name(filepaths, variable)

Filter for filenames that include the variable name.

Args:

filepaths (list): List of filepaths.

Returns:

list : List of filepaths that include the variable name.

axiom.drs.utilities.generate_user_config()

Generate the user .axiom directory with all installed data files.

Args:

overwrite (bool): Overwrite existing .axiom directory.

axiom.drs.utilities.generate_years_list(start_year, end_year)

Generate a list of years (decades) to process.

Args:

start_year (int): Start year. end_year (int): End year.

Returns:

iterator : Years to process.

axiom.drs.utilities.get_domain(key)

Load a domain out of the internal configuration.

Args:

key (str): Domain key or parseable domain directive.

Returns:

axiom.domain.Domain : Domain object.

axiom.drs.utilities.get_domains(resolution, frequency, variable_fixed, no_frequencies)

Get the domains for the arguments provided.

Args:

resolution (int): Resolution frequency (str): Frequency variable_fixed (bool): Is the variable fixed? no_frequencies (bool): True if no frequencies were provided.

Raises:

Exception: When the arguments provided don’t yield any domain information.

Returns:

dict: Dictionary of domain information.

axiom.drs.utilities.get_meta(config, meta_key, key)

Load metadata on top of a _default key, if applicable.

Args:

config (dict): Configuration dictionary. meta_key (str): Top-level key. key (str): Key.

Returns:

dict: Metadata dictionary, with key loaded over _default.

axiom.drs.utilities.get_start_and_end_dates(year, output_frequency)

Get the start and end dates for interpolation context.

Args:

year (int): Current year output_frequency (str): Frequency.

axiom.drs.utilities.get_template(config, key)

Get an interpolation template out of the config.

Args:

config (dict): Dictionary. key (str): Template key.

Returns:

str: Template.

axiom.drs.utilities.get_uninterpolated_placeholders(string)

Check if a string has any remaining uninterpolated values.

Args:

string (string): String object.

Returns:

list : List of uninterpolated values.

axiom.drs.utilities.input_files_exist(paths)

Ensure all the input files actually exist.

Args:

paths (list): List of paths.

Returns:

bool: True if they all exist, False otherwise.

axiom.drs.utilities.interpolate_context(context)

Interpolate the context dictionary into itself, filling all placeholders.

Args:

context (dict): Context dictionary.

Returns:

dict : Interpolated context.

axiom.drs.utilities.is_error_recoverable(exception, recoverable_errors)

Determine if an error is recoverable based on the presence of certain text in the stack trace.

Args:

exception (Exception): Exception object, with message attribute. recoverable_errors (list): List of recoverable errors (regexes) that are permitted.

Returns:

bool : True if recoverable, False otherwise.

axiom.drs.utilities.is_fixed_variable(config, variable)

Determine if a variable is listed in in the config as fixed.

Args:

config (dict): Configuration dictionary variable (str): Variable name.

Returns:

bool: True if fixed, False otherwise.

axiom.drs.utilities.is_registered_domain(key)

Quick shortcut to see if the domain is already registered in the system.

Args:

key (str): Domain key.

Returns:

bool: True if registered, False otherwise.

axiom.drs.utilities.is_time_invariant(ds)

Test if the dataset is time-invariant (has no time coordinate)

Args:

ds (xarray.Dataset or xarray.DataArray): Data

Return:

bool : True if no ‘time’ coordinate detected, False otherwise

axiom.drs.utilities.load_domain_config()

Load domain configuration out of installed data dir.

Returns:

configparser.Config: Configuration object.

axiom.drs.utilities.load_postprocessor(model_key)

Shorthand for the load_processor function.

Args:

model_key (str): Model

axiom.drs.utilities.load_preprocessor(model_key)

Shorthand for the load_processor function.

Args:

model_key (str): Model

axiom.drs.utilities.load_processor(model_key, proc_type='pre')

Load a pre-or-post processor for the model, if one exists.

Args:

model_key (str): Model.

Returns:

callable: Function that takes an xarray.Dataset as input.

axiom.drs.utilities.metadata(obj, **kwargs)

Add metadata to an xarray object.

Args:

da (xarray.DataArray or xarray.Dataset): xarray object.

Returns:

xarray.DataArray or xarray.Dataset : Same as caller

axiom.drs.utilities.parse_domain(directive)

Parse a domain directive.

Domains are of the form: “name,fx,lat_min,lat_max,lon_min,lon_max”

Args:

directive (str) : Domain directive of the form name,dx,lat_min,lat_max,lon_min,lon_max.

Returns:

dict : Domain specification.

Raises:

AssertionError : When the directive is missing componenents. TypeError : When the directive is unable to be parsed.

axiom.drs.utilities.parse_domain_directive(directive)

Parse a domain directive (i.e. from CLI).

Args:

directive (str): Domain directive of the form name,dx,lat_min,lat_max,lon_min,lon_max

Returns:

dict : Domain dictionary

Raises:

ValueError : If the domain cannot be parsed.

axiom.drs.utilities.postprocess_cordex(ds)

For CORDEX processing, there is some minor postprocessing that happens.

Args:

ds (xarray.Dataset): Data.

Returns:

xarray.Dataset: Data with postprocessing applied.

axiom.drs.utilities.standardise_units(ds)

Standardise units.

Only converts mm to m for now.

Args:

ds (xarray.Dataset): Dataset

Returns:

xarray.Dataset: Dataset with units standardised.

Module contents

Main entrypoint for DRS processing.

axiom.drs.consume(json_filepath)

Consume a json payload (for message passing)

Args:

json_filepath (str): Path to the JSON file.

axiom.drs.filter_years(filepaths, year, offset=0)

Filter filepaths based on a year, plus or minus an offset.

Args:

filepaths (list): List of filepaths. year (int): Year. offset (int, optional): Number of years either side of YEAR to include. Defaults to 0.

Returns:

list : List of filtered filepaths.

axiom.drs.is_error_recoverable(exception)

Check if the error is recoverable.

Args:

exception (Exception): Raised exception to check.

Returns:

bool : True if recoverable, False otherwise.

axiom.drs.load_variable_config(project_config)

Extract the variable configuration out of the project configuration.

Args:

project_config (dict-like): Project configuration.

Returns:

dict: Variable dictionary with name: [levels] (single level will have a list containing None.)

axiom.drs.log_exception(message, ex)

Log the exception.

Args:

message (str): Human-readable error message. ex (Exception): Stack trace.

axiom.drs.process(input_files, output_directory, variable, project, model, domain, start_year, end_year, output_frequency, level=None, input_resolution=None, overwrite=True, preprocessor=None, postprocessor=None, **kwargs)

Method to process a single variable/domain/resolution combination.

Args:

input_files (str or list): Globbable string or list of filepaths. output_directory (str) : Path from which to build DRS structure. variable (str): Variable to process. project (str): Project metadata to apply (loaded from user config). model (str): Model metadata to apply (loaded from user config). start_year (int): Start year. end_year (int): End year. output_frequency (str): Output frequency to process. input_resolution (float, optional): Input resolution in km. Leave black to auto-detect from filepaths. overwrite (bool): Overwrite the data at the destination. Defaults to True. preprocessor (str): Data preprocessor to activate on input data. Defaults to None. postprocesser (str): Data postprocess to activate before writing data. Defaults to None. **kwargs: Additional keyword arguments used in metadata interpolation.

axiom.drs.process_multi(variables, domain, project, **kwargs)

Start a processing chain of multiple variables.

Args:

variables (list): List of variables to process. domain (str): Domain to process from domains.json. project (str): Project metadata to use from projects.json. **kwargs: Additional keyword arguments to pass to the processing chain.

axiom.drs.track_failure(variable, exception)

Track the failure for reprocessing.

Args:

variable (str): Variable name. exception (Exception): Exception raised.

axiom.drs.update_cell_methods(ds, variable, dim='time', method='mean')

Update the cell_methods attribute on the data.

Args:

ds (xarray.Dataset): Data. variable (str): Variable being processed. dim (str): Dimension over which method was applied. method (str): Method that was applied.