axiom package

Subpackages

• axiom.drs package
  • Subpackages
    • axiom.drs.processing package
      • Submodules
      • axiom.drs.processing.ccam module
      • Module contents
  • Submodules
  • axiom.drs.cli module
  • axiom.drs.domain module
  • axiom.drs.payload module
  • axiom.drs.utilities module
  • Module contents
• axiom.validation package
  • Submodules
  • axiom.validation.validator module
  • Module contents

Submodules

axiom.config module

Configuration.

class axiom.config.Config

Bases: dict

get(key, default=None)

Return the value for key if key is in the dictionary, else default.

load(config_name, defaults_only=False)

Load the configuration in a cascading fashion, defaults first, then overlay with user.

Args:

config_name (str): Configuration name, without file extension. defaults_only (bool, optional): Load only the defaults. Defaults to False.

axiom.config.load_config(config_name, defaults_only=False)

Shorthand to load a configuration object.

Args:

config_name (str): Name of the config file. defaults_only (bool, optional): Load only the defaults. Defaults to False.

Returns:

axiom.Config: Configuration object.

axiom.converters module

Conversion routines to convert between units automatically. EXPERIMENTAL.

axiom.converters.ensure_units(da, expected_units)

Ensure that a given DataArray has the correct units, converting if needed.

Args:

da (xarray.DataArray): Data. expected_units (ste): Units value.

Returns:

xarray.DataArray: Data with units converted.

Raises:

KeyError : When no unit attribute is present. KeyError : When no converter is registered between the old and new units. Exception: When the converter fails to convert the units.

axiom.converters.mm2m(da)

Convert mm to m.

Args:

da (xarray.DataArray): Data in mm.

Returns:

xarray.DataArray: Data in m.

axiom.decorators module

Decorators.

axiom.decorators.metadata(**meta)

Apply metadata to the xarray outputs of a function.

Usage Example:

@metadata(units=’K’) def get_temp(ds):

return ds.T

axiom.decorators.singleton(cls)

Define a class as a singleton.

Returns:

object: Singleton

axiom.drs module

Main entrypoint for DRS processing.

axiom.drs.consume(json_filepath)

Consume a json payload (for message passing)

Args:

json_filepath (str): Path to the JSON file.

axiom.drs.filter_years(filepaths, year, offset=0)

Filter filepaths based on a year, plus or minus an offset.

Args:

filepaths (list): List of filepaths. year (int): Year. offset (int, optional): Number of years either side of YEAR to include. Defaults to 0.

Returns:

list : List of filtered filepaths.

axiom.drs.is_error_recoverable(exception)

Check if the error is recoverable.

Args:

exception (Exception): Raised exception to check.

Returns:

bool : True if recoverable, False otherwise.

axiom.drs.load_variable_config(project_config)

Extract the variable configuration out of the project configuration.

Args:

project_config (dict-like): Project configuration.

Returns:

dict: Variable dictionary with name: [levels] (single level will have a list containing None.)

axiom.drs.log_exception(message, ex)

Log the exception.

Args:

message (str): Human-readable error message. ex (Exception): Stack trace.

axiom.drs.process(input_files, output_directory, variable, project, model, domain, start_year, end_year, output_frequency, level=None, input_resolution=None, overwrite=True, preprocessor=None, postprocessor=None, **kwargs)

Method to process a single variable/domain/resolution combination.

Args:

input_files (str or list): Globbable string or list of filepaths. output_directory (str) : Path from which to build DRS structure. variable (str): Variable to process. project (str): Project metadata to apply (loaded from user config). model (str): Model metadata to apply (loaded from user config). start_year (int): Start year. end_year (int): End year. output_frequency (str): Output frequency to process. input_resolution (float, optional): Input resolution in km. Leave black to auto-detect from filepaths. overwrite (bool): Overwrite the data at the destination. Defaults to True. preprocessor (str): Data preprocessor to activate on input data. Defaults to None. postprocesser (str): Data postprocess to activate before writing data. Defaults to None. **kwargs: Additional keyword arguments used in metadata interpolation.

axiom.drs.process_multi(variables, domain, project, **kwargs)

Start a processing chain of multiple variables.

Args:

variables (list): List of variables to process. domain (str): Domain to process from domains.json. project (str): Project metadata to use from projects.json. **kwargs: Additional keyword arguments to pass to the processing chain.

axiom.drs.track_failure(variable, exception)

Track the failure for reprocessing.

Args:

variable (str): Variable name. exception (Exception): Exception raised.

axiom.drs.update_cell_methods(ds, variable, dim='time', method='mean')

Update the cell_methods attribute on the data.

Args:

ds (xarray.Dataset): Data. variable (str): Variable being processed. dim (str): Dimension over which method was applied. method (str): Method that was applied.

axiom.exceptions module

Custom exception classes.

exception axiom.exceptions.DRSContextInterpolationException(placeholders)

Bases: Exception

Raised when there are remaining placeholders in a template after DRS context interpolation.

Args:

placeholders (list) : List of placeholders that have yet to be interpolated.

exception axiom.exceptions.MalformedDRSJSONPayloadException

Bases: Exception

Raised when a DRS JSON payload can’t be processed.

exception axiom.exceptions.NoFilesToProcessException

Bases: Exception

Raised when there are no files to process for a DRS payload

exception axiom.exceptions.ResolutionDetectionException

Bases: Exception

Raised when tere are issues detecting resolution information.

axiom.report module

Reporting functions.

axiom.report.generate_report(validator, input_filepath, report_filepath)

Generate a report for a validator.

Args:

validator (axiom.validation.Validator): Validator object. input_filepath (str) : Path to the file that was validated. report_filepath (str): Path to which to write the report.

axiom.utilities module

General utilities.

class axiom.utilities.ListAwareConfigParser

Bases: configparser.ConfigParser

getlist(section, key, delimiter='\n', cast=<class 'str'>)

Get an option out of the configuration and automatically split into a list.

Args:

section (str): Section name. key (str): Option key. delimiter (str, optional): Delimiter. Defaults to ‘

‘.

cast (callable): Typecast on the fly to this type.

Returns:

list: Configuration option as a list.

section2dict(section, detect_dtypes=True)

class axiom.utilities.Timer

Bases: object

Basic timer class.

start()

Start the timer.

stop()

Stop the timer.

Returns:

float : Elapsed time in seconds

axiom.utilities.apply_schema(ds, schema)

Apply a metadata schema on a dataset.

Args:

ds (xarray.Dataset): Dataset. schema (dict): Axiom schema dictionary.

Returns:

xarray.Dataset : Dataset with schema-defined metadata applied.

axiom.utilities.auto_glob(path)

Shorthand for sorted(glob.glob(path))

Args:

mask (str): Globabble path

Returns:

list : List of paths that match the globbable part.

axiom.utilities.batch_split(iterable, n_batches)

Split iterable into n_batches.

Args:

iterable (iterable) : Iterable object to split. n_batches (int) : Number of batches.

Returns:

list : List of iterables.

axiom.utilities.conditional_rename(ds, **kwargs)

Conditionally rename an object on a dataset, if it exists.

Args:

ds (xarray.Dataset): Dataset. **kwargs : Key/value pairs of old=new names.

Returns:

xarray.Dataset : Dataset with renamed variables/coords.

axiom.utilities.dict2obj(d)

Convert a dictionary to an object.

axiom.utilities.diff_metadata(meta_a, meta_b, ignore_matches=True)

Difference the metadata between two metadata dictionaries.

Differences are encoded with values as tuples, where: - (None, None) = Attributes match. - (None, value2) = Attribute is missing from meta_a. - (value1, None) = Attribute is missing from meta_b. - (value1, value2) = Differing values between meta_a and meta_b.

Args:

meta_a (dict): Metadata dictionary of the form from extract_metadata. meta_b (dict): Metadata dictionary of the form from extract_metadata.

Returns:

dict : Dictionary of differences.

axiom.utilities.extract_metadata(ds)

Extract metadata from an xarray dataset.

Args:

ds (str or xarray.Dataset): Path to (netcdf) or dataset.

Returns:

dict : Metadata dictionary.

axiom.utilities.get_installed_data_root()

Get the root directory for installed data.

Returns:

str : Root directory.

axiom.utilities.get_lock_filepath(filepath)

Get a lock file path for filepath.

Args:

filepath (str): Path.

Returns:

str: Lock file path.

axiom.utilities.get_logger(name, level='debug')

Get a logging object.

Args:

name (str): Name of the module currently logging. level (str, optional): Level of logging to emit. Defaults to ‘debug’.

Returns:

logging.Logger: Logging object.

axiom.utilities.get_timestamp()

Generate a JSON-compliant timestamp.

Returns:

str : Timestamp.

axiom.utilities.get_user_data_root()

Get the root directory for user data.

Returns:

str : Root directory.

axiom.utilities.get_variables_and_coordinates(ds)

Get a list of variable and coordinate names.

Args:

ds (xarray.Dataset): Dataset

Returns:

list: List of variable and coordinate names

axiom.utilities.has_attr(obj, attr)

Test if obj has the attribute attr.

Args:

obj (xarray.DataArray or xarray.Dataset): xarray object. attr (str): Name of the attribute.

Returns:

bool: True if obj has attribute attr, False otherwise.

axiom.utilities.has_coord(dx, coord)

Check if a DataArray or Dataset contains a coordinate.

Args:

dx (xarray.DataArray or xarray.Dataset): Data. coord (str): Coordinate name.

Returns:

bool: True if coordinate in object, False otherwise.

axiom.utilities.has_dim(dx, dim)

Check if a DataArray or Dataset contains a dimension.

Args:

dx (xarray.DataArray or xarray.Dataset): Data. dim (str): Dimension name.

Returns:

bool: True if dim on object, False otherwise.

axiom.utilities.has_variable(ds, variable)

Check if a dataset contains a given variable.

Args:

ds (xarray.Dataset): Dataset. variable (str): Variable name.

Returns:

bool: True if ds contains variable, False otherwise.

axiom.utilities.in_ds(variable, ds)

Test if variable is in the data file.

Args:

variable (str): Variable name. ds (xarray.Dataset): Data.

Returns:

bool : True if the variable exists, False otherwise.

axiom.utilities.infer_dtype(value)

Infer the data type of the value passed.

Args:

value (unknown): Value.

Raises:

ValueError : When the type can’t be inferred.

axiom.utilities.interpolate_template(template, **kwargs)

Interpolate onto a template string.

Args:

string (str): Template string. **kwargs : Key/value pairs to interpolate.

Returns:

str : Interpolated string

axiom.utilities.is_locked(filepath)

Check if a file is locked.

Args:

filepath (str): Path to the file.

axiom.utilities.isolate_coordinate(obj, coordinate_name, drop=True)

Ensure that coordinate_name is the only coordinate on obj by selecting the first index of all other coordinates.

Args:

obj (xarray.DataArray or xarray.Dataset): xarray object. coordinate_name (str): Name of the coordinate to retain. drop (bool, optional): Drop the coordinate_name from the object. Defaults to True.

axiom.utilities.load_cf_standard_name_table(filepath)

Load and convert a CF standard name table XML into a metadata schema.

Args:

filepath (str): Path to the file.

Returns:

dict : Metadata dictionary (will be long).

axiom.utilities.load_cordex_csv(filepath, **kwargs)

Load a cordex csv (i.e. from CCAM) and convert it into a metadata schema.

Note: There are no _global attributes.

Args:

filepath (str): Path to file **kwargs : Additional key/value pairs to add to the schema header (i.e. contact etc.)

Returns:

dict : Schema dictionary

axiom.utilities.load_metadata_json(filepath)

Load a metadata.json file.

Args:

filepath (str): Path to the json file.

Returns:

dict: Metadata dictionary.

axiom.utilities.load_package_data(slug, package_name='axiom', return_type='json')

Load data from the installed Axiom package.

Args:

slug (str): Internal data path. package_name (str) : Name of the package. Defaults to ‘axiom’.

Returns:

mixed : Dictionary of data (JSON only ATM).

axiom.utilities.load_package_json(path)

Load package JSON data from the Axiom installation directory or from the user’s home directory.

Args:

path (str): Path to the data.

Returns:

dict : Data.

axiom.utilities.load_schema_json(filepath)

Load a schema.json file.

Args:

filepath (str): Path to the json file.

Returns:

dict : Schema dictionary.

axiom.utilities.load_user_config(name)

Load a configuration file from the user’s $HOME/.axiom directory.

Args:

name (str): Name of the configuration file.

Returns:

configparser.Config : Configuration.

axiom.utilities.lock(filepath)

Place a lock on a filepath.

Args:

filepath (str): Path to the file.

axiom.utilities.pluralise(obj)

Automatically convert obj into a list object.

For use when an object can be defined singularly or plurally.

Args:

obj (object): Object.

axiom.utilities.save_schema(schema, filepath)

Save the schema to a json file at filepath.

Args:

schema (dict): Schema dictionary. filepath (str): Path to which to save the schema.

axiom.utilities.shell(cmd, shell=True, check=True, capture_output=True, **kwargs)

Execute a shell command.

Args:

cmd (str): Command **kwargs : Passed to subprocess.run

Returns:

[type]: [description]

axiom.utilities.touch(filepath)

Thin wrapper for touching files.

Args:

filepath (str): Path.

axiom.utilities.unlock(filepath)

Remove a lock on a filepath.

Args:

filepath (str): Path to the file.

Module contents

Axiom main module.