The Python package for the threedigrid administration.
.. image:: https://github.com/nens/threedigrid/workflows/Linux/badge.svg :alt: Github Actions status :target: https://github.com/nens/threedigrid/actions/workflows/test.yml?query=branch%3Amaster
.. image:: https://readthedocs.org/projects/threedigrid/badge/?version=latest :target: https://threedigrid.readthedocs.io/en/latest/?badge=latest :alt: Documentation Status
.. PyPI
.. image:: https://img.shields.io/pypi/v/threedigrid.svg :alt: PyPI :target: https://pypi.org/project/threedigrid/
.. Anaconda
.. image:: https://img.shields.io/conda/vn/conda-forge/threedigrid :alt: Anaconda :target: https://anaconda.org/conda-forge/threedigrid
GridH5Admin objectThe standard threedigrid distribution is pretty lightweight, installing as little dependencies as possible. If you want to make use of all capabilities threedigrid has to ofter (e.g. spatial operations and command line tools) install like this::
$ pip install threedigrid[geo,results]
Console scripts +++++++++++++++
Using the 3digrid_explore shortcut, simply run::
$ 3digrid_explore --grid-file=<path to grid file> --ipy
This will invoke an ipython session with a GridH5Admin instance already loaded.
To get a quick overview of the threedimodels meta data omit the --ipy option or
explicitly run::
$ 3digrid_explore --grid-file=<the to grid file> --no-ipy
This will give you output like this::
Overview of model specifics:
model slug: v2_bergermeer-v2_bergermeer_bres_maalstop-58-b1f8179f1f3c2333adb08c9e6933fa7b9a8cd163
threedicore version: 0-20180315-3578e9b-1
threedi version: 1.63.dev0
has 1d: True
has 2d: True
has groundwater: True
has levees: True
has breaches: True
has pumpstations: True
(I)Python shell +++++++++++++++ Get a grid admin instance::
from threedigrid.admin.gridadmin import GridH5Admin
f = 'gridadmin.h5'
ga = GridH5Admin(f)
The grid admin directly holds some model specific attributes like whether the model has a 1D or 2D or groundwater section::
In [4]: ga.has_groundwater
Out[4]: False
In [5]: ga.has_1d
Out[5]: True
There are different types of filters but a filter, generally speaking, acts on field. That means you can filter by value. If you have a line model instance you can filter the data by the kcu field::
ga.lines.filter(kcu__in=[100,102])
or by the lik value::
ga.lines.filter(lik__eq=4)
The filtering is lazy, that is, to retrieve data you have to call data explicitly::
ga.lines.filter(lik__eq=4).data # will return an ordered dict
The structure control actions netcdf can also be analyzed and exported using threedigrid::
from threedigrid.admin.gridresultadmin import GridH5StructureControl
from threedigrid.admin.structure_controls.exporters import structure_control_actions_to_csv
gst = GridH5StructureControl("gridadmin.h5", "structure_control_actions_3di.nc")
gst.table_control
structure_control_actions_to_csv(gst, "test.csv")
Currently only the client-side is included. The server-side might be added in a later stage. Note: this is an advanced feature used inside the 3Di stack, probably you don't need this. Note2: you need Python 3.7 or higher for this to work.
Installation::
$ pip install threedigrid[rpc]
Basic usage::
ga = GridH5ResultAdmin('rpc://REDIS_HOST/SIMULATION_ID', 'rpc://REDIS_HOST/SIMULATION_ID')
# Replace REDIS_HOST and SIMULATION_ID with actual values.
future_result = ga.nodes.filter(lik__eq=4).data
data = await future_result.resolve()
Subscription usage::
subscription = await future_result.subscribe()
async for item in subscription.enumerate():
# do something with item
In order to set up a virtual environment, perform the following steps:
Clone the repo and fetch the LFS objects::
git lfs fetch origin refs/remotes/origin/master
git lfs checkout
Install platform dependencies::
sudo apt-get update && sudo apt-get install --yes --no-install-recommends libgdal-dev
Create and activate a virtual environment::
python -m venv ./venv
source ./venv/bin/activate
Install the dependencies. For your distribution, check the dependency matrix in .github/workflows/test.yml. For example, for Python 3.10::
pip install --disable-pip-version-check --upgrade pip setuptools wheel
pip install -e .[geo,results] pygdal==$(gdal-config --version).* ipython pytest flake8 sphinx==1.8.5 docutils==0.17.* sphinx_rtd_theme>=0.4.3 numpy==1.23.* h5py==3.7.* shapely==1.8.* pyproj==3.4.* geojson==2.5.* mercantile==1.2.1 cftime==1.6.2
This package was created with Cookiecutter_ and the audreyr/cookiecutter-pypackage_ project template.
.. _Cookiecutter: https://github.com/audreyr/cookiecutter
.. _audreyr/cookiecutter-pypackage: https://github.com/audreyr/cookiecutter-pypackage
Add customized water quality results NetCDF interface.
Extract area name from customized NetCDF files.
Fix slicing of Id fields in customized NetCDF files.
substances and overwrite get_model_instance_by_field_name method
in GridH5WaterQualityResultAdmin.group_by_grid_id to structure control interface.Set substance units on Nodes.
Set exchange level on levl attribute on breach geojson export.
Removed old gridadmin file from tests.
Set pumps on start node and add line geometry in geojson export.
Return YZ tabulated data in geojson export if YZ profile.
Substance count regex fix
Set substance name on Nodes
Bugfix: only in case of boundaries: correct sorting of timeseries in all TimeSeriesSubsetArrayField. This
affected: Nodes [infiltration_rate_simple, ucx, ucy, leak, intercepted_volume, q_sss]
Lines: [qp, up1, breach_depth, breach_width]
Note: The depth and width field are broken for breaches object.
Please use breach_depth and breach_width on the lines object instead.
Add water quality results NetCDF interface.
Drop Python 3.7 support
sewerage and sewerage_type fields to Line model.Added meta table to Geokpackage export with meta information about 3Di model.
Added 2D groundwater boundary line types 600, 700, 800, 900. Eextended the '2D_GROUNDWATER' subset.
Removed gridadmin.lines.breaches.
Removed MappedSubsetArrayField.
Added 'code' and 'display_name' to breaches.
Added missing Geopackage fields bottom_level and drain_level for nodes. Renamed connection_node_storage_are to connection_node_storage_area.
Dropped python 3.6 support.
Fixed compatibility with shapely 2.*.
Optimized geojson export memory usage.
Removed majority of properties from the flowlines (1D2D) export.
Added new OGR based exporter for Geopackage exports. Currently only has default export settings for lines, cells, nodes and pumps.
Fixed error in crs attribute.
Removed deprecation warning on gridadmin.breaches and added it to gridadmin.lines.breaches.
Fixed export_breaches for new gridadmins.
Added discharge_coefficient_positive and discharge_coefficient_negative to breaches.
Removed threedicore version check on GridH5ResultAdmin initialization.
Added crs attribute to GridH5Admin. Pyproj >=2.2 is required.
Drop python 2.7 support.
Fix geometry selection filtering, gridadmin can now contain nan values.
Added dimp attribute to nodes.
Exporters now export NaN and -9999.0 float values as NULL.
Fixed exporting string dtype fields (e.g. cont_type) in OGR exporter. For instance, the string "b'something'" now is written as "something".
Skip the dummy element (with id=0) in all exporters.
Set the FID (feature ID) in the OGR (shapefile/geopackage/some geojson) exporters.
Deprecate specific serializers.py under threedigrid.admin.breaches, .lines, .nodes and .pumps.
Renamed the 's_1d' field under lines to 'ds1d_half'.
Added 'initial_waterlevel' to nodes.
dpumax cross1 cross2 ds1d s1d cross_weight invert_level_start_point invert_level_end_pointcalculation_type drain_level storage_area dmaxReplace nan with null in geojson output.
Remove requirements files, only keep one for development in docker.
Fixed only filter for aggregate result admin. (#121)
Added cross_pix_coords field to lines.
Bugfix: geojson line_geometries export crashes with Python 3.8.10 and numpy 1.19.1. Needed explicit astype conversion
Bumped asyncio-rpc to 0.1.10
Fixed GeoJSON export with pyproj <= 1.9.6.
Fixed compatibility with h5py 3.*.
Added gridadmin.grid.transform.
Fixed gridadmin.grid.n2dtot and .dx propagation.
Added gridadmin.cells.iter_by_tile() and .get_extent_pixels().
Added extra field 'discharge_coefficient' to channels and pipes. These fields default to 1.0.
Bugfix: don't use the z-coordinate when making line_geometries during the prepare step
Invert_level_start_point and end point where inverted by Inpy. After this change the values will be correct again, however present gridadmin files will still have the incorrect value.
Added Breaches model under lines with specific breach fields
mapped from the 'breaches' h5py datagroup.
Added MappedSubsetArrayField allowing to map arrays from other
h5py datagroups to a model on another datagroup subset. Breaches
uses this to map the array's under 'breaches' to 'lines'
Fixed RPC breaches/pumps bug
Added fast nod_grid generation on cells
content_pk to the export_constants of all structures which have a
content_pk.An empy array [] is returned now instead of None if there is no dataset.
Line geojson items need to use line_geometry values
Bugfix for Model._get_subset_idx not instantiating new subsets with their parent's
mixins.
Allow older pyrpoj versions. (pre 2.2.0)
Bugfix for timeseries start_time=0 selection and allow indexes=slice(x,x,x) in combination with sample() method.
Added try-except surrounding all imports of the package geojson. This package is
only available when threedigrid is installed with the extra [geo] extension.
Add ORIFICES_EXPORT_FIELDS to export_constants.
Add extra field pixel_width to cells
Bugfix for GeometryIntersectionFilter: filter was only checking on intersecting bounding boxes
sample method needs to skip the last timestamp for SWMR
to work correctly. (time dataset can have one item more
than datasets with timeseries)Added GeometryIntersectionFilter.
Added general GeoJsonSerializer which allows you to specify the field names you want to serialize and extract to geojson. The GeoJsonSerializer allows you to specify nested fields.
Added a set of standard export fields for each model.
Automatically pick the correct serializer based on file extention
Added RPC datasource which enables to use the majority of threedigrid in a RPC setting. Uses asyncio-rpc for sending/handling RPC calls.
RPC datasource allows both one time executing (resolve()) and pub/sub
(subscribe()) functionialty.
Refactored to allow using RPC datasource
Added content_pk to the pumps model.
Bumped package versions
Reduced reprojection overhead of line_geometries.
0 has a default when converting database objects to numpy
arrays in the prepare phase._field_model_dict being a class variable.h5py_file method mapping.Manholes preparation fixed mapping in connection_node_pk.
Added to_structured_array method for retrieving (filtered) results
as Numpy structured array instead of an OrderedDict
code, display_name and shape. These fields
now have a fixed lenght of 32, 64 and 4 characters respectively.Add aggregation option 'current' to volume and intercepted_volume.
Using a non-tuple sequence for multidimensional indexing is deprecated; use
arr[tuple(seq)] instead of arr[seq].
Properties should be strings so we can use string methods on them.
Do not prepare levees if there aren't any.
Split requirements file to allow for finer grained builds (for instance to generate the documentation).
Add 'intercepted_volume' to NodesAggregateResultsMixin.
Split requirements file to allow for finer grained builds (for instance to generate the documentation).
is_manhole filter.
NO_DATA_VALUE is interpreted as False.ImportErrorPatch for converting numpy strings/bytes to float for both python2/3.
Dropped NetCDF library and replaced opening NetCDF files with h5py
Bumped h5py to 2.8.0
Bug fix for issue #44: use the method get_filtered_field_value() instead
of get_field_value() for the count property.
Properly closes netcdf-file in GridH5ResultAdmin.
export_all() collection.pkg_resources to determine the current version but use
zest_releaser to update the version string in threedigrid/init.pyNumpyEncoder to convert specific numpy types to native
python types when calling (geo-)json.dumps().Fix lookup_index functionality for composite fields.
Make model name property optional. That is, 'unknown' will be returned if the name cannot be derived.
Changed Depth/width fields on breach-timeseries to breach_depth and breach_width.
_get_composite_meta() does not raise an AssertionError anymore if
composite field attributes differ. Instead a warning is issued.threedicore_result_version must be a property.Added additional exporters for
Added method get_model_instance_by_field_name to the
GridH5ResultAdmin class. This makes it possible to do reverse lookups
in situations where you have a field name but do not know which model it
belongs to. N.B the field must be unique otherwise an IndexError will
be raised.
Added property dt_timestamps to the timeseries_mixin module.
The version number is added to the __init__ file dynamically using the
pkg_resources API.
Timestamps of all timeseries fields are shown for aggregation results.
Timestamps in the aggregation results are filtered when retrieving subsets of timeseries.
Introducing the ModelMeta class. Its main purpose at this moment is to compute all possible combinations of composite_fields and aggregation variables.
Fixed return statement of method slice (in class Model) which now takes
**new_class_kwargs.
Empty or missing datasets are now displayed as np.array(None) instead of
raising an error.
Changed ResultMixin to dynamically add attributes based on the netcdf variables.
Added basic result proccesing for line/node data.
The filter mask is computed only for array's affected and before applying it to all array's
The 'only' filter works much faster because the filter mask is only applied on fields that are affected.
The filter mask is cached on the line/node instance after getting the first value. You can thus do something like:
queryset = gridadmin.lines.filter(kcu=2)
ids = queryset.id
line_coords = queryset.line_coords
and the filter mask will only be computed once.
Add click console scripts 3digrid_explore and 3digrid_export for
quick overviews and data exports.
Make ogr/gdal imports optional to avoid breaking parts of the documentation.
Added documentation and setup for sphinx documentation pipeline.
Use linear referencing for embedded channels to keep the original geometry intact when preparing line geometries for visualisation.
Define extra's to make the standard threedigrid distribution as lightweight as possible.
Convert strings in attrs to numpy.string_ to fix crashes under
Windows.
has_groundwater from GridH5Admin.
Should always be provided by the threedicore itself. Gives a warning for
backwards compatibility.All imports are absolute.
Added install info using pip.
FAQs
Python package for the threedigrid administration
We found that threedigrid demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 6 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Malicious PyPI checkers validate stolen emails against TikTok and Instagram APIs, enabling targeted account attacks and dark web credential sales.
Security News
The Node.js TSC opted not to endorse a feature bounty program, citing concerns about incentives, governance, and project neutrality.
Research
Security News
A look at the top trends in how threat actors are weaponizing open source packages to deliver malware and persist across the software supply chain.