
Research
PyPI Package Disguised as Instagram Growth Tool Harvests User Credentials
A deceptive PyPI package posing as an Instagram growth tool collects user credentials and sends them to third-party bot services.
ngiab-data-preprocess
Advanced tools
This repository contains tools for preparing data to run a next gen simulation using NGIAB. The tools allow you to select a catchment of interest on an interactive map, choose a date range, and prepare the data with just a few clicks!
This tool prepares data to run a next gen simulation by creating a run package that can be used with NGIAB.
It uses geometry and model attributes from the v2.2 hydrofabric more information on all data sources here.
The raw forcing data is nwm retrospective v3 forcing data or the AORC 1km gridded data depending on user input
For automatic evaluation using Teehr, please run NGIAB interactively using the guide.sh
script.
For automatic interactive visualisation, please run NGIAB interactively using the guide.sh
script
It is highly recommended to use Astral UV to install and run this tool. It works similarly to pip and conda, and I would also recommend you use it for other python projects as it is so useful.
# Install UV
curl -LsSf https://astral.sh/uv/install.sh | sh
# It can be installed via pip if that fails
# pip install uv
# Create a virtual environment in the current directory
uv venv
# Install the tool in the virtual environment
uv pip install ngiab_data_preprocess
# To run the cli
uv run cli --help
# To run the map
uv run map_app
UV automatically detects any virtual environments in the current directory and will use them when you use uv run
.
This package supports pipx and uvx which means you can run the tool without installing it. No virtual environment needed, just UV.
# run this from anywhere
uvx --from ngiab_data_preprocess cli --help
# for the map
uvx --from ngiab_data_preprocess map_app
# If you're installing this on jupyterhub / 2i2c you HAVE TO DEACTIVATE THE CONDA ENV
(notebook) jovyan@jupyter-user:~$ conda deactivate
jovyan@jupyter-user:~$
# The interactive map won't work on 2i2c
# This tool is likely to not work without a virtual environment
python3 -m venv .venv
source .venv/bin/activate
# installing and running the tool
pip install 'ngiab_data_preprocess'
python -m map_app
# CLI instructions at the bottom of the README
To install and run the tool, follow these steps:
git clone https://github.com/CIROH-UA/NGIAB_data_preprocess
cd NGIAB_data_preprocess
uv venv
uv pip install -e .
uv run map_app
Running the command uv run map_app
will open the app in a new browser tab.
To use the tool:
Once all the steps are finished, you can run NGIAB on the folder shown underneath the subset button.
Note: When using the tool, the default output will be stored in the ~/ngiab_preprocess_output/<your-input-feature>/
folder. There is no overwrite protection on the folders.
-h
, --help
: Show the help message and exit.-i INPUT_FEATURE
, --input_feature INPUT_FEATURE
: ID of feature to subset. Providing a prefix will automatically convert to catid, e.g., cat-5173 or gage-01646500 or wb-1234.--vpu VPU_ID
: The id of the vpu to subset e.g 01. 10 = 10L + 10U and 03 = 03N + 03S + 03W. --help
will display all the options.-l
, --latlon
: Use latitude and longitude instead of catid. Expects comma-separated values via the CLI, e.g., python -m ngiab_data_cli -i 54.33,-69.4 -l -s
.-g
, --gage
: Use gage ID instead of catid. Expects a single gage ID via the CLI, e.g., python -m ngiab_data_cli -i 01646500 -g -s
.-s
, --subset
: Subset the hydrofabric to the given feature.-f
, --forcings
: Generate forcings for the given feature.-r
, --realization
: Create a realization for the given feature.--start_date START_DATE
, --start START_DATE
: Start date for forcings/realization (format YYYY-MM-DD).--end_date END_DATE
, --end END_DATE
: End date for forcings/realization (format YYYY-MM-DD).-o OUTPUT_NAME
, --output_name OUTPUT_NAME
: Name of the output folder.--source
: The datasource you want to use, either nwm
for retrospective v3 or aorc
. Default is nwm
-D
, --debug
: Enable debug logging.--run
: Automatically run Next Gen against the output folder.--validate
: Run every missing step required to run ngiab.-a
, --all
: Run all operations: subset, forcings, realization, run Next Gengage-
, you do not need to pass -g
.-l
, -g
, -s
, -f
, -r
flags can be combined like normal CLI flags. For example, to subset, generate forcings, and create a realization, you can use -sfr
or -s -f -r
.--all
flag, it automatically sets subset
, forcings
, realization
, and run
to True
.--run
flag automatically sets the --validate
flag.Prepare everything for a nextgen run at a given gage:
python -m ngiab_data_cli -i gage-10154200 -sfr --start 2022-01-01 --end 2022-02-28
# add --run or replace -sfr with --all to run nextgen in a box too
# to name the folder, add -o folder_name
Subset hydrofabric using catchment ID or VPU:
python -m ngiab_data_cli -i cat-7080 -s
python -m ngiab_data_cli --vpu 01 -s
Generate forcings using a single catchment ID:
python -m ngiab_data_cli -i cat-5173 -f --start 2022-01-01 --end 2022-02-28
Create realization using a lat/lon pair and output to a named folder:
python -m ngiab_data_cli -i 54.33,-69.4 -l -r --start 2022-01-01 --end 2022-02-28 -o custom_output
Perform all operations using a lat/lon pair:
python -m ngiab_data_cli -i 54.33,-69.4 -l -s -f -r --start 2022-01-01 --end 2022-02-28
Subset hydrofabric using gage ID:
python -m ngiab_data_cli -i 10154200 -g -s
# or
python -m ngiab_data_cli -i gage-10154200 -s
Generate forcings using a single gage ID:
python -m ngiab_data_cli -i 01646500 -g -f --start 2022-01-01 --end 2022-02-28
Run all operations, including Next Gen and evaluation/plotting:
python -m ngiab_data_cli -i cat-5173 -a --start 2022-01-01 --end 2022-02-28
FAQs
Graphical Tools for creating Next Gen Water model input data.
We found that ngiab-data-preprocess demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
A deceptive PyPI package posing as an Instagram growth tool collects user credentials and sends them to third-party bot services.
Product
Socket now supports pylock.toml, enabling secure, reproducible Python builds with advanced scanning and full alignment with PEP 751's new standard.
Security News
Research
Socket uncovered two npm packages that register hidden HTTP endpoints to delete all files on command.