Independent ZC proxies only for testing on it.
Modified and simplified from foresight repo, fix some bugs in model output, remove some unwanted code snippets.
Supported zc-metrics are:
=========================================================
= grad_norm, =
=-------------------------------------------------------=
= grasp, =
=-------------------------------------------------------=
= snip, =
=-------------------------------------------------------=
= synflow, =
=-------------------------------------------------------=
= nwot, (NASWOT) =
= [nwot, nwot_Kmats] =
=-------------------------------------------------------=
= lnwot, (Layerwise NASWOT) =
= [lnwot, lnwot_Kmats] =
=-------------------------------------------------------=
= nwot_relu, (original RELU based NASWOT metric) =
= [nwot_relu, nwot_relu_Kmats] =
=-------------------------------------------------------=
= zen, =
= Your network need have attribute fn: =
= `forward_before_global_avg_pool(inputs)` =
= to calculate zenas score =
= (see sample code in tests/test_zc.py) =
=-------------------------------------------------------=
= tenas, =
= must work in `gpu` env, =
= might encouter bug on `cpu`. =
= also contains metrics: =
= ntk, =
= lrn, =
=-------------------------------------------------------=
= zico, not work in torch-cpu, I will check it later. =
= zico must use at least two batches of data, =
= in order to calculate cross-batch (non-zero) std =
=-------------------------------------------------------=
= tcet, =
= snr-synflow, =
= snr-snip, =
=========================================================
First create conda env with python version >= 3.6, this repo has been completely tested on python 3.9.
conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.6 -c pytorch -c nvidia
Install torch, torchvision, cudatoolkit.
Tested on:
pytorch==1.13.1 (py3.9_cuda11.6_cudnn8.3.2_0)
python==3.9.16
cuda 11.6
torchvision==0.14.1 (py39_cu116)
torchaudio==0.13.1 (py39_cu116)
this repo is perfectly compatible with current mainstream zc testing framework, including zennas, naslib, nb201 related repos, nb101, nb1shot1, blox, etc.
If you still cannot use this repo, try to contact me, or try to setup some mainstream nas testing benchmarks, then most problems would be solved.
Finally, if all the previous basic enviroment requirements are met, then try this lib with just one single command.
pip install -e .
# running this command under the root directory where the setup.py locates in.
check installation success.
cd tests/
python test_zc.py
ImageNet16-120 cannot be automatically downloaded. Using script under scripts/download_data.sh to download:
source scripts/download_data.sh nb201 ImageNet16-120
# do not use `bash`, use `source` instead
tenas, add net instance deep copy to avoid weight changes.tenas, now using new torch api to calc eigenvalue.tcet, add net instance deep copy to avoid weight changes, add manually designed tcet copy process, remove bn in synflow, add bn in snip.tcet metric, which calculates TCET score.
Add snr metrics, which calculates SNR family scores.zico metric, which calculates ZICO score.nwot_relu for wrong for/backward fn register,zen for missed necessary attribute check, add test sample for zen metric,zen for return value have not .item() attribute,tenas metric, which calculates TE-NAS score. (tenas, ntk, lrn)zen metric, which calculates ZenNAS score.naswot implements based on RELU, can be calculated using metirc nwot_relu, also fix potential oom bug, and more reliable GPU memory cache removal code snippets.naswot, lnwot into matsif you encouther this error:
RuntimeError: "addmm_impl_cpu_" not implemented for 'Half'
Traceback (most recent call last):
File "/home/u2280887/GitHub/zero-cost-proxies/tests/test_zc.py", line 87, in <module>
test_zc_proxies()
File "/home/u2280887/GitHub/zero-cost-proxies/tests/test_zc.py", line 49, in test_zc_proxies
results = calc_zc_metrics(metrics=mts, model=net, train_queue=train_loader, device=device, aggregate=True)
File "/home/u2280887/GitHub/zero-cost-proxies/alethiometer/zc_proxy.py", line 115, in calc_zc_metrics
mt_vals = calc_vals(net_orig=model, trainloader=train_queue, device=device, metric_names=metrics, loss_fn=loss_fn)
File "/home/u2280887/GitHub/zero-cost-proxies/alethiometer/zc_proxy.py", line 101, in calc_vals
raise e
File "/home/u2280887/GitHub/zero-cost-proxies/alethiometer/zc_proxy.py", line 73, in calc_vals
val = M.calc_metric(mt_name, net_orig, device, inputs, targets, loss_fn=loss_fn, split_data=ds)
File "/home/u2280887/GitHub/zero-cost-proxies/alethiometer/zero_cost_metrics/__init__.py", line 42, in calc_metric
return _metric_impls[name](net, device, *args, **kwargs)
File "/home/u2280887/GitHub/zero-cost-proxies/alethiometer/zero_cost_metrics/__init__.py", line 24, in metric_impl
ret = func(net, *args, **kwargs, **impl_args)
File "/home/u2280887/GitHub/zero-cost-proxies/alethiometer/zero_cost_metrics/tenas.py", line 316, in compute_TENAS_score
RN = compute_RN_score(net, inputs, targets, split_data, loss_fn, num_batch)
File "/home/u2280887/GitHub/zero-cost-proxies/alethiometer/zero_cost_metrics/tenas.py", line 201, in compute_RN_score
num_linear_regions = float(lrc_model.forward_batch_sample()[0])
File "/home/u2280887/GitHub/zero-cost-proxies/alethiometer/zero_cost_metrics/tenas.py", line 170, in forward_batch_sample
return [LRCount.getLinearReginCount() for LRCount in self.LRCounts]
File "/home/u2280887/GitHub/zero-cost-proxies/alethiometer/zero_cost_metrics/tenas.py", line 170, in <listcomp>
return [LRCount.getLinearReginCount() for LRCount in self.LRCounts]
File "/home/u2280887/GitHub/zero-cost-proxies/alethiometer/zero_cost_metrics/tenas.py", line 93, in getLinearReginCount
self.calc_LR()
File "/home/u2280887/miniconda3/envs/zc-alth/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/home/u2280887/GitHub/zero-cost-proxies/alethiometer/zero_cost_metrics/tenas.py", line 62, in calc_LR
res = torch.matmul(self.activations.half(), (1-self.activations).T.half())
RuntimeError: "addmm_impl_cpu_" not implemented for 'Half'
please check your lib installation, we need gpu support for torch.half(), please check your cuda version and pytorch version, and reinstall pytorch with cuda support. It seem current cpu version of pytorch does not support torch.half(), even if we are using float32 not float16.
....
FAQs
ZC proxies calculation repo, altered from foresight package.
We found that alethiometer demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
PyPI confirms no security flaws were exploited in the Ultralytics supply chain attack and highlights improvements for safer package publishing.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.