data-ecosystem-services
Advanced tools
Run in bash or powershell
Check Python Version
```sh
python3 --version
```
or
```sh
python --version
```
```sh
sudo apt update
sudo apt install python3.9
```
Update Path
```sh
nano ~/.bashrc
export PATH="/usr/bin/python3.9:$PATH"
source ~/.bashrc
```
Run Application
Download: https://www.python.org/ftp/python/3.9.9/python-3.9.9-embed-amd64.zip
Install as a non global user
```sh
sudo apt update
sudo apt-get install python3-pip
```
```sh
py -m ensurepip --upgrade
```
See Virtual Environment Set Reference Article
If first virtual environment on desktop
Open Terminal in home directory
```sh
cd $HOME
```
Create Directory for virtual environment
mkdir .virtualenv
sudo apt install python3-pip
pip3 --version
pip3 install virtualenv
which virtualenv
sudo apt install python3.9-venv
pip install virtualenvwrapper
pip3 install virtualenvwrapper
export PATH="$HOME/.local/bin:$PATH"
export PATH="/usr/bin:$PATH"
echo WORKON_HOME="~/.virtualenvs" | sudo tee -a $HOME/.bashrc
echo VIRTUALENVWRAPPER_VIRTUALENV="$HOME/.local/bin/virtualenv" | sudo tee -a $HOME/.bashrc
echo "VIRTUALENVWRAPPER_PYTHON=$(which python3)" | sudo tee -a $HOME/.bashrc
echo "source $HOME/.local/bin/virtualenvwrapper.sh" | sudo tee -a $HOME/.bashrc
source $HOME/.bashrc
If second or later virtual environment on desktop
mkvirtualenv OCIO_PADE_DEV
mkvirtualenv EZDX_FOODNET_DEV
mkvirtualenv OCIO_PADE_DEV
mkvirtualenv OD_DATALAKE_DEV
mkvirtualenv OD_NHANES_DEV
mkvirtualenv NDSP_PERTUSSIS_DEV
mkvirtualenv NDSP_PERTUSSIS_QA
mkvirtualenv NDSP_PERTUSSIS_ONBOARD
mkvirtualenv NDSP_PERTUSSIS_PROD
deactivate
sudo pip uninstall nodeenv
pip install nodeenv
workon {virtualenv name}
nodeenv -p
Example
deactivate
cd $HOME
# cleanup any existing node
sudo rm -rf node-v18.12.1-linux-x64.tar.xz
sudo rm -rf /usr/local/bin/npm /usr/local/share/man/man1/node* ~/.npm
sudo rm -rf /usr/local/lib/node*
sudo rm -rf /usr/local/bin/node*
sudo rm -rf /usr/local/include/node*
sudo apt-get purge nodejs npm
sudo apt autoremove
sudo pip uninstall nodeenv
deactivate
cd $HOME
# cleanup any existing node
sudo rm -rf node-v18.12.1-linux-x64.tar.xz
# sudo rm -rf /usr/local/bin/npm /usr/local/share/man/man1/node* ~/.npm
# sudo rm -rf /usr/local/lib/node*
# sudo rm -rf /usr/local/bin/node*
# sudo rm -rf /usr/local/include/node*
sudo apt-get purge nodejs npm
sudo apt autoremove
sudo pip uninstall nodeenv
workon VIRTUAL_ENV
# EXAMPLE:
workon OCIO_PADE_DEV
cd $VIRTUAL_ENV
# install node
wget https://nodejs.org/dist/v18.12.1/node-v18.12.1-linux-x64.tar.xz
tar -xf node-v18.12.1-linux-x64.tar.xz
sudo mv node-v18.12.1-linux-x64/bin/* ./bin/
sudo mv node-v18.12.1-linux-x64/lib/node_modules/ ./lib/node_modules/
# Verify installation using
node -v
npm -v
npm install npm@9.1.1
pip install --upgrade pip
pip install nodeenv
cd $VIRTUAL_ENV/lib/node_modules/npm
nodeenv -p
cd $VIRTUAL_ENV/lib
npm install @mermaid-js/mermaid-cli --registry=https://registry.npmjs.org
cdvirtualenv bin
echo "export PATH='$PATH:$VIRTUAL_ENV/lib/node_modules/.bin'" | sudo tee -a activate
source activate
# Test client
mmdc -h
Run
deactivate
rmvirtualenv OCIO_PADE_DEV
For global
sudo rm -rf venv
For local
rm -rf OCIO_PADE_DEV
Run the following command
cd data_ecosystem_services
pytest --cov-report html tests/
Reference 1: Install Docker Engine without Docker Desktop Reference 2: Install Docker for WSL
Option 1: Convenience Script
Docker provides a convenience script at Docker Script to install Docker into development environments non-interactively. The convenience script isn't recommended for production environments, but it's useful for creating a provisioning script tailored to your needs.
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
Note: You will have to wait 20 seconds
2. Build and Run Image
cd .devcontainerlocal
sudo docker build -t OCIO_PADE_DEV .
sudo docker run -it --rm --name OCIO_PADE_DEV OCIO_PADE_DEV
Option 2: Manual Install
wsl --set-default-version 2
sudo apt remove docker docker-engine docker.io containerd runc
sudo apt-get update
sudo apt-get install \
ca-certificates \
curl \
gnupg \
lsb-release
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin
sudo docker run hello-world
Docker CLI is required for managing images. It is available for install at
The OC Client is used to manage all resources in a project and can be downloaded from
The OCP Web sh can also be used to perform most of the project administration tasks. Additional OC information is available at OpenShift Client
Steps to follow:
Run
oc
oc project "ddt-ops-center"
oc port-forward nccdphp-postgresql-test-32-btxvk 5432
oc get po
oc get po|grep post (if you are using git bash as your shell)
oc get po| findstr "post" (select only PostgreSQL pods)
Install Node on Windows
npm install -g npm
Reference: StackOverflow: Poetry with Docker
cd $VIRTUAL_ENV
curl -sSL https://install.python-poetry.org | python3 -
cp $VIRTUAL_ENV/bin/poetry ~/.local/bin
(Invoke-WebRequest -Uri https://install.python-poetry.org -UseBasicParsing).Content | py -
workon EZDX_FOODNET_DEV
workon OCIO_PADE_DEV
workon OD_DATALAKE_DEV
workon NDSP_PERTUSSIS_DEV
workon NDSP_PERTUSSIS_QA
workon NDSP_PERTUSSIS_ONBOARD
workon NDSP_PERTUSSIS_PROD
cd _templates/data_ecosystem_services
poetry update
cd "C:\Users\jcbow\OneDrive - CDC\DAVT Analytics\davt-dev-jcbowyer"
poetry update
For the markdown all in one exension
Symptom: Windows ignores JAVA_HOME
Resolution:
The Java installer will put a copy of java.exe (but no libraries) in the C:\Program Files (x86)\Common Files\Oracle\Java\javapath directory and add that directory to the beginning of the PATH variable.
If you don"t use a full path, the copy of java.exe to run is found by using the PATH system variable. Since this directory doesn"t contain the DLLs of a particular Java runtime version, one is located one by looking at the registry.
So, you either need to modify the registry, or replace the javapath entry with the version of Java you want in your PATH system (not user) variable.
Recommend setting registry
Computer\HKEY_CURRENT_USER\Environment\JAVA_HOME
C:\apps\Java\jdk1.8.0_333
Reference: Stack Overflow
Problem: Unable to remove existing local node_modules
Solution:
Run Command
cd path_with_modules
find . -name 'node_modules' -type d -prune -exec rm -rf '{}' +
Problem: Unable to remove all existing global node_modules
Solution:
Run Command
npm ls -gp --depth=0 | awk -F/ '/node_modules/ && !/\/npm$/ {print $NF}' | xargs npm -g rm
Problem: Unable remove existing python packages
Solution: Run from elevated command prompt to removes all local packages in your environment
pip freeze | xargs pip uninstall -y
Problem: NPM Won't Run After Upgrade using a manual install
Reference: Stack Overflow
Solution:
If npm is no longer installed in /usr/bin/npm, then chances are good bash(1) has hashed the executable name. The hashing saves repeated searches of all directories in your PATH every time you execute common programs. Since programs almost never change directories, this is usually a great idea. To fix this isue, remove the hash by running the following command:
hash -r.
Symptom: Unable to reset WSL root password in Ubuntu
Reference: AskUbuntu
Solution:
wsl -u root
passwd username
exit
wsl
sudo echo hi
spark.conf.set("spark.databricks.pyspark.enablePy4JSecurity", "false")
Symptom: Receive error when creating an Azure B2C tenant: The subscription is not registered to use namespace 'Microsoft.AzureActiveDirectory'. See https://aka.ms/rps-not-found for how to register subscriptions.
Reference: Blog
Resolution:
Login to azure. Start up a shell and type the following command.
az login
This will start up a browser and you will need to log into Azure.
Get subscription list. If you have one subscription then the default will be selected. If not then you will need to check your subscriptions and then set the right one. Using the account list command you can get a list of your subscriptions.
az account list
You will get a list of subscriptions that you have access to. You can then set the subscription you want to work on.
Set the subscription to work on.
az account set --subscription "ZFI4-PERSONAL-DEV"
Register the namespace provider
az provider register --namespace Microsoft.AzureActiveDirectory
Symptom: Receive error: secret should be an Azure Active Directory application's client secret when calling function: credential = ClientSecretCredential(tenant_id, client_id, client_secret)
Resolution:
- If running from local jupyter notebook ensure you have set the local python environment for your project: Example: OD_NHANES_DEV
- Use the client secret of the service principal, not the client secret of the app registration.
- Ensure that the client secret of the service principal is set in the system environment variable for your application: Example: DAVT_OD_NHANES_DEV_AZURE_CLIENT_SECRET
Problem: org.apache.spark.SparkException: Cannot find catalog plugin class for catalog "spark_catalog": org.apache.spark.sql.delta.catalog.DeltaCatalog
Resolution:
From your virtual environment. You can select using workon venv_name
Run
pyspark --packages io.delta:delta-core_2.12:1.0.0 --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"
or
Copy io.delta:delta-core_2.12:1.0.0 JAR file to $SPARK_HOME/lib and restart
This should make error go away.
Reference: Stack Overflow
Symptom: Table creation fails in Databricks with security exception: Error in SQL statement: SecurityException: User does not have permission SELECT on any file.
Reference: Databricks KB
Solution:
Run the following command:
GRANT SELECT ON ANY FILE TO `user1`
Warning
It is important to understand the security implications of granting ANY FILE permissions on a filesystem. You should only grant ANY FILE to privileged users. Users with lower privileges on the cluster should never access data by referencing an actual storage location. Instead, they should access data from tables that are created by privileged users, thus ensuring that Table ACLS are enforced.
In addition, if files in the Databricks root and data buckets are accessible by the cluster and users have MODIFY privileges, the admin should lock down the root.
Problem: Spark illegal character in path
Resolution:
run
spark-class org.apache.spark.deploy.master.Master
then
spark-shell --master spark://ip_and_port_from_step_above
example
spark-shell --master spark://192.168.86.27:7077
Reference: Stack Overflow
Symptom: Receive error: Cannot find catalog plugin class for catalog "spark_catalog"
Resolution:
Copy delta-core_2.13-2.1.0 JAR file to $SPARK_HOME/lib and restart, this error goes away.
Symptom: Receive errror: Encountering "WARN ProcfsMetricsGetter: Exception when trying to compute pagesize" error when running Spark
Resolution
Adding PYTHONPATH environment variable with value as:
%SPARK_HOME%\python;%SPARK_HOME%\python\lib\py4j-<version>-src.zip;%PYTHONPATH%
Reference: Stack Overflow
Problem: Error: py4j.security.Py4JSecurityException: Constructor public com.databricks.backend.daemon.dbutils.FSUtilsParallel(org.apache.spark.SparkContext) is not whitelisted when connecting from Databricks to ADSL from Datascience Cluster
Reference: Databricks Community Q and A
Solution:
Update function setup_spark_configuration in environment_metadata class
spark.conf.set("spark.databricks.pyspark.enablePy4JSecurity", "false")
Problem: Table creation fails with security exception
Reference: Databricks KB
Solution:
GRANT SELECT ON ANY FILE TO `user1`
Warning
It is important to understand the security implications of granting ANY FILE permissions on a filesystem. You should only grant ANY FILE to privileged users. Users with lower privileges on the cluster should never access data by referencing an actual storage location. Instead, they should access data from tables that are created by privileged users, thus ensuring that Table ACLS are enforced.
In addition, if files in the Databricks root and data buckets are accessible by the cluster and users have MODIFY privileges, the admin should lock down the root.
Symptom: During Docker Build Receive ERROR: failed to solve: error getting credentials - err: docker-credential-desktop.exe resolves to executable in current directory (./docker-credential-desktop.exe), out:
Reference: Stack Overflow
Solution:
Run command:
code ~/.docker/config.json
Change credsStore to credStore
Save
Symptom: Receiving error: "Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? " when trying to run a docker build command
Reference: Stack Overflow
Solution:
Run command:
sudo dockerd
Symptom: Receive error: error during connect: This error may indicate that the docker daemon is not running.: when attempting to run docker image
Reference: Stack Overflow
Details:
Resolution with Docker Desktop:
Resolution with Powershell:
Resolution with CMD:
Symptom: Receive error: git push origin dev-zfi4:dev-zfi4 remote: warning: File pade/pade_ocio_ingress/ehr/encounters.csv is 87.59 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB remote: error: Trace: 2c8fbfed93d4f4c52468dde090cd8d14a1a311d13a6aa58ea35cb9cb8c5d2577 remote: error: See http://git.io/iEPt8g for more information. remote: error: File pade/pade_ocio_ingress/ehr/observations.csv is 296.35 MB; this exceeds GitHub's file size limit of 100.00 MB remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com. To https://github.com/cdcent/data-ecosystem-services.git ! [remote rejected] dev-zfi4 -> dev-zfi4 (pre-receive hook declined)
Reference: Stack Overflow
Solution:
git filter-branch --tree-filter 'rm -rf path/to/your/file' HEAD
git push
Symptom: Receive error: Support for password authentication was removed. Please use a personal access token instead. when trying to push to a remote repository
Reference: Stack Overflow
Solution:
For Ubuntu:
For Linux, you need to configure the local GIT client with a username and email address,
git config --global user.name "your_github_user_id"
git config --global user.email "your_github_email"
git config -l
Symptom: CICD or VS Code Tag give error "would clobber existing tag"
Solution:
Run Git
git fetch --tags -f
Then pull again.
Reference: Stack Overflow
Symptom: Receiving error: Git fatal: mmap failed: Invalid argument
Reference: Stack overflow
Solution:
Problem: "Java gateway process exited before sending its port number"
Resolution:
On All OS:
Check that the following environment variable is set where 1 is the number of processors
export PYSPARK_SUBMIT_ARGS="--master spark:192.168.86.27:7077"
or
export PYSPARK_SUBMIT_ARGS="--master local[1] pyspark-shell"
On Ubuntu: install openjdk-8-jdk package
sudo apt-get install openjdk-8-jdk-headless -qq
On MacOs: run:
java -version
Verify Resolution In Python:
from pyspark import SparkContext
sc = SparkContext.getOrCreate()
# check that it really works by running a job
# example from http://spark.apache.org/docs/latest/rdd-programming-guide.html#parallelized-collections
data = range(10000)
distData = sc.parallelize(data)
distData.filter(lambda x: not x&1).take(10)
OR
Reference: Stack Overflow
Symptom: Server access error at url Maven URL (javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target)
Resolution:
Run:
From cli change dir to jre\bin
keytool -import -alias repo1_maven_org -keystore C:\apps\jdk-8.0.342.07-hotspot\jre\lib\security\cacerts -file "C:\Users\zfi4\OneDrive - CDC\DAVT Analytics\davt-dev-jcbowyer\_templates\certs\repo1.maven.org.crt"
keytool -import -alias repo_spark_packages_org -keystore C:\apps\jdk-8.0.342.07-hotspot\jre\lib\security\cacerts -file "C:\Users\zfi4\OneDrive - CDC\DAVT Analytics\davt-dev-jcbowyer\_templates\certs\repos.spark-packages.org.crt"
keytool -list -keystore C:\apps\Java\jdk1.8.0_333\lib\security\cacerts
Reference: Stack Overflow Reference: Java Samples
The pop-up window for user name and password entry may close immediately when Cached Credentials expired.
Symptom: Receive Error: Failed to save logic app logic-sps-list-export-all-environments. The workflow must be associated with an integration account to use the workflow run action 'Execute_JavaScript_Code' of type 'JavaScriptCode'.
Reference: Stack Overflow
Resolution: To run the inline code action you need an integration account. These have some different pricing tiers. You only need to create it and then on the Logic App Workflow settings associate it.
Problem: Mermaid raises error Expecting 'NEWLINE', 'SPACE', 'GRAPH', got 'ALPHA' when it encounters bracket.
Reference: GitHub
The issue seems to be, that mermaid.cli ships with an old version of mermaid.min.js which doesn't support these kinds of node types/attributes.
Resolution:
Explicitly install mermaid and then copy dist/mermaid.min.js into the mermaid.cli folder within node_modules
Version is avaialble at unpkg.com-mermaid
Symptom: Receiving error: "PhantomJS not found on PATH" when running npm install
Reference: StackOverflow
Solution:
sudo npm uninstall phantomjs
sudo npm install phantomjs -g
sudo npm install phantomjs -s
Symptom: workon command not found
Reference: StackOverflow
Details:
The workon command is not available if you have restarted the shell.
If you want this to work with each shell, you'll need to add these to your ~/.bashrc file
Solution:
export PATH="$HOME/.local/bin:$PATH"
echo "export WORKON_HOME=$HOME/.virtualenvs" | sudo tee -a $HOME/.bashrc
echo "VIRTUALENVWRAPPER_PYTHON='/usr/lib/python3.9'" | sudo tee -a $HOME/.bashrc
echo "source $HOME/.local/bin/virtualenvwrapper.sh" | sudo tee -a $HOME/.bashrc
source $HOME/.bashrc
Symptom: Receiving error: "Error: Failed to launch the browser process! /usr/bin/chromium: error while loading shared libraries: libgbm.so.1: cannot open shared object file: No such file or directory"
Details: Puppeteer is a Node library which provides a high-level API to control headless Chrome or Chromium over the DevTools Protocol. It can also be configured to use full (non-headless) Chrome or Chromium. It is not launching when running mermaid-cli.
Reference: GitHub
Solution:
Add the lgm package to the Dockerfile/Setup process
sudo apt-get update
sudo apt-get install -y libgbm-dev
Symptom: Error: Python: Current Python version (3.8.10) is not allowed by the project (^3.9). Current Python version (3.8.10) is not allowed by the project (^3.9). Please change python executable via the "env use" command.
Reference: Stack Overflow
Details: Please change python executable via the "env use" command.
Solution:
which python3.9
poetry env use /usr/bin/python3.9
Symptom: Receiving Error when running npm link command: Error: EACCES: permission denied, mkdir '/usr/local/lib/node_modules/node-sass/build'
Reference: Stack Overflow
Details: EACCES: permission denied, mkdir '/usr/local/lib/node_modules/node-sass/build'
Solution:
Change the ownership of folder node_modules, because you use sudo npm install -g node-sass, so its ownership is set to root.
sudo chown -R root:$(whoami) /usr/local/lib/node_modules/
sudo chmod -R 775 /usr/local/lib/node_modules/
Symptom: Receiving error: "WARNING: dot command 'dot' cannot be run (needed for graphviz output), check the graphviz_dot setting" when building HTML Sphinx documentation
Reference: Stack Overflow Reference: Blog
Details: The problem could be referred to an incorrect configuration of "dot" executable from GraphViz PATH.
Solution:
Adding the PATH using the export command, such as:
export PATH=$PATH:~/opt/bin
Symptom: Ubuntu: InRelease is not valid yet (invalid for another 1d 21h 40min 43s). Updates for this repository will not be applied. occurs when running sudo apt-get update
Reference: StackOverflow
Details: Occurs when the system clock in Ubuntu is not Correct
Solution: Run the following command to fix the issue:
sudo date -s "$(wget -qSO- --max-redirect=0 google.com 2>&1 | grep Date: | cut -d' ' -f5-8)Z"
sudo apt update
Symptom: Receiving error: "System has not been booted with systemd as init system (PID 1). Can't operate. Failed to connect to bus: Host is down" when running : sudo systemctl daemon-reload
Reference: Github Gist
Solution: Run the following command to fix the issue:
sudo apt-get update && sudo apt-get install -yqq daemonize dbus-user-session fontconfig
sudo daemonize /usr/bin/unshare --fork --pid --mount-proc /lib/systemd/systemd --system-unit=basic.target
exec sudo nsenter -t $(pidof systemd) -a su - $LOGNAME
Symptom: Receiving error: "Release file for Ubuntu File is not valid yet" during apt-get update
Reference: How2Shout
Solution:
sudo dpkg-reconfigure tzdata
Symptom: Deploying a project to a local web server return error message: VS Code Ubuntu/Docker/Web Server: Failed to bind to address already in use
Reference: Stack Overflow
Solution:
Determine which ports are blocked
lsof -i: <port number>
example:
lsof -i: 5001
Kill the process that is blocking the port
kill -9 <process number>
example:
kill -9 1600
Symptom: Permission Denied when calling apt-get updatee from WSL
Details:
Run wsl command from ubuntu
Receive Error
Reading package lists... Done E: Could not open lock file /var/lib/apt/lists/lock - open (13: Permission denied) E: Unable to lock directory /var/lib/apt/lists/
Reference: AskUbuntu
This line says you are not authorized to install/update. You need to be root to do it....
Resolution with WSL:
Add sudo before apt-get or you can do sudo -s to be root.
sudo apt-get update
or
Run the following command to promote the current terminal to run as root until the terminal is closed.
sudo -s
apt-get update
Symptom: Receive error: "ImportError: No module named pip" right after installing pip
Reference: Stack Overflow
Details: /home/jcbow/.virtualenvs/OCIO_PADE_DEV/bin/python: No module named pip
Resolution:
First, ensure that python is included in the PATH variable, then run Then run the following
python -m ensurepip
Problem: When browsing https://127.0.0.1:5000 receive error: SSL: Handshake failed. while not logged into ZScaler
Solution:
Here are the steps you can follow to fix this error:
systemctl status upower.service
If the command output shows that the UPower service is not running or is not installed, you can install it using the package manager for your system (e.g. apt-get for Ubuntu/Debian, dnf for Fedora, pacman for Arch Linux, etc.).
sudo apt-get install upower
This command installs the upower package, which provides the UPower daemon.
sudo systemctl start upower.service
systemctl status upower.service
Note that the specific steps for installing and starting the UPower daemon may vary depending on your system and distribution. You may need to consult the documentation or package manager for your system to find the appropriate package and commands to install and start the UPower daemon.
Problem: When browsing https://127.0.0.1:5000 receive error: SSL: Handshake failed. when logged into ZScaler
[39102:39117:0503/091443.791015:ERROR:ssl_client_socket_impl.cc(992)] handshake failed; returned -1, SSL error code 1, net_error -202 [39102:39117:0503/091447.197078:ERROR:ssl_client_socket_impl.cc(992)] handshake failed; returned -1, SSL error code 1, net_error -202 [39102:39117:0503/091447.998238:ERROR:ssl_client_socket_impl.cc(992)] handshake failed; returned -1, SSL error code 1, net_error -202 [39062:39086:0503/091448.177119:ERROR:cert_issuer_source_aia.cc(134)] AiaRequest::OnFetchCompleted got error -301 [39062:39086:0503/091448.178211:ERROR:cert_issuer_source_aia.cc(134)] AiaRequest::OnFetchCompleted got error -301 [42678:42704:0503/093402.088689:ERROR:cert_verify_proc_builtin.cc(677)] CertVerifyProcBuiltin for clientservices.googleapis.com failed: ----- Certificate i=2 (CN=NCA-DPI1,OU=ITSO,O=Centers for Disease Control and Prevention,L=Atlanta,ST=Georgia,C=US) ----- ERROR: No matching issuer found
[39062:39086:0503/091448.178351:ERROR:cert_verify_proc_builtin.cc(677)] CertVerifyProcBuiltin for optimizationguide-pa.googleapis.com failed: ----- Certificate i=2 (CN=NCA-DPI1,OU=ITSO,O=Centers for Disease Control and Prevention,L=Atlanta,ST=Georgia,C=US) ----- ERROR: No matching issuer found
Solution:
python -c "import ssl; print(ssl.get_default_verify_paths())"
This should print out the paths to the CA bundle and the OpenSSL configuration file used by Python. If the paths are not found or are invalid, you may need to update your Python installation or install a valid CA bundle.
Path should return
/usr/lib/ssl/certs
cd $HOME
openssl s_client -showcerts -connect clientservices.googleapis.com:443 </dev/null 2>/dev/null | openssl x509 -outform PEM > clientservices.googleapis.com.crt
openssl s_client -showcerts -connect optimizationguide-pa.googleapis.com:443 </dev/null 2>/dev/null | openssl x509 -outform PEM > optimizationguide-pa.googleapis.com.crt
openssl s_client -showcerts -connect safebrowsing.googleapis.com.com:443 </dev/null 2>/dev/null | openssl x509 -outform PEM > safebrowsing.googleapis.com.crt
openssl s_client -showcerts -connect accounts.google.com:443 </dev/null 2>/dev/null | openssl x509 -outform PEM > accounts.google.com.crt
openssl s_client -showcerts -connect update.googleapis.com:443 </dev/null 2>/dev/null | openssl x509 -outform PEM > update.googleapis.com.crt
openssl s_client -showcerts -connect www.google.com:443 </dev/null 2>/dev/null | openssl x509 -outform PEM > www.google.com.crt
dns.google
sudo cp clientservices.googleapis.com.crt /usr/local/share/ca-certificates/
sudo cp optimizationguide-pa.googleapis.com.crt /usr/local/share/ca-certificates/
sudo cp safebrowsing.googleapis.com.crt /usr/local/share/ca-certificates/
sudo cp accounts.google.com.crt /usr/local/share/ca-certificates/
sudo cp update.googleapis.com.crt /usr/local/share/ca-certificates/
sudo cp www.google.com.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
Symptom: Receive error: curl: (60) SSL certificate problem: unable to get local issuer certificate when installing docker
Reference: Stack Overflow
Details:
The problem may be related to the way the firewall is handling certificates. The certificate of the firewall may be untrusted/unknown from within the wsl environment.
Resolution:
Option 1:
Option 2:
Export the firewall certificate.
Configure WSL.
open wsl terminal from VS Code : should default to Ubuntu: directory "/Users/zfi4/OneDrive - CDC/DAVT Analytics/davt-dev-jcbowyer"
wsl
openssl x509 -inform DER -in _templates/certs/zscaler.cer -out ./zscaler.crt
sudo cp zscaler.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
Problem: Python installation error: ImportError: No module named apt_pkg occurs when running sudo apt-get update
Reference: Stack Overflow
Solution:
```sh
sudo apt remove python3-apt
sudo apt autoremove
sudo apt autoclean
sudo apt install python3-apt
```
Problem: NPM can't find module "semver" error in Ubuntu 19.04 when installing nodejs upgrade
Reference: AskUbuntu
Solution:
Run the following commands to uninstall node:
cd $HOME
sudo rm -rf /usr/local/bin/npm /usr/local/share/man/man1/node* ~/.npm
sudo rm -rf /usr/local/lib/node*
sudo rm -rf /usr/local/bin/node*
sudo rm -rf /usr/local/include/node*
sudo apt-get purge nodejs npm
sudo apt autoremove
Run the following commands to reinstall node:
workon OCIO_PADE_DEV
cd $VIRTUAL_ENV
rm -rf /usr/local/bin/npm /usr/local/share/man/man1/node* ~/.npm
rm -rf /usr/local/lib/node*
rm -rf /usr/local/bin/node*
rm -rf /usr/local/include/node*
# install node with nvm - node version manager
# wget https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh
# chmod +x install.sh
# ./install.sh
# source ~/.bashrc
# nvm list-remote
# install node with wget
wget https://nodejs.org/dist/v18.12.1/node-v18.12.1-linux-x64.tar.xz
tar -xf node-v18.12.1-linux-x64.tar.xz
rsync -av node-v18.12.1-linux-x64/bin/* ./bin
rsync -av node-v18.12.1-linux-x64/lib/node_modules/ ./lib/node_modules/
cd $VIRTUAL_ENV/bin
hash -r
# Verify installation using
node -v
npm -v
npm install puppeteer
npm install npm@9.1.2
npm install @mermaid-js/mermaid-cli
sudo apt-get update
sudo apt-get install -y libgbm-dev
Symptom: Receiving error: "End of line character is invalid"
Reference: Boot.dev Blog
Solution for One File:
Unless you work on a Windows-only team, the answer is almost always to change all your code to the Unix default of LF.
Soulution for All New Files:
For Workspace Tab
For User Tab
Solution for All Existing Files
Symptom: Trying to clone a github repository and receive error: "fatal: unable to access Could not resolve host: github.com "
Reference: Stack Overflow
Resolution:
Option 1:
Option 2:
git config --global --unset https.proxy
Option 3:
Option 4:
Option 5:
Fix wrong/empty /etc/resolv.conf file.
To view contents of /etc/resolv.conf file:
code /etc/resolv.conf
To fix:
sudo rm /etc/resolv.conf
sudo bash -c 'echo "nameserver 8.8.8.8" > /etc/resolv.conf'
sudo bash -c 'echo "[network]" > /etc/wsl.conf'
sudo bash -c 'echo "generateResolvConf = false" >> /etc/wsl.conf'
sudo chattr +i /etc/resolv.conf
Symptom: Trying to connect to WSL from Windows Terminal and receive error: "Logon failure: the user has not been granted the requested logon type at this computer."
Reference: Stack Overflow
Solution:
Get-Service vmcompute | Restart-Service
Symptom: WSL windows keeps exiting unexpectedly
Reference: SuperUser
Details:
Resolution with Powershell or CMD:
wsl --set-default Ubuntu-20.04
FAQs
Program Agnostic Data Ecosystem (PADE) - Python Services
We found that data-ecosystem-services demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
In this segment of the Risky Business podcast, Feross Aboukhadijeh and Patrick Gray discuss the challenges of tracking malware discovered in open source softare.
Research
Security News
A threat actor's playbook for exploiting the npm ecosystem was exposed on the dark web, detailing how to build a blockchain-powered botnet.
Security News
NVD’s backlog surpasses 20,000 CVEs as analysis slows and NIST announces new system updates to address ongoing delays.