Remote, in-memory Python package/module importing through HTTP/S
A feature that Python misses and has become popular in other languages is the remote loading of packages/modules.
httpimport lets Python packages and modules to be installed and imported directly in Python interpreter's process memory, through remote URIs, and more...
1.1.0.with httpimport.remote_repo('http://my-codes.example.com/python_packages'):
import package1
with httpimport.pypi_repo():
import distlib # https://pypi.org/project/distlib/
print(distlib.__version__)
# '0.3.6' <-- currently latest (https://github.com/pypa/distlib/blob/0.3.6/distlib/__init__.py#L9)
with httpimport.github_repo('operatorequals', 'httpimport', ref='master'):
import httpimport as httpimport_upstream
# Also works with 'bitbucket_repo' and 'gitlab_repo'
url = "https://gist.githubusercontent.com/operatorequals/ee5049677e7bbc97af2941d1d3f04ace/raw/e55fa867d3fb350f70b2897bb415f410027dd7e4"
with httpimport.remote_repo(url):
import hello
hello.hello()
# Hello world
# From HTTP/S URL
http_module = httpimport.load('package1', 'https://my-codes.example.com/python_packages')
print(http_module)
<module 'package1' from 'https://my-codes.example.com/python_packages/package1/__init__.py'>
# From PyPI
pypi_module = httpimport.load('distlib', importer_class=httpimport.PyPIImporter)
print(pypi_module)
<module 'distlib' from 'https://files.pythonhosted.org/packages/76/cb/6bbd2b10170ed991cf64e8c8b85e01f2fb38f95d1bc77617569e0b0b26ac/distlib-0.3.6-py2.py3-none-any.whl#distlib/__init__.py'>
No file is touching the disk in the process
# with httpimport.remote_repo('https://example.com/packages.tar'):
# with httpimport.remote_repo('https://example.com/packages.tar.bz2'):
# with httpimport.remote_repo('https://example.com/packages.tar.gz'):
# with httpimport.remote_repo('https://example.com/packages.tar.xz'):
with httpimport.remote_repo('https://example.com/packages.zip'):
import test_package
Any package can be served for httpimport using a simple HTTP/S Server:
echo 'print("Hello httpimport!")' > module.py
python -m http.server
Serving HTTP on 0.0.0.0 port 8000 ...
>>> import httpimport
>>> with httpimport.remote_repo("http://127.0.0.1:8000"):
... import module
...
Hello httpimport!
After v1.0.0 it is possible to set HTTP Authentication, Custom Headers, Proxies and several other things using URL and Named Profiles!
URL Profiles are INI configurations, setting specific per-URL options, as below:
[http://127.0.0.1:8000]
allow-plaintext: yes ; also 'true' and '1' evaluate to True
[https://example.com]
proxy-url: https://127.0.0.1:8080 ; values must not be in quotes (')
Now, requests to http://127.0.0.1:8000 will be allowed (HTTP URLs do not work by default) and requests to https://example.com will be sent to an HTTP Proxy.
with httpimport.remote_repo("https://example.com"): # URL matches the URL profile
import module_accessed_through_proxy
Named Profiles are like URL profiles but do not specify a URL and need to be explicitly used:
[github]
headers:
Authorization: token <Github-Token>
And the above can be used as follows:
with httpimport.github_repo('operatorequals','httpimport-private-test', profile='github'):
import secret_module
github_pat_<gibberish> and can be issued here: https://github.com/settings/tokens/newWhen importing from PyPI extra options can be used, as described in the profile below:
[pypi]
# The location of a 'requirements.txt' file
# to use for PyPI project versions
requirements-file: requirements-dev.txt
# Inline 'requirements.txt' syntax appended
requirements:
distlib==0.3.5
sampleproject==3.0.0
# Only version pinning notation is supported ('==')
# with 'requirements' and 'requirements-file' options
# A map that contains 'module': 'PyPI project' tuples
# i.e: 'import sample' --> search 'sample' module at 'sampleproject' PyPI Project:
# https://pypi.org/project/sampleproject/
project-names:
sample: sampleproject
The PyPI Profiles can be used exactly like all Named Profiles:
with httpimport.pypi_repo(profile='pypi'):
import distlib
import sample
distlib.__version__
# '0.3.5' <-- pinned in the profile 'requirements' option
sample.__url__
# 'https://files.pythonhosted.org/packages/ec/a8/5ec62d18adde798d33a170e7f72930357aa69a60839194c93eb0fb05e59c/sampleproject-3.0.0-py3-none-any.whl#sample/__init__.py' <-- loaded from 'sampleproject'
Additionally, all other options cascade to PyPI profiles, such as HTTPS Proxy (HTTP proxies won't work, as PyPI is hosted with HTTPS), headers, etc.
',")Profiles can be provided as INI strings to the set_profile function and used in all httpimport functions:
httpimport.set_profile("""
[profile1]
proxy-url: https://my-proxy.example.com
headers:
Authorization: Basic ...
X-Hello-From: httpimport
X-Some-Other: HTTP header
""")
with httpimport.remote_repo("https://code.example.com", profile='profile1'):
import module_accessed_through_proxy
Profiles are INI configuration strings parsed using Python configparser module.
The ConfigParser object for httpimport is the global variable httpimport.CONFIG and can be used freely:
import httpimport
httpimport.CONFIG.read('github.ini') # Read profiles from a file
with httpimport.github_repo('operatorequals','httpimport-private-test', profile='github'):
import secret_module
The httpimport module automatically loads Profiles found in $HOME/.httpimport.ini and under the $HOME/.httpimport/ directory. Profiles under $HOME/.httpimport/ override ones found in $HOME/.httpimport.ini.
HTTP options
zip-password - v1.0.0proxy-url - v1.0.0headers - v1.0.0allow-plaintext - v1.0.0ca-verify - v1.3.0ca-file - v1.3.0PyPI-only options
project-names - v1.2.0requirements - v1.2.0requirements-file - v1.2.0allow-compiled
auth
auth-type
tls-cert
tls-key
tls-passphrase
import httpimport
import logging
logging.getLogger('httpimport').setLevel(logging.DEBUG)
Using the httpimport with plain HTTP URLs is highly discouraged
As HTTP traffic can be read and changed from all intermediate hosts (unlike HTTPS), it is possible for a remote adversary to alter the HTTP responses consumed by httpimport and add arbitrary Python code to the downloaded packages/modules. This directly results in arbitrary Remote Code Execution on your current user's context of your host!
In other words, using plain HTTP through the Internet can compromise your host without a way to notice it.
httpimport!RELOAD flag and Bug Fixesload() functionimp module in Python3 in favour of importlibIn case my work helped you, you can always buy me a beer or a liter of gas through the Internet or in case you meet me personally.
In the second case we can talk about any video of Internet Historian or Ordinary Things, while listening to a Lofi Girl Playlist, like the citizens of the Internet that we are.
FAQs
Module for remote in-memory Python package/module loading through HTTP
We found that httpimport demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.
Security News
The Ultralytics' PyPI Package was compromised four times in one weekend through GitHub Actions cache poisoning and failure to rotate previously compromised API tokens.