Research
Recent Trends in Malicious Packages Targeting Discord
The Socket research team breaks down a sampling of malicious packages that download and execute files, among other suspicious behaviors, targeting the popular Discord platform.
Readme
Sitemaps is a Python command line tool and library to generate sitemap files by crawling web servers or ASGI apps. Sitemaps is powered by HTTPX and anyio.
Note: This is alpha software. Be sure to pin your dependencies to the latest minor release.
python -m sitemaps https://example.org
Example output:
$ cat sitemap.xml
<?xml version="1.0" encoding="utf-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd">
<url><loc>https://example.org/</loc><changefreq>daily</changefreq></url>
</urlset>
HTTP requests are issued to the ASGI app directly. The target URL is only used as a base URL for building sitemap entries.
python -m sitemaps --asgi '<module>:<attribute>' http://testserver
Useful to verify that the sitemap is in sync (e.g. as part of CI checks):
python -m sitemaps --check [...]
--check
mode.Install with pip:
$ pip install sitemaps
Sitemaps requires Python 3.7+.
$ python -m sitemaps --help
usage: __main__.py [-h] [-o OUTPUT] [-I IGNORE_PATH_PREFIX] [--asgi ASGI]
[--max-concurrency MAX_CONCURRENCY] [--check]
target
positional arguments:
target The base URL used to crawl the website and build
sitemap URL tags.
optional arguments:
-h, --help show this help message and exit
-o OUTPUT, --output OUTPUT
Output file path.
-I IGNORE_PATH_PREFIX, --ignore-path-prefix IGNORE_PATH_PREFIX
Ignore URLs for this path prefix. Can be used multiple
times.
--asgi ASGI Path to an ASGI app, formatted as
'<module>:<attribute>'.
--max-concurrency MAX_CONCURRENCY
Maximum number of URLs to process concurrently.
--check Compare existing output and fail if computed XML
differs.
import sitemaps
async def main():
urls = await sitemaps.crawl("https://example.org")
with open("sitemap.xml", "w") as f:
f.write(sitemaps.make_xml(urls))
import httpx
import sitemaps
from .app import app
async def main():
async with httpx.AsyncClient(app=app) as client:
urls = await sitemaps.crawl("http://testserver", client=client)
with open("sitemap.xml", "w") as f:
f.write(sitemaps.make_xml(urls))
By default, .make_xml()
generates <url>
tags with a daily
change frequency. You can customize the generation of URL tags by passing a custom urltag
callable:
from urllib.parse import urlsplit
def urltag(url):
path = urlsplit(url).path
changefreq = "monthly" if path.startswith("/reports") else "daily"
return f"<url><loc>{url}</loc><changefreq>{changefreq}</changefreq></url>"
async def main():
urls = await sitemaps.crawl(...)
with open("sitemap.xml", "w") as f:
f.write(sitemaps.make_xml(urls, urltag=urltag))
MIT
All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog.
FAQs
Sitemap generation for Python, with support for crawling ASGI web apps directly.
We found that sitemaps demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
The Socket research team breaks down a sampling of malicious packages that download and execute files, among other suspicious behaviors, targeting the popular Discord platform.
Security News
Socket CEO Feross Aboukhadijeh joins a16z partners to discuss how modern, sophisticated supply chain attacks require AI-driven defenses and explore the challenges and solutions in leveraging AI for threat detection early in the development life cycle.
Security News
NIST's new AI Risk Management Framework aims to enhance the security and reliability of generative AI systems and address the unique challenges of malicious AI exploits.