Research
Recent Trends in Malicious Packages Targeting Discord
The Socket research team breaks down a sampling of malicious packages that download and execute files, among other suspicious behaviors, targeting the popular Discord platform.
viu-sitemap-parser
Readme
Parse XML/Plaintext/RSS Sitemaps
Intent for creating that package
We've searched for a python based Sitemap parser which supports all main types of sitemap formats. It should also be as flexible as possible by declaring sitemaps etc. We haven't found one that fits our requirements, so we created this one.
It's not a big project - so we try to leave the code as simple as possible.
The result contains all information we found (e.g. HREF Lang etc.) They are not normalized and for each sitemap format different. See examples.
Installation
pip install VIU-Sitemap-parser
Usage - single files
from viusitemapparser.vsp import get_sitemap_contents
sitemap, sitemap_entries = get_sitemap_contents('https://run.mocky.io/v3/d8cb86d5-0400-4860-a053-78996f9ee6b2')
Add Headers (optional):
sitemap, sitemap_entries = get_sitemap_contents('https://run.mocky.io/v3/d8cb86d5-0400-4860-a053-78996f9ee6b2', headers={'User-Agent': 'Mozilla/5.0'})
Variable "sitemap" contains a dict with information about the sitemap itself - e.g.
{'file_name': 'https://run.mocky.io/v3/d8cb86d5-0400-4860-a053-78996f9ee6b2',
'file_headers': {'Content-Type': 'text/xml; charset=UTF-8', 'Date': 'Tue, 27 Jul 2021 15:05:03 GMT', 'Content-Length': '939', 'Sozu-Id': '01FBM7YVDJEWJP4670MWBRFXBW'},
'source_type': 'remote', # remote or local
'file_error': None, # If there was an error getting the file (e.g. 404 you will get the information about it here)
'file_received': True, # Final boolean if we get an file to check
'sitemap_type': 'xml_sitemap' # one of xml_sitemap, xml_sitemap_index, rss_feed, atom_feed, plain_text,
# invalid_file_format
}
"sitemap_entries" is an iterator over the lines in the sitemap file. The content of the lines is different by sitemap_type.
Here is an example if you just want the URLs out of the Sitemap files. The names are equal to the names in the Sitemap file.
for entry in sitemap_entries:
if sitemap.get("sitemap_type") == 'xml_sitemap':
print(entry.get('loc'))
elif sitemap.get("sitemap_type") == 'xml_sitemap_index':
print(entry.get('loc'))
elif sitemap.get("sitemap_type") == 'rss_feed':
print(entry.get('link'))
elif sitemap.get("sitemap_type") == 'atom_feed':
print(entry.get('link'))
elif sitemap.get("sitemap_type") == 'plain_text':
print(entry)
Take a look on https://github.com/VIU-one/VIU-Sitemap-Parser/blob/main/tests/test_sitemap_entries.py to see how the response of different formats looks.
Usage - with iterator over sitemap_index files
TODO
Alternatives
https://github.com/mediacloud/ultimate-sitemap-parser
More about the Sitemaps format
https://developers.google.com/search/docs/advanced/sitemaps/build-sitemap?hl=de
https://developers.google.com/search/blog/2014/10/best-practices-for-xml-sitemaps-rssatom
Video in Sitemaps: https://developers.google.com/search/docs/advanced/sitemaps/video-sitemaps
Images in Sitemaps: https://developers.google.com/search/docs/advanced/sitemaps/image-sitemaps
HREF Lang in Sitemaps: https://developers.google.com/search/docs/advanced/crawling/localized-versions#sitemap
Google News Sitemaps: https://developers.google.com/search/docs/advanced/sitemaps/news-sitemap?hl=de
FAQs
Unknown package
We found that viu-sitemap-parser demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
The Socket research team breaks down a sampling of malicious packages that download and execute files, among other suspicious behaviors, targeting the popular Discord platform.
Security News
Socket CEO Feross Aboukhadijeh joins a16z partners to discuss how modern, sophisticated supply chain attacks require AI-driven defenses and explore the challenges and solutions in leveraging AI for threat detection early in the development life cycle.
Security News
NIST's new AI Risk Management Framework aims to enhance the security and reliability of generative AI systems and address the unique challenges of malicious AI exploits.