viu-sitemap-parser

VIU-Sitemap-Parser

Parse XML/Plaintext/RSS Sitemaps

Intent for creating that package

We've searched for a python based Sitemap parser which supports all main types of sitemap formats. It should also be as flexible as possible by declaring sitemaps etc. We haven't found one that fits our requirements, so we created this one.

It's not a big project - so we try to leave the code as simple as possible.

The result contains all information we found (e.g. HREF Lang etc.) They are not normalized and for each sitemap format different. See examples.

Features

• Supports XML sitemaps, sitemap index files, RSS feeds, Atom feeds, and plain text sitemaps
• Automatically detects and decompresses gzipped sitemaps (.gz files)
• Handles both local files and remote URLs
• Preserves all metadata from sitemap entries

Installation

pip install VIU-Sitemap-parser

Usage - single files

from viusitemapparser.vsp import get_sitemap_contents

sitemap, sitemap_entries = get_sitemap_contents('https://run.mocky.io/v3/d8cb86d5-0400-4860-a053-78996f9ee6b2')

Add Headers (optional):
sitemap, sitemap_entries = get_sitemap_contents('https://run.mocky.io/v3/d8cb86d5-0400-4860-a053-78996f9ee6b2', headers={'User-Agent': 'Mozilla/5.0'})

# Gzipped sitemaps are automatically supported
sitemap, sitemap_entries = get_sitemap_contents('https://example.com/sitemap.xml.gz')
sitemap, sitemap_entries = get_sitemap_contents('local_sitemap.xml.gz')

Variable "sitemap" contains a dict with information about the sitemap itself - e.g.

{'file_name': 'https://run.mocky.io/v3/d8cb86d5-0400-4860-a053-78996f9ee6b2', 
'file_headers': {'Content-Type': 'text/xml; charset=UTF-8', 'Date': 'Tue, 27 Jul 2021 15:05:03 GMT', 'Content-Length': '939', 'Sozu-Id': '01FBM7YVDJEWJP4670MWBRFXBW'}, 
'source_type': 'remote',        # remote or local 
'file_error': None,             # If there was an error getting the file (e.g. 404 you will get the information about it here)
'file_received': True,          # Final boolean if we get an file to check
'sitemap_type': 'xml_sitemap'   # one of xml_sitemap, xml_sitemap_index, rss_feed, atom_feed, plain_text,
                                # invalid_file_format 
}

"sitemap_entries" is an iterator over the lines in the sitemap file. The content of the lines is different by sitemap_type.

Here is an example if you just want the URLs out of the Sitemap files. The names are equal to the names in the Sitemap file.

for entry in sitemap_entries:
    if sitemap.get("sitemap_type") == 'xml_sitemap':
        print(entry.get('loc'))
    elif sitemap.get("sitemap_type") == 'xml_sitemap_index':
        print(entry.get('loc'))
    elif sitemap.get("sitemap_type") == 'rss_feed':
        print(entry.get('link'))
    elif sitemap.get("sitemap_type") == 'atom_feed':
        print(entry.get('link'))
    elif sitemap.get("sitemap_type") == 'plain_text':
        print(entry)

Take a look on https://github.com/VIU-one/VIU-Sitemap-Parser/blob/main/tests/test_sitemap_entries.py to see how the response of different formats looks.

Usage - with iterator over sitemap_index files

TODO

Alternatives

https://github.com/mediacloud/ultimate-sitemap-parser

More about the Sitemaps format

https://www.sitemaps.org/

https://developers.google.com/search/docs/advanced/sitemaps/build-sitemap?hl=de

https://developers.google.com/search/blog/2014/10/best-practices-for-xml-sitemaps-rssatom

Video in Sitemaps: https://developers.google.com/search/docs/advanced/sitemaps/video-sitemaps

Images in Sitemaps: https://developers.google.com/search/docs/advanced/sitemaps/image-sitemaps

HREF Lang in Sitemaps: https://developers.google.com/search/docs/advanced/crawling/localized-versions#sitemap

Google News Sitemaps: https://developers.google.com/search/docs/advanced/sitemaps/news-sitemap?hl=de