Product
Introducing License Enforcement in Socket
Ensure open-source compliance with Socket’s License Enforcement Beta. Set up your License Policy and secure your software!
django-sitemap-generate
Advanced tools
Background sitemap generation for Django.
Almost every content site has a sitemap. Django provides an application serving sitemap views, and it's OK if your website is small. If you have complicate logic in sitemap generation or if you have millions of items in sitemap - you'll have a massive load spikes when Google and another search engines come with thousands of there indexer bots. These bots will request same sitemap pages in parallel and those requests couldn't be cached because of large index interval and small hit rate.
The solution is to re-generate sitemap files periodically, once per day and not once per search engine indexer. These files could be served as static files which will not affect backend performance at all.
These project uses index sitemap view and per-model sitemap views to generate sitemap xml files. To provide it you will need following.
Add django.contrib.sitemaps
to installed apps
INSTALLED_APPS.append('django.contrib.sitemaps')
Configure at least one sitemap
from django.contrib.sitemaps import Sitemap
from testproject.testapp import models
class VideoSitemap(Sitemap):
name = 'video'
changefreq = 'daily'
limit = 50000
def items(self):
return models.Video.objects.order_by('id')
Note that changefreq
parameter is a hint for search engine indexer, it
does not affect sitemap files generation period.
Configure sitemap serving
from django.contrib.sitemaps import views
from django.urls import path
from testproject.testapp.sitemaps import VideoSitemap, ArticleSitemap
sitemaps = {
VideoSitemap.name: VideoSitemap,
ArticleSitemap.name: ArticleSitemap
}
urlpatterns = [
path('sitemap.xml', views.index, {'sitemaps': sitemaps},
name='sitemap-index'),
path('sitemap-<section>.xml', views.sitemap, {'sitemaps': sitemaps},
name='django.contrib.sitemaps.views.sitemap'),
]
Now your website supports sitemap views.
pip install django-sitemap-generate
Working example is in testproject.testapp
.
Add sitemap_generate
application to installed apps in django settings:
INSTALLED_APPS.append('sitemap_generate')
Add a reference to sitemap mapping to django settings:
SITEMAP_MAPPING = 'testproject.testapp.urls.sitemaps'
Specify name of the sitemap index url. If you have urlpatterns
like
example above, you can write:
SITEMAP_INDEX_NAME = 'sitemap-index'
default: 'sitemap-index'
Specify name of the view.sitemap
view. If you have urlpatterns
like
example above, you can write:
SITEMAPS_VIEW_NAME = 'django.contrib.sitemaps.views.sitemap'
default: 'django.contrib.sitemaps.views.sitemap'
Set media path to store sitemaps under
SITEMAP_MEDIA_PATH = 'sitemaps'
default: 'sitemaps'
Also you may need to setup forwarded protocol handling in django settings:
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
Note that django paginates sitemap with p
query parameter, but
corresponding sitemap files are named sitemap-video.xml
,
sitemap-video-2.xml
and so on. You'll need to configure some "rewrites".
Optional. Change storage for generated sitemaps
SITEMAP_STORAGE = custom_storage
default: django.core.files.storage.default_storage
When you request sitemap over http, django substitutes website domain name from
request to links in sitemap xml. In background, you'll need some environment
variables. By defaults link are generated for localhost
over HTTPS.
export \
SITEMAP_PROTO=https \
SITEMAP_HOST=github.com \
SITEMAP_PORT=443
# generate all sitemaps
python manage.py generate_sitemap
# generate sitemap for single model
python manage.py generate_sitemap video
You may run sitemap generation from crontab:
0 0 * * * python manage.py generate_sitemap
You may run sitemap generation from celery:
@celery.task
def generate_sitemap():
generator = SitemapGenerator() # Uses django settings by default
generator.generate()
And you will need to configure xml files static responses, i.e. in nginx:
location ~* /sitemaps/(?<fn>sitemap(-(article|video)).xml {
try_files /media/sitemaps/$fn$arg_p.xml @backend;
}
location /media/ {
alias /app/media/;
}
location @backend {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
set $app http://app:8000;
proxy_pass $app;
}
FAQs
Background sitemap generation for Django
We found that django-sitemap-generate demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Product
Ensure open-source compliance with Socket’s License Enforcement Beta. Set up your License Policy and secure your software!
Product
We're launching a new set of license analysis and compliance features for analyzing, managing, and complying with licenses across a range of supported languages and ecosystems.
Product
We're excited to introduce Socket Optimize, a powerful CLI command to secure open source dependencies with tested, optimized package overrides.