Product
Introducing License Enforcement in Socket
Ensure open-source compliance with Socket’s License Enforcement Beta. Set up your License Policy and secure your software!
.. image:: https://raw.githubusercontent.com/chris-mcdo/obscraper/main/docs/source/img/logo.svg :target: https://obscraper.readthedocs.io/en/stable/ :align: center :height: 200 :alt: obscraper
.. image:: https://img.shields.io/pypi/v/obscraper.svg :target: https://pypi.org/project/obscraper/ :alt: Project Version on PyPI
.. image:: https://img.shields.io/pypi/pyversions/obscraper.svg :target: https://pypi.org/project/obscraper/ :alt: Supported Python Versions
.. image:: https://readthedocs.org/projects/obscraper/badge/?version=latest :target: https://obscraper.readthedocs.io/en/latest/?badge=latest :alt: Documentation Status
.. image:: https://codecov.io/gh/chris-mcdo/obscraper/branch/main/graph/badge.svg :target: https://codecov.io/gh/chris-mcdo/obscraper :alt: Unit Test Coverage
.. image:: https://img.shields.io/badge/code%20style-black-000000.svg :target: https://github.com/psf/black :alt: Code Style: Black
.. image:: https://img.shields.io/badge/license-MIT-purple :target: https://github.com/chris-mcdo/obscraper/blob/main/LICENSE :alt: MIT License
obscraper
lets you scrape blog posts and associated metadata from the
overcomingbias <https://www.overcomingbias.com/>
_ blog.
It's easy to get a single post::
>>> import obscraper
>>> intro_url = 'https://www.overcomingbias.com/2006/11/introduction.html'
>>> post = obscraper.get_post_by_url(intro_url)
>>> post.title
'How To Join'
>>> post.plaintext
'How can we better believe what is true? ...'
>>> post.internal_links
{'http://www.overcomingbias.com/2007/02/moderate_modera.html': 1,
'http://www.overcomingbias.com/2006/12/contributors_be.html': 1}
>>> post.comments
20
Or a full list of post URLs and edit dates::
>>> import obscraper
>>> edit_dates = obscraper.get_edit_dates()
...
>>> len(edit_dates)
4352
>>> {url: str(edit_dates[url]) for url in list(edit_dates)[:5]}
{'2022/01/much-talk-is-sales-patter':
'2022-01-14 20:46:35+00:00',
'2022/01/old-man-rant':
'2022-01-13 15:21:33+00:00',
'2022/01/my-11-bets-at-10-1-odds-on-10m-covid-deaths-by-2022':
'2022-01-12 19:15:10+00:00',
'2022/01/to-innovate-unify-or-fragment':
'2022-01-11 01:03:44+00:00',
'2022/01/on-what-is-advice-useful':
'2022-01-10 18:46:26+00:00'}
Features
Get posts by their URLs or edit dates, or get all posts hosted on the overcomingbias site
Provides detailed post metadata including post URLs, titles, authors, tags, publish dates, and last edit dates
Provides summary of post content including full post text as HTML or plaintext, and a list of hyperlinks to other overcomingbias posts
Asynchronous execution and caching for fast downloads
Use via import obscraper
or the simple command line interface
Comprehensively tested
Supports python 3.8+
Documentation
Read the full documentation here <https://obscraper.readthedocs.io/en/stable/>
,
including the Installation and Getting Started Guide <https://obscraper.readthedocs.io/en/stable/getting-started.html>
and the
Public API Reference <https://obscraper.readthedocs.io/en/stable/api.html>
_.
Bugs/Requests
Please use the GitHub issue tracker <https://github.com/chris-mcdo/obscraper/issues>
_
to submit bugs or request features.
Changelog
See the Changelog <https://obscraper.readthedocs.io/en/stable/changelog.html>
_
for a list of fixes and enhancements at each version.
License
Copyright (c) 2022 Christopher McDonald
Distributed under the terms of the
MIT <https://github.com/chris-mcdo/obscraper/blob/main/LICENSE>
_ license.
All overcomingbias posts are copyright the original authors.
FAQs
obscraper: scrape posts from the overcomingbias blog
We found that obscraper demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Product
Ensure open-source compliance with Socket’s License Enforcement Beta. Set up your License Policy and secure your software!
Product
We're launching a new set of license analysis and compliance features for analyzing, managing, and complying with licenses across a range of supported languages and ecosystems.
Product
We're excited to introduce Socket Optimize, a powerful CLI command to secure open source dependencies with tested, optimized package overrides.