
Security News
Deno 2.2 Improves Dependency Management and Expands Node.js Compatibility
Deno 2.2 enhances Node.js compatibility, improves dependency management, adds OpenTelemetry support, and expands linting and task automation for developers.
tedfulk-kb-pycrawler
Advanced tools
This script utilizes the Playwright library for scraping websites and generating a knowledgebase. It is capable of outputting the scraped data in multiple formats, including JSON, plain text, and PDF.
scrape_it(config: Config)
config
: An object containing URLs to scrape, output file names, file types, and a limit on pages to crawl.Config
object:
sitemap.xml
.NoSitemapError
if not found, else extract URLs.max_pages_to_crawl
is reached.WebPage
object with URL and content.Knowledgebase
object.write_to_file(knowledgebase: Knowledgebase, config: Config)
to generate outputs.config
(JSON, TXT, PDF).write_to_file(knowledgebase: Knowledgebase, config: Config)
Writes the scraped data from Knowledgebase
to files in the specified formats.
Knowledgebase
object in a JSON file.get_output_filename(file_name: str, file_type: str)
to ensure unique file names for each output file.string_to_pdf(knowledgebase: Knowledgebase, file_name)
Converts the content of webpages in a Knowledgebase
to a PDF file.
NoSitemapError
if a sitemap.xml file is not found for a given URL.poetry add tedfulk-kb-pycrawler
playwright install
import asyncio
from tedfulk_kb_pycrawler import scrape_it, Config
config = Config(
urls=[
"https://jxnl.github.io/instructor/",
],
output_file_name=["instructor"],
output_file_type=["pdf"],
max_pages_to_crawl=100,
)
asyncio.run(scrape_it(config))
Activate your virtual environment:
poetry shell
Run the script:
python main.py
FAQs
Unknown package
We found that tedfulk-kb-pycrawler demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Deno 2.2 enhances Node.js compatibility, improves dependency management, adds OpenTelemetry support, and expands linting and task automation for developers.
Security News
React's CRA deprecation announcement sparked community criticism over framework recommendations, leading to quick updates acknowledging build tools like Vite as valid alternatives.
Security News
Ransomware payment rates hit an all-time low in 2024 as law enforcement crackdowns, stronger defenses, and shifting policies make attacks riskier and less profitable.