crawlabpy

Crawlab implements with file uploader.

0.0.8
PyPI

Maintainers: 1

Crawlab Python SDK

基本用法

from crawlabpy.utils import notify_target, save_file, PART_CONTENT_TYPE

img = requests.get(url=img_url).content
img_name = hashlib.md5(img).hexdigest() + '.jpg'
save_file(img_name, img)

result = {
    'body': body,
    'body_cn': '',
    'body_en': body,
    'covering': img_name_list,
    'downdate': down_date,  # 爬虫时间
    'id': news_id,
    'languageid': '10001',
    'languagename': '英语',
    'pagedate': down_date,
    'partid': '3',  # 板块号
    'partname': '对外关系,国际关系',  # 板块名
    'resource': img_name_list,
    'sectionid': '',
    'sectionname': '缅甸',
    'siteid': '153',
    'sitename': '缅甸新闻网',
    'title': news_title[0],
    'title_cn': '',
    'title_en': news_title[0],
    'url': news_url,
    'viewcount': '',
    'writer': ''
}
save_item(result)
notify_target(PART_CONTENT_TYPE, result, img_name_list)

开始开发

开发环境初始化

git clone https://github.com/gditsec/crawlabpy.git

cd crawlerpy

make dev

source venv/bin/activate

python -m pip install -r requirements.txt

构建代码

make build

发布代码

make release

FAQs

What is crawlabpy?

Is crawlabpy well maintained?

Did you know?

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install