Readme
apibackuper is a command line tool to archive/backup API calls. It's goal to download all data behind REST API and to archive it to local storage. This tool designed to backup API data, so simple as possible.
.. contents::
.. section-numbering::
This tool was developed optimize backup/archival procedures for Russian government information from E-Budget portal budget.gov.ru and some other government IT systems too. Examples of tool usage could be found in "examples" directory
Most Linux distributions provide a package that can be installed using the system package manager, for example:
.. code-block:: bash
# Debian, Ubuntu, etc.
$ apt install apibackuper
.. code-block:: bash
# Fedora
$ dnf install apibackuper
.. code-block:: bash
# CentOS, RHEL, ...
$ yum install apibackuper
.. code-block:: bash
# Arch Linux
$ pacman -S apibackuper
A universal installation method (that works on Windows, Mac OS X, Linux, …, and always provides the latest version) is to use pip:
.. code-block:: bash
# Make sure we have an up-to-date version of pip and setuptools:
$ pip install --upgrade pip setuptools
$ pip install --upgrade apibackuper
(If pip installation fails for some reason, you can try
easy_install apibackuper as a fallback.)
Python version 3.6 or greater is required.
This example is about backup of Russian certificate authorities. List of them published at e-trust.gosuslugi.ru and available via undocumented API.
.. code-block:: bash
$ apibackuper create etrust
$ cd etrust
Edit apibackuper.cfg as:
.. code-block:: bash
[settings]
initialized = True
name = etrust
[project]
description = E-Trust UC list
url = https://e-trust.gosuslugi.ru/app/scc/portal/api/v1/portal/ca/list
http_mode = POST
work_modes = full,incremental,update
iterate_by = page
[params]
page_size_param = recordsOnPage
page_size_limit = 100
page_number_param = page
[data]
total_number_key = total
data_key = data
item_key = РеестровыйНомер
change_key = СтатусАккредитации.ДействуетС
[storage]
storage_type = zip
Add file params.json with parameters used with POST requests
.. code-block:: json
{"page":1,"orderBy":"id","ascending":false,"recordsOnPage":100,"searchString":null,"cities":null,"software":null,"cryptToolClasses":null,"statuses":null}
Execute command "estimate" to see how long data will be collected and how much space needed
.. code-block:: bash
$ apibackuper estimate full
Output:
.. code-block:: bash
Total records: 502
Records per request: 100
Total requests: 6
Average record size 32277.96 bytes
Estimated size (json lines) 16.20 MB
Avg request time, seconds 66.9260
Estimated all requests time, seconds 402.8947
Execute command "run" to collect the data. Result stored in "storage.zip"
.. code-block:: bash
$ apibackuper run full
Exports data from storage and saves as jsonl file called "etrust.jsonl"
.. code-block:: bash
$ apibackuper export jsonl etrust.jsonl
Example config file
.. code-block:: bash
[settings]
initialized = True
name = <name>
splitter = .
[project]
description = <description>
url = <url>
http_mode = <GET or POST>
work_modes = <combination of full,incremental,update>
iterate_by = <page or skip>
[params]
page_size_param = <page size param>
page_size_limit = <page size limit>
page_number_param = <page number>
count_skip_param = <key to iterate in skip mode>
[data]
total_number_key = <total number key>
data_key = <data key>
item_key = <item key>
change_key = <change key>
[follow]
follow_mode = <type of follow mode>
follow_pattern = <url prefix to follow links>
follow_data_key = <follow data item key>
follow_param = <follow param>
follow_item_key = <follow item key>
[files]
fetch_mode = <file fetch mode>
root_url = <file root url>
keys = <keys with file data>
storage_mode = <file storage mode>
[storage]
storage_type = zip
compression = True
Synopsis:
.. code-block:: bash
$ apibackuper [flags] [command] inputfile
See also apibackuper --help.
Create project "budgettofk":
.. code-block:: bash
$ apibackuper create budgettofk
Estimate execution time for 'budgettofk' project. Should be called in project dir or project dir provided via -p parameter:
.. code-block:: bash
$ apibackuper estimate full -p budgettofk
Output
.. code-block:: bash
Total records: 12282
Records per request: 500
Total requests: 25
Average record size 1293.60 bytes
Estimated size (json lines) 15.89 MB
Avg request time, seconds 1.8015
Estimated all requests time, seconds 46.0536
Run project. Should be called in project dir or project dir provided via -p parameter
.. code-block:: bash
$ apibackuper run full
Export data from project. Should be called in project dir or project dir provided via -p parameter
.. code-block:: bash
$ apibackuper export jsonl hhemployers.jsonl -p hhemployers
Follows each object of downloaded data and does requests for each objects .. code-block:: bash
$ apibackuper follow continue
Downloads all files associated with API objects .. code-block:: bash
$ apibackuper getfiles
TBD
.. :changelog:
FAQs
apibackuper: a command-line tool and python library for API backuping
We found that apibackuper demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
CISA launched a new project called Vulnrichment to enrich CVEs with details that help prioritize patching and mitigation efforts, as the NVD backlog of unenriched CVEs awaiting analysis surpasses 10,000.
Security News
Socket is joining forces with CISA and other industry leaders at the RSA Conference to sign the Secure by Design pledge, committing to uphold the highest security standards in our products.
Research
The Socket research team breaks down a sampling of malicious packages that download and execute files, among other suspicious behaviors, targeting the popular Discord platform.