The command line catalog tool for your offline data
[!WARNING] catcli has been superseded by gocatcli which provides all features of catcli and more...
Did you ever wanted to find back that specific file that should be on one of your backup DVDs or one of your external hard drives? You usually go through all of them hoping to find the right one on the first try? Catcli indexes external media in a catalog file and allows to quickly find specific files or even navigate in the catalog of indexed files while these are not connected to your host.
Features:
lsfzf for finding files
Quick start:
# install catcli with pip
pip3 install catcli --user
# index a directory in the catalog
catcli index --meta='some description' log /var/log
# display the content
catcli ls -r
# navigate
catcli ls log
# find files/directories named '*log*'
catcli find log
# show directories sizes
catcli du log
see usage for specific info
Catcli gives the ability to navigate, explore and find your files that are stored on external media (DVDs, hard drives, USB sticks, etc) when those are not connected. Catcli can just as easily index any arbitrary directories.
See the examples for an overview of the available features.
Table of Contents
Install from Pypi
$ pip3 install catcli --user
Or from github directly
$ cd /tmp; git clone https://github.com/deadc0de6/catcli && cd catcli
$ sudo python3 setup.py install
$ catcli --help
To work with catcli without installing it, you can do the following
$ cd /tmp; git clone https://github.com/deadc0de6/catcli && cd catcli
$ pip3 install -r requirements.txt --user
$ python3 -m catcli.catcli --help
or install it in a virtualenv
$ cd /tmp; git clone https://github.com/deadc0de6/catcli && cd catcli
$ virtualenv -p python3 env
$ source env/bin/activate
$ python setup.py install
$ catcli --help
Catcli is also available on aur: https://aur.archlinux.org/packages/catcli-git/
Each indexed directory is stored in the catalog. Multiple directories can be indexed and they are all available through the command line interface of catcli.
Five different types of entry are present in a catalog:
Following environment variables are supported:
CATCLI_CATALOG_PATH: define the catalog path (--catalog=<path>)CATCLI_NO_BANNER: disable the banner (--no-banner)CATCLI_VERBOSE: enable verbose mode (--verbose)CATCLI_FORMAT: define the output format (-F --format=<fmt>)Let's say the DVD or external hard drive that needs to be indexed
is mounted on /media/mnt. The following command
will index the entire directory /media/mnt
and store that in your catalog under the name <short-name>.
$ catcli index --meta=<some-description> <short-name> /media/mnt
If not specified otherwise (with the switch --catalog), the catalog is saved in the current
directory under catcli.catalog.
The --meta switch allows to add any additional information to store along in
the catalog like for example the blue disk in my office.
Using the -a --archive switch allows to also index archive files as explained
below.
Catcli is able to index and explore the content of archive files. Following archive formats are supported: tar, tar.gz, tar.xz, lzma, tar.bz2, zip. Catcli is also able to find files within indexed archive files.
See the archive example for more.
A catalog can be walked using the command ls as if the media
is mounted (File/directories separator is /).
$ catcli ls tmp/a/b/c
Resulting files can be sorted by size using -S --sortsize.
See the examples for more.
Files and directories can be found based on their names
using the find command.
Find support two formats that allow to use fzf for
searching:
--format=fzf-native: display the result in native format--format=fzf-csv: display the result in csvSee the examples for more.
The catalog can be mounted with fuse and navigate like any filesystem.
$ mkdir /tmp/mnt
$ catcli index -c github .github
$ catcli mount /tmp/mnt
$ ls -laR /tmp/mnt
drwxrwxrwx - user 8 Mar 22:08 github
mnt/github:
.rwxrwxrwx 17 user 19 Oct 2022 FUNDING.yml
drwxrwxrwx - user 2 Mar 10:15 workflows
mnt/github/workflows:
.rwxrwxrwx 691 user 19 Oct 2022 pypi-release.yml
.rwxrwxrwx 635 user 8 Mar 21:08 testing.yml
The entire catalog can be shown using the ls -r command.
Resulting files can be sorted by size using the -S --sortsize switch.
See the examples for more.
You can get the disk usage with the du command.
Resulting files can be sorted by size using the -S --sortsize switch.
The catalog can be exported in a dot file that can be used to generate a graph of the indexed files.
$ catcli graph
dot file created under "/tmp/catcli.dot"
create graph with "dot /tmp/catcli.dot -T png -o /tmp/tree.png" (you need graphviz)
$ dot /tmp/catcli.dot -T png -o /tmp/tree.png
Storage entry can be edited with following catcli commands:
rename - rename the storageedit - edit storage metadataThe catalog can be updated with the update command.
Updates are based on the access time of each of the files and on the
hash checksum if present (catalog was indexed with -c --hash and
update is called with the switch -c --hash).
Results can be printed to CSV using --format=csv.
Fields are separated by a comma (,) and are quoted with double quotes (").
Each line contains the following fields:
Let's first create some files and directories:
$ mkdir -p /tmp/test/{a,b,c}
$ echo 'something in files in a' > /tmp/test/a/{1,2,3}
$ echo 'something else in files in b' > /tmp/test/b/{4,5,6}
$ echo 'some bytes' > /tmp/test/c/{7,8,9}
$ tree /tmp/test
/tmp/test
├── a
│ ├── 1
│ ├── 2
│ └── 3
├── b
│ ├── 4
│ ├── 5
│ └── 6
└── c
├── 7
├── 8
└── 9
3 directories, 9 files
First this directory is indexed with catcli as if it was some kind of
external storage:
$ catcli index --meta='my test directory' tmptest /tmp/test
Catcli creates its catalog file in the current directory as catcli.catalog.
Printing the entire catalog as a tree is done with the command ls -r
$ catcli ls -r
top
└── storage: tmptest (my test directory) (nbfiles:3, free:3.7G/3.7G, date:2019-01-26 19:59:47)
├── a [nbfiles:3, totsize:72]
│ ├── 1 [size:24]
│ ├── 2 [size:24]
│ └── 3 [size:24]
├── b [nbfiles:3, totsize:87]
│ ├── 4 [size:29]
│ ├── 5 [size:29]
│ └── 6 [size:29]
└── c [nbfiles:3, totsize:33]
├── 7 [size:11]
├── 8 [size:11]
└── 9 [size:11]
The catalog can be walked with ls as if it was a normal directory
$ catcli ls
top
- storage: tmptest (my test directory) (nbfiles:3, free:3.7G/3.7G, date:2019-01-26 19:59:47)
$ catcli ls tmptest
storage: tmptest (my test directory) (nbfiles:3, free:3.7G/3.7G, date:2019-01-26 19:59:47)
- a [nbfiles:3, totsize:72]
- b [nbfiles:3, totsize:87]
- c [nbfiles:3, totsize:33]
$ catcli ls tmptest/b
b [nbfiles:3, totsize:87]
- 4 [size:29]
- 5 [size:29]
- 6 [size:29]
And files can be found using the command find
$ catcli find 9
c/9 [size:11, storage:tmptest]
When using the -b --script switch, a one-liner is generated
that allows to handle the found file(s)
$ catcli find 9 --script
c/9 [size:11, storage:tmptest]
op=file; source=/media/mnt; $op ${source}/c/9
Let's consider a directory containing archive files:
$ ls -1 /tmp/catcli
catcli-0.3.1
v0.3.1.tar.gz
v0.3.1.zip
To enable the indexing of archive contents use
the -a --archive switch
$ catcli index -au some-name /tmp/catcli
Then any command can be used to explore the catalog as for normal
files but, by providing the -a --archive switch, archive content are displayed.
$ catcli ls some-name
storage: some-name (free:800G, total:1T)
- catcli-0.3.1 [nbfiles:11, totsize:80.5K]
- v0.3.1.tar.gz [size:24.2K]
- v0.3.1.zip [size:31.2K]
$ catcli ls -r some-name/v0.3.1.zip
v0.3.1.zip [size:31.2K]
$ catcli ls -ar some-name/v0.3.1.zip
v0.3.1.zip [size:31.2K]
├── catcli-0.3.1 [archive:v0.3.1.zip]
│ ├── catcli [archive:v0.3.1.zip]
│ │ ├── __init__.py [archive:v0.3.1.zip]
│ │ ├── catalog.py [archive:v0.3.1.zip]
│ │ ├── catcli.py [archive:v0.3.1.zip]
│ │ ├── logger.py [archive:v0.3.1.zip]
│ │ ├── noder.py [archive:v0.3.1.zip]
│ │ ├── utils.py [archive:v0.3.1.zip]
│ │ └── walker.py [archive:v0.3.1.zip]
│ ├── .gitignore [archive:v0.3.1.zip]
│ ├── LICENSE [archive:v0.3.1.zip]
│ ├── MANIFEST.in [archive:v0.3.1.zip]
│ ├── README.md [archive:v0.3.1.zip]
│ ├── requirements.txt [archive:v0.3.1.zip]
│ ├── setup.cfg [archive:v0.3.1.zip]
│ ├── setup.py [archive:v0.3.1.zip]
│ ├── tests [archive:v0.3.1.zip]
│ │ ├── __init__.py [archive:v0.3.1.zip]
│ │ ├── helpers.py [archive:v0.3.1.zip]
│ │ ├── test_find.py [archive:v0.3.1.zip]
│ │ ├── test_graph.py [archive:v0.3.1.zip]
│ │ ├── test_index.py [archive:v0.3.1.zip]
│ │ ├── test_ls.py [archive:v0.3.1.zip]
│ │ ├── test_rm.py [archive:v0.3.1.zip]
│ │ └── test_tree.py [archive:v0.3.1.zip]
│ ├── tests.sh [archive:v0.3.1.zip]
│ └── .travis.yml [archive:v0.3.1.zip]
└── catcli-0.3.1/ [archive:v0.3.1.zip]
If you are having trouble installing or using catcli, open an issue.
If you want to contribute, feel free to do a PR (please follow PEP8).
The tests.sh script can be run to check the code.
If you like catcli, buy me a coffee.
This project is licensed under the terms of the GPLv3 license.
FAQs
The command line catalog tool for your offline data
We found that catcli demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
PyPI confirms no security flaws were exploited in the Ultralytics supply chain attack and highlights improvements for safer package publishing.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.