gitlab.com/interview95/go-fetch

Fetch URLs in Go

Notes

I used cobra-cli to implement this command line application because I already knew it and I know it is very convenient and commonly used in the Go community for this type of programs (my personal blog - hosted on IPFS - is made using Hugo, a static sites generator, which command line uses indeed cobra-cli. So, I studied it many years ago and I know how it works).

Build and run Docker

docker build -t cli .
docker run -it -p 8080:80 cli /bin/bash

and then it is possible to call the ready executable with ./fetch

How to use

# ./fetch --help
This command fetches the HTML page from a list of URLs and save them into files locally.

        Example: if fetch from https://www.google.com, then it saves the pointed HTML page into current_path/www.google.com.html

Usage:
   [flags]

Flags:
  -h, --help       help for this command
  -m, --metadata   Visualize additional info about the URLs fetched.

To fetch URLs with metadata:

./fetch --metadata https://www.gazzetta.it

site:  www.gazzetta.it
num_links:  679
images:  361
last_fetch:  Tue Jan 24 2023 02:55 UTC

In any case, the HTML of the web page is saved locally:

root@0cafd40693ff:/app# ls -lrt
...
...
rw-r--r-- 1 root root  435918 Jan 24 02:55 www.gazzetta.it.html