scdata

Smart Citizen Data

Welcome to SmartCitizen Data. This is a data analysis framework for working with sensor data in different ways:

• Interacting with several sensors APIs
• Clean data, export and calculate metrics
• Model sensor data and calibrate sensors
• Generate data visualisations - matplotlib, plotly or uplot
• Generate analysis reports in html or pdf and upload them to Zenodo

Installation

You can check it out in the before installing if you want. Works with Python 3.* (tested between 3.9 and 3.11).

You can just run:

pip install scdata

Work on the source code

Simply clone the repository with:

git clone https://github.com/fablabbcn/smartcitizen-data.git
cd smartcitizen-data

Install scdata package with requirements:

python setup.py install

Or if you want to edit:

cd scdata
pip install --editable .

Tokens and config

If you want to upload data to Zenodo, you will need to fill set an environment variable called ZENODO_TOKEN in your environment. You can get more instructions here and with this example.

A configuration file is available at ~/.config/scdata/config.yaml, which contains a set of configurable variables to allow or not the local storage of relevant data in the data folder, normally in ~/.cache/scdata:

data:
  cached_data_margin: 2
  load_cached_api: true
  reload_metadata: true
  store_cached_api: true
paths:
  config: /Users/username/.config/scdata
  data: /Users/username/.cache/scdata
  export: /Users/username/.cache/scdata/export
  interim: /Users/username/.cache/scdata/interim
  inventory: ''
  models: /Users/username/.cache/scdata/models
  processed: /Users/username/.cache/scdata/processed
  raw: /Users/username/.cache/scdata/raw
  reports: /Users/username/.cache/scdata/reports
  uploads: /Users/username/.cache/scdata/uploads
zenodo_real_base_url: https://zenodo.org
zenodo_sandbox_base_url: http://sandbox.zenodo.org

Also, .env files will be picked from ~/.cache/scdata.

Using with jupyter lab (optional)

It can also be used with jupyter lab or jupyter. For this install juypterlab.

Contribute

Issues and PR more than welcome!

Funding

This work has received funding from the European Union's Horizon 2020 research and innovation program under the grant agreement No. 689954