confo2

Multi backend configuration manager for python

About Confo

Confo is a flexible multi backend configuration manager for python. Built to provide a unified configuration management interface in different environments.Confo supports both local configuration with json files and distributed configurations, best suited for micro-service and distributed systems development.

Getting started

1.Installation

pip install confo2

2.Setup Backends

Confo currently supports two backends, a filesystem backend and a ZooKeeper backend.

2.1 Filesystem backend

The filesystem backend simply works by reading json files from the filesystem. An example configuration can be setup like below. Imagine you have a sales database with credentials you want to expose to your application, and you want to use a sales forecasting model which is exposed via REST api. Confo uses namespaces to separate logical groups of configurations. For example everything concerned with systems from the sales department can be stored in the sales namespace and every configuration concerned with systems from the anlytics team can be store in a namespace called analytics.

mkdir Configurations
mkdir Configurations/sales
mkdir Configurations/analytics
echo '{"host":"127.0.0.1", "port":5432,"username":"kabelo","password":"confoRocks"}' > Configurations/sales/database.json
echo '{"host":"127.0.0.1", "port":5432,"token":"2df228d6-890b-11eb-8dcd-0242ac130003","secret":"d51a8a7a-3286-4deb-8805-85f4528920ae 
dcfb49c0-a251-4657-87df-44996ea308ee"}' > Configurations/analytics/api_one.json

In the setup above both namespaces only have one configuration,but each namespace can have infinite configurations. Confo uses a json files to store each configuration.

backend 
   |______namespace
             |_______configuration
                          |__________fields
                          
e.g

filesystem 
  |________sales
             |___________database 
                             |_________host : 127.0.0.1 
                             |_________port : 5432
                             |_________user : kabelo 
                             |_________password :confoRocks 

from confo.Confo import Confo
import confo.Backends as BE

#create the singleton configuration manager object 
config = Confo()

The Confo object is a singleton, meaning you can instantiate it multiple time through the code base, but you will always have up to date configurations loaded.

# Instantiate a FILE_BACKEND backend 
cred = {"config_path":"Configurations/"}
config.load_backend(credentials=cred,name="example_backend",backend_type=BE.FILE_BACKEND)

The Confo.load_backend(credentials,name,backend_type) method is used to create a backend management object.The credentials differ by backend type The filesystem backend accepts a dictionary of credentials which contains the config_path key, this is the path where the configurations are stored.

backed operations

To list all backends loaded

config.get_backends()
#RESULTS:

{'all_backends': ['example_backend'], 'active_backend': None}

This return a dictionary with two keys: all_backends holds list of all backends registered in this application; and active_backend holds the name of the currently active backend.Note that activate_backend is None meaning the manager cannot access any configuration.If any of the configuration level methods are invoked a BackendsActivationException error is raised.Now to make sure a backend is active,

#activate the example_backend
config.activate_backend("example_backend")
# List backends again to see if we have an active backend
print(config.get_backends())
#RESULTS
{'all_backends': ['example_backend'], 'active_backend': 'example_backend'}

Note that the activate_backend field has the backend we activated with Confo.activate_backend.

Namespace operations

As mentioned above namespaces create a logical partition between groups of configuration, this becomes more useful when dealing with distributed configurations, where multiple microservices are accessing and sharing configuration and data through Confo.

#The method below is the namespace level analog to get_backends()
print(config.get_namespaces())
#RESULTS:
{'all_namespaces': ['sales', 'analytics'], 'current_namespace': '*'}

Please note we need to choose a namespace to use before we can access any configuration. If no namespace is chosen a NamespaceNotLoadedException error will be raised.

#choose namespace 
config.use_namespace("sales")
#check if a namespace is chosen
print(config.get_namespaces())

#RESULTS:
{'all_namespaces': ['sales', 'analytics'], 'current_namespace': 'sales'}

configuration level operations

lets get all configurations in the current namespace

print(config.get_all())

#RESULTS: 
{'database': {'host': '127.0.0.1',
              'port': 5432,
              'username': 'kabelo',
              'password': 'confoRocks'}}

The sales department is exposing daily sales reports through a REST API, the report is read a json data which will be used by our application to build visualization. Lets create a new configuration programmatically, and store the credentials to this API.

config.set("sale_report","host","10.222.194.146")
config.set("sale_report","port","2389")
config.set("sale_report","token","01e947b6-8914-11eb-8dcd-0242ac130003")
config.set("sale_report","host","78f45902-c9b8-4151-b924734+18080f28-8914-11eb-8dcd-0242ac133")

#OR 

sales_report_data = {
    "host":"10.222.194.146",
    "port":"2389",
    "token":"01e947b6-8914-11eb-8dcd-0242ac130003",
    "secret":"78f45902-c9b8-4151-b924734+18080f28-8914-11eb-8dcd-0242ac133"
}
config.set("sale_report",sales_report_data,None)

The Confo.set(config,field,value) method creates a new configuration in the current namespace. This method can be used this way Confo.set(config,field,value) or Confo.set(config,dictionary,None), by replacing the value of field with a dictionary or list and replacing the last argument with None we can overload the default behaviour and set the configuration values directly.

Now lets check all available configuration in the namespace:

print(config.get_all())
#RESULTS: 
{'database': {'host': '127.0.0.1',
              'port': 5432,
              'username': 'kabelo',
              'password': 'confoRocks'},
 'sale_report': {'host': '10.222.194.146',
                 'port': '2389',
                 'token': '01e947b6-8914-11eb-8dcd-0242ac130003',
                 'secret': '78f45902-c9b8-4151-b924734+18080f28-8914-11eb-8dcd-0242ac133'}}

Perfect we can now consume configuration and metadata in a clean and decoupled way.Imagine if you have a 100 configurations and metadata in a given namespace,the output dictionary from Confo.get_all() can be overwhelming to traverse. Lets find a more efficient way

#Get the database host 
print(config.get("database","host"))

#RESULT
127.0.0.1

#get the database password 
print(config.get("database","confoRocks"))

#RESULTS:
confoRocks

#get the entire sale_report configuration
print(config.get("sale_report"))

#RESULT
{'host': '10.222.194.146',
 'port': '2389',
 'token': '01e947b6-8914-11eb-8dcd-0242ac130003',
 'secret': '78f45902-c9b8-4151-b924734+18080f28-8914-11eb-8dcd-0242ac133'}

#By ommiting the `field` argument `Confo.get()` returns the whole  configuration 

Lets assume we were able to programmatically retrieve a new secret and token for the sales report API. Confo allows us to update the configuration, by using Confo.set() to overwrite the old values.

new_token = "e9f2b59d-130d-4b61-b20c-94c73496655f"
new_secret = "cd58cb1d-22fc-4420-b5ab-6b67a565671d7d34f5e4-8916-11eb-8dcd"
config.set("sale_report","token",new_token)
config.set("sale_report","secret",new_secret)
print(config.get("sale_report"))

#RESULTS:
{'host': '10.222.194.146',
 'port': '2389',
 'token': 'e9f2b59d-130d-4b61-b20c-94c73496655f',
 'secret': 'cd58cb1d-22fc-4420-b5ab-6b67a565671d7d34f5e4-8916-11eb-8dcd'}

swapping namespaces

What if we now want to access the analytics's sales forecasting REST API.

#You can start by checking available namespaces 
print(config.get_namespaces())
#RESULTS: 
{'all_namespaces': ['sales', 'analytics'], 'current_namespace': 'sales'}

#Lets swap from sales to analytics.
config.use_namespaces("analytics")
print(config.get_namespaces())

#RESULTS: 

{'all_namespaces': ['sales', 'analytics'], 'current_namespace': 'analytics'}

After swapping namespaces check which configurations exist in analytics namespace

config.get_all()

#RESULTS
{'api_one': {'host': '127.0.0.1',
             'port': 5782,
             'token': '2df228d6-890b-11eb-8dcd-0242ac130003',
             'secret': 'd51a8a7a-3286-4deb-8805-85f4528920aecfb49c0-a251-4657-87df-44996ea308ee'}}

config.get("api_one")

#RESULTS
{'host': '127.0.0.1',
 'port': 5782,
 'token': '2df228d6-890b-11eb-8dcd-0242ac130003',
 'secret': 'd51a8a7a-3286-4deb-8805-85f4528920aecfb49c0-a251-4657-87df-44996ea308ee'}

#Only `api_one` exists in the `analytics` namespace as expected

We have retrieved the api_one values lets go back to the sales namespace

config.use_namespace("sales")
print(config.get_all())

#RESULTS 
{'database': {'host': '127.0.0.1',
              'port': 5432,
              'username': 'kabelo',
              'password': 'confoRocks'},
 'sale_report': {'host': '10.222.194.146',
                 'port': '2389',
                 'token': 'e9f2b59d-130d-4b61-b20c-94c73496655f',
                 'secret': 'cd58cb1d-22fc-4420-b5ab-6b67a565671d7d34f5e4-8916-11eb-8dcd'}}

Now if we were to open the Configurations/ directory this is what I would find in the sale directory:

Configurations
        |___________analytics 
        |                |_________api_one.json 
        |____________sales
                       |___________database.json          

There is no file Configurations/sales/sale_report.json . The sale_report configuration only exists in memory. The This allows the developer to explicitly decide when they want to persist the updated or new configuration as a result this helps the developer to handle cases where he creates session specific configurations, these should remain in in memory and not be persisted.

persistence

Confo handles persistance at three levels :

backend level : This option persist the current state of all configuration in the current backend.
    |
    |namespace level: This option persist the current state of all configuration in the current namespace
         |
         | Configuration level: This option persists a specific configuration

backend level

config.persist()

The code above will persist every configuration from all namespaces.

namespace level

config.persist(namespace="sales")

The above code will persist every configuration in the sales namespace

configuration level

config.persist(namespace="sales",config="sale_report")

The above code will persist the sale_report configuration only.If the configuration file already exists i will be updated.

2.2 ZooKeeper backend

Lets assume our application is operating a very distributed environments. One of the main problems in distributed systems is finding one source of truth for application state and configuration.ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications.We use zookeeper as a backend for Confo essentially abstracting away the complex zookeeper API. The advantage of using confo in this environment is that you get the same API when testing locally and when deploying on production.

Lets load a zookeeper backend

# Instantiate a ZOOKEEPER_BACKEND backend 
cred = {"zookeeper_host":"127.0.0.1","zookeeper_port":"2181",
        #if your zookeeper instance has password authentication we can also send 

        "zookeeper_user":"kabelo",
        "zookeeper_passwd":"confoRocks"
        }
config.load_backend(credentials=cred,name="zookeeper_backend",backend_type=BE.ZOOKEEPER_BACKEND )

print(config.get_backend())

#RESULTS:
{'all_backends': ['example_backend', 'zookeeper_backend'],
 'active_backend': 'example_backend'}

Lets activate the zookeeper backend.

config.activate_backend("zookeeper_backend")
config.get_backends()

#RESULTS:

{'all_backends': ['zookeeper_backend', 'example_backend'],
 'active_backend': 'zookeeper_backend'}

Zookeeper quick overview

ZooKeeper allows distributed processes to coordinate with each other through a shared hierarchical name space of data registers (we call these registers znodes), much like a file system. Unlike normal file systems ZooKeeper provides its clients with high throughput, low latency, highly available, strictly ordered access to the znodes. The performance aspects of ZooKeeper allow it to be used in large distributed systems. The reliability aspects prevent it from becoming the single point of failure in big systems. Its strict ordering allows sophisticated synchronization primitives to be implemented at the client.

Znodes are analogous to files and directory at the same time. Because a Znode can have children Znodes like a directory can have subdirectories and files. Also every znode can store data just like a file. Confo depends on a znode call confo which it will create automatically if it does not exist. Then every znode which is a child of confo is a namespace so they will not be queried for any data.Then every child of a namespace znode is considered a configuration znode and are expected to have json object stored within. Now imagine we have znodes systemA and systemB under the confo znode:

"/"                                                                          ->zookeeper root 
  |________"confo"                                                           -> confo root ("/confo")
            |_________"systemA"                                              -> namespace called "systemA" ("/confo/systemA")
            |             |__________"database"                              -> a configuration called "database" ("/confo/systemA/database")
            |                           |______________host: 127.0.0.1       -> a configuration field called "host"
            |                           |______________port: 3306 
            |                           |______________user: root 
            |                           |______________password: newpassword
            |                           |______________vendor: postgres 
            |
            |_____________"systemB"                                          -> namespace called "systemB" ("/confo/systemB")  
                            |__________"email"                               -> a configuration called "email" ("/confo/systemB/email")
                                           |___________ "port":587           -> a configuration field called "port"
                                           |___________ "smtp_server": "smtp.gmail.com"
                                           |___________ "sender_email":"[e-mail]"
                                           |___________ "password" :"ThisIsAStrongPasswordTrustMeItIs"            
                                        

Now lets build a python app that sends an email verification link to newly registered users.We decoupled configuration from actual code by using confo:

from confo.Confo import Confo
import confo.Backends as BE
import smtplib, ssl
from sqlalchemy import create_engine

config = Confo()
cred = {"zookeeper_host":"127.0.0.1","zookeeper_port":"2181",

        }

config.load_backend(credentials=cred,name="zookeeper_backend",backend_type=BE.ZOOKEEPER_BACKEND )
config.active_backend("zookeeper_backend")
config.use_namespace("systemA")
def send_email(smtp_server,sender_email,receiver_email,password,port=587):
    # Create a secure SSL context
    context = ssl.create_default_context()
    # Try to log in to server and send email
    try:
        message= "Please click <a href=\"http://verification-link.com\">here</a> to verify your email"
        server = smtplib.SMTP(smtp_server,port)
        server.ehlo() # Can be omitted
        server.starttls(context=context) # Secure the connection
        server.ehlo() # Can be omitted
        server.login(sender_email, password)
        server.sendmail(sender_email, receiver_email, message)

    except Exception as e:
            # Print any error messages to stdout
            print(e)
    finally:
        server.quit()
def build_data_url(database):
    url = database['vendor'] + '://' + database['user'] + ':' + database['password']
    url = url +'@' +database['host'] + ':' + database['port'] + '/' + database['db_name'] + '?driver=' + database['driver']
    return url
def build_engine(config):
    database_config = config.get('database')
    url = build_data_url(database_config)
    engine = create_engine(url)
    return engine
def mail_recipients(engine,config):
    smtp_server = config["smtp_server"]
    port= config["port"]
    sender_email = config["sender_email"]
    password= config["password"]
    with engine.connect() as connection:
        result = connection.execute("select email from users where verified=0")
    for row in result:
        sender_email(smtp_server=smtp_server,sender_email=sender_email,receiver_email=row['email'],password=password,port=port) 
        
engine = build_engine(config)
config.active_backend("systemB")
email_config = config.get("email")
mail_recipients(engine,email_config)

    

The script above shows a simple use case for a robust configuration manager.