super-collections

Python Super Collections

Dictionaries and lists as you dreamed them when you were a kid.

Instantly Convert json and YAML files into objects with attributes.

import json
from super_collections import SuperDict
with open('my_file.json', 'r') as file:
    data = json.load(file)
document = SuperDict(data)

print(document.author) # instead of document['author'] 
for document in document.blocks: # instead of document['blocks']
    ...
print(document.blocks[3].name) # instead of document['blocks'][3]['name'] -- eek! 🤢

• Python Super Collections
  • What are SuperCollections?
    • What is the problem?
    • Definitions
  • How it works
    • Superdicts
    • Superlists
    • Why Combining SuperDicts with SuperLists?
    • SuperCollection
    • Factory function
  • Install
    • From the repository
  • Usage
  • Export to JSON or Hjson
  • Export to YAML
  • Remarks
    • Restrictions
    • Does it work?
    • When are superdictionaries and superlists not recommended?
  • Shelves and SuperShelves
    • Metaphor of a shelf
    • Definition of a shelf (data structure)
    • Iteration through a shelf
    • SuperShelf
  • Related data structures and ideas
    • Standard Python
    • Dot notation on dictionaries
    • Using superlists to complement superdictionaries

What are SuperCollections?

What is the problem?

There are several packages that quickly convert json or YAML files into dictionaries that contain dictionaries, lists etc.

If you want to properly use those data structures in Python, one solution is to create specific classes.

But sometimes, it is overkill. You just want your app to quickly load structured data and navigate through them.

That's where the super-collections package comes handy.

Definitions

• A SuperCollection is a nested data structure that can encode any type of information (in the same way as a JSON file). It is essentially constituted of dictionaries and lists, where dictionaries can contain lists and vice-versa, and the root can be either a dictionary or a list.

• An elementary datatype is a non-mutable type: str, int, str, float in Python that is used to build classes.

• A SuperList is a list that can contain SuperCollections, or elementary datatypes.

• A SuperDict is a dict that can contain SuperCollections, or elementary datatypes. A key advantage of SuperDicts over dictionaries, is that its keys can be accessed as attributes (providing they are valid Python identifiers and they don't conflict with pre-existing attributes).

How it works

Superdicts

📝 Definition
A superdictionnary is a dictionary whose keys (at least those that are valid identifiers) are automatically accessible as attributes, with the *dot notation.

d = SuperDict({'foo':5, 'bar': 'hello'})

# instead of writing d['foo']
d.foo = 7

Several other languages, such as Javascript, LUA, Ruby, and PHP offer that dot notation in some form or other. However, implementing that idea is not as straightforward as it seems. The idea of superdictionaries in Python has been around for some time (see the superdict packagage by Yuri Shevchuk, 2015).

📝 Property
If a SuperDict object contains a value that is itself a dictionary, that dictionary is then converted in turn into a SuperDict.

If the object is not a dict or immediately compatible, it will try the following conversions:

• the methods .asdict(), .dict() or dump(), providing they actually generate a dict.
• if the object is a dataclass, it will apply the asdict() function on it.

Superlists

A superlist is a list where all dictionary items have been (automagically) converted to superdictionnaries.

⚠️ Superlists are indispensable
They were the missing piece of the jigsaw puzzle; without them, it is not possible to convert deep data structures into supercollections.

Why Combining SuperDicts with SuperLists?

The structure of JSON, YAML or HTML data is generally a deeply nested combination of dictionaries and lists. Using superdictionaries alone would not be sufficient, since lists within the data contained in a list would still contain regular (unconverted) dictionaries; this would require you to switch back to the standard dictionary access method.

By combining superdictionaries and superlists, it is possible to ensure that all nested dictionaries within lists will also be converted to SuperDicts, allowing for a consistent dot notation throughout the entire data structure.

💡 Deep conversion
SuperLists objects, combined with SuperDicts make sure that the most complex datastructures (from json or YAML) can be recursively converted into well-behaved Python objects.

SuperCollection

SuperCollection is an abstract class containing SuperList and SuperDict.

It means that SuperList and SuperDict objects are instances of SuperCollection, but they do not inherit from it.

obj1 = SuperList([1, 3, 5])
assert isinstance(obj1, SuperCollection)

obj2 = SuperDict({'a': 5, 'b':7})
assert isinstance(obj2, SuperCollection)

Factory function

You can use the super_collect() function to create a SuperCollection (SuperList or SuperDict) from an object.

It is particularly useful for converting Python data structures such as JSON files.

import json
with open(FILENAME, 'r', encoding='utf-8') as f:
    data = json.load(f)
content = super_collect(data)

super_collect() is designed to work on any list or dict, but it will also attempt to process other types:

• all sequences (see definition), in other words objects whose class is registered as instance of collections.abs.Sequence will be converted into lists. This applies to tuple, range, collections.UserList, etc.
• Special types: set, deque as well as ndarray (Numpy or compatible) and Series (Pandas and others; your mileage may vary).
• Otherwise, will try to generate a SuperDict.

This function is also available as a static method of the SuperCollection class:

content = SuperCollection.collect(data)

Install

From the repository

pip install super-collections

Usage

from super_collections import SuperDict, SuperList

d = SuperDict({'foo':5, 'bar': 'hello'})
l = SuperList([5, 7, 'foo', {'foo': 5}])

You can cast any dictionary and list into its "Super" equivalent when you want, and you are off to the races.

The casting is recursive i.e. in the case above, you can assert:

l[-1].foo == 5

All methods of dict and list are available.

Those objects are self documented. d.properties() is a generator that lists all keys that are available as attributes.

The __dir__() method (accessible with dir()) is properly updated with those additional properties.

list(d.properties())
> ['foo', 'bar']
dir(d)
> ['__class__', ..., 'bar', 'clear', 'copy', 'foo', 'fromkeys', 'get', 'items', 'keys', 'pop', 'popitem', 'properties', 'setdefault', 'to_hjson', 'to_json', 'update', 'values']

This means the auto-complete feature might be available for the attributes of a SuperDict within a code editor (if the dictionary was statically declared in the code); or in an advanced REPL (such as bpython).

The methods dict.update(other_dict) and list.extend(other_list) automatically cast the contents into SuperDict and SuperList as needed.

Export to JSON or Hjson

You can export a SuperDict or SuperList to JSON or Hjson, for debug purposes. It is not guaranteed that it will preserve all the meaningful information you want, but all basic types will be preserved.

Python DateTimes are converted into ISO Dates.

print (d.to_json())
print (d.to_hjson())

The module also exports a json_encode() function, which will attempt to serialize any object in Python to JSON (in an opinionated way).

Export to YAML

If you wish to use PyYAML and guarantee the SuperDict and SuperList behave exactly as dict and list, use the yaml_support() function.

This works with both dump() and safedump()

from super_collections import SuperDict, SuperList, yaml_support
yaml_support()


d = SuperDict({"x": 1})
l = SuperList(["a", "b"])

dumped_dict = yaml.dump(d)
dumped_list = yaml.dump(l)

Remarks

Restrictions

• In a SuperDict, only keys that are valid Python identifiers can be accessed as attributes. If 'bar' is a key of object foo, you can write foo.bar; but you can't write foo.hello world because 'hello world' is not a valid Python identifier; you will have to access that specific value with the "dictionary" notation: foo['hello world'].
• Similarly, you can't use pre-existing methods of the dict class: keys, items, update, etc. as properties; as well as the properties method itself (wich is specific to SuperDict). In that case again, use the dictionary notation to access the value (d['items'], etc.). Those keys that cannot be accessed as attributes are said to be masked. If you are uncertain which are available, just use SuperDict.properties(). method.
• Updating a single element (d['foo'] for a SuperDict and l[5] for a SuperList) does not perfom any casting. That's to avoid crazy recursive situations, while giving you fine grain control on what you want to do (just cast with SuperDict() and SuperList()).

Does it work?

Yes. It is tested with pytest. See the test directory for examples.

When are superdictionaries and superlists not recommended?

SuperDicts (and SuperLists) classes are most useful when the program you are writing is consuming loosely structured data (json, YAML, HTML) you have every reason to believe they are sufficiently well-formed: typically data exported from existing APIs or Web sources.

⚠️ Caution
super-collections may not be the best tool when source data come from a source whose quality is unsufficiently guaranteed for your needs, or is untrusted.

If you want to impose strongly formatted data structures in your code, one solution is to create dataclasses; especially those of Pydantic, which make implicit and explicit controls on the integrity of the source data.

Shelves and SuperShelves

Metaphor of a shelf

Imagine a real shelf in your home, with compartments. You place items in each compartment: books, boxes, a photo frame, etc. Some compartments have labels ("History Books", "Photos"), others don’t. To refer to a compartment, you can use its position ("the third from the left"), or refer to it by its label.

If it was a really big shelf, as in a library, and you were given the label of a compartment and no other information, finding it would take some time, because you would have to scan the whole shelf each time.

To speed things up, you would have to keep a cardfile with cards for each label, indicating the position of the corresponding compartment in the shelf.

Definition of a shelf (data structure)

A Shelf data structure works the same way. Conceptually, a shelf is very intuitive: you can think about it as a list where you can also use labels for fast retrieval.

It’s a line of compartments, each holding one object.

If implements dual addressing: You can access an item by a key that can be either its index (2) or its label, providing it exists ("Receipts"). In other words, a shelf works both as a list and a dictionary, and has the attributes of both classes.

Internally, a shelf is a list of cells, with a value and an optional label. It is complemented by a cardfile that converts the labels of the cells into their index. However, you don't have to worry about it: when you add an item to the shelf, change it or delete it, both the list and the carfile are kept up-to-date.

Iteration through a shelf

When you iterate through a shelf, you return the values as in a list.

⚠️ IMPORTANT NOTE: This is distinct from a Python dictionary, where iteration returns the keys.

• If you want the keys use the .keys() method. For items that do not have a label, it returns the index.
• If you want the key, value pairs, use the .items() method.
• The .values() method is provided for compatibility with a dictionary.

SuperShelf

A SuperShelf is essentially a shelf that behaves as a super-collection:

• it converts recursively all the content into shelves
• it exports the labels as attributes (when they are valid Python identifiers and not masked)

In other words it behaves as SuperList and a SuperDict at the same time.

Related data structures and ideas

These projects contain ideas that inspired or predated super-collections.

Standard Python

• collections.namedtuple: tuples with dot notation (standard python class)
• types.SimpleNamespace: objects with arbitrary attributes (standard python class)
• All Python classes have a dict attribute, used at the foundation to implement the dot notation in the language, with the relative standard methods (__setattr__(), etc.) and functions (setattr(), etc.).
• In modern Python, the dict class has ordered keys (by insertion order) and is subclassable.

Dot notation on dictionaries

• addict (Github)
• DotMap: subclasses and MutableMapping and OrderedDict (Github)
• SuperDict: subclasses dict (Github)
• dotty_dict: wrapper (Github)

Using superlists to complement superdictionaries

• Packages that write to and read from files, such as shelve (standard), json, YAML, Beautifulsoup, etc. heavily rely on a combination of dictionaries and lists. BeautifulSoup in particular supports dot notation.
• In general, the construction of any syntactic or semantic tree requires both dictionaries and lists.