Nested dictionaries in Python

In this article I will show you what is a nested dictionary and how to work with it efficient (get ,set values).

Nested — when you have an object of objects of objects etc.

Introduction

How often you need to use nested dictionary in Python ? My experience was poor before I have a lot of work with such data base as Elastic search. It is a huge and fast “organism” if you wish. It consists of many objects like dictionaries and lists and inside of them may be again dictionaries and lists. And after you make a request you receive really huge response and you need for example get 10 levels of keys deep and get this value after this you need to assign it to another nested dictionary for 30 levels deep. Alright, you can write pure Python code but it will be ugly and if you try to format it, you can brake the rules of PEP8 etc. Or even worse if on some level the structure will be changed you will spend a lot of time to fix your code.

So here I will show the result of my fight with these problems.

If you have worked with nested dictionaries you can skip “What is a nested dictionary in Python?” paragraph and start with next one “Complex nested dictionaries”.

What is a nested dictionary in Python ?

Let’s start from the very beginning.

Standard Python dictionary — is an associative array of data with structure key-value. All the keys are unique and immutable (strings, numbers, characters), value may be any type.

Here are some examples:

shopping_bag = {"apple": 5, "banana": 10}
group_name_age = {"Robert": 25, "Emma": 31, "Mike": 19}

And if we want to get an element from these dictionaries we can simply call them by a valid key.

apples_count = shopping_bag["apple"]
print(apples_count)  # 5 

And we can assign new values for existing keys and write new data.

shopping_bag["banana"] -= 3  # reduce bananas count on 3 to 7
shopping_bag["banana"] = 20  # change bananas count to 20
shopping_bag["orange"] = 1  # write a new key-value pair

Alright, it was easy. Now we a bit improve our approach and make it a bit more flexible. For example we need to want to multiply count of certain fruit for it’s price taken from another prices dictionary.

# Old way
shopping_bag = {"apple": 5, "banana": 10}
prices = {"apple": 10, "banana": 25}
apples_count = shopping_bag["apple"]
apple_price = prices["apple"]
total_price = apples_count * apple_price
print(total_price)  # 50

But what if we somehow don’t have apples in prices list ?

shopping_bag = {"apple": 5, "banana": 10}
prices = {"banana": 25}
apples_count = shopping_bag["apple"]
apple_price = prices["apple"] # Here will be KeyError and stop
total_price = apples_count * apple_price
print(total_price)

We can simply handle it by .get() function of dict class, where we can specify default value.

shopping_bag = {"apple": 5, "banana": 10}
prices = {"banana": 25}
apples_count = prices.get("apple", 0)
apple_price = prices.get("apple", 0)
total_price = apples_count * apple_price
print(total_price)  # 0

So now we have everything to start working with nested dictionary. Example:

users_information = {"John": {"age": 25,
                              "gender": "male"},
                     "Rose": {"gender": "female"}
                     }

Here we have a dictionary with 2 string keys and 2 dictionaries as values where each one has 2 string keys and 2 values int and str. How to get John’s age value ?

age = users_information["John"]["age"]
print(age)  # 25
age = users_information.get("Rose").get("age", None)
print(age)  # None

So it was simple example of a nested dictionary with depth = 2 and simple approach to work with it. Now we can move forward.

Complex nested dictionaries

For example you have received such response of data.

users_data={
  1: {
    "general": {
      "age": 45,
      "name": "John"
    },
    "car_data": {
      "general": {
        "type": "mercedes",
        "model": "GLE"
      }
    },
    "additional": {
      "first_commit": {
        "list_of_attempts": [
          {
            "type": "main"
          }
        ]
      }
    }
  }
}

We can also see it via this cool tool.

View for our data

And for example we need to assign value “developer_branch” to key “type”.

users_data[1]["additional"]["first_commit"]["list_of_attempts"][0]["type"] = "developer_branch"

So that’s it. Huge and ugly line in your code. In case if you put it in iterator with some indents it will be moved righter and you can easily brake the 100 chars rule of PEP8. But even without PEP8 this code is ugly and if you have to call such things more than once you must think “How can I simplify and beautify it ?”.

The first thing that I have found out is to use a reduce function.

from functools import reduce
data={'apple':{'data':{'price':{'actual':5, 'discounted': 4}}}}

path = ['apple', 'data', 'price', 'actual']
actual_price = reduce(dict.get, path, data)
print(actual_price)  # 5

But here we have important disadvantage. We can only get value, not assign it. So if you need only access to deep nested dictionaries you can use reduce function from functools.

And it will not help you if you have at least one list on your way, so this function will break because of calling dict.get function.

How to deal with it ? Well, there is a cool package named glom.

Sure there are nothing “super genius”, just a cool tool that you can write by your own, but for what if it’s already written ?

pip install glom

You can easily handle getting and setting operations with it. Get example:

import glom
path = glom.Path(
    1, "additional",
    "first_commit",
    "list_of_attempts",
    0, "type")
glom.assign(user_data, path, "developer_branch")

It was example when you have integer keys in your nested dictionaries. If you have dictionary with only str keys and there might be happen lists you can use it without glom.Path instance. You just need to create a dot separated string of keys path.

import glom
data={'apple':{'data':{'price':{'actual':5, 'discounted': 4}}}}

path = "apple.data.price.actual"
glom.assign(data, path, 10)

Now let’s see how to simply get data.

import glom
data={'apple':{'data':{'price':{'actual':10, 'discounted': 4}}}}

path = "apple.data.price.actual"
actual = glom.glom(data, path)
print(actual)  # 10

It means that you can make separate list or simple dict of paths and make it like a constant. And in case you need to process some data you just need to call glom and use certain path from your collection.

That’s all magic ! Good luck.

With best wishes,

Bondarenko K., machine learning engineer