Create your own blockchain using Python (pt. 9)

A Distributed Network

8 min readNov 3, 2021

It’s one thing to build a standalone blockchain but it’s an entirely different ambition to build a fully decentralized cryptocurrency network from scratch. The main idea of decentralization in the context of cryptocurrencies is that anybody, anywhere, with only an internet connection, can spin up a new node and join the network. There is no hierarchy in a blockchain network, all nodes are equivalent. You can have only five nodes or as high as a million and the network will behave the same.

In pt. 6 we implemented the possibility of having transactions being sent to our node via HTTP and for transactions to be broadcasted to other nodes. However we cut come corners by hardcoding other node addresses and we completely ignored the question of “How do nodes join the network”. Here we will build on what we did in pt. 6 and hopefully build a truly distributable blockchain.

Decentralization

You might have heard at some point that Bitcoin is a decentralized network. But what does that mean? Before going to deep, let’s start with another example of a decentralized network: The Internet. Here are some key points about it:

The Internet (or any network) is made of nodes and links that connect those nodes together.
Each node in the internet is a network equipement called a router.
Each router is functionally equivalent to all other routers.
Data transits in the internet network in the forms of packets. Packets are small chunks of data with labels on them that indicate their source and destination addresses.
Each router maintains a routing table. A routing table records the paths that packets should take to reach every destination that the router is responsible for.
The main function executed by a router is called routing. When a packet comes in the router, the router looks at the destination address and sends it to a different router based on the routing table.
The main protocol at the heart of the internet is called TCP/IP. This protocol defines the packet structures as well as the expected behaviour of routers.

Bitcoin (and other cryptocurrencies) have a lot in common with the Internet. Bitcoin, just like the Internet, is a network. Each Bitcoin node is a network node. Communications between each node is set by standard protocols. Obviously the main functions executed by nodes are different from routers. Here’s what Wikipedia has to say about decentralization in the context of blockchain:

Every node in a decentralized system has a copy of the blockchain. Data quality is maintained by massive database replication and computational trust. No centralized “official” copy exists and no user is “trusted” more than any other. Transactions are broadcast to the network using software. Messages are delivered on a best-effort basis. Mining nodes validate transactions, add them to the block they are building, and then broadcast the completed block to other nodes. Blockchains use various time-stamping schemes, such as proof-of-work, to serialize changes.

Node discovery

When a new node wants to join the network, it must be aware of at least one other node to advertise to it that it wants to join the party. To enable this, we hardcode one known node in our code and the rest of this node discovery will be driven by two types of messages:

Advertisement: A node advertises to another node that it exists and wants to join the network
Known node requests: A node asks another node for its known nodes.

Now here is how a protocol can be built using those two types of messages. In this example, we have a network of nodes that includes Nodes A, B and C and we have a fourth node (Node X) that would like to be added to the network. Node C is hardcoded in the software as being a known node.

Advertisement (1): Node X advertises to Node C that it wants to join the network.
Advertisement (2): Node C responds to Node X with Accept and adds Node C to its list of known nodes.
Known nodes request (1): Node X asks Node C for its list of known Nodes
Known nodes request (2): Node C responds with the list of known nodes. Node X adds those known nodes to its list
Advertisement (3): Node C advertises to Nodes A and B that it wants to join the network
Advertisement (4): Node A and B accept and add Node C to their list of known nodes.

Note that the protocol we described here is not the actual Bitcoin protocol, or any other cryptocurrency’s for that matter, we’re simply inventing something that will fit our purpose.

Python implementation

Let’s implement each of those interactions in our code. We start by creating a Network class. When a new node is initialized, an instance of this class will be created. Note that we set a default node that we hardcode. It is important to have this default one since when you spin up a new node, it must at least be aware of one node to which it will connect.

# src/common/network.pyimport json

from common.node import Node


class Network:

    KNOWN_NODES_FILE = 'src/doc/known_nodes.json'
    FIRST_KNOWN_NODE_HOSTNAME = "127.0.0.1:5000"

    def __init__(self, node: Node):
        self.node = node
        self.initialize_known_nodes_file()

    def initialize_known_nodes_file(self):
        print("Initializing known nodes file")
        initial_known_node = Node(hostname=self.FIRST_KNOWN_NODE_HOSTNAME)
        with open(self.KNOWN_NODES_FILE, "w") as jsonFile:
            json.dump([initial_known_node.dict], jsonFile)

Here we create a join_network method that will be called once a node is spun up. This method is the code implementation of the protocol we described above.

# src/common/network.pyimport json

from common.node import Node
from common.io_blockchain import store_blockchain_dict_in_memory
from common.initialize_default_blockchain import initialize_default_blockchain


class Network:

    KNOWN_NODES_FILE = 'src/doc/known_nodes.json'
    FIRST_KNOWN_NODE_HOSTNAME = "127.0.0.1:5000"

    def join_network(self):
        print("Joining network")
        if self.other_nodes_exist:
            self.advertise_to_all_known_nodes()
            known_nodes_of_known_node = self.ask_known_nodes_for_their_known_nodes()
            self.store_nodes(known_nodes_of_known_node)
            self.advertise_to_all_known_nodes()
            self.initialize_blockchain()
        else:
            print("No other node exists. This could be caused by a network issue or because we are the first node out here.")
            initialize_default_blockchain()

Now let’s look at each of the method that is called inside of join_network . We first validate that other nodes exist (that we aren’t deploying the first node).

# src/common/network.pyclass Network:

    @property
    def known_nodes(self) -> [Node]:
        with open(self.KNOWN_NODES_FILE) as f:
            nodes = json.load(f)
            known_nodes = [Node(hostname=node["hostname"]) for node in nodes]
        return known_nodes

    @property
    def other_nodes_exist(self) -> bool:
        if len(self.known_nodes) == 0:
            return False
        elif len(self.known_nodes) == 1 and self.known_nodes[0].hostname == self.node.hostname:
            return False
        else:
            return True

If indeed other nodes exist, we will execute the following methods: advertise_to_all_known_nodes, ask_known_nodes_for_their_known_nodes, store_nodes, advertise_to_all_known_nodes again, and initialize_blockchain. Here is the code implementation of those:

# src/common/network.pyclass Network:

    def advertise_to_all_known_nodes(self):
        print("Advertising to all known nodes")
        for node in self.known_nodes:
            if node.hostname != self.node.hostname:
                node.advertise(self.node.hostname)

    def ask_known_nodes_for_their_known_nodes(self) -> list:
        print("Asking known nodes for their own known nodes")
        known_nodes_of_known_nodes = []
        for currently_known_node in self.known_nodes:
            known_nodes_of_known_node = currently_known_node.known_node_request(self.node.hostname)
            for node in known_nodes_of_known_node:
                known_nodes_of_known_nodes.append(Node(node["hostname"]))
        return known_nodes_of_known_nodes

    @property
    def known_nodes(self) -> [Node]:
        with open(self.KNOWN_NODES_FILE) as f:
            nodes = json.load(f)
            known_nodes = [Node(hostname=node["hostname"]) for node in nodes]
        return known_nodes

    def store_new_node(self, new_node: Node):
        print(f"Storing new node: {new_node.hostname}")
        with open(self.KNOWN_NODES_FILE, "r+") as f:
            current_nodes_json = json.load(f)
            current_nodes = [Node(hostname=node["hostname"]) for node in current_nodes_json]
            if new_node not in current_nodes:
                current_nodes_json.append(new_node.dict)
                f.seek(0)
                json.dump(current_nodes_json, f)

    def store_nodes(self, nodes: [Node]):
        for node in nodes:
            self.store_new_node(node)

    def initialize_blockchain(self):
        longest_blockchain = self.get_longest_blockchain()
        store_blockchain_dict_in_memory(longest_blockchain)

    def get_longest_blockchain(self):
        longest_blockchain_size = 0
        longest_blockchain = None
        for node in self.known_nodes:
            if node.hostname != self.node.hostname:
                blockchain = node.get_blockchain()
                blockchain_length = len(blockchain)
                if blockchain_length > longest_blockchain_size:
                    longest_blockchain_size = blockchain_length
                    longest_blockchain = blockchain
        return longest_blockchain

And if no other node exist and that we are indeed deploying the first node, we initialize a default blockchain.

Now on the other side of things we also need to enhance our code so that an existing node can answer back to a new comer. So part of this same Network class, we implement a return_known_nodes method.

# src/common/network.pyclass Network:

    KNOWN_NODES_FILE = 'src/doc/known_nodes.json'
    FIRST_KNOWN_NODE_HOSTNAME = "127.0.0.1:5000"

    def return_known_nodes(self) -> []:
        with open(self.KNOWN_NODES_FILE) as f:
            current_nodes_json = json.load(f)
        return current_nodes_json

We also must implement new endpoints to our node’s flask app:

# src/node/main.py@app.route("/new_node_advertisement", methods=['POST'])
def new_node_advertisement():
    content = request.json
    hostname = content["hostname"]
    try:
        new_node = Node(hostname)
        network.store_new_node(new_node)
    except TransactionException as transaction_exception:
        return f'{transaction_exception}', 400
    return "New node advertisement success", 200


@app.route("/known_node_request", methods=['GET'])
def known_node_request():
    return jsonify(network.return_known_nodes())

Blockchain Propagation

The subject of blockchain propagation was already addressed in pt.6. However one key point that we left out is that when our nodes receive a new block that was created by a different node, we must broadcast it to the rest of the network so that each of the other nodes of the network have the same version of the blockchain.

All we must modify to support this is implement a broadcast method in our new_block_validation NewBlock class.

# src/new_block_validation/new_block_validation.pyclass NewBlock:
    def broadcast(self):
        node_list = self.network.known_nodes
        for node in node_list:
            if node.hostname != self.network.node.hostname:
                block_content = {
                    "block": {
                        "header": self.new_block.block_header.to_dict,
                        "transactions": self.new_block.transactions
                    }
                }
                node.send_new_block(block_content)

And this method must be called by the flask app’s validate_block method, right after the block is validated and added to memory:

# src/node/main.py@app.route("/block", methods=['POST'])
def validate_block():
    content = request.json
    blockchain_base = get_blockchain_from_memory()
    try:
        block = NewBlock(blockchain_base, network)
        block.receive(new_block=content["block"])
        block.validate()
        block.add()
        block.broadcast()
    except (NewBlockException, TransactionException) as new_block_exception:
        return f'{new_block_exception}', 400
    return "Transaction success", 200

A truly truly distributed network

All right! We improved our blockchain so that it could be truly distributed. But is it really distributed? Not really. If you are following this guide, you probably have only one instance of it running on your laptop. To continue the internet analogy, it’s like having one router setup. It’s awesome but one node isn’t really a network. In the next section I will show you how you can actually deploy multiple instances of our blockchain, anywhere you want, and actually validate that it is working as intended.