Create your own blockchain using Python (pt. 7)

New block creation and Proof-of-Work

10 min readJul 24, 2021

Up to now, we built a transaction validation system that looks at the data incoming, validates it using transaction scripts and returns either success or failure, but the transaction data was never written in the ledger. In this section, we will look at how new blocks get added to the blockchain and we will introduce Proof-of-Work (PoW) which is one of the main components of bitcoin and most cryptocurrencies.

The double-spending problem

Proof-of-Work involves having nodes solve computational problem in order to create those new blocks. The use of PoW for new block creation solves the double-spending problem and also secures the blockchain network. So, what’s double-spending and how is it a problem? Here’s the Wikipedia definition:

Double-spending is a potential flaw in a digital cash scheme in which the same single digital token can be spent more than once. Unlike physical cash, a digital token consists of a digital file that can be duplicated or falsified.

Central third parties

The double-spending problem is easily solved using central third parties

Prevention of double-spending is usually implemented using an online central trusted third party that can verify whether a token has been spent.

If we look at the blockchain we developed until now, storing it on a central server would mean that every transaction would have to go through a central authority (node) that manages inscriptions inside of this central ledger.

Since all transactions go through the central authority, it is possible for this entity to keep record of all transactions and to know the amount of money in everybody’s account.

Managing ledgers in a decentralized system

The problem of central third parties is that they represent a single point of failure from both availability and trust viewpoints. So there might be a need for ledgers to be stored inside decentralized systems and have no authority managing it. However, the task is not easy and as identified on the Wikipedia page of double-spending:

To avoid the need for a trusted third party, many servers must store identical up-to-date copies of a public transaction ledger, but as transactions are broadcast, they will arrive at each server at slightly different times. If two transactions attempt to spend the same token, each server will consider the first transaction it sees to be valid, and the other invalid. Once the servers disagree, there is no way to determine true balances, as each server’s observations are considered equally valid. Most decentralized systems solve this with a consensus algorithm, a way to bring the servers back in sync.

In his White Paper on Bitcoin, Satoshi Nakamoto describes the same problem:

We need a system for participants to agree on a single history of the order in which they were received. The payee needs proof that at the time of each transaction, the majority of nodes agreed it was the first received.

Nakamoto proposes a solution to the problem of managing ledgers in a decentralized network. New blocks are created by having nodes hash new blocks and publishing the hash to the rest of the network. The information that is hashed is the time at which the block was created, the previous block, and a summary of the block’s transaction (the merkle root). But what does that do?

The timestamp proves that the data must have existed at the time, obviously, in order to get into the hash. Each timestamp includes the previous timestamp in its hash, forming a chain, with each additional timestamp reinforcing the ones before it. The timestamp proves that the transaction data existed when the block was published in order to get into its hash.

So what is Proof-of-Work?

Now let’s say we spin up our node, it registers on the network and receives two different versions of the blockchain. How does it know which one is the correct one? We need a way for nodes to agree on a single version of the blockchain. Various consensus mechanisms exist to solve this problem and Proof-of-Work is one of them. In Bitcoin and most cryptocurrencies, the correct blockchain is the one with the most processing power invested into it. In other words, Proof-of-Work is used by block creators to show that they spent a certain amount of effort building the new block. Here is the Wikipedia definition:

Proof of work is a form of cryptographic zero-knowledge proof in which one party (the prover) proves to others (the verifiers) that a certain amount of a specific computational effort has been expended. Verifiers can subsequently confirm this expenditure with minimal effort on their part.
Proof-of-work is essentially one-CPU-one-vote. The majority decision is represented by the longest chain, which has the greatest proof-of-work effort invested in it. If a majority of CPU power is controlled by honest nodes, the honest chain will grow the fastest and outpace any competing chains. To modify a past block, an attacker would have to redo the proof-of-work of the block and all blocks after it and then catch up with and surpass the work of the honest nodes. We will show later that the probability of a slower attacker catching up diminishes exponentially as subsequent blocks are added.

As seen in the last section, here’s the process by which new transactions are written in to blockchain:

New transactions are broadcast to all nodes.
Each node collects new transactions into a block.
Each node works on finding a difficult proof-of-work for its block.
When a node finds a proof-of-work, it broadcasts the block to all nodes.
Nodes accept the block only if all transactions in it are valid and not already spent.
Nodes express their acceptance of the block by working on creating the next block in the chain, using the hash of the accepted block as the previous hash.

The puzzle

Proof-of-Work involves having nodes solve a processing demandant puzzles. Here is what this puzzle actually is. The node first constructs a new block header that contains the following information:

The previous block hash
The Merkle root of this new block
A timestamp
A nonce (a variable integer)

The node then hashes the header for different values of the nonce until the hash starts with a certain number of zeros. The difficulty of the puzzle is exponentially more difficult based on the number of leading zeros.

Once the nodes find a nonce that solves the puzzle, it keeps the noonce in the header and the new block is broadcasted to the other nodes. The other nodes can validate that the puzzle was completed successfully by hashing the header and looking at the number of leading zeros at the start of the hash.

Please remember that ultimately, we don’t really care about the puzzle itself. It doesn’t “mean” anything and it could have been different. Proof-of-Work exists in order to deter manipulation of the blockchain.

Shared memory

Until now, our code had two sections:

Node: Transaction validation
Wallet: Transaction creation

We introduce two new types of tasks: creating new blocks and validating new blocks. Both the create_new_block and the transaction_validation processes will need access to the same memory location that contains transaction information called mem_pool.

To allow this shared memory access, we will cheat a little bit and simply use files that will contain the information on disk and we will read and write from/to them. In practice, part of our transaction validation class, we create a new store method so that transactions can be stored in the mem_pool after they have been validated:

# node/transaction_validation/transaction_validation.pyfrom common.block import Block
from common.io_mem_pool import get_transactions_from_memory, store_transactions_in_memory


class Transaction:
    def __init__(self, blockchain: Block):
        self.blockchain = blockchain
        self.transaction_data = {}
        self.inputs = []
        self.outputs = []
        self.is_valid = False
        self.is_funds_sufficient = False

    def store(self):
        if self.is_valid and self.is_funds_sufficient:
            current_transactions = get_transactions_from_memory()
            current_transactions.append(self.transaction_data)
            store_transactions_in_memory(current_transactions)

In our “common” directory, we create two new methods to get transactions from memory and store them:

# common/io_mem_pool.pyimport json


FILENAME = "src/doc/mem_pool"


def get_transactions_from_memory():
    with open(FILENAME, "rb") as file_obj:
        current_mem_pool_str = file_obj.read()
        current_mem_pool_list = json.loads(current_mem_pool_str)
    return current_mem_pool_list


def store_transactions_in_memory(transactions: list):
    text = json.dumps(transactions).encode("utf-8")
    with open(FILENAME, "wb") as file_obj:
        file_obj.write(text)

New block creation

Here we will implement the code responsible for creating new blocks using Proof-of-Work. We first create a new directory in our project called “new_block_creation” and a new file of the same name. We create a new class for our Proof-of-Work algorithm called ProofOfWork:

# node/new_block_creation/new_block_creation.pyimport json
from datetime import datetime

from common.block import Block, BlockHeader
from common.io_blockchain import get_blockchain_from_memory
from common.io_mem_pool import get_transactions_from_memory
from common.merkle_tree import get_merkle_root
from common.utils import calculate_hash
from common.values import NUMBER_OF_LEADING_ZEROS


class ProofOfWork:
    def __init__(self):
        self.blockchain = get_blockchain_from_memory()
        self.new_block = None

    @staticmethod
    def get_noonce(block_header: BlockHeader) -> int:
        block_header_hash = ""
        noonce = block_header.noonce
        starting_zeros = "".join([str(0) for _ in range(NUMBER_OF_LEADING_ZEROS)])
        while not block_header_hash.startswith(starting_zeros):
            noonce = noonce + 1
            block_header_content = {
                "previous_block_hash": block_header.previous_block_hash,
                "merkle_root": block_header.merkle_root,
                "timestamp": block_header.timestamp,
                "noonce": noonce
            }
            block_header_hash = calculate_hash(json.dumps(block_header_content))
        return noonce

    def create_new_block(self):
        transactions = get_transactions_from_memory()
        if transactions:
            block_header = BlockHeader(
                merkle_root=get_merkle_root(transactions),
                previous_block_hash=self.blockchain.block_header.hash,
                timestamp=datetime.timestamp(datetime.now()),
                noonce=0
            )
            block_header.noonce = self.get_noonce(block_header)
            block_header.hash = block_header.get_hash()
            self.new_block = Block(transactions=transactions, block_header=block_header)
        else:
            raise BlockException("", "No transaction in mem_pool")

Here the create_new_block method is responsible for reading the transactions in the memory people and building a new block out of it. The whole Proof-of-Work algorithm is here in the get_noonce method. noonce’s are interated until the hash of the header starts with a certain amount of zeros. In our code, we set the number of leading zeros to be 3 but note that in the actual bitcoin this value dynamically changes in order for new blocks to take ~10 minutes to be created. The more there are nodes in the network, the faster the solution will be found hence the dynamic value.

Block broadcast

After new blocks have been created, they are broadcasted to the other nodes of the network. To support this, we create a broadcast method in our ProofOfWork class responsible to broadcast new blocks via HTTP to the nodes it knows about:

# node/new_block_creation/new_block_creation.pyimport requests

from common.io_blockchain import get_blockchain_from_memory
from common.node import Node


class OtherNode(Node):
    def __init__(self, ip: str, port: int):
        super().__init__(ip, port)

    def send_new_block(self, block: dict) -> requests.Response:
        return self.post(endpoint="block", data=block)


class ProofOfWork:
    def __init__(self):
        self.blockchain = get_blockchain_from_memory()
        self.new_block = None

    def broadcast(self):
        node_list = [OtherNode("127.0.0.1", 5000)]
        for node in node_list:
            block_content = {
                "block": {
                    "header": self.new_block.block_header.to_dict,
                    "transactions": self.new_block.transactions
                }
            }
            node.send_new_block(block_content)

Block validation

We add a new interface to our flask server so that nodes can receive new blocks from other nodes.

# node/main.pyfrom flask import Flask, request

from common.initialize_blockchain import initialize_blockchain
from common.io_blockchain import get_blockchain_from_memory
from node.new_block_validation.new_block_validation import NewBlock, NewBlockException
from node.transaction_validation.transaction_validation import TransactionException

app = Flask(__name__)
initialize_blockchain()


@app.route("/block", methods=['POST'])
def validate_block():
    content = request.json
    blockchain_base = get_blockchain_from_memory()
    try:
        new_block = NewBlock(blockchain_base)
        new_block.receive(new_block=content["block"])
        new_block.validate()
        new_block.add()
    except (NewBlockException, TransactionException) as new_block_exception:
        return f'{new_block_exception}', 400
    return "Transaction success", 200

This means that we have a new endpoint called “block” that can be reached by HTTP POST calls. Whenever a request is received, an instance of NewBlock is created and three methods are called: receive, validate and add. We create a new file called new_block_validation that will contain this class. Let’s look at the receive method first:

# node/new_block_validation/new_block_validation.pyfrom common.block import Block, BlockHeader


class NewBlockException(Exception):
    def __init__(self, expression, message):
        self.expression = expression
        self.message = message


class NewBlock:
    def __init__(self, blockchain: Block):
        self.blockchain = blockchain
        self.new_block = None

    def receive(self, new_block: dict):
        block_header = BlockHeader(**new_block["header"])
        self.new_block = Block(transactions=new_block["transactions"], block_header=block_header)
        try:
            assert self.blockchain.block_header.hash == self.new_block.block_header.previous_block_hash
        except AssertionError:
            print("Previous block provided is not the most recent block")
            raise NewBlockException("", "Previous block provided is not the most recent block")

Here we simply create a new instance of a BlockHeader and of a Block. We validate that the blockchain stored in memory corresponds to the received block’s “previous block hash”. Now let’s look at the validate method:

# node/new_block_validation/new_block_validation.pyfrom common.block import Block
from common.values import NUMBER_OF_LEADING_ZEROS
from node.transaction_validation.transaction_validation import Transaction
class NewBlock:
    def __init__(self, blockchain: Block):
        self.blockchain = blockchain
        self.new_block = None

    def validate(self):
        self._validate_hash()
        self._validate_transactions()

    def _validate_hash(self):
        new_block_hash = self.new_block.block_header.get_hash()
        number_of_zeros_string = "".join([str(0) for _ in range(NUMBER_OF_LEADING_ZEROS)])
        try:
            assert new_block_hash.startswith(number_of_zeros_string)
        except AssertionError:
            print('Proof of work validation failed')
            raise NewBlockException("", "Proof of work validation failed")

    def _validate_transactions(self):
        for transaction in self.new_block.transactions:
            transaction_validation = Transaction(self.blockchain)
            transaction_validation.receive(transaction=transaction)
            transaction_validation.validate()
            transaction_validation.validate_funds()

Here, we validate that the proof of work was done successefully and that each transaction inside of the new block is valid. To do so, we re-use the same validate and validate_funds methods that we created in section 5 of this tutorial series.

Once the new block hash and transactions are validated, it can be added to the blockchain:

# node/new_block_validation/new_block_validation.pyfrom common.block import Block
from common.io_blockchain import store_blockchain_in_memory


class NewBlock:
    def __init__(self, blockchain: Block):
        self.blockchain = blockchain
        self.new_block = None

    def add(self):
        self.new_block.previous_block = self.blockchain
        store_blockchain_in_memory(self.new_block)

I hope you are proud

If you have made it up to here, you have built a process to validate transactions and securely add new blocks to a blockchain in a decentralized network. I hope you are proud. In the next section, we will actually look at how multiple nodes can be deployed and work together at maintaining the blockchain network.

Code repository

https://github.com/gruyaume/my-blockchain/tree/pow