How to Create Overlay Networks inside Bitcoin Script
In this article we are going to describe an efficient filtering mechanism to allow users/applications to:
- Efficiently retrieve a subset of transactions from the UTXO set and/or the blockchain
- Enforce a specific transaction structure across inputs/outputs
Credit for this filtering technique goes to Dr. Craig Wright who outlined some of these things in the following video.
Up until this point the injection of data protocols into Bitcoin have largely been centered around the use of OP_FALSE OP_RETURN outputs. Protocols like MAP, B://, and even the Metanet protocol itself define data structures that can be attached after the use of OP_RETURN in transactions to allow applications to emerge that consume and interpret the data in useful ways. When OP_RETURN was made unbounded on BSV, we saw a Cambrian explosion of data protocols and applications emerge that made use of this functionality. This began the emergence of overlay networks on Bitcoin where layers on top of Bitcoin could begin interpreting a common dataset.
The use of OP_RETURN for data brings three downsides:
- Anything in OP_RETURN can be pruned by nodes on the network at any time
- The data in OP_RETURN cannot be consumed by Bitcoin Script itself, limiting the ability to perform validation on the data or otherwise utilize Script to interact with the data. This means we cannot enforce restrictions on who can spend an output based on the data inserted into the transaction.
- Data structures can be interrupted by attackers, where junk that follows a certain data protocol can be inserted in OP_RETURN to confuse the overlay nodes
The purposes of this article will focus on (2) and (3). To create an overlay network in Bitcoin it is important that there is some mechanism to validate that new transactions consumed by the overlay network are valid. There should also be an easy way for new nodes entering the overlay network to fetch all of the data relevant to the overlay network without having to sort through junk data. The solution was identified years ago by Satoshi Nakamoto himself in 2010 when describing Bitcoin Script:
The script is actually a predicate. It’s just an equation that evaluates to true or false. Predicate is a long and unfamiliar word so I called it script.
The receiver of a payment does a template match on the script. Currently, receivers only accept two templates: direct payment and bitcoin address. Future versions can add templates for more transaction types and nodes running that version or higher will be able to receive them. All versions of nodes in the network can verify and process any new transactions into blocks, even though they may not know how to read them.
The design supports a tremendous variety of possible transaction types that I designed years ago. Escrow transactions, bonded contracts, third party arbitration, multi-party signature, etc.
Currently there is no distinction between transaction templates across the network aside from a basic boolean between standard and nonstandard. As Bitcoin SV continues to utilize Bitcoin Script for more advanced transactions, this entire concept needs to move towards identifying templates.
To create overlay networks we propose a filtering mechanism using Bitcoin Script that loosely follows the following format:
Unlocking Script: <unlocking script for transaction template><preimage of transaction>
Locking Script: <Hash of Transaction Template> <Filter Mechanism> <OP_PUSH_TX> <Transaction Template>
To help the reader understand the concept we will outline a filtering mechanism for P2PKH transactions, the most common transaction template in use across the network. We will utilize the mechanism to enforce across a chain of transactions that they must be a P2PKH template.
For every P2PKH transaction, the locking script follows a common format:
OP_DUP OP_HASH160 <public key hash> OP_EQUALVERIFY OP_CHECKSIG
Where the public key hash is 20 bytes of data that varies for each transaction. Thus, our transaction template — the transaction format stripped of data — is the following:
OP_DUP OP_HASH160 OP_EQUALVERIFY OP_CHECKSIG
To enforce that all transactions follow this format, we will take a SHA1 hash of this data which turns out to be 464ac09fd447f94b321b0c351ae652ad39bb9c5f. We will insert this hash at the beginning of our locking script and use it in our filter check. While this hash could be of any type, SHA1 has a few distinct advantages in the creation of HMACs and because of the low compute required.
We utilize the OP_PUSH_TX technique to push the current transaction to the stack to validate that the filter checking is being performed on the current transaction. The OP_PUSH_TX technique requires us to include the preimage of the transaction in the unlocking script of a transaction. Since the previous transaction’s locking script is included in the preimage of a transaction, we can simultaneously use the preimage to enforce that every UTXO used as an input must be of the same template.
Our filter is going to be different for every transaction template we use, but it will follow a similar format:
<Get Locking Script From Preimage> <Strip Data From Template> OP_SHA1 OP_EQUALVERIFY
Stripping the data from our P2PKH template is quite simple:
OP_2 OP_SPLIT OP_1 OP_SPLIT OP_SWAP OP_SPLIT OP_NIP OP_CAT
You can use the link above to see how our filter strips the public key hash from the locking script. We can validate by appending OP_SHA1 to the script that the script template properly hashes to 464ac09fd447f94b321b0c351ae652ad39bb9c5f.
Note that in practice our hash won’t actually be this value, because our locking script is going to include the opcodes above as well as the OP_PUSH_TX script, so we will want to include those opcodes to complete our transaction template before taking our hash to compare with. As an example of bringing all of this together, see this transaction on the blockchain. Any extensions to this chain of transactions will require the same transaction template. You can build your own transaction using this golang repository.
Remember, our goal is to enforce a common transaction template across inputs and outputs:
Because the preimage of a transaction only contains a hash of the transaction outputs, every transaction can only validate that the input for the transaction contains the proper hash. Because the checking of the hash is done in Script using the OP_SHA1 opcode, it means that when syncing data for our overlay from the blockchain we can trust that every transaction in a block utilizing that hash and script template has been validated by the miners. Since Script is a predicate, we can be certain that any failed script would not be accepted by the mining network. This means that attackers can only succeed in confusing our overlay network for any outputs that have not been spent yet.
As new transactions are received by the overlay node, a simple check for the hash at the beginning of the script can be employed. As a second layer of security for unspent transactions, any transactions found can then be further checked by performing a SHA1 hash against the template (thus our use of a low-compute hashing algorithm such as SHA1). For security concerns regarding any particular hashing algorithms, extraneous hashing can be performed or other algorithms can be used.
This is the simplest filtering mechanism that can be used. Masking and other techniques can enable vastly more complicated enforcement of transaction templates and data insertion. We will outline those techniques in future articles.
