bitcoin-bips/bip-0380.mediawiki

<pre>
  BIP: 380
  Layer: Applications
  Title: Output Script Descriptors General Operation
  Author: Pieter Wuille <pieter@wuille.net>
          Andrew Chow <andrew@achow101.com>
  Comments-Summary: No comments yet.
  Comments-URI: https://github.com/bitcoin/bips/wiki/Comments:BIP-0380
  Status: Draft
  Type: Informational
  Created: 2021-06-27
  License: BSD-2-Clause
</pre>

==Abstract==

Output Script Descriptors are a simple language which can be used to describe collections of output scripts.
There can be many different descriptor fragments and functions.
This document describes the general syntax for descriptors, descriptor checksums, and common expressions.

==Copyright==

This BIP is licensed under the BSD 2-clause license.

==Motivation==

Bitcoin wallets traditionally have stored a set of keys which are later serialized and mutated to produce the output scripts that the wallet watches and the addresses it provides to users.
Typically backups have consisted of solely the private keys, nowadays primarily in the form of BIP 39 mnemonics.
However this backup solution is insuffient, especially since the introduction of Segregated Witness which added new output types.
Given just the private keys, it is not possible for restored wallets to know which kinds of output scripts and addresses to produce.
This has lead to incompatibilities between wallets when restoring a backup or exporting data for a watch only wallet.

Further complicating matters are BIP 32 derivation paths.
Although BIPs 44, 49, and 84 have specified standard BIP 32 derivation paths for different output scripts and addresses, not all wallets support them nor use those derivation paths.
The lack of derivation path information in these backups and exports leads to further incompatibilities between wallets.

Current solutions to these issues have not been generic and can be viewed as being layer violations.
Solutions such as introducing different version bytes for extended key serialization both are a layer violation (key derivation should be separate from script type meaning) and specific only to a particular derivation path and script type.

Output Script Descriptors introduces a generic solution to these issues.
Script types are specified explicitly through the use of Script Expressions.
Key derivation paths are specified explicitly in Key Expressions.
These allow for creating wallet backups and exports which specify the exact scripts, subscripts (redeemScript, witnessScript, etc.), and keys to produce.
With the general structure specified in this BIP, new Script Expressions can be introduced as new script types are added.
Lastly, the use of common terminology and existing standards allow for Output Script Descriptors to be engineer readable so that the results can be understood at a glance.

==Specification==

Descriptors consist of several types of expressions.
The top level expression is a <tt>SCRIPT</tt>.
This expression may be followed by <tt>#CHECKSUM</tt>, where <tt>CHECKSUM</tt> is an 8 character alphanumeric descriptor checksum.
Although the checksum is optional for parsing, applications may choose to reject descriptors that do not contain a checksum.

===Script Expressions===

Script Expressions (denoted <tt>SCRIPT</tt>) are expressions which correspond directly with a Bitcoin script.
These expressions are written as functions and take arguments.
Such expressions have a script template which is filled with the arguments correspondingly.
Expressions are written with a human readable identifier string with the arguments enclosed with parentheses.
The identifier string should be alphanumeric and may include underscores.

The arguments to a script expression are defined by that expression itself.
They could be a script expression, a key expression, or some other expression entirely.

===Key Expressions===

A common expression used as an argument to script expressions are key expressions (denoted <tt>KEY</tt>).
These represent a public or private key and, optionally, information about the origin of that key.
Key expressions can only be used as arguments to script expressions.

Key expressions consist of:
* Optionally, key origin information, consisting of:
** An open bracket <tt>[</tt>
** Exactly 8 hex characters for the fingerprint of the key where the derivation starts (see BIP 32 for details)
** Followed by zero or more <tt>/NUM</tt> or <tt>/NUMh</tt>  path elements to indicate the unhardened or hardened derivation steps between the fingerprint and the key that follows.
** A closing bracket <tt>]</tt>
* Followed by the actual key, which is either:
** A hex encoded public key, which depending the script expression, may be either:
*** 66 hex character string beginning with <tt>02</tt> or <tt>03</tt> representing a compressed public key
*** 130 hex character string beginning with <tt>04</tt> representing an uncompressed public key
** A [[https://en.bitcoin.it/wiki/Wallet_import_format|WIF]] encoded private key
** <tt>xpub</tt> encoded extended public key or <tt>xprv</tt> encoded extended private key (as defined in BIP 32)
*** Followed by zero or more <tt>/NUM</tt> or <tt>/NUMh</tt> path elements indicating BIP 32 derivation steps to be taken after the given extended key.
*** Optionally followed by a single <tt>/*</tt> or <tt>/*h</tt> final step to denote all direct unhardened or hardened children.

If the <tt>KEY</tt> is a BIP 32 extended key, before output scripts can be created, child keys must be derived using the derivation information that follows the extended key.
When the final step is <tt>/*</tt> or <tt>/*'</tt>, an output script will be produced for every child key index.
The derived key must be not be serialized as an uncompressed public key.
Script Expressions may have further requirements on how derived public keys are serialized for script creation.

In the above specification, the hardened indicator <tt>h</tt> may be replaced with alternative hardened indicators of <tt>H</tt> or <tt>'</tt>.

====Normalization of Key Expressions with Hardened Derivation====

When a descriptor is exported without private keys, it is necessary to do additional derivation to remove any intermediate hardened derivation steps for the exported descriptor to be useful.
The exporter should derive the extended public key at the last hardened derivation step and use that extended public key as the key in the descriptor.
The derivation steps that were taken to get to that key must be added to the previous key origin information.
If there is no key origin information, then one must be added for the newly derived extended public key.
If the final derivation is hardened, then it is not necessary to do additional derivation.

===Character Set===

The expressions used in descriptors must only contain characters within this character set so that the descriptor checksum will work.

The allowed characters are:
<pre>
0123456789()[],'/*abcdefgh@:$%{}
IJKLMNOPQRSTUVWXYZ&+-.;<=>?!^_|~
ijklmnopqrstuvwxyzABCDEFGH`#"\<space>
</pre>
Note that <tt><space></tt> on the last line is a space character.

This character set is written as 3 groups of 32 characters in this specific order so that the checksum below can identify more errors.
The first group are the most common "unprotected" characters (i.e. things such as hex and keypaths that do not already have their own checksums).
Case errors cause an offset that is a multiple of 32 while as many alphabetic characters are in the same group while following the previous restrictions.

===Checksum===

Following the top level script expression is a single octothorpe (<tt>#</tt>) followed by the 8 character checksum.
The checksum is an error correcting checksum similar to bech32.

The checksum has the following properties:
* Mistakes in a descriptor string are measured in "symbol errors". The higher the number of symbol errors, the harder it is to detect:
** An error substituting a character from <tt>0123456789()[],'/*abcdefgh@:$%{}</tt> for another in that set always counts as 1 symbol error.
*** Note that hex encoded keys are covered by these characters. Extended keys (<tt>xpub</tt> and <tt>xprv</tt>) use other characters too, but also have their own checksum mechanism.
*** <tt>SCRIPT</tt> expression function names use other characters, but mistakes in these would generally result in an unparsable descriptor.
** A case error always counts as 1 symbol error.
** Any other 1 character substitution error counts as 1 or 2 symbol errors.
* Any 1 symbol error is always detected.
* Any 2 or 3 symbol error in a descriptor of up to 49154 characters is always detected.
* Any 4 symbol error in a descriptor of up to 507 characters is always detected.
* Any 5 symbol error in a descriptor of up to 77 characters is always detected.
* Is optimized to minimize the chance of a 5 symbol error in a descriptor up to 387 characters is undetected
* Random errors have a chance of 1 in 2<super>40</super> of being undetected.

The checksum itself uses the same character set as bech32: <tt>qpzry9x8gf2tvdw0s3jn54khce6mua7l</tt>

Valid descriptor strings with a checksum must pass the criteria for validity specified by the Python3 code snippet below.
The function <tt>descsum_check</tt> must return true when its argument <tt>s</tt> is a descriptor consisting in the form <tt>SCRIPT#CHECKSUM</tt>.

<pre>
INPUT_CHARSET = "0123456789()[],'/*abcdefgh@:$%{}IJKLMNOPQRSTUVWXYZ&+-.;<=>?!^_|~ijklmnopqrstuvwxyzABCDEFGH`#\"\\ "
CHECKSUM_CHARSET = "qpzry9x8gf2tvdw0s3jn54khce6mua7l"
GENERATOR = [0xf5dee51989, 0xa9fdca3312, 0x1bab10e32d, 0x3706b1677a, 0x644d626ffd]

def descsum_polymod(symbols):
    """Internal function that computes the descriptor checksum."""
    chk = 1
    for value in symbols:
        top = chk >> 35
        chk = (chk & 0x7ffffffff) << 5 ^ value
        for i in range(5):
            chk ^= GENERATOR[i] if ((top >> i) & 1) else 0
    return chk

def descsum_expand(s):
    """Internal function that does the character to symbol expansion"""
    groups = []
    symbols = []
    for c in s:
        if not c in INPUT_CHARSET:
            return None
        v = INPUT_CHARSET.find(c)
        symbols.append(v & 31)
        groups.append(v >> 5)
        if len(groups) == 3:
            symbols.append(groups[0] * 9 + groups[1] * 3 + groups[2])
            groups = []
    if len(groups) == 1:
        symbols.append(groups[0])
    elif len(groups) == 2:
        symbols.append(groups[0] * 3 + groups[1])
    return symbols

def descsum_check(s):
    """Verify that the checksum is correct in a descriptor"""
    if s[-9] != '#':
        return False
    if not all(x in CHECKSUM_CHARSET for x in s[-8:]):
        return False
    symbols = descsum_expand(s[:-9]) + [CHECKSUM_CHARSET.find(x) for x in s[-8:]]
    return descsum_polymod(symbols) == 1
</pre>

This implements a BCH code that has the properties described above.
The entire descriptor string is first processed into an array of symbols.
The symbol for each character is its position within its group.
After every 3rd symbol, a 4th symbol is inserted which represents the group numbers combined together.
This means that a change that only affects the position within a group, or only a group number change, will only affect a single symbol.

To construct a valid checksum given a script expression, the code below can be used:

<pre>
def descsum_create(s):
    """Add a checksum to a descriptor without"""
    symbols = descsum_expand(s) + [0, 0, 0, 0, 0, 0, 0, 0]
    checksum = descsum_polymod(symbols) ^ 1
    return s + '#' + ''.join(CHECKSUM_CHARSET[(checksum >> (5 * (7 - i))) & 31] for i in range(8))

</pre>

==Test Vectors==

The following tests cover the checksum and character set:

* Valid checksum: <tt>raw(deadbeef)#89f8spxm</tt>
* No checksum: <tt>raw(deadbeef)</tt>
* Missing checksum: <tt>raw(deadbeef)#</tt>
* Too long checksum (9 chars): <tt>raw(deadbeef)#89f8spxmx</tt>
* Too short checksum (7 chars): <tt>raw(deadbeef)#89f8spx</tt>
* Error in payload: <tt>raw(deedbeef)#89f8spxm</tt>
* Error in checksum: <tt>raw(deedbeef)##9f8spxm</tt>
* Invalid characters in payload: <tt>raw(Ü)#00000000</tt>

The following tests cover key expressions: 

Valid expressions:

* Compressed public key: <tt>0260b2003c386519fc9eadf2b5cf124dd8eea4c4e68d5e154050a9346ea98ce600</tt>
* Uncompressed public key: <tt>04a34b99f22c790c4e36b2b3c2c35a36db06226e41c692fc82b8b56ac1c540c5bd5b8dec5235a0fa8722476c7709c02559e3aa73aa03918ba2d492eea75abea235</tt>
* Public key with key origin: <tt>[deadbeef/0h/0h/0h]0260b2003c386519fc9eadf2b5cf124dd8eea4c4e68d5e154050a9346ea98ce600</tt>
* Public key with key origin (<tt>'</tt> as hardened indicator): <tt>[deadbeef/0'/0'/0']0260b2003c386519fc9eadf2b5cf124dd8eea4c4e68d5e154050a9346ea98ce600</tt>
* Public key with key origin (mixed hardened indicator): <tt>[deadbeef/0'/0h/0']0260b2003c386519fc9eadf2b5cf124dd8eea4c4e68d5e154050a9346ea98ce600</tt>
* WIF uncompressed private key <tt>5KYZdUEo39z3FPrtuX2QbbwGnNP5zTd7yyr2SC1j299sBCnWjss</tt>
* WIF compressed private key <tt>L4rK1yDtCWekvXuE6oXD9jCYfFNV2cWRpVuPLBcCU2z8TrisoyY1</tt>
* Extended public key: <tt>xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL</tt>
* Extended public key with key origin: <tt>[deadbeef/0h/1h/2h]xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL</tt>
* Extended public key with derivation: <tt>[deadbeef/0h/1h/2h]xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL/3/4/5</tt>
* Extended public key with derivation and children: <tt>[deadbeef/0h/1h/2h]xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL/3/4/5/*</tt>
* Extended public key with hardened derivation and unhardened children: <tt>xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL/3h/4h/5h/*</tt>
* Extended public key with hardened derivation and children: <tt>xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL/3h/4h/5h/*h</tt>
* Extended public key with key origin, hardened derivation and children: <tt>[deadbeef/0h/1h/2]xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL/3h/4h/5h/*h</tt>
* Extended private key: <tt>xprvA1RpRA33e1JQ7ifknakTFpgNXPmW2YvmhqLQYMmrj4xJXXWYpDPS3xz7iAxn8L39njGVyuoseXzU6rcxFLJ8HFsTjSyQbLYnMpCqE2VbFWc</tt>
* Extended private key with key origin: <tt>[deadbeef/0h/1h/2h]xprvA1RpRA33e1JQ7ifknakTFpgNXPmW2YvmhqLQYMmrj4xJXXWYpDPS3xz7iAxn8L39njGVyuoseXzU6rcxFLJ8HFsTjSyQbLYnMpCqE2VbFWc</tt>
* Extended private key with derivation: <tt>[deadbeef/0h/1h/2h]xprvA1RpRA33e1JQ7ifknakTFpgNXPmW2YvmhqLQYMmrj4xJXXWYpDPS3xz7iAxn8L39njGVyuoseXzU6rcxFLJ8HFsTjSyQbLYnMpCqE2VbFWc/3/4/5</tt>
* Extended private key with derivation and children: <tt>[deadbeef/0h/1h/2h]xprvA1RpRA33e1JQ7ifknakTFpgNXPmW2YvmhqLQYMmrj4xJXXWYpDPS3xz7iAxn8L39njGVyuoseXzU6rcxFLJ8HFsTjSyQbLYnMpCqE2VbFWc/3/4/5/*</tt>
* Extended private key with hardened derivation and unhardened children: <tt>xprvA1RpRA33e1JQ7ifknakTFpgNXPmW2YvmhqLQYMmrj4xJXXWYpDPS3xz7iAxn8L39njGVyuoseXzU6rcxFLJ8HFsTjSyQbLYnMpCqE2VbFWc/3h/4h/5h/*</tt>
* Extended private key with hardened derivation and children: <tt>xprvA1RpRA33e1JQ7ifknakTFpgNXPmW2YvmhqLQYMmrj4xJXXWYpDPS3xz7iAxn8L39njGVyuoseXzU6rcxFLJ8HFsTjSyQbLYnMpCqE2VbFWc/3h/4h/5h/*h</tt>
* Extended private key with key origin, hardened derivation and children: <tt>[deadbeef/0h/1h/2]xprvA1RpRA33e1JQ7ifknakTFpgNXPmW2YvmhqLQYMmrj4xJXXWYpDPS3xz7iAxn8L39njGVyuoseXzU6rcxFLJ8HFsTjSyQbLYnMpCqE2VbFWc/3h/4h/5h/*h</tt>

Invalid expressiosn:

* Children indicator in key origin: <tt>[deadbeef/0h/0h/0h/*]0260b2003c386519fc9eadf2b5cf124dd8eea4c4e68d5e154050a9346ea98ce600</tt>
* Trailing slash in key origin: <tt>[deadbeef/0h/0h/0h/]0260b2003c386519fc9eadf2b5cf124dd8eea4c4e68d5e154050a9346ea98ce600</tt>
* Too short fingerprint: <tt>[deadbef/0h/0h/0h]0260b2003c386519fc9eadf2b5cf124dd8eea4c4e68d5e154050a9346ea98ce600</tt>
* Too long fingerprint: <tt>[deadbeeef/0h/0h/0h]0260b2003c386519fc9eadf2b5cf124dd8eea4c4e68d5e154050a9346ea98ce600</tt>
* Invalid hardened indicators: <tt>[deadbeeef/0f/0f/0f]0260b2003c386519fc9eadf2b5cf124dd8eea4c4e68d5e154050a9346ea98ce600</tt>
* Invalid hardened indicators: <tt>[deadbeeef/0H/0H/0H]0260b2003c386519fc9eadf2b5cf124dd8eea4c4e68d5e154050a9346ea98ce600</tt>
* Invalid hardened indicators: <tt>[deadbeeef/-0/-0/-0]0260b2003c386519fc9eadf2b5cf124dd8eea4c4e68d5e154050a9346ea98ce600</tt>
* Private key with derivation: <tt>L4rK1yDtCWekvXuE6oXD9jCYfFNV2cWRpVuPLBcCU2z8TrisoyY1/0</tt>
* Private key with derivation children: <tt>L4rK1yDtCWekvXuE6oXD9jCYfFNV2cWRpVuPLBcCU2z8TrisoyY1/*</tt>
* Derivation index out of range: <tt>xprv9s21ZrQH143K31xYSDQpPDxsXRTUcvj2iNHm5NUtrGiGG5e2DtALGdso3pGz6ssrdK4PFmM8NSpSBHNqPqm55Qn3LqFtT2emdEXVYsCzC2U/2147483648)", "pkh(xpub661MyMwAqRbcFW31YEwpkMuc5THy2PSt5bDMsktWQcFF8syAmRUapSCGu8ED9W6oDMSgv6Zz8idoc4a6mr8BDzTJY47LJhkJ8UB7WEGuduB/2147483648</tt>
* Invalid derivation index: <tt>xprv9s21ZrQH143K31xYSDQpPDxsXRTUcvj2iNHm5NUtrGiGG5e2DtALGdso3pGz6ssrdK4PFmM8NSpSBHNqPqm55Qn3LqFtT2emdEXVYsCzC2U/1aa)", "pkh(xpub661MyMwAqRbcFW31YEwpkMuc5THy2PSt5bDMsktWQcFF8syAmRUapSCGu8ED9W6oDMSgv6Zz8idoc4a6mr8BDzTJY47LJhkJ8UB7WEGuduB/1aa</tt>
* Multiple key origins: <tt>[aaaaaaaa][aaaaaaaa]xprv9s21ZrQH143K31xYSDQpPDxsXRTUcvj2iNHm5NUtrGiGG5e2DtALGdso3pGz6ssrdK4PFmM8NSpSBHNqPqm55Qn3LqFtT2emdEXVYsCzC2U/2147483647'/0</tt>
* Missing key origin start: <tt>aaaaaaaa]xprv9s21ZrQH143K31xYSDQpPDxsXRTUcvj2iNHm5NUtrGiGG5e2DtALGdso3pGz6ssrdK4PFmM8NSpSBHNqPqm55Qn3LqFtT2emdEXVYsCzC2U/2147483647'/0</tt>
* Non hex fingerprint: <tt>[gaaaaaaa]xprv9s21ZrQH143K31xYSDQpPDxsXRTUcvj2iNHm5NUtrGiGG5e2DtALGdso3pGz6ssrdK4PFmM8NSpSBHNqPqm55Qn3LqFtT2emdEXVYsCzC2U/2147483647'/0</tt>
* Key origin with no public key: <tt>[deadbeef]</tt>

==Backwards Compatibility==

Output script descriptors are an entirely new language which is not compatible with any existing software.
However many components of the expressions reuse encodings and serializations defined by previous BIPs.

Output script descriptors are designed for future extension with further fragment types and new script expressions.
These will be specified in additional BIPs.

==Reference Implementation==

Descriptors have been implemented in Bitcoin Core since version 0.17.

==Appendix A: Index of Expressions==

Future BIPs may specify additional types of expressions.
All available expression types are listed in this table.

{|
! Name
! Denoted As
! BIP
|-
| Script
| <tt>SCRIPT</tt>
| 380
|-
| Key
| <tt>KEY</tt>
| 380
|-
| Tree
| <tt>TREE</tt>
| [[bip-0386.mediawiki|386]]
|}

==Appendix B: Index of Script Expressions==

Script expressions will be specified in additional BIPs.
This Table lists all available Script expressions and the BIPs specifying them.

{|
! Expression
! BIP
|-
| <tt>pk(KEY)</tt>
| [[bip-0381.mediawiki|381]]
|-
| <tt>pkh(KEY)</tt>
| [[bip-0381.mediawiki|381]]
|-
| <tt>sh(SCRIPT)</tt>
| [[bip-0381.mediawiki|381]]
|-
| <tt>wpkh(KEY)</tt>
| [[bip-0382.mediawiki|382]]
|-
| <tt>wsh(SCRIPT)</tt>
| [[bip-0382.mediawiki|382]]
|-
| <tt>multi(NUM, KEY, ..., KEY)</tt>
| [[bip-0383.mediawiki|383]]
|-
| <tt>sortedmulti(NUM, KEY, ..., KEY)</tt>
| [[bip-0383.mediawiki|383]]
|-
| <tt>combo(KEY)</tt>
| [[bip-0384.mediawiki|384]]
|-
| <tt>raw(HEX)</tt>
| [[bip-0385.mediawiki|385]]
|-
| <tt>addr(ADDR)</tt>
| [[bip-0385.mediawiki|385]]
|-
| <tt>tr(KEY)</tt>, <tt>tr(KEY, TREE)</tt>
| [[bip-0386.mediawiki|386]]
|}