bitcoin-bips/bip-0099.mediawiki

<pre>
  BIP: 99
  Title: Motivation and deployment of consensus rule changes ([soft/hard]forks)
  Author: Jorge Timón jtimon@jtimon.cc
  Status: Draft
  Type: Informational | Process
  Created: 2015-06-20
  Post-History: http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-June/008936.html
</pre>

==Abstract==

This BIP attempts to create a taxonomy of the different types of
consensus forks and proposes a deployment mechanism for each of them.

==Motivation==

The security assumptions of p2p consensus-based systems like Bitcoin are
not always well understood, and the upgrade mechanisms to the
consensus rules are a work in progress that may vary depending on the
type of change being deployed.
Discussing such changes without a uniform view on the deployment
paths often leads to misunderstandings and unnecessarily delays the
deployment of changes. 

==Defintions==

- Software fork: in free software, an existing project can be copied
  and extended/maintained without permission from the original
  project's maintainers.
- Consensus fork: a divergence in the implementation of the
  verification rules can impede the expected eventual convergence of
  the network in a single chain that has the most proof of work and
  also satisfies the rules. This can be intentional or be caused by a
  bug in consensus validation reimplementations.
- Softfork: an intentional consensus fork where everything that
  was previously invalid remains invalid, but some things that were
  valid also become invalid. Majority of miners can impose
  this kind of changes and they also have some deployment advantages.
- Hardfork: an intentional consensus fork that is not a softfork.

==Taxonomy of consensus forks==

===Accidental consensus fork===

Software forks are very different in nature from consensus rules forks. No software
maintainer has special powers over consensus rules changes. There's
many good reasons (experimentation, lack of features, independent
development, diversity, etc) to fork the Bitcoin core software and it's good
that there's many alternative implementations of the protocol (forks
of Bitcoin core or written from scratch).

But sometimes a bug in the reimplementaion of the consensus
validation rules can prevent alternative implementation users from
following the longest (most work) valid chain. This can result in
those users losing coins or being defrauded, making reimplementations
of the consensus validation rules very risky. Note that a natural
language specification of those rules doesn't help since the
consensus is not determined by such specification but by the software
that the majority of the network runs. That's why "the implementation
is the specification".

But Bitcoin core contains many more things than just consensus
validation and it would be unreasonable for all alternative
implementations to depend on it. Bitcoin core should not be the
specification. That's why the consensus validation is being separated
into a libbitcoinconsensus library with a C API easily accessible from
any language. This makes alternative implementations much more secure
without burdening them with specific design choices made by Bitcoin
core. It is to be noted that sharing the same code for consensus
validation doesn't prevent alternative implementations from
independently changing their consensus rules: they can always fork
the libbitcoinconsensus project (once it is in a separate repository). 

Hopefully libbitcoinconsensus will remove this type of consensus fork
which - being accidental - obviously don't need a deployment plan.

====11/12 March 2013 Chain Fork====

There is a precedent of an accidental consensus fork at height 225430. 
Without entering in much detail, the situation was different from
what's being described from the alternative implementation risks (today alternative implementation
still usually rely in different degrees on bitcoin core trusted proxies, which
is very reasonable considering the lack of a complete
libbitcoinsensus).
The two conflicting consensus validation implementations were two
different versions of Bitcoin core (Bitcoin-qt at the time): 0.8
against all versions prior to it. Most miners had been fast on
upgrading to 0.8 and they were also fast on downgrading to 0.7 as an
emergency when they were ask to by the developers community.

Without entering in much detail[2], the issue was that BDB was being
abandoned in favor of levelDB, and - at the same time - the miner's
policy block size limit was being lift (it was not a consensus rule,
not even enforced via softfork). Even after testing, a case where
levelDB couldn't correctly validate certain bigger blocks only appeared after
deployment in production. Fortunately this was handled very well and
rapidly by the whole worldwide community and nobody is unhappy about
the solution.

But there's some philosophical disagreements on the terms of what the
solution was: we can add a pedantic note on that.   
If "the implementation is the specification", then those
levelDB-specific limitations were part of the consensus rules.
Then additional rules were necessary and any alternative
implementation (including 0.8) would have to implement it. Then a
planned consensus fork to migrate all Bitcoin-qt 0.7- users could
remove those additional consensus restrictions.
Had libconsensus being implemented without depending on levelDB,
those additional restrictions wouldn't have been "the implementation
is the specification" and this would just have been a bug in the
consensus rules, just a consensus-critical bug in a set of
implementations, concretely all satoshi-bitcoin-0.7-or-less (which
happened to be a huge super majority of the users), but other
implementations (like libbitcoin) would be free from such bug and
implementing the correct libconsensus specification. But since the
buggy implementation was a super-majority, the solution would have
been to instantly (from a specific block) change the rules to not let
the super-majority deviate from the specification and then have
another consensus fork to remove them. Two theoretical consensus forks
instead of one but the first one deployed practically for free. The
practical result would have been identical and only the definitions
change. This means discussing something that went uncontroversially
well further is "philosophical bike-shed" (TM).  

===Unilateral softforks===

If it is in their best interest of miners to softfork it should be
assumed that they may likely enforce it. In some cases, even against the will of a
super-majority of users. This is practically an attack on the network
and the only solution is to carefully design the incentives so that
the case is simply impossible. If that fails, miners should still
consider the risk of motivating a schism hardfork before attempting
such a consensus fork. A deployment plan for this case is also
unnecessary.

===Schism[1] hardforks===

In all of the following examples there's clearly a confrontation that
is being resolved using an intentional consensus fork.

Being a schism hardfork, there will likely be 2 chains
coexisting for at least some time, maybe forever. Maybe bitcoin
becomes bitcoinA and bitcoinB. The implications for market
capitalization are completely unpredictable, 
maybe mc(bitcoinA) = mc(bitcoinB) = mc(old_bitcoin), 
maybe mc(bitcoinA) + mc(bitcoinB) = mc(old_bitcoin), 
maybe mc(bitcoinA) + mc(bitcoinB) = 1000 * mc(old_bitcoin),
maybe mc(bitcoinA) + mc(bitcoinB) = 0,
... 

This is very disruptive and hopefully will never be needed. But if
it's needed the best deployment path is just to activate the rule
changes after certain block height in the future. On the other hand,
it is healthy decentralization-wise that many independent software
projects are ready to deploy a schism hardfork.

====ASIC-reset hardfork====

Imagine ASIC production has been consolidated to a single company and
distribution is simply not happening: the company is keeping them to
mine itself. For that or another reason, a single entity controls
40%+ of the hashrate and there's no hope for an spontaneous
improvement in decentralization. Such an untenable centralization could
be fixed (with great risks) by switching the hash function used in the
proof of work, effectively "pressing the restart button" on the ASIC
market. [The next function should be simple to implement in ASIC as
well so that the market can more easily develop as a healthy and
competitive one (as opposed to what the "ASIC-hard" proponents would
want), but that's another story...]

Since in this case the confrontation is clearly against the current
miners any notion of "miners' voting" is utterly irrelevant.

====Anti-Block-creator hardfork====

There's less extreme cases where changing the pow function would not
be necessary. For example, let's imagine a bright future where
commoditized ASICs are running in millions home-heaters all over the
world, but the block size has been completely removed and the network has devolved to a
very centralized system where only 2 big pools have the resources to
fully validate full blocks and create block templates with competitive levels of
transaction fees. In that case, changing the pow function would be a
terrible waste and a risk that could be avoided. A hardfork restoring
a block size limit could help fixing this situation. Please don't
take it as an argument for or against raising the block size limit:
it's just an example. But in this case, again, those 2 big pools
would probably be against the fork and, again, their voting is
irrelevant.

Like in the previous example, miners are expected to oppose and they
have to be ignored.

====Anti-cabal hardfork====

Let's imagine BIP66 [TODO remove has been deployed, but it] had a crypto backdoor
that nobody noticed and allows an evil developer cabal to steal
everyone's coins. The users and non-evil developers could join, fork
libconsensus and use the forked version in their respective bitcoin
implementations. 
Should miner's "vote" to express their consent? What if some miners
are part of the cabal? In the unlikely event that most miners are
part of such an evil cabal, changing the pow function may be
required. In other cases, mining "vote" doesn't have much value
either since this kind of hardfork would not qualify as
uncontroversial anyway.

===Uncontroversial consensus upgrades===

"Uncontroversial" is something though to define in this context. What
if a single user decides he won't upgrade no matter what and
he doesn't even attempt to explain his decision? Obviously, such
a user should be just ignored. But what if the circumstances are
slightly different? What if they're 2, 10 users? where's the line.
It is possible that we can never have a better definition than "I know
it when I see it" [citation needed].

====Uncontroversial softforks====

If a majority of miners adopts a softfork, users will follow that
chain, even without understanding the new rules. For them is like
if blocks are created in a certain way or certain valid transactions
are being rejected by miners for some reason. For old nodes it just
looks like the new rules are policy rules rather than consensus rules.
This greatly reduces the deployment risks, making softforks the
preferred consensus rules upgrade mechanism.

The first precedent of a softfork was the introduction of P2SH
documented in BIP16. There were competing proposals, but BIP12 had
clear disadvantage and  BIP17 was considered a less tested but
functionally equivalent version by most of the reviewers. Although it
was later discovered that BIP16 had unnecessary limitations and BIP17
is now considered superior, this probably still qualified for our
vague concept of "uncontroversial".

At the time, there was no "mining voting" implementation and it was
simply deployed using the timestamp of the blocks at some time in the
future as the activation trigger. This can't guarantee the assumption
that most miners have upgraded before enforcing the new rules and
that's why the voting mechanism and first used for BIP30 and BIP66.
The current voting threshold for softfork enforcement is 95%. There's
also a 75% threshold for miners to activate it as a policy rule, but
it should be safe for miners to activate such a policy from the start
or later than 75%, as long as they enforce it as consensus rule after 95%. 

The current miners' voting mechanism can be modified to allow for
changes to be deployed in parallel, the rejection of a concrete
softfork without getting locked for the deployment of the next one,
and also a more efficient use of the version field in block
headers [3]. BIP65 is expected to be deployed with the improved
mechanism.

====Uncontroversial hardforks====

Some consensus changes require all participants to upgrade their software
before the new rules can be safely activated or they will face serious
risk of following the wrong chain and being defrauded. Even if the
exact same mechanism used for softforks would be more risky in these
cases, that doesn't mean that this type of changes cannot be deployed
in an uncontroversial and safe manner.

The simplest approach is to select a block height far enough in the
future that everybody has plenty of time to change their software.
But if you're aiming for universal adoption, that includes miners'
adoption, so it seems reasonable to use a mining voting on top of
that. In this case there's only one relevant threshold and it could
be different from the softfork one. Probably 100% is too strict,
since it would allow a relatively small miner to attack the network
and block a consensus upgrade. Something between 99% and 95% is
probably a sensible choice for this parameter.

====Uncontroversial emergency hardforks====

Emergency forks may not have time to consult miners and have to be
deployed simply by choosing a block height not so far in the future.

But emergency forks could be prepared ahead of time. For example, an
intermediary version of software could allow blocks
that are double the size of old blocks (after a certain height in the
future) while still making miners reject bigger blocks as a softfork
rule. Then miners can start the regular process for uncontroversial
softfork (or a unilateral softfork if they're a majority) at any
point in the future if it is required, and both intermediary and new
versions would be prepared for it (which would make deployment much
easier). Other related consensus changes could be deployed in the
meantime (say, quadrupling the block size) making the emergency
softfork unnecessary.

==Code==

This BIP is complemented with a concrete code proposal[4] for an
uncontroversial hardfork which acts as a precedent and removes the
perception that hardforks are impossible in Bitcoin. The deployment of
the proposal should not block any other potential hardforks (thus it
will required the version bits proposal[3] to be implemented). The
change itself doesn't add much complexity to Bitcoin Core and is simple
enough that is trivial to apply to diverse implementations (that
currently can only use libbitcoinconsensus to validate script-related
rules). The change has been already widely tested in many altcoins.

The chosen consensus change is the fix of the timewarp attack
discovered and also fixed with a simple patch[5] by @ArtForz. This
change has been deployed by most altcoins that made any minimally
meaningful change to bitcoin and thus can be considered somewhat
tested (in fact, most SHA256d altcoins that didn't implemented it have
died or being forced to implement it as an emergency hardfork). When
deploying this change has been discussed, usually arguments in the
lines of "if we get to the point when this matters to bitcoin, we
would be already in serious trouble" were used against it. This
shouldn't be seen as a disadvantage in this context, since it means we
can safely activate the fix very far away in the future (say, 4 years
worth of blocks).

==Attribution==

Incorporated suggestions from @btcdrak.

==Footnotes==

[1] https://en.wikipedia.org/wiki/Schism

[2] https://github.com/bitcoin/bips/blob/master/bip-0050.mediawiki

[3] http://sourceforge.net/p/bitcoin/mailman/message/34146741/ 

[4] https://github.com/bitcoin/bitcoin/compare/0.11...jtimon:hardfork-timewarp-0.11

[5] Original references:
https://bitcointalk.org/index.php?topic=114751.0
https://bitcointalk.org/index.php?topic=43692.msg521772#msg521772
Rebased patch: 
https://github.com/freicoin/freicoin/commit/beb2fa54745180d755949470466cbffd1cd6ff14