IIRC, GHash actually got more than 50%. 

The community can be sure that they got down to 10% by turning off 80% of their equipment, all for the good of the community. They would never think of moving that equipment to other pools. [/sarcasm]

Some pools apparently monitor other pools and "steal" the hash of their last mined block even before the block has propagated to the relay nodes.  Since they don't have the block, not even the header, they cannot validate it immediately as they start mining on top of it.

I have read that those two pools and also BTC-China use this trick.  I don't know whether others do the same, but since it gives them several seconds of advantage in the race for the next block, we must assume that every miner will do it if they can.  And the shortcut usually works; it failed this time only because of the version switch.  Therefore, the miners will not stop doing that, in spite of having lost 9 block rewards altogether (half of which they would have lost anyway).

Sorry, that was not the idea.  Players were supposed to behave in certain ways because of the economic incentives built into the protocol, not because of "the good of the whole".   If players find that it is more advantageous to them to behave in ways that are bad for the system, that is a flaw of the protocol, not a fault of the players.

Try adding some other term, like "paypal".  It looks like a general treend of google usage, or just a glitch.

According to this reddit comment, F2Pool did not even have the header of the invalid block B when they started mining on top of it.  The big pools monitor the instructions that other pools send to their members, to catch the hash of a recently mined block even before that block gets to the relay nodes.  They just did not notice (or had no way of knowing) that the hash came from a pool in the 5% that was still running v2, and therefore was likely to be invalid.

You are proposing is to modify the client so that it would do what a full node would do, only truncated to that last 50 nodes.

Just checking the version tag is not enough. This hacked client app must also know that the BIP66 rule became active at a certain block b_{n}.  Otherwise it would have to keep fetching blocks until it finds a "95% majority" event, or reaches a block before the v3 software was published.

You mean: after you sell some of your bitcoins for dollars through BitPay, and buy some stuff on overstock.com using those dollars. 

The deadline for filing claims against the MtGOX bankruptcy has been extended:

https://www.mtgox.com/img/pdf/20150706_deadline.pdf [ scroll down for the English translation ]

Only that "crystal clear" can be black and opaque as coal.  Which is actually what "crystal clear" usually means in politics and economy...

(Uh, ok, and coal can be clear and transparent as diamond, alright.)

Thanks @Cryddit for the replies! You write:


Well, it is debatable whether it can be called "wrong".  It is more risky for the miner, and not what the bitcoin devs think that the miner should do, and not as helpful to the system as the others would think.  But miners have no obligation to behave in any particular way.  Each miner has the right to run whatever software he wants and output any blocks he wants.  The economic incentives are supposed to lead him, through self-interest, to behave in a way that is best for the system.  If the incentives are right but the miner doesn't act in his best interests, it is his problem, not anyone else's.  But if the incentives actually encourage a behavior that is bad for the system, it is a flaw of the protocol.

When they opted for "SPV mining", it was because they thought it would increase their revenue. In this case it didn't, but maybe it paid off overall.  Perhaps they overreacted to that incident that made them disable the verification code (when they lost ~6000 USD in an orphaned block because of a few seconds of delay); or perhaps "SPV mining" indeed saved them from several other incidents. 

Anyway, it should be their problem to figure that out and do what they think is best for them.  Their losses in that chain should not be of concert to anyone... except that there were many v2 nodes and clients out there that got confused because, by their criteria, the 'bad' chain was fully valid and was the longest chain. 

So, I insist that the devs deserve a fat slice of the blame: for not having a delay after the "95% majority" event, and because they choose to do a riskier manoeuver (allow v3 nodes and miners interact with v2 clients) for PR reasons (to keep clients unaware of the change, and let them run the v2 version even after it went into effect).

Thanks @knightdk again for the reply. You write:


In understand that clients don't need to update if a soft fork only makes the rules for blocks more restrictive.  However, if it makes the rules for transactions more restrictive too, then the clients must update anyway, otherwise their transactions will be rejected (except by nodes and miners who are still running the old version, which may cause those transactions to be confirmed and then unconfirmed).  isn't that so? Isn't that the case of BIP66?


Indeed, I see the quote in other posts.

No matter how big the army, it cannot defend the country from an internal war.

Thanks for the replies, @knightdk.  You write:


I doubt that 1% of the bitcoiners out there know what a BIP is.

I doubt that 0.1% of the bitcoiners out there knew that a soft fork due to BIP66 was due to happen this year.

A 2-week delay would have allowed the devs to send an alert to all nodes and miners still runing v2 to upgrade immediately or risk being cut off from the network and having their feet eaten by the boogeyman while they sleep.

A 2-week delay would also allow any v3 nodes that were turned off to catch up with the blockchain, and determine whether BIP66 was already in force of not.


After BIP66 became active, any v3-running miner, node, or client should have ignored (or have disconnected from) any nodes that it knew were still running version v2.

More generally, the core devs approach to protocol changes seems to be like that of an airline that lets a plane take off with a known engine malfunction, and tries to fix it during flight -- because it does not want to alarm or inconvenience the passengers, or get bad exposure in the news...

And, AFAIK, "SPV mining" is only the most likely theory, not confirmed or proved yet.

I am still trying to understand what actually happened.

What I have read is that BTCNuggets, a miner that was still running v2 software, mined a block B'(N) that was valid by v2 criteria but invalid by v3 criteria. Somehow F2Pool, a miner that was running v3 software, received the header of this block and successfully mined a block B'(N+1) that was valid under v3 rules (empty, in fact) except for having an invalid parent.  As luck would have it, they also mined another (empty) block B'(N+2) with B'(N+1) as parent.  Then AntPool, who was running v3 software too, mined B'(N+3) on top of B'(N+2).  And so on; this was the 'bad' branch.

Meanwhile, other miners running v3 software either did not see block B'(N), or saw it and detected that it was not v3-valid; so they ignored it and eventually mined a block B''(N) that was v3-valid.  Several v3 miners solved additional v3-valid blocks B''(N+1), B''(N+2), etc. on top B''(N).  This was the 'good' branch.

For a while, the bad branch was ahead of the good one, in terms of total work.  Fortunately the devs and other bitcoin hackers were watching the blockchain to monitor the onset of BIP66, spotted the problem immediately, alerted F2Pool and AntPool that they were mining a v3-invalid chain, and sent out an alert recommending a 30-confirmation (5 hour) wait for important transactions.

Is this a correct summary of the events?

Some of the many questions that remain:

* How many blocks were actually mined on the bad branch (starting form B'(N)) before it was abandoned?

* Were F2Pool, AntPool, and the other pools that mined on the bad chain running modified versions of the v3 core? If so, what were the modifications?

* Did F2Pool realize that the BIP66 rules had already started to apply when they received B'(N) from BTCNuggets? (Say, perhaps it had counted only 949 v3 blocks, or had its counter reset recently, etc.)

* How did F2Pool get the header of the invalid block B'(N): directly from BTCNuggets, from a v2-running relay node, or from a v3-running relay node?

* In the standard version of the v3 core software, do miners get the whole parent node and verify its validity?

* Could there be an off-by-one bug in the v3 core software, that would have resulted in F2Pool verifying B'(N) with pre-BIP66 rules instead of BIP66 rules?  (Note that a v3-valid node *can* have a v3-invalid parent, if the parent was mined before BIP66 went into effect.)

* If F2Pool knew that BIP66 had kicked in, and got B'(N) from a v2-running miner or node: why was it still accepting transactions, blocks, and block headers from v2-running players, without checking whether such data was v3-valid?

* If F2Pool knew that BIP66 had kicked in, and got B'(N) from a v3-running node: which node was that, and why did it forward the v3-invalid node B'(N) to F2Pool?

* If a miner receives a block B(M), how many blocks B(M), B(M-1), B(M-2) etc. should it validate on its own before daring to mine B(M+1) on top of it?

* Suppose that a miner receives block B(M) and starts validating it, while in parallel mining a new block B(M+1) on top of it; and happens to solve B(M+1) before the validation of B(M) is complete.  Should it broadcast B(M+1), or wait until the verification of B(M) is complete?

The devs and others have been quick to blame F2Pool, and even AntPool, for the incident.  But, depending on the answers to the above questions, other players may be to blame too.

The devs deserve a fat slice of the blame for not programming a reasonable delay (say, 2 weeks) between the "95% majority" event and the activation of the BIP66 rule.  That delay would have given a chance for the remaining v2 miners and nodes, as well as more v2 clients, to upgrade to v3.  (Using blockchain voting to trigger protocol changes also seems a terrible idea, but that is another discussion.) EDIT: Such a delay would have also reduced the risk of a fork being triggered by off-by-one bugs and network delays.

The core devs or F2Pool's own devs may also deserve another fat slice of the blame, for the v3 software allowing v3 miners and nodes to receive blocks from v2-running miners and nodes after BIP66 was activated.

BTCNuggets deserved their slice of the blame for not upgrading to v3.

EDIT: repaced some "v3 rules" by "BIP66 rules"

As far as I understand, the incident had some consequences:

* several miners lost the rewards that they mined (at least 6 blocks, maybe more).
* there was a signficant risk of double-spends, since many ciients were watching the bad chain.
* at least one transaction (the one that triggered the fork) was undone after >6 confirms.
* a warning was broadcast telling everybody to wait for 30 confirmations (5 hours).

The incident was not as bad as it could have been because Greg Maxwell and other ore devs were carefully watching the blockchain as the BIP66  triggered the switch from version v2 to version v3 of the protocol, and they immediately spotted the fork.  They soon contacted the mining pools that were mining on the wrong chain.

The fork was started by miner BTCNuggets, that was still running v2 and issued a block N with a transaction that was invalid under version v3.  Miner F2Pool, that was already running v3, saw that block and successfully mined another block N+1 on top of it, not realizing that N was invalid.  Other v3 miners like AntPool added N+2, N+3, etc. on top of F2Pool's block N+1.  Meanwhile some v3 miners did not see the block N issued by BTCNuggets, or noticed that it was invalid under v3 rules and rejected it; so they created their own block N and kept mining that branch.  Until the miners were alerted, the bad branch was longer, so it was accepted by all v2 clients and maybe by v3 clients that did not check all blocks.

Blame is being thrown at the miners, especially F2Pool for mining block N+1 on top of a block N that was produced by a v2 miner, without checking the validity of that block.  The blame may be partly deserved if F2Pool got the block directly from BRCNuggets or from any intermediate relay node still running v2.

However, methinks that the blame should fall on the developers.  After the BIP66 change went into effect, every v3 player (client, node, or miner) should have dropped all links to other v2 players.  Letting v2 and v3 players communicate was begging for this accident to happen.

If F2pool got that block through an intermediate v3 node, the blame should fall on that node.

Miners should not be required to validate the blocks that they receive.  It is their right to decide whether to validate them, or to trust that the sender did so.  Note that the miner that started mining block N+3 on top of N+2 would have had to check also blocks N+1 and N to discover that N+2 was invalid.

Was the incident serious? Suppose that a trash basket caught fire in a gunpowder factory, but it was quickly put out.  was that incident serious?

I don't know about bitcoin itself, but the bitcoiners' faith in it can survive anything. 


Indeed, it has already been used...

However the problem is not the block size per se, but miners skipping the validation of the previous block in order to save a few milliseconds. It is roughly the same reasoning that results in empty blocks being mined while the queues are full.  Bigger blocks will make these "optimizations" more tempting, but they are tempting enough even with 1 MB blocks. 

Ideally, the protocol should force the miners to validate the previous block(s), and take as many transactions from the queues as possible.  Then the block size would not matter, for this issue at least.

The top 4-5 Chinese pools have a lot more than 50%.

It was solved promptly because Greg Maxwell (@nullc) and perhaps other devs were awake and carefully watching the blockchain for the BIP66 activation.  They spotted the problem right away and promptly warned the SPV miners.  

Had the devs been sleeping or busy at the time, the accident would have been nastier.

Of course.  It was just an accident, some core devs were awake watching anxiously the onset of BIP66, the two pools involved cooperated promptly, etc..  

That was not an attack: someone just sneezed. See, the building stopped shaking already.