I mean that, even if your wallet is bug-free and up-to-date, you cannot be sure that your transaction can be confirmed, until it is; and that risk increases with time -- because soft-fork changes to the protocol can render the transaction invalid.

Since those mothballed transactions are not publicly accessible, there is no way for soft-fork proponents to make sure that they will not be invalidated.  In some cases (such as security or bug fixes), they must be invalidated.  Conversely, those who hold such transactions may not have the private keys or other conditions needed to create valid versions of them.

This may be bad news for the Lightning Network.  The latest attempt at the LN design, IIUC, uses long-lived bidirectional channels, and unconfirmed and unbroadcast transactions ("cheques") that may have to be held by the participants for months or years.  It was already pointed out that fee hikes could cause problems, forcing the receiver of a cheque to pay (via CPFP) the fees that the sender was supposed to pay.  But soft-forks could make the cheque completely unspendable.  Then the receiver would lose all the payments that he received through the channel.  If the channels have 100 year timouts, maybe both parties would effectively lose all the coins that they put into the channel.

Even if the risk of one cheque being invalidated is low -- say, 1 chance in 1'000'000 -- it may be unacceptable when there are 100'000 people doing 100 transactions per month in the LN.  Moreover, asingle change can precipitate many such incidents in a short time.


Of course.  A cooperating mining majority can do anything.

Sorry, I don't understand your objections.  

There are no "meta-rules" that specify what the validity rules can be.  They are not limited to "syntax", whatever that means.   Any computable predicate on bit strings could in principle be a validity rule, as long as it does not completely break the system.

Right now there are no validiy rules that refer to fees.  The minimum fee, like the Pirate Code, "is more what you'd call 'guideline' than actual rule"; each miner decides whether to require it (or even to require more than it).  But the minimum could be made into a validity rule.  the difference woudl be that each miner would not only impose it on his blocks, but also reject blocks solved by other miners that contain transactions that pay less than that fee.


As I wrote, this cannot be guaranteed.  If a fork (rule change) was executed to fix a bug or prevent an attack, the miners cannot continue to use the old rules for transactions that have the old version tag; that would negate the purpose of the fork.  They must reject such transactions.  

So, it is not safe to retain signed but unconfirmed transactions without broadcasting them.

I am not sure if I understood your comment.  Miners cannot apply old semantics when the transaction has an old version field, because that field can be faked by the clients to sabotage the change.  E.g., suppose that the change imposed a mininum output amount of 0.0001 BTC as a way to reduce spam attacks on the UTXO database.  An attacker could frustrate that measure by issuing transactions with the pre-fork version tag.   Does that answer your comment?

A hard fork cannot "change anything" that easily, because the proponents must explain the change and convince most miners and most users to upgrade, before the change is activated.

A soft fork, on the other hand, can "change anything" much more easily, because it only needs the agreement of a simple mining majority, who need not inform or convince anyone else beforehand.

I believe the idea was posted first on reddit by /u/seweso . Here is my version of it.


But what would the outcry achieve?


1 MB transactions already can take a long time to validate. 

You mean a soft fork.

A hard fork should not cause that.  It should only make invalid transactions valid, not the other way around.

However, a hard fork could enable a new type of "lock breaking" transaction that allows the locked coins to be spent before the expiration date.  That would invalidate the refund transaction, which would be rejected as a double spend.

I don't know whether such a change would still qualify as a hard fork, though. 

I insist: you cannot be sure, because a fee hike is not the only change that might prevent confirmation. Especially if the transaction is held for months before being broadcast.

Rule changes are inevitable.  They are likely to be needed to fix bugs and to meet new demands and constraints.  Many rule changes have happened already, and many more are in the pipeline.

As I pointed out, if Antpool, F2Pool, and any third miner decide to impose a soft-fork change, they can do it, and no one can stop them.
 
Curiously, it is soft-fork changes that can prevent confirmation of signed and validated but unconfirmed transactions.  Hard-fork changes (that only make rules more permissive) will not affect them.

CPFP is a mempool management rule only.  If a min fee hike is implemented as a mempool management rule only, or is an individual option of each miner, then one can hope that some miner may also implement CPFP, and then the low-fee transaction will be pulled through.  But there is no way for the client to know whether some miner is doing that, so he cannot put a probability on that.

On the other hand, if the min fee is implemented as a rule change (meaning that miners are prohibited from accepting low-paying transactions) then it seems unlikely that CPFP will be implemented too.  The validity rules must be verifiable "on line", meaning that the validity of a block in the blockchain can only depend on the contents of the blockchain up to and including that block.  In particular, the rules cannot say "a transaction with low fee is valid if there is a transaction further ahead in the blockchain  that pays for it.

Anyway, other possible soft-fork changes that could prevent confirmation of a currently valid transaction include reduction of the block size limit (as Luke has been demanding), imposing a minimum output value (an antispam measure proposed by Charlie Lee), limtiing the number of inputs and outputs, extending the wait period for spending coinbase UTXOs, and many more

I don't think it is the same thing at all. 

Support for legacy files and programs in newer relases of an OS is similar to the "clean fork" approach that I described.  namely, the new software is aware of the old sematics and can use it when required.  Any hard fork must have such backwards compatibilty, because it must recognize as valid all blocks and transactions that were confirmed before the fork.

Backwards compatibility in general is feasible as long as there is a feasible mapping of old semantics to the new infrastructure, and there is no technical or other reason to deny the conversion.    However, that sometimes is impossible; e.g. if an old program tries to access hardware functions that are not accessible in newer hardware, or if the mapping would require decrypting and re-encrypting data without access to the keys.

Similar difficulties exist in handling an old transaction that was created before a soft fork but was broadcast only after it, and became invalid under new rules.  The rules must have changed for a reason, so the transaction cannot simply be included in the blockchain as such.   For example, suppose that the change consisted in imposing a strict limit to the complexity of signatures, to prevent "costly transaction" attacks.  The miners cannot continue to accept old transactions according to old rules, because that would frustrate the goal of the fork. 

(Note that there is no way for a miner to determine when a transaction T1 was signed.  Even if it spends an UTXO in a transaction T2 that was confirmed only yesterday, it is possible that both T1 and T2 were signed a long time ago.)


I don't think that anyone is proposing to change the definition.  Transactions that have not been broadcast yet and transactions that are in the queue (mempool) of some nodes or miners, but are not safely buried into the blockchain, are equally unconfirmed. 

I meant to point out that there is no way that a client can make sure that an unconfirmed transaction will ever be confirmed, even if it seems to be valid by current rules.  Everybody agrees on that.?

In fact, there is no way to put a probability value on that event, even with the usual assumptions of well-distributed mining etc.  Everybody still agrees?

But then it follows that clients who hold signed transactions for broadcast at a later date cannot trust that they will be confirmed, even if they seem to be valid at the time of igning.  Everybody OK with this?

Thus, there is no weight in the argument "we cannot do X because it would invalidate all pre-signed transactions that people are holding". 

Pre-signed but unbroadcast or unconfirmed transactions seem to be a tough problem. 

If the protocol is to support such transactions, then soft forks must be forbidden, since (by definition)  transactions that were valid before a soft fork may be invalid after it.  BIP66 invalidated any unbroadcast transactions that used the signature variants that it excluded. Even the deployment of SegWit as a soft fork could invalidate some valid transactions.

IMHO, the safest way to introduce changes is by a clean fork:  making sure that *every* transaction or block that is valid under the old rules is invalid under the new ones, and vice-versa.  The code for the change should be introduced in some release N of the software, but the change itself should be programmed to become active at some block number X that is ~6 months in the future, after the expected date of release N+3.   Then users can be warned of the impending fork at the time of release N, and in particular that any transaction created with older releases that is not confirmed by block X will never be confirmed. 

This does not solve completely the problem of transactions that are created specifically for delayed broadcast, but reduces the severity.  Clients who need that feature can tell their new wallet software, after upgrading to release N, whether they intend to bradcast them before or after block X.  (Or they can create both versions, just in case.)

The problem is that soft forks cannot be prevented.  If a simple majority of the miners wants to impose a soft-fork type of change, all they have to do is to start rejecting all blocks and transactions that are invalid under their chosen new rules.  They don't even have to warn other miners, users, or relay nodes; and even if they do, there is nothing that these players can do to prevent the fork.

TL;DR: Holding on to pre-signed transactions,without broadcasting them, seem to be a bad idea.  There is no way to guarantee that a transaction will be confirmed, until it is confirmed.   The older the transaction, the greater the risk of it becoming invalid.

The initial depoyment of SegWit will not enable Schnorr signatures, will it? Won't they require a hard fork anyway?

Even with Schnorr signatures, the miners would still have to accept old-style multisigs produced by old clients, right?  Then an attacker could still generate those hard-to-validate blocks, no?

As a temporary fix, a soft fork can be deployed limiting the max number of signatures.  Even a low limit like 100 is no restriction, only a small annoyance for the few users who would want to use more   It woudl be a good use of an "arbitrary numerical limit", like the 1 MB limit was when it was introduced. 

But there is no logical reason why signature validation should take quadratic time.  That is a bug in the protocol, that should be fixed by changing the algorithm -- with a hard fork if need be.

(By the way,  [for a couple of hours today](https://statoshi.info/dashboard/db/transactions?from=1458258715516&to=1458259562505) there was an apparent "stress test" where each transaction was 10 kB long (rather than the usual 0.5 kB).  Was the "tester" trying to generate such troll blocks?)

That is not quite true.  After a soft fork, old clients may issue transactions that are invalid by the new rules, and not understand why they are never confirmed.  A soft fork can also introduce new ways of storing transactions in the blockchain, implicitly or explicitly, that are invisible to old clients, as in this example.   In this case, the old clients will not see coins that new clients send them. 


Yes. 

More precisely, if 51% of the miners decide to do a soft fork, the soft fork happens -- even if no one was told about it in advance -- and all other miners and clients have to accept it. 


Not exactly.

With a hard fork, all users would have to be warned in advance, with explanation of what the change is and why it is a good idea. 

I there is not enough support from the miners, the hard fork does not happen and nothing changes.

If there is enough support from the miners to execute the hard fork, the users would have to be warned again to upgrade to a version that is at most K releases old by date D.  Hopefully, most everybody will convert in time, and then the few lagards will be unable to use their coins until they upgrade too. 

However, if a substantial minority *of the miners* remains absolutely opposed to the changes, the coin will split into new-rule and old-rule coins.  Each user will see his own coins replicated in both branches, and will be able to use both  independently.  Is freedom of choice such a bad thing?

Hm, I just came back to this old thread to find that some people still don't get it.  (And some of them resort to insults, on top of that.)


Sorry, that is exactly what I can't agree with.  It seems that miners have come to constitute a connected social group, much as ranchers in a county or plumbers in a city, and believe that they must have a code of ethics and have common policies and decision --- such as ostracizing peers that don't follow those rules.

But that is bad, terribly bad.  Bitcoin's "Fundamental Assumption" is that mining is distributed among thousands of independent miners that act individually, motivated by their individual gain.  That is, selfish and greedy, not "good" or "honest" in any sense.  If miners start acting as a group, sacrificing some selfish gain in pursuit of group profits (or other goals), then it is impossible to give any guarantees at all, not even probabilistic ones.

Today the "good miners guild" may decide that its members should not do hash-stealing and should not mine empty blocks while waiting for the parent to download -- because they decided that such things are bad for bitcoin. (I don't think so, but let's skip that issue for now.)  

But tomorrow they may decide that the minimum fee should be raised to well above the cost, because of suitable excuses --- and all guild members agree to orphan any blocks that contain low-paying transactions.  Or the guild may decide that transactions from or to a certain wallet should not be processed, because that wallet belongs to the "wrong" side in some war, and the guild does not want trouble with the government that is backing the "right" side. Or the guild may decide to increase the block reward via a soft  fork.  Or...

These last things are typical of what the "bankers guilds" -- formal or informal, national and international -- do all the time; and are percisely the sort of thing that the bitcoin protocol was hoped to avoid: miners colluding to act so as to achieve a commong goal, rather than their individual selfish interests.

That is the the only original feature of bitcoin, that justifies its existence: it is a system of incentives that is supposed to achieve the desired group behavior out of the selfish choices of anonymous, uncoordinated, unregulated and unscrupulous miners.  

And it almost worked!  It fialed, not because of trifles like hash-stealing, but because mining, quite unexpectedly, became concentrated in a handful of big pools and a few dozen big farms.  (In a rarely-quoted post, Satoshi estimated that the number of independent miners would grow to "100'000, maybe less", serving "millions" of clients.) Not only the big miners, but even miners in the 5% range, like Slush and KnC, are already too big to make the Fundamental Assumption believable.

Indeed, objectively, the present concentration of mining means that bitcoin is already dead.  It still works as a payment system, but it requires trusting some third parties -- the 5 big miners.  Than, what is the point?  

(What is worse, the concentration happened because of many unavoidable social and economic factors; the advantage of short propagation delay is only one of the weakest.  I see no hope of this problem being fixed -- unless the price crashes to pennies ber BTC, n which case industrial mining woudl not be worthwhile, whatever the difficulty.)  


Big pools and miners identify themselves for marketing reasons, but they don't have to.  In the original design, in fact, it was sort of assumed that  miners would be anonymous, coming and going at random times with no control or coordination.  A miner with 75% of the hashpower, who did not care fro marketing, could easily disguise himself as hundreds of anonymous miners with less than 1% each, with unknown geographic location.

Thus, boycotts and public campaign against transgressors of the "guild ethics", besides wrong in principle, are ineffective.  The attacked company can split and mutate into other pools, and no one will know whether they are independent or owned by the same persons (and maybe located under the same roof).

My advice is that miners forget about "the good of bitcoin", as doing so will only bring them frustration and stress; and just behave as Satoshi expected them to behave.

What do you mean? They are committed in the same block, at the same time, right?


The position of a transaction in the blockchain should define which version of the rules is applicable to it (in particular,
which version of the scripting language it uses).

In a soft fork, by definition, the new version of the software can reject transactions that the revious version considered OK.

For example, IIUC the soft-forked SegWit proposal implies redefining an op code that previously meant "no-op" to mean "check the signatures in the extension record" or something like that.  Thus, a transaction that used that opcode (for some bizarre reason of its own, possibly fraudulent) could be valid before SegWit was enabled, but become invalid after it.

That may be a good argument to phase out nLocktime in favor of CLTV.  Once a transaction is in the blockchain, its position in it defines the rules by which it should be validated, which allows proper handling of old time locks.

Yes.  That fix should be done by a hard fork: because the code will be much cleaner, and because hard forks are safer than soft forks. (More precisely: ensuring that old versions are inoperable after 3-4 releases is safer than deploying changes to the protcol without alerting users, and letting them discover later that they must upgrade to understand why their transactions are not confirming anymore.)

On the contrary, rearranging the data in transactions and blocks is an unnecessary and ugly hack to get that effect.  It means hundreds of lines of new code scattered all over the place, in the Core source and wallets, rather than a few lines in one library routine that everybody else can copy.


If a client fetches a block without signatures, with SegWit or not, he cannot check whether the transactions contained in it were properly signed.  With SegWit, he can check the hash of the non-signature data; but if he is an old client, he will not even be aware that he is not checking the signatures.  

With the special call solution, if the client wants to validate a particular block, he asks for it in full, and then he can validate everything (except the parent link), as now.  The extra call can be implemented with no fork, so clients who do not upgrade, or do not wish to use that special call, will still be able to verify everything as they do now.

In other words, soft-forked SegWit *forces* old clients to fetch only part of the data, and limits them to verify only that part, *without them being aware of it*.  The special call solution lets clients decide case by case whether they want to verify a block or trust the node (that they are already trusting to some extent); and it does not change the behavior or security of existing client software.

The savings would be greater because clients who choose to use this call for old blocks would get fewer data, whereas with soft-forked SegWit everybody would have to fetch old blocks in full, signatures included.

No, none of the alleged benefits of SegWit requires segregation of signatures:

* Malleability could be fixed by just skipping the signatures (the same data that SegWit would segregate) when computing the txid.  

* GREATER bandwidth savings could be obtained by providing API/RPC calls that simple clients can use to fetch the data that they need from full nodes, sans the signatures etc.  This enhancement would not even be a soft fork, and would let simple clients save bandwidth even when fetching legacy (non-SegWit) blocks and transactions.  

* Pruning signature data from old transactions can be done the same way.

* Increasing the network capacity can be achieved by increasing the block size limit, of course.


Biggest issue of this week, perhaps?  

Surely you know that the non-mining relay nodes invalidate the few security guarantees that the protocol can offer. Simple clients should not connect to them, but to miners (or relay nodes that are know to be run by miners). It makes no sense to twist the protocol inside out in order to to meet CONJECTURAL needs of those nodes.

The only cost that really matters is the marginal cost for a miner to add another transaction to his candidate block.  That is the cost that the transaction fees have to cover.  The magnitude of that cost is one of the great mysteries of bitcoin, extensively discussed but never estimated. But it seems to be very small (at least for competent miners) and is probably dependent only on the total size of the transaction.  But anyway the developers have no business worrying about that cost: the fees are payment for the miners, it should be the miners who decide how much to charge, and for what.

I asked some of these questions 3 months ago.  Never got a decent answer.

Blockstream wants soft-forked SegWit to fix the malleability problems (that would be needed for the LN, if they ever get it to work), and to force ordinary p2p bitcoin users subsidize the costs of complicated multisig transactions (ditto).  But these reasons do not seem explain the urgency and energy that they are putting on the SegWit soft fork.  Maybe they have other undeclared reasons?  Perhaps they intend to stuff more data into the extension records, which they would not have to justify or explain since, being in the extension part, "ordinary users can ignore it anyway"?

As for SegWit being a soft fork, that is technically true; but a soft fork can do some quite radical changes, like imposing a negative interest (demurrage) tax, or raising the 21 million limit.  One could also raise the block size limit that way.  These tricks would all let old clients work for a while, but eventually everybody will be forced to upgrade to use coins sent by the new verson.

I have more than 100 pages to catch up on this thread, sorry if this is old news:

The Chinese New Year holiday is the week February 7–13, 2016.  They should return to work on Sunday February 14.

As in previous New Year holidays, there is very little trading on the Chinese exchanges during this period, and therfore one can expect relatively little price movement.  

There are some curious differences between the volume at the Chinese exchanges between this and previous years.  In 2014, on the 1h and 30min plots, there was usually a clear and relatively smooth periodic pattern with a minimum around 18:00 UTC (03:00 am Chinese local time).  At that hour, the volume went down to practically zero on Huobi, whereas on OKCoin the minimum was about 10% of the peak value.

Recalling, the present price rise started in September 2015. At first there was a fast exponential rise from  ~230 USD to ~500 USD that was probably due to speculator euphoria.  That rally ended abruptly on November 04, and was down to ~320 USD on Nov 26, 2015

The last surge in price began suddenly on that date, climbed steadily to ~460 on Dec 17, and has been mostly falling since.  

The trade volume at OKCoin increased substantially during the first spike, but on Nov 26 it jumped suddenly from ~0.28 M BTC/day to ~1.0 M BTC/day, and remained in the range 1.0-1.5 M BTC/day until February 06. With the start of the holiday, the volume dropped to ~0.7 M BTC/day.

The interesting detail is that the daily pattern changed markedly on Nov 26, and has been different since then.  On that date, there appeared concentrated peaks, lasting 2-3 hours, almost but not quite periodic, spaced roughly 26 hours apart, but with some exceptions and resets.  Those peaks seem to account for half of the total daily volume or more.

Curiously, the price hardly moves during most of those peaks (and the large price moves often occur with little volume).  Curiously too, since the holiday week started, the normal daily volume pattern practically disappeared, but those peaks continued unabated.

These same features of the volume plot are seen at Huobi, although the "normal" (non-peak) background has been rather irregular in recent months, so those peaks are harder to discern.  On BTC-China, on the other hand, those daily peaks are much smaller or absent, and the volume increase since September was more modest and gradual.

I don't know what to make of those peaks. (More generally, the volume and price at the Chinese exchanges have become more "mysterious" since I looked closely at them in 2014.)  Since they were not affected by the holidays, perhaps they are generated outside China, or generated automatically by some script.  Perhaps they are connected to mining operations.  Perhaps it is Sergei Mavrodi or some Chinese copycat selling the day's catch of his ponzi.  Or perhaps they are periodic settlements of arbitrage trade, generated by Blockstream's Liquid tool.

Totally correct, of course!