New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cluster size 2 package rbf #28984
base: master
Are you sure you want to change the base?
Cluster size 2 package rbf #28984
Conversation
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. Code CoverageFor detailed information about the code coverage, see the test coverage report. ReviewsSee the guideline for information on the review process.
If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
38ce2cb
to
5b216db
Compare
Added this to the tracking issue |
57e5fc7
to
56848f9
Compare
56848f9
to
be9d163
Compare
src/policy/rbf.cpp
Outdated
const CFeeRate replacement_miner_score(replacement_fees, replacement_vsize); | ||
|
||
for (const auto& entry : direct_conflicts) { | ||
const bool conflict_is_v3{entry->GetSharedTx()->nVersion == 3}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this can't be hit yet; post-v3 it can. I can remove this, or leave a note, or both. It makes some package rbfs more useful
(edit: it's used in #29001 functional tests )
7aee728
to
1d859b0
Compare
So thinking conceptually about this, I'm most concerned by the new I think there are two issues with this approach:
Even though the current RBF logic is broken already, I'd prefer to avoid adding on to that while rolling out the package RBF implementation, just so that users don't get used to some kinds of replacements working that really shouldn't be. So I was wondering, can we deal with this by just restricting the topology of what we allow a package RBF to conflict with? Let's say we allow a replacement if the conflict set consists of either (1) a single transaction with no ancestors (and of course no descendants, since any descendants would be in the conflict set as well), or (2) two transactions that are in a parent-child topology, and that have no other in-mempool ancestors (or descendants, of course). In those situations, the maximum miner score you would have to consider is either the individual feerate of the parent tx or the ancestor feerate of the child, so the new logic should work perfectly. |
After offline discussion with @sdaftuar , I've worked on a set of prospective changes and pushed them to this branch as follow-on commits. Key change is that we will now only allow package RBF when the conflicted transactions are all in "clusters" of up to size 2. This allows us to calculate mining scores for each conflicted transaction, which means that post-cluster mempool, if we did this right, IIUC, the behavior should match with more general package RBF. |
f9ce967
to
385b6ca
Compare
705fcd9
to
53ed8fd
Compare
src/validation.cpp
Outdated
|
||
// We're in package RBF context; replacement proposal must be size 2 | ||
if (workspaces.size() != 2) { | ||
return package_state.Invalid(PackageValidationResult::PCKG_POLICY, "package RBF failed: replacing cluster not size two"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah, I think PCKG_POLICY
is fine given 6119f76. Just need to make sure the MempoolAcceptResult
s are correct so that we are attributing the failures correctly
src/validation.cpp
Outdated
// Ensure this two transaction package is a "chunk" on its own; we don't want the child | ||
// to be only paying anti-DoS fees | ||
if (CFeeRate(parent_ws.m_modified_fees, parent_ws.m_vsize) >= | ||
CFeeRate(parent_ws.m_modified_fees + child_ws.m_modified_fees, parent_ws.m_vsize + child_ws.m_vsize)) { | ||
return package_state.Invalid(PackageValidationResult::PCKG_POLICY, | ||
"package RBF failed: parent paying for child anti-DoS", ""); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in b86487c
The comments + error string are a bit contradictory ("child only paying anti-DoS" and "parent paying for child anti-DoS") unless maybe I'm misunderstanding?
Also maybe error str "package RBF failed: parent paying for child replacement" and debug str "package feerate is X while parent feerate is Y" ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
src/validation.cpp
Outdated
// Check if it's economically rational to mine this package rather than the ones it replaces. | ||
if (const auto err_tup{ImprovesFeerateDiagram(m_pool, direct_conflict_iters, m_all_conflicts, m_total_modified_fees, m_total_vsize)}) { | ||
return package_state.Invalid(PackageValidationResult::PCKG_POLICY, | ||
"package RBF failed: " + err_tup.value().second, ""); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if PaysForRBF
and ImprovesFeerateDiagram
should be swapped in the order of checks?
ImprovesFeerateDiagram
is a more expensive, is a superset of the rules, and a less intuitive error (I'd have an easier time guessing what went wrong with "pays less fees" vs "does not improve feerate diagram". Especially looking at the functional tests which switched out the error strings due to this check being earlier.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is a superset of the rules
PaysForRBF has incremental rate on top, so ImprovesFeerateDiagram
is not a strict superset fwiw.
That said, I don't think swapping the order is a problem and would indeed be cheaper.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
True, it's only a superset of Rule 3 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
swapped them and updated tests accordingly
src/validation.cpp
Outdated
// since it would result in a cluster larger than 2 | ||
for (const auto& ws : workspaces) { | ||
if (!ws.m_ancestors.empty()) { | ||
return package_state.Invalid(PackageValidationResult::PCKG_POLICY, "package RBF failed: replacing cluster with ancestors not size two"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
return package_state.Invalid(PackageValidationResult::PCKG_POLICY, "package RBF failed: replacing cluster with ancestors not size two"); | |
return package_state.Invalid(PackageValidationResult::PCKG_POLICY, "package RBF failed: new transaction cannot have mempool ancestors"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
"package RBF failed: parent paying for child anti-DoS", ""); | ||
} | ||
|
||
// Check if it's economically rational to mine this package rather than the ones it replaces. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe you can mention that this replaces PaysMoreForConflicts
function from ReplacementChecks
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
src/validation.cpp
Outdated
|
||
// We're in package RBF context; replacement proposal must be size 2 | ||
if (workspaces.size() != 2 || !Assume(IsChildWithParents(txns))) { | ||
return package_state.Invalid(PackageValidationResult::PCKG_POLICY, "package RBF failed: replacing cluster not size two"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think it's very clear which cluster isn't size two in this str
return package_state.Invalid(PackageValidationResult::PCKG_POLICY, "package RBF failed: replacing cluster not size two"); | |
return package_state.Invalid(PackageValidationResult::PCKG_POLICY, "package RBF failed: package must be 1-parent-1-child"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
doc/policy/packages.md
Outdated
- Package RBF may be enabled in the future. | ||
- More general package RBF may be enabled in the future. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could add more detailed docs for what the rules are in package RBF?
- all to-be-replaced signal (bip125 or v3)
- package is 1p1c, only in clusters up to size 2
- to-be-replaced all in clusters up to size 2
- no more than 100 total to-be-replaced
- total fees of package > all fees being replaced, at incremental relay feerate
- must improve feerate diagram
- parent feerate must be < package feerate
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@glozow pushed some more detailed docs, thanks!
53ed8fd
to
99c0f92
Compare
@glozow thanks for the review, all comments addressed |
🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the Possibly this is due to a silent merge conflict (the changes in this pull request being Leave a comment here, if you need help tracking down a confusing failure. |
silent merge conflict with #29939 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
some test review
package_child = self.wallet.create_self_transfer(fee_rate=1000*DEFAULT_FEE, utxo_to_spend=package_parent["new_utxos"][0]) | ||
|
||
pkg_results = node.submitpackage([package_parent["hex"], package_child["hex"]]) | ||
assert_equal(f"package RBF failed: too many potential replacements, rejecting replacement {package_parent['tx'].rehash()}; too many potential replacements (102 > 100)\n", pkg_results["package_msg"]) | ||
self.assert_mempool_contents(expected=expected_txns) | ||
|
||
# Make singleton tx to conflict with in next batch | ||
singleton_coin = self.coins[-1] | ||
del self.coins[-1] | ||
singleton_tx = self.wallet.create_self_transfer(utxo_to_spend=singleton_coin) | ||
node.sendrawtransaction(singleton_tx["hex"]) | ||
expected_txns.append(singleton_tx["tx"]) | ||
|
||
# Double-spend same set minus last, and double-spend singleton. This hits 101 evictions; should still fail. | ||
# N.B. we can't RBF just a child tx in the clusters, as that would make resulting cluster of size 3. | ||
double_spending_coins = parent_coins[:-1] + [singleton_coin] | ||
package_parent = self.wallet.create_self_transfer_multi(utxos_to_spend=double_spending_coins, fee_per_output=10000) | ||
package_child = self.wallet.create_self_transfer(fee_rate=1000*DEFAULT_FEE, utxo_to_spend=package_parent["new_utxos"][0]) | ||
pkg_results = node.submitpackage([package_parent["hex"], package_child["hex"]]) | ||
assert_equal(f"package RBF failed: too many potential replacements, rejecting replacement {package_parent['tx'].rehash()}; too many potential replacements (101 > 100)\n", pkg_results["package_msg"]) | ||
self.assert_mempool_contents(expected=expected_txns) | ||
|
||
# Finally, evict MAX_REPLACEMENT_CANDIDATES | ||
package_parent = self.wallet.create_self_transfer_multi(utxos_to_spend=parent_coins[:-1], fee_per_output=10000) | ||
package_child = self.wallet.create_self_transfer(fee_rate=10000*DEFAULT_FEE, utxo_to_spend=package_parent["new_utxos"][0]) | ||
pkg_results = node.submitpackage([package_parent["hex"], package_child["hex"]], maxfeerate=0) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: you change the fee in the last successful one, which is a little bit sus since fee shouldn't matter in this test
Maybe create 2 constants for the parent fee_per_output
and child fee/feerate, and use them for each round?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I turned off maxfeerate, jacked up the child feerates, and made constants for both
node.submitpackage(package_hex1) | ||
self.assert_mempool_contents(expected=package_txns1) | ||
# The first two transactions have the same conflicts | ||
package_duplicate_conflicts_hex = [package_hex2[0]] + package_hex3 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't really understand what's being tested here - this doesn't have anything to do with package RBF? This isn't a topologically valid package since the transactions aren't related to each other in addition to being double spends. #30066 seems closer to what this is going for?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed, it's perhaps an older badly done version of #30066, removed
fee_rate=0, | ||
fee=DEFAULT_FEE, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These 2 things are contradictory. AFAICT fee_rate
is ignored, delete?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
removed a series of these ignored args
self.assert_mempool_contents(expected=expected_txns) | ||
|
||
# Now make conflicting packages for each coin | ||
package_hex1, package_txns1 = self.create_simple_package(coin1, DEFAULT_FEE, DEFAULT_FEE * 4) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: more varied fees for the packages for the wrong_conflict_cluster_size_*
tests, could also use constant
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
picked a constant and used it
assert_greater_than_or_equal, | ||
assert_raises_rpc_error, | ||
assert_equal, | ||
fill_mempool, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is now in mempool_util
(= linter complaint)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
src/validation.cpp
Outdated
const auto& child_ws = workspaces[1]; | ||
|
||
// Use the child as the transaction for attributing errors to. | ||
const Txid child_hash = child_ws.m_ptx->GetHash(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit; make reference not copy
const Txid child_hash = child_ws.m_ptx->GetHash(); | |
const Txid& child_hash = child_ws.m_ptx->GetHash(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
src/validation.cpp
Outdated
"package RBF failed: parent paying for child replacement", | ||
strprintf("package feerate is %s while parent feerate is %s", package_feerate.ToString(), parent_feerate.ToString())); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry to nit my own previous suggestion, but I think "parent paying for child replacement" might still be too specific, since it could be that the child has nothing to replace but is lower feerate (like package_txns6
in the functional test).
Maybe
"package RBF failed: parent paying for child replacement", | |
strprintf("package feerate is %s while parent feerate is %s", package_feerate.ToString(), parent_feerate.ToString())); | |
"package RBF failed: package feerate is less than parent feerate", | |
strprintf("package feerate %s <= parent feerate %s", package_feerate.ToString(), parent_feerate.ToString())); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
taken
package_hex5, package_txns5 = self.create_simple_package(coin, parent_fee=DEFAULT_FEE, child_fee=DEFAULT_FEE) | ||
node.submitpackage(package_hex5) | ||
self.assert_mempool_contents(expected=package_txns5) | ||
package_hex6, package_txns6 = self.create_simple_package(coin, parent_fee=DEFAULT_FEE + fee_delta, child_fee=DEFAULT_FEE - (fee_delta / 2)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Given that fee_delta
is a multiple of DEFAULT_FEE
and you're using fee_delta / 2
here... I think the general readability of this test would be improved if you created a constant base unit FEE
which was equal to, say, DEFAULT_FEE / 10
(or 104 * 10
which would give most of the transactions whole number feerates), and then used FEE
, 2*FEE
, 8*FEE
, etc. in all the tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I created DEFAULT_CHILD_FEE
and used that pretty much everywhere it made sense
self.generate(node, 1) | ||
|
||
def test_child_conflicts_parent_mempool_ancestor(self): | ||
fill_mempool(self, self.nodes[0], self.wallet) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
with rebase
fill_mempool(self, self.nodes[0], self.wallet) | |
fill_mempool(self, self.nodes[0]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
|
||
- *Rationale*: Basic support for package RBF can be used by wallets | ||
by making chains of no longer than two, then directly conflicting | ||
those chains when needed. Combined with V3 transactions this can |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe there should be some v3 test coverage (including nice 0fee parent package RBF)?
diff --git a/test/functional/mempool_accept_v3.py b/test/functional/mempool_accept_v3.py
index 8285b82c19..990d36359e 100755
--- a/test/functional/mempool_accept_v3.py
+++ b/test/functional/mempool_accept_v3.py
@@ -579,6 +579,32 @@ class MempoolAcceptV3(BitcoinTestFramework):
)
self.check_mempool([tx_with_multi_children["txid"], tx_with_sibling3_rbf["txid"], tx_with_sibling2["txid"]])
+ @cleanup(extra_args=["-acceptnonstdtxn=1"])
+ def test_package_rbf(self):
+ self.log.info("Test package RBF: v3 0-fee parent + high-fee child replaces parent's conflicts")
+ node = self.nodes[0]
+ # Reuse the same coins so that the transactions conflict with one another.
+ parent_coin = self.wallet.get_utxo(confirmed_only=True)
+
+ # package1 pays default fee on both transactions
+ parent1 = self.wallet.create_self_transfer(utxo_to_spend=parent_coin, version=3)
+ child1 = self.wallet.create_self_transfer(utxo_to_spend=parent1["new_utxo"], version=3)
+ package_hex1 = [parent1["hex"], child1["hex"]]
+ fees_package1 = parent1["fee"] + child1["fee"]
+ submitres1 = node.submitpackage(package_hex1)
+ assert_equal(submitres1["package_msg"], "success")
+ self.check_mempool([parent1["txid"], child1["txid"]])
+
+ # package2 has a 0-fee parent (conflicting with package1) and very high fee child
+ parent2 = self.wallet.create_self_transfer(utxo_to_spend=parent_coin, fee=0, fee_rate=0, version=3)
+ child2 = self.wallet.create_self_transfer(utxo_to_spend=parent2["new_utxo"], fee=fees_package1*10, version=3)
+ package_hex2 = [parent2["hex"], child2["hex"]]
+
+ submitres2 = node.submitpackage(package_hex2)
+ assert_equal(submitres2["package_msg"], "success")
+ assert_equal(set(submitres2["replaced-transactions"]), set([parent1["txid"], child1["txid"]]))
+ self.check_mempool([parent2["txid"], child2["txid"]])
+
def run_test(self):
self.log.info("Generate blocks to create UTXOs")
@@ -598,6 +624,7 @@ class MempoolAcceptV3(BitcoinTestFramework):
self.test_reorg_2child_rbf()
self.test_v3_sibling_eviction()
self.test_reorg_sibling_eviction_1p2c()
+ self.test_package_rbf()
if __name__ == "__main__":
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
was wondering about this case, added with some modifications for test simplicity
722d1a5
to
7fd321d
Compare
The behavior is not new, but this rule exits earlier than before. Previously, a carve out could have been granted in PreChecks() but then nullified in PackageMempoolChecks() when CheckPackageLimits() is called with the default limits.
No change in behavior. For single transaction acceptance, this is a simple refactor: Workspace::m_all_conflicting -> MemPoolAccept::m_all_conflicts Workspace::m_conflicting_fees -> MemPoolAccept::m_conflicting_fees Workspace::m_conflicting_size -> MemPoolAccept::m_conflicting_size Workspace::m_replaced_transactions -> MemPoolAccept::m_replaced_transactions And local variables m_total_vsize and m_total_modified_fees are now MemPoolAccept members so they can be accessed from PackageMempoolChecks. We want these to be package-wide variables because - Transactions could conflict with the same tx (just not the same prevout), or their conflicts could share descendants. - We want to compare conflicts with the package fee rather than individual transaction fee. We reset these MemPoolAccept-wide fields for each subpackage evaluation to not cause state leaking, similar to temporary coins.
Relax assumptions aabout in-mempool children of in-mempool parents. With package RBF, we will allow a package of size 2 with conflicts on its parent and reconsider the parent if its fee is insufficient on its own. Consider: TxA (in mempool) <- TxB (in mempool) TxA (in mempool) <- TxB' (in package, conflicts with TxB) <- TxD (in package) If TxB' fails to RBF TxB due to insufficient feerate, the package TxB' + TxC will be considered. PackageV3Checks called on TxB' will see an in-mempool parent TxA, and see the in-mempool child TxB. We cannot assume there is no in-mempool sibling, rather detect it and fail normally. Prior to package RBF, this would have failed on the first conflict in package.
Support package RBF where the conflicting package would result in a mempool cluster of size two, and each of its direct conflicts are also part of an up-to-size-2 mempool cluster. This restricted topology allows for exact calculation of miner scores for each side of the equation, reducing the surface area for new pins, or incentive-incompatible replacements. This allows wallets to create simple CPFP packages that can fee bump other simple CPFP packages. This, leveraged with other restrictions such as V3 transactions, can create pin-resistant applications. Future package RBF relaxations can be considered when appropriate. Co-authored-by: glozow <gloriajzhao@gmail.com> Co-authored-by: Greg Sanders <gsanders87@gmail.com>
7fd321d
to
ce3a662
Compare
rebased due to conflict |
While I wouldn't trust myself to correctly code-review this PR, I'd be happy to work on e2e tests that would leverage this for lightning channels fee-bumping (based on eclair) if it can help validate the logic and get this PR merged. I'd like to highlight again how important this feature is for lightning (and probably for many other L2 protocols on top of bitcoin today). This is the critical step that allows us to mitigate pinning of a commitment transaction, and guarantee that we're able to set the fees of our commitment package to a value that should ensure timely confirmation. |
sorry forgot to link #30072 here, added to OP |
based on #30072 , please review that one first
Allows any 2 transaction package with no in-mempool ancestors to do package RBF when directly conflicting with other mempool clusters of size two or less.
Proposed validation steps:
Post-cluster mempool this will likely be expanded to general package rbf, but this is what we can safely support today.