Stratum v2 Template Provider (take 3) #29432

Sjors · 2024-02-14T11:34:49Z

Based on on @Fi3's master...Fi3:bitcoin:PatchTemplates which is based on @ccdle12's #27854. I rebased it and re-wrote the commit history. Compared to #28983 it introduces EllSwift in the handshake and fixes various bugs. I used that opportunity to change the branch name, which makes testing against SRI slightly easier. There's no conceptual discussion on #28983 so it can be ignored by reviewers.

See docs/stratum-v2.md for a brief description of Stratum v2 and the role of Bitcoin Core in that system..

What to test and review?

I'll make separate pull requests for parts that are ready for detailed review.

Stratum v2 Noise Protocol #29346
Introduce Mining interface #30200
CKey: add Serialize and Unserialize #29295
Custom signet mining improvement: signet: fixing mining for OP_TRUE challenge #29032
Make nested pull-requests easier: Support self-hosted Cirrus workers on forks #29274

See the testing guide for various ways to test this PR. This branch is actively used by (testnet) pools, so it should be ready for high level review.

Related useful PRs

test: add mocked Sock that can read/write custom data and/or CNetMessages #30205

Related useful issues

Make Transport independent of CNetMessage and CSerializedNetMsg #30209

Implementation notes

There's roughly three layers:

Noise encryption Stratum v2 Noise Protocol #29346
Messages and transport layer
The Template Provider

the ci: commits (Support self-hosted Cirrus workers on forks #29274) are there to facilitate PR's against this branch, but they are not blocking for Stratum v2
the commits that move transport.h and some other stuff from node to common are not blocking. But in the longer run I'd like to see process separation between the node and the template provider.
I will occasionally add commits to undo bug fixes, in order to stay compatible with the SRI main branch. Those will get dropped over time and can be ignored.

Contributing

If you want to help out with any of the issues below, please open a PR to my fork. I will then squash your commits into my own where needed.

Things left todo

Spec

modify spec to use ProvideMissingTransactions? (followup?)
pick a good default for default_coinbase_tx_additional_output_size (see getblocktemplate RPC)

Networking

add -sv2bind and -sv2allowip
optional -sv2cert
drop Sv2TemplateProvider::SendBuf, reuse p2p socket handling if possible
limit number of connected clients
maybe limit (number of) coinbase_output_max_additional_size
TMP / TODO comments at the top of sv2_messages.h

Testing

expand sv2_template_provider_tests
add transport fuzzer
add template provider fuzzer

Template generation and updating

group templates with the same coinbase_tx_additional_output_size
don't generate templates when no client is connected

Misc

move / refactor GetMerklePathForCoinbase helper (see Future Work in original PR, and Fix NewTemplate.merkle_path in bitcoind stratum-mining/stratum#567)
truncate long trace messages

Potential followups

implement Noise protocol and mock client in Python, add functional tests (based on test/sv2_template_provider_tests.cpp)
use process separation, e.g. a bitcoin-tp binary, see multiprocess.md
make template updates push based, on top of Cluster Mempool, see docs/stratum-v2.md (for new blocks it's already push based)
push empty template for the next block (downstream can ignore or use, Implement a clever way to create and manage future jobs stratum-mining/stratum#715)
- send prevhash for this template as soon as any new block arrives
push optimistic template for the next block
- send prevhash if and only if our template won (i.e. we got a SubmitSolution message)

DrahtBot · 2024-02-14T11:34:52Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.
A summary of reviews will appear here.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#30203 (Enhance signet chain configuration in bitcoin.conf by BrandonOdiwuor)
#30200 (Introduce Mining interface by Sjors)
#30141 (kernel: De-globalize validation caches by TheCharlatan)
#30130 (contrib/signet/miner: increase miner search space by edilmedeiros)
#30051 (crypto, refactor: add new KeyPair class by josibake)
#29876 (build: add -Wundef by fanquake)
#29838 (Feature: Use different datadirs for different signets by BrandonOdiwuor)
#29775 (Testnet4 including PoW difficulty adjustment fix by fjahr)
#29686 (Update manpage descriptions by willcl-ark)
#29415 (Broadcast own transactions only via short-lived Tor or I2P connections by vasild)
#29015 (kernel: Streamline util library by ryanofsky)
#28843 ([refactor] Remove BlockAssembler m_mempool member by TheCharlatan)
#28710 (Remove the legacy wallet and BDB dependency by achow101)
#28417 (contrib/signet/miner updates by ajtowns)
#26697 (logging: use bitset for categories by LarryRuane)
#10102 (Multiprocess bitcoin by ryanofsky)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

DrahtBot · 2024-02-14T12:27:21Z

🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the
documentation.

Possibly this is due to a silent merge conflict (the changes in this pull request being
incompatible with the current code in the target branch). If so, make sure to rebase on the latest
commit of the target branch.

Leave a comment here, if you need help tracking down a confusing failure.

_{Debug: https://github.com/bitcoin/bitcoin/runs/21562393655}

Sjors · 2024-02-14T14:17:54Z

src/test/sv2_template_provider_tests.cpp

+    {
+        bool started = m_tp->Start(Sv2TemplateProviderOptions { .port = 18447 });
+        if (! started) return false;
+        // Avoid "Connection refused" on CI:


The template provider tests are quite brittle because they use a real socket.

For the time being I just added handling for MSG_MORE (on e.g. macOS sequential messages are sent separately while on Linux they're combined). I also made the timeouts a bit longer.

Hopefully that does the trick. This can be revisited closer to the time when the Template Provider is ready for its own PR.

Except the macOS native CI... https://github.com/Sjors/bitcoin/actions/runs/7905485495/job/21578223892?pr=34#step:7:6060

Working on a fix in Sjors#34

I should probably look into using StaticContentsSock

@vasild any thoughts on how to make mock Socks that can be used to play messages in two directions?

Yes! See the first two commits in #26812:

bee6bdf test: put the generic parts from StaticContentsSock into a separate class
f42e4f3 test: add a mocked Sock that allows inspecting what has been Send() to it

and then how to use that in the last commit of the same PR:

8b10990 test: add unit tests exercising full call chain of CConnman and PeerManager

With those it is possible to send/receive raw bytes to/from the (mocked) socket, or NetMsgs, e.g.:

pipes->recv.PushNetMsg(NetMsgType::GETBLOCKS, block_locator, hash_stop);

Sjors · 2024-02-14T20:04:56Z

src/test/sv2_template_provider_tests.cpp

+        ss << TX_WITH_WITNESS(tx);
+        tx_size = ss.size();
+    }
+


TSAN is tripping up somewhere around here. The last thing it logs is - Connect 2 transactions:. It doesn't get to - Verify ... txins:. I wonder if this is related to mock time, which I'm testing in Sjors#34

@maflcko shouldn't TSan on CI output something useful about why it crashed? I currently only says "error 2": https://cirrus-ci.com/task/5124733717446656?logs=ci#L3531

When running this locally on Ubuntu with clang 16.0.6 I get a WARNING: ThreadSanitizer: data race and significantly more details (still a bit cryptic, but hopefully enough to figure out what's happening).

Ah, I guess the unit tests don't capture the tsan output?

But they should. At least back when I tested #27667

Maybe good to re-check this when/after the cmake migration is done?

Sjors · 2024-02-15T14:07:31Z

Added m_tp_mutex to Sv2TemplateProvider.

Sjors · 2024-02-15T15:12:13Z

Bumping macOS to 14 on the CI does not help (tried in Sjors#35). I also can't reproduce this failure on my own Intel macOS machines, not on 13.6.4 and not on 14.2.1. A Sock mock is probably the most robust solution, but it'd be nice to find another workaround.

This extra delay seems to do the trick for now: Sjors@c8d10af

Another option to consider is using the functional test framework instead, since these are not really unit tests. However that involves implementing the sv2 noise protocol in Python and a bunch of other work to export transport functions to the functional test framework. If anyone feels up to that challenge, let me know...

Sjors · 2024-06-07T14:25:44Z

Updated to use the interface proposed in #30200. This also fixes a small bug: -blockmintxfee and -blockmaxweight are no longer ignored. When the latter argument is unset, by default blocks have a 4000 byte safety margin, just like with getblocktemplate.

This makes the options argument mandatory for BlockAssembler constructor, dropping implicit handling of ArgsManager. The caller i.e. the Mining interface implementation now handles this. In Stratum v2 the pool communicates how many extra bytes it needs for its own outputs (payouts, extra commitments, etc). This needs to be substracted from what the user set as -blockmaxweight. To achieve that the caller would have to pass in an options object, and not forget to also process -blockmintxfee.

Set tip at the start of the function and only update it for a long poll.

Co-Authored-By: Christopher Coverdale <chris.coverdale24@gmail.com>

The template provider will listen for a Job Declarator client. It can establish a connection and detect various protocol errors. Co-Authored-By: Christopher Coverdale <chris.coverdale24@gmail.com> Co-Authored-By: Fi3

Co-Authored-By: Christopher Coverdale <chris.coverdale24@gmail.com>

Co-authored-by: Ryan Ofsky <ryan@ofsky.org>

This was referenced Feb 14, 2024

Stratum v2 Template Provider (take 2) #28983

Closed

Stratum v2 Noise Protocol #29346

Open

DrahtBot added the CI failed label Feb 14, 2024

Sjors force-pushed the sv2 branch 3 times, most recently from 87f0527 to 684890f Compare February 14, 2024 14:15

Sjors commented Feb 14, 2024

View reviewed changes

This was referenced Feb 14, 2024

log: deduplicate category names and improve logging.cpp #29419

Merged

rpc: provide per message stats for global traffic via new RPC 'getnetmsgstats' #29418

Open

Broadcast own transactions only via short-lived Tor or I2P connections #29415

Open

Sjors force-pushed the sv2 branch from 684890f to 50b9afa Compare February 14, 2024 15:50

Sjors force-pushed the sv2 branch from 50b9afa to 1bacb89 Compare February 14, 2024 20:01

Sjors commented Feb 14, 2024

View reviewed changes

This was referenced Feb 14, 2024

logging: use bitset for categories #26697

Open

tracing: Only prepare tracepoint arguments when actually tracing #26593

Open

Add a "tx output spender" index #24539

Open

Sjors force-pushed the sv2 branch 2 times, most recently from 90735e6 to 1e06c04 Compare February 15, 2024 14:05

Sjors force-pushed the sv2 branch from 1e06c04 to 6242776 Compare February 15, 2024 15:35

Sjors force-pushed the sv2 branch from 316857c to d18b951 Compare June 7, 2024 14:32

Sjors and others added 13 commits June 7, 2024 17:06

rpc: getTransactionsUpdated via miner interface

df2c40f

rpc: call processNewBlock via miner interface

5d8fe1e

rpc: minize getTipHash() calls in gbt

033bdf2

Set tip at the start of the function and only update it for a long poll.

rpc: call IsInitialBlockDownload via miner interface

c3f11c2

Stratum v2 template provider scaffold

3303b1f

Co-Authored-By: Christopher Coverdale <chris.coverdale24@gmail.com>

Template provider sv2 connection support

d27cf39

The template provider will listen for a Job Declarator client. It can establish a connection and detect various protocol errors. Co-Authored-By: Christopher Coverdale <chris.coverdale24@gmail.com> Co-Authored-By: Fi3

Persist static key for Template Provider

f946b9f

Subtract pool reserved bytes in createNewBlock

41ac95e

Sv2: construct and submit block templates

05832ab

Co-Authored-By: Christopher Coverdale <chris.coverdale24@gmail.com>

Incrementally update sv2 block template

a35673a

Reject RequestTransactionData for stale templates

91d2bcd

test: Sv2TemplateProvider tests

9f9dbf1

Sjors force-pushed the sv2 branch from d18b951 to 9c137b9 Compare June 7, 2024 15:11

Sjors and others added 8 commits June 7, 2024 17:16

fixup: use mining interface

5078e51

signet: miner skips PSBT step for OP_TRUE

716b799

doc: explain Stratum v2 design, testing and usage

452f5cd

ci: forks can opt-out of CI branch push

fecfd89

ci: skip arm if no worker is configured

ff110b4

ci: skip Github CI on branch pushes for forks

a4ee8cf

ci: clarify Cirrus self-hosted workers setup

b312459

ci: test-each-commit merge base optional

f6ed0e5

Co-authored-by: Ryan Ofsky <ryan@ofsky.org>

Sjors force-pushed the sv2 branch from 9c137b9 to f6ed0e5 Compare June 7, 2024 15:16

This was referenced Jun 7, 2024

refactor: Improve assumeutxo state representation #30214

Open

[refactor] Remove BlockAssembler m_mempool member #28843

Open

Remove the legacy wallet and BDB dependency #28710

Draft

build: add -Wundef #29876

Draft

Multiprocess bitcoin #10102

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stratum v2 Template Provider (take 3) #29432

Stratum v2 Template Provider (take 3) #29432

Sjors commented Feb 14, 2024 •

edited

DrahtBot commented Feb 14, 2024 •

edited

DrahtBot commented Feb 14, 2024

Sjors Feb 14, 2024

Sjors Feb 14, 2024

Sjors Feb 14, 2024

Sjors Feb 14, 2024

Sjors Feb 14, 2024

Sjors Feb 15, 2024

vasild Feb 16, 2024

Sjors Feb 14, 2024 •

edited

Sjors Feb 15, 2024

maflcko Feb 15, 2024

maflcko Feb 15, 2024

maflcko Feb 15, 2024

Sjors commented Feb 15, 2024

Sjors commented Feb 15, 2024 •

edited

Sjors commented Jun 7, 2024

Stratum v2 Template Provider (take 3) #29432

Are you sure you want to change the base?

Stratum v2 Template Provider (take 3) #29432

Conversation

Sjors commented Feb 14, 2024 • edited

What to test and review?

Related useful PRs

Related useful issues

Implementation notes

Contributing

Things left todo

Spec

Networking

Testing

Template generation and updating

Misc

Potential followups

DrahtBot commented Feb 14, 2024 • edited

Code Coverage

Reviews

Conflicts

DrahtBot commented Feb 14, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Sjors Feb 14, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Sjors commented Feb 15, 2024

Sjors commented Feb 15, 2024 • edited

Sjors commented Jun 7, 2024

Sjors commented Feb 14, 2024 •

edited

DrahtBot commented Feb 14, 2024 •

edited

Sjors Feb 14, 2024 •

edited

Sjors commented Feb 15, 2024 •

edited