Drop -dbcache limit #28358

Sjors · 2023-08-28T15:57:20Z

Due to recent UTXO set growth, the current maximum value for -dbcache of 16GB is ~~just months away from being~~ insufficient (for those who wish to complete IBD with the UTXO set held in RAM).

This drops the limit. It also adds a warning that it's up to users to check that they have enough RAM.

Fixes #28249.

A previous version of this PR increased the maximum to 64GB. It also made startup abort if the value provided is too high, rather than quietly round it down. But this didn't get much support.

DrahtBot · 2023-08-28T15:57:23Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

Type	Reviewers
Concept ACK	kristapsk, TheCharlatan, henrikhk, 0xB10C, darosior, jonatack, tdb3, BenWestgate, sdaftuar
Stale ACK	cbergqvist

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#30059 (Add option dbfilesize to control LevelDB target ("max") file size by luke-jr)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

maflcko · 2023-08-28T16:11:57Z

insufficient

Can you explain this? Why is there a requirement that the whole utxo set must fit in the cache? This can easily lead to OOM, as you explain, in which case the user would have to start from zero, because potentially nothing has been persisted at all. Also, for slow storage it could lead to hangs, which could lead the user (or OS) to kill the process, causing the same issues. Finally, with assumeutxo, I wonder why there is any need to chase this? No objection to this pull, but I wonder if this really makes users happier on average, or will just lead to more hangs, crashes and -reindex.

maflcko · 2023-08-28T16:17:14Z

doc/release-notes-25358.md

+Node
+------
+
+- The maximum value for `-dbcache` has been increased to 32,000 MiB due to recent UTXO set growth. Be sure to check that you have enough RAM before increasing this.


Instead of implying that users should check their RAM and then increase this, it would be more helpful to explain why a user should change the setting in the first place, what they can expect, and what the tradeoffs are, but this seems better put in a separate document or the -help.

That was not the intended implication. If they feel like increasing it, they should check their RAM first.

It is just that this is the only thing you mention in the release notes. In any case, I don't think the release notes are the right place for extended documentation. Either put it in the -help or a separate md.

I dropped the "check RAM" sentence.

jamesob · 2023-08-28T16:25:13Z

This can easily lead to OOM

Wonder if it's worth warning/failing if the dbcache param exceeds available memory.

Sjors · 2023-08-28T16:40:37Z

My recollection is that an IBD without a single flush is faster, but have not benched this in recent years.

We can always recommend a lower setting if that turns out to be better. By not increasing it to infinity at least we constrain the worst case here, e.g. if it turns out the failures you mean get exponentially worse as the UTXO set keeps growing. While at the same time not removing the existing ability to sync in one go.

Sjors · 2023-08-28T16:41:08Z

if the dbcache param exceeds available memory.

My understanding was that we can't measure available RAM reliably.

andrewtoth · 2023-08-28T16:50:28Z

My recollection is that an IBD without a single flush is faster, but have not benched this in recent years.

A single flush throughout the entire multi-hour IBD will not make a noticeable difference to a user. My initial version of #28280 had 3 extra pointers per cache entry, which caused the dbcache limit to be hit. It was still >2% faster than master which did not have a single flush.

kristapsk · 2023-08-28T17:00:45Z

Concept ACK. This does not change defaults, just gives more options to power users. And UTXO set will grow in future. For some server environments 16 GB more or less will mean nothing.

TheCharlatan · 2023-08-29T05:21:16Z

Concept ACK.

henrikhk · 2023-08-29T16:51:37Z

Concept ACK.

BenWestgate · 2023-08-30T02:37:10Z

Concept ACK.

Except for:

make startup abort if the value provided is too high, rather than quietly rounding it down.

This breaks scripts that set dbcache to a % of available RAM.

And is inconsistent with it rounding up silently rather than fail to start if the value is too low.

0xB10C · 2023-08-30T07:46:22Z

Concept ACK on increasing the maximum dbcache.

I had to manually bump the dbcache to run bitcoin-cli verifychain 4 0 last year (#24851 (comment)). This came close to using 32 GB and might not be enough anymore (or soon - I haven't checked again). Maybe worth considering going directly to 64 GB here to be able to actually use the verifychain 4 0 RPC?

darosior · 2023-08-30T07:57:38Z

Concept ACK on giving more options.

Sjors · 2023-08-30T08:14:02Z

This breaks scripts that set dbcache to a % of available RAM.

Do you know any example of people doing that?

In general I think we should fail to startup if the user provides an invalid configuration option, as this often leads to problems. I believe this has been gradually introduced for various other options too (see also #16545).

At the same time I don't want to needless break scripts out there.

From my own experience I was very surprised to find that this rounding down behavior happened, which is which I prefer an error. But we could also print a warning in the debug log.

Maybe worth considering going directly to 64 GB here to be able to actually use the verifychain 4 0 RPC?

@sipa any thoughts on a sane maximum?

BenWestgate · 2023-08-30T12:57:35Z

This breaks scripts that set dbcache to a % of available RAM.

Do you know any example of people doing that?

My project Bails puts this in bitcoin-qt.desktop as well as .config/autostart/Bitcoin/bitcoin.desktop:
-dbcache=$(($(grep Available /proc/meminfo | sed s/[^0-9]//g)/1024-2000))

It's an alpha with about 20 users, makes a portable USB stick node that keeps Bitcoin Core up to date, none probably own a 64GB computer, but it will break if they do.

I found a couple other projects with the idea:
Start9Labs/bitcoind-startos#39
JWWeatherman/yeticold#145
epiccurious/bitcoin-core-node-builder#38

They'll waste time if they miss the change notes.
Maybe a one time warning would be better when the value rounds up or down? It took me a long time to notice that it did that.

jonatack

Concept ACK

In general I think we should fail to startup if the user provides an invalid configuration option, as this often leads to problems. I believe this has been gradually introduced for various other options too (see also #16545).

#27632 was another recent example of switching from a warning to raising an error. It's too easy for users, even experienced ones, not to notice a warning line in the log.

jonatack · 2023-08-30T14:28:15Z

src/init.cpp

@@ -446,7 +446,7 @@ void SetupServerArgs(ArgsManager& argsman)
    argsman.AddArg("-conf=<file>", strprintf("Specify path to read-only configuration file. Relative paths will be prefixed by datadir location (only useable from command line, not configuration file) (default: %s)", BITCOIN_CONF_FILENAME), ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);
    argsman.AddArg("-datadir=<dir>", "Specify data directory", ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);
    argsman.AddArg("-dbbatchsize", strprintf("Maximum database write batch size in bytes (default: %u)", nDefaultDbBatchSize), ArgsManager::ALLOW_ANY | ArgsManager::DEBUG_ONLY, OptionsCategory::OPTIONS);
-    argsman.AddArg("-dbcache=<n>", strprintf("Maximum database cache size <n> MiB (%d to %d, default: %d). In addition, unused mempool memory is shared for this cache (see -maxmempool).", nMinDbCache, nMaxDbCache, nDefaultDbCache), ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);
+    argsman.AddArg("-dbcache=<n>", strprintf("Maximum database cache size <n> MiB (%d to %d, default: %d). Make sure you have enough RAM. In addition, unused mempool memory is shared for this cache (see -maxmempool).", nMinDbCache, nMaxDbCache, nDefaultDbCache), ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);


While touching this help, I think the following would be much clearer (particularly s/for/with).

Suggested change

argsman.AddArg("-dbcache=<n>", strprintf("Maximum database cache size <n> MiB (%d to %d, default: %d). Make sure you have enough RAM. In addition, unused mempool memory is shared for this cache (see -maxmempool).", nMinDbCache, nMaxDbCache, nDefaultDbCache), ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);

argsman.AddArg("-dbcache=<n>", strprintf("Maximum database cache size <n> MiB (%d to %d, default: %d). Make sure you have enough RAM. In addition, unused memory allocated to the mempool is shared with this cache (see -maxmempool).", nMinDbCache, nMaxDbCache, nDefaultDbCache), ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);

jonatack · 2023-08-30T14:31:26Z

src/init.cpp

@@ -925,6 +925,10 @@ bool AppInitParameterInteraction(const ArgsManager& args)
        return InitError(Untranslated("Cannot set -listen=0 together with -listenonion=1"));
    }

+    if (args.GetIntArg("-dbcache", nDefaultDbCache) >  nMaxDbCache) {
+        return InitError(Untranslated(strprintf("-dbcache must be at most %d MiB", nMaxDbCache)));
+    }


Could add a test.

Edit: maybe -dbcache configuration option can be at most %d MiB, but %d was passed.

Used the text, added a test.

jonatack · 2023-08-30T14:33:00Z

doc/release-notes-25358.md

+Node
+------
+
+- The maximum value for `-dbcache` has been increased to 32,000 MiB due to recent UTXO set growth. Be sure to check that you have enough RAM before increasing this.


Suggested change

- The maximum value for `-dbcache` has been increased to 32,000 MiB due to recent UTXO set growth. Be sure to check that you have enough RAM before increasing this.

- The maximum allowed value for the `-dbcache` configuration option has been increased to 32,000 MiB due to recent UTXO set growth. Be sure to check that you have enough RAM before increasing this. (#28358)

I might also add something like "Even with sufficient RAM the maximum setting may not be optimal."

Yes -- and why, or link to an improved In-memory caches section in doc/reduce-memory.md.

Taken the suggested text diff.

jonatack · 2023-08-30T14:35:01Z

doc/release-notes-25358.md

@@ -0,0 +1,5 @@
+Node


Maybe (I'm not sure this matters).

Suggested change

Node

Updated settings

Release notes get edited later anyway, but I'll change once I retouch.

(waiting for more inputs in the discussion on warning/failure and the max size)

BenWestgate · 2023-08-30T14:50:38Z

Perhaps it should be set so high that you can't easily buy a Desktop with more RAM than the limit.

I could go to Microcenter today and walk out with a 64GB gaming tower, or a Xeon workstation with 128GB, likely a 64GB laptop as well.

Lots of guides on internet (foolishly) suggest setting it to half the total RAM.

maflcko · 2023-08-30T14:51:32Z

src/init.cpp

@@ -925,6 +925,10 @@ bool AppInitParameterInteraction(const ArgsManager& args)
        return InitError(Untranslated("Cannot set -listen=0 together with -listenonion=1"));
    }

+    if (args.GetIntArg("-dbcache", nDefaultDbCache) >  nMaxDbCache) {
+        return InitError(Untranslated(strprintf("-dbcache must be at most %d MiB", nMaxDbCache)));


Given that this will break software out there, why not simply keep the previous behavior and leave it as-is? What problem are you trying to solve?

Printing a warning and/or updating the docs should be more than enough.

That indeed seems like the best approach. Silently rounding down is annoying, but doesn't break anything, so a warning should be fine.

OTOH, it seems that raising would be a clearer user interface than rounding down silently as now, or with an added warning. See https://en.wikipedia.org/wiki/Principle_of_least_astonishment. I don't want to break anything, but the only software mentioned so far that this might marginally affect is that of @BenWestgate (thank you for reporting), who would be aware of the change.

I'm leaving this error as is for now, unless there are other examples of breaking software.

I would also rather drop nMaxDbCache entirely than quietly enforce it.

See https://en.wikipedia.org/wiki/Principle_of_least_astonishment

If I imagined myself running into this error. I'd be indeed astonished and ask myself why the dbcache is capped at all. Again, if there is a reason for this change, it should be explained. It can't hurt to put the reason in the error string.

luke-jr · 2023-09-05T20:52:19Z

Why do we have an upper limit at all?

My understanding was that we can't measure available RAM reliably.

#19873 does exactly that

luke-jr · 2023-09-05T21:55:53Z

src/txdb.h

@@ -38,7 +38,7 @@ static const int64_t nDefaultDbCache = 450;
 //! -dbbatchsize default (bytes)
 static const int64_t nDefaultDbBatchSize = 16 << 20;
 //! max. -dbcache (MiB)
-static const int64_t nMaxDbCache = sizeof(void*) > 4 ? 16384 : 1024;
+static const int64_t nMaxDbCache = sizeof(void*) > 4 ? 32768 : 1024;


Might as well make it constexpr while you're touching it

Sjors · 2024-02-13T14:10:15Z

Rebased, incorporated most of the text feedback above. Dropped the RAM warning from the release note, since that's already in the help text.

Why do we have an upper limit at all?

@sipa seemed worried about users going overboard with this setting: #28249 (comment)

maflcko · 2024-03-12T08:43:19Z

@maflcko do you prefer to drop the limit? We should not quietly change a user setting, so imo an invalid value should explicitly fail, rather than silently fail.

Why would the value be "invalid"?

Sjors · 2024-03-12T08:56:46Z

If there is a limit, a value above the limit is invalid. That is the entire point of a limit. Which is why I asked if you prefer to drop the limit.

maflcko · 2024-03-12T09:47:53Z

I don't know if it is safe to drop the limit. For example, one could run into a signed integer overflow for some values. If that is the reason for the limit, you should add a comment for it.

However, if there is no reason for the limit, I don't see why a limit should exist.

Sjors · 2024-03-12T10:50:41Z

It seems there's more enthusiasm for dropping this limit entirely, so I did that. This also impacts the GUI (but none of the translated strings), cc @hebasto.

For future reference, c016405 + 39a3d48 is the last version of this PR which does have a limit.

Sjors · 2024-03-12T10:56:16Z

src/qt/optionsdialog.cpp

@@ -94,8 +94,7 @@ OptionsDialog::OptionsDialog(QWidget* parent, bool enableWallet)
    ui->setupUi(this);

    /* Main elements init */
-    ui->databaseCache->setMinimum(nMinDbCache);
-    ui->databaseCache->setMaximum(nMaxDbCache);
+    ui->databaseCache->setRange(nMinDbCache, std::numeric_limits<int>::max());


Even on a 32 bit system the maximum would be 2 petabyte (2 billion MiB), so not a practical concern imo.

sipa · 2024-03-12T11:24:22Z

On 32-bit systems we shouldn't raise the cache size too close to 4 GiB or more, as it'll just result in memory allocation errors.

On 64-bit systems, the limit is much less necessary, but I'd still worry about wildly too large numbers, as they also affect leveldb cache sizes. Setting those (way) too large is probably counterproductive.

Sjors · 2024-03-12T12:26:00Z

@sipa since 32 bit systems can't have more than 4 GiB of RAM, the "Make sure you have enough RAM" wording in the documentation should cover that. Though perhaps "Stay at least 1 GiB below your RAM size" may be safer, also because of swap behaviour (see e.g. #29603 (comment))

With something like #19873 we could warn the use if they pick a value too close to the limit (on both 32 and 64 bit systems).

wildly too large

We could suggest an optimum, but without further experimentation we don't know where that is.

Ideally we would measure performance in real time and flush when it starts degrading (unless the user really insists that we don't).

cbergqvist · 2024-03-14T20:12:45Z

Built 5f31349 and ran bitcoin-qt because I was worried that setting a very high max value might cause issues with the QT UI control. It's not some kind of horizontal slider that goes crazy, but rather a textbox with up/down arrows. It ignores key inputs that would make the value go further than std::numeric_limits<int>::max(). So all good from that perspective.

hebasto · 2024-03-18T16:34:20Z

This also impacts the GUI (but none of the translated strings), cc @hebasto.

5f31349: changes to the GUI code look good to me. It might worth considering adding the "Make sure you have enough RAM." warning to the GUI as well (as a follow-up).

Sjors · 2024-03-18T16:37:14Z

"Make sure you have enough RAM." warning to the GUI as well (as a follow-up).

Will do if I need to retouch.

andrewtoth · 2024-05-14T03:53:17Z

After #28233 keeping the entire utxo set in memory will also help connect new blocks as fast as possible, not just for IBD.

Sjors · 2024-05-14T06:54:01Z

Rebased to make it easier to test the interaction with (merged) #28233. However, any potential benefit goes away after the first restart. The only time -dbcache is filled beyond its previous 16GB limit is during IDB.

In order to benefit from having the full UTXO set in RAM after a restart, we'd have to proactively load the entire UTXO set into -dbcache if space permits. Or maybe fill half the available space with coins, picked randomly.

Added "Make sure you have enough RAM." warning to the GUI.

andrewtoth · 2024-05-15T01:38:50Z

The only time -dbcache is filled beyond its previous 16GB limit is during IDB

I'm sure it will happen if you restart your node now and leave it running for long enough ;)

In order to benefit from having the full UTXO set in RAM after a restart, we'd have to proactively load the entire UTXO set into -dbcache if space permits.

I did make a crude attempt at that years ago #18941. Not sure it is worth it.

BenWestgate

crACK

BenWestgate · 2023-08-30T02:25:01Z

doc/release-notes-25358.md

+------
+
+- The maximum value for `-dbcache` has been increased to 32,000 MiB due to recent UTXO set growth. Be sure to check that you have enough RAM before increasing this.
+- Setting `-dbcache` above the maximum causes the node to abort startup. Previously it would round the number down.


I would prefer it quietly round down to the maximum. Especially since it rounds up to the minimum.

I know a few scripts people use that set dbcache to a percentage of available memory that this part of the PR would break for 64GB RAM.

What problem is failing to start meant to solve?

sdaftuar

Concept ACK.

I don't see much of a philosophical difference between having no limit and having a limit of 16GB on systems where a user has less than 16GB actually available... If we're already requiring users to do something smart, we can keep doing that -- and if we are worried about someone using an inappropriate value we can try to solve that separately (maybe by testing that we can allocate that much at startup or something?).

doc/release-notes-25358.md

BenWestgate · 2024-05-23T15:53:37Z

If we are worried about someone using an inappropriate value we can try to solve that separately (maybe by testing that we can allocate that much at startup or something?).

The best would be never setting it greater than total RAM. -dbcache=<total RAM> is still higher than optimal as there will be applications in memory that won't be swapped to disk including Bitcoin. But measuring Available RAM or swap is precarious as it changes.

Since other reviewers mentioned only wildly excessive values are seriously detrimental with swap, capping to max RAM is sensible.

In order to benefit from having the full UTXO set in RAM after a restart, we'd have to proactively load the entire UTXO set into -dbcache if space permits. Or maybe fill half the available space with coins, picked randomly.

Some have used cat .bitcoin/chainstate/* > /dev/null to make a node that's far behind start syncing faster. Because of the disk io to read the chainstate folder, there's a trade off between initial speed and medium-term speed. A node mere hours behind shouldn't read the entire UTXO set into memory. But one that's weeks behind will reach tip faster if it does.

Though perhaps "Stay at least 1 GiB below your RAM size" may be safer, also because of swap behaviour (see e.g. #29603 (comment))

That's barely safe without swap, Bitcoin-QT and the OS will use most of the 1 GiB offset leaving it pretty close to locking up.

Without a swap file, an offset from total ram (1-2 GiB?) may avoid OOM, but this becomes arbitrary and may limit performance if set too low. It's better users w/o swap who set it too high OOM, than to limit performance of systems w/ swap that may benefit from more.

So total memory is still the best limit, if any is enforced.

We could suggest an optimum, but without further experimentation we don't know where that is.

I've done a lot of tests on 4 GB and 8 GB RAM machines with zram swap and the -datadir on USB sticks and available_RAM - 2 GiB is about optimal for keeping those systems responsive. See this closed issue for more data: BenWestgate/Bails#41

With -dbcache=<available_RAM - 2 GiB> both 4 GB and 8 GB RAM systems can unlock the screen and manipulate the desktop without exceeding a 10 second lag, after a user locks the screen and lets Bitcoin sync several hours. With 4 GB RAM, this is approximately our default -dbcache=450 as suggested elsewhere to NOT raise it on 4 GB. If -dbcache is too high with a slow swap the machine may take minutes to respond. That's true even when the value is set lower than total or available memory. But whether that lag is undesirable or not depends on whether the user intends to use the machine or not before sync completes.

Suggesting dbcache=$(total_ram - 3650) as an optimum place to start matches what some of our devs have recommended on stack exchange.

Fixes bitcoin#28249

Sjors · 2024-05-28T06:42:23Z

@BenWestgate thanks for the analysis. If we want to add a recommendation to the help text, I suggest we do that in another PR though. It's probably going to take multiple people doing benchmark on wildly different machines.

Checking against RAM can also be done in a followup.

I rebased and changed the .md file name.

Sjors marked this pull request as ready for review August 28, 2023 15:57

Sjors force-pushed the 2023/08/double-your-coins---cache branch from 4e27771 to f311e45 Compare August 28, 2023 16:05

DrahtBot added the CI failed label Aug 28, 2023

maflcko reviewed Aug 28, 2023

View reviewed changes

DrahtBot removed the CI failed label Aug 28, 2023

jonatack reviewed Aug 30, 2023

View reviewed changes

maflcko reviewed Aug 30, 2023

View reviewed changes

DrahtBot added CI failed and removed CI failed labels Sep 3, 2023

luke-jr reviewed Sep 5, 2023

View reviewed changes

DrahtBot mentioned this pull request Nov 27, 2023

[refactor] Moved simple math calculations from runtime to compile time. #28947

Closed

DrahtBot added the CI failed label Jan 12, 2024

Sjors force-pushed the 2023/08/double-your-coins---cache branch from f311e45 to 1e10a82 Compare February 13, 2024 14:07

DrahtBot requested review from tdb3 and removed request for tdb3 March 5, 2024 08:44

willcl-ark mentioned this pull request Mar 8, 2024

Current default settings are broken, some fix is needed #29603

Closed

Sjors mentioned this pull request Mar 11, 2024

Raise maximum -dbcache setting #28249

Open

Sjors changed the title ~~Double -dbache maximum to 64GB~~ Quadruple -dbache maximum to 64GB Mar 11, 2024

Sjors mentioned this pull request Mar 11, 2024

rpc: Optimize serialization and enhance metadata of dumptxoutset output #29612

Merged

Sjors force-pushed the 2023/08/double-your-coins---cache branch from 39a3d48 to 5f31349 Compare March 12, 2024 10:50

Sjors changed the title ~~Quadruple -dbache maximum to 64GB~~ Drop -dbache limit Mar 12, 2024

Sjors commented Mar 12, 2024

View reviewed changes

Sjors changed the title ~~Drop -dbache limit~~ Drop -dbcache limit Mar 12, 2024

DrahtBot mentioned this pull request May 8, 2024

Add option dbfilesize to control LevelDB target ("max") file size #30059

Open

Sjors force-pushed the 2023/08/double-your-coins---cache branch from 5f31349 to 2a71819 Compare May 14, 2024 06:53

BenWestgate approved these changes May 23, 2024

View reviewed changes

sdaftuar reviewed May 23, 2024

View reviewed changes

doc/release-notes-25358.md Outdated Show resolved Hide resolved

validation: drop maximum -dbcache

bb3b980

Fixes bitcoin#28249

Sjors force-pushed the 2023/08/double-your-coins---cache branch from 2a71819 to bb3b980 Compare May 28, 2024 06:40

	argsman.AddArg("-dbcache=<n>", strprintf("Maximum database cache size <n> MiB (%d to %d, default: %d). Make sure you have enough RAM. In addition, unused mempool memory is shared for this cache (see -maxmempool).", nMinDbCache, nMaxDbCache, nDefaultDbCache), ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);
	argsman.AddArg("-dbcache=<n>", strprintf("Maximum database cache size <n> MiB (%d to %d, default: %d). Make sure you have enough RAM. In addition, unused memory allocated to the mempool is shared with this cache (see -maxmempool).", nMinDbCache, nMaxDbCache, nDefaultDbCache), ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);

	- The maximum value for `-dbcache` has been increased to 32,000 MiB due to recent UTXO set growth. Be sure to check that you have enough RAM before increasing this.
	- The maximum allowed value for the `-dbcache` configuration option has been increased to 32,000 MiB due to recent UTXO set growth. Be sure to check that you have enough RAM before increasing this. (#28358)

Drop -dbcache limit #28358

Are you sure you want to change the base?

Drop -dbcache limit #28358

Conversation

Sjors commented Aug 28, 2023 • edited

DrahtBot commented Aug 28, 2023 • edited

Code Coverage

Reviews

Conflicts

maflcko commented Aug 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jamesob commented Aug 28, 2023

Sjors commented Aug 28, 2023

Sjors commented Aug 28, 2023

andrewtoth commented Aug 28, 2023 • edited

kristapsk commented Aug 28, 2023

TheCharlatan commented Aug 29, 2023

henrikhk commented Aug 29, 2023

BenWestgate commented Aug 30, 2023

0xB10C commented Aug 30, 2023

darosior commented Aug 30, 2023

Sjors commented Aug 30, 2023 • edited

BenWestgate commented Aug 30, 2023

jonatack left a comment

Choose a reason for hiding this comment

jonatack Aug 30, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonatack Aug 30, 2023 • edited

Choose a reason for hiding this comment

Sjors Feb 13, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonatack Aug 30, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenWestgate commented Aug 30, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonatack Aug 31, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luke-jr commented Sep 5, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Sjors commented Feb 13, 2024 • edited

maflcko commented Mar 12, 2024

Sjors commented Mar 12, 2024

maflcko commented Mar 12, 2024

Sjors commented Mar 12, 2024

Sjors Mar 12, 2024 • edited

Choose a reason for hiding this comment

sipa commented Mar 12, 2024

Sjors commented Mar 12, 2024

cbergqvist commented Mar 14, 2024

hebasto commented Mar 18, 2024

Sjors commented Mar 18, 2024

andrewtoth commented May 14, 2024

Sjors commented May 14, 2024 • edited

andrewtoth commented May 15, 2024

BenWestgate left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sdaftuar left a comment

Choose a reason for hiding this comment

BenWestgate commented May 23, 2024 • edited

Sjors commented May 28, 2024

Sjors commented Aug 28, 2023 •

edited

DrahtBot commented Aug 28, 2023 •

edited

andrewtoth commented Aug 28, 2023 •

edited

Sjors commented Aug 30, 2023 •

edited

jonatack Aug 30, 2023 •

edited

jonatack Aug 30, 2023 •

edited

Sjors Feb 13, 2024 •

edited

jonatack Aug 30, 2023 •

edited

BenWestgate commented Aug 30, 2023 •

edited

jonatack Aug 31, 2023 •

edited

Sjors commented Feb 13, 2024 •

edited

Sjors Mar 12, 2024 •

edited

Sjors commented May 14, 2024 •

edited

BenWestgate commented May 23, 2024 •

edited