{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":6934395,"defaultBranch":"main","name":"rocksdb","ownerLogin":"facebook","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2012-11-30T06:16:18.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/69631?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1717370288.0","currentOid":""},"activityList":{"items":[{"before":"961468f92e210d9fa5e3b8ca7aef1d5b2c3d4b5c","after":"d64eac28d32a025770cba641ea04e697f475cdd6","ref":"refs/heads/main","pushedAt":"2024-06-12T04:44:26.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Fix a failure to propagate ReadOptions (#12757)\n\nSummary:\nThe crash test revealed a case in which the uncache functionality in ~BlockBasedTableReader could initiate an block read (IO), despite setting ReadOptions::read_tier = kBlockCacheTier.\n\nThe root cause is a place in the code where many people have over time decided to opt-in propagating ReadOptions and no one took the initiative to propagate ReadOptions by default (opt out / override only as needed). The fix is in partitioned_index_reader.cc. Here,\nReadOptions::readahead_size is opted-out to avoid churn in prefetch_test that is not clearly an improvement or regression. It's hard to tell given the poor state of relevant documentation https://github.com/facebook/rocksdb/issues/12756. The affected unit test was added in https://github.com/facebook/rocksdb/issues/10602.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12757\n\nTest Plan: (Now postponed to a follow-up diff) I have added some new infrastructure to DEBUG builds to catch this specific kind of violation in unit tests and in the stress/crash test. `EnforceReadOpts` establishes a thread-local context under which we assert no IOs are performed if ReadOptions said it should be forbidden. With this new checking, the Uncache unit test would catch the critical step toward a violation (inner ReadOptions allowing IO, even if no IO is actually performed), which is fixed with the production code change.\n\nReviewed By: hx235\n\nDifferential Revision: D58421526\n\nPulled By: pdillinger\n\nfbshipit-source-id: 9e9917a0e320c78967e751bd887926a2ed231d37","shortMessageHtmlLink":"Fix a failure to propagate ReadOptions (#12757)"}},{"before":"21eb82ebec503ab9fa9b2d9f36e3c332920f8274","after":"961468f92e210d9fa5e3b8ca7aef1d5b2c3d4b5c","ref":"refs/heads/main","pushedAt":"2024-06-11T23:56:44.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Fix TSAN-reported data race with uncache_aggressiveness (#12753)\n\nSummary:\nData race reported on\nBlockBasedTableReader::Rep::uncache_aggressiveness because apparently a file can be marked obsolete through multiple table cache references in parallel. Using a relaxed atomic should resolve the race quite reasonably, especially considering this is a rare case and the racing writes should be storing the same value anyway.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12753\n\nTest Plan: watch for TSAN crash test results\n\nReviewed By: ltamasi\n\nDifferential Revision: D58397473\n\nPulled By: pdillinger\n\nfbshipit-source-id: 3e78b6adac4f7a7056790754bee42b3cb244f037","shortMessageHtmlLink":"Fix TSAN-reported data race with uncache_aggressiveness (#12753)"}},{"before":"af50823069818fc127438e39fef91d2486d6e76c","after":"21eb82ebec503ab9fa9b2d9f36e3c332920f8274","ref":"refs/heads/main","pushedAt":"2024-06-11T23:01:20.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Disable \"uncache\" behavior in DB shutdown (#12751)\n\nSummary:\nCrash test showed a potential use-after-free where a file marked as obsolete and eligible for uncache on destruction is destroyed in the VersionSet destructor, which only happens as part of DB shutdown. At that point, the in-memory column families have already been destroyed, so attempting to uncache could use-after-free on stuff like getting the `user_comparator()` from the `internal_comparator()`.\n\nI attempted to make it smarter, but wasn't able to untangle the destruction dependencies in a way that was safe, understandable, and maintainable.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12751\n\nTest Plan:\nReproduced by adding uncache_aggressiveness to an existing (but otherwise unrelated) test. This makes it a fair regression test.\n\nAlso added testing to ensure that trivial moves and DB close & reopen are well behaved with uncache_aggressiveness. Specifically, this issue doesn't seem to be because things are uncached inappropriately in those cases.\n\nReviewed By: ltamasi\n\nDifferential Revision: D58390058\n\nPulled By: pdillinger\n\nfbshipit-source-id: 66ac9cb13bf02638fa80ee5b7218153d8bc7cfd3","shortMessageHtmlLink":"Disable \"uncache\" behavior in DB shutdown (#12751)"}},{"before":"68112b3beb885c9ec8bc410e15b05e7e27e3c9ee","after":"af50823069818fc127438e39fef91d2486d6e76c","ref":"refs/heads/main","pushedAt":"2024-06-10T23:33:10.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"c.h: Add set_track_and_verify_wals_in_manifest to C API (#12749)\n\nSummary:\nThis option is recommended to be set for production use:\n\n We recommend to set track_and_verify_wals_in_manifest to true\n for production\n\nhttps://github.com/facebook/rocksdb/wiki/Track-WAL-in-MANIFEST\n\nThis adds this setting to the C API, so it can be used by other languages.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12749\n\nReviewed By: ltamasi\n\nDifferential Revision: D58382892\n\nPulled By: ajkr\n\nfbshipit-source-id: 885de4539745a3119b6b2a162ab4fca9fa975283","shortMessageHtmlLink":"c.h: Add set_track_and_verify_wals_in_manifest to C API (#12749)"}},{"before":"d3c4b7fe0bd0b7f3ec27895c517d43cc6b4c06ad","after":"68112b3beb885c9ec8bc410e15b05e7e27e3c9ee","ref":"refs/heads/main","pushedAt":"2024-06-10T20:35:38.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Attempt fix rare failure in DBBlockCacheTypeTest.Uncache (#12748)\n\nSummary:\nI haven't been able to reproduce the failure, seen in https://github.com/facebook/rocksdb/actions/runs/9420830905/job/25953696902?pr=12734\n\n```\n[ RUN ] DBBlockCacheTypeTestInstance/DBBlockCacheTypeTest.Uncache/2\ndb/db_block_cache_test.cc:1415: Failure\nExpected equality of these values:\n cache->GetOccupancyCount()\n Which is: 37\n kBaselineCount + kNumDataBlocks + meta_blocks_per_file\n Which is: 15\nGoogle Test trace:\ndb/db_block_cache_test.cc:1346: ua=10000\ndb/db_block_cache_test.cc:1344: partitioned=1\ndb/db_block_cache_test.cc:1418: Failure\n...\n```\n\nBut it's consistent with a SuperVersion reference sticking around beyond the CompactRange, as I can reproduce the result with a dangling Iterator. Like some other tests have had trouble with periodic stats popping up randomly, I suspect that could be the explanation in this case.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12748\n\nTest Plan: Watch for similar future failures\n\nReviewed By: ltamasi\n\nDifferential Revision: D58366031\n\nPulled By: pdillinger\n\nfbshipit-source-id: b812ca8837b8c8b9cbda1b201d76316d145fa3ec","shortMessageHtmlLink":"Attempt fix rare failure in DBBlockCacheTypeTest.Uncache (#12748)"}},{"before":"43906597f5d74245a094df27fc27f9bbb93384f8","after":"d3c4b7fe0bd0b7f3ec27895c517d43cc6b4c06ad","ref":"refs/heads/main","pushedAt":"2024-06-10T19:40:20.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Enable reopen with un-synced data loss in crash test (#12746)\n\nSummary:\n**Context/Summary:**\nhttps://github.com/facebook/rocksdb/pull/12567 disabled reopen with un-synced data loss in crash test since we discovered un-synced WAL loss and we currently don't support prefix recovery in reopen. This PR explicitly sync WAL data before close to avoid such data loss case from happening and add back the testing coverage.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12746\n\nTest Plan: CI\n\nReviewed By: ajkr\n\nDifferential Revision: D58326890\n\nPulled By: hx235\n\nfbshipit-source-id: 0865f715e97c5948d7cb3aea62fe2a626cb6522a","shortMessageHtmlLink":"Enable reopen with un-synced data loss in crash test (#12746)"}},{"before":"32e6825bc6a18d9e61de375b62139cc4a0f0fe36","after":"43906597f5d74245a094df27fc27f9bbb93384f8","ref":"refs/heads/main","pushedAt":"2024-06-08T05:18:11.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Fixed CMake builds for iOS (#12744)\n\nSummary:\nThis [ports the flags which were already defined in `build_tools`](https://github.com/facebook/rocksdb/blob/44aceb88d0de120847719c061aa3a8465daaee48/build_tools/build_detect_platform#L151) for CMake.\n\nThe flag `OS_MACOSX` in specific is necessary for proper endiness detection, [since they're required for proper inclusion of OSX endian header](https://github.com/facebook/rocksdb/blob/44aceb88d0de120847719c061aa3a8465daaee48/port/port_posix.h#L27).\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12744\n\nReviewed By: ltamasi\n\nDifferential Revision: D58317987\n\nPulled By: ajkr\n\nfbshipit-source-id: 407e623ddb6afc9c48939d52f610281f59cf99af","shortMessageHtmlLink":"Fixed CMake builds for iOS (#12744)"}},{"before":"b34cef57b798520791312f2f40681c4d12d5d33c","after":"32e6825bc6a18d9e61de375b62139cc4a0f0fe36","ref":"refs/heads/main","pushedAt":"2024-06-07T23:57:10.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"c.h: Add GetDbIdentity, Options::write_dbid_to_manifest (#12736)\n\nSummary:\nThe write_dbid_to_manifest option is documented as \"We recommend setting this flag to true\". However, there is no way to set this flag from the C API.\n\nAdd the following functions to the C API:\n\n* rocksdb_get_db_identity\n* rocksdb_options_get_write_dbid_to_manifest\n* rocksdb_options_set_write_dbid_to_manifest\n\nAdd a test that this option preserves the ID across checkpoints.\n\nc.cc:\n* Remove outdated comments about missing C API functions that exist.\n* Document that CopyString is intended for binary data and is not NUL terminated.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12736\n\nReviewed By: ltamasi\n\nDifferential Revision: D58202117\n\nPulled By: ajkr\n\nfbshipit-source-id: 707b110df5c4bd118d65548327428a53a9dc3019","shortMessageHtmlLink":"c.h: Add GetDbIdentity, Options::write_dbid_to_manifest (#12736)"}},{"before":"44aceb88d0de120847719c061aa3a8465daaee48","after":"b34cef57b798520791312f2f40681c4d12d5d33c","ref":"refs/heads/main","pushedAt":"2024-06-07T16:01:45.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Support pro-actively erasing obsolete block cache entries (#12694)\n\nSummary:\nCurrently, when files become obsolete, the block cache entries associated with them just age out naturally. With pure LRU, this is not too bad, as once you \"use\" enough cache entries to (re-)fill the cache, you are guranteed to have purged the obsolete entries. However, HyperClockCache is a counting clock cache with a somewhat longer memory, so could be more negatively impacted by previously-hot cache entries becoming obsolete, and taking longer to age out than newer single-hit entries.\n\nPart of the reason we still have this natural aging-out is that there's almost no connection between block cache entries and the file they are associated with. Everything is hashed into the same pool(s) of entries with nothing like a secondary index based on file. Keeping track of such an index could be expensive.\n\nThis change adds a new, mutable CF option `uncache_aggressiveness` for erasing obsolete block cache entries. The process can be speculative, lossy, or unproductive because not all potential block cache entries associated with files will be resident in memory, and attempting to remove them all could be wasted CPU time. Rather than a simple on/off switch, `uncache_aggressiveness` basically tells RocksDB how much CPU you're willing to burn trying to purge obsolete block cache entries. When such efforts are not sufficiently productive for a file, we stop and move on.\n\nThe option is in ColumnFamilyOptions so that it is dynamically changeable for already-open files, and customizeable by CF.\n\nNote that this block cache removal happens as part of the process of purging obsolete files, which is often in a background thread (depending on `background_purge_on_iterator_cleanup` and `avoid_unnecessary_blocking_io` options) rather than along CPU critical paths.\n\nNotable auxiliary code details:\n* Possibly fixing some issues with trivial moves with `only_delete_metadata`: unnecessary TableCache::Evict in that case and missing from the ObsoleteFileInfo move operator. (Not able to reproduce an current failure.)\n* Remove suspicious TableCache::Erase() from VersionSet::AddObsoleteBlobFile() (TODO follow-up item)\n\nMarked EXPERIMENTAL until more thorough validation is complete.\n\nDirect stats of this functionality are omitted because they could be misleading. Block cache hit rate is a better indicator of benefit, and CPU profiling a better indicator of cost.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12694\n\nTest Plan:\n* Unit tests added, including refactoring an existing test to make better use of parameterized tests.\n* Added to crash test.\n* Performance, sample command:\n```\nfor I in `seq 1 10`; do for UA in 300; do for CT in lru_cache fixed_hyper_clock_cache auto_hyper_clock_cache; do rm -rf /dev/shm/test3; TEST_TMPDIR=/dev/shm/test3 /usr/bin/time ./db_bench -benchmarks=readwhilewriting -num=13000000 -read_random_exp_range=6 -write_buffer_size=10000000 -bloom_bits=10 -cache_type=$CT -cache_size=390000000 -cache_index_and_filter_blocks=1 -disable_wal=1 -duration=60 -statistics -uncache_aggressiveness=$UA 2>&1 | grep -E 'micros/op|rocksdb.block.cache.data.(hit|miss)|rocksdb.number.keys.(read|written)|maxresident' | awk '/rocksdb.block.cache.data.miss/ { miss = $4 } /rocksdb.block.cache.data.hit/ { hit = $4 } { print } END { print \"hit rate = \" ((hit * 1.0) / (miss + hit)) }' | tee -a results-$CT-$UA; done; done; done\n```\n\nAveraging 10 runs each case, block cache data block hit rates\n\n```\nlru_cache\nUA=0 -> hit rate = 0.327, ops/s = 87668, user CPU sec = 139.0\nUA=300 -> hit rate = 0.336, ops/s = 87960, user CPU sec = 139.0\n\nfixed_hyper_clock_cache\nUA=0 -> hit rate = 0.336, ops/s = 100069, user CPU sec = 139.9\nUA=300 -> hit rate = 0.343, ops/s = 100104, user CPU sec = 140.2\n\nauto_hyper_clock_cache\nUA=0 -> hit rate = 0.336, ops/s = 97580, user CPU sec = 140.5\nUA=300 -> hit rate = 0.345, ops/s = 97972, user CPU sec = 139.8\n```\n\nConclusion: up to roughly 1 percentage point of improved block cache hit rate, likely leading to overall improved efficiency (because the foreground CPU cost of cache misses likely outweighs the background CPU cost of erasure, let alone I/O savings).\n\nReviewed By: ajkr\n\nDifferential Revision: D57932442\n\nPulled By: pdillinger\n\nfbshipit-source-id: 84a243ca5f965f731f346a4853009780a904af6c","shortMessageHtmlLink":"Support pro-actively erasing obsolete block cache entries (#12694)"}},{"before":"390fc55ba15912751591a4522e701a760b80640a","after":"44aceb88d0de120847719c061aa3a8465daaee48","ref":"refs/heads/main","pushedAt":"2024-06-07T00:31:54.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Add a OnManualFlushScheduled callback in event listener (#12631)\n\nSummary:\nAs titled. Also added the newest user-defined timestamp into the `MemTableInfo`. This can be a useful info in the callback.\n\nAdded some unit tests as examples for how users can use two separate approaches to allow manual flush / manual compactions to go through when the user-defined timestamps in memtable only feature is enabled. One approach relies on selectively increase cutoff timestamp in `OnMemtableSeal` callback when it's initiated by a manual flush. Another approach is to increase cutoff timestamp in `OnManualFlushScheduled` callback. The caveats of the approaches are also documented in the unit test.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12631\n\nReviewed By: ajkr\n\nDifferential Revision: D58260528\n\nPulled By: jowlyzhang\n\nfbshipit-source-id: bf446d7140affdf124744095e0a179fa6e427532","shortMessageHtmlLink":"Add a OnManualFlushScheduled callback in event listener (#12631)"}},{"before":"98393f0139fc9529d5c56a4b43bc7a245b22f734","after":"390fc55ba15912751591a4522e701a760b80640a","ref":"refs/heads/main","pushedAt":"2024-06-06T18:50:03.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Revert PR 12684 and 12556 (#12738)\n\nSummary:\n**Context/Summary:** a better API design is decided lately so we decided to revert these two changes.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12738\n\nTest Plan: - CI\n\nReviewed By: ajkr\n\nDifferential Revision: D58162165\n\nPulled By: hx235\n\nfbshipit-source-id: 9bbe4d2fe9fbe39213f4cf137a2d419e6ffb8e16","shortMessageHtmlLink":"Revert PR 12684 and 12556 (#12738)"}},{"before":"a211e065520b80a2a5a28a795aae1a71e1b139dc","after":"98393f0139fc9529d5c56a4b43bc7a245b22f734","ref":"refs/heads/main","pushedAt":"2024-06-06T00:43:33.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Fix Checkpoint hard link of inactive but unsynced WAL (#12731)\n\nSummary:\nBackground: there is one active WAL file but there can be\nseveral more WAL files in various states. Those other WALs are always\nin a \"flushed\" state but could be on the `logs_` list not yet fully\nsynced. We currently allow any WAL that is not the active WAL to be\nhard-linked when creating a Checkpoint, as although it might still be\nopen for write, we are not appending any more data to it.\n\nThe problem is that a created Checkpoint is supposed to be fully synced\non return of that function, and a hard-linked WAL in the state described\nabove might not be fully synced. (Through some prudence in https://github.com/facebook/rocksdb/issues/10083,\nit would synced if using track_and_verify_wals_in_manifest=true.)\n\nThe fix is a step toward a long term goal of removing the need to query\nthe filesystem to determine WAL files and their state. (I consider it\ndubious any time we independently read from or query metadata from a\nfile we have open for writing, as this makes us more susceptible to\nFileSystem deficiencies or races.) More specifically:\n* Detect which WALs might not be fully synced, according to our DBImpl\n metadata, and prevent hard linking those (with `trim_to_size=true`\n from `GetLiveFilesStorageInfo()`. And while we're at it, use our known\n flushed sizes for those WALs.\n* To avoid a race between that and GetSortedWalFiles(), track a maximum\n needed WAL number for the Checkpoint/GetLiveFilesStorageInfo.\n* Because of the level of consistency provided by those two, we no\n longer need to consider syncing as part of the FlushWAL in\n GetLiveFilesStorageInfo. (We determine the max WAL number consistent\n with the manifest file size, while holding DB mutex. Should make\n track_and_verify_wals_in_manifest happy.) This makes the premise of\n test PutRaceWithCheckpointTrackedWalSync obsolete (sync point callback\n no longer hit) so the test is removed, with crash test as backstop for\n related issues. See https://github.com/facebook/rocksdb/issues/10185\n\nStacked on https://github.com/facebook/rocksdb/issues/12729\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12731\n\nTest Plan:\nExpanded an existing test, which now fails before fix.\nAlso long runs of blackbox_crash_test with amplified checkpoint frequency.\n\nReviewed By: cbi42\n\nDifferential Revision: D58199629\n\nPulled By: pdillinger\n\nfbshipit-source-id: 376e55f4a2b082cd2adb6408a41209de14422382","shortMessageHtmlLink":"Fix Checkpoint hard link of inactive but unsynced WAL (#12731)"}},{"before":"61d10fe0c355d089d2789e5d41d20de907708c2b","after":"a211e065520b80a2a5a28a795aae1a71e1b139dc","ref":"refs/heads/main","pushedAt":"2024-06-05T19:56:08.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Remove close when fd == -1. (#12732)\n\nSummary:\nIts polluting my valgrind runs:\n==3733139== Warning: invalid file descriptor -1 in syscall close()\n==3733139== Warning: invalid file descriptor -1 in syscall close()\n==3733139== Warning: invalid file descriptor -1 in syscall close()\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12732\n\nReviewed By: ltamasi\n\nDifferential Revision: D58170009\n\nPulled By: ajkr\n\nfbshipit-source-id: 1fc6944c2667641996676a75aa3e91984070ba49","shortMessageHtmlLink":"Remove close when fd == -1. (#12732)"}},{"before":"9f4c597d839ed9fa9d1a5ed7ea393a150fa9adf8","after":"61d10fe0c355d089d2789e5d41d20de907708c2b","ref":"refs/heads/main","pushedAt":"2024-06-05T19:50:49.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Fix compile errors on loongarch (#12739)\n\nSummary:\nFailed to compile when using cmake on loongarch architecture with the following error details:[https://buildd.debian.org/status/fetch.php?pkg=rocksdb&arch=loong64&ver=9.2.1-2&stamp= 1717362107&raw=0](url). The reason for the error is that loongarch does not support the mcpu option, refer to the link for details: [https://gcc.gnu.org/onlinedocs/gcc-13.2.0/gcc/LoongArch-Options.html](url)\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12739\n\nReviewed By: ltamasi\n\nDifferential Revision: D58200695\n\nPulled By: ajkr\n\nfbshipit-source-id: 00e1a51e15defaa8983524cdd3fc25240833c08b","shortMessageHtmlLink":"Fix compile errors on loongarch (#12739)"}},{"before":"8523f0a86afcf16ba679393baf7b8d3b54e5daad","after":"9f4c597d839ed9fa9d1a5ed7ea393a150fa9adf8","ref":"refs/heads/main","pushedAt":"2024-06-04T22:31:04.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"FaultInjectionTestFS read unsynced data by default (#12729)\n\nSummary:\nIn places (e.g. GetSortedWals()) RocksDB relies on querying the file size or even reading the contents of files currently open for writing, and as in POSIX semantics, expects to see the flushed size and contents regardless of what has been synced. FaultInjectionTestFS historically did not emulate this behavior, only showing synced data from such read operations. (Different from FaultInjectionTestEnv--sigh.)\n\nThis change makes the \"proper\" behavior the default behavior, at least for GetFileSize and FSSequentialFile. However, this new functionality is disabled in db_stress because of undiagnosed, unresolved issues.\n\nAlso removes unused and confusing field `pos_at_last_flush_`\n\nThis change is needed to support testing a relevant bug fix (in a follow-up diff). Other suggested follow-up:\n* Fix db_stress not to rely on the old behavior, and fix a related FIXME in db_stress_test_base.cc in LockWAL testing.\n* Fill in some corner cases in the FileSystem API for reading unsynced data (see new TODO items).\n* Consider deprecating and removing Flush() API functions from FileSystem APIs. It is not clear to me that there is a supported scenario in which they do anything but confuse API users and developers. If there is a use for them, it doesn't appear to be tested.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12729\n\nTest Plan: applies to all unit tests successfully, just updating the unit test from https://github.com/facebook/rocksdb/issues/12556 due to relying on the errant behavior. Also added a specific unit test\n\nReviewed By: hx235\n\nDifferential Revision: D58091835\n\nPulled By: pdillinger\n\nfbshipit-source-id: f47a63b2b000f5875b6293a98577bff663d7fd33","shortMessageHtmlLink":"FaultInjectionTestFS read unsynced data by default (#12729)"}},{"before":"a8a52e5b4d938c1aea28d7a96ec70a459ffaa1a4","after":"8523f0a86afcf16ba679393baf7b8d3b54e5daad","ref":"refs/heads/main","pushedAt":"2024-06-04T20:44:04.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Use extended file boundary for key range overlap check during file ingestion (#12735)\n\nSummary:\nWhen https://github.com/facebook/rocksdb/issues/12343 added support to bulk load external files while column family enables user-defined timestamps, it's a requirement that the external file doesn't overlap with the DB in key ranges. More specifically, the external file should not contain a user key (without timestamp) that already have some entries in the DB.\n\nAll the `*Overlap*` functions like `RangeOverlapWithMemtable`, `RangeOverlapWithCompaction` are using `CompareWithoutTimestamp` to check for overlap already. One thing that is missing here is we need to extend the external file's user key boundary for this check to avoid missing the checks for the boundary user keys. For example, with the current way of checking things where `external_file_info.smallest.user_key()` is used as the left boundary, and `external_file_info.largest.user_key()` is used as the right boundary, a file with this entry: (b, 40) can fit into a DB with these two entries: (b, 30), (c, 20).\n\nTo avoid this, we extend the user key boundaries used for overlap check, by updating the left boundary with the maximum timestamp and the right boundary with the minimum timestamp.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12735\n\nTest Plan: Added unit test\n\nReviewed By: ltamasi\n\nDifferential Revision: D58152117\n\nPulled By: jowlyzhang\n\nfbshipit-source-id: 9cba61e7357f6d76ad44c258381c35073ebbf347","shortMessageHtmlLink":"Use extended file boundary for key range overlap check during file in…"}},{"before":"030f7c4a7b270b4ba7fb1b929b92404165291bb1","after":"c5201abc4a983450f9423435a4405829be23d0a8","ref":"refs/heads/9.3.fb","pushedAt":"2024-06-04T18:43:09.000Z","pushType":"push","commitsCount":2,"pusher":{"login":"ajkr","name":"Andrew Kryczka","path":"/ajkr","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4780362?s=80&v=4"},"commit":{"message":"update HISTORY.md and version.h for 9.3.1","shortMessageHtmlLink":"update HISTORY.md and version.h for 9.3.1"}},{"before":"b03d4156609dd27f65ad199139eaae722c876b8c","after":"a8a52e5b4d938c1aea28d7a96ec70a459ffaa1a4","ref":"refs/heads/main","pushedAt":"2024-06-04T16:44:54.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Fix AddressSanitizer container-overflow (#12722)\n\nSummary:\n```\nERROR: AddressSanitizer: container-overflow on address 0x506000682221 at pc 0x5583da569f76 bp 0x7f0ec8a9ffb0 sp 0x7f0ec8a9f780\nWRITE of size 53 at 0x506000682221 thread T29\n #0 0x5583da569f75 in pread\n https://github.com/facebook/rocksdb/issues/1 0x5583e334fde4 in rocksdb::PosixRandomAccessFile::Read(unsigned long, unsigned long, rocksdb::IOOptions const&, rocksdb::Slice*, char*, rocksdb::IODebugContext*) const /rocksdb/env/io_posix.cc:580:9\n https://github.com/facebook/rocksdb/issues/2 0x5583e2cac42b in rocksdb::(anonymous namespace)::CompositeRandomAccessFileWrapper::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const /rocksdb/env/composite_env.cc:61:21\n https://github.com/facebook/rocksdb/issues/3 0x5583e2c8a8e4 in rocksdb::(anonymous namespace)::LegacyRandomAccessFileWrapper::Read(unsigned long, unsigned long, rocksdb::IOOptions const&, rocksdb::Slice*, char*, rocksdb::IODebugContext*) const /rocksdb/env/env.cc:152:41\n https://github.com/facebook/rocksdb/issues/4 0x5583e2d6cbfb in rocksdb::RandomAccessFileReader::Read(rocksdb::IOOptions const&, unsigned long, unsigned long, rocksdb::Slice*, char*, std::__2::unique_ptr>*, rocksdb::Env::IOPriority) const /rocksdb/file/random_access_file_reader.cc:204:25\n https://github.com/facebook/rocksdb/issues/5 0x5583e307c614 in rocksdb::ReadFooterFromFile(rocksdb::IOOptions const&, rocksdb::RandomAccessFileReader*, rocksdb::FilePrefetchBuffer*, unsigned long, rocksdb::Footer*, unsigned long) /rocksdb/table/format.cc:383:17\n https://github.com/facebook/rocksdb/issues/6 0x5583e2f88456 in rocksdb::BlockBasedTable::Open(rocksdb::ReadOptions const&, rocksdb::ImmutableOptions const&, rocksdb::EnvOptions const&, rocksdb::BlockBasedTableOptions const&, rocksdb::InternalKeyComparator const&, std::__2::unique_ptr>&&, unsigned long, std::__2::unique_ptr>*, std::__2::shared_ptr, std::__2::shared_ptr const&, bool, bool, int, bool, unsigned long, bool, rocksdb::TailPrefetchStats*, rocksdb::BlockCacheTracer*, unsigned long, std::__2::basic_string, std::__2::allocator> const&, unsigned long) /rocksdb/table/block_based/block_based_table_reader.cc:610:9\n https://github.com/facebook/rocksdb/issues/7 0x5583e2ef7837 in rocksdb::BlockBasedTableFactory::NewTableReader(rocksdb::ReadOptions const&, rocksdb::TableReaderOptions const&, std::__2::unique_ptr>&&, unsigned long, std::__2::unique_ptr>*, bool) const /rocksdb/table/block_based/block_based_table_factory.cc:599:10\n https://github.com/facebook/rocksdb/issues/8 0x5583e2ab873c in rocksdb::TableCache::GetTableReader(rocksdb::ReadOptions const&, rocksdb::FileOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&, bool, bool, rocksdb::HistogramImpl*, std::__2::unique_ptr>*, std::__2::shared_ptr const&, bool, int, bool, unsigned long, rocksdb::Temperature) /rocksdb/db/table_cache.cc:142:34\n https://github.com/facebook/rocksdb/issues/9 0x5583e2aba5f6 in rocksdb::TableCache::FindTable(rocksdb::ReadOptions const&, rocksdb::FileOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&, rocksdb::Cache::Handle**, std::__2::shared_ptr const&, bool, bool, rocksdb::HistogramImpl*, bool, int, bool, unsigned long, rocksdb::Temperature) /rocksdb/db/table_cache.cc:190:16\n https://github.com/facebook/rocksdb/issues/10 0x5583e2abb7e1 in rocksdb::TableCache::NewIterator(rocksdb::ReadOptions const&, rocksdb::FileOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileMetaData const&, rocksdb::RangeDelAggregator*, std::__2::shared_ptr const&, rocksdb::TableReader**, rocksdb::HistogramImpl*, rocksdb::TableReaderCaller, rocksdb::Arena*, bool, int, unsigned long, rocksdb::InternalKey const*, rocksdb::InternalKey const*, bool) /rocksdb/db/table_cache.cc:235:9\n https://github.com/facebook/rocksdb/issues/11 0x5583e28d14cf in rocksdb::BuildTable(std::__2::basic_string, std::__2::allocator> const&, rocksdb::VersionSet*, rocksdb::ImmutableDBOptions const&, rocksdb::TableBuilderOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase*, std::__2::vector>, std::__2::allocator>>>, rocksdb::FileMetaData*, std::__2::vector>*, std::__2::vector>, unsigned long, unsigned long, rocksdb::SnapshotChecker*, bool, rocksdb::InternalStats*, rocksdb::IOStatus*, std::__2::shared_ptr const&, rocksdb::BlobFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, rocksdb::Env::WriteLifeTimeHint, std::__2::basic_string, std::__2::allocator> const*, rocksdb::BlobFileCompletionCallback*, unsigned long*, unsigned long*, unsigned long*) /rocksdb/db/builder.cc:335:57\n https://github.com/facebook/rocksdb/issues/12 0x5583e29bf29d in rocksdb::FlushJob::WriteLevel0Table() /rocksdb/db/flush_job.cc:919:11\n https://github.com/facebook/rocksdb/issues/13 0x5583e29b33ac in rocksdb::FlushJob::Run(rocksdb::LogsWithPrepTracker*, rocksdb::FileMetaData*, bool*) /rocksdb/db/flush_job.cc:276:9\n https://github.com/facebook/rocksdb/issues/14 0x5583e27a4781 in rocksdb::DBImpl::FlushMemTableToOutputFile(rocksdb::ColumnFamilyData*, rocksdb::MutableCFOptions const&, bool*, rocksdb::JobContext*, rocksdb::SuperVersionContext*, std::__2::vector>&, unsigned long, rocksdb::SnapshotChecker*, rocksdb::LogBuffer*, rocksdb::Env::Priority) /rocksdb/db/db_impl/db_impl_compaction_flush.cc:258:19\n https://github.com/facebook/rocksdb/issues/15 0x5583e27a7a96 in rocksdb::DBImpl::FlushMemTablesToOutputFiles(rocksdb::autovector const&, bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::Env::Priority) /rocksdb/db/db_impl/db_impl_compaction_flush.cc:377:14\n https://github.com/facebook/rocksdb/issues/16 0x5583e27d6777 in rocksdb::DBImpl::BackgroundFlush(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::FlushReason*, rocksdb::Env::Priority) /rocksdb/db/db_impl/db_impl_compaction_flush.cc:2778:14\n https://github.com/facebook/rocksdb/issues/17 0x5583e27d14e2 in rocksdb::DBImpl::BackgroundCallFlush(rocksdb::Env::Priority) /rocksdb/db/db_impl/db_impl_compaction_flush.cc:2817:16\n https://github.com/facebook/rocksdb/issues/18 0x5583e323d353 in std::__2::__function::__policy_func::operator()[abi:ne180100]() const /root/build/3rdParty/llvm/runtimes/include/c++/v1/__functional/function.h:714:12\n https://github.com/facebook/rocksdb/issues/19 0x5583e323d353 in std::__2::function::operator()() const /root/build/3rdParty/llvm/runtimes/include/c++/v1/__functional/function.h:981:10\n https://github.com/facebook/rocksdb/issues/20 0x5583e323d353 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) /rocksdb/util/threadpool_imp.cc:266:5\n https://github.com/facebook/rocksdb/issues/21 0x5583e3243d18 in decltype(std::declval()(std::declval())) std::__2::__invoke[abi:ne180100](void (*&&)(void*), rocksdb::BGThreadMetadata*&&) /root/build/3rdParty/llvm/runtimes/include/c++/v1/__type_traits/invoke.h:344:25\n https://github.com/facebook/rocksdb/issues/22 0x5583e3243d18 in void std::__2::__thread_execute[abi:ne180100]>, void (*)(void*), rocksdb::BGThreadMetadata*, 2ul>(std::__2::tuple>, void (*)(void*), rocksdb::BGThreadMetadata*>&, std::__2::__tuple_indices<2ul>) /root/build/3rdParty/llvm/runtimes/include/c++/v1/__thread/thread.h:193:3\n https://github.com/facebook/rocksdb/issues/23 0x5583e3243d18 in void* std::__2::__thread_proxy[abi:ne180100]>, void (*)(void*), rocksdb::BGThreadMetadata*>>(void*) /root/build/3rdParty/llvm/runtimes/include/c++/v1/__thread/thread.h:202:3\n https://github.com/facebook/rocksdb/issues/24 0x5583da5e819e in asan_thread_start(void*) crtstuff.c\n https://github.com/facebook/rocksdb/issues/25 0x7f0eda362a93 in start_thread nptl/pthread_create.c:447:8\n https://github.com/facebook/rocksdb/issues/26 0x7f0eda3efc3b in clone3 misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:78\n\n0x506000682221 is located 1 bytes inside of 56-byte region [0x506000682220,0x506000682258)\nallocated by thread T29 here:\n #0 0x5583da6281d1 in operator new(unsigned long)\n https://github.com/facebook/rocksdb/issues/1 0x5583da6c987d in __libcpp_operator_new /root/build/3rdParty/llvm/runtimes/include/c++/v1/new:271:10\n https://github.com/facebook/rocksdb/issues/2 0x5583da6c987d in __libcpp_allocate /root/build/3rdParty/llvm/runtimes/include/c++/v1/new:295:10\n https://github.com/facebook/rocksdb/issues/3 0x5583da6c987d in allocate /root/build/3rdParty/llvm/runtimes/include/c++/v1/__memory/allocator.h:125:32\n https://github.com/facebook/rocksdb/issues/4 0x5583da6c987d in allocate_at_least /root/build/3rdParty/llvm/runtimes/include/c++/v1/__memory/allocator.h:131:13\n https://github.com/facebook/rocksdb/issues/5 0x5583da6c987d in allocate_at_least > /root/build/3rdParty/llvm/runtimes/include/c++/v1/__memory/allocate_at_least.h:34:20\n https://github.com/facebook/rocksdb/issues/6 0x5583da6c987d in __allocate_at_least > /root/build/3rdParty/llvm/runtimes/include/c++/v1/__memory/allocate_at_least.h:42:10\n https://github.com/facebook/rocksdb/issues/7 0x5583da6c987d in std::__2::basic_string, std::__2::allocator>::__shrink_or_extend[abi:ne180100](unsigned long) /root/build/3rdParty/llvm/runtimes/include/c++/v1/string:3236:27\n https://github.com/facebook/rocksdb/issues/8 0x5583e307c5aa in std::__2::basic_string, std::__2::allocator>::reserve(unsigned long) /root/build/3rdParty/llvm/runtimes/include/c++/v1/string:3207:3\n https://github.com/facebook/rocksdb/issues/9 0x5583e307c5aa in rocksdb::ReadFooterFromFile(rocksdb::IOOptions const&, rocksdb::RandomAccessFileReader*, rocksdb::FilePrefetchBuffer*, unsigned long, rocksdb::Footer*, unsigned long) /rocksdb/table/format.cc:382:18\n https://github.com/facebook/rocksdb/issues/10 0x5583e2f88456 in rocksdb::BlockBasedTable::Open(rocksdb::ReadOptions const&, rocksdb::ImmutableOptions const&, rocksdb::EnvOptions const&, rocksdb::BlockBasedTableOptions const&, rocksdb::InternalKeyComparator const&, std::__2::unique_ptr>&&, unsigned long, std::__2::unique_ptr>*, std::__2::shared_ptr, std::__2::shared_ptr const&, bool, bool, int, bool, unsigned long, bool, rocksdb::TailPrefetchStats*, rocksdb::BlockCacheTracer*, unsigned long, std::__2::basic_string, std::__2::allocator> const&, unsigned long) /rocksdb/table/block_based/block_based_table_reader.cc:610:9\n https://github.com/facebook/rocksdb/issues/11 0x5583e2ef7837 in rocksdb::BlockBasedTableFactory::NewTableReader(rocksdb::ReadOptions const&, rocksdb::TableReaderOptions const&, std::__2::unique_ptr>&&, unsigned long, std::__2::unique_ptr>*, bool) const /rocksdb/table/block_based/block_based_table_factory.cc:599:10\n https://github.com/facebook/rocksdb/issues/12 0x5583e2ab873c in rocksdb::TableCache::GetTableReader(rocksdb::ReadOptions const&, rocksdb::FileOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&, bool, bool, rocksdb::HistogramImpl*, std::__2::unique_ptr>*, std::__2::shared_ptr const&, bool, int, bool, unsigned long, rocksdb::Temperature) /rocksdb/db/table_cache.cc:142:34\n https://github.com/facebook/rocksdb/issues/13 0x5583e2aba5f6 in rocksdb::TableCache::FindTable(rocksdb::ReadOptions const&, rocksdb::FileOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&, rocksdb::Cache::Handle**, std::__2::shared_ptr const&, bool, bool, rocksdb::HistogramImpl*, bool, int, bool, unsigned long, rocksdb::Temperature) /rocksdb/db/table_cache.cc:190:16\n https://github.com/facebook/rocksdb/issues/14 0x5583e2abb7e1 in rocksdb::TableCache::NewIterator(rocksdb::ReadOptions const&, rocksdb::FileOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileMetaData const&, rocksdb::RangeDelAggregator*, std::__2::shared_ptr const&, rocksdb::TableReader**, rocksdb::HistogramImpl*, rocksdb::TableReaderCaller, rocksdb::Arena*, bool, int, unsigned long, rocksdb::InternalKey const*, rocksdb::InternalKey const*, bool) /rocksdb/db/table_cache.cc:235:9\n https://github.com/facebook/rocksdb/issues/15 0x5583e28d14cf in rocksdb::BuildTable(std::__2::basic_string, std::__2::allocator> const&, rocksdb::VersionSet*, rocksdb::ImmutableDBOptions const&, rocksdb::TableBuilderOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase*, std::__2::vector>, std::__2::allocator>>>, rocksdb::FileMetaData*, std::__2::vector>*, std::__2::vector>, unsigned long, unsigned long, rocksdb::SnapshotChecker*, bool, rocksdb::InternalStats*, rocksdb::IOStatus*, std::__2::shared_ptr const&, rocksdb::BlobFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, rocksdb::Env::WriteLifeTimeHint, std::__2::basic_string, std::__2::allocator> const*, rocksdb::BlobFileCompletionCallback*, unsigned long*, unsigned long*, unsigned long*) /rocksdb/db/builder.cc:335:57\n https://github.com/facebook/rocksdb/issues/16 0x5583e29bf29d in rocksdb::FlushJob::WriteLevel0Table() /rocksdb/db/flush_job.cc:919:11\n https://github.com/facebook/rocksdb/issues/17 0x5583e29b33ac in rocksdb::FlushJob::Run(rocksdb::LogsWithPrepTracker*, rocksdb::FileMetaData*, bool*) /rocksdb/db/flush_job.cc:276:9\n https://github.com/facebook/rocksdb/issues/18 0x5583e27a4781 in rocksdb::DBImpl::FlushMemTableToOutputFile(rocksdb::ColumnFamilyData*, rocksdb::MutableCFOptions const&, bool*, rocksdb::JobContext*, rocksdb::SuperVersionContext*, std::__2::vector>&, unsigned long, rocksdb::SnapshotChecker*, rocksdb::LogBuffer*, rocksdb::Env::Priority) /rocksdb/db/db_impl/db_impl_compaction_flush.cc:258:19\n https://github.com/facebook/rocksdb/issues/19 0x5583e27a7a96 in rocksdb::DBImpl::FlushMemTablesToOutputFiles(rocksdb::autovector const&, bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::Env::Priority) /rocksdb/db/db_impl/db_impl_compaction_flush.cc:377:14\n https://github.com/facebook/rocksdb/issues/20 0x5583e27d6777 in rocksdb::DBImpl::BackgroundFlush(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::FlushReason*, rocksdb::Env::Priority) /rocksdb/db/db_impl/db_impl_compaction_flush.cc:2778:14\n https://github.com/facebook/rocksdb/issues/21 0x5583e27d14e2 in rocksdb::DBImpl::BackgroundCallFlush(rocksdb::Env::Priority) /rocksdb/db/db_impl/db_impl_compaction_flush.cc:2817:16\n https://github.com/facebook/rocksdb/issues/22 0x5583e323d353 in std::__2::__function::__policy_func::operator()[abi:ne180100]() const /root/build/3rdParty/llvm/runtimes/include/c++/v1/__functional/function.h:714:12\n https://github.com/facebook/rocksdb/issues/23 0x5583e323d353 in std::__2::function::operator()() const /root/build/3rdParty/llvm/runtimes/include/c++/v1/__functional/function.h:981:10\n https://github.com/facebook/rocksdb/issues/24 0x5583e323d353 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) /rocksdb/util/threadpool_imp.cc:266:5\n https://github.com/facebook/rocksdb/issues/25 0x5583e3243d18 in decltype(std::declval()(std::declval())) std::__2::__invoke[abi:ne180100](void (*&&)(void*), rocksdb::BGThreadMetadata*&&) /root/build/3rdParty/llvm/runtimes/include/c++/v1/__type_traits/invoke.h:344:25\n https://github.com/facebook/rocksdb/issues/26 0x5583e3243d18 in void std::__2::__thread_execute[abi:ne180100]>, void (*)(void*), rocksdb::BGThreadMetadata*, 2ul>(std::__2::tuple>, void (*)(void*), rocksdb::BGThreadMetadata*>&, std::__2::__tuple_indices<2ul>) /root/build/3rdParty/llvm/runtimes/include/c++/v1/__thread/thread.h:193:3\n https://github.com/facebook/rocksdb/issues/27 0x5583e3243d18 in void* std::__2::__thread_proxy[abi:ne180100]>, void (*)(void*), rocksdb::BGThreadMetadata*>>(void*) /root/build/3rdParty/llvm/runtimes/include/c++/v1/__thread/thread.h:202:3\n https://github.com/facebook/rocksdb/issues/28 0x5583da5e819e in asan_thread_start(void*) crtstuff.c\n\nHINT: if you don't care about these errors you may set ASAN_OPTIONS=detect_container_overflow=0.\nIf you suspect a false positive see also: https://github.com/google/sanitizers/wiki/AddressSanitizerContainerOverflow.\n AddressSanitizer:container-overflow in pread\nShadow bytes around the buggy address:\n 0x506000681f80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa\n 0x506000682000: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa\n 0x506000682080: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa\n 0x506000682100: fa fa fa fa fa fa fa fa fa fa fa fa 00 00 00 00\n 0x506000682180: 00 00 00 fa fa fa fa fa fa fa fa fa fa fa fa fa\n=>0x506000682200: fa fa fa fa[01]fc fc fc fc fc fc fa fa fa fa fa\n 0x506000682280: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa\n 0x506000682300: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 01\n 0x506000682380: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa\n 0x506000682400: fd fd fd fd fd fd fd fa fa fa fa fa fd fd fd fd\n 0x506000682480: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd fd\nShadow byte legend (one shadow byte represents 8 application bytes):\n Addressable: 00\n Partially addressable: 01 02 03 04 05 06 07\n Heap left redzone: fa\n Freed heap region: fd\n Stack left redzone: f1\n Stack mid redzone: f2\n Stack right redzone: f3\n Stack after return: f5\n Stack use after scope: f8\n Global redzone: f9\n Global init order: f6\n Poisoned by user: f7\n Container overflow: fc\n Array cookie: ac\n Intra object redzone: bb\n ASan internal: fe\n Left alloca redzone: ca\n Right alloca redzone: cb\n```\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12722\n\nReviewed By: hx235\n\nDifferential Revision: D58118264\n\nPulled By: ajkr\n\nfbshipit-source-id: 0dd914c886c022d82697b769d664ba52de0770de","shortMessageHtmlLink":"Fix AddressSanitizer container-overflow (#12722)"}},{"before":"e4428b7eb9b32d85495186757a13cc782f505535","after":"b03d4156609dd27f65ad199139eaae722c876b8c","ref":"refs/heads/main","pushedAt":"2024-06-04T16:40:10.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Fix build on i386 (#12719)\n\nSummary:\nCited from https://pkg-status.freebsd.org/beefy21/data/140i386-default/02faf78f4c9b/logs/rocksdb-9.2.1.log\nThe error message is as follows:\n```\nmkdir -p db_stress_tool && clang++ -O2 -pipe -DOS_FREEBSD -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -Wno-inconsistent-missing-override -Wno-unused-parameter -Wno-unused-variable -Wno-unused-private-field -isystem /usr/local/include -std=c++17 -fPIC -DROCKSDB_DLL -DROCKSDB_USE_RTTI -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wunused-parameter -Werror -I. -I./include -std=c++17 -O2 -pipe -DOS_FREEBSD -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -Wno-inconsistent-missing-override -Wno-unused-parameter -Wno-unused-variable -Wno-unused-private-field -isystem /usr/local/include -std=c++17 -faligned-new -DHAVE_ALIGNED_NEW -DROCKSDB_PLATFORM_POSIX -DROCKSDB_LIB_IO_POSIX -O2 -pipe -DOS_FREEBSD -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -D_REENTRANT -DOS_FREEBSD -DSNAPPY -DGFLAGS=1 -DZLIB -DBZIP2 -DLZ4 -DZSTD -DROCKSDB_MALLOC_USABLE_SIZE -DROCKSDB_PTHREAD_ADAPTIVE_MUTEX -DROCKSDB_BACKTRACE -DROCKSDB_SCHED_GETCPU_PRESENT -isystem third-party/gtest-1.8.1/fused-src -O2 -fno-omit-frame-pointer -momit-leaf-frame-pointer -DNDEBUG -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers -Wno-invalid-offsetof -c db_stress_tool/db_stress_common.cc -o db_stress_tool/db_stress_common.o\ndb_stress_tool/db_stress_common.cc:204:17: error: format specifies type 'unsigned long' but the argument has type 'size_t' (aka 'unsigned int') [-Werror,-Wformat]\n block_cache->GetCapacity());\n ^~~~~~~~~~~~~~~~~~~~~~~~~~\n1 error generated.\n```\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12719\n\nReviewed By: hx235\n\nDifferential Revision: D58093539\n\nPulled By: jaykorean\n\nfbshipit-source-id: 400cae3a4b0d23b168937a5388065ef1c4b8b56e","shortMessageHtmlLink":"Fix build on i386 (#12719)"}},{"before":"21a16f9e6410613498a4be219998049f95fbdd3f","after":"e4428b7eb9b32d85495186757a13cc782f505535","ref":"refs/heads/main","pushedAt":"2024-06-03T18:41:40.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"More details for 'tail prefetch size is calculated based on' (#12667)\n\nSummary:\nThese messages indicate that SST file was created by a pre-9.0.0 RocksDB. Eventually, `TailPrefetchStats` might be removed, so it would be more informative if log message also included name of the affected SST file.\n\nIssue: https://github.com/facebook/rocksdb/issues/12664\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12667\n\nReviewed By: ajkr\n\nDifferential Revision: D57464025\n\nPulled By: hx235\n\nfbshipit-source-id: 12f2f2635e3092f8c29362aa132462492b5c1417","shortMessageHtmlLink":"More details for 'tail prefetch size is calculated based on' (#12667)"}},{"before":"b9e82f51627ccfb84047f49f829c5a716a2c64f2","after":"21a16f9e6410613498a4be219998049f95fbdd3f","ref":"refs/heads/main","pushedAt":"2024-06-03T18:13:52.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"add c-api to get default cf handle (#12514)\n\nSummary:\nrocksdb_batched_multi_get_cf has performance improvement than normal multi_get, however it needs a cf_handle arg, so add a C-API to get and destroy the default cf_handle, as many user only use the default cf.\n\nFixes https://github.com/facebook/rocksdb/issues/12316\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12514\n\nReviewed By: hx235\n\nDifferential Revision: D55922517\n\nPulled By: ajkr\n\nfbshipit-source-id: c4cc4289f2cfd9efbb8f390a44a9d8d1ed08d9f0","shortMessageHtmlLink":"add c-api to get default cf handle (#12514)"}},{"before":"c3ae569792816a8a8ee67b395294ee06ecdf1c1e","after":"b9e82f51627ccfb84047f49f829c5a716a2c64f2","ref":"refs/heads/main","pushedAt":"2024-06-03T17:54:27.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Mention two fixes (#12677 and #12681) in the changelog (#12730)\n\nSummary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/12730\n\nReviewed By: jowlyzhang\n\nDifferential Revision: D58092079\n\nfbshipit-source-id: 708437e705f9d9f770c9ba1a1a9b3c369b0a4b79","shortMessageHtmlLink":"Mention two fixes (#12677 and #12681) in the changelog (#12730)"}},{"before":"b7fc9ada9e111247f84747a6d2cc3ac7943d5a4a","after":"c3ae569792816a8a8ee67b395294ee06ecdf1c1e","ref":"refs/heads/main","pushedAt":"2024-06-03T05:13:34.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Update the main branch for the 9.3 release (#12726)\n\nSummary:\nCut the 9.3.fb branch as of 5/17 11:59pm. Also, cherry-picked all bug fixes that have happened since then. Removed their files from unreleased_history/ since those fixes will appear in 9.3.0, so there seems no use repeating them in any later release.\n\nRelease branch: https://github.com/facebook/rocksdb/tree/9.3.fb\nTests: https://github.com/facebook/rocksdb/actions/runs/9342097111\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12726\n\nReviewed By: ltamasi\n\nDifferential Revision: D58069263\n\nPulled By: ajkr\n\nfbshipit-source-id: c4f557bc8dbc20ce53021ac7e97a24f930542bf9","shortMessageHtmlLink":"Update the main branch for the 9.3 release (#12726)"}},{"before":"023a8084173d5c0664c8afafb8634b8b750aca9b","after":"b7fc9ada9e111247f84747a6d2cc3ac7943d5a4a","ref":"refs/heads/main","pushedAt":"2024-06-03T05:01:22.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Temporarily disable multi_cf_iter in stress test (#12728)\n\nSummary:\nWe plan to re-enable the test after fixing the test.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12728\n\nTest Plan: N/A. Disabling the test\n\nReviewed By: hx235\n\nDifferential Revision: D58071284\n\nPulled By: jaykorean\n\nfbshipit-source-id: af6b45ec7654f9c7b40c36d3b59c7087e27a7af9","shortMessageHtmlLink":"Temporarily disable multi_cf_iter in stress test (#12728)"}},{"before":"b135fa7a8b75b30d084a606450e99511eaf69406","after":"030f7c4a7b270b4ba7fb1b929b92404165291bb1","ref":"refs/heads/9.3.fb","pushedAt":"2024-06-02T23:52:13.000Z","pushType":"push","commitsCount":5,"pusher":{"login":"ajkr","name":"Andrew Kryczka","path":"/ajkr","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4780362?s=80&v=4"},"commit":{"message":"include bug fixes in 9.3.0 from after the branch cut until now","shortMessageHtmlLink":"include bug fixes in 9.3.0 from after the branch cut until now"}},{"before":"0ed93552f4cd6004e966815e1c18347e01628830","after":"b135fa7a8b75b30d084a606450e99511eaf69406","ref":"refs/heads/9.3.fb","pushedAt":"2024-06-02T23:21:28.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ajkr","name":"Andrew Kryczka","path":"/ajkr","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4780362?s=80&v=4"},"commit":{"message":"9.3.0 release notes","shortMessageHtmlLink":"9.3.0 release notes"}},{"before":null,"after":"0ed93552f4cd6004e966815e1c18347e01628830","ref":"refs/heads/9.3.fb","pushedAt":"2024-06-02T23:18:08.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"ajkr","name":"Andrew Kryczka","path":"/ajkr","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4780362?s=80&v=4"},"commit":{"message":"Implement obsolete file deletion (GC) in follower (#12657)\n\nSummary:\nThis PR implements deletion of obsolete files in a follower RocksDB instance. The follower tails the leader's MANIFEST and creates links to newly added SST files. These links need to be deleted once those files become obsolete in order to reclaim space. There are three cases to be considered -\n1. New files added and links created, but the Version could not be installed due to some missing files. Those links need to be preserved so a subsequent catch up attempt can succeed. We insert the next file number in the `VersionSet` to `pending_outputs_` to prevent their deletion.\n2. Files deleted from the previous successfully installed `Version`. These are deleted as usual in `PurgeObsoleteFiles`.\n3. New files added by a `VersionEdit` and deleted by a subsequent `VersionEdit`, both processed in the same catchup attempt. Links will be created for the new files when verifying a candidate `Version`. Those need to be deleted explicitly as they're never added to `VersionStorageInfo`, and thus not deleted by `PurgeObsoleteFiles`.\n\nTest plan -\nNew unit tests in `db_follower_test`.\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12657\n\nReviewed By: jowlyzhang\n\nDifferential Revision: D57462697\n\nPulled By: anand1976\n\nfbshipit-source-id: 898f15570638dd4930f839ffd31c560f9cb73916","shortMessageHtmlLink":"Implement obsolete file deletion (GC) in follower (#12657)"}},{"before":"fc59d8f9c6a8d8b4ff506708f1e78d3b3b54686b","after":"023a8084173d5c0664c8afafb8634b8b750aca9b","ref":"refs/heads/main","pushedAt":"2024-06-01T16:56:44.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Disable iterator refresh for CoalescingIterator in TestIterateAgainstExpected (#12723)\n\nSummary:\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12723\n\n`CoalescingIterator` doesn't support `Refresh` currently; the patch adds a check that was missing from https://github.com/facebook/rocksdb/pull/12721 to disable this operation when multi-CF iterators are in use in the stress test.\n\nReviewed By: jaykorean\n\nDifferential Revision: D58053334\n\nfbshipit-source-id: 3146f0e7e87230b49b244cecdfcee345c0ce78fa","shortMessageHtmlLink":"Disable iterator refresh for CoalescingIterator in TestIterateAgainst…"}},{"before":"f3b7e959b3ee434dbc7217be36ba70a71d8e8697","after":"fc59d8f9c6a8d8b4ff506708f1e78d3b3b54686b","ref":"refs/heads/main","pushedAt":"2024-06-01T02:34:38.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Add public API `WriteWithCallback` to support custom callbacks (#12603)\n\nSummary:\nThis PR adds a `DB::WriteWithCallback` API that does the same things as `DB::Write` while takes an argument `UserWriteCallback` to execute custom callback functions during the write.\n\nWe currently support two types of callback functions: `OnWriteEnqueued` and `OnWalWriteFinish`. The former is invoked after the write is enqueued, and the later is invoked after WAL write finishes when applicable.\n\nThese callback functions are intended for users to use to improve synchronization between concurrent writes, their execution is on the write's critical path so it will impact the write's latency if not used properly. The documentation for the callback interface mentioned this and suggest user to keep these callback functions' implementation minimum.\n\nAlthough transaction interfaces' writes doesn't yet allow user to specify such a user write callback argument, the `DBImpl::Write*` type of APIs do not differentiate between regular DB writes or writes coming from the transaction layer when it comes to supporting this `UserWriteCallback`. These callbacks works for all the write modes including: default write mode, Options.two_write_queues, Options.unordered_write, Options.enable_pipelined_write\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12603\n\nTest Plan: Added unit test in ./write_callback_test\n\nReviewed By: anand1976\n\nDifferential Revision: D58044638\n\nPulled By: jowlyzhang\n\nfbshipit-source-id: 87a84a0221df8f589ec8fc4d74597e72ce97e4cd","shortMessageHtmlLink":"Add public API WriteWithCallback to support custom callbacks (#12603)"}},{"before":"6f17056e401f458d9512b9950af46d8b5e3527fe","after":"f3b7e959b3ee434dbc7217be36ba70a71d8e8697","ref":"refs/heads/main","pushedAt":"2024-05-31T22:20:31.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Add CoalescingIterator to TestIterateAgainstExpected (#12721)\n\nSummary:\nContinuing from https://github.com/facebook/rocksdb/pull/12706. Adding the CoalescingIterator to `TestIterateAgainstExpected` as well when `use_multi_cf_iterator` is set True\n\nPull Request resolved: https://github.com/facebook/rocksdb/pull/12721\n\nTest Plan:\n```\npython3 tools/db_crashtest.py blackbox --simple --max_key=25000000 --write_buffer_size=4194304 --use_attribute_group=0 --use_put_entity_one_in=1 --use_multi_get=1 --use_multi_cf_iterator=1 --verify_iterator_with_expected_state_one_in=2\n```\n\nReviewed By: ltamasi\n\nDifferential Revision: D58033811\n\nPulled By: jaykorean\n\nfbshipit-source-id: 7caf39883e277e695b653e295ad72b1004169ca0","shortMessageHtmlLink":"Add CoalescingIterator to TestIterateAgainstExpected (#12721)"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAEYuQuaAA","startCursor":null,"endCursor":null}},"title":"Activity · facebook/rocksdb"}