Finish physical redesign #4997

thejohnfreeman · 2024-04-19T19:56:33Z

This is the final and most disruptive phase of the physical redesign. It makes these changes:

CMake modules are moved into their own top-level cmake/ directory.
libxrpl headers are moved into include/xrpl/.
libxrpl sources are moved to src/libxrpl/.
rippled sources are moved to src/xrpld/.
The ripple C++ namespace is unchanged.
libxrpl headers are installed to both the xrpl/ and ripple/ subdirectories of the header installation prefix. (ripple/ is a symbolic link to xrpl/, even on Windows.) They include each other with path prefix xrpl/, but existing projects can continue to include them with path prefix ripple/ and migrate to prefix xrpl/ at their own pace.
rippled is installed as both rippled and xrpld. (xrpld is a symbolic link to rippled, even on Windows.)
CMake source lists are computed from globs. Newly added source and header files are automatically built and installed without any intervention from contributors.
CMake target names are cleaned up and standardized.

There is a non-trivial (but, I think, easy) process to merge existing branches-in-progress with this one. I will describe it here. Look at the sequence of commits. Notice the commit with subject "Add bin/physical.sh". Call that the checkpoint commit. There are no file movements before the checkpoint commit. The four commits following the checkpoint commit are generated by running the bin/physical.sh script added in the checkpoint commit.

When you want to merge, you should:

Merge with develop. Resolve conflicts yourself. Be sure that all of the source files added in your branch (if any) appear in the correct source lists in Builds/CMake/RippledCore.cmake, between the BEGIN/END marker comments (example) . The restructure script moves only the files in these lists, and stops early if any files are leftover (because they were omitted from the lists). Remember the commit ID of this merge. Call it your marker commit.
```
git merge develop
marker=$(git log -1 --pretty=format:%h)
```
Merge with the checkpoint commit. Resolve conflicts yourself. (I will try to keep this instruction up-to-date, but the commit ID below may be incorrect. Please double-check the commits in this PR to identify the correct checkpoint commit.)
```
git merge 64a6c40
```
Regenerate the next four commits by executing bin/physical.sh, passing it your marker commit from the first step:
```
./bin/physical.sh ${marker}
```
Check the sort order in Builds/levelization/results/ordering.txt. Lines beginning with xrpl. must appear before lines beginning with xrpld. Different versions of sort have been inconsistent on this matter in my experience. On my machine, sort puts xrpl. after xrpld, which does not match the output of sort in our CI job that checks the file.
```
${EDITOR} Builds/levelization/results/ordering.txt
git add --update .
git commit --amend
```

Then you can git merge -X ours $upstream where upstream points to the head of this branch. It will automatically resolve conflicts in your branch's favor. Then you can git diff $upstream and see your work ported.

This PR requires the same finishing steps as #4966:

Confirm it has at least two approvals from qualified reviewers.
Rebase the commits on top of develop. The commit sequence that predates the review should be preserved. Fixes made since the review opened should be merged into earlier commits.
Rewrite non-substantive (e.g. formatting) commits as authored by "Pretty Printer <cpp@ripple.com>".
Add rewritten non-substantive commit IDs to .git-blame-ignore-revs.
Confirm the branch still passes all tests after making these changes.

codecov-commenter · 2024-04-20T15:33:53Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.3%. Comparing base (d576416) to head (92bd4ca).

Additional details and impacted files

@@            Coverage Diff            @@
##           develop   #4997     +/-   ##
=========================================
- Coverage     71.3%   71.3%   -0.0%     
=========================================
  Files          796     796             
  Lines        66883   66883             
  Branches     10866   10870      +4     
=========================================
- Hits         47685   47677      -8     
- Misses       19198   19206      +8

Files	Coverage Δ
include/xrpl/basics/BasicConfig.h	`87.7% <ø> (ø)`
include/xrpl/basics/Buffer.h	`100.0% <ø> (ø)`
include/xrpl/basics/ByteUtilities.h	`100.0% <ø> (ø)`
include/xrpl/basics/CompressionAlgorithms.h	`0.0% <ø> (ø)`
include/xrpl/basics/CountedObject.h	`100.0% <ø> (ø)`
include/xrpl/basics/DecayingSample.h	`87.9% <ø> (ø)`
include/xrpl/basics/Expected.h	`100.0% <ø> (ø)`
include/xrpl/basics/FeeUnits.h	`90.1% <ø> (ø)`
include/xrpl/basics/IOUAmount.h	`100.0% <ø> (ø)`
include/xrpl/basics/LocalValue.h	`100.0% <ø> (ø)`
... and 159 more

... and 1254 files with indirect coverage changes

conanfile.py

CMakeLists.txt

Bronek · 2024-05-20T15:09:46Z

Note, this is moving files from impl and details to their parent directories, which I think is a problem because that expands the libxrpl API surface with the following files (which were designed to be implementation and not API)

json/impl/json_assert.h

protocol/impl/STVar.h  protocol/impl/b58_utils.h  protocol/impl/secp256k1.h  protocol/impl/token_errors.h

resource/impl/Entry.h  resource/impl/Import.h  resource/impl/Key.h  resource/impl/Kind.h  resource/impl/Logic.h  resource/impl/Tuning.h

server/impl/BaseHTTPPeer.h  server/impl/BaseWSPeer.h  server/impl/JSONRPCUtil.h  server/impl/PlainHTTPPeer.h  server/impl/SSLHTTPPeer.h  server/impl/ServerImpl.h
server/impl/BasePeer.h      server/impl/Door.h        server/impl/LowestLayer.h  server/impl/PlainWSPeer.h    server/impl/SSLWSPeer.h    server/impl/io_list.h

Bronek · 2024-06-07T13:32:37Z

Note, this is moving files from impl and details to their parent directories, which I think is a problem because that expands the libxrpl API surface with the following files (which were designed to be implementation and not API)

json/impl/json_assert.h

protocol/impl/STVar.h  protocol/impl/b58_utils.h  protocol/impl/secp256k1.h  protocol/impl/token_errors.h

resource/impl/Entry.h  resource/impl/Import.h  resource/impl/Key.h  resource/impl/Kind.h  resource/impl/Logic.h  resource/impl/Tuning.h

server/impl/BaseHTTPPeer.h  server/impl/BaseWSPeer.h  server/impl/JSONRPCUtil.h  server/impl/PlainHTTPPeer.h  server/impl/SSLHTTPPeer.h  server/impl/ServerImpl.h
server/impl/BasePeer.h      server/impl/Door.h        server/impl/LowestLayer.h  server/impl/PlainWSPeer.h    server/impl/SSLWSPeer.h    server/impl/io_list.h

@thejohnfreeman thanks for addressing this concern.

Bronek

I like this a lot.

Bronek

I noticed that the most recent rebase has reverted the change to which I refer here
#4997 (comment)

Is it intentional ?

Bronek · 2024-06-10T16:26:35Z

I noticed that the most recent rebase has reverted the change to which I refer here #4997 (comment)

Thanks @thejohnfreeman , I see that's fixed now.

scottschurr · 2024-06-11T18:20:40Z

I'm trying to follow the directions for merging an in-process branch with the physical restructure. Everything was going well until I ran bin/physical.sh. There I had two surprises.

The output from bin/physical.sh was this:

move protocol buffers
move libxrpl headers
move libxrpl sources
check leftovers
leftover files:
beast//utility/src/beast_PropertyStream.cpp crypto//impl/secure_erase.cpp

The leftover files was a surprise, and the instructions did not say what to do if that happened, so I'm reporting it here.

Additionally, I checked the sort order of Builds/levelization/results/ordering.txt as instructed. To my surprise it contained no lines that started with either xrpl. or xrpld. All 229 lines in the file started with either ripple. or test.. So I suspect something went wrong.

Suggestions? Thanks.

thejohnfreeman · 2024-06-11T19:03:37Z

@scottschurr what branch did you try?

scottschurr · 2024-06-11T19:22:06Z

@thejohnfreeman, this is the branch I tried: https://github.com/scottschurr/rippled/tree/blocked-book

thejohnfreeman · 2024-06-11T22:34:42Z

I just merged that branch without a hitch: https://github.com/thejohnfreeman/rippled/tree/blocked-book

I'll try to replicate your experience on macOS soon.

scottschurr · 2024-06-11T22:58:24Z

Thanks. And if there are any experiments you would like me to try I'm happy to do so. Best of luck.

- Remove CMake module "MultiConfig". - Update clang-format configuration, CodeCov configuration, levelization script. - Replace source lists in CMake with globs.

thejohnfreeman · 2024-06-12T02:06:43Z

The problem is what I expected, the environment. The requirements I found missing on macOS are:

GNU sed
GNU find
Bash >= 4 (for readarray)
clang-format-10 must be on the PATH

Can you run the script in a Linux environment? Alternatively, if you can get these programs on your PATH (brew install gnu-sed findutils will install gsed and gfind; Homebrew doesn't have clang-format-10) and invoke the script with Bash (the default shell in macOS is now Zsh), then it should work.

scottschurr · 2024-06-12T15:48:10Z

I don't have a readily available Linux environment, but I can install gsed and gfind using homebrew and invoke the script with Bash. I'm in the middle of a different (unrelated) problem right now, but I should be able to get you results in a day or so. Thanks for the help.

thejohnfreeman force-pushed the restructure branch 2 times, most recently from e604953 to e068be2 Compare April 20, 2024 15:12

thejohnfreeman force-pushed the restructure branch 2 times, most recently from 4d2f327 to e25ebac Compare April 22, 2024 17:11

thejohnfreeman marked this pull request as ready for review April 22, 2024 17:12

thejohnfreeman mentioned this pull request Apr 23, 2024

[build docs error] (Version: [2.1.1]) #4973

Open

thejohnfreeman mentioned this pull request May 1, 2024

Add external directory for Conan recipe's exports #5006

Merged

5 tasks

thejohnfreeman force-pushed the restructure branch from e25ebac to b2463c1 Compare May 2, 2024 15:29

Bronek self-requested a review May 3, 2024 10:31

Bronek reviewed May 16, 2024

View reviewed changes

conanfile.py Show resolved Hide resolved

CMakeLists.txt Show resolved Hide resolved

thejohnfreeman force-pushed the restructure branch from b2463c1 to b00160d Compare June 6, 2024 22:00

Bronek approved these changes Jun 7, 2024

View reviewed changes

thejohnfreeman mentioned this pull request Jun 7, 2024

Can't process new ledgers: The current ETL source is not compatible with the version of the libxrpl Clio is currently using. Please upgrade Clio to a newer version (Version: 2.1.1~26ed78f-1) XRPLF/clio#1410

Open

thejohnfreeman force-pushed the restructure branch from b00160d to d3e2788 Compare June 7, 2024 18:49

Bronek self-requested a review June 10, 2024 10:49

Bronek reviewed Jun 10, 2024

View reviewed changes

thejohnfreeman force-pushed the restructure branch from d3e2788 to 75ac07d Compare June 10, 2024 15:35

thejohnfreeman force-pushed the restructure branch from 75ac07d to d40a7c6 Compare June 10, 2024 17:22

thejohnfreeman and others added 3 commits June 11, 2024 19:31

Prepare to rearrange sources

9c2d13c

- Remove CMake module "MultiConfig". - Update clang-format configuration, CodeCov configuration, levelization script. - Replace source lists in CMake with globs.

Add bin/physical.sh

64a6c40

Move CMake directory

1ff391a

Pretty Printer added 3 commits June 11, 2024 19:31

Rearrange sources

4617566

Rewrite includes

8ccb974

Recompute loops

92bd4ca

thejohnfreeman force-pushed the restructure branch from d40a7c6 to 92bd4ca Compare June 12, 2024 00:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Finish physical redesign #4997

Finish physical redesign #4997

thejohnfreeman commented Apr 19, 2024 •

edited

codecov-commenter commented Apr 20, 2024 •

edited by codecov bot

Bronek commented May 20, 2024

Bronek commented Jun 7, 2024

Bronek left a comment

Bronek left a comment •

edited

Bronek commented Jun 10, 2024 •

edited

scottschurr commented Jun 11, 2024

thejohnfreeman commented Jun 11, 2024

scottschurr commented Jun 11, 2024

thejohnfreeman commented Jun 11, 2024

scottschurr commented Jun 11, 2024

thejohnfreeman commented Jun 12, 2024 •

edited

scottschurr commented Jun 12, 2024

Finish physical redesign #4997

Are you sure you want to change the base?

Finish physical redesign #4997

Conversation

thejohnfreeman commented Apr 19, 2024 • edited

codecov-commenter commented Apr 20, 2024 • edited by codecov bot

Codecov Report

Bronek commented May 20, 2024

Bronek commented Jun 7, 2024

Bronek left a comment

Choose a reason for hiding this comment

Bronek left a comment • edited

Choose a reason for hiding this comment

Bronek commented Jun 10, 2024 • edited

scottschurr commented Jun 11, 2024

thejohnfreeman commented Jun 11, 2024

scottschurr commented Jun 11, 2024

thejohnfreeman commented Jun 11, 2024

scottschurr commented Jun 11, 2024

thejohnfreeman commented Jun 12, 2024 • edited

scottschurr commented Jun 12, 2024

thejohnfreeman commented Apr 19, 2024 •

edited

codecov-commenter commented Apr 20, 2024 •

edited by codecov bot

Bronek left a comment •

edited

Bronek commented Jun 10, 2024 •

edited

thejohnfreeman commented Jun 12, 2024 •

edited