Switch PyProject to Poetry

[ppfeister]

Continuation of #2036

Package info

Sherlock Project added as Author
Homepage URL set to the github[.]io homepage (PyPi homepage already set, but not pip show)
Dynamic versioning fixed (pulled from sherlock[.]py)
Maintainers added
Classifiers added
Keywords added
Description added
etc

---

Refocused PR : PrProject being merged elsewhere to expedite package review process and resolve dependency issues (it takes time). This PR is now only to discuss switching to Poetry. (main pr #2116)

::: >>> Jump to start of refocused discussion <<< :::

[matheusfelipeog]

Just a comment to start a discussion:

I personally use the Poetry package manager instead of directly using pip for projects like this. It has a slightly different format from what setuptools accepts; you can view it here: https://python-poetry.org/docs/pyproject/

I find its dependency resolution mechanism better, with features that prevent the addition of dependencies that are not compatible with others in the project programmatically.

Since in the future we intend to add Sherlock to PyPI, Poetry can also help with that, as it has native build and publish mechanisms, eliminating the need to use other tools like twine.

For the average user, nothing would change; pip would work normally.
For development, well, it has all these features I mentioned above and a few more.

What do you think about this?

---

Refs:

• Python Packaging using Python-Poetry #1241
• https://python-poetry.org/
• https://python-poetry.org/docs/basic-usage/
• https://python-poetry.org/docs/pyproject/
• https://python-poetry.org/docs/cli/

[ppfeister]

Personally indifferent. No reservations or preference either way.
As long as normal pip usage isn't broken, then additional package manager support can only really be a plus.

As someone who hasn't used poetry, would this be as simple as making the pyproject compatible? Misread. It does in fact appear that simple.

[ppfeister]

So, I may be wrong on this, but it seems that poetry does not support dynamic versioning OOTB.

The package version will either have to be manually updated each time, or it would require an additional plugin that versions based on git tags (which sherlock doesn't currently use). Thoughts?

[ppfeister]

It also doesn't seem that poetry has support for dynamic loading of dependencies -- they would need to be explicitly listed in the pyproj. Thoughts there?

Seems finicky on my end. I'm able to install with poetry right now, but unable to uninstall it with poetry (need to use pip). I'm also unable to run sherlock via sherlock due to an odd import error, needing to run it with poetry run sherlock instead. I'm sure both problems can be solved, just haven't seen why yet.

Latest commit cb22902 is what I have for poetry.

[ppfeister]

Rebased onto master to fix conflict (some ~183 behind).

Poetry rewritten in a way that actually works.

Discovered that the current version of pyproject has some ~undesirable~ install issues, namely it doesn't install the src into a named directory and dumps them straight into site-packages for some unknown reason. Not only is this not pretty, it can cause some conflicts. This version fixes that behavior.

Due to how poetry handles packaging, the imports had to be rewritten.

Due to how the imports were rewritten, the (already poorly located) unit tests no longer function as desired.

Unit tests were relocated to the root where they probably should be anyways. Unit tests work from this location with modified imports.

**** PR action currently uses the old unit test location, so failure here is expected. ****

GitHub Actions for PRs will be fixed in another PR. I'll probably take advantage of that fixup to switch from the unittest module to tox, which is a bit more portable in my opinion.

This also removes the requirements.txt file, as dependencies should now be listed within the pyproj. If some sort of workflow requires the requirements.txt file, poetry itself can dump it from the pyproj.

Removed the setup.py and setup.cfg items as they are no longer necessary with the new layout.

Removed the rpm spec file.

Dynamic versioning still needs to be set up.

[ppfeister]

(a couple minor doc updates included, i.e. pyproj readme logo and capitalization)

[ppfeister]

Dynamic versioning added but not yet enabled. Relies on named tags and applies throughout Sherlock, rather than pulling from src.

[ppfeister]

Fixed an import issue in entry point __main__.

Note that python3 sherlock/sherlock is not likely to work anymore, as that would require a bunch of try catch imports. python3 -m sherlock should be used to load the module properly now. Documentation updated to reflect.

$ python -m sherlock --version
Sherlock  0.14.4
Requests  2.28.2
Python    3.12.3

[matheusfelipeog]

So, I may be wrong on this, but it seems that poetry does not support dynamic versioning OOTB.

The package version will either have to be manually updated each time, or it would require an additional plugin that versions based on git tags (which sherlock doesn't currently use). Thoughts?

Yes, poetry doesn't natively support dynamic versioning. We use community plugins for that. We don't need to rely on git tags; we can use the __version__ variable which should be located in sherlock/__init__.py (this is the location used by the community to store package metadata like version, name, etc.).

The plugin we can use for this is: https://github.com/tiangolo/poetry-version-plugin

It also doesn't seem that poetry has support for dynamic loading of dependencies -- they would need to be explicitly listed in the pyproj. Thoughts there?

This isn't necessary. They should be defined directly in the dependencies group of the project in pyproject.toml. I see that you've already done that, good job.

Seems finicky on my end. I'm able to install with poetry right now, but unable to uninstall it with poetry (need to use pip). I'm also unable to run sherlock via sherlock due to an odd import error, needing to run it with poetry run sherlock instead. I'm sure both problems can be solved, just haven't seen why yet.

Poetry is a package manager like pip and also an environment manager like virtualenv. To use it correctly, you need to activate the virtual environment by using poetry shell in the project directory.

The command poetry run script.py does exactly that but does not keep the virtual environment active in the shell session. The virtual environment will only be used to run the script and then it will exit.

[ppfeister]

Most of my original comments seem to be resolved, which is nice. It was easier to just rewrite it rather than adapt it for some reason.

Yes, poetry doesn't natively support dynamic versioning. We use community plugins for that. We don't need to rely on git tags; we can use the version variable which should be located in sherlock/init.py (this is the location used by the community to store package metadata like version, name, etc.).

The plugin we can use for this is: https://github.com/tiangolo/poetry-version-plugin

My original find was poetry-dynamic-versioning, but that dynamically populates version tags within the package rather than dynamically populating the pyproj from tags within the package (backwards). Your linked plugin might be better since it pulls from src as before

Only problem I've noticed is that there doesn't seem to be a way to require your plugin for builds. Since there is no backend, it also doesn't change how things are built sans-poetry build. python -m build will now forever use version 0, and poetry build itself will return 0 by default unless a dev goes out of their way to realize it's needed and install the plugin.

Any suggestions to enforce this plugin, or just let people figure it out?

[matheusfelipeog]

There is no way to require poetry plugins because they are not part of the project itself but rather part of poetry. This is a community need, and there are discussions about implementing this in the future.

Ideally, we should have a workflow for build and publish so we don't handle this manually.

For those who want to build locally from the source for any reason, this should be documented in the respective section of the documentation.

[ppfeister]

Well in that case, that's all I've got. Probably good to merge... don't think anything else needs to be tweaked

(except the github actions, which should probably be addressed separately)

[matheusfelipeog]

Here are a few more suggestions.