build: initial support for poetry build tool #4513

MatriQ · 2024-05-19T15:16:51Z

Description

This PR is just adding poetry configurations in pyproject.toml so that we can use poety to easily manage the environment, as well as a little of README.md.

Fixes # (issue)

Type of Change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update, included: Dify Document
Improvement, including but not limited to code refactoring, performance optimization, and UI/UX improvement
Dependency upgrade

How Has This Been Tested?

This change doesn't bring out any changes of code, and I have tested with generic poetry commands.

TODO

Suggested Checklist:

I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods
optional I have made corresponding changes to the documentation
optional I have added tests that prove my fix is effective or that my feature works
optional New and existing unit tests pass locally with my changes

bowenliang123 · 2024-05-20T08:46:45Z

How to guarantee the [tool.poetry.dependencies] settings align to requireiments.txt? As well as the poetry.lock lock file to the requireiments.txt?

MatriQ · 2024-05-20T12:44:08Z

How to guarantee the [tool.poetry.dependencies] settings align to requireiments.txt? As well as the poetry.lock lock file to the requireiments.txt?

Hi @bowenliang123 , in current stage. we can use poetry add command to keep dependencies consistency as I mentioned in README.md:

In the future, I'd like to suggest that remove requirements.txt and requirements-dev.txt and entirely use poetry to manage dependencies.

bowenliang123 · 2024-05-20T12:50:03Z

Can you figure out at least one way in CI jobs to making sure poetry dependency list satisfying the existed requirements.txt ?

I'm against setting the removal the requirements.txt as final target, considering the following facts:

requirements.txt is generally used out of box with python and pip
poetry is just one of the build systems with dependencies management
For the runtime usage, the build system does not bring any benefits , while it causes reversed dependencies on poetry to the runtime base, like in docker images.

bowenliang123 · 2024-05-20T12:59:45Z

api/.python-version

@@ -0,0 +1 @@
+3.10


This file is unnecessary, as the pyporject.toml has set the required minimum python version.
Also, dify API module is tested on Python 3.10 and 3.11 in CI jobs.
If this is for local development only, please do not submit to the commit.

I'm facing the following issue with 3.11, hence I thought Dify doesn't support 3.11:

However, I will remove it.

The problem you listed is not caused by python 3.11 , I think.
And it should be fixed in #4517, bumping yfinance on minor verions.
Please rebase to the latest commit and check again.

This is an perfect example for inconsistency of your poetry dependencies and the required dependencies from requirements.txt.

MatriQ · 2024-05-20T13:00:26Z

Can you figure out at least one way in CI jobs to making sure poetry dependency list satisfying the existed requirements.txt ?

I'm against setting the removal the requirements.txt as final target, considering the following facts:

requirements.txt is generally used out of box with python and pip

poetry is just one of the build systems with dependencies management

For the runtime usage, the build system does not bring any benefits , while it causes reversed dependencies on poetry to the runtime base, like in docker images.

This PR is just for providing another for developers to manage dependencies, it doesn't impact unit tests, CI and product env. As for your objections, I'd like to discuss with you in another PR(not created yet).

bowenliang123 · 2024-05-20T13:03:51Z

This PR is just for providing another for developers to manage dependencies, it doesn't impact unit tests, CI and product env. As for your opsitions, I'd like to discuss with you in another PR(not created yet).

The introduced support for a new build system should at least respect and align with the existing core mechanisms. Otherwise, it will puzzle the users with the issues caused by inconsistencies in dependencies(which happen often), as the de-facto two different dependencies are listed in the Dify API module.

MatriQ · 2024-05-20T13:20:31Z

This PR is just for providing another for developers to manage dependencies, it doesn't impact unit tests, CI and product env. As for your opsitions, I'd like to discuss with you in another PR(not created yet).

The introduced support for a new build system should at least respect and align with the existing core mechanisms. Otherwise, it will puzzle the users with the issues caused by inconsistencies in dependencies(which happen often), as the de-facto two different dependencies are listed in the Dify API module.

Currently, the only supported build system is still not changed. the dependencies in pyproject.toml will be kept update from requirements.txt. So there won't introduce any new issues from this PR.

takatost · 2024-05-20T13:44:46Z

Hi @bowenliang123, is it possible to migrate to poetry in the future? Actually, currently lint and test are using poetry's configuration file, while package dependencies are using pip. I hope in the future we can consolidate everything with poetry, and poetry can parallelize the dependency fetching process to speed up package installation.

bowenliang123 · 2024-05-20T13:51:35Z

Hi @bowenliang123, is it possible to migrate to poetry in the future? Actually, currently lint and test are using poetry's configuration file, while package dependencies are using pip. I hope in the future we can consolidate everything with poetry, and poetry can parallelize the dependency fetching process to speed up package installation.

Hi @takatost , I am always open to modern build systems like hatch or poetry. All I want to prevent is the inconsistency of dependencies, especially we don't have our decisions in the choice of build system.

loganclark360 · 2024-05-20T14:22:34Z

Support [W. Logan C - Chat @ Spike](https://spikenow.com/r/a/?ref=spike-organic-signature&_ts=2lrqyj) [2lrqyj] On May 20, 2024 at 13:52 GMT, langgenius/dify ***@***.***> wrote: Hi ***@***.***(https://github.com/bowenliang123), is it possible to migrate to poetry in the future? Actually, currently lint and test are using poetry's configuration file, while package dependencies are using pip. I hope in the future we can consolidate everything with poetry, and poetry can parallelize the dependency fetching process to speed up package installation. Hi ***@***.***(https://github.com/takatost) , I am always open to modern build system like hatch or poetry. All I want to prevent is the inconsistency of dependencies. — Reply to this email directly, [view it on GitHub](#4513 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/BH6TQ24SXBJKPMDVNWIPNATZDH5ZFAVCNFSM6AAAAABH6OKP7CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRQGUYDMOJYGI). You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

loganclark360 · 2024-05-20T14:23:06Z

? [W. Logan C - Chat @ Spike](https://spikenow.com/r/a/?ref=spike-organic-signature&_ts=2lrqze) [2lrqze] On May 20, 2024 at 14:22 GMT, W Logan C ***@***.***> wrote: Support [W. Logan C - Chat @ Spike](https://spikenow.com/r/a/?ref=spike-organic-signature&_ts=2lrqyj) [2lrqyj] On May 20, 2024 at 13:52 GMT, langgenius/dify ***@***.***> wrote: Hi ***@***.***(https://github.com/bowenliang123), is it possible to migrate to poetry in the future? Actually, currently lint and test are using poetry's configuration file, while package dependencies are using pip. I hope in the future we can consolidate everything with poetry, and poetry can parallelize the dependency fetching process to speed up package installation. Hi ***@***.***(https://github.com/takatost) , I am always open to modern build system like hatch or poetry. All I want to prevent is the inconsistency of dependencies. — Reply to this email directly, [view it on GitHub](#4513 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/BH6TQ24SXBJKPMDVNWIPNATZDH5ZFAVCNFSM6AAAAABH6OKP7CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRQGUYDMOJYGI). You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

loganclark360 · 2024-05-20T15:51:02Z

Langgenius is also GitHub! Thank you [W. Logan C - Chat @ Spike](https://spikenow.com/r/a/?ref=spike-organic-signature&_ts=2lrv1y) [2lrv1y]

bowenliang123 · 2024-05-21T00:57:36Z

I'm suggesting the following changes in this PR to meet the baseline of acceptance for initial support for poetry build system.

change the PR title build: initial support for poetry build tool
add CI job to check poetry.lock file aligned to the dependency list of [tool.poetry.dependencies] in pyproject.toml, by using poetry check
add CI job in api-tests.yml workflow to install dependencies via poetry and run all through all the tests

These actions still don't guarantee the poetry dependencies aligned to requirements.txt, but it should be good enough for initial support. The dependency list of [tool.poetry.dependencies] in pyproject.toml requires more polishment in version declaration like preferring ^ operator rather than >= + <, which could be done in the future.

WDYT? @takatost @MatriQ

api/pyproject.toml

MatriQ · 2024-05-21T14:11:16Z

I'm suggesting the following changes in this PR to meet the baseline of acceptance for initial support for poetry build system.

change the PR title build: initial support for poetry build tool

add CI job to check poetry.lock file aligned to the dependency list of [tool.poetry.dependencies] in pyproject.toml, by using poetry check

add CI job in api-tests.yml workflow to install dependencies via poetry and run all through all the tests

These actions still don't guarantee the poetry dependencies aligned to requirements.txt, but it should be good enough for initial support. The dependency list of [tool.poetry.dependencies] in pyproject.toml requires more polishment in version declaration like preferring ^ operator rather than >= + <, which could be done in the future.

WDYT? @takatost @MatriQ

It's ok for me. But I'd like to get opinion from @takatost

bowenliang123

LGTM overall. It covers the tests with dependencies installed by peotry. cc @takatost
Minor optional suggestions:

use poetry style version specification like ^ rather than >= + < for readability and maintainability
use actions-poetry v3
basic info of [tool.poetry] should be updated as

name: dify-api
version: 0.6.8
authors: Dify with its official email

takatost

LGTM

bowenliang123 · 2024-06-11T02:07:23Z

api/pyproject.toml

+
+[tool.poetry]
+name = "dify-api"
+version = "0.6.8"


Suggested change

version = "0.6.8"

version = "0.6.10"

Yep. It's best to update to the latest dependencies.

api/pyproject.toml

bowenliang123 · 2024-06-11T02:09:05Z

api/pyproject.toml

+pytest-benchmark = "^4.0.0"
+pytest-env = "^1.1.3"
+pytest-mock = "^3.14.0"
+jinja2 = "^3.1.2"


jinjia2 has been removed from development dependencies.

Co-authored-by: Bowen Liang <bowenliang@apache.org>

dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. 🐍 python Pull requests that update Python code 📚 documentation Improvements or additions to documentation labels May 19, 2024

bowenliang123 reviewed May 20, 2024

View reviewed changes

bowenliang123 reviewed May 21, 2024

View reviewed changes

api/pyproject.toml Show resolved Hide resolved

MatriQ force-pushed the chore/poetry-support branch 5 times, most recently from a77ade4 to ac63620 Compare May 25, 2024 06:43

MatriQ changed the title ~~chore: support poetry~~ build: initial support for poetry build tool May 26, 2024

MatriQ requested a review from bowenliang123 May 26, 2024 03:35

bowenliang123 approved these changes May 26, 2024

View reviewed changes

MatriQ requested a review from bowenliang123 May 26, 2024 16:54

takatost previously approved these changes Jun 11, 2024

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label Jun 11, 2024

bowenliang123 reviewed Jun 11, 2024

View reviewed changes

api/pyproject.toml Show resolved Hide resolved

bowenliang123 reviewed Jun 11, 2024

View reviewed changes

MatriQ and others added 9 commits June 11, 2024 11:47

chore: support poetry

00a7132

chore: add poetry config and pyenv config

6305361

chore: fix package name in pyproject.toml

5b0e57b

chore: update poetry config

e795fec

chore: remove .python-version

58c777a

Update api/pyproject.toml

a5cd7d2

Co-authored-by: Bowen Liang <bowenliang@apache.org>

build: test via poetry

03db020

fix: missing google-cloud-aiplatform issue

8ce9e54

fix: address reviewing comment

3c9f51a

MatriQ force-pushed the chore/poetry-support branch from 74c57ad to 3c9f51a Compare June 11, 2024 03:49

fix: update dependencies

37dbe45

MatriQ dismissed takatost’s stale review via 37dbe45 June 11, 2024 04:45

takatost approved these changes Jun 11, 2024

View reviewed changes

takatost merged commit f62f71a into langgenius:main Jun 11, 2024
7 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build: initial support for poetry build tool #4513

build: initial support for poetry build tool #4513

MatriQ commented May 19, 2024

bowenliang123 commented May 20, 2024 •

edited

MatriQ commented May 20, 2024

bowenliang123 commented May 20, 2024

bowenliang123 May 20, 2024

MatriQ May 20, 2024

bowenliang123 May 20, 2024 •

edited

bowenliang123 May 20, 2024

MatriQ May 20, 2024

bowenliang123 May 20, 2024

MatriQ commented May 20, 2024 •

edited

bowenliang123 commented May 20, 2024 •

edited

MatriQ commented May 20, 2024

takatost commented May 20, 2024

bowenliang123 commented May 20, 2024 •

edited

loganclark360 commented May 20, 2024 via email

loganclark360 commented May 20, 2024 via email

loganclark360 commented May 20, 2024 via email

bowenliang123 commented May 21, 2024

MatriQ commented May 21, 2024

bowenliang123 left a comment •

edited

takatost left a comment

bowenliang123 Jun 11, 2024

takatost Jun 11, 2024

MatriQ Jun 11, 2024

bowenliang123 Jun 11, 2024

bowenliang123 Jun 11, 2024

MatriQ Jun 11, 2024

		@@ -0,0 +1 @@
		3.10

build: initial support for poetry build tool #4513

build: initial support for poetry build tool #4513

Conversation

MatriQ commented May 19, 2024

Description

Type of Change

How Has This Been Tested?

Suggested Checklist:

bowenliang123 commented May 20, 2024 • edited

MatriQ commented May 20, 2024

bowenliang123 commented May 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bowenliang123 May 20, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MatriQ commented May 20, 2024 • edited

bowenliang123 commented May 20, 2024 • edited

MatriQ commented May 20, 2024

takatost commented May 20, 2024

bowenliang123 commented May 20, 2024 • edited

loganclark360 commented May 20, 2024 via email

loganclark360 commented May 20, 2024 via email

loganclark360 commented May 20, 2024 via email

bowenliang123 commented May 21, 2024

MatriQ commented May 21, 2024

bowenliang123 left a comment • edited

Choose a reason for hiding this comment

takatost left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bowenliang123 commented May 20, 2024 •

edited

bowenliang123 May 20, 2024 •

edited

MatriQ commented May 20, 2024 •

edited

bowenliang123 commented May 20, 2024 •

edited

bowenliang123 commented May 20, 2024 •

edited

bowenliang123 left a comment •

edited