Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDFS is unsupported #64126

Merged
merged 4 commits into from
May 21, 2024
Merged

HDFS is unsupported #64126

merged 4 commits into from
May 21, 2024

Conversation

alexey-milovidov
Copy link
Member

Changelog category (leave one):

  • Documentation (changelog entry is not required)

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

Modify your CI run

NOTE: If your merge the PR with modified CI you MUST KNOW what you are doing
NOTE: Checked options will be applied if set before CI RunConfig/PrepareRunConfig step

Include tests (required builds will be added automatically):

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • All with ASAN
  • All with TSAN
  • All with Analyzer
  • All with Azure
  • Add your option here

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Performance tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All with Aarch64
  • Add your option here

Extra options:

  • do not test (only style check)
  • disable merge-commit (no merge from master before tests)
  • disable CI cache (job reuse)

Only specified batches in multi-batch jobs:

  • 1
  • 2
  • 3
  • 4

@robot-clickhouse-ci-1 robot-clickhouse-ci-1 added the pr-documentation Documentation PRs for the specific code PR label May 20, 2024
@robot-clickhouse-ci-1
Copy link
Contributor

robot-clickhouse-ci-1 commented May 20, 2024

This is an automated comment for commit 39b84d0 with description of existing statuses. It's updated for the latest CI running

✅ Click here to open a full report in a separate page

Successful checks
Check nameDescriptionStatus
A SyncThere's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS✅ success
CI runningA meta-check that indicates the running CI. Normally, it's in success or pending state. The failed status indicates some problems with the PR✅ success
Docs checkBuilds and tests the documentation✅ success
Mergeable CheckChecks if all other necessary checks are successful✅ success
PR CheckThere's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS✅ success
Style checkRuns a set of checks to keep the code style clean. If some of tests failed, see the related log from the report✅ success

@@ -464,9 +466,11 @@ In this sample configuration:
</clickhouse>
```

Keep in mind that HDFS does not work at all.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May be this message is too much? "At all" is not really true... If if did not work at all, there would have been no sense having hdfs in documentation at all.

Copy link
Contributor

@den-crane den-crane May 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kssenii latest HDFS have a hard limit 1million files (objects) per folder.
Clickhouse stores all objects in a single HDFS folder (for HDFS as external storage, unlike S3).
The average table with 500 columns reaches this limit of 1million objects easily, it's less than 1000 active+inactive wide parts, after that a table is unable to insert or to merge.

#43112

docs/en/operations/storing-data.md Outdated Show resolved Hide resolved
@kssenii kssenii self-assigned this May 20, 2024
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
@alexey-milovidov alexey-milovidov added this pull request to the merge queue May 21, 2024
Merged via the queue into master with commit b4e87f3 May 21, 2024
18 checks passed
@alexey-milovidov alexey-milovidov deleted the alexey-milovidov-patch-8 branch May 21, 2024 02:18
@robot-ch-test-poll robot-ch-test-poll added the pr-synced-to-cloud The PR is synced to the cloud repo label May 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-documentation Documentation PRs for the specific code PR pr-synced-to-cloud The PR is synced to the cloud repo
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants