Elastic adapter #2727
Merged
Elastic adapter #2727
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
- Added tests - Added some documentation comments - Threaded the passed context through to all the API requests
zricethezav
reviewed
May 23, 2024
| unitsOfWork := distributeDocumentScans(&indices, s.concurrency, scanCoverageRate) | ||
|
|
||
| for uowIndex, outerUOW := range unitsOfWork { | ||
| uow := outerUOW |
There was a problem hiding this comment.
@camgunz
Suggestion: Even though uowIndex is only used in logs, it still needs to have a copy similar to uow inside the loop otherwise we have the potential for incorrect log output
There was a problem hiding this comment.
Oof, nice catch; I think this started life as _ but then I used it for logging and missed pulling it in 👍
zricethezav
reviewed
May 23, 2024
zricethezav
reviewed
May 23, 2024
zricethezav
reviewed
May 23, 2024
| } | ||
| } | ||
|
|
||
| func (uow *UnitOfWork) AddSearch( |
There was a problem hiding this comment.
Question: any reason we need to export this function? Looks like the only place it's being used is distributeDocumentScans and the tests.
There was a problem hiding this comment.
Yeah you're right, no need to export here 👍
zricethezav
reviewed
May 23, 2024
zricethezav
reviewed
May 23, 2024
pkg/sources/elasticsearch/api.go
Outdated
|
|
||
| func (i *Index) DocumentAlreadySeen(document *Document) bool { | ||
| i.lock.Lock() | ||
| defer i.lock.Unlock() |
There was a problem hiding this comment.
Suggestion: rather than locking at the top and deferring the unlock we should just surround contentious access with straight up lock/unlocks rather than defer. Easier to figure out what needs to be protected that way.
There was a problem hiding this comment.
Interesting, yeah I buy that. I'll change what I can -- in this case I'll probably keep defer and just move it below ParseTimestamp because everything else needs to be write synchronized, but I think the others can all be explicit.
zricethezav
reviewed
May 23, 2024
zricethezav
reviewed
May 23, 2024
pkg/sources/elasticsearch/api.go
Outdated
| return "", err | ||
| } | ||
|
|
||
| pitID, found := data["id"].(string) |
There was a problem hiding this comment.
nit: use ok instead of found for consistency
zricethezav
reviewed
May 23, 2024
zricethezav
reviewed
May 23, 2024
zricethezav
reviewed
May 23, 2024
… in subsequent scans
…stamp range clause in searches
…ronized part of the code clearer
zricethezav
approved these changes
May 24, 2024
itsacoyote
pushed a commit
to matter-labs/docs-nuxt-template
that referenced
this pull request
May 29, 2024
[](https://renovatebot.com) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [trufflesecurity/trufflehog](https://togithub.com/trufflesecurity/trufflehog) | action | minor | `v3.76.3` -> `v3.77.0` | --- ### Release Notes <details> <summary>trufflesecurity/trufflehog (trufflesecurity/trufflehog)</summary> ### [`v3.77.0`](https://togithub.com/trufflesecurity/trufflehog/releases/tag/v3.77.0) [Compare Source](https://togithub.com/trufflesecurity/trufflehog/compare/v3.76.3...v3.77.0) #### What's Changed - Remove "finished verificationOverlap chunks" log line by [@​rgmz](https://togithub.com/rgmz) in [trufflesecurity/trufflehog#2860 - fix(deps): update module github.com/wasilibs/go-re2 to v1.5.3 by [@​renovate](https://togithub.com/renovate) in [trufflesecurity/trufflehog#2861 - fix(deps): update module google.golang.org/api to v0.181.0 by [@​renovate](https://togithub.com/renovate) in [trufflesecurity/trufflehog#2857 - fix(deps): update module github.com/aws/aws-sdk-go to v1.53.5 by [@​renovate](https://togithub.com/renovate) in [trufflesecurity/trufflehog#2859 - Update azure storage extra data by [@​abmussani](https://togithub.com/abmussani) in [trufflesecurity/trufflehog#2808 - Update regex for Organization in Azure Devops detector by [@​abmussani](https://togithub.com/abmussani) in [trufflesecurity/trufflehog#2866 - fix(deps): update module github.com/aws/aws-sdk-go to v1.53.6 by [@​renovate](https://togithub.com/renovate) in [trufflesecurity/trufflehog#2867 - \[chore] - Use http.NewRequestWithContext by [@​ahrav](https://togithub.com/ahrav) in [trufflesecurity/trufflehog#2870 - adding Groq detector by [@​0x1](https://togithub.com/0x1) in [trufflesecurity/trufflehog#2873 - Log reasons for GitLab repo exclusion by [@​rosecodym](https://togithub.com/rosecodym) in [trufflesecurity/trufflehog#2875 - \[github] Scan user repositories by [@​rgmz](https://togithub.com/rgmz) in [trufflesecurity/trufflehog#2814 - Elastic adapter by [@​camgunz](https://togithub.com/camgunz) in [trufflesecurity/trufflehog#2727 - Improve handling of Gist URLs by [@​rgmz](https://togithub.com/rgmz) in [trufflesecurity/trufflehog#2653 - Fix some GitHub source test issues by [@​rgmz](https://togithub.com/rgmz) in [trufflesecurity/trufflehog#2774 - fix(deps): update module github.com/aws/aws-sdk-go to v1.53.10 by [@​renovate](https://togithub.com/renovate) in [trufflesecurity/trufflehog#2871 - fix(deps): update module github.com/go-logr/logr to v1.4.2 by [@​renovate](https://togithub.com/renovate) in [trufflesecurity/trufflehog#2869 - fix(deps): update module cloud.google.com/go/secretmanager to v1.13.1 by [@​renovate](https://togithub.com/renovate) in [trufflesecurity/trufflehog#2884 - fix(deps): update golang.org/x/exp digest to [`4c93da0`](https://togithub.com/trufflesecurity/trufflehog/commit/4c93da0) by [@​renovate](https://togithub.com/renovate) in [trufflesecurity/trufflehog#2883 - fix(deps): update module github.com/elastic/go-elasticsearch/v8 to v8.13.1 by [@​renovate](https://togithub.com/renovate) in [trufflesecurity/trufflehog#2886 - fix(deps): update module github.com/gabriel-vasile/mimetype to v1.4.4 by [@​renovate](https://togithub.com/renovate) in [trufflesecurity/trufflehog#2890 - Added extra data for LaunchDarkly by [@​abmussani](https://togithub.com/abmussani) in [trufflesecurity/trufflehog#2836 - feat: support docker image history scanning by [@​jamestelfer](https://togithub.com/jamestelfer) in [trufflesecurity/trufflehog#2882 #### New Contributors - [@​camgunz](https://togithub.com/camgunz) made their first contribution in [trufflesecurity/trufflehog#2727 - [@​jamestelfer](https://togithub.com/jamestelfer) made their first contribution in [trufflesecurity/trufflehog#2882 **Full Changelog**: trufflesecurity/trufflehog@v3.76.3...v3.77.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/matter-labs/docs-nuxt-template). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zNzcuOCIsInVwZGF0ZWRJblZlciI6IjM3LjM3Ny44IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
haraldh
added a commit
to matter-labs/vault-auth-tee
that referenced
this pull request
Jun 10, 2024
](https://renovatebot.com){kind=link}
[](https://renovatebot.com) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [trufflesecurity/trufflehog](https://togithub.com/trufflesecurity/trufflehog) | action | minor | `v3.76.3` -> `v3.78.0` | --- ### Release Notes <details> <summary>trufflesecurity/trufflehog (trufflesecurity/trufflehog)</summary> ### [`v3.78.0`](https://togithub.com/trufflesecurity/trufflehog/releases/tag/v3.78.0) [Compare Source](https://togithub.com/trufflesecurity/trufflehog/compare/v3.77.0...v3.78.0) #### What's Changed - Add postman to tui by [@​hxnyk](https://togithub.com/hxnyk) in [trufflesecurity/trufflehog#2895 - chore(deps): update alpine docker tag to v3.20 by [@​renovate](https://togithub.com/renovate) in [trufflesecurity/trufflehog#2874 - fix(deps): update golang.org/x/exp digest to [`fd00a4e`](https://togithub.com/trufflesecurity/trufflehog/commit/fd00a4e) by [@​renovate](https://togithub.com/renovate) in [trufflesecurity/trufflehog#2899 - Update metadata for DataDog for API + APPKey by [@​abmussani](https://togithub.com/abmussani) in [trufflesecurity/trufflehog#2879 - Consistent docker image of MSSQL for integration testing. by [@​abmussani](https://togithub.com/abmussani) in [trufflesecurity/trufflehog#2898 - fix(deps): update module github.com/charmbracelet/bubbletea to v0.26.4 by [@​renovate](https://togithub.com/renovate) in [trufflesecurity/trufflehog#2885 - Remove 'www' from `DefaultFalsePositives` by [@​rgmz](https://togithub.com/rgmz) in [trufflesecurity/trufflehog#2896 - Redis integration test by [@​abmussani](https://togithub.com/abmussani) in [trufflesecurity/trufflehog#2901 - Fix Github `enumerateWithToken` test failure by [@​rgmz](https://togithub.com/rgmz) in [trufflesecurity/trufflehog#2880 - fix(deps): update module github.com/aws/aws-sdk-go to v1.53.14 by [@​renovate](https://togithub.com/renovate) in [trufflesecurity/trufflehog#2900 - fix(deps): update module github.com/hashicorp/go-retryablehttp to v0.7.7 by [@​renovate](https://togithub.com/renovate) in [trufflesecurity/trufflehog#2904 - integration testing for mongodb. by [@​abmussani](https://togithub.com/abmussani) in [trufflesecurity/trufflehog#2907 - fix(deps): update module github.com/azure/go-autorest/autorest/azure/auth to v0.5.13 by [@​renovate](https://togithub.com/renovate) in [trufflesecurity/trufflehog#2902 - chore: fix some comments by [@​jinjiadu](https://togithub.com/jinjiadu) in [trufflesecurity/trufflehog#2903 - \[chore] Always log git repositories being scanned by [@​mcastorina](https://togithub.com/mcastorina) in [trufflesecurity/trufflehog#2909 - Add Jenkins scanning by [@​dustin-decker](https://togithub.com/dustin-decker) in [trufflesecurity/trufflehog#2892 #### New Contributors - [@​jinjiadu](https://togithub.com/jinjiadu) made their first contribution in [trufflesecurity/trufflehog#2903 **Full Changelog**: trufflesecurity/trufflehog@v3.77.0...v3.78.0 ### [`v3.77.0`](https://togithub.com/trufflesecurity/trufflehog/releases/tag/v3.77.0) [Compare Source](https://togithub.com/trufflesecurity/trufflehog/compare/v3.76.3...v3.77.0) #### What's Changed - Remove "finished verificationOverlap chunks" log line by [@​rgmz](https://togithub.com/rgmz) in [trufflesecurity/trufflehog#2860 - fix(deps): update module github.com/wasilibs/go-re2 to v1.5.3 by [@​renovate](https://togithub.com/renovate) in [trufflesecurity/trufflehog#2861 - fix(deps): update module google.golang.org/api to v0.181.0 by [@​renovate](https://togithub.com/renovate) in [trufflesecurity/trufflehog#2857 - fix(deps): update module github.com/aws/aws-sdk-go to v1.53.5 by [@​renovate](https://togithub.com/renovate) in [trufflesecurity/trufflehog#2859 - Update azure storage extra data by [@​abmussani](https://togithub.com/abmussani) in [trufflesecurity/trufflehog#2808 - Update regex for Organization in Azure Devops detector by [@​abmussani](https://togithub.com/abmussani) in [trufflesecurity/trufflehog#2866 - fix(deps): update module github.com/aws/aws-sdk-go to v1.53.6 by [@​renovate](https://togithub.com/renovate) in [trufflesecurity/trufflehog#2867 - \[chore] - Use http.NewRequestWithContext by [@​ahrav](https://togithub.com/ahrav) in [trufflesecurity/trufflehog#2870 - adding Groq detector by [@​0x1](https://togithub.com/0x1) in [trufflesecurity/trufflehog#2873 - Log reasons for GitLab repo exclusion by [@​rosecodym](https://togithub.com/rosecodym) in [trufflesecurity/trufflehog#2875 - \[github] Scan user repositories by [@​rgmz](https://togithub.com/rgmz) in [trufflesecurity/trufflehog#2814 - Elastic adapter by [@​camgunz](https://togithub.com/camgunz) in [trufflesecurity/trufflehog#2727 - Improve handling of Gist URLs by [@​rgmz](https://togithub.com/rgmz) in [trufflesecurity/trufflehog#2653 - Fix some GitHub source test issues by [@​rgmz](https://togithub.com/rgmz) in [trufflesecurity/trufflehog#2774 - fix(deps): update module github.com/aws/aws-sdk-go to v1.53.10 by [@​renovate](https://togithub.com/renovate) in [trufflesecurity/trufflehog#2871 - fix(deps): update module github.com/go-logr/logr to v1.4.2 by [@​renovate](https://togithub.com/renovate) in [trufflesecurity/trufflehog#2869 - fix(deps): update module cloud.google.com/go/secretmanager to v1.13.1 by [@​renovate](https://togithub.com/renovate) in [trufflesecurity/trufflehog#2884 - fix(deps): update golang.org/x/exp digest to [`4c93da0`](https://togithub.com/trufflesecurity/trufflehog/commit/4c93da0) by [@​renovate](https://togithub.com/renovate) in [trufflesecurity/trufflehog#2883 - fix(deps): update module github.com/elastic/go-elasticsearch/v8 to v8.13.1 by [@​renovate](https://togithub.com/renovate) in [trufflesecurity/trufflehog#2886 - fix(deps): update module github.com/gabriel-vasile/mimetype to v1.4.4 by [@​renovate](https://togithub.com/renovate) in [trufflesecurity/trufflehog#2890 - Added extra data for LaunchDarkly by [@​abmussani](https://togithub.com/abmussani) in [trufflesecurity/trufflehog#2836 - feat: support docker image history scanning by [@​jamestelfer](https://togithub.com/jamestelfer) in [trufflesecurity/trufflehog#2882 #### New Contributors - [@​camgunz](https://togithub.com/camgunz) made their first contribution in [trufflesecurity/trufflehog#2727 - [@​jamestelfer](https://togithub.com/jamestelfer) made their first contribution in [trufflesecurity/trufflehog#2882 **Full Changelog**: trufflesecurity/trufflehog@v3.76.3...v3.77.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/matter-labs/vault-auth-tee). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zNzcuOCIsInVwZGF0ZWRJblZlciI6IjM3LjM4OC4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119-->
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
This PR is to add an elasticsearch source.
Checklist:
make test-community)?tarsource?make lintthis requires golangci-lint)?