[Cloud Security] Refactor D4C metering function #183814

CohenIdo · 2024-05-20T10:45:35Z

Summary

Refactor the Cloud Defend metering to sample each agent separately and do so once per hour.

elasticmachine · 2024-05-20T10:45:38Z

Pinging @elastic/kibana-cloud-security-posture (Team:Cloud Security)

CohenIdo · 2024-05-20T10:48:48Z

@elasticmachine merge upstream

x-pack/plugins/security_solution_serverless/server/cloud_security/constants.ts

CohenIdo · 2024-05-22T05:13:35Z

@elasticmachine merge upstream

CohenIdo · 2024-05-22T05:18:53Z

@elasticmachine merge upstream

joeypoon · 2024-05-22T15:03:04Z

x-pack/plugins/security_solution_serverless/server/cloud_security/containters_defend.ts

+    {
+      index: CLOUD_DEFEND_HEARTBEAT_INDEX,
+      sort: 'event.ingested',
+      query: {


We probably need an explicit size param here otherwise it will only pull 10 docs per run and miss the other heartbeats.

joeypoon · 2024-05-22T15:05:37Z

x-pack/plugins/security_solution_serverless/server/cloud_security/cloud_security_metering.ts

@@ -25,20 +26,31 @@ export const cloudSecurityMetringCallback = async ({
  const tier: Tier = getCloudProductTier(config, logger);

  try {
-    const cloudSecuritySolutions: CloudSecuritySolutions[] = [CSPM, KSPM, CNVM, CLOUD_DEFEND];
+    const cloudSecuritySolutions: CloudSecuritySolutions[] = [CLOUD_DEFEND];

    const promiseResults = await Promise.allSettled(


It might be good to additionally return a shouldRunAgain and latestTimestamp along with the records so that if there are more heartbeat documents, the task will run again immediately to pick up the rest of the heartbeats.

@joeypoon It's great input,
here's how I addressed it.

Your approach makes sense to me so I went ahead and approved. Something to keep in mind with this approach is that if there is a very large amount of heartbeats the task might potentially run into the 1 minute timeout.

CohenIdo · 2024-05-28T07:33:11Z

@elasticmachine merge upstream

eyalkraft · 2024-05-28T08:40:53Z

...plugins/security_solution_serverless/server/cloud_security/defend_for_containers_metering.ts

+  };
+}
+
+const buildMeteringRecord = (


Suggested change

const buildMeteringRecord = (

const buildUsageRecord = (

eyalkraft · 2024-05-28T08:42:37Z

...plugins/security_solution_serverless/server/cloud_security/defend_for_containers_metering.ts

+  );
+};
+
+export const getCloudDefendUsageRecord = async ({


Suggested change

export const getCloudDefendUsageRecord = async ({

export const getCloudDefendUsageRecords = async ({

eyalkraft · 2024-05-28T08:45:38Z

...plugins/security_solution_serverless/server/cloud_security/defend_for_containers_metering.ts

+      if (usageRecords.hits.hits.length < BATCH_SIZE) {
+        fetchMore = false;
+      } else {
+        searchAfter = usageRecords.hits.hits[usageRecords.hits.hits.length - 1].sort;


does this return the latest timestamp from the previous batch?

eyalkraft · 2024-05-28T08:58:40Z

...plugins/security_solution_serverless/server/cloud_security/defend_for_containers_metering.ts

+  timestamp.setMilliseconds(0);
+
+  const usageRecord = {
+    id: `container-defend-${agentId}-${timestamp.toISOString()}`,


I'm not sure I understood correctly - we do create a usage, record record per heartbeat, but we trust the id uniqueness to squash them into a single record every half an hour?

The agent sends a heartbeat every 30 minutes. We want to count records with an hourly granularity, so every 2 records will have the same record ID. Later, in the glue function, they will be merged into one.

CohenIdo · 2024-05-28T20:11:33Z

@elasticmachine merge upstream

kfirpeled · 2024-05-29T18:59:15Z

x-pack/plugins/security_solution_serverless/server/task_manager/usage_reporting_task.ts

@@ -153,7 +153,7 @@ export class SecurityUsageReportingTask {
      return { state: taskInstance.state, runAt: new Date() };
    }

-    this.logger.debug(`received usage records: ${JSON.stringify(usageRecords)}`);
+    this.logger.warn(`received usage records: ${JSON.stringify(usageRecords)}`); // TODO: change to info before merge


@CohenIdo this is still warn

kibana-ci · 2024-05-29T21:46:31Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: acb9728
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-183814-acb972877c4d
Security Deployment

Failed CI Steps

Jest Integration Tests #6

Metrics [docs]

Canvas Sharable Runtime

The Canvas "shareable runtime" is an bundle produced to enable running Canvas workpads outside of Kibana. This bundle is included in third-party webpages that embed canvas and therefor should be as slim as possible.

id	before	after	diff
`module count`	-	5412	+5412
`total size`	-	8.8MB	+8.8MB

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

CohenIdo added 2 commits May 20, 2024 13:37

fix d4c metering

fc8bf5b

clean comments

4864325

CohenIdo added release_note:skip Skip the PR/issue when compiling release notes Team:Cloud Security Cloud Security team related v8.14.0 labels May 20, 2024

CohenIdo requested a review from a team as a code owner May 20, 2024 10:45

CohenIdo marked this pull request as draft May 20, 2024 10:45

CohenIdo added the ci:project-deploy-security Create a Security Project label May 20, 2024

CohenIdo marked this pull request as ready for review May 20, 2024 10:49

CohenIdo added backport:skip This commit does not require backporting and removed v8.14.0 labels May 20, 2024

kfirpeled reviewed May 21, 2024

View reviewed changes

x-pack/plugins/security_solution_serverless/server/cloud_security/constants.ts Outdated Show resolved Hide resolved

debug

41ebf91

CohenIdo force-pushed the update-d4c-metering branch from df40095 to 41ebf91 Compare May 22, 2024 05:13

CohenIdo marked this pull request as draft May 22, 2024 05:13

debug

0e39cfc

CohenIdo force-pushed the update-d4c-metering branch from 5e24f64 to 0e39cfc Compare May 22, 2024 05:29

CohenIdo added 2 commits May 22, 2024 14:46

refactor cloud defend metering

4c8df26

refactor cloud defend metering

bcfe85d

CohenIdo requested a review from kfirpeled May 22, 2024 11:53

Merge remote-tracking branch 'upstream/main' into update-d4c-metering

4cfd3f0

CohenIdo requested a review from joeypoon May 22, 2024 11:54

CohenIdo marked this pull request as ready for review May 22, 2024 11:54

CohenIdo changed the title ~~[Cloud Security] update d4c metering function~~ [Cloud Security] Refactor D4C metering function May 22, 2024

CohenIdo force-pushed the update-d4c-metering branch from cc7dcec to 4cfd3f0 Compare May 22, 2024 11:55

joeypoon reviewed May 22, 2024

View reviewed changes

xode review comments

52e81f8

CohenIdo force-pushed the update-d4c-metering branch from e72fc9b to 52e81f8 Compare May 23, 2024 13:57

CohenIdo added 2 commits May 23, 2024 16:58

Merge remote-tracking branch 'upstream/main' into update-d4c-metering

c44f296

handle pagination

a4b6b5d

CohenIdo requested a review from joeypoon May 23, 2024 14:29

kfirpeled requested a review from eyalkraft May 23, 2024 15:07

joeypoon approved these changes May 23, 2024

View reviewed changes

eyalkraft approved these changes May 28, 2024

View reviewed changes

CohenIdo added 2 commits May 28, 2024 13:20

Merge remote-tracking branch 'upstream/main' into update-d4c-metering

8db2801

code review comments

444fd4d

CohenIdo force-pushed the update-d4c-metering branch from 54422b0 to 444fd4d Compare May 28, 2024 13:12

eslint fix

d7c51cd

tests

69e93eb

CohenIdo force-pushed the update-d4c-metering branch from a83d403 to 69e93eb Compare May 29, 2024 11:14

CohenIdo added 2 commits May 29, 2024 14:15

tests

80b06b5

Merge remote-tracking branch 'upstream/main' into update-d4c-metering

3d4afd6

kfirpeled reviewed May 29, 2024

View reviewed changes

CohenIdo added 2 commits May 29, 2024 23:46

clean code

d6e76a0

Merge remote-tracking branch 'upstream/main' into update-d4c-metering

acb9728

CohenIdo merged commit fb12588 into elastic:main May 30, 2024
38 checks passed

kibanamachine added the v8.15.0 label May 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Cloud Security] Refactor D4C metering function #183814

[Cloud Security] Refactor D4C metering function #183814

CohenIdo commented May 20, 2024 •

edited

elasticmachine commented May 20, 2024

CohenIdo commented May 20, 2024

CohenIdo commented May 22, 2024

CohenIdo commented May 22, 2024

joeypoon May 22, 2024

joeypoon May 22, 2024

CohenIdo May 23, 2024 •

edited

joeypoon May 23, 2024

CohenIdo commented May 28, 2024

eyalkraft May 28, 2024

eyalkraft May 28, 2024

eyalkraft May 28, 2024

CohenIdo May 28, 2024

eyalkraft May 28, 2024 •

edited

CohenIdo May 28, 2024

CohenIdo commented May 28, 2024

kfirpeled May 29, 2024

kibana-ci commented May 29, 2024 •

edited

	export const getCloudDefendUsageRecord = async ({
	export const getCloudDefendUsageRecords = async ({

[Cloud Security] Refactor D4C metering function #183814

[Cloud Security] Refactor D4C metering function #183814

Conversation

CohenIdo commented May 20, 2024 • edited

Summary

elasticmachine commented May 20, 2024

CohenIdo commented May 20, 2024

CohenIdo commented May 22, 2024

CohenIdo commented May 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CohenIdo May 23, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CohenIdo commented May 28, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eyalkraft May 28, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CohenIdo commented May 28, 2024

Choose a reason for hiding this comment

kibana-ci commented May 29, 2024 • edited

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Canvas Sharable Runtime

History

CohenIdo commented May 20, 2024 •

edited

CohenIdo May 23, 2024 •

edited

eyalkraft May 28, 2024 •

edited

kibana-ci commented May 29, 2024 •

edited