feat(Kafka): New Kafka AWS checks #4021

puchy22 · 2024-05-16T15:34:32Z

Context

New checks to cover basic security best practices for Apache Kafka service managed by AWS.

kafka_cluster_enhanced_monitoring_enabled
kafka_cluster_in_transit_encryption_enabled
kafka_cluster_is_public
kafka_cluster_mutual_tls_authentication_enabled
kafka_cluster_unrestricted_access_disabled
kafka_cluster_encryption_at_rest_uses_cmk

Description

Added new checks with metadata and respective unit testing

License

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…rest

codecov · 2024-05-20T10:44:40Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 86.33%. Comparing base (73b3484) to head (9eb9a23).
Report is 25 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4021      +/-   ##
==========================================
- Coverage   86.51%   86.33%   -0.18%     
==========================================
  Files         776      789      +13     
  Lines       24163    24707     +544     
==========================================
+ Hits        20904    21332     +428     
- Misses       3259     3375     +116

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…ecks

sergargar · 2024-05-20T12:49:09Z

...kafka/kafka_cluster_encryption_at_rest_uses_cmk/kafka_cluster_encryption_at_rest_uses_cmk.py

+            report.resource_arn = arn_cluster
+            report.resource_tags = cluster.tags
+            report.status = "FAIL"
+            report.status_extended = f"Kafka cluster '{cluster.name}' does not have encryption at rest enabled with a CMK."


Would it be interesting to put a FAIL if it is encrypted with another type of key?

There is only two types of managers or AWS or CUSTOMER, if the manager is AWS it FAIL and if is CUSTOMER like is reflected on check it will PASS. https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/kms/client/describe_key.html

sergargar · 2024-05-20T12:49:55Z

...kafka/kafka_cluster_enhanced_monitoring_enabled/kafka_cluster_enhanced_monitoring_enabled.py

+                f"Kafka cluster '{cluster.name}' has enhanced monitoring enabled."
+            )
+
+            if cluster.enhanced_monitoring == "DEFAULT":


Which are the possible values here?

'DEFAULT'|'PER_BROKER'|'PER_TOPIC_PER_BROKER'|'PER_TOPIC_PER_PARTITION'

I only put DEFAULT as FAIL because every other method is an enhance over default monitoring.

sergargar · 2024-05-20T12:50:48Z

...ster_in_transit_encryption_enabled/kafka_cluster_in_transit_encryption_enabled.metadata.json

+  "RelatedUrl": "https://docs.aws.amazon.com/msk/latest/developerguide/msk-encryption.html",
+  "Remediation": {
+    "Code": {
+      "CLI": "",


Suggested change

"CLI": "",

"CLI": "https://docs.prowler.com/checks/aws/general-policies/bc_aws_general_32/#cli",

sergargar · 2024-05-20T12:51:41Z

...a/kafka_cluster_in_transit_encryption_enabled/kafka_cluster_in_transit_encryption_enabled.py

+            report.resource_arn = arn_cluster
+            report.resource_tags = cluster.tags
+            report.status = "FAIL"
+            report.status_extended = f"Kafka cluster '{cluster.name}' does not have encryption in transit enabled"


Please add a final dot in all the statuses extended.

sergargar · 2024-05-20T12:52:19Z

prowler/providers/aws/services/kafka/kafka_cluster_is_public/kafka_cluster_is_public.py

+            report.resource_tags = cluster.tags
+            report.status = "FAIL"
+            report.status_extended = (
+                f"Kafka cluster '{cluster.name}' does not have public access disabled."


Suggested change

f"Kafka cluster '{cluster.name}' does not have public access disabled."

f"Kafka cluster '{cluster.name}' is publicly accessible."

sergargar · 2024-05-20T12:53:14Z

...luster_unrestricted_access_disabled/kafka_cluster_unrestricted_access_disabled.metadata.json

+{
+  "Provider": "aws",
+  "CheckID": "kafka_cluster_unrestricted_access_disabled",
+  "CheckTitle": "Ensure Kafka Cluster is not publicly accessible",


Is not this confusing with the check kafka_cluster_is_public?

sergargar

Good job @puchy22 !!

github-actions bot added the provider/aws Issues/PRs related with the AWS provider label May 16, 2024

puchy22 marked this pull request as ready for review May 20, 2024 09:12

puchy22 requested review from a team as code owners May 20, 2024 09:12

puchy22 added 14 commits May 20, 2024 12:00

feat(kafka): New check for encryption at rest use CMK

22f13bf

feat(kafka): New check for better monitoring

c72029d

feat(kafka): New check for encrypt in transit with TLS

3dbb6af

feat(kafka): New check to ensure a cluster is not public

f21e0b6

feat(kafka): New check for TLS auth

cd5283f

feat(kafka): New check for unauthorized access

3d19acb

chore(kafka): Rename check

f2879b4

test(kafka): Change name test

350c369

test(kafka): New test for all types of encryption in transit

f8e6e4c

test(kafka): New tests for public and not public cluster

d5d8343

test(kafka): New tests for mutual TLS auth

60731c0

test(kafka): New tests for unauth access

3fa2ec4

feat(kafka): New check and testing to ensure using CMK encryption at …

e71adc0

…rest

feat(kafka): Update the number of checks and services

f826e15

github-actions bot added documentation provider/azure Issues/PRs related with the Azure provider provider/gcp Issues/PRs related with the Google Cloud Platform provider provider/kubernetes Issues/PRs related with the Kubernetes provider labels May 20, 2024

Merge branch 'master' of github.com:puchy22/prowler into kafka_aws_ch…

f13aac0

…ecks

puchy22 force-pushed the kafka_aws_checks branch from bcccded to f13aac0 Compare May 20, 2024 10:52

puchy22 added 2 commits May 20, 2024 12:55

test(kafka): Change mock global provider

904758b

test(kafka): Fix import mocking

3827631

jfagoagas removed documentation provider/aws Issues/PRs related with the AWS provider provider/azure Issues/PRs related with the Azure provider labels May 20, 2024

jfagoagas removed provider/gcp Issues/PRs related with the Google Cloud Platform provider provider/kubernetes Issues/PRs related with the Kubernetes provider labels May 20, 2024

github-actions bot added the provider/aws Issues/PRs related with the AWS provider label May 20, 2024

sergargar reviewed May 20, 2024

View reviewed changes

sergargar self-assigned this May 20, 2024

puchy22 and others added 7 commits May 21, 2024 09:27

test(kafka): Fix clients mocking

ac420b1

chore(kafka): Add CLI doc

5265c08

chore(kafka): Add point at end of status_extended

10b9ebf

chore(kafka): Make more readable status extended

e9060fc

chore(kafka): Update metadata to not confuse with publicly access

1cf7482

chore(kafka): Change status extended from tests

4cd718f

fix typo in kafka service

5b90015

sergargar previously approved these changes May 21, 2024

View reviewed changes

test(kafka): Fix text import

9eb9a23

puchy22 dismissed sergargar’s stale review via 9eb9a23 May 22, 2024 10:27

sergargar self-requested a review May 22, 2024 12:40

sergargar approved these changes May 22, 2024

View reviewed changes

sergargar merged commit 0e71756 into prowler-cloud:master May 22, 2024
10 of 11 checks passed

puchy22 deleted the kafka_aws_checks branch May 31, 2024 09:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(Kafka): New Kafka AWS checks #4021

feat(Kafka): New Kafka AWS checks #4021

puchy22 commented May 16, 2024 •

edited

codecov bot commented May 20, 2024 •

edited

sergargar May 20, 2024

puchy22 May 21, 2024

sergargar May 20, 2024

puchy22 May 21, 2024

sergargar May 20, 2024

sergargar May 20, 2024

sergargar May 20, 2024

sergargar May 20, 2024

sergargar left a comment

	"CLI": "",
	"CLI": "https://docs.prowler.com/checks/aws/general-policies/bc_aws_general_32/#cli",

	f"Kafka cluster '{cluster.name}' does not have public access disabled."
	f"Kafka cluster '{cluster.name}' is publicly accessible."

feat(Kafka): New Kafka AWS checks #4021

feat(Kafka): New Kafka AWS checks #4021

Conversation

puchy22 commented May 16, 2024 • edited

Context

Description

License

codecov bot commented May 20, 2024 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sergargar left a comment

Choose a reason for hiding this comment

puchy22 commented May 16, 2024 •

edited

codecov bot commented May 20, 2024 •

edited