Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failed to scrape resource for metric -> System.NullReferenceException #2263

Open
andreibogdan opened this issue Mar 17, 2023 · 0 comments
Open

Failed to scrape resource for metric -> System.NullReferenceException #2263

andreibogdan opened this issue Mar 17, 2023 · 0 comments
Assignees
@tomkerkhove
Labels
bug Something isn't working

Comments

@andreibogdan
Copy link

andreibogdan commented Mar 17, 2023 •
edited

Report

Hi,
I am using promitor-agent-scraper:2.8 with the following config metric declaration config:

podLabels:
  name: promitor-agent-scraper
azureMetadata:
  tenantId: XXXX
  subscriptionId: XXXXX
  resourceGroupName: XXX
metricDefaults:
  aggregation:
    interval: 00:05:00
  scraping:
    schedule: "55 * * ? * *"
metrics:
  - name: azure_service_bus_queue_successful_requests
    description: "The number of successful requests"
    resourceType: ServiceBusNamespace
    azureMetricConfiguration:
      metricName: SuccessfulRequests
      aggregation:
        type: Total
    resources:
      - namespace: XXXXX
        queueName: xxxx-queue
        resourceGroupName: xxxxxx

When running using docker the metric is scraped but when running in a K8s cluster with Promitor scraper deployed via helm(latest version) I got the following error.
On K8s namespace, I deployed only the scrapper.

Expected Behavior

Metrics to show up

Actual Behavior

When running using docker the metric is scraped but when running in a K8s cluster with Promitor scraper deployed via helm(latest version) I got the following error.
On K8s namespace, I deployed only the scrapper.

Steps to Reproduce the Problem

  1. deploy on azure k8s using helm

...

Component

Scraper

Version

2.8.0

Configuration

Configuration:

podLabels:
  name: promitor-agent-scraper
azureMetadata:
  tenantId: xxxxx
  subscriptionId: xxxxx
  resourceGroupName: xxxxxx
metricDefaults:
  aggregation:
    interval: 00:05:00
  scraping:
    schedule: "55 * * ? * *"
metrics:
  - name: azure_service_bus_queue_successful_requests
    description: "The number of successful requests"
    resourceType: ServiceBusNamespace
    azureMetricConfiguration:
      metricName: SuccessfulRequests
      aggregation:
        type: Total
    resources:
      - namespace: xxx-stage
        queueName: xxx-queue

Logs

[15:37:52 INF] Booting up Promitor v2.8.0 running .NET .NET 6.0.9 - Thank you for using Promitor!
[15:37:52 INF] Running .NET 6.0.9 on Linux (alpine.3.16-x64 | Linux 5.15.43-flatcar #1 SMP Tue May 31 00:28:53 -00 2022).
[15:37:52 INF] Using configuration folder '/config/'
[15:37:53 INF] The following metric sinks were configured and are being enabled:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┓
┃ Metric Sink                  │ Details        ┃
┠──────────────────────────────┼────────────────┨
┃ Prometheus Scraping Endpoint │ Url: /metrics. ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┛
             Configured Metric Sinks
[15:37:53 INF] Starting validation of Promitor setup
[15:37:53 INF] Scrape configuration found at '/config/metrics-declaration.yaml'
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name                              │ Outcome │ Details                        ┃
┠───────────────────────────────────┼─────────┼────────────────────────────────┨
┃ Metrics Declaration Path          │ Success │ Everything is well-configured. ┃
┃ Azure Authentication              │ Success │ Everything is well-configured. ┃
┃ Metrics Declaration               │ Success │ Everything is well-configured. ┃
┃ Resource Discovery                │ Success │ Everything is well-configured. ┃
┃ Atlassian Statuspage Metric Sink  │ Success │ Everything is well-configured. ┃
┃ OpenTelemetry Collector Metric    │ Success │ Everything is well-configured. ┃
┃ Sink                              │         │                                ┃
┃ Prometheus Scraping Endpoint      │ Success │ Everything is well-configured. ┃
┃ Metric Sink                       │         │                                ┃
┃ StatsD Metric Sink                │ Success │ Everything is well-configured. ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                                   Validation
[15:37:53 INF] Promitor configuration is valid, we are good to go.
[15:37:53 INF] Here's an overview of what was configured:
╭───────────────────┬───────────────────┬───────────────────┬──────────────────╮
│ Tenant Id         │ Subscription Id   │ Resource Group    │ Azure Cloud      │
│                   │                   │ Name (Default)    │                  │
├───────────────────┼───────────────────┼───────────────────┼──────────────────┤
│ fxxx │ xxxxx │ xxxxx │ Azure Global     │
│ xxxx │ xxxxx │ e                 │ Cloud            │
│ e1                │ 87                │                   │                  │
╰───────────────────┴───────────────────┴───────────────────┴──────────────────╯
                                 Azure Metadata
╭─────────────┬─────────────┬────────┬─────────────┬─────────────┬─────────────╮
│ Metric Name │ Resource    │ Labels │ Azure       │ Resources   │ Resource    │
│             │ Type        │        │ Monitor     │ To Scrape   │ Discovery   │
│             │             │        │ Metric      │             │ Groups To   │
│             │             │        │             │             │ Scrape      │
├─────────────┼─────────────┼────────┼─────────────┼─────────────┼─────────────┤
│ azure_servi │ Service Bus │ None   │ SuccessfulR │ xxxx │ None        │
│ ce_bus_queu │ Namespace   │        │ equests     │ xxx     │             │
│ e_successfu │             │        │             │             │             │
│ l_requests  │             │        │             │             │             │
╰─────────────┴─────────────┴────────┴─────────────┴─────────────┴─────────────╯
                               Configured Metrics
[15:37:53 INF] [Job][Scraper-1Metrics-At_55_seconds_past_the_minute-0d2019a1316a4b87a5349e97d3d74bc2] is running under time zone: UTC
[15:37:53 INF] Adding Prometheus sink to expose on /metrics
[15:37:53 INF] Adding Prometheus sink to expose on /metrics
[15:37:53 INF] [Job][Scraper-1Metrics-At_55_seconds_past_the_minute-0d2019a1316a4b87a5349e97d3d74bc2] is running under time zone: UTC
[15:37:53 INF] Scraping azure_service_bus_queue_successful_requests for resource type ServiceBusNamespace.
[15:37:54 INF] Now listening on: http://[::]:8081
[15:37:54 INF] Application started. Press Ctrl+C to shut down.
[15:37:54 INF] Hosting environment: Production
[15:37:54 INF] Content root path: /app
[15:37:54 INF] [Job][Scraper-1Metrics-At_55_seconds_past_the_minute-0d2019a1316a4b87a5349e97d3d74bc2] is running under time zone: UTC
[15:37:54 INF] Scraping azure_service_bus_queue_successful_requests for resource type ServiceBusNamespace.
[15:37:55 INF] Scraping azure_service_bus_queue_successful_requests for resource type ServiceBusNamespace.
[15:38:00 INF] Request starting HTTP/1.1 GET http://172.18.191.168:8081/api/v1/health?includeDependencies=false - -
[15:38:01 INF] Route matched with {action = "Get", controller = "Health"}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] Get(Boolean) on controller Promitor.Agents.Scraper.Controllers.v1.HealthController (Promitor.Agents.Scraper).
[15:38:55 FTL] Failed to scrape resource for metric 'azure_service_bus_queue_successful_requests'
System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Http.AdalHttpClient.GetResponseAsync[T](Boolean respondToDeviceAuthChallenge)
   at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Http.AdalHttpClient.GetResponseAsync[T](Boolean respondToDeviceAuthChallenge)
   at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Http.AdalHttpClient.GetResponseAsync[T]()
   at Microsoft.IdentityModel.Clients.ActiveDirectory.InstanceDiscovery.DiscoverAsync(Uri authority, Boolean validateAuthority, RequestContext requestContext)
   at Microsoft.IdentityModel.Clients.ActiveDirectory.InstanceDiscovery.GetMetadataEntryAsync(Uri authority, Boolean validateAuthority, RequestContext requestContext)
   at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Instance.Authenticator.UpdateFromTemplateAsync(RequestContext requestContext)
   at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Flows.AcquireTokenHandlerBase.PreRunAsync()
   at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Flows.AcquireTokenHandlerBase.RunAsync()
   at Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext.AcquireTokenForClientCommonAsync(String resource, ClientKey clientKey)
   at Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext.AcquireTokenAsync(String resource, ClientCredential clientCredential)
   at Microsoft.Rest.Azure.Authentication.Internal.MemoryApplicationAuthenticationProvider.AuthenticateAsync(String clientId, String audience, AuthenticationContext context)
   at Microsoft.Rest.Azure.Authentication.ApplicationTokenProvider.LoginSilentAsync(String domain, String clientId, IApplicationAuthenticationProvider authenticationProvider, ActiveDirectoryServiceSettings settings, TokenCache cache)
   at Microsoft.Rest.Azure.Authentication.ApplicationTokenProvider.LoginSilentAsync(String domain, ClientCredential credential, ActiveDirectoryServiceSettings settings, TokenCache cache)
   at Microsoft.Rest.Azure.Authentication.ApplicationTokenProvider.LoginSilentAsync(String domain, String clientId, String secret, ActiveDirectoryServiceSettings settings, TokenCache cache)
   at Microsoft.Azure.Management.ResourceManager.Fluent.Authentication.AzureCredentials.ProcessHttpRequestAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.Monitor.Fluent.MetricDefinitionsOperations.ListWithHttpMessagesAsync(String resourceUri, String metricnamespace, Dictionary`2 customHeaders, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.Monitor.Fluent.MetricDefinitionsOperationsExtensions.ListAsync(IMetricDefinitionsOperations operations, String resourceUri, String metricnamespace, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.Monitor.Fluent.MetricDefinitionsImpl.ListByResourceAsync(String resourceId, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.Monitor.Fluent.MetricDefinitionsImpl.Microsoft.Azure.Management.Monitor.Fluent.IMetricDefinitions.ListByResourceAsync(String resourceId, CancellationToken cancellationToken)
   at Promitor.Integrations.AzureMonitor.AzureMonitorClient.GetMetricDefinitionsAsync(String resourceId) in /src/Promitor.Integrations.AzureMonitor/AzureMonitorClient.cs:line 128
   at Promitor.Integrations.AzureMonitor.AzureMonitorClient.QueryMetricAsync(String metricName, String metricDimension, AggregationType aggregationType, TimeSpan aggregationInterval, String resourceId, String metricFilter, Nullable`1 metricLimit) in /src/Promitor.Integrations.AzureMonitor/AzureMonitorClient.cs:line 84
   at Promitor.Core.Scraping.AzureMonitorScraper`1.ScrapeResourceAsync(String subscriptionId, ScrapeDefinition`1 scrapeDefinition, TResourceDefinition resourceDefinition, AggregationType aggregationType, TimeSpan aggregationInterval) in /src/Promitor.Core.Scraping/AzureMonitorScraper.cs:line 54
   at Promitor.Core.Scraping.Scraper`1.ScrapeAsync(ScrapeDefinition`1 scrapeDefinition) in /src/Promitor.Core.Scraping/Scraper.cs:line 78

Platform

Microsoft Azure

Contact Details

andrei.bogdan.mail@gmail.com
@andreibogdan andreibogdan added the bug Something isn't working label Mar 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tomkerkhove tomkerkhove

Labels
bug Something isn't working
Projects
Promitor Roadmap
Status: Proposed
Milestone
No milestone
Development

No branches or pull requests
2 participants
@andreibogdan @tomkerkhove