Adds Prometheus Remote Write #388

tiedotguy · 2021-08-30T08:22:19Z

Probably easiest to just look at 6ca4ea3, it's based off the Datadog backend, with a lot of reused stuff.

This removes tests for valid metrics from the TestInvalidMetricsLexer test. It also codifies handling of "foo:" and ":foo" for tags as valid tags. This may not be the design we want, but it's the code we have, so now we enforce it with tests.

MovieStoreGuy

Just some questions on things, I would like to enforce compile time interface checks where possible.

MovieStoreGuy · 2021-09-02T02:21:28Z

BACKENDS.md

+Prometheus Remote Writer Backend
+--------------------------------
+The `promremotewriter` backend supports sending metrics to a Prometheus Remote Writer compatible backend.  It currently
+drops events.  At present there is no authentication supported.


What should be expected with dropped events?

Prometheus doesn't do events, they need to be managed by something external (ie, grafana). So that will probably end up as a grafana-events backend which drops metrics or something. Haven't really thought about it too hard.

MovieStoreGuy · 2021-09-02T02:23:05Z

CHANGELOG.md

 29.0.2
 ------
+- New Relic backend handles infinities and NaN better


Is this change part of this change set?

No, it was an "unreleased" change in 29.0.2, this just documents when the behavior went in.

MovieStoreGuy · 2021-09-02T02:26:47Z

pb/remote_write.pb.go

@@ -0,0 +1,229 @@
+// Code generated by protoc-gen-go. DO NOT EDIT.


Is it okay to check in the auto generated code?

Yep, we do it already with the other PB stuff. It simplifies the need to deal with protoc.

MovieStoreGuy · 2021-09-02T02:27:44Z

pb/remote_write.proto

@@ -0,0 +1,29 @@
+syntax = "proto3";


Is there a reference that this proto file was built from?

Yeah, came from the prom remote write spec, I can add a ref to it.

this

Yes please, just in case we need to chase it up.

MovieStoreGuy · 2021-09-02T02:29:33Z

pkg/backends/promremotewriter/flush.go

+)
+
+// flush represents a send operation.
+type flush struct {


Does this struct implement an interface?

MovieStoreGuy · 2021-09-02T02:35:52Z

pkg/backends/promremotewriter/flush.go

+}
+
+func (f *flush) maybeFlush() {
+	if uint(len(f.writeRequest.Timeseries))+20 >= f.metricsPerBatch { // flush before it reaches max size and grows the slice


Why the magic +20?

It's inherited from datadog, but it's meant to be the maximum cardinality of a single metric. That is, a timer is going to have like 10 metrics it emits. 20 just gives it a buffer.

The intent is len(...) < metricsPerBatch, rather than len(...) > metricsPerBatch+epsilon, because if there's a limit on the backend, then it might be a hard limit, so going 1 over is a hard fail.

Could I ask you to make the number a constant with the file to make it easier in future to understand the code a bit more?

MovieStoreGuy · 2021-09-02T03:11:28Z

pkg/backends/promremotewriter/promremotewriter.go

+const (
+	// BackendName is the name of this backend.
+	BackendName                  = "promremotewriter"
+	defaultUserAgent             = "gostatsd" // TODO: Add version


Worth raising an issue for?

Nah, pretty sure it's a theme.

MovieStoreGuy · 2021-09-02T03:14:56Z

pkg/backends/promremotewriter/promremotewriter.go

+)
+
+// Client represents a Prometheus Remote Writer client.
+type Client struct {


Is there an interface that this implements?

Yup, gostatsd.Backend. It's not formally checked, but the code won't compile if it doesn't implement it (NewClientFromViper will fail its return type)

Not worth doing the compiler time interface check within the file?

We do, NewClientFromViper will fail if it doesn't match.

MovieStoreGuy · 2021-09-02T03:15:41Z

pkg/backends/promremotewriter/promremotewriter.go

+	now := clock.FromContext(ctx).Now().UnixNano() / 1_000_000
+	prw.processMetrics(now, metrics, func(writeReq *pb.PromWriteRequest) {
+		atomic.AddUint64(&prw.batchesCreated, 1)
+		go func() {


Any concern if this is called mutliple times?

I don't love it, but SendMetricsAsync needs to complete fast - If it blocks, everything blocks. MetricFlusher.flushData ensures receiving of data to continue, and enforces that we won't initiate a second flush before the first one completes.

MovieStoreGuy · 2021-09-02T03:22:32Z

pkg/backends/promremotewriter/promremotewriter.go

+	bufCompressed := snappy.Encode(bufBorrowed, raw)
+	if cap(bufBorrowed) < cap(bufCompressed) {
+		// it overflowed the buffer and had to allocate, so we'll keep the larger buffer for next time
+		*buffer = bytes.NewBuffer(bufCompressed)


is it worth looking at https://pkg.go.dev/sync#Pool for this functionality instead of doing the memory management for go?

snappy is creating/growing the byte array if needed, the best we can do is detect if it needed to grow, and re-use it. The bytes.Buffer is used in the semaphore, so we'll reach a local maxima and then stop allocating more data. There's not a large amount of byte slices allocated.

Sounds good.

tiedotguy · 2021-09-08T23:29:48Z

Once again we need to ignore coveralls, I think it's just sad because there's new code without sufficient coverage to actively increase the %.

tiedotguy · 2021-10-19T01:25:04Z

Oh hey I never got this merged. I'll clean it up later.

MovieStoreGuy · 2021-10-20T00:51:34Z

Soz

tiedotguy · 2022-01-12T22:31:19Z

See #388 (comment)

tiedotguy added 4 commits August 30, 2021 08:10

Copy datadog as initial implementation

c46447c

Initial conversion

6ca4ea3

Adjust lexer tests

0224477

This removes tests for valid metrics from the TestInvalidMetricsLexer test. It also codifies handling of "foo:" and ":foo" for tags as valid tags. This may not be the design we want, but it's the code we have, so now we enforce it with tests.

Update changelog

aac4c1f

MovieStoreGuy reviewed Sep 2, 2021

View reviewed changes

MovieStoreGuy previously approved these changes Sep 6, 2021

View reviewed changes

Clarify some stuff

5ac61c8

tiedotguy dismissed MovieStoreGuy’s stale review via 5ac61c8 September 7, 2021 21:49

tiedotguy requested a review from MovieStoreGuy September 8, 2021 23:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds Prometheus Remote Write #388

Adds Prometheus Remote Write #388

tiedotguy commented Aug 30, 2021

MovieStoreGuy left a comment

MovieStoreGuy Sep 2, 2021

tiedotguy Sep 2, 2021

MovieStoreGuy Sep 2, 2021

tiedotguy Sep 2, 2021

MovieStoreGuy Sep 2, 2021

MovieStoreGuy Sep 2, 2021

tiedotguy Sep 2, 2021

MovieStoreGuy Sep 2, 2021

tiedotguy Sep 2, 2021 •

edited

MovieStoreGuy Sep 2, 2021

MovieStoreGuy Sep 2, 2021

tiedotguy Sep 2, 2021

MovieStoreGuy Sep 2, 2021

tiedotguy Sep 2, 2021

MovieStoreGuy Sep 6, 2021

MovieStoreGuy Sep 2, 2021

tiedotguy Sep 8, 2021

MovieStoreGuy Sep 2, 2021

tiedotguy Sep 2, 2021 •

edited

MovieStoreGuy Sep 6, 2021

tiedotguy Sep 8, 2021

MovieStoreGuy Sep 2, 2021

tiedotguy Sep 2, 2021

MovieStoreGuy Sep 2, 2021

tiedotguy Sep 2, 2021

MovieStoreGuy Sep 6, 2021

tiedotguy commented Sep 8, 2021

tiedotguy commented Oct 19, 2021

MovieStoreGuy commented Oct 20, 2021

tiedotguy commented Jan 12, 2022

		@@ -0,0 +1,229 @@
		// Code generated by protoc-gen-go. DO NOT EDIT.

Adds Prometheus Remote Write #388

Are you sure you want to change the base?

Adds Prometheus Remote Write #388

Conversation

tiedotguy commented Aug 30, 2021

MovieStoreGuy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tiedotguy Sep 2, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tiedotguy Sep 2, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tiedotguy commented Sep 8, 2021

tiedotguy commented Oct 19, 2021

MovieStoreGuy commented Oct 20, 2021

tiedotguy commented Jan 12, 2022

tiedotguy Sep 2, 2021 •

edited

tiedotguy Sep 2, 2021 •

edited