sql: v23.1.13: TxnFingerprintIDCache: interface {} is nil, not appstatspb.TransactionFingerprintID

[cockroach-sentry]

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/5378038283/?referrer=webhooks_plugin

Panic Message:

panic.go:884: interface conversion: interface {} is nil, not appstatspb.TransactionFingerprintID
(1) attached stack trace
  -- stack trace:
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:884
  | runtime.panicdottypeE
  | 	GOROOT/src/runtime/iface.go:262
  | github.com/cockroachdb/cockroach/pkg/sql.(*TxnFingerprintIDCache).GetAllTxnFingerprintIDs.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/txn_fingerprint_id_cache.go:108
  | github.com/cockroachdb/cockroach/pkg/util/cache.(*baseCache).Do
  | 	github.com/cockroachdb/cockroach/pkg/util/cache/cache.go:308
  | github.com/cockroachdb/cockroach/pkg/sql.(*TxnFingerprintIDCache).GetAllTxnFingerprintIDs
  | 	github.com/cockroachdb/cockroach/pkg/sql/txn_fingerprint_id_cache.go:107
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).serialize
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:4012
  | github.com/cockroachdb/cockroach/pkg/sql.(*SessionRegistry).SerializeAll
  | 	github.com/cockroachdb/cockroach/pkg/sql/exec_util.go:2256
  | github.com/cockroachdb/cockroach/pkg/server.(*baseStatusServer).getLocalSessions
  | 	github.com/cockroachdb/cockroach/pkg/server/status.go:210
  | github.com/cockroachdb/cockroach/pkg/server.(*statusServer).ListLocalSessions
  | 	github.com/cockroachdb/cockroach/pkg/server/status.go:2835
  | github.com/cockroachdb/cockroach/pkg/server/serverpb._Status_ListLocalSessions_Handler.func1
  | 	github.com/cockroachdb/cockroach/pkg/server/serverpb/bazel-out/k8-opt/bin/pkg/server/serverpb/serverpb_go_proto_/github.com/cockroachdb/cockroach/pkg/server/serverpb/status.pb.go:8781
  | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1
  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:116
  | google.golang.org/grpc.getChainUnaryHandler.func1
  | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:284
  | google.golang.org/grpc.getChainUnaryHandler.func1
  | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163
  | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105
  | google.golang.org/grpc.getChainUnaryHandler.func1
  | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:251
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:322
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:249
  | google.golang.org/grpc.chainUnaryInterceptors.func1
  | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1154
  | github.com/cockroachdb/cockroach/pkg/server/serverpb._Status_ListLocalSessions_Handler
  | 	github.com/cockroachdb/cockroach/pkg/server/serverpb/bazel-out/k8-opt/bin/pkg/server/serverpb/serverpb_go_proto_/github.com/cockroachdb/cockroach/pkg/server/serverpb/status.pb.go:8783
  | google.golang.org/grpc.(*Server).processUnaryRPC
  | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1336
  | google.golang.org/grpc.(*Server).handleStream
  | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1704
  | google.golang.org/grpc.(*Server).serveStreams.func1.2
  | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:965
  | runtime.goexit
  | 	GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2) interface conversion: interface {} is nil, not appstatspb.TransactionFingerprintID
Error types: (1) *withstack.withStack (2) *runtime.TypeAssertionError
-- report composition:
*runtime.TypeAssertionError
panic.go:884: *withstack.withStack (top exception)

Stacktrace (expand for inline code snippets):

GOROOT/src/runtime/asm_amd64.s#L1593-L1595
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L964-L966
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1703-L1705
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1335-L1337
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/server/serverpb/bazel-out/k8-opt/bin/pkg/server/serverpb/serverpb_go_proto_/github.com/cockroachdb/cockroach/pkg/server/serverpb/status.pb.go#L8782-L8784
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1153-L1155
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L248-L250

cockroach/pkg/util/stop/stopper.go

Lines 321 to 323 in 8d065df

	return f(ctx)
	}

https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L250-L252
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1162-L1164
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/auth.go#L104-L106
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1162-L1164
https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/rpc/pkg/rpc/context.go#L283-L285
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1162-L1164

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 115 to 117 in 8d065df

	resp, err := handler(ctx, req)
	if err != nil {

https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/server/serverpb/bazel-out/k8-opt/bin/pkg/server/serverpb/serverpb_go_proto_/github.com/cockroachdb/cockroach/pkg/server/serverpb/status.pb.go#L8780-L8782

cockroach/pkg/server/status.go

Lines 2834 to 2836 in 8d065df

	) (*serverpb.ListSessionsResponse, error) {
	sessions, err := s.getLocalSessions(ctx, req)
	if err != nil {

cockroach/pkg/server/status.go

Lines 209 to 211 in 8d065df

	sessions := b.sessionRegistry.SerializeAll()
	var closedSessions []serverpb.Session

cockroach/pkg/sql/exec_util.go

Lines 2255 to 2257 in 8d065df

	for _, s := range sessions {
	response = append(response, s.serialize())
	}

cockroach/pkg/sql/conn_executor.go

Lines 4011 to 4013 in 8d065df

	txnFingerprintIDs := ex.txnFingerprintIDCache.GetAllTxnFingerprintIDs()
	sessionActiveTime := ex.totalActiveTimeStopWatch.Elapsed()

cockroach/pkg/sql/txn_fingerprint_id_cache.go

Lines 106 to 108 in 8d065df

	b.mu.cache.Do(func(entry *cache.Entry) {
	id := entry.Value.(appstatspb.TransactionFingerprintID)

cockroach/pkg/util/cache/cache.go

Lines 307 to 309 in 8d065df

	for e := bc.ll.root.next; e != &bc.ll.root; e = e.next {
	fn(e)
	}

cockroach/pkg/sql/txn_fingerprint_id_cache.go

Lines 107 to 109 in 8d065df

	b.mu.cache.Do(func(entry *cache.Entry) {
	id := entry.Value.(appstatspb.TransactionFingerprintID)

GOROOT/src/runtime/iface.go#L261-L263
GOROOT/src/runtime/panic.go#L883-L885

GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).serveStreams.func1.2 at line 965
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).handleStream at line 1704
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).processUnaryRPC at line 1336
pkg/server/serverpb/bazel-out/k8-opt/bin/pkg/server/serverpb/serverpb_go_proto_/github.com/cockroachdb/cockroach/pkg/server/serverpb/status.pb.go in pkg/server/serverpb._Status_ListLocalSessions_Handler at line 8783
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.chainUnaryInterceptors.func1 at line 1154
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 249
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 322
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 251
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainUnaryHandler.func1 at line 1163
pkg/rpc/pkg/rpc/auth.go in pkg/rpc.kvAuth.unaryInterceptor at line 105
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainUnaryHandler.func1 at line 1163
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 284
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainUnaryHandler.func1 at line 1163
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1 at line 116
pkg/server/serverpb/bazel-out/k8-opt/bin/pkg/server/serverpb/serverpb_go_proto_/github.com/cockroachdb/cockroach/pkg/server/serverpb/status.pb.go in pkg/server/serverpb._Status_ListLocalSessions_Handler.func1 at line 8781
pkg/server/status.go in pkg/server.(*statusServer).ListLocalSessions at line 2835
pkg/server/status.go in pkg/server.(*baseStatusServer).getLocalSessions at line 210
pkg/sql/exec_util.go in pkg/sql.(*SessionRegistry).SerializeAll at line 2256
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).serialize at line 4012
pkg/sql/txn_fingerprint_id_cache.go in pkg/sql.(*TxnFingerprintIDCache).GetAllTxnFingerprintIDs at line 107
pkg/util/cache/cache.go in pkg/util/cache.(*baseCache).Do at line 308
pkg/sql/txn_fingerprint_id_cache.go in pkg/sql.(*TxnFingerprintIDCache).GetAllTxnFingerprintIDs.func1 at line 108
GOROOT/src/runtime/iface.go in runtime.panicdottypeE at line 262
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884

Tags

Tag	Value
Command	start-single-node
Environment	v23.1.13
Go Version	go1.19.13
Platform	linux amd64
Distribution	CCL
Cockroach Release	v23.1.13
Cockroach SHA	8d065df
# of CPUs	6
# of Goroutines	390

Jira issue: CRDB-38875

[yuzefovich]

Hm, I'm not sure what to make of it - I don't see how entry.Value can be nil. The code was adjusted slightly in 88ebd70 which shouldn't affect this though.