Skip to content

Httptrace #1087

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

Httptrace #1087

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
e8d8f88
added httpTrace logging for debugging network issues
yesoreyeram Dec 9, 2024
17c78ae
added error logging
yesoreyeram Dec 9, 2024
9e9d208
Bump supported grafana version to 10.4.8 (#1105)
ivanahuckova Jan 10, 2025
41b8579
Merge remote-tracking branch 'origin' into dev-3.0.0
yesoreyeram Jan 16, 2025
00dea29
Merge branch 'dev-3.0.0' into httptrace
yesoreyeram Jan 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/sixty-lemons-move.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'grafana-infinity-datasource': minor
---

Added HTTPTrace logging for debugging connection issues
5 changes: 5 additions & 0 deletions .changeset/three-clouds-yawn.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'grafana-infinity-datasource': major
---

Plugin now requires Grafana 10.4.8 or newer
5 changes: 0 additions & 5 deletions docker/blocks/tempo/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,11 +39,6 @@ metrics_generator:
storage:
trace:
backend: local # backend configuration to use
block:
bloom_filter_false_positive: .05 # bloom filter false positive rate. lower values create larger filters but fewer false positives
v2_index_downsample_bytes: 1000 # number of bytes per index record
v2_encoding: zstd # block encoding/compression. options: none, gzip, lz4-64k, lz4-256k, lz4-1M, lz4, snappy, zstd, s2
version: vParquet
wal:
path: /tmp/tempo/wal # where to store the the wal locally
v2_encoding: snappy # wal encoding/compression. options: none, gzip, lz4-64k, lz4-256k, lz4-1M, lz4, snappy, zstd, s2
Expand Down
8 changes: 4 additions & 4 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,10 +55,10 @@
},
"dependencies": {
"@emotion/css": "11.10.6",
"@grafana/data": "10.3.3",
"@grafana/runtime": "10.3.3",
"@grafana/schema": "10.3.3",
"@grafana/ui": "10.3.3",
"@grafana/data": "10.4.8",
"@grafana/runtime": "10.4.8",
"@grafana/schema": "10.4.8",
"@grafana/ui": "10.4.8",
"cheerio": "^1.0.0-rc.10",
"csv-parse": "^4.12.0",
"groq-js": "1.1.8",
Expand Down
53 changes: 53 additions & 0 deletions pkg/infinity/httptrace.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
package infinity

import (
"context"
"crypto/tls"
"net/http"
"net/http/httptrace"

"github.com/grafana/grafana-plugin-sdk-go/backend"
)

func ApplyHTTPTraceToRequest(_ context.Context, req *http.Request) *http.Request {
ctx := req.Context()
logger := backend.Logger.FromContext(ctx)
trace := &httptrace.ClientTrace{
GetConn: func(hostPort string) {
logger.Debug("HTTPTrace event", "event_name", "GetConn")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am wondering if all of these should be level trace because for data source services, we have debug level logging turned on. This would produce a lot of logs that are only sometimes needed. So we could turn it on in that case.

We have trace log level: https://github.com/grafana/grafana-plugin-sdk-go/blob/main/backend/log/log.go#L14, but we are missing logger.Trace() method - all other levels have it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm with Ivana here. Why not just even be a trace instead of log?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea.. Happy to change to Trace ( but we need to make sure the changes done in SDK first )

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, Any thoughts on making it to debug but only calling this method on healthcheck rather query. Won't that solve the problems we disucussed here. :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking about a proper trace so these would be spans instead of logs. Like in here https://github.com/AdamKorcz/skipper/blob/d0b249a4aa8993a050b87cecc11cc6ca839195a4/proxy/proxy.go#L1464

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea.. that too possible too.

I am thinking of deferring this to 3.0.0-beta.2

},
DNSStart: func(di httptrace.DNSStartInfo) {
logger.Debug("HTTPTrace event", "event_name", "DNSStart")
},
DNSDone: func(dnsInfo httptrace.DNSDoneInfo) {
logger.Debug("HTTPTrace event", "event_name", "DNSDone")
},
ConnectStart: func(network, addr string) {
logger.Debug("HTTPTrace event", "event_name", "ConnectStart")
},
ConnectDone: func(network, addr string, err error) {
if err != nil {
logger.Error("HTTPTrace event", "event_name", "ConnectDone", "error", err.Error())
return
}
logger.Debug("HTTPTrace event", "event_name", "ConnectDone", "address", addr)
},
TLSHandshakeStart: func() {
logger.Debug("HTTPTrace event", "event_name", "TLSHandshakeStart")
},
TLSHandshakeDone: func(cs tls.ConnectionState, err error) {
if err != nil {
logger.Error("HTTPTrace event", "event_name", "TLSHandshakeDone", "error", err.Error())
return
}
logger.Debug("HTTPTrace event", "event_name", "TLSHandshakeDone")
},
GotConn: func(connInfo httptrace.GotConnInfo) {
logger.Debug("HTTPTrace event", "event_name", "GotConn", "remote_address", connInfo.Conn.RemoteAddr().String())
},
GotFirstResponseByte: func() {
logger.Debug("HTTPTrace event", "event_name", "GotFirstResponseByte")
},
}
return req.WithContext(httptrace.WithClientTrace(ctx, trace))
}
1 change: 1 addition & 0 deletions pkg/infinity/request.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ func GetRequest(ctx context.Context, settings models.InfinitySettings, body io.R
req = ApplyBearerToken(ctx, settings, req, includeSect)
req = ApplyApiKeyAuth(ctx, settings, req, includeSect)
req = ApplyForwardedOAuthIdentity(ctx, requestHeaders, settings, req, includeSect)
req = ApplyHTTPTraceToRequest(ctx, req)
req = ApplyTraceHead(ctx, req)
return req, err
}
Expand Down
4 changes: 2 additions & 2 deletions src/plugin.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,8 +92,8 @@
]
},
"dependencies": {
"grafanaDependency": ">=9.5.15",
"grafanaVersion": "9.5.x",
"grafanaDependency": ">=10.4.8",
"grafanaVersion": "10.4.x",
"plugins": []
},
"includes": [],
Expand Down
4,775 changes: 2,255 additions & 2,520 deletions yarn.lock
View file
Open in desktop

Large diffs are not rendered by default.