Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Network Viewer integrated with eBPF #17073

Open
wants to merge 103 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
103 commits
Select commit Hold shift + click to select a range
ba4ffae
nv_ebpf: Move content from plugin to libnetdata; Update enum to match…
thiagoftsm Feb 26, 2024
01993c5
nv_ebpf: Remove overload on host, disabling socket
thiagoftsm Feb 27, 2024
0d0d9b1
nv_ebpf: Update documentation
thiagoftsm Feb 27, 2024
353b707
nv_ebpf: Add code to load eBPF
thiagoftsm Feb 27, 2024
f5c6440
nv_ebpf: Fix compilation on some distributions
thiagoftsm Feb 27, 2024
eb0920c
nv_ebpf: Move endif
thiagoftsm Feb 27, 2024
ce4e586
nv_ebpf: Add data types to work with eBPF hash tables
thiagoftsm Feb 28, 2024
f16b0dc
nv_ebpf: Add a selector between eBPF and other models
thiagoftsm Feb 28, 2024
7c98060
nv_ebpf: Add enum to do code more readable
thiagoftsm Feb 28, 2024
0d168b1
nv_ebpf: Add missing call
thiagoftsm Feb 28, 2024
4123cd0
nv_ebpf: Move variable from libnetdata to plugin
thiagoftsm Feb 29, 2024
35d2353
nv_ebpf: Add function to read data from eBPF table
thiagoftsm Feb 29, 2024
926d619
nv_ebpf: Move content before to store
thiagoftsm Feb 29, 2024
3cd274a
nv_ebpf: Add restriction to display data
thiagoftsm Feb 29, 2024
790a0a9
prepare_functions: Improve description
thiagoftsm Feb 29, 2024
fcabf8b
prepare_functions: Move union from files.
thiagoftsm Mar 1, 2024
3ba8956
prepare_functions: Modify data type avoiding compiler issues
thiagoftsm Mar 1, 2024
9d3f195
prepare_functions: Store sockets
thiagoftsm Mar 1, 2024
67ea9ff
nv_ebpf: Add missing namespace informtion
thiagoftsm Mar 3, 2024
bceaa35
nv_ebpf: Modify algorithm to match new data type
thiagoftsm Mar 4, 2024
a24d721
nv_ebpf: Add algorithm to also work with tcp_close
thiagoftsm Mar 5, 2024
0030d3b
nv_ebpf: Clean up table when socket is closed; Align socket
thiagoftsm Mar 6, 2024
8bb9758
nv_ebpf: Move function call outside local_sockets_read_sockets_from_proc
thiagoftsm Mar 6, 2024
aaecd15
nv_ebpf: Update hash table.
thiagoftsm Mar 7, 2024
0931334
nv_ebpf: Change internal logic to get namespace.
thiagoftsm Mar 7, 2024
f94423d
nv_ebpf: As discussed internally, let us set as optional, avoiding is…
thiagoftsm Mar 8, 2024
379c073
nv_ebpf: Use optional variable to define when it is necessary to load…
thiagoftsm Mar 8, 2024
2a39187
nv_ebpf: Store data inside hashtable, because eBPF only collects acti…
thiagoftsm Mar 9, 2024
bcd766f
nv_ebpf: Change flag to read data from kernel.
thiagoftsm Mar 9, 2024
9f5c32a
nv_ebpf: Adjust flag
thiagoftsm Mar 10, 2024
bb85096
nv_ebpf: Fix issue to read sockets
thiagoftsm Mar 10, 2024
946be1b
nv_ebpf: Fix chart visualization
thiagoftsm Mar 10, 2024
8853d8a
nv_ebpf: Change data type and append spinlock to improve cleanup.
thiagoftsm Mar 10, 2024
9cf8622
nv_ebpf: Expand period of time used to update hash tables.
thiagoftsm Mar 10, 2024
e8b6935
nv_ebpf: Adjust flags and prepare to add table cleanup when function …
thiagoftsm Mar 10, 2024
e831acc
nv_ebpf: Initialize spinlock before add new thread.
thiagoftsm Mar 11, 2024
5fb0e9a
nv_ebpf: Add new thread and adjust main thread to update last run.
thiagoftsm Mar 11, 2024
69e58ed
nv_ebpf: Fill thread content to cleanup table when functions are not …
thiagoftsm Mar 11, 2024
8289e14
nv_ebpf: Add missing PID
thiagoftsm Mar 11, 2024
cffb5fe
nv_ebpf: Change the way we test arguments.
thiagoftsm Mar 12, 2024
8e809ab
nv_ebpf: Address issues when run debug
thiagoftsm Mar 12, 2024
4f46110
nv_ebpf: Add README
thiagoftsm Mar 12, 2024
e1a1d1b
nv_ebpf: Fix typo
thiagoftsm Mar 12, 2024
c9c594b
nv_ebpf: Update reference avoiding possible issues with future changes
thiagoftsm Mar 12, 2024
b23d17e
nv_ebpf: Remove unecessary test.
thiagoftsm Mar 12, 2024
345e2b6
nv_ebpf: Address issue when eBPF is disabled
thiagoftsm Mar 12, 2024
1e10704
nv_ebpf: Append a call to proc function, avoiding have empty response
thiagoftsm Mar 12, 2024
69264d7
nv_ebpf: Fix typo; address codacy
thiagoftsm Mar 12, 2024
9cfb29b
nv_ebpf: Reorganize index
thiagoftsm Mar 13, 2024
1157fd5
nv_ebpf: Change default level
thiagoftsm Mar 13, 2024
69b0b7b
nv_ebpf: Show namespace when it is possible
thiagoftsm Mar 25, 2024
73fc40c
nv_ebpf: Move content from plugin to libnetdata; Update enum to match…
thiagoftsm Feb 26, 2024
a723c6a
nv_ebpf: Add code to load eBPF
thiagoftsm Feb 27, 2024
da15917
nv_ebpf: Fix compilation on some distributions
thiagoftsm Feb 27, 2024
de0c65a
nv_ebpf: Add data types to work with eBPF hash tables
thiagoftsm Feb 28, 2024
eeef36e
nv_ebpf: Add a selector between eBPF and other models
thiagoftsm Feb 28, 2024
8708455
nv_ebpf: Add enum to do code more readable
thiagoftsm Feb 28, 2024
30fbaec
nv_ebpf: Move variable from libnetdata to plugin
thiagoftsm Feb 29, 2024
17f5fbe
nv_ebpf: Add function to read data from eBPF table
thiagoftsm Feb 29, 2024
823e2b7
nv_ebpf: Move content before to store
thiagoftsm Feb 29, 2024
a925a96
nv_ebpf: Add restriction to display data
thiagoftsm Feb 29, 2024
1bdd02d
prepare_functions: Modify data type avoiding compiler issues
thiagoftsm Mar 1, 2024
f28d4bb
prepare_functions: Store sockets
thiagoftsm Mar 1, 2024
981a774
nv_ebpf: Modify algorithm to match new data type
thiagoftsm Mar 4, 2024
528b80c
nv_ebpf: Add algorithm to also work with tcp_close
thiagoftsm Mar 5, 2024
912767d
nv_ebpf: Move function call outside local_sockets_read_sockets_from_proc
thiagoftsm Mar 6, 2024
75eeb44
nv_ebpf: Use optional variable to define when it is necessary to load…
thiagoftsm Mar 8, 2024
b51ff32
nv_ebpf: Store data inside hashtable, because eBPF only collects acti…
thiagoftsm Mar 9, 2024
e0de617
nv_ebpf: Fix issue to read sockets
thiagoftsm Mar 10, 2024
eda4442
nv_ebpf: Adjust flags and prepare to add table cleanup when function …
thiagoftsm Mar 10, 2024
957c0aa
nv_ebpf: Initialize spinlock before add new thread.
thiagoftsm Mar 11, 2024
207bbb3
nv_ebpf: Add new thread and adjust main thread to update last run.
thiagoftsm Mar 11, 2024
f112080
nv_ebpf: Fill thread content to cleanup table when functions are not …
thiagoftsm Mar 11, 2024
7fe7ac5
nv_ebpf: Change the way we test arguments.
thiagoftsm Mar 12, 2024
365790c
nv_ebpf: Update reference avoiding possible issues with future changes
thiagoftsm Mar 12, 2024
c36622c
nv_ebpf: Change default level
thiagoftsm Mar 13, 2024
22e389e
nv_ebpf: Update libbpf
Apr 1, 2024
a1325d7
nv_ebpf: Fix rebase
thiagoftsm Apr 2, 2024
392050f
nv_ebpf: Address issue with static tpc_v6_connect on some kernels
thiagoftsm Apr 2, 2024
ead4e83
nv_ebpf: Reduce number of PIDs shown
thiagoftsm Apr 2, 2024
3adc674
nv_ebpf: Create limitation considering actual dashboard
thiagoftsm Apr 3, 2024
b5ac10b
nv_ebpf: Update ebpf
thiagoftsm Apr 3, 2024
0414963
nv_ebpf: Adjust message and level, considering we are starting using …
thiagoftsm Apr 3, 2024
1fb77ea
nv_ebpf: Change flags to identify when the new option is used
thiagoftsm Apr 4, 2024
343a55e
nv_ebpf: Select between PID and TGID
thiagoftsm Apr 4, 2024
c4223a4
nv_ebpf: Move content outside local-sockets.h
thiagoftsm Apr 4, 2024
beb7cc0
nv_ebpf: Rename functions
thiagoftsm Apr 5, 2024
0767e15
nv_ebpf: Create structure to store long time sockets
thiagoftsm Apr 5, 2024
7522b35
nv_ebpf: Add pointer to use structure with different functions
thiagoftsm Apr 5, 2024
bdb064b
nv_ebpf: Update documentation
thiagoftsm Apr 5, 2024
4d9a2f3
nv_ebpf: Always collect current socket, and store in our database whe…
thiagoftsm Apr 6, 2024
a8bd417
nv_ebpf: Remove reset value after to change algorithm
thiagoftsm Apr 6, 2024
911be00
nv_ebpf: Cleanup function before copy hash tables.
thiagoftsm Apr 6, 2024
9968632
nv_ebpf: Organize sync algorithm and do some cleanup
thiagoftsm Apr 7, 2024
b8e7dcd
nv_ebpf: Modify network_viewer_ebpf_get_sockets function returning a …
thiagoftsm Apr 8, 2024
39c4f95
nv_ebpf: Change network_viewer_function to use pointers for different…
thiagoftsm Apr 8, 2024
9288292
nv_ebpf: Move code
thiagoftsm Apr 8, 2024
3992f84
nv_ebpf: Address issues to detect listen ports
thiagoftsm Apr 9, 2024
184e1bf
nv_ebpf: Fix issues that did not show sockets..
thiagoftsm Apr 12, 2024
1e8b13d
nv_ebpf: Simplify logic to read listen ports
thiagoftsm Apr 13, 2024
630a6d5
nv_ebpf: Reduce number of names displayed on charts
thiagoftsm Apr 13, 2024
5c88bc9
nv_ebpf: Adjust chart representation with all PIDs
thiagoftsm Apr 14, 2024
f85d69f
nv_ebpf: Update documentation
thiagoftsm Apr 15, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 3 additions & 0 deletions src/collectors/ebpf.plugin/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -302,6 +302,9 @@ are divided in the following sections:

#### `[network connections]`

Netdata is integrating eBPF sockets collection with the [`networkviewer.plugin`](https://github.com/netdata/netdata/blob/master/src/collectors/network-viewer.plugin/network-connections-chart.html).
During this process, we are temporarily disabling this option to prevent host overload.

You can configure the information shown with function `ebpf_socket` using the settings in this section.

```conf
Expand Down
40 changes: 9 additions & 31 deletions src/collectors/ebpf.plugin/ebpf.c
Original file line number Diff line number Diff line change
Expand Up @@ -67,14 +67,14 @@ ebpf_module_t ebpf_modules[] = {
{ .info = {.thread_name = "socket",
.config_name = "socket",
.thread_description = NETDATA_EBPF_SOCKET_MODULE_DESC},
.functions = {.start_routine = ebpf_socket_thread,
.apps_routine = ebpf_socket_create_apps_charts,
.fnct_routine = ebpf_socket_read_open_connections,
.fcnt_name = EBPF_FUNCTION_SOCKET,
.fcnt_desc = EBPF_PLUGIN_SOCKET_FUNCTION_DESCRIPTION,
.fcnt_thread_chart_name = NULL,
.fcnt_thread_lifetime_name = NULL},
.enabled = NETDATA_THREAD_EBPF_NOT_RUNNING,
.functions = {.start_routine = NULL, //ebpf_socket_thread,
.apps_routine = NULL, //ebpf_socket_create_apps_charts,
.fnct_routine = NULL, //ebpf_socket_read_open_connections,
.fcnt_name = EBPF_FUNCTION_SOCKET,
.fcnt_desc = EBPF_PLUGIN_SOCKET_FUNCTION_DESCRIPTION,
.fcnt_thread_chart_name = NULL,
.fcnt_thread_lifetime_name = NULL},
.enabled = NETDATA_THREAD_EBPF_NOT_RUNNING,
.update_every = EBPF_DEFAULT_UPDATE_EVERY, .global_charts = 1, .apps_charts = NETDATA_EBPF_APPS_FLAG_NO,
.apps_level = NETDATA_APPS_LEVEL_REAL_PARENT, .cgroup_charts = CONFIG_BOOLEAN_NO, .mode = MODE_ENTRY, .optional = 0,
.maps = NULL,
Expand Down Expand Up @@ -813,28 +813,6 @@ static void ebpf_exit()
exit(0);
}

/**
* Unload loegacy code
*
* @param objects objects loaded from eBPF programs
* @param probe_links links from loader
*/
void ebpf_unload_legacy_code(struct bpf_object *objects, struct bpf_link **probe_links)
{
if (!probe_links || !objects)
return;

struct bpf_program *prog;
size_t j = 0 ;
bpf_object__for_each_program(prog, objects) {
bpf_link__destroy(probe_links[j]);
j++;
}
freez(probe_links);
if (objects)
bpf_object__close(objects);
}

/**
* Unload Unique maps
*
Expand Down Expand Up @@ -4023,7 +4001,7 @@ int main(int argc, char **argv)
ebpf_module_t *em = &ebpf_modules[i];
em->thread = st;
em->thread_id = i;
if (em->enabled != NETDATA_THREAD_EBPF_NOT_RUNNING) {
if (em->enabled != NETDATA_THREAD_EBPF_NOT_RUNNING && em->functions.start_routine) {
st->thread = mallocz(sizeof(netdata_thread_t));
em->enabled = NETDATA_THREAD_EBPF_RUNNING;
em->lifetime = EBPF_NON_FUNCTION_LIFE_TIME;
Expand Down
22 changes: 0 additions & 22 deletions src/collectors/ebpf.plugin/ebpf_socket.h
Original file line number Diff line number Diff line change
Expand Up @@ -51,18 +51,6 @@ enum ebpf_socket_publish_index {
NETDATA_MAX_SOCKET_VECTOR
};

enum socket_functions {
NETDATA_FCNT_INET_CSK_ACCEPT,
NETDATA_FCNT_TCP_RETRANSMIT,
NETDATA_FCNT_CLEANUP_RBUF,
NETDATA_FCNT_TCP_CLOSE,
NETDATA_FCNT_UDP_RECEVMSG,
NETDATA_FCNT_TCP_SENDMSG,
NETDATA_FCNT_UDP_SENDMSG,
NETDATA_FCNT_TCP_V4_CONNECT,
NETDATA_FCNT_TCP_V6_CONNECT
};

typedef enum ebpf_socket_idx {
NETDATA_KEY_CALLS_TCP_SENDMSG,
NETDATA_KEY_ERROR_TCP_SENDMSG,
Expand Down Expand Up @@ -207,16 +195,6 @@ typedef struct netdata_passive_connection_idx {
uint16_t port;
} netdata_passive_connection_idx_t;

/**
* Union used to store ip addresses
*/
union netdata_ip_t {
uint8_t addr8[16];
uint16_t addr16[8];
uint32_t addr32[4];
uint64_t addr64[2];
};

typedef struct ebpf_network_viewer_ip_list {
char *value; // IP value
uint32_t hash; // IP hash
Expand Down
66 changes: 66 additions & 0 deletions src/collectors/network-viewer.plugin/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
<!--
title: "Network Viewer monitoring (network-viewer.plugin)"
sidebar_label: "Network Viewer monitoring "
custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/collectors/network-viewer.plugin/README.md"
learn_status: "Published"
learn_topic_type: "References"
learn_rel_path: "Integrations/Monitor/System metrics"
-->

# Network Viewer monitoring (network-viewer.plugin)

The `network-viewer.plugin` provides users with information about TCP and UDP
resources.
It is enabled by default on every Netdata installation.

To accomplish this, it scans through the entire process tree, gathering socket
information for each process identified during its parsing of `/proc` or using
`libmnl`. Alternatively, it monitors calls to kernel internal functions when
utilizing `eBPF`.

## Functions

To visualize your network traffic, you'll need to log in to the Netdata cloud.
Navigate to the `Top` tab, select a specific host, and then click on
`Network connections`.

## Performance

The `network-viewer.plugin` is a sophisticated piece of software that requires
significant processing.

On Linux systems, the plugin reads multiple `network-viewer.plugin` reads
several `/proc` files for each running process to gather necessary data.
Performing this task every second, particularly on hosts with numerous
processes, can result in increased CPU utilization by the plugin.

To minimize the load caused by parsing `/proc` the plugin can utilize `eBPF`,
provided it's available on your host. However, this approach will introduce
additional overhead for each TCP and UDP function call on the host. To switch
from using `/proc` files to eBPF, adjustments to the configuration file are also
required:

```
[plugin:network-viewer]
command options = ebpf
```

You can further customize the collection level by specifying the desired option
`apps-level`:

```
[plugin:network-viewer]
command options = ebpf apps-level 1
```

The accepted values are as follows:
- 0: Group data under the real parent
- 1: Group data under the parent.

Lastly, you have the option to personalize whether the plugin will display data per parent PID,
which is the default behavior, or utilize per-thread information by enabling the `use-pid` option:

```
[plugin:network-viewer]
command options = ebpf apps-level 1 use-pid
```