Skip to content

Commit

Permalink
Windows Support Phase 1 (#17497)
Browse files Browse the repository at this point in the history 
* abstraction layer for O/S

* updates

* updates

* updates

* temp fix for protobuf

* emulated waitid()

* fix

* fix

* compatibility layer

* fix for idtype

* fix for missing includes

* fix for missing includes

* added missing includes

* added missing includes

* added missing includes

* added missing includes

* added missing includes

* added missing includes

* UUID renamed to ND_UUID to avoid conflict with windows.h

* include libnetdata.h always - no conflicts

* simplify abstraction headers

* fix missing functions

* fix missing functions

* fix missing functions

* fix missing functions

* rename MSYS to WINDOWS

* moved byteorder.h

* structure for an internal windows plugin

* 1st windows plugin

* working plugin

* fix printf

* Special case windows for protobuf

* remove cygwin, compile both as windows

* log windows libraries used

* fix cmake

* fix protobuf

* compilation

* updated compilation script

* added system.ram

* windows uptime

* perflib

* working perflibdump

* minify dump

* updates to windows plugins, enable ML

* minor compatibility fixes for cygwin and msys

* perflib-dump to its own file

* perflib now indexes names

* improvements to the library; disks module WIP

* API for selectively traversing the metrics

* first working perflib chart: disk.space

* working chart on logical and physical disks

* added windows protocols

* fix datatypes for loops

* tinysleep for native smallest sleep support

* remove libuuid dependency on windows

* fix uuid functions for macos compilation

* fix uuid comparison function

* do not overwrite uuid library functions, define them as aliases to our own

* fixed uuid_unparse functions

* fixed typo

* added perflib processor charts

* updates for compiling without posix emulation

* gather common contexts together

* fix includes on linux

* perflib-memory

* windows mem.available

* Update variable names for protobuf

* network traffic

* add network adapters that have traffic as virtual interfaces

* add -pipe to windows compilation

* reset or overflow flag is now per dimension

* dpc is now counted separately

* verified all perflib fields are processed and no text fields are present in the data

* more common contexts

* fix crash

* do not add system.net multiple times

* install deps update and shortcut

* all threads are now joinable behind the scenes

* fix threads cleanup

* prepare for abstracting threads API

* netdata threads full abstraction from pthreads

* more threads abstraction and cleanup

* more compatibility changes

* fix compiler warnings

* add base-devel to packages

* removed duplicate base-devel

* check for strndup

* check headers in quotes

* fix linux compilation

* fix attribute gnu_printf on macos

* fix for threads on macos

* mingw64 compatibility

* enable compilation on windows clion

* added instructions

* enable cloud

* compatibility fixes

* compatibility fixes

* compatibility fixes

* clion works on windows

* support both MSYSTEM=MSYS and MSYSTEM=MINGW64 for configure

* cleanup and docs

* rename uuid_t to nd_uuid_t to avoid conflict with windows uuid_t

* leftovers uuid_t

* do not include uuid.h on macos

* threads signaled cancellations

* do not install v0 dashboard on windows

* script to install openssh server on windows

* update openssh installation script

* update openssh installation script

* update openssh installation script

* update openssh installation script

* update openssh installation script

* update openssh installation script

* update openssh installation script

* update openssh installation script

* update openssh installation script

* use cleanup variable instead of pthreads push and pop

* replace all calls to netdata_thread_cleanup_push() and netdata_thread_cleanup_pop() with __attribute__((cleanup(...)))

* remove left-over freez

* make sure there are no locks acquired at thread exit

* add missing parameter

* stream receivers and senders are now voluntarily cancelled

* plugins.d now voluntarily exits its threads

* uuid_t may not be aligned to word boundaries - fix the uuid_t functions to work on unaligned objects too.

* collectors evloop is now using the new threading cancellation; ml is now not using pthread_cancel; more fixes

* eliminate threads cancellability from the code base

* fix exit timings and logs; fix uv_threads tags

* use SSL_has_pending() only when it is available

* do not use SSL_has_pending()

* dyncfg files on windows escape collon and pipe characters

* fix compilation on older systems

* fix compilation on older systems

* Create windows installer.

The installer will install everything
under C:\netdata by default.

It will:
  - Install msys2 at C:\netdata
  - Install netdata dependencies with pacman
  - Install the agent itself under C:\netdata\opt

You can start the agent by running an MSYS
shell with C:\netdata\msys2_shell.cmd and
then start the agent normally with:

/opt/netdata/usr/sbin/netdata -D

There are a more couple things to work on:

  - Verify publisher.
  - Install all deps not just libuv & protobuf.
  - Figure out how we want to auto-start the agent as a service.
  - Check how to uninstall things.

* fixed typo

* code cleanup

* Create uninstaller

---------

Co-authored-by: vkalintiris <vasilis@netdata.cloud>
  • Loading branch information
@ktsaou @vkalintiris
ktsaou and vkalintiris committed May 16, 2024
1 parent c401ef6 commit fe06e84
Show file tree
Hide file tree
Showing 283 changed files with 7,449 additions and 2,824 deletions.
152 changes: 134 additions & 18 deletions CMakeLists.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -225,23 +225,51 @@ endif()
# detect OS
#

set(LINUX False)
set(FREEBSD False)
set(MACOS False)
set(LINUX False)
set(FREEBSD False)
set(MACOS False)
set(WINDOWS False)
set(FOREIGN_OS False)

if(${CMAKE_SYSTEM_NAME} MATCHES "Darwin")
if("${CMAKE_SYSTEM_NAME}" STREQUAL "Darwin")
set(MACOS True)
set(COMPILED_FOR_MACOS True)

find_library(IOKIT IOKit)
find_library(FOUNDATION Foundation)
elseif(${CMAKE_SYSTEM_NAME} MATCHES "FreeBSD")
message(INFO " Compiling for MacOS... ")
elseif("${CMAKE_SYSTEM_NAME}" STREQUAL "FreeBSD")
set(FREEBSD True)
set(COMPILED_FOR_FREEBSD True)
else()
message(INFO " Compiling for FreeBSD... ")
elseif("${CMAKE_SYSTEM_NAME}" STREQUAL "Linux")
set(LINUX True)
set(COMPILED_FOR_LINUX True)
add_definitions(-D_GNU_SOURCE)
message(INFO " Compiling for Linux... ")
elseif("${CMAKE_SYSTEM_NAME}" STREQUAL "CYGWIN" OR "${CMAKE_SYSTEM_NAME}" STREQUAL "MSYS" OR "${CMAKE_SYSTEM_NAME}" STREQUAL "Windows")
set(WINDOWS True)
set(COMPILED_FOR_WINDOWS True)
add_definitions(-D_GNU_SOURCE)

if($ENV{CLION_IDE})
# clion needs these to find the includes
if("${CMAKE_SYSTEM_NAME}" STREQUAL "MSYS" OR "${CMAKE_SYSTEM_NAME}" STREQUAL "Windows")
if("$ENV{MSYSTEM}" STREQUAL "MSYS")
include_directories(c:/msys64/usr/include)
include_directories(c:/msys64/usr/include/w32api)
elseif("$ENV{MSYSTEM}" STREQUAL "MINGW64")
include_directories(c:/msys64/mingw64/include)
elseif("$ENV{MSYSTEM}" STREQUAL "UCRT64")
include_directories(c:/msys64/ucrt64/include)
endif()
endif()
endif()

message(INFO " Compiling for Windows (${CMAKE_SYSTEM_NAME}, MSYSTEM=$ENV{MSYSTEM})... ")
else()
set(FOREIGN_OS True)
set(COMPILED_FOR_FOREIGN_OS True)
message(WARNING " Compiling for Unknown O/S... (${CMAKE_SYSTEM_NAME})")
endif()

if(ENABLE_PLUGIN_EBPF)
Expand Down Expand Up @@ -326,6 +354,20 @@ check_include_file("sys/statvfs.h" HAVE_SYS_STATVFS_H)
check_include_file("inttypes.h" HAVE_INTTYPES_H)
check_include_file("stdint.h" HAVE_STDINT_H)
check_include_file("sys/capability.h" HAVE_SYS_CAPABILITY_H)
check_include_file("arpa/inet.h" HAVE_ARPA_INET_H)
check_include_file("netinet/tcp.h" HAVE_NETINET_TCP_H)
check_include_file("sys/ioctl.h" HAVE_SYS_IOCTL_H)
check_include_file("grp.h" HAVE_GRP_H)
check_include_file("pwd.h" HAVE_PWD_H)
check_include_file("net/if.h" HAVE_NET_IF_H)
check_include_file("poll.h" HAVE_POLL_H)
check_include_file("syslog.h" HAVE_SYSLOG_H)
check_include_file("sys/mman.h" HAVE_SYS_MMAN_H)
check_include_file("sys/resource.h" HAVE_SYS_RESOURCE_H)
check_include_file("sys/socket.h" HAVE_SYS_SOCKET_H)
check_include_file("sys/wait.h" HAVE_SYS_WAIT_H)
check_include_file("sys/un.h" HAVE_SYS_UN_H)
check_include_file("spawn.h" HAVE_SPAWN_H)

#
# check symbols
Expand All @@ -340,9 +382,15 @@ check_symbol_exists(finite "math.h" HAVE_FINITE)
check_symbol_exists(isfinite "math.h" HAVE_ISFINITE)
check_symbol_exists(dlsym "dlfcn.h" HAVE_DLSYM)

check_function_exists(pthread_getthreadid_np HAVE_PTHREAD_GETTHREADID_NP)
check_function_exists(pthread_threadid_np HAVE_PTHREAD_THREADID_NP)
check_function_exists(gettid HAVE_GETTID)
check_function_exists(waitid HAVE_WAITID)
check_function_exists(nice HAVE_NICE)
check_function_exists(recvmmsg HAVE_RECVMMSG)
check_function_exists(getpriority HAVE_GETPRIORITY)
check_function_exists(setenv HAVE_SETENV)
check_function_exists(strndup HAVE_STRNDUP)

check_function_exists(sched_getscheduler HAVE_SCHED_GETSCHEDULER)
check_function_exists(sched_setscheduler HAVE_SCHED_SETSCHEDULER)
Expand Down Expand Up @@ -624,10 +672,10 @@ set(LIBNETDATA_FILES
src/libnetdata/log/journal.h
src/libnetdata/log/log.c
src/libnetdata/log/log.h
src/libnetdata/os.c
src/libnetdata/os.h
src/libnetdata/os/os.c
src/libnetdata/os/os.h
src/libnetdata/simple_hashtable.h
src/libnetdata/byteorder.h
src/libnetdata/os/byteorder.h
src/libnetdata/onewayalloc/onewayalloc.c
src/libnetdata/onewayalloc/onewayalloc.h
src/libnetdata/popen/popen.c
Expand Down Expand Up @@ -682,6 +730,34 @@ set(LIBNETDATA_FILES
src/libnetdata/linked-lists.h
src/libnetdata/storage-point.h
src/libnetdata/bitmap64.h
src/libnetdata/os/waitid.c
src/libnetdata/os/waitid.h
src/libnetdata/os/gettid.c
src/libnetdata/os/gettid.h
src/libnetdata/os/adjtimex.c
src/libnetdata/os/adjtimex.h
src/libnetdata/os/setresuid.c
src/libnetdata/os/setresuid.h
src/libnetdata/os/setresgid.c
src/libnetdata/os/setresgid.h
src/libnetdata/os/getgrouplist.c
src/libnetdata/os/getgrouplist.h
src/libnetdata/os/get_pid_max.c
src/libnetdata/os/get_pid_max.h
src/libnetdata/os/os-freebsd-wrappers.c
src/libnetdata/os/os-freebsd-wrappers.h
src/libnetdata/os/os-macos-wrappers.c
src/libnetdata/os/os-macos-wrappers.h
src/libnetdata/os/get_system_cpus.c
src/libnetdata/os/get_system_cpus.h
src/libnetdata/os/tinysleep.c
src/libnetdata/os/tinysleep.h
src/libnetdata/os/uuid_generate.c
src/libnetdata/os/uuid_generate.h
src/libnetdata/os/setenv.c
src/libnetdata/os/setenv.h
src/libnetdata/os/strndup.c
src/libnetdata/os/strndup.h
)

if(ENABLE_PLUGIN_EBPF)
Expand Down Expand Up @@ -956,6 +1032,16 @@ else()
)
endif()

set(INTERNAL_COLLECTORS_FILES
src/collectors/common-contexts/common-contexts.h
src/collectors/common-contexts/disk.io.h
src/collectors/common-contexts/system.io.h
src/collectors/common-contexts/system.ram.h
src/collectors/common-contexts/mem.swap.h
src/collectors/common-contexts/mem.pgfaults.h
src/collectors/common-contexts/mem.available.h
)

set(PLUGINSD_PLUGIN_FILES
src/collectors/plugins.d/plugins_d.c
src/collectors/plugins.d/plugins_d.h
Expand Down Expand Up @@ -1197,6 +1283,24 @@ set(FREEBSD_PLUGIN_FILES
src/collectors/proc.plugin/zfs_common.h
)

set(WINDOWS_PLUGIN_FILES
src/collectors/windows.plugin/windows_plugin.c
src/collectors/windows.plugin/windows_plugin.h
src/collectors/windows.plugin/GetSystemUptime.c
src/collectors/windows.plugin/GetSystemRAM.c
src/collectors/windows.plugin/GetSystemCPU.c
src/collectors/windows.plugin/perflib.c
src/collectors/windows.plugin/perflib.h
src/collectors/windows.plugin/perflib-rrd.c
src/collectors/windows.plugin/perflib-rrd.h
src/collectors/windows.plugin/perflib-names.c
src/collectors/windows.plugin/perflib-dump.c
src/collectors/windows.plugin/perflib-storage.c
src/collectors/windows.plugin/perflib-processor.c
src/collectors/windows.plugin/perflib-network.c
src/collectors/windows.plugin/perflib-memory.c
)

set(PROC_PLUGIN_FILES
src/collectors/proc.plugin/ipc.c
src/collectors/proc.plugin/plugin_proc.c
Expand Down Expand Up @@ -1315,6 +1419,7 @@ if(LINUX)
${PROC_PLUGIN_FILES}
${TC_PLUGIN_FILES}
${TIMEX_PLUGIN_FILES}
${INTERNAL_COLLECTORS_FILES}
)

if(ENABLE_SENTRY)
Expand All @@ -1327,12 +1432,20 @@ elseif(MACOS)
src/daemon/static_threads_macos.c
${MACOS_PLUGIN_FILES}
${TIMEX_PLUGIN_FILES}
${INTERNAL_COLLECTORS_FILES}
)
elseif(FREEBSD)
list(APPEND NETDATA_FILES
src/daemon/static_threads_freebsd.c
${FREEBSD_PLUGIN_FILES}
${TIMEX_PLUGIN_FILES}
${INTERNAL_COLLECTORS_FILES}
)
elseif(WINDOWS)
list(APPEND NETDATA_FILES
src/daemon/static_threads_windows.c
${WINDOWS_PLUGIN_FILES}
${INTERNAL_COLLECTORS_FILES}
)
endif()

Expand Down Expand Up @@ -1518,6 +1631,7 @@ target_include_directories(libnetdata BEFORE PUBLIC ${CONFIG_H_DIR} ${CMAKE_SOUR
target_link_libraries(libnetdata PUBLIC
"$<$<NOT:$<BOOL:${HAVE_BUILTIN_ATOMICS}>>:atomic>"
"$<$<OR:$<BOOL:${LINUX}>,$<BOOL:${FREEBSD}>>:pthread;rt>"
"$<$<BOOL:${WINDOWS}>:kernel32;advapi32;winmm;rpcrt4>"
"$<$<BOOL:${LINK_LIBM}>:m>"
"${SYSTEMD_LDFLAGS}")

Expand Down Expand Up @@ -1578,7 +1692,7 @@ if(LIBBROTLI_FOUND)
endif()

# uuid
if(MACOS)
if(MACOS OR WINDOWS)
# UUID functionality is part of the system libraries here, so no extra
# stuff needed.
else()
Expand Down Expand Up @@ -1879,9 +1993,9 @@ endif()

if(ENABLE_PLUGIN_CUPS)
pkg_check_modules(CUPS libcups)
if(NOT CUPS_LIBRARIES)
if(NOT CUPS_FOUND)
pkg_check_modules(CUPS cups)
if(NOT CUPS_LIBRARIES)
if(NOT CUPS_FOUND)
find_program(CUPS_CONFIG cups-config)
if(CUPS_CONFIG)
execute_process(COMMAND ${CUPS_CONFIG} --api-version OUTPUT_VARIABLE CUPS_API_VERSION OUTPUT_STRIP_TRAILING_WHITESPACE)
Expand Down Expand Up @@ -2914,10 +3028,12 @@ install(FILES
COMPONENT netdata
DESTINATION ${WEB_DEST}/.well-known/dnt)

# v0 dashboard
install(FILES
src/web/gui/v0/index.html
COMPONENT netdata
DESTINATION ${WEB_DEST}/v0)
if(NOT WINDOWS)
# v0 dashboard
install(FILES
src/web/gui/v0/index.html
COMPONENT netdata
DESTINATION ${WEB_DEST}/v0)
endif()

include(Packaging)
97 changes: 55 additions & 42 deletions packaging/cmake/Modules/NetdataProtobuf.cmake
View file
Original file line number Diff line number Diff line change
Expand Up @@ -56,54 +56,67 @@ endfunction()

# Handle detection of Protobuf
macro(netdata_detect_protobuf)
if(NOT ENABLE_BUNDLED_PROTOBUF)
if (NOT BUILD_SHARED_LIBS)
set(Protobuf_USE_STATIC_LIBS On)
if(COMPILED_FOR_WINDOWS)
set(PROTOBUF_PROTOC_EXECUTABLE "$ENV{PROTOBUF_PROTOC_EXECUTABLE}")
if(NOT PROTOBUF_PROTOC_EXECUTABLE)
set(PROTOBUF_PROTOC_EXECUTABLE "/bin/protoc")
endif()

# The FindProtobuf CMake module shipped by upstream CMake is
# broken for Protobuf version 22.0 and newer because it does
# not correctly pull in the new Abseil dependencies. Protobuf
# itself sometimes ships a CMake Package Configuration module
# that _does_ work correctly, so use that in preference to the
# Find module shipped with CMake.
#
# The code below works by first attempting to use find_package
# in config mode, and then checking for the existence of the
# target we actually use that gets defined by the protobuf
# CMake Package Configuration Module to determine if that
# worked. A bit of extra logic is required in the case of the
# config mode working, because some systems ship compatibility
# logic for the old FindProtobuf module while others do not.
#
# Upstream bug reference: https://gitlab.kitware.com/cmake/cmake/-/issues/24321
find_package(Protobuf CONFIG)

if(NOT TARGET protobuf::libprotobuf)
message(STATUS "Could not find Protobuf using Config mode, falling back to Module mode")
find_package(Protobuf REQUIRED)
set(PROTOBUF_CFLAGS_OTHER "")
set(PROTOBUF_INCLUDE_DIRS "")
set(PROTOBUF_LIBRARIES "-lprotobuf")

set(ENABLE_PROTOBUF True)
set(HAVE_PROTOBUF True)
else()
if(NOT ENABLE_BUNDLED_PROTOBUF)
if (NOT BUILD_SHARED_LIBS)
set(Protobuf_USE_STATIC_LIBS On)
endif()

# The FindProtobuf CMake module shipped by upstream CMake is
# broken for Protobuf version 22.0 and newer because it does
# not correctly pull in the new Abseil dependencies. Protobuf
# itself sometimes ships a CMake Package Configuration module
# that _does_ work correctly, so use that in preference to the
# Find module shipped with CMake.
#
# The code below works by first attempting to use find_package
# in config mode, and then checking for the existence of the
# target we actually use that gets defined by the protobuf
# CMake Package Configuration Module to determine if that
# worked. A bit of extra logic is required in the case of the
# config mode working, because some systems ship compatibility
# logic for the old FindProtobuf module while others do not.
#
# Upstream bug reference: https://gitlab.kitware.com/cmake/cmake/-/issues/24321
find_package(Protobuf CONFIG)

if(NOT TARGET protobuf::libprotobuf)
message(STATUS "Could not find Protobuf using Config mode, falling back to Module mode")
find_package(Protobuf REQUIRED)
endif()
endif()
endif()

if(TARGET protobuf::libprotobuf)
if(NOT Protobuf_PROTOC_EXECUTABLE AND TARGET protobuf::protoc)
set(Protobuf_PROTOC_EXECUTABLE protobuf::protoc)
if(TARGET protobuf::libprotobuf)
if(NOT Protobuf_PROTOC_EXECUTABLE AND TARGET protobuf::protoc)
set(Protobuf_PROTOC_EXECUTABLE protobuf::protoc)
endif()

# It is technically possible that this may still not
# be set by this point, so we need to check it and
# fail noisily if it isn't because the build won't
# work without it.
if(NOT Protobuf_PROTOC_EXECUTABLE)
message(FATAL_ERROR "Could not determine the location of the protobuf compiler for the detected version of protobuf.")
endif()

set(PROTOBUF_PROTOC_EXECUTABLE ${Protobuf_PROTOC_EXECUTABLE})
set(PROTOBUF_LIBRARIES protobuf::libprotobuf)
endif()

# It is technically possible that this may still not
# be set by this point, so we need to check it and
# fail noisily if it isn't because the build won't
# work without it.
if(NOT Protobuf_PROTOC_EXECUTABLE)
message(FATAL_ERROR "Could not determine the location of the protobuf compiler for the detected version of protobuf.")
endif()

set(PROTOBUF_PROTOC_EXECUTABLE ${Protobuf_PROTOC_EXECUTABLE})
set(PROTOBUF_LIBRARIES protobuf::libprotobuf)
set(ENABLE_PROTOBUF True)
set(HAVE_PROTOBUF True)
endif()

set(ENABLE_PROTOBUF True)
set(HAVE_PROTOBUF True)
endmacro()

# Helper function to compile protocol definitions into C++ code.
Expand Down

0 comments on commit fe06e84

Please sign in to comment.