wanderer/fastnetmon-rewritten
1
0
Fork 0
mirror of https://github.com/pavel-odintsov/fastnetmon synced 2024-09-26 22:10:47 +02:00
Code Issues Releases Activity
3,492 Commits 2 Branches 20 Tags 53 MiB
866ac8ce3a
Commit Graph

1975 Commits

Author SHA1 Message Date
Pavel Odintsov
a937723bba Added cross platform support for getsockopt in sFlow plugin 2023-04-02 21:17:40 +01:00
Pavel Odintsov
b395248a4d Cross platform support improvement for sFlow plugin 2023-04-02 21:06:56 +01:00
Pavel Odintsov
2786580f3b Get rid of old style include protection 2023-04-02 20:50:03 +01:00
Pavel Odintsov
1a5d8fd894 Reworked non cross platform sleep to cross platform sleep_for 2023-04-02 20:43:20 +01:00
Pavel Odintsov
b12edc4eb8 Added correct include path for curses library 2023-04-02 20:38:36 +01:00
Pavel Odintsov
7645a9655f Removed C includes from fastnetmon_client 2023-04-02 20:34:56 +01:00
Pavel Odintsov
995041cfe1 Added missing includes for Netflow logic 2023-04-02 20:32:12 +01:00
Pavel Odintsov
1078253ad2 Moved away from using name interface as it conflicts with MingGW, related #977 2023-04-02 20:27:05 +01:00
Pavel Odintsov
d820098b46 Added missing include for sockaddr_in6 2023-04-02 20:18:39 +01:00
Pavel Odintsov
15a014404a Added logic to control build of pcap module 2023-04-02 20:08:29 +01:00
Pavel Odintsov
77134e1b1b Added option to disable pcap build 2023-04-02 19:57:29 +01:00
Pavel Odintsov
0227a923ad Added missing in6_addr definition in simple packet structure 2023-04-02 19:50:06 +01:00
Pavel Odintsov
749646cabb We do not need remote syslog and syslog appenders in all modules 2023-04-02 19:40:02 +01:00
Pavel Odintsov
e376abaeed Reworked pcap format to make it more cross platform and make it more ideomatic C++ 2023-04-02 19:30:16 +01:00
Pavel Odintsov
a34c9b6c9f C include cleanup for pcap plugin and cross platform logic 2023-04-02 16:35:15 +01:00
Pavel Odintsov
ad4abcd061 Unified pcap logic to make it cross platform 2023-04-02 16:33:04 +01:00
Pavel Odintsov
097bdc3b0c Cross platform changes for sFlow and Netflow plugins 2023-04-02 16:32:54 +01:00
Pavel Odintsov
c8f236e094 Fixed network_data_structures on Windows 2023-04-02 16:30:57 +01:00
Pavel Odintsov
b5f773b34d Made fastnetmon networks more cross platform 2023-04-02 16:28:34 +01:00
Pavel Odintsov
f5f435f93e Reworked include 2023-04-02 16:26:24 +01:00
Pavel Odintsov
168b797561 Made simple packet cross platform 2023-04-02 16:25:17 +01:00
Pavel Odintsov
08b291c450 Cross platform support for Netflow plugin 2023-04-02 16:23:17 +01:00
Pavel Odintsov
2bc0e19300 Removed unused includes for sFlow plugin 2023-04-02 16:21:31 +01:00
Pavel Odintsov
5aa2d52cee Made sflow plugin cross platform. Addresses include duplication 2023-04-02 16:20:03 +01:00
Pavel Odintsov
459e0cd90e Reworked endian-less logic to make it more clear 2023-04-02 16:11:14 +01:00
Pavel Odintsov
dd755df567 Made Patricial library cross platform 2023-04-02 15:57:30 +01:00
Pavel Odintsov
e09d02e2d9 Added cross platform support for libsflow to fix build on Windows 2023-04-02 15:19:02 +01:00
Pavel Odintsov
91b98a62fb Added option to disable MongoDB builds 2023-04-02 13:39:38 +01:00
Pavel Odintsov
d16e4d9c73 Split Protobuf library search into two steps for clarity about failures 2023-04-01 19:11:02 +01:00
Pavel Odintsov
da2bb0e0b1 cmake generator was replaced by CMakeToolchain long time ago 2023-04-01 18:51:28 +01:00
Pavel Odintsov
2e118691a6 Added vcpkg dependencies manifest which can be used that way: vcpkg install 2023-04-01 17:55:18 +01:00
Pavel Odintsov
640ea2a3f6 Changes for conan file to address conflicts 2023-04-01 16:56:12 +01:00
Pavel Odintsov
4dd9f1d9eb Added conanfile with all dependencies 2023-04-01 16:29:20 +01:00
Pavel Odintsov
784479731f Reformat code with clang-format 2023-03-13 14:43:24 +00:00
Pavel Odintsov
96c08b37bd Enabled host announce by default when GoBGP is enabled 2023-03-13 13:12:09 +00:00
Pavel Odintsov
f03ed772ff Added hashes for new GoBGP binary 2023-03-13 12:42:02 +00:00
Pavel Odintsov
4ee1c4b344 Adjusted path for GoBGP 2023-03-13 12:39:16 +00:00
Pavel Odintsov
2933d8e121 Upgrade bundled GoBGP to 3.12 2023-03-13 12:34:00 +00:00
Pavel Odintsov
9f8eaad078 Reworked GoBGP logic to new version of gRPC binding with GoBGP 2023-03-13 12:30:46 +00:00
Pavel Odintsov
61eb8c5edd Upgrade bundled gRPC schema for GoBGP to v3.12.0 2023-03-13 12:28:31 +00:00
Pavel Odintsov
5f928d1010 Added timestamp for JSON export format 2023-03-10 22:41:18 +01:00
Pavel Odintsov
c05859372c Switched to use arrival time instead of timestamp from flow for Protobuf export 2023-03-10 22:28:34 +01:00
Pavel Odintsov
732fef3ecc Added arrival time for pcap 2023-03-10 22:23:51 +01:00
Pavel Odintsov
b832182bc0 Added arrival time for pcap 2023-03-10 22:22:31 +01:00
Pavel Odintsov
0e43edc39a Added logic to populate packet arrival time for all plugins 2023-03-10 22:21:36 +01:00
Pavel Odintsov
ddc4f5eaf3 Added logic to export AF_PACKET counters from Prometheus 2023-03-10 12:25:00 +01:00
Pavel Odintsov
7aa3b19849 Added logic to export all available system counters for sFlow mode to Prometheus endpoint 2023-03-10 12:18:27 +01:00
Pavel Odintsov
f216793245 Improved error message when we cannot bind on socket from sFlow plugin 2023-03-10 12:06:03 +01:00
Pavel Odintsov
8369c5e0f1 Addressed race condition and removed old attack direction change logic 2023-03-10 02:58:47 -08:00
Pavel Odintsov
b98e7db2bb Sync Homebrew with upstream 2023-03-07 13:12:52 +00:00
Pavel Odintsov
c0d04543de Attempt to addd workaround for Boost 1.74. Closes #970 2023-03-07 12:43:42 +00:00
Babak Farrokhi
c220afea84
Update FreeBSD port to 1.2.4 (#971) 
Available on FreeBSD ports tree and can be installed via ports or pkg.
 2023-03-06 13:06:34 +00:00
Pavel Odintsov
52cac989d1 Added mutex to address build issues on Debian Sid 2023-03-06 12:16:56 +00:00
Babak Farrokhi
8cbc9ffdc6
Use lightweigth bourne shell when possible 
This script does not use bash specific features, therfore bash is not
necessary. If this script gets called too often, it may use up
unneccessary resources on sensor host.

For reference, here is comparison (binary size) on MacOS (almost ten
times difference):

```
% ls -lh /bin/sh /bin/bash
-r-xr-xr-x  1 root  wheel   1.3M Feb  9 10:39 /bin/bash
-rwxr-xr-x  1 root  wheel   131K Feb  9 10:39 /bin/sh
```

and on FreeBSD (~5x):

```
% ls -lh /bin/sh /usr/local/bin/bash
-r-xr-xr-x  1 root  wheel   166K Feb 23 20:52 /bin/sh
-rwxr-xr-x  1 root  wheel   963K Dec 15 02:15 /usr/local/bin/bash
```
 2023-03-06 11:49:28 +01:00
Babak Farrokhi
96bffc5e98
Moving platform specific packaging files to the right place 2023-03-06 11:40:36 +01:00
Babak Farrokhi
0e0153bb4c
Unbreak build on FreeBSD 2023-03-06 11:36:00 +01:00
Pavel Odintsov
f516cd5655 Switched to 1.2.5 as version number for new release 2023-03-01 14:50:21 +00:00
Pavel Odintsov
c5090e8e84 Increased default threshold per host from 20k to 100k as 20k is too low for modern traffic speeds 2023-03-01 14:23:34 +00:00
Pavel Odintsov
a032af4998 Fixed URL for IPv4 logic 2023-03-01 14:16:10 +00:00
Pavel Odintsov
08c307d941 Switched to use short git commit 2023-03-01 14:06:16 +00:00
Pavel Odintsov
565dd90868 Changed Advanced edition page from main to landing about product 2023-03-01 14:06:04 +00:00
Pavel Odintsov
bd4d62f305 Reformat all code with clang-format 2023-03-01 13:31:39 +00:00
Pavel Odintsov
2ffcab168a Adjusted paths for protoc 2023-03-01 12:49:33 +00:00
Pavel Odintsov
ff1e1a03c5 Rearranged fields in Protobuf encoding logic and addressed bug with ip_dont_fragment 2023-03-01 12:45:18 +00:00
Pavel Odintsov
5745221f83 Adjusted to new path for proto file 2023-03-01 12:36:56 +00:00
Pavel Odintsov
15edfed29d Reordered sequence of fields in protobuf file to more logical 2023-03-01 12:30:20 +00:00
Pavel Odintsov
2a67d0bab3 Moved Protobuf file to dedicated folder 2023-03-01 12:17:24 +00:00
Pavel Odintsov
eb982f4820 Reworked increment counters functions to use references 2023-02-28 20:45:57 +00:00
Pavel Odintsov
59af2ed719 Switched to unified speed counters 2023-02-28 20:18:41 +00:00
Pavel Odintsov
d5ce086244 Complete rework of speed counters for IPv4 prefixes and IPv6 hosts and networks 2023-02-28 19:24:05 +00:00
Pavel Odintsov
075216e8a9 Added logic to check that make and cmake were successfull 2023-02-28 17:33:43 +00:00
Pavel Odintsov
9c1dd27220 Imporoved gTest search logic 2023-02-28 17:24:46 +00:00
Pavel Odintsov
ef07501bcd Added speed calculation test which emulates whole logic 2023-02-28 16:03:10 +00:00
Pavel Odintsov
9f75a1b7d4 Disabled rdtsc tests as it's not cross platform 2023-02-28 15:30:38 +00:00
Pavel Odintsov
c4b1b1a31d Added open addressing maps for teal traffic test 2023-02-28 15:18:40 +00:00
Pavel Odintsov
5d4e878af6 Re-enabled logic to build tests by default 2023-02-28 15:02:14 +00:00
Pavel Odintsov
ab1aa224d3 Added tests for open addressing hash maps 2023-02-28 14:54:57 +00:00
Mamoru TASAKA
35635679c1
BUILDFIX: add missing header for g++13 (#968) 
With g++13, the following error is produced:

```
src/metrics/../abstract_subnet_counters.hpp:79:28: error: 'std::function' has not been declared
```

Fix this error by including additional header.
 2023-02-28 12:25:39 +00:00
Pavel Odintsov
f25df1062c Added hashes for gTest on Debian 12 2023-02-27 22:32:15 +00:00
Pavel Odintsov
b56286d126 Added hashes for gtest 2023-02-27 21:59:39 +00:00
Pavel Odintsov
a4f570a2a4 Moved gTest to run after cmake as it depends on cmake 2023-02-27 21:13:55 +00:00
Pavel Odintsov
75dcf19712 Upgraded gTest and fixed multiple tests 2023-02-27 21:05:58 +00:00
Pavel Odintsov
d32f3ba9ec Fixed test build system 2023-02-27 20:30:53 +00:00
Pavel Odintsov
fd5d501f55 Excluded autogenerated protobuf files from autoformatter 2023-02-26 23:32:28 +00:00
Pavel Odintsov
1ae44451c4 Reformat code with clang format 2023-02-26 23:29:01 +00:00
Pavel Odintsov
960d35166b Extracted traffic format encoding into separate static library 2023-02-21 09:26:35 -08:00
Pavel Odintsov
71096caf9c
Updated Netflow v9 plugin configuration about IPv6 support 2023-02-14 20:33:01 +00:00
Pavel Odintsov
e6ffb31e5c Added option to export traffic to Kafka in Protobuf format 2023-02-12 21:17:05 +00:00
Pavel Odintsov
f18d4585c4 Added logic to encode telemetry type 2023-02-12 21:04:34 +00:00
Pavel Odintsov
231443a443 Improved IP address encoding and added direction encoding as enum 2023-02-12 20:57:29 +00:00
Pavel Odintsov
e0de8b3356 Protobuf field naming adjustments 2023-02-12 19:48:58 +00:00
Pavel Odintsov
b757876b30 Added PoC logic to encode Protobuf data 2023-02-12 12:40:41 +00:00
Pavel Odintsov
d6f5c9b381 Added logic to build Flow Data Protobuf binding 2023-02-12 12:16:02 +00:00
Pavel Odintsov
2e9af3ba91 Added native Prometheus support 2023-02-11 18:08:48 +00:00
Pavel Odintsov
e19a85c180 Migrated XDP plugin to new counters with description 2023-02-11 17:49:02 +00:00
Pavel Odintsov
784e6386f4 Adjusted Netflow counters naming to new naming conventions 2023-02-11 17:45:48 +00:00
Pavel Odintsov
0b0a656e54 Added descriptions for all Netflow counters 2023-02-11 17:37:29 +00:00
Pavel Odintsov
f46f71d66e Switched sFlow plugin to new described counters 2023-02-11 17:29:45 +00:00
Pavel Odintsov
46f5b2c473 Migrated main logic to counters with description 2023-02-11 17:23:16 +00:00
Pavel Odintsov
812ec72c84 Doing migration to described counters structure to make Prometheus possible 2023-02-11 17:18:48 +00:00
Pavel Odintsov
f70c84c8ae Added PoC support for Prometheus 2023-02-11 17:03:05 +00:00
Pavel Odintsov
d0e6987519 Fenced Kafka logic when we have no Kafka compilation enabled 2023-02-06 10:43:25 +00:00
Pavel Odintsov
3b3d8419f6 There are no reasons to use two fields to encode IP addresses 2023-02-05 23:42:45 +00:00
Pavel Odintsov
257f292f3e Improved schema 2023-02-05 23:35:57 +00:00
Pavel Odintsov
fcf8b98ffe Added Protobuf schema concept for FastNetMon 2023-02-05 23:32:25 +00:00
Pavel Odintsov
1036203301 Removed old warning suppression logic 2023-02-05 22:48:49 +00:00
Pavel Odintsov
d9b184f9d2 Fix typo 2023-02-05 20:16:58 +00:00
Pavel Odintsov
38bf9d15dd Added logic to send all traffic in JSON format to Kafka 2023-02-05 20:14:44 +00:00
Pavel Odintsov
8ed555546a Fixed code alignment 2023-02-05 19:54:17 +00:00
Pavel Odintsov
e2bfdebcf7 Extracted IPv6 logic to separate function 2023-02-05 19:53:09 +00:00
Pavel Odintsov
a5d9663dba We do not need icu as direct dependency as Boost automatically links to our icu 2023-02-05 19:36:24 +00:00
Pavel Odintsov
2f1481d86a We do not need to specify C-Ares as direct dependency as gRPC can find it automatically due to custom RPATH for gRPC 2023-02-05 19:30:18 +00:00
Pavel Odintsov
d2fb77b2bd Enabled Kafke for our builds 2023-02-05 19:23:36 +00:00
Pavel Odintsov
ba7d4b282d Added hashes for new cppkafka 2023-02-05 19:17:19 +00:00
Pavel Odintsov
63b7a2449d cppkafka rebuild 2023-02-05 18:57:41 +00:00
Pavel Odintsov
6fcb46708f Force rebuild for cppkafka to address RPATH issue 2023-02-05 18:42:55 +00:00
Pavel Odintsov
12a8821939 There are no reasons to link with rdkafka manually as we link with it from cppkafka 2023-02-05 18:41:48 +00:00
Pavel Odintsov
75afe7c3cd Fixed build logic to build cppkafka with correct RPATH 2023-02-05 18:34:43 +00:00
Pavel Odintsov
3a6432e1c4 Added Kafka producer configuration logic 2023-02-05 18:08:41 +00:00
Pavel Odintsov
03ded87f1d Added logic to load Kafka configuration for traffic export 2023-02-05 17:54:40 +00:00
Pavel Odintsov
bb26d99354 Added basic Kafka configuration 2023-02-05 17:42:57 +00:00
Pavel Odintsov
8ed98fecc7 Added logic to retrieve Kafka libraries 2023-02-05 17:28:36 +00:00
Pavel Odintsov
e6a14ee440 Adding option for Kafka integration 2023-02-04 23:06:49 +00:00
Pavel Odintsov
dd946af4c8 Addressed issue with Protobuf library search path 2023-02-04 23:06:39 +00:00
Pavel Odintsov
f2fb700247 Added hashes for new binary dependencies 2023-02-04 22:06:29 +00:00
Pavel Odintsov
4aa29b2500 Added option to encode simple packet to JSON 2023-02-04 21:57:22 +00:00
Pavel Odintsov
49bf4a5c8a Added hashes for new dependencies 2023-02-04 21:43:16 +00:00
Pavel Odintsov
3c5cddb9cb Added rdkafka and cppkafka as dependencies 2023-02-04 21:36:15 +00:00
Pavel Odintsov
9f9d1ebd46 Added new configuration options: dump_all_traffic and dump_other_traffic useful for parser debugging 2023-01-25 17:37:48 +00:00
Pavel Odintsov
212ed44b61 Added documentation about unloading logic 2023-01-25 16:07:06 +00:00
Pavel Odintsov
f3e88805ec Added BPF build instructions 2023-01-25 16:01:42 +00:00
Pavel Odintsov
ce88bb3287 Ported XDP microcode to new libbpf 1.x.x compatible format: https://github.com/libbpf/libbpf/wiki/Libbpf:-the-road-to-v1.0#drop-support-for-legacy-bpf-map-declaration-syntax 2023-01-25 15:51:53 +00:00
Pavel Odintsov
9c411871c5 Added logic to print error code when bpf_object__open_file fails 2023-01-25 14:49:10 +00:00
Pavel Odintsov
d62d81dc50 We do not use quotes for configuration parameters 2023-01-25 14:34:02 +00:00
Pavel Odintsov
d0c51d6180 Added XDP microcode to use with FastNetMon 2023-01-25 13:39:43 +00:00
Pavel Odintsov
50dbba7c73 Removed testing logic as it did not help to speed up gcc build and we had to build in in cloud 2023-01-22 21:16:07 +00:00
Pavel Odintsov
aa4b172ce2 We do not need this check as we can try to download and it will fail if file do not exist 2023-01-22 21:13:20 +00:00
Pavel Odintsov
c3bcd45ef2 Moved alert about slow speed calculation to log file 2023-01-17 16:06:41 +00:00
Pavel Odintsov
1f4ac15d68 Better integer formats 2023-01-16 18:23:55 +00:00
Pavel Odintsov
b6096d715d Get rid of gettimeofday in screen draw function 2023-01-16 18:19:22 +00:00
Pavel Odintsov
704793b2bd Get rid of gettimeofday in Graphite logic 2023-01-16 18:13:59 +00:00
Pavel Odintsov
485b5e8e04 Reworked speed reporting calculation to monotonic time 2023-01-16 18:08:31 +00:00
Pavel Odintsov
5de0a3d579 Switched sleep duration calculation to monotonic time source to address outages during time adjustments. Related #954 2023-01-16 17:48:44 +00:00
Pavel Odintsov
a9f3837ed0 We do not need libatomic1 on x86_64 builds. Even if we need it then it's availble in our custom gcc: /opt/fastnetmon-community/libraries/gcc_12_1_0/lib64/libatomic.a 2023-01-16 13:45:59 +00:00
Pavel Odintsov
81db74bf00 Addressed issue with protoc search when we have custom libraries in place 2023-01-16 13:43:14 +00:00
Pavel Odintsov
2a82be1dd6 Merged ENABLE_CUSTOM_BOOST_BUILD with DO_NOT_USE_SYSTEM_LIBRARIES_FOR_BUILD as both of them have same meaning to simplify build procedure 2023-01-16 13:30:10 +00:00
Pavel Odintsov
385c6968eb We've switched configuration to use system libraries by default 2023-01-16 13:29:13 +00:00
Pavel Odintsov
6e932c269e We've switched configuration to use system libraries by default 2023-01-16 13:27:30 +00:00
Pavel Odintsov
7415273538 Get rid of long RPATH line and switched to shorter declarations 2023-01-16 13:19:58 +00:00
Pavel Odintsov
e8e6db2f8f Switched cmake to use system libraries for build procedure to offer better developer experience 2023-01-16 13:10:27 +00:00