FastNetMon Advanced 2.0.369

Changes:

• Fixed bug with endpoints total_traffic_counters_v4 and total_traffic_counters_v6 which did not work and required not needed parameters

FastNetMon Advanced 2.0.368

Changes:

• Complete multi user support for API via users_configuration and roles_configuration sections
• Improved log messages Traffic buffer size for IPv6 is too small to accommodate whole traffic calculation period. Please increase traffic buffer size to add more details
• Added logic to use source_asn and destination_asn for filtering for Flow Spec mode and Flow Spec white list
• Added source_asns and destination_asns to JSON representation of Flow Spec announces
• Added logic to return ipv4_nexthops for sudo fcli show flowspec
• Added new option for AF_PACKET to unpack GTPv1 tunnels: af_packet_extract_gtp_v1_tunnels
• Implemented API for sudo fcli show blackhole uuid
• Switched protocol version to lowercase in blackhole details
• Added command sudo fcli show blackhole UUID to show basic information about attack
• Behaviour change: removed legacy fields attack_type, initial_attack_power, peak_attack_power, attack_protocol, attack_direction for attack callbacks, MongoDB and POST callbacks
• Behaviour change: removed legacy fields in attack notifications average_incoming_traffic, average_incoming_traffic_bits, average_outgoing_traffic, average_outgoing_traffic_bits, average_incoming_pps, average_outgoing_pps, average_incoming_flows, average_outgoing_flows
• Removed legacy fields attack_direction and attack_power from attack details information
• Behaviour change: we reworked logic for attack direction and attack power in PPS arguments for legacy Bash callback scripts to use modern approach for calculations
• Added logic to calculate number of flows we keep in storage for TCP, UDP, ICMP and other protocols and exposed them via system counters
• Added new metrics for TCP, UDP, ICMP flow tracking structures
• Added new counters entries_flow_tracking_tcp_ipv4, entries_flow_tracking_udp_ipv4, entries_flow_tracking_icmp_ipv4
• Renamed counter entries_flow_tracking to entries_flow_tracking_ipv4
• Added support for new forwardingStatus 4 byte encoding which is used by Cisco ASR9006 with IOS XR 6.4.2

FastNetMon Advanced 2.0.367

Changes:

• Added new type string for system_counters and exposed FastNetMon version
• Added logic to store unban actions to attacks collection in MongoDB
• Fixed bug that clickhouse_metrics_push_period was used in Graphite plugin instead of correct graphite_push_period
• Change in BGP Flow Spec mitigation logic: DF (don't fragment) can be combined with other fragmentations flags. Previously if we had DF flag then we set only it and ignored all other fragmentation flags
• Added export for ip_dont_fragment, ip_more_fragments, ip_fragment_offset fields for packet dump
• Added logic to remove source and destination ports when flow spec rules has is-fragment flag. We do it for Arista due to their hardware limitations. It can be enabled using flag flow_spec_strip_ports_for_fragmented_traffic
• Deprecated field attack_protocol for JSON callbacks and explicitly set it to unknown value. Please use threshold names for attack protocol detection instead
• Deprecated fields peak_attack_power and initial_attack_power for JSON callback and set them to zeroes. Please use per protocol counters instead
• Manually set field attack_type for JSON based attack notifications to unknown as this logic was broken and we're deprecating it
• We stopped populating fields: average_incoming_traffic, average_incoming_traffic_bits, average_outgoing_traffic, average_outgoing_traffic_bits, average_incoming_pps, average_outgoing_pps, average_incoming_flows, average_outgoing_flows
• Removed deprecated attack detection fields from email alerts: Attack type, Initial attack power, Peak attack power, Attack direction, Attack protocol. These fields were calculated by very flawed logic and we replaced them by new fields which precisely reflect reality of attack.
• Added configuration options gobgp_as_path_host_ipv4, gobgp_as_path_subnet_ipv4, gobgp_as_path_host_ipv6, gobgp_as_path_subnet_ipv6
• Added bgp_as_path_host_ipv4, bgp_as_path_subnet_ipv4, bgp_as_path_host_ipv6, bgp_as_path_subnet_ipv6 for all hostgroups
• Improved logic to craft IPv4 BGP attributes
• We deprecated Ubuntu 16.04, Debian 9 and 10 and removed them from releases
• Fixed bug with too high values in sudo fcli show baseline_per_host global outgoing for outgoing traffic
• Adjusted Clickhouse datasource UID for password reset logic
• Changed Clickhouse datasource to new UUID format
• Added logic to set correct owner for /var/lib/grafana/plugins
• Added fcli flag NO_DATABASE_MODE which can be set to on to suppress connection attempts to MongoDB

FastNetMon Advanced 2.0.366

Changes:

• Adding option to pcap reader to load networks list from /tmp/networks_list_pcap.dat
• Added cron installation for Panel
• Added counter to track UDP packets for Netflow or IPFIX plugin which exceed 1500 bytes
• Switched Nginx signature to binary format
• Changed the way how we sign key for Ubuntu and Debian for Nginx
• Added explicit checks that repos work on Debian platforms
• Added logic to verify that MongoDB repo works fine on Ubuntu and Debian
• Added explicit check that FastNetMon and Grafana repos fro Debian and Ubuntu work fine before installing anything from them
• Added logic to check that FastNetMon repo works before installing from it
• Added logic to explicitly check that Nginx repo works
• Finished logic to set password in Grafana and for Nginx http auth in same time
• Moved to new logic which just replicates password of Grafana user on Nginx auth as old scheme was broken around Grafana 11 and we weren't able to pin point issue
• Added logic to pass IPv6 address in Netflow and IPFIX plugin. Also improve logic to use IPv6 address as is in pcap readers
• Deprecated visual stack for Ubuntu 16.04 because Clickhouse is not working on it
• Switched MongoDB to FerretDB for Ubuntu 16.04
• Switched to use FerretDB on Debian and Ubuntu and RedHat machines without AVX or when old CPU is forced via CLI
• Added per protocol counters for asn_counters_v4 and asn_counters_v6
• Added logic to respect unit passed to total_traffic_counters using GET query bandwidth_unit set to bps
• Added logic to pass unit to FNM internal API
• Added installer flag to install only FerretDB
• Added logic to start GoBGP daemon to eliminate errors like context deadline exceeded in Panel
• Added logic to export per protocol counters for network counters and per interface counters endpoints
• Added per protocol counters for networks and interfaces
• Unified networks_counters for IPv6 per network counters
• Ported ASN counters to new per protocol counters logic
• Added per protocol counters for ASNs and for networks for internal API
• Added optional capability to use TTL for Flow spec attack detection
• Added proxy_set_header for Grafana to address origin not allowed issue
• Introduced option to calculate speed in parallel. Please note that it performs best if you have ban disabled
• Added configuration options parallel_speed_calculation and parallel_speed_calculation_threads
• Implemented previously missing logic for flow_spec_ignore_do_not_fragment_flag
• Changed permissions for systemd unit files for CentOS family from 755 to 644 as RHEL 8 does not like it: Configuration file /usr/lib/systemd/system/fastnetmon.service is marked executable. Please remove executable permission bits. Proceeding anyway
• Added flows_per_second sorter for Partner integration
• Added FerretDB support for logic to import community configuration
• Added logic to use Nginx repository from Ubuntu 22.04 Jammy for Ubuntu 24.04 as we do not have official one yet
• Added Ubuntu 24.04 support to installer

FastNetMon Advanced 2.0.365

Changes:

• Multiple improvements for licensing logic

FastNetMon Advanced 2.0.364

Changes:

• Added BGP peering configuration options ipv4_unicast_add_path and ipv6_unicast_add_path to control add path logic with 8 routes for each prefix
• When you specify single value in bgp_next_hops_subnet_ipv4, bgp_next_hops_host_ipv4, bgp_next_hops_subnet_ipv6, bgp_next_hops_host_ipv6 for hostgroup it overrides default value in configuration
• Introduced gobgp_flow_spec_v4_redirect_target_as, gobgp_flow_spec_v4_redirect_target_community, gobgp_flow_spec_v6_redirect_target_as, gobgp_flow_spec_v6_redirect_target_community to control IPv6 Flow Spec redirect
• Added warning message to log when capacity of traffic buffer is not enough to accommodate generate_hostgroup_traffic_samples_delay or generate_attack_traffic_samples_delay

FastNetMon Advanced 2.0.363

Changes:

• Migrated from Patricia tree with manual memory management to new lookup_tree_128bit_t
• Implemented complete support for gobgp_next_hop_host_ipv6 and gobgp_next_hop_subnet_ipv6 which allow settings next top for per host and per network announces independently
• Added logic to provide additional BGP communities on hostgorup basis via: bgp_communities_subnet_ipv6 and bgp_communities_host_ipv6 hostgroup options
• Added logic to provide additional BGP communities on hostgorup basis via: bgp_communities_subnet_ipv4 and bgp_communities_host_ipv4 hostgroup options
• Eliminated parent_host_group as individual field
• Renamed ban_settings_t to hostgroup_t
• Added ban_settings_t to attack_details_t
• Added clickhouse_table_name as configuration option for traffic_db
• Added clickhouse_database_name as configuration option for traffic_db
• Extracted all configuration values for traffic_db into separate class
• Add per protocol detailed counters for host_counters_per_hostgroup_v4 and host_counters_per_hostgroup_v6
• Add per protocol counters for API mode of sudo fcli show remote_host_counters
• Added per protocol counters for API mode of sudo fcli show host_counters_v6
• Added per protocol counters for sudo fcli show host_counters in API mode
• Unified function to return metrics for GetHostCountersRemote, GetHostCountersV4, GetHostCountersV6
• Unified types HostCounterRemote, HostCounterV6 to HostCounter
• Added API endpoint sudo fcli set reload_bgp to reload BGP settings without FastNetMon restart
• Extracted read_bgp_configuration from fastnetmon_configuration_parser
• Breaking change for logic used to check if BGP Flow Spec is enabled. Now to enable it you just need to set flag gobgp_flow_spec_announces. We removed logic which required presence of at least one BGP peer with BGP Flow Spec enabled
• Increased default MongoDB timeout for retrieval of hostgroups_max_talkers, hostgroup_traffic_stats, attack_traffic_stats to 96 seconds and added option to independently control it from configuration /etc/fastnetmon/fastnetmon.conf using option mongodb_timeout_heavy
• Switched all the code to use only Patricia wrappers for IPv4 lookups
• Improved design of ip_lookup_tree to hide private members
• Added method lookup_network() for ip_lookup_tree and covered by tests
• Migrated lookup_ip_in_integer_form_inpatricia_and_return_subnet_if_found to lookup_tree_ipv4.lookup_network_which_includes_ip
• Added new IP lookup method lookup_network_which_includes_ip and added tests for lookup_ip
• Moved GoBGP configuration generation into separate library
• Split one large definition of all our configuration structures in a well defined isolated classes
• Moved gobgp configuration generation to gobgp_actions
• Split fastnetmon_configuration_structures to bunch of independent files
• Split logic to generate configuration structures and readers to two different scripts
• Reworked logic to init networks_whitelist_remote by adding strict input data validation and avoid potential crashes when IPv6 prefix specified
• Reworked Patricia test to avoid using deprecated function
• Split ip_lookup_tree_with_payload to ip_lookup_tree_with_payload and ip_lookup_tree_with_dynamically_allocated_payload to provide more clarity about interfaces we use
• Extracted IP lookup logic with payload to library ip_lookup_tree_with_payload.hpp

FastNetMon Advanced 2.0.362

Changes:

• Complete logic to reload per host hostgroups without FastNetMon restart with sudo fcli set reload_hostgroups
• Moved hostgroup loading logic to fastnetmon_host_group_configuration_parser
• Hide internals of hostgroup_lookup_t by using private
• Added notes about potential options to improve flexible counters calculation
• Improved IPv6 hostgroup lookup performance by eliminating expensive memory copy
• Added explicit logic to avoid complex JSON crafting operations in max talkers, attack samplers and hostgroup sampling logic
• Added counter hostgroup_traffic_samples_calculation_time to measure time required to create per hostgroup samples
• Added counter attack_traffic_samples_calculation_time to measure time required for attack sampling procedure
• Added counter max_talkers_per_hostgroup_accumulation_time to track accumulation time as it's very lengthy too
• Switched to new logic for checking if we should block IPv4 host or not. It features up to 2.5 speed up. It was accomplished by eliminating not needed memory copy
• Added function to lookup hostgroup and check thresholds in same time
• Extracted threshold checking logic into separate file
• Extracted some generic code from fast_libraries to fastnetmon_integers, fastnetmon_networks and fastnetmon_strings
• Extracted thresholds evaluation logic into separate library
• Removed duplicate checks to skip ban action if it disabled on hostgroup basis as we have it in we_should_ban_this_entity
• We decided to do bold move and duplicate speed_calculation_callback_local_ipv4_universal into two independent functions. We have lots of condition sections which make almost half of source code for it. I think it's right move and as independent functions they will be easier to deal with
• Improved design per hostgroup lookup logic
• Exposed Sensitive flag for fields API
• Added -demo for fastnetmon_client to hide real IP addresses
• Added new option gobgp_communities_host_ipv6 which allows adding multiple communities to configuration
• Upgrade FerretDB from 1.16.0 to 1.20.1
• Added new fcli and API endpoint to return all networks which belong to particular ASN: fcli show asn_networks 269872
• Increased precision of system_counters for API from 3 to 6 digits after point
• Make XDP filter less verbose
• Added logrotate for MongoDB to address flooded logs
• Changed MongoDB configuration
• Switched logRotate configuration from rename to reopen: https://www.mongodb.com/docs/manual/reference/configuration-options/#mongodb-setting-systemLog.logRotate to make possible logrotation using Linux logrotate

FastNetMon Advanced 2.0.361

Changes:

• Added logic to create stub config file for gobgp from installer
• Enabled netflow_ipfix_inline by default and removed configuration option
• Enabled netflow_v9_lite support by default and removed configuration option netflow_v9_lite
• Exposed average_calculation_time as system_counter as we want to see it all the time
• Added option for installer flag reset_visual_passwords to set custom password passed via password flag
• Implemented complete logic for BGP Flow Spec redirect action
• Improved BGP Flow spec rate and discard encoding
• Added redirect_as and redirect_value to action section
• Added redirect and mark actions for flow spec logic
• Added 4 byte encoding of sampler id in Netflow v9 used by Huawei
• Reworked flow tracking table cleanup logic
• Reworked global flow counters and introduced counter for flow calculation logic duration
• Extracted flow calculation speed logic to separate function
• Introduced logic to calculate tcp, udp, icmp and other flows per second for each host individually
• Migrated flow tracking structure from std::map to std::unordered_map because it's 2-10 times faster.
• Extracted flow callback to be standard function
• Made internal tracking variables for flow tracking logic external to reduce lambda complexity
• Removed checks if connection tracking is enabled from lambda as we do checks outside

FastNetMon Advanced 2.0.360

Changes:

• Introduced PoC ability to calculate speed in parallel
• Added very precise profiling for each part of speed calculation process and exported it via system_counters
• Reworked logic which returns top k speed counters via API
• Added comparator class to compare elements stored in unified storage with speed and traffic
• Migrated traffic increment logic for ipv4_remote_host_counters, ipv4_network_counters, ipv6_network_counters to use unified methods
• Reworked remote IP speed retrieval logic to standard API function
• Merged data and speed counters to use single std::unordered_map element for both of them. Expected speed up for speed calculation process is around 1.74 times
• Upgraded both traffic-database-ipv4 and traffic-database-ipv6 dashboards to fix duplicate AND in SQL query
• Added capability to specify next hop for host and subnet IPv4 announces independently using gobgp_next_hop_host_ipv4 and gobgp_next_hop_subnet_ipv4
• Split blackhole logic for IPv4 and IPv6 to two different functions
• Added API queries asn_counters_bulk_v4 and asn_counters_bulk_v6 to export whole structure of ASN counters
• Added option to control number of returned via API elements using max_elements parameter for host_counters_v6, host_counters, host_counters_per_hostgroup_v4, host_counters_per_hostgroup_v6, remote_host_counters
• Added query variable to set number of entries which can be returned by asn_counters_v4 and asn_counters_v6: max_elements
• Added option to pass number of arguments via query parameter
• Added logic to install version of installed Grafana from installer log output
• Added logic to explicitly create folders for provisioned files
• Fixed bug with mixed prefix lengths for IPv4 and IPv6 in Mikrotik integration. We had /128 for IPv4 and /32 for IPv6 which is completely incorrect