IPFIX export

For Clickstream data analisys (subscribers' http requests) and SIP (VOIP unciphered data) on external systems IPFIX export is available.

A list of the correspondence between the Protocol and the port number in netfow5 can be found here.

Any universal IPFIX collector that accepts templates or the IPFIX Receiver utility is suitable for collecting information in IPFIX format.

To receive, process and store ClickStream, we suggest using the QoE Store software and DPIUI2 graphical interface.

If the link quality between SSG and NetFlow/IPFIX collector is insufficient, SSG skips sending some statistics to save performance. A message is displayed in fastdpi_alert.log when a chunk of information is skipped:

[NFLW] very long operation ….

Starting from version 12.0, the statistics for sending NetFlow/IPFIX information is now available (additional section in fastdpi_stat.log):

    [STAT    ][2022/11/20-17:55:03:213770] Statistics on NFLW_export : {a/b/c%/d/e}

    a - number of sending cycles
    b - number of sending cycles, when the time spent on sending exceeded the cycle execution period
    c - percentage of exceeding the number of sending cycles: 100 * b/a
    d - time of maximum sending cycle duration, microseconds
    e - time of the period of sending statistics, microseconds (''netflow_timeout'' parameter value (the parameter is set in seconds)).

    Example:
    [STAT    ][2022/11/20-17:55:03:213770] Statistics on NFLW_export : {7/0/0.00%/45297us/30008163us}

Clickstream experts is configured by following parameters:

ipfix_dev=em1
ipfix_udp_collectors=1.2.3.4:1500,1.2.3.5:1501
ipfix_tcp_collectors=1.2.3.6:9418
dbg_log_mask=0x80

here

• em1 - NIC using for export.
• ipfix_udp_collectors - IP of udp collectors.
• ipfix_tcp_collectors - IP of tcp collectors.
• dbg_log_mask=0x80 - logging statistics about export.

The format of IPFIX templates for IPV6 differs only in the IP_SOURCE and IP_DESTINATION fields.

ND:

• LOCKED = 1 - blocked by HTTPS, 2 - HTTP redirect, 3 - blocked by HTTP (transmitted by bitmask)
• HOST TYPE = 1 in case of HTTP, 2 - CNAME, 3 - SNI, 4 - QUIC
• METHOD = 1 - GET, 2 - POST, 3 - PUT, 4 - DELETE

If the configuration parameter "http_parse_reply=1" is enabled, information from responses to requests will be additionally transmitted. You can associate them with responses by the session identifier SESSION_ID, taking into account the order.

If the configuration parameter "ssl_parse_reply=1" is enabled, information from responses to requests will be additionally transmitted. You can associate them with responses by the session identifier SESSION_ID, taking into account the order.

Export of metadata of other protocols for SORM is configured by the following parameters

ipfix_dev=em1
ipfix_meta_udp_collectors=1.2.3.4:1500,1.2.3.5:1501
ipfix_meta_tcp_collectors=1.2.3.6:9418
dbg_log_mask=0x80

where

• em1 - network interface name for export
  
• ipfix_meta_udp_collectors - udp addresses of collectors
  
• ipfix_meta_tcp_collectors - tcp addresses of collectors
  
• dbg_log_mask=0x80 - output of statistical information about export to the log

Notes:
IP_SRC --- IP SOURCE
IP_DST --- IP DESTINATION
GATEWAYS --- comma separated list of gateways (IP or hostname)

Note: the MODE field contains the FTP connection type 0 --- active, 1 --- passive

Note: the IM_PROTOCOL field contains the type of protocol used: 0 --- ICQ, 7 --- XMPP, 106 --- ZELLO

Note: the EVENT field indicates the event type 1 --- send, 2 --- receive,
ATTACHMENT sign of an attachment, mail_protocol = 0 --- smtp, 1 --- pop3, 2 --- imap

Note:

• FLW_DIR --- direction of packet on interfaces : 0 : subs -→ inet, 1 : inet -→ subs
  
• DIR_DATA --- direction of the packet by session: for TCP 0 : client -→ server, 1 : server -→ client, for UDP --- from whom the first packet was recorded, he is considered the client
  
• VDPI_PROTO --- protocol that defined dpi
  
• META_PROTO --- internal protocol identifier (3 --- SIP, 4 --- FTP, 5 --- SMTP, 6 --- POP3, 7 --- IMAP, 8 --- XMPP, 9 --- ICQ, 10 --- RSS, 11 --- NNTP, 12 --- H323, 13 --- ZELLO)
  
• RAW_DATA --- raw data

Aggregating raw_data, clickstream, http_reply and ssl_reply with session data requires additional processing or executing a database query with the session_id key, or support in the rcollector utility.

DNS export is configured with the following settings:

ipfix_dev=em1
ipfix_dns_udp_collectors=1.2.3.4:1234
ipfix_dns_tcp_collectors=1.2.3.6:4567

where

• em1 --- the name of the network interface to export.
  
• ipfix_dns_udp_collectors --- UDP addresses of collectors.
  
• ipfix_dns_tcp_collectors --- TCP collector addresses.
  

The format of IPFIX templates for IPV6 differs in the format of the IP_SOURCE and IP_DESTINATION fields.

An alternative is to save the data in a local text log. Parameters:

• ajb_save_dns - flag for writing to a text file
• ajb_dns_ftimeout - timeout (minutes) for switching to the next file
• ajb_dns_bufsize - file write buffer
• ajb_dns_fsize - file size limit
• ajb_dns_path- path where to write

Switching to the next file occurs when the file size reaches ajb_dns_fsize or the file is not empty and ajb_dns_ftimeout has passed

ajb_save_dns_format : format for writing to a text file

• "ts" - time
• "ipsrc" - ip source
• "ipdst" - ip destination
• "ssid" - session id
• "login" - understandable

• "host" - the name of which the information was requested
• "rrtype" - RR types
• "rrclass" - RR class
• "ttl" - TTL
• "rdlen" - rdata size
• "rdata" - the resource itself
• "psrc" - port source
• "pdst" - port destination
• "transport" - how the DNS query was received.

Default: ts:ssid:login:ipsrc:ipdst:psrc:pdst:transport:host:rrtype:rrclass:ttl:rdlen:rdata