====== Configuring Clickstream, Meta data, DNS export in IPFIX ======
{{indexmenu_n>3}}
For Clickstream data analisys (subscribers' http requests) and SIP (VOIP unciphered data) on external systems IPFIX export is available.
A list of the correspondence between the Protocol and the port number in netfow5 can be found __[[dpi:dpi_options:opt_statistics:statistics_info|here]]__.
Any universal IPFIX collector that accepts templates or the __[[dpi:dpi_components:utilities:ipfixreceiver2|IPFIX Receiver]]__ utility is suitable for collecting information in IPFIX format.
To receive, process and store ClickStream, we suggest using the __[[dpi:dpi_components:qoestor|QoE Store software]]__ and __[[dpi:dpi_components:dpiui|DPIUI2 graphical interface]]__.
If the link quality between SSG and NetFlow/IPFIX collector is insufficient, SSG skips sending some statistics to save performance. A message is displayed in ''fastdpi_alert.log'' when a chunk of information is skipped:
[NFLW] very long operation ….
Starting from version 12.0, the statistics for sending NetFlow/IPFIX information is now available (additional section in fastdpi_stat.log):
[STAT ][2022/11/20-17:55:03:213770] Statistics on NFLW_export : {a/b/c%/d/e}
a - number of sending cycles
b - number of sending cycles, when the time spent on sending exceeded the cycle execution period
c - percentage of exceeding the number of sending cycles: 100 * b/a
d - time of maximum sending cycle duration, microseconds
e - time of the period of sending statistics, microseconds (''netflow_timeout'' parameter value (the parameter is set in seconds)).
Example:
[STAT ][2022/11/20-17:55:03:213770] Statistics on NFLW_export : {7/0/0.00%/45297us/30008163us}
===== ClickStream export Setup =====
Clickstream experts is configured by following parameters:
ipfix_dev=em1
ipfix_udp_collectors=1.2.3.4:1500,1.2.3.5:1501
ipfix_tcp_collectors=1.2.3.6:9418
dbg_log_mask=0x80
here
* **''em1''** — NIC using for export.
* **''ipfix_udp_collectors''** — IP of udp collectors.
* **''ipfix_tcp_collectors''** — IP of tcp collectors.
* **''dbg_log_mask=0x80''** — logging statistics about export.
==== IPFIX format template for Clickstream ====
The format of IPFIX templates for IPV6 differs only in the **IP_SOURCE** and **IP_DESTINATION** fields.
^ № ^ Size in bytes ^ Type ^ IANA ^ Description ^ Note ^
| 1003 | 16 | IPv6 | 43823 |IP_SOURCE |Sender address|
| 1004 | 16 | IPv6 | 43823 |IP_DESTINATION |Recipient address|
^ IPFIX format template for Clickstream ^^^^^^
^ № ^ Size in bytes ^ Type ^ IANA ^ Description ^ Note ^
| 1001 | 4 | int32 | 43823 |TIME_STAMP|
| 1002 | - | string | 43823 |LOGIN|
| 1003 | 4 | IPv4 | 43823 |IP_SOURCE|Sender address|
| 1004 | 4 | IPv4 | 43823 |IP_DESTINATION|Recipient address|
| 1005 | - | string | 43823 |HOSTNAME/CNAME|
| 1006 | - | string | 43823 |PATH|
| 1007 | - | string | 43823 |REFER|
| 1008 | - | string | 43823 |USER_AGENT|
| 1009 | - | string | 43823 |COOCKIE|
| 2000 | 8 | int64 | 43823 |SESSION_ID|
| 1010 | 8 | int64 | 43823 |LOCKED|
| 1011 | 1 | int8 | 43823 |HOST_TYPE|
| 1012 | 1 | int8 | 43823 |METHOD|
| 1013 | 2 | int16 | 43823 |PORT_SOURCE| Sender port |
| 1014 | 2 | int16 | 43823 |PORT_DESTINATION| Recipient port |
| 2016 | 2 | int16 | 43823 |BRIDGE_CHANNEL_NUM|Channel number (vchannel) or bridge. If vchannel is configured in the DPI configuration, then the channel number will be transmitted, otherwise the bridge number. Used in QoEStor. |
| 1024 | 2 | int16 | 43823 |CipherSuitesLen|Size in bytes of the set of available CipherSuites encryption methods in the Client Hello message|
| 1025 | - | raw | 43823 |CipherSuites|CipherSuites array in Client Hello (max 16 values)|
| 58 | 2 | int16 | - |VlanId|VLAN|
| 59 | 2 | int16 | - |postVlanID|POST VLAN|
| 56 | 6 | mac_address | - |Source MAC Address|
| 57 | 6 | mac_adress | - |Destination MAC Address|
| 2017 | - | raw | 43823 |MPLS Labels|
| 2018 | 4 | int32 | 43823 |TCP Sequence|
**ND:**
* LOCKED = 1 — blocked by HTTPS, 2 — HTTP redirect, 3 — blocked by HTTP (transmitted by bitmask)
* HOST TYPE = 1 in case of HTTP, 2 — CNAME, 3 — SNI, 4 — QUIC
* METHOD = 1 — GET, 2 — POST, 3 — PUT, 4 — DELETE
If the configuration parameter ''http_parse_reply=1'' is enabled, information from responses to requests will be additionally transmitted. You can associate them with responses by the session identifier **SESSION_ID**, taking into account the order.
^ Clickstream export template IPFIX format for HTTP responses((for the IPv6 variant see difference above)) ^^^^^^
^ № ^ Size in bytes ^ Type ^ IANA ^ Description ^ Note ^
| 1001 | 4 | int32 | 43823 |TIME_STAMP|
| 1002 | - | string | 43823 |LOGIN|
| 1003 | 4 | IPv4 | 43823 |IP_SOURCE|
| 1004 | 4 | IPv4 | 43823 |IP_DESTINATION|
| 1020 | 4 | int32 | 43823 |RESULT_CODE|
| 1021 | 8 | int64 | 43823 |CONTENT_LENGTH|
| 1022 | - | string | 43823 |CONTENT_TYPE|
| 2000 | 8 | int64 | 43823 |SESSION_ID|
| 1023 | - | string | 43823 |LOCATION|
| 2016 | 2 | int16 | 43823 |BRIDGE_CHANNEL_NUM| Channel (vchannel) or bridge number. If vchannel is set in the DPI configuration, the channel number will be transmitted, otherwise the bridge number will be transmitted|
| 58 | 2 | int16 | - |VlanId|VLAN|
| 59 | 2 | int16 | - |postVlanID|POST VLAN|
| 56 | 6 | mac_address | - |Source MAC Address|
| 57 | 6 | mac_adress | - |Destination MAC Address|
| 2017 | - | raw | 43823 |MPLS Labels|
If the configuration parameter ''ssl_parse_reply=1'' is enabled, information from responses to requests will be additionally transmitted. You can associate them with responses by the session identifier **SESSION_ID**, taking into account the order.
^ Clickstream export template IPFIX format for responses over SSL/TLS, HTTPS((for the IPv6 variant, see difference above)) ^^^^^^
^ № ^ Size in bytes ^ Type ^ IANA ^ Description ^ Note ^
| 1001 | 4 | int32 | 43823 |TIME_STAMP|
| 1002 | - | string | 43823 |LOGIN|
| 1003 | 4 | IPv4 | 43823 |IP_SOURCE|
| 1004 | 4 | IPv4 | 43823 |IP_DESTINATION|
| 2000 | 8 | int64 | 43823 |SESSION_ID|
| 1030 | 2 | int16 | 43823 |SSL_VERSION|
| 1031 | 2 | int16 | 43823 |CIPHER_SUITE|
| 1032 | 1 | int8 | 43823 |COMPRESSION_METHOD|
| 2016 | 2 | int16 | 43823 |BRIDGE_CHANNEL_NUM| Channel (vchannel) or bridge number. If vchannel is set in the DPI configuration, the channel number will be transmitted, otherwise the bridge number will be transmitted|
| 58 | 2 | int16 | - |VlanId|VLAN|
| 59 | 2 | int16 | - |postVlanID|POST VLAN|
| 56 | 6 | mac_address | - |Source MAC Address|
| 57 | 6 | mac_adress | - |Destination MAC Address|
| 2017 | - | raw | 43823 |MPLS Labels|
| 1011 | 1 | int8 | 43823 | type_host |
| 1005 | - | string | 43823 | cname |
===== Metadata Export Setting =====
Export of metadata of other protocols for SORM is configured by the following parameters
ipfix_dev=em1
ipfix_meta_udp_collectors=1.2.3.4:1500,1.2.3.5:1501
ipfix_meta_tcp_collectors=1.2.3.6:9418
dbg_log_mask=0x80
where
* **''em1''** — network interface name for export\\
* **''ipfix_meta_udp_collectors''** — udp addresses of collectors\\
* **''ipfix_meta_tcp_collectors''** — tcp addresses of collectors\\
* **''dbg_log_mask=0x80''** — output of statistical information about export to the log
==== IPFIX metadata export template formats ====
^ SIP metadata export template IPFIX format ^^^^^^
^ № ^ Size in bytes ^ Type ^ IANA ^ Description ^ Note ^
| 1001 | 4 | int32 | 43823 | TIME_STAMP |
| 1002 | - | string | 43823 | LOGIN |
| 1003 | 4 | IPv4 | 43823 | IP_SRC | Sender's address |
| 1004 | 4 | IPv4 | 43823 | IP_DST | Recipient's address |
| 2000 | 8 | int64 | 43823 | SESSION_ID |
| 3000 | - | string | 43823 | MSG_CODE |
| 3001 | 2 | int16 | 43823 | STATUS_CODE |
| 3002 | - | string | 43823 | [[https://en.wikipedia.org/wiki/Uniform_Resource_Identifier|URI]]| Uniform Resource Identifier |
| 3003 | - | string | 43823 | FROM|
| 3004 | - | string | 43823 | TO|
| 3005 | - | string | 43823 | [[https://en.wikipedia.org/wiki/Caller_ID|CALLID]]|
| 3006 | - | string | 43823 | [[https://en.wikipedia.org/wiki/User_agent|UAGENT]]| Client application|
| 3007 | - | string | 43823 | [[https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type|CTYPE]]| Type of content to be transmitted |
| 3008 | - | string | 43823 | GATEWAYS |
| 58 | 2 | int16 | - |VlanId|VLAN|
| 59 | 2 | int16 | - |postVlanID|POST VLAN|
| 56 | 6 | mac_address | - |Source MAC Address|
| 57 | 6 | mac_adress | - |Destination MAC Address|
| 2017 | - | raw | 43823 |MPLS Labels|
**Notes:** \\
**IP_SRC** — IP SOURCE\\
**IP_DST** — IP DESTINATION\\
**GATEWAYS** — comma separated list of gateways (IP or hostname)
^ FTP Metadata Export Template IPFIX Format ^^^^^^
^ № ^ Size in bytes ^ Type ^ IANA ^ Description ^ Note ^
| 1001 | 4 | int32 | 43823 | TIME_STAMP |
| 1002 | - | string | 43823 | LOGIN |
| 1003 | 4 | IPv4 | 43823 | IP_SRC | Sender's address |
| 1004 | 4 | IPv4 | 43823 | IP_DST | Recipient's address |
| 2000 | 8 | int64 | 43823 | SESSION_ID |
| 3050 | - | string | 43823 | SERVER_NAME |
| 3051 | - | string | 43823 | USER|
| 3052 | - | string | 43823 | PASSWORD |
| 3053 | 1 | int8 | 43823 | MODE |
| 1020 | 4 | int32 | 43823 |RESULT_CODE|
| 58 | 2 | int16 | - |VlanId|VLAN|
| 59 | 2 | int16 | - |postVlanID|POST VLAN|
| 56 | 6 | mac_address | - |Source MAC Address|
| 57 | 6 | mac_adress | - |Destination MAC Address|
| 2017 | - | raw | 43823 |MPLS Labels|
**Note:** the MODE field contains the FTP connection type 0 — active, 1 — passive
^ Messenger Metadata Export Template IPFIX Format (XMPP) ^^^^^^
^ № ^ Size in bytes ^ Type ^ IANA ^ Description ^ Note ^
| 1001 | 4 | int32 | 43823 | TIME_STAMP |
| 1002 | - | string | 43823 | LOGIN |
| 1003 | 4 | IPv4 | 43823 | IP_SRC | Sender's address |
| 1004 | 4 | IPv4 | 43823 | IP_DST | Recipient's address |
| 2000 | 8 | int64 | 43823 | SESSION_ID |
| 3100 | - | string | 43823 | IM_LOGIN |
| 3101 | - | string | 43823 | IM_PASSW |
| 3102 | - | string | 43823 | IM_SCREEN_NAME |
| 3103 | - | string | 43823 | IM_UIN | Universal Internet number |
| 3104 | 1 | int8 | 43823 | IM_PROTOCOL | Type of protocol used |
| 3105 | - | string | 43823 | IM_RECEIVERS |
| 1020 | 4 | int32 | 43823 | RESULT_CODE |
| 58 | 2 | int16 | - |VlanId|VLAN|
| 59 | 2 | int16 | - |postVlanID|POST VLAN|
| 56 | 6 | mac_address | - |Source MAC Address|
| 57 | 6 | mac_adress | - |Destination MAC Address|
| 2017 | - | raw | 43823 |MPLS Labels|
**Note:** the IM_PROTOCOL field contains the type of protocol used: 0 — ICQ, 7 — XMPP, 106 — ZELLO
^ IPFIX format of mail protocol metadata export template (POP, IMAP, SMTP) ^^^^^^
^ № ^ Size in bytes ^ Type ^ IANA ^ Description ^ Note ^
| 1001 | 4 | int32 | 43823 | TIME_STAMP |
| 1002 | - | string | 43823 | LOGIN |
| 1003 | 4 | IPv4 | 43823 | IP_SRC | Sender's address |
| 1004 | 4 | IPv4 | 43823 | IP_DST | Recipient's address |
| 2000 | 8 | int64 | 43823 | SESSION_ID |
| 3150 | - | string | 43823 | MAIL_SENDER |
| 3151 | - | string | 43823 | MAIL_RECEIVER |
| 3152 | - | string | 43823 | MAIL_CC | Recipient of the copy |
| 3153 | - | string | 43823 | MAIL_SUBJECT |
| 3154 | - | string | 43823 | MAIL_SERVERS |
| 3155 | - | string | 43823 | MAIL_REPLY |
| 3156 | 1 | int8 | 43823 | EVENT | Event type |
| 3157 | 1 | int8 | 43823 | ATTACHMENT | Indication of attachment |
| 3158 | 1 | int8 | 43823 | MAIL_PROTOCOL |
| 1020 | 4 | int32 | 43823 | RESULT_CODE |
| 58 | 2 | int16 | - |VlanId| VLAN |
| 59 | 2 | int16 | - |postVlanID| POST VLAN |
| 56 | 6 | mac_address | - | Source MAC Address |
| 57 | 6 | mac_adress | - | Destination MAC Address |
| 2017 | - | raw | 43823 | MPLS Labels |
**Note:** the EVENT field indicates the event type 1 — send, 2 — receive, \\
ATTACHMENT sign of an attachment, mail_protocol = 0 — smtp, 1 — pop3, 2 — imap
^ The raw unparsed metadata export template IPFIX format ^^^^^^
^ № ^ Size in bytes ^ Type ^ IANA ^ Description ^ Note ^
| 1001 | 4 | int32 | 43823 | TIME_STAMP |
| 1002 | - | string | 43823 | LOGIN |
| 1003 | 4 | IPv4 | 43823 | IP_SRC | Sender's address |
| 1004 | 4 | IPv4 | 43823 | IP_DST | Recipient's address |
| 2000 | 8 | int64 | 43823 | SESSION_ID |
| 2013 | 1 | int8 | 43823 | FLW_DIR | Directing the packet across interfaces |
| 2014 | 1 | int8 | 43823 | DIR_DATA | Forwarding a packet by session |
| 2015 | 2 | int16 | 43823 | VDPI_PROTO | The protocol that determined the DPI |
| 2900 | 2 | int16 | 43823 | META_PROTO | Internal protocol identifier |
| 2901 | - | string | 43823 | RAW_DATA |
| 4 | 1 | int8 | - | protocolIdentifier | PROTOCOL |
| 7 | 2 | int16 | - | sourceTransportPort |
| 11 | 2 | int16 | - | destinationTransportPort |
| 6 | 2 | int16 | - | tcpControlBits |
| 2018 | 4 | int32 | - | TCP Sequence |
| 58 | 2 | int16 | - |VlanId|VLAN|
| 59 | 2 | int16 | - |postVlanID|POST VLAN|
| 56 | 6 | mac_address | - |Source MAC Address|
| 57 | 6 | mac_adress | - |Destination MAC Address|
| 2017 | - | raw | 43823 |MPLS Labels|
**Note:**
* **''FLW_DIR''** — direction of packet on interfaces : 0 : subs → inet, 1 : inet → subs \\
* **''DIR_DATA''** — direction of the packet by session: for TCP 0 : client → server, 1 : server → client, for UDP — from whom the first packet was recorded, he is considered the client\\
* **''VDPI_PROTO''** — protocol that defined DPI\\
* **''META_PROTO''** — internal protocol identifier (3 — SIP, 4 — FTP, 5 — SMTP, 6 — POP3, 7 — IMAP, 8 — XMPP, 9 — ICQ, 10 — RSS, 11 — NNTP, 12 — H323, 13 — ZELLO)\\
* **''RAW_DATA''** — raw data
Aggregating ''raw_data'', ''clickstream'', ''http_reply'' and ''ssl_reply'' with session data requires additional processing or executing a database query with the ''session_id'' key, or support in the ''rcollector'' utility.
=====DNS=====
DNS export is configured with the following settings:
ipfix_dev=em1
ipfix_dns_udp_collectors=1.2.3.4:1234
ipfix_dns_tcp_collectors=1.2.3.6:4567
where
* **''em1''** — the name of the network interface to export.\\
* **''ipfix_dns_udp_collectors''** — UDP addresses of collectors.\\
* **''ipfix_dns_tcp_collectors''** — TCP collector addresses.\\
The format of IPFIX templates for IPV6 differs in the format of the ''IP_SOURCE'' and ''IP_DESTINATION'' fields.
^ № ^ Number of bytes ^ Data type ^ IANA ^ Description ^ Note ^
| 1103 | 16 | IPv6 | 43823 | IP_SOURCE | Sender's address |
| 1104 | 16 | IPv6 | 43823 | IP_DESTINATION | Recipient's address |
^ DNS Export Template IPFIX Format ^^^^^^
^ № ^ Number of bytes ^ Data type ^ IANA ^ Description ^ Note ^
| 1001 | 4 | int32 | 43823 | TIME_STAMP | Timestamp |
| 1002 | - | string | 43823 | LOGIN | Log in |
| 1003 | 4 | IPv4 | 43823 | IP_SOURCE | Sender's address |
| 1004 | 4 | IPv4 | 43823 | IP_DESTINATION | Recipient's address |
| 1013 | 2 | int16 | 43823 | SOURCE PORT | |
| 1014 | 2 | int16 | 43823 | DESTINATION PORT | |
| 2000 | 8 | int64 | 43823 | SESSION_ID | Session ID |
| 3200 | 1 | int8 | 43823 | UDP/TCP | Transport: 0 --- UDP, 1 --- TCP |
| 3201 | - | string | 43823 | DOMAIN | |
| 3202 | 2 | int16 | 43823 | RRCLASS | |
| 3203 | 2 | int16 | 43823 | RRTYPE | |
| 3204 | 4 | int32 | 43823 | TTL | |
| 3205 | - | raw | 43823 | RDATA | |
| 58 | 2 | int16 | - | VlanId | VLAN |
| 59 | 2 | int16 | - | postVlanID | POST VLAN |
| 56 | 6 | mac_address | - | Source MAC Address ||
| 57 | 6 | mac_adress | - | Destination MAC Address ||
| 2017 | - | raw | 43823 | MPLS Labels ||
| 2016 | 2 | int16 | 43823 | BRIDGE_CHANNEL_NUM | Channel (vchannel) or bridge number. If vchannel is set in the DPI configuration, the channel number will be transmitted, otherwise the bridge number will be transmitted |
An alternative is to save the data in a local text log. Parameters:
* **''ajb_save_dns''** — flag for writing to a text file\\ **''ajb_save_dns=2''** allows you to enable sending DNS queries via IPFIX
* **''ajb_dns_ftimeout''** — timeout (minutes) for switching to the next file
* **''ajb_dns_bufsize''** — file write buffer
* **''ajb_dns_fsize''** — file size limit
* **''ajb_dns_path''** — path where to write
Switching to the next file occurs when the file size reaches ''ajb_dns_fsize'' or the file is not empty and ''ajb_dns_ftimeout'' has passed
ajb_save_dns_format : format for writing to a text file
* **''ts''** - time
* **''ipsrc''** — ip source
* **''ipdst''** — ip destination
* **''ssid''** — session id
* **''login''** — understandable
* **''host''** — the name of which the information was requested
* **''rrtype''** — RR types
* **''rrclass''** — RR class
* **''ttl''** — TTL
* **''rdlen''** — rdata size
* **''rdata''** — the resource itself
* **''psrc''** — port source
* **''pdst''** — port destination
* **''transport''** — how the DNS query was received.
Default: ''ts:ssid:login:ipsrc:ipdst:psrc:pdst:transport:host:rrtype:rrclass:ttl:rdlen:rdata''
=====Sending Template in IPFIX=====
- Transport protocol TCP.\\ The Template is sent once after the TCP session is established.
- Transport protocol UDP.\\ The Template is sent by default every 20 seconds. This is controlled by the ''ipfix_udp_template_timer'' parameter.