M01973
2016 IEEE Region 10 Symposium (TENSYMP), Bali, Indonesia
Toward Hardware Support for a Flexible
Sketch-based Network Traffic Monitoring System
Theophilus Wellem*§, Yu-Kuen Lai¶, Chao-Yuan Huang¶, Wen-Yaw Chung*
*
Department of Electronic Engineering, ¶Department of Electrical Engineering
Chung Yuan Christian University, Zhongli 32023, Taiwan
Email: {g10202604, ylai, g10279008, eldanny}@cycu.edu.tw
§
Department of Informatics
Satya Wacana Christian University, Salatiga 50711, Indonesia
Email: [email protected]
Abstract— Sketch-based data streaming algorithms are
capable of summarizing high-speed network traffic in real-time
without packet sampling. Therefore, they are used in many
networking applications to make accurate estimates of traffic
flow statistics. However, fixed hardware implementation of the
sketch counters limits the flexibility of re-using the same data
structure for various monitoring applications. In this paper, we
propose a generic scheme for collecting flow-level data using
high-speed memories on the NetFPGA-10G platform. The
purpose is to enable sketch-based algorithms making statistics
measurements in various network monitoring applications in a
flexible manner. Experiments using real Internet traces are
conducted for entropy estimation, superspreader detection, and
change detection applications. The results show that the proposed
scheme can support flexible selection of keys and counters for the
monitoring applications achieving high measurement accuracy.
Fig. 1. Sketch-based traffic monitoring is too application-specific. Each
monitoring application/measurement task requires its own sketch data
structures in data plane.
Keywords—Network traffic monitoring; Sketch; Hash table;
Linear probing; NetFPGA-10G
algorithms on network device is not flexible since once the data
structure is implemented in hardware for a measurement task,
another monitoring application that requires different flow key
and update value cannot re-use the data structure.
I. INTRODUCTION
Sketch-based data streaming algorithms are widely used in
network traffic monitoring to obtain accurate estimates of
traffic flow metrics such as entropy [1], [2], flow size
distribution [3], heavy hitters [4], heavy changers [5], and
superspreaders [6]. Entropy of source IP and destination port,
for example, can be utilized for detecting port scan attack. In
contrast to per-flow monitoring tool such as NetFlow [7] that
uses packet sampling to reduce number of flows collected,
sketch-based algorithms employ sketch data structure to
summarize network traffic into a compact representation
without packet sampling, using only a small amount of
memory space. Because the sketch data structure is updated for
each incoming packet, its update process is the critical
bottleneck. Therefore, the sketch is implemented by using fast
memory (SRAM) in the network device line card.
Motivated by the previous works of a "minimalist"
approach for flow monitoring system [9], [10] and OpenSketch
[8], this paper presents a data plane scheme to collect network
traffic flows without packet sampling. The goal is to create a
flexible sketch-based traffic monitoring system where the data
plane can enable flexible selection of flow keys and counters
for various monitoring applications. Rather than maintaining
sketch data structure for each flow key in data plane, the
proposed data plane scheme employs two tables: flow counter
table and flow key table for collecting traffic flows. The main
contributions of this work are:
• We propose a data plane scheme for flow-level data
collection to support more flexible implementation of
sketch-based
monitoring
applications.
Various
monitoring applications have greater flexibility of
choosing which flow keys and counters to monitor and
measure.
Sketch-based data streaming algorithms are efficient and
have provable memory-accuracy tradeoffs for traffic flow
measurements. However, they are too application-specific to be
implemented to perform primitive operations on network
devices [8], [9]. Sketch-based algorithms require a separate
instance of sketch data structure for each flow key, which is
defined according to the measurement tasks of interests, as
shown in Fig. 1. Current implementation of sketch-based
978-1-5090-0931-2/16/$31.00 ©2016 IEEE
• We present the evaluation of the scheme using three
monitoring
applications,
entropy
estimation,
superspreader detection, and change detection, to show
the feasibility of the data plane scheme.
7
2016 IEEE Region 10 Symposium (TENSYMP), Bali, Indonesia
Fig. 2. Sketch data structure and its update operation.
The remainder of this paper is organized as follows. Sketch
for network traffic measurements is discussed in Section II.
Section III describes the system architecture. Hardware
implementation overview is outlined in Section IV. Section V
explains three monitoring applications used to evaluate the
system. In Section VI, the system's evaluation with tracedriven tests using real traffic traces is presented. Section VII
reviews some related works, and Section VIII concludes this
paper.
Fig. 3. System architecture.
count, while the 5-tuple flow key is stored in the flow key table,
which is implemented as a FIFO in DRAM.
II. BACKGROUND
Sketch is a data structure representing the compact
summaries of a data stream. It can be queried to obtain
statistics about the data stream. A sketch is an array of counters
C[i][ j] , 1 ≤ i ≤ d , 1 ≤ j ≤ w . It consists of d hash tables of
size w . Each row is associated to a hash function selected
from a universal class of hash functions. When an item
a = (k , v) arrives, the key k is hashed by d hash functions
and the corresponding counters are updated by v . A query of
the sketch can yield the estimated counts for the key k . For
example, query of a key to Count-Min sketch [11] returns the
minimum value among all counters that correspond to that key.
Fig. 2 shows the sketch data structure.
During an observation interval, when a flow entry is
created, its 16-bit ID and counters are stored in the flow
counter table. This 16-bit ID is obtained from hashing the 32bit flow ID (i.e., the result from TabChar hash) using a 2universal hash function. A Bloom filter is used to check
whether the 5-tuple flow key already exists in the flow key
table. If the flow does not exist, then the 5-tuple is added into
the flow key table. After the observation interval, flow data
collected in both tables are moved to the host CPU and can be
utilized as required by the monitoring applications.
The modules presented in the upper part of Fig. 3 are
implemented in software on multi-core CPU for parallel
processing of sketches, while the bottom part is implemented in
FPGA hardware. The hardware development platform used for
implementation is NetFPGA-10G [15], which contains a
Virtex-5 XC5VTX240T FPGA, four 10GbE SFP+ interfaces,
three banks of 9MB QDRII SRAM with a total capacity of
27MB, and four 72MB RLDRAM II memories. The main
hardware modules for flow processing inside the system is
depicted in Fig. 4.
To use sketch for traffic measurement, the sketch data
structure must be updated based on a predefined flow key,
which is a combination of tuples from the packet header. The
update value can be the packet size or packet count. For
example, heavy hitters detection application uses the source IP
address and packet count as its key and updated value,
respectively. In change detection application [5], the sketch is
updated using source IP address and packet size. In
superspreader detection applications [6], [12], [13], the update
process sets the bit in bitmap sketch based on source IP address
and pair of source and destination IP addresses.
IV. HARDWARE OVERVIEW
In the NetFPGA-10G platform, incoming packets from four
10GbE interfaces are collected in four ingress queues, one for
each interface. An input arbiter module then takes the packet
out from these queues in round robin policy, and sends it to the
next module in the data path. The field extractor module
extracts the 104-bit 5-tuple and 16-bit packet length from each
packet header and sends it to the hash function module.
III. SYSTEM ARCHITECTURE
The architecture for a flexible sketch-based traffic
monitoring system is shown in Fig. 3. The 104-bit 5-tuple
(source IP, destination IP, source port, destination port, and
protocol) is extracted from each packet by the header parser
module as the flow key. The hash function module then
computes a 32-bit hash value from the 104-bit flow key as the
flow ID using a tabulation-based 2-universal hash function
(TabChar). This mapping from 104-bit to 32-bit is collisionfree with high probability [14]. The 32-bit flow ID is then
hashed again using a 5-universal hash function to obtain a 32bit hash value. The lower 19-bit of this hash value is the
memory (SRAM) address where the flow's packet and byte
counts are stored. This system utilizes two tables. The flow
counter table stores the flow's 16-bit ID, packet count, and byte
Hash Function: Three hash modules are utilized to implement
the hash functions required. The first module implements the
tabulation-based 2-universal hashing (TabChar) to process the
104-bit 5-tuple flow key to a 32-bit flow ID. The second
module hashes the 32-bit flow ID using a 5-universal hash
function to obtain the memory address where the flow's packet
and byte counts are stored. This module also computes a 16-bit
ID from the 32-bit flow ID to be stored in the flow counter
table. The computation of 5-universal hash function is based on
the four degree polynomial over some prime field as follows.
8
2016 IEEE Region 10 Symposium (TENSYMP), Bali, Indonesia
k
kn
k
1
FPR = 1 − 1 − ≈ (1 − e − kn m )
m
(2)
Theoretically, for fixed m , the number of hash functions k
that is needed to minimize the false positive rate is
k = ( m n ) ln 2 . For the system herein, k is set to 3, while
various m values are utilized to observe the effect of filter size
on the measurement accuracy. The 1.42 MB on-chip Block
RAM (BRAM) is used to store the Bloom filter.
Flow Key Table: The flow key table is stored as a FIFO in
RLDRAM II. Four RLDRAM II chips are available on-board
with a total size of 288 MB. For each entry in the FIFO, it is
capable of holding a 104-bit 5-tuple flow key (13 bytes per
entry). The FIFO can contain up to 512K flow entries utilizing
only 6.5 MB of the RLDRAM space. Two RLDRAM II
controllers are used in the design, each controller connects to
two RLDRAMs. Using burst length of two, one controller can
write a 72-bit data with latency of 5 clock cycles. The latency
is depend on the RLDRAM configuration mode.
Fig. 4. Hardware modules for flow processing.
4
h( x) = ai x i mod p
i =0
(1)
CW-trick [16] is used to accelerate the 5-universal hash
computation without modulo operation. The 5-universal hash
function is implemented in a pipeline fashion and utilized
seven embedded multipliers on the Virtex-5 FPGA. The third
module calculates hash values for the Bloom filter using three
2-universal hash functions, h( x) = (ax + b) mod p . Again, to
avoid slow modulo operation, multiply, shift, and bitwise AND
operations, as in CW-trick, are used.
Hardware-Software communication: The time that is
required for the monitoring application runs on the host CPU to
read the flow data from SRAM and DRAM via the PCIe
interface is important for effective operation of the system.
Given the table size and number of flows that are processed in
the system herein, the required time is less than 10 ms.
Furthermore, the processing time on CPU side (for sketch
updates and queries) is much less than the commonly used
observation interval. Communication between hardware and
software utilizes the register infrastructure in the NetFPGA10G platform. Applications can read data from the hardware or
write data to the hardware using these registers.
Flow Counter Table: The flow counter table is implemented
using a hash table with linear probing, and a 5-universal hash
function is used to handle hash collisions with constant number
of probes per update/lookup. Using 5-universal hash function,
the average number of probes per update/lookup is a constant
less than four [14], allowing fast and deterministic update
operation of the flow counters. The flow counter table is stored
in the QDRII SRAM and hash collision is handled by
comparing the input ID to multiple entries in a bucket in
parallel. The linear probing mechanism makes use of the fourword burst access of the QDRII SRAM. A read burst retrieves
four IDs from the memory. These flow IDs are then compared
in parallel to the input ID. If the input is a new ID, a flow entry
is created and its counters are updated. Otherwise, only the
counters are modified and then written back to the memory.
Updating counters in the table requires a read-modify-write
operation for each incoming packet at wire-speed. With a flow
entry consisting of a 16-bit flow ID, a 32-bit packet counter
and a 60-bit byte counter, the flow counter table can
accommodate up to 2 million flow entries in 27MB QDRII
SRAM.
V. MONITORING APPLICATIONS
A. Entropy Estimation
Entropy is a measure of the amount of randomness of
information content in the distribution of a random variable.
Let X be a discrete random variables that can take values from
the set of [ s] = {1, 2,3, , n} , mi denotes the frequency of item
i ∈ [s] , and m = i =1 mi is the total number of items. The
n
entropy of X is defined as [17]: H ( X ) = − i =1 pi log 2 pi ,
n
where pi = mi m is the probability of X taking the value i ,
X
is defined as
and the entropy norm of
H norm ( X ) = H ( X ) log 2 m . The entropy norm value is
minimum (zero) if all items in the data stream are the same,
and is maximum ( log 2 m ) if all the items in the data stream are
distinct [1]. In traffic monitoring context, the distributions of
traffic attributes (e.g. distribution of source IP address,
destination IP address, and ports) are important for network
monitoring system, especially for fine-grained traffic analysis
and classification. Entropy is utilized as a metric for anomaly
detection. For example, entropy of source IP and destination
port can be used to detect port scan attack.
Bloom filter: A Bloom filter is used to check whether a 5-tuple
flow key is already exist in the flow key table and to prevent
the addition of a duplicated 5-tuple flow key into the flow key
table. Recall that Bloom filter allows false positive, the size of
Bloom filter and number of hash functions must be chosen
properly to reduce the false positive rate. Assume that the
Bloom filter size with m bits stores n elements, then the falsepositive rate (FPR) is
9
2016 IEEE Region 10 Symposium (TENSYMP), Bali, Indonesia
B. Superspreader Detection
Detecting traffic sources that communicate with a large
number of distinct destinations during a measurement interval
is important for detecting network anomalies. These sources
are known as superspreaders [18] or super sources [6]. For
example, an abrupt increase in the number of connections from
many distinct sources to a particular destination in a short
period, may be the evidence of a distributed denial-of-service
(DDoS) attack. Conversely, detecting destinations that
contacted by a large number of distinct sources (super
destinations) can be useful for detecting the victims of DDoS
attack. The goal of superspreader detection is to find hosts that
have a large number of connections to distinct destinations
above a predefined threshold in an observation interval. To
identify such host, the superspreader detection module needs to
count the number of connections to distinct destination (or fanout) of that host. Counting the fan-out of each host in the
network requires large memory space to store the host’s source
IP and its fan-out counter. Therefore, estimation using bitmap
and sketch-based algorithms are commonly used [6], [12], [13].
Probe length per update
Table size=512K
Number of flows
300000
Load factor = 39.69% (5s)
Load factor = 66.98% (10s)
Load factor = 85.13% (15s)
250000
200000
Table size=1M
300000
250000
200000
150000
150000
100000
100000
50000
0
Load factor = 20.29% (5s)
Load factor = 33.00% (10s)
Load factor = 42.56% (15s)
50000
0
4
8
12 16 20 24 28 32
Probe length
0
0
4
8
12
16
20
Probe length
24
28
32
Fig. 5. Number of flows vs. probe length for different table size and
observation interval. Data in 4th interval are used. The number of flows
in this interval: 5s=208k, 10s=346k, 15s=446k. For table size=512K with
10s and 15s intervals, the figure only shows the number of flows that
have probe length up to 30.
10Gbps Ethernet link of a Tier1 ISP. It comprises of 30 million
packets and 1.4 million distinct 5-tuple flows, with a duration
of 60s. The number of source-destination IP pairs and distinct
source IP addresses in the trace are 1 million and 450k,
respectively. The observation interval is set as 15s in the
experiments. In this interval, the number of distinct 5-tuple
flows is ranged from 445k to 510k. Entropy estimation,
superspreader detection and change detection applications are
run on the host CPU, and utilize the flow keys and counters
collected in both tables to do their measurement task. For
change detection, a 15-minute trace from MAWI is used [20].
It is a part of 24h-long trace in a trans-Pacific line (18 Mbps
CAR on 100 Mbps link) that is collected at MAWI's
samplepoint-B. The trace consists of 51,792 distinct source IP
addresses and 374,561 distinct source-destination IP flows.
C. Change detection
The change detection's goal is to identify flows whose
contribution to changes in size (packet counts) or changes in
volume (byte counts) of total traffic over several consecutive
observation intervals are above a predefined threshold. These
flows are also known as heavy changers. Let At = a1 , a2 , be
the input packet stream in observation interval t . Each packet
ai is associated with a key k and an update value v .
Following the algorithm in [5], the source IP address and
packet size are used as the key and update value, respectively.
Let So (t ) denotes the sketch that accumulate the byte counts in
interval t and S f (t ) is the forecast sketch computed using a
forecasting model (e.g. moving average). The forecast error
sketch is then computed as Se (t ) = So (t ) − S f (t ) . A threshold
B. Experiment Results
1) Flow Counter Table and Linear Probing: Probe length
per key for different table sizes are shown in Fig. 5. When a
512K-entry table is utilized with observation interval of 15s,
84.1% of the flows have probe length less or equal to four.
The maximum probe length of 350 is observed by one flow.
After all flows were inserted into the table, the load factor is
85.13%. For 5s and 10s observation intervals, the percentage
of flows that have probe length less or equal to four are
98.97% and 92.99%, respectively. If a table size of 1M entries
is used in 15s observation interval, up to 98.56% of the flows
have probe length less or equal to four, and the maximum
probe length decreases to 27. For the 1.44% of flows that have
probe length more than four, only one flow has the maximum
probe length of 27. The load factor after all flows were
inserted into the table is 42.56%. For 5s and 10s intervals, the
percentage of flows that have probe length less or equal to
four are 99.95% and 99.48%, respectively. These experiment
results confirmed that by using 5-universal hash function,
most of flows have probe length less or equal to four.
2) Entropy estimation: The entropy norm of 5-tuple flow,
source IP address and destination IP address are estimated
using the data from the QDRII SRAM and RLDRAM FIFO.
The results are compared to the exact entropy norm and the
estimation accuracy is measured in term of relative error. Fig.
6 shows the relative error of the entropy norm estimation for
alarm TA is chosen based on the estimated second moment of
the forecast error sketch and a threshold parameter T
determined by application ( TA = T × EstF 2 ( So (t ) ) ). The
change detection task is to find all keys (source IP addresses)
whose volume, queried from Se (t ) , is above TA .
Although only three applications above are utilized for the
system’s evaluation, other monitoring applications may also
use the flow data collected by the hardware. For example, the
flow size distribution (FSD) estimation, which is useful for
network service providers to know the usage pattern of their
networks by customers [3]. Given a flow size x , the goal of
FSD estimation is to find the number of flows that have x
packets in an observation interval. In this case, the monitoring
application can rely on the flow counter and flow key tables to
obtain flow data and estimate the FSD.
VI. EVALUATION
A. Experiment Setup
Trace-driven tests using CAIDA traces [19] are conducted
to evaluate the system. The traces were collected over a
10
2016 IEEE Region 10 Symposium (TENSYMP), Bali, Indonesia
source IP address and destination IP address. Different sizes
of Bloom filter are used in the experiments. As presented, the
system can yield estimation errors in an acceptable range for
anomaly detection. The estimation accuracy increases with the
size of Bloom filter. For Bloom filter size greater than 128
KB, the estimation errors of 5-tuple flow, source IP address,
and destination IP address are less than 2%.
3) Superspreader detection: For superspreader detection,
source IP-destination IP pairs data from the flow key table in
RLDRAM FIFO are utilized. Here, a superspreader is defined
as a source whose fan-out exceeds 200. False negative ratio
(FNR) and false positive ratio (FPR) are used to measure the
superspreader identification accuracy, while the fan-out
estimation accuracy is obtained from scatter plots. The FNR is
the number of unidentified superspreaders divided by the
number of actual superspreaders, while the FPR is number of
non-superspreaders that are incorrectly identified as
superspreaders divided by the number of actual
superspreaders. FPR and FNR of superspreader detection
based on two methods are shown in Table I. The first method
(2HashTable) maintains two hash tables. One hash table is
used to remove duplicated source IP-destination IP pairs, and
the other hash table for counting the fan-out. This method is
traditionally used for exact fan-out counting, and is very
accurate. The FNR is due to the false positive of Bloom filter
when the 5-tuple is added into the flow key table. The second
method (FM-BF-HT) [12] combines an FM sketch, a Bloom
filter, and a hash table. This method has larger FPR and FNR.
The FNR is due to the FM sketch filtering, while the FPR is
caused by hash collisions since no collision resolution is
implemented for the hash table. The accuracy of fan-out
estimation is shown in Fig. 7. The closer the points to the
diagonal line, the more accurate the estimation is. The mean
errors in the fan-out estimation for 2HashTable and FM-BFHT are 0.11% and 3.6%, respectively.
4) Change detection: The experiments using k-ary sketchbased change detection algorithm [5] with Moving Average
forecast model are based on the parameters listed as follows:
Number of hash functions H = 5 , number of sketch entries
K = 32K , Moving Average window size W = 3 , and length
of observation interval Δt = 60s . The threshold parameter T
is varied for all experiments. The detection accuracy is
measured using false positive and false negative rates. The
false positives are flow keys whose volume do not change but
are incorrectly reported, while the false negatives are flow
keys that have volume changes but are not reported.
Entropy Estimation
Relative error (%)
Source IP
Destination IP
7
7
5
5
3
3
1
0.4
1
0.4
0.2
0.2
0
1
2
3
4
0
1
Interval
Bloom filter size 128KB
256KB
2
3
4
Interval
512KB
1024KB
Bloom filter size 128KB
256KB
512KB
1024KB
Fig. 6. Entropy estimation results of source IP address and destination IP
address for different Bloom filter size.
the beginning of fifth interval, the forecast error sketch Se (t ) is
available and queried using the current interval incoming
source IP address to find the heavy changers. From this interval
onward, the change detection process reports its results in each
interval. The experiment results are reported in Table II. As
shown, the change detection application achieves high
accuracy in detecting heavy changers with very low false
positives and false negatives. As an example, in 14th interval
with T = 0.1 , the application reports 10 heavy changers and
only two of them are false negatives. For other intervals, all
heavy changers are reported correctly.
VII. RELATED WORK
NetFlow [7] is a tool developed by Cisco for traffic flow
monitoring. The flow exporter in NetFlow is responsible for
creating flow records from network traffic by capturing packet
headers. Sampling and filtering are performed before a flow
entry is created or updated in the flow cache. These flow
records are stored in the flow cache until they are considered to
have expired. When a flow record expires, it is exported to the
flow collector. The flow data collected in the flow collector are
then used for analysis. To some extent, the system presented in
this work is similar to NetFlow. However, the system only
collect the flow's 5-tuple and packet/byte counts. It also does
not use the expiration mechanism of NetFlow, and simply
moves the flow entry from the tables to host CPU at the end of
observation interval.
Liu et al. [10] developed a universal monitoring
architecture (UnivMon), where a simple and generic
monitoring primitive run on network device. UnivMon data
plane utilizes parallel Count Sketch [21] to collect flow-level
data. However, its feasibility of hardware implementation is
not discussed. OpenSketch [8] uses a hash-based measurement
data plane and sketch data structure to collect traffic flow
information. The goal of OpenSketch is to provide a generic
and efficient means of measuring network traffic by separating
the measurement control plane from the data plane functions. It
supports customized flow-level data collection such that the
flow to collect and measure is selected by using TCAM-based
classification. After an observation interval, the sketch counters
stored in SRAM are sent to the controller for further analysis.
Different to OpenSketch, the system presented herein uses two
tables in the data plane to collect flow information.
Using a 15-minute trace, there are 15 observation intervals
in one experiment. The first forecast sketch S f (t ) is computed
at the beginning of fourth interval since W = 3 , while the
forecast error sketch Se (t ) is calculated at the end of fourth
interval after the observed sketch So (t ) of fourth interval is
available. The threshold alarm TA then can be determined. At
11
2016 IEEE Region 10 Symposium (TENSYMP), Bali, Indonesia
TABLE I. FPR AND FNR FOR SUPERSPREADER IDENTIFICATION
2HashTable
Interval
[2]
FM-BF-HT
FPR
FNR
FPR
FNR
1
0
0
0
0.01
2
0
0.01
0
0.04
3
0
0.01
0.04
0.08
4
0
0
0.01
0.03
[3]
[4]
Estimated source fan-out
Superspreader detection
2HashTable
[5]
FM-BF-HT
2500
2500
1250
1250
500
500
[6]
200
200
500
1250
Exact source fan-out
2500
200
200
500
1250
2500
Exact source fan-out
[7]
[8]
Fig. 7. Exact vs. estimated fan-out in Interval 4. The number of actual
superspreaders in this interval is 76.
[9]
TABLE II. FALSE POSITIVE AND FALSE NEGATIVE FOR CHANGE DETECTION
Threshold (T)
0.01
0.02
0.05
0.1
0.2
0.5
False positive
0.01
0.01
0
0
0
0
False negative
0.02
0.02
0.02
0.02
0.03
0
[10]
VIII. CONCLUSION AND FUTURE WORK
This work proposes a system in which different
combination of flow keys can be chosen with great flexibility
for the sketch-based monitoring applications to perform
various measurement tasks. Based on the real network traffic
traces, entropy estimation, superspreader detection, and change
detection monitoring applications are utilized for system
evaluation. The results of entropy estimation and detection of
changes in traffic volume are accurate. Moreover, the system is
also capable of identifying the superspreaders with low fan-out
estimation error. Currently, based on the preliminary hardware
simulations, the system can process traffic flows at 50 Gbps
with 64-byte minimum-sized Ethernet frame. In future work,
we plan to evaluate more applications and further optimize the
hardware design to achieve higher performance.
[11]
[12]
[13]
[14]
[15]
[16]
[17]
ACKNOWLEDGMENT
This research was funded in part by the National Science
Council, Taiwan, under contract number: MOST 104-2221-E033-007, MOST 103-2221-E-033-030, and NSC 102-2221-E033-031.
[18]
[19]
[20]
REFERENCES
[1]
[21]
A. Lall, V. Sekar, M. Ogihara, J. Xu, and H. Zhang, “Data streaming
algorithms for estimating entropy of network traffic,” SIGMETRICS
Perform. Eval. Rev., vol. 34, no. 1, pp. 145–156, 2006.
12
Y.-K. Lai, T. Wellem, and H.-P. You, “Hardware-assisted estimation of
entropy norm for high-speed network traffic,” Electronics Letters, vol.
50, no. 24, pp. 1845–1847, 2014.
A. Kumar, M. Sung, J. J. Xu, and J. Wang, “Data streaming algorithms
for efficient and accurate estimation of flow size distribution,” in
Proceedings of the Joint International Conference on Measurement and
Modeling of Computer Systems, SIGMETRICS ’04/Performance ’04.
New York, NY, USA: ACM, 2004, pp. 177–188.
C. Estan and G. Varghese, “New directions in traffic measurement and
accounting,” in Proceedings of the 2002 Conference on Applications,
Technologies, Architectures, and Protocols for Computer
Communications, SIGCOMM ’02. New York, NY, USA: ACM, 2002,
pp. 323–336.
B. Krishnamurthy, S. Sen, Y. Zhang, and Y. Chen, “Sketch-based
change detection: Methods, evaluation, and applications,” in
Proceedings of the 3rd ACM SIGCOMM Conference on Internet
Measurement, IMC ’03. ACM, October 2003, pp. 234–247.
Q. Zhao, A. Kumar, and J. Xu, “Joint data streaming and sampling
techniques for detection of super sources and destinations,” in
Proceedings of the 5th ACM SIGCOMM Conference on Internet
Measurement, IMC ’05. Berkeley, CA, USA: USENIX Association,
2005, pp. 77–90.
“Cisco NetFlow http://www.cisco.com.”
M. Yu, L. Jose, and R. Miao, “Software defined traffic measurement
with OpenSketch,” in Proceedings of the 10th USENIX Conference on
Networked Systems Design and Implementation, NSDI’13. Berkeley,
CA, USA: USENIX Association, 2013, pp. 29–42.
V. Sekar, M. K. Reiter, and H. Zhang, “Revisiting the case for a
minimalist approach for network flow monitoring,” in Proceedings of
the 10th ACM SIGCOMM Conference on Internet Measurement, IMC
’10. New York, NY, USA: ACM, 2010, pp. 328–341.
Z. Liu, G. Vorsanger, V. Braverman, and V. Sekar, “Enabling a ”RISC”
Approach for Software-Defined Monitoring Using Universal
Streaming,” in Proceedings of the 14th ACM Workshop on Hot Topics in
Networks, HotNets-XIV. New York, NY, USA: ACM, 2015, pp. 21:1–
21:7.
G. Cormode and S. Muthukrishnan, “An improved data stream
summary: The count-min sketch and its applications,” Journal of
Algorithms, vol. 55, no. 1, pp. 58–75, April 2005.
T. Wellem, G.-W. Li, and Y.-K. Lai, “Superspreader Detection System
on NetFPGA Platform,” in Proceedings of the Tenth ACM/IEEE
Symposium on Architectures for Networking and Communications
Systems, ANCS ’14. New York, NY, USA: ACM, 2014, pp. 247–248.
M. Yoon, T. Li, S. Chen, and J.-K. Peir, “Fit a Compact Spread
Estimator in Small High-Speed Memory,” IEEE/ACM Transactions on
Networking, vol. 19, no. 5, pp. 1253–1264, Oct. 2011.
M. Thorup and Y. Zhang, “Tabulation based 5-universal hashing and
linear probing,” in Proceedings of the 12th Workshop on Algorithm
Engineering and Experiments ALENEX, 2010, pp. 62–76.
“NetFPGA project site. http://netfpga.org.”
J. Carter and M. Wegman, “Universal classes of hash functions,”
Journal of Computer and System Sciences, vol. 18, no. 2, pp. 143–154,
1979.
C. Shannon, “A mathematical theory of communication,” The Bell
System Technical Journal, vol. 27, pp. 379–423, July 1948.
S. Venkataraman, D. Song, P. B. Gibbons, and A. Blum, “New
streaming algorithms for fast detection of superspreaders,” in
Proceedings of Network and Distributed System Security Symposium
(NDSS), 2005, pp. 149–166.
“CAIDA UCSD Anonymized Internet Traces,” 2012.
MAWI Working Group, “MAWI Working Group Traffic Archive,”
http://mawi.wide.ad.jp/mawi/samplepoint-B/20030227/.
M. Charikar, K. Chen, and M. Farach-Colton, “Finding frequent items in
data streams,” in Proceedings of the 29th International Colloquium on
Automata, Languages and Programming. Springer-Verlag, July 2002,
pp. 693–703.
Toward Hardware Support for a Flexible
Sketch-based Network Traffic Monitoring System
Theophilus Wellem*§, Yu-Kuen Lai¶, Chao-Yuan Huang¶, Wen-Yaw Chung*
*
Department of Electronic Engineering, ¶Department of Electrical Engineering
Chung Yuan Christian University, Zhongli 32023, Taiwan
Email: {g10202604, ylai, g10279008, eldanny}@cycu.edu.tw
§
Department of Informatics
Satya Wacana Christian University, Salatiga 50711, Indonesia
Email: [email protected]
Abstract— Sketch-based data streaming algorithms are
capable of summarizing high-speed network traffic in real-time
without packet sampling. Therefore, they are used in many
networking applications to make accurate estimates of traffic
flow statistics. However, fixed hardware implementation of the
sketch counters limits the flexibility of re-using the same data
structure for various monitoring applications. In this paper, we
propose a generic scheme for collecting flow-level data using
high-speed memories on the NetFPGA-10G platform. The
purpose is to enable sketch-based algorithms making statistics
measurements in various network monitoring applications in a
flexible manner. Experiments using real Internet traces are
conducted for entropy estimation, superspreader detection, and
change detection applications. The results show that the proposed
scheme can support flexible selection of keys and counters for the
monitoring applications achieving high measurement accuracy.
Fig. 1. Sketch-based traffic monitoring is too application-specific. Each
monitoring application/measurement task requires its own sketch data
structures in data plane.
Keywords—Network traffic monitoring; Sketch; Hash table;
Linear probing; NetFPGA-10G
algorithms on network device is not flexible since once the data
structure is implemented in hardware for a measurement task,
another monitoring application that requires different flow key
and update value cannot re-use the data structure.
I. INTRODUCTION
Sketch-based data streaming algorithms are widely used in
network traffic monitoring to obtain accurate estimates of
traffic flow metrics such as entropy [1], [2], flow size
distribution [3], heavy hitters [4], heavy changers [5], and
superspreaders [6]. Entropy of source IP and destination port,
for example, can be utilized for detecting port scan attack. In
contrast to per-flow monitoring tool such as NetFlow [7] that
uses packet sampling to reduce number of flows collected,
sketch-based algorithms employ sketch data structure to
summarize network traffic into a compact representation
without packet sampling, using only a small amount of
memory space. Because the sketch data structure is updated for
each incoming packet, its update process is the critical
bottleneck. Therefore, the sketch is implemented by using fast
memory (SRAM) in the network device line card.
Motivated by the previous works of a "minimalist"
approach for flow monitoring system [9], [10] and OpenSketch
[8], this paper presents a data plane scheme to collect network
traffic flows without packet sampling. The goal is to create a
flexible sketch-based traffic monitoring system where the data
plane can enable flexible selection of flow keys and counters
for various monitoring applications. Rather than maintaining
sketch data structure for each flow key in data plane, the
proposed data plane scheme employs two tables: flow counter
table and flow key table for collecting traffic flows. The main
contributions of this work are:
• We propose a data plane scheme for flow-level data
collection to support more flexible implementation of
sketch-based
monitoring
applications.
Various
monitoring applications have greater flexibility of
choosing which flow keys and counters to monitor and
measure.
Sketch-based data streaming algorithms are efficient and
have provable memory-accuracy tradeoffs for traffic flow
measurements. However, they are too application-specific to be
implemented to perform primitive operations on network
devices [8], [9]. Sketch-based algorithms require a separate
instance of sketch data structure for each flow key, which is
defined according to the measurement tasks of interests, as
shown in Fig. 1. Current implementation of sketch-based
978-1-5090-0931-2/16/$31.00 ©2016 IEEE
• We present the evaluation of the scheme using three
monitoring
applications,
entropy
estimation,
superspreader detection, and change detection, to show
the feasibility of the data plane scheme.
7
2016 IEEE Region 10 Symposium (TENSYMP), Bali, Indonesia
Fig. 2. Sketch data structure and its update operation.
The remainder of this paper is organized as follows. Sketch
for network traffic measurements is discussed in Section II.
Section III describes the system architecture. Hardware
implementation overview is outlined in Section IV. Section V
explains three monitoring applications used to evaluate the
system. In Section VI, the system's evaluation with tracedriven tests using real traffic traces is presented. Section VII
reviews some related works, and Section VIII concludes this
paper.
Fig. 3. System architecture.
count, while the 5-tuple flow key is stored in the flow key table,
which is implemented as a FIFO in DRAM.
II. BACKGROUND
Sketch is a data structure representing the compact
summaries of a data stream. It can be queried to obtain
statistics about the data stream. A sketch is an array of counters
C[i][ j] , 1 ≤ i ≤ d , 1 ≤ j ≤ w . It consists of d hash tables of
size w . Each row is associated to a hash function selected
from a universal class of hash functions. When an item
a = (k , v) arrives, the key k is hashed by d hash functions
and the corresponding counters are updated by v . A query of
the sketch can yield the estimated counts for the key k . For
example, query of a key to Count-Min sketch [11] returns the
minimum value among all counters that correspond to that key.
Fig. 2 shows the sketch data structure.
During an observation interval, when a flow entry is
created, its 16-bit ID and counters are stored in the flow
counter table. This 16-bit ID is obtained from hashing the 32bit flow ID (i.e., the result from TabChar hash) using a 2universal hash function. A Bloom filter is used to check
whether the 5-tuple flow key already exists in the flow key
table. If the flow does not exist, then the 5-tuple is added into
the flow key table. After the observation interval, flow data
collected in both tables are moved to the host CPU and can be
utilized as required by the monitoring applications.
The modules presented in the upper part of Fig. 3 are
implemented in software on multi-core CPU for parallel
processing of sketches, while the bottom part is implemented in
FPGA hardware. The hardware development platform used for
implementation is NetFPGA-10G [15], which contains a
Virtex-5 XC5VTX240T FPGA, four 10GbE SFP+ interfaces,
three banks of 9MB QDRII SRAM with a total capacity of
27MB, and four 72MB RLDRAM II memories. The main
hardware modules for flow processing inside the system is
depicted in Fig. 4.
To use sketch for traffic measurement, the sketch data
structure must be updated based on a predefined flow key,
which is a combination of tuples from the packet header. The
update value can be the packet size or packet count. For
example, heavy hitters detection application uses the source IP
address and packet count as its key and updated value,
respectively. In change detection application [5], the sketch is
updated using source IP address and packet size. In
superspreader detection applications [6], [12], [13], the update
process sets the bit in bitmap sketch based on source IP address
and pair of source and destination IP addresses.
IV. HARDWARE OVERVIEW
In the NetFPGA-10G platform, incoming packets from four
10GbE interfaces are collected in four ingress queues, one for
each interface. An input arbiter module then takes the packet
out from these queues in round robin policy, and sends it to the
next module in the data path. The field extractor module
extracts the 104-bit 5-tuple and 16-bit packet length from each
packet header and sends it to the hash function module.
III. SYSTEM ARCHITECTURE
The architecture for a flexible sketch-based traffic
monitoring system is shown in Fig. 3. The 104-bit 5-tuple
(source IP, destination IP, source port, destination port, and
protocol) is extracted from each packet by the header parser
module as the flow key. The hash function module then
computes a 32-bit hash value from the 104-bit flow key as the
flow ID using a tabulation-based 2-universal hash function
(TabChar). This mapping from 104-bit to 32-bit is collisionfree with high probability [14]. The 32-bit flow ID is then
hashed again using a 5-universal hash function to obtain a 32bit hash value. The lower 19-bit of this hash value is the
memory (SRAM) address where the flow's packet and byte
counts are stored. This system utilizes two tables. The flow
counter table stores the flow's 16-bit ID, packet count, and byte
Hash Function: Three hash modules are utilized to implement
the hash functions required. The first module implements the
tabulation-based 2-universal hashing (TabChar) to process the
104-bit 5-tuple flow key to a 32-bit flow ID. The second
module hashes the 32-bit flow ID using a 5-universal hash
function to obtain the memory address where the flow's packet
and byte counts are stored. This module also computes a 16-bit
ID from the 32-bit flow ID to be stored in the flow counter
table. The computation of 5-universal hash function is based on
the four degree polynomial over some prime field as follows.
8
2016 IEEE Region 10 Symposium (TENSYMP), Bali, Indonesia
k
kn
k
1
FPR = 1 − 1 − ≈ (1 − e − kn m )
m
(2)
Theoretically, for fixed m , the number of hash functions k
that is needed to minimize the false positive rate is
k = ( m n ) ln 2 . For the system herein, k is set to 3, while
various m values are utilized to observe the effect of filter size
on the measurement accuracy. The 1.42 MB on-chip Block
RAM (BRAM) is used to store the Bloom filter.
Flow Key Table: The flow key table is stored as a FIFO in
RLDRAM II. Four RLDRAM II chips are available on-board
with a total size of 288 MB. For each entry in the FIFO, it is
capable of holding a 104-bit 5-tuple flow key (13 bytes per
entry). The FIFO can contain up to 512K flow entries utilizing
only 6.5 MB of the RLDRAM space. Two RLDRAM II
controllers are used in the design, each controller connects to
two RLDRAMs. Using burst length of two, one controller can
write a 72-bit data with latency of 5 clock cycles. The latency
is depend on the RLDRAM configuration mode.
Fig. 4. Hardware modules for flow processing.
4
h( x) = ai x i mod p
i =0
(1)
CW-trick [16] is used to accelerate the 5-universal hash
computation without modulo operation. The 5-universal hash
function is implemented in a pipeline fashion and utilized
seven embedded multipliers on the Virtex-5 FPGA. The third
module calculates hash values for the Bloom filter using three
2-universal hash functions, h( x) = (ax + b) mod p . Again, to
avoid slow modulo operation, multiply, shift, and bitwise AND
operations, as in CW-trick, are used.
Hardware-Software communication: The time that is
required for the monitoring application runs on the host CPU to
read the flow data from SRAM and DRAM via the PCIe
interface is important for effective operation of the system.
Given the table size and number of flows that are processed in
the system herein, the required time is less than 10 ms.
Furthermore, the processing time on CPU side (for sketch
updates and queries) is much less than the commonly used
observation interval. Communication between hardware and
software utilizes the register infrastructure in the NetFPGA10G platform. Applications can read data from the hardware or
write data to the hardware using these registers.
Flow Counter Table: The flow counter table is implemented
using a hash table with linear probing, and a 5-universal hash
function is used to handle hash collisions with constant number
of probes per update/lookup. Using 5-universal hash function,
the average number of probes per update/lookup is a constant
less than four [14], allowing fast and deterministic update
operation of the flow counters. The flow counter table is stored
in the QDRII SRAM and hash collision is handled by
comparing the input ID to multiple entries in a bucket in
parallel. The linear probing mechanism makes use of the fourword burst access of the QDRII SRAM. A read burst retrieves
four IDs from the memory. These flow IDs are then compared
in parallel to the input ID. If the input is a new ID, a flow entry
is created and its counters are updated. Otherwise, only the
counters are modified and then written back to the memory.
Updating counters in the table requires a read-modify-write
operation for each incoming packet at wire-speed. With a flow
entry consisting of a 16-bit flow ID, a 32-bit packet counter
and a 60-bit byte counter, the flow counter table can
accommodate up to 2 million flow entries in 27MB QDRII
SRAM.
V. MONITORING APPLICATIONS
A. Entropy Estimation
Entropy is a measure of the amount of randomness of
information content in the distribution of a random variable.
Let X be a discrete random variables that can take values from
the set of [ s] = {1, 2,3, , n} , mi denotes the frequency of item
i ∈ [s] , and m = i =1 mi is the total number of items. The
n
entropy of X is defined as [17]: H ( X ) = − i =1 pi log 2 pi ,
n
where pi = mi m is the probability of X taking the value i ,
X
is defined as
and the entropy norm of
H norm ( X ) = H ( X ) log 2 m . The entropy norm value is
minimum (zero) if all items in the data stream are the same,
and is maximum ( log 2 m ) if all the items in the data stream are
distinct [1]. In traffic monitoring context, the distributions of
traffic attributes (e.g. distribution of source IP address,
destination IP address, and ports) are important for network
monitoring system, especially for fine-grained traffic analysis
and classification. Entropy is utilized as a metric for anomaly
detection. For example, entropy of source IP and destination
port can be used to detect port scan attack.
Bloom filter: A Bloom filter is used to check whether a 5-tuple
flow key is already exist in the flow key table and to prevent
the addition of a duplicated 5-tuple flow key into the flow key
table. Recall that Bloom filter allows false positive, the size of
Bloom filter and number of hash functions must be chosen
properly to reduce the false positive rate. Assume that the
Bloom filter size with m bits stores n elements, then the falsepositive rate (FPR) is
9
2016 IEEE Region 10 Symposium (TENSYMP), Bali, Indonesia
B. Superspreader Detection
Detecting traffic sources that communicate with a large
number of distinct destinations during a measurement interval
is important for detecting network anomalies. These sources
are known as superspreaders [18] or super sources [6]. For
example, an abrupt increase in the number of connections from
many distinct sources to a particular destination in a short
period, may be the evidence of a distributed denial-of-service
(DDoS) attack. Conversely, detecting destinations that
contacted by a large number of distinct sources (super
destinations) can be useful for detecting the victims of DDoS
attack. The goal of superspreader detection is to find hosts that
have a large number of connections to distinct destinations
above a predefined threshold in an observation interval. To
identify such host, the superspreader detection module needs to
count the number of connections to distinct destination (or fanout) of that host. Counting the fan-out of each host in the
network requires large memory space to store the host’s source
IP and its fan-out counter. Therefore, estimation using bitmap
and sketch-based algorithms are commonly used [6], [12], [13].
Probe length per update
Table size=512K
Number of flows
300000
Load factor = 39.69% (5s)
Load factor = 66.98% (10s)
Load factor = 85.13% (15s)
250000
200000
Table size=1M
300000
250000
200000
150000
150000
100000
100000
50000
0
Load factor = 20.29% (5s)
Load factor = 33.00% (10s)
Load factor = 42.56% (15s)
50000
0
4
8
12 16 20 24 28 32
Probe length
0
0
4
8
12
16
20
Probe length
24
28
32
Fig. 5. Number of flows vs. probe length for different table size and
observation interval. Data in 4th interval are used. The number of flows
in this interval: 5s=208k, 10s=346k, 15s=446k. For table size=512K with
10s and 15s intervals, the figure only shows the number of flows that
have probe length up to 30.
10Gbps Ethernet link of a Tier1 ISP. It comprises of 30 million
packets and 1.4 million distinct 5-tuple flows, with a duration
of 60s. The number of source-destination IP pairs and distinct
source IP addresses in the trace are 1 million and 450k,
respectively. The observation interval is set as 15s in the
experiments. In this interval, the number of distinct 5-tuple
flows is ranged from 445k to 510k. Entropy estimation,
superspreader detection and change detection applications are
run on the host CPU, and utilize the flow keys and counters
collected in both tables to do their measurement task. For
change detection, a 15-minute trace from MAWI is used [20].
It is a part of 24h-long trace in a trans-Pacific line (18 Mbps
CAR on 100 Mbps link) that is collected at MAWI's
samplepoint-B. The trace consists of 51,792 distinct source IP
addresses and 374,561 distinct source-destination IP flows.
C. Change detection
The change detection's goal is to identify flows whose
contribution to changes in size (packet counts) or changes in
volume (byte counts) of total traffic over several consecutive
observation intervals are above a predefined threshold. These
flows are also known as heavy changers. Let At = a1 , a2 , be
the input packet stream in observation interval t . Each packet
ai is associated with a key k and an update value v .
Following the algorithm in [5], the source IP address and
packet size are used as the key and update value, respectively.
Let So (t ) denotes the sketch that accumulate the byte counts in
interval t and S f (t ) is the forecast sketch computed using a
forecasting model (e.g. moving average). The forecast error
sketch is then computed as Se (t ) = So (t ) − S f (t ) . A threshold
B. Experiment Results
1) Flow Counter Table and Linear Probing: Probe length
per key for different table sizes are shown in Fig. 5. When a
512K-entry table is utilized with observation interval of 15s,
84.1% of the flows have probe length less or equal to four.
The maximum probe length of 350 is observed by one flow.
After all flows were inserted into the table, the load factor is
85.13%. For 5s and 10s observation intervals, the percentage
of flows that have probe length less or equal to four are
98.97% and 92.99%, respectively. If a table size of 1M entries
is used in 15s observation interval, up to 98.56% of the flows
have probe length less or equal to four, and the maximum
probe length decreases to 27. For the 1.44% of flows that have
probe length more than four, only one flow has the maximum
probe length of 27. The load factor after all flows were
inserted into the table is 42.56%. For 5s and 10s intervals, the
percentage of flows that have probe length less or equal to
four are 99.95% and 99.48%, respectively. These experiment
results confirmed that by using 5-universal hash function,
most of flows have probe length less or equal to four.
2) Entropy estimation: The entropy norm of 5-tuple flow,
source IP address and destination IP address are estimated
using the data from the QDRII SRAM and RLDRAM FIFO.
The results are compared to the exact entropy norm and the
estimation accuracy is measured in term of relative error. Fig.
6 shows the relative error of the entropy norm estimation for
alarm TA is chosen based on the estimated second moment of
the forecast error sketch and a threshold parameter T
determined by application ( TA = T × EstF 2 ( So (t ) ) ). The
change detection task is to find all keys (source IP addresses)
whose volume, queried from Se (t ) , is above TA .
Although only three applications above are utilized for the
system’s evaluation, other monitoring applications may also
use the flow data collected by the hardware. For example, the
flow size distribution (FSD) estimation, which is useful for
network service providers to know the usage pattern of their
networks by customers [3]. Given a flow size x , the goal of
FSD estimation is to find the number of flows that have x
packets in an observation interval. In this case, the monitoring
application can rely on the flow counter and flow key tables to
obtain flow data and estimate the FSD.
VI. EVALUATION
A. Experiment Setup
Trace-driven tests using CAIDA traces [19] are conducted
to evaluate the system. The traces were collected over a
10
2016 IEEE Region 10 Symposium (TENSYMP), Bali, Indonesia
source IP address and destination IP address. Different sizes
of Bloom filter are used in the experiments. As presented, the
system can yield estimation errors in an acceptable range for
anomaly detection. The estimation accuracy increases with the
size of Bloom filter. For Bloom filter size greater than 128
KB, the estimation errors of 5-tuple flow, source IP address,
and destination IP address are less than 2%.
3) Superspreader detection: For superspreader detection,
source IP-destination IP pairs data from the flow key table in
RLDRAM FIFO are utilized. Here, a superspreader is defined
as a source whose fan-out exceeds 200. False negative ratio
(FNR) and false positive ratio (FPR) are used to measure the
superspreader identification accuracy, while the fan-out
estimation accuracy is obtained from scatter plots. The FNR is
the number of unidentified superspreaders divided by the
number of actual superspreaders, while the FPR is number of
non-superspreaders that are incorrectly identified as
superspreaders divided by the number of actual
superspreaders. FPR and FNR of superspreader detection
based on two methods are shown in Table I. The first method
(2HashTable) maintains two hash tables. One hash table is
used to remove duplicated source IP-destination IP pairs, and
the other hash table for counting the fan-out. This method is
traditionally used for exact fan-out counting, and is very
accurate. The FNR is due to the false positive of Bloom filter
when the 5-tuple is added into the flow key table. The second
method (FM-BF-HT) [12] combines an FM sketch, a Bloom
filter, and a hash table. This method has larger FPR and FNR.
The FNR is due to the FM sketch filtering, while the FPR is
caused by hash collisions since no collision resolution is
implemented for the hash table. The accuracy of fan-out
estimation is shown in Fig. 7. The closer the points to the
diagonal line, the more accurate the estimation is. The mean
errors in the fan-out estimation for 2HashTable and FM-BFHT are 0.11% and 3.6%, respectively.
4) Change detection: The experiments using k-ary sketchbased change detection algorithm [5] with Moving Average
forecast model are based on the parameters listed as follows:
Number of hash functions H = 5 , number of sketch entries
K = 32K , Moving Average window size W = 3 , and length
of observation interval Δt = 60s . The threshold parameter T
is varied for all experiments. The detection accuracy is
measured using false positive and false negative rates. The
false positives are flow keys whose volume do not change but
are incorrectly reported, while the false negatives are flow
keys that have volume changes but are not reported.
Entropy Estimation
Relative error (%)
Source IP
Destination IP
7
7
5
5
3
3
1
0.4
1
0.4
0.2
0.2
0
1
2
3
4
0
1
Interval
Bloom filter size 128KB
256KB
2
3
4
Interval
512KB
1024KB
Bloom filter size 128KB
256KB
512KB
1024KB
Fig. 6. Entropy estimation results of source IP address and destination IP
address for different Bloom filter size.
the beginning of fifth interval, the forecast error sketch Se (t ) is
available and queried using the current interval incoming
source IP address to find the heavy changers. From this interval
onward, the change detection process reports its results in each
interval. The experiment results are reported in Table II. As
shown, the change detection application achieves high
accuracy in detecting heavy changers with very low false
positives and false negatives. As an example, in 14th interval
with T = 0.1 , the application reports 10 heavy changers and
only two of them are false negatives. For other intervals, all
heavy changers are reported correctly.
VII. RELATED WORK
NetFlow [7] is a tool developed by Cisco for traffic flow
monitoring. The flow exporter in NetFlow is responsible for
creating flow records from network traffic by capturing packet
headers. Sampling and filtering are performed before a flow
entry is created or updated in the flow cache. These flow
records are stored in the flow cache until they are considered to
have expired. When a flow record expires, it is exported to the
flow collector. The flow data collected in the flow collector are
then used for analysis. To some extent, the system presented in
this work is similar to NetFlow. However, the system only
collect the flow's 5-tuple and packet/byte counts. It also does
not use the expiration mechanism of NetFlow, and simply
moves the flow entry from the tables to host CPU at the end of
observation interval.
Liu et al. [10] developed a universal monitoring
architecture (UnivMon), where a simple and generic
monitoring primitive run on network device. UnivMon data
plane utilizes parallel Count Sketch [21] to collect flow-level
data. However, its feasibility of hardware implementation is
not discussed. OpenSketch [8] uses a hash-based measurement
data plane and sketch data structure to collect traffic flow
information. The goal of OpenSketch is to provide a generic
and efficient means of measuring network traffic by separating
the measurement control plane from the data plane functions. It
supports customized flow-level data collection such that the
flow to collect and measure is selected by using TCAM-based
classification. After an observation interval, the sketch counters
stored in SRAM are sent to the controller for further analysis.
Different to OpenSketch, the system presented herein uses two
tables in the data plane to collect flow information.
Using a 15-minute trace, there are 15 observation intervals
in one experiment. The first forecast sketch S f (t ) is computed
at the beginning of fourth interval since W = 3 , while the
forecast error sketch Se (t ) is calculated at the end of fourth
interval after the observed sketch So (t ) of fourth interval is
available. The threshold alarm TA then can be determined. At
11
2016 IEEE Region 10 Symposium (TENSYMP), Bali, Indonesia
TABLE I. FPR AND FNR FOR SUPERSPREADER IDENTIFICATION
2HashTable
Interval
[2]
FM-BF-HT
FPR
FNR
FPR
FNR
1
0
0
0
0.01
2
0
0.01
0
0.04
3
0
0.01
0.04
0.08
4
0
0
0.01
0.03
[3]
[4]
Estimated source fan-out
Superspreader detection
2HashTable
[5]
FM-BF-HT
2500
2500
1250
1250
500
500
[6]
200
200
500
1250
Exact source fan-out
2500
200
200
500
1250
2500
Exact source fan-out
[7]
[8]
Fig. 7. Exact vs. estimated fan-out in Interval 4. The number of actual
superspreaders in this interval is 76.
[9]
TABLE II. FALSE POSITIVE AND FALSE NEGATIVE FOR CHANGE DETECTION
Threshold (T)
0.01
0.02
0.05
0.1
0.2
0.5
False positive
0.01
0.01
0
0
0
0
False negative
0.02
0.02
0.02
0.02
0.03
0
[10]
VIII. CONCLUSION AND FUTURE WORK
This work proposes a system in which different
combination of flow keys can be chosen with great flexibility
for the sketch-based monitoring applications to perform
various measurement tasks. Based on the real network traffic
traces, entropy estimation, superspreader detection, and change
detection monitoring applications are utilized for system
evaluation. The results of entropy estimation and detection of
changes in traffic volume are accurate. Moreover, the system is
also capable of identifying the superspreaders with low fan-out
estimation error. Currently, based on the preliminary hardware
simulations, the system can process traffic flows at 50 Gbps
with 64-byte minimum-sized Ethernet frame. In future work,
we plan to evaluate more applications and further optimize the
hardware design to achieve higher performance.
[11]
[12]
[13]
[14]
[15]
[16]
[17]
ACKNOWLEDGMENT
This research was funded in part by the National Science
Council, Taiwan, under contract number: MOST 104-2221-E033-007, MOST 103-2221-E-033-030, and NSC 102-2221-E033-031.
[18]
[19]
[20]
REFERENCES
[1]
[21]
A. Lall, V. Sekar, M. Ogihara, J. Xu, and H. Zhang, “Data streaming
algorithms for estimating entropy of network traffic,” SIGMETRICS
Perform. Eval. Rev., vol. 34, no. 1, pp. 145–156, 2006.
12
Y.-K. Lai, T. Wellem, and H.-P. You, “Hardware-assisted estimation of
entropy norm for high-speed network traffic,” Electronics Letters, vol.
50, no. 24, pp. 1845–1847, 2014.
A. Kumar, M. Sung, J. J. Xu, and J. Wang, “Data streaming algorithms
for efficient and accurate estimation of flow size distribution,” in
Proceedings of the Joint International Conference on Measurement and
Modeling of Computer Systems, SIGMETRICS ’04/Performance ’04.
New York, NY, USA: ACM, 2004, pp. 177–188.
C. Estan and G. Varghese, “New directions in traffic measurement and
accounting,” in Proceedings of the 2002 Conference on Applications,
Technologies, Architectures, and Protocols for Computer
Communications, SIGCOMM ’02. New York, NY, USA: ACM, 2002,
pp. 323–336.
B. Krishnamurthy, S. Sen, Y. Zhang, and Y. Chen, “Sketch-based
change detection: Methods, evaluation, and applications,” in
Proceedings of the 3rd ACM SIGCOMM Conference on Internet
Measurement, IMC ’03. ACM, October 2003, pp. 234–247.
Q. Zhao, A. Kumar, and J. Xu, “Joint data streaming and sampling
techniques for detection of super sources and destinations,” in
Proceedings of the 5th ACM SIGCOMM Conference on Internet
Measurement, IMC ’05. Berkeley, CA, USA: USENIX Association,
2005, pp. 77–90.
“Cisco NetFlow http://www.cisco.com.”
M. Yu, L. Jose, and R. Miao, “Software defined traffic measurement
with OpenSketch,” in Proceedings of the 10th USENIX Conference on
Networked Systems Design and Implementation, NSDI’13. Berkeley,
CA, USA: USENIX Association, 2013, pp. 29–42.
V. Sekar, M. K. Reiter, and H. Zhang, “Revisiting the case for a
minimalist approach for network flow monitoring,” in Proceedings of
the 10th ACM SIGCOMM Conference on Internet Measurement, IMC
’10. New York, NY, USA: ACM, 2010, pp. 328–341.
Z. Liu, G. Vorsanger, V. Braverman, and V. Sekar, “Enabling a ”RISC”
Approach for Software-Defined Monitoring Using Universal
Streaming,” in Proceedings of the 14th ACM Workshop on Hot Topics in
Networks, HotNets-XIV. New York, NY, USA: ACM, 2015, pp. 21:1–
21:7.
G. Cormode and S. Muthukrishnan, “An improved data stream
summary: The count-min sketch and its applications,” Journal of
Algorithms, vol. 55, no. 1, pp. 58–75, April 2005.
T. Wellem, G.-W. Li, and Y.-K. Lai, “Superspreader Detection System
on NetFPGA Platform,” in Proceedings of the Tenth ACM/IEEE
Symposium on Architectures for Networking and Communications
Systems, ANCS ’14. New York, NY, USA: ACM, 2014, pp. 247–248.
M. Yoon, T. Li, S. Chen, and J.-K. Peir, “Fit a Compact Spread
Estimator in Small High-Speed Memory,” IEEE/ACM Transactions on
Networking, vol. 19, no. 5, pp. 1253–1264, Oct. 2011.
M. Thorup and Y. Zhang, “Tabulation based 5-universal hashing and
linear probing,” in Proceedings of the 12th Workshop on Algorithm
Engineering and Experiments ALENEX, 2010, pp. 62–76.
“NetFPGA project site. http://netfpga.org.”
J. Carter and M. Wegman, “Universal classes of hash functions,”
Journal of Computer and System Sciences, vol. 18, no. 2, pp. 143–154,
1979.
C. Shannon, “A mathematical theory of communication,” The Bell
System Technical Journal, vol. 27, pp. 379–423, July 1948.
S. Venkataraman, D. Song, P. B. Gibbons, and A. Blum, “New
streaming algorithms for fast detection of superspreaders,” in
Proceedings of Network and Distributed System Security Symposium
(NDSS), 2005, pp. 149–166.
“CAIDA UCSD Anonymized Internet Traces,” 2012.
MAWI Working Group, “MAWI Working Group Traffic Archive,”
http://mawi.wide.ad.jp/mawi/samplepoint-B/20030227/.
M. Charikar, K. Chen, and M. Farach-Colton, “Finding frequent items in
data streams,” in Proceedings of the 29th International Colloquium on
Automata, Languages and Programming. Springer-Verlag, July 2002,
pp. 693–703.