0471408824ch03. 197KB Jun 05 2011 09:30:41 PM

Chapter 3

The Brief Life of a Packet
\It is better to give than to receive." - traditional saying.
\Be conservative in what you send, and liberal in what you receive."

Postel.

- Jon

3.1 Roadmap
In this chapter we take a lightweight structured walk-through of the process of
constructing and sending a packet in Linux, and then receiving and decoding
the packet at the far side. We do this by way of three dierent examples, each
seen from three dierent viewpoints: rstly we look at the transfer of some data
over TCP connection (e.g. part of an HTTP down-load); then we look at a UDP
packet exchange (e.g. part of a DNS lookup); nally, we look at a one way RTP
ow. The viewpoints are taken from the source code; from the execution; and
from the \wire". In the last case, we also consider a router as part of the wire,
and go into a bit more detail about its operation: looking at an extended case
which entails capturing some audio, putting it into an RTP packet, calling the

socket API to send a packet, going through the socket kernel code (skbus etc),
adding the UDP header, rewall lter checking, and multicast. Then the packet
traverses a link (SLIP or PPP say) to a router which is also Linux, running
diserv - so we get a look at a packet being scheduled in packet queueing code,
by a WRR or an other scheduler, then forwarded, say, over an Ethernet, then
nally received by driver, IP layer, demultiplexed and passed up through UDP
through to the process which is scheduled, reads the socket and outputs the
audio.
This is a combination of the way to explain Internet communications ideas
from many RFCs and related Internet documents1 , where the view of single
protocol is expanded through an operational example and a top down view of
the entire protocol stack in one go.
Thus we also provide a rapid introduction to the rest of the book at a fasttrack technical level.
1

see for example, TCP/IP Illustrated Volume 1.

55

56

CHAPTER 3. THE BRIEF LIFE OF A PACKET

3.1.1 User, Application and Protocol Viewpoints

There are at least three very dierent views of communication that one can
take:

User from a user perspective, in modern systems, communication is made as

transparent as possible. when clicking on an anchor in a web page, there is
no indication whether the down-load is local, or very remote. Many FTP
implementations include GUIs that make copying les the same whether
remote or local - usually the only visible dierences are in the areas of
naming (specifying a remote system name as part of a URL or FTP site
name, although this can often be hidden in a global extension to a normal
object or le naming system; and security - access to a remote system
usually (though not within an organisation) entails presenting more credentials than when accessing local resources.
Application An application is programmed to talk to an Application Programming Interface to a lower level system. Normally, such an API hides
detail and, especially in the area of data communication, this detail may

be of a very complex and rich structure - if the API is to a lower level
protocol, then it will hide the concurrency and state machine present in
the lower level and attempt to present a more sequential, simpler style of
programming for everyday implementors to understand and use.
Protocol The protocol perspective is the most complex. here we need to understand the way that information is exchanged. Here we need to consider
the fact that each end point in a communication session is autonomous,
and has its own quirks and failures and successes. The channels between
end points are imperfect, introducing errors and independent failures and
performance degradation.
Each of these perspectives can be illustrated in dierent ways. In the user
perspective, one is interested in the resource naming, and visible performance
parameters. In the application perspective, one is interested in the APIs and
interface programming. From the protocol perspective, one really wants to
understand the range of sequences of possible events and the way that state is
changed by events.
From a user perspective, we can show information models and organisation
of how resources are mapped into cyberspace. From an API perspective, we can
document the interfaces - the function prototypes, pre-requisites and results of
functions and how they relate in families to each other are usually part of good
systems documentation in any case. However, as we move through the layers of

abstraction to lower levels, often these APIs become more complex and the interactions more subtle. From a protocol perspective, we need to understand the
behaviour of senders and receivers and the structure of information exchanged.
This latter view is often best illustrated trough looking at the operational
examples of a protocol, showing the actual data exchanged on the wire for
correct, and common error cases. in this sense, understanding a protocol stack
implementation is often best done by eectively debugging it.
In the Linux community, the use of high level tools for debugging the kernel is
somewhat frowned upon. To some extent, this is justiable since most high level

3.1. ROADMAP

57

debugging tools (e.g. GDB) introduce changes to the function call mechanisms
and therefore actually alter timing which in low level code (especially interrupt
handling, but also anything that involves any fancy concurrency and scheduling)
will alter the behaviour of the system under examination, often critically to the
point where the problem being chased is no longer evident!
However, one can conduct the same type of activity `by hand". Thus in this
chapter we start doing what the rest of this book does in detail, which is to take

a structured walk through o the Linux kernel communications stack. In doing
so we occasionally take a look at a trace of packets on the wire, and occasionally
take a look at a trace of function calls, but largely, especially given space and
time considerations, we concentrate on the code itself. This is because there
is really one main example piece of code, while there are an innite variety of
examples of execution traces of the code.
Many tracing, debugging and proling tools do abound, and I would encourage you to use them to help nd problems (especially non intrusive tracing for
performance problems) then all well and good. For now, lets look at some of
the code!
Sending

In the diagram below?? ,we try to illustrate the overall structure of the code,
showing the way the control
ows from the application down to the driver, and
thence out onto the transmission medium.
Application

Transport

Network

Driver

X

X

X

Wire

Time

Figure 3.1: The overall transmission process: from Application to Wire
So here we can see the
ow of information from the application, which uses
system calls from the le system API and Socket API such as send, sendto,
sendmsg, writev, write to interface to the lower level protocols in the kernel.
The parameters to these system calls are passed through what is normally a
synchronous,. blocking, procedure call to the lower level, and actually makes

things like reading and writing to a remote application look extremely similar
to reading and writing fro ma local le or application (in chapter nine, we look

CHAPTER 3. THE BRIEF LIFE OF A PACKET

58

in detail at Inter-process communication APIs. The parameters are typically
data and addressing information, and these are passed through to the transport
protocol in the correct family, for example UDP, TCP, etc
The kernel then triggers the appropriate state machine for the protocol and
then passes information through to the IP level which then entails doing the
necessary route work, and then adds any appropriate IP header information,
nally queueing the data for driver output. The device is then usually woken
and actual transmission occurs. In fact, several of these stages may be deferred
to allow more ecient use of CPU, memory, and network resources, so that a
network scheduler is then part of the picture. In fact there are several complex
levels of scheduling which are not always explicit. Sometimes, it is simply a
matter of using a
ow control and on-o scheduler mechanism, for instance

when looking at device output for busy links (or device input for full queues)
we may rely simply on semaphores or other mechanisms to handshake between
the layers, and wake up device driver, which does link and physical layer work
to send or receive a packet.
Receiving

In the diagram here ?? we try to show the converse process - from unsolicited
arrival of a frame from the wire, up to delivery to an application that had, in
fact, been earlier primed with the idea that it might want to receive some data
sometime soon!
Application

Application Blocked/Descheduled during this period......until data ready

Transport

X

X

Network

X
Driver

Wire

Time

Figure 3.2: The overall reception process: from Wire to Application
Physical reception causes hardware to wakeup, and typically interrupt. Most
modern network devices include both interrupt and DMA hardware. The interrupt service routine pulls packet o device and puts in memory and
ags the
"network bottom half" that packet(s) ready. Bottom halves and other Linux
mechanism (software interrupts, and tasklets and other SMP capabilities) are
discussed in chapter tow and later on in the book.

3.1. ROADMAP

59

The general system scheduler (after system call) runs network bottom half
(and can dispatch tasklets or software interrupts) on completion of any kernel
work on behalf of any user process that called a system call. The bottom halves
basically complete work that the hardware interrupts started- for example, they
can dequeues any packets in the interface queue for IP input , and sends any
more packets waiting to send.
IP input checks packets, carries out any re-assembly work needed, get a route
if the packet is to be forwarded rather than being destined for a local application
process. If it is for local delivery, IP then demultiplexes the packets according
to the IP protocol number to the appropriate transport handler receive function
(UDP, TCP etc).
TCP (and UDP) input then do further packet header checking, update state
machinery, and then demultiplex the packet further to the correct socket (i.e to
the queue of data pending for any user process that has an outstanding read on
that socket, which is \connected" to the appropriate port numbers.
Once the transport protocol demultiplexing routine has put data on the
appropriate socket receive queue, it calls wakeup on any process waiting for
data, thus making any such process ready for general running by the system
scheduler,

Sometimes a process that had earlier called a recv, recvfrom, recvmsg read/readv and was asleep - data is copied from receive queue to user space buer;
process is now woken IP, and continues with data in buers.
Recent Linux work on avoiding the receive packet copies (copy buer only
on write) has improved this performance somewhat.
Forwarding
The nal case of a packet handling pattern we need to worry about is that of
forwarding. Forwarding is (almost) entirely kernel bound - with the exception
of the process of managing the tables used to make forwarding decisions (see
chapter seven ) and performance (see chapter eight) the forwarding task for IP
entails back to back, reception and transmission.

3.1.2 Source Code, Execution and Wire Viewpoints

In the rest of this chapter then we look at the source code and trace through
some simple vertical cuts across the stack for some examples, with some very
slimmed down view of what occurs on the wire during these executions.

3.1.3 State, Memory etc

Throughout the code, protocols need to keep track of where they are. Rather
than just using a set of randomly allocated messy global memory, most protocols
gather together all the appropriate information on a given
ow (at their level e.g. an IP route, a TCP connection, a UDP session) in one data structure - in
BSD Unix this is generally referred to as the Protocol Control Block. In Linux,
the general all purpose data structure of skbuffs is also used to hold this data.
It contains information to t a packet into a sequence, to multiplex packets
downwards (locally route them) and to de-multiplex them when receiving and

CHAPTER 3. THE BRIEF LIFE OF A PACKET

60

passing them upwards (e.g. link to the user socket structure to queue data and
so on) as well as very complex state machinery for parts of the TCP machine.

3.2 TCP Example
First we look at the example of the Transmission Control Protocol and follow
through the code applied to a single data packet as it is transmitted, then
subsequently as one is receive.d
3.2.1

Socket Level

In this simple example, we assume that an application process (netscape, apache,
etc) has called write() on socket which maps into a system call (see chapter 2),
which subsequently calls tcp sendmsg() and and deschedules the process.....(see
chapter 2).
3.2.2

TCP Output

Here we at the socket glue, then the tcp work - in the source distribution this
(as is most of the rest of the code discussed in this chapter) resides in the
src/net/ipv4/ directory in the (version 2.4) kernel distribution tree.
An extract from the code referenced below is displayed in gure ??.
inet_sendmsg()
sk->prot->sendmsg() ...
tcp_ipv4.c:
struct proto tcp_prot {}
tcp.c:
tcp_sendmsg()
tcp_output.c:
tcp_send\_skb()
tcp_transmit_skb()
err = tp->af_specific->queue_xmit(skb);
which maps to ip_queue_xmit()

Then in tcp transmit skb(), the state/header changes are managed This
is discussed in detail in chapter six.
3.2.3

IP Output Work

TCP calls IP functions to carry out output. Some of this work is structured as
follows, and a relevant extract from the code is illustraed below in gure ??.
ip_output.c:
ip_queue_xmit()
ip_route_output()
NF_HOOK()

3.2. TCP EXAMPLE

236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255



61

th = (struct tcphdr ) skb push(skb, tcp header size );
skb;>h.th = th;
skb set owner w(skb, sk);



/ Build TCP header and checksum it. /
th;>source = sk;>sport;
th;>dest = sk;>dport;
th;>seq = htonl(tcb;>seq);
th;>ack seq = htonl(tp;>rcv nxt);
((( u16 )th) + 6) = htons(((tcp header size >> 2)
ags);
if (tcb;>
ags & TCPCB FLAG SYN) f
/ RFC1323: The window in SYN & SYN/ACK segments
 is never scaled .
/
th;>window = htons(tp;>rcv wnd);
g else f
th;>window = htons(tcp select window(sk));

g



th;>check = 0;
th;>urg ptr = 0;



Figure 3.3: Filling in Some Key TCP elds
NF HOOK is part of the network lter support in Linux which is machinery to
do the low level work for rewalls and other functions and is discussed in more
detail in chapter ten.

netfilter.h:
NF_HOOK(pf, hook, skb, indev, outdev, okfn)
NF_hook\_slow((pf), (hook), (skb), (indev), (outdev), (okfn))
ip_queue_xmit2
skb->dst->output(skb)

This queues, or calls okfn(), which maps to op queue xmit2() The skb->dst
structure was lled in by local route lookup and is link level output routine.
In ip queue xmit, header changes are as follows:
This is covered in detail in chapter ve.
3.2.4

Link Level Output

Output at the link level can be complex (see chapter 8). It includes specic
technical details of driving devices, but also the important part is that this is
where Linux provides dierent treatment for dierent trac types by providing
fancy queueing and scheduling management.
In general, we call the device specic function to start output on the device.
This does the book work, then informs the device that there is work to do, which
then typically is done by DMA - this is covered in some detail in chapter four.

CHAPTER 3. THE BRIEF LIFE OF A PACKET

62

367
368
369
370
371
372
373
374
375
376
377







/ OK, we know where to send it, allocate and build IP header. /
iph = (struct iphdr ) skb push(skb, sizeof (struct iphdr) + (opt ? opt;>optlen : 0));
(( u16 )iph) = htons((4 tot len = htons(skb;>len);
iph ;>frag o = 0;
iph ;>ttl
= sk;>protinfo.af inet.ttl ;
iph ;>protocol = sk;>protocol;
iph ;>saddr = rt;>rt src;
iph ;>daddr = rt;>rt dst;
skb;>nh.iph = iph;
/ Transport layer set skb;>h.foo itself . /



Figure 3.4: Filling in Some Key IP bits
3.2.5

Link Level Input

Link level input requires dealing with hardware and software interrupts and
DMA devices - this is covered also in chapter four in detail. Some relevant
extracts from the code are displayed in gures ??, ?? and ??
Network device interrupts end up calling a function via the net devices
structure,
core/netif_rx:

Then the software interrupt schedules the rest. which eventually results in
a call to
Next, net rx action() which dequeues the packet, and calls a packet
handler based on the packet type (via a table) - if IP, this calls ip rcv() in
ip input.c - which checks various things.
3.2.6

IP Input Work

IP input has to decipher the packet, so the rst thing to do is gure out if it is
really \for me"., e.g. is it IP version 4!
In ip input ip rcv() checks, then calls NH HOOK (to run lters, which
may call, or queues directly ip rcv finish(), which calls ip route input()
in route.c which lls in the \internal" route.
This includes a function (forward or local deliver), and calls ip local deliver()
in ip input again, and this then calls ipprot->handler.
This pointer was previously lled in (via the inet protocol structure) in
protocol.c, which matches the protocol to the protocol handler function - in
this case, TCP - tcp v4 rcv().
This is investigated in more detail in chapter ve.
3.2.7

TCP Input Work

In the IPv4 case, the relevant TCP input code is kicked o from tcp ipv4.c,
which looks up if we have a connection tcp v4 lookup, then tries to queue
in process context, or else, calls tcp v4 do rcv() then tcp rcv established()
which actually processes the TCP header, (header prediction etc) and calls

3.2. TCP EXAMPLE

1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083



63



static void netdev wakeup(void)

f

unsigned long xo ;
spin lock(&netdev fc lock);
xo = netdev fc xo ;
netdev fc xo = 0;
while ( xo ) f
int i = z(~xo );
xo &= ~(1 1)+ (blog >> 1);
if (avg blog > mod cong) f
/ Above moderate congestion levels. /
softnet data [cpu]. cng level = NET RX CN HIGH;
#ifdef RAND LIE
rd = net random();
rq = rd % netdev max backlog;
if (rq < avg blog) / unlucky bastard /
softnet data [cpu]. cng level = NET RX DROP;
#endif
g else if (avg blog > lo cong) f
softnet data [cpu]. cng level = NET RX CN MOD;
#ifdef RAND LIE
rd = net random();
rq = rd % netdev max backlog;
if (rq < avg blog) / unlucky bastard /
softnet data [cpu]. cng level = NET RX CN HIGH;
#endif
g else if (avg blog > no cong)
softnet data [cpu]. cng level = NET RX CN LOW;
else / no congestion /
softnet data [cpu]. cng level = NET RX SUCCESS;

g

softnet data [cpu].avg blog = avg blog;

#ifdef OFFLINE SAMPLE

static void sample queue(unsigned long dummy)

f

/ 10 ms 0r 1ms ;; i dont care ;; JHS /
Figure 3.5: Network Device Driver Rx



64

CHAPTER 3. THE BRIEF LIFE OF A PACKET









1135 enqueue:
1136
dev hold(skb;>dev);
1137
skb queue tail (&queue;>input pkt queue,skb);
1138
cpu raise softirq (this cpu , NET RX SOFTIRQ);
1139
local irq restore (
ags );
1140 #ifndef OFFLINE SAMPLE
1141
get sample stats(this cpu );
1142 #endif
1143
return softnet data [this cpu ]. cng level ;
1144
g
1145
1146
if (queue;>throttle) f
1147
queue;>throttle = 0;
1148 #ifdef CONFIG NET HW FLOWCONTROL
1149
if (atomic dec and test(&netdev dropping))
1150
netdev wakeup();
1151 #endif
1152
g
1153
goto enqueue;
1154 g
1155
1156
if (queue;>throttle == 0) f
1157
queue;>throttle = 1;
1158
netdev rx stat[this cpu ]. throttled++;
1159 #ifdef CONFIG NET HW FLOWCONTROL
1160
atomic inc(&netdev dropping);
1161 #endif
1162 g
1163
1164 drop:
1165 netdev rx stat[this cpu ].dropped++;
1166
local irq restore (
ags );
1167
1168 kfree skb(skb);
1169 return NET RX DROP;
1170 g
Figure 3.6: Network Device Driver Rx i

3.2. TCP EXAMPLE

65









1172 / Deliver skb to an old protocol , which is not threaded well
1173
or which do not understand shared skbs.
1174 /
1175 static int deliver to old ones (struct packet type pt, struct sk bu skb, int last )
1176 f
1177
static spinlock t net bh lock = SPIN LOCK UNLOCKED;
1178
int ret = NET RX DROP;
1179
1180
1181
if (! last ) f
1182
skb = skb clone(skb, GFP ATOMIC);
1183
if (skb == NULL)
1184
return ret ;
1185 g
1186
1187 / The assumption (correct one) is that old protocols
1188
did not depened on BHs dierent of NET BH and TIMER BH.
1189
/
1190
1191 / Emulate NET BH with special spinlock /
1192
spin lock(&net bh lock);
1193
1194 / Disable timers and wait for all timers completion /
1195
tasklet disable (bh task vec+TIMER BH);
1196
1197 ret = pt;>func(skb, skb;>dev, pt);
1198
1199
tasklet enable (bh task vec+TIMER BH);
Figure 3.7: Network Device Driver Rx ii

CHAPTER 3. THE BRIEF LIFE OF A PACKET

66

tcp event data recv(), does any acknowledgement needed and calls sk->data ready.
The last function queues data to the socket.
3.2.8

Socket Input Work

In core/sock.c we see that the main eect is triggered because



sk->data ready = sock def readable;

1083 void sock def readable(struct sock sk, int len)
1084 f
1085 read lock(&sk;>callback lock);
1086 if (sk;>sleep && waitqueue active(sk;>sleep))
1087
wake up interruptible(sk;>sleep);
1088 sk wake async(sk,1,POLL IN);
1089 read unlock(&sk;>callback lock);
1090 g







Figure 3.8: Socket Receive wake up process
3.2.9

On the wire

16:09:07.462590 brahms.cs.ucl.ac.uk.ssh > ovavu.cs.ucl.ac.uk.1023: P
2466:2574(108) ack 971 win 17376
(DF)

3.3 DNS/UDP Example
Here we look at the dierence if the packet was a UDP packet, for example part
of a DNS exchange. To a large extent, these are restricted to the actual UDP
protocol function itself.
3.3.1

UDP Output

In udp.c. the main function of interest is udp sendmsg(), for output.
3.3.2

UDP Input

UDP is small enough that it is all in one C le, udp.c. For receive, the function
of interest is udp recvmsg()
3.3.3

On the wire

16:11:35.321380 brahms.cs.ucl.ac.uk.1904 > bells.cs.ucl.ac.uk.domain:
60803+ (43)
16:11:35.323169 bells.cs.ucl.ac.uk.domain > brahms.cs.ucl.ac.uk.1904:
60803* 1/2/2 (172)

3.3. DNS/UDP EXAMPLE

488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528



if (msg;>msg name) f
struct sockaddr in  usin = (struct sockaddr in)msg;>msg name;
if (msg;>msg namelen < sizeof(usin))
return ;EINVAL;
if (usin ;>sin family != AF INET) f
if (usin ;>sin family != AF UNSPEC)
return ;EINVAL;

67



g

ufh.daddr = usin ;>sin addr.s addr;
ufh.uh.dest = usin;>sin port;
if (ufh.uh.dest == 0)
return ;EINVAL;
g else f
if (sk;>state != TCP ESTABLISHED)
return ;ENOTCONN;
ufh.daddr = sk;>daddr;
ufh.uh.dest = sk;>dport;
/ Open fast path for connected socket.
Route will not be used, if at least one option is set .
/
connected = 1;

g

ipc.addr = sk;>saddr;
ufh.uh.source = sk;>sport;
ipc.opt = NULL;
ipc. oif = sk;>bound dev if;
if (msg;>msg controllen) f
err = ip cmsg send(msg, &ipc);
if ( err)
return err ;
if (ipc .opt)
free = 1;
connected = 0;

g

if (! ipc.opt)
ipc.opt = sk;>protinfo.af inet.opt;



ufh.saddr = ipc.addr;
ipc.addr = daddr = ufh.daddr;
Figure 3.9: UDP Output



68



CHAPTER 3. THE BRIEF LIFE OF A PACKET

530
if (ipc .opt && ipc.opt;>srr) f
531
if (! daddr)
532
return ;EINVAL;
533
daddr = ipc.opt;>faddr;
534
connected = 0;
535 g
536 tos = RT TOS(sk;>protinfo.af inet.tos);
537
if (sk;>localroute jj (msg;>msg
ags&MSG DONTROUTE) jj
538
(ipc.opt && ipc.opt;>is strictroute)) f
539
tos j= RTO ONLINK;
540
connected = 0;
541 g
542
543
if (MULTICAST(daddr)) f
544
if (! ipc . oif )
545
ipc . oif = sk;>protinfo.af inet.mc index;
546
if (! ufh.saddr)
547
ufh.saddr = sk;>protinfo.af inet.mc addr;
548
connected = 0;
549 g
550
551
if (connected)
552
rt = (struct rtable )sk dst check(sk , 0);
553
554
if ( rt == NULL) f
555
err = ip route output(&rt, daddr, ufh.saddr, tos , ipc. oif );
556
if ( err)
557
goto out;
558
559
err = ;EACCES;
560
if (rt ;>rt
ags&RTCF BROADCAST && !sk;>broadcast)
561
goto out;
562
if (connected)
563
sk dst set (sk, dst clone(&rt;>u.dst));
564 g
565
566
if (msg;>msg
ags&MSG CONFIRM)
567
goto do conrm;
568 back from conrm:
569
570 ufh.saddr = rt;>rt src;
571
if (! ipc.addr)
572
ufh.daddr = ipc.addr = rt;>rt dst;
573 ufh.uh.len = htons(ulen);
574 ufh.uh.check = 0;
575 ufh.iov = msg;>msg iov;
576 ufh.wcheck = 0;
577
578 / RFC1122: OK. Provides the checksumming facility (MUST) as per /
579
/ 4.1.3.4. It 's congurable by the application via setsockopt() /
580 / (MAY) and it defaults to on (MUST). /
581
582 err = ip build xmit(sk,
583
(sk;>no check == UDP CSUM NOXMIT ?
584
udp getfrag nosum :
585
udp getfrag),
586
&ufh, ulen, &ipc, rt , msg;>msg
ags);



Figure 3.10: UDP Output





3.3. DNS/UDP EXAMPLE

683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723



skb = skb recv datagram(sk,
ags, noblock, &err);
if (! skb)
goto out;

69



copied = skb;>len ; sizeof(struct udphdr);
if (copied > len) f
copied = len;
msg;>msg
ags j= MSG TRUNC;

g

if (skb;>ip summed==CHECKSUM UNNECESSARY) f
err = skb copy datagram iovec(skb, sizeof(struct udphdr), msg;>msg iov,
copied);
g else if (msg;>msg
ags&MSG TRUNC) f
if ( udp checksum complete(skb))
goto csum copy err;
err = skb copy datagram iovec(skb, sizeof(struct udphdr), msg;>msg iov,
copied);
g else f
err = copy and csum toiovec(msg;>msg iov, skb, sizeof(struct udphdr));

g

if ( err)
goto csum copy err;

if (err)
goto out free ;
sock recv timestamp(msg, sk, skb);
/ Copy the address. /
if (sin)

f

sin ;>sin family = AF INET;
sin ;>sin port = skb;>h.uh;>source;
sin ;>sin addr.s addr = skb;>nh.iph;>saddr;
memset(sin;>sin zero, 0, sizeof(sin ;>sin zero));

g



if (sk;>protinfo.af inet.cmsg
ags)
ip cmsg recv(msg, skb);
err = copied;
Figure 3.11: UDP Input i



CHAPTER 3. THE BRIEF LIFE OF A PACKET

70

725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784



out free :
skb free datagram(sk, skb);
out:
return err ;



csum copy err:
UDP INC STATS BH(UdpInErrors);
/ Clear queue. /
if (
ags &MSG PEEK) f
int clear = 0;
spin lock irq (&sk;>receive queue.lock);
if (skb == skb peek(&sk;>receive queue)) f
skb unlink(skb, &sk;>receive queue);
clear = 1;

g

g

spin unlock irq(&sk;>receive queue.lock);
if ( clear )
kfree skb(skb);

skb free datagram(sk, skb);

g

return ;EAGAIN;

int udp connect(struct sock sk, struct sockaddr uaddr, int addr len)

f

struct sockaddr in usin = (struct sockaddr in ) uaddr;
struct rtable  rt ;
int err ;
if (addr len < sizeof(usin))
return ;EINVAL;
if (usin ;>sin family != AF INET)
return ;EAFNOSUPPORT;
sk dst reset (sk);
err = ip route connect(&rt, usin;>sin addr.s addr, sk;>saddr,
sk;>protinfo.af inet.tos j sk;>localroute, sk;>bound dev if);
if ( err)
return err ;
if (( rt ;>rt
ags&RTCF BROADCAST) && !sk;>broadcast) f
ip rt put (rt );
return ;EACCES;

g

if (!sk;>saddr)
sk;>saddr = rt;>rt src; / Update source address /
if (!sk;>rcv saddr)
sk;>rcv saddr = rt;>rt src;
sk;>daddr = rt;>rt dst;
sk;>dport = usin;>sin port;
sk;>state = TCP ESTABLISHED;

g

sk dst set (sk, &rt ;>u.dst);
return(0);

Figure 3.12: UDP Input ii



3.4. RTP/UDP (MULTICAST) EXAMPLE

71

14:19:51.488399 0:20:af:ab:e1:6e 8:0:20:7d:a5:36 ip 85:
brahms.cs.ucl.ac.uk.2167 > bells.cs.ucl.ac.uk.domain: 6517+ (43)
14:19:51.490923 8:0:20:7d:a5:36 0:20:af:ab:e1:6e ip 214:
bells.cs.ucl.ac.uk.domain > brahms.cs.ucl.ac.uk.2167: 6517* 1/2/2
(172)

14:22:05.123790 0:20:af:ab:e1:6e Broadcast arp 42: arp who-has
merci.cs.ucl.ac.uk tell brahms.cs.ucl.ac.uk

3.4 RTP/UDP (multicast) Example
For multicast, most higher level protocols are built on top of UDP, so again we
need to look in udp.c.
Sending is the same as for unicast, but the receive case has to handle the
chance that there is more than one process waiting for copies of a multicast
packet, so receiving is a tad dierent. as we can see in udp v4 mcast deliver()

859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880



read lock(&udp hash lock);
sk = udp hash[ntohs(uh;>dest) & (UDP HTABLE SIZE ; 1)];
dif = skb;>dev;>index;
sk = udp v4 mcast next(sk, uh;>dest, daddr, uh;>source, saddr, dif);
if (sk) f
struct sock sknext = NULL;



do f
struct sk bu skb1 = skb;
sknext = udp v4 mcast next(sk;>next, uh;>dest, daddr,
uh;>source, saddr, dif);
if (sknext)
skb1 = skb clone(skb, GFP ATOMIC);



if (skb1)
udp queue rcv skb(sk, skb1);
sk = sknext;
g while(sknext);
g else
kfree skb(skb);
read unlock(&udp hash lock);



Figure 3.13: UDP Receive Multicast
3.4.1

On the wire

16:25:49.747290 d230-17.uoregon.edu.1025 > 224.2.163.188.23824: udp 332

CHAPTER 3. THE BRIEF LIFE OF A PACKET

72
16:25:49.790001
16:25:49.829479
16:25:49.853864
16:25:49.898744
16:25:49.992876
16:25:50.104227
16:25:50.135776
16:25:50.212101
16:25:50.272045
16:25:50.276215

3.4.2

hocus.cs.ucl.ac.uk > 224.1.127.255: igmp nreport 224.1.127.255 [ttl 1]
d230-17.uoregon.edu.1025 > 224.2.163.188.23824: udp 332
d230-17.uoregon.edu.1025 > 224.2.163.188.23824: udp 332
d230-17.uoregon.edu.1025 > 224.2.163.188.23824: udp 332
d230-17.uoregon.edu.1025 > 224.2.163.188.23824: udp 332
d230-17.uoregon.edu.1025 > 224.2.163.188.23824: udp 332
d230-17.uoregon.edu.1025 > 224.2.163.188.23824: udp 332
d230-17.uoregon.edu.1025 > 224.2.163.188.23824: udp 332
d230-17.uoregon.edu.1025 > 224.2.163.188.23824: udp 332
Ez.Stanford.EDU.33235 > 224.2.163.188.23825: udp 88

...but with a router in the path...

We know where we are going, but not how to get there - some kind strangers
along the way will help - these are routers. See chapter seven, eight for more
details Linux as a router.
Linux works well as a router. Basically, you need to consider input, then
output - the decision above, where a packet was discovered to be destined \for
me", in the call in route.c validates the source address for the input device,
them decides if its for me, multicast, loopback or for someone else nd adds it to
a hash based cache table of most recent routes
Then if we are forwarding, we do fib lookup(). This wills in the routing
structure, including a reference to the input handler - if its a non local delivery,
this will be set to ip forward() in ip forward.c:
ip forward does various checks (router alerts, route options etc) ttl processing, mtu work, fragmenting, generating ICMP errors if necessary, NATs, if
needed, then calls NF HOOK (for net lter work, again see chapter 10). and then
this will schedule a call to ip forward finish, which checks the route cache for
a fast route decision, and then does ip send(skb).
Then ip send() does fragmentation or not, and calls ip finish output()
does yet more NF work, then calls dst->neighbour->output(skb); and/or
hh->hh output(skb); (hardware header cache stu)

3.5 Socket System Call Trace
As an illustration of the actual total API, including another protocol case,e
that of a raw socket, used to do ICMP access, lets look at the strace of the ping
programr, usage: ping bells
PING bells.cs.ucl.ac.uk (128.16.5.31) from 128.16.6.226 : 56(84) bytes of data.
64 bytes from bells.cs.ucl.ac.uk (128.16.5.31): icmp_seq=0 ttl=255 time=819 usec

On the wire, this looks as follows (taken from another example:

10:38:11.705142 0:c0:4f:d3:db:c3 0:20:af:ab:e1:6e ip 98: ovavu.cs.ucl.ac.uk > brahms.cs.u
10:38:11.705270 0:20:af:ab:e1:6e 0:c0:4f:d3:db:c3 ip 98: brahms.cs.ucl.ac.uk > ovavu.cs.u

3.5. SOCKET SYSTEM CALL TRACE

73

execve("/bin/ping", ["ping", "-c", "1", "bells"], [/* 24 vars */]) = 0
brk(0)
= 0x805d3d4
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40014000
open("/etc/ld.so.preload", O_RDONLY)
= -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)
= 3
fstat(3, {st_mode=S_IFREG|0644, st_size=19066, ...}) = 0
old_mmap(NULL, 19066, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40015000
close(3)
= 0
open("/lib/libresolv.so.2", O_RDONLY)
= 3
fstat(3, {st_mode=S_IFREG|0755, st_size=169720, ...}) = 0
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\340&\0"..., 4096) = 4096
old_mmap(NULL, 60956, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x4001a000
mprotect(0x40026000, 11804, PROT_NONE) = 0
old_mmap(0x40026000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0xb000) = 0x40026000
old_mmap(0x40027000, 7708, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x4
close(3)
= 0
open("/lib/libc.so.6", O_RDONLY)
= 3
fstat(3, {st_mode=S_IFREG|0755, st_size=4101324, ...}) = 0
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\210\212"..., 4096) = 4096
old_mmap(NULL, 1001564, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x40029000
mprotect(0x40116000, 30812, PROT_NONE) = 0
old_mmap(0x40116000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0xec000) = 0x40116000
old_mmap(0x4011a000, 14428, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x
close(3)
= 0
mprotect(0x40029000, 970752, PROT_READ|PROT_WRITE) = 0
mprotect(0x40029000, 970752, PROT_READ|PROT_EXEC) = 0
munmap(0x40015000, 19066)
= 0
personality(PER_LINUX)
= 0
getpid()
= 4510
getuid()
= 0
socket(PF_INET, SOCK_RAW, IPPROTO_ICMP) = 3
setuid(0)
= 0
brk(0)
= 0x805d3d4
brk(0x805d7ec)
= 0x805d7ec
brk(0x805e000)
= 0x805e000
gettimeofday({960371553, 351449}, NULL) = 0
getpid()
= 4510
open("/etc/resolv.conf", O_RDONLY)
= 4
fstat64(0x4, 0xbffff3d0)
= -1 ENOSYS (Function not implemented)
fstat(4, {st_mode=S_IFREG|0644, st_size=43, ...}) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40015000
read(4, "search cs.ucl.ac.uk\nnameserver 1"..., 4096) = 43
read(4, "", 4096)
= 0
close(4)
= 0
munmap(0x40015000, 4096)
= 0
socket(PF_UNIX, SOCK_STREAM, 0)
= 4
connect(4, {sin_family=AF_UNIX, path="
close(4)
= 0
open("/etc/nsswitch.conf", O_RDONLY)
= 4

74

CHAPTER 3. THE BRIEF LIFE OF A PACKET

fstat(4, {st_mode=S_IFREG|0644, st_size=1744, ...}) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40015000
read(4, "#\n# /etc/nsswitch.conf\n#\n# An ex"..., 4096) = 1744
read(4, "", 4096)
= 0
close(4)
= 0
munmap(0x40015000, 4096)
= 0
open("/etc/ld.so.cache", O_RDONLY)
= 4
fstat(4, {st_mode=S_IFREG|0644, st_size=19066, ...}) = 0
old_mmap(NULL, 19066, PROT_READ, MAP_PRIVATE, 4, 0) = 0x40015000
close(4)
= 0
open("/lib/libnss_files.so.2", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0755, st_size=246652, ...}) = 0
read(4, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0p \0\000"..., 4096) = 4096
brk(0x805f000)
= 0x805f000
old_mmap(NULL, 36384, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0x4011e000
mprotect(0x40126000, 3616, PROT_NONE)
= 0
old_mmap(0x40126000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0x7000) = 0x40
close(4)
= 0
munmap(0x40015000, 19066)
= 0
open("/etc/host.conf", O_RDONLY)
= 4
fstat(4, {st_mode=S_IFREG|0644, st_size=26, ...}) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40015000
read(4, "order hosts,bind\nmulti on\n", 4096) = 26
read(4, "", 4096)
= 0
close(4)
= 0
munmap(0x40015000, 4096)
= 0
open("/etc/hosts", O_RDONLY)
= 4
fcntl(4, F_GETFD)
= 0
fcntl(4, F_SETFD, FD_CLOEXEC)
= 0
fstat(4, {st_mode=S_IFREG|0644, st_size=80, ...}) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40015000
read(4, "127.0.0.1\tlocalhost.localdomain\t"..., 4096) = 80
read(4, "", 4096)
= 0
close(4)
= 0
munmap(0x40015000, 4096)
= 0
open("/etc/ld.so.cache", O_RDONLY)
= 4
fstat(4, {st_mode=S_IFREG|0644, st_size=19066, ...}) = 0
old_mmap(NULL, 19066, PROT_READ, MAP_PRIVATE, 4, 0) = 0x40015000
close(4)
= 0
open("/lib/libnss_nisplus.so.2", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0755, st_size=252234, ...}) = 0
read(4, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\20\37\0"..., 4096) = 4096
old_mmap(NULL, 41972, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0x40127000
mprotect(0x40130000, 5108, PROT_NONE)
= 0
old_mmap(0x40130000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0x8000) = 0x40
close(4)
= 0
open("/lib/libnsl.so.1", O_RDONLY)
= 4
fstat(4, {st_mode=S_IFREG|0755, st_size=370141, ...}) = 0
read(4, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\20?\0\000"..., 4096) = 4096
old_mmap(NULL, 88104, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0x40132000

3.5. SOCKET SYSTEM CALL TRACE

75

mprotect(0x40144000, 14376, PROT_NONE) = 0
old_mmap(0x40144000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0x11000) = 0x40144000
old_mmap(0x40146000, 6184, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x4
close(4)
= 0
munmap(0x40015000, 19066)
= 0
uname({sys="Linux", node="ovavu.cs.ucl.ac.uk", ...}) = 0
open("/var/nis/NIS_COLD_START", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/var/nis/NIS_COLD_START", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)
= 4
fstat(4, {st_mode=S_IFREG|0644, st_size=19066, ...}) = 0
old_mmap(NULL, 19066, PROT_READ, MAP_PRIVATE, 4, 0) = 0x40015000
close(4)
= 0
open("/lib/libnss_nis.so.2", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0755, st_size=255963, ...}) = 0
read(4, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0`\37\0\000"..., 4096) = 4096
old_mmap(NULL, 38488, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0x40148000
mprotect(0x40150000, 5720, PROT_NONE)
= 0
old_mmap(0x40150000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0x7000) = 0x40150000
close(4)
= 0
munmap(0x40015000, 19066)
= 0
uname({sys="Linux", node="ovavu.cs.ucl.ac.uk", ...}) = 0
open("/etc/ld.so.cache", O_RDONLY)
= 4
fstat(4, {st_mode=S_IFREG|0644, st_size=19066, ...}) = 0
old_mmap(NULL, 19066, PROT_READ, MAP_PRIVATE, 4, 0) = 0x40015000
close(4)
= 0
open("/lib/libnss_dns.so.2", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0755, st_size=67580, ...}) = 0
read(4, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\300\r\0"..., 4096) = 4096
old_mmap(NULL, 15088, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0x40152000
mprotect(0x40155000, 2800, PROT_NONE)
= 0
old_mmap(0x40155000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0x2000) = 0x40155000
close(4)
= 0
munmap(0x40015000, 19066)
= 0
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 4
connect(4, {sin_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("128.16.5.31")}}, 16) = 0
send(4, "V\'\1\0\0\1\0\0\0\0\0\0\5bells\2cs\3ucl\2ac\2uk\0"..., 36, 0) = 36
time(NULL)
= 960371553
poll([{fd=4, events=POLLIN, revents=POLLIN}], 1, 5000) = 1
recvfrom(4, "V\'\205\200\0\1\0\1\0\4\0\6\5bells\2cs\3ucl\2ac\2uk\0"..., 1024, 0, {sin_family=AF_IN
close(4)
= 0
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 4
connect(4, {sin_family=AF_INET, sin_port=htons(1025), sin_addr=inet_addr("128.16.5.31")}}, 16) = 0
getsockname(4, {sin_family=AF_INET, sin_port=htons(1031), sin_addr=inet_addr("128.16.6.226")}}, [1
close(4)
= 0
bind(3, {sin_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("128.16.6.226")}}, 16) = 0
setsockopt(3, IPPROTO_RAW1, [-6202], 4) = 0
getpid()
= 4510
fstat(1, {st_mode=S_IFREG|0664, st_size=8905, ...}) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40015000
rt_sigaction(SIGINT, {0x804a670, [], SA_INTERRUPT|0x4000000}, NULL, 8) = 0

76

CHAPTER 3. THE BRIEF LIFE OF A PACKET

rt_sigaction(SIGALRM, {0x8049e74, [], SA_INTERRUPT|0x4000000}, NULL, 8) = 0
gettimeofday({960371553, 403056}, NULL) = 0
gettimeofday({960371553, 403316}, NULL) = 0
sendmsg(3, {msg_name(16)={sin_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("128
rt_sigaction(SIGALRM, {0x804a670, [], SA_INTERRUPT|0x4000000}, NULL, 8) = 0
setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={10, 0}}, NULL) = 0
time(NULL)
= 960371553
recvfrom(3, "E\0\0TR\250\0\0\377\1\\\337\200\20\5\37\200\20\6\342\0"..., 192, 0, {sin_fam
gettimeofday({960371553, 407464}, NULL) = 0
brk(0x8060000)
= 0x8060000
socket(PF_UNIX, SOCK_STREAM, 0)
= 4
connect(4, {sin_family=AF_UNIX, path="
close(4)
= 0
open("/etc/hosts", O_RDONLY)
= 4
fcntl(4, F_GETFD)
= 0
fcntl(4, F_SETFD, FD_CLOEXEC)
= 0
fstat(4, {st_mode=S_IFREG|0644, st_size=80, ...}) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40016000
read(4, "127.0.0.1\tlocalhost.localdomain\t"..., 4096) = 80
read(4, "", 4096)
= 0
close(4)
= 0
munmap(0x40016000, 4096)
= 0
open("/var/nis/NIS_COLD_START", O_RDONLY) = -1 ENOENT (No such file or directory)
uname({sys="Linux", node="ovavu.cs.ucl.ac.uk", ...}) = 0
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 4
connect(4, {sin_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("128.16.5.31")}},
send(4, "V(\1\0\0\1\0\0\0\0\0\0\00231\0015\00216\003128\7in-add"..., 42, 0) = 42
time(NULL)
= 960371553
poll([{fd=4, events=POLLIN, revents=POLLIN}], 1, 5000) = 1
recvfrom(4, "V(\205\200\0\1\0\1\0\3\0\3\00231\0015\00216\003128\7in"..., 1024, 0, {sin_fa
close(4)
= 0
write(1, "PING bells.cs.ucl.ac.uk (128.16."..., 159PING bells.cs.ucl.ac.uk (128.16.5.31)
64 bytes from bells.cs.ucl.ac.uk (128.16.5.31): icmp_seq=0 ttl=255 time=4.1 ms
) = 159
rt_sigaction(SIGALRM, {SIG_IGN}, NULL, 8) = 0
write(1, "\n", 1
)
= 1
write(1, "--- bells.cs.ucl.ac.uk ping stat"..., 141--- bells.cs.ucl.ac.uk ping statistic
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 4.1/4.1/4.1 ms
) = 141
munmap(0x40015000, 4096)
= 0
_exit(0)
= ?