Linux Network Administrators Guide, 2nd Edition
By Olaf Kirch Terry Dawson 2nd Edition June 2000
1-56592-400-2, Order Number: 4002 506 pages, 34.95
Chapter 22 NNTP and the nntpd Daemon
Contents:
The NNTP Protocol Installing the NNTP Server
Restricting NNTP Access NNTP Authorization
nntpd Interaction with C News
Network News Transfer Protocol NNTP provides for a vastly different approach to news exchange from C News and other news servers without native NNTP support. Rather than rely on a batch technology like
UUCP to transfer news articles between machines, it allows articles to be exchanged via an interactive network connection. NNTP is not a particular software package, but an Internet standard described in
RFC-977. It is based on a stream-oriented connection, usually over TCP, between a client anywhere in the network and a server on a host that keeps Netnews on disk storage. The stream connection allows the client
and server to interactively negotiate article transfer with nearly no turnaround delay, thus keeping the number of duplicate articles low. Together with the Internets high-transfer rates, this adds up to a news
transport that surpasses the original UUCP networks by far. While some years ago it was not uncommon for an article to take two weeks or more before it arrived in the last corner of Usenet; it is now often less
than two days. On the Internet itself, it is even within the range of minutes.
Various commands allow clients to retrieve, send, and post articles. The difference between sending and posting is that the latter may involve articles with incomplete header information; it generally means that
the user has just written the article.[ 1
] Article retrieval may be used by news transfer clients as well as newsreaders. This makes NNTP an excellent tool for providing news access to many clients on a local
network without going through the contortions that are necessary when using NFS. [1] When posting an article over NNTP, the server always adds at least one header field,
NNTP-Posting-Host: . The field contains the clients hostname.
NNTP also provides for an active and a passive way to transfer news, colloquially called pushing and pulling. Pushing is basically the same as the ihavesendme protocol used by C News described in
Chapter 21, C News. The client offers an article to the server through the IHAVE msgid command, and the server returns a response code that indicates whether it already has the article or if it wants it. If the server
wants the article, the client sends the article, terminated by a single dot on a separate line.
Pushing news has the single disadvantage that it places a heavy load on the server system, since the system has to search its history database for every single article.
http:www.oreilly.comcataloglinag2bookch22.html 1 of 10 [2202001 11:05:41 AM]
The opposite technique is pulling news, in which the client requests a list of all available articles from a group that have arrived after a specified date. This query is performed by the NEWNEWS command. From
the returned list of message IDs, the client selects those articles it does not yet have, using the ARTICLE command for each of them in turn.
Pulling news needs tight control by the server over which groups and distributions it allows a client to request. For example, it has to make sure that no confidential material from newsgroups local to the site is
sent to unauthorized clients.
There are also a number of convenience commands for newsreaders that permit them to retrieve the article header and body separately, or even single header lines from a range of articles. This lets you keep all news
on a central host, with all users on the presumably local network using NNTP-based client programs for reading and posting. This is an alternative to exporting the news directories via NFS, as described in
Chapter 21.
An overall problem of NNTP is that it allows a knowledgeable person to insert articles into the news stream with false sender specification. This is called news faking or spoofing.[
2 ] An extension to NNTP allows
you to require user authentication for certain commands, providing some measure of protection against people abusing your news server in this way.
[2] The same problem exists with the Simple Mail Transfer Protocol SMTP, although most mail transport agents now provide mechanisms to prevent spoofing.
There are a number of NNTP packages. One of the more widely known is the NNTP daemon, also known as the reference implementation. Originally, it was written by Stan Barber and Phil Lapsley to illustrate the
details of RFC-977. As with much of the good software available today, you may find it prepackaged for your Linux distribution, or you can obtain the source and compile it yourself. If you choose to compile it
yourself, you will need to be quite familiar with your distribution to ensure you configure all of the file paths correctly.
The nntpd package has a server, two clients for pulling and pushing news, and an inews replacement. They live in a B News environment, but with a little tweaking, they will be happy with C News, too. However, if
you plan to use NNTP for more than offering newsreaders access to your news server, the reference implementation is not really an option. We will therefore discuss only the NNTP daemon contained in the
nntpd package and leave out the client programs.
If you wish to run a large news site, you should look at the InterNet News package, or INN, that was written by Rich Salz. It provides both NNTP and UUCP-based news transport. News transport is definitely
better than nntpd. We discuss INN in detail in Chapter 23, Internet News.
The NNTP Protocol
Weve mentioned two NNTP commands that are key to how news articles are pushed or pulled between servers. Now well look at these in the context of an actual NNTP session to show you how simple the
protocol is. For the purposes of our illustration, well use a simple telnet client to connect to an INN-based news server at the Virtual Brewery called news.vbrew.com. The server is running a minimal configuration
to keep the examples short. Well look at how to complete the configuration of this server in Chapter 23. In our testing well be very careful to generate articles in the junk newsgroup only, to avoid disturbing anyone
else.
Connecting to the News Server
Connecting to the news server is a simple as opening a TCP connection to its NNTP port. When you are connected, you will be greeted with a welcome banner. One of the first commands you might try is help.
The response you get generally depends upon whether the server believes you are a remote NNTP server or a newsreader, as there are different command sets required. You can change your operating mode using the
mode command; well look at that in a moment:
http:www.oreilly.comcataloglinag2bookch22.html 2 of 10 [2202001 11:05:41 AM]
telnet news.vbrew.com nntp Trying 172.16.1.1...
Connected to localhost. Escape character is ].
200 news.vbrew.com InterNetNews server INN 1.7.2 08-Dec-1997 ready help
100 Legal commands authinfo
help ihave
check takethis
list mode
xmode quit
head stat
xbatch xpath
xreplic For more information, contact usenet at this machine.
.
The responses to NNTP commands always end with a period . on a line by itself. The numbers you see in the output listing are response codes and are used by the server to indicate success or failure of a command.
The response codes are described in RFC-977; well talk about the most important ones as we proceed.
Pushing a News Article onto a Server
We mentioned the IHAVE command when we talked about pushing news articles onto a news server. Lets now have a look at how the IHAVE command actually works:
ihave 123456gw.vk2ktj.ampr.org 335
From: terrygw.vk2ktj.ampr.org Subject: test message sent with ihave
Newsgroups: junk Distribution: world
Path: gw.vk2ktj.ampr.org Date: 26 April 1999
Message-ID: 123456gw.vk2ktj.ampr.org Body:
This is a test message sent using the NNTP IHAVE command. .
235 All NNTP commands are case insensitive, so you may enter them in either upper- or lowercase. The
IHAVE command takes one mandatory argument, it being the Message ID of the article that is being pushed. Every news article is assigned a unique message ID when it is created. The IHAVE command
provides a way of the NNTP server to say which articles it has when it wants to push articles to another server. The sending server will issue an IHAVE command for each article it wishes to push. If the
command response code generated by the receiving NNTP server is in the 3xx range, the sending NNTP server will transmit the complete article, including its full header, terminating the article with a period on a
line by itself. If the response code was in the 4xx range, the receiving server has chosen not to accept this article, possibly because it already has it, or because of some problem, such as running out of disk space.
http:www.oreilly.comcataloglinag2bookch22.html 3 of 10 [2202001 11:05:41 AM]
When the article has been transmitted, the receiving serve issues another response code indicating whether the article transmission was successful.
Changing to NNRP Reader Mode
Newsreaders use their own set of commands when talking to a news server. To activate these commands, the news server has to be operating in reader mode. Most news servers default to reader mode, unless the
IP address of the connecting host is listed as a news-forwarding peer. In any case, NNTP provides a command to explicitly switch into reader mode:
mode reader 200 news.vbrew.com InterNetNews NNRP server INN 1.7.2 08-Dec-1997 ready
posting ok. help
100 Legal commands authinfo user Name|pass Password|generic prog args
article [MessageID|Number] body [MessageID|Number]
date group newsgroup
head [MessageID|Number] help
ihave last
list [active|active.times|newsgroups|distributions|distrib.pats| overview.fmt|subscriptions]
listgroup newsgroup mode reader
newgroups yymmdd hhmmss [GMT] [distributions] newnews newsgroups yymmddhhmmss [GMT] [distributions]
next post
slave stat [MessageID|Number]
xgtitle [group_pattern] xhdr header [range|MessageID]
xover [range] xpat header range|MessageID pat [morepat...]
xpath MessageID Report problems to usenetvlager.vbrew.com
.
NNTP reader mode has a lot of commands. Many of these are designed to make the life of a newsreader easier. We mentioned earlier that there are commands that instruct the server to send the head and the body
of articles separately. There are also commands that list the available groups and articles, and others that allow posting, an alternate means of sending news articles to the server.
Listing Available Groups
The list command lists a number of different types of information; notably the groups supported by the server:
list newsgroups 215 Descriptions in form group description.
control News server internal group junk News server internal group
local.general General local stuff
http:www.oreilly.comcataloglinag2bookch22.html 4 of 10 [2202001 11:05:41 AM]
local.test Local test group .
Listing Active Groups
list active shows each supported group and provides information about them. The two numbers in each line of the output are the high-water mark and the low-water mark -- that is, the highest numbered article and
lowest numbered article in each group. The newsreader is able to form an idea of the number of articles in the group from these. Well talk a little more about these numbers in a moment. The last field in the output
displays flags that control whether posting is allowed to the group, whether the group is moderated, and whether articles posted are actually stored or just passed on. These flags are described in detail in Chapter
23. An example looks like this:
list active 215 Newsgroups in form group high low flags.
control 0000000000 0000000001 y junk 0000000003 0000000001 y
alt.test 0000000000 0000000001 y .
Posting an Article
We mentioned there was a difference between pushing an article and posting an article. When you are pushing an article, there is an implicit assumption that the article already exists, that it has a message
identifier that has been uniquely assigned to it by the server to which it was originally posted, and that it has a complete set of headers. When posting an article, you are creating the article for the first time and the
only headers you supply are those that are meaningful to you, such as the Subject and the Newgroups to which you are posting the article. The news server you post the article on will add all the other headers for
you and create a message ID that it will use when pushing the article onto other servers.
All of this means that posting an article is even easier than pushing one. An example posting looks like this:
post 340 Ok
From: terryrichard.geek.org.au Subject: test message number 1
Newsgroups: junk Body:
This is a test message, please feel free to ignore it. .
240 Article posted Weve generated two more messages like this one to give our following examples some realism.
Listing New Articles
When a newsreader first connects to a new server and the user chooses a newsgroup to browse, the newsreader will want to retrieve a list of new articles, those posted or received since the last login by the
user. The newnews command is used for this purpose. Three mandatory arguments must be supplied: the name of the group or groups to query, the start date, and the start time from which to list. The date and time
are each specified as six-digit numbers, with the most significant information first; yymmdd and hhmmss, respectively:
newnews junk 990101 000000 230 New news follows
7g2o5raa6news.vbrew.com 7g5bhm8f2news.vbrew.com
http:www.oreilly.comcataloglinag2bookch22.html 5 of 10 [2202001 11:05:41 AM]
7g5bk58f3news.vbrew.com .
Selecting a Group on Which to Operate
When the user selects a newsgroup to browse, the newsreader may tell the news server that the group was selected. This simplifies the interaction between newsreader and news server; it removes the need to
constantly send the name of the newsgroup with each command. The group command simply takes the name of the selected group as an argument. Many following commands use the group selected as the
default, unless another newsgroup is specified explicitly:
group junk 211 3 1 3 junk
The group command returns a message indicating the number of active messages, the low-water mark, the high-water mark, and the name of the group, respectively. Note that while the number of active messages
and the high-water mark are the same in our example, this is not often the case; in an active news server, some articles may have expired or been deleted, lowering the number of active messages but leaving the
high-water mark untouched.
Listing Articles in a Group
To address newsgroup articles, the newsreader must know which article numbers represent active articles. The listgroup command offers a list of the active article numbers in the current group, or an explicit group
if the group name is supplied:
listgroup junk 211 Article list follows
1 2
3 .
Retrieving an Article Header Only
The user must have some information about an article before she can know whether she wishes to read it. We mentioned earlier that some commands allow the article header and body to be transferred separately.
The head command is used to request that the server transmit just the header of the specified article to the newsreader. If the user doesnt want to read this article, we havent wasted time and network bandwidth
transferring a potentially large article body unnecessarily.
Articles may be referenced using either their number from the listgroup command or their message identifier:
head 2 221 2 7g5bhm8f2news.vbrew.com head
Path: news.vbrew.comnot-for-mail From: terryrichard.geek.org.au
Newsgroups: junk Subject: test message number 2
Date: 27 Apr 1999 21:51:50 GMT Organization: The Virtual brewery
Lines: 2 Message-ID: 7g5bhm8f2news.vbrew.com
NNTP-Posting-Host: localhost X-Server-Date: 27 Apr 1999 21:51:50 GMT
Body: Xref: news.vbrew.com junk:2
.
http:www.oreilly.comcataloglinag2bookch22.html 6 of 10 [2202001 11:05:41 AM]
Retrieving an Article Body Only
If, on the other hand, the user decides she does want to read the article, her newsreader needs a way of requesting that the message body be transmitted. The body command is used for this purpose. It operates in
much the same way as the head command, except that only the message body is returned:
body 2 222 2 7g5bhm8f2news.vbrew.com body
This is another test message, please feel free to ignore it too.
.
Reading an Article from a Group
While it is normally most efficient to separately transfer the headers and bodies of selected articles, there are occasions when we are better off transferring the complete article. A good example of this is in
applications through which we want to transfer all of the artices in a group without any sort of preselection, such as when we are using an NNTP cache program like leafnode.[
3 ]
[3] leafnode is available by anonymous FTP from wpxx02.toxi.uni-wuerzburg.de in the pub directory.
Naturally, NNTP provides a means of doing this, and not surprisingly, it operates almost identically to the head command as well. The article command also accepts an article number or message ID as an argument,
but returns the whole article including its header:
article 1 220 1 7g2o5raa6news.vbrew.com article
Path: news.vbrew.comnot-for-mail From: terryrichard.geek.org.au
Newsgroups: junk Subject: test message number 1
Date: 26 Apr 1999 22:08:59 GMT Organization: The Virtual brewery
Lines: 2 Message-ID: 7g2o5raa6news.vbrew.com
NNTP-Posting-Host: localhost X-Server-Date: 26 Apr 1999 22:08:59 GMT
Body: Xref: news.vbrew.com junk:1
This is a test message, please feel free to ignore it. .
If you attempt to retrieve an unknown article, the server will return a message with an appropriately coded response code and perhaps a readable text message:
article 4 423 Bad article number
Weve described how the most important NNTP commands are used in this section. If youre interested in developing software that implements the NNTP protocol, you should refer to the relevant RFC documents;
they provide a great deal of detail that we couldnt include here.
Lets now look at NNTP in action through the nntpd server.
http:www.oreilly.comcataloglinag2bookch22.html 7 of 10 [2202001 11:05:41 AM]
Installing the NNTP Server
The NNTP server nntpd may be compiled in two ways, depending on the expected load on the news system. There are no compiled versions available, because of some site-specific defaults that are hardcoded
into the executable. All configuration is done through macros defined in commonconf.h.
nntpd may be configured as either a standalone server that is started at system boot time from an rc file, or a daemon managed by inetd. In the latter case, you have to have the following entry in etcinetd.conf:
nntp stream tcp nowait news usretcin.nntpd nntpd The inetd.conf syntax is described in detail in Chapter 12, Important Network Features. If you configure
nntpd as standalone, make sure that any such line in inetd.conf is commented out. In either case, you have to make sure the following line appears in etcservices:
nntp 119tcp readnews untp Network News Transfer Protocol To temporarily store any incoming articles, nntpd also needs a .tmp directory in your news spool. You
should create it using the following commands:
mkdir varspoolnews.tmp chown news.news varspoolnews.tmp
Restricting NNTP Access
Access to NNTP resources is governed by the file nntp_access in etcnews. Lines in this file describe the access rights granted to foreign hosts. Each line has the following format:
site read|xfer|both|no post|no [exceptgroups]
If a client connects to the NNTP port, nntpd attempts to obtain the hosts fully qualified domain name from its IP address using reverse lookup. The clients hostname and IP address are checked against the site field
of each entry in the order in which they appear in the file. Matches may be either partial or exact. If an entry matches exactly, it applies; if the match is partial, it applies only if there is no other match following
it that is at least as good. site may be specified in one of the following ways:
Hostname This is a fully qualified domain name of a host. If this matches the clients canonical hostname
literally, the entry applies, and all following entries are ignored. IP address
This is an IP address in dotted quad notation. If the clients IP address matches this, the entry applies, and all following entries are ignored.
Domain name This is a domain name, specified as .domain. If the clients hostname matches the domain name, the
entry matches. Network name
This is the name of a network as specified in etcnetworks. If the network number of the clients IP address matches the network number associated with the network name, the entry matches.
Default The string
default matches any client.
Entries with a more general site specification should be specified earlier, because any matches will be overridden by later, more exact matches.
The second and third fields describe the access rights granted to the client. The second field details the permissions to retrieve news by pulling read, and transmit news by pushing xfer. A value of both
enables both; no denies access altogether. The third field grants the client the right to post articles, i.e.,
http:www.oreilly.comcataloglinag2bookch22.html 8 of 10 [2202001 11:05:41 AM]
deliver articles with incomplete header information, which is completed by the news software. If the second field contains no, the third field is ignored.
The fourth field is optional and contains a comma-separated list of groups to which the client is denied access.
This is a sample nntp_access file: by default, anyone may transfer news, but not read or post
default xfer no public.vbrew.com offers public access via modem. We allow
them to read and post to any but the local. groups public.vbrew.com read post local
all other hosts at the brewery may read and post .vbrew.com read post
NNTP Authorization
The nntpd daemon provides a simple authorization scheme. If you capitalize any of the access tokens in the nntp_access file, nntpd requires authorization from the client for the respective operation. For instance,
when specifying a permission of Xfer or XFER, as opposed to xfer, nntpd will not let the client transfer articles to your site unless it passes authorization.
The authorization procedure is implemented by means of a new NNTP command named AUTHINFO. Using this command, the client transmits a username and a password to the NNTP server. nntpd validates
them by checking them against the etcpasswd database and verifies that the user belongs to the nntp group.
The current implementation of NNTP authorization is only experimental and has therefore not been implemented very portably. The result of this is that it works only with plain-style password databases;
shadow passwords are not recognized. If you are compiling from source and have the PAM package installed, the password check is fairly simple to change.
nntpd Interaction with C News
When nntpd receives an article, it has to deliver it to the news subsystem. Depending on whether it was received as a result of an IHAVE or POST command, the article is handed to rnews or inews, respectively.
Instead of invoking rnews, you may also configure it at compile time, to batch the incoming articles and move the resulting batches to varspoolnewsin.coming, where they are left for relaynews to pick them up
at the next queue run.
nntpd has to have access to the history file to be able to properly perform the ihavesendme protocol. At compile time, you have to make sure the path to that file is set correctly. If you use C News, make sure that
C News and nntpd agree on the format of your history file. C News uses dbm hashing functions to access it; however, there are quite a number of different and slightly incompatible implementations of the dbm
library. If C News has been linked with a different dbm library than you have in your standard libc, you have to link nntpd with this library, too.
nntpd and C news disagreement sometimes produces error messages in the system log that nntpd can not open it properly, or you might see duplicate articles being received via NNTP. A good test of a
malfunctioning news transfer is to pick an article from your spool area, telnet to the nntp port, and offer it to nntpd as shown in the next example. Of course, you have to replace msgid with the message ID of the
article you want to feed to nntpd:
telnet localhost nntp Trying 127.0.0.1...
http:www.oreilly.comcataloglinag2bookch22.html 9 of 10 [2202001 11:05:41 AM]
Connected to localhost Escape characters is ].
201 vstout NNTP[auth] server version 1.5.11t 16 November 1991 ready at Sun Feb 6 16:02:32 1194 no posting
IHAVE msgid435 Got it. QUIT
This conversation shows nntpds proper reaction; the message Got it
tells you that it already has this article. If you get a message of 335 Ok instead, the lookup in the history file failed for some reason.
Terminate the conversation by typing Ctrl-D. You can check what has gone wrong by checking the system log; nntpd logs all kinds of messages to the daemon facility of syslog. An incompatible dbm library usually
manifests itself in a message complaining that dbminit failed.
Back to: Sample Chapter Index
Back to: Linux Network Administrators Guide, 2nd Edition
ORe illy Ho m e |
ORe illy B o o k s t o re s |
Ho w t o Ord e r |
ORe illy Co n t a c t s I n t e rn a t io n a l
| Ab o u t ORe illy
| Affilia t e d Co m p a n ie s
© 2 0 0 1 , ORe illy As s o cia t e s , In c. w e b m a s t e ro re illy . co m
http:www.oreilly.comcataloglinag2bookch22.html 10 of 10 [2202001 11:05:41 AM]
Linux Network Administrators Guide, 2nd Edition
By Olaf Kirch Terry Dawson 2nd Edition June 2000
1-56592-400-2, Order Number: 4002 506 pages, 34.95
Chapter 23 Internet News
