DIABLO-FILES(5)

NAME
diablo-files - Synopsis of configuration and control files for Diablo

GENERAL

This manual page provides a synopsis for various diablo configuration and control files. It does not necessarily describe all commands or the specific file format. You can obtain most of the latter by looking at the example config/control files in the sample/ directory. What this manual page does is describe specific featurisms that would otherwise not be readily apparent.

The location of all diablo files is controlled by /news/diablo.config. The location of diablo.config itself can be specified with the DIABLO_CONFIG_PATH environment variable or with the '-C diabloconfigpath' option to any command. Diablo files are typically rooted at either the news root, the library root, or the database root (path_news, path_lib, or path_db in diablo.config). See samples/diablo.config for more information.

DNEWSFEEDS
The dnewsfeeds file specifies how diablo generates outbound feeds as well as specifies how diablo filters incoming feeds. You specify each outbound feed with a label keyword, followed by various other keywords and ending with an end keyword.

alias specifies an outbound newspath alias, which may contain '*' and '?' wildcards. An inbound article whos path element matches any given alias for any given outbound feed is not requeued to that outbound feed.

addgroup and delgroup filters articles to outbound feeds. All add/delgroup commands are run against each newsgroup in the Newsgroups: header for the article and the last add/delgroup command effecting any given newsgroup is applied to the article. Normally this means you start out more general at the beginning (i.e. '*') and become more specific at the end of your list. When an article is cross posted to multiple newsgroups, success with any of the groups will cause the article to be propogated.

NOTE! If you 'addgroup *', then use delgroup and delgroupany to remove groups that you do not want, beware that control messages for ALL groups will still be propogated unless you also do a 'delgroup control.*'.

delgroupany filters articles in a manner similar to delgroup, but if the specified groups exist in the Newsgroups: line *AT ALL*, no distribution to *any* of the other newsgroups will occur. i.e. if you say 'delgroupany alt.warez*', it means that if an article is posted to comp.sys.amiga.misc and alt.warez*, the article will NOT be propogated.

requiregroup An extremely exclusive filter that is the inverse of delgroupany. The article's Newsgroups MUST contain a group matching the wildcard or it will not be propogated. This is generally only used when splitting off control feeds. Please see samples/dnewsfeeds for typical usage.

groupref allows a feed to recursively include a group access list defined by a groupdef command. The groupdef command may occur before OR after the newsfeed that references it, and groupdef's can be recursive. see groupdef for more information.

filter effects INBOUND feeds. Articles coming FROM the specified label (see diablo.hosts on how to tie labels to incomming connections) are rejected if any listed newsgroup matches the wildcard specified in any of the filter commands. You normally use this to reject articles cross posted to your local newsgroups that are propogated from outside entities. Note that the history file is not updated, just in case the article is also brought in from a valid 'local' newsfeed.

maxpath (1.08 or higher) effects OUTBOUND feeds. Diablo will not send an article to an outbound feed if the article's Path: line contains more then the specified number of path elements.

maxcross (1.04 or higher) effects OUTBOUND feeds. Diablo will not send an article to an outbound feed if the article's Newsgroups: line contains more then the specified number of groups.

maxsize (1.04 or higher) effects OUTBOUND feeds. Diablo will not send an article to an outbound feed if the article is larger then the specified number of bytes. You may use 'k', 'm', and 'g' to denote kilobytes, megabytes, and gigabytes. For example, <I>maxsize 100k</I> .

rtflush (1.12 or higher) effects OUTBOUND feeds. Diablo will write the queue file for this feed unbuffered. Generally used along with 'realtime' in dnntpspool.ctl.

There is an arbitrary limit of 256 add/delgroup entries per feed. Currently this is due to the fact that Diablo scans the wildcard list linearly and cannot really support group-specific wildcards (where a feed wants 10,000 specific groups rather then fewer wildcarded groups).

A special label, called ME may be specified. This label generally only contains filter commands. Diablo will revert to this label for any incoming connections that have not been associated with a specific label.

There is also the groupdef command, which must occur outside any feed commands. The groupdef command is followed by a set of groupadd, groupdel, groupdelany, and/or groupref commands to create a grouplist. The groupdef command is terminated by an end command. Grouplist definitions may be referenced by feeds via the groupref command, and may be referenced by other grouplists.

DEXPIRE.CTL

The dexpire.ctl file controls the expiration time of an article based on percentage of best case. Valid expirations range from 1 to 100, where 1 expires the article quickly and 100 keeps the article around as long as possible. Each line consists of a wildcard group, a base expiration percentage, and options. All fields are colon-separated.

Options are inherited. For example, if you place a 2 megabyte article size limit m2m for group '*', this option will effect all matching groups unless overriden by a match later in the file. Thus, you do not have to specify global or hierarchical limits on every line. You can also specify an exponential factor to reduce the expire based on the number of cross posts. A cross post factor of c0.95 is typical. There is no /remember line as with INN. The history retention time is compiled into Diablo and may only be set by modifying lib/vendor.h. The default is to use a 16 day retention (which causes incoming articles over 14 days old to be rejected).

Article expiration percentages essentially insert an article X% of the way into the spool and thus operate in a relative manner. It is not useful to assign 100 to all groups unless you want straight FIFO operation. It would be useful to assign, say, 80 to alt.*, 50 to alt.binaries.*, and 100 to everything else. We strongly suggest that you use only four or five different expire percentages, as the diversity of expire percentages tends to multiply the number of open descriptors each forked diablo process must maintain.

Also, dreaderd uses the 'a' field specification in dexpire.ctl to determine the maximum number of articles that any given group can hold. The default is 1024. See dexpireover.8 and samples/dexpire.ctl for more information.

DEXPIRE.FACTOR
The dexpire.factor file is no longer used by Diablo and can be deleted. It may be used again in the future.

DNNTPSPOOL.CTL
The dnntpspool.ctl file is used by the dspoolout program to manage the outbound queues. Diablo will maintain a single outbound queue file for each feed. dspoolout renames this file to a proper sequenced queue file, properly flushes Diablo's file descriptors, starts up dnewslink programs, and removes any sequenced queue files backlogged beyond the specified limit.

This file allows you to specify the remote host, maximum number of queue files, and maximum number of dnewslink processes to run for each outbound feed, and other options. Please refer to the samples/dnntpspool.ctl file.

DISTRIB.PATS
The distrib.pats file is read and returned by the NNTP 'list distrib.pats' command. It is also used to generate a Distribution: header internally when the POST command does not supply it (or supplies an empty Distributions header).

DISTRIBUTIONS
The distributions file is read and returned by the NNTP 'list distributions' command.

DIABLO.HOSTS
The diablo.hosts file controls authentication for incoming connections. Each line consists of a domain name or IP address wildcard and a label identifying which feed in dnewsfeeds the incoming connection is associated with. THE LABEL IS NO LONGER OPTIONAL. You must supply a label.

Diablo normally requires that the reverse lookup match the forward lookup for security purposes, but many sites set up their reverse to point to a CNAME or set up their reverse to point to an unassociated host yet still request that you put a common hostname in your hosts file which 'resolves' to all the IP addresses of their news machines.

Diablo will attempt to wildcard match the last two domain elements of the reverse domain name with non-wildcard domain names in diablo.hosts then issue a forward lookup of the name in diablo.hosts and attempt to match the IP. In otherwords, if you have an entry for 'newsfeeds.fubar.com' in diablo.hosts and an incoming connection's reverse lookup comes back 'news55.fubar.com', diablo will convert news55.fubar.com to *.fubar.com and attempt to match that against entries in diablo.hosts. When a match occurs, it performs a forward lookup (in this case against 'newsfeeds.fubar.com') and tries to match up the IP address that way.

This methodology has the advantage of not requiring diablo to do a sequential forward lookup of all the entries in diablo.hosts. Each connection's DNS load is consistent only with the domain/IP the connection is coming from which is very important for stability in the face of a large number of feeds.

DREADER.HOSTS
The dreaderd.hosts file contains access permissions for NNTP readers for the dreaderd server. This file also contains access permissions for header-only feeds coming into the dreaderd server.

DSERVER.HOSTS
The dserver.hosts file contains the outgoing spool and posting server configuration which dreaderd uses to make connections to outside servers for message retrieval and outgoing POSTed message propogation.

DQUEUE (directory)
The dqueue directory contains the queue files, both the ones generated by the diablo server and the ones maintained by dspoolout. Files in this directory are generally named as the label (diablo outbound queue file for a label), as the label.Snnnnn (sequenced queue file maintained by dspoolout), or .label.seq (sequence number information maintained by dspoolout).

Note that the diablo queue format has changed as of V1.08. Older versions of diablo dumped the filename and message id. As of V1.08, an third field formatted as OFFSET,BYTES is added to support the new multi-article spool files. DNewslink understands both formats.

FEEDS (directory)
The feeds directory contains automatic group add/del information as requested by a feed through the feedrset and other Diablo commands. If a file exists for any given feed, it overrides the addgroup and delgroup commands in the dnewsfeeds file for that feed.

NEWS SPOOL (/news/spool/news) (directory)
Diablo implements a two-level news spool. A directory of the form D.xxxxxxxx is created on the first level every 10 minutes. Each discrete fork creates a distinct file in one or more of these directories when it receives an article. The directory is chosen based on the expiration and the filename is chosen starting with a hash of the incoming IP. The diablo process then exclusively locks the file(s) in question. In the case of contention, the loser will generate another filename and loop until it finds or creates one. The files or of the form B.xxxx where xxxx is basically random. Multiple articles may be stored in each file. An ascii NUL (code 0x00) is used as an out-of-band article separator. The history and outgoing queue files reference articles by relative file path, offset, and size.

It should be noted that dnewslink explicitly looks for the separator as a double-check against corruption.

DACTIVE.KP (/news/dactive.kp)
The dactive.kp file is a key-token-pair database (see diablo-kp(5)). It is NOT compatible with the INN active file. The contrib/innactive program may be used to generate dactive.kp from an INN active file.

Diablo KP database files are human readable but should only be manipulated with the dkp program while Diablo is active. If Diablo is inactive, KP files may be manipulated with dkp OR can be edited by hand. Diablo's dactive.kp file serves the same purpose as INN's active file but, being a general token=value database, may contain additional information. The intention is to use this database to track article number assigns and to store additional group description, moderator email, and PGP keys for the reader portion of Diablo.

In Diablo, groups missing from dactive.kp do not normally effect the feeder side of the system. However, Xref: headers are only generated for those newsgroups listed in dactive.kp. This behavior can be changed through the 'activedrop' option in diablo.config

The dactive.kp database is keyed off the group name and uses the following tokens, some of which are optional: NE (last stored article number, %010d format), NB (first stored article number %010d format), S (status), M (moderator email), and CPGP (pgp key, hierarchically recursive). The status may contain multiple characters. 'n' indicates a disabled group, 'y' indicates an enabled group, 'm' indicates moderated. itself is entirely optional (defaults to 'y'). The group description, if any, is stored with the GD key. The supplemental program 'inntodiabloactive' can be used to convert the INN active/newsgroups/configuration-files to the diablo dactive.kp file.

When using the feeder to generate Xref: headers, the feeder creates a copy of the NE field called NX and uses that to track article number assignments. This way both the reader and feeder can use the same physical active file when both the reader and feeder are running on the same box. The dsyncgroups command has options to manipulate NE and NX. WARNING! If NX is present, the reader will reset NE if NX < NE and an incoming article has been assigned a number less then NE.

if a group is marked moderated, the moderator email is obtained via the M key. If control messages related to the group (hierarchically speaking), the CPGP key contains the public key. For example, there might be an entry for 'comp' with a CPGP key but since 'comp' is not a real newsgroup, the status would be blank.

SEE ALSO
diablo(8), dclean(8), dicmd(8), didump(8), diload(8), dnewslink(8), doutq(8), dexpire(8), dexpireover(8), diconvhist(8), dilookup(8), dspoolout(8), dkp(8), dpath(8), diablo-files(5), diablo-kp(5)