
		    README FILE FOR DIABLO TRANSIT FEEDER

				    ---

    DIABLO is a news transit and reader system.  This readme file describes
    the transit side of things.  You can operate the transit and reader
    sides separately or together, or just operate one or the other depending
    on your needs.

    The transit side of Diablo is designed to transit news between one or
    more incoming feeds and one or more outgoing feeds.  The transit side
    of Diablo is not designed to operate as a reader.  However, it may be 
    used to back a reader's spool for readers that fetch articles by message-id
    (which the reader side of Diablo does).  

    Since the transit portion of Diablo is not designed to support readers,
    an active file is not usually used with it ('active off' in diablo.config).
    If you want the transit portion of Diablo to act as the master article 
    number assignment point which downstream sites are slaved off of, you turn
    ON feeder-side active file support in diablo.config and
    run the reader side, dreaderd, with the -x option to slave it's article
    numbering to the master's.  When turned on, the active file only effects
    how the Xref: header is generated by the transit side of Diablo.  The
    transit side of Diablo will still transit articles even if none of the
    newsgroups are listed in the active file.

    The feeder side of diablo does not process control messages, so if you 
    are mastering article numbers on the feeder, you either have to run the
    reader on the same box in order to process control messages and keep the
    active file up to date, or you must periodically synchronize the active
    file from a remote source using dsyncgroups (taking care to use the proper
    options so you do not overwrite the begin/end/other article numbering
    parameters that Diablo uses to master XRef: lines).

    The transit side of Diablo maintains a history file (the reader side does
    not).  This means that the transit side is able to take multiple feeds
    in parallel without transiting duplicate articles.    The reader side's
    ability to handle duplicate articles is limited only to the case where
    the -x option is used on dreaderd.  If the option is not used dreaderd
    CANNOT handle duplicate articles being fed to it.  It is often beneficial
    to run the feeder and reader on the same machine, with the feeder feeding
    the reader internally, even if the feeder is not going to be used as
    a (major) spool cache.  It is more common to run the feeder and the
    reader on the same machine if the feeder is given a large enough spool
    to act as the backend cache for the reader.  Or you may just want to 
    run a feeder-only machine with no reader elements on it at all.

    The transit side of Diablo (the 'feeder') is strictly for news transit
    and does not understand reader-related NNTP commands.

    The Diablo spool, usually /news/spool/news, is maintained by the transit
    system.   You must size the spool and run dexpire as appropriate to your
    needs.  Diablo stores multiple articles per file in the spool in a 
    two-tiered directory structure.  Article files & directories should
    never be directly edited or removed or you risk corrupting the reference
    data Diablo stores in the history file.  You must use dexpire to free
    space in the spool.  Since Diablo does not write out article files singly
    it tends to be much more efficient then INN without CNFS.   It is
    roughly on-par with INN + CNFS, though I personally believe it is better.

    Typically, anyone taking a full feed these days must dedicate a
    machine to it that is separate from the newsreader machine that your 
    users use to read news.  

    The DIABLO transit system is designed to replace the dedicated newsfeeds
    machine and is designed to be a mostly hands-off affair once you get past
    configuring and stabiliing it.  See TUNING_NOTES for machine configuration
    suggestions.

			      OS REQUIREMENTS

    See TUNING_NOTES.

			    WHO SHOULD RUN DIABLO

    If you need to run a USENET news feeding/transit system and/or a
    USENET newsreading system, Diablo may be for you.

			    WHERE TO GET DIABLO

    http://www.openusenet.org/diablo/

			    REPORTING BUGS

    send the bug to:	   diablo-bugs@openusenet.org
    send non-bug stuff to: diablo-users@openusenet.org
    NEWSGROUPS:		   news.software.nntp
    MAILING LISTS:	   http://www.plig.net/mailman/listinfo/diablo-users


			USE OF REALTIME FEEDS AND FEED DELAYS

	If you have several outgoing feeds, you should consider using the
	realtime and delay ( d# ) options in dnntpspool.ctl.  all of your
	local and internal feeds should be realtime.  Cheap external paths
	to the internet can also be realtime.  To reduce the cost of running
	outgoing feeds over your internet transit, you may wish to weight
	the feeds according to cost.  For example, our MAE-WEST connection
	is a lot cheaper then our MCI T3, so I run outgoing feeds with 
	MAE-WEST destinations in realtime and run outgoing feeds which go
	via MCI in batch mode with a 10 second delay.  This way the articles
	may actually propogate to the more expensive destinations via other
	means prior to my actually attempting to send them direct.

	Likewise, if you have T1 and frame customrs, it is usually cheaper
	to supply them with a newsfeed yourself rather then force them to
	go to someone over the internet.  This way they are not eating your
	transit bandwidth on newsfeeds.  A realtime feed to those people is
	best.

			    CATCHING UP AFTER BEING DOWN

	The key item to monitor when catching up on incoming feeds after
	being down for a while is the incoming article rate.  Diablo will
	generate a log line for every 1024 articles received that looks like
	this:

    Jun 24 11:03:59 news1 diablo[18153]: DIABLO uptime=7:46 arts=241.000K tested=0 bytes=1.842G fed=12.613M

	You can calculate the article rate by looking at the delta activity
	from two log lines that are around an hour apart from each other.
	If the article rate is above 9 articles/sec, diablo is catching up
	reasonably well.. as of today, a full feed is around 5 articles/sec.

	With a moderate number of incoming feeds, diablo can do around 30
	articles/sec.  If you have a huge number of incoming feeds that are
	all in catchup, in-kernel filesystem locking will begin to interfere
	with the history file lookups and updates.  Diablo will be able to
	maintain a reasonable history file write transaction rate, but the
	lookup rate will suffer.

	This causes diablo to catch up on articles first without appreciably 
	reducing the backlog at remote sites due to slow check-responses. 
	Once it passes a certain threshold, however, and the load on the
	history file turns to mostly-read rather then read/write, the 
	transaction rate will increase dramatically and diablo will generally 
	be able to cleanup the backlogs very quickly after that.


