                               Mirroring FreeBSD

  Jun Kuriyama

   <kuriyama@FreeBSD.org>

  Valentino Vaschetto

   <logo@FreeBSD.org>

  Daniel Lang

   <dl@leo.org>

  Daniel Lang

   <kensmith@FreeBSD.org>

   Revision: 71530d581c

   FreeBSD *** FreeBSD************************.

   ***************************************************************************************************.******************************,******
   FreeBSD Project ***************************,************ "(TM)" *** "(R)"
   ***************.

   Last modified on 2018-02-03 05:50:22 +0000 by Ruey-Cherng Yu.
   ******

   ********************* mirror
   FreeBSD,******************************************************************************.

   [ ************ / ************ ]

     ----------------------------------------------------------------------

   ************

   1. ************

   2. FreeBSD mirrors *********

   3. ******Mirror FreeBSD ******

   4. Where to mirror from

   5. Official Mirrors

   6. Some statistics from mirror sites

  ******:

   ************************Mirror***************.

1. ************

   The Mirror System Coordinators can be reached through email at
   <mirror-admin@FreeBSD.org>. There is also a FreeBSD mirror sites mailing
   lists.

2. FreeBSD mirrors *********

  2.1. ************

   ***************************. *************** mirror
   ************,_CPU******
   ,************************************.*************** ******
   *********************
   mirror._*********************************._***************************************************
   (*** 10.4-RELEASE/11.1-RELEASE
   )._***************************************************._************************
   ( ***10-20% ) ************._**************************

     * *************************** FTP ***********1.4 TB

     * CTM deltas** 10 GB

     * ******** 1GB

   ****** FTP Distribution *********************
   ftp://ftp.FreeBSD.org/pub/FreeBSD/dir.sizes ******._

  2.2. ************/******

   ***************************** Internet._
   ******************************************** mirror ************._
   ************ mirror ************ FreeBSD ************************ intranet
   *****************
   ******************************************************************._
   ************ official mirror
   *****************************************************._*******************************************************

     * *************************************
       *********************************** < 2Mbps ************************

     * *********************: 34Mbps ******************.

     * ************: > 100Mbps
       ************,************************************************************.

  2.3. **************CPU**RAM

   ************************************,************************************._*********************************************.*********
   FTP *** HTTP
   ************************************._*********************rsync.
   *************** CPU
   **************************************************************************._
   ************************************************._

   ************************************ rsync*****************************
   800Mhz *** 1Ghz******************** 512MB RAM**************************
   ****** *********************.

   ****************************************************** RAM
   (2GB******************) ************************ CPU **
   *************************** SMP ******._

   ***************************************************._********* SVN
   repository ********************************* ( ************
   RAID)._****************************** SCSI
   **************************************************************************************************._

  2.4. ***************

   *********************************************************._**************************************************************************************************._************************************************************************._

    2.4.1. FTP (***************FTP*********)

   ******************************._***************************************
   FTP distributions ._ FTP **************************
   ***************/************
   ********************************************************* ***********
   ************ FreeBSD **************._********FreeBSD archive
   ***************/pub/FreeBSD***._

   ********************************************************* FTP ******
   *******************._

     * /usr/libexec/ftpd: FreeBSD ********* ftpd ************._************
       ftpd(8)._

     * ftp/ncftpd._***********************************************._

     * ftp/oftpd***************************************** ftpd._

     * ftp/proftpd************************************** ftpd._

     * ftp/pure-ftpd: ****************************** ftpd._

     * ftp/twoftpd********._

     * ftp/vsftpd**"***************" ftpd._

   FreeBSD *** ftpd,_proftpd ********* ncftpd ****************** FTP
   ******._************************************************._******************************,********************************************************************************************************._

    2.4.2. Rsync (***FTP***************)

   Rsync ********************* FreeBSD
   ************FTP***********************************************************************._***************
   FTP
   ******************************************************************************************************************************._Rsync
   ************************._************************************************************._Rsync
   ************ rsh *** ssh ******************************
   **************************************************rsync**************************._***************,_***************************._***********************************

     * net/rsync

    2.4.3. HTTP****************FTP ***********************

   ****************** FreeBSD
   ***********************************************._*********************
   HTTP ****** FTP
   *********._************************************************************._********************************

     * www/apache22**Apache
       ***************************************************._ ****** FreeBSD
       ******************._

     * www/thttpd*************************************************************************
       thttpd *************************** Apache ************._*********
       FreeBSD *********************************._

     * www/boa**Boa *** thttpd *** Apache
       *********************._**************************************************
       Apache
       ***************._**************************************************
       thttpd ***************FreeBSD ************._

     * www/nginx**Nginx
       ***************************************************************************************************************************************************
       HTTP ***********HTTP
       *****************************************************************(request
       throtting)****************************SSL ********* HTTP
       ************._

3. ******Mirror FreeBSD ******

   ****************************************************************************._:-)
   *************************** mirror FreeBSD
   ************************************************** mirror._

  3.1. ****** FTP ***

   FTP ************************************
   mirror._*******************************************************************************************************************ISO*********
   *********** live *********************************** port tree
   *********._*********************** FreeBSD ***************CPU******._

   The best way to mirror the FTP area is rsync. You can install the port
   net/rsync and then use rsync to sync with your upstream host. rsync is
   already mentioned in *** 2.4.2, "Rsync (***FTP***************)". Since
   rsync access is not required, your preferred upstream site may not allow
   it. You may need to hunt around a little bit to find a site that allows
   rsync access.

  ******:

   ****** rsync
   ********************************************************************************************************************._******
   mirror ***************************** mirror
   *******************************************************************************************************
   mirror *****._

   ************mirror FreeBSD*********************:

 % rsync -vaHz --delete rsync://ftp4.de.FreeBSD.org/FreeBSD/ /pub/FreeBSD/

   Consult the documentation for rsync, which is also available at
   http://rsync.samba.org/, about the various options to be used with rsync.
   If you sync the whole module (unlike subdirectories), be aware that the
   module-directory (here "FreeBSD") will not be created, so you cannot omit
   the target directory. Also you might want to set up a script framework
   that calls such a command via cron(8).

  3.2. Mirroring ******

   FreeBSD *********************rsync*********mirror.

   ****** mirror FreeBSD **************************************

 % rsync -vaHz --delete rsync://bit0.us-west.freebsd.org/FreeBSD-www-data/ /usr/local/www/

  3.3. Mirroring ******

   **********************************************************FreeBSD
   *************************** mirror ******.
   ************************************************** pkg(8) ****** HTTP
   proxy
   ***************************._****************************************************************

 % pkg fetch -d -o /usr/local/mirror vim

   *************************************************************************************

 % pkg repo /usr/local/mirror

   *****************************************************************************
   HTTP
   ***************************************._*****************************
   pkg(8) *** man pages***********pkg-repo(8) ******._

  3.4. ***************mirror?

   Every mirror should be updated at a minimum of once per day. Certainly a
   script with locking to prevent multiple runs happening at the same time
   will be needed to run from cron(8). Since nearly every admin does this in
   their own way, specific instructions cannot be provided. It could work
   something like this:

    1. Put the command to run your mirroring application in a script. Use of
       a plain /bin/sh script is recommended.

    2. Add some output redirections so diagnostic messages are logged to a
       file.

    3. Test if your script works. Check the logs.

    4. Use crontab(1) to add the script to the appropriate user's crontab(5).
       This should be a different user than what your FTP daemon runs as so
       that if file permissions inside your FTP area are not world-readable
       those files can not be accessed by anonymous FTP. This is used to
       "stage" releases - making sure all of the official mirror sites have
       all of the necessary release files on release day.

   Here are some recommended schedules:

     * FTP fileset: daily

     * WWW pages: daily

4. Where to mirror from

   This is an important issue. So this section will spend some effort to
   explain the backgrounds. We will say this several times: under no
   circumstances should you mirror from ftp.FreeBSD.org.

  4.1. A few words about the organization

   Mirrors are organized by country. All official mirrors have a DNS entry of
   the form ftpN.CC.FreeBSD.org. CC (i.e. country code) is the top level
   domain (TLD) of the country where this mirror is located. N is a number,
   telling that the host would be the Nth mirror in that country. (Same
   applies to wwwN.CC.FreeBSD.org, etc.) There are mirrors with no CC part.
   These are the mirror sites that are very well connected and allow a large
   number of concurrent users. ftp.FreeBSD.org is actually two machines, one
   currently located in Denmark and the other in the United States. It is NOT
   a master site and should never be used to mirror from. Lots of online
   documentation leads "interactive"users to ftp.FreeBSD.org so automated
   mirroring systems should find a different machine to mirror from.

   Additionally there exists a hierarchy of mirrors, which is described in
   terms of tiers. The master sites are not referred to but can be described
   as Tier-0. Mirrors that mirror from these sites can be considered Tier-1,
   mirrors of Tier-1-mirrors, are Tier-2, etc. Official sites are encouraged
   to be of a low tier, but the lower the tier the higher the requirements in
   terms as described in *** 2, "FreeBSD mirrors *********". Also access to
   low-tier-mirrors may be restricted, and access to master sites is
   definitely restricted. The tier-hierarchy is not reflected by DNS and
   generally not documented anywhere except for the master sites. However,
   official mirrors with low numbers like 1-4, are usually Tier-1 (this is
   just a rough hint, and there is no rule).

  4.2. Ok, but where should I get the stuff now?

   Under no circumstances should you mirror from ftp.FreeBSD.org. The short
   answer is: from the site that is closest to you in Internet terms, or
   gives you the fastest access.

    4.2.1. I just want to mirror from somewhere!

   If you have no special intentions or requirements, the statement in
   *** 4.2, "Ok, but where should I get the stuff now?" applies. This means:

    1. Check for those which provide fastest access (number of hops,
       round-trip-times) and offer the services you intend to use (like
       rsync).

    2. Contact the administrators of your chosen site stating your request,
       and asking about their terms and policies.

    3. Set up your mirror as described above.

    4.2.2. I am an official mirror, what is the right site for me?

   In general the description in *** 4.2.1, "I just want to mirror from
   somewhere!" still applies. Of course you may want to put some weight on
   the fact that your upstream should be of a low tier. There are some other
   considerations about official mirrors that are described in *** 5,
   "Official Mirrors".

    4.2.3. I want to access the master sites!

   If you have good reasons and good prerequisites, you may want and get
   access to one of the master sites. Access to these sites is generally
   restricted, and there are special policies for access. If you are already
   an official mirror, this certainly helps you getting access. In any other
   case make sure your country really needs another mirror. If it already has
   three or more, ask the "zone administrator" (<hostmaster@CC.FreeBSD.org>)
   or FreeBSD mirror sites mailing lists first.

   Whoever helped you become, an official should have helped you gain access
   to an appropriate upstream host, either one of the master sites or a
   suitable Tier-1 site. If not, you can send email to
   <mirror-admin@FreeBSD.org> to request help with that.

   There is one master site for the FTP fileset.

      4.2.3.1. ftp-master.FreeBSD.org

   This is the master site for the FTP fileset.

   ftp-master.FreeBSD.org provides rsync access, in addition to FTP. Refer to
   *** 3.1, "****** FTP ***".

   Mirrors are also encouraged to allow rsync access for the FTP contents,
   since they are Tier-1-mirrors.

5. Official Mirrors

   Official mirrors are mirrors that

     * a) have a FreeBSD.org DNS entry (usually a CNAME).

     * b) are listed as an official mirror in the FreeBSD documentation (like
       handbook).

   So far to distinguish official mirrors. Official mirrors are not
   necessarily Tier-1-mirrors. However you probably will not find a
   Tier-1-mirror, that is not also official.

  5.1. Special Requirements for official (tier-1) mirrors

   It is not so easy to state requirements for all official mirrors, since
   the project is sort of tolerant here. It is more easy to say, what
   official tier-1 mirrors are required to. All other official mirrors can
   consider this a big should.

   Tier-1 mirrors are required to:

     * carry the complete fileset

     * allow access to other mirror sites

     * provide FTP and rsync access

   Furthermore, admins should be subscribed to the FreeBSD mirror sites
   mailing lists. See this link for details, how to subscribe.

  ******:

   It is very important for a hub administrator, especially Tier-1 hub
   admins, to check the release schedule for the next FreeBSD release. This
   is important because it will tell you when the next release is scheduled
   to come out, and thus giving you time to prepare for the big spike of
   traffic which follows it.

   It is also important that hub administrators try to keep their mirrors as
   up-to-date as possible (again, even more crucial for Tier-1 mirrors). If
   Mirror1 does not update for a while, lower tier mirrors will begin to
   mirror old data from Mirror1 and thus begins a downward spiral... Keep
   your mirrors up to date!

  5.2. How to become official then?

   We are not accepting any new mirrors at this time.

6. Some statistics from mirror sites

   Here are links to the stat pages of your favorite mirrors (a.k.a. the only
   ones who feel like providing stats).

  6.1. FTP site statistics

     * ftp.is.FreeBSD.org - <hostmaster@is.FreeBSD.org> - (Bandwidth) (FTP
       processes) (HTTP processes)

     * ftp2.ru.FreeBSD.org - <mirror@macomnet.ru> - (Bandwidth) (HTTP and FTP
       users)
