<!-- $Id: p2cf.odd,v 1.9 1998/01/31 19:14:05 tei Exp $ -->
<!--*
    * $Log: p2cf.odd,v $
    * Revision 1.9  1998/01/31 19:14:05  tei
    * finish note on derived DTDs.
    *
    * Revision 1.8  1998/01/31 18:27:01  tei
    * fix log keyword
    * begin changes to allow derived dtds.
    *
    * Revision 1.7  98/01/28  11:09:58  cmsmcq
    * LB changes of January 1998
    *
    * Revision 1.6  1994/04/08  18:54:36  msmcq
    * Last-second fixes for P3
    *-->
<!-- TEI Doc. No:  ml w43                                       -->
<!-- Title:        TEI Conformance                              -->
<!-- Drafted:      msm, 22 May 1991                             -->
<!-- ********************************************************** -->
<!-- Revision History (add lines at top)                        -->
<!-- 4 Jan 98 : LB : remove reference to tei1 -->
<!-- 94-04-07 : MSM : normalize caps in headings              -->
<!-- 94-04-07 : MSM : Added RCS id                            -->
<!-- 93-06-02 : WP : Spellchecked, made minor corrections.    -->
<!-- 93-04-12 : MSM : cosmetic changes                        -->
<!-- Date      Who    What                                      -->
<!-- 10 Apr 93 LB: ODDified from                                -->
<!-- 30 May 91 CMSMcQ minor corrections                         -->
<!-- 29 May 91 CMSMcQ finished major revision                   -->
<!-- 28 May 91 CMSMcQ major cuts and reorganization (ver 2)     -->
<!-- 22 May 91 CMSMcQ made file                                 -->
<div1 id="CF" n="28"><head>Conformance</head>
<p>The notion of <term>TEI conformance</term>
is intended as an aid in describing the
format and contents of a particular document or set of documents.
These concepts are expected to be useful in:
<list>
<item>agreements for the interchange of documents among researchers</item>
<item>agreements for the deposit of texts in archives and their
distribution from archives</item>
<item>describing the documents to be produced by or for a given project</item>
<item>defining the classes of documents accepted or rejected by a given
piece of software</item>
</list></p>
<p>This chapter describes the areas in which these terms are defined and
specifies their meaning.  It also proposes other terms for related
concepts and points out some dangers in the careless use or application
of these terms.</p>
<div2 id="CFdefns"><head>Definitions of Terms</head>
<p>The terms described here should be considered technical terms for
users and implementors of the TEI Guidelines and should be used only in
the senses given and with the usages described.</p>
<div3 id="CFdoc"><head>TEI-Conformant Document</head>
<p>A document is <term>TEI conformant</term> if it is
either in <term>TEI local processing format</term> or in <term>TEI
interchange format</term>.  A full description of the document should
specify which format it is in.</p>
<p>The term <term>TEI conformance</term> does not apply to software:
programs can be usefully described as <term>accepting</term> or
<term>validating</term> TEI-conformant documents or some subset of
TEI-conformant documents, but the TEI defines no required processing
model against which software could be measured.  Programs are thus not
themselves conformant or non-conformant and should not be so described.</p></div3>
<div3 id="CFlocal"><head>TEI Local Processing Format</head>
<p>A document is in <term>TEI local processing format</term> if
<list>
<item>it is a conforming SGML document with a legal SGML declaration;</item>
<item>it uses the document type declarations provided by the TEI, either
without modifications or with all modifications effected by inclusion
in the DTD subset as described in section <ptr target="dtdmod">;</item>
<item>all modifications to meaning or use of defined tags, and all new
tags, are documented in TEI Tag Set Declarations which accompany the
document, as defined in chapter <ptr target="TD">;
<!-- < fn > The components of a tag set declaration are at    -->
<!-- present defined only in document TEI ED W5; they         -->
<!-- should receive a fuller definition and documentation     -->
<!-- as soon as possible.  The ultimate structure of the      -->
<!-- tag set declaration should agree with the structures     -->
<!-- used in the TEI's own reference manual for the TEI       -->
<!-- scheme.                                                  -->
<!-- < /fn >                                                  --></item>
<item>it includes, in the TEI header, all the elements required by the
TEI declarations for the TEI header;</item>
<item>it contains no markup other than SGML or declared notations for
graphics, tables, figures, etc.  That is, unless a declared notation is
in use, the semantics of any content character in the document are
exhausted by its identity as a graphic character.</item>
</list></p>
<p>A TEI-local-processing-format document may be described as requiring
<term>DTD extensions</term> if it modifies the TEI-supplied DTDs or
SGML prolog in any of the ways described  under
<ptr target="dtdmod">.</p>
<p>The following terms are synonymous:  <term>document in TEI local
processing format</term>, <term>TEI local-processing document</term>,
and <term>TEI local-processing-conformant document</term>.</p></div3>
<div3 id="CFint"><head>TEI Interchange Format</head>
<p>A document is in <term>TEI interchange format</term> if it conforms
to the TEI local-processing format and if further:
<list>
<item>its SGML declaration is either the predefined SGML
declaration for TEI interchange documents or an SGML declaration
which differs from it only in ways allowed by section <ptr
target="sgmlmod">;</item>
<item>it makes no use of any of the following SGML constructs:
<list><item>short references</item>
<item>the RANK feature</item>
<item>omission of generic identifiers in start- and end-tags
<note>This is one of several abbreviations allowed by the SHORTTAG
feature; the others (omission of attribute names under certain
circumstances and omission of non-required attribute values) are allowed.
</note></item>
<item>keywords other than INCLUDE, IGNORE, and CDATA on a marked
section</item>
</list></item>
<item>it includes no SUBDOC subordinate document by means of an entity
reference embedded directly within content data (SUBDOCs must be
included by giving the entity reference as the value of an attribute);
<!-- kill this one?: I dont understand it (LB) --></item>
<item>it does not provide different definitions for the same entity in
different document types.</item>
</list></p>
<p>A TEI-interchange-format document may be described as requiring
<term>DTD extensions</term> if its DTD is modified in any of the ways
described in section <ptr target="dtdmod">.</p>
<p>The following terms are synonymous: <term>document in TEI
interchange format</term>, <term>TEI interchange document</term>, and
<term>TEI interchange-conformant document</term>.</p></div3>
<div3 id="CFex"><head>TEI Packed Interchange format</head>
<p>A document is in TEI packed interchange format with a given
<term>transmission character set</term> and a given
<term>transmission entity set</term> if all of the following are true:
<list>
<item>all separate entities in the document are packed into a single
entity (file) in a manner conforming to ISO 9069 (SDIF) or to some
other TEI-authorized form;</item>
<item>all characters occurring in SGML names (generic identifiers and
attribute names) occur within the transmission character set;
<!-- .  (If this is not true, the SGML names must be modified -->
<!-- so as to make it true.) --></item>
<item>all characters in the document content and attribute values either
occur within the transmission character set or are represented by an
appropriate entity reference using an entity name included in the
transmission entity set;</item>
<item>the transmission character and entity sets are named in the header
of the packed file and in any  accompanying paper documentation.</item>
</list>
Any
pre-transmission processing required to convert a document to meet the
above requirements for conformance to the  TEI-interchange-format  is
called <term>packing</term>.</p>
<p>With prior agreement between parties to an exchange, interchange
documents may use character code set switching as defined in ISO 2022, its
national analogues, or successor standards.</p>
<p>A full description of a document in TEI packed interchange format
<emph>must</emph> specify the transmission character set and the
transmission entity set used in the document.</p></div3>
<div3 id="CFrec"><head>TEI Recommended Practice</head>
<p>A document follows <term>TEI recommended practice</term> if:
<list>
<item>it is a TEI-conformant document;</item>
<item>wherever the guidelines say to prefer one encoding practice to
another, the preferred practice is used;</item>
<item>all textual features which the guidelines recommend be captured
are in fact encoded.
<!-- since there are none: follow DD's proposal to drop following -->
<!-- <item>no textual features which the guidelines recommend not -->
<!--     be captured are in fact encoded --></item>
</list></p></div3>
<div3 id="CFmod"><head>TEI Abstract Model</head>
<p>A document follows the <term>TEI abstract model</term> if it tags
the features specified in the TEI documentation and documentation, and
their structural interrelations agree with those specified in the TEI DTDs.

<!-- end of CF ? --></p></div3></div2>

<div2 id="sgmlmod"><head>Modifications to TEI SGML Declaration</head>
<p>The SGML declaration for TEI interchange documents may differ
from that provided in TEI documentation in these ways:
<list>
<item>the CHARSET clause must be used to define the transmission character set
(possibly in connection with the SHUNCHAR specification in the SYNTAX clause);</item>
<item>the CAPACITY clause may  be used to raise (but not lower) capacities;</item>
<item>the SYNTAX clause may be used to define the SGML syntax used in
      the document.  Notably:
<list>
<item>the SHUNCHAR specification within the SYNTAX clause may be used to
restrict the transmission character set;</item>
<item>the BASESET and DESCSET specifications within the SYNTAX
clause must be used to describe the transmission character set;</item>
<item>the DELIM and NAMES specifications may be used to modify the SGML
syntax.</item>
</list></item>
<item>in the FEATURES clause, CONCUR may be set to NO if concurrent
markup is not used in the document.</item>
</list></p>
<p>The following portions of the SGML declaration may not be modified in
TEI interchange documents:
<list>
<item>the CAPACITY and QUANTITY values may be increased but not decreased;</item>
<item>the SCOPE clause may not be changed;</item>
<item>no new FEATURES may be turned on.</item>
</list></p>
<p>The SGML declaration for TEI-local-format documents may be modified
without restriction.  Some recommendations for usage are made in
document TEI P1, but these recommendations are not normative.</p></div2>

<div2 id="dtdmod"><head>Modifications to TEI Document Type Declarations</head>

<p>A TEI-conformant document (whether for local processing or for
interchange) may make any change to the TEI-supplied document type
declarations which is allowed by SGML and the controlling SGML
declaration.  All such changes should be made (or at least it must
be possible to make them) within the SGML DTD subset, by defining TEI
DTD modifications files as described in chapter <ptr target="MD">,
and embedding the modification files within the DTD subset of a
document whose document type declaration refers to the unmodified TEI
main DTD, as in the following fragment:
<!--*
In all TEI-conformant documents, the document type name must be
<gi>tei</gi>, followed by a digit indicating the currently released
version number for the TEI guidelines in use.  The associated public
or system identifier(s) should refer to a file containing the
unmodified TEI main DTD fragment, as in the following example:
*-->
<eg>
<![CDATA[
<!DOCTYPE tei.2 PUBLIC "-//TEI P3//DTD Main Document Type//EN"
                       "tei2.dtd" [
<!ENTITY % TEI.extensions.ent
           PUBLIC "-//ProjectName//ENTITIES Local modifications to
                   TEI main DTD//EN" >
<!ENTITY % TEI.extensions.dtd
           PUBLIC "-//ProjectName//DTD Local element types for
                   TEI main DTD//EN" >
]>
]]>
</eg></p>
<p>For reasons of convenience, it may be desirable in practice to
create a <term>derived DTD</term> in which the local modifications
have been integrated with the TEI main DTD in a single file.  If such
a  one-file DTD is desired, it should be derived automatically from
the TEI DTD and the local modifications files using appropriate
software, rather than derived by hand-editing the TEI DTD files, as
hand-editing increases the chances of error and inconsistency between
the DTD modifications files and the one-file DTD. Documents in the TEI
interchange format must use the form shown above, with a reference to
the unmodified main TEI DTD and declarations of the local
modifications files.
</p>
<p>The following must remain true of the DTD after modification:
<list>
<item>the overall document must contain a single <gi>teiHeader</gi>
element and a single <gi>text</gi> element, in that order; in the
case of a corpus or collection the overall collection may have a
<gi>teiHeader</gi> followed by a series of <gi>tei</gi> documents;</item>
<item>the <gi>teiHeader</gi> element must include elements for:
<list type="gloss">
<label>title statement</label>
<item>the title of the machine-readable work and the names of those
responsible for it;</item>
<label>publication statement</label>
<item>place and date of publication or distribution of the machine-readable
document;</item>
<label>source description</label>
<item>bibliographic description of the copy text or source of the
electronic text, including at least title, author,
        and edition.</item>
</list></item>
</list></p>

<p>A TEI-conformant document may be said to require <term>DTD
extensions</term> if it:
<list>
<item>defines new elements;</item>
<item>modifies the content model, declared content, or omissibility of any
element;</item>
<item>adds or modifies any attribute definitions;</item>
<item>renames any elements, attributes, or attribute values;</item>
<item>defines any new document types;</item>
<item>declares any non-SGML notations.</item>
</list>
Without requiring DTD extensions, therefore, any TEI document may:
<list>
<item>define entities and parameter entities;</item>
<item>include processing instructions and comments in its DTD subset.</item>
</list>
For local-processing purposes, a TEI document may also, without
requiring DTD extension:
<list>
<item>include link type declarations etc. in its SGML prolog;</item>
<item>define and use short reference mapping in its DTD.</item>
</list>

Note that TEI interchange documents may <emph>not</emph> include link
type or short reference declarations because the SGML declaration for
interchange does not allow them.</p>
<p>It is expected that the notion of DTD extension will be particularly
useful in describing the classes of documents accepted or
validated by software.</p></div2>
<div2 id="procmod"><head>TEI Processing Model</head>

<p>This section is included for illustrative purposes only; it does
not restrict the processing of TEI or other documents.  It simply
distinguishes a number of typical ways in which a project may choose
to apply the TEI Guidelines to different kinds of processing.</p>

<div3 id="pmdoc"><head>Document Capture and Reclamation</head>
<p>First, data might be captured by keyboarding into a locally defined
data capture format, or by scanning into a locally defined scanner-file
format.  From these initial forms, transducers might convert the files
into a standard local storage format.</p></div3>
<div3 id="pmlocal"><head>Local Storage Format and Application Software</head>
<p>The local storage format might be the input format of some
application program used frequently by the project.  In this case,
transducers might be necessary to prepare data for processing by other
applications.  Alternatively, the local storage format might be
independent of the formats used by application programs; transducers
would be needed to prepare data for any processing.  Such an
independent format is useful if the local storage format needs to
contain more information than any single application can conveniently
handle.</p>
<p>The local storage format might be SGML-conformant without being
TEI-conformant, e.g. because it uses local DTDs instead of the standard
TEI DTDs, or because it uses a TEI local processing format.  Local
software may be used to validate a TEI
local-processing format, to transduce documents into the input formats
needed by applications, and when appropriate to transform documents
into the TEI interchange format for exchange with other sites.</p>

<p>Finally, the TEI interchange format may be used as a local storage
format.  It is not expected that this will be a very common practice,
since it is expected that most sites interested in TEI conformance
will eventually acquire SGML-conformant software which allows for a
more compact local storage than does the interchange format.  In the
absence of SGML software, however, some projects may find the TEI
interchange format (or perhaps a restrictive variant of it) useful,
because such a format can be relatively easy to parse with ad hoc
software.</p>

<p>Whether the local storage format is strictly TEI conformant or not,
it may follow TEI-recommended practice in its selection of textual
features to be marked up, in its tag names, in its documentation
practices, etc.</p></div3>
<div3 id="pmrich"><head>Enrichment and Other Processing</head>
<p>Over the course of the project, analysis and processing may result
in interim results which may be incorporated into the locally stored
copy of the text so that the interim results can be used in later
processing.  This process of enrichment can be carried out either by
manual editing of the documents using conventional text editors, or by
application programs. <!--  which perform (part of) the analysis. --></p></div3>
<div3 id="pmexp"><head>Data Export</head>
<p>When a document is to be exchanged with another site using the TEI
interchange format, it must first be transduced from the local storage
format to TEI interchange form.  If local documents are already
TEI-conformant, this requires either no processing at all, or a
relatively simple normalization which can be handled readily by the
normalization facilities of most SGML parsers.  If the local storage
form is non-SGML conformant, some transducer must be used to transform
it into the TEI interchange format.</p>
<p>The TEI-interchange-format document must then be packed for shipping
into the TEI packed interchange format, using a packing program.  This
program will gather the constituent parts (files) of a document into a
single file, and ensure that the file contains no characters whose safe
passage to the recipient of the data is endangered by the transmission
path.  If the ultimate recipient of the document is unknown, the set of
safe characters is very small.  The specific <term>transmission
character set</term> however is independent of TEI conformance:  any
convenient set may be used where both parties agree.  The packer will
ensure that the transmission character set is properly identified.</p></div3>
<div3 id="pmimp"><head>Data Import</head>
<p>When a document is received from another site using the TEI packed
interchange format, it must first be unpacked into a TEI
interchange-format document in the local character set.  It may then be
necessary to <term>naturalize</term> it by translating it into the
local storage format; if the local format is TEI- or SGML-conformant,
no processing is needed (although some SGML processors may offer a
facility for suppressing omissible markup).</p></div3>
<div3 id="CFpm"><head>TEI Conformance in the Processing Model</head>
<p>The notions of TEI interchange format and TEI packed interchange
format are central to the exchange of documents using the TEI
guidelines, whether the local storage format is TEI-conformant or not.
The TEI interchange format and the TEI local-processing format may each
be used as a local storage format, though the local storage format
might well differ from either of these without materially affecting the
use of TEI formats for interchange.  The TEI interchange format being
less flexible than the local-processing format, it is expected that
sites using SGML-conformant software may use the latter, while sites
without such software may prefer the former.</p>
<p>The notion of TEI recommended practice, it is hoped, will be
relevant to decisions about what textual features should be recorded
during data capture and will thus affect data-capture formats and the
transducers which render captured files into the local storage format.</p>
<p>The TEI abstract structure may be useful in developing local
non-SGML markup schemes for data capture or for processing with ad hoc
application programs.  It is strongly recommended that the TEI
recommendations, as well as the TEI abstract structure, be used for
such development as well.</p></div3></div2>
<div2 id="CFasp"><head>Aspects of Conformance and Document Description</head>
<!-- DD asks: Are we back to normative status here? (cf sentence-->
<!-- at end of first para of previous div2) -->
<!-- I'm jiggered if I know...    (LB)  -->
<div3 id="CFch"><head>Character Sets</head>
<p>Neither the character sets used for local processing nor those used
for transmission of interchange documents are restricted by the
definition of TEI conformance.  For local processing, users will
typically use the system character set of their local system or some
modification thereof.  For exchange with known partners, users should
choose any convenient character set; typically the most convenient is
the set of all characters which:
<list>
<item>are transmitted successfully over the existing transmission link;</item>
<item>occur in both sender's and receiver's local coded character sets.</item>
</list></p>
<p>For blind exchange with unknown partners a conservative choice of
transmission set is needed to ensure that characters arrive
correctly.  How conservative the choice need be depends on the medium
of transmission.  The <term>ISO 646 subset</term> defined in section
<ptr target="CHin"> remains the only guaranteed safe set of characters
for the regional and international networks most widely used, although
larger character sets are increasingly coming into use. This is
largely because silent and not always reversible translation between
character sets remains a feature of transmissions across current
disparate networks.  At the present time (1993) therefore, only the
ISO 646 subset is recommended for fully blind interchange, although
the full complement of ISO character sets may be used successfully in
some subdomains.</p>
<p>In transmission by disk or tape, however, no silent translation is
likely to occur, and so larger sets may be successfully used in blind
interchange.  The primary danger is a failure of software in the
receiving machine to process the characters correctly; at this time
(1991), ASCII or 94-character U.S. EBCDIC appear to represent the
largest safe choices; other national character sets may of course be
used if good internal documentation is also provided.</p>
<p>Note that the transmission character set does not associate specific
binary encodings with the characters in the set.  In the technical
senses, it is a character set, not a <emph>coded</emph> character set.
This means that a document may undergo various automatic translations
from one coded character set to another (notably, in the case of
transmission over international networks, from ASCII to EBCDIC or vice
versa) without leaving the <term>transmission character set</term>.
<!-- kill this section? it adds nothing --></p>
<p>For further discussion of the topics addressed in this section, reference
should be made to chapter <ptr target="IN">.</p></div3>
<div3 id="CFsd"><head>SGML Declaration</head>
<p>The utility of various SGML constructs is discussed in section
 2.2 of document TEI P1 version 1.  The restrictions on SGML declarations
and SGML usage in TEI interchange documents discussed above under
<ptr target="sgmlmod"> are derived from that discussion.</p>
<p>No restrictions are made on SGML usage in the local processing
format because such usage is best determined locally and has no impact
on interchange.</p></div3>
<div3 id="CFdtd"><head>SGML Document Type Declaration</head>
<p>The document type declaration provided by the TEI is intended to
cover as wide a variety of document types and processing needs as
proved feasible.  It is impossible, however, for any finite list of
text elements to cover every need of textual research and processing.
As a result, extension of the TEI DTD has no effect
on strict TEI conformance, as long as certain restrictions are
observed; these have the effect of ensuring that later users of a file
can easily see what changes have been made to the DTDs and what the new
tags are intended to mean.</p>
<p>The requirement that all new or modified tags be documented,
however, is formally verifiable only to a limited extent.  It is
possible for a program to verify that for every tag introduced in a DTD
modification, a corresponding record exists in a Tag Set Declaration.
It is impossible, however, to verify using formal means that the entry
in the tag set declaration makes sense.  Purely formal conformance
measures, therefore, must be supplemented with human inspection of the
documentation.</p>
<p>The concept of DTD extension is introduced to allow the concise
description of software which is designed to handle documents encoded
using the published DTDs but which is not prepared to deal with tags
not included there.
<note>Some will regard such simplifications as useful ways of making it
easier to develop software which accepts TEI-conformant documents;
others will deplore the failure of such software to accept all
TEI-conformant documents including those which extend the TEI DTD.  In
providing the notion of <term>DTD extension</term> for describing what
documents are and are not accepted by such software, the TEI acts in
the belief that such software will in fact be developed; it neither
endorses nor deplores its construction or use.
</note></p>
<p>All sections of the TEI DTD are subject to modification by the
user, except that a documentary header must be provided and
distinguished from the text itself, and that documentary header must
include tagged elements identifying the document encoded and those
responsible for the encoding.  This ensures that all TEI-conformant
documents will have at least this bare minimum of accompanying
documentation.</p></div3>
<div3 id="CFtag"><head>Tag Usage and Feature Marking</head>
<p>The basic design principles of the TEI require the notion of
<term>TEI conformance</term> to be applicable to existing electronic
documents if they are translated into a proper format, without
requiring the insertion of information not captured in the initial
preparation of the text.
<note>See document TEI PC P1 <q>The Preparation of Text Encoding
Guidelines.</q>
</note></p>
<p>At the same time, the TEI is charged with formulating advice to
those engaged in the creation of new electronic texts and is required
to distinguish what is actively recommended for general use from what
is merely optional, provided for use by those engaged in a particular
sort of work.</p>
<p>The notion of <term>TEI recommended practice</term> is introduced to
allow the concise description of documents in which not only the
requirements, but also the recommendations of the Guidelines are
followed.  It is hoped that while projects to convert existing
electronic data may content themselves with achieving TEI conformance,
projects to produce new electronic texts will produce documents
following TEI recommended practice.  To distinguish those projects
which follow the TEI's recommendation to use SGML markup from those
which capture the same underlying textual features but do so using
non-SGML markup, the notion of the <term>TEI abstract model</term> is
introduced; it is this which a non-SGML-based encoding can have in
common with the TEI.</p></div3>
<div3 id="CFnon"><head>Non-SGML Markup</head>
<p>In exchanging texts for use by others, the goal of an interchange
format is to ensure that the information encoded in an electronic
version of a text can be correctly understood and processed by the
recipient as well as by the originator of the text.  To assure the
achievement of this goal, the definition offered here of TEI
conformance restricts markup in TEI conformant documents to SGML markup
and to properly declared non-SGML notations.  The latter are explicitly
recommended for the encoding of tables,
figures, etc. and so cannot reasonably be excluded.  Since they do
place a burden on the recipient for proper processing, the use of any
such non-SGML notation is defined to fall within the class of DTD
extensions.</p>
<p>Because of the escape clause for graphics, etc., it is in principle
possible to create a TEI conformant document by embedding a document
using any arbitrary markup into a driver file containing a TEI header
and a declaration for the appropriate markup as a non-SGML notation.
Though it falls within the letter, such a practice falls outside the
spirit of TEI-conformant document interchange.
</p>
</div3></div2></div1>
<!-- Following Glossary shd move to back matter or be junked -->

<!-- OK,  given the choice, I say : junket (LB) -->

<!-- Keep this comment at the end of the file
Local variables:
mode: sgml
sgml-default-dtd-file:"~/lib/psgml/p2odd.ced"
sgml-parent-document:("/home/tei/P3/Master/Odds/p2.odd" "TEI.1" "TEXT" "BODY" "DIV1")
sgml-omittag:t
sgml-shorttag:t
End:
-->
             <!-- 28 TEI Conformance -->
