# TWiki Release 4.0 (Dakar)

'Dakar' is the first major release of the TWiki Enterprise Collaboration Platform in over a year. The focus of this release has been on refactoring the code in the interests of security, efficiency and maintainability, though a range of powerful new features are also included. The refactoring work has included tightening up the specification of certain key TWiki behaviours, which has resulted in some specification changes. The impact on end users has been minimised as far as possible.

Major New Features

- Much simpler install and configuration
- Integrated session support
- Webserver-independent login/logout
- Security sandbox blocks all possible routes for remote command execution on the server
- New editing model allows freer collaboration, without fear of overwriting other people's changes
- Multilingual UI
- E-mail confirmations for registration
- WYSIWYG editor
- Hierarchical sub-webs

See feature list at TWikiHistory#DakarRelease. TWiki 4.0 patch release details are at the end of this release notes document.

Note: In what follows, {This} (words in curly braces) refers to settings in the new 'configure' interface.

## Notes for end users

### Editing at the same time as other people

Dakar release introduces a brand-new strategy for handling simultaneous changes to a topic by several people. Instead of one person locking the topic, and other having to wait until they are finished, Dakar allows multiple simultaneous edits of the same topic, and then merges the different changes.

You probably won't even notice this happening unless you are changing existing text in the file at the same time as someone else. In this case, you may see TWiki inserting "change marks" into the text to highlight conflicts between your edits and another persons. These change marks are only used if you edit the same part of a topic as someone else, and they indicate what the text used to look like, what the other person's edits were, and what your edits were. For example, let's say you have a topic that contains this text:

You edit | You see
--- | ---

    Casablanca is
    Humphrey Bogart's finest film.
    Of all the gin joints in all the world,
    you had to walk into mine.

Casablanca is Humphrey Bogart's finest film. Of all the gin joints in all the world, you had to walk into mine.

and you start editing this text before going for coffee. Meanwhile, a colleague also starts editng the same topic and changes the text to:

    The Maltese Falcon is
    Humphrey Bogart's finest film.
    Of all the gin joints in all the world,
    you had to walk into mine.

The Maltese Falcon is Humphrey Bogart's finest film. Of all the gin joints in all the world, you had to walk into mine.

When you get back from coffee, you finish your edit, changing the text to

    To Have or Have Not is
    Humphrey Bogart's finest film.
    You know how to whistle, don't you Steve?
    You just put your lips together and blow.

To Have or Have Not is Humphrey Bogart's finest film. You know how to whistle, don't you Steve? You just put your lips together and blow.

and saving it. The topic will now look like this when you display it:

    <div class="twikiConflict"><b>CONFLICT</b> original 5:</div>
    Casablanca is
    <div class="twikiConflict"><b>CONFLICT</b> version 6:</div>
    The Maltese Falcon is
    <div class="twikiConflict"><b>CONFLICT</b> version 7:</div>
    To Have or Have Not is
    <div class="twikiConflict"><b>CONFLICT</b> end</div>
    Humphrey Bogart's finest film.
    You know how to whistle, don't you Steve?
    You just put your lips together and blow.

CONFLICT original 5:
Casablanca is
CONFLICT version 6:
The Maltese Falcon is
CONFLICT version 7:
To Have or Have Not is
CONFLICT end
Humphrey Bogart's finest film. You know how to whistle, don't you Steve? You just put your lips together and blow.

As you can see, your changes have been merged with your colleagues. The merge is done on a line-by-line basis, and if your changes do not overlap, then change marks will not be used (as is the case for the last line of the example).

Merging only applies to text fields. When there are conflicts in field data in forms such as checkboxes, radio buttons and selects, the most recent change (usually your change) always wins, even if someone else has changed the form since you started editing. Of course, all changes are available from the topic history.

Because there are cases where you actually want to avoid overlapping edits altogether (e.g. if you are changing forms data) TWiki will warn if you attempt to edit a topic that someone else is editing. It will also warn if a merge was required during a save.

### User Interface Internationalisation

TWiki will now pick up the language you are using in your browser, and try to present system messages in that language, if it is available. If your preferred language is not available, TWiki will revert to English. You'll also have an option to choose a language different from that used in your browser.

User Interface Internationalisation support is optional and enabled by the {UserInterfaceInternationalisation} setting.

The translation is performed by the Perl standard internationalization framework. If you want to contribute a new language, it would be most welcome: see TWiki:Codev.UserInterfaceLocalisation for instructions on how to help.

### New options on the editing screen

You will notice a couple of changes to the editing screen: first, there is no switch for releasing the edit lock any more (if locks are enabled, they are always released when an edit finishes). You will also notice a new "force new revision" checkbox. TWiki normally doesn't add a new revision if the same user re-edits a topic within a certain time period; this checkbox allows you to force TWiki to add a revision for every change, regardless of how small it is.

### Change notification

You now have much more control over which topics in a web you are interested in changes to. You can choose to receive notifications about topics selected by name, or by their parent relationship. See MailerContrib for more details. Note that the `mailnotify` cron script has moved out of the `bin` directory and into the `tools` directory - active crontab entries needs to be updated accordingly.

### E-mail addresses

Because of the security risks inherent in publishing e-mail addresses in personal topics, the storage of user's emails has been moved to the password manager. For sites that use the `.htpasswd` password manager, e-mail addresses that new users provide during registration are stored in the `.htpasswd` file. To aid in migration, if TWiki can't find a registered e-mail address in `.htpasswd`, it will still look in the personal topic. All users should register a valid e-mail address at ChangeEmailAddress.

If a different password manager is in use (e.g. LDAP, or 'none'), user e-mails will still be stored in personal topics. Sites that use other password systems (such as LDAP) should consider implementing a TWiki password manager, so that TWiki can look up email addresses, rather than storing them in personal topics.

### Parameterised Includes

`%INCLUDE{}%` variables have a predefined set of parameters. In the past, any parameters not in this set were simply ignored. With Dakar, these parameters are now defined as TWikiVariables within the included topic - for example,

    %INCLUDE{ "BugList" FAVOURITE="Damsel Flies" }%

will define `%FAVOURITE%` as `Damsel Flies` in the included topic, so if `BugList` contained the line

    My favourite bugs are %FAVOURITE%

it will be expanded to

    My favourite bugs are Damsel Flies

### Named Section Include

The `%INCLUDE{}%` variable allows to include only a named section of the included topic. These sections are defined in the included topic using the `%STARTSECTION%` and `%ENDSECTION%` variables. For example, if the included topic has:

    ---+ News

    ---++ IT News
    All news related to IT.
    %STARTSECTION{"itnews"}%
       * 2005-10-02 Final deployment of Dakar
       * 2005-10-01 Moving platform to Dakar
    %ENDSECTION{"itnews"}%

Using `%INCLUDE{ "AllNews" section="itnews" }%` will produce:

       * 2005-10-02 Final deployment of Dakar
       * 2005-10-01 Moving platform to Dakar

This syntax also allows for nested sections. For example, given the following topic:

    %STARTSECTION{"outer"}%
       * Top Outer Text
    %STARTSECTION{"inner"}%
       * Inner Text
    %ENDSECTION{"inner"}%
       * Top Outer Text
    %ENDSECTION{"outer"}%

Using `%INCLUDE{"SampleTopic" section="outer"}%` will produce:

       * Top Outer Text
       * Inner Text
       * Top Outer Text

And `%INCLUDE{"SampleTopic" section="inner"}%` will produce:

       * Inner Text

Overlapped sections are also allowed.

### RSS Feeds

RSS feeds have been enhanced to display links as links to make the feeds more useful. They also now use the web-specific logos.

## Notes for TWikiAdmins and WikiMasters

### Upgrading

See TWikiUpgradeGuide for help in upgrading an existing Cairo (Sep 2004) installation.

### Security

Dakar Release introduces the use of 'safe pipes' to prevent any malicious request from executing code on the server. This strategy stops any of the known attacks dead in its tracks. The Dakar codebase has not been impacted by any of the recent security advisories in any way. Several public sites have been running Dakar code for some months now, and to the best of our knowledge none has been hacked.

### CPAN Requirements

CPAN:Text::Diff is a prerequisite for the UpgradeTWiki script only. CPAN:URI is a prerequisite for `configure`. Other new prerequisites are CPAN:CGI::Session and CPAN:CGI::Cookie, if you want to take advantage of the new session support.

If you want user interface internationalization support, CPAN:Locale::Maketext::Lexicon and CPAN:Encode (in perl 5.8's core) are required, as well as perl 5.8 or higher. See TWiki:Codev.UserInterfaceLocalisation for details on TWiki internationalization support.

### Installation and configuration

The installation and configuration processes have been simplified.

#### setlib.cfg

The installer should now provide a file called `LocalLib.cfg` that contains local path settings. `setlib.cfg` contains documentation of what has to be done. Old `setlib.cfg` files will not work with Dakar.

#### TWiki.cfg

`TWiki.cfg` now contains all the default configuration settings, and the installer should provide a file called `LocalSite.cfg` that contains just those settings that are different than the defaults. The syntax of the settings in the file has also changed. Old `TWiki.cfg` files will not work with Dakar. The UpgradeTWiki script can be used to automate most of the necessary changes.

#### `testenv` / `configure`

`testenv` has been removed, and replaced with the new `configure` interface. This interface performs all the checking functions of the old `testenv`, adds several new ones (including permissions checks) and also acts as a browser interface allowing you to do all TWiki configuration from the browser. `configure` is now the main installation interface for TWiki.

The `configure` script can be used like the old `testenv` for public review of the configuration of the site. Saving from the interface is password-protected, using a password set in the configuration files, so to ordinary users `configure` just looks like a posh version of `testenv`. If you want to hide your configuration from public view, you can restrict access to the script using webserver access controls (Apache users see the Apache documentation on the 'require' directive for more infomation on how to do this).

#### `configure` optional features

New optional features include {AutoAttachPubFiles} and {EnableHierarchicalWebs}. Both are switched off by default but can be enabled in `configure`.

#### Improved support for shorter URLs

See TWiki:TWiki.ShorterUrlCookbook

### Preferences

There have been some significant changes to the handling of preferences, aimed at making them more logical, consistent and easy to use.

#### Changes to the evaluation order

The rules for preference evaluation (whether the user setting overrides the topic setting etc) have always been a bit confusing and difficult to use. For example, a topic setting would override a user setting, but a user setting would override a web setting, making per-web settings awkward to use. Mainly due to the introduction of hierarchical webs, preferences are now evaluated in the following order:

1. Topic
2. Web
3. Parent Web(s) (if appropriate. All parent webs are evaluated in bottom-up order)
4. Session
5. User
6. Local Site (as set in {LocalSitePreferences}, typically =%MAINWEB%.TWikiPreferences))
7. Default (as set in {SitePrefsTopic}, typically =%TWIKIWEB%.TWikiPreferences))

This is a change from previous versions, where the User preferences were evaluated between the topic and the web.

Note that a user can still dictate preference values that can't be overridden by the topic or the web level settings, but they will now need to set those preferences as FINALPREFERENCES. You can use `%ALLVARIABLES%` in a topic to get a dump of all preferences set in that context.

Permissions controls are not affected by this change.

#### Preferences Plugin

TWiki:Plugins.PreferencesPlugin has been included to allow more convenient editing of preferences. This plugin provides input controls, such as menus, radio buttons, and checkboxes to select preference settings.

The following standard preferences have been removed: MAILTHISTOPIC, MAILTHISTOPICTEXT, TOPICURL, READTOPICPREFS, TOPICOVERRIDESUSER (click on the name to search for occurrences on this site). If they are in use on your site, you can restore them to their Cairo settings by simply cutting and pasting the old definitions.

#### {LocalSitePreferences} (was Main.TWikiPreferences)

Customized site preferences can now be saved in the {LocalSitePreferences} topic which will override settings in {SitePrefsTopicName}. This simplifies upgrades as you can overwrite the {SitePrefsTopicName} topic and gain any new or updated preference settings without losing your local customizations.

#### `FAVICON`

`favicon.ico` is a small graphic that can appear in a variety of places in the browser: the titlebar, the taskbar, the address bar, bookmarks/favourites, and page tabs. Each web browser has a unique user interface, and as a result uses the Favicon in different ways. Most browsers display it in most of the locations listed.

Out of the box, TWiki is configured to easily customise the `favicon.ico` for each web. To switch to a new `favicon.ico`, upload it to the desired web's WebPreferences.

- - Set FAVICON = http://twiki.org/p/pub/TWiki/WebPreferences/favicon.ico

To provide a single, site-wide `favicon.ico`, hardcode a specific web, for example:

- - Set FAVICON = http://twiki.org/p/pub/TWiki/WebPreferences/favicon.ico

#### `FORCENEWREVISIONCHECKBOX`

Normally, if you make another edit within a one hour period (was `$editLockTime` in `lib/TWiki.cfg`, now `{ReplaceIfEditedAgainWithin}`), TWiki will fold together your changes. This is often the "right thing to do", as it can reduce the visual clutter of diffs.

The "Force New Revision" checkbox is a way to force it to create a separate revision each time you save.

The TWiki.TWikiPreferences variable `FORCENEWREVISIONCHECKBOX` controls whether this is checked by default or not.

On a related note, you can force every save to be a new revision number by editing `lib/TWiki.cfg` and setting `{ReplaceIfEditedAgainWithin}` to 0.

NOTE: Although this feature is being introduced in this release, it is also being deprecated at the same time. TWiki:Codev.EdinburghRelease is planned to provide the ability to elide revisions at the GUI level, rather than the Store level, thus obviating the need for this stopgap measure.

#### `WEBLOGONAME`, `WEBLOGOIMG`, `WEBLOGOURL`, `WEBLOGOALT`

Each web can have its own customised logo. The simplest way is to upload a `logo.gif` to a web's WebPreferences, and it will appear in the top-left corner.

To change the logo's filename, set the `WEBLOGONAME` variable. You'll especially need to do this if you use a different logo file format:

- - Set WEBLOGONAME = MyLogo.jpg

If you don't want to have custom logos on a per-web basic, but instead want to use a single, site-wide logo, hardcode a specific web in the `WEBLOGOURL` variable. For example:

- - Set WEBLOGOURL = %PUBURLPATH%/Main/WebPreferences/logo.png

#### `WIKILOGOIMG`, `WIKILOGOURL`, `WIKILOGOALT`

These variables are now more closely associated with `WIKITOOLNAME`. If you change `WIKITOOLNAME`, you'll probably want to change these variables, too. `WIKILOGOIMG`, `WIKILOGOURL`, `WIKILOGOALT`, and `WIKITOOLNAME` are now used more consistently together.

### Final Preferences

The `FINALPREFERENCES` setting prevents particular preference settings from being over-ridden at a lower level. The hierarchy of how `FINALPREFERENCES` settings are applied has been clarified/formalized as reflected in the following chart:

Level | Set By | Local site examples
--- | --- | ---
default site | %TWIKIWEB%.TWikiPreferences or %WIKIPREFSTOPIC% | TWikiPreferences
local site | %MAINWEB%.TWikiPreferences or %LOCALSITEPREFS% | TWikiPreferences
web | WebPreferences | WebPreferences
user | In one's user topic | TWikiAdminGroup
topic | "Edit topic preferences settings" under "More topic actions" | TWikiReleaseNotes04x00

By default, the site level `FINALPREFERENCES` are set in Main.TWikiPreferences so as not to conflict with preference settings in that topic.

### `mod_perl` support improvements

TWiki no longer uses global variables other than for constants. Each CGI script creates a new "session object" that holds all session-specific information.

There is still an issue with the `@INC` path in `mod_perl`, that mainly impacts plugins that lazy-load modules. You should use the `PerlSetEnv` directive that `mod_perl` provides to make sure that your TWiki `lib` directory is permanently on the path, if you are using `mod_perl`.

### Plugins

Plugins are no longer searched for every time a TWiki script is run. Instead they are automatically discovered in `configure`. To enable and disable plugins, use the `configure` interface. The entire @INC path is searched for plugins, so you can easily point at plugins outside the installation. However only the first instance of a plugin on the @INC path will be found (it is a path, after all).

`%INSTALLEDPLUGINS%` and `%DISABLEDPLUGINS%` are no longer supported in TWikiPreferences. If you have set `%INSTALLEDPLUGINS%` in TWikiPreferences, you need to move that setting into the `{PluginsOrder}` configuration key, using the `configure` interface. To disable plugins, uncheck them in the `configure` interface, and save the changes.

Whenever you install a plugin, make sure you check TWikiPlugins#FAILEDPLUGINS. Several handlers have been deprecated, and updates of the plugins may be required. Contact the plugin author directly to get an update if none is available on the web.

### Logins, Logouts, Sessions and Passwords

TWiki:Plugins.SessionPlugin and TWiki:Plugins.AuthPagePlugin have been integrated into the core. TWiki now supports cookied sessions, in the context of a much improved authentication architecture. The setup for authentication is now much simpler, and for most sites can be done entirely from the `configure` interface. There are some incompatibilities with TWiki:Plugins.SessionPlugin, with resepect to the in-line variables. See TWikiUserAuthentication in the release for full details of how authentication works now. TWiki also now supports the concept of pluggable password managers, making the integration of corporate authentication services much simpler.

Administrators, especially of public sites, need to be aware of the security implications of cookied sessions, and the potential risks of cross-site scripting attacks that may be used to steal user sessions. See TWikiUserAuthentication for more details.

### Protections

The evaluation of protections has been re-worked to make it more naturally understandable, and also fill a number of holes in the protection scheme, These holes meant that it was relatively easy to deny access to a topic, but rather difficult to subsequently restore access without either compromising other topics, or compromising old revisions.

When deciding whether to grant access, TWiki now evaluates the following rules in order (read from the top of the list; if the logic arrives at PERMITTED or DENIED that applies immediately and no more rules are applied). You need to read the rules bearing in mind that VIEW, CHANGE and RENAME access may be granted/denied separately.

1. If the user is a super-user
   - access is PERMITTED.
2. If DENYTOPIC is set to a list of wikinames
   - people in the list will be DENIED.
3. If DENYTOPIC is set to empty ( i.e. `Set DENYTOPIC =` )
   - access is PERMITTED i.e. no-one is denied access to this topic
4. If ALLOWTOPIC is set
   1. people in the list are PERMITTED
   2. everyone else is DENIED
5. If DENYWEB is set to a list of wikiname
   - people in the list are DENIED access
6. If ALLOWWEB is set to a list of wikinames
   - people in the list will be PERMITTED
   - everyone else will be DENIED
7. If you got this far, access is PERMITTED

In addition, permissions set on a specific revision of a topic now always apply to that revision, even if access to the topic is relaxed in a later revision. This change was necessary to prevent accidentally permitting access to sensitive data in old revisions.

The major impact of this change is that WebPreferences topics shipped with earlier releases of TWiki will have excessively restrictive controls, as the default settings were:

- Set DENYWEBVIEW =
- Set ALLOWWEBVIEW =
  - This will now deny view access to everyone not in the list (i.e. everyone except admins)
- Set DENYWEBCHANGE =
- Set ALLOWWEBCHANGE =
  - This will now deny change access to everyone not in the list (i.e. everyone except admins)
- Set DENYWEBRENAME =
- Set ALLOWWEBRENAME =
  - This will now deny rename access to everyone not in the list (i.e. everyone except admins)
- Set ALLOWTOPICCHANGE =
  - This will now deny change access to everyone not in the list (i.e. everyone except admins)
- Set ALLOWTOPICRENAME = Main.TWikiAdminGroup

The standard webs shipped with this release have these settings disabled. However you are likely to have inherited the old default settings in your user webs. The easiest way to deal with this is to simply insert a # sign in these settings; for example:

- #Set DENYWEBVIEW =
- #Set ALLOWWEBVIEW =
- etc

Note: For security reasons, the Trash web is shipped with `ALLOWWEBVIEW` set to TWikiAdminGroup.

#### Site Permissions Overview

The SitePermissions topic gives you a quick view of the permissions on each web - i.e. it aggregates ALLOWWEB* etc. into a handy table. It is also installed on TWiki.org as TWiki:TWiki.SitePermissions.

### Frustrating Robots and Spammers

Standard TWiki incorporates some simple measures to protect e-mail addresses and control the activities of benign robots. These should be enough to handle intranet requirements. Administrators of public (internet) sites are strongly recommended to investigate the TWiki:Plugins/BlackListPlugin.

### New User Registration

The new user registration process has been extensively reworked to improve usability and security of the registration process.

1. FirstName and LastName are recorded separately and tabulated. See UserListByLocation
2. Sends a e-mail that the user has to respond to; registration data is staged until this is done
3. Populates a form in the user topic, if there is one in the NewUserTemplate. If there is no form, then bullets are used, as in pre-Dakar TWiki.
4. Sends different e-mails to the WIKIWEBMASTER and the USER, with the users containing the password unless {HidePasswdInRegistration} is set.
5. There is now a mechanism for bulk registration, with callbacks for augmentation (e.g. from other sources).
6. Absorbed TWiki:Codev.ResetPasswordByEmail.
7. Obsoleted TWiki:Codev.InstallPassword

### E-mail addresses

E-mail addresses for new users are no longer stored in home topics. Instead, the password manager API has been extended to support storing e-mails.

The default password manager stores e-mails in the `.htpasswd` file - you can safely edit this file with a text editor to modify the info field that contains the e-mail addresses (the format of each line in this file is `username>:<password>:<info>`, and TWiki expects the info field to be a ;-separated list of e-mail addresses). Password managers for other systems e.g. LDAP can esily be extended to support the new API. If the password manager does not have an e-mail address for a user, then TWiki will still look in the users' personal topic.

The script `tools/upgrade_emails.pl` can be used to extract e-mail addresses for existing TWiki users from personal topics, and add them to the password manager.

- You are strongly recommended to run this script if you are running a public site.
- It will not modify existing personal topics.
- The script will tell you if any users are found that don't have an e-mail.
- You can re-run the script as many times as you like; once the password manager has an e-mail for a user, the e-mail in their personal topic is ignored.

You should also advise all your registered users to make sure they have a valid e-mail. They can do this by logging in and visiting ChangeEmailAddress.

### Change notification support

The old `mailnotify` script has been retired in favour of the MailerContrib. See MailerContrib for information about functional changes.

### Site Changes Summary

The SiteChanges topic mimics TWiki.org's TWiki:Codev:WebChangesForAllWebs - i.e. it shows a WebChanges view across a whole site. It's name was chosen to parallel SiteMap; at some point you can expect the arrival of SiteStatistics too.

### What's a Web

Old versions of TWiki would consider all subdirectories of the TWiki data directory to be webs. This behaviour has changed; now only subdirectories that contain a web preferences topic (`WebPreferences.txt`) will be considered to be webs. This may result in directories that used to return search matches no longer doing so.

### Disable <script> tags

In recognition of security concerns around <script> tags, the administrator has the choice whether to allow users to add script to topics or not. See the `{AllowInlineScript}` setting in the Security section of `configure`.

## Notes for TWikiApplication Developers

### Space/Tab conversion

Previous TWiki versions would automatically convert three spaces at the start of lines to a tab when the topic was saved. This meant that the saved topic was not the same as the edited topic, which could result in considerable confusion. This conversion has been disabled, and the saved topic is now exactly what you see in the editor. One impact of this change is that any add-on scripts you may have developed that rely on bulleted list lines starting with a tab will no longer work. They must be adapted to treat groups of three spaces and single tabs as equivalent.

### Evaluation order of TWikiVariables

In previous TWiki versions the evaluation order of `%VARIABLE%s` depended on where they were expanded in the code. The parser was somewhat crude, and could easily be confused when embedded variables (variables embedded in the parameters of other variables) were used.

The parser has been replaced in Dakar with a deterministic variable parser with predictable behaviour. Specifically, variables are now always evaluated left to right and inside out. For example, consider `%VAR2{ "%VAR1{ "%VAR0{ "params" }%" }%" }% %VAR3%`. Previously, the expansion order would have depended on the order of expressions in the code, so the expansion may have proceeded VAR3 - VAR0 - VAR2 - VAR1. If you were lucky, this was the intended order. In Dakar, the order is now guranteed to be VAR0 - VAR1 - VAR2 - VAR3 (i.e. inside out and left to right).

The main impact of this is that some TWikiApplications may cease to work if they have been written to take advantage of the old chaotic order. There is no way to predict which will work and which will fail, so you will have to deal with this on a case-by-case basis. In most cases TWikiApplication authors will have worked hard to do the "sensible thing" so instances of this problem should be rare.

Note that because the TWiki spec allows double quotes within double-quoted strings in certain variable parameters it has been impossible to make the parser 100% deterministic. There may still be pathological cases where the parser may fail. In these cases, consider how open and close curly brackets are matched up.

### Encoding of form-field values

The encoding used to escape characters in form-field values has had to change. The old encoding could very easily be confused by text strings in data values. Existing topics with the old encoding will still be loaded into TWiki as before, but if edited they will be saved with the new encoding. This should not affect you unless you have searches that look for the old encodings; specifically, the %_G_%, %_Q_%, and %_P_% character sequences. Any such searches will not work any more, and need to be converted to search for the new encoding. Assuming they are `regex` type searches, you can use `(%_G_%|%0A)` to match encoded newlines in field data in both old and new format topics, `(%_Q_%|%22)` to match quotes, and `(%_P_%|25)` for percent signs.

### <script> tags in topics

Previous releases of TWiki would attempt to interpret the content of <script> tags in topics as TWiki formatting language, often resulting in non-functional scripts. This release protects script sections from expansion by TWiki. Note that this means that TWikiVariables will not be expanded in script tags - they are passed through verbatim.

In recognition of security concerns around <script> tags, the administrator has the choice whether to allow users to add script to topics or not. Check the setting of `{AllowInlineScript}` in `configure` to see if it is allowed on your site. If not, script sections will simply disappear from topics.

### <verbatim> tags in topics

Previous releases required `verbatim` tags to be on their own line. TWiki can now deal with inline verbatim blocks such as

    blah<verbatim>inside</verbatim>after

results in

blah

    inside

after

NOTE: VARIABLES are still Set within verbatim tags (this is a historical peculiarity)

### `ALLVARIABLES`

You can use `%ALLVARIABLES%` in a topic to get a dump of all variables set in that context. Invaluable for debugging those tricky TWikiApplications!

### `IF`

The new `%IF()%` variable defines simple conditional statements that are evaluated at view time. This allows you to include content conditionally based on environmental factors. See IfStatements for more information on usage.

### New `$count(reg-exp)` variable in Formatted Search

This new variable for FormattedSearch returns the number of instances a specified RegularExpression occur in a topic. This is useful for such things as counting the number of comments on a page (assuming they are marked my a unique heading level).

## Notes for Skin Developers

### Skin search path

Earlier releases of TWiki only allowed for a single skin, so customising a single template in a skin meant deriving a whole new skin. Dakar introduces the concept of a skin search path, which lets you combine skins additively. For example, you can customise just the view template by defining a new `view.mylocalskin.tmpl` and then setting

- `Set SKIN = mylocalskin,pattern`

Since you have defined a view template for `mylocalskin`, it will be picked up when you view a topic because `mylocalskin` is first on the search path. But you didn't define `edit.mylocalskin.tmpl`, so when you edit the next skin on the search path will be used instead (in this case `edit.pattern.tmpl`). You can put as many skins on the search path as you like.

As with older releases, setting SKIN (or the `skin` parameter in the URL) replaces the existing skin path setting. Dakar supports extension of the path as well, using covers.

- `Set COVER = mylocalskin`

pushes a different skin to the front of the skin search path. There is also an equivalent `cover` URL parameter. For example, http://twiki.org/cgi-bin/view/TWiki/WhatIsWikiWiki?cover=print.pattern. This gives you an extra level of flexibility when defining skins.

See TWikiSkins for more information.

### Supporting web names that are WikiWords

Although web names have been permitted to be WikiWords since the TWiki:Codev.CairoRelease, the base templates have been fixed for Dakar.

Skins should be upgraded if they have standalone `%WEB%` variables; only standalone `%WEB%` text that potentially could be turned into a link (because of a WikiWord) needs to be escaped. Same for `%MAINWEB%` and `%TWIKIWEB%`.

Examples:

- `%WEB%` -- needs to be escaped with `<nop>%WEB%`
- `(%WEB%)` -- needs to be escaped because of parenthesis
- `"%WEB%"` -- no need to escape, does not get linked
- `<b>%WEB%</b>` -- no need to escape, does not get linked
- `%WEB%.%TOPIC%` -- no need, is a `Web.TopicName`
- `(%WEB%.%TOPIC%)` -- no need, is a `Web.TopicName`

Basically, any prefix other then space and parenthesis needs to be looked at. `%WEB%` in a `%SEARCH%` should not be escaped.

### Deprecation of %EDITTOPIC%

In Cairo release (and earlier) the only way to have an edit link that changed with the context was to use `%EDITTOPIC%`. This was only available in view templates, and had no flexibility in formatting. It was also impossible to disable other active links, such as Attach.

Dakar release includes new support for "context if" parameters to the `%TMPL:P%` construct. See TWikiTemplates for details. The default templates shipped with Dakar have been modified to use this support. %EDITTOPIC has been deprecated, though it is still available as a simple edit link, defined in TWikiPreferences. Skin authors are strongly recommended to replace this link with context-if conditionals.

### Template Parameters

`%TMPL:P%` now accepts parameters. Values passed in these parameters will be expanded when the `%TMPL:DEF%` is instantiated. See TWikiTemplates for full details. (Remember, this happens at template expansion time, which is usually very early in the rendering process.)

### HTTP and HTTPS

These two new TWikiVariables give access to the HTTP headers sent by your browser. For example, your browser says your browser is %HTTP:{"User-Agent"}% (this will be filled in if you are running Dakar).

### QUERYSTRING

This new TWikiVariable gives access to the full query string from the URL sent to your browser. For example, for the URL `http://twiki.org/cgi-bin/view/TWiki/TWikiReleaseNotes04x00?make=Reliant&model=Robin`, the query string is `?make=Reliant;model=Robin` (yes, the semicolon is correct!)

## Notes for Plugin Developers

### Changes to variable parsing

In earlier TWiki versions there was considerable confusion over the syntax and semantics of Preferences, TWiki Variables, and Plugin variables. The documentation suggested that all %VARIABLES% were equal, but in fact some were more equal than others.

The syntax and semantics of preferences and TWikiVariables has been made consistent. `%VARIABLE%` has been made semantically identical to `%VARIABLE()%`, so if you set a preference named `%VARIABLE%` it will automatically be instantiated in place of `%VARIABLE{}%`. This is an elegant solution in several ways: first, it allows an administrator to electively disable TWikiVariables, simply by defining an overriding preference. Second, it rationalises the semantics in line with the common syntax. Third, it allows a single parser to do all the work, allowing localised optimisation. Fourth, it prevents a plugin from accidentally kidnapping system TWikiVariables (while this can still be done by registering a tag handler, it's a much more explicit process). Fifth, the ground rules are set for a possible future extension to support parameterised TWikiVariables e.g.

- `Set CAR{make model accessory} = I drive %make% %model% with %accessory% in my dreams`

`%CAR{make="an Aston Martin" model="DB9" accessory="a gorgeous blonde"}%`

### Internals

A lot of the TWiki internals have changed. As a result, plugins that bypass the `TWiki::Func` API and call core functions directly are unlikely to work.

The restructuring of the code internals is such that there are no 1:1 equivalents for the old core functions. Only the TWiki::Func API is guaranteed to work.

You should convert your plugins to call the `TWiki::Func` API. If you have called unpublished functions that have no equivalent in `TWiki::Func`, then you may still be able to call the function via the TWiki "session" object, `$TWiki::Plugins::SESSION`. See the implementation of the `TWiki::Func` module for ideas on how to do this. However calling internals is not recommended, even using this new mechanism, as they are liable to change without prior notice.

### Extensions to the Func API

- The `TWiki::Func` API has been extended to expose a number of new core functions. Review TWikiFuncDotPm? for details.
- The `TWiki::Meta` API, which was previously for internal core use only, has now been exposed and may be used in plugins. See TWikiMetaDotPm? for full details.
- Plugins which observed the existing published API (TWiki::Func) should continue to work.

## TWiki 4.0.1 Patch Release Details

The following fixes have been addressed in this release:

### TWiki 4.0.1 Fixes

Item1597 | configure shows text outside the text boxes - thanks KoenMartens
Item1592 | Whoops, deprecated syntax was both alive and documented, trying something else
Item1592 | Killing some never documented syntax, allowing for more flexibility in explicit external [[]]-style links
Item1591 | Friendly fallback added to meta->getParent()
Item1591 | small fix to parent search but bug remains
Item1590 | Don't loose old TOPICPARENT on save
Item1589 | Don't report cfg-files as non-executable
Item1587 | Including upgrade_emails.pl in distro
Item1583 | Cairo compatibility for rev argument syntax
Item1579 | corrected spanish access key.
Item1574 | Net.pm: Tidying up e-mail split
Item1574 | Net.pm: Split only on , not on spaces when sending e-mail
Item1572 | made optional - thus removing the horrible slowdown it causes from the main loop
Item663 | Fix %MAINWEB%.TWikiPreferences to %LOCALSITEPREFS%; remove %TWIKIWEB% prefix where not needed
Item663 | TWiki.org doc merge: GNU patch requirement
Item240 | layout of more screen; added line "Current parent" with 2 searches inside IF - must be a more efficient way but how?
Item240 | just to be sure: replacing MAKETEXT text with English, to be translated later.
Item240 | removes unwanted left border in non-view pages
Item000 | replaced 9 tabs by 9 x 3 spaces
Item000 | Typo in ClassicSkin.pm
Item000 | Committed a whole lot of local setup by accident, reverting - sorry folks
Item000 | Typo fix in Search.pm
Item0 | Untainting the tainted title

The 4.0.1 release was built from SVN http://svn.twiki.org:8181/svn/twiki/branches/TWikiRelease04x00 revision 8740.

## TWiki 4.0.2 Patch Release Details

The following fixes and minor enhancements have been addressed in this release:

### TWiki 4.0.2 Fixes

Item2004 | Syntax error in DelimitedFile.pm
Item1983 | Search box bust when searching for WikiWords
Item1978 | Form.pm fails when the name field is [[Topic][fieldname]] for controls
Item1971 | Not possible to INCLUDE javascript from external sites
Item1963 | "public" in WEBLIST should include all webs if the user is an admin
Item1961 | LANGUAGES variable very slow
Item1958 | RSS feed mentions the TWikiAdministrator as creator
Item1951 | Excluding topic from SEARCH does not exclude WebHome (Doc improvement)
Item1950 | Denial of Service attack very easy on both Cairo and Dakar
Item1944 | Add Comment to Configure Page - AuthScripts section
Item1937 | rename script doesn't honour access control restrictions
Item1935 | preview script ignores access control settings
Item1925 | rdiff and changes scripts ignore access settings
Item1921 | Merging 3part duplicates the text screwing up the page
Item1920 | Hide form twisty gives poor usability for TWiki apps that use the form
Item1912 | 3-way merge doesn't call the plugins merge handler
Item1911 | Small typo correction in MANIFEST for upgrade_emails.pl
Item1909 | TWiki.pot should be included in distribution
Item1903 | Special chars not escaped in registration
Item1902 | New chars to escape during registration
Item1900 | Compatibility issue: variables only work when upper case.
Item1899 | statistics ignore pages with dashes and underscores
Item1898 | configure is unhelpful if rcs is not installed
Item1897 | Simultaneous edit feature is not at all reliable.
Item1896 | Skin fallback not working
Item1895 | $topic not substituted right in INCLUDEWARNING
Item1882 | "Plural to singular" unmarked breaks documentation
Item1878 | FORMLIST in ChangeForm.pm generates invalid html
Item1871 | Attachment table broken with version history
Item1864 | "Modify search" link in search results not working
Item1863 | missing oopslanguagechanged template
Item1853 | Sandbox doesn't report full errors for command execution on Windows
Item1850 | checkPassword not checkPasswd
Item1849 | Typo in code: checkPassword not checkPasswd, so remove user does not work
Item1848 | .htaccess.txt internal documentation unclear
Item1840 | Inadequate upgrade documentation
Item1838 | commonTagHandler garbles square bracket links in WebTopBar
Item1830 | Rename web does not work
Item1829 | the templates for SEARCH are broken
Item1820 | Reproducible case of not being able to log out
Item1819 | tainted username prevents exec()
Item1803 | Sorting bugs by merge field causes an internal error
Item1789 | User::isAdmin can create an empty SuperAdminGroup
Item1788 | Formatted search does not return values that are zero
Item1787 | I18N: Translation updates for 4.0.2 / new PatternSkin
Item1781 | Allow admin users to change passwords and mail addresses
Item1778 | Registration gives strange TWiki vars in user topic
Item1771 | Error message 'Oh dear' in warnyyyymm.txt isn't helpful
Item1743 | Attachment sort order is cockeyed
Item1729 | Better localisation of variable in Net.pm
Item1724 | Attach: FILENAME AND FILEPATH not showing the file name with non-alpha characters
Item1720 | Save script fails on old topic when sole argument is topicparent and there's a mandatory field in the form.
Item1714 | quotes in attachment comment break manage form
Item1687 | Must set TWiki::Plugins::SESSION before invoking registered tag handler
Item1677 | STARTSECTION/ENDSECTION variables being stripped from templates
Item1672 | Change PatternSkin to use non-table based layout
Item1657 | RenderListPlugin broken. The case where you draw a tree view does not work
Item1654 | mailnotify does not enter the command_line context
Item1652 | REVINFO{$time} should display time, not date - time
Item1649 | TWikiJavascripts prototype.js causes crash on Internet Explorer.
Item1645 | Someweb.WebTopicEditTemplate topic does not exist
Item1636 | ResetPasswd deletes email entry in .htpasswd
Item1634 | Pattern Skin in Dakar: verbatim text and large images makes everything wider than the screen.
Item1624 | FORMFIELD variable always expands to value of most recent topic revision
Item1623 | PatternSkin: WebLeftBar border margin
Item1621 | beforeSaveHandler not functioning correctly
Item1619 | Label form field content destroyed
Item1616 | inaccurate password_changed message
Item1611 | Generic doc work for TWiki 4.0.2
Item1610 | Configure script corrupts NameFilter (Unmatched Bracket in Regex)
Item1608 | TOC{"MyTopic" web="MyWeb" } no longer forwards
Item1599 | "Upgrading a Beta" documentation errors
Item1595 | WebTopicCreator disallows topic names containing numbers
Item1587 | upgrade_emails.pl script missing in TWiki 4 distro
Item1583 | INCLUDE of a topic of a specific revision is not backwards compatible.
Item1579 | Incorrect properties in the Discard button in the action bar for the spanish translation
Item1574 | Registration process tries to send extra emails
Item1566 | DakarReleaseNotes.html and TWikiDocumentation.html not included in distribution of TWiki4.0
Item1553 | Clean up "oops attention: merge_notice"
Item1515 | Localise formatting help
Item1430 | inconsistent width constraints on topbar and content area

### TWiki 4.0.2 Enhancements

Item1964 | Configurable user homepage layout
Item1960 | WebRss lacks search options
Item1956 | Cleaned up Plugins.EmptyPlugin
Item1926 | Usability: Add tabindex + setfocus to template login form in TWiki.PatternSkin
Item1801 | Improved doc
Item1728 | increase security by defaulting "send password" in email off
Item1689 | Add more colors for text ink in Plugins.WysiwygPlugin
Item1148 | Consistent buttons in edit and editsettings
Item1147 | "Raw view" should turn into "Normal View" in raw mode
Item1146 | Two "Create" buttons

The 4.0.2 release was built from SVN http://svn.twiki.org:8181/svn/twiki/branches/TWikiRelease04x00 revision 9626.

## TWiki 4.0.3 Patch Release Details

The following fixes and minor enhancements have been addressed in this release:

### TWiki 4.0.3 Fixes

Item668 | _default web gives include error for WebLeftBar but the topic does exist
Item444 | oops upload message with empty file is not intelligent
Item2530 | Prevent infinite recursion in TMPL:INCLUDE when including generic templates
Item2528 | Support for shortest view URLs
Item2524 | Make templates independent on TablePlugin
Item2509 | Render::getRenderedVersion converts translation token
Item2487 | Re-introducing attachment table list count
Item2481 | Need a configure data type to select a pluggable class (like TWiki::Store::*)
Item2476 | Topic action buttons shown with search result
Item2472 | Get user web name from config and not from registration form
Item2469 | TWikiAdminGroup definition topic must be called TWikiAdminGroup (not configurable)
Item2467 | Non-existing abbreviations in other than current web have web part stripped (AAA.BBB.FFF renders as FFF)
Item2464 | Definition of a group always requires "Group" in the group name
Item2463 | rdiff should show HTML comments
Item2441 | Func::getWikiToolName documentation is incorrect
Item2439 | Rename TWikiRelease04x00x00.html to TWikiRelease04x00.html
Item2436 | PatternSkin does not support PAGEBGCOLOR
Item2427 | viewfile delivers .tgz files as text (wrong mime type)
Item2426 | ICONTOPIC variable: Incorrect value deadlocks topic (no edit possible)
Item2422 | Inconsistent LocalSite.cfg.txt settings
Item2421 | Rename: Referrer topics need to be linked explicit
Item2409 | Form field checkbox option lost on topic edit
Item2402 | Sandbox not aware of I18N for user names
Item2399 | statistics does not like date variable sent to it because its tainted
Item2395 | Always show e-mail addresses to admins (USERINFO / HideUserDetails setting)
Item2394 | Allow registration without a running e-mail service (error during registration using TWiki::Net::_sendEmailBySendmail)
Item2390 | beforeAttachmentSaveHandler is broken on Solaris and RedHat
Item2380 | Logins / logouts with template login does not work with I18N topic names
Item2379 | Inconsistent meta data in registration
Item2369 | WIKIUSERNAME incorrectly expanded
Item2365 | Email address handling for registration and in users homepage is confusing and not working
Item2356 | in sequential rdiff a changed region is classified as twikiDiffAdd*
Item2353 | Cannot lock out re-registration by using htpasswd file anymore
Item2352 | Support secret values in configure (display stars for passwords)
Item2347 | Statistics without params updates only Main web
Item2339 | Unable to cleanly turn off WEBHEADERART completely causing unexpected major problems
Item2338 | Improve doc on session vars to preclude override of perms
Item2336 | Using htpasswd on the command line wipes out email address in .htpasswd file
Item2333 | TWiki::Func::readAttachment my line shows wrong parameters
Item2332 | GoodStyle talks about "Initials"; obsolete
Item2331 | Calling a speedy-fied view cgi from the commandline breaks template login
Item2327 | TWiki on Apache 2.0 hangs (Diab's TWikiOnApache2dot0Hangs patch)
Item2324 | PatternSkin menu layout issue with MS Internet Explorer 7
Item2322 | Comment box should have ability to be disabled by skin template
Item2321 | Performance improvements to Users.pm implementation (large user bases)
Item2318 | Links in square brackets breaks if there is a space in front of chars like - and (
Item2317 | ChangePassword confirms change without password-handler
Item2315 | Template file permissions more restrictive in tgz dist
Item2309 | Prevent change password to empty string (add MinPasswordLength configuration option)
Item2302 | Restore Using Forms for Settings Feature
Item2298 | TWiki::Data::DelimitedFile is not as robust as it could be
Item2297 | Lock down TWiki.TWikiPreferences to admin group
Item2293 | Registration fails if no mail available
Item2292 | SMTP mail fails if the server requires auth
Item2287 | When printing a view page, the print style is not called
Item2286 | When you move an attachment the rename screen shows all topics that refers to the topic name
Item2278 | userToWikiName broken
Item2274 | I18N: Non US-ASCII chars in usernames breaks groups
Item2271 | Cannot "put back" a moved non-wikiword topic
Item2261 | Error in Apache log due to TWikiWebPreferences (File does not exist / ATTACHEDFILELINKFORMAT preference)
Item2259 | TWiki::UI::Save::buildNewTopic does not treat onlywikiname as a Boolean
Item2247 | Unclosed DIV in viewprint.pattern.tmpl
Item2244 | Documented METASEARCH parameter defaults are not the actual defaults
Item2234 | natlogon broken due to recent changes in Client.pm
Item2227 | PatternSkin CSS updates for IE 7
Item2226 | When you delete an attachment the rename screen shows all topics that refers to the topic name
Item2225 | Attachments are being named the full path name instead of the filename only
Item2223 | Empty textarea generated in raw mode (VIEW_TEMPLATE issue)
Item2214 | WebTopicCreator not stripping some disallowed characters
Item2186 | TWiki.SiteChanges shows oldest (not latest) changes
Item2163 | Groups should be defined in UsersWebName only (performance improvement)
Item2158 | TWiki leaks memory - mod_perl processes continually grow
Item2157 | Rendering of links containing periods does not work properly
Item2151 | Delete usecase broken in PatternSkin
Item2142 | Add option to write email into user topic
Item2141 | Error.pm not in sync with CPAN state
Item2140 | AUTHORS missing translators' credits
Item2133 | Editform templates are missing some save parameters (templatetopic and text)
Item2126 | Bulk register shouldn't change passwords of existing users
Item2116 | I18N: updated translations for TWiki 4.0.3
Item2109 | Add css classes to PreferencesPlugin buttons
Item2105 | A form field defined as name? assumes that the topic is in the same web as the form
Item2102 | Clicking Upload without choosing a file gives error
Item2097 | Plugin API / Store.pm: beforeAttachmentSaveHandler broken
Item2096 | New topic missing from notification when renamed
Item2090 | Default values for fields not picked up in new form (TWIKI4 only)
Item2088 | PatternSkin MANIFEST is missing PatternSkin.pm
Item2057 | TMPL:P parameters broken
Item2054 | JSCalendarContrib only works with IE in PatternSkin
Item2050 | Easier overriding of PatternSkin
Item2048 | Mailto links written as mailto:address causes flooding with warning messages about uninitialized value
Item2032 | Some UTF8 characters in form values broken (CGI.pm interaction)
Item2029 | Build script: New topics added to distribution are not versioned correctly
Item2019 | Add tracing to Client.pm (login scenarios)
Item2012 | Break RCS locks on topics automatically when saving (Cairo upgrade issue)
Item2010 | Email address in the email field for the user is not turned into a link
Item2009 | Email addresses are not fully padded with the NOSPAM
Item2003 | lib/CPAN/lib/Error.pm has debugging print enabled
Item1989 | Latest CGI::Session is broken
Item1982 | Sequence of form fields in topics do not follow form definition
Item1980 | Login text remains untranslated
Item1941 | If REPEATs are munged in templates, RDiff bombs out
Item1939 | Inconsistent handling of non-wikiword topic names when creating new topic
Item1890 | Wysiwyg plugin in infinite loop
Item1869 | Clarify security docs in configure
Item1843 | A 1.2 version of a topic cannot be deleted (spam) with cmd=delRev
Item1651 | Extract UserMapping and GroupMapping code out of User.pm and Users.pm (enable non-TWikiTopic based User and Groups definititions)
Item1613 | Renaming the Main web breaks several links to users
Item1602 | link to TWikiUpgradeGuide needed from TWiki Installation Guide
Item1560 | Non-existing favicon.ico is referenced in upgraded Cairo webs

### TWiki 4.0.3 Enhancements

Item2452 | mime.types lacks some widely used file types
Item2301 | Put {linkProtocolPattern} into configure
Item2282 | SpreadSheetPlugin with SETIFEMPTY
Item2155 | Remove hardcoded fonts from twiki's default content
Item2125 | Add format parameter to META{"parent"}

## TWiki 4.0.4 Patch Release Details

The following fixes have been addressed in this release:

### TWiki 4.0.4 Fixes

Item2578 | SECURITY HOTFIX: Improved protection against attaching php scripts that can be executed afterwords by simple view
Item2568 | Fix potential script error when attachment twisty is removed
Item2558 | TWiki 4.0.3 distributed LocalSite.cfg.txt uses incorrect syntax
Item2546 | Handmade twisty buttons has underline under them

The 4.0.4 release was built from SVN http://svn.twiki.org:8181/svn/twiki/branches/TWikiRelease04x00 revision 10807...