Skip to content

Commit

Permalink
Item14398: Implement Friendly macro parser as default
Browse files Browse the repository at this point in the history 
This has a number of issues.  Mainly, that many macros accept an
un-quoted string as the default parameter.  Our documentation does not
mention this feature, but it's in very common use.

The friendly parser requires that any unnamed default parameter be
quoted.
  • Loading branch information
@gac410
gac410 committed May 14, 2017
1 parent f2c7a43 commit 21c5775
Show file tree
Hide file tree
Showing 15 changed files with 141 additions and 39 deletions.
9 changes: 8 additions & 1 deletion SpreadSheetPlugin/lib/Foswiki/Plugins/SpreadSheetPlugin.pm
View file
Expand Up @@ -34,6 +34,12 @@ sub initPlugin {
# Get plugin debug flag
$debug = Foswiki::Func::getPreferencesFlag("SPREADSHEETPLUGIN_DEBUG") || 0;

my $legacyParser =
Foswiki::Func::isTrue(
Foswiki::Func::getPreferencesFlag("SPREADSHEETPLUGIN_LEGACYPARSER") )
|| $Foswiki::cfg{UseLegacyMacroParser}
|| '0';

# Following code is for a registered tag handler that does the same as
# CALC but in a tag handler instead of in commonTagsHandler. That means
# you can't use table references, but you can rely on the execution order
Expand All @@ -47,7 +53,8 @@ sub initPlugin {
$Foswiki::Plugins::SpreadSheetPlugin::Calc::cPos = 0;
return Foswiki::Plugins::SpreadSheetPlugin::Calc::_doCalc(
$attributes->{_DEFAULT} );
}
},
($legacyParser) ? 'classic' : 'context-free'
);

# Flag to skip calc if in include
Expand Down
20 changes: 10 additions & 10 deletions SpreadSheetPlugin/test/unit/SpreadSheetPlugin/SpreadSheetPluginTests.pm
View file
Expand Up @@ -992,11 +992,11 @@ sub test_LOG {

sub test_LOWER {
my ($this) = @_;
$this->assert( $this->CALC('$LOWER(lowercase)') eq 'lowercase' );
$this->assert( $this->CALC('$LOWER(LOWERCASE)') eq 'lowercase' );
$this->assert( $this->CALC('$LOWER(lOwErCaSe)') eq 'lowercase' );
$this->assert( $this->CALC('$LOWER()') eq '' );
$this->assert( $this->CALC('$LOWER(`~!@#$%^&*_+{}|:"<>?)') eq
$this->assert( $this->CALC('$LOWER(lowercase)') eq 'lowercase' );
$this->assert( $this->CALC('$LOWER(LOWERCASE)') eq 'lowercase' );
$this->assert( $this->CALC('$LOWER(lOwErCaSe)') eq 'lowercase' );
$this->assert( $this->CALC('$LOWER()') eq '' );
$this->assert( $this->CALC('$LOWER(`~!@#$%^&*_+{}|:\"<>?)') eq
q(`~!@#$%^&*_+{}|:"&lt;&gt;?) );
}

Expand Down Expand Up @@ -1596,11 +1596,11 @@ sub test_unknown {

sub test_UPPER {
my ($this) = @_;
$this->assert( $this->CALC('$UPPER(uppercase)') eq 'UPPERCASE' );
$this->assert( $this->CALC('$UPPER(UPPERCASE)') eq 'UPPERCASE' );
$this->assert( $this->CALC('$UPPER(uPpErCaSe)') eq 'UPPERCASE' );
$this->assert( $this->CALC('$UPPER()') eq '' );
$this->assert( $this->CALC('$UPPER(`~!@#$%^&*_+{}|:"?<>)') eq
$this->assert( $this->CALC('$UPPER(uppercase)') eq 'UPPERCASE' );
$this->assert( $this->CALC('$UPPER(UPPERCASE)') eq 'UPPERCASE' );
$this->assert( $this->CALC('$UPPER(uPpErCaSe)') eq 'UPPERCASE' );
$this->assert( $this->CALC('$UPPER()') eq '' );
$this->assert( $this->CALC('$UPPER(`~!@#$%^&*_+{}|:\"?<>)') eq
q(`~!@#$%^&*_+{}|:"?&lt;&gt;) );
}

Expand Down
12 changes: 10 additions & 2 deletions TestFixturePlugin/lib/Foswiki/Plugins/TestFixturePlugin.pm
View file
Expand Up @@ -174,13 +174,15 @@ sub _processDiff {
sub initPlugin {
( $topic, $web ) = @_;

Foswiki::Func::registerTagHandler( 'STRICTTAG', \&_STRICTTAG );
Foswiki::Func::registerTagHandler( 'STRICTTAG', \&_STRICTTAG, 'classic' );
Foswiki::Func::registerTagHandler( 'FRIENDLYTAG', \&_FRIENDLYTAG,
'context-free' );

return 1;
}

sub commonTagsHandler {
$_[0] =~ s/%FRIENDLYTAG\{(.*?)\}%/&_extractParams($1)/ge;
$_[0] =~ s/%COMMONTAG\{(.*?)\}%/&_extractParams($1)/ge;
}

sub _extractParams {
Expand All @@ -194,6 +196,12 @@ sub _STRICTTAG {
return $params->stringify();
}

sub _FRIENDLYTAG {
my ( $session, $params ) = @_;

return $params->stringify();
}

my $iph = 0;
my $oph = 0;

Expand Down
25 changes: 16 additions & 9 deletions core/data/System/Macros.txt
View file
@@ -1,4 +1,4 @@
%META:TOPICINFO{author="ProjectContributor" date="1489246418" format="1.1" version="1"}%
%META:TOPICINFO{author="ProjectContributor" date="1494802437" format="1.1" version="1"}%
%META:TOPICPARENT{name="UserDocumentationCategory"}%
%STARTINCLUDE%
---+ Macros
Expand All @@ -9,24 +9,31 @@ Macros are text strings in one of three basic forms:

<verbatim class="tml">
%MACRONAME%
%MACRONAME{ parameter="value" }%
%MACRONAME{"default_parameter" named_parameter="value" }%
%MACRONAME{
param1="value "
+"& more "
param2="whatever"
param1+="and even more"
}%
</verbatim>

Parameter names may be upper or lower case, and may consist of the ascii alphas, numbers, and the symbol #, Other characters and Unicode extended
alpha characters are not valid.

The third form is a new feature in Foswiki 2.1 to significantly improve readability with complex macros, see [[#Readable_Macros][Readable Macros]] for details.

These usually expand into content when a topic is rendered for viewing. There are two types of macros:
1 [[PreferenceSettings][Preference settings]]: May be defined and modified by the user
1 Registered macros: Defined by the system or by Plugins (for example, the SpreadSheetPlugin introduces a =%<nop>CALC{}%= macro)
Parameter _names_ may be upper or lower case, and may consist of the ascii alphas, numbers, and the symbol #, Other characters and Unicode extended
alpha characters are not valid.

Parameter _values_ may be entered in either the "Classic" "double-quote delimited"
and "comma separated" format, or in a "Friendly" format.
In addition to the "classic" double-quoted parameters, the "Friendly" parser permits:
* Single-quoted values.
* Space delimited values
* Space or comma separated parameters
* Optional spaces around the equal signs.

Macros usually expand into content when a topic is rendered for viewing. There are two types of macros:
1 [[PreferenceSettings][Preference settings]]: May be defined and modified by the user.
1 Registered macros: Defined by the system or by Plugins (for example, the SpreadSheetPlugin introduces a =%<nop>CALCULATE{}%= macro).
Regardless of how they are defined, both types follow the same rules.
%TOC%

---++ Using Macros
Expand Down
24 changes: 23 additions & 1 deletion core/data/System/ReleaseNotes02x02.txt
View file
@@ -1,4 +1,4 @@
%META:TOPICINFO{author="ProjectContributor" date="1494776564" format="1.1" version="1"}%
%META:TOPICINFO{author="ProjectContributor" date="1494802437" format="1.1" version="1"}%
%META:TOPICPARENT{name="ReleaseHistory"}%
---+!! Foswiki Release 2.2.0

Expand Down Expand Up @@ -121,6 +121,28 @@ if the Foswiki is accessed through a reverse proxy. Foswiki will the use the =X-
This setting should only be enabled if the majority of the clients access the server via the reverse proxy. It is possible for clients to spoof the
=X-Forwarded-For= header, so only enable this setting when appropriate to avoid client IP Address spoofing.

---+++ Friendlly Macro parser is now the default.

In Foswiki 2.1 and earlier, there was an optional "Friendly" macro parser which could be enabled when a plugin registered a Tag. All other macros used
the classic restricted parser. This change makes the new "Friendly" parser the default parser, and adds configuration parameters to restore the Legacy
behaviour. With the new Friendly parser:
* Parameters may be any mixture of:
* single-quoted, ex: =%SOMEMACRO{'default value', parm1='foo bar'}%=
* double-quoted, ex: =%SOMEMACRO{"default value", parm1="foo bar"}%=
* or unquoted spaceless values, ex: =%SOMEMACRO{defaultvalue" parm1 = foobar}%=
* Spaces may be used around the = signs,
* Parameters may be comma or space separated.

If this proves incompatible with legacy topics or macros, the old behaviour can
be enabled in the configure -&gt; Miscellaneous tab. Enable the ={UseLegacyMacroParser}= setting.
The SpreadsheetPlugin can also be configured to use the Legacy parser for the =%<nop>CALCULATE%= macro: Enable
the =SPREADSHEETPLUGIN_LEGACYPARSER= setting in a Site, Web, or Topic preferences.

Plugins can specify the "classic" parser when registering a tag. Specify
='classic'= or ='context-free'= as the 3rd argument to the registerTag call.
In Foswiki 2.2, the default is ='context-free'= unless overridden in the
configuration.

---++ Important changes in Foswiki 2.1

---+++ Deprecations
Expand Down
13 changes: 11 additions & 2 deletions core/data/TestCases/TestCaseAutoAttributeParsing.txt
View file
@@ -1,4 +1,4 @@
%META:TOPICINFO{author="ProjectContributor" date="1229276996" format="1.1" version="1"}%
%META:TOPICINFO{author="ProjectContributor" date="1494802437" format="1.1" version="1"}%

Description: ensure attribute parsing provides expected outcome

Expand All @@ -8,13 +8,14 @@ Designed by: Crawford Currie

Expected outcome in each case is defined as the results of =extractParameters=, the Cairo equivalent to the attribute parser.

This test uses the =STRICTTAG= feature of the !TestFixturePlugin, that simply creates a Foswiki::Attrs from the parameter and then outputs it using =stringify()=
This test uses the =STRICTTAG= and =FRIENDLYTAG= feature of the !TestFixturePlugin, that simply creates a Foswiki::Attrs from the parameter and then outputs it using =stringify()=

<table border=1>
<tr><th>Input string</th>
<th>Expected</th>
<th>Strict</th>
<th>Friendly</th>
<th>Common</th>
</tr>
<tr><td>
def="m\"no" pqr=" stu="vwx""
Expand All @@ -28,6 +29,8 @@ def="m\"no" pqr=" stu="
<!-- /actual -->
</td><td>
%FRIENDLYTAG{ def="m\"no" pqr=" stu="vwx""}%
</td><td>
%COMMONTAG{ def="m\"no" pqr=" stu="vwx""}%
</td></tr>
<tr><td>
</td><td>
Expand All @@ -40,6 +43,8 @@ def="m\"no" pqr=" stu="
<!-- /actual -->
</td><td>
%FRIENDLYTAG{ }%
</td><td>
%COMMONTAG{ }%
</td></tr>
<tr><td>
bloody " hell
Expand All @@ -53,6 +58,8 @@ bloody " hell
<!-- /actual -->
</td><td>
%FRIENDLYTAG{ bloody " hell}%
</td><td>
%COMMONTAG{ bloody " hell}%
</td></tr>
<tr><td>
"abc\" def="ghi" jkl" def="mno" pqr=" stu="\"vwx""
Expand All @@ -66,6 +73,8 @@ bloody " hell
<!-- /actual -->
</td><td>
%FRIENDLYTAG{ "abc\" def="ghi" jkl" def="mno" pqr=" stu="\"vwx""}%
</td><td>
%COMMONTAG{ "abc\" def="ghi" jkl" def="mno" pqr=" stu="\"vwx""}%
</td></tr>
</table>

Expand Down
32 changes: 24 additions & 8 deletions core/lib/Foswiki.pm
View file
Expand Up @@ -360,7 +360,9 @@ BEGIN {
STOPINCLUDE => sub { '' },
ENDINCLUDE => sub { '' },
);
$contextFreeSyntax{IF} = 1;
$contextFreeSyntax{IF} = 1;
$contextFreeSyntax{SCRIPTURLPATH} = 0;
$contextFreeSyntax{PUBURLPATH} = 0;

# Load LocalSite.cfg
if ( Foswiki::Configure::Load::readConfig( 0, 0, 0 ) ) {
Expand Down Expand Up @@ -2553,6 +2555,7 @@ sub parseSections {

return ( '', [] ) unless defined $text;

my $friendlyParser = ( $Foswiki::cfg{UseLegacyMacroParser} ) ? 0 : 1;
my %sections;
my @list = ();

Expand All @@ -2566,7 +2569,7 @@ sub parseSections {
require Foswiki::Attrs;

# SMELL: unchecked implicit untaint?
my $attrs = new Foswiki::Attrs($1);
my $attrs = new Foswiki::Attrs( $1, $friendlyParser );
$attrs->{type} ||= 'section';
$attrs->{name} =
$attrs->{_DEFAULT}
Expand Down Expand Up @@ -2598,7 +2601,7 @@ sub parseSections {
require Foswiki::Attrs;

# SMELL: unchecked implicit untaint?
my $attrs = new Foswiki::Attrs($1);
my $attrs = new Foswiki::Attrs( $1, $friendlyParser );
$attrs->{type} ||= 'section';
$attrs->{name} = $attrs->{_DEFAULT} || $attrs->{name} || '';
delete $attrs->{_DEFAULT};
Expand Down Expand Up @@ -3308,18 +3311,19 @@ sub _expandMacroOnTopicRendering {
my ( $this, $tag, $args, $topicObject ) = @_;

require Foswiki::Attrs;
my $friendlyParser = ( $Foswiki::cfg{UseLegacyMacroParser} ) ? 0 : 1;

my $e = $this->{prefs}->getPreference($tag);
if ( defined $e ) {
if ( $args && $args =~ m/\S/ ) {
my $attrs = new Foswiki::Attrs( $args, 0 );
my $attrs = new Foswiki::Attrs( $args, $friendlyParser );

$e = $this->_processMacros(
$e,
sub {
# Expand %DEFAULT and any parameter tags
my ( $this, $tag, $args, $topicObject ) = @_;
my $tattrs = new Foswiki::Attrs($args);
my $tattrs = new Foswiki::Attrs( $args, $friendlyParser );

if ( $tag eq 'DEFAULT' ) {

Expand Down Expand Up @@ -3352,6 +3356,9 @@ sub _expandMacroOnTopicRendering {
die $@ if $@;
$macros{$tag} = eval "\\&$tag";
die $@ if $@;
unless ( defined $contextFreeSyntax{$tag} ) {
$contextFreeSyntax{$tag} = $friendlyParser;
}
}

my $attrs = new Foswiki::Attrs( $args, $contextFreeSyntax{$tag} );
Expand All @@ -3361,7 +3368,7 @@ sub _expandMacroOnTopicRendering {

# Arbitrary %SOMESTRING{default="xxx"}% will expand to xxx
# in the absence of any definition.
my $attrs = new Foswiki::Attrs($args);
my $attrs = new Foswiki::Attrs( $args, $friendlyParser );
if ( defined $attrs->{default} ) {
$e = expandStandardEscapes( $attrs->{default} );
}
Expand Down Expand Up @@ -3490,8 +3497,17 @@ The syntax isn't vastly different from what's there; the differences are:
sub registerTagHandler {
my ( $tag, $fnref, $syntax ) = @_;
$macros{$tag} = $fnref;
if ( $syntax && $syntax eq 'context-free' ) {
$contextFreeSyntax{$tag} = 1;
$contextFreeSyntax{$tag} = ( $Foswiki::cfg{UseLegacyMacroParser} ) ? 0 : 1;
if ($syntax) {
if ( $syntax eq 'context-free' ) {
$contextFreeSyntax{$tag} = 1;
}
elsif ( $syntax eq 'classic' ) {
$contextFreeSyntax{$tag} = 0;
}
elsif (DEBUG) {
ASSERT("Incorrect syntax requested in macro $tag");
}
}
}

Expand Down
18 changes: 13 additions & 5 deletions core/lib/Foswiki.spec
View file
Expand Up @@ -2316,9 +2316,9 @@ $Foswiki::cfg{NotifyTopicName} = 'WebNotify';

#---++ Compatibility
# This section contains options that you can use to enforce compatibility
# with older releases of Foswiki.
# with older releases of Foswiki or TWiki.

# **BOOLEAN EXPERT LABEL="Require Compatible Anchors"**
# **BOOLEAN LABEL="Require Compatible Anchors"**
# 'Anchors' are positions within a Foswiki page that can be targeted in
# a URL using the =#anchor= syntax. The format of these anchors has
# changed several times. If this option is set, Foswiki will generate extra
Expand All @@ -2329,14 +2329,14 @@ $Foswiki::cfg{NotifyTopicName} = 'WebNotify';
# links to the internals of pages to continue to work.
$Foswiki::cfg{RequireCompatibleAnchors} = $FALSE;

# **BOOLEAN EXPERT LABEL="Accept User Password on GET"**
# **BOOLEAN LABEL="Accept User Password on GET"**
# Enable this setting if you want
# =username= and =password= parameters to be accepted on a GET request when
# provided as part of the query string. It is more secure to restrict login
# operations to POST requests only.
$Foswiki::cfg{Session}{AcceptUserPwParamOnGET} = $FALSE;

# **BOOLEAN EXPERT LABEL="Map IP to Session ID" DISPLAY_IF="{UseClientSessions}" CHECK="iff:'{UseClientSessions}'" EXPERT**
# **BOOLEAN LABEL="Map IP to Session ID" DISPLAY_IF="{UseClientSessions}" CHECK="iff:'{UseClientSessions}'" EXPERT**
# For compatibility with older versions, Foswiki supports the mapping of the
# clients IP address to a session ID. You can only use this if all
# client IP addresses are known to be unique.
Expand All @@ -2347,14 +2347,22 @@ $Foswiki::cfg{Session}{AcceptUserPwParamOnGET} = $FALSE;
# _off_.
$Foswiki::cfg{Sessions}{MapIP2SID} = $FALSE;

# **BOOLEAN EXPERT LABEL="Disable automatic macros on topic template expansion"**
# **BOOLEAN LABEL="Enable automatic macros on topic template expansion"**
# In Foswiki version 2.1 and earlier, the macros =DATE=, =GMTIME=, =SERVERTIME=,
# =USERNAME=, =URLPARAM=, =WIKINAME=, and =WIKIUSERNAME= were automatically expanded
# on template creation. This has been disabled by default, as the =CREATE:=
# prefix in topic templates serves the same function much more cleanly.
# You can re-enable the old behaviour here.
$Foswiki::cfg{ExpandSomeMacrosOnTopicCreation} = $FALSE;

# **BOOLEAN LABEL="Use Legacy Macro Parser" **
# In Foswiki version 2.1 and earlier, the macro parser used a restricted format
# where parameter values must all be double-quoted. The new Parser
# also parses single-quoted values, unquoted spaceless values, spaces around the =,
# and commas as well as spaces separating values.
# Enable this option to restrict macro parsing to the legacy double-quoted syntax.
$Foswiki::cfg{UseLegacyMacroParser} = $FALSE;

#############################################################################
#---+ Extensions

Expand Down

0 comments on commit 21c5775

Please sign in to comment.