Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
Item13516: corrected attachment transfer. There are still cases which…
… may confuse it, but it should recover from just about anything now. Plus fixed and disabled annoying RCS files check in PFS - it should only be enabled in the specific case where someone has ignored the documentation and tried to overlay RCS and PFS stores.
  • Loading branch information
Crawford Currie committed Jul 17, 2015
1 parent 36aca0d commit ba0a5df
Show file tree
Hide file tree
Showing 5 changed files with 160 additions and 51 deletions.
Expand Up @@ -7,4 +7,12 @@
# programs. The use of text files makes it easy to implement 'out of band'
# processing, as well as taking maximum advantage of filestore caching. This
# is the reference implementation of a store.</dd></dl>
# ---+ Extensions
# ---++ PlainFileStoreContrib
# **BOOLEAN**
# Check before every store modification that there are no suspicious
# files left over from RCS. This check should be enabled whenever there
# is a risk that old RCS data has been mixed in to a PlainFileStore.
$Foswiki::cfg{Extensions}{PlainFileStoreContrib}{CheckForRCS} = 0;

1;
8 changes: 7 additions & 1 deletion PlainFileStoreContrib/lib/Foswiki/Store/PlainFile.pm
Expand Up @@ -1102,7 +1102,13 @@ sub _saveDamage {
my $latest = _latestFile( $meta, $attachment );
return unless ( _e $latest );

if ( _e "$latest,v" && !$Foswiki::inUnitTestMode ) {
if ( $Foswiki::cfg{Extensions}{PlainFileStoreContrib}{CheckForRCS}
&& !$Foswiki::inUnitTestMode
&& _e("$latest,v") )
{
my $path =
Encode::encode_utf8( $Foswiki::cfg{DataDir} ) . "/"
. $meta->getPath();
die <<DONE;
PlainFileStore is selected but you have ,v files present in the directory tree, Save aborted to avoid loss of topic history.
Did you remember to convert the store? The administrator should review tools/bulk_copy.pl, or select an RCS based store.
Expand Down
2 changes: 1 addition & 1 deletion core/lib/Foswiki/Meta.pm
Expand Up @@ -2480,7 +2480,7 @@ sub replaceMostRecentRevision {
---++ ObjectMethod getRevisionHistory([$attachment]) -> $iterator
Get an iterator over the range of version identifiers (just the identifiers,
not the content).
not the content) starting with the most recent revision.
The iterator will be empty ($iterator->hasNext() will be false) if the object
does not exist.
Expand Down
2 changes: 1 addition & 1 deletion core/lib/Foswiki/Store.pm
Expand Up @@ -559,7 +559,7 @@ from an input stream =$stream=.
* =comment= - a comment associated with the save
Returns the number of the revision saved.
Note: =\%options= was added in Foswiki 1.2
Note: =\%options= was added in Foswiki 2.0
=cut
Expand Down
191 changes: 143 additions & 48 deletions core/tools/bulk_copy.pl
Expand Up @@ -136,14 +136,18 @@ sub call {

$/ = "\n";
my $response = <FROM_B>;
return undef unless defined $response;
die "Bad response from peer" unless defined $response;
( my $status, $response ) = split( ':', $response, 2 );
die "Bad response from peer" unless $status =~ /^\d+$/;

# Response is utf8-encoded JSON
$response = $json->decode($response);

# Convert to {Site}{CharSet}, if necessary
$response = convert( $response, 'from_unicode' ) unless $Foswiki::UNICODE;

die $response if $status;

return $response;
}

Expand Down Expand Up @@ -219,11 +223,34 @@ sub copy_web {
}

# Copy a single topic and all it's attachments.
# Transferring attachments is trickier than it should be, because
# the way attachments are stored and managed in RCS means that
# there isn't a 1:1 correspondence between attachment versions
# mentioned in the META:FILEATTACHMENT and the attachments actually
# present in the history. So we establish the maximum rev number
# for each attachment as and when we encounter it in the topic history.
# At that time we also interrogate the store to determine the maximum
# number of revs of the attachment that are available in the attachment
# history. Then as we replay the history from the oldest topic version
# forwards, we are able to run the attachment version up to the version
# encountered in the topic.
#
# There are a number of ways revision histories of attachments can
# get mangled.
# 1. The topic can reference a revision that doesn't exist in the
# attachment history.
# 2. The topic can reference a revision that is *newer* than the
# revision in a newer version of the topic e.g.
# Topic rev 1 references attachment rev 2
# Topic rev 2 reference attachment rev 1
# 3. Attachments may not be referenced in topics at all.
sub copy_topic {
my ( $web, $topic ) = @_;

announce "Copy topic $web.$topic";
my $topicMO = Foswiki::Meta->new( $session, $web, $topic );

# The revision list is sorted starting with the most recent revision
my @rev_list = $topicMO->getRevisionHistory()->all();
if ( grep { $topic =~ /^$_$/ } @{ $control{latest} } ) {
announce "\t-only latest";
Expand All @@ -233,7 +260,7 @@ sub copy_topic {
}
my %att_tx;

# Replay history
# Replay history, *oldest* rev first
while ( my $tv = pop @rev_list ) {

$topicMO->unload();
Expand All @@ -259,49 +286,97 @@ sub copy_topic {
# embeds in $data, which may not be wise.
call( 'saveTopicRev', $web, $topic, $data );

# Transfer attachments. We use eachAttachment rather than
# META:FILEATTACHMENT because it won't stumble over deleted
# attachments. An attachment, and its history, can be
# completely removed from some stores, leaving
# META:FILEATTACHMENT still in older revs of the topic.
# Transfer attachments.
my $tri;
my $att_it = $topicMO->eachAttachment();
while ( $att_it->hasNext() ) {
my $att_name = $att_it->next();
my $att_info = $topicMO->get( 'FILEATTACHMENT', $att_name );
$att_tx{$att_name} ||= 0;

# Is there info about this attachment in this rev of the
# topic? If not, we can't do anything useful.
next unless $att_info;
my $att_info = $topicMO->get( 'FILEATTACHMENT', $att_name );
next unless ($att_info);

my $att_version = $att_info->{version};
unless ( $att_info->{version} ) {
unless ($att_version) {
announce
" Attachment $att_name has corrupt meta-data - cannot copy";
next;
}

# Already copied?
next if $att_version == $att_tx{$att_name};

announce " Attachment $att_name:$att_version ";

# Check if the attachment history is mangled
if ( $att_version < $att_tx{$att_name} ) {
announce
"- $web.$topic\[$tv\]:$att_name has corrupt FILEATTACHMENT meta-data - cannot copy";
" - version $att_version is behind one that has already been copied ($att_tx{$att_name}) - cannot copy";
next;
}
next if $att_tx{"$att_name:$att_info->{version}"};
$att_tx{"$att_name:$att_version"} = 1;

unless ( $att_info->{author} ) {
unless ($tri) {
$tri = $topicMO->getRevisionInfo();
}
$att_info->{author} = $tri->{user};
require Foswiki::Users::BaseUserMapping;
$att_info->{author} =
$Foswiki::Users::BaseUserMapping::UNKNOWN_USER_CUID;
}
my $fh =
$topicMO->openAttachment( $att_name, '<',
version => $att_info->{version} );

# TODO: chunked transfer
local $/ = undef;
my $att_data = <$fh>;
# Copy hidden intermediates up to and including the version
# referenced.
while ( $att_tx{$att_name} < $att_version ) {
$att_tx{$att_name}++;
$att_info->{version} = $att_tx{$att_name};
copy_attachment_version( $topicMO, $att_info );
}
}
}

announce " Attach $att_name\[$att_info->{version}\]";
call( 'saveAttachmentRev',
$web, $topic, $att_name, $att_info, \$att_data );
# Any attachments that are seen by the store but haven't been
# copied are never referenced in any topic version. Copy all
# their revs.
while ( my ( $att_name, $rev ) = each %att_tx ) {
next if $rev;
announce " Copy hidden attachment $att_name";
require Foswiki::Users::BaseUserMapping;
my %att_info = (
name => $att_name,
author => $Foswiki::Users::BaseUserMapping::UNKNOWN_USER_CUID,
date => 0
);
my @revs = $topicMO->getRevisionHistory($att_name)->all();
while ( my $rev = pop(@revs) ) {
$att_info{version} = $rev;
copy_attachment_version( $topicMO, \%att_info );
}
}
}

sub copy_attachment_version {
my ( $meta, $att_info ) = @_;
my $att_name = $att_info->{name};
my $att_ver = $att_info->{version};

my $fh = $meta->openAttachment( $att_name, '<', version => $att_ver );

# Write data into temp file
my $tfh = File::Temp->new( UNLINK => 0, SUFFIX => '.dat' );

# Temp file will be unlinked in receiver
binmode $tfh;
local $/ = undef;
my $data = <$fh>;
print $tfh $data;
close($tfh);
my $tfn = $tfh->filename();

# $tfn passed by reference to block encoding
my $rev =
call( 'saveAttachmentRev', $meta->web, $meta->topic, $att_info, \$tfn );
announce " Attached $att_ver as $rev";
}

#######################################################################
# Receiver

Expand All @@ -321,10 +396,17 @@ sub dispatch {

my $fn = shift(@$data); # function name

no strict 'refs';
my $response = &$fn(@$data);
use strict 'refs';
$response = 0 unless defined $response;
my $response;
eval {
no strict 'refs';
$response = &$fn(@$data);
use strict 'refs';
};
my $status = 0;
if ($@) {
$status = 1;
$response = $@;
}

# Convert response to unicode (if necessary)
$response = convert( $response, 'to_unicode' ) if $Foswiki::UNICODE;
Expand All @@ -333,7 +415,7 @@ sub dispatch {
debug "$fn(", join( ', ', @$data ), ") -> $response";

# JSON encode and print
print TO_A "$response\n";
print TO_A "$status:$response\n";

return 1;
}
Expand All @@ -360,7 +442,7 @@ sub saveWeb {
my $web = shift;
return '' if $session->webExists($web);
my $mo = Foswiki::Meta->new( $session, $web );
return 0 if $control{check_only};
return '' if $control{check_only};
$mo->populateNewWeb();

# Return the name of the web preferences topic, as it was just created
Expand Down Expand Up @@ -407,23 +489,32 @@ sub saveTopicRev {
# Given revision info and data for the attachment
# save this rev.
sub saveAttachmentRev {
my ( $web, $topic, $attachment, $info, $data ) = @_;
my ( $web, $topic, $info, $fname ) = @_;
my $mo = Foswiki::Meta->new( $session, $web, $topic );
my $fh;
open( $fh, '<', $data ); # Open string for input
return 0 if $control{check_only};
$mo->attach(
name => $attachment,
dontlog => 1,
notopicchange => 1,
author => $info->{author},
filedate => $info->{date},
forcedate => $info->{date},
stream => $fh,

# Don't call handlers (works on 2+ only)
nohandlers => 1
);
my $rev = $info->{version};
unless ( $control{check_only} ) {

# Can't use $mo->attach because it updates the FILEATTACHMENT
# metadata, and we've already written that when transferring
# the topic. So we have to kick under it to the
# Store::saveAttachment method - which is OK because it's part
# of the (relatively stable) store API.
my $fh;
open( $fh, '<', $fname ) || die "Failed to open $fname";
$rev = $mo->session->{store}->saveAttachment(
$mo,
$info->{name},
$fh,
$info->{author},
{ # Only works for Foswiki 2.0
forcedate => $info->{date},
comment => $info->{comment}
}
);
close($fh);
}
unlink($fname);
return $rev;
}

# convert * and ? to regex, and quotemeta other re chars
Expand Down Expand Up @@ -634,6 +725,10 @@ =head1 SYNOPSIS
topic or attachment already exists in the target installation, it will
be reported but not copied.
Note that while all attachments are copied, only attachment revisions that
are explicitly listed in topic revisions are copied. Intermediate attachment
versions (e.g. those created by processes outside of Foswiki) are not copied.
=head1 OPTIONS
=head2 Selecting Webs and Topics
Expand Down

0 comments on commit ba0a5df

Please sign in to comment.