If these are workarounds, shouldn't they be using record_soft_failure with a relevant reference to the bugs in question so we know when those workarounds are being triggered?

it's unlikely though that we will fix the 42.2 FTP repo - so it will be soft failure forever :(

The test tries to do zypper_call "in java-*"; and if fails in opensuse because of boo#1019090. So, the current workaround is to check if DISTRI is opensuse and then install everything except from the bootstrap packages. However, instead of "hardcoding" it like this and making it work only for this specific case, it would be better if I would go with a more generic approach. For example:

1. Try to install java-* packages
2. If there is a dependency conflict, save a screenshot.
3. Mark this as a soft-failed test.
4. Continue the zypper installation by choosing the solution 1 (which is kind of the default)

All these are already implemented in tests/installation/zdup.pm by @okurz . So, one way it is to copy and paste a part of his code:

my $self = shift;

# precompile regexes
my $zypper_dup_continue      = qr/^Continue\? \[y/m;
my $zypper_dup_conflict      = qr/^Choose from above solutions by number[\s\S,]* \[1/m;
my $zypper_dup_notifications = qr/^View the notifications now\? \[y/m;
my $zypper_dup_error         = qr/^Abort, retry, ignore\? \[a/m;
my $zypper_dup_finish        = qr/^There are some running programs that might use files|^ZYPPER-DONE/m;
my $zypper_packagekit        = qr/^Tell PackageKit to quit\?/m;
my $zypper_packagekit_again  = qr/^Try again\?/m;
my $zypper_repo_disabled     = qr/^Repository '[^']+' has been successfully disabled./m;
my $zypper_installing        = qr/Installing: \S+/;
my $zypper_dup_fileconflict  = qr/^File conflicts .*^Continue\? \[y/ms;
my $zypper_retrieving        = qr/Retrieving \S+/;
my $zypper_check_conflicts   = qr/Checking for file conflicts: \S+/;

script_run("(zypper in java-*;echo ZYPPER-DONE) | tee /dev/$serialdev", 0);

$out = wait_serial([$zypper_dup_continue, $zypper_dup_conflict, $zypper_dup_error], 240);
while ($out) {
    if ($out =~ $zypper_dup_conflict) {
        if (get_var("WORKAROUND_DEPS")) {
            record_soft_failure 'workaround dependencies';
            send_key '1';
            send_key 'ret';
        }
        else {
            $self->result('fail');
            save_screenshot;
            return;
        }
    }
    elsif ($out =~ $zypper_dup_continue) {
        # confirm zypper dup continue
        send_key 'y';
        send_key 'ret';
        last;
    }
    elsif ($out =~ $zypper_dup_error) {
        $self->result('fail');
        save_screenshot;
        return;
    }
    save_screenshot;
    $out = wait_serial([$zypper_dup_continue, $zypper_dup_conflict, $zypper_dup_error], 120);
}
unless ($out) {
    $self->result('fail');
    save_screenshot;
    return;
}

# wait for zypper dup finish, accept failures in meantime
my $post_checks = [
    $zypper_dup_finish,       $zypper_installing,      $zypper_dup_notifications, $zypper_dup_error,
    $zypper_dup_fileconflict, $zypper_check_conflicts, $zypper_retrieving
];
$out = wait_serial($post_checks, 480);
while ($out) {
    if ($out =~ $zypper_dup_notifications) {
        send_key 'n';    # do not show notifications
        send_key 'ret';
    }
    elsif ($out =~ $zypper_dup_error) {
        $self->result('fail');
        save_screenshot;
        return;
    }
    elsif ($out =~ $zypper_dup_finish) {
        last;
    }
    elsif ($out =~ $zypper_retrieving or $out =~ $zypper_check_conflicts) {
        # probably to avoid hitting black screen on video
        send_key 'shift';
        # continue but do a check again
        $out = wait_serial($post_checks, 240);
        next;
    }
    elsif ($out =~ $zypper_dup_fileconflict) {
        #             record_soft_failure;
        #             send_key 'y';
        #             send_key 'ret';
        $self->result('fail');
        save_screenshot;
        return;
    }
    else {
        # probably to avoid hitting black screen on video
        send_key 'shift';
    }
    save_screenshot;
    $out = wait_serial([$zypper_dup_finish, $zypper_installing, $zypper_dup_notifications, $zypper_dup_error], 240);
}

In the meanwhile, instead of repeating the previous codeblock, I would prefer to use an API call similar to zypper_call. However, I am not aware if there is any. Do you think that it makes sense to create a new call for that? IHMO it makes sense to have a zypper call that detects conflicts, saves a soft-failure and then proceeds with solution 1. The aforementioned source code does all this beautifully, so I suppose it would not be that hard to make a function out of it. Right?

After all, I am open to suggestions, what do you guys think?

As for the bnc#1014602 this is an invalid bug, so there is no reason to mark this as a soft-failure. I would say it's rather a weird to have a package with version 1.7.1 but when I ask for the version of it (using the typical --version) it responds with 1.7.0 and on top of that this is expected. So, it's a specific case from IBM, but it's not a bug or a failure according to the them, so it makes no sense to flag it as a problem (soft or not). Hm?

I didn't implement the part you mentioned, I just changed it and adopted maintainership.

In general, anytime you want to reuse code move it up in the hierarchy to the first common point, e.g. a baseclass which is common to the test modules that share the functionality. You can do that by extracting the part into a function, find a baseclass (e.g. lib/opensusebasetest.pm), define the function as a object method there and call the object method from the test modules.

In general the procedure with record_soft_fail should be like this:

• if a test fails, create a ticket
• change the test code and reference the ticket

recording a soft failure without a reference is a bad approach as - honestly - no one will care until it might be too late. So you want a test to fail and only explicitly accept dependency issues in certain and known cases. You maybe could make it a bit easier by doing something like assert_screen 'dependency_issue_please_create_workaround_needle'; and needle appropriately which looks hacky but could be faster than changing test code.

@okurz thanks for the clarification. So, after talking with @asemen we concluded to another simpler solution for boo#1019090:

if the test runs in opensuse, then
    if there are java-bootstrap packages available in the repositories, then
        if trying to install java along with bootstrap packages fails, then
            mark soft failure with the bug number
            # this means they fixed (or not) the bug, by removing the bootstrap pkgs
            continue by not installing the bootstrap packages
        else
            # this means they fixed the bug so both pks exist but not conflict
            continue by installing java along with bootstrap packages

In that case, no matter of Leap or TW, if such bug exists in opensuse it will be flagged as soft-failed (hitting boo#1019090) and the test will continue with the workaround.

I recommitted and rebased. The new code works as expected, tested in:

• Leap 42.2
• Tumbleweed
• SLES 12 SP1
• SLES 12 SP2

Please review the code, and merge if you are OK with the changes :)

+use base "installbasetest"; this is not needed, let me remove it and re-commit.