Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fetch-kde-qt.sh: support current layout of download.kde.org #104684

Closed
wants to merge 5 commits into from
Closed

fetch-kde-qt.sh: support current layout of download.kde.org #104684

wants to merge 5 commits into from

Conversation

hamburger1984
Copy link
Contributor

@hamburger1984 hamburger1984 commented Nov 23, 2020 •
edited

Motivation for this change

The script fetch-kde-qt.sh stopped working for the (new?) layout of download.kde.org. This PR restores functionality to fetch source package names and sha256 sums for plasma, kde and qt to write them into srcs.nix files.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@hamburger1984 hamburger1984 mentioned this pull request Nov 23, 2020
Closed
9 tasks
@Thra11
Copy link
Member

Thra11 commented Nov 23, 2020

Thanks for all your work on this!
When I try to use this to update kde-frameworks to 5.76:

./maintainers/scripts/fetch-kde-qt.sh pkgs/development/libraries/kde-frameworks/

it eventually fails with:

Saving to: ‘attica-5.76.0.tar.xz.mirrorlist’

attica-5.76.0.tar.xz.mirrorlist                    [ <=>                                                                                               ]  26.89K  --.-KB/s    in 0.08s   

2020-11-23 16:50:48 (357 KB/s) - ‘attica-5.76.0.tar.xz.mirrorlist’ saved [27537]

--2020-11-23 16:50:48--  https://download.kde.org/stable/frameworks/5.76/kxmlrpcclient-5.76.0.tar.xz.mirrorlist
Resolving download.kde.org (download.kde.org)... 168.119.32.158, 2a01:4f8:242:1ed5::3
Connecting to download.kde.org (download.kde.org)|168.119.32.158|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2020-11-23 16:50:48 ERROR 404: Not Found.

(plasma 20.03 needs a newer extra-cmake-modules)

SuperSandro2000
SuperSandro2000 reviewed Nov 23, 2020
View reviewed changes
maintainers/scripts/fetch-kde-qt.sh Outdated Show resolved Hide resolved
maintainers/scripts/fetch-kde-qt.sh Outdated Show resolved Hide resolved
@hamburger1984
Copy link
Contributor Author

Thanks for all your work on this!
When I try to use this to update kde-frameworks to 5.76:

./maintainers/scripts/fetch-kde-qt.sh pkgs/development/libraries/kde-frameworks/

it eventually fails with:

Saving to: ‘attica-5.76.0.tar.xz.mirrorlist’

attica-5.76.0.tar.xz.mirrorlist                    [ <=>                                                                                               ]  26.89K  --.-KB/s    in 0.08s   

2020-11-23 16:50:48 (357 KB/s) - ‘attica-5.76.0.tar.xz.mirrorlist’ saved [27537]

--2020-11-23 16:50:48--  https://download.kde.org/stable/frameworks/5.76/kxmlrpcclient-5.76.0.tar.xz.mirrorlist
Resolving download.kde.org (download.kde.org)... 168.119.32.158, 2a01:4f8:242:1ed5::3
Connecting to download.kde.org (download.kde.org)|168.119.32.158|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2020-11-23 16:50:48 ERROR 404: Not Found.

Thanks! Will take a look at this.

@hamburger1984
Copy link
Contributor Author

hamburger1984 commented Nov 23, 2020 •
edited

@Thra11 should work for kxmlrpcclient (and other packages in subfolders) as well now.

@hamburger1984 hamburger1984 mentioned this pull request Nov 24, 2020
Closed
10 tasks
@Thra11
Copy link
Member

Thra11 commented Nov 24, 2020

Works for me!

@hamburger1984 hamburger1984 changed the title Upgrade fetch-kde-qt.sh to support current layout of download.kde.org fetch-kde-qt.sh: support current layout of download.kde.org Nov 30, 2020
@hamburger1984
Copy link
Contributor Author

Should I open an issue for this to get a better chance of getting reviewed and/or merged?
@SuperSandro2000 (don't know if you're the right one to ask)

@SuperSandro2000
Copy link
Member

Should I open an issue for this to get a better chance of getting reviewed and/or merged?

Not required. Backlog is just to big.

@SuperSandro2000
Copy link
Member

SuperSandro2000 commented Dec 2, 2020 •
edited

If you do not supply $1 you get the following:

./maintainers/scripts/fetch-kde-qt.sh
these paths will be fetched (0.63 MiB download, 3.09 MiB unpacked):
  /nix/store/xax35k4di9hxm0xsim0lbfdsami32dip-wget-1.20.3
copying path '/nix/store/xax35k4di9hxm0xsim0lbfdsami32dip-wget-1.20.3' from 'https://cache.nixos.org'...
./maintainers/scripts/fetch-kde-qt.sh: line 7: $1: unbound variable

Can you default this to --help and add an usage example there?

Something like eg ./maintainers/scripts/fetch-kde-qt.sh pkgs/development/libraries/kde-frameworks/ is totally fine.

@SuperSandro2000
Copy link
Member

@SuperSandro2000 (don't know if you're the right one to ask)

I can definitely merge it and as it is not something bootstrap, stdenv or other very crucial parts I feel comfortable when it works.

But I found another bug:

FINISHED --2020-12-02 10:18:34--
Total wall clock time: 3m 37s
Downloaded: 101 files, 888K in 0.3s (3.41 MB/s)
--2020-12-02 10:18:34--  https://download.kde.org/kde/ftp/stable/frameworks/5.73/portingAids/kxmlrpcclient-5.73.0.tar.xz.mirrorlist
Resolving download.kde.org (download.kde.org)... 168.119.32.158, 2a01:4f8:242:1ed5::3
Connecting to download.kde.org (download.kde.org)|168.119.32.158|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2020-12-02 10:18:35 ERROR 404: Not Found

@hamburger1984
Copy link
Contributor Author

Thanks for the feedback, @SuperSandro2000! I'll have a look and come back to you.

@hamburger1984
Copy link
Contributor Author

@SuperSandro2000 (don't know if you're the right one to ask)

I can definitely merge it and as it is not something bootstrap, stdenv or other very crucial parts I feel comfortable when it works.

But I found another bug:

FINISHED --2020-12-02 10:18:34--
Total wall clock time: 3m 37s
Downloaded: 101 files, 888K in 0.3s (3.41 MB/s)
--2020-12-02 10:18:34--  https://download.kde.org/kde/ftp/stable/frameworks/5.73/portingAids/kxmlrpcclient-5.73.0.tar.xz.mirrorlist
Resolving download.kde.org (download.kde.org)... 168.119.32.158, 2a01:4f8:242:1ed5::3
Connecting to download.kde.org (download.kde.org)|168.119.32.158|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2020-12-02 10:18:35 ERROR 404: Not Found

How did you trigger this? What's the content of your fetch.sh in pkgs/development/libraries/kde-framework? That "ftp" in the path looks suspicious. It should be something like https://download.kde.org/stable/frameworks/5.73/

@hamburger1984
Copy link
Contributor Author

If you do not supply $1 you get the following:

./maintainers/scripts/fetch-kde-qt.sh
these paths will be fetched (0.63 MiB download, 3.09 MiB unpacked):
  /nix/store/xax35k4di9hxm0xsim0lbfdsami32dip-wget-1.20.3
copying path '/nix/store/xax35k4di9hxm0xsim0lbfdsami32dip-wget-1.20.3' from 'https://cache.nixos.org'...
./maintainers/scripts/fetch-kde-qt.sh: line 7: $1: unbound variable

Can you default this to --help and add an usage example there?

Something like eg ./maintainers/scripts/fetch-kde-qt.sh pkgs/development/libraries/kde-frameworks/ is totally fine.

Done.

@SuperSandro2000
Copy link
Member

How did you trigger this?

I did nixpkgs-review pr 104684, cd nixpkgs and run ./maintainers/scripts/fetch-kde-qt.sh pkgs/development/libraries/kde-frameworks/.

What's the content of your fetch.sh in pkgs/development/libraries/kde-framework?

WGET_ARGS=(https://download.kde.org/stable/frameworks/5.73/)

Could it be that I am using wget2?

@hamburger1984
Copy link
Contributor Author

How did you trigger this?

I did nixpkgs-review pr 104684, cd nixpkgs and run ./maintainers/scripts/fetch-kde-qt.sh pkgs/development/libraries/kde-frameworks/.

What's the content of your fetch.sh in pkgs/development/libraries/kde-framework?

WGET_ARGS=(https://download.kde.org/stable/frameworks/5.73/)

Could it be that I am using wget2?

I did exactly the same and it works for me. My wget --version is 1.20.3. which wget is /run/current-system/sw/bin/wget which links to /nix/store/s9mf59032b6nyd3bff3rp27zh439k7a4-wget-1.20.3/bin/wget

@hamburger1984
Copy link
Contributor Author

How do I upgrade to wget2? @SuperSandro2000

@SuperSandro2000
Copy link
Member

SuperSandro2000 commented Dec 3, 2020 •
edited

How do I upgrade to wget2? @SuperSandro2000

apt get install wget2 😂

@SuperSandro2000
Copy link
Member

Without wget2 I get the following error:

Removing stable/frameworks/5.73/portingAids/index.html?C=S;O=D.tmp.tmp since it should be rejected.

--2020-12-03 10:37:34--  https://download.kde.org/stable/frameworks/5.73/portingAids/?C=D;O=D
Reusing existing connection to download.kde.org:443.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
--2020-12-03 10:37:34--  https://download.kde.org/stable/frameworks/5.73/portingAids/?C=D;O=D
Reusing existing connection to download.kde.org:443.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘stable/frameworks/5.73/portingAids/index.html?C=D;O=D.tmp.tmp’

stable/frameworks/5.73/portingAids/index.html?     [ <=>                                                                                                 ]  15.77K  --.-KB/s    in 0.002s

2020-12-03 10:37:34 (6.40 MB/s) - ‘stable/frameworks/5.73/portingAids/index.html?C=D;O=D.tmp.tmp’ saved [16147]

Removing stable/frameworks/5.73/portingAids/index.html?C=D;O=D.tmp.tmp since it should be rejected.

FINISHED --2020-12-03 10:37:34--
Total wall clock time: 32s
Downloaded: 101 files, 888K in 0.3s (2.85 MB/s)
--2020-12-03 10:37:34--  https://download.kde.org/pub/kde/stable/frameworks/5.73/portingAids/kross-5.73.0.tar.xz.mirrorlist
Resolving download.kde.org (download.kde.org)... 168.119.32.158, 2a01:4f8:242:1ed5::3
Connecting to download.kde.org (download.kde.org)|168.119.32.158|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2020-12-03 10:37:35 ERROR 404: Not Found

@hamburger1984
Copy link
Contributor Author

hamburger1984 commented Dec 3, 2020 •
edited

Without wget2 I get the following error:

Removing stable/frameworks/5.73/portingAids/index.html?C=S;O=D.tmp.tmp since it should be rejected.

--2020-12-03 10:37:34--  https://download.kde.org/stable/frameworks/5.73/portingAids/?C=D;O=D
Reusing existing connection to download.kde.org:443.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
--2020-12-03 10:37:34--  https://download.kde.org/stable/frameworks/5.73/portingAids/?C=D;O=D
Reusing existing connection to download.kde.org:443.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘stable/frameworks/5.73/portingAids/index.html?C=D;O=D.tmp.tmp’

stable/frameworks/5.73/portingAids/index.html?     [ <=>                                                                                                 ]  15.77K  --.-KB/s    in 0.002s

2020-12-03 10:37:34 (6.40 MB/s) - ‘stable/frameworks/5.73/portingAids/index.html?C=D;O=D.tmp.tmp’ saved [16147]

Removing stable/frameworks/5.73/portingAids/index.html?C=D;O=D.tmp.tmp since it should be rejected.

FINISHED --2020-12-03 10:37:34--
Total wall clock time: 32s
Downloaded: 101 files, 888K in 0.3s (2.85 MB/s)
--2020-12-03 10:37:34--  https://download.kde.org/pub/kde/stable/frameworks/5.73/portingAids/kross-5.73.0.tar.xz.mirrorlist
Resolving download.kde.org (download.kde.org)... 168.119.32.158, 2a01:4f8:242:1ed5::3
Connecting to download.kde.org (download.kde.org)|168.119.32.158|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2020-12-03 10:37:35 ERROR 404: Not Found

uhm.. I don't understand what caused the script to (wrongly) look at https://download.kde.org/pub/kde/stable/frameworks/.. while it was (correctly) looking at https://download.kde.org/stable/frameworks/.. in the previous request.

@hamburger1984 hamburger1984 mentioned this pull request Dec 26, 2020
Closed
10 tasks
@hamburger1984
Copy link
Contributor Author

We might as well close this - probably made obsolete by commit 3db0433

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SuperSandro2000 SuperSandro2000 Awaiting requested review from SuperSandro2000

Assignees
No one assigned
Labels
10.rebuild-darwin: 0 10.rebuild-linux: 0 12. first-time contribution
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@hamburger1984 @Thra11 @SuperSandro2000 @Janik-Haag