Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

treewide: Per RFC45, remove all unquoted URLs #83909

Closed
wants to merge 1 commit into from

Conversation

OmnipotentEntity
Copy link
Contributor

@OmnipotentEntity OmnipotentEntity commented Apr 1, 2020

Motivation for this change

Per RFC45, unquoted URL syntax is deprecated. Unfortunately, I was unaware of this, and myself along with probably many other newer contributors really form habits around packages that are already in the repository.

So this pull request aims to clean up the vast majority of these unquoted strings. So that people learning learn the newer habits, and so that when an automated message comes along, we won't all be drowning in warnings.

This pull request should contain no changes other than quoting previously unquoted strings. If you see something in this PR that does not fit that criteria, please flag me, and I'll fix it. Because this is only fixing quotes on strings, this change should not actually cause any rebuilds, if it does, that is also a bug.

Unfortunately, while I have performed various sorts of static analysis on this PR, it is far too large for any single person to verify and test. I also expect it to need to be fixed for merge conflicts. If you are a maintainer interested in merging this PR, please flag me down on IRC, and I'll fix up the latest merge conflicts for you right before you want to merge. I will also keep it refreshed at a rate of at least once per day while under review.

@OmnipotentEntity
Copy link
Contributor Author

OmnipotentEntity commented Apr 1, 2020

If I missed an unquoted string (most likely within a list or as a sub field) please let me know as well, and I will quote it. But the goal of this PR isn't necessarily 100% coverage, it's 99%+ to help others form good habits / avoid bad habits. This PR is probably too huge to aim for completely perfection in a realistic time frame.

@bhipple
Copy link
Contributor

bhipple commented Apr 1, 2020

Nice! Since this diff is enormous and you're willing/prepared to handle merge conflicts, I'm assuming it's done by a script? It might be useful to publish the script on a gist as well, so that it can be reviewed alongside the actual diff.

@OmnipotentEntity
Copy link
Contributor Author

OmnipotentEntity commented Apr 1, 2020

Actually, it was done by hand with the aid of grep and vim macros. I didn't trust a script enough to do the proper thing without reviewing every point it changed anyway. And since I wanted at least one pair of human eyes on every change I figured I might as well just process it by hand. It took about 8 hours. (And I didn't think that there was a handy nix lexer I could just hook into easily, and I didn't fancy writing one.)

For the record, my grep call was: vim $(egrep '^\s*[A-Za-z0-9.]+\s*=\s*\w+://' * -Rl)

And my macro was roughly: /^\s*[\w.]\+\s*=\s*\w\+:\/\/<Enter>f=wi"<Esc>/\s*;\s*\(\#.*\)*<Enter>i"<Esc>

However that macro underwent a few refinements as I found failure modes. The following failure modes for the various versions of the macro are/were:

  1. fails when the semicolon is followed by an inline comment, it puts the ending quote on the next line instead. This was double checked for at the end, no are present to my knowledge. (Old only)
  2. It quotes a trailing space if there is a space between the URL and the semicolon. This is more difficult to search for, but an attempt was made to find any examples that I missed. (Old only)
  3. Does not find all unquoted URL strings. And I'm ok with that, though I have tried to track down other types as well. (All versions)

EDIT: @bhipple thanks for the question. It gave me an opportunity to think about failure mode 2 a little bit harder, and I was able to come up with a better way of searching for missed cases and found 6 strings that were improperly quoted. They have been fixed with the latest force push.

EDIT2: Additionally, I am a programmer. @a<leader>n My job is very important and serious @@<leader>n I have very high standards for myself and my own professionalism @@@@@@@@<leader>n And I contribute to open source projects during the quarantine 20@a<leader>n using only the best and most sophisticated techniques. <right arrow><right arrow><right arrow><right arrow><right arrow>i"<esc><mouse over and click>i"<esc>

@OmnipotentEntity
Copy link
Contributor Author

One conflict for today, renamed commit per @bhipple.

@OmnipotentEntity OmnipotentEntity changed the title Per RFC45, remove all unquoted URLs treewide: Per RFC45, remove all unquoted URLs Apr 4, 2020
@OmnipotentEntity
Copy link
Contributor Author

Fixed 23 packages this morning. That's more like it!

3 due to a removed patches
4 due to updating adjacent metadata
8 generated files updated (generated with bareURLs)
3 files removed entirely
3 due to marking package as broken or partially broken
1 due to the URL directly changing
1 due to whitespace differences (indenting fixed)

Additionally, 5 pkgs that were either just added or previously correct had bare URLs introduced, they were also fixed.

@OmnipotentEntity
Copy link
Contributor Author

OmnipotentEntity commented Apr 6, 2020

Three conflicts fixed today. Fixed one new bare URL in the new commits.

Question. Would you like me to restructure this commit into smaller batch commits (like 1000 apiece) and/or keep the daily merge fixes as separate commits, then rebase at the end? Will that help with the review process? @bhipple @grahamc @FRidh ?

@OmnipotentEntity
Copy link
Contributor Author

@GrahamcOfBorg eval

@grahamc
Copy link
Member

grahamc commented Apr 6, 2020

I'm not sure, to be honest. Such a large change is ... well, a very large change. Even broken up in to 15 PRs, it is quite significant. I'm not sure who will have the guts to press the merge button :) Something like this and #84122 I can only really imagine happening nicely over time, gradually converging to the "correct" format. I am just one person, though.

@OmnipotentEntity
Copy link
Contributor Author

OmnipotentEntity commented Apr 7, 2020

Fixed 6 conflicts:

5 changed homepage URLs (3 committed as bare URLs)
1 changed description

Fixed 1 newly pushed bare URLs.

However, my + and - numbers have diverged. So I'm attempting to figure out why.

EDIT: that's sorted.

@7c6f434c
Copy link
Member

7c6f434c commented Apr 7, 2020

Re: who could review this — @infinisil by comparing with the automated version, I guess.

Thanks a lot for detailed and frequent reports on merge conflicts. With just the automated tool, we were never sure how much merge churn there is

@OmnipotentEntity
Copy link
Contributor Author

Fixed 7 conflicts:

1 whitespace change in adjacent line
3 files entirely deleted
1 indentation adjustment
2 URLs changed

No new bare URLs were committed today.

@OmnipotentEntity
Copy link
Contributor Author

Fixed 8 conflicts: (Total commit size: 15410 -> 15387)

2 due to updating adjacent metadata
4 changed URLs
2 due to simplified nix expressions on adjacent lines (cleanup)

@OmnipotentEntity
Copy link
Contributor Author

Fixed 3 conflicts (Total commit size 15387 -> 15373)

2 files removed entirely
1 changed URL

@bhipple
Copy link
Contributor

bhipple commented Apr 10, 2020

If you send a PR with just the lib/ and nixos/ subdirs, I can review and merge that; then we're down to just the huge pkgs/ subdir (which the bot is slowly reducing in scope, at least). For pkgs, we have OfBorg telling us that it's 0 rebuilds, at least.

@Mic92
Copy link
Member

Mic92 commented Apr 10, 2020

I splitted the diff of aaf2b1e into smaller 10000 lines chunks:

Pick a random one and post if you have reviewed it. Best viewed with: nix-shell -p bat and bat -l diff ./splitaa.txt

@Mic92
Copy link
Member

Mic92 commented Apr 10, 2020

Cherry-picked into 84cf00f
I had to fix the merge conflict with doctl.

@Mic92 Mic92 closed this Apr 10, 2020
@OmnipotentEntity
Copy link
Contributor Author

Thanks!

@grahamc
Copy link
Member

grahamc commented Apr 10, 2020

🥳

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

9 participants