scons: Switch to Python 3 #75877

primeos · 2019-12-18T15:23:45Z

~~This should be merged after #75875 and~~ many build failures have to be resolved first.

Build logs

Results from nix-review: Gist (incomplete).

Old logs:

2019-12-18: Gist

Current status

Done (:heavy_check_mark:):

bombono: @symphorien (a33e63b)
fceux: @scubed2 (fceux: 2.2.3 -> 2020-01-29 #78943 ~~no meta.maintainers~~)
mypaint: @cillianderoiste @jtojnar (mypaint: 1.2.1 → 2.0.0-beta.0 #54677)
dxx-rebirth: @peterhoeg

WIP (:hourglass_flowing_sand:):

gringo: @hakuch (no longer required for opam -> probably safe to mark it as broken and remove it if no-one want's to maintain it)
nsis: @pedropombeiro (nsis: 3.04 -> 3.05 #79083, version 3.05 should support Python 3 but additional changes are required (debug))

Unresolved (:negative_squared_cross_mark:):

Marked as broken:

mesos: @cstrahan @offlinehacker
pingus: @7c6f434c
swiften: @twey

Unrelated:

mixxx
descent12

Proposed resolution

CC the maintainers of all packages that fail due to Python 3.x and ask them if they could have a look. I.e.:

Check if a new version is available that supports Python 3
Check if an upstream PR/issue already exits
- If there is a stale PR/issue it would probably make sense to add a comment
Optional: Check if a trivial patch can be written to support Python 3
Open an upstream issue (https://pythonclock.org/)

TODO: If it's not possible to support Python 3 the packages should override scons to use Python 2.7.

Things done

Notify maintainers

leahneukirchen · 2019-12-18T15:51:18Z

Not sure why I was pinged here?

primeos · 2019-12-18T15:54:33Z

@leahneukirchen Oh, sorry, I probably added you on accident due to the GH auto-completion without noticing it... :o (IMO that feature can be quite a bit annoying as the suggestions only disappear via escape or de-focusing)
(Note: I've edited the description and removed that part.)

jonringer · 2019-12-20T02:54:11Z

the other PR got merged, do you mind rebasing?

primeos · 2019-12-21T22:33:34Z

@jonringer did a rebase but unfortunately this is far from being ready to merge anyway :o

@FRidh @jonringer: What do you think of my proposal to resolve this? Currently the switch to Python 3 causes ~22 builds to fail (there could be more failures in transient dependencies later). IMO it would be best to notify the maintainers of these packages as they're the most familiar with it (e.g. also with testing if it still works correctly with Python 3 at runtime). Do you have any suggestions, changes, etc. regarding this proposal (before I ping the maintainers)?

jonringer · 2019-12-21T22:37:33Z

I'll have to look more into this later, will be pretty busy the next week

FRidh · 2019-12-29T10:21:32Z

@primeos I think that's a good approach. Otherwise, maybe adding a Python2 version of scons as well. E.g. in its passthru so it won't show up in the top-level package set.

primeos · 2020-01-27T21:22:19Z

Dear maintainers,
you're cc'd in this issue because at least one of the packages you maintain (s.
list below) will fail to build after switching SCons to Python 3.

If you're still actively using/maintaining your package it would be nice if you
could try to make the build work e.g. by following these steps:

Check if a new version is available that supports Python 3
Check if an upstream patch/PR/issue already exits
- If there is a stale PR/issue it would most likely make sense to add a
  comment (triage)
Optional: Check if a trivial patch can be written to support Python 3 (e.g. if
it only fails due to a few "print" statements)
Open an upstream issue (https://pythonclock.org/)
Report the results back to this PR
- Feel free to reply very briefly, but a link to the PR/commit would be nice
- If e.g. the new version / patch doesn't work with Python 2 anymore you can
  either provide a link to the commit and I'll cherry-pick it into this PR or
  we can already merge a version of SCons that uses Python 3 into Nixpks.

Please also reply here if you're not maintaining your package anymore or if
there isn't an easy solution to make it work with Python 3.

Thanks in advance ❤️ :)

List of packages that don't build with Python 3

Link to the build log

btanks: @viric (no meta.maintainers)
bombono: @symphorien
dxx-rebirth: @peterhoeg
fceux: @scubed2 (no meta)
globulation2 (glob2): @7c6f434c
gpsd: @bjornfor @rasendubi
gringo: @hakuch
jackmix: @kampfschlaefer
lprof: @MarcWeber
mongodb: @bluescreen303 @offlinehacker @cstrahan
mypaint: @cillianderoiste @jtojnar
nsis: @pedropombeiro (old username in maintainer-list.nix: @pombeirp)
pingus: @7c6f434c
rhvoice: @berce
serf: @orivej @7c6f434c
swift-im / swiften: @orivej @Twey
tdm: @cizra
vdrift: @viric
xboxdrv: @Fuuzetsu (no meta)
ffado: @cillianderoiste (AttributeError: module 'posixpath' has no attribute 'walk')
toluapp: @vrthra (AttributeError: '_io.TextIOWrapper' object has no attribute 'xreadlines')

jtojnar · 2020-01-28T01:15:59Z

MyPaint 2 is scheduled to release soon #54677

pedropombeiro · 2020-01-28T23:07:28Z

nsis seems to still be using Python 2: https://github.com/kichik/nsis/blob/master/INSTALL#L8

If this is a blocker, I suggest removing the NSIS expression from all-packages.

primeos · 2020-01-29T00:02:59Z

@pedropombeiro I've had a look and it seems like a patch already exists: kichik/nsis@9ee3275

By looking at the dates this patch should already be in version 3.05: https://sourceforge.net/projects/nsis/files/NSIS%203/

Could you try updating to version 3.05?

scubed2 · 2020-01-31T06:00:02Z

Sent #78943 for fceux.

pedropombeiro · 2020-01-31T06:12:18Z

Could you try updating to version 3.05?

That's what I tried with but I got some weird error.

hakuch · 2020-02-01T14:16:44Z

Hello, and thank you for the clear message and instructions. I'm no longer actively maintaining the gringo package: as of version 2, opam no longer requires an external solver and so the dependency on gringo is not necessary.

primeos · 2020-02-01T22:00:33Z

@pedropombeiro what does "weird error" mean? It didn't build or there where errors at run-time? And what would you recommend to do regarding this PR? (fix the build, mark it as broken, use Python 2 for now, etc.)

@hakuch ok, thanks for the information :)

I'll collect all information in the PR description (current status). Thanks @everyone for the updates.

pedropombeiro · 2020-02-02T09:08:31Z

@primeos I've fixed this permissions error:

Checking for C library z... (cached) yes
Checking for C++ library cppunit... (cached) no
scons: done reading SConscript files.
scons: Building targets ...
scons: *** [/nix/store/lkcib3zdqd1q4n250ny5wl4vpz5z4zl1-nsis-3.05/share/nsis/Stubs/uninst] /nix/store/lkcib3zdqd1q4n250ny5wl4vpz5z4zl1-nsis-3.05/share/nsis/Stubs/uninst: Permission denied
scons: building terminated because of errors.
builder for '/nix/store/563ygbxywfmh3yxpssd2bl2dmzbrk9mp-nsis-3.05.drv' failed with exit code 2
error: build of '/nix/store/563ygbxywfmh3yxpssd2bl2dmzbrk9mp-nsis-3.05.drv' failed
nix-build default.nix -A nsis  21.48s user 2.16s system 99% cpu 23.652 total

I've created a PR to upgrade to 3.05 and it works in master, however on your branch SCons is now complaining that it doesn't know the install-compiler even though that's still part of the documentation:

Checking for C++ library cppunit... (cached) no
scons: done reading SConscript files.
scons: Building targets ...
scons: *** Do not know how to make File target `install-compiler' (/build/nsis-3.05-src/install-compiler).  Stop.
scons: building terminated because of errors.

I'd say we either use Python 2 for this package for now or mark it as broken.

symphorien · 2020-02-02T15:41:28Z

A commit to fix bombono: 431094f

jonringer · 2020-02-02T17:50:01Z

diff LGTM, let me know when you feel comfortable enough for me to test

Reasons: Python 2.7 will EOL very soon [0]: DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support SCons 4.0.0 will drop Python 2.7 Support [1]: https://raw.githubusercontent.com/SConsProject/scons/rel_3.1.2/src/CHANGES.txt [0]: From the SCons build output previous to this commit (i.e. with Python 2.7). [1]: https://raw.githubusercontent.com/SConsProject/scons/rel_3.1.2/src/CHANGES.txt

Not all packages build with Python 3, see NixOS#75877. The goal is to get rid of Python 2 but this approach ensures a smoother transition.

primeos · 2020-03-27T14:35:23Z

Unfortunately this didn't work as well as I hoped.

My new approach is to switch SCons to Python 3 but continue to build all failing packages with Python 2 for now. Then we can use subsequent PRs to either mark deprecated/unmaintained packages as broken or fix the build with Python 3 if the package is still relevant.

I used the suggestion of @FRidh:

Otherwise, maybe adding a Python2 version of scons as well. E.g. in its passthru so it won't show up in the top-level package set.

Though my implementation is a bit hacky :o

@jonringer @FRidh: Is the current approach still ok? If so I could run nixpkgs-review again to verify this doesn't cause any additional build regressions and run a few manual tests and then it should basically be ready to be merged.

@jonringer if your testing offer still stands that'd be welcome :)

jonringer · 2020-03-27T17:48:47Z

I don't have enough context to know what ramifications moving to python3 will do. However, I think it's a good idea to move away from python2 anyway.

jonringer · 2020-03-27T17:49:27Z

I don't have enough context to know what ramifications moving to python3 will do. However, I think it's a good idea to move away from python2 anyway.

I'm okay with this

Those packages are broken for >4 months, which is why it seems best to mark them as broken for now. I noticed these while testing #75877.

primeos mentioned this pull request Dec 18, 2019

scons: 3.1.1 -> 3.1.2 #75875

Merged

10 tasks

ofborg bot added 11.by: package-maintainer 10.rebuild-darwin: 11-100 10.rebuild-linux: 101-500 labels Dec 18, 2019

jtojnar added this to WIP in Python 2 deprecation Dec 21, 2019

primeos force-pushed the scons-python3 branch from 957d9d0 to c527f5d Compare December 21, 2019 22:12

jtojnar mentioned this pull request Dec 25, 2019

Another batch of Python 2 clean-ups #76488

Merged

17 tasks

scubed2 mentioned this pull request Jan 31, 2020

fceux: 2.2.3 -> 2020-01-29 #78943

Merged

10 tasks

ofborg bot requested a review from symphorien February 2, 2020 17:55

peterhoeg mentioned this pull request Feb 3, 2020

dxx-rebirth: 0.59.100 -> 0.59.20200202 #79158

Merged

10 tasks

primeos and others added 3 commits March 27, 2020 15:20

bombono: fix build with scons/python3

013b3b2

scons: Add passthru.py2 for backward compatibility

0cd9ac8

Not all packages build with Python 3, see NixOS#75877. The goal is to get rid of Python 2 but this approach ensures a smoother transition.

primeos force-pushed the scons-python3 branch from a33e63b to 0cd9ac8 Compare March 27, 2020 14:23

primeos marked this pull request as ready for review March 27, 2020 14:35

FRidh approved these changes Mar 27, 2020

View reviewed changes

jonringer merged commit 0950324 into NixOS:staging Mar 27, 2020

Python 2 deprecation automation moved this from WIP to Done Mar 27, 2020

primeos added a commit that referenced this pull request Mar 29, 2020

mesos,swiften,pingus: Mark as broken

0dd5ccc

Those packages are broken for >4 months, which is why it seems best to mark them as broken for now. I noticed these while testing #75877.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

scons: Switch to Python 3 #75877

scons: Switch to Python 3 #75877

primeos commented Dec 18, 2019 •

edited

leahneukirchen commented Dec 18, 2019

primeos commented Dec 18, 2019 •

edited

jonringer commented Dec 20, 2019

primeos commented Dec 21, 2019

jonringer commented Dec 21, 2019

FRidh commented Dec 29, 2019

primeos commented Jan 27, 2020 •

edited

jtojnar commented Jan 28, 2020 •

edited

pedropombeiro commented Jan 28, 2020

primeos commented Jan 29, 2020

scubed2 commented Jan 31, 2020

pedropombeiro commented Jan 31, 2020

hakuch commented Feb 1, 2020

primeos commented Feb 1, 2020

pedropombeiro commented Feb 2, 2020 •

edited

symphorien commented Feb 2, 2020

jonringer commented Feb 2, 2020

primeos commented Mar 27, 2020

jonringer commented Mar 27, 2020

jonringer commented Mar 27, 2020

scons: Switch to Python 3 #75877

scons: Switch to Python 3 #75877

Conversation

primeos commented Dec 18, 2019 • edited

Build logs

Current status

Proposed resolution

Things done

Notify maintainers

leahneukirchen commented Dec 18, 2019

primeos commented Dec 18, 2019 • edited

jonringer commented Dec 20, 2019

primeos commented Dec 21, 2019

jonringer commented Dec 21, 2019

FRidh commented Dec 29, 2019

primeos commented Jan 27, 2020 • edited

List of packages that don't build with Python 3

jtojnar commented Jan 28, 2020 • edited

pedropombeiro commented Jan 28, 2020

primeos commented Jan 29, 2020

scubed2 commented Jan 31, 2020

pedropombeiro commented Jan 31, 2020

hakuch commented Feb 1, 2020

primeos commented Feb 1, 2020

pedropombeiro commented Feb 2, 2020 • edited

symphorien commented Feb 2, 2020

jonringer commented Feb 2, 2020

primeos commented Mar 27, 2020

jonringer commented Mar 27, 2020

jonringer commented Mar 27, 2020

primeos commented Dec 18, 2019 •

edited

primeos commented Dec 18, 2019 •

edited

primeos commented Jan 27, 2020 •

edited

jtojnar commented Jan 28, 2020 •

edited

pedropombeiro commented Feb 2, 2020 •

edited