Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python 3.7 support in conda installations #652

Closed
sbourdeauducq opened this issue Jan 7, 2017 · 40 comments
Closed

Python 3.7 support in conda installations #652

sbourdeauducq opened this issue Jan 7, 2017 · 40 comments

Comments

@sbourdeauducq
Copy link
Member

  • pythonparser has a built-in list of supported versions and will bail out on 3.6.
  • there are hardcoded python3.5 interpreter references in many places (shebang, doc, ...).

Those seem to be the only major issue affecting ARTIQ on Python 3.6; after working around the above problems, the GUI, the master and the compiler essentially work without any particular trouble.

@sbourdeauducq
Copy link
Member Author

There are however problems with migen, and conda will need some attention (recompiling packages, switching qt to the anaconda package).

@sbourdeauducq
Copy link
Member Author

This is becoming an annoying issue now that Anaconda installs 3.6 by default. This breaks ARTIQ 2.x as well.

Sorry, something went wrong.

@jordens
Copy link
Member

jordens commented Jan 25, 2017

Using 3.5 with Anaconda only needs a documentation update.

Sorry, something went wrong.

@sbourdeauducq
Copy link
Member Author

Also check this: http://bugs.python.org/issue27500

Sorry, something went wrong.

@dhslichter
Copy link
Contributor

Does this bug appear in Python 3.6?

Sorry, something went wrong.

@sbourdeauducq
Copy link
Member Author

I think so. ARTIQ works around it with py 3.5, what I meant is check if the workaround is still needed.

@jordens
Copy link
Member

jordens commented Jan 30, 2017

We should not make a release with py36 mandatory before there is a reasonable ecosystem to do most of the day-to-day work in py36, i.e. wait for ContinuumIO/anaconda-issues#1315

@cjbe
Copy link
Contributor

cjbe commented May 28, 2017

There is now a Miniconda release for py3.6 (released 22nd Feburary)

@whitequark
Copy link
Contributor

there are hardcoded python3.5 interpreter references in many places (shebang, doc, ...).

Not anymore (at least I can't grep any).

pythonparser has a built-in list of supported versions and will bail out on 3.6.

There is now a stub Python 3.6 parser. (There are no 3.6-specific features like format strings, but also 3.5 features are already missing...)

@dhslichter
Copy link
Contributor

Are there still obstacles to the 3.6 upgrade? What benefits are there for the 3.6 upgrade other than playing nicer with the Anaconda default install?

@whitequark
Copy link
Contributor

You can run ARTIQ on Debian testing (like I do), which ships with Python 3.6.

@dhslichter
Copy link
Contributor

@whitequark this seems like a minor convenience for the developer, but not really anything for the end user. Are there features which are compelling/useful for the end user?

@whitequark
Copy link
Contributor

Other than being able to use Python 3.6 f"format strings", I don't think so.

@sbourdeauducq
Copy link
Member Author

sbourdeauducq commented Jan 27, 2018

I don't know for how long Python 3.5 will keep working on conda. 32-bit Windows is already broken (which is why we no longer have ARTIQ packages for that OS). Not having Python 3.6 support (or in general, not tracking what everybody else is doing with conda) may get the project stuck or requiring lots of conda workarounds.

whitequark added a commit that referenced this issue Jan 28, 2018
)

This removes the last hardcoded python3.5 reference.
@dhslichter
Copy link
Contributor

@sbourdeauducq ack. Is the conda ecosystem ready for ARTIQ to upgrade to 3.6? This was one of the previous issues raised about it.

@sbourdeauducq
Copy link
Member Author

Is there a package in particular you are worried about? Otherwise, the whole Anaconda distribution has been Python 3.6 by default for a while, and I don't expect problems other than the usual yak-shaving.

@jordens
Copy link
Member

jordens commented Jan 30, 2018

The situation has reversed from what it was. Now 3.6 is eminent and sticking with 3.5 will ultimately lead to what @sbourdeauducq has described.

@dhslichter
Copy link
Contributor

@jordens @sbourdeauducq all ack.

@sbourdeauducq
Copy link
Member Author

32-bit Windows is already broken (which is why we no longer have ARTIQ packages for that OS).

The Python 3.5 conda breakage continues. On 64-bit Windows, if you install the latest Anaconda, then create a Python 3.5 environment inside it and activate it, then the "conda install" command cannot be used from inside the environment due to SSL module issues.

@sbourdeauducq sbourdeauducq changed the title Python 3.7 support Python 3.7 support in conda installations Mar 15, 2019
@sbourdeauducq
Copy link
Member Author

sbourdeauducq commented Sep 23, 2019

Funded by NIST
(NB: ARTIQ already runs fine on 3.6 and 3.7, at least on Linux, if installed from source or via Nix)

@sbourdeauducq
Copy link
Member Author

Available for Linux now, and the anaconda.org m-labs channel is no longer required.

conda config --prepend channels https://conda.m-labs.hk/artiq-beta
conda config --append channels conda-forge
conda install artiq

Windows support will come later using similar techniques and the WFVM toolkit
https://git.m-labs.hk/M-Labs/nix-scripts/src/branch/wfvm

@sbourdeauducq
Copy link
Member Author

Works on Windows now.
I couldn't get Visual Studio to work with CMake, so I simply copied the old conda binary packages, which happen to still work with the Python 3.7 environment. This will cause an issue with ARM (Zynq) support but maybe we can just move to MSYS2 instead.

@sbourdeauducq
Copy link
Member Author

Still does not work properly on Windows, usually conda will downgrade Python to 3.5 and this seems to be due to the llvmlite-artiq package.

@sbourdeauducq
Copy link
Member Author

sbourdeauducq commented Jun 28, 2020

Seems OK with the beta packages now, except that they are affected by https://git.m-labs.hk/M-Labs/nix-scripts/issues/28
Installing the legacy binutils package might be a workaround, this one does not involve Python at all so hopefully that works.

@sbourdeauducq
Copy link
Member Author

Finally, and at great expense, this is working now.
People should really reconsider using that cretinous Windows "operating system" in the labs...

@charlesbaynham
Copy link
Contributor

Would that that were an option everywhere. Thanks for all the work! I'll try it out now

@sbourdeauducq
Copy link
Member Author

Might still be some issue with https://conda.m-labs.hk/artiq-beta/ not being updated (related to the Kasli timing failures)...

@hartytp
Copy link
Collaborator

hartytp commented Jul 24, 2020

@sbourdeauducq out of curiosity, what was the issue(s) here?

@sbourdeauducq
Copy link
Member Author

Issues about what?

@hartytp
Copy link
Collaborator

hartytp commented Jul 24, 2020

With the windows conda packages?

@sbourdeauducq
Copy link
Member Author

#1471
The last issue was https://git.m-labs.hk/M-Labs/nix-scripts/issues/28
As for why Windows is crappy, there are some reasons in https://itvision.altervista.org/why-windows-10-sucks.html

@charlesbaynham
Copy link
Contributor

Looking good!

C:\Users\myuser>conda create -n artiq artiq python=3.7

Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: D:\ProgramData\Miniconda3\envs\artiq

  added / updated specs:
    - artiq
    - python=3.7


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    [...]

The following NEW packages will be INSTALLED:

  artiq              artiq-beta/noarch::artiq-6.7251.91c93e1a.beta-0
  binutils-or1k-lin~ artiq-beta/win-64::binutils-or1k-linux-2.30-0
  blas               pkgs/main/win-64::blas-1.0-mkl
  ca-certificates    pkgs/main/win-64::ca-certificates-2020.6.24-0
  cached-property    pkgs/main/noarch::cached-property-1.5.1-py_1
  certifi            pkgs/main/win-64::certifi-2020.6.20-py37_0
  cffi               pkgs/main/win-64::cffi-1.14.0-py37h7a1dbc1_0
  curl               pkgs/main/win-64::curl-7.71.1-h2a8f88b_1
  h5py               pkgs/main/win-64::h5py-2.10.0-py37h5e291fa_0
  hdf5               pkgs/main/win-64::hdf5-1.10.4-h7ebc959_0
  icc_rt             pkgs/main/win-64::icc_rt-2019.0.0-h0cc432a_1
  icu                pkgs/main/win-64::icu-58.2-ha925a31_3
  intel-openmp       pkgs/main/win-64::intel-openmp-2020.1-216
  jpeg               pkgs/main/win-64::jpeg-9b-hb83a4c4_2
  krb5               pkgs/main/win-64::krb5-1.18.2-hc04afaa_0
  libcurl            pkgs/main/win-64::libcurl-7.71.1-h2a8f88b_1
  libgit2            conda-forge/win-64::libgit2-1.0.0-h79a9ecc_0
  libiconv           pkgs/main/win-64::libiconv-1.15-h1df5818_7
  libpng             pkgs/main/win-64::libpng-1.6.37-h2a8f88b_0
  libssh2            pkgs/main/win-64::libssh2-1.9.0-h7a1dbc1_1
  llvmlite-artiq     artiq-beta/win-64::llvmlite-artiq-0.23.0.dev-0
  mkl                pkgs/main/win-64::mkl-2020.1-216
  mkl-service        pkgs/main/win-64::mkl-service-2.3.0-py37hb782905_0
  mkl_fft            pkgs/main/win-64::mkl_fft-1.1.0-py37h45dec08_0
  mkl_random         pkgs/main/win-64::mkl_random-1.1.1-py37h47e9c7a_0
  numpy              pkgs/main/win-64::numpy-1.18.5-py37h6530119_0
  numpy-base         pkgs/main/win-64::numpy-base-1.18.5-py37hc3f5095_0
  openssl            pkgs/main/win-64::openssl-1.1.1g-he774522_0
  pip                pkgs/main/win-64::pip-20.1.1-py37_1
  prettytable        conda-forge/noarch::prettytable-0.7.2-py_3
  pycparser          pkgs/main/noarch::pycparser-2.20-py_2
  pygit2             conda-forge/win-64::pygit2-1.2.1-py37h4ab8f01_0
  pyqt               pkgs/main/win-64::pyqt-5.9.2-py37h6538335_2
  pyqtgraph          pkgs/main/noarch::pyqtgraph-0.11.0-py_0
  pyreadline         pkgs/main/win-64::pyreadline-2.1-py37_1
  python             pkgs/main/win-64::python-3.7.7-h81c818b_4
  python-dateutil    pkgs/main/noarch::python-dateutil-2.8.1-py_0
  python-levenshtein pkgs/main/win-64::python-levenshtein-0.12.0-py37he774522_1
01
  python_abi         conda-forge/win-64::python_abi-3.7-1_cp37m
  pythonparser       artiq-beta/noarch::pythonparser-1.3-0
  qt                 pkgs/main/win-64::qt-5.9.7-vc14h73c81de_0
  quamash            artiq-beta/noarch::quamash-0.6.1-0
  regex              pkgs/main/win-64::regex-2020.6.8-py37he774522_0
  scipy              pkgs/main/win-64::scipy-1.5.0-py37h9439919_0
  setuptools         pkgs/main/win-64::setuptools-49.2.0-py37_0
  sip                pkgs/main/win-64::sip-4.19.8-py37h6538335_0
  sipyco             artiq-beta/noarch::sipyco-1.1-0
  six                pkgs/main/noarch::six-1.15.0-py_0
  sqlite             pkgs/main/win-64::sqlite-3.32.3-h2a8f88b_0
  vc                 pkgs/main/win-64::vc-14.1-h0510ff6_4
  vs2015_runtime     pkgs/main/win-64::vs2015_runtime-14.16.27012-hf0eaf9b_3
  wheel              pkgs/main/win-64::wheel-0.34.2-py37_0
  wincertstore       pkgs/main/win-64::wincertstore-0.2-py37_0
  zlib               pkgs/main/win-64::zlib-1.2.11-h62dcd97_4


Proceed ([y]/n)?


Downloading and Extracting Packages
[...]
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate artiq
#
# To deactivate an active environment, use
#
#     $ conda deactivate


C:\Users\myuser>conda activate artiq

(artiq) C:\Users\myuser>python --version
Python 3.7.7

(artiq) C:\Users\myuser>artiq_client --version
ARTIQ v6.7251.91c93e1a.beta

@dtcallcock
Copy link

As for why Windows is crappy, there are some reasons in https://itvision.altervista.org/why-windows-10-sucks.html

Amusingly, this guy also has a similar list of reasons why Linux is crappy: https://itvision.altervista.org/why.linux.is.not.ready.for.the.desktop.current.html

macOS anyone?

@charlesbaynham
Copy link
Contributor

charlesbaynham commented Jul 25, 2020

Maybe I spoke too soon. I'm seeing these error messages for Experiments which use the core:

root:Terminating with exception (Exception: or1k-linux-ld invocation failed: or1k-linux-ld: final link failed: File truncated)
Traceback (most recent call last):
  File "d:\working directories\programming\python\artiq_master\artiq\master\worker_impl.py", line 306, in main
    exp_inst.run()
  File "d:\working directories\programming\python\artiq_master\artiq\language\core.py", line 54, in run_on_core
    return getattr(self, arg).run(run_on_core, ((self,) + k_args), k_kwargs)
  File "d:\working directories\programming\python\artiq_master\artiq\coredevice\core.py", line 129, in run
    self.compile(function, args, kwargs, set_result)
  File "d:\working directories\programming\python\artiq_master\artiq\coredevice\core.py", line 112, in compile
    library = target.compile_and_link([module])
  File "d:\working directories\programming\python\artiq_master\artiq\compiler\targets.py", line 198, in compile_and_link
    return self.link([self.assemble(self.compile(module)) for module in modules])
  File "d:\working directories\programming\python\artiq_master\artiq\compiler\targets.py", line 188, in link
    **{"obj{}".format(index): obj for index, obj in enumerate(objects)}) \
  File "d:\working directories\programming\python\artiq_master\artiq\compiler\targets.py", line 36, in __enter__
    format(cmdline[0], stderr))
Exception: or1k-linux-ld invocation failed: or1k-linux-ld: final link failed: File truncated

That was just running led.py from the tutorial.

@sbourdeauducq
Copy link
Member Author

sbourdeauducq commented Jul 25, 2020

Download this conda package and install it manually, replacing the currently installed one.
https://nixbld.m-labs.hk/build/78740

@charlesbaynham
Copy link
Contributor

Works! Now fully running in python 3.7, including kernels.

@charlesbaynham
Copy link
Contributor

I should also mention that the upgrade to 3.7 has cleared up a few annoying typing-related issues which used to throw warnings.

@jkellerPTB
Copy link

Thanks for making this work.

Sorry if I missed it somewhere, but I am unsure what the current best option is for installing the release version of ARTIQ with Python 3.7 on Windows.

Using conda, it seems that the only package required from the beta channel is llvmlite-artiq, due to the python<3.6 requirement of the version from the non-beta channel (even though both have the same version number). #1469 (comment) makes it sound like this is intentional.

Is it a good idea to use the below combination of packages, or are there risks with mixing the beta channel version of llvmlite-artiq with release packages? And if this is not a good idea, what would you suggest instead (given some unfortunate boundary conditions regarding the choice of OS)?

>conda create -n artiq artiq python=3.7

[...]

  artiq              artiq/noarch::artiq-5.7122.929b04da-0
  binutils-or1k-lin~ artiq/win-64::binutils-or1k-linux-2.27-h93a10e1_6
  blas               pkgs/main/win-64::blas-1.0-mkl
  ca-certificates    conda-forge/win-64::ca-certificates-2020.6.20-hecda079_0
  cached-property    conda-forge/noarch::cached-property-1.5.1-py_0
  certifi            conda-forge/win-64::certifi-2020.6.20-py37hc8dfbb8_0
  cffi               conda-forge/win-64::cffi-1.14.3-py37h26f1ce3_0
  curl               conda-forge/win-64::curl-7.71.1-h4b64cdc_8
  h5py               conda-forge/win-64::h5py-2.10.0-nompi_py37hde23a51_104
  hdf5               conda-forge/win-64::hdf5-1.10.6-nompi_he0bbb20_101
  icc_rt             pkgs/main/win-64::icc_rt-2019.0.0-h0cc432a_1
  icu                conda-forge/win-64::icu-67.1-h33f27b4_0
  intel-openmp       pkgs/main/win-64::intel-openmp-2019.4-245
  jpeg               conda-forge/win-64::jpeg-9d-he774522_0
  krb5               conda-forge/win-64::krb5-1.17.1-hc04afaa_3
  libblas            conda-forge/win-64::libblas-3.8.0-14_mkl
  libcblas           conda-forge/win-64::libcblas-3.8.0-14_mkl
  libclang           conda-forge/win-64::libclang-10.0.1-default_hf44288c_1
  libcurl            conda-forge/win-64::libcurl-7.71.1-h4b64cdc_8
  libgit2            conda-forge/win-64::libgit2-1.0.1-hcb5ae20_0
  libiconv           conda-forge/win-64::libiconv-1.16-he774522_0
  liblapack          conda-forge/win-64::liblapack-3.8.0-14_mkl
  libpng             conda-forge/win-64::libpng-1.6.37-ha81a0f5_2
  libssh2            conda-forge/win-64::libssh2-1.9.0-hb06d900_5
  llvmlite-artiq     artiq-beta/win-64::llvmlite-artiq-0.23.0.dev-0
  mkl                pkgs/main/win-64::mkl-2019.4-245
  mkl-service        conda-forge/win-64::mkl-service-2.3.0-py37hfa6e2cd_0
  numpy              conda-forge/win-64::numpy-1.19.1-py37h7cee19a_2
  openssl            conda-forge/win-64::openssl-1.1.1h-he774522_0
  pip                conda-forge/noarch::pip-20.2.3-py_0
  prettytable        conda-forge/noarch::prettytable-0.7.2-py_3
  pycparser          conda-forge/noarch::pycparser-2.20-pyh9f0ad1d_2
  pygit2             conda-forge/win-64::pygit2-1.3.0-py37h4ab8f01_0
  pyqt               conda-forge/win-64::pyqt-5.12.3-py37h6538335_1
  pyqtgraph          conda-forge/noarch::pyqtgraph-0.11.0-pyh9f0ad1d_1
  pyreadline         conda-forge/win-64::pyreadline-2.1-py37_1001
  python             conda-forge/win-64::python-3.7.8-h60c2a47_1_cpython
  python-dateutil    conda-forge/noarch::python-dateutil-2.8.1-py_0
  python-levenshtein conda-forge/win-64::python-levenshtein-0.12.0-py37h8055547_1002
  python_abi         conda-forge/win-64::python_abi-3.7-1_cp37m
  pythonparser       artiq/noarch::pythonparser-1.3-0
  qt                 conda-forge/win-64::qt-5.12.6-hb2cf2c5_0
  quamash            artiq/noarch::quamash-0.6.1-0
  regex              conda-forge/win-64::regex-2020.9.27-py37h4ab8f01_0
  scipy              pkgs/main/win-64::scipy-1.5.2-py37h9439919_0
  setuptools         conda-forge/win-64::setuptools-49.6.0-py37hc8dfbb8_1
  sipyco             artiq/noarch::sipyco-1.1-0
  six                conda-forge/noarch::six-1.15.0-pyh9f0ad1d_0
  sqlite             conda-forge/win-64::sqlite-3.33.0-he774522_0
  tk                 conda-forge/win-64::tk-8.6.10-he774522_0
  vc                 conda-forge/win-64::vc-14.1-h869be7e_1
  vs2015_runtime     conda-forge/win-64::vs2015_runtime-14.16.27012-h30e32a0_2
  wheel              conda-forge/noarch::wheel-0.35.1-pyh9f0ad1d_0
  wincertstore       conda-forge/win-64::wincertstore-0.2-py37_1003
  zlib               conda-forge/win-64::zlib-1.2.11-h62dcd97_1009

@sbourdeauducq
Copy link
Member Author

It's only available on ARTIQ-6; ARTIQ-5 uses the legacy packages.

@sbourdeauducq
Copy link
Member Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

9 participants