Eliminate dependency on bitarray #179

mattvenn · 2020-01-31T14:01:42Z

Appears to be a python3.7 package installation problem?
ubuntu 18.04 LTS
python3.7
glasgow repo cloned today

After following the install instructions I try to run glasgow and get the following:

480s:2000: cd ~/.local/bin/
T480s:2001: ./glasgow 
Traceback (most recent call last):
  File "./glasgow", line 11, in <module>
    load_entry_point('glasgow', 'console_scripts', 'glasgow')()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 480, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2693, in load_entry_point
    return ep.load()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2324, in load
    return self.resolve()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2330, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/matt/work/fpga/glasgow/software/glasgow/cli.py", line 28, in <module>
    from .applet import all
  File "/home/matt/work/fpga/glasgow/software/glasgow/applet/all.py", line 30, in <module>
    from .program.xc6s import ProgramXC6SApplet
  File "/home/matt/work/fpga/glasgow/software/glasgow/applet/program/xc6s/__init__.py", line 12, in <module>
    from bitarray import bitarray
  File "/usr/lib/python3/dist-packages/bitarray/__init__.py", line 11, in <module>
    from bitarray._bitarray import _bitarray, bitdiff, bits2bytes, _sysinfo
ModuleNotFoundError: No module named 'bitarray._bitarray'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in <module>
    import apport.fileutils
  File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in <module>
    from apport.packaging_impl import impl as packaging
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 24, in <module>
    import apt
  File "/usr/lib/python3/dist-packages/apt/__init__.py", line 23, in <module>
    import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'

Original exception was:
Traceback (most recent call last):
  File "./glasgow", line 11, in <module>
    load_entry_point('glasgow', 'console_scripts', 'glasgow')()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 480, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2693, in load_entry_point
    return ep.load()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2324, in load
    return self.resolve()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2330, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/matt/work/fpga/glasgow/software/glasgow/cli.py", line 28, in <module>
    from .applet import all
  File "/home/matt/work/fpga/glasgow/software/glasgow/applet/all.py", line 30, in <module>
    from .program.xc6s import ProgramXC6SApplet
  File "/home/matt/work/fpga/glasgow/software/glasgow/applet/program/xc6s/__init__.py", line 12, in <module>
    from bitarray import bitarray
  File "/usr/lib/python3/dist-packages/bitarray/__init__.py", line 11, in <module>
    from bitarray._bitarray import _bitarray, bitdiff, bits2bytes, _sysinfo
ModuleNotFoundError: No module named 'bitarray._bitarray'

Module is indeed not available:

T480s:2002: python3.7
Python 3.7.5 (default, Nov  7 2019, 10:50:52) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import bitarray
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/bitarray/__init__.py", line 11, in <module>
    from bitarray._bitarray import _bitarray, bitdiff, bits2bytes, _sysinfo
ModuleNotFoundError: No module named 'bitarray._bitarray'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in <module>
    import apport.fileutils
  File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in <module>
    from apport.packaging_impl import impl as packaging
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 24, in <module>
    import apt
  File "/usr/lib/python3/dist-packages/apt/__init__.py", line 23, in <module>
    import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'


Original exception was:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/bitarray/__init__.py", line 11, in <module>
    from bitarray._bitarray import _bitarray, bitdiff, bits2bytes, _sysinfo
ModuleNotFoundError: No module named 'bitarray._bitarray'
>>>

Try to install:

T480s:2003: pip3.7 bitarray
ERROR: unknown command "bitarray"
T480s:2004: pip3.7 install bitarray
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: bitarray in /usr/lib/python3/dist-packages (0.8.1)

The text was updated successfully, but these errors were encountered:

electroniceel · 2020-01-31T20:09:49Z

Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python3/dist-packages/bitarray/init.py", line 11, in
from bitarray._bitarray import _bitarray, bitdiff, bits2bytes, _sysinfo
ModuleNotFoundError: No module named 'bitarray._bitarray'

This means there is is some part of the bitarray python module on your system, but it seems to be broken or incomplete. Take a look in the /usr/lib/python3/dist-packages/bitarray/ directory, there should be a _bitarray.cpython.<something>.so file.

Since it is below /usr/lib I guess this was installed through your ubuntu package management. Try to find out which package these files belong to and reinstall, upgrade or downgrade it.

It also might be a multilib issue, on a x86_64 machine I'd expect the .so below /usr/lib64, not /usr/lib. So check if there are 32bit and 64bit versions of this package available and just install the one matching your system bit size.

whitequark · 2020-01-31T20:26:03Z

We really should just get rid of bitarray at last, it has caused nothing but problems, and the implementation is unsafe and likely unsound.

jeremyherbert · 2020-03-20T06:58:58Z

This problem happened to me. It appears to be because the apt-get command in the readme installs packages for python3.6, not python3.7 on Ubuntu 18.04. I'm not sure there are actually any python3.7 apt packages for bitarray or apt.

ie python 3.7 tries to import apt_pkg.cpython-37m-x86_64-linux-gnu.so but the only file that exists is apt_pkg.cpython-36m-x86_64-linux-gnu.so. Same issue for _bitarray.cpython-36m-x86_64-linux-gnu.so

Symlinking does fix the problem as a quick hack to make sure this is indeed the real issue.

whitequark · 2020-03-21T15:14:35Z

Thanks for the investigation. My conclusion is the same: we should get rid of bitarray. In the meantime we may need to fix the installation instructions but I'm not entirely sure how; using 3.6 is out of question, and in fact we should really migrate to 3.8.

jeremyherbert · 2020-03-22T02:24:26Z

I think given that Ubuntu 20.04 (new LTS version) is coming out in about a month and will have >3.8 (3.8.2 I think), perhaps it's best to just state that this is not working for 18.10 in the readme and link to this issue for an explanation? Ubuntu 19.04 uses python 3.7.2, so you could also say that you need at least that version.

I also needed to do some update-alternatives trickery to get it to use python3.7 instead of python3.6 which is the default when running python3, which is recommended against I think as it could be system-breaking.

Fixes #179.

whitequark added the software Component: software label Jan 31, 2020

whitequark added documentation Component: documentation and removed software Component: software labels Mar 22, 2020

whitequark added software Component: software and removed documentation Component: documentation labels Nov 4, 2020

whitequark changed the title ~~python modules not found after install~~ Eliminate dependency on bitarray Nov 4, 2020

This was referenced Nov 10, 2020

Windows support checklist #231

Closed

Long-term maintainability goals #234

Open

whitequark mentioned this issue Dec 29, 2020

Add support for Out-of-Tree applets #213

Closed

mndza mentioned this issue Jan 24, 2021

Add bitarray implementation to eliminate external dependency #265

Closed

mwkmwkmwk added a commit that referenced this issue Jul 31, 2023

software: migrate away from bitarray dependency

1d437e1

Fixes #179.

mwkmwkmwk mentioned this issue Jul 31, 2023

software: migrate away from bitarray dependency. #376

Merged

mwkmwkmwk pushed a commit that referenced this issue Oct 22, 2023

software: migrate away from bitarray dependency

1afcd05

Fixes #179.

mwkmwkmwk pushed a commit that referenced this issue Oct 22, 2023

software: migrate away from bitarray dependency

ea8b241

Fixes #179.

mwkmwkmwk pushed a commit that referenced this issue Oct 22, 2023

software: migrate away from bitarray dependency

f835de2

Fixes #179.

github-merge-queue bot pushed a commit that referenced this issue Oct 22, 2023

software: migrate away from bitarray dependency

2c18b56

Fixes #179.

whitequark closed this as completed in #376 Oct 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Eliminate dependency on bitarray #179

Eliminate dependency on bitarray #179

mattvenn commented Jan 31, 2020

electroniceel commented Jan 31, 2020 •

edited

whitequark commented Jan 31, 2020

jeremyherbert commented Mar 20, 2020

whitequark commented Mar 21, 2020

jeremyherbert commented Mar 22, 2020

Eliminate dependency on bitarray #179

Eliminate dependency on bitarray #179

Comments

mattvenn commented Jan 31, 2020

electroniceel commented Jan 31, 2020 • edited

whitequark commented Jan 31, 2020

jeremyherbert commented Mar 20, 2020

whitequark commented Mar 21, 2020

jeremyherbert commented Mar 22, 2020

electroniceel commented Jan 31, 2020 •

edited