Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pythonPackages.umap-learn: 0.3.10 -> 0.4.5 #100487

Merged
merged 1 commit into from Oct 16, 2020
Merged

pythonPackages.umap-learn: 0.3.10 -> 0.4.5 #100487

merged 1 commit into from Oct 16, 2020

Conversation

uvNikita
Copy link
Contributor

@uvNikita uvNikita commented Oct 14, 2020 •
edited

Motivation for this change

Updating umap-learn package and fixing build. Unfortunately, I had to disable three tests since they were failing.

ZHF: #97479
@NixOS/nixos-release-managers

Fixing this build: https://hydra.nixos.org/build/127633974

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@FRidh
Copy link
Member

FRidh commented Oct 14, 2020

By using the hook for testing you can disable tests by simply adding them to disabledTests = [ ]; Did you notify upstream of the failures?

@uvNikita
Copy link
Contributor Author

By using the hook for testing you can disable tests by simply adding them to disabledTests = [ ];

That's convenient!

@ofborg ofborg bot requested a review from costrouc October 14, 2020 11:08
@uvNikita
Copy link
Contributor Author

uvNikita commented Oct 14, 2020 •
edited

Did you notify upstream of the failures?

I might be wrong here, but it seems like their own CI system is mostly red: https://travis-ci.org/github/lmcinnes/umap/builds

Btw, the plot tests fail mainly because of missing dependencies needed for the plot submodule only (pandas, matplotlib, datashader, holoviews, bokeh, colorcet). But I wasn't sure if I should just add all of them to the package since they are optional? That said, even after adding all dependencies plot tests still fail with some other error.

Let me know how you want me to proceed with this.

@uvNikita
Copy link
Contributor Author

For the reference, here are failures from two plot tests when all dependencies are added:

__________________________ test_umap_plot_testability __________________________

    def test_umap_plot_testability():
        try:
>           from umap import plot

umap/tests/test_plot.py:28:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
umap/plot.py:7: in <module>
    import datashader as ds
/nix/store/fbc35lajrwk092scbknm79ljvgga493m-python3.8-datashader-0.11.1/lib/python3.8/site-packages/datashader/__init__.py:8: in <module>
    from .core import Canvas                                 # noqa (API import)
/nix/store/fbc35lajrwk092scbknm79ljvgga493m-python3.8-datashader-0.11.1/lib/python3.8/site-packages/datashader/core.py:19: in <module>
    from . import reductions as rd
/nix/store/fbc35lajrwk092scbknm79ljvgga493m-python3.8-datashader-0.11.1/lib/python3.8/site-packages/datashader/reductions.py:11: in <module>
    from datashader.transfer_functions._cuda_utils import (cuda_atomic_nanmin,
/nix/store/fbc35lajrwk092scbknm79ljvgga493m-python3.8-datashader-0.11.1/lib/python3.8/site-packages/datashader/transfer_functions/__init__.py:18: in <module>
    from datashader.composite import composite_op_lookup, over
/nix/store/fbc35lajrwk092scbknm79ljvgga493m-python3.8-datashader-0.11.1/lib/python3.8/site-packages/datashader/composite.py:17: in <module>
    def extract_scaled(x):
/nix/store/j507gq4jndxp1gbf489k1rwqzdy9qx2z-python3.8-numba-0.51.1/lib/python3.8/site-packages/numba/core/decorators.py:211: in wrapper
    disp.enable_caching()
/nix/store/j507gq4jndxp1gbf489k1rwqzdy9qx2z-python3.8-numba-0.51.1/lib/python3.8/site-packages/numba/core/dispatcher.py:743: in enable_caching
    self._cache = FunctionCache(self.py_func)
/nix/store/j507gq4jndxp1gbf489k1rwqzdy9qx2z-python3.8-numba-0.51.1/lib/python3.8/site-packages/numba/core/caching.py:620: in __init__
    self._impl = self._impl_class(py_func)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <numba.core.caching.CompileResultCacheImpl object at 0x7fff9dde1b80>
py_func = <function extract_scaled at 0x7fff9ddecf70>

    def __init__(self, py_func):
        self._lineno = py_func.__code__.co_firstlineno
        # Get qualname
        try:
            qualname = py_func.__qualname__
        except AttributeError:
            qualname = py_func.__name__
        # Find a locator
        source_path = inspect.getfile(py_func)
        for cls in self._locator_classes:
            locator = cls.from_function(py_func, source_path)
            if locator is not None:
                break
        else:
>           raise RuntimeError("cannot cache function %r: no locator available "
                               "for file %r" % (qualname, source_path))
E           RuntimeError: cannot cache function 'extract_scaled': no locator available for file '/nix/store/fbc35lajrwk092scbknm79ljvgga493m-python3.8-datashader-0.11.1/lib/python3.8/site-packages/datashader/composite.py'

/nix/store/j507gq4jndxp1gbf489k1rwqzdy9qx2z-python3.8-numba-0.51.1/lib/python3.8/site-packages/numba/core/caching.py:355: RuntimeError
____________________________ test_plot_runs_at_all _____________________________

mapper = UMAP(n_epochs=100)
iris = {'data': array([[5.1, 3.5, 1.4, 0.2],
       [4.9, 3. , 1.4, 0.2],
       [4.7, 3.2, 1.3, 0.2],
       [4.6, 3.1, 1.5,...r66865jka1d5jcvczmlk1k4hknb4-python3.8-scikit-learn-0.23.2/lib/python3.8/site-packages/sklearn/datasets/data/iris.csv'}

    def test_plot_runs_at_all(mapper, iris):
>       from umap import plot as umap_plot

umap/tests/test_plot.py:39:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
umap/plot.py:7: in <module>
    import datashader as ds
/nix/store/fbc35lajrwk092scbknm79ljvgga493m-python3.8-datashader-0.11.1/lib/python3.8/site-packages/datashader/__init__.py:8: in <module>
    from .core import Canvas                                 # noqa (API import)
/nix/store/fbc35lajrwk092scbknm79ljvgga493m-python3.8-datashader-0.11.1/lib/python3.8/site-packages/datashader/core.py:19: in <module>
    from . import reductions as rd
/nix/store/fbc35lajrwk092scbknm79ljvgga493m-python3.8-datashader-0.11.1/lib/python3.8/site-packages/datashader/reductions.py:11: in <module>
    from datashader.transfer_functions._cuda_utils import (cuda_atomic_nanmin,
/nix/store/fbc35lajrwk092scbknm79ljvgga493m-python3.8-datashader-0.11.1/lib/python3.8/site-packages/datashader/transfer_functions/__init__.py:18: in <module>
    from datashader.composite import composite_op_lookup, over
/nix/store/fbc35lajrwk092scbknm79ljvgga493m-python3.8-datashader-0.11.1/lib/python3.8/site-packages/datashader/composite.py:17: in <module>
    def extract_scaled(x):
/nix/store/j507gq4jndxp1gbf489k1rwqzdy9qx2z-python3.8-numba-0.51.1/lib/python3.8/site-packages/numba/core/decorators.py:211: in wrapper
    disp.enable_caching()
/nix/store/j507gq4jndxp1gbf489k1rwqzdy9qx2z-python3.8-numba-0.51.1/lib/python3.8/site-packages/numba/core/dispatcher.py:743: in enable_caching
    self._cache = FunctionCache(self.py_func)
/nix/store/j507gq4jndxp1gbf489k1rwqzdy9qx2z-python3.8-numba-0.51.1/lib/python3.8/site-packages/numba/core/caching.py:620: in __init__
    self._impl = self._impl_class(py_func)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <numba.core.caching.CompileResultCacheImpl object at 0x7fff8d3be730>
py_func = <function extract_scaled at 0x7fff8d827430>

    def __init__(self, py_func):
        self._lineno = py_func.__code__.co_firstlineno
        # Get qualname
        try:
            qualname = py_func.__qualname__
        except AttributeError:
            qualname = py_func.__name__
        # Find a locator
        source_path = inspect.getfile(py_func)
        for cls in self._locator_classes:
            locator = cls.from_function(py_func, source_path)
            if locator is not None:
                break
        else:
>           raise RuntimeError("cannot cache function %r: no locator available "
                               "for file %r" % (qualname, source_path))
E           RuntimeError: cannot cache function 'extract_scaled': no locator available for file '/nix/store/fbc35lajrwk092scbknm79ljvgga493m-python3.8-datashader-0.11.1/lib/python3.8/site-packages/datashader/composite.py'

/nix/store/j507gq4jndxp1gbf489k1rwqzdy9qx2z-python3.8-numba-0.51.1/lib/python3.8/site-packages/numba/core/caching.py:355: RuntimeError

@uvNikita
Copy link
Contributor Author

The third test seems to be flaky since it passes sometimes. Here is the error:

____________________________ test_sparse_hellinger _____________________________

sparse_spatial_data = <12x20 sparse matrix of type '<class 'numpy.float64'>'
        with 63 stored elements in Compressed Sparse Row format>

    def test_sparse_hellinger(sparse_spatial_data):
        dist_matrix = dist.pairwise_special_metric(
            np.abs(sparse_spatial_data[:-2].toarray())
        )
        test_matrix = np.array(
            [
                [
                    spdist.sparse_hellinger(
                        np.abs(sparse_spatial_data[i]).indices,
                        np.abs(sparse_spatial_data[i]).data,
                        np.abs(sparse_spatial_data[j]).indices,
                        np.abs(sparse_spatial_data[j]).data,
                    )
                    for j in range(sparse_spatial_data.shape[0] - 2)
                ]
                for i in range(sparse_spatial_data.shape[0] - 2)
            ]
        )

>       assert_array_almost_equal(
            test_matrix,
            dist_matrix,
            err_msg="Sparse distances don't match " "for metric hellinger",
            decimal=4,
        )
E       AssertionError: 
E       Arrays are not almost equal to 4 decimals
E       Sparse distances don't match for metric hellinger
E       Mismatched elements: 1 / 100 (1%)
E       Max absolute difference: 0.00015389
E       Max relative difference: 1.94364094e-08
E        x: array([[4.4424e-05, 1.0000e+00, 8.4342e-01, 9.5723e-01, 8.5692e-01,
E               8.9906e-01, 1.0000e+00, 5.1850e-01, 8.3013e-01, 8.2275e-01],
E              [1.0000e+00, 1.0601e-04, 8.8778e-01, 7.1078e-01, 8.8114e-01,...
E        y: array([[0.    , 1.    , 0.8434, 0.9572, 0.8569, 0.8991, 1.    , 0.5185,
E               0.8301, 0.8228],
E              [1.    , 0.    , 0.8878, 0.7108, 0.8811, 0.8246, 0.6627, 0.9032,...

umap/tests/test_umap_metrics.py:436: AssertionError

@FRidh
Copy link
Member

FRidh commented Oct 14, 2020

Keep it simple, if its failing upstream we can't do much about it. Also, we do not need to test everything. Adding a comment explaining why a test is disabled is very useful though.

@uvNikita
Copy link
Contributor Author

Sounds good, added a couple of comments which explains why those tests are disabled.

jonringer
jonringer approved these changes Oct 16, 2020
View reviewed changes
Copy link
Contributor

@jonringer jonringer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Result of nixpkgs-review pr 100487 1

4 packages built:
  • python37Packages.scikit-tda
  • python37Packages.umap-learn
  • python38Packages.scikit-tda
  • python38Packages.umap-learn

@jonringer jonringer merged commit 105d500 into NixOS:master Oct 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jonringer jonringer jonringer approved these changes

@FRidh FRidh Awaiting requested review from FRidh

@costrouc costrouc Awaiting requested review from costrouc

Assignees
No one assigned
Labels
6.topic: python 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@uvNikita @FRidh @jonringer