Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python fbprophet #91701

Closed

Conversation

cartesian-theatrics
Copy link

@cartesian-theatrics cartesian-theatrics commented Jun 28, 2020

Hello,

I'm trying to add a python library called fbprophet. I'm just starting with Nix and ran into an issue building the stanpy dependency:

======================================================================
ERROR: test_cross_validation (fbprophet.tests.test_diagnostics.TestDiagnostics)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/fbprophet-0.6/fbprophet/tests/test_diagnostics.py", line 35, in test_cross_validation
    m = Prophet()
  File "/build/fbprophet-0.6/fbprophet/forecaster.py", line 141, in __init__
    self._load_stan_backend(stan_backend)
  File "/build/fbprophet-0.6/fbprophet/forecaster.py", line 154, in _load_stan_backend
    logger.debug("Loaded stan backend: %s", self.stan_backend.get_type())
AttributeError: 'Prophet' object has no attribute 'stan_backend'

There's some discussion of the problem here:
facebook/prophet#1462

Any help would be much appreciated!

Motivation for this change

Add fbprophet python library.

Things done

This is not quite building yet.

  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

Copy link
Contributor

@jonringer jonringer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please apply the changes across all the packages.

and please have a commit per package addition.

Comment on lines +2842 to +2850
fbprophet = callPackage ../development/python-modules/fbprophet { };

pystan = callPackage ../development/python-modules/pystan { };

cmdstanpy = callPackage ../development/python-modules/cmdstanpy { };

lunar-calendar = callPackage ../development/python-modules/LunarCalendar { };

korean-lunar-calendar = callPackage ../development/python-modules/korean-lunar-calendar { };
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please attempt to sort these. Yes, python-packages.nix is not in amazing state, but not reason to continue adding to the mess

pkgs/development/python-modules/pystan/default.nix Outdated Show resolved Hide resolved
pkgs/development/python-modules/pystan/default.nix Outdated Show resolved Hide resolved
buildInputs = [ numpy cython ];
propagatedBuildInputs = [ ];

doCheck = false;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
doCheck = false;
doCheck = false;

There are no tests included. Please try to checkout from source and check if they have unit tests, and try to run them. Unit tests give a good indication that they package has a high degree of validity and correctness given the python package set.

If tests are not available, then please use pythonImportsCheck to import the most important modules. This isn't as good as unit tests, but will usually give a good indication of run-time errors.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ran into this bug running pystan checkPhase:

...
  File "/nix/store/f87w21b91cws0wbsvyfn5vnlyv491czi-python3-3.8.3/lib/python3.8/unittest/main.py", line 154, in createTests
    self.test = loader.discover(self.start, self.pattern, self.top)
  File "/nix/store/f87w21b91cws0wbsvyfn5vnlyv491czi-python3-3.8.3/lib/python3.8/unittest/loader.py", line 349, in discover
    tests = list(self._find_tests(start_dir, pattern))
  File "/nix/store/f87w21b91cws0wbsvyfn5vnlyv491czi-python3-3.8.3/lib/python3.8/unittest/loader.py", line 405, in _find_tests
    tests, should_recurse = self._find_test_path(
  File "/nix/store/f87w21b91cws0wbsvyfn5vnlyv491czi-python3-3.8.3/lib/python3.8/unittest/loader.py", line 483, in _find_test_path
    tests = self.loadTestsFromModule(package, pattern=pattern)
  File "/nix/store/ld0khsbfx69hrvp4qfv3ijw0iczrkiyv-python3.8-setuptools-46.1.3/lib/python3.8/site-packages/setuptools/command/test.py", line 55, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/nix/store/f87w21b91cws0wbsvyfn5vnlyv491czi-python3-3.8.3/lib/python3.8/unittest/loader.py", line 205, in loadTestsFromName
    test = obj()
TypeError: lookup() missing 1 required positional argument: 'name'

I'm not sure where the responsibility lies for this bug.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using pytest instead of unittest will likely fix this

homepage = "https://github.com/wolfhong/LunarCalendar";
description = "LunarCalendar is a Lunar-Solar Converter, containing a number of lunar and solar festivals in China.";
license = licenses.mit;
maintainers = with maintainers; [ bletham seanjtaylor ];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this right?

Suggested change
maintainers = with maintainers; [ bletham seanjtaylor ];
maintainers = with maintainers; [ bletham seanjtaylor ];

homepage = "https://github.com/stan-dev/cmdstanpy";
description = "CmdStanPy is a lightweight interface to Stan for Python users which provides the necessary objects and functions to do Bayesian inference given a probability model written as a Stan program and data.";
license = licenses.mit;
maintainers = with maintainers; [ CmdStanPy ];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
maintainers = with maintainers; [ CmdStanPy ];
maintainers = with maintainers; [ CmdStanPy ];
undefined variable 'CmdStanPy' at /var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/mr-est/packet-spot-eval-3/pkgs/development/python-modules/cmdstanpy/default.nix:18:39

Please add yourself to maintainers.nix, with this following commit message

maintainers: add <name>

Comment on lines +23 to +30
cmdstanpy' = cmdstanpy.overridePythonAttrs (old: rec {
pname = "cmdstanpy";
version = "0.4.0";
src = fetchPypi {
inherit pname version;
sha256 = "07wmv36adp8jxf6zdas8n407dx51q7la486s7s40anvq6q0x412n";
};
});
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pinning within python-modules is highly discouraged. This will potentially introduce incompatible versions in a user's environment as either the new pinned version or the original version will be imported at runtime, breaking one of the packages.

The preference to handling this is to relax version bounds in the "install_requires" field. (could be in setup.py, pyproject.toml, requirements or others). In most cases, packages are still compatible with small API changes which may warrant a major version bump. We use test suites to verify that the package still works correctly.

If the package is still incompatible with the latest major version, then the most proper way to handle this is make an issue with the upstream package to adopt the latest major version. Or if upstream is not very responsive, you are free patch the source to make it compatible.

In very few circumstances, two versions of the same package are allowed to exist if the packages are extremely difficult to package. Some examples of this are tensorflow, which has huge ecosystems built around it and is hard to package. Another is django, which has 2 actively developed versions, and large ecosystems built around each.

One exception to this is applications, due to the way buildPythonApplication and toPythonApplication functions work, the related derivations will not bleed dependencies between packages. If the package doesn't need to be imported by other python modules, then this package would be a good candidate to convert into application. You can look at https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/admin/awscli/default.nix as an example of using an overlay within a python application.

Info on buildPythonApplication can be found here.
Info on toPythonApplication can be found here.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll make an issue with fbprophet to unpin cmdstanpy. In the meantime parhaps I should just pin cmdstanpy to 0.4.0 in its nix file? I thought 0.4.0 was super old, but the library actually just on a very fast release cadence. 0.4.0 was released in July 2019.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just found out the master branch of fbprophet is using the latest cmdstanpy version, so we'll just have to wait for the next release to make it into pypi.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if upstream is unlikely to release a new version soon-ish. Then you can always add the patches to enable it, but this might be difficult if there's a large divergence between master and the current released version.

pkgs/development/python-modules/fbprophet/default.nix Outdated Show resolved Hide resolved
Comment on lines +8992 to +9026
bletham = {
name = "Ben Letham";
email = "6339760+bletham@users.noreply.github.com";
github = "bletham";
githubId = 6339760;
};
seanjtaylor = {
name = "Sean Taylor";
email = "7317+seanjtaylor@users.noreply.github.com";
github = "seanjtaylor";
githubId = 7317;
};
usingsky = {
name = "Jinil Lee";
email = "7775824+usingsky@users.noreply.github.comseanjtaylor";
github = "usingsky";
githubId = 7775824;
};
ahartikainen = {
name = "Ari Hartikainen";
email = "13161958+ahartikainen@users.noreply.github.com";
github = "ahartikainen";
githubId = 13161958;
};
ariddell = {
name = "Allen Riddell";
email = "19950+ariddell@users.noreply.github.com";
github = "ariddell";
githubId = 19950;
};
wolfhong = {
email = "1505169+wolfhong@users.noreply.github.com";
github = "wolfhong";
githubId = 1505169;
};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is highly unusual

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not quite sure what you mean?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you're adding a lot of individuials who are not you, and adding them as maintainers

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, my apologies. These are the python package maintainers listed in pypi page. I'll put myself.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't add people unless you have their explicit permission to be added to nixpkgs as a package manager.

There's a difference between being a maintainer of a python package, and being a maintainer nixpkgs package. They are often not the same person.

Apply's jonringer's generous suggestions.

Co-authored-by: Jon <jonringer@users.noreply.github.com>
@stale
Copy link

stale bot commented Apr 18, 2021

I marked this as stale due to inactivity. → More info

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Apr 18, 2021
@Artturin Artturin closed this Feb 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.status: merge conflict 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md 6.topic: python 12. first-time contribution
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants