Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pythonPackages.sqlite-utils: depend on sqlite-fts4 #104253

Merged
merged 2 commits into from Dec 20, 2020

Conversation

meatcar
Copy link
Contributor

@meatcar meatcar commented Nov 19, 2020

Motivation for this change

Adding sqlite-fts4 as a dependency.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@SuperSandro2000
Copy link
Member

Result of nixpkgs-review pr 104253 run on x86_64-linux 1

4 packages built:
  • python37Packages.sqlite-fts4
  • python37Packages.sqlite-utils
  • python38Packages.sqlite-fts4
  • sqlite-utils (python38Packages.sqlite-utils)

@SuperSandro2000
Copy link
Member

Result of nixpkgs-review pr 104253 run on x86_64-darwin 1

4 packages built:
  • python37Packages.sqlite-fts4
  • python37Packages.sqlite-utils
  • python38Packages.sqlite-fts4
  • sqlite-utils (python38Packages.sqlite-utils)

@meatcar
Copy link
Contributor Author

meatcar commented Dec 13, 2020

/marvin opt-in
/status needs_reviewer

@marvin-mk2
Copy link

marvin-mk2 bot commented Dec 13, 2020

Hi! I'm an experimental bot. My goal is to guide this PR through its stages, hopefully ending with a merge. You can read up on the usage here.

@meatcar meatcar force-pushed the sqlite-utils-3.0 branch 2 times, most recently from 6110994 to 024f58c Compare December 13, 2020 18:03
@meatcar
Copy link
Contributor Author

meatcar commented Dec 13, 2020

I see there was a merge conflict due to #105368 forcing the update to sqlite-utils 3.0 in b8c6488 without adding the sqlite-fts4 dependency.

I've resolved the conflict, but for some reason there are two failing tests across all python versions, I'm investigating.

@jonringer
Copy link
Contributor

looks to be a small flakey difference. I'm okay with just disabling the tests

builder for '/nix/store/x10pnlagpnchjc73wqp38isjq1045pzz-python3.7-sqlite-utils-3.0.drv' failed with exit code 1; last 10 log lines:
          assert 0 == result.exit_code
          size_after_optimize = os.stat(db_path).st_size
  >       assert size_after_optimize < size_before_optimize
  E       assert 1667072 < 1662976

Copy link
Member

@vikanezrimaya vikanezrimaya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, this pull-request is a little bit confusing. First of all, do I understand correctly that there is already a commit bumping sqlite-utils to 3.0 but forgetting to add a dependency? how was this missed before?! 🙀

The sqlite-fts4 builds fine. I don't think this PR should break anything (it's not removing dependencies, only adding them, only side-effect could be closure growth), but there was a mention of a flaky test, so I can't give it an LGTM but I don't feel like I'm qualified to request changes here, since it's so confusing and the tests just plain hung up on my machine so I'm not sure what's happening and the tests did fail for me, and they apparently take some time to perform.

@meatcar
Copy link
Contributor Author

meatcar commented Dec 14, 2020

Thanks for taking a look, I agree this PR is pretty confusing at the moment. I was just going to poke it awake earlier today when I saw the merge conflict, and had to run.

Is there a better way I can clarify the confusion through the commit history? Should I revert b8c6488 instead, to keep the changes together and more atomic?

I have a hunch on how to fix the tests, I'll test it as soon as I get back to my pc.

@vikanezrimaya
Copy link
Member

I don't think the faulty commit should be reverted, rather this PR should be renamed to better represent that a dependency is added. The commit message should acknowledge the faulty commit and point out the mistake that was left there - and that now it is fixed.

Fixed b8c6488 where `sqlite-utils`
was updated without adding the new `sqlite-fts4` dependency.

Disabled two tests that are failing for sqlite v3.34.0, until they are
fixed upstream.
@meatcar meatcar changed the title pythonPackages.sqlite-utils: 2.22 -> 3.0 pythonPackages.sqlite-utils: depend on sqlite-fts4 Dec 14, 2020
@meatcar
Copy link
Contributor Author

meatcar commented Dec 14, 2020

Thanks @kisik21, I've reworded the commit.

I've also tracked down the cause for the failing tests, its due to an update of sqlite from 3.33 to 3.34. The tests fail at cfc56fb, and pass at the immediately previous e4d17dc (after waiting for 150 packages to rebuild overnight). I've opened an issue upstream (simonw/sqlite-utils#209), and disabled the failing two tests for now.

Testing this tool is straight forward:

$ cp /nix/var/nix/db/db.sqlite /tmp/db.sqlite
$ ./result/bin/sqlite-utils enable-fts --fts4 /tmp/db.sqlite ValidPaths path
$ ./result/bin/sqlite-utils search /tmp/db.sqlite ValidPaths firefox

Also optimize still works, I think it only fails for the contrived example in the tests because the entire DB is set up for full-text searching.

$ find /tmp/db.sqlite -printf '%s'
46256128
$ ./result/bin/sqlite-utils optimize /tmp/db.sqlite
$ find /tmp/db.sqlite -printf '%s'
30429184

@vikanezrimaya
Copy link
Member

I've run your example and I can confirm that this is working. Should we merge this now or wait for an upstream to fix the failing test? I'll leave this decision to whoever will be assigned to merge this, since I'm not yet well-versed in Nixpkgs policies, but if I were to decide, I'd err on the side of merging, as the functionality seems to be working as intended (as far as demonstrated by the remaining enabled tests and the manual test case described here).

/status needs_merger

@marvin-mk2
Copy link

marvin-mk2 bot commented Dec 20, 2020

Reminder: Please review!

Reminder: This Pull Request is awaiting merger. If you are the assigned reviewer with commit permission, please have a look. If you can't, please say so. If the status is not accurate, please change it. If nothing happens, this PR will be put back in the needs_reviewer queue in one day.

@kevincox kevincox merged commit 0d42077 into NixOS:master Dec 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants