Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Osmscout server #108354

Merged
merged 5 commits into from Mar 8, 2021
Merged

Osmscout server #108354

merged 5 commits into from Mar 8, 2021

Conversation

Thra11
Copy link
Member

@Thra11 Thra11 commented Jan 3, 2021

Motivation for this change

Package osmscout-server and its dependencies. It's an application which allows you to manage offline OSM maps and serve them to pure-maps (PR #107836).

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@Thra11
Copy link
Member Author

Thra11 commented Jan 3, 2021

Needs #108347 to build successfully on aarch64 Rebased

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/470

pkgs/development/libraries/valhalla/default.nix Outdated Show resolved Hide resolved
pkgs/development/libraries/valhalla/default.nix Outdated Show resolved Hide resolved
pkgs/development/libraries/valhalla/default.nix Outdated Show resolved Hide resolved
pkgs/development/libraries/valhalla/default.nix Outdated Show resolved Hide resolved
pkgs/development/libraries/libpostal/default.nix Outdated Show resolved Hide resolved
pkgs/development/libraries/libosmscout/default.nix Outdated Show resolved Hide resolved
pkgs/development/libraries/libosmscout/default.nix Outdated Show resolved Hide resolved
pkgs/applications/misc/osmscout-server/default.nix Outdated Show resolved Hide resolved
@Thra11 Thra11 force-pushed the osmscout-server branch 2 times, most recently from 503fe33 to 27ff407 Compare February 22, 2021 20:25
@SuperSandro2000
Copy link
Member

@Thra11 can you please take a look at the failing ofborg builds?

@rinigus
Copy link

rinigus commented Feb 23, 2021

Thanks for working on getting OSM Scout Server and Pure Maps into NixOS. I can see that you incorporated libosmscout backend as well. As I am using very old version of it (as newer versions break API once in a while) and has stated that this backend is deprecated, I suggest to skip it when you package for NixOS. This would allow you later to use newer libosmscout library, if the apps using it will get traction. Otherwise, we will have a conflict over libosmscout version.

Alteranative is to package it as you did and be ready to disable the backend in OSM Scout Server as soon as libosmscout will be updated to newer versions.

@Thra11
Copy link
Member Author

Thra11 commented Feb 23, 2021

Yeah, I tend to go with a policy along the lines of, "If an optional backend is on by default, supported on this platform, and not too much extra effort to include, then include it, as some users may want it".

Alteranative is to package it as you did and be ready to disable the backend in OSM Scout Server as soon as libosmscout will be updated to newer versions.

I think this is probably a reasonable approach. At the moment, there are no other packages which depend libosmscout, so it is unlikely to be updated to an incompatible version in the near future. NixOS does support having multiple versions of a package, so long as the added complexity justifies it. osmscout-server's disable_osmscout flag will be handy if we do want to disable it.

@rinigus
Copy link

rinigus commented Feb 23, 2021

Good! Then at least you know the background and are aware of possible complications in future.

@Thra11
Copy link
Member Author

Thra11 commented Feb 23, 2021

@SuperSandro2000 w.r.t ofborg's failing aarch64-linux build, it appears that #111198 landed in staging a little while back and has now made its way into master, breaking the valhalla build. I will investigate what needs to be done for valhalla to build against geos 3.9.0.

As far as the x86_64-linux build is concerned, I have no idea what's going on there:

In file included from /build/source/valhalla/midgard/linesegment2.h:7,
                 from /build/source/src/midgard/linesegment2.cc:1:
/build/source/valhalla/midgard/point2.h: In member function 'std::size_t std::hash<valhalla::midgard::Point2>::operator()(const valhalla::midgard::Point2&) const':
/build/source/valhalla/midgard/point2.h:198:5: error: 'uint64_t' was not declared in this scope; did you mean 'u_int64_t'?
  198 |     uint64_t h;
      |     ^~~~~~~~
      |     u_int64_t
/build/source/valhalla/midgard/point2.h:199:54: error: 'h' was not declared in this scope
  199 |     char* b = static_cast<char*>(static_cast<void*>(&h));
      |                                                      ^
In file included from /build/source/valhalla/midgard/obb2.h:7,
                 from /build/source/src/midgard/obb2.cc:1:
/build/source/valhalla/midgard/point2.h: In member function 'std::size_t std::hash<valhalla::midgard::Point2>::operator()(const valhalla::midgard::Point2&) const':
/build/source/valhalla/midgard/point2.h:198:5: error: 'uint64_t' was not declared in this scope; did you mean 'u_int64_t'?
  198 |     uint64_t h;
      |     ^~~~~~~~
      |     u_int64_t
/build/source/valhalla/midgard/point2.h:199:54: error: 'h' was not declared in this scope
  199 |     char* b = static_cast<char*>(static_cast<void*>(&h));
      |                                                      ^
make[2]: *** [src/midgard/CMakeFiles/valhalla-midgard.dir/build.make:82: src/midgard/CMakeFiles/valhalla-midgard.dir/linesegment2.cc.o] Error 1

I'll take a closer look once I've sorted the geos 3.9.0 issue.

@rinigus
Copy link

rinigus commented Feb 23, 2021

@Thra11 : try to bump Valhalla to 3.1.0. I tested a bit with that version and OSM Scout Server looks to be working fine with it. From release notes, looks like Valhalla 3.1.0 supports geos 3.9. And I suspect that this undeclared uint64_t issue is fixed as well. If I remember correctly, I had to patch to get around that uint64_t issue. Hopefully, in newer Valhalla it is OK already.

@Thra11
Copy link
Member Author

Thra11 commented Feb 23, 2021

Yes, I saw there was a commit fixing geos-3.9 in valhalla 3.1.0. Just making the necessary changes for it to build, as there are some new/no longer optional vendored dependencies.

@Thra11 Thra11 force-pushed the osmscout-server branch 2 times, most recently from f1c6cfb to 2a33848 Compare February 23, 2021 13:41
@SuperSandro2000 SuperSandro2000 marked this pull request as draft February 23, 2021 14:13
@SuperSandro2000
Copy link
Member

When you fixed the build failures please undraft the PR.

@Thra11
Copy link
Member Author

Thra11 commented Feb 23, 2021

@rinigus Any plans for a new osmscout-server release any time soon? Looks like I need a few of the unreleased commits for it to build against valhalla 3.1.0. (At a minimum, it needs the commit changing to C++14, otherwise std::max isn't constexpr. I assume it should have the "add Valhalla 3.1 config" commit too).

If not, no worries (don't want to rush you into releasing before you're ready). I can always either package from an unreleased commit or package the 1.17.1 release plus the required commits as patches.

@rinigus
Copy link

rinigus commented Feb 23, 2021

@Thra11: I guess I should take it into my plans. There is also a wish to get ability to activate it via DBus to help those without systemd (pmOS). And switch to Valhalla 3.1.0 would also be helpful. So, there are some targets to work on the new release.

I am busy with working on my port of SFOS, but I will look into OSM Scout Server then this weekend.

@Thra11
Copy link
Member Author

Thra11 commented Feb 23, 2021

Right. Now we once again have something that builds and runs.

Not entirely happy with the workaround for the HowardHinnant/date dependency, but valhalla puts us in a slightly awkward position there: Valhalla bundles date as a vendored build-time dependency, doesn't install the date headers which it uses, but then expects users to have the exact same version installed when building against valhalla (I assume they expect you to use valhalla as vendored dependency or something?). The alternative solution is to de-vendor date as a separate package, but since it probably still needs to be pinned to the version valhalla wants, that could actually end up being worse.

@Thra11 Thra11 marked this pull request as ready for review February 23, 2021 20:22
@Thra11
Copy link
Member Author

Thra11 commented Mar 1, 2021

I've removed the unused fetchpatch input from libosmscout.

libpostal sqlite marisa kyotocabinet boost protobuf date
];

# Replace the default valhalla.json with the valhalla 3.1 version
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we explain here why we do this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a longer comment, which hopefully makes it clearer (along with some comments annotating what the patches do).

mv data/valhalla.json-3.1 data/valhalla.json
'';

qmakeFlags = [ "SCOUT_FLAVOR=kirigami" ];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And possibly document this one too, although don't feel super strongly about it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a comment.

Copy link
Contributor

@asymmetric asymmetric left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@SuperSandro2000 are you satisfied with this?

@asymmetric
Copy link
Contributor

/status needs_merger

@marvin-mk2
Copy link

marvin-mk2 bot commented Mar 7, 2021

Reminder: Please review!

Reminder: This Pull Request is awaiting merger. If you are the assigned reviewer with commit permission, please have a look. If you can't, please say so. If the status is not accurate, please change it. If nothing happens, this PR will be should be put back in the needs_reviewer queue in one day.


Note: This feature is currently broken. The bot will not actually change the status. If you see this message multiple times, please request a status change manually.

@Mic92
Copy link
Member

Mic92 commented Mar 7, 2021

/status awaiting_changes

@Thra11
Copy link
Member Author

Thra11 commented Mar 7, 2021

Rebased and fixed conflicts in pkgs/top-level/all-packages.nix

@SuperSandro2000
Copy link
Member

This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.

Result of nixpkgs-review pr 108354 run on x86_64-linux 1

5 packages built:
  • libosmscout
  • libpostal
  • osmscout-server
  • prime-server
  • valhalla

@SuperSandro2000 SuperSandro2000 merged commit 642f28a into NixOS:master Mar 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants