Merge pull request #14 from metacpan/oalders/v1

Upgrade to v1
metacpan · Nov 18, 2016 · c307fbe · c307fbe
2 parents c51d807 + 30553cc
commit c307fbe
Show file tree

Hide file tree

Showing 55 changed files with 344 additions and 373 deletions.
diff --git a/README.md b/README.md
@@ -1,29 +1,141 @@
-metacpan-examples
-=================
-
-Sample code to get you up and running quickly.  If you are new to MetaCPAN,
-please use the scripts/endpoints/author examples to get yourself started.  If
-you are searching for a specific topic, try scripts/topic as your starting
-point.
-
-The purpose of this repository is to showcase how to use the MetaCPAN API.
-There are examples using raw curl as well as the following Perl modules:
-MetaCPAN::API, MetaCPAN::API::Tiny, WWW::Mechanize and ElasticSearch.  Scripts
-are all found in /scripts.  You can run the Perl scripts in the usual manner,
+# metacpan-examples
+
+This repository contains sample code to get you up and running quickly with the MetaCPAN API.
+
+## Please use MetaCPAN::Client
+
+There are many different ways and clients which you can use to access the API.  However, *we strongly encourage you* to start by using the MetaCPAN::Client module.  If you have compelling reasons to use a different client, feel free to do so, but please be aware that MetaCPAN::Client is fully supported by MetaCPAN.  You will likely save yourself (and us) much debugging time if you begin with this module rather than rolling your own Elasticsearch queries.  If MetaCPAN::Client *doesn't* do something which you need it to do, please open a GitHub issue to let us know about it.
+
+There are some examples to be found here which use `curl`, `WWW::Mechanize`, `Search::Elasticsearch` etc.  These are useful for reference, but please do take `MetaCPAN::Client` as your starting point. 
+
+## Getting Started
+
+Scripts are all found in /scripts.  You can run the Perl scripts in the usual manner,
 but if you want to get up and running quickly, you are encouraged to run your
 scripts via Carton.  The workflow is:
 
     cpanm Carton
     carton install
-    bin/carton scripts/endpoints/author/1-fetch-single-author-mcpan-api.pl
+    bin/carton scripts/author/1a-search-authors.pl
 
 Or, you can use Carton directly:
 
-    carton exec perl -Ilib scripts/endpoints/author/1-fetch-single-author-mcpan-api.pl
+    carton exec perl -Ilib scripts/author/1a-search-authors.pl
 
-Using the carton wrapper in bin will save you a few keystrokes and will
+Using the `bin/carton` will save you a few keystrokes and will
 automatically add new libs to the path in future, if they are required.  Use
 the workflow you are most comfortable with.
 
 Please open issues for examples you would like to see and send pull requests
 for examples you've already written.
+
+## Upgrading from v0
+
+The MetaCPAN API v1 is now available.  v0 will be deprecated after a 6 month window.  This window closes on or after June 1, 2017.  Here's a guide to converting your scripts.
+
+## Elasticsearch version
+
+v1 of the MetaCPAN API uses [Elasticsearch v2.4.0](https://www.elastic.co/guide/en/elasticsearch/reference/2.4/index.html) (v0 was at 0.20.2).  There are many breaking changes in this upgrade, since it spans almost 4 years.
+
+The MetaCPAN API versions will increase with breaking changes, but we will not use the same versioning as Elasticsearch itself.
+
+### Data Structure Changes
+
+Elasticsearch 1.x changed the data structure returned when fields are used.
+For example before one could get a `ArrayRef[ HashRef [ Str ] ]` where now
+that will come in the form of `ArrayRef[ HashRef [ ArrayRef [ Str ] ] ]`.
+
+Our convenience endpoints revert this behaviour, but if you're crafting your own searches, you should be aware that the structure of your fields data may have changed.
+
+### Client-specific Changes
+
+#### MetaCPAN::Client
+
+v0:
+
+```perl
+use MetaCPAN::Client->new();
+```
+
+v1:
+
+```perl
+use MetaCPAN::Client->new( version => 'v1' );
+```
+
+#### Search::Elasticsearch
+
+v0:
+
+```perl
+use Search::Elasticsearch;
+my $es = Search::Elasticsearch->new(
+    cxn_pool => 'Static::NoPing',
+    nodes    => 'api.metacpan.org',
+    trace_to => 'Stdout',
+);
+```
+
+v1:
+
+```perl
+use Search::Elasticsearch;
+my $es = Search::Elasticsearch->new(
+    cxn_pool         => 'Static::NoPing',
+    nodes            => 'https://fastapi.metacpan.org',
+    send_get_body_as => 'POST',
+    trace_to         => 'Stdout',
+);
+```
+
+* The URL of the v1 API is https://fastapi.metacpan.org  Note that not only is the host name new, but we've also switched to https.
+
+* You'll need to set `send_get_body_as => 'POST'`.  This is because v1 does not accept `GET` with a body.
+
+##### $es->search
+
+```perl
+my $faves = $es->search(
+    index => 'v0',
+    type  => 'favorite',
+    body  => {
+        query  => { match_all => {} },
+        facets => {
+            dist => {
+                terms => { field => 'favorite.distribution', size => 10 },
+            },
+        },
+    },
+    size => 0,
+);
+```
+
+```perl
+my $faves = es()->search(
+    index => 'v1',
+    type  => 'favorite',
+    body  => {
+        aggs => {
+            dist => {
+                terms => { field => 'distribution', size => 10 },
+            },
+        },
+    },
+);
+``` 
+
+* The index is now `v1`
+* `match_all` is the default query type, so it's not required to specify it
+* `facets` have been replaced by `aggs`
+* The data structure returned for `facets` does not mirror the `aggs` return values, so you'll need to rework your logic when checking the return values.
+* When searching on a type (`favorite` in this case), do not prefix the key with the type name.  So, as seen above, `field => 'favorite.distribution'` is now `field => 'distribution'`
+
+#### GET via command line or browser
+
+v0: `http://api.metacpan.org/v0/author/MSTROUT`
+
+v1: `https://fastapi.metacpan.org/v1/author/MSTROUT`
+
+* Note the new host name
+* Note that we now use HTTPS
+* Note that the path begins with v0 rather than v1
diff --git a/cpanfile b/cpanfile
@@ -5,10 +5,9 @@ requires 'Data::Printer::Filter::URI';
 requires 'DateTime';
 requires 'HTTP::Tiny::Mech';
 requires 'IO::Socket::SSL';
-requires 'JSON';
+requires 'JSON::MaybeXS';
 requires 'MetaCPAN::API::Tiny';
-requires 'MetaCPAN::Client', '>= 1.015000';
-requires 'Search::Elasticsearch';
+requires 'MetaCPAN::Client', '>= 1.028003';
 requires 'WWW::Mechanize::Cached';
 requires 'WWW::Mechanize::Cached::GZip';
 

diff --git a/lib/MetaCPAN/Util.pm b/lib/MetaCPAN/Util.pm
@@ -8,9 +8,10 @@ use Sub::Exporter -setup => { exports => ['es'] };
 
 sub es {
     return Search::Elasticsearch->new(
-        cxn_pool => 'Static::NoPing',
-        nodes    => 'api.metacpan.org',
-        trace_to => 'Stdout',
+        cxn_pool         => 'Static::NoPing',
+        nodes            => 'https://fastapi.metacpan.org',
+        send_get_body_as => 'POST',
+        trace_to         => 'Stdout',
     );
 }
 

diff --git a/scripts/author/1-fetch-single-author-curl.sh b/scripts/author/1-fetch-single-author-curl.sh
@@ -0,0 +1,3 @@
+#!/bin/sh
+
+curl https://fastapi.metacpan.org/v1/author/MSTROUT
diff --git a/...points/author/1-fetch-single-author-es.pl → scripts/author/1-fetch-single-author-es.pl b/...points/author/1-fetch-single-author-es.pl → scripts/author/1-fetch-single-author-es.pl
@@ -8,12 +8,12 @@
 
 my $es = Search::Elasticsearch->new(
     cxn_pool => 'Static::NoPing',
-    nodes    => 'api.metacpan.org',
+    nodes    => 'fastapi.metacpan.org',
     trace_to => 'Stdout',
 );
 
 my $author = $es->get(
-    index => 'v0',
+    index => 'v1',
     type  => 'author',
     id    => 'MSTROUT',
 );

diff --git a/.../1-fetch-single-author-www-mech-cached.pl → .../1-fetch-single-author-www-mech-cached.pl b/.../1-fetch-single-author-www-mech-cached.pl → .../1-fetch-single-author-www-mech-cached.pl
@@ -6,6 +6,6 @@
 
 use WWW::Mechanize::Cached::GZip;
 my $mech = WWW::Mechanize::Cached::GZip->new;
-$mech->get('http://api.metacpan.org/v0/author/MSTROUT');
+$mech->get('https://fastapi.metacpan.org/v1/author/MSTROUT');
 
 say $mech->content;
diff --git a/...author/1-fetch-single-author-mcpan-api.pl → scripts/author/1-fetch-single-author.pl b/...author/1-fetch-single-author-mcpan-api.pl → scripts/author/1-fetch-single-author.pl
@@ -6,7 +6,7 @@
 use Data::Printer;
 use MetaCPAN::Client;
 
-my $mcpan  = MetaCPAN::Client->new();
+my $mcpan  = MetaCPAN::Client->new( version => 'v1' );
 my $author = $mcpan->author('MSTROUT');
 
 p $author;
diff --git a/...nts/author/1a-search-authors-mcpan-api.pl → scripts/author/1a-search-authors.pl b/...nts/author/1a-search-authors-mcpan-api.pl → scripts/author/1a-search-authors.pl
@@ -7,7 +7,7 @@
 use Data::Printer;
 use MetaCPAN::Client;
 
-my $mc = MetaCPAN::Client->new;
+my $mc = MetaCPAN::Client->new( version => 'v1' );
 
 my $search = $mc->author( { name => 'Olaf *' } );
 

diff --git a/...points/author/1b-scroll-all-authors-es.pl → scripts/author/1b-scroll-all-authors-es.pl b/...points/author/1b-scroll-all-authors-es.pl → scripts/author/1b-scroll-all-authors-es.pl
@@ -9,7 +9,7 @@
 my $scroller = es()->scroll_helper(
     search_type => 'scan',
     scroll      => '5m',
-    index       => 'v0',
+    index       => 'v1',
     type        => 'author',
     size        => 100,
     body        => {

diff --git a/.../1c-scroll-all-authors-with-twitter-es.pl → .../1c-scroll-all-authors-with-twitter-es.pl b/.../1c-scroll-all-authors-with-twitter-es.pl → .../1c-scroll-all-authors-with-twitter-es.pl
@@ -9,16 +9,15 @@
 my $scroller = es()->scroll_helper(
     search_type => 'scan',
     scroll      => '5m',
-    index       => 'v0',
+    index       => 'v1',
     type        => 'author',
     size        => 100,
     body        => {
         query => {
             filtered => {
-                query => { term => { 'author.profile.name' => 'twitter' } },
+                filter => { match => { 'profile.name' => 'twitter' } },
             },
-            match_all => {}
-        }
+        },
     },
 );
 

diff --git a/...ndpoints/author/2-twitter-or-github-es.pl → scripts/author/2-twitter-or-github-es.pl b/...ndpoints/author/2-twitter-or-github-es.pl → scripts/author/2-twitter-or-github-es.pl
@@ -9,32 +9,30 @@
 my $scroller = es()->scroll_helper(
     search_type => 'scan',
     scroll      => '5m',
-    index       => 'v0',
+    index       => 'v1',
     type        => 'author',
     size        => 100,
     body        => {
         query => {
-            match_all => {},
-            filtered  => {
+            filtered => {
                 filter => {
                     or => [
                         {
                             and => [
                                 {
                                     term => {
-                                        'author.profile.name' => 'twitter'
+                                        'profile.name' => 'twitter'
                                     }
                                 },
-                                { term => { 'author.country' => 'US' } }
+                                { term => { 'country' => 'US' } }
                             ]
                         },
                         {
                             and => [
                                 {
-                                    term =>
-                                        { 'author.profile.name' => 'github' }
+                                    term => { 'profile.name' => 'github' }
                                 },
-                                { term => { 'author.country' => 'CA' } }
+                                { term => { 'country' => 'CA' } }
                             ]
                         },
                     ],

diff --git a/scripts/by-topic/1-fetch-author-curl.sh b/scripts/by-topic/1-fetch-author-curl.sh
diff --git a/scripts/by-topic/2-fetch-author-mcpan-api.pl b/scripts/by-topic/2-fetch-author-mcpan-api.pl
diff --git a/scripts/by-topic/3-fetch-author-es.pl b/scripts/by-topic/3-fetch-author-es.pl
diff --git a/scripts/by-topic/4-scrolling-es.pl b/scripts/by-topic/4-scrolling-es.pl
diff --git a/scripts/by-topic/5-autocomplete-www-mech.pl b/scripts/by-topic/5-autocomplete-www-mech.pl
diff --git a/scripts/by-topic/6-a-module-is-a-file-es.pl b/scripts/by-topic/6-a-module-is-a-file-es.pl
diff --git a/...points/distribution/1-bugs-for-dist-es.pl → scripts/distribution/1-bugs-for-dist-es.pl b/...points/distribution/1-bugs-for-dist-es.pl → scripts/distribution/1-bugs-for-dist-es.pl
@@ -7,7 +7,7 @@
 use MetaCPAN::Util qw( es );
 
 my $dist = es()->get(
-    index => 'v0',
+    index => 'v1',
     type  => 'distribution',
     id    => 'Moose',
 );

diff --git a/scripts/endpoints/author/1-fetch-single-author-curl.sh b/scripts/endpoints/author/1-fetch-single-author-curl.sh
diff --git a/scripts/endpoints/author/1-fetch-single-author-mcpan-api-tiny.pl b/scripts/endpoints/author/1-fetch-single-author-mcpan-api-tiny.pl
diff --git a/scripts/endpoints/favorite/3-leaderboard-es.pl b/scripts/endpoints/favorite/3-leaderboard-es.pl