Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: kanzure/papermonk-downloader-plosone
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 01755da2445c
Choose a base ref
...
head repository: kanzure/papermonk-downloader-plosone
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: 2233c786435a
Choose a head ref
  • 3 commits
  • 2 files changed
  • 1 contributor

Commits on Aug 10, 2013

  1. implement download using request and jsdom 

    The download function was originally just throwing errors for
everything, but now it uses request.get to grab HTML and then jsdom to
parse the DOM.

Error handling needs more thought. For now errors are passed as the
first parameter to the callback, and metadata second.

version bump to: 0.0.2
    @kanzure
    kanzure committed Aug 10, 2013

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    GPG key ID: 4AEE18F83AFDEB23
    Expired
    Verified
    Learn about vigilant mode
    Copy the full SHA
    a1f7432 View commit details

  2. fix the required version of url-parser

    @kanzure
    kanzure committed Aug 10, 2013
    Copy the full SHA
    7e0e4cf View commit details

  3. Merge branch 'use-request-jsdom' into master

    @kanzure
    kanzure committed Aug 10, 2013
    Copy the full SHA
    2233c78 View commit details
Showing with 24 additions and 5 deletions.
  1. +20 −3 index.js
  2. +4 −2 package.json
23 changes: 20 additions & 3 deletions index.js
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
var urlparser = require("url-parser");
var request = require("request");
var jsdom = require("jsdom");

module.exports.test = function test(url) {
parsedurl = urlparser.parse(url);
@@ -18,8 +20,23 @@ module.exports.test = function test(url) {
return false;
};

// TODO: this should be split into multiple methods
module.exports.download = function download(url, options, callback) {
// TODO: this needs to be implemented
throw new Error("not implemented");
request.get(url, function requestcallback(error, response, body) {
var metadata = null;

if (!error && response.statusCode == 200) {
var window = jsdom.jsdom(body).createWindow();
var title = window.document.getElementsByTagName("title")[0].innerHTML;

metadata = {
"html": {
"title": title,
},
};
}

callback(error, metadata);
});

return undefined;
};
6 changes: 4 additions & 2 deletions package.json
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "papermonk-downloader-plosone",
"description": "plosone.org scraper",
"version": "0.0.1",
"version": "0.0.2",
"readmeFilename": "README.md",
"homepage": "https://github.com/kanzure/papermonk-downloader-plosone",
"author": {
@@ -42,7 +42,9 @@
"devDependencies": {
},
"dependencies": {
"url-parser": "*",
"url-parser": ">0",
"request": "~2.26.0",
"jsdom": "~0.8.0",
"tape": ">0"
},
"optionalDependencies": {