Use wayback package, remove internetarchive module, take 2 (#512)

The `internetarchive` module has been split off as a new package named `wayback` (we’re replacing an older, unmaintained package of the same name). This updates our code here to use it and delete the old `internetarchive` module.

Some parts of the `internetarchive` module (namely formatting memento responses as dicts) were not generic, and so have been moved directly into `cli` because they did not make sense as part of the new `wayback` package. They’ll probably be further refactored later.

This change was originally done in #511, but that PR was incomplete and had to be reverted. Fixes applied here:

- Fixed references to error types. These require importing from a private part of wayback, so we might want to merge and release edgi-govdata-archiving/wayback#7, which provides public access to those, first.
- Fixed some broken method calls.
- Fix the fact that date got renamed to timestamp on CdxRecord right before the 0.2.0 release.
- Add an end-to-end test that runs the whole import process, and covers some of the error cases we encounter from the various clients. It’s not perfect and doesn’t cover all errors, but was a lot quicker to get done than a completely custom test with mock client instances that purposely trigger all the various possible errors.

Author: Mr0grog

README.md

@@ -23,8 +23,6 @@ This component is intended to hold various backend tools serving different tasks
 
 Working and Under Active Development:
 
-* A Python API to the Internet Archive Wayback Machine's archived webpage
-  snapshots in ``web_monitoring.internetarchive``
 * A Python API to the web-monitoring-db Rails app in ``web_monitoring.db``
 * Python functions and a command-line tool for importing snapshots from the
   Internet Archive into web-monitoring-db.
@@ -102,7 +100,7 @@ Point your browser or ``curl`` at ``http://localhost:4000``.
 
 ## Releases
 
-New releases of the diffing server are published automatically as Docker images by CircleCI when someone pushes to the `release` branch. They are availble at https://hub.docker.com/r/envirodgi/ui. See [web-monitoring-ops](https://github.com/edgi-govdata-archiving/web-monitoring-ops) for how we deploy releases to actual web servers. We do not yet publish releases of the library-style tools in this repo (e.g. `db.py`, `internetarchive.py`).
+New releases of the diffing server are published automatically as Docker images by CircleCI when someone pushes to the `release` branch. They are availble at https://hub.docker.com/r/envirodgi/ui. See [web-monitoring-ops](https://github.com/edgi-govdata-archiving/web-monitoring-ops) for how we deploy releases to actual web servers. We do not yet publish releases of the library-style tools in this repo (e.g. `db.py`).
 
 Images are tagged with the SHA-1 of the git commit they were built from. For example, the image `envirodgi/processing:446ae83e121ec8c2207b2bca563364cafbdf8ce0` was built from [commit `446ae83e121ec8c2207b2bca563364cafbdf8ce0`](https://github.com/edgi-govdata-archiving/web-monitoring-processing/commit/446ae83e121ec8c2207b2bca563364cafbdf8ce0) in web-monitoring-processing.
 

docs/source/index.rst

@@ -21,5 +21,4 @@ developers. Contributions are welcome! See the
 
     installation
     db_api
-    wayback
     data_sources

docs/source/wayback.rst

@@ -13,3 +13,4 @@ toolz ~=0.10.0
 tornado ~=6.0.3
 tqdm ~=4.40.0
 tzlocal ~=2.0
+wayback ~=0.2.1

requirements.txt

@@ -9,7 +9,8 @@ from datetime import datetime, timedelta
 import random
 import sentry_sdk
 import sys
-from web_monitoring import db, internetarchive
+from web_monitoring import db
+from wayback import WaybackClient
 
 
 # The current Sentry client truncates string values at 512 characters. It
@@ -61,14 +62,14 @@ def wayback_has_captures(url, from_date=None):
     -------
     list of JSON
     """
-    try:
-        with internetarchive.WaybackClient() as wayback:
-            versions = wayback.list_versions(url, from_date=from_date)
+    with WaybackClient() as wayback:
+        versions = wayback.search(url, from_date=from_date)
+        try:
             next(versions)
-    except ValueError:
-        return False
-    else:
-        return True
+        except StopIteration:
+            return False
+        else:
+            return True
 
 
 def output_results(statuses):