receiver, verifier, retriever, storage: URL normalization

utils: add url utilities for accurate comparison

- add url/normalize taht reconstructs URL, strips unnecessary
components, reorders query string and ensures determinism
- add url/url= that uses url/normalize

receiver: normalize URLs before checking and putting in queue

- use url= and normalize from utils.url
- rename verify-request to check-request to avoid confusion with
verifier.clj

verifier: check if source includes target URL more accurately

- implement url/extract with robinst/autolink-java, which extracts a
vector a URLs from string
- check if source includes target by
  1. extracting all links from source body
  2. normalizing all links before comparing with target link (with url=
func)
- refactor verify-headers -> match-content-types, with a higher order
function and clearer description

retriever, storage: use url= instead of = when comparing urls

1 files changed, 2 insertions, 1 deletions

diff --git a/project.clj b/project.clj
index fa2d666..811348c 100644
--- a/project.clj
+++ b/project.clj
@@ -17,7 +17,8 @@
                  [cheshire "6.2.0"]
                  [ac.eltr/mf2clj "0.0.3"]
                  [com.taoensso/timbre "6.8.0"]
-                 [clj-rss "0.4.0"]]
+                 [clj-rss "0.4.0"]
+                 [org.nibor.autolink/autolink "0.12.0"]]
   :managed-dependencies [[org.clojure/tools.reader "1.6.0"]
                          [org.ow2.asm/asm "9.8"]] 
   :profiles {:uberjar {:aot :all}