receiver, verifier, retriever, storage: URL normalization

utils: add url utilities for accurate comparison

- add url/normalize taht reconstructs URL, strips unnecessary
components, reorders query string and ensures determinism
- add url/url= that uses url/normalize

receiver: normalize URLs before checking and putting in queue

- use url= and normalize from utils.url
- rename verify-request to check-request to avoid confusion with
verifier.clj

verifier: check if source includes target URL more accurately

- implement url/extract with robinst/autolink-java, which extracts a
vector a URLs from string
- check if source includes target by
  1. extracting all links from source body
  2. normalizing all links before comparing with target link (with url=
func)
- refactor verify-headers -> match-content-types, with a higher order
function and clearer description

retriever, storage: use url= instead of = when comparing urls

1 files changed, 16 insertions, 12 deletions

diff --git a/src/weepinbell/processor/verifier.clj b/src/weepinbell/processor/verifier.clj
index 9a561ed..b868201 100644
--- a/src/weepinbell/processor/verifier.clj
+++ b/src/weepinbell/processor/verifier.clj
@@ -4,18 +4,24 @@
    1) source contains target URL.

    2) source respond with 2** status code.

    3) source's Content-Type is either HTML or plain text."

-  (:require [clojure.string :refer [includes? trim lower-case split]]

+  (:require [clojure.string :refer [trim lower-case split]]

+            [weepinbell.utils.url :as url]

             [weepinbell.http.client :as client]

             [taoensso.timbre :as timbre]))

 

-(defn- verify-headers [headers]

+(def text-content-types #{"text/html" "text/plain"})

+(defn- match-content-type? 

+  "Return true if "

+  [types headers]

   (when-let [content-type (:content-type headers)]

-    (-> content-type

-        (split #";")

-        first

-        trim

-        lower-case

-        (#{"text/html" "text/plain"}))))

+    (-> content-type (split #";") first trim lower-case types)))

+(def text-content-types? 

+  (partial match-content-type? text-content-types))

+

+(defn- includes-url? 

+  "Return true if text includes a certain URL. URLs are normalized before comparison."

+  [text url]

+  (some #(url/url= % url) (url/extract text)))

 

 (defn- verify-mention

   "Get source and follow redirects to check if target link is present."

@@ -37,14 +43,12 @@
         (timbre/warn "invalid source of" id ", status:" status)

         false)

       ;; verify Content-Type 

-      (or (not (verify-headers headers))

-          (not (verify-headers target-headers)))

+      (not (every? text-content-types? [target-headers headers]))

       (do

         (timbre/warn "source or target of" id "is neither HTML nor plain text content.")

         false)

       ;; verify inclusion of target URL

-      ;; TODO needs to be more robust and performant

-      (not (includes? (str body) target))

+      (not (includes-url? (str body) target))

       (do

         (timbre/warn "invalid source of" id ": target not found in source body.")

         false)