| Age | Commit message (Collapse) | Author |
|
microformats/JSON-LD/<meta> to JF2 (#80)
closes #75
This is a large squash commit. For details, see https://codeberg.org/eltrac/weepinbell/pulls/80
|
|
|
|
|
|
|
|
worker
|
|
|
|
|
|
- print clear stack trace using timbre when capturing exceptions
- don't pass nil to url/normalize; if url/normalize gets a nil, instead
of NPE, it now returns an empty string
- capture exceptions in processor pipeline to keep it from dying
- http.downloader now checks response status instead of blindly
downloading anything
|
|
- add weepinbell.http.client/with-retry macro
- retry only requests that returns :error with type TimeoutException
- retry is done asynchronously
- with-retry should return a Future
- retry verifier requesting source and target, 3 times with 5 seconds
backoff
|
|
- some namespaces tests require did not get renamed
- url/extract might throw AssertionError if input contains non-HTTP(S)
urls. it should not check schemes.
- verifier doesn't need to filter non-HTTP(S) urls since they will
return false anyway, and plus it makes localhost urls in tests fail
|
|
add docstrings
format code
|
|
- URISyntaxException was not caught in url/normalize, causing
normalization failure which leads to verification failure when there is
one invalid URL, even though valid URL exists
- skip non-HTTP(S) urls when verifying
|
|
utils: add url utilities for accurate comparison
- add url/normalize taht reconstructs URL, strips unnecessary
components, reorders query string and ensures determinism
- add url/url= that uses url/normalize
receiver: normalize URLs before checking and putting in queue
- use url= and normalize from utils.url
- rename verify-request to check-request to avoid confusion with
verifier.clj
verifier: check if source includes target URL more accurately
- implement url/extract with robinst/autolink-java, which extracts a
vector a URLs from string
- check if source includes target by
1. extracting all links from source body
2. normalizing all links before comparing with target link (with url=
func)
- refactor verify-headers -> match-content-types, with a higher order
function and clearer description
retriever, storage: use url= instead of = when comparing urls
|
|
- implement a functional rewrite of core.clj, no more `def`, no more
states. callable from an REPL
- handle SIGINT/SIGTERM and perform graceful shutdown
- print clear logs when starting up and shutting down
|
|
|
|
- replace mf2j with mf2clj
- remove utils.clj completely
- refactored processor, use Chain of Responsibilities pattern and pass
around maps to reuse data (so we don't have to make a separate HTTP
request just to parse microformats, etc.)
- test: add more test cases to verifier test
- test: add basic test cases to all processing steps
- add more logging to each processing steps. good for debugging.
|