summaryrefslogtreecommitdiff
path: root/src/weepinbell/processor/verifier.clj
AgeCommit message (Collapse)Author
2026-10-15test: remove http request retries from testseltrac
2026-10-09verifier, http.client: retry timeout HTTP requestseltrac
- add weepinbell.http.client/with-retry macro - retry only requests that returns :error with type TimeoutException - retry is done asynchronously - with-retry should return a Future - retry verifier requesting source and target, 3 times with 5 seconds backoff
2026-10-07test, verifier, utils: fix failed testseltrac
- some namespaces tests require did not get renamed - url/extract might throw AssertionError if input contains non-HTTP(S) urls. it should not check schemes. - verifier doesn't need to filter non-HTTP(S) urls since they will return false anyway, and plus it makes localhost urls in tests fail
2026-10-07processor, webserver: move two main subsystems to own dir and make facadeseltrac
add docstrings format code
2026-10-07verifier, utils: fix verification failure when page contains invalid URLeltrac
- URISyntaxException was not caught in url/normalize, causing normalization failure which leads to verification failure when there is one invalid URL, even though valid URL exists - skip non-HTTP(S) urls when verifying
2026-10-07receiver, verifier, retriever, storage: URL normalizationeltrac
utils: add url utilities for accurate comparison - add url/normalize taht reconstructs URL, strips unnecessary components, reorders query string and ensures determinism - add url/url= that uses url/normalize receiver: normalize URLs before checking and putting in queue - use url= and normalize from utils.url - rename verify-request to check-request to avoid confusion with verifier.clj verifier: check if source includes target URL more accurately - implement url/extract with robinst/autolink-java, which extracts a vector a URLs from string - check if source includes target by 1. extracting all links from source body 2. normalizing all links before comparing with target link (with url= func) - refactor verify-headers -> match-content-types, with a higher order function and clearer description retriever, storage: use url= instead of = when comparing urls
2026-09-23processor: refactor processor; replace mf2j with mf2cljeltrac
- replace mf2j with mf2clj - remove utils.clj completely - refactored processor, use Chain of Responsibilities pattern and pass around maps to reuse data (so we don't have to make a separate HTTP request just to parse microformats, etc.) - test: add more test cases to verifier test - test: add basic test cases to all processing steps - add more logging to each processing steps. good for debugging.