summaryrefslogtreecommitdiff
path: root/src/weepinbell/processor
AgeCommit message (Collapse)Author
11 daysall: use JF2 as a single source of truth, compile ↵eltrac
microformats/JSON-LD/<meta> to JF2 (#80) closes #75 This is a large squash commit. For details, see https://codeberg.org/eltrac/weepinbell/pulls/80
2026-10-15test: test concurrent proceesor; assert config option typeseltrac
2026-10-15test: remove http request retries from testseltrac
2026-10-15processor: concurrently starts multiple pipeline loopseltrac
2026-10-14processor, storage: clear queues before shutting down proceesor and storage ↵eltrac
worker
2026-10-14processor: log before write; logging afterwards doesn't mean write is completedeltrac
2026-10-13wrapper: fix not downloading photo with relative urlseltrac
2026-10-11proceesor, storage: better error handling; no interrupting pipelineeltrac
- print clear stack trace using timbre when capturing exceptions - don't pass nil to url/normalize; if url/normalize gets a nil, instead of NPE, it now returns an empty string - capture exceptions in processor pipeline to keep it from dying - http.downloader now checks response status instead of blindly downloading anything
2026-10-09verifier, http.client: retry timeout HTTP requestseltrac
- add weepinbell.http.client/with-retry macro - retry only requests that returns :error with type TimeoutException - retry is done asynchronously - with-retry should return a Future - retry verifier requesting source and target, 3 times with 5 seconds backoff
2026-10-07test, verifier, utils: fix failed testseltrac
- some namespaces tests require did not get renamed - url/extract might throw AssertionError if input contains non-HTTP(S) urls. it should not check schemes. - verifier doesn't need to filter non-HTTP(S) urls since they will return false anyway, and plus it makes localhost urls in tests fail
2026-10-07processor, webserver: move two main subsystems to own dir and make facadeseltrac
add docstrings format code
2026-10-07verifier, utils: fix verification failure when page contains invalid URLeltrac
- URISyntaxException was not caught in url/normalize, causing normalization failure which leads to verification failure when there is one invalid URL, even though valid URL exists - skip non-HTTP(S) urls when verifying
2026-10-07receiver, verifier, retriever, storage: URL normalizationeltrac
utils: add url utilities for accurate comparison - add url/normalize taht reconstructs URL, strips unnecessary components, reorders query string and ensures determinism - add url/url= that uses url/normalize receiver: normalize URLs before checking and putting in queue - use url= and normalize from utils.url - rename verify-request to check-request to avoid confusion with verifier.clj verifier: check if source includes target URL more accurately - implement url/extract with robinst/autolink-java, which extracts a vector a URLs from string - check if source includes target by 1. extracting all links from source body 2. normalizing all links before comparing with target link (with url= func) - refactor verify-headers -> match-content-types, with a higher order function and clearer description retriever, storage: use url= instead of = when comparing urls
2026-10-06core, processor: refactor startup; graceful shutdowneltrac
- implement a functional rewrite of core.clj, no more `def`, no more states. callable from an REPL - handle SIGINT/SIGTERM and perform graceful shutdown - print clear logs when starting up and shutting down
2026-09-23processor: fix not finding photo to proxyeltrac
2026-09-23processor: refactor processor; replace mf2j with mf2cljeltrac
- replace mf2j with mf2clj - remove utils.clj completely - refactored processor, use Chain of Responsibilities pattern and pass around maps to reuse data (so we don't have to make a separate HTTP request just to parse microformats, etc.) - test: add more test cases to verifier test - test: add basic test cases to all processing steps - add more logging to each processing steps. good for debugging.