| Age | Commit message (Collapse) | Author |
|---|
|
microformats/JSON-LD/<meta> to JF2 (#80)
closes #75
This is a large squash commit. For details, see https://codeberg.org/eltrac/weepinbell/pulls/80
|
|
- use LEFT JOIN instead of making multiple queries
- put insert/update inside transactions
- add indexes
- fix configuration description
|
|
|
|
add ADR-0002 about migrating to H2
|
|
|
|
- new config `:storage-type` (default `:edn`, also support `:sqlite`)
- `LICENSE` change: add exceptions to `next.jdbc` (which uses EPL)
Reviewed-on: https://codeberg.org/eltrac/weepinbell/pulls/73
|
|
worker
|
|
thread-safely
fix
|
|
also remove post-conditional assertion from edn-storage/use-store, since
tests have that covered
|
|
- print clear stack trace using timbre when capturing exceptions
- don't pass nil to url/normalize; if url/normalize gets a nil, instead
of NPE, it now returns an empty string
- capture exceptions in processor pipeline to keep it from dying
- http.downloader now checks response status instead of blindly
downloading anything
|
|
- handler and core no longer care if storage is a valid vtable
- use-store assert storage is valid before it was returned
- AssertionError should be spotted while testing
|
|
|
|
utils: add url utilities for accurate comparison
- add url/normalize taht reconstructs URL, strips unnecessary
components, reorders query string and ensures determinism
- add url/url= that uses url/normalize
receiver: normalize URLs before checking and putting in queue
- use url= and normalize from utils.url
- rename verify-request to check-request to avoid confusion with
verifier.clj
verifier: check if source includes target URL more accurately
- implement url/extract with robinst/autolink-java, which extracts a
vector a URLs from string
- check if source includes target by
1. extracting all links from source body
2. normalizing all links before comparing with target link (with url=
func)
- refactor verify-headers -> match-content-types, with a higher order
function and clearer description
retriever, storage: use url= instead of = when comparing urls
|
|
- replace mf2j with mf2clj
- remove utils.clj completely
- refactored processor, use Chain of Responsibilities pattern and pass
around maps to reuse data (so we don't have to make a separate HTTP
request just to parse microformats, etc.)
- test: add more test cases to verifier test
- test: add basic test cases to all processing steps
- add more logging to each processing steps. good for debugging.
|
|
|
|
|
|
wrapper: use spec to check webmention data before writing to storage
warn when data does not meet spec (corrupted). for now we're not dealing
with corruption, just collect logs and analyze.
microformats: use spec to validate data and warn if it's in bad form
microformats: move parsers/wrappers to one sub-directory
storage: move weepinbell.spec to weepinbell.storage.spec
|
|
Instead of exit, it should create the data file. The previous approach
cause issues when testing and building.
|
|
|
|
used java.net to create a small validator that passes the test
refactor: rename validator to url, and move other url funcs here
|
|
|
