all: use JF2 as a single source of truth, compile microformats/JSON-LD/<meta> to JF2 (#80)

closes #75

This is a large squash commit. For details, see https://codeberg.org/eltrac/weepinbell/pulls/80

4 files changed, 70 insertions, 23 deletions

diff --git a/doc/manual/API.md b/doc/manual/API.md
index cd78f02..8d3f9f1 100644
--- a/doc/manual/API.md
+++ b/doc/manual/API.md
@@ -16,12 +16,12 @@ Receiver endpoint is where people send their webmentions to. It's a route that a
 

 | Method | Path                                                  |

 | ------ | ----------------------------------------------------- |

-| `POST` | `/endpoint` (Configurable, see [configuration](./configuration.md)) |

+| `POST` | `/endpoint` (can be configured using option `:endpoint`, see `config.edn`) |

 

 | Post Params | Description                                                                                                                                               | Default |

 | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |

 | `source`    | Where webmention is coming from. The page that mentioned `target`.                                                                                        | None    |

-| `target`    | Where webmention is sent to. The page that was mentioned by `source`. Target's domain should be in the allow list of receiver. (see [configuration](./configuration.md)) | None    |

+| `target`    | Where webmention is sent to. The page that was mentioned by `source`. Target's domain should be in the allow list of receiver. (allow list is `:allowed-target-domains` in `config.edn`) | None    |

 

 The endpoint responds with status code `201` and a `Location` header pointing to the webmention. It'll take some time for weepinbell to verify, process and store webmention. Senders should check the `Location` URL a few minutes after.

 

@@ -63,10 +63,6 @@ The message themselves should be informative enough.
 | `GET`  | `/api/mention/{id}`                              |

 | `GET`  | `/api/mention/?source={source}&target={target}` |

 

-| Query Params    | Description                                                                                                                                          | Default |

-| --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |

-| `microformats`* | Only takes value `"raw"`. By default, microformats are reduced to deterministic structure. If `"raw"`, return original unmodified microformats data. | None    |

-

 An example response: 

 

 ```json 

@@ -77,18 +73,28 @@ An example response:
     "id": "<32_digit_SHA_256_hash>",

     "modified": <timestamp_of_last_modified>

   },

-  "microformats": {

+  "jf2": {

+    "type": "entry", // it's always "entry"

+    "name": "Title of this entry",

+    "url": "URL to this entry",

     "author": {

-      "name": "<author_name>",

-      "url": "<author_url>",

-      "photo": "<author_photo>"

+        "type": "card", // it's always "card"

+        "name": "Name of author",

+        "url": "URL to this author",

+        "photo": "Photo URL" // optional

     },

-    "type": "<mention | in-reply-to | like-of | repost-of>",

-    "published": "<date>",

+    "published": "Date string of published date",

     "content": {

-        "html": "<html>",

-        "text": "<text>"

-    }

+        "html": "<p>content of this entry</p>",

+        "text": "content of this entry"

+    },

+    "summary": "Summary or description of this entry", // optional

+    "wm-property": "mention", // or "like-of", "repost-of", "in-reply-to", "rsvp"

+    // below are fields that only exists when the coresponding "wm-property" is present

+    "like-of": "URL that this webmention represents a like of",

+    "repost-of": "URL that this webmention represents a repost of",

+    "in-reply-to": "URL that this webmention replies to",

+    "rsvp": "URL to the event that this webmention represents an RSVP for"

   },

   "extra": {

     "proxied-photo": "<config:base_url>/avatars/<photo_path>"

@@ -106,11 +112,10 @@ An example response:
 

 | Query Params    | Description                                                                                                                                                      | Default |

 | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |

-| `microformats`* | Only takes value `"raw"`. By default, microformats are reduced to deterministic structure. If `"raw"`, return original unmodified microformats data.             | None    |

 | `page`*         | Page number starting from `0`. **If no page number, weepinbell returns a full list.** If not a number (e.g. `last`, a string), weepinbell returns the last page. | None    |

-| `mftype`*       | Filters webmentions by their microformats type. Available types are `like-of`, `repost-of`, `in-reply-to` and `rsvp`. If no mftype, no filter is applied.        | None    |

+| `jf2type`*       | Filters webmentions by their JF2 type (.i.e the `wm-property`). Available types are `like-of`, `repost-of`, `in-reply-to` and `rsvp`. If absent, no filter is applied.        | None    |

 

-The responses are almost the same as `/api/mention/{id}` except that there are multiple webmention items wrapped inside a JSON array.

+The responses are mostly the same as `/api/mention/{id}` except that there are multiple webmention items wrapped inside a JSON array.

 

 ### Couting a list of webmentions

 

diff --git a/doc/manual/installation.md b/doc/manual/installation.md
index 442dfc5..321c2fc 100644
--- a/doc/manual/installation.md
+++ b/doc/manual/installation.md
@@ -60,6 +60,8 @@ For weepinbell to return the right URLs to webmentions and proxied photos, set `
 > [!warning]

 > You have to **restart** weepinbell for configuration to take effects.

 

+You may want to know how and where your webmentions are stored by default and how to change it, see [storage](./storage.md).

+

 ## 3. Run the app

 

 ### The command 

@@ -74,6 +76,17 @@ When you see this line of log, you're good to go:
 2026-09-28T08:20:45.680Z MACHINE INFO [weepinbell.core:43] - weepinbell started, listening on 3000

 ```

 

+> [!warning]

+> If you're just trying weepinbell, running it with a command is fine. It's recommended to use a service manager or a container. See below.

+

+### `systemd` (Linux)

+

+TODO

+

+### `rc` (OpenBSD)

+

+TODO

+

 ### Docker

 

 > [!warning]

diff --git a/doc/manual/migration.md b/doc/manual/migration.md
index 8ef7d38..6edb020 100644
--- a/doc/manual/migration.md
+++ b/doc/manual/migration.md
@@ -1,6 +1,35 @@
 # Migration

 

-## Migrating from `webmention.io`

+## Migrating from an older version

+

+### Migrate from before `v1.0.0`

+

+Before `v1.0.0`, weepinbell stored microformats rather than JF2, and did not support H2 database.

+

+To convert microformats in your `.edn` file, back up your data and run this:

+

+```shell

+# change .jar name to what you have; use absolute paths

+java -jar weepinbell-v1.x.x-standalone.jar edn-jf2ify /path/to/old.edn /path/to/new.edn

+

+# or, if you have Leiningen installed, you can run this in the top directory of the repository

+lein run edn-jf2ify /path/to/old.edn /path/to/new.edn

+```

+

+To migrate from EDN file-based storage to H2 database, run this (it automatically converts microformats to JF2 as well):

+

+```shell

+java -jar weepinbell-v1.x.x-standalone.jar edn-jf2ify /path/to/old.edn /path/to/database

+

+# or

+lein run edn-jf2ify /path/to/old.edn /path/to/new.edn

+```

+

+Also see [storage](./storage.md).

+

+## Migrating from another service

+

+### `webmention.io`

 

 You can migrate from [webmention.io](https://webmention.io/) with [this Babashka script](script/migrate-from-webmentionio.clj). Make sure you have [Babashka](https://babashka.org/) installed and:

 

diff --git a/doc/manual/storage.md b/doc/manual/storage.md
index 20f905b..3aa04f5 100644
--- a/doc/manual/storage.md
+++ b/doc/manual/storage.md
@@ -18,7 +18,7 @@ By default, weepinbell uses EDN storage and writes data to `data/webmention.edn`
 > [!note]

 > If you're not familiar with this format, see [edn](https://github.com/edn-format/edn). In short, it's Clojure's equivalent of JSON, but more extensible.

 

-When `:storage-type` is set to `:edn`, weepinbell writes to and reads from a `.edn` file, but more accurately, it's in "EDN lines" format, like JSONL. Every line is a valid EDN data. Every line is a webmention entry. Every entry is a map containing the following keys: `:webmention`, `:microformats`, `:extra`.

+When `:storage-type` is set to `:edn`, weepinbell writes to and reads from a `.edn` file, but more accurately, it's in "EDN lines" format, like JSONL. Every line is a valid EDN data. Every line is a webmention entry. Every entry is a map containing the following keys: `:webmention`, `:jf2`, `:extra`.

 

 It's not recommended to edit data file directly by hand, but if you must, back up before you do. Also, it's considered good practice to set up a cron job to routinely back up data file.

 

@@ -26,9 +26,9 @@ If you don't have much data, using a single EDN would be enough, though the curr
 

 ## H2 database

 

-H2 is a file-based SQL database in pure Java. Like SQLite, it's lightweight and fast. Unlike SQLite, it does not need a native C library, which is slower but more suitable for Java and Clojure.

+H2 is a file-based SQL database in pure Java. Like SQLite, it's lightweight and fast. Unlike SQLite, it does not need a native C library, so it's slower but more suitable for Java and Clojure.

 

-When `:storage-type` is set to `:h2`, weepinbell uses a H2 database. The database is usually two files. The `:data-path` is the name of database, not the filename like EDN file-based storage. Just set `:data-path` to a name you want and weepinbell will create and use relevant files for you.

+When `:storage-type` is set to `:h2`, weepinbell uses a H2 database. The `:data-path` is the path to the database file but **without extension name**, so instead of `"data/webmention.db"`, you put `"data/webmention"`.

 

-This is the recommended storage type.

+H2 database is more performant and faster, and it's lightweight enough.