doc: update User Manual, Contributors' Guide, Philosphy and comment config.edn

8 files changed, 211 insertions, 151 deletions

diff --git a/doc/CONTRIBUTING.md b/doc/CONTRIBUTING.md
index cb82836..59b42df 100644
--- a/doc/CONTRIBUTING.md
+++ b/doc/CONTRIBUTING.md
@@ -6,16 +6,16 @@ Contributions are welcome and greatly appreciated, as long as you follow the rul
 

 Go to [Issues](https://codeberg.org/eltrac/weepinbell/issues) and find ones labeled [contribution welcome](https://codeberg.org/eltrac/weepinbell/issues?q=&type=all&state=open&labels=2145577&milestone=0&assignee=0&poster=0), or [good first issue](https://codeberg.org/eltrac/weepinbell/issues?q=&type=all&state=open&labels=2145589&milestone=0&assignee=0&poster=0) if you're contributing for the first time. 

 

-Once you find an issue you think you'll be able to help with, read the conversation and ask if anyone else has already been working on it. If not, you can fork the repository and work on your fork! 

+Once you find an issue you think you'll be able to help with, read the conversation and ask if anyone else has already been working on it. If not, you can ask the maintainer to assign you to that task and work on your fork. Good luck and thanks for contributing!

 

-Don't create a PR if it hasn't been discussed and approved in an issue yet!

+Don't create a PR if it hasn't been discussed and approved in an issue yet! If you have new ideas, please read [our philosophy](./PHILOSOPHY.md) to see if they align with our principles, and then open an issue.

 

 ## Style Guide 

 

 ### Code style 

 

 1. Functions should be small and does one thing only.

-2. A file should have less than 100 lines of code. 

+2. A file should have less than 150 lines of code.[^1]

 3. Test-Driven Development is not enforced, but committing tests alongside your code is encouraged.

 4. Use utility functions and `->`/`->>` macros to reduce layers of indentation.

 5. Format your code using [cljfmt](https://github.com/weavejester/cljfmt). (run `lein cljfmt fix`)

@@ -27,9 +27,36 @@ Don't create a PR if it hasn't been discussed and approved in an issue yet!
 3. Make sure you've run `lein test` and manually verified your changes before submitting PR. 

 4. Keep description simple and concise. Highlight affected scopes, things other contributors should look out for.

 

+## Principles

+

+### Documentation first, code later 

+

+1. A feature should be discussed in an issue before it's implemented in a PR.

+2. When writing a function, write docstring first.

+3. When designing a new component, write namespace's docstring first.

+

+### Minimal dependencies 

+

+1. Never depend on things you don't need. For instance, if only one or two features of a library is used, we should not depend on that library, but should seek alternatives that's more focused, implement those features within our project, or copy-paste code we need exactly into our codebase.

+2. If a requirement is not of great significance and implementing it will require introducing tons of dependencies, it's best to reconsider. Read [#19](https://codeberg.org/eltrac/weepinbell/issues/19) about how and why I abandoned the idea of auto-generating API documentation.

+

+### Make things pluggable 

+

+Whenever possible and necessary, pass something as a function argument instead of requiring it in that function's namespace, because:

+

+1. It makes the function **testable**. It's easy to pass mock objects as arguments. It's difficult to redefine a namespace's dependencies.

+2. It makes it easier to replace concrete implementation. If a component depends on SQL directly, it's hard to switch to KV storage. If a component uses something like `((:get-data storage) data-id)`, all we need to do is replace that `storage` vtable.

+3. It's lighter than implementing interfaces or protocols. We don't need to create an object or a record. We only need a regular symbol or a collection of symbols.

+4. It can still be type-safe if we check it using spec or schema.

+5. It makes the system easier to maintain.

+

+### Keep everything small and focused

+

+Every part of the system should be small. A file should have no more than 150 lines of code.[^1] A function should be easy to understand and should not require knowledge of more than one layer of abstraction. The key to keep things small is to keep them clear. Everything should have a clear purpose and a clear way to achieve that purpose.

+

 ## Set up environment

 

-Install Clojure and [Leinigen](https://leiningen.org/).

+Install Clojure, [Leinigen](https://leiningen.org/) and, optionally, [Babashka](https://babashka.org/).

 

 To install dependencies:

 

@@ -81,4 +108,18 @@ Also, refer to Codeberg's [Terms of Use](https://codeberg.org/Codeberg/org/commi
 

 > 7. You must not share projects that mostly consist of code written by "generative AI"-tools (including services such as *Claude*, *OpenAI Codex*). Such projects having an unclear copyright status (see requirements § 2 (1) 1 and § 2 (1) 3) and furthermore have little safeguards to ensure that they do not include harmful code (c.f. § 2 (1) 5).

 

-Generated code, especially by an LLM, is not the prompter's to license. Such code may not be compatible with our GNU Affero General Public License. LLM usage for research, validation and code review is tolerated.

+Generated code, especially by an LLM, is not the prompter's to license. Such code may not be compatible with our GNU Affero General Public License. 

+

+The following usage of LLMs is tolerated but not appreciated:

+

+1. Using LLMs to review your code and find bugs (not to fix bugs for you).

+2. Using LLMs to gather information about a topic or technology you're not familiar with.

+3. Using LLMs to propose a solution to your problem, but not to be accepted blindly nor to have them implemented by an "agent".

+

+The following usage of LLMs is strictly forbidden:

+

+1. Writing documentation, issues and PR descriptions.

+2. Generating code that has not been significantly modified by the contributors themselves. (By "significantly", I mean "significantly enough to claim copyright of this piece of code")

+3. Spread information generated by an LLM without verifying it yourself.

+

+[^1]: It should be more than enough for Clojure. If not, consider refactoring.

diff --git a/doc/PHILOSOPHY.md b/doc/PHILOSOPHY.md
index 8c06a29..e6e2344 100644
--- a/doc/PHILOSOPHY.md
+++ b/doc/PHILOSOPHY.md
@@ -2,61 +2,83 @@
 

 This document explains what weepinbell is, what it's not, and the design principles it follows. This document is for anyone who wants to get involved in the development of weepinbell and users who are simply curious. 

 

-## What It Is 

+---

 

-A tool for hosting [webmention](https://www.w3.org/TR/webmention/) service that receives and sends[^1] webmentions. It aims to be an all-in-one webmention toolkit that's easy to self-host. 

+## Scope

+

+### What It Is 

+

+A tool for hosting [webmention](https://www.w3.org/TR/webmention/) service that receives and sends[^1] webmentions. It aims to be an all-in-one webmention toolkit that's easy to self-host. It's built for the [IndieWeb](https://indieweb.org/) community. It should be compatible with popular IndieWeb standards like [microformats](https://microformats.org/).

 

 It should be capable of:

 

 1. Receiving webmentions for one or many webmasters[^2]

-2. Parsing metadata (microformats) of webmentions 

-3. Storing and querying webmentions 

-4. Sending webmentions 

-5. Notifiying users about webmentions they receive or send 

+2. Parsing structural data (e.g. microformats) of web pages that send webmentions 

+3. Basic CRUD of webmentions and their metadata

+4. Sending webmentions for anyone

+5. Notifiying users about webmentions they receive or send

 

-## What It's Not 

+### What It's Not 

 

 1. Full website interaction layer

 2. Commenting system 

-3. Commercial user tracking system 

-4. Enterprise-level system 

+3. Tracking and analytics system 

+4. Commercial or enterprise-level system 

 5. LLM-powered application

-6. Complex system that handles large amount of concurrent requests

 

-## Principles 

+---

+

+## Architectural Decisions

+

+### Minimal core

+

+Weepinbell deliberately does **not** do a lot of things.

+

+1. Weepinbell itself is headless and runs without a frontend.

+2. Weepinbell does not store user credentials nor manage user groups or roles.

 

-### Documentation first, code later 

+See weepinbell as a "micro-service".

 

-1. A feature should be discussed in an issue before it's implemented in a PR.

-2. When writing a function, write docstring first.

-3. When designing a new component, write namespace's docstring first.

+- If you want a dashboard or any user interface, put a frontned in front of it.

+- If you want multi-tenant features, develop a user management system that talks to weepinbell through its API, adding items to allowlist and etc.

+- If you want a CLI, develop one that talks to its API.

+- If you want none of that, great! Enjoy unbloated[^4] software!

 

-### Minimal dependencies 

+### Scalability

 

-1. Never depend on things you don't need. For instance, if only one or two features of a library is used, we should not depend on that library, but should seek alternatives that's more focused, implement those features within our project, or copy-paste code we need exactly into our codebase.

-2. If a requirement is not of great significance and implementing it will require introducing tons of dependencies, it's best to reconsider. Read [#19](https://codeberg.org/eltrac/weepinbell/issues/19) and see how and why we abandoned auto-generating API documentation.

+Weepinbell is for both individuals and large groups.

 

-### Make things pluggable 

+1. It is small enough for individual to deploy with ease.

+2. It is capable of handling hundreds of concurrent requests.

 

-Whenever possible, pass something as a function argument instead of requiring it in that function's namespace, because:

+### Subsystems

 

-1. It makes the function **testable**. It's easy to pass mock objects as arguments. It's difficult to redefine a namespace's dependencies.

-2. It makes it easier to replace concrete implementation. If a component depends on SQL directly, it's hard to switch to KV storage. If a component uses something like `((:get-data storage) data-id)`, all we need to do is replace that `storage`.

-3. It's lighter than implementing interfaces or protocols. We don't need to create an object or a record. We only need a regular symbol or a collection of symbols.

-4. It can still be type-safe if we check it using spec or schema.

-5. It makes the system easier to maintain.

+Weepinbell is devided into small focused subsystems.

 

-### Keep everything small and focused

+1. Web server, that handles API query and receives webmention requests.

+2. Processor, that starts pipelines to process received webmentions.

+3. Storage worker, that manipulates persistent data safely.

 

-Every part of the system should be small. A file should have no more than 100 lines of code.[^3] A function should be easy to understand and should not require knowledge of more than one layer of abstraction. The key to keep things small is to keep them clear. Everything should have a clear purpose and a clear way to achieve that purpose.

+Subsystems are decoupled. They do not talk to each other directly. They put and take data to and from channels, just like standalone systems gets input and outputs.

 

-## Why Clojure 

+See more in [Architecture](./ARCHITECTURE.md)

 

-1. Clojure is a functional language with laziness, which makes systems easier to maintain.

-2. Clojure runs on JVM, a stable platform.

+### Programming language, Clojure

+

+Weepinbell is developed in Clojure, a Lisp that runs on the JVM. 

+

+1. Clojure is a lazy functional language, which makes systems easier to maintain.

+2. Clojure runs on JVM, a stable platform that runs on all major operating systems.

 3. Clojure's semantics is pretty dense and it's easy to write less code and achieve more.

 4. *It's a Lisp and I like it.*

 

+Clojure has its downsides, which I decided to tolerate.

+

+1. No native binaries. Users must install Java to run weepinbell. Though compiling to native executables using GraalVM is possible, it take extra effort and the binary size is not what you typically get from a compiled language, like Go, Rust or Zig.

+2. Small community. Though clojure has a vibrant community, it's hard to find contributors who are both interested in IndieWeb and Lisp. Development can be slow with only one or two active contributors.

+

+---

+

 ## Why the name "weepinbell"

 

 Webmention is basically notification to someone when someone else mentioned them, which is like ringing a bell. And it's a name of a Pokemon. No particular reaon about why it has to be a pokemon.

@@ -64,3 +86,4 @@ Webmention is basically notification to someone when someone else mentioned them
 [^1]: Sender feature is planned but not yet implemented. See [#24](https://codeberg.org/eltrac/weepinbell/issues/24).

 [^2]: Webmasters refers to those who own a website.

 [^3]: Trust me. 100 LoC is more than enough for Clojure.

+[^4]: One might argue Clojure/Java itslef it not "unbloated" enough. Fair point, but I'm doing the best I can.

diff --git a/doc/manual/configuration.md b/doc/manual/configuration.md
deleted file mode 100644
index 6c11f58..0000000
--- a/doc/manual/configuration.md
+++ /dev/null
@@ -1,102 +0,0 @@
-# Configuration

-

-## 0. Create a `.edn` file

-

-> [!note]

-> If you're not familiar with this format, see [edn](https://github.com/edn-format/edn). In short, it's Clojure's equivalent of JSON, but more extensible.

-

-Configure weepinbell by creating `config.edn` at the app's working directory[^1].

-

-Copy-paste the following [default configuration](https://codeberg.org/eltrac/weepinbell/src/branch/doc/resources/config.edn) to your file and start editing.

-

-```edn

-{:port "3000"

- :base-url "http://localhost:3000"

- :endpoint "/endpoint"

- :allowed-target-domains ["example.com"]

- :data-path "data/webmention.edn"

- :webhook {:on-receive []

-           :on-create []

-           :on-update []

-           :on-delete []}

- :webhook-request-fn nil

- :feed-length 8}

-```

-

-Port can also be configured with enviroment variable `PORT`. Config overrides enviroment variable.

-

-## 1. First options to modify 

-

-For weepinbell to receive webmentions for your site, add your site's domain to `:allowed-target-domains`.

-

-```edn 

- :allowed-target-domains ["your.site.domain" "can-have.more-than.one"]

-```

-

-For weepinbell to return the right URLs to webmentions and proxied photos, set `:base-url` to where you're hosting weepinbell, **without trailing slash**.

-

-```edn 

- :base-url "https://example.com"

-``` 

-

-> [!warning]

-> You have to **restart** weepinbell for configuration to take effects.

-

-## 2. Tweak some features

-

-### Webhook

-

-Weepinbell can be configured to send `POST` requests to certain URLs when certain events ocurred, like when a webmention is received (but not processed yet), created, updated or deleted.

-

-```edn

- :webhook {:on-receive []

-           :on-create ["https://send-to-me-when-created.com"]

-           :on-update ["https://can-have-multiple-URLs-under-one.event"

-                       "https://like-this.net"]

-           :on-delete []}

-```

-

-If an event key's value is empty, request will not be sent when that event occurs.

-

-Webhook requests can carry customized payload. You need a *payload generator function*. The function should take two arguments, `hook` and `data`. 

-

-- `hook`: a Clojure keyword, which can only be one of the four keywords: `:on-receive`, `:on-create`, `:on-update` and `:on-delete`.

-- `data`: a wrapped webmention map, much like what you'll get in `/api/mention/{id}` (see [API Referrence](./API.md)), but a Clojure map. Note that when `hook` is `:on-receive`, `data` only contains `webmention` but no `microformats` or `extra` fields.

-

-The return value should be a map, which will be merged to HTTP request options. Example:

-

-```edn 

-{:body "I'm in request body"

- :headers {"Content-Type" "text/plain"}

- :post-params {"title" "Webmention received"}}

-```

-

-Bascially:

-

-```edn 

-(fn [hook data] 

-  {:body ...

-   :headers {}

-   ;; more 

-  })

-```

-

-For now you can only use a preset generator by setting `:webhook-request-fn` to a keyword. Currently available presets are `:ntfy` and `:bark`. (for push notification apps [ntfy](https://ntfy.sh/) and [Bark](https://bark.day.app/))

-

-```edn 

- :webhook-request-fn :ntfy ;; or :bark

-```

-

-See examples [here](https://codeberg.org/eltrac/weepinbell/src/commit/0471b4017ccab10f92f140881aeafba9ceb0896f/src/weepinbell/webhook_presets.clj#L21-L33). Feel free to [contribute](../CONTRIBUTING.md) more presets!

-

-### RSS/Atom Feed

-

-Weepinbell generates [Atom](https://validator.w3.org/feed/docs/atom.html) feed for each domain.

-

-Currently, the only option available is length of feed. If set to `n`, Atom feed lists only the newest `n` webmentions. Cannot use `0` and cannot list all available webmentions.

-

-```edn

- :feed-length 8

-```

-

-[^1]: Working directory is where the starting command is run. For example, if you `cd` into `/home/yourname/weepinbell` and run `java -jar weepinbell-xxx.jar`, then your configuration file should be `/home/yourname/weepinbell/config.edn`. If you're using Docker or systemd, you can configure working directory in `docker-compose.yaml` and `.service` files.

diff --git a/doc/manual/index.md b/doc/manual/index.md
index 35f233d..cb1cb0c 100644
--- a/doc/manual/index.md
+++ b/doc/manual/index.md
@@ -3,9 +3,12 @@
 ## Index 

 

 1. [Installation](./installation.md)

-2. [Configuration](./configuration.md)

-3. [API Referrence](./API.md)

-4. [Migration](./migration.md)

+2. [API Referrence](./API.md)

+3. [Webhook](./webhook.md)

+4. [RSS](./rss.md)

+5. [Migration](./migration.md)

+

+---

 

 ## Ask a question 

 

@@ -27,5 +30,5 @@ I don't care for issue templates. When [submitting a new issue](https://codeberg
 

 ## Suggest a Feature

 

-

+Read about our [philosophy](../PHILOSOPHY.md) first and then open an issue.

 

diff --git a/doc/manual/installation.md b/doc/manual/installation.md
index 3ab457d..442dfc5 100644
--- a/doc/manual/installation.md
+++ b/doc/manual/installation.md
@@ -34,7 +34,33 @@ lein uberjar
 

 And grab `target/weepinbell-v0.x.x-standalone.jar`.

 

-## 2. Run the app

+## 2. Configure the app

+

+Create a file named `config.edn` at the current working directory[^1], or set environment variable `WEEPINBELL_CONFIG` to a custom file pathname. Either way, it should be written in EDN format.

+

+> [!note]

+> If you're not familiar with this format, see [edn](https://github.com/edn-format/edn). In short, it's Clojure's equivalent of JSON, but more extensible.

+

+Use [resources/config.edn](../../resources/config.edn) as a template.

+

+Refer to comments in the template config file or a feature's documentation for details. But first, you should change at least the following two options in your config file.

+

+For weepinbell to receive webmentions for your site, add your site's domain to `:allowed-target-domains`.

+

+```edn 

+ :allowed-target-domains ["your.site.domain" "can-have.more-than.one"]

+```

+

+For weepinbell to return the right URLs to webmentions and proxied photos, set `:base-url` to where you're hosting weepinbell, **without trailing slash**.

+

+```edn 

+ :base-url "https://example.com"

+``` 

+

+> [!warning]

+> You have to **restart** weepinbell for configuration to take effects.

+

+## 3. Run the app

 

 ### The command 

 

@@ -63,7 +89,7 @@ docker compose up -d
 

 Podman should work as well, but it's not tested.

 

-## 3. Set up web server

+## 4. Set up web server

 

 It's recommended to put weepinbell behind a reverse proxy, using something like [nginx](https://nginx.org/) and [caddy](https://caddyserver.com/). Here's an example `Caddyfile`:

 

@@ -73,9 +99,9 @@ example.com
 reverse_proxy :3000

 ```

 

-Weepinbell listens port `:3000` by default. See [configuration](./configuration.md) on how to change it.

+Weepinbell listens port `:3000` by default. Edit configuration `:port` if you want something else.

 

-## 4. Point your site to weepinbell 

+## 5. Point your site to weepinbell 

 

 Add this to your `<head>`:

 

@@ -85,7 +111,4 @@ Add this to your `<head>`:
 

 Senders will look for this `<link>` tag and know where to send webmentions to.

 

----

-

-Now weepinbell is installed on your site. Before it can actually receive webmentions for you, you need to [configure it](./configuration.md).

-

+[^1]: Working directory is where the starting command is run. For example, if you `cd` into `/home/yourname/weepinbell` and run `java -jar weepinbell-xxx.jar`, then your configuration file should be `/home/yourname/weepinbell/config.edn`. If you're using Docker or systemd, you can configure working directory in `docker-compose.yaml` and `.service` files.

diff --git a/doc/manual/rss.md b/doc/manual/rss.md
new file mode 100644
index 0000000..3b653c3
--- /dev/null
+++ b/doc/manual/rss.md
@@ -0,0 +1,10 @@
+# RSS/Atom Feed

+

+Weepinbell generates [Atom](https://validator.w3.org/feed/docs/atom.html) feed for each domain.

+

+Currently, the only option available is length of feed. If set to `n`, Atom feed lists only the newest `n` webmentions. Cannot use `0` and cannot list all available webmentions.

+

+```edn

+ :feed-length 8

+```

+

diff --git a/doc/manual/webhook.md b/doc/manual/webhook.md
new file mode 100644
index 0000000..cd2ddc1
--- /dev/null
+++ b/doc/manual/webhook.md
@@ -0,0 +1,44 @@
+# Webhook

+

+Weepinbell can be configured to send `POST` requests to certain URLs when certain events ocurred, like when a webmention is received (but not processed yet), created, updated or deleted.

+

+```edn

+ :webhook {:on-receive []

+           :on-create ["https://send-to-me-when-created.com"]

+           :on-update ["https://can-have-multiple-URLs-under-one.event"

+                       "https://like-this.net"]

+           :on-delete []}

+```

+

+If an event key's value is empty, request will not be sent when that event occurs.

+

+Webhook requests can carry customized payload. You need a *payload generator function*. The function should take two arguments, `hook` and `data`. 

+

+- `hook`: a Clojure keyword, which can only be one of the four keywords: `:on-receive`, `:on-create`, `:on-update` and `:on-delete`.

+- `data`: a wrapped webmention map, much like what you'll get in `/api/mention/{id}` (see [API Referrence](./API.md)), but a Clojure map. Note that when `hook` is `:on-receive`, `data` only contains `webmention` but no `microformats` or `extra` fields.

+

+The return value should be a map, which will be merged to HTTP request options. Example:

+

+```edn 

+{:body "I'm in request body"

+ :headers {"Content-Type" "text/plain"}

+ :post-params {"title" "Webmention received"}}

+```

+

+Bascially:

+

+```edn 

+(fn [hook data] 

+  {:body ...

+   :headers {}

+   ;; more 

+  })

+```

+

+For now you can only use a preset generator by setting `:webhook-request-fn` to a keyword. Currently available presets are `:ntfy` and `:bark`. (for push notification apps [ntfy](https://ntfy.sh/) and [Bark](https://bark.day.app/))

+

+```edn 

+ :webhook-request-fn :ntfy ;; or :bark

+```

+

+See examples [here](https://codeberg.org/eltrac/weepinbell/src/commit/0471b4017ccab10f92f140881aeafba9ceb0896f/src/weepinbell/webhook_presets.clj#L21-L33). Feel free to [contribute](../CONTRIBUTING.md) more presets!

diff --git a/resources/config.edn b/resources/config.edn
index f73758a..85aa74b 100644
--- a/resources/config.edn
+++ b/resources/config.edn
@@ -1,14 +1,32 @@
-{:port "3000"
+{;; :port is a string representing port number weepinbell listens to
+ ;; configurable by environmental variable PORT, which is overridden by config
+ :port "3000"
+ ;; :base-url is used to form permalinks to a resource (a webmention, proxied photos...)
  :base-url "http://localhost:3000"
+ ;; :endpoint is where webmention requests are sent to
  :endpoint "/endpoint"
+ ;; :allowed-target-domains is a vector representing an allow list
+ ;; weepinbell only receive webmentions whose target is from one of the domain below
  :allowed-target-domains ["example.com"]
+ ;; available storage types: 
+ ;; - :edn (:data-path should point to an edn text file)
+ ;; - :sqlite (:data-path should point to a SQLite db file)
+ ;; :data-path can be non-existent. weepinbell will create them.
  :storage-type :edn
  :data-path "data/webmention.edn"
+ ;; :webhook is a map of events to URLs. weepinbell sends POST request to these URLs
+ ;; when certain event occurs.
+ ;; :webhook-request-fn is the payload generator for those requests. If nil, webhook
+ ;; will carry no data and sends a plain POST request.
  :webhook {:on-receive []
            :on-create []
            :on-update []
            :on-delete []}
- :webhook-request-fn nil
+ :webhook-request-fn nil ;; supported values: `:ntfy` or `:bark`
+ ;; :feed-length is how many latest items an RSS feed should contain.
  :feed-length 8
+ ;; :pipeline-concurrency is how many processing pipelines should be started.
+ ;; For personal use, 1 would be enough. For large traffic, adjust accordingly
  :pipeline-concurrency 5
+ ;; Minimal log level. Set to `:debug` when debugging or submitting a bug.
  :log-level :info}