[draft] update

1 files changed, 176 insertions, 7 deletions

diff --git a/content/en/article/weepinbell.md b/content/en/article/weepinbell.md
index adf76f11..8e7452f0 100644
--- a/content/en/article/weepinbell.md
+++ b/content/en/article/weepinbell.md
@@ -21,7 +21,7 @@ I wrote to [Brennan](https://brennan.day/) recently and he was really nice to me
 
 I would like to make things clear. I didn't start developing weepinbell because of Aaron's rewrite. I started early in this May to learn Clojure. I've been blogging (in Chinese) for years and of course I love the concept of IndieWeb. Naturally, my practice project for learning a new language involves IndieWeb technologies.
 
-The motive is that I want to get my hands dirty with the Webmention standard. I read through the [W3C specification document](https://www.w3.org/TR/webmention/) and started putting things in *parentheses*. You can still find snippets from that document in my code as comments. I bascially code against the specification.
+The motive is that I want to get my hands dirty with the Webmention standard. I read through the [W3C specification document](https://www.w3.org/TR/webmention/) and started putting things in *parentheses*. You can still find snippets from that document in my code as comments. I bascially code against the specification. It was a fun experience for me.
 
 ```clojure
     (if-let [error-msg (check-request source target allowlist)]
@@ -54,14 +54,183 @@ After the initial *quick and rough* implementation, I stopped working on weepinb
 
 Then the webmention.io rewrite came to my attention.
 
-Why does it bother me? I don't even use webmention.io any more. I see some of my friends seeking alternatives, and... there aren't many to choose from! There is [webmentiond](https://github.com/zerok/webmentiond/) by Horst Gutmann and some other implementations here and there. My friend [Chlorine](https://codeberg.org/chlorine) developed [CircleAt](https://codeberg.org/chlorine/circleat) in Rust. Frankly, none of them truly count as "alternative to webmention.io".
+Why does it bother me? I don't even use webmention.io any more. I see some of my friends seeking alternatives, and... there aren't many to choose from! There is [webmentiond](https://github.com/zerok/webmentiond/) by Horst Gutmann and some other implementations here and there. My friend [Chlorine](https://codeberg.org/chlorine) developed [CircleAt](https://codeberg.org/chlorine/circleat) in Rust. And I'm sure there are many more. But frankly, none of them truly count as "alternative to webmention.io".
 
-Don't get me wrong. I think many of them are great for their purpose, i.e. small service one can self-host. However, they can be quite difficult for the non-tech IndieWeb folks. ==They need a public instance.==
+Don't get me wrong. I think many of them are great for their purpose, i.e. small service one can self-host. However, they can be quite difficult for the non-tech IndieWeb folks and those who simply do not own a server. ==They need a public instance.==
 
-## For both individuals and large groups
+## Refactoring for scalability
 
-……
+I decided to continue development of weepinbell and make it "the next webmention.io", at least that's what I'm going for.
 
-## Slowing Down for Refactoring and Testing
+So I took a closer look at my code. They're not that messy, well partitioned (for the most part) and maintainable. I can swap some of the parts easily because I used hook functions and vtables.[^1] The architecture is flexible enough, and the *Lisp power* also makes modifying software really easy.
 
-## Parsing Microformats: Frustrating Experience
+The following part will be somewhat technical. Feel free to jump to the next section if you don't want to read about system design. The bottom line is: ==weepinbell would be lightweight enough for people to self-host; it would also be scalable enough to support large traffic.==
+
+I'm a Go programmer by the living.[^2] And Clojure's [`core.async`](https://clojuredocs.org/clojure.core.async) library greatly resembles Go's concurrey programming style, which means my skills of building scalable applications in Go translates.
+
+Webmention is not big data. Storage is definitely not the bottleneck. In fact, up until [a few commits ago](https://codeberg.org/eltrac/weepinbell/commit/1db02376cd50e604973d17fd4a04f6e1969fc53c), weepinbell does **not even support any type of database**. It unloaded everything in a single file and then loaded it back when needed. And it just... worked, for months.
+
+Of course this is neither performant nor safe. If I want to build something reliable, I need a database. If you have taken a look at that commit I linked, you might notice that the business logic is **untouched**. All that's added is a new storage implementation, a new config option, and a few lines in `core.clj` (the main component). 
+
+I'll probably elaborate on architecture flexibility in another post. That's something I'm really proud of. The gist is that using a new type of database and ORM is easy, and will be just as easy if I ever decided to redo the storage layer again.
+
+```clojure
+;; all I need to make sure is that the new storage implementation
+;; returns a valid storage vtable, like this:
+(defn use-store [db-file]
+  (let [ds (make-datasource db-file)]
+    (create-tables ds)
+    {:append (partial insert-webmention ds)
+     :update (partial update-webmention ds)
+     :delete (partial delete-webmention ds)
+     :get    (partial get-webmention ds)
+     :get-by-domain (partial get-webmentions-by-domain ds)}))
+```
+
+Beside a new SQLite implementation, I added a *storage worker* that handles I/O asynchronously. So when processing a new webmention, the *processor* won't have to wait for DB transaction (or appending to a file; it does not care about what's going on underneath) to end. It simply delegates to the *storage worker* and keeps processing the next webmention.
+
+The *processor* delegates by putting an operation object into a *channel*. It's similar to what I already have for receiver. When a new webmention request is sent to the *web server*, receiver does some basic checks (e.g. does `source` and `target` look like real URLs?) and then put the request into *webmention queue*. A *processor* will pick it up and make sure `source` has truly mentioned `target`, parse microformats, wrap data into a valid record, and hand it over to *storage worker*.
+
+Reading from a channel is **thread-safe**. And since I/O is handled internally by *storage worker*, it can also be thread-safe, *without the need for a mutex lock*. So we can add as many processors as we want to handle webmentions concurrently.
+
+The overall architecture looks like this:
+
+![](https://img.eltr.ac/geedeapro/en/article/weepinbell/weepinbell-architecture.svg "Architecture of the receiver part of weepinbell, drawn using Excalidraw")
+{.dark:invert}
+
+That's **horizontal scaling**. For self-hosters, simply set `pipeline-concurrency` to `1`, and there will be only one processor handling webmentions, using a reasonable amount of RAM and CPU. 
+
+For large groups or someone who wants to host a public instance, change a number in config and you're good to go.
+
+All of that is compiled to a single `.jar`.[^3]
+
+## The microformats nightmare
+
+Before I move on to a non-tech topic, there's one thing I have to *complain*: Using microformats markup is easy. Parsing it is really not.
+
+Here is an example snippet from [microformats.io](https://microformats.io/).
+
+```html
+<div class="h-card">
+    <a class="p-name u-url" href="http://blog.lizardwrangler.com/">Mitchell Baker</a>
+    (<a class="p-org h-card" href="http://mozilla.org/">Mozilla Foundation</a>)
+</div>
+```
+
+It's just regular HTML with standardized class names like `h-card` and `p-name`. It's easy to add a few classes to your HTML. However, it's still hard to parse structural data from HTML with a few more classes.
+
+Good thing there are already some parsers available... right?
+
+> Production ready parsers are available for: Go, Node, PHP, Python, Ruby.
+
+Right... I'm using Clojure...
+
+Clojure can use Java code, since it runs on top of it. I found [mf2j](https://github.com/karadaisy/mf2j), an "In-development Java parser for microformats2". It haven't been updated since... 10 years ago.
+
+There is a [Clojure implementation](https://codeberg.org/alinnow/microformats). It's archived, with its last update committed 11 years ago.
+
+Do I implement a new one myself? Do I pick up where they left off? How do I make sure I don't burn out and abandon it as well? After all, Clojure is a niche language and IndieWeb is also kind of niche. I can't imagine there being another Clojurist building something that relies on microformats. At least, I don't know anyone like that.
+
+I briefly considered running a micro-service that wraps the Go parser, and weepinbell can talk to this service using RESTful API or gRPC. But it feels heavy and everything just won't fit in a single `.jar`.
+
+So how can Clojure run programs written in a non-JVM language? What else runs on JVM? Scala, Groovy... *Wait, will WebAssembly work?*
+
+Yes! **WebAssembly runs on JVM.** There is a native JVM WebAssembly runtime called [chicory](https://github.com/dylibso/chicory).
+
+I spent an afternoon studying [Extism](https://github.com/extism/) and chirocy. Then I wrote a tiny little Go wrapper of Will Norris' [parser library](https://github.com/willnorris/microformats).
+
+```go
+package main
+
+import (
+	"bytes"
+
+	"github.com/extism/go-pdk"
+	"willnorris.com/go/microformats"
+)
+
+//go:wasmexport parse
+func parse() int32 {
+	input := pdk.Input()
+	r := bytes.NewReader(input)
+	data := microformats.Parse(r, nil)
+
+	pdk.OutputJSON(data)
+	return 0
+}
+```
+
+Then I compiled it to `.wasm` using [tinygo](https://tinygo.org/) compiler, bundled the `.wasm` inside `.jar`, and called the Extism plugin from Clojure using chicory. Voilà! Here's [mf2clj](https://codeberg.org/eltrac/mf2clj), a Clojure wrapper of WebAssembly compiled from a Go wrapper of existing microformats parser.
+
+By the way, I made a little logo just for fun.
+
+![A combination of Clojure logo and the microformats logo](https://img.eltr.ac/geedeapro/en/article/weepinbell/mf2clj.jpeg)
+
+You use it just like a normal library.
+
+```clojure
+(require '[mf2clj.core :as mf2clj])
+(mf2clj/parse html-string)
+```
+
+Surprisingly, bundling a `.wasm` binary and the WebAssembly runtime inside the `.jar` is not as heavy as I thougt it would be. The `.jar` only grew 2 MB in size. *Am I really running Java?*
+
+The nightmare isn't over yet. In the begining, I settled for the incomplete `mf2j` library and used it for a while. At that time, I called the JSON output *data structure from Hell*. I still do —— Everything is an array! Everything is deep inside something's `properties` or `children`! Nothing is deterministic! Every website's data is different!
+
+```json
+{
+  "items": [{
+    "type": ["h-card"],
+    "properties": {
+      "url": ["http://blog.lizardwrangler.com/"],
+      "name": ["Mitchell Baker"],
+      "org": [{
+        "value": "Mozilla Foundation",
+        "type": ["h-card"],
+        "properties": {
+          "name": ["Mozilla Foundation"],
+          "url": ["http://mozilla.org/"]
+        }
+      }]
+    }
+  }],
+  "rels": {},
+  "rel-urls": {}
+}
+```
+
+In the end, I had to write a smaller parser to parse output a the original parser. I called this feature *microformats reduction*. I remember webmention.io having something similar.
+
+```clojure
+"microformats": {
+    "author": {
+        "name": "Mitchell Baker",
+        "url": "http://blog.lizardwrangler.com/",
+        "photo": "https://blog.lizardwrangler.com/photo.png"
+      },
+    "type": "mention",
+    "published": "2026-07-25T00:00:00-07:00",
+    "content": {
+        "html": "..."
+        "text": "..."
+    }
+}
+```
+
+If something does not exists, it'll just be `null`. The format is not final yet, I will make some adjustments. This can be turned off by `?microformats=raw` when querying webmentions.
+
+Also, I think more than half of the bugs I found in weepinbell have something to do with microformats-related features. Maybe we should switch to Open Graph and JSON-LD, folks?
+
+Anyway, I'm just glad microformats are finally working.
+
+## Slowing down
+
+At this point, I've almost finished everything I have planned for the `v1.0.0` version: a proper database, horizontal scalability, up-to-date microformats parser
+
+## Footnotes
+
+[^1]: A vtable in Clojure is basically a hashmap whose values are functions. It's a technique mentioned in one of Uncle Bob's books, *Functional Design*.
+
+[^2]: Actually I'm at the very begining of my career. Don't think of me as an expert of anything. XD
+
+[^3]: Clojure runs on JVM, so it compiles to JVM bytecode, just like Java.