diff options
Diffstat (limited to 'content')
| -rw-r--r-- | content/en/article/weepinbell.md | 75 |
1 files changed, 54 insertions, 21 deletions
diff --git a/content/en/article/weepinbell.md b/content/en/article/weepinbell.md index 8e7452f0..dc838503 100644 --- a/content/en/article/weepinbell.md +++ b/content/en/article/weepinbell.md @@ -1,17 +1,18 @@ +++ -title = 'Why a New Webmention Receiver?' -subtitle = 'The Birth of Weepinbell and How I Designed It' -slug = '' -tags = [] +title = 'What’s next for the indie web?' +subtitle = 'Building a new Webmention receiver, and more communities' +slug = 'weepinbell' +tags = ["Clojure","IndieWeb","Lisp"] categories = '' -date = 2026-10-18T09:39:09+08:00 -draft = true +date = 2026-10-18 17:31:05 +draft = false toc = true +fedi = 'https://akk.eltr.ac/objects/14a4784a-95f4-41cd-afe3-a5248ae12519' +++ -I have been blogging in English for some time, but this is my first time participating in an IndieWeb conversation. So I think I should introduce myslef. +I have been blogging in English for some time, but this is my first time participating in an indie web conversation. So I think I should introduce myself first. -I'm Eltrac. I'm mainly a web developer but I like all kinds of thing about programming. I code in Go, Clojure and Lua, and I explore new tech all the time. I usually blog more than three times a week, but most of my posts are written in Chinese. I recently became an active member of [Codeberg e.V.](https://codeberg.org/Codeberg-e.V.) and have been working on my own FOSS project [weepinbell](https://codeberg.org/eltrac/weepinbell) for a while. +I'm Eltrac. I'm mainly a web developer but I like all kinds of thing about programming. I code in Go, Clojure and Lua, and I explore new tech all the time. I usually blog more than three times a week, but most of my posts are written in Chinese. I recently became an active member of [Codeberg e.V.](https://codeberg.org/Codeberg-e.V.) and have been working on my own FOSS project [weepinbell](https://codeberg.org/eltrac/weepinbell) for a while. It's a new **Webmention receiver** written in Clojure. In this post, I will talk about why I developed weepinbell, how I designed its architecture, what I think of the IndieWeb movement and many things. @@ -54,9 +55,9 @@ After the initial *quick and rough* implementation, I stopped working on weepinb Then the webmention.io rewrite came to my attention. -Why does it bother me? I don't even use webmention.io any more. I see some of my friends seeking alternatives, and... there aren't many to choose from! There is [webmentiond](https://github.com/zerok/webmentiond/) by Horst Gutmann and some other implementations here and there. My friend [Chlorine](https://codeberg.org/chlorine) developed [CircleAt](https://codeberg.org/chlorine/circleat) in Rust. And I'm sure there are many more. But frankly, none of them truly count as "alternative to webmention.io". +Why does it bother me? I don't even use webmention.io any more. I see some of my friends seeking alternatives, and... there aren't many to choose from! There is [webmentiond](https://github.com/zerok/webmentiond/) by Horst Gutmann and some other implementations here and there. My friend [Chlorine](https://chlo.is/) developed [CircleAt](https://codeberg.org/chlorine/circleat) in Rust. And I'm sure there are many more. But frankly, none of them truly count as "alternative to webmention.io". -Don't get me wrong. I think many of them are great for their purpose, i.e. small service one can self-host. However, they can be quite difficult for the non-tech IndieWeb folks and those who simply do not own a server. ==They need a public instance.== +Don't get me wrong. I think many of them are great for their purpose, i.e. small service one can self-host. However, they can be quite difficult for the non-tech indie web folks and those who simply do not own a server. ==They need a public instance.== ## Refactoring for scalability @@ -70,7 +71,7 @@ I'm a Go programmer by the living.[^2] And Clojure's [`core.async`](https://cloj Webmention is not big data. Storage is definitely not the bottleneck. In fact, up until [a few commits ago](https://codeberg.org/eltrac/weepinbell/commit/1db02376cd50e604973d17fd4a04f6e1969fc53c), weepinbell does **not even support any type of database**. It unloaded everything in a single file and then loaded it back when needed. And it just... worked, for months. -Of course this is neither performant nor safe. If I want to build something reliable, I need a database. If you have taken a look at that commit I linked, you might notice that the business logic is **untouched**. All that's added is a new storage implementation, a new config option, and a few lines in `core.clj` (the main component). +Of course this is neither performant nor safe. If I want to build something reliable, I need a database. If you have taken a look at that commit I linked, you might notice that the business logic is **untouched**. All that's added is a new storage implementation, a new config option, and a few lines in the main component (`core.clj`). I'll probably elaborate on architecture flexibility in another post. That's something I'm really proud of. The gist is that using a new type of database and ORM is easy, and will be just as easy if I ever decided to redo the storage layer again. @@ -89,7 +90,7 @@ I'll probably elaborate on architecture flexibility in another post. That's some Beside a new SQLite implementation, I added a *storage worker* that handles I/O asynchronously. So when processing a new webmention, the *processor* won't have to wait for DB transaction (or appending to a file; it does not care about what's going on underneath) to end. It simply delegates to the *storage worker* and keeps processing the next webmention. -The *processor* delegates by putting an operation object into a *channel*. It's similar to what I already have for receiver. When a new webmention request is sent to the *web server*, receiver does some basic checks (e.g. does `source` and `target` look like real URLs?) and then put the request into *webmention queue*. A *processor* will pick it up and make sure `source` has truly mentioned `target`, parse microformats, wrap data into a valid record, and hand it over to *storage worker*. +The *processor* delegates by putting an operation object into a *channel*. It's similar to what I already have for receiver. When a new webmention request is sent to the *web server*, receiver does some basic checks (e.g. does `source` and `target` look like real URLs?) and then put the request into the *webmention queue*. A *processor* will pick it up and make sure `source` has truly mentioned `target`, parse microformats, wrap data into a valid record, and hand it over to *storage worker*. Reading from a channel is **thread-safe**. And since I/O is handled internally by *storage worker*, it can also be thread-safe, *without the need for a mutex lock*. So we can add as many processors as we want to handle webmentions concurrently. @@ -117,7 +118,7 @@ Here is an example snippet from [microformats.io](https://microformats.io/). </div> ``` -It's just regular HTML with standardized class names like `h-card` and `p-name`. It's easy to add a few classes to your HTML. However, it's still hard to parse structural data from HTML with a few more classes. +It's just regular HTML with standardized class names like `h-card` and `p-name`. It's easy to add a few classes to your HTML. However, it's still hard to parse structural data from HTML with *just* a few more classes. Good thing there are already some parsers available... right? @@ -160,22 +161,22 @@ func parse() int32 { } ``` -Then I compiled it to `.wasm` using [tinygo](https://tinygo.org/) compiler, bundled the `.wasm` inside `.jar`, and called the Extism plugin from Clojure using chicory. Voilà! Here's [mf2clj](https://codeberg.org/eltrac/mf2clj), a Clojure wrapper of WebAssembly compiled from a Go wrapper of existing microformats parser. +Then I compiled it to `.wasm` using [tinygo](https://tinygo.org/) compiler, bundled the `.wasm` inside `.jar`, and called the Extism plugin from Clojure using chicory. Voilà! Here's [mf2clj](https://codeberg.org/eltrac/mf2clj), a Clojure wrapper of WebAssembly compiled from a Go wrapper of an existing microformats parser. By the way, I made a little logo just for fun.  -You use it just like a normal library. +You use it just like a normal library in Clojure. And it's surprisingly fast. ```clojure (require '[mf2clj.core :as mf2clj]) (mf2clj/parse html-string) ``` -Surprisingly, bundling a `.wasm` binary and the WebAssembly runtime inside the `.jar` is not as heavy as I thougt it would be. The `.jar` only grew 2 MB in size. *Am I really running Java?* +More surprisingly, bundling a `.wasm` binary and the WebAssembly runtime inside the `.jar` is not as heavy as I thougt it would be. The `.jar` only grew 2 MB in size. *Am I really running Java?* -The nightmare isn't over yet. In the begining, I settled for the incomplete `mf2j` library and used it for a while. At that time, I called the JSON output *data structure from Hell*. I still do —— Everything is an array! Everything is deep inside something's `properties` or `children`! Nothing is deterministic! Every website's data is different! +Nevertheless, the nightmare isn't over yet. In the begining, I settled for the incomplete `mf2j` library and used it for a while. At that time, I called the JSON output "*data structure from Hell*". I still do —— Everything is an array! Everything is deep inside something's `properties` or `children`! Nothing is deterministic! Every website's format is different! ```json { @@ -219,18 +220,50 @@ In the end, I had to write a smaller parser to parse output a the original parse If something does not exists, it'll just be `null`. The format is not final yet, I will make some adjustments. This can be turned off by `?microformats=raw` when querying webmentions. -Also, I think more than half of the bugs I found in weepinbell have something to do with microformats-related features. Maybe we should switch to Open Graph and JSON-LD, folks? +Also, I think more than half of the bugs I found in weepinbell have something to do with microformats-related features. *Maybe we should switch to Open Graph and JSON-LD, folks?* -Anyway, I'm just glad microformats are finally working. +Anyway, I'm just glad it's finally working. ## Slowing down -At this point, I've almost finished everything I have planned for the `v1.0.0` version: a proper database, horizontal scalability, up-to-date microformats parser +At this point, I've almost finished everything I have planned for the `v1.0.0` release: a proper database, horizontal scalability, up-to-date microformats parser. As for hosting a public instance, that'll involve authenticating users and making a dashboard for them to manage webmentions. That would be a separate project. -## Footnotes +I plan to develop a service that sits in front of weepinbell, handling logging in and stuffs. I mentioned the idea in the project's [PHILOSOPHY.md](https://codeberg.org/eltrac/weepinbell/src/branch/master/doc/PHILOSOPHY.md). + +==I want to keep the core of weepinbell minimal.== Those who want to be their own sysadmins don't have to care about authentication. They don't even need a frontend. They only need to self-host one `.jar` and use API to query webmentions, use RSS to receive notifications and use webhook to automate stuff. These are the built-ins. + +A *platform* like webmention.io needs more than that. If I want to keep weepinbell "for both individuals and large groups", I have to separate things. Also, this makes everything modular and swappable, a pretty good architecture decision. + +My writing to Brennan was an attempt to get more Clojure developers on the team. *Well, that didn't go well.* For the new platform, which will be named *Ringabell*[^4], I want to use more mainstream stack, probably Go, so that it's easier for people to get involved. + +For now, I'm still working on weepinbell's test suite. The current test coverage is more than 83%, but there are still some edge cases that haven't been covered by tests. I had to test a lot of paths by hand. Also it'll take some time for the new SQLite implementation to be stable. + +I don't think we need to or should move fast when it comes to developing reliable software, with or without LLMs. + +## Independence, and now decentralization + +In reply to Brennan's recent question, ["What do we do now?"](https://brennan.day/the-indieweb-now-has-genai-and-claude-built-in-what-do-we-do/#what-do-we-do-now), I think it's finanlly time to get more decentralized. + +Decentralization is never about moving away from the corporate web (that's independence). It's about not relying on a [single point](https://indieweb.org/single_point_of_aaron), no matter what the point is and who it involves. + +I agree with Brennan on not conflating IndieWeb and the lower-case indie web. The [IndieWeb camp](https://indieweb.org/) is a nice community and a great introduction to the indie web. But I think we should form more indie web communities other than the IndieWeb camp. + +[omg.lol](http://omg.lol/) is an example. It's a fun web community by [Neatnik](https://neatnik.net/). They provide infrastructure like a simple profile page, Fediverse instance and email forwarding. Brennan is also building [folk.zone](https://folk.zone/), which also hosts a fork of webmention.io. + +Those two may not be exactly alternatives to the IndieWeb camp, but they can be. And there can be more for people to choose from. + +Tech folks can self-host their own infrastructure if they want. For the non-tech folks, there should be more alternatives to what the IndieWeb camp has to offer. Poeple who are capable can host public services and take a small community. + +After all, communities are what brings the indie web together. We can and should build more communities (this is probably more like polycentricity rather than decentralization). We can have the *LispyWeb* for lisp programmers, *ArtsyWeb* for artists, and *Indie Indie Literature Club!*[^5] for book lovers. Those are just some random names I came up with. + +Perhaps the upcoming *Ringabell* might be the next cozy small community. Who knows? To everyone who's nice enough to finish reading this, ==I hope we can all *find our people*.== [^1]: A vtable in Clojure is basically a hashmap whose values are functions. It's a technique mentioned in one of Uncle Bob's books, *Functional Design*. [^2]: Actually I'm at the very begining of my career. Don't think of me as an expert of anything. XD [^3]: Clojure runs on JVM, so it compiles to JVM bytecode, just like Java. + +[^4]: Meaning "ring a bell". + +[^5]: The referrence is a galgame called "[Doki Doki Literature Club](https://en.wikipedia.org/wiki/Doki_Doki_Literature_Club!)". :3 |