About 20 days ago, I had made a blog post about an idea I had for a better federated search engine model.
It didn't take much time for it to develop into a thing I am fixated on.
I am putting the code out, its not ready or working, but it is something I am really happy to make and has filled my time with joy designing.
My current plan is the following:
- Get the basic web-ring creation process down
- Get scraping jobs functional
- Provide a basic query system
- Implement basic user accounts
- Implement basic federation
- Implement basic moderation
Once I am done with the core features that I have in mind, I will start working on adding more features and quality of life improvements.
Some features I want to work on to make this software more enticing to administrators:
- The ability to customize what is publicly accessible.
- The ability to edit the pages HTML style on the fly, without having to recompile.
- Containers for easy deployment.
In regards to application design, I am taking pages from my book in developing Android applications, along with cherry-picking from projects @[email protected] made.
- MVC design, with static pages to provide the fastest loading experience for users
- Bootstrap to make the pages responsive for any device
- Diesel to abstract database interaction and migration.
- Handlebars for view templating
- Axum as the HTTP core
Hopefully these design decisions make my application as debt free as possible.
If you have any advice or suggestion, please do give, I want to know how I can do better or avoid common pitfalls for newcomers!
If you have criticisms, please be constructive and have empathy towards the fact of me doing this because it makes me happy.
Really interesting proposal! To a degree the structure of Lemmy/Mbin/etc may be quite close to the categorising and moderating aspect, and might be a good place to start collecting URLs to crawl.
Each community could be considered analogous to a (rather chaotic) webring. When an instance doesn't meet your moderation expectation, defederate; if a MengZi user wants to see search results from different defederated segments, use a MengZi instance that federates with both, or just have both plugged into a searx instance.
The categorising side of MengZi could be (from an activitypub perspective) like a very cut down version of lemmy –each webring/category being a community, each website being a post, comments disabled or limited/filtered to hashtags.
A webring could be a specific sort of category/community, where a submitted website's url's page must contain specific metadata definining its membership in that ring or it is automoderated and removed. Such a category could automoderate the url and title to be the default page defined by its membership metadata. Existing webring html element standards could suffice.
A website could be crossposted to other categories, including to other instances, even to/from lemmy or other compatible activitypub sites. If a (cross)posted post is not a url returning the correct mime type for a category then it can be automoderated and deleted; same for other arbitrary criteria a category could define.
A website/post on MengZi could be accompanied by relevant crawling metadata, even full search database data available via the api for sharing to other MengZi instances to save duplication of crawling effort while distributing the database.