Hister: a private search engine [AIP]

What are the Hardware requirements? I imagine the index will become quite large, no?

Big fan of searx and use it on all of my devices. This will be a welcome addition.

I’ve been slowly migrating from buku to hister, but it’s still early days. I do like þat hister indexes sites, which beats having to manually tag everyþing. So far it’s looking pretty good, þough. Þanks for writing it.

Ver cool!

I set hister up for a while, completely forget about it and checked it again after reading this Lemmy post. Well there are now 6.600 pages and 480 MB RAM usage.

Looks like I need to integrate hister more into my web usage. But maybe I need some training. It isn’t possible to setup hister as my default search engine in browser with automated fallback to normal Meta-search with whatever engine? And I use login credentials maybe a faster way like passkey possible in the future?

Very cool! Is there a place to share index DB? Having a pre-fetched index on awesome lists and fmhy would be amazing!

Being able to navigate only through the content of my favorites websites, oh my, excellent news. I’m truly eager to try it.

I have a newbie question tho : I would like to use Hister while still supporting the people who create website content. Say I’m online and don’t wish to use the local preview, can I access to the website instead of loading my local content ? If I access the website via Hister instead of Searx for instance, would my request be added to their traffic numbers, or not ? (those lowering their referencing and their visibility).

(Not like my individual usage would change anything anyway but still asking to get a graspe of what it implies).

Thank you for the hard work !

Uhh, neat. Will definitely try it out.

I have actually been looking for something like this, look forward to giving it a go!

This is a great idea, I’m gonna try

Interesting, I didn’t see it in the documentation so if you didn’t document that already, you can have your local instance as search suggestion for Firefox on mobile and desktop. I use it for my own wiki, e.g. https://mastodon.pirateparty.be/@utopiah/116351732150481942

Also how I would imagine it is default search there and if no hit then fallback to a default search engine, e.g. DDG.

Deleted by moderator

This looks really rad. I have been trying to build a leftist search engine using searxng and it has a lot of issue because of things getting blocked over tor due to the amount of background requests you have to be making to those sites. I have thought about building something like this to deal with that issue, but just had my hands full with other projects. Quite obviously (for those who like to come online to shit on really cool project people are building) the use case is people looking to build their own privately indexed search results without it having to get polluted by shit that comes up in other search engines. I feel like that was immediately apparent to me, because I have that use case and I immediately saw the utility here. For those who don’t get, just because it doesn’t fit the use case you have for internet related projects, doesn’t mean it’s not extremely useful to others. You literally don’t have to come on here just to shit on what other people are doing simply because, “you don’t get it.” This project looks really cool and I am going to look into replacing my searxng instance with this soon. I am super stoked that somebody has done this work. It has needed to be done for a long time. So thank you!

I don’t get it. It indexes pages which were already visited, right? So in order to find some website I need to first use another search engine. Afterwards, that website is in my browsing history and if I need it again, I don’t need to search for it. So what’s the use case for this project?

this sounds really cool!

How did you come up with the name?

iirc “Hister” is the name of an evil despot used in at least one of Nostradamus’ “prophecies” and is often believed (by those who believe in these things) to be a reference to Hitler.

its also an ancient name for the River Danube, which flows through much of WWII’s battlefields

Those are some impressive projects - some people are just built different huh

Any interested in making this a federated search?

I’ve been toying with the idea of a web-ring style search engine where it uses the fediverse’s post and sub system so that you can subscribe to “indexers” that are made by other users (or even mastodon users themselves) so that you can have highly targeted search engine systems. So, for example, you could make multiple search “scopes” and each of those scopes would follow specific other “sources” and search from them. Likewise, users could provide links and descriptions to add new entries to each scope which would, in essense, work like a fediverse post and then be distributed to all other interested parties.

I haven’t done a lick of implementation yet, but thought I’d share the idea here as I consider doing this more and more every day. The only thing I haven’t figured out yet is images, because ideally that would be a special type of scope view.

Already installed and loving this idea.

This seems super cool, you can curate your own search index :O

Being able to preview your search results offline is quite neat too!

One question though, how will you solve the issue of “echo chambers”, because from my understanding, it looks like it will only present results that you search for / have searched for, meaning for some people, you could be stuck with results that only support your view of the world.

Asciimoo always with the cool stuff!

Think I’ll try getting it setup on Kubernetes and using it for searching documentation when programming.

Would you be interested in getting a Kubernetes example in the docs too then? If you have any requirements for it let me know.

I’ve been running this for a while since I first heard of it, it’s already saved me a few times trying to recall a page for something.

I treat it like a more advanced history and it’s great!

Oh damn this could replace the bookmakers. I have Linkding with the Linkding Injector extension, but this would be next level.

You can view the saved text too, nice. Would it be possible to attach an HTML file from like single file for sites that have heavy image content as part of the “view” button? That’d completely replace Linkding/Karakeep/Linkwarden use cases for me

Understandable if not, that’s ancillary to the text search focus

The name reminds me of this crazy nostradamus “documentary “ that used to play on HBO when I was a kid.

Is there a future where Hister takes on the functions of SearXNG? I.E. new content is discovered through APIs but local or previous content is prioritized?

Very likely a huge thing to ask, but have you considered adding federation to the indexing? I presume that generating and storing the index database would quickly put a severe storage bottleneck (maybe the idea here is that no single person explores the internet enough for their index database to become infeasibly large?). But, at least to me, it sounds like allowing different hister instances to share a common index (or index subsets) would greatly reduce this bottleneck. Just curious if you have any thoughts on this. Overall I love the idea and am looking forward to giving it a try soon, so well done!

If anyone else is confused what this does, I pointed the clankers at it and asked. Way to bury the lede OP.

See [6] for some missing features (?) that would be nice to clear up.

[1]

• You open a webpage normally.
• The browser extension reads the page that your browser loaded.
• It sends the page text, title, URL, and other details to your Hister server.
• Hister stores and indexes that content.
• Later, you search Hister for words from the page.
• Hister finds the saved page, even when you cannot remember its title or URL.

[2]

so, like a Google search of visited sites and my own drive?

Yes. That is actually a reasonable use case.

If you pointed Hister at:

C:\Documents

it would:

1. Walk the folder tree.

2. Read supported files.

3. Build a full-text index.

4. Watch for changes.

5. Re-index new or edited files automatically.

You could then search for things like:

tax invoice lenovo

lumbar radiculopathy

that python script with sqlite

“Methylphenidate” (exact phrase)

without caring which folder the file lives in.

Is it like Google?

Yes, but only for your own data.

Imagine Google, except the only websites are:

C:\Documents

D:\Notes

your browser history

pages you crawled yourself

You open a browser, type a search, and receive ranked results from your own files instead of the internet.

[3]

Can you access it from elsewhere on your LAN?

Yes.

By default it listens only on localhost:

127.0.0.1:4433

However, you can configure it to listen on all network interfaces:

0.0.0.0:4433

Then any device on your LAN can use the web interface or API, provided your firewall allows it. You should also enable an access token or user accounts if you expose it beyond your own PC.

[4] What interests me more for your home setup is that Hister is not just a search engine.

It already exposes:

a web UI,

a REST API,

and an MCP endpoint for AI assistants.

That means you could run one Hister instance on your server, index your documents once, and then:

search them yourself from any device on the LAN,

or let an AI search the same index through MCP.

If I wanted a self-hosted “Google for my own stuff” with minimal setup, Hister is much closer to that goal than Meilisearch. Meilisearch is an indexing engine. Hister is already a complete document search application.

[5]

Yes. By default, Hister stores its data in SQLite:

Search index and metadata: db.sqlite3

Stored HTML previews: separate files on disk

Configuration: YAML

It can also use PostgreSQL instead of SQLite if you want.

Regarding browser history, there are two separate things:

1. Your browser’s history database (Chrome, Firefox, etc.).

2. Hister’s own index.

They are independent after import.

For example:

Monday: You visit example.com.

The extension sends the page to Hister.

Hister stores the text, title, URL, metadata, and optionally the HTML preview.

Tuesday: You clear Chrome’s history.

The Hister copy remains. Clearing your browser history does not delete Hister’s copy. Hister is designed to be a long-term knowledge store, not a mirror of your browser history.

If you want Hister to forget something, you must delete it from Hister itself. The web UI and terminal both support deleting indexed entries.

I actually like that design. If it deleted documents every time you cleared browser history, it would defeat one of its main purposes. It treats browser history as an ingestion source, not as the source of truth.

[6]

From the current documentation:

Are sources timestamped?

Yes.

Each indexed document records at least:

Added timestamp.

Updated timestamp.

For browser pages, the visit timestamp.

URL, title, and metadata.

Is there a TTL (auto-expiry)?

No.

I could not find any TTL, age-based expiry, or retention policy.

Documents remain until you delete them or overwrite them with newer content.

Can I set a maximum database size?

Not that I can find.

There is no option such as:

10 GB maximum.

Delete oldest first.

Keep only the last 365 days.

Limit to 1 million documents.

You can reduce storage by:

Disabling HTML previews.

Limiting indexed file size.

Restricting indexed file types.

Excluding paths or domains.

Can I cleanly delete by topic?

Partly.

You can:

Delete individual documents.

Delete crawl jobs (but not the indexed documents).

Label imported content.

Query by labels and metadata.

Configure rules to stop future indexing.

I did not find built-in support for bulk deletion such as:

“Delete everything about Tesla.”

“Delete all Reddit pages.”

“Delete every page tagged AI.”

The API may allow scripted deletion, but I did not see a first-class “delete by query” feature in the documentation.

Is browser ingestion automatic?

Yes.

That is one of its main features.

Once the browser extension is installed and pointed at your server:

Every page you visit is captured automatically.

The extension extracts the rendered page.

It sends it to Hister.

If the page later changes, the extension periodically checks it and updates the stored copy.

Duplicate visits update the existing document rather than creating a new one.

You do not need to press a button.

My only concern

This project feels like it is missing a lifecycle policy.

For a personal knowledge base, I would want options such as:

Keep browser pages for 90 days.

Keep GitHub forever.

Keep PDFs forever.

Delete pages that nobody has opened for two years.

Maximum database size: 20 GB.

Delete by label or query.

I could not find those features in the current documentation.

The project is very good at collecting information. It appears less mature at managing its lifetime. That would probably become the first thing I wanted if I intended to leave it running for years.

[7]

Yes, on both counts.

Can I exclude particular sites?

Yes, and it is quite flexible.

Hister has skip rules that apply before a page enters the index. You can exclude:

Entire domains.

Subdomains.

URL patterns.

Login pages.

Admin pages.

Cookie consent pages.

Anything that matches a regular expression.

For example:

^https://reddit//.com/ ^https://mail//.google/.com/ ^https://.*.bank.com/

Those pages never enter the index.

The nice part is that the rules apply both to new browser captures and to re-index operations, so they stay excluded.

How does multi-tenancy work?

It is better than I expected.

Each user has:

Their own login.

Their own search index.

Their own browser history.

Their own skip rules.

Their own search aliases.

Their own API token.

If you and your wife both use the same Hister server:

You ├── browser history ├── indexed pages ├── rules └── API token

Wife ├── browser history ├── indexed pages ├── rules └── API token

Neither user can search the other’s documents.

There is also a global area.

You can configure directories such as:

/shared/docs

Those appear in everyone’s searches, while personal folders remain private.

One feature I really like

Directory indexing also supports user ownership.

For example:

alice -> /nextcloud/alice bob -> /nextcloud/bob global -> /shared/docs

The server enforces that visibility automatically. Even administrators do not automatically see other users’ indexed files.

Overall, I came into this expecting “browser history search”. After reading the documentation, it is closer to a lightweight, self-hosted personal search appliance with sensible multi-user support. The remaining gaps are lifecycle management features such as retention policies, TTL, storage quotas, and richer bulk deletion. Those do not appear to be built in.

If just it was called hipster and utilized a rapper with backwards facing basecap and bling bling as logo

Lets see how it stacks up against the big ones