As of now, TLGS only indexes textual content within Gemini. And it ignores all links to other protocols like HTTP and Gopher. It only parses and follows links withing `text/gemini` pages. Thus links within other textual pages are ignored. Though contents in `text/plain`, `text/markdown` and `text/x-rst` are still indexed.
Textual pages over 2.5MB are not indexed nor links within are followed. And we only fetch headers of binary files to make sure the file exists. Then it's URL is directly added to the index. If you see broken connections requesting binary files. That's likely TLGS checking if the file exists.
Please understand that TLGS allows manually excluding capsules from it's index. That the maintainer utilizes to prevent TLGS from crawling sites that causes issues with ranking or other functionalities. Please don't take it personally or think it is censorship if your content does not get indexed. TLGS although designed to be scalable and efficient, does run on fairly limited resources. We promise to keep working and make the indexing more efficient and resilient.
Currently especially contents of the following types is excluded:
With that said. We do index individual news mirror as long as they are not overly large.
Yes... but not really 100%. To provide better search result. After retrieving the page. TLGS removes ASCII arts and content separators from the page before indexing. There will be false positives in the detection despite our efforts. Causing some code segments (especially esoteric languages like BrainF**k) be removed from the index. Thus almost everything relevant will be indexed.
TLGS supports the simplified robots.txt specification for Gemini. To prevent crawling pages of your site, place a "robots.txt" file in your capsule's root directory. TLGS will fetch and parse the file before crawling your site. The file should have a content type of `text/plain` and is encoded in UTF-8, without BOM. The file is cached by TLGS for 7 days. TLGS will reuse the file within the caching period to reduce bandwidth consumption and speedup crawling.
TLGS follows the following user agents:
Note that TLGS's crawler does not support any of the following extended features common on the Web. And we only support the * and $ special characters.
You can't. Gemini does not provide a way for TLGS (or any client in that matter) to notify the server about the implementation and/or intent.
Unless manually triggered, the crawler considers a page static after first crawl. For pages that the crawler considers could be changing -- pages that looks like Gemsub, being atom, rss or twtxt feed, or at path / or ~username/ -- they are automatically reindexed every 7 days to discover new content.
This page is rendered from Gemini Gemtext to HTML. We recommened to get a proper Gemini client for the best experience.