Spyglass athlabs

A personal search engine.

README

---

Create your library from:

• [x] Local documents/folders
• [x] Different internet topics (https://lenses.spyglass.fyi).
  • Lots of developer docs (Rustlang, Go, etc.)
  • Wikpedia, game wikis, etc.
• [x] Google Calendar events.
• [x] Google Drive docs.
• [x] GitHub repos, starred repos, & issues.
• [x] Reddit saved/upvoted posts.
• [ ] Gmail
• [ ] YouTube playlists & favorited.

Introduction

Spyglass lives on your device crawling & indexing websites you want with a basic set of rules.

Web pages when condensed down to text are surprisingly small. With todays' incredibly fast CPUs and ample amounts of of disk space, you can easily create a personal library of wikis, blog posts, etc. that can be referenced instantly. Cut through the SEO spam of the internet by building your own index.

For users who have been frustrated with the current state of search and the internet, Spyglass offers a powerful solution to find exactly what you want.

See Launching & Using Spyglass to get started.

Traditional web search sucks

The short answer is that Google search results are clearly dying. The long answer is that most of the web has become too inauthentic to trust.

• https://dkb.io/post/google-search-is-dying

Spyglass is a solution to the following common issues when searching the web:

• Do you add terms such as reddit or wiki to your searches to narrow it down?
• Do you get frustrated with overzealous autocorrect on your search terms?
• Do you get frustrated with the terrible search some wikis/sites offer?
• Do you scroll past dozens of SEO spam pages to find the recipe/review/blog post you were looking for?
• Do you skip over a full-page of ads before getting to your actual search results?
• Do you have private websites / data / documents that you'd like to search through?

How does it know what to crawl?

Spyglass expands on the ideas outlined in this paper by the Brave Search Team. There are currently a simple set of rules that will point Spyglass at a website and crawl only what you want. When available, crawling is bootstrapped w/ data from the Internet Archive to not overwhelm smaller websites.

For community lenses, we precrawl & preprocess these lenses so that you can get started searching through those topics immediately.

Not all websites & not all data can be crawled by Spyglass. If you have something that you'd like to index and would like some help, feel free to ping me on our Discord server!

See Community Lenses to install lenses others in the community have built.

See Building your own lens to see how easy it is to build your own lens. Please share w/ the community when you're done!

Developer Guide

If you'd like to help, reach out on our Discord server to see what is currently being developed and how you can help usher in a new, better search.

See Building from source to get started building & contributing to Spyglass.

TL;DR: If you want to build and run Spyglass from source, you can simply run this command:

cargo make run