Skip to content

Conversation

@GenericMale
Copy link

@GenericMale GenericMale commented Jan 21, 2025

What does this PR do?

  • Adds Tavily Engine
  • Requires API key. Free Account allows 1000 requests per month.
  • Support for general search and news.
  • By default fills Answer Box with AI response.
  • If images are turned on (include_images: true), the infobox is used instead.
  • Supports time range
  • Allows filtering of analyzed domains (include_domains & exclude_domains)

How to test this PR locally?

  • Create a (free) account on tavily.com or hmu for a key
  • Enable engine in settings
  • !tav which engines does searxng support?

Examples

20250121_204524
Default Setting: Results & Answer

20250121_205214
include_images: true: Results & Infobox

@return42
Copy link
Member

I have dug into the https://app.tavily.com/playground request/response cycles to get the one request to the API (incl. the API key / don't know how long the key is valid for).

Here is an example for an request (copied from app.tavily.com/playground)

URL:

https://api.tavily.com/search

POST data:

{
  "query": "Which engines does SearXNG support?",
  "api_key": "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",
  "search_depth": "basic",
  "topic": "general",
  "include_answer": false,
  "include_raw_content": false,
  "include_images": false,
  "include_image_descriptions": false,
  "include_domains": [],
  "max_results": 5
}

and the response was:

{
  "query": "Which engines does SearXNG support?",
  "follow_up_questions": null,
  "answer": null,
  "images": [],
  "results": [
    {
      "title": "SearXNG - Wikipedia",
      "url": "https://en.wikipedia.org/wiki/SearXNG",
      "content": "The SearXNG results page with \"wikipedia\" as the search term SearXNG is a free and open-source federated metasearch engine forked from Searx.[1] SearXNG supports over 70 different search engines.[2] Similar to Searx, it does not collect information about users.[3] SearXNG features several customization options, which can be configured server-side or client-side, including filtering which engines to use, content filtering, and language preferences.[4] As a metasearch engine, SearXNG functions by sending queries to upstream search engines and returning them to the user. Engines include Google, Bing, DuckDuckGo, and Qwant.[5][7] More than 70 different search engines are supported.[2] \"Search in Peace with SearXNG: An Alternative Search Engine That Keeps Your Searches Private\".",
      "score": 0.9379027,
      "raw_content": null
    },
    {
      "title": "Configured Engines — SearXNG Documentation (2025.1.20+8731e3779)",
      "url": "https://docs.searxng.org/user/configured_engines.html",
      "content": "SearXNG supports 214 search engines of which 85 are enabled by default. Engines can be assigned to multiple categories. The UI displays the tabs that are configured in categories_as_tabs. In addition to these UI categories (also called tabs), engines can be queried by their name or the categories they belong to, by using a !bing syntax. tab",
      "score": 0.9145431,
      "raw_content": null
    },
    {
      "title": "Configured Engines — SearXNG Documentation (2023.4.3+fc18450f)",
      "url": "https://dalf.github.io/searxng/admin/engines/configured_engines.html",
      "content": "Configured Engines — SearXNG Documentation (2023.4.3+fc18450f) Engine Overview SearXNG supports 137 search engines (of which 60 are enabled by default). general search engines¶ | y | y | y | y | y | y | y |  |  | y | y | y | y | y | y |  |  | y | y | y | y | y | y | y | y | y | y | y | y | y | y | y | y | y | y | y images search engines¶ | google images | google_images videos search engines¶ news search engines¶ map search engines¶ music search engines¶ it search engines¶ science search engines¶ files search engines¶ Source-Code",
      "score": 0.9039154,
      "raw_content": null
    },
    {
      "title": "Engine Implementations — SearXNG Documentation (2025.1.17+a576f69c1)",
      "url": "https://docs.searxng.org/dev/engines/index.html",
      "content": "Engine Implementations — SearXNG Documentation (2025.1.21+e7081bb2c) Engine Implementations Engine Implementations¶ Engine Types Online Engines Offline Engines Engine Library Engine Engine traits SearXNG’s engines loader load_engine() load_engines() engine_shortcuts Engine Overview Engine Types¶ Demo Online Engine XPath Engine MediaWiki Engine JSON Engine Bing Engines Brave Engines DuckDuckGo Engines Google Engines Mwmbl Engine Peertube Engines Presearch Engine Recoll Engine Seekr Engines Startpage Engines Yahoo Engine SQL Engines Engine Implementations Engine Library SearXNG’s engines loader Engine Overview Engine Types Online Engines Demo Online Engine XPath Engine MediaWiki Engine JSON Engine Bing Engines Brave Engines DuckDuckGo Engines Google Engines Mwmbl Engine Peertube Engines Presearch Engine Recoll Engine Seekr Engines Startpage Engines Yahoo Engine Offline Engines SQL Engines Previous: How to contribute Next: Engine Library",
      "score": 0.84858394,
      "raw_content": null
    },
    {
      "title": "SearXNG project details - facts.dev",
      "url": "https://www.facts.dev/p/searxng/",
      "content": "Search Engines SearXNG SearXNG Internet metasearch engine which aggregates results from various search services and databases (Fork of Searx). SearXNG is a free, open-source internet metasearch engine that aggregates results from over 70 search services. Unlike other search engines, SearXNG does not track or profile users, providing complete anonymity and privacy protection. With features like self-hosted options, secure connections, and easy integration of search engines, SearXNG is an ideal solution for those seeking a private search experience. About 200 Search Engines: Aggregate results from a wide range of search services With SearXNG, you can enjoy a private and secure search experience while still benefiting from the power of aggregated search results. Tools for Developers, By Developers© 2024 facts.dev",
      "score": 0.77913743,
      "raw_content": null
    }
  ],
  "response_time": 2.01
}

@return42
Copy link
Member

AI-generated content is always a source of joy and inspires new questions .. a query about the !tav history of SearXNG:

grafik

associates the history of SearXNG with the image of an extensive drinking binge .. OK guys, then the next two questions; when did this take place .. and why wasn't I invited !?!?! 🥳

GenericMale and others added 3 commits January 22, 2025 14:13
- Config options like ``search_type`` renamed to follow the upstream
  API (``topic``).
- Default ``max_results`` is set to 5
- use image description if one exists
- add an init function to check engine's settings
- settings example: additional category 'ai'

To review the added documentation of this path::

    make docs.live

and jump to: http://0.0.0.0:8000/dev/engines/online/tavily.html

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
To get translations for, the missed CATEGORY_GROUPS has been added:

- ai
- movies
- translate
- wikimedia

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Copy link
Member

@return42 return42 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed two commits on top of this branch, for details, take a look at the commit messages.

Please run another test with my changes ... to see if the engine still meets your expectations after my changes.

Thanks a lot for this PR .. it was really an excellent work and a joy to review 👍

Copy link
Author

@GenericMale GenericMale left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the improvements, just two minor remarks

- add include_image_descriptions & include_answer to engine settings
- move [ai] prefix in results from title to content content
- minor doc fixes
@GenericMale
Copy link
Author

GenericMale commented Jan 23, 2025

@return42 maybe you can take a look again. basically everything should be configurable now and produce either an answer box, an infobox with or w/o image description or no answer/infobox at all (which almost makes the engine fast enough for regular use).

oh what i forgot to mention in the commit msg: i also changed the formatting of the "Image caption" text in the info box

Comment on lines 205 to 209
img_caption = f"<i>{gettext('Image caption')}</i>: {img_list[0]['description']}"
if not content:
result["content"] = img_caption
else:
result["content"] = f"{content}<br/>{img_caption}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not use HTML markups in the results .. "separation of content structure and layout"

  • "structure" is given by the results their types (I'm working on) and their order.

  • "layout" is done by the HTML templates & CSS

.. or in more practical terms: remember, we have also a JSON output.

But for now leave it as is .. I will come back and review this PR next week / BTW thanks for your additions and corrections 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants