-
Notifications
You must be signed in to change notification settings - Fork 2.4k
[feat] engines: add tavily (AI powered) #4221
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
|
I have dug into the https://app.tavily.com/playground request/response cycles to get the one request to the API (incl. the API key / don't know how long the key is valid for). Here is an example for an request (copied from app.tavily.com/playground) URL: https://api.tavily.com/searchPOST {
"query": "Which engines does SearXNG support?",
"api_key": "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",
"search_depth": "basic",
"topic": "general",
"include_answer": false,
"include_raw_content": false,
"include_images": false,
"include_image_descriptions": false,
"include_domains": [],
"max_results": 5
}and the response was: {
"query": "Which engines does SearXNG support?",
"follow_up_questions": null,
"answer": null,
"images": [],
"results": [
{
"title": "SearXNG - Wikipedia",
"url": "https://en.wikipedia.org/wiki/SearXNG",
"content": "The SearXNG results page with \"wikipedia\" as the search term SearXNG is a free and open-source federated metasearch engine forked from Searx.[1] SearXNG supports over 70 different search engines.[2] Similar to Searx, it does not collect information about users.[3] SearXNG features several customization options, which can be configured server-side or client-side, including filtering which engines to use, content filtering, and language preferences.[4] As a metasearch engine, SearXNG functions by sending queries to upstream search engines and returning them to the user. Engines include Google, Bing, DuckDuckGo, and Qwant.[5][7] More than 70 different search engines are supported.[2] \"Search in Peace with SearXNG: An Alternative Search Engine That Keeps Your Searches Private\".",
"score": 0.9379027,
"raw_content": null
},
{
"title": "Configured Engines — SearXNG Documentation (2025.1.20+8731e3779)",
"url": "https://docs.searxng.org/user/configured_engines.html",
"content": "SearXNG supports 214 search engines of which 85 are enabled by default. Engines can be assigned to multiple categories. The UI displays the tabs that are configured in categories_as_tabs. In addition to these UI categories (also called tabs), engines can be queried by their name or the categories they belong to, by using a !bing syntax. tab",
"score": 0.9145431,
"raw_content": null
},
{
"title": "Configured Engines — SearXNG Documentation (2023.4.3+fc18450f)",
"url": "https://dalf.github.io/searxng/admin/engines/configured_engines.html",
"content": "Configured Engines — SearXNG Documentation (2023.4.3+fc18450f) Engine Overview SearXNG supports 137 search engines (of which 60 are enabled by default). general search engines¶ | y | y | y | y | y | y | y | | | y | y | y | y | y | y | | | y | y | y | y | y | y | y | y | y | y | y | y | y | y | y | y | y | y | y images search engines¶ | google images | google_images videos search engines¶ news search engines¶ map search engines¶ music search engines¶ it search engines¶ science search engines¶ files search engines¶ Source-Code",
"score": 0.9039154,
"raw_content": null
},
{
"title": "Engine Implementations — SearXNG Documentation (2025.1.17+a576f69c1)",
"url": "https://docs.searxng.org/dev/engines/index.html",
"content": "Engine Implementations — SearXNG Documentation (2025.1.21+e7081bb2c) Engine Implementations Engine Implementations¶ Engine Types Online Engines Offline Engines Engine Library Engine Engine traits SearXNG’s engines loader load_engine() load_engines() engine_shortcuts Engine Overview Engine Types¶ Demo Online Engine XPath Engine MediaWiki Engine JSON Engine Bing Engines Brave Engines DuckDuckGo Engines Google Engines Mwmbl Engine Peertube Engines Presearch Engine Recoll Engine Seekr Engines Startpage Engines Yahoo Engine SQL Engines Engine Implementations Engine Library SearXNG’s engines loader Engine Overview Engine Types Online Engines Demo Online Engine XPath Engine MediaWiki Engine JSON Engine Bing Engines Brave Engines DuckDuckGo Engines Google Engines Mwmbl Engine Peertube Engines Presearch Engine Recoll Engine Seekr Engines Startpage Engines Yahoo Engine Offline Engines SQL Engines Previous: How to contribute Next: Engine Library",
"score": 0.84858394,
"raw_content": null
},
{
"title": "SearXNG project details - facts.dev",
"url": "https://www.facts.dev/p/searxng/",
"content": "Search Engines SearXNG SearXNG Internet metasearch engine which aggregates results from various search services and databases (Fork of Searx). SearXNG is a free, open-source internet metasearch engine that aggregates results from over 70 search services. Unlike other search engines, SearXNG does not track or profile users, providing complete anonymity and privacy protection. With features like self-hosted options, secure connections, and easy integration of search engines, SearXNG is an ideal solution for those seeking a private search experience. About 200 Search Engines: Aggregate results from a wide range of search services With SearXNG, you can enjoy a private and secure search experience while still benefiting from the power of aggregated search results. Tools for Developers, By Developers© 2024 facts.dev",
"score": 0.77913743,
"raw_content": null
}
],
"response_time": 2.01
} |
|
AI-generated content is always a source of joy and inspires new questions .. a query about the associates the history of SearXNG with the image of an extensive drinking binge .. OK guys, then the next two questions; when did this take place .. and why wasn't I invited !?!?! 🥳 |
- Config options like ``search_type`` renamed to follow the upstream
API (``topic``).
- Default ``max_results`` is set to 5
- use image description if one exists
- add an init function to check engine's settings
- settings example: additional category 'ai'
To review the added documentation of this path::
make docs.live
and jump to: http://0.0.0.0:8000/dev/engines/online/tavily.html
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
To get translations for, the missed CATEGORY_GROUPS has been added: - ai - movies - translate - wikimedia Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
return42
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I pushed two commits on top of this branch, for details, take a look at the commit messages.
Please run another test with my changes ... to see if the engine still meets your expectations after my changes.
Thanks a lot for this PR .. it was really an excellent work and a joy to review 👍
GenericMale
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks for the improvements, just two minor remarks
- add include_image_descriptions & include_answer to engine settings - move [ai] prefix in results from title to content content - minor doc fixes
|
@return42 maybe you can take a look again. basically everything should be configurable now and produce either an answer box, an infobox with or w/o image description or no answer/infobox at all (which almost makes the engine fast enough for regular use). oh what i forgot to mention in the commit msg: i also changed the formatting of the "Image caption" text in the info box |
searx/engines/tavily.py
Outdated
| img_caption = f"<i>{gettext('Image caption')}</i>: {img_list[0]['description']}" | ||
| if not content: | ||
| result["content"] = img_caption | ||
| else: | ||
| result["content"] = f"{content}<br/>{img_caption}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do not use HTML markups in the results .. "separation of content structure and layout"
-
"structure" is given by the results their types (I'm working on) and their order.
-
"layout" is done by the HTML templates & CSS
.. or in more practical terms: remember, we have also a JSON output.
But for now leave it as is .. I will come back and review this PR next week / BTW thanks for your additions and corrections 👍

What does this PR do?
include_images: true), the infobox is used instead.include_domains&exclude_domains)How to test this PR locally?
!tav which engines does searxng support?Examples
Default Setting: Results & Answer
include_images: true: Results & Infobox