[fix] ahmia engine: requires rotating tokens to work #5441
Conversation
Ahmia recently implemented a 60 minute rotating token system when searching. This fix uses the cache and updates the tokens on every request.
Bnyro
left a comment
There was a problem hiding this comment.
I'm always running into a ReplyError of the proxy after the timeout period of SearXNG is finished (even set it to 10 seconds), so probably something wrong on my configuration. (I simply ran tor and used the address it told me to use 🤷)
PS: Not related to your changes, but in case of a proxy error it crashes with
File "/home/username/Projects/searxng/searx/network/client.py", line 111, in handle_async_request
raise httpx.ProxyError("ProxyError: " + e.args[0], request=request) from e
~~~~~~~~~~~~~~~^~~~~~~~~~~
TypeError: can only concatenate str (not "ReplyError") to str
instead of actually displaying the proxy error
|
I want to test a bit more before coming back to this. It seems to be very slow, often just times out and it sometimes gives the proxy error as you mentioned that can be hard to reproduce sometimes. |
@Bnyro It could be that it differs from distro to distro. You can test if the tor proxy is working by running |
|
Although it is incredible slow, I cannot think of a better solution. The nature of the tor network is just slow afterall, and having to spend two requests sometimes does not make it better. I believe those who are interested in running tor-related engines on their SearXNG instances, are willing to accept how slow it is. I recommend setting up the timeout for Ahmia to 20 seconds. |
What does this PR do?
Ahmia recently implemented a 60 minute rotating token system when searching. If these tokens are not provided, it will redirect to the homepage, and SearXNG returning a parsing error.
The tokens update every minute, but Ahmia allow up to 1 hour old tokens to be used.
This implementation gets the keys from the hidden input using xpaths, and uses the
EngineCacheto prevent having to do 2 requests on every search. It will however search 2 times for the first initial request towards Ahmia, and also if tokens are older than 1 hour.I took inspiration from semantic scholar engine to set up the cache.
Why is this change important?
As of now, SearXNG will return a parsing error when trying to search Ahmia because it is returning a redirect to the homepage.
There is also very few tor-specific search engines on SearXNG, and feel it is important that this works.
How to test this PR locally?
!ah testfor example