[Feature Request] Support for more complex HTTP requests in search_engines.js

[blacklight]

• Surfingkeys Version: 1.12
• Surfingkeys-conf Version (git hash): main HEAD
• Operating System: Any
• Browser Version: Any

Details

First of all, thanks for the amazing work you've put in this repo! It served me as a solid base to customize my SurfingKeys experience to the next level.

There's a feature I miss quite a bit though, and that's the ability to make HTTP requests other than GET in search_engines.js.

I'm not sure if I missed some polymorphic logic on completions.<search-engine>.search and completions.<search-engine>.compl, but it seems to me that it only supports string parameters (i.e. it only supports GET requests to the URL provided as a template).

That probably works in >90% of the cases, but recently I've implemented on my fork the support for ChatGPT results, and their API only supports POST for the queries.

My workaround has been to spin up my own little service that proxies GET requests to POST requests to the ChatGPT API, and then use that URL for the completions, but this definitely isn't scalable.

I also see it as a problem if somebody wants to implement search results from (x)RPC or GraphQL APIs, since those are likely to take POST/PUT requests.

I may work on a PR for it if there's enough interest. I have two possible approaches in mind:

1. compl and search can be either strings or objects containing the static configuration of an HTTP request to be passed to e.g. axios. Example:

completions.cg = {
  compl: {
    method: 'POST',
    url: 'https://api.openai.com/v1/completions',
    data: {
      prompt: '$QUERY$',  // I'm open on how to define the user query placeholder
      model: 'text-davinci-003',
      max_tokens: 2048,
    },
    headers: {
      'Content-Type': 'application/json',
      'Authorization': `Bearer ${conf.priv.chatgptKey}`,
    }
  },
  // ...
}

2. compl and search can be callbacks. Example:

completions.cg = {
  compl: async (query, conf) => {
    return await axios.post(
      'https://api.openai.com/v1/completions',
      {
        prompt: query,
        model: 'text-davinci-003',
        max_tokens: 2048,
      },
      {
        headers: {
          'Content-Type': 'application/json',
          'Authorization': `Bearer ${conf.priv.chatgptKey}`,
        }
      }
  }
  // ...
}

[b0o]

Hey!

I agree with you that lacking support for non-GET HTTP request methods is unfortunate. Currently, the search engines defined in search-engines.js are simply being registered with SurfingKeys via addSearchAlias, which unfortunately only supports GET requests. In order to implement support for all HTTP verbs, we'd need to add it in SurfingKeys itself.

When a GET request isn't enough, I have taken the same approach as you: spinning up a local webserver to proxy the requests. Some examples:

• The unicode search engine
  • Search engine definition: src/search-engines.js:409
  • Local web server handler: server/index.js:48
• The domainr search engine (uses an AWS Lambda function, but I plan to migrate it to server/index.js like the unicode search engine)
  • Search engine definition: src/search-engines.js:317
  • Lambda function: sk-proxy/app.js:9

I'd encourage you to submit a PR on the main SurfingKeys repo adding support for other HTTP verbs. I am a maintainer over there and would be happy to review your PR.

P.s. if you get a ChatGPT completion working, please feel free to open a PR here!

[blacklight]

Thanks for the quick response! Ok, I think I can work out on a PR for SurfingKeys. Which approach do you prefer between the two alternatives (static objects modeling HTTP requests vs. callbacks that take the user query as a parameter)?

[b0o]

I think I prefer the second approach, because it opens up the possibility to do other interesting things with the user’s query. For example, you could implement a simple calculator as a callback which never needs to hit the network.