Authority search very poor

[RichardTaylor]

The search box at:

https://www.whatdotheyknow.com/select_authority

is particularly poor, it produces worse results than the general site search.

A search for cabinet office on the select authority page on WhatDoTheyKnow.com currently gives no results.

I suspect a number of the cases of reported problems logged at #1179 are not actually issues with the general site search, but with the select authority search.

See also: #4426

[garethrees]

#1179 is the the same mechanism and #4426 covers the most annoying specific issue, so closing this as this just covers the same ground.

[RichardTaylor]

I want to stress the /select_authority search produces different results to the search box in the header and at /search

the /select_authority search often produces no results when a public body name is entered into it.

Currently for example on WhatDoTheyKnow.com /select_authority has no results for cabinet office whereas the main search returns many and the Cabinet Office is second.

Many public body names result in no hits on the authority search eg
https://www.whatdotheyknow.com/select_authority?utf8=%E2%9C%93&query=cambridgeshire+county+council&bodies=1&commit=Search
https://www.whatdotheyknow.com/select_authority?utf8=%E2%9C%93&query=oxfordshire+county+council&bodies=1&commit=Search

Where there are results, putting bodies with many requests to the top (#40470 might help too.

[RichardTaylor]

Email to WhatDoTheyKnow today:

I am trying to make a new request to the Department of Health and Social Care. When I put that in the search box and click Search nothing happens.

They are correct, there are no results at:

https://www.whatdotheyknow.com/select_authority?utf8=%E2%9C%93&query=Department+of+Health+and+Social+Care&bodies=1&commit=Search

I'm going to reopen this, it's clearly a bug which is affecting many users who use a work-flow which takes them via the /select_authority page.

[garethrees]

Yeah okay something looks wrong here. Thanks for clarifying.

[garethrees]

Seems related notanumber/xapian-haystack#154

[RichardTaylor]

A WhatDoTheyKnow user has asked if we can make the results for searches on HS2 and High Speed 2 include the entry we have for High Speed Two (HS2) Limited, currently those terms don't result in a hit for the body.

Also related - where the /select_authority search has no hits there is no message to say "no results" it just appears to do nothing.

[RichardTaylor]

This isn't just an issue of case sensitivity, the search for Cabinet Office doesn't result in any hits irrespective of capitalisation.

Interestingly, when typing cabinet office into the search box letter by letter the instant results shown while typing include the cabinet office while the search term is cabi, cabin, cabine, cabinet, cabinet o, and cabinet of but not when further letters are added.

[garethrees]

This isn't just an issue of case sensitivity, the search for Cabinet Office doesn't result in any hits irrespective of capitalisation.

Yet "cabinet office" and "Cabinet Office" (quoted) do provide reasonable results.

Interestingly, when typing cabinet office into the search box letter by letter the instant results shown while typing include the cabinet office while the search term is cabi, cabin, cabine, cabinet, cabinet o, and cabinet of but not when further letters are added.

Yeah, notanumber/xapian-haystack#154 mentions issues to do with stemming which is related to this behaviour.

[garethrees]

Noting that I think we upgraded Xapian (or at least the underlying data format – can't remember) recently, and that I need to dig out the issue where this was discussed. (EDIT discussion: https://github.com/mysociety/sysadmin/issues/1305#issuecomment-630688979 / commit d54ba8e)

[gbp]

@garethrees I don't think this is doing a Xapian search. It looks like this is all happening in SQL via the PublicBody.with_query method.

[gbp]

Actually, I'm wrong I was on the "View authorities" action at /body/list/all. Guess this means we have 3 differnt ways of searching for an authority

[garethrees]

Yeah, /select_authority uses the xapian-based typeahead_search.

[gbp]

Also seems to be affecting the Pro batch authority search at /alaveteli_pro/batch_request_authority_searches

[gbp]

In production console:

irb(main):001:0> TypeaheadSearch.new('Cabinet Office', { model: PublicBody }).xapian_search
=> nil

Dev:

[1] pry(main)> TypeaheadSearch.new('Cabinet Office', { model: PublicBody }).xapian_search
=> #<ActsAsXapian::Search:0x00007f99d3f4dea8 snip>

[gbp]

This fix has now been deployed

[RichardTaylor]

Difficulty has been reported finding the body Transport for the North via the search at

https://www.whatdotheyknow.com/select_authority

When searching for

Transport for the North

the body with that name appears in 14th place in those authority search results, but it appears in 2nd place in the results of a research via the general site header.

There is a related recent support inbox thread, subject: "Add authority - Transport for the North"