Internet Archive imports often missing language

[cdrini]

Problem

Eg:

• https://openlibrary.org/books/OL43438350M/500_G%C3%A2teaux?_compare=Compare&b=4&a=3&m=diff
• https://openlibrary.org/books/OL43193978M/Acres_de_Diamantes?b=3&a=2&_compare=Compare&m=diff

Reproducing the bug

1. Trigger an import of an IA book

• Expected behavior: Language on IA is copied over to Open Library
• Actual behavior: Language is just missing

Context

• Browser (Chrome, Safari, Firefox, etc):
• OS (Windows, Mac, etc):
• Logged in (Y/N):
• Environment (prod, dev, local): prod

Instructions for Contributors

• Please run these commands to ensure your repository is up to date before creating a new branch to work on this issue and each time after pushing code to Github, because the pre-commit bot may add commits to your PRs upstream.

[NickDevi]

hello @cdrini I would like to work on this issue. Thank you!

[mekarpeles]

@NickDevi thank you for offering! I think first we'll have @scottbarnes investigate and provide next steps if necessary

[scottbarnes]

@NickDevi, sorry for the inexcusable delay. If you are still interested in this, I think this is what's going on.

1. Languages work for NEW imports via the /api/import/ia endpoint (see Direct Import by OCAID).
2. Languages ARE NOT added to EXISTING items when using that endpoint to supplement an existing record.
3. This is because 'languages' is not included in fields which are added to an existing record if the field is empty/absent in the existing record. See

   openlibrary/openlibrary/catalog/add_book/__init__.py

   Lines 808 to 814 in 1e7fce7

   	edition_list_fields = [
   	'local_id',
   	'lccn',
   	'lc_classifications',
   	'oclc_numbers',
   	'source_records',
   	]

4. HOWEVER, simply adding 'languages' there will cause an import error. Note: we here enter the realm of belief, where I may be mistaken. This seems to be because the language field is in the wrong format in rec at this point. We need it to look like [{"key": "/languages/eng"}], for example, but it seems to look like "languages": ["eng"]. That won't work.
5. The reason for this seems to be related to why this would work as an initial import (when there is no matching record) but not when matched to an existing record. That is I think because build_query() is not called when re-importing, but is called upon an initial import, and I think that's the only place the language field gets formatted. See

   openlibrary/openlibrary/catalog/add_book/load_book.py

   Lines 286 to 291 in 1e7fce7

   	if k in ('languages', 'translated_from'):
   	for language in v:
   	if web.ctx.site.get('/languages/' + language.lower()) is None:
   	raise InvalidLanguage(language.lower())
   	book[k] = [{'key': '/languages/' + language.lower()} for language in v]
   	continue

What this probably means is that:

1. 'languages' needs to be added to add_book/__init__.py as mentioned in (3) above, and;
2. the language formatting code in build_query() mentioned in (4) above needs to be broken out into an a helper function and called from both build_query() and update_edition_with_rec_data() (found in add_book/__init__.py).

I don't think any of this is necessarily hard in and of itself, but imports are a bit of a quagmire.

If you are interested in this, and I'm more than happy to do this if you think this is will involve too much wandering in the wilderness, you'd want to read:

1. https://github.com/internetarchive/openlibrary/wiki/Developer's-Guide-to-Data-Importing#api-authentication-for-imports (for learning how to set a cookie when using imports locally) and
2. https://github.com/internetarchive/openlibrary/wiki/Developer's-Guide-to-Data-Importing#direct-import-by-ocaid (for importing by OCAID/identifier, such as isbn_9781849353946.

And then test this all locally. This would get you started testing your code using curl:

$ curl -c cookies.txt -X POST "http://localhost:8080/account/login" -d "[email protected]&password=admin123"
$ curl -L -v -X POST "http://localhost:8080/api/import/ia" \
     -b ./cookies.txt \
     -d "identifier=isbn_9781849353946&require_marc=false&force_import=true"
...
{"authors": [{"key": "/authors/OL5A", "name": "Leslie Kaplan", "status": "created"}], "success": true, "edition": {"key": "/books/OL11M", "status": "created"}, "work": {"key": "/works/OL1W", "status": "created"}}%

That should import. If you remove the language (possibly using http://localhost:8080/books/OL11M.yml?m=edit, where OL11M is the edition identifier you got when importing), and then attempt to re-import the same record, it should match, but not add English back as a language, which is the problem and the reason this issue exists.

From there, the fun begins editing the Python code.

Again, please let me know if you're still interested. I would say because of the various parts here, this isn't a great first issue.

Edit: fix file names.

[qotkdrn]

Hello, my name is Alex Bae. I was wondering if I could contribute to this issue?

[scottbarnes]

@NickDevi, is this still of interest to you?

[mekarpeles]

@qotkdrn why don't you give it a shot if you'd like

[scottbarnes]

@qotkdrn, just to follow up, if you have not noticed, you should have a Slack invite in your email (possibly in a spam folder).

[qotkdrn]

I got it!

[qotkdrn]

I’m currently experiencing an issue with connecting to the Open Library server running on my local machine. When I attempt to make a POST request to http://localhost:8080/account/login using curl, I get the following error:

curl: (7) Failed to connect to localhost port 8080 after 8 ms: Couldn't connect to server

It was working when I ran it earlier

[scottbarnes]

Hmmm, I can't speak to the error, but ideally once the cookie is set, it need not be set again. Does the process work without that step?

[qotkdrn]

Never mind, sorry about that. It works now. I don't know why I couldn't connect to the server earlier despite having run the same commands. After importing the sample book, this is what my output looks like, but what should the output be? (base) ***@***.*** openlibrary % curl -L -v -X POST " http://localhost:8080/api/import/ia" \ -b ./cookies.txt \ -d "identifier=isbn_9781849353946&require_marc=false&force_import=true" Note: Unnecessary use of -X or --request, POST is already inferred. * Trying 127.0.0.1:8080... * Connected to localhost (127.0.0.1) port 8080 (#0)

POST /api/import/ia HTTP/1.1 Host: localhost:8080 User-Agent: curl/8.1.2 Accept: */* Cookie:

session=/people/openlibrary%2C2024-12-27T01%3A19%3A37%2Ca1ee5%24184e3f54fdecebe424fd176a4f2e1a26; pd=

Content-Length: 66 Content-Type: application/x-www-form-urlencoded

< HTTP/1.1 200 OK < Server: gunicorn < Date: Fri, 27 Dec 2024 01:20:16 GMT < Connection: close < Transfer-Encoding: chunked < Access-Control-Allow-Origin: * < Access-Control-Allow-Method: GET, OPTIONS < Access-Control-Max-Age: 86400 < Content-type: application/json < Content-Type: application/json < X-OL-Stats: "IB 13 0.168 MC 10 0.007 OT 1 0.284 TT 0 2.069" < * Closing connection 0 {"success": true, "edition": {"key": "/books/OL5M", "status": "matched"}, "work": {"key": "/works/OL1W", "status": "matched"}}%

…

On Thu, Dec 26, 2024 at 5:15 PM Scott Barnes ***@***.***> wrote: Hmmm, I can't speak to the error, but ideally once the cookie is set, it need not be set again. Does the process work without that step? — Reply to this email directly, view it on GitHub <#10141 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWFGWLHY2QEVGNKZW4EHNVD2HSS3RAVCNFSM6AAAAABTLXSGE6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRTGIYTCMZSGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[scottbarnes]

That looks correct. /books/OL5M is the edition, so if you go to http://localhost:8080/books/OL5M you should see it there. If you try to import it again at this point, the results should mention success and the result should once again match, as it did in the example you shared, as the edition and work had already been created.

If you go to http://localhost:8080/books/OL5M.yml?m=edit and remove the language, and import it again, what will happen (I imagine) is it will match again, and yet the language will continue to be absent. That is the thing we're trying to solve.

The goal is that if the edition already exists, as OL5M does here, then if there is no language present, and the import record has one (as it does in this case), then the language should be added.

[qotkdrn]

I made some changes and I think it works now? How do I proceed?

…

On Thu, Dec 26, 2024 at 8:11 PM Scott Barnes ***@***.***> wrote: That looks correct. /books/OL5M is the edition, so if you go to http://localhost:8080/books/OL5M you should see it there. If you try to import it again at this point, the results should mention success and the result should once again match, as it did in the example you shared, as the edition and work had already been created. If you go to http://localhost:8080/books/OL5M.yml?m=edit and remove the language, and import it again, what will happen (I imagine) is it will match again, and yet the language will continue to be absent. That is the thing we're trying to solve. The goal is to that if the edition already exists, as OL5M does here, then if there is no language present, and the import record has one (as it does in this case), then the language should be added. — Reply to this email directly, view it on GitHub <#10141 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWFGWLF6RV5ABEKN7A22UP32HTHO7AVCNFSM6AAAAABTLXSGE6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRTGI4TIMBQHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[scottbarnes]

Just open up a pull request and I will try to look at it tonight or over the weekend.

Please also include how you tested the pull request/fix, including before and after examples demonstrating it broken, and showing it fixed.

[qotkdrn]

I couldn't find my pr under pull requests and I noticed that my pr failed one of the python tests? Not quite sure what went wrong.

…

On Fri, Dec 27, 2024 at 3:51 PM Scott Barnes ***@***.***> wrote: Just open up a pull request <https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#making-changes-and-creating-a-pull-request> and I will try to look at it tonight or over the weekend. Please also include how you tested the pull request/fix, including before and after examples demonstrating it broken, and showing it fixed. — Reply to this email directly, view it on GitHub <#10141 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWFGWLAE75IITRHKXLASST32HXRWLAVCNFSM6AAAAABTLXSGE6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRUGA4TMOJVGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[el4ctr0n]

@cdrini Please note that page counts are also missing.

Problem

Eg:

• https://openlibrary.org/books/OL43438350M/500_G%C3%A2teaux?_compare=Compare&b=4&a=3&m=diff
• https://openlibrary.org/books/OL43193978M/Acres_de_Diamantes?b=3&a=2&_compare=Compare&m=diff

Reproducing the bug

1. Trigger an import of an IA book

• Expected behavior: Language on IA is copied over to Open Library
• Actual behavior: Language is just missing