Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add method field to CDXJ #107

Closed
wants to merge 1 commit into from
Closed

Add method field to CDXJ #107

wants to merge 1 commit into from

Conversation

edsu
Copy link

@edsu edsu commented May 17, 2022

Allow for a method field in the CDXJ, which can be included in pywb's CDXJ when a WARC record was created with a POST.

Fixes #106

Allow for a a `method` property in the CDXJ, which is occasionally
emitted by pywb.

Fixes nla#106
@edsu
Copy link
Author

edsu commented May 17, 2022

I guess this just parses it, but doesn't serialize the data to the db. Changing this to a draft for now.

@edsu edsu marked this pull request as draft May 17, 2022 12:07
@edsu
Copy link
Author

edsu commented May 17, 2022

@ato would adding method to the db require a new indexVersion (5)?

@ato
Copy link
Member

ato commented May 17, 2022

Yeah. New fields requires a new version number so that the reading code knows what to expect. The approach I was thinking about was to add an extra serialized field that stores a blob of json with all the extra fields (since as far as I know they could be arbitrarily complex json structures).

@steph-nb
Copy link

steph-nb commented Dec 1, 2022

you also have to add a requestBody field, no?

java.lang.IllegalArgumentException: no such capture field: requestBody
at outbackcdx.Capture.put(Capture.java:554)
at outbackcdx.Capture.fromCdxjLine(Capture.java:437)
at outbackcdx.Capture.fromCdxLine(Capture.java:388)
at outbackcdx.Webapp.post(Webapp.java:249)
at outbackcdx.Webapp.lambda$new$3(Webapp.java:102)
at outbackcdx.Web$Route.handle(Web.java:312)
at outbackcdx.Web$Router.handle(Web.java:236)
at outbackcdx.Webapp.handle(Webapp.java:594)
at outbackcdx.Web$Server.serve(Web.java:50)
at outbackcdx.NanoHTTPD$HTTPSession.execute(NanoHTTPD.java:848)
at outbackcdx.NanoHTTPD$1$1.run(NanoHTTPD.java:207)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

@ato
Copy link
Member

ato commented Jun 12, 2023

I'm going to close this since we now support arbitrary CDXJ fields (commit 9d73df3) which I think should cover this. Feel free to reopen if I'm wrong though!

@ato ato closed this Jun 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CDXJ: Error: no such capture field: method
3 participants