Push API - Content and Metadata Multipart Request

syerkes 2019-03-04 13:23:05 UTC #1

Hi!

I spent all weekend trying to get the POST /v2/collections/{collection}/documents/content-and-metadata endpoint to work in the Push API. Even reading through the docs on this endpoint multiple times I'm still lost on what goes in the request, metadata and content fields. I keep getting a "Could not parse multipart servlet request" exception or a 500 error.

I've successfully been able to PUT, DELETE and commit new docs to the collection, but it would be nice to not have to manually create the metadata mappings before hand each time. I believe this endpoint will do create mappings automatically, correct?

Any help would be awesome! Thanks,
Sam

gtran 2019-03-07 11:11:20 UTC #2

Hi Sam,

It might be easier to try using curl to test this endpoint as it lets you be very specific with the request.

I managed to get it working using the following steps:

Create the metadata file called metadata.json with the following content:

{
    "author":["Tom", "Ben"],
    "id":["123"]
}

Create the content html called content.html with the following content:

<html>
        <head></head>
        <body>blah blah</body>
</html>

Generate the authentication token using the methods found on https://community.funnelback.com/knowledge-base/system-adminstration/api-authentication-methods

Craft the curl request as shown on the API page:

curl -X POST 'https://<SERVER>:<PORT>/push-api/v2/collections/gt-test-push/documents/content-and-metadata?key=http://foo.com/' -H 'Content-Type: multipart/form-data' -H 'Authorization: Basic <security token>=' -F "metadata=@metadata.json; type=application/json " -F "content=@content.html; type=text/html" -i

The response will be something similar to the below:

HTTP/1.1 200 OK
Date: Thu, 07 Mar 2019 10:56:46 GMT
X-Security-Token: <removed for security>
Set-Cookie: remember-me=<removed for security>;Path=/;Expires=Mon, 11-Mar-2019 10:56:46 GMT;Max-Age=345600;Secure;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: DENY
Content-Length: 53
Server: Jetty(9.4.z-SNAPSHOT)

{"message":"Added!","storedKeys":["http://foo.com/"]}

Please note that metadata specified in this way still needs to be mapped using Funnelback metadata mappings. e.g. In my case, author and id metadata fields needs mapped to metadata classes before I am able to use them in Funnelback for things like faceting.

Hope this helps.

Thanks,

~Gioan

syerkes 2019-03-07 20:17:29 UTC #3

Hi @gtran, thanks for the response! I'll try this curl response again soon. I'll have to go through it closer at a later time to see what I was missing.

Follow up question, since metadata still needs to be mapped using the Funnelback mappings what's the advantage to using this endpoint over the PUT /v2/collections/{collection}/documents endpoint? Just the fact that you can use files over inline metadata? Apparently my assumption that this was a magic endpoint that mapped whatever metadata you threw at it automatically was incorrect.

LukeButters 2019-03-11 06:16:02 UTC #4

I think that defining the multipart body is not possible with that auto generated UI.

That is the only endpoint that lets you add metadata where each metadata has multiple values. The other end point for example wont let you define author as both 'Jack' and 'Jill'. Other limitations also exist with the non multi part end point as it is limited to what is supported in HTTP headers. I think that some clunky way of allowing new lines in http header values exist but not all clients support that so the multipart request might end up being your safest choice.

Is automatically mapping metadata something you need? How will you make use of that metadata?

syerkes 2019-03-11 12:18:21 UTC #5

@LukeButters, I see the value of the endpoint now. Thanks for pointing out the multiple values point.

I do find it a slightly cumbersome to go in and map metadata through the UI, especially when I have 5+ fields. I think it would be a neat idea to have metadata mapped automatically on a push collection at a special endpoint. I know you can use the admin part of the api to map metadata, so I may use that for now.

Thanks!