Metadata class max size?

quimby 2017-04-28 03:17:10 UTC #1

Is there a maximum size for strings stored in metadata classes? If so, can we increase it?

I'm currently indexing a field that is truncated at ~247 chars.

gordongrace 2017-04-28 09:19:57 UTC #2

The Indexer Option you're looking for here would be mdsfml (MetaData Summary Field Maximum Length).

https://docs.funnelback.com/15.10/collections/collection-configuration/indexer_options_collection_cfg.html#g-setting-size-limits

There's a corresponding Query Processor Option (SBL - Summary Buffer Length) that controls how long those strings are when returned in the data model.

https://docs.funnelback.com/15.10/more/extra/padre_query_processor_options.html#f-presentation-options

quimby 2017-06-08 13:10:13 UTC #3

Thanks Gordon. Spot on the money as always. Option I needed was nearby, 'MBL', which sets the buffer length per metadata field. I think I needed this one because in this case I'm printing the metadata directly, not summaries.

I had to set mine to 2056 bytes. The default is 250, and there's this warning:

Warning: setting very large values will increase query processor memory demands and may cause problems.

Any idea what constitutes a 'very large value'? And what might happen?

P.S. Sorry took me so long to respond. Was an old problem I only got time to come back and work on just recently.

gordongrace 2017-06-09 08:16:17 UTC #4

Increasing the MBL value will increase the length for all metadata values being returned for each result. Given that there's a very large number of metadata fields available in v14+, and the num_ranks value doesn't have an upper limit (by default), you could request some very large responses from the query processor, even before they hit the templating layer.

Ideally, you'd want to do all you can to keep the search response packet as small (<200KB) and fast as possible (<200ms) - a multiplier effect kicks in, otherwise.

Search Response Size = baseline data model + Number of fields in SF (if SM is either both or meta) x MBL x num_ranks x avg. metadata field length

You may want a single-result query to return the entirety of a series of a very large metadata fields, for example:

num_ranks=1&SF=[description,content,title]&MBL=100000...