*** 2,7 **** --- 2,27 ---- ---- + *Incremental merges* + + Except for level 0, when any level fills up we don't need to *immediately* merge it to the next level. We just need to have finished that merge before the next incoming merge hits. This means that when merging the 4kdoc segments into a 64kdoc segment, we can split it across 4k incremental steps. + + Of course, that's probably not really reasonable, it implies we're only processing a handful of terms per pass. Instead, we could have a setting describing how much effort per insert we're willing to spend doing bookkeeping. Then we could split a big merge across a couple dozen inserts (or a couple hundred when doing the 64k->1M merge). Or it might be reasonable to do it in something like sqrt(N) steps. So the the merge to a 64kdoc segment, 256 steps, to 1mdoc 1024 steps. + + ---- + + *Developer influence on segment merges* + + Right now, segment merges are always forced events. So while the average might be a couple hundred inserts per second, that average includes some inserts which themselves take many seconds. That kind of variance isn't really acceptable. + + One option would be an ability to tell the system to stop doing merges entirely until asked to continue. That has some pretty severe downsides, to my mind. + + An option I like better is to retain the current system as the baseline, but allow the developer to ask us to do preemptive merges. So the lower merge levels will pretty much work as they do now (they have less variance anyhow), but the higher-level merges might happen earlier than otherwise. That would mean that so long as the developer keeps calling us, things will remain very tight, but if they forget, things will remain adequate. + + Either way, this would probably manifest as a table function. Tuning parameters might also manifest as table functions. + + ---- + *Break doclists for a term across multiple leaf nodes* After loading in the first 64kdocs of the Enron dataset, position and offsets, the "enron" doclist is 3M. If this were extended to 1M docs, that could be expected to run to 50M. That's not acceptable.
Small. Fast. Reliable.
Choose any three.
Choose any three.
|
|