**Introduction**

The module *fts2* is a new version of the SQLite FullTextIndex module.  At this time, usage is identical to *fts1*.  The data storage has been radically changed in a way which greatly improves insertion speed, at minor cost to query performance.  *fts2* is in the SQLite CVS tree at =ext/fts2= (/ext/fts2/fts2.c).

In general, building, linking, and otherwise using *fts2* is identical to *fts1*, and will not be covered, here.

**Caveats**

There is *no* provision for backwards compatible with *fts1* data.  Data can be transferred from an *fts1* table to an *fts2* table by creating a new table using the new module, and doing an insert...select statement.

The *fts2* storage format is not yet to be considered final, and it is very likely to change multiple times in the near future.  Unfortunately, at this time there is no way to load old and new versions of *fts2* at the same time, so if the format does change, the only option is to do a =.dump=.

*Performance*

*fts1* used a scheme of segmenting tokenized data into per-term doclists which were stored in a =%_terms= table.  The segmentation allowed newer data to be stored in smaller doclists in order to control update costs.

*fts2* changes to a scheme where data is segmented into document groups, and updates occur by merging document groups together.  New documents form singleton segment, which is stored as a series of blobs, which is very cheap.  *fts1* required a read-modify-update pass on hundreds (or thousands) of term doclists, which became very expensive as the number of documents grew.  Segments are periodically merged together to moderate the query cost (which rises proportional to the number of segments).  Term data within a segment is grouped together and written in sorted fashion, making segment merges reasonably cheap.

In a test which loads the Enron email corpus (1.4G of data across 517,431 documents), *fts1* required 13.5 hours, while *fts2* required 35 minutes.  This was with pagesize=4096, synchronous=off, 100 inserts per transaction.

See the top-of-file comment in =fts2.c= for an in-depth description of how things work.