bgcolor # Type Status Created By Subsys Changed Assigned Svr Pri Title _Description _Remarks #f2dcdc 2917 code active 2008 Feb anonymous 2008 Feb 4 4 Tcl interface - busy callback confusion script/procedure In the Tcl interface the "busy" method doesn't work if a script is supplied instead of a procedure: % package req sqlite3 3.5.1 % sqlite3 db hl7journal.db % db busy {puts busy; expr 0} % db eval {select count(*) from incoming} busy database is locked Here the callback script is only invoked once, even though it returns 0. If we put this code in a procedure it works as desired/expected: % proc b args {puts busy; after 1000; expr 0} % db busy b % db eval {select count(*) from incoming} busy busy busy ^C _2008-Feb-01 12:31:45 by anonymous:_ {linebreak} After researching this a little it appears this happens because the busy callback is invoked with an extra argument. The extra argument leads to an error but that error is only visible through errorInfo, not the result. I humbly suggest the following changes: * mention the extra argument in the documentation of the Tcl interface * forward the error from the busy callback to Tcl (replacing the "database is locked") * enhance errorInfo to make the invokation of the busy callback apparent. Currently, I'm getting this errorInfo: % db busy {puts hi; after 1000; return 0} % db eval "select count(*) from incoming" hi database is locked % set errorInfo bad option "0": must be -code, -errorcode, or -errorinfo while executing "return 0 0" invoked from within "db eval "select count(*) from incoming"" It would be nicer to get something like bad option "0": must be -code, -errorcode, or -errorinfo while executing "return 0 0" from busy callback "puts hi; after 1000; return 0 0" invoked from within "db eval "select count(*) from incoming"" #f2dcdc 2916 code active 2008 Feb anonymous 2008 Feb 1 1 sqlitedll-3_5_5.zip is older 3.5.4 binary sqlitedll-3_5_5.zip in download section is same with old 3.5.4 binary. _2008-Feb-01 12:13:04 by anonymous:_ {linebreak} Yes , I can confirm it #f2dcdc 2915 build active 2008 Feb pweilbacher 2008 Feb 4 3 fix Makefile for platforms that need .exe extension There are some targets that need a $(TEXE) added in Makefile.in. Otherwise a platform that needs a .exe extension cannot run e.g. tests. #f2dcdc 2914 code active 2008 Jan anonymous 2008 Jan 3 3 ATTACH returns SQLITE_ERROR when it means SQLITE_BUSY I'm seeing the same behavior as in #2096, with SQLite 3.5.4. ATTACH DATABASE fails with SQLITE_ERROR rather than SQLITE_BUSY when the database to be attached, or the main database of the connection being attached to, is EXCLUSIVE-locked by another database connection. For added confusion, sqlite3_errmsg() says "database is locked" when the ATTACH is done via sqlite3_exec(), but "SQL logic error or missing database" when the ATTACH is done via sqlite3_step(). As a result of this bug, it is difficult to distinguish between fatal and transient ATTACH errors, particularly when sqlite3_step() is used. I am attaching a test program that demonstrates the problem. #cfe8bd 2913 code fixed 2008 Jan anonymous 2008 Jan 3 3 vtab regression failures FYI: vtab6-2.2... Expected: [1 2 3 {} 2 3 4 1 3 4 5 2] Got: [1 2 3 {} 2 3 4 {} 3 4 5 {}] vtab6-2.3... Ok vtab6-2.4... Expected: [1 2 3 {} {} {} 2 3 4 {} {} {} 3 4 5 1 2 3] Got: [1 2 3 {} {} {} 2 3 4 {} {} {} 3 4 5 {} {} {}] vtab6-2.5... Expected: [2 3 4 {} {} {} 3 4 5 1 2 3] Got: [2 3 4 {} {} {} 3 4 5 {} {} {}] vtab6-2.6... Expected: [1 2 3 {} {} {} 2 3 4 {} {} {}] Got: [1 2 3 {} {} {} 2 3 4 {} {} {} 3 4 5 {} {} {}] vtab6-3.1... Ok vtab6-3.2... Ok vtab6-3.3... Ok vtab6-3.4... Ok vtab6-3.5... Ok vtab6-3.6... Ok vtab6-3.7... Ok vtab6-3.7... Ok vtab6-4.1... Ok vtab6-4.2... Ok vtab6-4.3... Ok vtab6-4.4... Ok vtab6-4.6... Ok vtab6-4.7... Ok vtab6-4.8... Ok vtab6-4.9... Ok vtab6-4.10... Ok vtab6-7.1... Expected: [1 999 999 2 131 130 999] Got: [1 999 999 999 999 999 999] vtab6-8.1... Ok vtab6-8.2... Ok vtab6-8.3... Ok vtab6-9.1... Ok vtab6-9.1.1... Expected: [] Got: [2 22 {}] vtab6-9.2... Expected: [] Got: [2 22 {}] vtab6-10.1... Ok #f2dcdc 2912 new active 2008 Jan anonymous 2008 Jan 5 4 merge join Would it be possible to implement merge joins? #f2dcdc 2911 code active 2008 Jan anonymous 2008 Jan 2 2 Adding parentheses to a FROM clause Hi, Parentheses in a FROM statement seem to mess with the ability to use table aliases in the "what" part. Here is an example: Start SQLite: $ sqlite3 employee.db SQLite version 3.5.4 Enter ".help" for instructions create a couple of tables and populate them with test data: sqlite> create table person (id integer, name text, employerid integer); sqlite> create table employer (id integer, name text); sqlite> insert into person (id, name, employerid) values (1, "Dave", 1); sqlite> insert into employer (id, name) values (1, "ACME"); Run a simple query with *no parentheses* in the FROM statement: sqlite> select b.id from person as a inner join employer as b on a.employerid = b.id; 1 Everything works as expected. Now, repeat that query *with parentheses*: sqlite> select b.id from (person as a inner join employer as b on a.employerid = b.id); SQL error: no such column: b.id There you have it. This may be related to ticket #1822, although that ticket deals with aliases and subqueries. This problem seems to be more fundamental. Many thanks, -- Dave #c8c8c8 2910 new closed 2008 Jan anonymous 2008 Jan 1 2 Number of rows in result set - new API Performance problem to retrieve number of records in result set after executed any query using prepare/step is evident. On DB around 200MB using one table with 15.000 records having around 10 fieds, among it's one blob field with data less than 10KB in each record, executed query with 'where' clausule using prepare/step (have no blob field listed) last around 30s to only count records in result set. SQLIte3_Get_Table need only 3s to returning all data as well. I I'm aware that many asserts may slow down the whole process dramatically, however not 10 times. Since there is no an API to retrieve number of records in result set and as prepare/step is far to slow, suggestion is to create one API, with maximum optimization for the purpose. _2008-Jan-31 02:00:07 by anonymous:_ {linebreak} To discover the number of rows, the VDBE engine should iterate over every record of the result. This is the same as calling sqlite_step() until EOF. But you can't do that, because after each sqlite_step(), you lost in memory data fields fetched for each row, so the way is counting rows after iteration. Also, tickets should be used for reporting bugs only. ---- _2008-Jan-31 04:24:29 by danielk1977:_ {linebreak} Internally, sqlite3_get_table() uses sqlite3_exec() which uses sqlite3_prepare()/step()/finalize(). So you should look for another reason why get_table() is 10 times faster than prepare()/step()/finalize(). You could also try a "SELECT count(*) ..." query to determine the expected size of a result set. As the above poster points out, there is no way to efficiently add this capability to SQLite (at least, no more efficiently than evaluating a count(*) query before the "real" query). Dan. ---- _2008-Jan-31 08:19:05 by anonymous:_ {linebreak} Anonymous: As allowed during ticket adding, I propose new feature. Dan: Since this feature should speed-up performance of a SQLite wrapper, "SELECT count(*) ..." is a real problem (require careful parsing to redesine gived query). "SELECT count(*) ..." itself is reasonably fast. ---- _2008-Jan-31 08:54:56 by anonymous:_ {linebreak} In addition to upper. "SELECT count(*)..." exactly last around 3s, Get_Table last as well around 3s, prepare/step/finalyze to only count numbers of records last 30s. This clearly indicate radically performance lost in prepare/step/finalyze. #f2dcdc 2909 build active 2008 Jan aswift 2008 Jan aswift 4 4 Omit the _XOPEN_SOURCE 500 define on Mac OS-X (check for __APPLE__) sqliteInt.h attempts to avoid setting _XOPEN_SOURCE on MacOSX by checking if __DARWIN__ is defined, however on Leopard (as of Mac OS X 10.5) this test fails and should be enhanced by testing for both __DARWIN__ and __APPLE__ Building sqlite succeeds with _XOPEN_SOURCE defined, but it causes _POSIX_C_SOURCE to be defined which leads to failing to define F_FULLFSYNC which prevents use of the fullfsync pragma. #f2dcdc 2908 code active 2008 Jan anonymous 2008 Jan 3 3 Add support to examine whether statements modify the database Currently there is no way to check whether a compiled statement will modify the database when being executed. Of course, there is the work-around of misusing the authorizer callback for this purpose, but this is kinda error prone and causes quite some overhead for such a simple purpose. #f2dcdc 2907 code active 2008 Jan anonymous 2008 Jan 1 1 Issues of sqlite3 with Windows Mobile 5/6 hi. we are currently using sqlite3 for our mobile application. it has been running without a hitch on pocket pc 2003 and previous versions. come windows mobile 5 and 6 we have been getting errors, although not consistent yet. one example is 'EXCEPTION_DATATYPE_MISALIGNMENT'. another is 'SELECT STATMENTS TO THE LEFT AND RIGHT OF UNION ARE NOT EQUAL'. i was wondering if you have any known compatibility issues of your product with this version of windows mobile. thanks in advance. _2008-Jan-28 13:26:26 by anonymous:_ {linebreak} EXCEPTION_DATATYPE_MISALIGNMENT is thrown when you try to use and Odd pointer address. I wrote a custom allocator for WinCE/ARM platform, and I have to take care about memory alignment (I used to align at 2 bytes, and at that time it solved the problem) #c8c8c8 2906 code closed 2008 Jan anonymous 2008 Jan anonymous 2 2 Error in MSVC v.6.0 (optimization: Maximize Speed mode) Hello, I am Vyacheslav Pindura (Kiev, Ukraine). I will detect error in MSVC v.6.0 (optimization: Maximize Speed mode). Look accessPayload() function in amalgamation project (the same in btree.c in separeted project). This is fragment from sqlite.cod (listing generated by MSVC compiler). Look my comment "//SLA: ..."
<<<<<< BEGIN LISTING ; COMDAT _accessPayload _TEXT SEGMENT _pCur$ = 8 _offset$ = 12 _amt$ = 16 _pBuf$ = 20 _skipKey$ = 24 _eOp$ = 28 _nKey$ = -12 _iIdx$ = -8 _pPage$ = -4 _pBt$ = -4 _ovflSize$62039 = 24 _nextPage$62040 = 12 _pDbPage$62056 = -12 _accessPayload PROC NEAR ; COMDAT ; 29385: ){ 00000 83 ec 0c sub esp, 12 ; 0000000cH 00003 53 push ebx 00004 55 push ebp 00005 56 push esi ; 29386: unsigned char *aPayload; ; 29387: int rc = SQLITE_OK; ; 29388: u32 nKey; ; 29389: int iIdx = 0; ; 29390: MemPage *pPage = pCur->pPage; /* Btree page of current cursor entry */ 00006 8b 74 24 1c mov esi, DWORD PTR _pCur$[esp+20] 0000a 57 push edi 0000b 33 db xor ebx, ebx 0000d 8b 7e 1c mov edi, DWORD PTR [esi+28] ; 29391: BtShared *pBt; /* Btree this cursor belongs to */ ; 29392: ; 29393: assert( pPage ); ; 29394: assert( pCur->eState==CURSOR_VALID ); ; 29395: assert( pCur->idx>=0 && pCur->idxSo, optimization imposible !!! We need disable optimization in MSVC project :( Best regards, Vyacheslav. _2008-Jan-27 12:51:32 by drh:_ {linebreak} See also ticket #2657. There have been other reports of problems with MSCV 6.0 and SQLite, some of which result in an "Internal Compiler Error" and others which just cause incorrect code to be generated. At this point we recommend that you *not* use MSVC 6.0 to build SQLite or any other project for that matter. If you must use MSVC 6.0, at least disable optimization. To be fair to MSVC, other compilers also have bugs. Ticket #2469 is an example of an optimizer bug in GCC which has since been fixed. Thank you for reporting this problem. But this really is a bug in MSVC, not in SQLite, so there is nothing that we can do about it. #f2dcdc 2905 code active 2008 Jan anonymous 2008 Jan pweilbacher 2 2 mutex_os2.c - incorrect mutex implementation The OS/2 version of sqlite3_mutex_alloc() is badly broken. It creates named mutexes which, by design, are global rather than process-specific as intended. This might be minimally acceptable except that the function reuses the same name every time it attempts to create a SQLITE_MUTEX_FAST or SQLITE_MUTEX_RECURSIVE. The result is that every call returns the exact same semaphore to every thread in every process using sqlite3. Once this mutex is owned by one process, other processes calling sqlite3_mutex_enter() will be blocked. Much the same is true for the static mutexes. Every process ends up using the exact same SQLITE_MUTEX_STATIC_MASTER, SQLITE_MUTEX_STATIC_MEM, etc. There's another flaw that is fairly minor compared with the above: in an attempt to avoid concurrency when creating the static mutexes, this function uses an API call that is thoroughly deprecated. The attached patch remedies all of these issues. Since the logic that protects the creation of the static mutexes may not be self-evident, here's an explanation: The existence (or non-existence) of a given named mutex is itself a semaphore. If the isInit flag is false, the code attempts to create a mutex whose name is unique to that process. If the attempt is successful, there are two possibilities: (1) either the current thread is the first to reach this code & may proceed; (2) or while the current thread was making its preparations, another thread created the mutex, did the init, then closed the mutex. Testing isInit immediately after creating the mutex determines which possibility is valid. If mutex creation fails due to a duplicate name, then another thread is currently performing the init. In this case, the current thread simply has to wait a while until the other thread is done & isInit becomes true. Submitted by Rich Walsh (richATe-vertise.com) #cfe8bd 2904 code fixed 2008 Jan anonymous 2008 Feb pweilbacher 3 3 os_os2.c - 2 dysfunctional functions The attached patch fixes os2Truncate() and os2FullPathname(). Currently, os2Truncate() does nothing useful. It moves the pointer associated with a file handle but has no effect on the file's size. The patch provides code that will change the actual size of a file on the disk. os2FullPathname() is overly complex and fails in marginal cases. E.g. for file "c:\programs\myapp\mydata.db", it fails to return a fully-qualified path when given "c:mydata.db". The patch replaces existing code with a single API call that always returns a valid result. Submitted by Rich Walsh (richATe-vertise.com) #f2dcdc 2903 code active 2008 Jan anonymous 2008 Jan 2 2 tclinstaller.tcl fails on path and permissions issue When compiling using custom PREFIX, pointing to private directory, tclinstaller.tcl fails, because it tries to remove contents from /usr/share/tcl8.4/sqlite3.nCell ); ; 29396: assert( offset>=0 ); ; 29397: assert( cursorHoldsMutex(pCur) ); ; 29398: ; 29399: getCellInfo(pCur); 00010 56 push esi 00011 89 5c 24 18 mov DWORD PTR _iIdx$[esp+32], ebx 00015 89 7c 24 1c mov DWORD PTR _pPage$[esp+32], edi 00019 e8 00 00 00 00 call _getCellInfo ; 29400: aPayload = pCur->info.pCell + pCur->info.nHeader; 0001e 8b 4e 24 mov ecx, DWORD PTR [esi+36] ; 29401: nKey = (pPage->intKey ? 0 : pCur->info.nKey); 00021 8a 47 03 mov al, BYTE PTR [edi+3] 00024 33 ed xor ebp, ebp 00026 83 c4 04 add esp, 4 00029 66 8b 6e 38 mov bp, WORD PTR [esi+56] 0002d 03 e9 add ebp, ecx 0002f 84 c0 test al, al 00031 74 08 je SHORT $L80642 00033 33 c0 xor eax, eax 00035 89 44 24 10 mov DWORD PTR _nKey$[esp+28], eax 00039 eb 0a jmp SHORT $L80643 $L80642: 0003b 8b 4e 2c mov ecx, DWORD PTR [esi+44] 0003e 8b 4e 28 mov ecx, DWORD PTR [esi+40] 00041 89 4c 24 10 mov DWORD PTR _nKey$[esp+28], ecx $L80643: ; 29402: ; 29403: if( skipKey ){ 00045 8b 4c 24 30 mov ecx, DWORD PTR _skipKey$[esp+24] //SLA: ecx:=skipKey ; 29404: offset += nKey; 00049 8b 7c 24 24 mov edi, DWORD PTR _offset$[esp+24] //SLA: edi:=offset 0004d 3b cb cmp ecx, ebx 0004f 74 02 je SHORT $L62032 00051 03 f9 add edi, ecx //SLA: offset += skipKey !!! $L62032: ; 29405: } ; 29406: if( offset+amt > nKey+pCur->info.nData ){ 00053 8b 54 24 28 mov edx, DWORD PTR _amt$[esp+24] 00057 8b 46 30 mov eax, DWORD PTR [esi+48] 0005a 03 44 24 10 add eax, DWORD PTR _nKey$[esp+28] 0005e 8d 0c 17 lea ecx, DWORD PTR [edi+edx] //SLA: edi have (offset+skipKey) !!! 00061 3b c8 cmp ecx, eax 00063 76 0d jbe SHORT $L62033 00065 5f pop edi 00066 5e pop esi 00067 5d pop ebp ; 29407: /* Trying to read or write past the end of the data is an error */ ; 29408: return SQLITE_ERROR; 00068 b8 01 00 00 00 mov eax, 1 0006d 5b pop ebx ; 29510: } <<<<<<< END LISTING
./configure --prefix=/my/private/sqlite/sqlite-3.5.4 ... # success make ... # success make install ... tclsh ./tclinstaller.tcl 3.5 error deleting "/usr/share/tcl8.4/sqlite3": not owner while executing "file delete -force $LIBDIR/sqlite3" (file "./tclinstaller.tcl" line 17) make: *** [tcl_install] Error 1I've found two work-arounds: 1: If you run make install as root. 2: If you use ./configure --disable-tcl _2008-Jan-28 17:47:39 by anonymous:_ {linebreak} I also ran into this problem. make install as root will end up copying files into the system's library directory and is almost certainly not what you want if you specified your own --prefix. #f2dcdc 2902 code active 2008 Jan anonymous 2008 Jan drh 3 3 Add watch support to SQLite SQLite currently provides only TRIGGERs and the update_hook() as a way for applications to stay informed about changes to the database. But both of these alternatives do not provide enough details about the actual changes to the underlying database file(s). We've prepared a patch for SQLite 3.5.x to allow applications to install a watch_hook into the database, that will be invoked everytime the database is changed with exact details about the change that was performed. _2008-Jan-22 16:06:59 by anonymous:_ {linebreak} Great idea and nice job. This functionality is very useful. ---- _2008-Jan-31 18:27:16 by anonymous:_ {linebreak} Any chance to get this committed for the next release (i.e. 3.5.6)? ---- _2008-Jan-31 19:38:35 by drh:_ {linebreak} Unlikely, for two reasons: 1: I am unconvinced that this patch solves a problem that needs solving. It is vitally important to a project like SQLite that we work to avoid clutter and cruft. That means that any change must have a compelling rational or else it is rejected. 2: The patches are against version 3.5.4. There were many changes to the core for 3.5.5 and the patches no longer work. ---- _2008-Jan-31 20:58:55 by anonymous:_ {linebreak} We can (and will) port the changes to 3.5.5, so the second point will be done. First the first point, I'm not sure how many projects will actually need this functionality, but I guess there are quite a lot of projects that would benefit, and for the others, there's zero overhead due to this patch. ---- _2008-Jan-31 21:21:30 by drh:_ {linebreak} There is a lot of overhead for me because if I accept this patch, that means I have to maintain it forever. Most of the work is in maintenance, not coming up with the original patch. #f2dcdc 2901 code active 2008 Jan anonymous 2008 Jan 3 3 ROLLBACK and COMMIT statements should not expire Currently, whenever a statement changes the schema of the database, all prepared statements will be expired, no matter whether they actually need to be prepared again or not. This is especially problematic for ROLLBACK statements in a multi-statement transaction. Currently there is no way to guaranty that a multi-statement transaction can at least be rolled back in case of an error, because one has to (re)prepare the ROLLBACK statement to roll back the transaction, which can fail because of OOM (in a multi-threaded application). #c8c8c8 2900 code closed 2008 Jan anonymous 2008 Jan 3 4 Can't ESCAPE in GLOB If you apply an ESCAPE clause to the GLOB operator, you get the message: SQL error: wrong number of arguments to function glob() You can reproduce this on Mac OS X 10.5 with the following:
$ sqlite3 test.db SQLite version 3.4.0 Enter ".help" for instructions sqlite> create table Identifiers ( ident text ); sqlite> insert into Identifiers values ( "foo?" ); sqlite> insert into Identifiers values ( "foo" ); sqlite> insert into Identifiers values ( "foo_" ); sqlite> select * from Identifiers where ident like "foo\_" escape "\"; foo_ sqlite> select * from Identifiers where ident glob "foo\?" escape "?"; SQL error: wrong number of arguments to function glob()I believe that this is because sqlite3RegisterLikeFunctions (in "func.c") registers the "like" function with either 2 or 3 arguments, but only registers the "glob" function with 2 arguments. _2008-Jan-22 13:04:28 by drh:_ {linebreak} Escape characters in a GLOB as you would in a shell glob, which is to put them inside of a [...]. Example: select * from identifiers where ident glob 'foo[?]'; Note also the use of single-quotes instead of double-quotes around the string literal. SQL requires single-quotes. #f2dcdc 2899 code active 2008 Jan anonymous 2008 Jan 4 4 sqlite3_reset() after exec() takes > 100 ms to complete I'm not entirely sure whether this is a bug or not... I'm not familiar enough with SQLite to know if this is way too unconventional, but I noticed today that running exec() in conjunction with a prepared statement really kills the performance of sqlite3_reset(), if it's called after exec(): // init sqlite3_prepare_v2( db,{linebreak} "SELECT [file_id],[file_name],[file_mime],[file_type],"{linebreak} "[file_size],[date_created],[date_accessed],[data] "{linebreak} "FROM file_cache WHERE [file_id] = ? LIMIT 1;",{linebreak} -1, &sqQueryStatement, &unused ); // query function -- called repeatedly (usually second or third run starts to cause big delays) /* begin function */ sqlite3_reset( sqQueryStatement );{linebreak} sqlite3_bind_int( sqQuerystatement, 1, 292 );{linebreak} sqlite3_step( sqQueryStatement );{linebreak} bla = sqlite3_column_int( sqQueryStatement, 0 ); /* ... */ char *erm; sqlite3_exec( db, "UPDATE file_cache SET [date_accessed] = DATETIME( 'NOW', 'LOCALTIME' ) WHERE [file_id] = 292", NULL, NULL, &erm ); /* ... */ sqlite3_clear_bindings( sqQueryStatement ); /* end function */ If sqlite3_exec() is commented out, consecutive calls to the function run in less than a millisecond. With sqlite3_exec() included, sqlite3_reset() call in this function takes > 100 ms to complete. Tested: Windows Vista Ultimate 32 bit, Visual Studio 2005 8.0.50727.876 #f2dcdc 2898 code active 2008 Jan anonymous 2008 Jan 1 1 Latest CVS for 3.5.4 fails to build test1.c gcc -pipe -O3 -g -Wall -DSQLITE_DISABLE_DIRSYNC=1 -I. -I../src -DNDEBUG -I/usr/include -DSQLITE_THREADSAFE=1 -DSQLITE_THREAD_OVERRIDE_LOCK=1 -DSQLITE_OMIT_LOAD_EXTENSION=1 -DTCLSH=1 -DSQLITE_TEST=1 -DSQLITE_CRASH_TEST=1 -DSQLITE_NO_SYNC=1 -DTEMP_STORE=1 -o .libs/testfixture ../src/attach.c ../src/btree.c ../src/build.c ../src/date.c ../src/expr.c ../src/func.c ../src/insert.c ../src/malloc.c ../src/os.c ../src/os_os2.c ../src/os_unix.c ../src/os_win.c ../src/pager.c ../src/pragma.c ../src/prepare.c ../src/printf.c ../src/select.c ../src/test1.c ../src/test2.c ../src/test3.c ../src/test4.c ../src/test5.c ../src/test6.c ../src/test7.c ../src/test8.c ../src/test9.c ../src/test_autoext.c ../src/test_async.c ../src/test_btree.c ../src/test_config.c ../src/test_hexio.c ../src/test_malloc.c ../src/test_md5.c ../src/test_onefile.c ../src/test_schema.c ../src/test_server.c ../src/test_tclvar.c ../src/test_thread.c ../src/tokenize.c ../src/utf.c ../src/util.c ../src/vdbe.c ../src/vdbeapi.c ../src/vdbeaux.c ../src/vdbemem.c ../src/where.c parse.c ../src/tclsqlite.c ./.libs/libsqlite3.so -L/usr/lib64 -ltcl8.4 -ldl -lpthread -lieee -lm -Wl,--rpath -Wl,/common/pkgs/sqlite-3.5.4.3/lib ../src/build.c: In function 'sqlite3RefillIndex': ../src/build.c:2275: warning: cast to pointer from integer of different size ../src/func.c: In function 'trimFunc': ../src/func.c:919: warning: cast from pointer to integer of different size ../src/func.c: In function 'sqlite3RegisterBuiltinFunctions': ../src/func.c:1464: warning: cast to pointer from integer of different size ../src/func.c:1483: warning: cast to pointer from integer of different size ../src/insert.c: In function 'sqlite3GenerateConstraintChecks': ../src/insert.c:1200: warning: cast to pointer from integer of different size ../src/insert.c:1034: warning: 'j2' may be used uninitialized in this function ../src/insert.c: In function 'sqlite3Insert': ../src/insert.c:373: warning: 'regFromSelect' may be used uninitialized in this function ../src/test1.c: In function 'test_collate_func': ../src/test1.c:2085: warning: cast from pointer to integer of different size ../src/test1.c: In function 'test_collate_needed_cb': ../src/test1.c:2209: warning: cast to pointer from integer of different size ../src/test1.c: In function 'alignmentCollFunc': ../src/test1.c:2258: warning: cast from pointer to integer of different size ../src/test1.c:2259: warning: cast from pointer to integer of different size ../src/test8.c: In function 'echoBestIndex': ../src/test8.c:722: warning: 'nRow' may be used uninitialized in this function ../src/vdbe.c: In function 'sqlite3VdbeExec': ../src/vdbe.c:502: warning: 'pOut' may be used uninitialized in this function ../src/vdbe.c:501: warning: 'pIn3' may be used uninitialized in this function ../src/vdbe.c:501: warning: 'pIn2' may be used uninitialized in this function ../src/vdbe.c:501: warning: 'pIn1' may be used uninitialized in this function ../src/vdbeaux.c: In function 'sqlite3VdbeChangeP4': ../src/vdbeaux.c:529: warning: cast from pointer to integer of different size ../src/vdbemem.c: In function 'sqlite3ValueText': ../src/vdbemem.c:911: warning: cast from pointer to integer of different size /tmp/ccsuOeus.o: In function `reset_prng_state': /build/work/sqlite-3.5.4.3/bld/../src/test1.c:4280: undefined reference to `sqlite3ResetPrngState' /tmp/ccsuOeus.o: In function `restore_prng_state': /build/work/sqlite-3.5.4.3/bld/../src/test1.c:4267: undefined reference to `sqlite3RestorePrngState' /tmp/ccsuOeus.o: In function `save_prng_state': /build/work/sqlite-3.5.4.3/bld/../src/test1.c:4254: undefined reference to `sqlite3SavePrngState' collect2: ld returned 1 exit status make: *** [testfixture] Error 1 _2008-Jan-17 23:54:58 by anonymous:_ {linebreak} Problem appears to be here in libsqlite.3.so.0.8.6 as shown by: nm -A .libs/libsqlite3.so.0.8.6 | grep sqlite3ResetPrngState which shows no entry point. And: nm -A .libs/random.o | grep sqlite3ResetPrngState which also shows no entry point. ---- _2008-Jan-17 23:56:55 by anonymous:_ {linebreak} Ah... It appears -DSQLITE_TEST should be passed when building test1.c and left off when building prior to install. ---- _2008-Jan-21 20:16:00 by anonymous:_ {linebreak} In the makefile the right flag appears to be set, it's just not making it through to the compile for some reason. ---- _2008-Jan-21 20:16:24 by anonymous:_ {linebreak} Still fails the same based on today's cvs update. ---- _2008-Jan-23 03:14:49 by anonymous:_ {linebreak} This bug fixed as of latest cvs pull #f2dcdc 2897 code active 2008 Jan anonymous 2008 Jan 1 1 String or BLOB exceed size limit This error was shown after attemp to read script from SQLite 3.5.4 shell in order to recreate old DB. Details: 1. Database was created with SQLite 3.3.4. Around 20 standard fieds and one BLOB. 2. The only one existed table was dumped with shell of SQLite 3.5.4. SQL script seems to be coorrect. 3. Opened SQLite 3.5.4 and read script in new DB. The error "String or BLOB exceed size limit" are sown for several lines. Many records missing. 4. Attempted to dump table with shell of version 3.3.6 (have no more 3.3.4 shell) and read into new DB with 3.5.4 shell The same errors are shown. The same steps was attempted with 3.3.6. shell only. All seems to be correct. _2008-Jan-17 20:23:25 by drh:_ {linebreak} This size limit on BLOBs in SQLite version 3.5.4 is 1GB. How big is your blob, exactly? ---- _2008-Jan-17 22:22:24 by anonymous:_ {linebreak} BLOB in each record is no more than few MB. Mostly it is few KB (e-client and news application). Whole DB have around 200MB. ---- _2008-Jan-18 02:28:11 by drh:_ {linebreak} This issue is probably resolved by check-in [4636], then. ---- _2008-Jan-18 14:28:13 by anonymous:_ {linebreak} If directive SQLITE_MAX_SQL_LENGTH is not defined it is set to 1,000,000 (10^6) in amalgamation code of 3.5.4. #c8c8c8 2896 code closed 2008 Jan anonymous 2008 Jan 1 1 FTS3 misses document, but finds non-matching ones instead The following script presents a severe bug in FTS3 where it does not find a document even though it is in the index. Instead it finds other documents which do not match the search criteria. The log below was generated by the original SQLite3.exe 3.5.4 downloaded from the web-site. It shows that the entry 'A-123' is present in the table but is not matched by the query. It also demonstrates that non-matching entries are found instead. DROP TABLE IF EXISTS fts3; CREATE VIRTUAL TABLE fts3 using fts3 (a); INSERT INTO fts3 VALUES ('A-123'); SELECT * FROM fts3; A-123 SELECT * FROM fts3 WHERE fts3 MATCH 'A-123'; INSERT INTO fts3 VALUES ('A-12'); SELECT * FROM fts3 WHERE fts3 MATCH 'A-123'; A-12 INSERT INTO fts3 VALUES ('A-1'); SELECT * FROM fts3 WHERE fts3 MATCH 'A-123'; A-12 A-1 Here is the SQL only (without output, for easy reproduction): DROP TABLE IF EXISTS fts3; CREATE VIRTUAL TABLE fts3 using fts3 (a); INSERT INTO fts3 VALUES ('A-123'); SELECT * FROM fts3; SELECT * FROM fts3 WHERE fts3 MATCH 'A-123'; INSERT INTO fts3 VALUES ('A-12'); SELECT * FROM fts3 WHERE fts3 MATCH 'A-123'; INSERT INTO fts3 VALUES ('A-1'); SELECT * FROM fts3 WHERE fts3 MATCH 'A-123'; _2008-Jan-17 19:36:18 by anonymous:_ {linebreak} Sorry, I must have been dreaming: This is of course *not a bug* but the expected behavior. The '-' character is supposed to exclude the word following it. My sincere apologies if this has caused doubt or uncertainties to anyone! #c8c8c8 2895 build closed 2008 Jan anonymous 2008 Jan 3 1 Please describe how to run an indvidual test out of the test suite Please describe how to run an individual test out of the test suite? Thanks. For example, how to run the test incrvacuum-ioerr-1.31.4 _2008-Jan-17 17:14:12 by drh:_ {linebreak} Tickets are for reporting bugs, not asking questions about how to compile or operate SQLite. For questions of that nature, please post comments on the SQLite mailing list. See http://www.sqlite.org/support.html. PS: You cannot run individual tests. You have to run an entire script. #cfe8bd 2894 code fixed 2008 Jan anonymous 2008 Jan 1 1 virtual table calls Next() and Column() after Eof() returns true I saw this problem in my virtual table implementation. When I had a query that used a left outer join on a column that could be NULL, Next() and Column() were called after Eof() had returned true. I tried to reproduce the problem using test_schema from the source distribution which resulted in a segmentation fault, and I assume it's the same defect. Here is a sample script: .load test_schema.so create virtual table x using schema; -- this produces results select * from x where dflt_value is null; -- this produces a segmentation fault select * from x lhs left outer join x rhs on lhs.dflt_value = rhs.dflt_value; #f2dcdc 2893 code active 2008 Jan anonymous 2008 Jan 1 1 incorrect integer range tests recently a function that performs integer range tests was added to the cvs (check-in [4706]), but if i am correct there is a problem in the return value of the function in the file vdbemem.c: static i64 doubleToInt64(double r){ ... if( r<(double)minInt ){ return minInt; }else if( r>(double)maxInt ){ return minInt; <-- is this correct, shouldn't it be maxInt? }else{ return (i64)r; } } _2008-Jan-16 17:33:56 by drh:_ {linebreak} See the remarks on ticket #2280. The code duplicates the behavior of the FPU on x86. ---- _2008-Jan-16 18:21:28 by anonymous:_ {linebreak} did you mean ticket #2880? didn't read that ticket before, but since there was no comment regarding that behavior in the function it seemed (to my eyes) that it was a mistake. maybe adding a small comment in there would clarify this issue ---- _2008-Jan-16 18:39:42 by anonymous:_ {linebreak} Just because the double to int overflow behavior happens to be that way with GCC on x86, is it desirable? #f2dcdc 2892 new active 2008 Jan anonymous 2008 Jan 5 5 There should be a way in the api to read more precise error message. Currently all errors in sql queries issued via C,C++ api result in error code SQLITE_ERROR = 1 and message "SQL error or missing database". That leaves user completely clueless what mistake in his sql he actually made. Some evidence of confusion can be found here: http://bugs.php.net/bug.php?id=33117 _2008-Jan-16 17:41:00 by drh:_ {linebreak} SQLite gives detailed error information for SQL syntax or logic errors. (Try, for example, entering invalid SQL into the CLI.) I think perhaps that PHP is simply failing to to access those errors and is instead picking up some other error indication from someplace else. ---- _2008-Jan-16 21:59:17 by anonymous:_ {linebreak} Well I shall retest it, but I got the same error message upon a duplicate key in Delphi. Although I'm aware of an enhanced api to get the errormessage. ---- _2008-Jan-17 00:05:58 by drh:_ {linebreak} What error message do you get from the CLI? #c8c8c8 2891 code closed 2008 Jan anonymous 2008 Jan 2 3 col with "NOT NULL", no default: works on table create, not col add 1: WORKS: CREATE TABLE my_table ("my_column" integer NOT NULL); 2: GIVES ERROR: ALTER TABLE my_table ADD "my_other_column" integer NOT NULL; (2) gives the error: "SQL error: Cannot add a NOT NULL column with default value NULL" This is inconsistent, either both cases should work, or both should give an error. _2008-Jan-15 19:59:03 by anonymous:_ {linebreak} This is default in all SQL database managers. If a column can't have a default NULL value, how it will be added to a table with existing rows ? How that columns will have the default column value is there´s no default and has the NOT NULL constraint ? ---- _2008-Jan-16 00:15:50 by drh:_ {linebreak} Anonymous poster above is correct. This is not a bug. #c8c8c8 2890 build closed 2008 Jan anonymous 2008 Jan 1 1 Assistance request fixing six 3.5.4 i/o test failures Failures include exclusive-ioerr-2.280.4 exclusive-ioerr-2.281.4 exclusive-ioerr-2.282.4 incrvacuum-ioerr-1.31.4 io-4.1 io-4.2.3 Please let me know how to run individual tests, so I can debug these and submit fix suggestions. Thanks. _2008-Jan-15 21:50:52 by drh:_ {linebreak} These errors do not appear when the tests are run individually. They are only reproducible when you run the complete test suite. ---- _2008-Jan-16 03:15:50 by anonymous:_ {linebreak} Please give an example of how to run one individual test. Thanks. #c8c8c8 2889 build closed 2008 Jan anonymous 2008 Jan 3 2 Cannot disable TCL Slackware 12 TCL/TK is not installed, as I don't need it, and don't really want to install it Sequence of commands to reproduce : tar -xzvf sqlite-3.5.4.tar.gz cd sqlite-3.5.4 mkdir bld cd bld ../configure --disable-tcl --without-tcl make The last command dies with --------------------------------------------------- rm tsrc/sqlite.h.in tsrc/parse.y cp parse.c opcodes.c keywordhash.h tsrc tclsh ../tool/mksqlite3c.tcl make: tclsh: Command not found make: *** [sqlite3.c] Error 127 root@slack12:~/install/sqlite-3.5.4/bld# ls .. --------------------------------------------------- Thanks in advance Roman Same as #2845, #2346, #2871. #c8c8c8 2888 code closed 2008 Jan anonymous 2008 Jan 3 4 statement not commited code: #include
/* ** The MEM structure is already a MEM_Real. Try to also make it a ** MEM_Int if we can. */ SQLITE_PRIVATE void sqlite3VdbeIntegerAffinity(Mem *pMem){ assert( pMem->flags & MEM_Real ); assert( pMem->db==0 || sqlite3_mutex_held(pMem->db->mutex) ); pMem->u.i = pMem->r; // <- This causing termination for values >= 1e19 if( ((double)pMem->u.i)==pMem->r ){ pMem->flags |= MEM_Int; } }This conversion line causing BCC 5.5.1 run-time to raise a conversion error if values are >= 1e19. It is necessary a clean way to convert double to integer/int64. If nothing else, this will solve the problem (code is not essential for correct functionality):
/* ** The MEM structure is already a MEM_Real. Try to also make it a ** MEM_Int if we can. */ SQLITE_PRIVATE void sqlite3VdbeIntegerAffinity(Mem *pMem){ assert( pMem->flags & MEM_Real ); assert( pMem->db==0 || sqlite3_mutex_held(pMem->db->mutex) ); #ifndef __BORLANDC__ pMem->u.i = pMem->r; if( ((double)pMem->u.i)==pMem->r ){ pMem->flags |= MEM_Int; } #endif }[ Bijgewerkt do jan 10 2008, 08:49 ] SZUTILS - link ---- _2008-Jan-11 10:23:37 by anonymous:_ {linebreak} In "sqlite - Check-in [4705]" you have tryed to "attempt to work around this bug in the Borland compiler + ** by moving the value into a temporary variable first so that if + ** the assignment into the integer really does corrupt the right-hand + ** side value, it will corrupt a temporary variable that we do not + ** care about." First of all, thank you for attempt to solve the issue. I will first write several facts: 1. Statically linked SQLite 3.5.4 commandline shell (aka sqlite3.exe) is compiled and SQL script worked fine on Windows with MinGW 3.2.3, however not with BCC 5.5.1 (without any special compiler options). 2. Library (aka sqlite3.dll) compiled with MinGW 2.95 (from official SQLite site), MinGW 3.2.3, BCC 5.5.1 and probably other compilers for Windows does FAIL executing provided SQL script only with large floats >=1e19. With MinGW 3.2.3, library was created from official SQLite 3.5.4 distribution with following script: sh configure make dll 3. Both (integer and double) have 8 byte size in BCC. BCC is ANSI C complient compiler. 4. It is not the fact that BCC corrupt right-hand side value. It fail trying to convert values >=1e19 and assign it to int64 variable. This will raise invalid floating-point operation exception for most C compilers. 5. Since front-end application use only DLL (compiled by several different C compilers, including provided from offical SQLite site) to execute an SQL command by sqlite_exec(), it is nothing which connect the bug and the application itself (made in Delphi or C). Current attempt in 4705 will not work from two reasons: 1. The same conversion logic is used. 2. Variable definition in the middle of the function in not allowed by ANSI C. This will fail not only on BCC, but on any ANSI C compiler. According to upper, conclusion is clear that this is not a bug in BCC, but in all C compiles for Windows - or in conversion double to integer currently used. Thank you for your time. Sasa Zeman ---- _2008-Jan-11 13:22:27 by drh:_ {linebreak} The purpose of the sqlite3VdbeIntegerAffinity() function is to convert a floating point value into a 64-bit integer value if and only if that conversion can be done without loss of precision. Perhaps Sasa Zeman or somebody else can suggest a way to do that successfully on BCC 5.5.1. (The current implementation appears to work on every other compiler.) Or perhaps this is not a compiler issue at all, but an instance of Delphi messing with FPU registers, as was suggested by the anonymous comment on 2008-Jan-09. Please note that I do not own a machine that runs windows, and I am unwilling to purchase one for the purpose of fixing this issue. So I cannot download and install BCC 5.5.1 or Delphi. And thus, I cannot reproduce the problem here. So if you want this problem resolved, you are going to need to suggest code for resolving it. Completely disabling the sqlite3VdbeIntegerAffinity() function is not an acceptable resolution because that breaks quite a few other things. ---- _2008-Jan-11 14:00:05 by anonymous:_ {linebreak} AFAIK it is a general compiler isue on Windows. Reading Sasa's comment this is NOT a Delphi isue since the problem lies in SQLite3.dll. In fact, the tests who are shown here are all done in C++. What I would like to know is how the Windows dll on this website and which DOES have this problem, is created and with what compiler. Appearantly you haven't created this one since you do not own a Windows compiler (?). Then we are in pursuit of an answer and solution. Ignoring the problem is not an option IMHO and would be bad news to all on the Windows community. The Borland C++ compiler with the patch here (SASA does this work?) is completely FREE and doesn't need to be purchased. But if this isue cannot be resolved, and I'm not a C++ guru, then there should be a note here that on Windows the precision is limited to 10e18??? There are now three people busy with this, sure there must be a solution in conjunction with all of us? It's in everyones benefit if this isue can be resolved in stead of ignored. ---- _2008-Jan-11 14:36:11 by drh:_ {linebreak} The windows binaries on the SQLite website are cross-compiled on a Linux box using Mingw-GCC. ---- _2008-Jan-11 17:49:26 by anonymous:_ {linebreak} Which steps do I need to take to reproduce the problem? It seems to work just fine with sqlite3.exe 3.5.0 on Windows XP: sqlite> create table t1 (ID integer primary key, v1 double); sqlite> insert into t1(v1) values(1e18); sqlite> insert into t1(v1) values(1e19); sqlite> select * from t1; 1|1.0e+18 2|1.0e+19 Could someone please post a little C-code which exhibits the problem? Thanks! ---- _2008-Jan-11 18:10:29 by anonymous:_ {linebreak} Thank you for your respond. In order to reproduce the problem, you can create very simple C program using the DLL, which execute upper script by each line, using sqlite3_exec() command. Thank you for your time. Sasa ---- _2008-Jan-11 19:05:51 by anonymous:_ {linebreak} The following: + static const i64 maxInt = (((i64)0x7fffffff)<<32)|0xffffffff; + static const i64 minInt = ((i64)0x80000000)<<32; + + if( r<(double)minInt ){ + return minInt; + }else if( r>(double)maxInt ){ runs faster on my compiler if you do this: + static const double maxInt = (((i64)0x7fffffff)<<32)|0xffffffff; + static const double minInt = ((i64)0x80000000)<<32; + + if( r
Index: configure =================================================================== RCS file: /sqlite/sqlite/configure,v retrieving revision 1.45 diff -u -3 -p -r1.45 configure --- configure 27 Nov 2007 14:50:07 -0000 1.45 +++ configure 5 Jan 2008 07:41:00 -0000 @@ -18520,9 +18520,9 @@ if test "$TARGET_EXEEXT" = ".exe"; then OS_UNIX=0 OS_WIN=0 OS_OS2=1 - TARGET_CFLAGS="$TARGET_CFLAGS -DOS_OS2=1" + CFLAGS="$CFLAGS -DOS_OS2=1" if test "$ac_compiler_gnu" == "yes" ; then - TARGET_CFLAGS="$TARGET_CFLAGS -Zomf -Zexe -Zmap" + CFLAGS="$CFLAGS -Zomf -Zexe -Zmap" BUILD_CFLAGS="$BUILD_CFLAGS -Zomf -Zexe" fi else @@ -18530,14 +18530,14 @@ if test "$TARGET_EXEEXT" = ".exe"; then OS_WIN=1 OS_OS2=0 tclsubdir=win - TARGET_CFLAGS="$TARGET_CFLAGS -DOS_WIN=1" + CFLAGS="$CFLAGS -DOS_WIN=1" fi else OS_UNIX=1 OS_WIN=0 OS_OS2=0 tclsubdir=unix - TARGET_CFLAGS="$TARGET_CFLAGS -DOS_UNIX=1" + CFLAGS="$CFLAGS -DOS_UNIX=1" fi @@ -19392,7 +19392,7 @@ fi echo "$as_me:$LINENO: result: $ac_cv_func_usleep" >&5 echo "${ECHO_T}$ac_cv_func_usleep" >&6 if test $ac_cv_func_usleep = yes; then - TARGET_CFLAGS="$TARGET_CFLAGS -DHAVE_USLEEP=1" + CFLAGS="$CFLAGS -DHAVE_USLEEP=1" fi @@ -19491,7 +19491,7 @@ fi echo "$as_me:$LINENO: result: $ac_cv_func_fdatasync" >&5 echo "${ECHO_T}$ac_cv_func_fdatasync" >&6 if test $ac_cv_func_fdatasync = yes; then - TARGET_CFLAGS="$TARGET_CFLAGS -DHAVE_FDATASYNC=1" + CFLAGS="$CFLAGS -DHAVE_FDATASYNC=1" fi Index: configure.ac =================================================================== RCS file: /sqlite/sqlite/configure.ac,v retrieving revision 1.31 diff -u -3 -p -r1.31 configure.ac --- configure.ac 27 Nov 2007 14:50:07 -0000 1.31 +++ configure.ac 5 Jan 2008 07:41:00 -0000 @@ -310,9 +310,9 @@ if test "$TARGET_EXEEXT" = ".exe"; then OS_UNIX=0 OS_WIN=0 OS_OS2=1 - TARGET_CFLAGS="$TARGET_CFLAGS -DOS_OS2=1" + CFLAGS="$CFLAGS -DOS_OS2=1" if test "$ac_compiler_gnu" == "yes" ; then - TARGET_CFLAGS="$TARGET_CFLAGS -Zomf -Zexe -Zmap" + CFLAGS="$CFLAGS -Zomf -Zexe -Zmap" BUILD_CFLAGS="$BUILD_CFLAGS -Zomf -Zexe" fi else @@ -320,14 +320,14 @@ if test "$TARGET_EXEEXT" = ".exe"; then OS_WIN=1 OS_OS2=0 tclsubdir=win - TARGET_CFLAGS="$TARGET_CFLAGS -DOS_WIN=1" + CFLAGS="$CFLAGS -DOS_WIN=1" fi else OS_UNIX=1 OS_WIN=0 OS_OS2=0 tclsubdir=unix - TARGET_CFLAGS="$TARGET_CFLAGS -DOS_UNIX=1" + CFLAGS="$CFLAGS -DOS_UNIX=1" fi AC_SUBST(BUILD_EXEEXT) @@ -565,13 +565,13 @@ AC_SUBST(TARGET_DEBUG) ######### # Figure out whether or not we have a "usleep()" function. # -AC_CHECK_FUNC(usleep, [TARGET_CFLAGS="$TARGET_CFLAGS -DHAVE_USLEEP=1"]) +AC_CHECK_FUNC(usleep, [CFLAGS="$CFLAGS -DHAVE_USLEEP=1"]) #-------------------------------------------------------------------- # Redefine fdatasync as fsync on systems that lack fdatasync #-------------------------------------------------------------------- -AC_CHECK_FUNC(fdatasync, [TARGET_CFLAGS="$TARGET_CFLAGS -DHAVE_FDATASYNC=1"]) +AC_CHECK_FUNC(fdatasync, [CFLAGS="$CFLAGS -DHAVE_FDATASYNC=1"]) ######### # Generate the output files._2008-Jan-05 08:24:29 by anonymous:_ {linebreak} It appears that http://www.sqlite.org/sqlite3-3.5.4.bin.gz and http://www.sqlite.org/sqlite-3.5.4.so.gz use sleep and fsync even though usleep and fdatasync are available on Linux. On the Linux man page, it claims that fdatasync is more efficient than fsync: "Unfortunately, fsync() will always initiate two write operations: one for the newly written data and another one in order to update the modification time stored in the inode. If the modification time is not a part of the transaction concept fdatasync() can be used to avoid unnecessary inode disk write operations." #f2dcdc 2872 code active 2008 Jan anonymous 2008 Jan 4 4 Some table scan operations could use an index for better data density If a suitable index covers a column being during a table scan, it makes sense to use the index for IO not the table pages themselves for speed. As (contrived) example: CREATE TABLE t1 ( c1 blob, c2 integer, c3 integer ); CREATE INDEX i1 on t1(c2,c3); CREATE INDEX i2 on t1(c3); T1 populated with 4096 rows, c1 being 64K random blobs (to make c2, c3 access slower in this case), c2 and c3 being small random integers. Now: sqlite> select sum(c3) from t1; sum(c3) ---------- 519895 is very slow (several seconds). A table scan is done. Forcing use of an index, in a way that I know *all* rows will be included: sqlite> select sum(c3) from t1 where c2<1000; sum(c3) ---------- 519895 This is instantaneous. It seems to me that if a table scan has to be performance, it makes sense to grab the data from an index whenever possible. ideally the most densely packed index. (BTW; this is contrived, I know putting the blob as the last column will greatly speed things up). #c8c8c8 2871 build closed 2008 Jan anonymous 2008 Jan 3 3 SQLite 3.5.4 build process does not listen to --disable-tcl I build sqlite without TCL support, with an incantation along the lines of: =../sqlite-3.5.4/configure --disable-tcl; make=. This works fine in 3.5.3 but 3.5.4 fails late in the compilation, looking for =tclsh=: =tclsh ../sqlite-3.5.4/tool/mksqlite3c.tcl= is the command it's trying to run. I'm building on Solaris 10u4 and the full incantation for =configure= is as follows (most of this is for readline): =../sqlite-3.5.4/configure --prefix=/home/tfb/packages/sqlite-3.5.4 --enable-static=no --with-readline-lib="-L/opt/sfw/lib -R/opt/sfw/lib -lreadline -lcurses" --with-readline-inc=-I/opt/sfw/include --disable-tcl=. I use =gmake= (which is GNU make, in =/usr/sfw/bin= on Solaris 10) to compile. This is not urgent for me - I will just stick with 3.5.3 for now - but it would be nice to avoid mandatory dependence on TCL I think. Thanks --tim Duplicate of ticket #2845 #c8c8c8 2870 code closed 2008 Jan anonymous 2008 Jan danielk1977 5 5 Comment error In the #4666 checkin comment is: +** Change the value of the P3 operand for a specific instruction. +*/ +void sqlite3VdbeChangeP5(Vdbe *p, int addr, u8 val){ but should be: .... P5 .... operand Thanks. #f2dcdc 2869 new active 2008 Jan anonymous 2008 Jan 5 5 add "sqlite3_open16_v2" to the C API I'm using UTF-16, and if the database file does not exist, "sqlite3_open16" will create a new one, but i wish it fails in such conditions. I notice that there's a "sqlite3_open_v2", but it doesn't support UTF-16, although i can implement a "sqlite3_open16_v2" myself, I think it should exists in the offical releases. #c8c8c8 2868 code closed 2008 Jan anonymous 2008 Jan 1 1 Encryption user can't link latest SQLite I have licensed the encryption package and have it working with an earlier version of SQLite. I downloaded the current stable release, copied it over the previous version and compiled it. It compiled fine, but now the pager has unresolved externals, it seems many of the function names have changed. Please advise. Thanks Tim _2008-Jan-03 14:00:20 by drh:_ {linebreak} The license is perpetual with unlimited free updates. You just have to send us private email and we will send you back the latest code. #f2dcdc 2867 code active 2008 Jan anonymous 2008 Jan 2 2 doesn't build on Cygwin - wrong sqlite3 exe suffix The new Makefile used $(EXE), which doesn't seem to be defined (typo?) _2008-Jan-02 11:12:39 by anonymous:_ {linebreak} Same on mingw: Following patch fixes things:
--- sqlite-3.5.4/Makefile.in Thu Dec 13 19:17:42 2007 +++ sqlite-3.5.4-mingw-fix/Makefile.in Wed Jan 2 11:37:50 2008 @@ -322,7 +322,7 @@ -o $@ $(TOP)/src/shell.c libsqlite3.la \ $(LIBREADLINE) $(TLIBS) -sqlite3$(EXE): $(TOP)/src/shell.c sqlite3.c sqlite3.h +sqlite3$(TEXE): $(TOP)/src/shell.c sqlite3.c sqlite3.h $(LTLINK) $(READLINE_FLAGS) -o $@ \ -DSQLITE_MAX_SQL_LENGTH=1000000000 \ -USQLITE_THREADSAFE -DSQLITE_THREADSAFE=0 \ @@ -577,7 +577,7 @@ -e 's,$$,\\n",' \ $(TOP)/tool/spaceanal.tcl >spaceanal_tcl.h $(LTLINK) -DTCLSH=2 -DSQLITE_TEST=1 $(TEMP_STORE)\ - -o sqlite3_analyzer$(EXE) $(TESTSRC) $(TOP)/src/tclsqlite.c \ + -o sqlite3_analyzer$(TEXE) $(TESTSRC) $(TOP)/src/tclsqlite.c \ libtclsqlite3.la $(LIBTCL)#f2dcdc 2866 build active 2008 Jan anonymous 2008 Jan 1 3 Problems building Windows native in cygwin/mingw environment Trying to build Windows native version using the Cygwin build environment. $ gcc -v Reading specs from /usr/lib/gcc/i686-pc-cygwin/3.4.4/specs Configured with: /usr/build/package/orig/test.respin/gcc-3.4.4-3/configure --verbose --prefix=/usr --exec-prefix=/usr --sysconfdir=/etc --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --enable-languages=c,ada,c++,d,f77,pascal,java,objc --enable-nls --without-included-gettext --enable-version-specific-runtime-libs --without-x --enable-libgcj --disable-java-awt --with-system-zlib --enable-interpreter --disable-libgcj-debug --enable-threads=posix --enable-java-gc=boehm --disable-win32-registry --enable-sjlj-exceptions --enable-hash-synchronization --enable-libstdcxx-debug Thread model: posix gcc version 3.4.4 (cygming special, gdc 0.12, using dmd 0.125) $ $ CFLAGS=-mno-cygwin ./configure --disable-tcl --enable-threadsafe $ make A) The make appears to build sqlite3.exe just fine, without errors or warnings. This binary does work from cmd.exe, BUT not from within the bash cygwin shell for some reason, unlike other Windows native binaries I've built. Next... $ make install B) The cc sqlite3.c -o sqlite3 fails to rebuild sqlite3.exe correctly with the -mno-cygwin option. The output follows: rm -rf tsrc mkdir -p tsrc cp ./src/alter.c ./src/analyze.c ./src/attach.c ./src/auth.c ./src/btmutex.c ./src/btree.c ./src/btree.h ./src/build.c ./src/callback.c ./src/complete.c ./src/date.c ./src/delete.c ./src/expr.c ./src/func.c ./src/hash.c ./src/hash.h ./src/insert.c ./src/journal.c ./src/legacy.c ./src/loadext.c ./src/main.c ./src/malloc.c ./src/mem1.c ./src/mem2.c ./src/mem3.c ./src/mutex.c ./src/mutex_os2.c ./src/mutex_unix.c ./src/mutex_w32.c ./src/os.c ./src/os_unix.c ./src/os_win.c ./src/os_os2.c ./src/pager.c ./src/pager.h ./src/parse.y ./src/pragma.c ./src/prepare.c ./src/printf.c ./src/random.c ./src/select.c ./src/shell.c ./src/sqlite.h.in ./src/sqliteInt.h ./src/table.c ./src/tclsqlite.c ./src/tokenize.c ./src/trigger.c ./src/utf.c ./src/update.c ./src/util.c ./src/vacuum.c ./src/vdbe.c ./src/vdbe.h ./src/vdbeapi.c ./src/vdbeaux.c ./src/vdbeblob.c ./src/vdbefifo.c ./src/vdbemem.c ./src/vdbeInt.h ./src/vtab.c ./src/where.c ./ext/fts1/fts1.c ./ext/fts1/fts1.h ./ext/fts1/fts1_hash.c ./ext/fts1/fts1_hash.h ./ext/fts1/fts1_porter.c ./ext/fts1/fts1_tokenizer.h ./ext/fts1/fts1_tokenizer1.c sqlite3.h ./src/btree.h ./src/btreeInt.h ./src/hash.h ./src/sqliteLimit.h ./src/mutex.h opcodes.h ./src/os.h ./src/os_common.h ./src/sqlite3ext.h ./src/sqliteInt.h ./src/vdbe.h parse.h ./ext/fts1/fts1.h ./ext/fts1/fts1_hash.h ./ext/fts1/fts1_tokenizer.h ./src/vdbeInt.h tsrc cp: warning: source file `./src/btree.h' specified more than once cp: warning: source file `./src/hash.h' specified more than once cp: warning: source file `./src/sqliteInt.h' specified more than once cp: warning: source file `./src/vdbe.h' specified more than once cp: warning: source file `./ext/fts1/fts1.h' specified more than once cp: warning: source file `./ext/fts1/fts1_hash.h' specified more than once cp: warning: source file `./ext/fts1/fts1_tokenizer.h' specified more than once cp: warning: source file `./src/vdbeInt.h' specified more than once rm tsrc/sqlite.h.in tsrc/parse.y cp parse.c opcodes.c keywordhash.h tsrc tclsh ./tool/mksqlite3c.tcl cc sqlite3.c -o sqlite3 /usr/lib/gcc/i686-pc-cygwin/3.4.4/../../../libcygwin.a(libcmain.o):(.text+0xab): undefined reference to `_WinMain@16' collect2: ld returned 1 exit status make: *** [sqlite3] Error 1 $ #f2dcdc 2865 code active 2007 Dec anonymous 2007 Dec 1 2 FTS3 does not build with amalgamation in CVS Grab the latest CVS sources, then run: ./configure make sqlite3.c grep sqlite3Fts3Init sqlite3.c extern int sqlite3Fts3Init(sqlite3*); rc = sqlite3Fts3Init(db); If you compile sqlite3.c with -DSQLITE_ENABLE_FTS3, then sqlite3Fts3Init is unresolved. For some reason, sqlite3Fts3Init and fts3.c was not included in the sqlite3.c amalg. It used to work correctly in 3.5.4. _2007-Dec-30 18:17:57 by anonymous:_ {linebreak} Nevermind, "make sqlite3.c" has never built with the fts3 sources in 3.5.4 or before. You have to run ext/fts3/mkfts3amal.tcl ---- _2007-Dec-30 18:20:56 by anonymous:_ {linebreak} It seems that the sqlite3+fts3 amalg can only be built from main.mk, not Makefile. #f2dcdc 2864 doc active 2007 Dec anonymous 2007 Dec 5 3 ext/fts3/README.txt File ext/fts3/README.txt reads: This folder contains source code to the second full-text search [...] Shouldn't that be: This folder contains source code to the third full-text search [...] _2007-Dec-30 18:27:08 by anonymous:_ {linebreak} Oh, after Googl'ing a little bit, I found that _fts3_ really is _fts2-with-rowid-fixed_. If both _fts2_ and _fts3_ are considered to be the _"second full-text search extension for SQLite"_, the _README_ files could maybe explain the situation. #f2dcdc 2863 code active 2007 Dec anonymous 2007 Dec 2 3 test cast-3.14, cast-3.18 and cast-3.24 fail test cast-3.{14,18,24} fail on freebsd-6.3-PRERELEASE2: cast-3.14...^M Expected: [9223372036854774784]^M Got: [9223372036854773760]^M cast-3.18...^M Expected: [-9223372036854774784]^M Got: [-9223372036854773760]^M cast-3.24...^M Expected: [9223372036854774784]^M Got: [9223372036854773760]^M I used tcl8.4 from ports with no threads and here was the config line: ../sqlite-3.5.4/configure --prefix=/home/marc/local --with-tcl=/usr/local/lib/tcl8.4/ This was built on an ibm t30 laptop #c8c8c8 2862 warn closed 2007 Dec anonymous 2008 Jan 4 3 Small warning fix Hi, There is a harmless warning in btree.c about szNext being used uninitialized, with a comment in the code saying that adding an extra initialization to silence it would slow it down. But I've done it another way which will probably make a tiny speed-up instead (only when failing some step though, I think) Diff:{linebreak} http://megasource.no-ip.org/PTstuff/btree.c.morefix {linebreak} http://megasource.no-ip.org/PTstuff/btree.c.fix (the original fix, just to silence the warning) _2008-Jan-01 06:19:11 by danielk1977:_ {linebreak} Thanks. #c8c8c8 2861 doc closed 2007 Dec anonymous 2008 Jan 1 1 How do I connect to Crystal Reports - I am using vs2003 & the Finisar Not sure of the right venue for this question. My apologies if this is not the right place. I am developing a simple accounting application for a non-profit organisation. I am using vs2003 and programming in c#. To date I have used Access (Jet) database files and the Crystal Report Generator built in to VS2003. I thought SQLite would be a better choice than the Access database - simple to change my c# code using the Finisar data provider, but I am at a loss to see how I can use this provider in Crystal. Is there any way to get this provider onto the list of ADO DB providers in Crystal - or any other approach I could use. If I cannot use Crystal is there an alternative for producing reports which can link to SQLite. Any help much appreciated. Roger Lovelock _2007-Dec-29 13:58:55 by anonymous:_ {linebreak} Not a bug. Better ask this on the mailing list. To subscribe to the list, send a message to:
$ gcc -I. -DSQLITE_OMIT_LOAD_EXTENSION shell.c sqlite3.c -o shell -lpthread sqlite3.c:81710:26: error: sqlite3ext.h: No such file or directory sqlite3.c:81748: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘static’ sqlite3.c:87815: error: expected ‘;’, ‘,’ or ‘)’ before ‘*’ tokenYou must use both -DSQLITE_OMIT_LOAD_EXTENSION -DSQLITE_CORE on the compile line: gcc -I. -DSQLITE_OMIT_LOAD_EXTENSION -DSQLITE_CORE shell.c sqlite3.c -o shell -lpthread That's a bug. The workaround is simple, though - always compile with -DSQLITE_CORE As you've found out, if you choose to build with FTS3 support you do not have to #define SQLITE_CORE on the compile line, as it defines it for you: gcc -I. -DSQLITE_OMIT_LOAD_EXTENSION -DSQLITE_ENABLE_FTS3 shell.c sqlite3.c -o shell -lpthread ---- _2007-Dec-30 17:55:25 by anonymous:_ {linebreak} I'm using GCC, Visual Studio, Intel C++, HP ANSI C/aC++, SGI MIPSpro, Sun Studio, IBM XL C and other compilers. They will certainly not #include a file even when it is not wanted. How is your remark related to the use case exposed above? ---- _2007-Dec-30 18:08:50 by anonymous:_ {linebreak} Ignore the #include thing - it was just an idea. The sqlite bug is explained in detail in the 2007-Dec-30 17:53:42 comment. ---- _2007-Dec-30 18:09:36 by anonymous:_ {linebreak} Indeed, =SQLITE_CORE= is usually defined by the _loadext.c_ source code. Unfortunately this will not happen if =SQLITE_OMIT_LOAD_EXTENSION= is defined. The _fts3.c_ source code in the amalgamated source file needs to be compiled with =SQLITE_CORE= if we want it: *: not to include _"sqlite3ext.h"_, *: and not to build the FTS3 module unless =SQLITE_OMIT_LOAD_EXTENSION= is defined. So yes, the solution is to define =SQLITE_CORE= when building the amalgamated source file: *: either from the command line, in which case this requirement needs to be documented, *: or within the amalgamated source file itself, in which case {linebreak} =#define SQLITE_CORE 1= {linebreak} should be added before any reference to _sqlite3ext.h". ---- _2008-Jan-01 05:51:17 by danielk1977:_ {linebreak} Thanks. #f2dcdc 2857 code active 2007 Dec anonymous 2007 Dec 2 2 GROUP BY cost estimate wrong with WHERE clause There seems to be an issue with the sqlite cost heuristic with an INDEX present on GROUP BY with certain types of WHERE clauses. Given the database formed by running these statements: create table stuff(a,b,c,d); insert into stuff values(1,2,3,4); create temp view v1 as select random()%100, random()%100, random()%1000, random()%10000 from stuff x, stuff y; insert into stuff select * from v1; insert into stuff select * from v1; insert into stuff select * from v1; insert into stuff select * from v1; insert into stuff select * from v1; create index stuff_b on stuff(b); create index stuff_c on stuff(c); create index stuff_d on stuff(d); analyze; Using sqlite.org's sqlite3-3.5.4.bin, this query takes 47 seconds: select c from stuff where a=23 group by c; while this query takes just 2 seconds: select c from stuff where a=23 group by +c; It is more efficient in this case to do a full table scan instead of using the INDEX on column c. _2007-Dec-23 23:14:06 by anonymous:_ {linebreak} The queries above both run in a couple of seconds with this naive patch:
--- src/where.c 12 Dec 2007 17:42:53 -0000 1.266 +++ src/where.c 23 Dec 2007 22:48:37 -0000 @@ -1514,6 +1514,12 @@ static double bestIndex( flags = 0; } + if( pWC && pWC->nTerm>0 && pOrderBy ){ + /* Reduce cost if both an ORDER/GROUP BY exists with a WHERE. */ + cost /= 100; /* A very rough guess. */ + WHERETRACE(("... WHERE + ORDER BY decreases cost to: %.9g\n", cost)); + } + /* If the table scan does not satisfy the ORDER BY clause, increase ** the cost by NlogN to cover the expense of sorting. */ if( pOrderBy ){But it has not been tested on queries with more than one table. Its logic could be flawed. ---- _2007-Dec-24 00:09:00 by drh:_ {linebreak} The complaint is centered around these two queries: /* 1 */ SELECT c FROM stuff WHERE a=23 GROUP BY c; /* 2 */ SELECT c FROM stuff WHERE a=23 GROUP BY +c; Query 1 runs in about 40 seconds and query 2 in about 1.5 seconds on my macbook. But with the patch, both queries run in about 1.5 seconds. Fair enough. But now consider these two queries: /* 3 */ SELECT c FROM stuff WHERE a!=23 GROUP BY c; /* 4 */ SELECT c FROM stuff WHERE a!=23 GROUP BY +c; In this case, query 3 runs in 42 seconds on an unpatched version of 3.5.4 and query 4 runs in about 109 seconds. So in cases where the WHERE clause is not particularly selective, the first version is faster than the second by a good margin. On a patched version of 3.5.4, both queries 3 and 4 run in about 110 seconds. So it seems to me that the patch is robbing Peter to pay Paul. It makes ORDER BY queries with very selective WHERE clauses run faster but at the expense of making queries with unselective WHERE clauses running slower. But notice this: in the current (unpatched) implementation, the programmer at least has the ability to select a different algorithm by the judicious placement of a "+" sign. After the patch, this is no longer possible. The patch forces the second algorithm to be used in all cases, even cases where it is slower. It seems to me to be better to leave things as they are since the current approach at least allows the programmer to override SQLite's algorithm selection if SQLite chooses incorrectly. The only way, it seems to me, to automatically choose the correct algorithm is to devise some test that will determine (in advance) whether or not the WHERE clause weeds out many or few rows from the result set. I'm thinking that determination is going to be very hard (or impossible) to do without first doing a full table scan. ---- _2007-Dec-24 05:40:47 by anonymous:_ {linebreak} It think it would be surprising to average users that _adding_ an index (on column C in this case) may significantly _decrease_ query performance for some queries. It was surprising to me, at least. In my opinion, a query being 20 times slower in a default bad guess situation is worse than a query only being 2.5 times slower with a default bad guess in a worst case scenario. It's a question of relative magnitude of the difference. This is why I think that the database should err on the side of the WHERE clause having a more selective bias. (Side note: the query timings difference is less pronounced if you use PRAGMA temp_store=memory, in which case query 3 running on an unpatched 3.5.4 takes just 50% more time to run than query 4 on my machine.) But you raise a good point in that if there's a wrong guess in the selectivity bias it would be nice to be able to manually override it. How much do you hate this type of syntax that some other databases use? select c from stuff where a!=23 group by /*+stuff_c*/ c; SQLite does not currently offer a way to pick a specific index. I think it would be quite useful. ---- _2007-Dec-24 17:05:16 by anonymous:_ {linebreak} Another option is to collect WHERE clause statistics in a table like create table sqlite_stat2( where_clause_md5 BLOB primary key, where_clause TEXT, rows_examined INT, rows_true INT ); where the last 2 columns are cumulative for each query. The statistics option could be enabled/disabled via a PRAGMA sqlite_collect_statistics. The where_clause column could be a string generated fairly easily from the walking the parse tree of the resolved Select statement's pWhere. This way the where_clause is normalized and a single query with many subselects could generate more than 1 where_clause, and different queries that happen to use the same normalized where clause would update the same entry in the stat2 table. where_clause normalization would strip off aliases and only refer to the original table and column names. For example the 2 queries below: -- CREATE TABLE t1(a, b); -- CREATE TABLE t2(b, c); SELECT t1.a*c as AC, t2.b-a as BA FROM t1, t2 WHERE AC>BA; SELECT *, t1.a Foo FROM t2, t1 WHERE Foo*c > t2.b - t1.a; would generate the same normalized where_clause string "(T1.A*T2.C)>(T2.B-T1.A)". The table information is already encoded within it. The generated VDBE code would have to generate Mem counters that would be incremented by each WHERE test, and lazily updated at the end of transactions or periodically written to the stat2 table to minimize disk use, as this information is not critical. One could also manually set the stat2 table with statistical values they would like their queries to use even if PRAGMA sqlite_collect_statistics=off; Any time the schema is changed, the entire sqlite_stat2 table would be cleared. #f2dcdc 2856 doc active 2007 Dec anonymous 2007 Dec anonymous 4 3 SQLite Documentation - Tcl API - Link broken On the page "The Tcl Interface to the SQLite library", the link "enable load extension" does not work. #c8c8c8 2855 code closed 2007 Dec anonymous 2007 Dec drh 2 3 Get an error "11:database disk image is malformed" during queries I don't know what has happened to my database and when. I followed every new sqlite version as soon as it got released. So now I am with version 3.5.4 (WinXP) and my database has grown up to a size of about 1.74 GB. It contains only three tables and neither views nor triggers. When I run a query on all records of one table I got the a.m. error message and no results. I was able to query all details by RowID up to and above a small range of only 105 records (out of 345767 ). No problem below and none above. Then I renamed the table, created another table and "insert into .. select from .." one into the other in two steps. ("where RowID between .. and .. or RowID between .. and .." popped up the error message again but running two queries one after the other worked fine. That's my workaround. It is a pitty that I cannot send you an example but I do not know how to shrink the database to only remain with the route cause of trouble. I dropped the other two tables and deleted all records before and after the malformed part but now I cannot vacuum without getting again that message. 1: How may I got this malformed records into my database? How to avoid repetition? 2: How can I eliminate a malformed part out of the database without having to recreate almost everything from scrap? As soon as the malformed records are touched every command stops. Isn't there a possibility to put such records into *quarantine* ?! This would really help to isolate problems and would hopefully give me a chance to send you the damaged records for your analysis to find out what may have happened. The isolation of records may be related to "check integrity" of the database. Another good idea may be to give sqlite.exe a chance to extract isolated part under quaratine from one into another database. - I keep my damaged database for such a case for a while, expecting such a command with the next release. _2007-Dec-21 12:52:51 by drh:_ {linebreak} Information on how to corrupt an SQLite database file can be found at *: http://www.sqlite.org/cvstrac/wiki?p=DatabaseCorruption *: http://www.sqlite.org/atomiccommit.html *: http://www.sqlite.org/lockingv3.html (Section 6.0) To restore data as much as you can from a corrupt database file, do this: sqlite3 corrupt.db .dump | sqlite3 restored.db This ticket reads more like a support request than a bug report. Or perhaps it is a feature request - requesting that corruption be "quarantined". Quarantining bad records merely masks errors and gives no new capabilities so that is not something we are interested in doing. For technical support see *: http://www.sqlite.org/support.html ---- _2007-Dec-21 13:06:46 by anonymous:_ {linebreak} Thank you for your quick response and hints about more info on causes for db corruptions. However, ="sqlite3.exe corrupt.db .dump | sqlite3 restored.db"= did not work because of the malformed records, which interrupted the process after a while of work. - For me it would be enough if ="delete from tbl where RowID between nn and mm"= would work without failing because of using the malformed space. This would be good for healing a db file if the problem could be isolated like in my case. Sorry, if dealing with such problems isn't the right place here. (By the way: the first 5 fields of that table are forming the primary index; there are no other indexes on that table.) I wish you a merry christmas and a happy and successfull new year. ---- _2007-Dec-21 13:08:59 by drh:_ {linebreak} The ".dump" command attempts to work around corruption. If you have some kind of corruption that it is unable to work-around, then append an example database to this ticket and we will reopen the ticket and work on the problem. Without having access to a database that demonstrates the problem, I am afraid there is not much we can do. ---- _2007-Dec-21 14:03:59 by anonymous:_ {linebreak} Well. Didn't know about =.dump= is working around corruption and tried to do this with my database where I eliminated good content around the malformed records. Your recommendation worked without raising the aborting error message. It will not solve the appearance but is a great help to heal the database for continued work. Thank you for this additional information! #c8c8c8 2854 code closed 2007 Dec danielk1977 2007 Dec danielk1977 1 1 BEGIN EXCLUSIVE does not prevent other shared-cache users from reading Subject says it all. Executing a "BEGIN EXCLUSIVE" does not prevent other shared cache users from reading the database. _2007-Dec-20 17:53:32 by danielk1977:_ {linebreak} The attached test file demonstrates the problem. ---- _2007-Dec-20 19:28:29 by anonymous:_ {linebreak} Could this cause unexpected deadlocks? #f2dcdc 2853 new active 2007 Dec anonymous 2008 Jan 2 3 optimizer fails to use an index on MAX subquery i have these 2 identical queries with an index on (place_id,visit_date), the second query is about 2x fast the the first, while i'd expect that the MAX is faster than a limited order by clause... It's like the index is mis-used with MAX SELECT h.id, h.url, h.title, h.rev_host, h.visit_count, (SELECT MAX(visit_date) FROM moz_historyvisits WHERE place_id = h.id AND visit_type NOT IN(0,4)), f.url, null, null FROM moz_places h LEFT OUTER JOIN moz_favicons f ON h.favicon_id = f.id WHERE h.id IN (SELECT DISTINCT p.id FROM moz_places p JOIN moz_historyvisits ON place_id = p.id WHERE hidden <> 1 AND visit_type NOT IN (0,4) ORDER BY visit_date DESC LIMIT 10) ORDER BY 6 DESC; SELECT h.id, h.url, h.title, h.rev_host, h.visit_count, (SELECT visit_date FROM moz_historyvisits WHERE place_id = h.id AND visit_type NOT IN(0,4) ORDER BY visit_date DESC LIMIT 1), f.url, null, null FROM moz_places h LEFT OUTER JOIN moz_favicons f ON h.favicon_id = f.id WHERE h.id IN (SELECT DISTINCT p.id FROM moz_places p JOIN moz_historyvisits ON place_id = p.id WHERE hidden <> 1 AND visit_type NOT IN (0,4) ORDER BY visit_date DESC LIMIT 10) ORDER BY 6 DESC; _2007-Dec-20 04:13:44 by anonymous:_ {linebreak} Can you supply the schema of these tables and all related indexes? ---- _2007-Dec-20 10:55:29 by danielk1977:_ {linebreak} Correct. SQLite does not use the index to optimize the max() in the first query. But it does use it to optimize ORDER BY in the second. ---- _2007-Dec-21 00:54:10 by anonymous:_ {linebreak} the schema is that of mozilla firefox 3 places.sqlite, with an added index on moz_historyvisits(place_id,visit_date), this is extracted from a i bug i was working on. Do you need that i append the full schema here? Will this be fixed to use the index with max or is that the expected behaviour? ---- _2007-Dec-21 15:59:58 by drh:_ {linebreak} We will be enhancing the min()/max() optimization to be able to cover the case you describe above. But this will take some time to get right. If you are in a hurry, it seems like modifying your SQL is the easiest way to go. The current optimizer recognizes queries of the form: SELECT min(x) FROM table; SELECT max(x) FROM table; And it causes special VDBE code to be generated for these cases. The presence of a WHERE clause defeats the current optimization. We need to modify the optimizer to recognize queries of the form: SELECT min(x) FROM table WHERE expr And convert them into: SELECT x FROM table WHERE expr AND x NOT NULL ORDER BY x LIMIT 1 But we only want to do this optimization if the ORDER BY can be evaluated using an index. If we have to accumulate the entire result set, sort it, then take the largest entry, that is more work than just keeping track of the largest entry as we loop through the result set. Notice the addition of the "x NOT NULL" term in the WHERE clause. This is critical to preserving correct semantics. The query optimizer currently does not know how to deal with a NOT NULL. It just evaluates all rows, tests them individually for NULL, and throws out those that match. This would work in most cases, but would slow down in a min() where there were many NULL entries. The current implemention of the min() optimization knows to skip over the NULL entries in a single operation. The WHERE generator part of the optimizer will need to be enhanced to recognize NOT NULL constraints and skip over them. All of this is a substantial change. The min()/max() optimization function will be rewritten from scratch. Significant and non-trivial modifications will need to be made to the code that looks for indices and generates the WHERE clause constraints. There are many opportunities to make coding mistakes, so we will need to spend a lot of time in testing before we put these changes into production. So, we will be working the problem. But do not expect an overnight fix. I suppose that while we are redesigning the min/max optimizer, we might as well also fix it so that SELECT arbitrary_expr(max(x)) FROM table WHERE expr; gets converted into SELECT arbitrary_expr(x) FROM table WHERE expr AND x NOT NULL ORDER BY x DESC LIMIT 1; ---- _2007-Dec-31 06:41:03 by anonymous:_ {linebreak} Regarding: SELECT min(x) from table WHERE expr being converted to: SELECT x FROM table WHERE expr AND x NOT NULL ORDER BY x LIMIT 1 There's no need for the "x NOT NULL" condition considering NULL is returned by min() (or max for that matter) when no rows match. sqlite> .nullvalue
capi3-13-4... Ok capi3-13-5...*** glibc detected *** capi3-14.1-misuse... Ok capi3-15.1... Error: error code SQLITE_TOOBIG (18) does not match sqlite3_errcode SQLITE_OK (0) capi3-15.2... Expected: [2] Got: [0] capi3-15.3... Expected: [SQLITE_OK] Got: [SQLITE_MISUSE] capi3-15.4... Error: (21) library routine called out of sequence capi3-15.5... Error: (21) library routine called out of sequence capi3-15.6... Error: (21) library routine called out of sequence capi3-15.7... Error: (21) library routine called out of sequence capi3-15.8... Error: (21) library routine called out of sequence capi3-16.1... Error: (21) library routine called out of sequence capi3-16.2... Error: (21) library routine called out of sequence capi3-16.3... Error: (21) library routine called out of sequence capi3-16.4... Error: (21) library routine called out of sequence capi3-17.1... Error: (21) library routine called out of sequence capi3-17.2... Ok capi3-17.3...*** glibc detected *** ./testfixture: double free or corruption (!prev): 0x0812d650 ***#cfe8bd 2850 code fixed 2007 Dec anonymous 2007 Dec 2 2 CSV output missing quotes if contains equals character Here's yet another bug in the CSV output using the sqlite3 command line tool. Here's a sample of the new bug: .mode csv select 'a=1,234', 'b=5'; gives: a=1,234,b=5 but should give: "a=1,234",b=5 I'm using sqlite 3.4.0 on Mac OS X 10.5.1. CSV (comma separated values) output should show value1,value2,value3 etc. If a value contains a comma, then csv should encapsulate that value in quotes: "value,1",value2,value3. And if a value contains a quote, csv should also encapsulate that value in quotes and show the quote as a double quote, eg "value,1","value ""2""",value3 Sqlite3 seems to ignore the need for quotes in a value that contains an equals "=". I don't know if it also fails with other characters. Needless to say, this messes up the output considerably, placing output in the wrong column. #cfe8bd 2849 doc fixed 2007 Dec anonymous 2007 Dec 5 5 unary operator ! From documentation: "Supported unary prefix operators are these: - + ! ~ NOT" There is no ! operator in sqlite #c8c8c8 2848 code closed 2007 Dec anonymous 2007 Dec 4 4 windows CRLF is incorrectly handled by .dump If I insert a string containing standard windows CRLF (0x0D, 0x0A) into the table, and then try to .dump the table, instead of original CRLF, I get a 'CRCRLF' (0x0D, 0x0D, 0x0A). It is probably not helpful to mention that the column does not use any special collation and that the database is in UTF-8 mode. I am on windows xp, and use original build of sqlite3.exe 3.5.4. I am sorry if this problem already has its own ticket, or if I missed some configuration in sqlite3/dump, but I could not find any such. _2007-Dec-17 14:06:58 by anonymous:_ {linebreak} I can't reproduce this on MS Windows Vista with SQLite v3.5.4. Snippet from .dump output:
sqlite3 test.db3 .dump | od --address-radix=x --format=x1a 000060 0d 0a 49 4e 53 45 52 54 20 49 4e 54 4f 20 22 74 cr nl I N S E R T sp I N T O sp " t 000070 65 73 74 31 22 20 56 41 4c 55 45 53 28 31 2c 27 e s t 1 " sp V A L U E S ( 1 , ' 000080 61 6c 0d 0a 70 68 61 27 29 3b 0d 0a 43 4f 4d 4d a l cr nl p h a ' ) ; cr nl C O M M 000090 49 54 3b 0d 0a I T ; cr nlPerhaps the extra control character was accidently loaded into the database? ---- _2007-Dec-17 15:18:53 by drh:_ {linebreak} Unable to reproduce on WinXP. ---- _2007-Dec-17 22:12:27 by anonymous:_ {linebreak} Thanks for fast response. The extra 0x0D was surely not in the database. I reviewed my situation over and it looks like a windows shell (cmd.exe) is adding the extra 0x0D as I was redirecting stdout from dump like this: cmd /c sqlite3 test.db3 .dump > test.txt Sorry to blame you. #f2dcdc 2847 new active 2007 Dec anonymous 2008 Jan 5 4 Include major, minor, and patch version numbers in sqlite.h Hi, I'm working on a project where sqlite is being compiled into a DLL. Currently, sqlite.h makes the version number available as both an x.y.z string and an integer value in the form of x*1000000 + y*1000 + z. Unfortunately, neither of these options works particularly well when trying to create a resource file so that the DLL can display the proper version information within Windows. I've tried many different ways of disassembling the integer version number, but limitations in the resource compiler unfortunately prevent them from working. As a result, I've been forced for the time being to define SQLITE_VERSION_MAJOR, SQLITE_VERSION_MINOR, and SQLITE_VERSION_PATCH with manually-given values of x, y, and z respectively in order to accomplish this task. It would be really nice if these could be generated automatically for sqlite.h when running configure in the same way that VERSION and VERSION_NUMBER are so that setting the values manually wouldn't be required for the future. Would you be willing to do that? Thanks in advance. _2007-Dec-17 13:30:22 by anonymous:_ {linebreak} You could invoke this awk script in your make file:
# ## extrvers.awk ## Extract verison parts from sqlite3.h # ## Usage: # %GNU_AWK% -f extrvers.awk sqlite3.h >sqlite3.h.new # rm / del sqlite3.h # mv / ren sqlite3.h.new sqlite3.h # # ## Ignore any previous defines /^#define SQLITE_VERSION_(MAJOR|MINOR|PATCH)/{ next } ## generate extra #define MAJOR/MINOR/PATH lines /^#define[[:blank:]]+SQLITE_VERSION[[:blank:]]/{ split(substr($3,2,length($3) - 2),tmp,".") print "#define SQLITE_VERSION_MAJOR " tmp[1] print "#define SQLITE_VERSION_MINOR " tmp[2] print "#define SQLITE_VERSION_PATCH " tmp[3] } ## Repeat all other lines untouched { print }---- _2007-Dec-18 00:06:53 by anonymous:_ {linebreak} Original poster here. Thanks for the useful script! I'm assuming you're granting a license for this (or a modified version of it) to be included in the source tree if the powers that be are willing to accept it? This would be for the Mozilla project. ---- _2007-Dec-18 00:21:52 by drh:_ {linebreak} First off, I didn't post the script. I don't know who "anonymous" is. Secondly, if you are working for Mozilla, you will get *much* faster service if you identify yourself as such. ---- _2007-Dec-18 00:39:38 by anonymous:_ {linebreak} I just want to make sure I'm not running afoul of anybody by using their work without proper permission. To whoever posted it, you can contact me at ryanvm [at] gmail [dot] com. Thanks again for help! ---- _2007-Dec-26 19:06:27 by anonymous:_ {linebreak} I'm the poster of the script. Of course I don't mind it being used. For the peace of mind of anyone using it: please prepend the code with:
# Copyright (C) 2007 by Kees Nuyt, Rotterdam, Netherlands # The author of this code dedicates any and all copyright # interest in this code to the public domain. I make this # dedication for the benefit of the public at large and # to the detriment of my heirs and successors. I intend # this dedication to be an overt act of relinquishment in # perpetuity of all present and future rights to this # code under copyright law. #Cheers! "Kees Nuyt"
gcc -g -O2 -I. -I./src -DNDEBUG -I/usr/include/tcl8.4.13 -DSQLITE_THREADSAFE=1 -DSQLITE_THREAD_OVERRIDE_LOCK=-1 -DSQLITE_OMIT_LOAD_EXTENSION=1 -c ./src/alter.c -o alter.o ./src/alter.c: In function ‘renameTableFunc’: ./src/alter.c:77: error: ‘TK_SPACE’ undeclared (first use in this function) ./src/alter.c:77: error: (Each undeclared identifier is reported only once ./src/alter.c:77: error: for each function it appears in.) ./src/alter.c: In function ‘renameTriggerFunc’: ./src/alter.c:135: error: ‘TK_SPACE’ undeclared (first use in this function) make: *** [alter.lo] Error 1_2007-Dec-13 13:22:42 by drh:_ {linebreak} Works for me. Try running configure in a directory separate from the source tree: ../sqlite/configure; make testfixture ---- _2007-Dec-13 16:16:04 by anonymous:_ {linebreak} When I "cvs up -A; cvs diff" the sources show no changes at all. So I tried the following: make clean make distclean ./configure --disable-shared make test But I get the same errors. So I run the old tree and the new tree's alter.c through gcc -E and here's the difference:
--- ../../sqlite/alter.old 2007-12-13 11:04:15.000000000 -0500 +++ alter.new 2007-12-13 11:04:25.000000000 -0500 @@ -599,7 +599,7 @@ HashElem *sqlite3HashFindElem(const Hash*, const void *pKey, int nKey); void sqlite3HashClear(Hash*); # 154 "./src/sqliteInt.h" 2 -# 1 "./src/parse.h" 1 +# 1 "./parse.h" 1 # 155 "./src/sqliteInt.h" 2 # 1 "/usr/include/stdio.h" 1 3 4 # 28 "/usr/include/stdio.h" 3 4 @@ -3532,7 +3532,7 @@ do { zCsr += len; len = sqlite3GetToken(zCsr, &token); - } while( token==TK_SPACE ); + } while( token==145 ); ((void) (0)); } while( token!=19 && token!=115 ); @@ -3582,7 +3582,7 @@ do { zCsr += len; len = sqlite3GetToken(zCsr, &token); - }while( token==TK_SPACE ); + }while( token==145 ); ((void) (0)); # 147 "./src/alter.c" dist++;So somehow an old parse.h got copied to to src/ which was the cause of the problem. If I remove src/parse.h as is well and "make test" now builds. Documented in this ticket in case someone else happens to have the same problem in the future. ---- _2007-Dec-13 16:18:16 by anonymous:_ {linebreak} I must have been testing lemon on src/parse.y some time in the past to cause this problem. #c8c8c8 2836 build closed 2007 Dec anonymous 2007 Dec 1 1 Undefined symbol 'OP_StackDepth' There is a error on current cvs version(12/12/07) with "Undefined symbol 'OP_StackDepth'" message. opcodes.h and opcodes.c havn't 'OP_StackDepth' definition currently. Sorry for my short English. _2007-Dec-13 07:41:07 by danielk1977:_ {linebreak} opcodes.c and opcodes.h are auto-generated files (generated based on vdbe.c). Likely your copies have not been regenerated since yesterdays modifications. Try running "make clean" before rebuilding. If that fails, manually remove opcodes.c and opcodes.h from your build directory. #cfe8bd 2835 code fixed 2007 Dec anonymous 2007 Dec 1 1 Lemon memory read/write errors
$ cat lemonbug.y phrase ::= foo AND A B C foo. phrase ::= foo AND A B C bar. foo ::= BIRD | CAT | DOG. bar ::= CAT | DOG. $ ./lemon lemonbug.y 1 parsing conflicts. $ head -23 lemonbug.out State 0: phrase ::= * foo AND A B C foo phrase ::= * foo AND A B C bar foo ::= * BIRD| CAT| DOG BIRD shift 2 shift 2 phrase accept foo shift 5 State 1: phrase ::= foo AND A B C * foo phrase ::= foo AND A B C * bar foo ::= * BIRD| CAT| DOG bar ::= * CAT| DOG BIRD shift 2 shift 2 reduce 134589256 ** Parsing conflict ** CAT shift 4 foo shift 9 bar shift 10 valgrind output: ==11245== Invalid read of size 1 ==11245== at 0x8049109: SetAdd (lemon.c:4080) ==11245== by 0x804B083: FindFirstSets (lemon.c:649) ==11245== by 0x8051919: main (lemon.c:1464) ==11245== Address 0x416980F is 0 bytes after a block of size 7 alloc'd ==11245== at 0x401F396: malloc (in /usr/lib/valgrind/x86-linux/vgpreload_memcheck.so) ==11245== by 0x804AEF3: SetNew (lemon.c:4057) ==11245== by 0x804AF7D: FindFirstSets (lemon.c:617) ==11245== by 0x8051919: main (lemon.c:1464) ==11245== ==11245== Invalid write of size 1 ==11245== at 0x804910C: SetAdd (lemon.c:4081) ==11245== by 0x804B083: FindFirstSets (lemon.c:649) ==11245== by 0x8051919: main (lemon.c:1464) ==11245== Address 0x416980F is 0 bytes after a block of size 7 alloc'd ==11245== at 0x401F396: malloc (in /usr/lib/valgrind/x86-linux/vgpreload_memcheck.so) ==11245== by 0x804AEF3: SetNew (lemon.c:4057) ==11245== by 0x804AF7D: FindFirstSets (lemon.c:617) ==11245== by 0x8051919: main (lemon.c:1464) ==11245== ==11245== Invalid read of size 4 ==11245== at 0x8048BF4: actioncmp (lemon.c:365) ==11245== by 0x8048DBC: merge (lemon.c:1577) ==11245== by 0x8048E6D: msort (lemon.c:1623) ==11245== by 0x80512C1: FindActions (lemon.c:966) ==11245== by 0x805194E: main (lemon.c:1479) ==11245== Address 0x416B298 is 4 bytes after a block of size 36 alloc'd ==11245== at 0x401F396: malloc (in /usr/lib/valgrind/x86-linux/vgpreload_memcheck.so) ==11245== by 0x804ABF2: State_new (lemon.c:4503) ==11245== by 0x8051015: getstate (lemon.c:759) ==11245== by 0x805113A: buildshifts (lemon.c:829) ==11245== by 0x805104E: getstate (lemon.c:766) ==11245== by 0x805113A: buildshifts (lemon.c:829) ==11245== by 0x805104E: getstate (lemon.c:766) ==11245== by 0x805113A: buildshifts (lemon.c:829) ==11245== by 0x805104E: getstate (lemon.c:766) ==11245== by 0x805113A: buildshifts (lemon.c:829) ==11245== by 0x805104E: getstate (lemon.c:766) ==11245== by 0x805113A: buildshifts (lemon.c:829) ==11245==_2007-Dec-12 18:00:29 by anonymous:_ {linebreak} Another grammar with parsing conflicts, different valgrind error:
article ::= blocks FIN. blocks ::= block. blocks ::= blocks block. block ::= NEWLINE. block ::= stanza. block ::= heading. block ::= stanza heading. heading ::= HEADING_START text HEADING_END. stanza ::= text. stanza ::= stanza NEWLINE text. text ::= textpiece. text ::= text textpiece. textpiece ::= TEXT. textpiece ::= LINK. ==11727== Conditional jump or move depends on uninitialised value(s) ==11727== at 0x804ADBB: CompressTables (lemon.c:3947) ==11727== by 0x8051A2C: main (lemon.c:1482) 3925 void CompressTables(lemp) 3926 struct lemon *lemp; 3927 { 3928 struct state *stp; 3929 struct action *ap, *ap2; 3930 struct rule *rp, *rp2, *rbest; 3931 int nbest, n; 3932 int i; 3933 int usesWildcard; 3934 3935 for(i=0; i---- _2007-Dec-12 18:14:11 by anonymous:_ {linebreak} "./lemon src/parse.y" also produces: ==12904== Conditional jump or move depends on uninitialised value(s) ==12904== at 0x80523AF: CompressTables (lemon.c:3947) ==12904== by 0x804B015: main (lemon.c:1482) ---- _2007-Dec-12 19:00:41 by anonymous:_ {linebreak} This appears to remedy the Conditional jump error for parse.y. The read/write errors in first grammar remains.nstate; i++){ 3936 stp = lemp->sorted[i]; 3937 nbest = 0; 3938 rbest = 0; 3939 usesWildcard = 0; 3940 3941 for(ap=stp->ap; ap; ap=ap->next){ 3942 if( ap->type==SHIFT && ap->sp==lemp->wildcard ){ 3943 usesWildcard = 1; 3944 } 3945 if( ap->type!=REDUCE ) continue; 3946 rp = ap->x.rp; 3947 if( rp->lhsStart ) continue;
--- tool/lemon.c 5 Oct 2007 16:16:36 -0000 1.51 +++ tool/lemon.c 12 Dec 2007 19:02:37 -0000 @@ -2100,6 +2100,7 @@ to follow the previous rule."); psp->prevrule = 0; }else{ int i; + rp->lhsStart = 0; rp->ruleline = psp->tokenlineno; rp->rhs = (struct symbol**)&rp[1]; rp->rhsalias = (char**)&(rp->rhs[psp->nrhs]);---- _2007-Dec-13 23:44:45 by anonymous:_ {linebreak} I originally thought this reading the unitialized memory for rp->lhsStart lemon parser bug was completely inconsequential, but now I see if I apply the following patch and run lemon against parse.y it generates a completely different parse.c file. If the lemon is built with the following patch and your run parse.y against it, lemon will generate a parse.c that is 275868 bytes in size:
diff -u -3 -p -r1.51 lemon.c --- tool/lemon.c 5 Oct 2007 16:16:36 -0000 1.51 +++ tool/lemon.c 13 Dec 2007 23:37:39 -0000 @@ -2100,6 +2100,7 @@ to follow the previous rule."); psp->prevrule = 0; }else{ int i; + rp->lhsStart = 0xffff; // *** change to zero and rerun rp->ruleline = psp->tokenlineno; rp->rhs = (struct symbol**)&rp[1]; rp->rhsalias = (char**)&(rp->rhs[psp->nrhs]);While if the rp->lhsStart value is set to 0, it will produce a completely different parse.c file of 122960 bytes. #c8c8c8 2834 event closed 2007 Dec anonymous 2007 Dec 2 1 Is Version 3.1.3 not compatible with 3.5.3 ? I created this table "CREATE TABLE CALCULATION (sceneid INTEGER, calcid INTEGER, data BLOB, timestamp INTEGER, PRIMARY KEY (sceneid, calcid))" with the sqlite version 3.1.3. And now I have updated to version 3.5.3 and it is not possible to read the data from table CALCULATION. But if I create the table only with one primary key (Primary key (sceneid)), than it is possible to read the data. Is it a bug in version 3.5.3 oder in 3.1.3? How can I read the data with a double primary key? _2007-Dec-11 18:43:51 by drh:_ {linebreak} All versions of SQLite are compatible back through version 3.0.0. If you think otherwise, please provide details and we will investigate. I generate a database using your schema and SQLite version 3.1.3 and it works just fine with SQLite version 3.5.3. Unable to recreate the problem... #cfe8bd 2833 doc fixed 2007 Dec anonymous 2007 Dec 5 1 typo on SQLite home page As pointed out by silvestru@molddata.md on the mailing list the phrase "This the homepage for SQLite - ..." should be changed to "This is the homepage for SQLite - ...". We fixed this yesterday. And the homepage is going to be completely replaced tomorrow morning, at which point the issue will become moot. #cfe8bd 2832 code fixed 2007 Dec anonymous 2007 Dec 1 1 Data corruption with UPDATE or DELETE clause ... On any UPDATE or DELETE operation, if a side effect causes other rows in the same table to be deleted, a stack leak can occur. In rare circumstances, database corruption can result. Side effect deletes might be caused by an OR REPLACE clause or by triggers. This bug appears to be in all prior versions of SQLite 3. The original problem description follows: ----- My scenario is rather simple - I have products table, and details table. Producs have unique column called code. Then I have transform table to renumber my products table. I used update or replace constraint, and my data got corrupted. Here's example, I will try to rewrite column names into english, sorry if I make some typo here: sql "create table products (code text primary key, former_code text, typy text, name text, warehouse text)" sql "create table transform (code text, name text, new_code text, former_code text, type text)" no indices (there are in newer version of my code, but simpler the case the better) And now sql code, which corrupts my data (contains a bit of REBOL code):
sql trim/lines { update or replace products set code = (select new_code from transform where transform.code=products.code), type = (select type from transform where transform.code=products.code) where exists (select code from transform where transform.code=products.code) }So - I just wonder - I am not fluent in SQL, so I am not sure I should update primary key in products that way? I expect this code would either update codes found for transformation, or it would fail, because such code to translate to would be already present on table, so that would be caught by "or replace" constraint, and such item would be replaced. So - what you suggest now? Is that a bug? Or am I behaving really badly to my data? :-) Tried with 3.5.2, 3.5.3, Windows Vista 32bit .... PS: This text is from my original post to ml - tried to subscribe to ml two times, no luck receiving any message, so posting directly to bugbase. Sorry if the bug is already known ... _2007-Dec-11 14:34:41 by drh:_ {linebreak} What makes you think your data is being corrupted, as opposed to merely transformed in exactly the way you have requested? In other words, why do you think this is a bug in SQLite? ---- _2007-Dec-12 08:22:19 by anonymous:_ {linebreak} If you can see that instead of product code you get characters you even can't write on your keyboard probably, then I think data is being corrupted :-) Here's a screenshot: {link: http://www.xidys.com/pekr/sqlite-insert-or-update-data-corruption.jpg corrupted data?} Cheers, Petr ---- _2007-Dec-12 08:35:14 by danielk1977:_ {linebreak} Can you post a database file and the SQL statement that is causing the corruption? This is likely an encoding problem. Are you inserting UTF-8 text into the database? And handling the returned data as UTF-8? ---- _2007-Dec-12 08:56:10 by anonymous:_ {linebreak} OK, before I post some data, I will try to describe what I did: In products table I have product numbers starting with "9", which are going to be translated into other numbers: 9xxxx.xx ---> 1xxxx.xx 9xxxx.xxA --> 1xxxx.xxA 9xxxx.xxBZ--> 9xxxx.xx I import all data from Excel .csv files. Encoding should be OK imo, as data live in one file. However, I noticed "bug" in my logic. Simply put, to make my life easier, I imported 9xxxx.xxBZ codes into my transform table, thinking that when those codes are at the end of the table, those are OK. But - my update clause (posted in my first request), does not "join data" that safe way, so actually what happens is - if the clause reaches 9xxxx.xxBZ code, it updates it to 9xxxx.xx code, but this code is then once again translated to 1xxxx.xx later in the process. If I don't import 9xxxx.xxBZ codes to the end of the table, my data remain being OK. Where can I send my database? I don't want it being public. Thanks a lot ... Petr ---- _2007-Dec-12 10:25:21 by anonymous:_ {linebreak} OK, here is a reduced recordset without sensitive data. {link: http://www.xidys.com/pekr/inventura.zip corruption data example} Just open the database and apply query as stored in query.txt ---- _2007-Dec-12 11:11:11 by anonymous:_ {linebreak} I can confirm the database corruption as provided in the sample. Before running the SQL in query.txt, =pragma integrity_check= returns 'ok', afterwards it returns these errors: rowid 156 missing from index sqlite_autoindex_products_1 rowid 159 missing from index sqlite_autoindex_products_1 rowid 160 missing from index sqlite_autoindex_products_1 rowid 162 missing from index sqlite_autoindex_products_1 rowid 163 missing from index sqlite_autoindex_products_1 Tested with CVS [4603]. ---- _2007-Dec-12 15:41:33 by anonymous:_ {linebreak} Regarding the test case in Check-in [4614], it produces the same expected result for both 3.5.3 and the latest CVS. No sign of database corruption or valgrind error for 3.5.3 on Linux. Are you certain this isolates the problem? ---- _2007-Dec-12 15:56:02 by drh:_ {linebreak} tkt2832.test provokes the problem if you use the OP_StackDepth opcode from check-in [4612]. The change in [4612], together with additional enhancements that will go in as part of this fix, is designed to head off this kind of bug in the future by making the VDBE very unforgiving to the kinds of code generator errors that caused this problem. #f2dcdc 2831 new active 2007 Dec anonymous 2007 Dec 3 4 alter view View can't be used after ALTER RENAME TO:
SQLite version 3.5.3 Enter ".help" for istructions sqlite> create table t(a); sqlite> create view v1 as select * from t; sqlite> alter table v1 rename to v2; sqlite> select * from v2; SQL error: no such table: v2 sqlite> select * from v1; SQL error: no such table: v1 sqlite> .schema CREATE TABLE t(a); CREATE VIEW v1 as select * from t; sqlite> select * from sqlite_master; table|t|t|2|CREATE TABLE t(a) view|v1|v1|0|CREATE VIEW v1 as select * from tThis is a feature request, not a bug. ---- _2007-Dec-11 18:40:17 by anonymous:_ {linebreak} Notice that alter table doesn't return an error. After the command neither v1 nor v2 can be used. ---- _2007-Dec-13 08:18:16 by danielk1977:_ {linebreak} [4623] improves the situation by returning an error when the user attempts to rename a view. One reason this feature (renaming views) is not a high priority is because a view can be dropped and recreated with a different name efficiently. This was not the case with tables. #c8c8c8 2830 code closed 2007 Dec anonymous 2007 Dec 5 5 tkt2822.test refers to tkt2820 ? typo? sqlite/test/tkt2822.test -> 1.1 do_test tkt2820-1.1 { do_test tkt2820-1.3 { do_test tkt2820-1.3 { Thanks. #cfe8bd 2829 code fixed 2007 Dec anonymous 2007 Dec 1 1 Win32: temp files left over in temp area Seems that temp tables generated on the fly for large group by or order by queries do not have their files removed on CloseHandle. This is because the CreateFile is not done with the right flag. Result is that a lot of files are left over in temp directory, even when queries are run properly. This patch seems to fix it (added: SQLITE_OPEN_DELETEONCLOSE): if( flags & (SQLITE_OPEN_TEMP_DB | SQLITE_OPEN_TEMP_JOURNAL - | SQLITE_OPEN_SUBJOURNAL) ){ + | SQLITE_OPEN_SUBJOURNAL | SQLITE_OPEN_DELETEONCLOSE) ){ _2007-Dec-10 19:54:25 by drh:_ {linebreak} Yikes! I just checked and I have 174260 SQLite temp files in my /var/tmp folder! And cleaning up such a mess is challenging. You cannot do "rm etilqs_*" because the argument list is too long. Even "ls etilqs_*" doesn't work. Here is what I'm doing: ls -U | grep etliqs_ | while read i; do rm $i; done That will have to run for 20 minutes or so to clear out all of the dead files. This problem is serious enough to warrant a new release once we get it fixed. ---- _2007-Dec-10 20:06:32 by anonymous:_ {linebreak} I don't see these temp file on UNIX - which OS is affected by this bug and under what circumstance? ---- _2007-Dec-10 20:50:20 by drh:_ {linebreak} Hmmm. I finally got my /var/tmp folder cleaned up. But now when I rerun the regression suite, no new temp files appear. Perhaps those 174260 temp files came from some older bug that has now been fixed. ---- _2007-Dec-10 21:24:27 by anonymous:_ {linebreak} Did you by chance catch a glimpse of the date of the most recent undeleted tmp files on UNIX? ---- _2007-Dec-11 03:31:38 by drh:_ {linebreak} Sadly, I did not observe the mtimes on any of the files in /var/tmp. Recall that ls was not working very well for me (probably due to the large number of files). I was having to use the -U open (to prevent sorting) in order to get ls to work at all. And I did not have the presence of mind to add a -l to that. Oh well.... ---- _2007-Dec-11 05:26:45 by anonymous:_ {linebreak} find is your friend: find /var/tmp -type f -iname "etliqs_*" -print0 | xargs -r0 rm (yes, you can simplify this a whole lot if you make some assumptions but why bother be unsafe?) #c8c8c8 2828 code closed 2007 Dec anonymous 2007 Dec 1 2 SQLITE_OMIT_SUBQUERY compile error: sqlite3CodeSubselect() missing. The new sqlite3FindInIndex() calls sqlite3CodeSubselect(), but this is kept out if SQLITE_OMIT_SUBQUERY is defined. As it stands now, SQLite does not compiled with SQLITE_OMIT_SUBQUERY defined. #cfe8bd 2827 code fixed 2007 Dec anonymous 2007 Dec 1 1 make test: vacuum2.test did not close all files: 1 Latest unmodified CVS source tree. Run "make test" to see the error.
vacuum2-1.1... Ok vacuum2-2.1... Ok vacuum2-2.1... Ok vacuum2-3.1... Ok vacuum2-3.2... Ok vacuum2-3.3... Ok vacuum2-3.4... Ok vacuum2-3.5... Ok vacuum2-3.6... Ok vacuum2-3.7... Ok vacuum2-3.13... Ok vacuum2-3.14... Ok vacuum2-3.15... Ok vacuum2-3.16... Ok vacuum2-3.17... Ok vacuum2.test did not close all files: 1 ... 1 errors out of 38209 tests Failures on these tests: vacuum2.test All memory allocations freed - no leaks Maximum memory usage: 14164890 bytes make: *** [test] Error 1Note: this does *not* happen if you only run vacuum2.test directory:
$ ./testfixture test/vacuum2.test vacuum2-1.1... Ok vacuum2-2.1... Ok vacuum2-2.1... Ok vacuum2-3.1... Ok vacuum2-3.2... Ok vacuum2-3.3... Ok vacuum2-3.4... Ok vacuum2-3.5... Ok vacuum2-3.6... Ok vacuum2-3.7... Ok vacuum2-3.13... Ok vacuum2-3.14... Ok vacuum2-3.15... Ok vacuum2-3.16... Ok vacuum2-3.17... Ok 0 errors out of 16 tests All memory allocations freed - no leaks Maximum memory usage: 85541 bytes#cfe8bd 2826 build fixed 2007 Dec anonymous 2007 Dec 4 4 test_thread.c won't build if !SQLITE_THREADSAFE || !TCL_THREADS Trying a quick build of the test sources I discovered that test_thread.c won't build the null-test, since it requires tcl.h for a Tcl typedef for the function signature but only includes tcl.h if it thinks it's doing something useful. The following patch fixes the problem:
diff -cr unpack-orig/sqlite-3.5.3/src/test_thread.c sqlite-3.5.3/src/test_thread.c *** unpack-orig/sqlite-3.5.3/src/test_thread.c Mon Sep 10 11:53:02 2007 --- sqlite-3.5.3/src/test_thread.c Thu Dec 6 14:26:16 2007 *************** *** 18,27 **** */ #include "sqliteInt.h" #if SQLITE_THREADSAFE && defined(TCL_THREADS) - #include#f2dcdc 2825 code active 2007 Dec anonymous 2007 Dec 3 3 FormatMessage (win32) should use extra flag and convert from Unicode The call to FormatMessageA in the win32 source code needs to have the flags changed from: FORMAT_MESSAGE_FROM_SYSTEM to FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS This ensures that any system messages that expect arguments do not try to grab the argument from some random memory location. ref: http://blogs.msdn.com/oldnewthing/archive/2007/11/28/6564257.aspx _2007-Dec-06 14:07:53 by anonymous:_ {linebreak} I also noticed that the result is NOT converted to UTF-8. FormatMessageA returns the text in the local ANSI codepage. FormatMessageW should be used on NT systems, and either result should be converted to the SQLite UTF-8 default. ---- _2007-Dec-11 00:34:37 by anonymous:_ {linebreak} to simplify what is meant even more... http://www.sqlite.org/cvstrac/fileview?f=sqlite/src/os_win.c&v=1.118 Search for FormatMessageA (only 1 instance) - FORMAT_MESSAGE_FROM_SYSTEM, + FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS, No breakage, ensures that no crashes with some messages (e.g. filesystem errors). The encoding issue should be addressed separately. ---- _2007-Dec-11 01:27:07 by anonymous:_ {linebreak} The function should be changed to the following to correctly handle the conversion from Unicode/MBCS.#include #include --- 18,27 ---- */ #include "sqliteInt.h" + #include #if SQLITE_THREADSAFE && defined(TCL_THREADS) #include #include
static void winDlError(sqlite3_vfs *pVfs, int nBuf, char *zBufOut){ int error = GetLastError(); #if OS_WINCE if( error>0x7FFFFFF ){ sqlite3_snprintf(nBuf, zBufOut, "OsError 0x%x", error); }else{ sqlite3_snprintf(nBuf, zBufOut, "OsError %d", error); } #else if( isNT() ){ LPWSTR zWinTemp = NULL; DWORD dwLen = FormatMessageW( FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS, NULL, error, 0, (LPWSTR) &zWinTemp, 0, 0 ); if (dwLen > 0) { char * zOut = unicodeToUtf8(zWinTemp); LocalFree(zWinTemp); sqlite3_snprintf(nBuf, zBufOut, "%s", zOut); free(zOut); } }else{ LPSTR zWinTemp = NULL; DWORD dwLen = FormatMessageA( FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS, NULL, error, 0, (LPSTR) &zWinTemp, 0, 0 ); if (dwLen > 0) { char * zOut = mbcsToUtf8(zWinTemp); LocalFree(zWinTemp); sqlite3_snprintf(nBuf, zBufOut, "%s", zOut); free(zOut); } } #endif }#c8c8c8 2824 code closed 2007 Dec anonymous 2007 Dec 1 1 enable shared cache problem System info: -Celeron M. -Windows Xp. -Single process. -No thread. Code: int WINAPI WinMain(HINSTANCE h1, HINSTANCE h2, LPSTR lpCMD, int nSCMD) { sqlite3* db; sqlite3_stmt * stm; sqlite3_enable_shared_cache(1); sqlite3_open("db", &db); sqlite3_prepare_v2(db, "ATTACH DATABASE db2 AS d", 24, &stm, 0); sqlite3_step(stm); } The problem: the sqlite3_step(stm) execute for an infinite time. MA _2007-Dec-06 07:56:12 by danielk1977:_ {linebreak} I just tried the following program with the cvs version: #include "sqlite3.h" int main (int argc, char **argv){ sqlite3* db; sqlite3_stmt *stm; sqlite3_enable_shared_cache(1); sqlite3_open("db", &db); sqlite3_prepare_v2(db, "ATTACH DATABASE db2 AS d", 24, &stm, 0); sqlite3_step(stm); return 0; } And the sqlite3_step() did not block for any amount of time. Can you confirm that the code above produces the bug in your environment? Thanks. ---- _2007-Dec-07 13:59:02 by anonymous:_ {linebreak} I have ran more test but the problem is never arised. I think it was generated from code or configuration properties from my system, outside sqlite. MA #c8c8c8 2823 code closed 2007 Dec anonymous 2007 Dec 1 1 query not return values I have a database with this description:
and contain this data:CREATE TABLE [coletores] ( [coletor] INTEGER NOT NULL ON CONFLICT ABORT PRIMARY KEY, [senha] NVARCHAR2(20) NOT NULL ON CONFLICT ABORT); CREATE UNIQUE INDEX [SenhaUnica] ON [coletores] ([senha]);
When i run the query: "select coletor, senha from coletores where senha = '43043553'" query not return values. Works when I try it. Perhaps you have some whitespace on one side of the '43043553' value in the database. Whitespace is significant to SQLite. #cfe8bd 2822 code fixed 2007 Dec anonymous 2007 Dec 3 3 ORDER BY term number 1 does not match any result column in select.c The original problem description is preserved below. This text is summary of what has changed. After this change the algorithm for matching ORDER BY terms is as follows: 1: If an ORDER BY term is a constant integer k then sort by the k-th column of the result set. 2: If an ORDER BY term is a simple identifer (like "x", not "x.y" and not "x.y.z") and if there if the k-th column uses that same identifer as an AS alias, the sort by the k-th column. 3: Otherwise, evaluate the expression which is in the ORDER BY term and sort by the resulting value. For a compound query, the expression in step (3) must exactly match one of the result columns. Also, the three steps are attempted first on the left-most SELECT. If there is a match, the process stops there. If no match is found, the next SELECT to the right is tried. This repeats as necessary until a match is found or until you run out of SELECT statement in which case there is an error. This algorithm differs from all prior versions of SQLite (1.0.0 through 3.5.3) by the addition of step (2). Adding step (2) brings SQLite much closer to the SQL standard. I believe that SQLite is now a superset of the SQL standard. In a compound query, SQL only looks at the left-most SELECT and does not fail over to SELECT statements to the right looking for a match. The difference in SQLite can be considered an extension. The revised algorithm is mostly compatible with the way SQLite has always operated before. But there are a few obscure corner cases where there is a difference. An example of the difference is the following: CREATE TABLE a(x,y); INSERT INTO a VALUES(1,8); INSERT INTO a VALUES(9,2); SELECT x AS y FROM a ORDER BY y; In older versions of SQLite, the SELECT statement above would return 9, 1 since the ORDER BY term evaluated to the expression a.y by rule (3) In the next release, because of the addition of rule (2) above, the result will be 1, 9. -------- *Original Problem Description* Starting from version 3.4.2 I receive errors with queries like: SELECT a.field FROM a UNION ALL SELECT b.field FROM b ORDER BY a.field; or even SELECT a.field FROM a UNION ALL SELECT a.field FROM a ORDER BY a.field; error is: ORDER BY term number 1 does not match any result column Tables are created by: CREATE TABLE a (field); CREATE TABLE b (field); Please note that the above queries worked fine with sqlite 3.2.x or 3.3.x. it's odd that this one doesn't work as well: create table t1(a); create table t2(b); select t1.a from t1 union all select t2.b from t2 order by a; SQL error: ORDER BY term number 1 does not match any result column _2007-Dec-04 04:35:48 by anonymous:_ {linebreak} Analysis from mailing list: http://www.mail-archive.com/sqlite-users%40sqlite.org/msg29701.html It appears as though the /src/select.c (Line1499) changed from: if( iCol<0 && mustComplete ){ to: }else if( mustComplete ){ in version 1.336 of this file - http://www.sqlite.org/cvstrac/filediff?f=sqlite/src/select.c&v1=1.335&v2=1.336 And this change results in this bug. ---- _2007-Dec-04 04:38:57 by anonymous:_ {linebreak} Related to Check-in [3842] : Match ORDER BY terms to columns using names in compound queries. Make sure this works for subqueries, especially in the right-hand side of an IN operator. Ticket #2296. ---- _2007-Dec-04 16:27:41 by anonymous:_ {linebreak} Other databases produce results for the following that differ from sqlite: create table x1(a INT, b INT, c INT); insert into x1 values(1, 2, 3); create table x2(a INT, b INT, c INT); insert into x2 values(9, 0, 4); SELECT x1.b, a FROM x1 UNION SELECT a, b FROM x2 ORDER BY b; SQLite version 3.5.3 sqlite> SELECT x1.b, a FROM x1 UNION SELECT a, b FROM x2 ORDER BY b; x1.b|a 9|0 2|1 versus: mysql> SELECT x1.b, a FROM x1 UNION SELECT a, b FROM x2 ORDER BY b; +------+------+ | b | a | +------+------+ | 2 | 1 | | 9 | 0 | +------+------+ Also, for this SQL: create table x1(a INT, b INT, c INT); insert into x1 values(1, 2, 3); create table g2(x INT, y INT, z INT); insert into g2 values(9, 0, 4); sqlite 3.5.3 produces: sqlite> SELECT x1.b, a FROM x1 UNION SELECT x, y FROM g2 ORDER BY y; x1.b|a 9|0 2|1 compared to an error on other databases that only use the first SELECT to determine ORDER BY/GROUP BY column names: mysql> SELECT x1.b, a FROM x1 UNION SELECT x, y FROM g2 ORDER BY y; ERROR 1054 (42S22): Unknown column 'y' in 'order clause' ---- _2007-Dec-05 05:14:38 by anonymous:_ {linebreak} Another related issue in 3.5.3+ CVS: CREATE TABLE foo(b); These are fine: SELECT 3 b union select b from foo order by b; 3 SELECT b from foo union SELECT 3 b order by b; 3 These have issues: SELECT b from foo union SELECT 3 b order by +b; SQL error: ORDER BY term number 1 does not match any result column SELECT 3 b union select b from foo order by +b; SQL error: ORDER BY term number 1 does not match any result column ---- _2007-Dec-05 05:34:23 by anonymous:_ {linebreak} A problem with expressions in an ORDER BY of a compound statement: CREATE TABLE t1(b); INSERT INTO t1 VALUES(-3); INSERT INTO t1 VALUES(2); -- works, of course select b from t1 order by b*b; 2 -3 -- expression does not work in compound SELECT select b from t1 union all select b from t1 order by b*b; SQL error: ORDER BY term number 1 does not match any result column -- an alias does not help select b as b from t1 union all select b from t1 order by b*b; SQL error: ORDER BY term number 1 does not match any result column select b as b from t1 union all select b as b from t1 order by b*b; SQL error: ORDER BY term number 1 does not match any result column ---- _2007-Dec-08 04:08:53 by anonymous:_ {linebreak} The column heading names in the result set of a compound query are not consistent with regular queries:coletor | senha 1 43043553
SQLite version 3.5.3 Enter ".help" for instructions sqlite> create table foo(a); sqlite> insert into foo values(1); sqlite> .header on sqlite> select foo.a from foo; a 1 sqlite> select foo.a from foo union all select foo.a from foo; a 1 1 sqlite> select foo.a from foo union all select foo.a from foo order by 1; foo.a <===== 1 1 sqlite> select foo.a from foo union all select foo.a from foo group by 1; a 1 1---- _2007-Dec-08 17:43:29 by anonymous:_ {linebreak} Possible fix: http://www.mail-archive.com/sqlite-users%40sqlite.org/msg29790.html http://marc.info/?l=sqlite-users&m=119708950929249&q=p3 ---- _2007-Dec-10 19:35:33 by anonymous:_ {linebreak} What should the following return? create table t1(a INT, b INT, c INT); insert into t1 values(1, 2, 4); insert into t1 values(2, -1000, 5); select a, a+b AS c from t1 order by c; sqlite 3.5.3: a|c 1|3 2|-998 mysql and postgres: a | c ---+------ 2 | -998 1 | 3 ---- _2007-Dec-13 08:13:24 by anonymous:_ {linebreak} It's unfortunate that the expression support in compound ORDER BY which existed briefly in CVS for a couple of days was removed: sqlite/test/tkt2822.test 1.2 -> 1.3 - SELECT a+1, b+1 FROM t1 UNION ALL SELECT a, c FROM t2 ORDER BY a+1; + SELECT a, b, c FROM t1 UNION ALL SELECT a AS x, b, c FROM t2 ORDER BY x; It was a useful extension. ---- _2007-Dec-13 16:05:10 by drh:_ {linebreak} The support is still there. We just removed the test case. I'm not sure why. Probably we should put it back... ---- _2007-Dec-13 16:22:42 by anonymous:_ {linebreak} Are you sure? From latest CVS: sqlite> select 3 as A, 6 as B union select 1 as B, 5 as A order by A; 1|5 3|6 sqlite> select 3 as A, 6 as B union select 1 as B, 5 as A order by +A; SQL error: 1st ORDER BY term does not match any column in the result set sqlite> select 3 as A, 6 as B union select 1 as B, 5 as A order by A+1; SQL error: 1st ORDER BY term does not match any column in the result set sqlite> select 3 as A, 6 as B union select 1 as B, 5 as A order by 9*A-A*A; SQL error: 1st ORDER BY term does not match any column in the result set select * from (select 3 as A, 6 as B union select 1 as B, 5 as A) order by 9*A-A*A; 1|5 3|6 ---- _2007-Dec-13 17:01:22 by drh:_ {linebreak} In a compound SELECT, the ORDER BY terms must match an expression in the result set exactly. "A" does not exactly match "+A". But "a+1" does exactly match "a+1". To keep things relatively simple, we do not allow equivalent expressions. "1+a" does not match "a+1". The expressions must be identical (from the point of view of the parser - differences in whitespace don't matter of course since that is all stripped out by the tokenizer.) In a simple SELECT, if an ORDER BY term does not exactly match a result set expression, we can add an additional "virtual" result column to contain the ORDER BY term. But we cannot do that in a compound SELECT because we would have to add such "virtual" columns to all SELECTs in the compound and the ORDER BY expression typically only makes sense for one of them. ---- _2007-Dec-13 17:28:33 by anonymous:_ {linebreak} Some popular databases do in fact add "virtual" columns you describe and support ORDER BY expressions not explicitly mentioned in any compound SELECT. mysql> select 3 as A, 6 as B union select 1 as B, 5 as A order by -A*A; +---+---+ | A | B | +---+---+ | 3 | 6 | | 1 | 5 | +---+---+ It's a useful extension, but not a show stopper. One could manually do to the transformation identified above to make the compound query a subquery and move the ORDER BY to the parent. I don't know if it is any less efficient. #f2dcdc 2821 new active 2007 Dec anonymous 2007 Dec 3 4 hashtable indicies It would be nice to implement non btree indices. I.e. CREATE INDEX ON table(rowid) AS HASH. Using a hashtable's O(1) properties, you could use the index for very quick lookups when one result is expected. This does have the tradeoff that a hashtable index has no ordering properties (can not be used for sorts or non-equality searching). However, it would be a *huge* win when you have 250,000 rowids in memory, and you want to go fetch another column in the database for each one of those rowids (SELECT * FROM table WHERE rowid=?). _2007-Dec-03 21:58:01 by anonymous:_ {linebreak} For 250,000 rows I doubt you would see that much of an improvement (try it.) You'll almost certainly find log_n is going to be fairly fast (especially for large n.) I personally would prefer some sort of 'virtual' index though, that could be a hash or actually from a user-supplied function so that I can index large blobs by some function (i.e. a hash). And yes, this would be an incompatible file-format change and it's not clear how to update an index when the function isn't loaded (i.e. db reopened with that function.) Perhaps mark the index as 'stale' and ignore it until the function loads then you can do the updates. Of course this starts to get quite complicated. ---- _2007-Dec-03 22:12:17 by anonymous:_ {linebreak} Everything in sqlite depends on btree indexes. You're talking a major rewrite if you support hash-based or other indexing. #cfe8bd 2820 code fixed 2007 Dec anonymous 2007 Dec 1 1 rollback doesn't work Code:
prints: database table is locked In sqlite rolback is called (I'v checked in older version), but in test.db there is no table 't'. Tested on win2k. _2007-Dec-03 16:31:50 by anonymous:_ {linebreak} What is the desired outcome? ---- _2007-Dec-03 18:28:38 by anonymous:_ {linebreak} No error. Or if error is reported during delete, nothing should be deleted. #c8c8c8 2819 code closed 2007 Dec anonymous 2007 Dec 2 1 Blocking and crashing First it was blocking when using perl lib DBI, i tried to update the packages and finally reinstall them and now i get this:#include#include "sqlite3.h" sqlite3 *db; static int callback(void *NotUsed, int argc, char **argv, char **azColName){ sqlite3_exec(db, "DROP TABLE t", 0, 0, 0); printf("%s\n", sqlite3_errmsg(db)); return 0; } int main() { sqlite3_open("test.db", &db); sqlite3_exec(db, "CREATE TABLE IF NOT EXISTS t(a INT)", 0, 0, 0); sqlite3_exec(db, "select name, type from sqlite_master", &callback, 0, 0); sqlite3_close(db); return 0; }
mush@server:~$ /usr/bin/sqlite3 ./Bureau/lexfix/lexFix/test.dat SQLite version 3.3.10 Enter ".help" for instructions sqlite> .schema /usr/bin/sqlite3: symbol lookup error: /usr/bin/sqlite3: undefined symbol: sqlite3_enable_load_extension mush@server:~$ /usr/bin/sqlite3 ./Bureau/lexfix/lexFix/test.dat /usr/bin/sqlite3: symbol lookup error: /usr/bin/sqlite3: undefined symbol: sqlite3_enable_load_extensionI think this is a support request, not a bug report. Please take your comments to the SQLite mailing list. #c8c8c8 2818 code closed 2007 Dec anonymous 2007 Dec 2 4 Comparison of 8 bytes integer numbers is truncated sqlite> select -9223372036854775808>=-9223372036854775808; 1 sqlite> select -9223372036854775809>=-9223372036854775808; 1 sqlite> select -9223372036854875808>=-9223372036854775808; 0 sqlite> select -9223372036854775808>=-9223372036854775808; 1 This is the correct answer. sqlite> select -9223372036854775809>=-9223372036854775808; 1 -9223372036854775809 is *not* an 8-byte integer. The smallest possible 8-byte integer is -9223372036854775808. The -9..809 value is a 64-bit IEEE floating point number: -9.22337203685488e+18. In order to perform the comparison, SQLite converts the number on the right to floating point value. Due to the limited precision of floating point numbers, both values are the same and so the values compare equal to one another. Hence the result shown above is correct. sqlite> select -9223372036854875808>=-9223372036854775808; 0 This is the correct answer. No bugs seen here. #c8c8c8 2817 code closed 2007 Dec danielk1977 2007 Dec danielk1977 1 1 Temp-tables with the same name as database tables can cause corruption Suspect a problem in the CREATE INDEX statement here: CREATE TEMP TABLE abc(a, b, c); CREATE TABLE main.abc(a, b, c); CREATE INDEX main.abc_i ON abc(a, b, c); The attached script demonstrates the problem. #c8c8c8 2816 code closed 2007 Dec anonymous 2007 Dec drh 4 3 .timer missing from CLI .timer not valid for 3.5.2/3.5.3 It works when I try it. ---- _2007-Dec-04 14:48:14 by anonymous:_ {linebreak} No .timer support on Windows. ---- _2007-Dec-04 15:31:30 by drh:_ {linebreak} That is correct - there is no timer support on windows. But on the other hand, we never claimed there was. The ".timer" command does not show up in the help menu on windows. #cfe8bd 2815 doc fixed 2007 Dec anonymous 2007 Dec 2 3 sqlite3_blob_close closes blob even when reporting error! As part of my testing, I open a blob read only and then do some writing to it. On calling sqlite3_blob_close, I get back error 8 (SQLITE_READONLY), but the blob is closed anyway. Normal practise for file like apis is that if close returns an error, the handle remains open. Please either make SQLite match other apis, or update the documentation to mention that the blob handle is always closed no matter what the return. It is probably also a good to mention that the return of close must be checked since that is where errors are returned (ie view it as a commit). The Linux close(2) man page contains advise that translates to sqlite3_blob_close as well http://linux.die.net/man/2/close #f2dcdc 2814 code active 2007 Nov anonymous 2007 Dec 3 3 _XOPEN_SOURCE again Ideally setting _XOPEN_SOURCE should be an opt-in detected by configure, rather than a hardcoded opt-out as it is now. I find you create more problems in setting it than just leaving it out on modern platforms. Can you please give users the option of not defining _XOPEN_SOURCE at all?
+#ifndef SQLITE_DONT_DEFINE_XOPEN_SOURCE #if !defined(_XOPEN_SOURCE) && !defined(__DARWIN__) && SQLITE_THREADSAFE # define _XOPEN_SOURCE 500 /* Needed to enable pthread recursive mutexes */ #endif +#endif_2007-Dec-01 09:23:15 by anonymous:_ {linebreak} Also when using Python, it sets _XOPEN_SOURCE to 600. No idea what the 500 vs 600 difference is about. ---- _2007-Dec-01 15:58:28 by anonymous:_ {linebreak} I've used a couple of different Linux OSes and _XOPEN_SOURCE is not needed. Maybe it's for OSes more than 5 years old. Recursive mutexes are pretty much standard these days since the popularity of Java which uses them extensively. ---- _2007-Dec-01 17:21:05 by drh:_ {linebreak} See also tickets #2673, #2681, and #2741. ---- _2007-Dec-02 02:08:26 by anonymous:_ {linebreak} On Linux, PTHREAD_MUTEX_RECURSIVE is the same as PTHREAD_MUTEX_RECURSIVE_NP: PTHREAD_MUTEX_RECURSIVE = PTHREAD_MUTEX_RECURSIVE_NP, Since PTHREAD_MUTEX_RECURSIVE_NP is always available, you could avoid defining _XOPEN_SOURCE and use this code instead:
- pthread_mutexattr_settype(&recursiveAttr, PTHREAD_MUTEX_RECURSIVE); + pthread_mutexattr_settype(&recursiveAttr, +#ifdef linux + PTHREAD_MUTEX_RECURSIVE_NP +#else + PTHREAD_MUTEX_RECURSIVE +#endif + );---- _2007-Dec-02 02:17:22 by anonymous:_ {linebreak} A quick google search reveals how various projects deal with this recursive mutex declaration problem (in no particular order): *: #define _XOPEN_SOURCE 500 and use PTHREAD_MUTEX_RECURSIVE *: #define _XOPEN_SOURCE 600 and use PTHREAD_MUTEX_RECURSIVE *: #define _GNU_SOURCE and use PTHREAD_MUTEX_RECURSIVE *: don't define anything and use PTHREAD_MUTEX_RECURSIVE_NP on linux, and PTHREAD_MUTEX_RECURSIVE elsewhere. Unfortunately, since PTHREAD_MUTEX_RECURSIVE is an enum on Linux, so you can't use the #ifdef PTHREAD_MUTEX_RECURSIVE compile-time technique. #f2dcdc 2813 build active 2007 Nov anonymous 2007 Nov 1 1 compile error on Windows CE environment: visual c++ 2005 window ce 6.0 customize sdk sqlite-amalgamation-3_5_3 I get error: Error 27 error C2040: 'localtime' : 'tm *(const time_t *)' differs in levels of indirection from 'int ()' d:\SubProjects\Sqlite\sqlite3.c 18574 but if I add code in line 7095: struct tm *__cdecl localtime(const time_t *t); then Success! #c8c8c8 2812 new closed 2007 Nov anonymous 2007 Dec 3 4 SQLITE_SECURE_DELETE overwrites with 0, but should use random data Hello Dr. Hipp, SQLITE_SECURE_DELETE overwrites with 0, but should use random data. I'm not an expert in this area, but it also sounds like secure delete should involve random data then writing to disk several times in a row. For the firefox bug that relates to this, please see https://bugzilla.mozilla.org/show_bug.cgi?id=405925 Thanks, -Seth _2007-Nov-29 18:40:32 by anonymous:_ {linebreak} This is a questionable enhancement to security because you cannot ensure where the disk controller will actually write the data on the media. The old fragment may exist elsewhere on the disk. Flash memory cards try to even out where blocks are written to on the media because of the finite number of writes allowed, for example. It's better to encrypt the whole database file. ---- _2007-Nov-29 21:19:34 by anonymous:_ {linebreak} dr. hipp writes: I don't think writing random bits or writing multiple times will help any. There are too many layers (operating system, file system, disk driver, and disk controller) in between SQLite and the oxide for SQLite to really have much control over what is truly stored on media. SQLite might write the same page multiple times, for example, but the OS would likely coalese the writes into a single write. Perhaps we could force multiple writes using fsync() but that would be a *huge* performance penalty. And even if we did, just because we write to the same offset in some file does *not* mean that the information is written to the same place on mass storage. Some filesystems (YAFFS) intentionally spread the writes around to different places for the purpose of wear leveling. And even if the operating system doesn't do this, then on-drive disk controller might. What problem, exactly, are you trying to solve? The existing SQLITE_SECURE_DELETE is adequate to prevent casual snooping. It is not, as you observe, adequate to prevent snooping from major governments. Against a determined adversary, the only sure defense is to physically shred the disk platter. ---- _2007-Nov-29 21:20:33 by anonymous:_ {linebreak} in https://bugzilla.mozilla.org/show_bug.cgi?id=405925#c4, jesse writes: I don't think it's worth the effort and perf hit to attempt to protect Firefox profile data from forensic data recovery. It's really hard to get right, especially at the app level, and we don't want to give users a false sense of security by trying. Any given data is likely to appear in many physical locations on the disk due to swapping, journaling, automated backups, and continuous defragmentation [1]. When you try to overwrite part of a file, you find yourself fighting against optimizations in operating systems and in disks themselves [2]. Data also appears in RAM, which can survive power cycling [2] and is even harder to control. Zeroing sounds reasonable; it will protect against casual snooping, and depending on the file system, might protect against forensic software. But I don't think we should go beyond zeroing (e.g. multiple overwrites) in an attempt to defeat forensic analysis of the physical hard drive and RAM. Extremely paranoid users can use operating system features to encrypt an entire hard drive or automatically do multiple overwrites every time data on disk as it is changed. [1] http://en.wikipedia.org/wiki/Disk_wipe [2] http://en.wikipedia.org/wiki/Data_remanence ---- _2007-Nov-29 21:21:02 by anonymous:_ {linebreak} based on these responses, this seems like a "wontfix" thanks, -Seth #cfe8bd 2811 code fixed 2007 Nov anonymous 2007 Nov 5 4 "appropriate" misspelled "approriate" on SQLite Documentation page In the description of the "Appropriate Uses For SQLite" document on the SQLite Documentation page at http://www.sqlite.org/docs.html, "appropriate" is misspelled "approriate". #f2dcdc 2810 code active 2007 Nov anonymous 2007 Nov 1 1 Unregistered collation problems with simple subselects As discussed on the mailing-list: Imagine that a SQLite3 database opened in a custom application with a registered a collation sequence named "unknown" has created the following table: CREATE TABLE a (b COLLATE unknown); Now open this table in the default SQLite3 CLI. Up to here, everything works as expected. Simple queries like "SELECT * FROM a;" work fine. But subselects, in their most basic form and with no sorting or comparisons, result in an error: sqlite> INSERT INTO a VALUES ('one'); sqlite> SELECT * FROM a, (SELECT * FROM a); SQL error: no such collation sequence: unknown sqlite> SELECT * FROM (SELECT * FROM a); SQL error: no such collation sequence: unknown sqlite> SELECT *, * FROM a; one|one This is surprising because the collation sequence should not matter to the queries. In fact, the union without the subselect works just fine and without errors. To demonstrate, here is the explain output of a table with a registered collation sequence. No mention of the collation name here: sqlite> CREATE TABLE b (b collate nocase); sqlite> EXPLAIN SELECT * FROM b, (SELECT * FROM b); 0|Goto|0|17| 1|Integer|0|0| 2|OpenRead|0|3| 3|SetNumColumns|0|1| 4|Integer|0|0| 5|OpenRead|2|3| 6|SetNumColumns|2|1| 7|Rewind|0|14| 8|Rewind|2|13| 9|Column|0|0| 10|Column|2|0| 11|Callback|2|0| 12|Next|2|9| 13|Next|0|8| 14|Close|0|0| 15|Close|2|0| 16|Halt|0|0| 17|Transaction|0|0| 18|VerifyCookie|0|4| 19|TableLock|0|3|b 20|Goto|0|1| 21|Noop|0|0| #f2dcdc 2809 code active 2007 Nov anonymous 2007 Nov 1 1 PRAGMA collation_list shows unregistered collations As presented on the mailing list: Imagine that a SQLite3 database opened in a custom application with a registered a collation sequence named "unknown" has created the following table: CREATE TABLE a (b COLLATE unknown); Now open this table in the default SQLite3 CLI. Up to here, everything works as expected. Next issue "PRAGMA collation_list;" and notice that "unknown" lists next to the other registered collations, even though "unknown" is not registered with the default SQLite3 CLI: sqlite> PRAGMA collation_list; 0|unknown 1|NOCASE 2|BINARY Responses from the mailing list indicate that this is not the expected behaviour. "PRAGMA collation_list;" should list registered collations only. _2007-Nov-28 16:12:17 by anonymous:_ {linebreak} I don't think this is a bug. If the CLI is not aware of the collation, it should not process the query that makes use of the collation because it would certainly be wrong if it simply ignored the collation. This is not unlike a user-registered SQL function that does not exist in the CLI. I would not expect or want the sqlite3 CLI to ignore the unknown function, nor would I want the CLI to process queries ignoring the custom collation. #f2dcdc 2808 doc active 2007 Nov anonymous 2007 Nov 4 2 Documentation GIF images take up too much space The GIF format used for the documentation images takes up too much space. I believe that this is in strict contrast to the "small" feature of SQLite. Converting the GIFs to PNG images saves up to 181 KB (64%) of the documentation storage space: Documentation images as is (mostly GIF): 381 KB == 100% Documentation images as PNG: 292 KB == 77% Documentation images as PNG 16 colors: 137 KB == 36% 16 colors are more than plenty -- the diagrams show no visibility degradation. You might even cut it down to 8 colors to save even more ... #cfe8bd 2807 doc fixed 2007 Nov anonymous 2007 Nov 4 3 Link errors in Documentation There are link errors in these documentation files: *atomiccommit.html:*
SQLite allows a single database connection to talk to two or more database files simultaneously through the use of the ATTACH DATABASE command. --- and step 3.10. --- During recovery at step 4.2 SQLite locates *lang_insert.html:* --- single keyword REPLACE as an #cfe8bd 2806 code fixed 2007 Nov anonymous 2007 Nov 3 4 Low memory assert in insertCell() The call to allocateSpace() at line 4511 of btree.c can fail, returning zero if the nested call to defragmentPage() fails to allocate its temp buffer. This then triggers the following assert. idx = allocateSpace(pPage, sz); assert( idx>0 ); For the older versions (ours is derived from 3.3.8) the solution is quite simple: idx = allocateSpace(pPage, sz); /* Check for memory allocation errors - pre v 3.5.0 */ if (sqlite3MallocFailed()) return SQLITE_NOMEM; assert( idx>0 ); For later versions you probably need something more like: if (idx == 0) return SQLITE_NOMEM; replacing the assert. This was found running test join-4.4 (from 3.3.8). _2007-Nov-28 15:47:39 by anonymous:_ {linebreak} Out of curiosity, which sqlite memory allocator are you using to find these low memory problems? The default malloc one, or the heap-in-an-array one? ---- _2007-Nov-28 17:03:19 by anonymous:_ {linebreak} Sorry, it's a proprietary allocator. But we do try to share the 'benefits'! #cfe8bd 2805 code fixed 2007 Nov anonymous 2007 Nov 1 1 Checkin [4573] Home grown recursive mutexes Checkin [4573] would only work on uniprocessors, not SMP hardware. You cannot implement recursive mutexes without some sort of memory barrier or atomic "test and set" instruction. The reason being the each SMP CPU has its own memory image that is not synchronized with main memory until there is a memory barrier or lock/spinlock. Take a look at NPTL or the previous incarnation, LinuxThreads, to see how difficult it is to implement a reliable and efficient recursive mutex on different CPU architectures. It simply can't be done without dropping into architecture-specific assembler. _2007-Nov-28 17:46:31 by anonymous:_ {linebreak} Solaris 2.6 was mentioned in the original homegrown recursive mutex implementation. Be aware that some Sparc machines offer cache coherency, while others do not. So it's a run-time hardware-specific thing, not an OS thing or a compile-time thing. Here is a fairly comprehensive list of cache coherency info (circa 2003) for various CPUs: http://linux.derkeiler.com/Mailing-Lists/Kernel/2003-09/3187.html #c8c8c8 2804 code closed 2007 Nov anonymous 2007 Dec 1 1 PRAGMA legacy_file_format=OFF setting lost with VACUUM It seems that the PRAGMA legacy_file_format=OFF and the ability to use DESC indexes is lost after two VACUUMs and reconnects.
$ rm -f foo.db $ ./sqlite3-3.5.3.bin foo.db SQLite version 3.5.3 Enter ".help" for instructions sqlite> PRAGMA legacy_file_format; 1 sqlite> PRAGMA legacy_file_format=OFF; sqlite> PRAGMA legacy_file_format; 0 sqlite> CREATE TABLE abc(a,b,c); sqlite> CREATE INDEX abc_i on abc(b desc, c asc, a desc); sqlite> explain query plan select * from abc order by b desc, c asc, a desc; 0|0|TABLE abc WITH INDEX abc_i ORDER BY sqlite> vacuum; sqlite> .qIn the next connection we see that the legacy file format reverted back to 1, but the DESC index is still picked up...
$ ./sqlite3-3.5.3.bin foo.db SQLite version 3.5.3 Enter ".help" for instructions sqlite> PRAGMA legacy_file_format; 1 sqlite> explain query plan select * from abc order by b desc, c asc, a desc; 0|0|TABLE abc WITH INDEX abc_i ORDER BY sqlite> vacuum; sqlite> .qBut if connected to another time, the DESC index is not picked up...
$ ./sqlite3-3.5.3.bin foo.db SQLite version 3.5.3 Enter ".help" for instructions sqlite> PRAGMA legacy_file_format; 1 sqlite> explain query plan select * from abc order by b desc, c asc, a desc; 0|0|TABLE abcActually, this works correctly, though the documentation could be clearer. The legacy_file_format pragma does not tell you the file format of the current database. Rather, it tells you what file format will be used when creating new databases (for example, using ATTACH). The behavior shown above is correct. There is no pragma to tell you what the file format of the current database is. The documentation has been clarified. Two new test cases have been inserted into the test suite to verify that the operation is correct. #cfe8bd 2803 code fixed 2007 Nov anonymous 2007 Nov 4 3 test: io-4.2.2 fails on 64-bit --- signed extension io-4.2.2 expects *0xfffffff* but gets *0xffffffffffffffff* format 0x%X $res is this a 64-bit sign-extension quirk wrt to tcl? _2007-Nov-27 20:48:24 by anonymous:_ {linebreak} Suggested fix: --- a/test/io.test +++ b/test/io.test @@ -449,6 +449,7 @@ if {$::tcl_platform(platform)=="unix"} { set blob [read $fd 4] close $fd binary scan $blob i res + set res [expr $res & 0xFFFFFFFF] format 0x%X $res } {0xFFFFFFFF} } #c8c8c8 2802 code closed 2007 Nov anonymous 2007 Dec 1 1 SELECT UNION and EXCEPT seem to have a memory allocation bug It seems that SELECT UNION and EXCEPT operators work wrong. It looks like memory access overflow or integer overflow. Well, I have a large text file 'text2.text' (~1.5mg), which contains: 20091208{linebreak} 20216510{linebreak} 20149599{linebreak} 20250484{linebreak} 20211910{linebreak} 20199980{linebreak} ........{linebreak} 20183528{linebreak} 20151994{linebreak} 20179184{linebreak} 20077607{linebreak} 182500 lines in total. The meaning of these numbers is a date in YYYYMMDD format. Numbers are random and unsorted at all. The next file is 'test1.text' is the same as 'test2.text' but without the last 1000 lines. The 'diff' between these files shows the difference in last 1000 lines. Than I execute script: @echo off{linebreak} if exist test.db del test.db{linebreak} rem{linebreak} rem Load data into table 'a'{linebreak} rem It prints 181500{linebreak} sqlite3 test.db "create table a (i1 text)"{linebreak} sqlite3 test.db ".import test1.text a"{linebreak} sqlite3 test.db "select count(*) from a"{linebreak} rem{linebreak} rem Load data into table 'b'{linebreak} rem It prints 182500{linebreak} sqlite3 test.db "create table b (i1 text)"{linebreak} sqlite3 test.db ".import test2.text b"{linebreak} sqlite3 test.db "select count(*) from b"{linebreak} rem{linebreak} rem PROBLEM IS HERE{linebreak} rem Now I am trying to merge the tables 'a' and 'b' via UNION.{linebreak} rem It should print '182500' but it prints '16384' !!!{linebreak} sqlite3 test.db "create table c (i1 text)"{linebreak} sqlite3 test.db "insert into c select * from b union select * from a"{linebreak} sqlite3 test.db "select count(*) from c"{linebreak} rem{linebreak} rem Just a check that EXCEPT also has the same bug.{linebreak} rem It should print 1000, but it prints '0' !!!{linebreak} sqlite3 test.db "select count(*) from (select * from c except select * from b)"{linebreak} I suggest that UNION and EXCEPT has some kind of memory overflow. And also it depends on how strong the numbers in the source text file are shuffled and how wide is the range of the numbers. I can attach my files 'test1.text' and 'test2.text' if needed. My system: CPU Intel Core 2 Duo, 1.06GHz, 2GB RAM. Operating system: Windows XP SP2 (Version 5.1.2600) _2007-Nov-27 23:45:42 by drh:_ {linebreak} UNION and EXCEPT both imply DISTINCT. Since you are dealing with dates, it seems unlikely that you really have 182500 distinct dates. That would span 500 years and your sample data seems to all be from the early part of the 21st century. 16384 distinct dates would span 44 years, which seems more reasonable, given your sample data. The fact that the number is a power of two probably results from a poor-quality PRNG used to generate the same data. You can use UNION ALL to to get the union of two tables without the implied DISTINCT. There is no way to omit the implied DISTINCT from EXCEPT. You will probably need to refactor your query to use the NOT IN operator. ---- _2007-Dec-05 20:10:08 by anonymous:_ {linebreak} Actually, in the real data, when I've got this situation on EXCEPT and UNION, there are some extra fields in the database. Except 'date', there were text fields: 'amount', 'owner', 'descr', 'labels' (it's personal cash accouting database). When the test data was generatated, all fields could have duplicate values, but 'amount' field was generated like: $amount = sprintf( "%d.%02d", rand( 1, 10000 ), rand( 0, 99 ) ); so, it can take 9999999 possible values, which is more than 16384 for sure. And the behaviour of the EXCEPT was completely the same as I've described the the original message of this bug report. That's why I've decided that the key of the problem is just in the amount of data, not in values, and I've reduced the database structure for the bug report to make it more clear (I left only one field 'date'), but the 'bug' (I mean this strange behaviour of the EXCEPT and UNION) still happens in the reduced database structure (with only one field 'date'). So DISTINCT function will not merge all values of 'amount' field into 16384 unique values. What do you think? #cfe8bd 2801 code fixed 2007 Nov anonymous 2007 Nov 5 5 superfluous -lpthread for CLI in publish_osx.sh -lpthread not needed in publish_osx.sh: + CFLAGS="-Os -DSQLITE_ENABLE_FTS3=1 -DSQLITE_THREADSAFE=0" + NAME=sqlite3-$VERS-osx-x86.bin + echo '***** '"COMPILING $NAME..." + gcc $CFLAGS -Itsrc sqlite3.c tsrc/shell.c -o $NAME -ldl -lpthread #cfe8bd 2800 build fixed 2007 Nov anonymous 2007 Nov 5 5 -lpthread not needed for publish.sh for CLI binary -lpthread not needed by publish.sh for CLI binary since not threadsafe build. +CFLAGS="-Os -DSQLITE_ENABLE_FTS3=1 -DSQLITE_THREADSAFE=0" +echo '***** '"COMPILING sqlite3-$VERS.bin..." +gcc $CFLAGS -Itsrc sqlite3.c tsrc/shell.c -o sqlite3 -ldl -lpthread ldd ./sqlite3-3.5.3.bin linux-gate.so.1 => (0xb7f1d000) libdl.so.2 => /lib/libdl.so.2 (0xb7f05000) libpthread.so.0 => /lib/i686/libpthread.so.0 (0xb7ef2000) libc.so.6 => /lib/i686/libc.so.6 (0xb7dc5000) /lib/ld-linux.so.2 (0xb7f1e000) Unrelated - why are the shared libs and binary chmod'ed to be not executable? chmod 644 sqlite3-$VERS.bin.gz chmod 644 tclsqlite3.so chmod 644 sqlite3.so The webserver used by www.sqlite.org treats any file with execute permission as a CGI program and tries to run it. So we have to remove the execution permission bit from files which are intended to be served intact. ---- _2007-Nov-27 19:00:03 by anonymous:_ {linebreak} Instead of gzipping the files directly, can you wrap the individual files in a tar.gz instead? This way they could retain the executable permission. UNIX novices may not know about chmod. #cfe8bd 2799 build fixed 2007 Nov anonymous 2007 Nov 3 3 'configure ' cannot locate VERSION file 'configure' diff: 18437c18437 < ALLOWRELEASE="-release `cat VERSION`" --- > ALLOWRELEASE="-release `cat $srcdir/VERSION`" #c8c8c8 2798 build closed 2007 Nov anonymous 2007 Nov 3 4 Wrong nm used when cross-compiling When cross-compiling, the build platforms nm is used, instead of the one for the target platform. I included a patch for Makefile.in and configure.ac to start using the detected nm instead of the hardcoded 'nm'. _2007-Nov-27 14:53:06 by drh:_ {linebreak} The configure script is a mess and I am disinclined to make it a bigger mess by trying to fix cross-compilation. A better approch to cross-compiling is to use the amalgamation. Build the amalgamation using: make sqlite3.c Then compile sqlite3.c for your target system using whatever cross-compiler you want. #cfe8bd 2797 code fixed 2007 Nov anonymous 2007 Nov 3 3 Low memory SEGV in whereClauseClear() The function whereClauseClear() in where.c dereferences 'a' which may be null following an allocation error in whereClauseInsert(). It looks as if this allocation error will also result in leaking the old WhereTerm array addressed through pWC->a (if this was not static). The correction is probably to re-instate the old value on error before returning from whereClauseInsert(), ie to add the line: pWC->a = pOld; before the return 0. #cfe8bd 2796 code fixed 2007 Nov anonymous 2007 Nov 3 3 Another low memory SEGV in exprAnalyze() The first statement after declaring local variables in function exprAnalyze() in where.c is: if( db->mallocFailed ) return; Unfortunately initialising some of the local variables can already reference null pointers: WhereTerm *pTerm = &pWC->a[idxTerm]; ExprMaskSet *pMaskSet = pWC->pMaskSet; Expr *pExpr = pTerm->pExpr; If pWC->a is NULL the third of these initialisations causes problems. This happened here in the two calls: whereSplit(&sOr, pExpr, TK_OR); exprAnalyzeAll(pSrc, &sOr); from exprAnalyze(), early in the section beginning: /* Attempt to convert OR-connected terms into an IN operator so that ** they can make use of indices. Example: where the allocation in whereClauseInsert() (called by 5 depths of whereSplit()) failed while processing "SELECT y FROM t9\nWHERE x=(SELECT x FROM t9 WHERE y=1)\n OR x=(SELECT x FROM t9 WHERE y=2)\n OR x=(SELECT x FROM t9 WHERE y=3)\n OR x=(SELECT x FROM t9 WHERE y=4)\n OR x=(SELECT x FROM t9 WHERE y=5". The fix is to delay initialising the variables until after the check for mallocFailed. #cfe8bd 2795 code fixed 2007 Nov anonymous 2007 Nov 3 3 Low memory SEGV in exprAnalyze() Function exprAnalyze() in where.c includes the following code: /* Add constraints to reduce the search space on a LIKE or GLOB ** operator. ... pStr2 = sqlite3ExprDup(db, pStr1); if( pStr2 ){ assert( pStr2->token.dyn ); ++*(u8*)&pStr2->token.z[nPattern-1]; } A memory allocation error (in sqlite3StrNDup() called from sqlite3ExprDup()) can leave pStr2->token.z as NULL. Dereferencing this to increment an element brings a segment violation. sqlite3ExprDup() currently has no way of returning an error. #cfe8bd 2794 code fixed 2007 Nov anonymous 2007 Nov 3 3 Low memory SEGV in isLikeOrGlob() Function isLikeOrGlob() in where.c includes (towards the end of the function) the lines: sqlite3DequoteExpr(db, pRight); z = (char *)pRight->token.z; for(cnt=0; (c=z[cnt])!=0 && c!=wc[0] && c!=wc[1] && c!=wc[2]; cnt++){} A memory allocation error (in sqlite3StrNDup(0 called from sqlite3TokenCopy() called from sqlite3DequoteExpr()) can leave pRight->token.z as NULL. Dereferencing z in the for statement brings a segment violation. #f2dcdc 2793 code active 2007 Nov anonymous 2007 Nov 3 3 fts3 lacks scoping It would be nice if the fts3 symbols could optionally be made private/static as the rest of the sqlite3 library. Not sure why sqlite3_api becomes public when used with the amalgamation, for that matter. make TOP=`pwd` BCC=gcc TCC=gcc AR=ar RANLIB=echo NAWK=gawk -f \ main.mk sqlite3.h sqlite3.c fts3amal.c cat fts3amal.c >> sqlite3.c gcc -DSQLITE_THREADSAFE -DSQLITE_API=static -DSQLITE_PRIVATE=static \ -DSQLITE_EXTERN=static -DSQLITE_ENABLE_FTS3 -c sqlite3.c nm sqlite3.o | grep -v ' [trUbd] ' 00000004 C sqlite3_api 00064da2 T sqlite3Fts3HashClear 000652a4 T sqlite3Fts3HashFind 00064d60 T sqlite3Fts3HashInit 0006533b T sqlite3Fts3HashInsert 00064b4c T sqlite3Fts3Init 00066b34 T sqlite3Fts3InitHashTable 000669bd T sqlite3Fts3PorterTokenizerModule 0006702d T sqlite3Fts3SimpleTokenizerModule #cfe8bd 2792 code fixed 2007 Nov anonymous 2007 Nov 3 3 invalidateCursorsOnModifiedBtrees has no scoping The function invalidateCursorsOnModifiedBtrees is not scoped ala #2554. It should presumably be SQLITE_PRIVATE. May I suggest this as part of the release/test process: gcc --pthread -DNDEBUG -DSQLITE_THREADSAFE -DEXPERIMENTAL -DSQLITE_API=static -DSQLITE_PRIVATE=static -DSQLITE_EXTERN=static -c sqlite3.c nm sqlite3.o | grep -v ' [trUbd] ' If there is any output from the nm/grep then those symbols have not been scoped. _2007-Nov-24 05:54:27 by anonymous:_ {linebreak} I agree it should be part of the release check. But I don't see EXPERIMENTAL #defined in the code anywhere. Also, --pthread does not work for gcc on Linux. $ gcc -DSQLITE_THREADSAFE -DSQLITE_API=static -DSQLITE_PRIVATE=static -DSQLITE_EXTERN=static -c sqlite3.c $ nm sqlite3.o | grep -v ' [trUbd] ' 0001d964 T invalidateCursorsOnModifiedBtrees ---- _2007-Nov-24 07:55:20 by anonymous:_ {linebreak} It should have been -pthread (one dash not two). And yes experimental should also be defined. ---- _2007-Nov-24 18:38:47 by anonymous:_ {linebreak} The macro EXPERIMENTAL is used by what exactly? ---- _2007-Nov-24 19:55:44 by anonymous:_ {linebreak} Several of the SQLite C APIs are marked as EXPERIMENTAL meaning that they could change between releases. This is noted on the documentation pages as appropriate. For example see http://sqlite.org/c3ref/commit_hook.html ---- _2007-Nov-24 22:27:15 by anonymous:_ {linebreak} These are just comments, not #ifdefs:
$ grep EXPERIMENTAL sqlite3.[ch] sqlite3.c:****** EXPERIMENTAL - subject to change without notice ************** sqlite3.c:****** EXPERIMENTAL - subject to change without notice ************** sqlite3.c:****** EXPERIMENTAL - subject to change without notice ************** sqlite3.c:** This is an EXPERIMENTAL api and is subject to change or removal. sqlite3.c:** EXPERIMENTAL - This is not an official function. The interface may sqlite3.c:/*** EXPERIMENTAL *** sqlite3.c:******* THIS IS AN EXPERIMENTAL API AND IS SUBJECT TO CHANGE ****** sqlite3.c:****** EXPERIMENTAL - subject to change without notice ************** sqlite3.c:****** EXPERIMENTAL - subject to change without notice **************#f2dcdc 2791 code active 2007 Nov anonymous 2007 Nov 1 1 Allow building FTS[123] as part of sqlite library with configure See attached patch. #c8c8c8 2790 code closed 2007 Nov anonymous 2007 Nov danielk1977 2 2 sqlite3_compile returns NULL for empty statements There was apparently a backwards-incompatible behaviour change in 3.5 and later versions. sqlite3_compile() for an empty statement, like "" now returns a NULL pointer for the statement, but the function returns SQLITE_OK. In my applications I always assumed that if sqlite3_compile() returns SQLITE_OK, then there will be a valid statement. Now if you sqlite3_step() on the NULL pointer, you get a segfault. This has been fixed so that passing a NULL pointer to sqlite3_step() returns SQLITE_MISUSE, as it did for earlier versions. Strictly speaking, applications are not supposed to pass NULL to sqlite3_step(). See also #2773. #cfe8bd 2789 code fixed 2007 Nov anonymous 2007 Nov 3 3 Low memory double free from addWhereTerm The end of the function addWhereTerm() in select.c contains the following: pE = sqlite3ExprAnd(pParse->db,*ppExpr, pE); if( pE ){ *ppExpr = pE; } The function sqlite3Expr() called by sqlite3ExprAnd() deletes both Expr parameters if it fails to create its result Expr. Here the caller still has a pointer to the now freed Expr structure. In our system it currently gets deleted again later, in a call from clearSelect() called from sqlite3SelectDelete() resulting in double free errors. However, it is possible that this does not occur on current systems as ours is still based on version 3.3.8 (which still includes the bad (and no longer required) sqlite3ExprOrFree()). The problem looks as if it slipped in when sqlite3Expr() was changed to delete its parameters on error. The solution is to clear *ppExpr in this case, and this is most easily done by simply copying pE into it in all cases, even when NULL, ie make the line "*ppExpr = pE;" unconditional. #cfe8bd 2788 build fixed 2007 Nov anonymous 2007 Nov drh 3 3 FormatMessageA not defined for Windows Mobile Compiling SQLite 3.5.2 with embedded Visual C++ 4 produces the following link error: os_win.obj : error LNK2019: unresolved external symbol FormatMessageA referenced in function winDlError The only occurence of FormatMessageA is in os_win.c. Windows Mobile doesn't support ASCII, only wide characters. Since LoadLibrary isn't supported in winDlOpen on Windows Mobile, I suggest the following patch for winDlError: #if OS_WINCE if (nBuf > 0) zBufOut[0] = '\0'; #else FormatMessageA( FORMAT_MESSAGE_FROM_SYSTEM, NULL, GetLastError(), 0, zBufOut, nBuf-1, 0 ); #endif Already fixed by check-in [4541]. #f2dcdc 2787 build active 2007 Nov anonymous 2007 Nov 4 5 sqlite3.pc is not remade the subject says it all, there is no make rule to rebuild the sqlite3.pc from the .in file. it is only possible by hand (./config.status sqlite3.pc) _2007-Nov-22 12:17:50 by drh:_ {linebreak} I don't know what the sqlite3.pc file does. I certainly do not use it for any of my builds on Linux, Mac OSX, or windows. Why should I leave it in the source tree? Isn't the best solution to this problem to simply delete the file? ---- _2007-Nov-22 17:14:27 by anonymous:_ {linebreak} It is indirectly used by pkgconfig. Here's some info on pkgconfig: http://pkg-config.freedesktop.org/wiki/ ---- _2007-Nov-23 15:22:32 by drh:_ {linebreak} Could somebody who understands what the sqlite3.pc file is used for suggest a makefile rule for rebuilding it? #cfe8bd 2786 code fixed 2007 Nov anonymous 2007 Nov 4 4 sqlite3.pc does not mention -lpthread when used with --static flag I'm trying to statically link sqlite3 on Linux. I'm using pkg-config to get the settings. However, when I use "pkg-config sqlite3 --libs --static", it does not specify "-lpthreads", which is necesary. The fix I use is to add the following line to sqlite3.pc.in: Libs.private: -lpthread #cfe8bd 2785 doc fixed 2007 Nov anonymous 2007 Nov 3 3 Broken Anchors in C API Reference All of the links at the top of the page are linked to #link rather than the correct section within the documentation. http://www.sqlite.org/capi3ref.html Fixed by http://www.sqlite.org/docsrc/vinfo/32573eb907ac4faea29a91677a69cd667d198239 #cfe8bd 2784 code fixed 2007 Nov anonymous 2007 Nov 1 3 Low memory double free and SEGV from flattenSubquery() Low memory double free and SEGV from flattenSubquery() sqlite3SrcListAppend() in build.c can call sqlite3DbRealloc() which can fail under low-memory conditions. In this case it returns zero. It is called by flattenSubquery() in select.c in the section following: /* Move all of the FROM elements of the subquery into the ** the FROM clause of the outer query. Immediately (3 lines) after this call (after the loop) the return value is dereferenced in the start of the next loop: pSrc = sqlite3SrcListAppend(db, pSrc, 0, 0); } p->pSrc = pSrc; for(i=pSrc->nSrc-1; i-extra>=iFrom; i--){ I am also getting a 'double free' error during the cleanup in sqlite3SrcListAppend(), where the old (unextended) structure is freed by sqlite3SrcListDelete(). Apparently one of the pItem->zName strings has already been freed by the lines: sqlite3DeleteTable(pSubitem->pTab); sqlite3_free(pSubitem->zDatabase); sqlite3_free(pSubitem->zName); sqlite3_free(pSubitem->zAlias); immediately before the call in this section of flattenSubquery(). Clearing these pointers after freeing gets rid of the double free error. I can't offer a simple solution to the SEGV The SQL statement is "SELECT * FROM v10_11 LEFT JOIN t9 ON( a=x );" from the join-8.3 runtest. #cfe8bd 2783 doc fixed 2007 Nov anonymous 2007 Nov 5 4 Broken links into the website Hi, there are some broken links in the new website: Documentation -> SQL Syntax -> PRAGMA and http://www.sqlite.org/lang_droptable.html try to click on "CREATE TABLE" and "VACUUM". Fixed by http://www.sqlite.org/docsrc/vinfo/4f5adc7af26df28e61a1a5bacde935573394e7e2 #c8c8c8 2782 code closed 2007 Nov anonymous 2007 Nov 1 1 bind_text(...) into transaction In Transaction i insert some values, using sqlite3_bind_text(...) The first row has undef value(some times it's empty), but other rows have normal values. _2007-Nov-21 13:50:25 by drh:_ {linebreak} If you can describe some kind of problem with SQLite, we will be happy to look into the matter. Your description above sounds like a bug in your own code though. What about this problem makes you think it is a bug in SQLite? #cfe8bd 2781 doc fixed 2007 Nov anonymous 2007 Nov 5 4 Make documentation available offline (i.e. pdf, chm, etc) It would be nice if all of the documentation under "Documentation" was available offline in conjunction with the online format. The resulting pdf, chm, etc. could have the documentation segments merged or keep each segment separate. For example: SQL Syntax segment could be its own pdf or could be merged into a pdf with the c++ API Reference. Added a link to the {link: /download.html download page} containing the complete static text of this website. #cfe8bd 2780 build fixed 2007 Nov anonymous 2007 Nov 3 3 sqliteint.h: test to disable _XOPEN_SOURCE uses wrong macro for OSX sqliteint.h tests for __MACOS__ for whether to define _XOPEN_SOURCE on OSX. But this is not the correct macro (I think it's an old OS9 macro). __APPLE__would be a better choice. #c8c8c8 2779 code closed 2007 Nov anonymous 2007 Nov 2 2 Performance optimizations Performance optimizations )) PRAGMA: Does anyone have experience and good results optimizing sqlite performance using PRAGMA's? )) Other techniques: Any success stories on sqlite optimization methods of any type would be appreciated. Here is a list of the PRAGMA's and example use from the sqlite documentation: PRAGMA auto_vacuum; PRAGMA auto_vacuum = 0 | none | 1 | full | 2 | incremental; PRAGMA cache_size; PRAGMA cache_size = Number-of-pages; PRAGMA case_sensitive_like; PRAGMA case_sensitive_like = 0 | 1; PRAGMA count_changes; PRAGMA count_changes = 0 | 1; PRAGMA default_cache_size; PRAGMA default_cache_size = Number-of-pages; PRAGMA default_synchronous; PRAGMA empty_result_callbacks; PRAGMA empty_result_callbacks = 0 | 1; PRAGMA encoding; PRAGMA encoding = "UTF-8"; PRAGMA encoding = "UTF-16"; PRAGMA encoding = "UTF-16le"; PRAGMA encoding = "UTF-16be"; PRAGMA full_column_names; PRAGMA full_column_names = 0 | 1; PRAGMA fullfsync PRAGMA fullfsync = 0 | 1; PRAGMA incremental_vacuum(N); PRAGMA legacy_file_format; PRAGMA legacy_file_format = ON | OFF PRAGMA locking_mode; PRAGMA locking_mode = NORMAL | EXCLUSIVE PRAGMA main.locking_mode=EXCLUSIVE; PRAGMA page_size; PRAGMA page_size = bytes; PRAGMA max_page_count; PRAGMA max_page_count = N; PRAGMA read_uncommitted; PRAGMA read_uncommitted = 0 | 1; PRAGMA short_column_names; PRAGMA short_column_names = 0 | 1; PRAGMA synchronous; PRAGMA synchronous = FULL; (2) PRAGMA synchronous = NORMAL; (1) PRAGMA synchronous = OFF; (0) PRAGMA temp_store; PRAGMA temp_store = DEFAULT; (0) PRAGMA temp_store = FILE; (1) PRAGMA temp_store = MEMORY; (2) PRAGMA temp_store_directory; PRAGMA temp_store_directory = 'directory-name'; This is not a bug report. Ask any questions on the mailing list. #c8c8c8 2778 code closed 2007 Nov anonymous 2007 Nov adixon 1 1 Error 0 CREATE PROCEDURE Driver_Daily_Activity { @comp_code varchar(8), @begin_date datetime, @end_date datetime, @emp_short_name varchar(8), @emp_id int, @driver_time_in output, @driver_time_out output, @driver_beaktime_in output, @driver_beaktime_out output, @driver_lunchtime_in output, @driver_lunchtime_out output, @total_break_time output } AS select @driver_time_in =driver_datetime_in,@driver_time_out =driver_datetime_out from driver_daily_log with (nolock) where company_code=@comp_code and emp_id=@emp_id and driver_datetime_in >=@begin_date and driver_datetime_out<@end_date and driver_datetime_out is not null select @driver_breaktime_in=substring((CAST(break_time_in AS varchar(20))),12,8) ,@driver_breaktime_out=substring((CAST(break_time_out AS varchar(20))),12,8) from driver_daily_break_log with (nolock) where company_code=@comp_code and emp_id=@emp_id and driver_status='OB'. select @driver_lunchtime_in=substring((CAST(break_time_in AS varchar(20))),12,8) ,@driver_lunchtime_out=substring((CAST(break_time_out AS varchar(20))),12,8) from driver_daily_break_log with (nolock) where company_code=@comp_code and emp_id=@emp_id and driver_status='OL'. select @total_break_time =sum(total_break_time) from driver_daily_break_log with (nolock) where company_code=@comp_code and emp_id=@emp_id. This procedure is giving me the following error : SQL State 42000: Error 0 : Syntax error or access violation SQL identifiers may not contain the "@" character. #c8c8c8 2777 code closed 2007 Nov anonymous 2007 Nov shess 1 1 FTS[23] compile bug You need to #include "sqlite3.h" to pick up the sqlite3_malloc() declaration: ./ext/fts3/fts3_hash.c: In function "fts3HashMalloc": ./ext/fts3/fts3_hash.c:38: warning: initialization makes pointer from integer without a cast Also, can you move the following: #ifndef SQLITE_ENABLE_BROKEN_FTS2 #error fts2 has a design flaw and has been deprecated. #endif to be after this line: #if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS2) in fts2.c? That way you can build fts3 while having fts2 objects in the Makefile without defining -DSQLITE_ENABLE_BROKEN_FTS2 for an fts3-only build. _2007-Nov-23 18:21:04 by anonymous:_ {linebreak} This error still exists in FTS2 and FTS3: ./ext/fts3/fts3_hash.c: In function "fts3HashMalloc": ./ext/fts3/fts3_hash.c:38: warning: initialization makes pointer from integer without a cast On some 64-bit systems this will cause FTS to crash because an int is smaller than a pointer. Nevermind - it was fixed. #f2dcdc 2776 warn active 2007 Nov anonymous 2007 Nov 5 1 mailing list sqlite-users-digest doesn't work. It remembers registered addresses but doesn't send any emails. #c8c8c8 2775 code closed 2007 Nov anonymous 2007 Nov danielk1977 3 4 Low-memory SEGV from sqlite3CodeRowTrigger() I have hit a low memory segment violation in sqlite3CodeRowTrigger() after /* code the WHEN clause */. sqlite3CodeRowTrigger() calls sqlite3ExprDup() which fails to deep copy the string p->pRight->token.z (and therefore everything thereafter). sqlite3CodeRowTrigger() continues to call sqlite3ExprResolveNames() sqlite3ExprResolveNames() calls walkExprTree() for the expression, which recursively calls walkExprTree() for pExpr->pRight which in turn calls nameResolverStep() which then tries to dereference pExpr->token.z[0] (for case TK_STRING) generating the SEGV. The solution appears to be checking for allocation failures before calling nameResolverStep(), using the same early exit as when this fails. ie replacing: if( sqlite3ExprResolveNames(&sNC, whenExpr) ){ with: if( pParse->db->mallocFailed || sqlite3ExprResolveNames(&sNC, whenExpr) ){ _2007-Nov-16 14:36:10 by anonymous:_ {linebreak} This change will also prevent an assertion if the previous allocation is the first to fail, or a SEGV if the following one is the first. --------- Thanks. #cfe8bd 2774 code fixed 2007 Nov shess 2007 Nov shess 1 1 Crash merging empty OR queries in fts3. create virtual table test using fts2(a); insert into test (a) values ('alpha'); insert into test (a) values ('beta'); select a from test where test match ('a:gamma* OR a:delta*'); sqlite3: ../sqlite/ext/fts3/fts3.c:478: dataBufferAppend: Assertion `nSource>0 && pSource!=((void *)0)' failed. "Workaround" is to compile with -DNDEBUG=1, but the person who reported this to me was using the fts2 dll from sqlite.org. Fix on the way. #c8c8c8 2773 code closed 2007 Nov danielk1977 2007 Nov 2 1 sqlite crashes in sqlite3_step when compiled with SQLITE_THREADS=1 When running the self-test of pysqlite2 [1], it crashes. This only happens when sqlite3 is compiled with the --enable-threads option, but not with --disable-threads. #0 0xbb987124 in sqlite3_step () from /usr/pkg/lib/libsqlite3.so.0 #1 0xbb9a7307 in _sqlite_step_with_busyhandler () from /usr/pkg/lib/python2.4/site-packages/pysqlite2/_sqlite.so #2 0xbb9a437b in pysqlite_cursor_executescript () from /usr/pkg/lib/python2.4/site-packages/pysqlite2/_sqlite.so #3 0xbbb45baa in PyCFunction_Call () from /usr/pkg/lib/libpython2.4.so.1.0 The reason for the crash is that v->db is NULL. In the non-threaded case, it all works. [1]
diff -u -3 -p -r1.653 vdbe.c --- src/vdbe.c 23 Oct 2007 15:39:45 -0000 1.653 +++ src/vdbe.c 11 Nov 2007 19:38:11 -0000 @@ -2005,10 +2005,10 @@ case OP_Column: { ** which is the number of records. */ pC = p->apCsr[p1]; + assert( pC!=0 ); #ifndef SQLITE_OMIT_VIRTUALTABLE assert( pC->pVtabCursor==0 ); #endif - assert( pC!=0 ); if( pC->pCursor!=0 ){ /* The record is stored in a B-Tree */ rc = sqlite3VdbeCursorMoveto(pC);Ok. Thanks. #cfe8bd 2767 code fixed 2007 Nov anonymous 2007 Nov 2 1 UPDATE causes ./src/vdbe.c:503: sqlite3VdbeExec: Assertion `pTos<=&p-> This might be hard to pin down, but since I don't know the code base all that well I would put this out there in case someone else might understand this better or would be willing to help me track this bug down. With that said, here is what is going on: I get the following assertion error: ./src/vdbe.c:503: sqlite3VdbeExec: Assertion `pTos<=&p->aStack[pc]' failed. with the attached "script". _2007-Nov-11 07:41:40 by anonymous:_ {linebreak} Version 3.5.2 does not cause this to fail and runs as expected. ---- _2007-Nov-11 09:30:49 by drh:_ {linebreak} I beg to differ. I'm seeing the problem in 3.5.2 and in CVS head. I am able to reproduce the problem with the following test script: CREATE TABLE t1(x); CREATE TRIGGER r1 BEFORE UPDATE ON t1 BEGIN SELECT raise(ignore); END; INSERT INTO t1 VALUES(1); INSERT INTO t1 VALUES(2); INSERT INTO t1 SELECT x+2 FROM t1; INSERT INTO t1 SELECT x+4 FROM t1; UPDATE t1 SET x=x+1; The problem is that the code generator is generating VDBE code that fails to clear values from the VDBE stack. After a few iterations of the trigger, the VDBE stack overflows, triggering the assertion fault. It may be that the person posting the previous comment did not compile their version 3.5.2 with -DSQLITE_DEBUG. Hence assert() statements were disabled. The assert() statement is correct, however. This is a real problem. ---- _2007-Nov-11 13:01:54 by drh:_ {linebreak} It appears that operations such as UPDATE push a few values onto the stack prior to calling BEFORE triggers. Those values are normally consumed after the triggers run. But if a trigger runs RAISE(IGNORE) the later part of the processing is skipped and those values pushed on the stack are never consumed. Should be a relatively easy fix. Either defer pushing the extra values until after the BEFORE triggers run. Or else pass an extra parameter into sqlite3CodeRowTrigger() to tell it how many values to pop from the stack when a RAISE(IGNORE) is encountered. Preferably the first. I have long wanted to switch the VDBE from being a stack-based VM to being a register based VM. Notice that were the VDBE register based, this bug would have never come up.... ---- _2007-Nov-11 20:24:35 by anonymous:_ {linebreak} Thank you for the amazingly fast turn around. (Yes, I forgot to compile 3.5.2 with -DSQLITE_DEBUG.) #f2dcdc 2766 code active 2007 Nov drh 2007 Nov 1 1 TCL transaction started from within a query does not commit This is a problem with the TCL interface. Consider the following TCL script: file delete -force test.db test.db-journal sqlite3 db test.db db eval { CREATE TABLE t1(x,y); INSERT INTO t1 VALUES(1,2); CREATE TABLE t2(a,b); INSERT INTO t2 VALUES(8,9); } db eval {SELECT * FROM t1} { db transaction { db eval {UPDATE t2 SET a=a*2} } } The [db transaction] statement starts a transaction and it is suppose to commit the tranaction at the end of the code block. But because the transaction started while a query was active, the tranaction is unable to commit. The TCL interface never commits the tranaction nor does it give any kind of error indication. It is unclear if an error should be returned or if the commit should be deferred until outer query finishes. If the code within the [db transaction] block throws an error, we really need the transaction to rollback right away. Perhaps there should be a new API that cancels all pending queries. Perhaps a call to sqlite3_interrupt() would suffice for this. Need to investigate further.... #c8c8c8 2765 code closed 2007 Nov anonymous 2007 Nov 1 3 bad rollback if temporary table created during transaction Hi, trying to resolve a test failure in an application using pysqlite2 (2.3.5) / sqlite 3.42 for its test suite, I've been able to produce a minimal script which shows that the problem is actually in sqlite... I don't think it comes from the python wrapper (if you think it is tell me and I'll redirect it to the appropriate tracker). Anyway here is the script : import pysqlite2.dbapi2 as sqlite cnx = sqlite.connect('testdb') cursor = cnx.cursor() cursor.execute('SELECT eid from Trinfo') nb = len(cursor.fetchall()) cursor.execute("INSERT INTO Trinfo ( from_state, to_state, wf_info_for, eid, creation_date, modification_date ) VALUES ( 734, 735, 5, 758, 1, 1)") # comment this line and the final assertion is ok cursor.execute("CREATE TEMPORARY TABLE T58d712f777a7170cb8ce7d94cef119ce (C0 integer,C1 timestamp);") # get the same result with or without dropping the temp table #cursor.execute('DROP TABLE T58d712f777a7170cb8ce7d94cef119ce') cnx.rollback() cursor.execute('SELECT eid from Trinfo') nb3 = len(cursor.fetchall()) assert nb3 == nb # FAIL Notice that if you don't create the temporary table, everything run fine, else the insertion is not actually rollbacked. _2007-Nov-09 00:23:38 by drh:_ {linebreak} I tested using the following TCL script: file delete -force test.db test.db-journal sqlite3 db test.db db eval { CREATE TABLE t1(a,b,c); INSERT INTO t1 VALUES(1,2,3); INSERT INTO t1 VALUES(4,5,6); INSERT INTO t1 SELECT a*2, b*2, c*2 FROM t1; INSERT INTO t1 SELECT a*4, b*4, c*4 FROM t1; INSERT INTO t1 SELECT a*8, b*8, c*8 FROM t1; } puts "begin=[md5 [db eval {SELECT * FROM t1}]]" db eval {BEGIN} db eval {INSERT INTO t1 VALUES('hello','out','there');} db eval {CREATE TEMPORARY TABLE T58d712f777(C0 integer,C1 timestamp)} puts " mid=[md5 [db eval {SELECT * FROM t1}]]" db eval {ROLLBACK} puts " end=[md5 [db eval {SELECT * FROM t1}]]" The results are: begin=c17bc52be49269f8f2cd3a4df3caee6e mid=f86cc8ad5b925962323edc041bfaa09d end=c17bc52be49269f8f2cd3a4df3caee6e Begin and end are equal so it appears to work using the TCL interface. I also observe that there are countless tests in the SQLite test suite that verify that this sort of thing works correctly. Perhaps this is something to do with python after all. ---- _2007-Nov-11 20:48:52 by anonymous:_ {linebreak} See: http://www.initd.org/pub/software/pysqlite/doc/usage-guide.html#controlling-transactions #c8c8c8 2764 todo closed 2007 Nov anonymous 2007 Nov 2 2 Since I installed Leopard I keep getting sqlite3 data bases in trash Every night I seem to have a bunch of database folders in my trash with sqlite doc inside. I can't seem to delete them all even after quitting my applications. (I realize this is a developer's page, but this is a real annoyance and I have no idea what to do about it!) Thanks, Liz _2007-Nov-08 02:50:51 by drh:_ {linebreak} I am told by engineers at Apple that this is an issue with Leopard and that it has nothing to do with SQLite. Apple is working on the problem and will fix it. #f2dcdc 2763 build active 2007 Nov anonymous 2007 Nov 4 4 AIX build failure due to explicit _XOPEN_SOURCE definition Our AIX machine has started having problems building the new sqlite 3.5.2 source. I'm no AIX expert, but I did some digging and I think it's due to sqliteInt.h defining _XOPEN_SOURCE without also defining _ALL_SOURCE. AIX system header files usually define this themselves (in /usr/include/ standards.h) but only if _XOPEN_SOURCE was previously undefined. Has anyone else come across this bug? I'm willing to believe it's our build system if not; we can work around it by setting _ALL_SOURCE on the command-line, but I thought it might be useful to raise a bug anyway. See also tickets #2673, #2681, and #2741. I do not have access to an AIX machine and so have no ability to debug this problem. If you have suggested patches we will consider them. Otherwise, there is not much we can do about this ticket. #cfe8bd 2762 code fixed 2007 Nov anonymous 2007 Nov 3 3 fts3 is cavalier about malloc vs sqlite3_malloc The fts3 code (and, from a brief look, the fts2 and fts1 also) are inconsistent with the memory upcalls they use; in some cases they use malloc/realloc/free/calloc(..., 1), and in some they use the sqlite3_ equivalents. I have a patch that makes them use only the sqlite3_ equivalents, but was there a specific reason they were written without originally? And another problem: trying to build fts3_hash.c gives warnings about unknown parameters for sqlite3_malloc (used from fts3HashMalloc) because it never finds sqlite3.h from anywhere. On at least one 64-bit Solaris platform this has caused a segfault on execution, because the compiler guess for return type (int) was 32 bits, whereas the pointer it was trying to return was 64 bits. _2007-Nov-20 03:41:52 by anonymous:_ {linebreak} Indeed, I also have segfaults with FTS3 memory allocation with a 64bit build on OSX 10.5. It works fine on a 32bit build on OS X 10.5. This is the thread trace: Thread 0 Crashed: 0 libSystem.B.dylib 0x00007fffffe00637 __bzero + 55 1 org.sqlite.sqlite3 0x0000000100051fc3 fts3HashMalloc + 46 2 org.sqlite.sqlite3 0x0000000100052465 sqlite3Fts3HashInsert + 316 3 org.sqlite.sqlite3 0x000000010004bb0e sqlite3Fts3Init + 126 4 org.sqlite.sqlite3 0x000000010002537c openDatabase + 808 5 sqlite3 0x000000010000115c open_db + 37 6 sqlite3 0x0000000100003af9 process_input + 938 7 sqlite3 0x00000001000043a8 main + 1696 8 sqlite3 0x000000010000099c start + 52 I didn't try the idea of changing malloc & friends to sqlite equivalents. ---- _2007-Nov-20 03:56:07 by anonymous:_ {linebreak} All calls should use the new sqlite3_ alloc/free calls instead of malloc/free. Because fts is not a part of the sqlite core, this code receives less attention. Also, Makefile.in should be updated so that fts2 and fts3 can build out of the box on all architectures without requiring them loaded as external modules. ---- _2007-Nov-23 18:00:39 by anonymous:_ {linebreak} Those patches seem to work on OSX 10.5 64bit. No compile errors. No run errors in a simple test. I don't build with TclTk (no 64bit tcl on OSX anyways), so I can't run the full tests. #f2dcdc 2761 code active 2007 Nov anonymous 2007 Dec 3 3 CLI (shell.c) should be bundled with amalgamation The CLI (shell.c) should be bundled with the amalgamation for database administrative purposes without downloading the matching shell.c from the full source tree. I second that! Qt ships with the amalgamated source files, but we also ship shell.c, whch we have to retrieve from the non-amalgamated source files. ---- _2007-Dec-26 15:20:04 by anonymous:_ {linebreak} I also agree. It is inconvenient to retrieve the matching shell.c from the source tree. #f2dcdc 2760 new active 2007 Nov anonymous 2007 Nov 5 4 request: sqlite3_unlink() to delete db files. Hi! Today i came across a use case where i would like client code to be able to delete an underlying sqlite3 db, but that code doesn't have immediate access to the file name of that db (without refactoring the db wrapper code). An interesting feature addition would, IMO, be: int sqlite3_unlink( sqlite3 * db, bool closeTheFile ); Unlinks the file associated with the given database. It does not alter the database in any way (thus is it a no-op on a :memory: database). The closeTheFile flag specifies whether the file handle associated with db should also be closed (and thus db must also be closed), or just unlinked (e.g., as temporary databases are unlinked right after creation but kept open). After browsing through the VFS API a bit, i see that there is an xDelete function, but i'm not sure if its semantics require that the underlying file handle be closed. i don't see an extra xClose member of VFS, so i assume that xDelete also handles closing the file handle. If these were split into two features, sqlite3_unlink() could be implemented very easily. :) #c8c8c8 2759 code closed 2007 Nov anonymous 2007 Nov danielk1977 1 2 crash due to 'select * from virtualTable where x is null' For virtual table T1: select * from t1 where y is NULL; causes: 1301 assert( pTerm->eOperator & (WO_EQ|WO_LT|WO_LE|WO_GT|WO_GE|WO_MATCH) ); (gdb) bt #0 0x005e17a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x006217d5 in raise () from /lib/tls/libc.so.6 #2 0x00623149 in abort () from /lib/tls/libc.so.6 #3 0x0061adb1 in __assert_fail () from /lib/tls/libc.so.6 #4 0x00849534 in bestVirtualIndex (pParse=0xbfe2d750, pWC=0xbfe2d1d0, pSrc=0x9a23d2c, notReady=18446744073709551615, pOrderBy=0x0, orderByUsable=1, ppIdxInfo=0x9a372bc) at ../../../../Source/sqlite-3.5.1-hp/src/where.c:1301 #5 0x0084a989 in sqlite3WhereBegin (pParse=0xbfe2d750, pTabList=0x9a23d28, pWhere=0x9a28ed8, ppOrderBy=0xbfe2d59c) at ../../../../Source/sqlite-3.5.1-hp/src/where.c:2074 #6 0x0082c6c6 in sqlite3Select (pParse=0xbfe2d750, p=0x9a28f28, eDest=4, iParm=0, pParent=0x0, parentTab=0, pParentAgg=0x0, aff=0x0) at ../../../../Source/sqlite-3.5.1-hp/src/select.c:3118 #7 0x0081f16c in yy_reduce (yypParser=0x9a33aa8, yyruleno=104) at parse.y:368 #8 0x00820f65 in sqlite3Parser (yyp=0x9a33aa8, yymajor=1, yyminor= {z = 0x9a2e120 ";", dyn = 0, n = 1}, pParse=0xbfe2d750) at parse.c:3419 #9 0x0082e877 in sqlite3RunParser (pParse=0xbfe2d750, zSql=0x9a2e100 "select * from t1 where y is NULL;", pzErrMsg=0xbfe2d74c) at ../../../../Source/sqlite-3.5.1-hp/src/tokenize.c:449 #10 0x008246a0 in sqlite3Prepare (db=0x9920010, zSql=0x9a2e100 "select * from t1 where y is NULL;", nBytes=-1, saveSqlFlag=0, ppStmt=0xbfe2d8e0, pzTail=0xbfe2d8e4) at ../../../../Source/sqlite-3.5.1-hp/src/prepare.c:525 #11 0x00824a2c in sqlite3LockAndPrepare (db=0x9920010, zSql=0x9a2e100 "select * from t1 where y is NULL;", nBytes=-1, saveSqlFlag=0, ppStmt=0xbfe2d8e0, pzTail=0xbfe2d8e4) at ../../../../Source/sqlite-3.5.1-hp/src/prepare.c:604 #12 0x00824bf9 in sqlite3_prepare (db=0x9920010, zSql=0x9a2e100 "select * from t1 where y is NULL;", nBytes=-1, ppStmt=0xbfe2d8e0, pzTail=0xbfe2d8e4) at ../../../../Source/sqlite-3.5.1-hp/src/prepare.c:658 #13 0x0084d862 in sqlite3_exec (db=0x9920010, zSql=0x9a2e100 "select * from t1 where y is NULL;", xCallback=0x8049b23
diff -u -3 -p -r1.130 vdbeInt.h --- src/vdbeInt.h 28 Aug 2007 23:28:08 -0000 1.130 +++ src/vdbeInt.h 5 Nov 2007 14:39:04 -0000 @@ -123,6 +123,7 @@ struct Mem { FuncDef *pDef; /* Used only when flags==MEM_Agg */ } u; double r; /* Real value */ + char zShort[NBFS]; /* Space for short strings */ sqlite3 *db; /* The associated database connection */ char *z; /* String or BLOB value */ int n; /* Number of characters in string value, including '\0' */ @@ -130,7 +131,6 @@ struct Mem { u8 type; /* One of SQLITE_NULL, SQLITE_TEXT, SQLITE_INTEGER, etc */ u8 enc; /* SQLITE_UTF8, SQLITE_UTF16BE, SQLITE_UTF16LE */ void (*xDel)(void *); /* If not null, call this function to delete Mem.z */ - char zShort[NBFS]; /* Space for short strings */ }; typedef struct Mem Mem;#c8c8c8 2750 code closed 2007 Oct anonymous 2007 Oct 2 1 for the new.somefield value returns the old value of it For the update trigger on a table containing n columns when retrieving the new value of filed the value of which was not changed at update (so the new value is null)it returns its old value. _2007-Oct-31 18:21:32 by danielk1977:_ {linebreak} That's how it's documented to work. #f2dcdc 2749 warn active 2007 Oct anonymous 2007 Oct 3 4 SQLITE_OMIT_FLOATING_POINT, int constants too large Hiya! i'm working from the 3.5.1 (CVS version) of the amalgamation. The code was pulled from CVS sometime around Oct 27th 2007. Platform: Kubuntu Linux 7.04, i386-32, gcc 4.1.2. The compiler output says it all: gcc -c -DSQLITE_OMIT_FLOATING_POINT sqlite3.c sqlite3.c: In function 'bestVirtualIndex': sqlite3.c:65531: warning: integer constant is too large for 'long' type sqlite3.c: In function 'bestIndex': sqlite3.c:65600: warning: integer constant is too large for 'long' type sqlite3.c: In function 'sqlite3WhereBegin': sqlite3.c:66223: warning: integer constant is too large for 'long' type sqlite3.c:66248: warning: integer constant is too large for 'long' type sqlite3.c:66254: warning: integer constant is too large for 'long' type These warnings would seem to indicate potentially serious problems, though i admittedly have not investigated whether overflows are really fatal in the affected contexts. #f2dcdc 2748 warn active 2007 Oct anonymous 2007 Oct 4 4 amalgamation: SQLITE_OMIT_ALTERTABLE warnings Hiya! i'm working from the 3.5.1 (CVS version) of the amalgamation. The code was pulled from CVS sometime around Oct 27th 2007. The compiler output says it all: gcc -c -DSQLITE_OMIT_ALTERTABLE sqlite3.c sqlite3.c:6909: warning: 'sqlite3AlterRenameTable' used but never defined sqlite3.c:6916: warning: 'sqlite3AlterFinishAddColumn' used but never defined sqlite3.c:6917: warning: 'sqlite3AlterBeginAddColumn' used but never defined The relevant code is: sed -ne '6909p;6916p;6917p' sqlite3.c SQLITE_PRIVATE void sqlite3AlterRenameTable(Parse*, SrcList*, Token*); SQLITE_PRIVATE void sqlite3AlterFinishAddColumn(Parse *, Token *); SQLITE_PRIVATE void sqlite3AlterBeginAddColumn(Parse *, SrcList *); Platform: Kubuntu Linux 7.04, i386, gcc 4.1.2. #c8c8c8 2747 code closed 2007 Oct anonymous 2007 Nov maria.khomenko 1 1 How to use Composite Key in Sqlite How to use Composite Key in Sqlite Tickets are for reporting bugs. Please ask general questions such as this on the SQLite mailing list. ---- _2007-Nov-01 11:34:53 by anonymous:_ {linebreak} but kindly Maria Give me the Way how can i use ---- _2007-Nov-02 07:15:59 by anonymous:_ {linebreak} How and where Can i do that ---- _2007-Nov-02 07:18:06 by anonymous:_ {linebreak} If you have some Articles regarding that mail me at rizwan_aman007@yahoo.co.in ---- _2007-Nov-05 05:39:22 by anonymous:_ {linebreak} I did't get any resolution. plz help me ---- _2007-Nov-08 12:07:18 by anonymous:_ {linebreak} ` #c8c8c8 2746 code closed 2007 Oct anonymous 2007 Oct 1 1 PRIMARY KEY does not treat NULL as unique The below inserts 2 records. One record would be expected. CREATE TABLE test(Colour VARCHAR COLLATE NOCASE,SubType VARCHAR COLLATE NOCASE,weight DOUBLE, PRIMARY KEY (Colour,SubType)); BEGIN TRANSACTION; INSERT OR REPLACE INTO test VALUES('red', NULL, 100); INSERT OR REPLACE INTO test VALUES('red', NULL, 100); COMMIT I understand that the NULL behaviour is hazy but this seems to not agree with the doc where it says: "nulls are distinct in a UNIQUE column". _2007-Oct-28 23:35:17 by drh:_ {linebreak} "nulls are distinct" means that every NULL is considered to be different from every other NULL. Hence, if two rows contain only NULL values, they are considered UNIQUE because they are not equal to one another. #c8c8c8 2745 code closed 2007 Oct anonymous 2007 Oct shazow 1 1 Sharing File between two processes Sharing File between two processes #c8c8c8 2744 code closed 2007 Oct drh 2007 Nov danielk1977 1 1 Cannot quote collating sequence name The following statement gives an error: CREATE TABLE demo(x TEXT COLLATE "binary"); It is using the quote characters as part of the name. The quotes ought to be removed before searching for the collating sequence. #c8c8c8 2743 doc closed 2007 Oct anonymous 2007 Oct 1 1 Comparison of SQLite against McObject Fusion Can you please provide a comparison of SQLite against McObhect Fusion. Thanks. _2007-Oct-24 11:07:16 by drh:_ {linebreak} I've never heard of McObject Fusion before. #c8c8c8 2742 code closed 2007 Oct anonymous 2007 Oct 1 1 codeInteger and codeReal In the latest CVS I noticed that codeInteger and codeReal ignores the argument `n'. static void codeReal(Vdbe *v, const char *z, int n, int negateFlag){ assert( z || v==0 || sqlite3VdbeDb(v)->mallocFailed ); if( z ){ double value; char *zV; sqlite3AtoF(z, &value); //***** can this accidentally go beyond n bytes? if( negateFlag ) value = -value; zV = dup8bytes(v, (char*)&value); sqlite3VdbeOp3(v, OP_Real, 0, 0, zV, P3_REAL); } } I was under the impression that both sqlite3AtoF and sqlite3Atoi64 required null terminated strings. Is `z' guaranteed to be null terminated in all cases? Could a random byte follow the number string buffer that happens to be a digit and cause the parsed number to be incorrect? So at best you could drop the 'n' arg from codeReal and codeInteger, at worst, you'd have to take n bytes into consideration for the sqlite3Ato* functions or less desirable - allocate a buffer and copy the string to it and nil terminate it prior to calling the sqlite3Ato* functions. _2007-Oct-23 19:23:24 by anonymous:_ {linebreak} You forgot to assert z[n] is not '.', 'e', and 'E' for codeReal. I'm not comfortable with this not-a-valid-number-character assumption, but am too lazy to look for a counter example that might break it. If you're confident that the z[n] byte always exists and reading that byte in sqlite3Ato* will not segfault or cause valgrind to complain, why not do this instead: char zn = z[n]; z[n] = 0; ... do sqlite3Ato* stuff z[n] = zn; It's fast, guaranteed to be safe and is more resiliant in the face of future possibly incompatible code changes in the parser. ---- _2007-Oct-23 19:26:05 by anonymous:_ {linebreak} As you well know, the assert only works in debug mode, not in production. The debug runs pad memory allocations at the end with a byte or two, do they not? ---- _2007-Oct-23 19:52:10 by anonymous:_ {linebreak} The saving/restoring z[n] hack above won't work if the original parse string on which the tokens are based is stored in read-only memory or if the original string is being parsed simultaneously in 2 or more threads. But if the original parse string is guaranteed to be copied before parsing begins, then sure, it could work. ---- _2007-Oct-23 21:15:15 by drh:_ {linebreak} The code is correct as written. If you think otherwise, show me a counter-example. Could the code benefit from be refactored to be cleaner. Probably so. But that would require hundreds of lines of carefully optimized code to be redesigned and rewritten. I have more important fish to fry right now and so I am not going to waste several days modifying code that works. It is what it is. Either show me a case where it fails or else learn to live with it. #cfe8bd 2741 code fixed 2007 Oct anonymous 2007 Oct 3 3 _XOPEN_SOURCE may be necessary for pread/pwrite as well as threads In 3.5.1, _XOPEN_SOURCE is only defined if sqlite is configured to be thread-safe. However, this definition may also be necessary to make prototypes visible for pread, pwrite, fdatasync, and usleep, all of which are conditionally used by os_unix.c. This is only a major problem for pread and pwrite, both of which take arguments that may be 64 bits wide and require a special calling convention on systems with 32-bit "long". _2007-Oct-23 01:09:32 by anonymous:_ {linebreak} But what value should it be set to in order to be portable across all POSIX machines? ---- _2007-Oct-23 14:52:57 by anonymous:_ {linebreak} _XOPEN_SOURCE is so ill defined and poorly implemented, the user should add -D_XOPEN_SOURCE=whatever to their own makefile's compile flags if they need 64 bit pread/pwrite support. ---- _2007-Oct-23 16:01:32 by drh:_ {linebreak} I have added comments to the code to warn about this situtation. But in accordance with the suggestions on ticket #2681, we do not want to define _OPEN_SOURCE by default. #cfe8bd 2740 code fixed 2007 Oct anonymous 2007 Oct 4 4 incorrect assertions in vdbe.c /sqlite/src/vdbe.c contains two assertions of the form assert( SQLITE_MAX_SQL_LENGTH < SQLITE_MAX_LENGTH ); as part of the implementations of the =OP_String8= and =OP_HexBlob= opcodes. This assertion is wrong, because the default definition of =SQLITE_MAX_SQL_LENGTH= is equal to =SQLITE_MAX_LENGTH=. It should be assert( SQLITE_MAX_SQL_LENGTH <= SQLITE_MAX_LENGTH ); #cfe8bd 2739 code fixed 2007 Oct anonymous 2007 Oct 2 3 large-file enable macros should be defined before first #include Currently (3.4.2 and it appears 3.5.1 as well) /sqlite/src/os_unix.c tries to enable support for large files by defining =_LARGEFILE_SOURCE= and/or =_FILE_OFFSET_BITS=64=. Unfortunately, it does this after including sqliteInt.h and os.h, which include several system headers, which means that on some platforms the macros have no effect. Contrast the behavior of these two test files: =testA.c= #define _FILE_OFFSET_BITS 64 #include
Index: src/vacuum.c =================================================================== RCS file: /sqlite/sqlite/src/vacuum.c,v retrieving revision 1.73 diff -u -3 -p -r1.73 vacuum.c --- src/vacuum.c 29 Aug 2007 12:31:28 -0000 1.73 +++ src/vacuum.c 20 Oct 2007 17:32:10 -0000 @@ -136,18 +136,18 @@ int sqlite3RunVacuum(char **pzErrMsg, sq ** in the temporary database. */ rc = execExecSql(db, - "SELECT 'CREATE TABLE vacuum_db.' || substr(sql,14,100000000) " - " FROM sqlite_master WHERE type='table' AND name!='sqlite_sequence'" - " AND rootpage>0" + "SELECT 'CREATE TABLE vacuum_db.' || substr(sql,14) " + "FROM sqlite_master WHERE type='table' AND name!='sqlite_sequence' " + "AND rootpage>0" ); if( rc!=SQLITE_OK ) goto end_of_vacuum; rc = execExecSql(db, - "SELECT 'CREATE INDEX vacuum_db.' || substr(sql,14,100000000)" - " FROM sqlite_master WHERE sql LIKE 'CREATE INDEX %' "); + "SELECT 'CREATE INDEX vacuum_db.' || substr(sql,14) " + "FROM sqlite_master WHERE sql LIKE 'CREATE INDEX %'"); if( rc!=SQLITE_OK ) goto end_of_vacuum; rc = execExecSql(db, - "SELECT 'CREATE UNIQUE INDEX vacuum_db.' || substr(sql,21,100000000) " - " FROM sqlite_master WHERE sql LIKE 'CREATE UNIQUE INDEX %'"); + "SELECT 'CREATE UNIQUE INDEX vacuum_db.' || substr(sql,21) " + "FROM sqlite_master WHERE sql LIKE 'CREATE UNIQUE INDEX %'"); if( rc!=SQLITE_OK ) goto end_of_vacuum; /* Loop through the tables in the main database. For each, do#f2dcdc 2736 build active 2007 Oct anonymous 2007 Oct 2 2 build problems on freebsd on freebsd: --disable-threads does not work. it is accepted as a valid option but no defs are added to the makefile -lgcc needs to be included in SHLIB_LD_LIBS pkgIndex.tcl is not built when -DSQLITE_THREADSAFE=0 is added manually, this causes the install target to fail. _2007-Oct-17 21:16:23 by anonymous:_ {linebreak} I forgot to mention this is with the TEA version #c8c8c8 2735 code closed 2007 Oct anonymous 2007 Oct nmadhav 3 1 Error : Unrecognized Token Hi, I am trying to Create a table in the sqlite DB. The table name is 9I0_STATIC_101607 and I get an error Unrecognized token : 9I0_STATIC_101607 But if I create a table with name STATIC_9I0_101607, my code works fine. It is something to do with the table name starting with number. Is there any kind of restriction with table name starting with a digit?? Thanks for your help. Regards, Mayura. _2007-Oct-17 17:07:43 by drh:_ {linebreak} This is not a bug report. This is a support request. Please see http://www.sqlite.org/support.html for information on how to receive support for SQLite. Please use these ticketing system to report bugs only. Thanks. #c8c8c8 2734 code closed 2007 Oct anonymous 2007 Oct 3 1 2 bug under the wince
the test version is 3.5.1 thank you! 1. in os_win.c line:1165 /* samsting_xie change the last line: zFilename to zName. the zFilename is not declared, so i change it to zName, maybe it's write error? the old last line is: && !winceCreateLock(zFilename, pFile) */ 2. in os_win.c line:1341 return SQLITE_OK; /* samsting_xie add the last line: return SQLITE_OK; if OS_WINCE defined 1, then this function is not return value. it's maybe return a unknown value. */Duplicate. See, for example, #2718, #2711, #2710, #2702, #2700, #2683. #cfe8bd 2733 code fixed 2007 Oct anonymous 2007 Oct 1 1 OP_Real and OP_Int64 inefficient for constants Let's say I have a table with 50 million rows: CREATE TABLE t1(a real); If I perform a query like: select -3.14159265 * a from t1; then the string "-3.14159265" will be converted to a double 50 million times via OP_Real -> sqlite3VdbeRealValue -> sqlite3AtoF. explain select -3.14 * a from t1; ... 5|Real|0|0|-3.14 -- converts string to double each loop pass! 6|Column|0|0|# t1.a 7|RealAffinity|0|0| 8|Multiply|0|0| 9|Callback|1|0| 10|Next|0|5| ... This is actually quite computationally expensive and unnecessary. If OP_Real and OP_Int64 would store the double/int64 values into P1/P2 at expression parse time then they would not have to be reparsed to a double/int64 each pass in the loop at vdbe runtime. _2007-Oct-18 03:37:47 by anonymous:_ {linebreak} Proof of concept patch: http://www.mail-archive.com/sqlite-users%40sqlite.org/msg28390.html http://marc.info/?l=sqlite-users&m=119266402408308&q=p3 #cfe8bd 2732 code fixed 2007 Oct anonymous 2007 Oct 2 2 sqlite3Step crashes if called with a null argument (easy fix) With the reorganisation of the code in 3.5 to use a memory manager accessed via the db structure, sqlite3Step in vdbeapi.c can now cause an access violation if called with a null Vdbe structure. Unfortunately pysqlite quite often calls this routine with a null statement and so can crash if you execute an empty query. The obvious fix is to move the two marked lines below the null check. static int sqlite3Step(Vdbe *p){ sqlite3 *db; int rc; /* Assert that malloc() has not failed */ -> db = p->db; -> assert( !db->mallocFailed ); if( p==0 || p->magic!=VDBE_MAGIC_RUN ){ return SQLITE_MISUSE; } _2007-Oct-17 01:38:03 by drh:_ {linebreak} This is really a bug in pysqlite. Have you filed a bug report there? ---- _2007-Oct-17 22:47:06 by anonymous:_ {linebreak} Just filed a bug there. I had to verify a few things first. In hindsight, the check for a null statement should perhaps be made in sqlite3_step rather than sqlite3Step. That would be more useful, as it appears to be the case that the mutex check causes an exception (I built the version I first observed this in without mutexes, and then fixed the bug in my copy of pysqlite) #c8c8c8 2731 code closed 2007 Oct anonymous 2007 Oct 1 1 sqlite3ExprCode TK_UMINUS inefficient with large negative floats The follow change in expr.c makes my program (which deals with large negative floating point numbers) run 14 times faster. 14 seconds originally to just 1 second after this patch. Assuming the fix is correct, can you please apply it? make test runs with 0 errors with this. Thanks.
Index: src/expr.c =================================================================== RCS file: /sqlite/sqlite/src/expr.c,v retrieving revision 1.313 diff -u -3 -p -r1.313 expr.c --- src/expr.c 18 Sep 2007 15:55:07 -0000 1.313 +++ src/expr.c 16 Oct 2007 23:54:17 -0000 @@ -1879,7 +1879,10 @@ void sqlite3ExprCode(Parse *pParse, Expr assert( pLeft ); if( pLeft->op==TK_FLOAT || pLeft->op==TK_INTEGER ){ Token *p = &pLeft->token; - char *z = sqlite3MPrintf(pParse->db, "-%.*s", p->n, p->z); + char *z = sqlite3_malloc(p->n+2); + z[0] = '-'; + memcpy(z+1, p->z, p->n); + z[p->n+1] = 0; if( pLeft->op==TK_FLOAT ){ sqlite3VdbeOp3(v, OP_Real, 0, 0, z, p->n+1); }else{_2007-Oct-17 01:34:39 by drh:_ {linebreak} FWIW, the patch is not correct. It does not work if sqlite3_malloc fails and returns a NULL pointer. But it isn't too hard to fix. I'm curious, though, what kind of program you are running where this makes a 14x speed improvement. One wonders if you couldn't do 100x faster using sqlite3_bind_double() instead. We could go through and make patches like this one all over the place, in order to make this or that special case run a little faster. But that would make the library footprint larger. The sqlite3MPrintf() call in this context is there for reasons of space efficiency, not CPU time. Some years ago, we when through this huge effort to reduce the footprint of the library by replacing the expanded string construction code like you are inserting with instances of sqlite3MPrintf(). I am reluctant to undo that effort for a single special case that could probably be handled more efficiently using sqlite3_bind_double(). ---- _2007-Oct-17 02:21:17 by anonymous:_ {linebreak} I have a script to generate ASCII SQL statements which I use in conjunction with the sqlite3 shell and MySQL to populate huge tables. I use the multi-insert patch on the mailing list for tables that hold millions of rows of negative numbers, with anywhere from 3 to 20 columns. http://www.mail-archive.com/sqlite-users%40sqlite.org/msg28337.html INSERT INTO BIGTABLE VALUES (-334712687065.09, -334712687065.12, -334712687065.13), (-334712687065.09, -334712687065.12, -334712687065.13), ... 10 thousand rows ... (-334712687065.09, -334712687065.12, -334712687065.13), (-334712687065.09, -334712687065.12, -334712687065.13); With the mulit-insert patch and the patch in this ticket, it is 25% faster for inserting than using the equivalent INSERT commands within a transaction: BEGIN INSERT INTO BIGTABLE VALUES(-334712687065.09, -334712687065.12, -334712687065.13); ... 10 thousand rows ... INSERT INTO BIGTABLE VALUES(-334712687065.09, -334712687065.12, -334712687065.13); COMMIT; I have to have no more than 10,000 rows per insert because it blows the stack due to sqlite3Select -> multiSelect recursion. I had to bump a few other sqliteLimits as well to get more than a few thousand rows at a time. I suppose the replacement of the sqlite3MPrintf could be useful in other scenarios as well. ---- _2007-Oct-17 02:27:30 by anonymous:_ {linebreak} The bind double approach is not 100X faster, by the way. It's only around 1.3X faster than the individual ASCII inserts in my tests. But it doesn't matter for my purposes, because it's not MySQL compatible anyway. ---- _2007-Oct-17 03:04:11 by anonymous:_ {linebreak} If the string were not pre-prepended with a '-' char, but instead parsed into a double, and the double simply negated it would be much faster and avoid a needless memory allocation/deallocation. To facilitate this, OP_Real would have to be changed or a new OP code created to accommodate an 8 byte double argument - possibly via straddling P1 and P2. ---- _2007-Oct-23 16:00:39 by anonymous:_ {linebreak} Fixed by Check-in [4507] #c8c8c8 2730 code closed 2007 Oct anonymous 2007 Oct anonymous 3 2 Can't compile for WindowsCE There is a compilation bug in os_win.c line 1165 (then compile for WindowsCE) "sqlite3.5_\os_win.c(1165) : error C2065: 'zFilename' : undeclared identifier" the code #if OS_WINCE if( (flags & (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB)) == (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB) && !winceCreateLock(zFilename, pFile) seems to be #if OS_WINCE if( (flags & (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB)) == (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB) && !winceCreateLock(zName, pFile) Best regards, Yuri Noyanov Duplicate of #2648, #2700, #2702, and #2718. #f2dcdc 2729 doc active 2007 Oct anonymous 2007 Oct 1 1 Lemon: %fallback, %wildcard, and @X uncodumented I noticed that the lemon documentation does not mention the %fallback and %wildcard directives. Both are in the code and are apparently doing useful work in SQLite's parse.y. Can other users benefit from them as well? The symbol @X is also undocumented. From a source code comment I read that it "If the argument is of the form @X then substituted the token number of X, not the value of X". A short documentation example would help to understand where and how it can be useful to apply this syntax. Are there other nice but undocumented Lemon goodies lacking documentation? #f2dcdc 2728 code active 2007 Oct anonymous 2007 Oct 4 4 Some indexes could contain pointers not the data It would be nice if there was a class of index(-column) that would contain a reference to the data in (in the table) rather than a copy of the data. This is useful where you have a need to index large strings or blobs and the size penalty of having 2+ copies ends up being *very* expensive. Consider the cost of the implied index on: create table t1 ( c1 BLOB UNIQUE); and store 100s of 1MB objects. The raw file ends up being twice as large as you ideally would like it to be. *WHEN* to use such indexes isn't entirely clear to me, not is the syntax for doing it in general, 30s of thought I can come up with: create table t1 ( c1 BLOB &UNIQUE); and say create index idx1 on t2(c1, &c2, c3); (meaning c2 would be done via reference, c1 & c3 as copies of the data). It's *FAR* from clear what other databases do here (I didn't look). This would also be useful in cases where I have multiple indexes on tables like email (headers) and values which can be very large (100s of bytes in some cases (Subject:) and I might have 5 or more indexes, means a 100-byte column will take over 600 bytes (+ padding)). Allowing for references would be slower in some cases but faster in others because of the smaller footprint and much grater CPU cache utilization. _2007-Oct-14 15:27:12 by anonymous:_ {linebreak} This enhacement has been discussed on the mailing list before, and would break sqlite3 file format compatability. ---- _2007-Oct-15 02:42:02 by anonymous:_ {linebreak} This is non standard sql. Also, the random access to pages to check indexes constraints (unique indexes, also primary keys) will trash the entire database performance. If you need small index disk usage, consider using hashes to your data keys. it´s the best that you could do to solve your problem. ---- _2007-Oct-18 06:07:58 by anonymous:_ {linebreak} Using a hash is what I've tried doing in a couple of cases. It's not very ideal. For one thing is ordering is messed up. I wonder about a virtual index concept, where you can define a function that takes the column value and returns something to actually store in the index? I'm told oracle has this feature, so whilst it's non-standard there is some precedent (albeit in a very different space). #f2dcdc 2727 build active 2007 Oct anonymous 2007 Oct 4 4 building with -malign-double causes strange behavior sqlite 3.5.1 amalgamation has problems when enabling a wide set of compiler features on gcc (GCC) 4.1.2 (Ubuntu 4.1.2-0ubuntu4) on linux/i686 /w glibc-2.5 strange behavior occurs. typical strangeness is that SQLITE_FULL is returned from sqlite3_prepare_v2() (called immediately after a statement was created with sqlite3_mprintf). This happens even though there is a reasonable amount of disk space free (6G). the compile line looks like:{linebreak} cc -g -pg -O1 -march=i686 -msse2 -malign-double -m128bit-long-double -momit-leaf-frame-pointer -minline-all-stringops -D_XOPEN_SOURCE=520 '-DVERSION="0.10r276M"' -Isqlite/ -DSQLITE_OMIT_LOAD_EXTENSION -DTHREADSAFE=0 -DSQLITE_OMIT_EXPLAIN -DSQLITE_ENABLE_COLUMN_METADATA -c -o sqlite/sqlite3.o sqlite/sqlite3.c and valgrind reports:{linebreak} ==15323== Use of uninitialised value of size 4{linebreak} ==15323== at 0x80585B7: insertElement (sqlite3.c:13072){linebreak} ==15323== by 0x807366E: sqlite3HashInsert (sqlite3.c:13290){linebreak} ==15323== by 0x809FE72: unixOpen (sqlite3.c:15403){linebreak} ==15323== by 0x80579A7: sqlite3OsOpen (sqlite3.c:8210){linebreak} ==15323== by 0x806B12F: sqlite3BtreeFactory (sqlite3.c:21317){linebreak} ==15323== by 0x80767BA: openDatabase (sqlite3.c:71237){linebreak} ==15323== by 0x8076BD1: sqlite3_open (sqlite3.c:71337){linebreak} ==15323== by 0x804BEAE: db_init (dbmgr.c:81){linebreak} ==15323== by 0x804CE64: main (main.c:59){linebreak} my dbmgr.c:81:db_init() is: res=sqlite3_open(filename, &system_db); and system_db=NULL and filename="zomg.sqlite" (string literal). so the parameters seem normal. if I turn off -malign-double everything works fine. (this "bug" also seems to be on 3.4.1 amalgamation) _2007-Oct-14 04:35:16 by anonymous:_ {linebreak} This is a compiler issue. -malign-double creates problems for most programs. What are you trying to accomplish? #c8c8c8 2726 code closed 2007 Oct anonymous 2007 Oct 4 3 solaris 10 compilation B_FALSE, B_TRUE clash from sys/types.h % make{linebreak} 386-pc-solaris2.10-gcc -O3 -pipe -o lemon ./tool/lemon.c{linebreak} ./tool/lemon.c:111: error: redeclaration of enumerator ‘B_FALSE’{linebreak} /usr/include/sys/types.h:176: error: previous definition of ‘B_FALSE’ was here{linebreak} ./tool/lemon.c:111: error: redeclaration of enumerator ‘B_TRUE’{linebreak} /usr/include/sys/types.h:176: error: previous definition of ‘B_TRUE’ was here{linebreak} make: *** [lemon] Error 1{linebreak} % sed -e 176\!d /usr/include/sys/types.h{linebreak} typedef enum { B_FALSE, B_TRUE } boolean_t;{linebreak} % sed -e 111\!d tool/lemon.c{linebreak} typedef enum {B_FALSE=0, B_TRUE} Boolean;{linebreak} even when the enum is only set when it is not yet defined, I guess this results in conflicts in the code because Boolean may not equal boolean_t. I suggest to rename B_FALSE and B_TRUE to something which makes it unique for the sqlite case. Duplicate of #2583 #f2dcdc 2725 code active 2007 Oct anonymous 2007 Oct 1 1 memory leak in sqlite3_open_v2() when it fails only happens with flags = SQLITE_OPEN_READWRITE; and when res = sqlite3_open_v2(sourcename, &conn, flags, NULL); seems to leak 674 bytes per call _2007-Oct-15 07:07:07 by danielk1977:_ {linebreak} Are you calling sqlite3_close(conn) after the error occurs? All calls to sqlite3_open_v2() need to be matched by a call to sqlite3_close(), even if an error occurs. #c8c8c8 2724 code closed 2007 Oct anonymous 2007 Dec drh 2 2 Load Ext api missing error_toobig function. Function call for sqlite3_error_toobig is not in the sqlite3Ext.h void (*result_error_toobig)(sqlite3_context*); #define sqlite3_result_error_toobig sqlite3_api->result_error_toobig Without these, externally loadded libs that would like to use error_toobig will not load into sqlite _2007-Dec-13 18:51:28 by drh:_ {linebreak} See ticket #2610. #c8c8c8 2723 new closed 2007 Oct anonymous 2007 Oct anonymous 5 4 After execution of SQL, the Data Returned should display the col.name Hi! After executing a SQL command, the program shows the results as a table with columns. The first row ist allways grey but empty. Couldn't the program display the selected entities (or their declared AS-alias) as column headers in the result table? Yours sincerely M. Heinrich, München _2007-Oct-12 18:19:38 by drh:_ {linebreak} Do you mean in the CLI? Just type .header on And the headers will appear. #cfe8bd 2722 code fixed 2007 Oct anonymous 2007 Oct 1 1 data mis-alignment in UTF16 collation callback arguments We're using a Unicode build of SQLite, and found that internally, the 'NOCASE' collation was not Unicode aware -- it would convert any wide strings to UTF-8 when needed, and compare those. For performance, we would like to implement a WNOCASE for wide strings. However, upon implementing this, we found that the pointers passed to our collation function are not wide character (short-) aligned. On our platform we receive data mis-alignment errors when comparing the strings. We have already implemented similiar sqlite wide string functions (WLIKE, WCONTAINS), and we never receive mis-aligned strings there (are those guaranteed to be aligned); it would be helpful if this restriction could be applied to collation callbacks. #f2dcdc 2721 code active 2007 Oct anonymous 2007 Dec 2 1 if db file is in a folder with non-ansi character some functions fail If database file is located in directory with some non-ANSI characters (in my case with a Russian subdirectory c:\Мои документы\Data_Jobs), or it's name is non-ansi. Some functions fail to execute sql. For example (with defined UNICODE):
TCHAR sql[512]; _stprintf(sql, _T("INSERT INTO tab_SurveyedPoints (name, comment, code,") _T("coordinatetype, b, l, h, solutiontype, sigmah, sigmav)") _T(" VALUES ('%s','%s','%s',0,%lf,%lf,%lf,0,%lf,%lf);"), point.m_name.c_str(), point.m_description.c_str(), point.m_code.c_str(), point.m_coordinates.b, point.m_coordinates.l, point.m_coordinates.h, point.m_sigmah, point.m_sigmav); int rc1 = sqlite3_prepare16(m_db, sqlfmt, -1, &stmt, (const void**)&pszTail); rc != SQLITE_OKBut if I move the file to c:\My documents\Data_Jobs this works ok. It's improbable behaviour, but I can't work around yet. Although, prepare() functions work ok as well in both cases. Yuri Noyanov. _2007-Oct-11 19:33:34 by drh:_ {linebreak} All string arguments to SQLite, and especially filename arguments, must be UTF-8 or UTF-16 (depending on the function). If you use string parameters which are not UTF-8 or UTF-16 (as appropriate) then the behavior of SQLite is undefined and probably not what you want. ---- _2007-Oct-12 04:25:56 by anonymous:_ {linebreak} but ALL programs to handle SQLite DBs (SQLIteBrowser, SQLite Control) fail to handle the files as well. Till I move the file to different directory !!! ---- _2007-Oct-12 04:27:54 by anonymous:_ {linebreak} Also I must note, that I CAN open the database, I CAN execute some SQLs with sqlite_prepare function OK. But sqlite_prepare16 FAILS if I just rename my database !!! ---- _2007-Oct-12 04:31:46 by anonymous:_ {linebreak} Also note to make my issue clearer: sqlite_prepare16() with the same code either works OK either doesn't work. depends on database filename or folder path. The database is opened OK in both cases (I used utf8 conversion). sql_prepare() works ok in both cases. ---- _2007-Oct-13 06:37:43 by anonymous:_ {linebreak} That appears to be only with INSERT sql statement. Both SELECT and UPDATE work fine with sqlite_prepare16. #c8c8c8 2720 code closed 2007 Oct anonymous 2007 Oct 1 1 WinCE - Can't write Application which was working correctly with one of the last sqlite 3.3.16 fails on 3.5.1 to write anything to the database - no create tables and no inserts if the db existed beforehand. It seems to happen always, with either UTF-8 or UTF-16 encoding. _2007-Oct-11 07:26:49 by anonymous:_ {linebreak} I meant, "...one of the last 3.3.x (I think 3.3.16)..." ---- _2007-Oct-11 12:20:13 by drh:_ {linebreak} The developers have no ability to compile or run on winCE, much less any ability to debug on winCE. All winCE support is community provided. There have been many patches to the winCE code since 3.5.1 was released. The writer of this ticket previously reported two other problems that have already been fixed. Several other problems have also been fixed since 3.5.1. We are guessing that this "Can't write" problem will probably go away if the writer will simply get the latest code out of CVS. If not, then there is nothing really we can do about it unless we have patches to fix the problem. #c8c8c8 2719 code closed 2007 Oct anonymous 2007 Oct 3 1 WinCE winFullPathName doesn't return a value #if OS_WINCE /* WinCE has no concept of a relative pathname, or so I am told. */ sqlite3_snprintf(pVfs->mxPathname, zFull, "%s", zRelative);
#if OS_WINCE if( (flags & (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB)) == (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB) && !winceCreateLock(zFilename, pFile) ){ CloseHandle(h); free(zConverted); return SQLITE_CANTOPEN; } if( dwFlagsAndAttributes & FILE_FLAG_DELETE_ON_CLOSE ){ pFile->zDeleteOnClose = zConverted; }else #endifThat zFilename of the winceCreateLock call doesn't exist. I guess it should be zName instead Duplicate of #2700 #c8c8c8 2717 doc closed 2007 Oct anonymous 2007 Dec anonymous 5 5 Document clarification around SQLITE_OMIT_MEMORY_ALLOCATION In the " Moving From SQLite 3.4.2 to 3.5.0" document, section 3.0 it states that "These routines are provided for use by the application. SQLite never invokes them itself. So if the application is providing its own memory allocation subsystem, it can omit these interfaces if desired." This doesn't seem to be correct unless SQLITE_OMIT_LOAD_EXTENSION is defined. If this value is not defined, then the function pointers are assigned in a structure (sqlite3_apis). As stated, they are not used by SQLite proper, but must be defined. In this case, I am unable to link without either defining the functions or building with SQLITE_OMIT_LOAD_EXTENSION defined. _2007-Oct-11 19:45:36 by anonymous:_ {linebreak} It appears that the sqlite3_memory_used() function is required in sqlite3_soft_heap_limit() so my previous comments were not entirely correct. I'm still having a link problem even with the define. ---- _2007-Dec-13 18:48:21 by drh:_ {linebreak} SQLITE_OMIT_MEMORY_ALLOCATION is no longer supported. #f2dcdc 2716 new active 2007 Oct anonymous 2007 Oct 5 1 Create Clear Command I want a command caled "clear" like in MySQL. This command should erase the screen and then put the sqlite pointer on top of the screen _2007-Oct-11 07:41:45 by anonymous:_ {linebreak} How about a cookie instead? ---- _2007-Oct-30 08:02:04 by anonymous:_ {linebreak} Clearing the screen and moving the cursor are platform-dependent operations. On Unix they are not only platform-dependent, but also terminal-dependent. Thus such a feature does not really belong in the cross-platform and minimalistic sqlite3 shell (in my opinion). #f2dcdc 2715 code active 2007 Oct anonymous 2007 Oct 1 1 no authorization needed to remove authorizer there should be a new auth code created and the auth function should be consulted for permission for removal. _2007-Oct-10 01:08:48 by drh:_ {linebreak} I'm assuming that this feature request comes from {quote: RockShox} and that the development language is Tcl. No. If your adversary has the ability to invoke the interface that removes an authorizer, then you system is already pwned. What you really need is the ability to [interp alias] the eval method into a safe interpreter. That way you can: *: Open the database in the main interpreter *: Set up the authorizer in the main interpreter to invoke a script in the main interpreter *: Set up the [interp alias] so that the safe interpreter can do [db eval ...] but not [db auth ...] It seems like an "-interp" option on the "eval" method of the database connection object would likely be the right interface. Or perhaps there should be separate "safeeval" method. Either way, it has been years and years since I have done anything with safe interpreters so I will have to look into what needs to be done to make that happen. ---- _2007-Oct-17 20:11:23 by anonymous:_ {linebreak} ok i think i agree with that. currently you cannot use an interp alias since the target command runs in the target interp and all your variables and commands are in the wrong scope. this means one needs to load sqlite again in the new interp, and sqlite will not load in a safe interp so a regular interp is required. to be useful, a -interp flag would need to execute in the current scope of the interp and not the global scope. #f2dcdc 2714 code active 2007 Oct anonymous 2007 Oct drh 3 4 Shell cannot import large files greater than 2 GB [patch] If I issue an .import command within the SQLite shell and the file size is larger than 2 GB, the shell gives the error message "cannot open file: foo.txt". Putting #define _FILE_OFFSET_BITS 64 before the #include statements in shell.c fixes this problem under Linux. I tested this solution with a 2.6 GB file under Ubuntu Feisty Fawn 7.04 with Linux kernel 2.6.20-16-generic. _2007-Oct-12 18:48:35 by drh:_ {linebreak} I'm not convinced SQLite _should_ be able to import files larger than 2GiB. Does anybody really ever need to read more than 2GiB of SQL text? That is a lot of SQL, don't you think? Is this really a desirable feature? Can you not split it up into two or more smaller files? ---- _2007-Oct-14 09:25:42 by anonymous:_ {linebreak} I'm not 100% certain about this but: #define _FILE_OFFSET_BITS 64 isn't necessarily correct, better to tweak the makefile(s): CFLAGS += $(shell getconf LFS_CLFAGS) (similar for LDFLAGS) should work on 32/64 bit machines and different platforms. For shell.c I really wouldn't bother and think this should be left alone. Using .import for load more than 2GB makes me things the shell should be fleshed out and extended for this sort of abuse, handy as it is right now, it's not a full SQL shell but a minimalist shell and a useful example). ---- _2007-Oct-16 14:08:09 by anonymous:_ {linebreak} I am the original poster. I use the shell to import CSV files in the 3-4 GiB range on a daily basis. I don't see the point of splitting the files or writing my own importer when the shell does a perfectly fine job once large file support has been activated. With all due respect, I don't understand the reluctance to enable a useful feature. Shouldn't the user decide whether it is reasonable for his application to import big files? I work in the health care industry processing insurance claims for several very large companies. There's nothing broken or wrong with my application -- big files are just the nature of the data I'm handling. Neither of you has proposed any disadvantage to enabling LFS in the shell. If there are no drawbacks, why not turn it on? If the core library can open large database files, why shouldn't the shell be able to import large ones? ---- _2007-Oct-16 16:24:22 by anonymous:_ {linebreak} For reasons also unknown to me, the authors appear to regard the shell as a largely unused sample program. Based on my personal experience in talking to various sqlite library users, it is the primary administrative front-end for sqlite. In particular, doing backups and merging databases. #cfe8bd 2713 build fixed 2007 Oct anonymous 2007 Oct 1 5 build fails for 3.5.1 on win32/cygwin - duplicate opcode labels Problem: build fails on compiling vdbe.c => src/vdbe.c: In function `sqlite3VdbeExec': src/vdbe.c:1136: error: duplicate case value src/vdbe.c:921: error: previously used here src/vdbe.c:1655: error: duplicate case value src/vdbe.c:900: error: previously used here ... Seems like my generated opcodes.h has lots of duplicates: #define OP_Multiply 80 /* same as TK_STAR */ #define OP_Pull 80 #define OP_Dup 71 #define OP_Lt 71 /* same as TK_LT */ I'm running on XP with a c:\cygwin library available. NB. i'm not a regular SQLLite user and was just trying the library out of curiosity so if no one else reports this then don't sweat it... it may be down to my environment screwing up the build process. _2007-Oct-09 17:15:24 by drh:_ {linebreak} Why don't you try compiling the {link: /cvstrac/wiki?p=TheAmalgamation amalgamation} instead? ---- _2007-Oct-09 18:06:22 by anonymous:_ {linebreak} Original poster - just run "make clean" and try again. ---- (OP) OK. tracked it down to the initialisation of the tk array in mkopcodeh.awk which is getting strings like "22 " (not sure if this is a space or a CR). the following patch fixes it for me: --- mkopcodeh.awk-orig 2007-03-29 19:39:30.000000000 +0100 +++ mkopcodeh.awk 2007-10-12 15:14:46.980260800 +0100 @@ -41,7 +41,9 @@ # Remember the TK_ values from the parse.h file /^#define TK_/ { - tk[$2] = $3 + num = $3 + num += 0 # convert "22 " -> 22 + tk[$2] = num } # Scan for "case OP_aaaa:" lines in the vdbe.c file ie. ensure that the keys are numeric. With this patch, the build completes ok. Thanks. ---- _2007-Oct-12 20:08:41 by anonymous:_ {linebreak} It's not an AWK bug in Cygwin. The only way to convert strings to numbers in AWK is to add zero. Cygwin uses the same version of GNU gawk as Linux. It's the exact same code. ---- _2007-Oct-12 20:43:55 by drh:_ {linebreak} Seems like if it were exactly the same code it would work exactly the same. But clearly it does not. I don't know what is going on. Probably some strange DOS \r line ending problems. Presumably the change has fixed the problem. #c8c8c8 2712 code closed 2007 Oct anonymous 2007 Oct 5 4 Expose a function to change the journal filename location Being able to change the pager journal filename using a nice function ie: sqlite3PagerSetJournal it maybe the following prototype: int sqlite3PagerSetJournal(sqlite3* db, const char *zJournal); It is intended to maximize disk IOs by splitting IO concerning the journal and the IO concerning the main DB storage. Typically we dedicate a specific disk channel for the journal, which may be a RAM disk like iRAM from Gigabyte, or a SATA2 disk with 16MB cache dedicated, or using a RAID controller with 256MB RAM with Backup Unit. The main DB storage resides on a different channel on SAS disks configured as RAID10. Separating journal and main data file is typically a feature available in common SQL servers : MySQL, MSSQL, ... _2007-Oct-09 15:38:50 by drh:_ {linebreak} Changing the name of the rollback journal and especially moving it to a different volume can lead to database corruption following a power failure. See http://www.sqlite.org/ac/atomiccommit.html for additional information. #cfe8bd 2711 code fixed 2007 Oct anonymous 2007 Oct 1 1 WinCE: flags incorrectly checked One last fix for WinCE: checkin 4479 is correct in the omission of certain flags for WinCE, but the flags are still being checked for later on in winOpen:
if( dwFlagsAndAttributes & FILE_FLAG_DELETE_ON_CLOSE ){ pFile->zDeleteOnClose = zConverted; }else ...Since that flag is no longer set, it needs to read:
if( flags & (SQLITE_OPEN_TEMP_DB | SQLITE_OPEN_TEMP_JOURNAL | SQLITE_OPEN_SUBJOURNAL) ){ pFile->zDeleteOnClose = zConverted; }else ...Finally, on WinCE, since FILE_FLAG_RANDOM_ACCESS is always set, it doesn't make sense to set FILE_FLAG_SEQUENTIAL_SCAN (for journals). At least on CE, it hampered performance in our tests. #cfe8bd 2710 code fixed 2007 Oct anonymous 2007 Oct 1 1 Typo in code specific to WinCE (checkin 4479) Checkin [4479] introduces a typo in code specific to WiNCE, returning SQLTIE_OK instead of SQLITE_OK _2007-Oct-09 15:21:50 by anonymous:_ {linebreak} Also, checkin 4479 is correct in the omission of certain flags for WinCE, but the flags are still being checked for later on in winOpen: if( dwFlagsAndAttributes & FILE_FLAG_DELETE_ON_CLOSE ){ pFile->zDeleteOnClose = zConverted; }else ... Since that flag is no longer set, it needs to read: if( flags & (SQLITE_OPEN_TEMP_DB | SQLITE_OPEN_TEMP_JOURNAL | SQLITE_OPEN_SUBJOURNAL) ){ pFile->zDeleteOnClose = zConverted; }else ... #c8c8c8 2709 code closed 2007 Oct anonymous 2007 Oct 2 3 Some SQLITE_OMIT_* keys break the sqlite3.c amalgamation compilation I found that SQLITE_OMIT_TRIGGER, SQLITE_OMIT_REINDEX and SQLITE_OMIT_ALTERTABLE keys break the compilation, when defined.{linebreak} Following patch solved this problem for me.{linebreak} {linebreak} sqlite-amalgamation-3_5_1.define.patch{linebreak} ----------------------------------
--- ./sqlite3.c.orig Wed Oct 3 18:08:44 2007 +++ ./sqlite3.c Tue Oct 9 14:52:09 2007 @@ -53246,6 +53266,7 @@ /* Run the BEFORE and INSTEAD OF triggers, if there are any */ endOfLoop = sqlite3VdbeMakeLabel(v); +#ifndef SQLITE_OMIT_TRIGGER if( triggers_exist & TRIGGER_BEFORE ){ /* build the NEW.* reference row. Note that if there is an INTEGER @@ -53309,6 +53330,7 @@ goto insert_cleanup; } } +#endif /* If any triggers exists, the opening of tables and indices is deferred ** until now. @@ -62064,6 +62086,7 @@ sqlite3VdbeAddOp(v, OP_MemInt, 0, memCnt); } +#ifndef SQLITE_OMIT_TRIGGER if( triggers_exist ){ /* Create pseudo-tables for NEW and OLD */ @@ -62129,6 +62152,7 @@ goto update_cleanup; } } +#endif if( !isView && !IsVirtual(pTab) ){ /* @@ -67517,10 +67541,12 @@ case 214: {sqlite3IdListDelete((yypminor->yy432));} break; +#ifndef SQLITE_OMIT_TRIGGER case 231: case 236: {sqlite3DeleteTriggerStep((yypminor->yy243));} break; +#endif case 233: {sqlite3IdListDelete((yypminor->yy370).b);} break; @@ -68378,11 +68404,13 @@ sqlite3DropTable(pParse, yymsp[0].minor.yy373, 0, yymsp[-1].minor.yy46); } break; +#ifndef SQLITE_OMIT_VIEW case 102: { sqlite3CreateView(pParse, &yymsp[-7].minor.yy0, &yymsp[-3].minor.yy410, &yymsp[-2].minor.yy410, yymsp[0].minor.yy219, yymsp[-6].minor.yy46, yymsp[-4].minor.yy46); } break; +#endif case 103: { sqlite3DropTable(pParse, yymsp[0].minor.yy373, 1, yymsp[-1].minor.yy46); @@ -68921,6 +68949,7 @@ case 250: {sqlite3Pragma(pParse,&yymsp[-1].minor.yy410,&yymsp[0].minor.yy410,0,0);} break; +#ifndef SQLITE_OMIT_TRIGGER case 258: { Token all; @@ -68935,6 +68964,7 @@ yygotominor.yy410 = (yymsp[-6].minor.yy410.n==0?yymsp[-7].minor.yy410:yymsp[-6].minor.yy410); } break; +#endif case 260: case 263: { yygotominor.yy46 = TK_BEFORE; } @@ -68972,6 +69002,7 @@ case 272: { yygotominor.yy243 = 0; } break; +#ifndef SQLITE_OMIT_TRIGGER case 273: { yygotominor.yy243 = sqlite3TriggerUpdateStep(pParse->db, &yymsp[-3].minor.yy410, yymsp[-1].minor.yy174, yymsp[0].minor.yy172, yymsp[-4].minor.yy46); } break; @@ -68987,6 +69018,7 @@ case 277: {yygotominor.yy243 = sqlite3TriggerSelectStep(pParse->db, yymsp[0].minor.yy219); } break; +#endif case 278: { yygotominor.yy172 = sqlite3PExpr(pParse, TK_RAISE, 0, 0, 0); @@ -69011,11 +69043,13 @@ case 282: {yygotominor.yy46 = OE_Fail;} break; +#ifndef SQLITE_OMIT_TRIGGER case 283: { sqlite3DropTrigger(pParse,yymsp[0].minor.yy373,yymsp[-1].minor.yy46); } break; +#endif case 284: { sqlite3Attach(pParse, yymsp[-3].minor.yy172, yymsp[-1].minor.yy172, yymsp[0].minor.yy386); @@ -69032,19 +69066,22 @@ case 287: { yygotominor.yy386 = yymsp[0].minor.yy172; } break; +#ifndef SQLITE_OMIT_REINDEX case 290: {sqlite3Reindex(pParse, 0, 0);} break; case 291: {sqlite3Reindex(pParse, &yymsp[-1].minor.yy410, &yymsp[0].minor.yy410);} break; +#endif case 292: {sqlite3Analyze(pParse, 0, 0);} break; case 293: {sqlite3Analyze(pParse, &yymsp[-1].minor.yy410, &yymsp[0].minor.yy410);} break; - case 294: +#ifndef SQLITE_OMIT_ALTERTABLE + case 294: { sqlite3AlterRenameTable(pParse,yymsp[-3].minor.yy373,&yymsp[0].minor.yy410); } @@ -69059,6 +69096,7 @@ sqlite3AlterBeginAddColumn(pParse, yymsp[0].minor.yy373); } break; +#endif case 299: {sqlite3VtabFinishParse(pParse,0);} break;---------------------------------- Thank you! _2007-Oct-09 13:22:03 by anonymous:_ {linebreak} This is because parse.c has to be regenerated from parse.y. Then, with an altered Makefile with the appropriate options, you have to run "make clean && make sqlite3.c". Could Lemon be altered to emit the %if/%ifndef/%else/%endif's as their C preprocessor equivalents instead of just not generating the if'd out parse code? This way the amalgamation could work with all OMITs. ---- _2007-Oct-09 15:13:49 by drh:_ {linebreak} Changing Lemon to generate "universal" code that works with any combination of OMIT macros is not practical. Any change to the grammer rules typically results in massiver reorganization of the parser tables. If you want to use OMIT macros, then you need to compile from the canonical source code - not the amalgamation. ---- _2007-Oct-10 15:30:31 by anonymous:_ {linebreak} I had a issue like that in previous sqlite 3.1x versions, which I want to omit alter table statement but need to generate a parser from lemon, but makefile did not supported it. there´s a way to configure --d(omit something) to omit some features of sqlite ? I know there are few on it, but not all are supported on sqlite configure script #f2dcdc 2708 code active 2007 Oct anonymous 2007 Oct 4 2 SQL error:disk I/O error I cross-compile sqlite to embedded Linux,but after I insert data to the table ,it failed.the warning is "SQL error:disk I/O error". _2007-Oct-09 05:12:28 by anonymous:_ Why do you think it is SQLite error ?? ---- _2007-Oct-09 05:46:06 by danielk1977:_ {linebreak} We'll need a bit more data than that to figure this out. Did earlier SQLite versions work? Can you post the entire output of the compile process so that we can see if there are any clues there? Can you run strace so that we can see if there really is an IO error, or at least when SQLite believes there to be one? #c8c8c8 2707 code closed 2007 Oct anonymous 2007 Oct appledev 1 3 SQLite on Mac OS X seems to have a 4GB database limit This is on Mac OX 10.4 using Apple's Core Data: I run into a strange error, always when I try to save a managed- object-context and the file-size is already at the 4GB limit: An error saving the context: NSError "An error occurred while saving." Domain=NSCocoaError Domain Code=134030 UserInfo={NSUnderlyingException = disk I/O error; } _2007-Oct-09 05:12:47 by anonymous:_ Why do you think it is SQLite error ?? ---- _2007-Oct-09 06:52:50 by danielk1977:_ {linebreak} What kind of file system are you using for the database? I used the attached script to create a 5.5 GB database on HFS+, then checked it using the integrity check and everything seemed Ok. If it's not a simple file-system limitation, is is possible for you to post some code that demonstrates the issue? Thanks. ---- _2007-Oct-12 18:46:31 by drh:_ {linebreak} Unable to reproduce. Submitter does not respond. #c8c8c8 2706 warn closed 2007 Oct anonymous 2007 Oct danielk1977 4 4 make test aborts when built with SQLITE_OMIT_ATTACH testfixture aborts during make test when it is compiled with -DSQLITE_OMIT_ATTACH=1
attach-4.5... Error: no such table: db2.t3 ./testfixture: near "DETACH": syntax error while executing "db eval { DETACH db2; }" ("uplevel" body line 1) invoked from within "uplevel [list $db eval $sql]" (procedure "execsql" line 3) invoked from within "execsql { DETACH db2; }" (file "./test/attach.test" line 505) invoked from within "source $testfile" ("foreach" body line 5) invoked from within "foreach testfile [lsort -dictionary [glob $testdir/*.test]] { set tail [file tail $testfile] if {[lsearch -exact $EXCLUDE $tail]>=0} continue if..." (file "./test/quick.test" line 93) make: *** [test] Error 1#f2dcdc 2705 code active 2007 Oct anonymous 2007 Oct 4 4 testfixture unresolved externals with SQLITE_OMIT_GET_TABLE Cannot build/run "make test" with -DSQLITE_OMIT_GET_TABLE due to testfixture link error:
In function `test_get_table_printf': ./src/test1.c:526: undefined reference to `sqlite3_get_table' ./src/test1.c:541: undefined reference to `sqlite3_free_table' collect2: ld returned 1 exit status make: *** [testfixture] Error 1#f2dcdc 2704 code active 2007 Oct anonymous 2007 Oct 4 4 "make test" aborts before completion with SQLITE_OMIT_BLOB_LITERAL When compiled with -DSQLITE_OMIT_BLOB_LITERAL make test aborts with this error:
substr-2.5.2... Ok ./testfixture: near "'61626364656667'": syntax error while executing "db eval " DELETE FROM t1; INSERT INTO t1(b) VALUES(x'$hex') "" (procedure "subblob-test" line 2) invoked from within "subblob-test 3.1 61626364656667 1 1 61" (file "./test/substr.test" line 86) invoked from within "source $testfile" ("foreach" body line 5) invoked from within "foreach testfile [lsort -dictionary [glob $testdir/*.test]] { set tail [file tail $testfile] if {[lsearch -exact $EXCLUDE $tail]>=0} continue if..." (file "./test/quick.test" line 93) make: *** [test] Error 1#f2dcdc 2703 code active 2007 Oct anonymous 2007 Oct 3 4 make test does not work with SQLITE_OMIT_FLOATING_POINT make test cannot run with SQLITE_OMIT_FLOATING_POINT. Shouldn't these tests be skipped?
./src/test1.c:1255: warning: passing argument 3 of ‘Tcl_GetDouble’ from incompatible pointer type ./src/test1.c: In function ‘sqlite3_mprintf_scaled’: ./src/test1.c:1284: warning: passing argument 3 of ‘Tcl_GetDouble’ from incompatible pointer type ./src/test1.c: In function ‘test_bind_double’: ./src/test1.c:2607: warning: passing argument 3 of ‘Tcl_GetDoubleFromObj’ from incompatible pointer type ./src/tclsqlite.c: In function ‘tclSqlFunc’: ./src/tclsqlite.c:728: warning: passing argument 3 of ‘Tcl_GetDoubleFromObj’ from incompatible pointer type ./src/tclsqlite.c: In function ‘DbObjCmd’: ./src/tclsqlite.c:1609: warning: passing argument 3 of ‘Tcl_GetDoubleFromObj’ from incompatible pointer type ... $ ./testfixture test/select1.test select1-1.1... Ok select1-1.2... Ok select1-1.3... Ok select1-1.4... Ok select1-1.5... Ok select1-1.6... Ok select1-1.7... Ok select1-1.8... Ok select1-1.8.1... Ok select1-1.8.2... Ok select1-1.8.3... Ok ./testfixture: near ".": syntax error while executing "db eval {INSERT INTO test2(r1,r2) VALUES(1.1,2.2)}" ("uplevel" body line 1) invoked from within "uplevel [list $db eval $sql]" (procedure "execsql" line 3) invoked from within "execsql {INSERT INTO test2(r1,r2) VALUES(1.1,2.2)}" (file "test/select1.test" line 69)#cfe8bd 2702 code fixed 2007 Oct anonymous 2007 Oct 1 1 Errors in CreateFileW function use on WinCE CreateFileW function in WindowsCE uses only subspace of flags of desktop windows' version and reports errors, when passed flag that can't be dispatched. For 3.5.1 this means that any attempt to open/create temporary file on WCE fails.{linebreak} Also there is bad argument name passed in winOpen to winceCreateLock.{linebreak} Also, there is 'return' missing in winFullPathname when building for WCE, so any attempt to open file fails. Here is the patch solving all(for me) filesystem errors on WindowsCE.
sqlite-amalgamation-3_5_1.ce.patch ---------------------------------- --- ./sqlite3.c.orig Wed Oct 3 18:08:44 2007 +++ ./sqlite3.c Mon Oct 8 17:04:16 2007 @@ -18730,17 +18730,24 @@ } if( flags & (SQLITE_OPEN_TEMP_DB | SQLITE_OPEN_TEMP_JOURNAL | SQLITE_OPEN_SUBJOURNAL) ){ +#if OS_WINCE + dwFlagsAndAttributes = FILE_ATTRIBUTE_HIDDEN; +#else dwFlagsAndAttributes = FILE_ATTRIBUTE_TEMPORARY | FILE_ATTRIBUTE_HIDDEN | FILE_FLAG_DELETE_ON_CLOSE; +#endif }else{ dwFlagsAndAttributes = FILE_ATTRIBUTE_NORMAL; } if( flags & (SQLITE_OPEN_MAIN_DB | SQLITE_OPEN_TEMP_DB)){ dwFlagsAndAttributes |= FILE_FLAG_RANDOM_ACCESS; - }else{ + } +#if !OS_WINCE + else{ dwFlagsAndAttributes |= FILE_FLAG_SEQUENTIAL_SCAN; } +#endif if( isNT() ){ h = CreateFileW((WCHAR*)zConverted, dwDesiredAccess, @@ -18786,7 +18793,7 @@ #if OS_WINCE if( (flags & (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB)) == (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB) - && !winceCreateLock(zFilename, pFile) + && !winceCreateLock(zName, pFile) ){ CloseHandle(h); free(zConverted); @@ -18957,6 +18964,7 @@ #if OS_WINCE /* WinCE has no concept of a relative pathname, or so I am told. */ sqlite3_snprintf(pVfs->mxPathname, zFull, "%s", zRelative); + return SQLITE_OK; #endif #if !OS_WINCE && !defined(__CYGWIN__)---------------------------------- Thank you! #f2dcdc 2701 new active 2007 Oct anonymous 2007 Oct 5 5 Make INSERT-ing multiple rows MySQL-compatible SQLite syntax allows to insert only one row with
insert into test (a, b, c) values (1, 2, 3);MySQL allows to insert multiple with
insert into test (a, b, c) values (1, 2, 3), (4, 5, 6), (7, 8, 9) -- etcBut SQLite is also capable of inserting multiple by using INSERT...SELECT:
insert into test (a, b, c) select 1, 2, 3 union select 4, 5, 6 -- etcIt would be nice to make INSERT statement syntactically compatible with MySQL, allowing to insert multiple rows with VALUES clause. It can be implemented by simply translating multiple 'VALUES ()()()' to 'select union' - no serious change required at all. _2007-Oct-08 21:45:05 by anonymous:_ {linebreak} You mean "UNION ALL", not "UNION". UNION would remove duplicate rows, and create an ephemeral table that you don't want because it's less efficient. Your idea is a good one and could be implemented largely in the parser. The number of VDBE opcodes would be quite large for such a statement. I wonder if that would present a problem. ---- _2007-Oct-14 08:12:11 by anonymous:_ {linebreak} Patch implementing multi-row INSERT statements against 3.5.1 source tree: http://www.mail-archive.com/sqlite-users%40sqlite.org/msg28337.html #cfe8bd 2700 code fixed 2007 Oct anonymous 2007 Oct 1 1 invalid name in winOpen when compiling under WinCE in os_win.c:1165, WinCE specific code to create lock still refers to former zFilename variable whereas argument is now named zName. --- os_win.c-orig 2007-10-07 22:04:02.000000000 +0200 +++ os_win.c 2007-10-07 23:32:51.000000000 +0200 @@ -1162,7 +1162,7 @@ #if OS_WINCE if( (flags & (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB)) == (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB) - && !winceCreateLock(zFilename, pFile) + && !winceCreateLock(zName, pFile) ){ CloseHandle(h); free(zConverted); #cfe8bd 2699 code fixed 2007 Oct anonymous 2007 Oct 2 2 Performance issues in 3.5.1 (Windows) As reported on mailing list: http://www.mail-archive.com/sqlite-users%40sqlite.org/msg28164.html "Forcibly adding FILE_FLAG_RANDOM_ACCESS to the flags in os_win.c's winOpen fixed it. It looks like in 3.4.2, that flag was set on all files (journals and db's), while in 3.5, it (was) only being set on db's. Adding it to journals too improves performance almost tenfold. This may also help performance on desktop Windows." #cfe8bd 2698 code fixed 2007 Oct anonymous 2007 Oct 3 4 memory leak in WIN CE because tempfilename is not freed at WinClose in a build for Windows CE ( OS_WIN is defined ) temporary files have to be deleted after winclose because the operating systems does not delete themself. At the first sqlite3_get_table() somewhere in the code in function sqlite3WinOpenExclusive() the call to convertUtf8Filename() returns a newly allocated name for the tempfile. This is afterwards assigned to f.DeleteOnClose but is never deleted or freed in WinClose, when hMutex is not used. _2007-Oct-08 04:49:56 by anonymous:_ {linebreak} This was introduced in checkin [3836] and subsequently reported in bugs #2533 and #2598. #cfe8bd 2697 code fixed 2007 Oct anonymous 2007 Oct 1 3 sqlite3_open_v2 fails then filename includes non-english chars Can't create database with new sqlite3_open_v2 function if filename is international (non English) char filename[MAX_PATH]; memset(filename, 0, sizeof(filename)); ::WideCharToMultiByte(CP_ACP, 0, jobfilename.c_str(), -1, filename, sizeof(filename)-1, 0, 0); int dbcreateres = sqlite3_open_v2(filename, &m_db, SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, NULL); dbcreateres == 14 By the way, why we just don't have sqlite3_open16_v2 function !!! It would be best solution for me ! changing the code to ::WideCharToMultiByte(CP_UTF8, ... fixes the issue... sorry for your time... #cfe8bd 2696 code fixed 2007 Oct anonymous 2007 Oct 3 3 sqlite3.c with SQLITE_OMIT_FLOATING_POINT cannot compile on Linux Unmodified compile of sqlite3.c with SQLITE_OMIT_FLOATING_POINT will generate these errors:
In file included from /usr/include/math.h:141, from sqlite3.c:10227: /usr/include/bits/mathcalls.h:55: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘acosl’ /usr/include/bits/mathcalls.h:55: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘__acosl’ /usr/include/bits/mathcalls.h:57: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘asinl’ /usr/include/bits/mathcalls.h:57: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘__asinl’ /usr/include/bits/mathcalls.h:59: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘atanl’ /usr/include/bits/mathcalls.h:59: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘__atanl’ /usr/include/bits/mathcalls.h:61: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘atan2l’ etc...#include
Index: src/shell.c =================================================================== RCS file: /sqlite/sqlite/src/shell.c,v retrieving revision 1.167 diff -u -3 -p -r1.167 shell.c --- src/shell.c 7 Sep 2007 01:12:32 -0000 1.167 +++ src/shell.c 5 Oct 2007 06:19:58 -0000 @@ -1458,6 +1458,7 @@ static int do_meta_command(char *zLine, fprintf(p->out,"\n"); }else +#ifndef SQLITE_OMIT_GET_TABLE if( c=='t' && n>1 && strncmp(azArg[0], "tables", n)==0 ){ char **azResult; int nRow, rc; @@ -1514,6 +1515,7 @@ static int do_meta_command(char *zLine, } sqlite3_free_table(azResult); }else +#endif if( c=='t' && n>1 && strncmp(azArg[0], "timeout", n)==0 && nArg>=2 ){ open_db(p);_2007-Oct-05 14:34:39 by drh:_ {linebreak} I would imagine that there are many SQLITE_OMIT_... macros that will break the CLI. The CLI is a separate program which is distinct from the SQLite library and is not expected to work with every possible configuration of the SQLite library. So this really should be a feature request and not a bug report. My dilemma now is whether to accept or reject this feature request. The changes are relatively minor. But on the other hand, the requested feature is very obscure. Why is it that you want to have a CLI that omits the sqlite3_get_table() API? As I sit here pondering, I cannot think of a single reason why anybody would want to do such a thing, and so for that reason I am rejecting this feature request. If somebody (the original poster or anybody else) can give me a legitimate reason for wanting to build the CLI without the sqlite3_get_table() interface, I will reconsider the rejection. ---- _2007-Oct-05 15:49:01 by anonymous:_ {linebreak} Why? To test all the OMIT macros on a representative program already in the source tree. I found other OMIT compile problems outside of shell.c when performing this exercise. I'll file a seperate ticket SQLITE_OMIT_FLOATING_POINT. No one is saying anyone would #define this particular OMIT. It is just there for completeness. It gives programmers confidence in using the sqlite library without investing too much time in it. For that matter, it would be nice to test an OMIT-filled sqlite3 library against the exactly same configured corresponding sqlite3 shell. Also, you define it if you wanted to build a minimalistic shell for a severely memory-constrained embedded device. Truth be told, the .table command is a nice but rarely used command whose info can be gleaned with a simple query on sqlite_master anyway. ---- _2007-Oct-05 16:08:23 by anonymous:_ {linebreak} By the way, only a couple of other OMITs broke shell.c: SQLITE_OMIT_COMPLETE SQLITE_OMIT_DISKIO You did a better job than you realized. ;-) ---- _2007-Oct-05 16:26:04 by anonymous:_ {linebreak} Was able to successfully compile/link/run shell.c with: THREADSAFE=0 SQLITE_OMIT_ALTERTABLE SQLITE_OMIT_ANALYZE SQLITE_OMIT_ATTACH SQLITE_OMIT_AUTHORIZATION SQLITE_OMIT_AUTOINCREMENT SQLITE_OMIT_AUTOVACUUM SQLITE_OMIT_BETWEEN_OPTIMIZATION SQLITE_OMIT_BLOB_LITERAL SQLITE_OMIT_CAST SQLITE_OMIT_CHECK SQLITE_OMIT_COMPOUND_SELECT SQLITE_OMIT_CONFLICT_CLAUSE SQLITE_OMIT_DATETIME_FUNCS SQLITE_OMIT_EXPLAIN SQLITE_OMIT_FLAG_PRAGMAS SQLITE_OMIT_FLOATING_POINT (see Ticket #2696) SQLITE_OMIT_FOREIGN_KEY SQLITE_OMIT_GET_TABLE (with above patch) SQLITE_OMIT_GLOBALRECOVER SQLITE_OMIT_INCRBLOB SQLITE_OMIT_INTEGRITY_CHECK SQLITE_OMIT_LIKE_OPTIMIZATION SQLITE_OMIT_LOAD_EXTENSION SQLITE_OMIT_MEMORYDB SQLITE_OMIT_OR_OPTIMIZATION SQLITE_OMIT_PAGER_PRAGMAS SQLITE_OMIT_PROGRESS_CALLBACK SQLITE_OMIT_QUICKBALANCE SQLITE_OMIT_REINDEX SQLITE_OMIT_SCHEMA_PRAGMAS SQLITE_OMIT_SCHEMA_VERSION_PRAGMAS SQLITE_OMIT_SHARED_CACHE SQLITE_OMIT_SUBQUERY SQLITE_OMIT_TCL_VARIABLE SQLITE_OMIT_TEMPDB SQLITE_OMIT_TRACE SQLITE_OMIT_TRIGGER SQLITE_OMIT_UTF16 SQLITE_OMIT_VACUUM SQLITE_OMIT_VIEW SQLITE_OMIT_VIRTUALTABLE SQLITE_OMIT_XFER_OPT ---- _2007-Oct-05 16:28:11 by drh:_ {linebreak} If you are building a minimalist shell, you should start from scratch and not try to use shell.c. There is a lot of cruft in shell.c (above and beyond ".table") that a minimalist shell does not want. So this is not a valid reason for wanting shell.c to support SQLITE_OMIT_GET_TABLE. Likewise, there a lot of ways to test to make sure SQLite work with SQLITE_OMIT_GET_TABLE without compiling shell.c. I remain unconvinced. Feature request still rejected. ---- _2007-Oct-05 16:33:33 by anonymous:_ {linebreak} Wow, you won't check in an trivially obvious perfectly harmless 2 line patch despite a half dozen reasons why it's useful. A lot of people use shell.c for testing and in production - probably more people use the CLI than do the Tcl API. #cfe8bd 2694 code fixed 2007 Oct anonymous 2007 Oct drh 2 2 Makefile.in is missing invokation of mksqlite3internalh.tcl for a10n The amalgamation should produce both sqlite3.c and sqlite3internal.h which is correct in the master main.mk but missing in Makefile.in; the following patch corrects Makefile.in: --- sqlite3.orig/Makefile.in 2007-09-04 00:11:33.000000000 +0200 +++ sqlite3/Makefile.in 2007-10-04 22:24:00.000000000 +0200 @@ -333,6 +333,7 @@ sqlite3.c: target_source $(TOP)/tool/mksqlite3c.tcl tclsh $(TOP)/tool/mksqlite3c.tcl + tclsh $(TOP)/tool/mksqlite3internalh.tcl # Rules to build the LEMON compiler generator # _2007-Oct-05 14:24:38 by drh:_ {linebreak} See my comment on ticket #2693. #cfe8bd 2693 code fixed 2007 Oct anonymous 2007 Oct drh 2 2 mutex.h missing in a10n of sqlite3internal.h mutex.h is still included in amalgamatized sqlite3internal.h; maybe that patch fixes it. --- tool/mksqlite3internalh.tcl.orig 2007-08-14 15:21:27.000000000 +0200 +++ tool/mksqlite3internalh.tcl 2007-10-04 22:52:46.000000000 +0200 @@ -57,6 +57,7 @@ btreeInt.h hash.h keywordhash.h + mutex.h opcodes.h os_common.h os.h _2007-Oct-05 14:23:58 by drh:_ {linebreak} Our intent is to eliminate sqlite3internal.h. As of 3.5.0 there should never be a need to access the internal definitions of SQLite. Everything that you need to use SQLite properly is contained in sqlite3.h. If you need definitions found in the internal include files, that indicates that you are using SQLite in a way that is unsupported and likely to break from one release to the next. #c8c8c8 2692 build closed 2007 Oct anonymous 2007 Oct 1 1 COmpilation of Sqlite amalgamation v3.4.2. failed Version 3.4.2 (amalgamation) could not be compiled under MS DevStudio 6 the compilation errors are: sqlite3.c(6187) : error C2133: 'sqlite3UpperToLower' : unknown size sqlite3.c(9310) : error C2133: 'sqlite3OpcodeNames' : unknown size sqlite3.c(47159) : error C2133: 'sqlite3IsIdChar' : unknown size _2007-Oct-04 18:19:44 by drh:_ {linebreak} Duplicate of #2574. #c8c8c8 2691 code closed 2007 Oct anonymous 2007 Oct 1 1 SQLITE_OMIT_AUTHORIZATION compilation error sqlite3.Int.h does not compile with SQLITE_OMIT_AUTHORIZATION. Rationale: sqlite3Int.h, line 1768: # define sqlite3AuthRead(a,b,c) is missing a 4th parameter like this: # define sqlite3AuthRead(a,b,c,d) thanks. #c8c8c8 2690 code closed 2007 Oct anonymous 2007 Oct 3 4 "PRAGMA synchronous = OFF;" is more dangerous than documented The "PRAGMA synchronous = OFF;" is much more dangerous than implied in the documentation (at http://www.sqlite.org/pragma.html). On our embedded system written data may remain in a middleware buffer (without being passed to the OS) until a subsequent read, write, seek or close call occurs. In particular, the last write to the database file may remain buffered semi-locally for quite a while after the lock is released. This will inevitably cause problems if the database is read through a different handle meanwhile - even by the same thread! I have seen this occur (fairly regularly), when a subsequent SELECT returns SQLITE_CORRUPT. Four pages are read through another handle before the error is raised. The write didn't complete until the file was closed as part of the application shutdown more than 10 seconds later. It might reasonably be argued that our middleware is broken in this case, but I'm sure that there are other systems with similar problems. The problem is still present in version 3.4.1 where the sqlite3OsSync call at pager.c:3954 should not be conditional on !pPager->noSync. Another solution may be to change the documentation to highlight this risk. _2007-Oct-04 14:26:18 by drh:_ {linebreak} From the description, this seems definitely to be a bug in your alternative OS interface, not in the delivered SQLite code. Perhaps you should modify your middleware so that it flushes pending writes prior to unlocking. #c8c8c8 2689 build closed 2007 Oct anonymous 2007 Oct 3 4 make test fails in 3.5.1 - unresolved externals in testfixture tar zxvf sqlite...... cd sqlite-3.5.1 mkdir bld cd bld ../configure make make test ./libtool --mode=link gcc -g -O2 -I. -I../src -DNDEBUG -I/System/Library/Frameworks/Tcl.framework/Versions/8.4/Headers -DSQLITE_THREADSAFE=1 -DSQLITE_THREAD_OVERRIDE_LOCK=-1 -DSQLITE_OMIT_LOAD_EXTENSION=1 -DTCLSH=1 -DSQLITE_TEST=1 -DSQLITE_CRASH_TEST=1 \ -DSQLITE_NO_SYNC=1 -DTEMP_STORE=1 \ -o testfixture ../src/attach.c ../src/btree.c ../src/build.c ../src/date.c ../src/expr.c ../src/func.c ../src/insert.c ../src/malloc.c ../src/os.c ../src/os_os2.c ../src/os_unix.c ../src/os_win.c ../src/pager.c ../src/pragma.c ../src/prepare.c ../src/printf.c ../src/select.c ../src/test1.c ../src/test2.c ../src/test3.c ../src/test4.c ../src/test5.c ../src/test6.c ../src/test7.c ../src/test8.c ../src/test9.c ../src/test_autoext.c ../src/test_async.c ../src/test_btree.c ../src/test_config.c ../src/test_hexio.c ../src/test_malloc.c ../src/test_md5.c ../src/test_schema.c ../src/test_server.c ../src/test_tclvar.c ../src/tokenize.c ../src/utf.c ../src/util.c ../src/vdbe.c ../src/vdbeapi.c ../src/vdbeaux.c ../src/vdbemem.c ../src/where.c parse.c ../src/tclsqlite.c \ libsqlite3.la -framework Tcl -lpthread -framework CoreFoundation gcc -g -O2 -I. -I../src -DNDEBUG -I/System/Library/Frameworks/Tcl.framework/Versions/8.4/Headers -DSQLITE_THREADSAFE=1 -DSQLITE_THREAD_OVERRIDE_LOCK=-1 -DSQLITE_OMIT_LOAD_EXTENSION=1 -DTCLSH=1 -DSQLITE_TEST=1 -DSQLITE_CRASH_TEST=1 -DSQLITE_NO_SYNC=1 -DTEMP_STORE=1 -o .libs/testfixture ../src/attach.c ../src/btree.c ../src/build.c ../src/date.c ../src/expr.c ../src/func.c ../src/insert.c ../src/malloc.c ../src/os.c ../src/os_os2.c ../src/os_unix.c ../src/os_win.c ../src/pager.c ../src/pragma.c ../src/prepare.c ../src/printf.c ../src/select.c ../src/test1.c ../src/test2.c ../src/test3.c ../src/test4.c ../src/test5.c ../src/test6.c ../src/test7.c ../src/test8.c ../src/test9.c ../src/test_autoext.c ../src/test_async.c ../src/test_btree.c ../src/test_config.c ../src/test_hexio.c ../src/test_malloc.c ../src/test_md5.c ../src/test_schema.c ../src/test_server.c ../src/test_tclvar.c ../src/tokenize.c ../src/utf.c ../src/util.c ../src/vdbe.c ../src/vdbeapi.c ../src/vdbeaux.c ../src/vdbemem.c ../src/where.c parse.c ../src/tclsqlite.c -framework Tcl -framework CoreFoundation ./.libs/libsqlite3.dylib -lpthread /usr/bin/ld: Undefined symbols: _Sqlitetest7_Init _SqlitetestOnefile_Init _SqlitetestThread_Init collect2: ld returned 1 exit status make: *** [testfixture] Error 1 _2007-Oct-04 11:21:47 by anonymous:_ {linebreak} This is mac os x ---- _2007-Oct-04 15:40:13 by anonymous:_ {linebreak} Same unresolved externals for testfixture on Linux. Fix below. Do a "make distclean" and re-run configure/make test.
Index: Makefile.in =================================================================== RCS file: /sqlite/sqlite/Makefile.in,v retrieving revision 1.183 diff -u -3 -p -r1.183 Makefile.in --- Makefile.in 3 Sep 2007 22:15:45 -0000 1.183 +++ Makefile.in 4 Oct 2007 15:37:57 -0000 @@ -212,6 +212,8 @@ SRC += \ # Source code to the test files. # TESTSRC = \ + $(TOP)/src/test_onefile.c \ + $(TOP)/src/test_thread.c \ $(TOP)/src/attach.c \ $(TOP)/src/btree.c \ $(TOP)/src/build.c \ Index: src/test7.c =================================================================== RCS file: /sqlite/sqlite/src/test7.c,v retrieving revision 1.9 diff -u -3 -p -r1.9 test7.c --- src/test7.c 12 Sep 2007 17:01:45 -0000 1.9 +++ src/test7.c 4 Oct 2007 15:37:57 -0000 @@ -21,8 +21,8 @@ ** This test only works on UNIX with a SQLITE_THREADSAFE build that includes ** the SQLITE_SERVER option. */ -#if defined(SQLITE_SERVER) && !defined(SQLITE_OMIT_SHARED_CACHE) -#if defined(OS_UNIX) && OS_UNIX && SQLITE_THREADSAFE +#if defined(SQLITE_SERVER) && !defined(SQLITE_OMIT_SHARED_CACHE) && \ + defined(OS_UNIX) && OS_UNIX && SQLITE_THREADSAFE #include---- _2007-Oct-04 15:43:15 by anonymous:_ {linebreak} Output of make test after patch: 0 errors out of 31801 tests All memory allocations freed - no leaks Maximum memory usage: 14160983 bytes ---- _2007-Oct-04 19:09:05 by anonymous:_ {linebreak} I am the original reporter - that fix worked for me, thanks. My results (OS X) 0 errors out of 31207 tests All memory allocations freed - no leaks Maximum memory usage: 14161431 bytes #c8c8c8 2688 build closed 2007 Oct anonymous 2007 Oct 3 1 SQLITE_OMIT_AUTHORIZATION build failure Version 3.5.1 does not compile when specifying -DSQLITE_OMIT_AUTHORIZATION. The problem lies on line 6872 (of the amalgamation): =# define sqlite3AuthRead(a,b,c)= should be: =# define sqlite3AuthRead(a,b,c,d)= thanks. #cfe8bd 2687 code fixed 2007 Oct anonymous 2007 Nov 3 2 Building with _UNICODE fails in Win32 (3.5.1) The function winDlError uses the unadorned function FormatMessage() which gets turned into FormatMessageW() in a Unicode build. Because the output buffer is a char*, it needs to be specified as FormatMessageA() to avoid the compile time error.#include @@ -720,5 +720,4 @@ int Sqlitetest7_Init(Tcl_Interp *interp) } #else int Sqlitetest7_Init(Tcl_Interp *interp){ return TCL_OK; } -#endif /* OS_UNIX */ #endif
static void winDlError(sqlite3_vfs *pVfs, int nBuf, char *zBufOut){ FormatMessageA( FORMAT_MESSAGE_FROM_SYSTEM, NULL, GetLastError(), 0, zBufOut, nBuf-1, 0 ); }Note that if the output message is supposed to be in UTF-8 then extra processing needs to be added to this function to convert the message into that character encoding. _2007-Oct-04 03:59:59 by anonymous:_ {linebreak} In summary:
- FormatMessage( + FormatMessageA(---- _2007-Nov-12 18:37:43 by anonymous:_ {linebreak} Please reopen this ticket: Windows Mobile / CE does not necessarily provide the ANSI version FormatMessageA(). This was observed with the arm-wince-mingw32ce GCC cross compiler. Maybe the following replacement could be added to os_win.c #ifdef UNDER_CE int FormatMessageA(int dwFlags, void *lpSource, int dwMessageId, int dwLangId, char *lpBuffer, int nSize, va_list *args){ if( dwFlags & 256 ){ char *buf = (char *)LocalAlloc(64, 126); *(char **)lpBuffer = buf; if( dwMessageId > 0x7FFFFFF ){ sprintf(buf, "OsError 0x%x", dwMessageId, dwMessageId); }else{ sprintf(buf, "OsError %d", dwMessageId); } return strlen(buf); } if( dwMessageId > 0x7FFFFFF ){ sprintf((char *)lpBuffer, "OsError 0x%x", dwMessageId); }else{ sprintf((char *)lpBuffer, "OsError %d", dwMessageId); } return strlen((char *)lpBuffer); } #endif #cfe8bd 2686 code fixed 2007 Oct anonymous 2007 Oct 3 3 Integrity check fails if you hit max_page_count and dont rollback If you set PRAGMA max_page_count to reachable value, fill database until SQLITE_FULL is returned, and don't do a rollback immediately (say, because your program quits or crashes before rollback), the database will be left in damaged state (PRAGMA integrity_check will detect errors). This happens for me for both 3.4.2 and 3.5.0. This is serious defect which breaks A.C.I.D. ideology, although it is not a show stopper and will not happen often to users. Furthermore, one time this ended in "database is malformed" error, although I could not reproduce this again. Try modifying the test below to not stop after integrity check failure, let it cycle hundreds times, and you will get malformed database error sooner or later. This should not happen, database integrity must stay o.k. whenever program crashes before doing rollback, or simply never does a rollback. The following code reproduces the problem. Notice the commented out conn.rollback() line, if you uncomment it, bug will disappear. You may consider adding this test to other SQLite unit-tests.
#!/usr/bin/python import sqlite3, random, os, os.path, sys dbfile = '''db.tmp''' if os.path.exists(dbfile): os.unlink(dbfile) conn = sqlite3.connect(dbfile) cur = conn.cursor() cur.execute('''PRAGMA page_size=1024''') # just to make sure cur.execute('''PRAGMA max_page_count=5120''') # 5 megabytes cur.execute('''PRAGMA auto_vacuum=0''') cur.execute('''CREATE TABLE filler (fill)''') while 1: try: cur.execute('''INSERT INTO filler (fill) VALUES (randomblob(%d))''' % int(random.uniform(10240, 102400))) conn.commit() sys.stdout.write('.') except sqlite3.OperationalError, e: if e.message != "database or disk is full": break sys.stdout.write('*') #conn.rollback() #uncomment this and error will disappear cur.execute('''DELETE FROM filler WHERE rowid <= (SELECT MAX(rowid) FROM filler LIMIT 20)''') cur.execute('''PRAGMA integrity_check''') r = cur.fetchall() s = str(r[0][0]) if s != 'ok': print "Something is wrong! Integrity check returned", s print r break#cfe8bd 2685 code fixed 2007 Oct anonymous 2007 Oct anonymous 2 1 Compilation under MSVC gives warning on TryEnterCriticalSection If SQLite is compiled under MSVC 8.0 for instance it gives following warning while compiling mutex_w32.c: src\mutex_w32.c(147) : warning C4013: 'TryEnterCriticalSection' undefined; assuming extern returning int This warning shall be treated as error, if correct behavior of try mutex is exected. Solution: to declare #define _WIN32_WINNT 0x0400 before # include
@@ -1162,13 +1168,13 @@ #if OS_WINCE if( (flags & (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB)) == (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB) - && !winceCreateLock(zFilename, &f) + && !winceCreateLock(zName, pFile)#c8c8c8 2682 code closed 2007 Oct anonymous 2007 Oct 1 1 SQLite Text Problem I am using SQLite 3 as backend database in my asp application. I am using SQLite3 ODBC driver (latest version). Table Design ------------ CREATE TABLE log ( id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, from_value TEXT NULL, to_value TEXT NULL ); INSERT 10 RECORDS IN TO DATABASE -------------------------------- Const adOpenStatic = 3 Const adLockOptimistic = 3 Dim SQLString ,objConnection Dim fromVal,toval Set objConnection = CreateObject("ADODB.Connection") for a = 1 to 10 objConnection.Open "DSN=Test;uid=;pwd=;" fromVal = "FROM Value : "& a toVal = "To Value : "& a SQLString = "INSERT INTO log(from_value, to_value) VALUES ('"& fromVal & "' ,'"&toVal& "')" objConnection.Execute SQLString objConnection.Close next Access records in the log table ------------------------------- Dim LogRs,SQLStmt SQLStmt = "SELECT * FROM log " LogRs.ActiveConnection = "DSN=Test;uid=;pwd=;" LogRs.Source= SQLStmt LogRs.CursorType = 3'ADOPENSTATIC LogRs.CursorLocation = 2 'ADUSESERVER LogRs.LockType = 3 'ADLOCKOPTIMISTIC LogRs.Open() RecordCount = LogRs.RecordCount If I use datatype of from_value/to_value as TEXT ,I get RecordCount as zero which is wrong. If I use datatype of from_value/to_value as Varchar(1600) ,I get RecordCount as 10 which is correct. If I use datatype of from_value/to_value as Varchar(1600) and tried to put more than 1600 charcters in it ,SQLite stores data in DB. While retrieving the data I get the error Arguments are of the wrong type,are out of acceptable range,or are in conflict with one another. I enabled ODBC trace log ,but its empty after running my application.So I am not sure where it is ODBC problem or SQLite ODBC driver error? If I open table (database) in SQLite Studio I could see data without any error. If I use MS Access with same table design and same code, I dont get any error . The code works fine as expected . _2007-Oct-01 22:04:18 by drh:_ {linebreak} There are multiple ODBC drivers for SQLite, none of which are support by this site. Your problem appears to be in the ODBC driver. If you find evidence to the contrary, please reopen this ticket. #cfe8bd 2681 code fixed 2007 Oct anonymous 2007 Oct 3 3 #ifndef _XOPEN_SOURCE needed around #define _XOPEN_SOURCE 500 _XOPEN_SOURCE is really poorly defined/implemented in general. Some platforms need _XOPEN_SOURCE to be set to 600, or a value other than 500. Could you please put an #ifdef wrapper around this, as follows? #ifndef _XOPEN_SOURCE # ifndef __MACOS__ # define _XOPEN_SOURCE 500 /* Needed to enable pthread recursive mutexes */ # endif #endif _2007-Oct-01 16:47:50 by anonymous:_ {linebreak} Furthermore, can you avoid defining _XOPEN_SOURCE if it is not a threadsafe build? (i.e., define _XOPEN_SOURCE only if threadsafe build requested). #cfe8bd 2680 code fixed 2007 Sep anonymous 2007 Oct 2 2 sqlite_int64 vs sqlite3_int64 type confusion in headers and docs Header file sqlite3.h (3.4.2) defines sqlite_int64 but not sqlite3_int64. 3.5.0 defines both. I discovered this used the 3.5.0 documentation (on the site) but building against a local installation of 3.4.2. The documentation and examples used BOTH types in a somewhat ad-hoc manner. Could someone please clarify which is the preferred type? If sqlite3_int64 is preferred, then should the header files for 3.4.x be updated to reflect this? _2007-Oct-03 20:21:49 by anonymous:_ {linebreak} sqlite 3.4.2 doesn't define sqlite3_int though, only sqlite_int this means i can't use common code for both sqlite3.4.2 and sqlite3.5.0 ---- _2007-Oct-03 20:29:51 by anonymous:_ {linebreak} Just use sqlite_int then. They are typedef'd to the same thing in 3.5. #c8c8c8 2679 code closed 2007 Sep anonymous 2007 Oct 2 3 locking errors lead to disk i/o failure with bulk operations on winxp First got lost cursor and disk i/o errors with bulk imports to sqlite database using sqliteodbc driver with =OpenOffice= under win xp sp 2. Then used sqlite3.exe to reproduce and found: if sqlite database is on an ext2 volume accessed through ext2ifs driver, and I .read a sql command file with 348 times INSERT INTO ... VALUES('...','...'); to a table with two text columns, after some (hundred or more) rows without error I receive the following error: SQL error near line xxx: disk I/O error for all following lines; likewise, when I replace the INSERT by DELETE FROM ... WHERE col='...', I will receive the same error after some ten lines. The database cannot be accessed any more as long as the sqlite3 executable is running; closing and restarting it restores access (same with odbc driver - no access unless connection closed and reopened). I can omit these errors by surrounding the statements with a BEGIN TRANSACTION / COMMIT clause (not available with OOo and the odbc driver, unfortunately). Likewise, when the sqlite database is on a ntfs volume, the error is gone. I first thought there might be an error with the ext2 ifs driver, but I use it heavily for all kinds of data without problems so far. Then I took a closer look at the sqlite3 executable by monitoring its activity using sysinternals' procmon. And this revealed that there were no failing reads or writes as the error message would suggest, but I noticed RANGE NOT LOCKED results with =UnlockFileSingle= operations (corresponding to =UnlockFileEx= calls in the source) when the error surfaces. Examing the sequence of =LockFile= and =UnlockFileSingle= on locations 1gb (1), 1gb+1 (1) and 1gb+2 (510) I found that sqlite3 indeed tried to unlock locations already unlocked previously - the reported errors (and the file locking implementation in the ext2 driver) were correct. It seems that sqlite looses track with its locks after multiple lock/unlock sequences. I append the procmon log of locking activities of the sqlite3 executable during 'mass' inserts in csv format. _2007-Sep-28 23:22:07 by anonymous:_ {linebreak} Could you clarify a few things? How many SQLite processes are there in your scenario? What is the actual schema and SQL statements issued by each process? ---- _2007-Sep-29 18:25:50 by anonymous:_ {linebreak} could be a ext2fs ifs driver issue. did you tested id in NTFS / FAT32 filesystems ? ---- _2007-Sep-29 23:32:09 by anonymous:_ {linebreak} ok, first to clarify: there is a *single* sqlite3 process, absolutly no concurrency; and a simple table used, like {quote: CREATE TABLE "WhoIsWho" ("Nummer" varchar(50) NOT NULL, "Teilnehmer" varchar(50), PRIMARY KEY ("Nummer"));} then I feed a text file with 348 statements like {quote: INSERT INTO WhoIsWho VALUES('01805312111','conrad electronic hotline');} with different values to a sqlite3-executable, either by issuing a .read statement at the sqlite3 prompt or using input redirection on the commandline. likewise, with full table to delete the entries I use a file with statements like {quote: DELETE FROM WhoIsWho WHERE Nummer=='01805220111';} this should make the setup clear - but what do you mean with schema? and about ext2ifs issue and testing on ntfs: see above ;-) and read on but now: I digged a lot deeper in this issue: first of all, there is a big difference in the sequence of operations as seen by procmon between ext2 and ntfs. on ntfs, there are 2 (!) {quote: "CloseFile C:\"} between all other operations, and a lot of additional {quote: "CloseFile
#define MX_DELETION_ATTEMPTS 3 static int winDelete( sqlite3_vfs *pVfs, /* Not used on win32 */ const char *zFilename, /* Name of file to delete */ int syncDir /* Not used on win32 */ ){ int cnt = 0; int rc; void *zConverted = convertUtf8Filename(zFilename); if( zConverted==0 ){ return SQLITE_NOMEM; } SimulateIOError(return SQLITE_IOERR_DELETE); if( isNT() ){ do{ rc = DeleteFileW(zConverted); }while( rc==0 && GetFileAttributesW(zConverted)!=0xffffffff && cnt++ < MX_DELETION_ATTEMPTS && (Sleep(100), 1) ); }else{ #if OS_WINCE return SQLITE_NOMEM; #else do{ rc = DeleteFileA(zConverted); }while( rc==0 && GetFileAttributesA(zConverted)!=0xffffffff && cnt++ < MX_DELETION_ATTEMPTS && (Sleep(100), 1) ); #endif } free(zConverted); OSTRACE2("DELETE \"%s\"\n", zFilename); return rc!=0 ? SQLITE_OK : SQLITE_IOERR; }---- _2007-Sep-30 12:35:32 by anonymous:_ {linebreak} Hmm, I think your suggestions point to the *wrong place*. If there were multiple attempts to delete the file, I would have seen them in the =ProcMon= log. But this is not the case. The =DeleteFileW= succeeds immediately (rc!=0), so the rest of the while clause is never executed. But, as the documentation says, a successful =DeleteFile= call does not mean the file *is deleted*, it only means the file has been *marked for deletion*, and the OS is free to schedule the actual deletion *anytime* after the =CloseFile= call. We should look at the code that checks for a left over journal file. This code already reads the file attributes, so it should be easy to check for the =DeletePending= flag: if it is set it should treat the journal file as if it were already deleted. Nevertheless, there is a fault in the fs driver or the OS: the MS documentation says that a =CreateFile= call on a file marked for deletion fails with ERROR_ACCESS_DENIED - but in this case it succeeds! The second error - loosing track of locks during rollback - still has to be addressed. ---- _2007-Sep-30 15:59:26 by anonymous:_ {linebreak} What happens when you insert the Sleep calls after CloseFile? ---- _2007-Oct-01 12:47:25 by anonymous:_ {linebreak} If I would place a Sleep(1000) after every =CloseFile= (I didn't) then the errror would shurely vanish - and my 348 inserts would last 6 minutes! That's not the way to go. I suggested to look for the code to check for the journal file still there, but I see this has already been accounted for in the code to exclusively open a file on windows, here:
/* ... ** Sometimes if we have just deleted a prior journal file, windows ** will fail to open a new one because there is a "pending delete". ** To work around this bug, we pause for 100 milliseconds and attempt ** a second open after the first one fails. The whole operation only ** fails if both open attempts are unsuccessful. */ int sqlite3WinOpenExclusive(const char *zFilename, OsFile **pId, int delFlag){ ... do{ h = CreateFileW((WCHAR*)zConverted, GENERIC_READ | GENERIC_WRITE, 0, NULL, CREATE_ALWAYS, fileflags, NULL ); }while( h==INVALID_HANDLE_VALUE && cnt++ < 2 && (Sleep(100), 1) ); ... }This requires the =CreateFileW= call to fail and return an invalid handle - which is what Microsoft specifies for a file marked for deletion but what the file system driver does not acknowledge: it allows the file marked for deletion to be successfully opened. So it's again the ext2ifs driver developer's turn: I have already contacted him ... ---- _2007-Oct-01 14:46:18 by anonymous:_ {linebreak} Adding the Sleep would simply confirm your theory. No one is suggesting it be added permanently. ---- _2007-Oct-07 23:21:26 by anonymous:_ {linebreak} I finally tracked down also the unlock errors: they are there by design. When a hot journal is found, the immediate rollback is purposely only with an EXCLUSIVE lock, but without a RESERVED or PENDING lock. Nevertheless, the unlock code assumes that whenever there is an EXCLUSIVE lock, then there might also be a RESERVED and a PENDING lock, and both are unlocked too. That's nothing to bother with, as it only provokes error returns from the OS, the handling itself is correct. See:
static int winUnlock(OsFile *id, int locktype){ ... if( type>=RESERVED_LOCK ){ UnlockFile(pFile->h, RESERVED_BYTE, 0, 1, 0); } ... if( type>=PENDING_LOCK ){ UnlockFile(pFile->h, PENDING_BYTE, 0, 1, 0); } ... }---- _2007-Oct-12 18:50:28 by drh:_ {linebreak} **Summary:** This turns out to be a bug in the ext2 driver for winNT, not in SQLite. The locking problem is not a real issue. ---- _2007-Oct-14 16:27:15 by anonymous:_ {linebreak} Just to approve: the ext2ifs driver developer sent me a patched driver that returns a "DELETE PENDING" error when a journal file already marked for deletion tried to be opened, and with it the errors are gone. I suppose the corrected driver will soon be general available. #cfe8bd 2678 code fixed 2007 Sep anonymous 2007 Oct 1 1 amalgamation zip file should include sqlite3ext.h It is not possible to build portable external loadable modules using just sqlite-amalgamation-3_5_0.zip - instead you need the entire sqlite3 source tree. If sqlite3ext.h were bundled with the amalgamation, this would not be necessary. _2007-Sep-28 19:44:00 by anonymous:_ {linebreak} Actually, a better solution would be to get rid of sqlite3ext.h altogether, fold its functionality into sqlite3.h and when people go to compile a module they #define whether they wish to compile an external library or not. #f2dcdc 2677 new active 2007 Sep anonymous 2007 Sep danielk1977 5 3 CREATE TRIGGER .. {databaseevent ON ... }+ for multiple events I have to write triggers which react on either INSERT or UPDATE of a specific field in a table. In some SQL syntax descriptions I found a possibility to combine the trigger events for more than one condition but not in SQLite3. Isn't there a possibility to use one trigger definition for more than one event or didn't I find the trick? Do I have to copy the whole definition for each event; even if the body text is exactly the same? {linebreak} Suggestion: On INSERT event use the OLD.fieldnames filled with NULL content to avoid problems with UPDATE event. - If not possible please explain why not for my better understanding. Thank you. #c8c8c8 2676 code closed 2007 Sep anonymous 2007 Sep 1 1 TEMP_STORE does not appear to be set by the compile time switch The compile time switch -DTEMP_STORE=2 does not appear to change TEMP_STORE. When a database is created after compiling the library I tried to view temp_store by using PRAGMA temp_store which returns the value 0. If this is true that would force all temp storage to a file. I want temp storage to be in memory. It's more complex than that. See the table at: http://www.sqlite.org/pragma.html#pragma_temp_store #c8c8c8 2675 new closed 2007 Sep anonymous 2007 Sep 1 1 TEMP_STORE does not appear to be set by the compile time switch The compile time switch -DTEMP_STORE=2 does not appear to change TEMP_STORE. When a database is created after compiling the library I tried to view temp_store by using PRAGMA temp_store which returns the value 0. If this is true that would force all temp storage to a file. I want temp storage to be in memory. See #2676 #f2dcdc 2674 code active 2007 Sep anonymous 2007 Sep 3 4 NFS fails without lock manager *Problem:* SQLite fails entirely if the NFS lock manager is not running on the share hosting the DB -- even when all access to the DB is serialized. Under these circumstances, it becomes difficult to create applications that are capable of running in a multitude of environments because restrictions are now imposed upon the storage location. ---- *Reproduction:* *:setup an NFS share *:disable the lock manager *:attempt to perform any transactions on a db on that share *:start the lock manager *:perform the same operations ---- *Request:* *:Make a change to the error handling, as necessary, to allow processes to access a DB over an NFS share without use of a lock manager. *:Make a big bold flashing sign in the FAQ about this failure-mode. The wording of the current FAQ led us to believe that the transaction would go through, but protection from other processes was not guaranteed. ---- *Shameless Fanmail:* Love the product! ---- *Screenshot:* dev-srs08 ~ #
** The sqlite3_errmsg() and sqlite3_errmsg16() return English-langauge
int *pAutoinc /* OUTPUT: True if colums is auto-increment */
** When the virtual-table mechanism stablizes, we will declare the
** The optimizer automatically inverts terms of the form "expr OP column"
** and makes other simplificatinos to the WHERE clause in an attempt to
** be taylored to the specific needs of the module implementation. The
In sqlite3.h
#define SQLITE_CONSTRAINT 19 /* Abort due to contraint violation */
In sqliteInt.h
/*
Thanks.
#cfe8bd 2659 code fixed 2007 Sep rse 2007 Sep danielk1977 2 2 openDirectory function broken IMHO the recently changed/introduced function src/os_unix.c:openDirectory() is broken in case the "zFilename" is just a filename without any path information in it (e.g. "foo.db"). In this case "ii" is 0 and as a result an uninitialized value "fd" is passed back and because of "fd" is uninitialized even the return code could be arbitrarily "SQLITE_OK" or "SQLITE_CANOPEN". The fix IMHO is to open "." in case no path information is present in "zFilename" and pass back an "fd" only if open(2) really was successful. Same for the return code. I append a patch which fixes this. But as I don't know what the original intentions were when this function was adjusted recently the patch should be reviewed by someone and then rejected or applied accordingly.
#cfe8bd 2658 code fixed 2007 Sep anonymous 2007 Sep 2 3 sqlite3VtabOverloadFunction broken Triggered by a compiler warning ("rc" not initialized) I reviewed the function src/vtab.c:sqlite3VtabOverloadFunction() and came to the conclusion that it is broken at all: 1. The compiler is right that "rc" is not initialized before it is used for comparison. 2. If sqlite3DbStrDup fails a the uninitialized "xFunc" function pointer is passed back in the result structure. I do not understand this piece of code in detail, so I can only guess what it should do. IMHO the function has to be fixed this way: if sqlite3DbStrDup fails one has to short-circuit the processing and immediately return/pass-through "pDef" as is. Then use xFindFunction and this way correctly initialize "rc" and then return "pDef" again if "rc" is _now_ zero. I append a patch for fixing the code this way. I'm rather sure this is the way the function was intended to work, but as I'm not 100% sure please review it and apply or reject accordingly.
#c8c8c8 2657 build closed 2007 Sep anonymous 2007 Oct anonymous 1 1 fail to compile I fail to compile. It is a "win32 static library" project. I simply create an empty project with "not using MFC" option and then add all files included in "sqlite-source-3_5_0.zip" into the project. Debug configuration is fine, but I cannot compile release. Information: ...sqlite-source-3_5_0\btree.c(3111) : fatal error C1001: INTERNAL COMPILER ERROR (compiler file 'F:\9782\vc98\p2\src\P2\regasg.c', line 646) Please choose the Technical Support command on the Visual C++ Help menu, or open the Technical Support help file for more information Error executing cl.exe. _2007-Sep-21 00:05:42 by anonymous:_ {linebreak} I have the same problem. I am using the preprocessed source code in sqlite-source-3_5_0.zip with Visual Studio 6.0 SP6. Compiler version: Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 12.00.8804 for 80x86 The offending line of btree.c is line 3111, which looks fairly innocent to me: pCur->aOverflow[iIdx] = nextPage; After investigating, it looks like this error is only emitted when global optimizations (/Og) are enabled, which is one of the switches set by optimizing for maximum speed. As a workaround, you can insert the line #pragma optimize("g", off) before line 3026 and the line #pragma optimize( "", on) after line 3156, which are the beginning and ending of the function static int accessPayload(BtCursor,int,int,unsigned char,int,int) This will disable global optimizations for the function, but leave them enabled for the rest of the file. {link: http://msdn2.microsoft.com/en-us/library/y19zxzb2(vs.71).aspx MSDN} has more information about error C1001, but it is not quite the same as what the VS6 docs say: {quote: This error is most often generated [due to] ... Failure of the code generator to find a way to generate correct code for a construct. This is most often caused by the interaction of an expression and an optimization option. The optimization has generated a tree which the compiler does not know how to handle. Such a problem can often be fixed by removing one or more optimization options when compiling the particular function containing the line indicated in the error message. } ---- _2007-Oct-12 18:53:47 by drh:_ {linebreak} In other words, there is a bug in MSVC++ that prevents it from correctly compiling SQLite. Shouldn't this be a problem reported to Microsoft? What exactly can we (the SQLite developers) do about this? Is there some simple code change that we can make to work around the bug in MSVC++?
#c8c8c8 2656 doc closed 2007 Sep anonymous 2007 Oct 2 1 missing documentation: END TRANSACTION The documentation lists END [TRANSACTION] as a legal command, but does not describe it. It should be documented that is is the same as COMMIT [TRANSACTION]. Reference: http://www.mail-archive.com/sqlite-users@sqlite.org/msg03000.html _2007-Oct-01 13:32:34 by drh:_ {linebreak} The END TRANSACTION syntax works but it is unsupported. If I document it that means I will have to begin supporting it, which is something I do not want to do. ---- _2007-Oct-01 17:41:52 by anonymous:_ {linebreak} The END DOCUMENTATION *syntax* is documented - but it is not described what it does. While this might be intentional, I know about a novice user who was confused if he had to call END DOCUMENTATION after either COMMIT or ROLLBACK. If you decide not to document END DOCUMENTATION, you might want to think about not documenting it at all, in other words, remove the syntax declaration. ---- _2007-Oct-01 17:46:07 by drh:_ {linebreak} OK. We have added documentation for END TRANSACTION. ---- _2007-Oct-02 06:30:17 by anonymous:_ {linebreak} Thanks!
#c8c8c8 2655 code closed 2007 Sep anonymous 2007 Sep 2 3 Reuse prepared statement after locked(5) gives called out of seq(21) Reusing a prepared statement after locked(5) error gives library routine called out of sequence(21) Create a test db: sqlite3 test.db CREATE TABLE test (id INTEGER PRIMARY KEY AUTOINCREMENT, val INTEGER); go Lock db by running a statement in a transaction: begin transaction go delete from test where id = 99 go Try to run a prepared statement on the table via Perl DBI will give database is locked(5) at the first try. On following tries library routine called out of sequence(21) will be returned. Only works if statement handle is undef:ed and prepared again. If this is a problem in the SQLite DBD or in the SQLite library I don't know. Environment: FreeBSD 6.1 Perl 5.8.8 DBI 1.50 SQLite DBD 1.13 SQLite 3.4.2 Test script: #!/usr/local/bin/perl use DBI; my $dbh = DBI->connect("dbi:SQLite:dbname=test.db","","",{RaiseError => 0}) or die "Couldn't open dbfile."; $dbh->func(1000, 'busy_timeout'); my $sql = "DELETE FROM test WHERE id = ?"; my $sth = $dbh->prepare($sql); die "Couldn't prepare" unless defined $sth; my $ret; while(! defined $ret){ sleep 1; print "Execute sth\n"; $ret = $sth->execute(452); print "ERROR: " . $sth->err . $sth->errstr . "\n" if $ret; $sth->finish; } if($ret){ print "Failure\n"; } else{ print "Success\n"; } DBI->trace(5) is attached. _2007-Sep-19 11:47:29 by drh:_ {linebreak} You appear to be calling sqlite3_finalize() (I'm guessing that is what the "finish" method does) on the statement after the error. sqlite3_finalize() is the destructor for a statement. Then you try to use the object after it has been destroyed. Most systems would segfault under those circumstances. SQLite is nice and gives you an SQLITE_MISUSE error. ---- _2007-Sep-19 14:33:56 by anonymous:_ {linebreak} How is the library supposed to be used in order to reuse the already prepared statement? (According to perldoc DBI->finish is supposed to finish collection of data from an execution. It is not supposed to be destructive on the statement handle. So I guess that is a misuse of the library from SQLite DBD.) If I leave out DBI->finish I get the same error. I added the trace when not running finish.
#c8c8c8 2654 code closed 2007 Sep anonymous 2007 Oct jadams 1 1 sqlite3_prepare return 1(fail) result = sqlite3_exec( db, "create table MyTable_1sadsd ( ID integer primary key autoincrement, name nvarchar(32),comp nvarchar(32), file_content blob )",NULL, NULL, errmsg );//ok result=sqlite3_prepare(db,"insert into MyTable_1sadsd ( name,comp,file_content ) values ( 'aaaa','aaaaaa1',? )",-1,&stat,&zLeftover );//fail //the result==1 so fail; //but result = sqlite3_exec( db, "create table MyTable( ID integer primary key autoincrement, name nvarchar(32),comp nvarchar(32), file_content blob )",NULL, NULL, errmsg );//ok result=sqlite3_prepare(db,"insert into MyTable ( name,comp,file_content ) values ( 'aaaa','aaaaaa1',? )",-1,&stat,&zLeftover );//ok //is ok ,because MyTable is shorter than MyTable_1sadsd ; //why ? tks It's not clear from the report. There's nothing wrong with the SQL or th e C code shown. Maybe some other process is changing the schema or locking the dataabase or something. If you can post a complete program that demonstrates the problem, we will be able to help you.
#f2dcdc 2653 code active 2007 Sep anonymous 2007 Sep 3 4 Exclusive Transactions do not work with a Database File attached twice Regarding the docs, it is possible to attach the same database file multiple times. After doing so, I wanted to begin an exclusive transaction. Unfortunately, this fails ("database is locked") and surprises me as I did not find any notice on this particular situation and possible side-effects neither in the attach nor in the transaction/locking documentation. If this behaviour is seen as an error, It would be useful to have it error fixed, because if one reads a list of (possibly duplicate) database files but with unique identifiers, it would be helpful to use these defined identifiers when accessing the databases (and that in an exclusive transaction). Real-world case: Each "module" uses its own database file. Some modules share a database file. So there is a list of module -> database file assignments. Now an update process gets some database update scripts from the modules. Every module wants its changes to be done in the right database, so it relies on having its own database attached with an unique identifier - the module's name. On the other hand, the update process need exclusive access to the databases and starts a transaction -- bummer.
#f2dcdc 2652 code active 2007 Sep drh 2007 Sep 1 1 Aggregate function cannot be used from within a subquery The following SQL fails: CREATE TABLE t1(x,y); INSERT INTO t1 VALUES(1,2); CREATE TABLE t2(z); INSERT INTO t2 VALUES(1); SELECT (SELECT y FROM t1 WHERE x=min(z)) FROM t2; Problem reported on the mailing list. _2007-Sep-23 16:01:09 by anonymous:_ {linebreak} Your syntax appears to be incorrect.{linebreak} SQLite v3.4.2 CREATE TABLE t1(x,y); CREATE TABLE t2(z); INSERT INTO t1 VALUES(1,21); INSERT INTO t1 VALUES(2,22); INSERT INTO t1 VALUES(3,23); INSERT INTO t2 VALUES(3); INSERT INTO t2 VALUES(2); INSERT INTO t2 VALUES(1); What you wanted to do: SELECT y FROM t1 WHERE x=(SELECT min(z) FROM t2); 21 -- works as expected What you did: SELECT (SELECT y FROM t1 WHERE x=min(z)) FROM t2; SQL error near line []: misuse of aggregate function min()
#f2dcdc 2651 new active 2007 Sep anonymous 2007 Sep 5 4 Add support for overriding home directory location Currently, the history file's and the rc file's location is hard-wired to =$HOME=. It would be nice if this could be overridden. One way is to look for a =SQLITE_HOME= environment variable that points to the location to use. This can be achieved by a simple addition to =find_home_dir()= in =src/shell.c=.
#c8c8c8 2650 build closed 2007 Sep anonymous 2007 Sep 1 1 lack ; in os_win.c:1329 Hi, I want just to inform you that it misses one; in the file os_win.c (line 1329). in winFullPathname, #if defined(__CYGWIN__) cygwin_conv_to_full_win32_path(zRelative, zFull); return SQLITE_OK <--- lack ';' #endif Regards YoCarBo Duplicate of #2642 and #2647
#f2dcdc 2649 new active 2007 Sep anonymous 2007 Sep 4 4 Add an "--enable-extensions" (default=no) to the configure script The attached patch adds "--enable-extensions" to the configure script, but is disabled by default (because of the security considerations of having it enabled).
#cfe8bd 2648 build fixed 2007 Sep anonymous 2007 Sep 1 1 compilation error for PocketPC/WindowsCE function winOpen has some type errors under OS_WINCE preprocessor
** SQLite supports many different ways to resolve a contraint
** error. ROLLBACK processing means that a constraint violation
#if OS_WINCE if( (flags & (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB)) == (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB) && !winceCreateLock(zFileName, &f) ){ CloseHandle(h); free(zConverted); return SQLITE_CANTOPEN; } if( dwFlagsAndAttributes & FILE_FLAG_DELETE_ON_CLOSE ){ pFile->zDeleteOnClose = zConverted; }else #endifThe line 18511 should be: && !winceCreateLock(zName, pFile) Also it's have mistake in typing FILE_FLAG_DELETE_ON_CLOSE (with last underlines - FILE_FLAG_DELETEONCLOSE in line 18517). Please change the sources to be compiled for WindowsCE. BR. Yuri Noyanov #c8c8c8 2647 code closed 2007 Sep anonymous 2007 Sep 1 1 sqlite-3.5.0 Cygwin: Compilation fails on src/os_win.c - ';' missing There is a ';' symbol missing on line 1329 in file os_win.c (winFullPathname function) after return statement:
#if defined(__CYGWIN__) cygwin_conv_to_full_win32_path(zRelative, zFull); return SQLITE_OK #endifDuplicate of Ticket #2642 #c8c8c8 2646 code closed 2007 Sep anonymous 2007 Sep a.rottmann 4 1 sqlite3_get_table loses one table row sqlite3_get_table seems to loose one row. this patch fixes the problem for me. If it is wrong, please fix the bug in a right way. --- sqlite-3.4.2/src/table.c 2007-06-12 15:18:01.000000000 +0300 +++ sqlite-3.4.2-x/src/table.c 2007-09-14 13:55:32.000000000 +0300 @@ -77,6 +77,7 @@ static int sqlite3_get_table_cb(void *pA } p->azResult[p->nData++] = z; } + p->nRow++; }else if( p->nColumn!=nCol ){ sqlite3SetString(&p->zErrMsg, "sqlite3_get_table() called with two or more incompatible queries", Can you take a look at the documentation here and confirm that you are interpreting it correctly? http://www.sqlite.org/capi3ref.html#sqlite3_free_table In particular, this bit: "Notice that there is an extra row of data containing the column headers. But the *nrow return value is still 3. *ncolumn is set to 2. In general, the number of values inserted into azResult will be ((*nrow) + 1)*(*ncolumn)." The code looks like it matches this description to me. If you find that it does not, please re-open this ticket. Thanks. #f2dcdc 2645 build active 2007 Sep anonymous 2007 Sep 5 4 Conflict between tclConfig.sh and tclinstaller.tcl I'm using a non-standard location for Tcl: --with-tcl=/home/scott/lib The build process finds and uses the tclConfig.sh file from /home/scott/lib just fine, but tclinstaller.tcl is called without an explicit path and uses the system's tclsh instead of the one in /home/scott/bin and thus tries to install that portion of code into the system's area. While I can change my path to resolve this, I think it makes more sense for tclinstaller.tcl to use the path information that's embedded in tclConfig.sh to be consistent. /s. #c8c8c8 2644 code closed 2007 Sep anonymous 2007 Sep 1 1 sqlite Will you tell me why sqlite program is running on my laptop computer? Is it part of another program. Your answer and time is really appreciated. Thank you. _2007-Sep-13 18:00:08 by drh:_ {linebreak} SQLite is used by Mcaffee Anti-virus, AOL Mail, Skype, Adobe Lightroom, and countless other programs, most of which we are completely unaware of. This website is for reporting bugs against SQLite, not for getting help with figuring out what it is or why it is running on your computer. Please use the SQLite mailing list for such help. #cfe8bd 2643 code fixed 2007 Sep anonymous 2007 Sep 1 1 bestIndex: Assertion `flags!=0' failed after index creation I get an assertion crash on a database using sqlite 3.3.12 and also 3.4.2:
sqlite> create index idx_media_items_p1 on media_items (p1); sqlite> select count(1) from media_items where p1 is not null; lt-sqlite3: ./src/where.c:1486: bestIndex: Assertion `flags!=0' failed. Aborted (core dumped) Program received signal SIGABRT, Aborted. [Switching to Thread 47454430964672 (LWP 24045)] 0x00002b28d839247b in raise () from /lib/libc.so.6 (gdb) bt #0 0x00002b28d839247b in raise () from /lib/libc.so.6 #1 0x00002b28d8393da0 in abort () from /lib/libc.so.6 #2 0x00002b28d838bbf6 in __assert_fail () from /lib/libc.so.6 #3 0x00002b28d7fd3360 in bestIndex () from /home/steve/dev/sqlite-3.4.2/.libs/libsqlite3-3.4.2.so.0 #4 0x00002b28d7fd3f7c in sqlite3WhereBegin () from /home/steve/dev/sqlite-3.4.2/.libs/libsqlite3-3.4.2.so.0 #5 0x00002b28d7fb3bdc in sqlite3Select () from /home/steve/dev/sqlite-3.4.2/.libs/libsqlite3-3.4.2.so.0 #6 0x00002b28d7fa2568 in yy_reduce () from /home/steve/dev/sqlite-3.4.2/.libs/libsqlite3-3.4.2.so.0 #7 0x00002b28d7fa4e52 in sqlite3Parser () from /home/steve/dev/sqlite-3.4.2/.libs/libsqlite3-3.4.2.so.0 #8 0x00002b28d7fb5719 in sqlite3RunParser () from /home/steve/dev/sqlite-3.4.2/.libs/libsqlite3-3.4.2.so.0 #9 0x00002b28d7fa8e1d in sqlite3Prepare () from /home/steve/dev/sqlite-3.4.2/.libs/libsqlite3-3.4.2.so.0 #10 0x00002b28d7fa92f6 in sqlite3_prepare () from /home/steve/dev/sqlite-3.4.2/.libs/libsqlite3-3.4.2.so.0 #11 0x00002b28d7fd70df in sqlite3_exec () from /home/steve/dev/sqlite-3.4.2/.libs/libsqlite3-3.4.2.so.0 #12 0x0000000000406a74 in process_input () #13 0x0000000000407bc2 in main ()core: http://skrul.com/core.gz I can send the db file upon request. _2007-Sep-13 17:09:29 by anonymous:_ {linebreak} Two additional notes: This does not crash on 3.3.4, plus this particular database was created using the "attach database" command from another database. ---- _2007-Sep-13 17:18:42 by drh:_ {linebreak} I need to know the schema for the media_items table. ---- _2007-Sep-13 17:22:13 by anonymous:_ {linebreak}
steve@steve-linux:~/dev/sqlite-3.4.2/.libs$ ./sqlite3 /home/steve/dev/songbird/db/flat2.db SQLite version 3.4.2 Enter ".help" for instructions sqlite> .schema CREATE TABLE media_items ( media_item_id integer primary key autoincrement, guid text unique not null, created integer not null, updated integer not null, content_url text not null, content_mime_type text, content_length integer, hidden integer not null check(hidden in (0, 1)), media_list_type_id integer, p1 text, p1_sort text, p2 text, p2_sort text, p3 text, p3_sort text, p4 text, p4_sort text, p5 text, p5_sort text, p6 text, p6_sort text, p7 text, p7_sort text, p8 text, p8_sort text, p9 text, p9_sort text, p10 text, p10_sort text , p11 text); CREATE INDEX idx_media_items_p1 on media_items (p1); CREATE INDEX idx_media_items_p11 on media_items (p11); CREATE INDEX idx_media_items_p2 on media_items (p2); sqlite>---- _2007-Sep-13 17:24:37 by drh:_ {linebreak} I also want to see a ".dump" of the sqlite_stat1 table. ---- _2007-Sep-13 17:25:52 by anonymous:_ {linebreak}
sqlite> .dump sqlite_stat1 BEGIN TRANSACTION; ANALYZE sqlite_master; INSERT INTO "sqlite_stat1" VALUES('media_items','idx_media_items_p11','61799 1'); INSERT INTO "sqlite_stat1" VALUES('media_items','sqlite_autoindex_media_items_1','61799 1'); COMMIT; sqlite>---- _2007-Sep-13 18:03:19 by anonymous:_ {linebreak} Great! I applied those changes to my sqlite 3.4.2 where.c and it fixed the problem. Thanks for the unbelievably fast fix! #c8c8c8 2642 code closed 2007 Sep anonymous 2007 Sep 5 5 CYGWIN compile error due to missing semicolon Missing semi-colon near line 1329 of os_win.c after SQLITE_OK. #if defined(__CYGWIN__) cygwin_conv_to_full_win32_path(zRelative, zFull); return SQLITE_OK #endif Thanks. #c8c8c8 2641 code closed 2007 Sep anonymous 2007 Sep 2 3 sqlite - Ticket #2638 continued as it was arbitrarily closed unsolved to continue the thread from 2638. I tried as a blob and it was identical behaviour, this was not a workaround. behaviour is different in different versions for test or blobs. used to work, is now broken Indeed, the behaviour has changed. SQLite is now less tolerant of non utf-8 encoded strings being handled as text. It is very likely that that is the reason your shiftjis data is being mangled. It should not of happened if the data was being treated as a blob. Suggest going to the mailing-list for clarification. There are many clever people there who understand this kind of issue. If it does turn out to be a bug, we can reopen this ticket. #cfe8bd 2640 code fixed 2007 Sep anonymous 2007 Sep 1 1 sqlite doesn't treat global identifiers correctly in a subquery Apparently, sqlite doesn't treat global identifiers correctly in a subquery; in particular, with my database filling, the following query gives a wrong result (3 rows) whereas changing the innermost p.person_id into d.person_id (which are equal by the preceding condition) gives the correct result (56 rows). select distinct p.name from persons p, directors d where d.person_id=p.person_id and not exists ( select * from directors d1 where d1.person_id=p.person_id except select * from writers w ) _2007-Sep-12 11:36:42 by danielk1977:_ {linebreak} Can you provide a sample populated database? This would be a lot quicker to look into that way... If it's too large to attach to this ticket, you can mail it to me: dan@sqlite.org ---- _2007-Sep-12 14:30:33 by anonymous:_ {linebreak}
CREATE TABLE persons(person_id, name); INSERT INTO "persons" VALUES(1,'fred'); INSERT INTO "persons" VALUES(2,'barney'); INSERT INTO "persons" VALUES(3,'wilma'); INSERT INTO "persons" VALUES(4,'pebbles'); INSERT INTO "persons" VALUES(5,'bambam'); CREATE TABLE directors(person_id); INSERT INTO "directors" VALUES(5); INSERT INTO "directors" VALUES(3); CREATE TABLE writers(person_id); INSERT INTO "writers" VALUES(2); INSERT INTO "writers" VALUES(3); INSERT INTO "writers" VALUES(4); select distinct p.name from persons p, directors d where d.person_id=p.person_id and not exists ( select * from directors d1 where d1.person_id=p.person_id except select * from writers w ); wilma select distinct p.name from persons p, directors d where d.person_id=p.person_id and not exists ( select * from directors d1 where d1.person_id=d.person_id except select * from writers w ); wilma bambam#c8c8c8 2639 code closed 2007 Sep anonymous 2007 Oct 1 2 Russian charset At implementation of query, Russian charset is inserted like a "" _2007-Sep-12 02:29:24 by drh:_ {linebreak} I do not understand the complaint. But a good guess is that you are trying to use a codepage of some kind in a TEXT entry. SQLite understands only Unicode - either UTF8 or UTF16. Translate all of your strings into one of these encodings first, before inserting them into the database, if you want them to be compared correctly. If you do not want to use unicode, insert your data as BLOBs instead of a strings. #c8c8c8 2638 code closed 2007 Sep anonymous 2007 Sep 2 3 selects on japanese text okay in 3.3.5 bad in later versions I have a database that was created in version 3.3.5 that contains japanese shift-jis text fields (a kanji lookup db). It works fine with the 3.3.5 dll. When I put the 3.4.2 or alpha 3.5 dll in a select of a japanese character does not match characters in the DB correctly, instead of returning 300 or so matching entries it returns 2000 non matching ones. I have a workaround in the sense that I can keep the current 3.3.5 dll in the app and it works but I'd like to move forward to new versions soon. It's a database of japanese characters with japanese pronounciation and english meanings. format is code page 932. User puts a character in and it matches to a character in the system. replacing the dll and putting the same character in exactly (ie, paste current clipboard in both cases to be sure input codex is the same) and results are completely different. _2007-Sep-12 01:09:08 by drh:_ {linebreak} SQLite uses Unicode, either UTF8 or UTF16. You can put in string that are not well-formed UTF8 or UTF16 and sometimes it might work. But later versions have become noticably less tolerate of illformed UTF8. This is due to security concerns. See, for example, ticket #2029. If you want to store strings of codepage 932, I suggest you store them as BLOBs and not as text strings. #c8c8c8 2637 code closed 2007 Sep anonymous 2007 Sep 1 1 Segfault in pthread_getspecific () Hi, I am trying to run Trac 0.11-dev with Apache 2.2.3-3.2ubuntu0.1/mod_python and libsqlite3 3.4.2. Accessing Trac over Apache causes segfault in libsqlite3. This has something to do with threading (models), I assume. I am not sure if this is an configuration issue, but I couldn't find any help after hard googling. I tried also libqlite 3.3.x with the same result. I marked the bug with severity 1, since there is no known workaround AFAIK. I compiled libsqlite3 to get the symbolic traceback:
#0 0xb7c521fa in pthread_getspecific () from /lib/tls/i686/cmov/libpthread.so.0 #1 0xb7d07f01 in sqlite3Insert (pParse=0x0, pTabList=0xb7d39bc8, pList=0xbfe16c98, pSelect=0xb7cebe8c, pColumn=0xb7cdf298, onError=1) at ./src/insert.c:1564 #2 0xb7d1e40e in sqlite3_get_table_cb (pArg=0xb7cdf298, nCol=1, argv=0xa582000, colv=0xb7d39bc8) at ./src/table.c:107 #3 0xb7cebe8c in sqlite3AuthRead (pParse=0xb7cebe8c, pExpr=0xbfe16c98, pTabList=0xb7d39bc8) at ./src/auth.c:118 #4 0xb7cebf18 in sqlite3AuthRead (pParse=0xb7d39bc8, pExpr=0xa582000, pTabList=0x1) at ./src/auth.c:140 #5 0xb7d118b6 in sqlite3Parser (yyp=0xa6ff558, yymajor=_2007-Sep-11 14:47:34 by anonymous:_ {linebreak} Also, Trac works fine if started using tracd standalone daemon tool ---- _2007-Sep-11 14:47:34 by anonymous:_ {linebreak} Could you try sqlite 3.5? The threading code has been completely redone. ---- _2007-Sep-11 14:57:17 by anonymous:_ {linebreak} No dice. libsqlite 3.5.0:, yyminor= {z = 0xffffffff , dyn = 0, n = 0}, pParse=0xbfe16e34) at parse.y:468 #6 0xb7d11fc1 in sqlite3Parser (yyp=0xa6ff558, yymajor= , yyminor= {z = 0xffffffff , dyn = 0, n = 1609611034}, pParse=0xbfe16e38) at parse.y:651 #7 0xb7d2f625 in getMask (pMaskSet=0x1, iCursor=1) at ./src/where.c:289 #8 0xb7d12243 in sqlite3Parser (yyp=0xbfe16ef8, yymajor= , yyminor= {z = 0xbfe16f08 "\222\023ӷ\232cӷ\200dӷ", dyn = 0, n = 0}, pParse=0x0) at parse.y:706 #9 0xb7d12347 in sqlite3Parser (yyp=0xb7cdffdc, yymajor= , yyminor= {z = 0xbfe1734c "X�o\n", dyn = 0, n = 0}, pParse=0xbfe16f78) at parse.y:735 #10 0xb7d1279c in sqlite3Parser (yyp=0xa6ff558, yymajor= , yyminor= {z = 0xa43a300 "system", dyn = 0, n = 86104312}, pParse=0xb7d39bc8) at parse.y:768 #11 0xb7d1288f in sqlite3Parser (yyp=0xbfe1734c, yymajor= , yyminor= {z = 0xa760dc8 "�>u\no", dyn = 0, n = 86104336}, pParse=0xbfe17048) at parse.y:812 #12 0xb7cf46da in sqlite3BtreeClose (p=0x0) at ./src/btree.c:1236 #13 0xb7d16c56 in base_vprintf (xRealloc=0, useInternal=-1210868792, zInitBuf=0x1 , nInitBuf=-1211228996, zFormat=0xa760dc8 "�>u\no", ap=0xb7d36480 "CREATE TABLE sqlite_master(\n type text,\n name text,\n tbl_name text,\n rootpage integer,\n sql text\n)") at ./src/printf.c:785 #14 0xb7d173ce in base_vprintf (xRealloc=0xbfe16f28, useInternal=-1210882944, zInitBuf= , nInitBuf=-1075743924, zFormat= , ---Type to continue, or q to quit--- ap=0xe9e1744a ) at ./src/printf.c:589 #15 0xb7d179d6 in base_vprintf (xRealloc=0xbfe17038, useInternal=0, zInitBuf= , nInitBuf=-1075743924, zFormat= , ap=0xbfe17090 "\030r��") at ./src/printf.c:548 #16 0xb7d0d75d in pager_playback (pPager=0x68, isHot=16777221) at ./src/pager.c:1474 #17 0xb7d1b000 in prepSelectStmt (pParse=0x5cc2c, p=0xb7cdd000) at ./src/select.c:1368 #18 0xb7d1191f in sqlite3Parser (yyp=0xa6ff558, yymajor= , yyminor= {z = 0xffffffff , dyn = 0, n = 0}, pParse=0xa73708c) at parse.y:480 #19 0xb7d11fc1 in sqlite3Parser (yyp=0xa6ff558, yymajor= , yyminor= {z = 0xffffffff , dyn = 0, n = 87668806}, pParse=0xbfe17488) at parse.y:651 #20 0xb6810cef in statement_create () from /usr/lib/python2.5/site-packages/pysqlite2/_sqlite.so #21 0xb680c7ad in connection_call () from /usr/lib/python2.5/site-packages/pysqlite2/_sqlite.so #22 0xb7547987 in PyObject_Call () from /usr/lib/libpython2.5.so.1.0 #23 0xb7547c03 in ?? () from /usr/lib/libpython2.5.so.1.0 #24 0x0a681a70 in ?? () #25 0x0a4154ec in ?? () #26 0x00000000 in ?? ()
#0 0xb7c521fa in pthread_getspecific () from /lib/tls/i686/cmov/libpthread.so.0 #1 0xb7d07f01 in sqlite3Insert (pParse=0x0, pTabList=0xb7d39bc8, pList=0xbfe16c98, pSelect=0xb7cebe8c, pColumn=0xb7cdf298, onError=1) at ./src/insert.c:1465 #2 0xb7d1e40e in sqlite3Select (pParse=0xb7cdf298, p=0x1, eDest=171204008, iParm=-1210868792, pParent=0x0, parentTab=0, pParentAgg=0xbfe16ca8, aff=0xb7cebf18 "�\205�\017\225�\017���\215�&") at ./src/select.c:2915 #3 0xb7cebe8c in attachFunc (context=0x0, argc=0, argv=0x0) at ./src/attach.c:82 #4 0xb7cebf18 in attachFunc (context=0x0, argc=-1211232372, argv=0x1) at ./src/attach.c:114 #5 0xb7d118b6 in sqlite3PagerWrite (pDbPage=0xa8fa470) at ./src/pager.c:4156 #6 0xb7d11fc1 in sqlite3PagerOpen (pVfs=0xa8fa470, ppPager=0xb7d36480, zFilename=0xffffffff , nExtra=-1075745228, flags=-1075745224, vfsFlags=1) at ./src/pager.c:2110 #7 0xb7d2f625 in sqlite3VdbeHalt (p=0xa8fa470) at ./src/vdbeaux.c:1142 #8 0xb7d12243 in yy_find_shift_action (pParser=0x0, iLookAhead=1 '\001') at parse.c:1442 #9 0xb7d12347 in yy_destructor (yymajor=---- _2007-Sep-11 14:58:54 by anonymous:_ {linebreak} Just a shot in the dark - do you have the correct system header files for your version of GLIBC? ---- _2007-Sep-11 15:02:42 by anonymous:_ {linebreak} sqlite 3.5 does not call pthread_getspecific ---- _2007-Sep-11 15:59:51 by anonymous:_ {linebreak} After a little fight, I rollbacked everything on my system to older versions. I'll see whether this problem reappears when Trac 0.11 is released. Meanwhile you can close the ticket, since I cannot provide anymore further info. It must be some sort of configuration issue. Glibc headers might be a good guess. Maybe the latest 3.5.0 was not installed correctly. ---- _2007-Sep-11 16:17:03 by drh:_ {linebreak} Anonymous users can close tickets in CVSTrac. You could have closed this ticket yourself. #c8c8c8 2636 code closed 2007 Sep anonymous 2007 Sep 1 1 Bug in sqlite3_prepare_v2 If we prepare two statements using sqlite3_prepare_v2, lets say DELETE FROM table WHERE ID = ?;{linebreak} and{linebreak} UPDATE table SET ID = ID - ? WHERE ID > ?; at the same time, bind values to it Then execute each of the statements, they get executed, but the changes to the database from the second (UPDATE) statement does not show up in the database. _2007-Sep-11 07:21:12 by danielk1977:_ {linebreak} Can you post a short program to demonstrate the problem? From the report, I'm not sure exactly how to reproduce it. _2007-Sep-12: by drh:_ {linebreak} Unable to reproduce and original submitter is not responding. #c8c8c8 2635 code closed 2007 Sep anonymous 2007 Sep 3 2 problem with calculation of juliand date in the calculation of the function computeYMD (in date.c) the date corresponding of the julian day 77528 is 29 february 300 (which should not exist) instead of 1 march 300, as matter of fact the next day is 2 march 300. This problem is present every 400 years backward. Thanks. _2007-Sep-08 21:46:10 by drh:_ {linebreak} The date and time functions in SQLite render all dates according to the gregorian calendar. This is by design. #f2dcdc 2634 code active 2007 Sep anonymous 2007 Sep 3 3 .schema uses incorrect ORDER BY giving wrong dependency order When the schema is exported, views are sorted by name instead of by dependency. If there are nested views, the schema may be invalid when used to re-create the database. sqlite3 create table t ( f text ); create view v2 as select f from t; create view v1 as select f from v2; .output test.txt .schema .exit sqlite3 .read test.txt SQL error near line 2: no such table: main.v2 _2007-Sep-07 15:33:06 by anonymous:_ {linebreak} Use .dump instead as a workaround. Unlike .schema, .dump does not use ORDER BY in its queries on sqlite_master and it outputs its rows in order of entry., yypminor=0x1) at parse.c:1337 #10 0xb7d1279c in sqlite3Parser (yyp=0xa8fa470, yymajor= , yyminor= {z = 0xa8fec48 "system", dyn = 0, n = 86973048}, pParse=0xb7d39bc8) at parse.c:3268 #11 0xb7d1288f in sqlite3Parser (yyp=0xbfe1734c, yymajor=-1075744936, yyminor= {z = 0xa906428 "�l\212\no", dyn = 0, n = 86973072}, pParse=0xbfe17048) at parse.c:3477 #12 0xb7cf46da in ?? () at ./src/btree.c:2498 from /usr/lib/libsqlite3.so.0 #13 0xbfe1734c in ?? () #14 0xb7d39bc8 in ?? () from /usr/lib/libsqlite3.so.0 #15 0x0a906428 in ?? () #16 0x0a5e3520 in ?? () #17 0xbfe17048 in ?? () #18 0xb7f18300 in ?? () from /lib/ld-linux.so.2 #19 0xb7d16c56 in sqlite3Pragma (pParse=0xbfe1734c, pId1=0xa8fec48, pId2=0x0, pValue=0xb757a700, minusFlag=173792084) at ./src/pragma.c:601
SQLite version 3.5.0 Enter ".help" for instructions sqlite> create table t ( f text ); sqlite> create view v2 as select f from t; sqlite> create view v1 as select f from v2; sqlite> sqlite> .schema CREATE TABLE t ( f text ); CREATE VIEW v1 as select f from v2; CREATE VIEW v2 as select f from t; sqlite> sqlite> .dump BEGIN TRANSACTION; CREATE TABLE t ( f text ); CREATE VIEW v2 as select f from t; CREATE VIEW v1 as select f from v2; COMMIT;Suggested patch:
Index: src/shell.c =================================================================== RCS file: /sqlite/sqlite/src/shell.c,v retrieving revision 1.167 diff -u -3 -p -r1.167 shell.c --- src/shell.c 7 Sep 2007 01:12:32 -0000 1.167 +++ src/shell.c 7 Sep 2007 15:28:24 -0000 @@ -1411,8 +1411,7 @@ static int do_meta_command(char *zLine, "SELECT sql FROM " " (SELECT * FROM sqlite_master UNION ALL" " SELECT * FROM sqlite_temp_master) " - "WHERE tbl_name LIKE shellstatic() AND type!='meta' AND sql NOTNULL " - "ORDER BY substr(type,2,1), name", + "WHERE tbl_name LIKE shellstatic() AND type!='meta' AND sql NOTNULL", callback, &data, &zErrMsg); zShellStatic = 0; } @@ -1421,8 +1420,7 @@ static int do_meta_command(char *zLine, "SELECT sql FROM " " (SELECT * FROM sqlite_master UNION ALL" " SELECT * FROM sqlite_temp_master) " - "WHERE type!='meta' AND sql NOTNULL AND name NOT LIKE 'sqlite_%'" - "ORDER BY substr(type,2,1), name", + "WHERE type!='meta' AND sql NOTNULL AND name NOT LIKE 'sqlite_%'", callback, &data, &zErrMsg ); }after patch:
SQLite version 3.5.0 Enter ".help" for instructions sqlite> create table t ( f text ); sqlite> create view v2 as select f from t; sqlite> create view v1 as select f from v2; sqlite> .schema CREATE TABLE t ( f text ); CREATE VIEW v2 as select f from t; CREATE VIEW v1 as select f from v2; sqlite> .q#cfe8bd 2633 build fixed 2007 Sep anonymous 2007 Sep 1 1 building with gcc3.3.4 - undefined reference to `sqlite3KeywordCode' I'm using gcc 3.3.4 with glibc 2.3.2 trying to cross compile for arm. My configure command line: ../sqlite-3.4.2/configure --disable-readline --disable-tcl --host=arm-linux --prefix=/usr/local/opt/crosstool/arm-linux/gcc-3.3.4-glibc-2.3.2/arm-linux/ During the build I have to compile lemon and mkkeywordhash for my native system and restart the build. Otherwise the build goes smoothly until I get here: ... ranlib .libs/libsqlite3.a creating libsqlite3.la (cd .libs && rm -f libsqlite3.la && ln -s ../libsqlite3.la libsqlite3.la) ./libtool --mode=link gcc -I/usr/local/opt/crosstool/arm-linux/gcc-3.3.4-glibc-2.3.2/arm-linux/include -I. -I../sqlite-3.4.2/src -DNDEBUG -DTHREADSAFE=0 -DSQLITE_THREAD_OVERRIDE_LOCK=-1 -DSQLITE_OMIT_LOAD_EXTENSION=1 -L/usr/local/opt/crosstool/arm-linux/gcc-3.3.4-glibc-2.3.2/arm-linux/lib -DHAVE_READLINE=0 \ -o sqlite3 ../sqlite-3.4.2/src/shell.c libsqlite3.la \ gcc -I/usr/local/opt/crosstool/arm-linux/gcc-3.3.4-glibc-2.3.2/arm-linux/include -I. -I../sqlite-3.4.2/src -DNDEBUG -DTHREADSAFE=0 -DSQLITE_THREAD_OVERRIDE_LOCK=-1 -DSQLITE_OMIT_LOAD_EXTENSION=1 -DHAVE_READLINE=0 -o .libs/sqlite3 ../sqlite-3.4.2/src/shell.c -L/usr/local/opt/crosstool/arm-linux/gcc-3.3.4-glibc-2.3.2/arm-linux/lib ./.libs/libsqlite3.so -Wl,--rpath -Wl,/usr/local/opt/crosstool/arm-linux/gcc-3.3.4-glibc-2.3.2/arm-linux//lib ./.libs/libsqlite3.so: undefined reference to `sqlite3KeywordCode' ./.libs/libsqlite3.so: undefined reference to `keywordCode' collect2: ld returned 1 exit status make: *** [sqlite3] Error 1 I'm trying to build the shared libraries for use on an arm system. I'd just build the sqlite3.c amalgamation into my apps, but I had a grocery list of build errors in gcc 4.1.2 on my native system and I didn't think my gcc3.3.4 cross-compiler would fair much better. So I'd like to at least get the shared libs working. Any help would be appreciated. I figured out that the build does not warn you if keywordhash.h is not created. Perhaps it should or the build will fail. So after compiling the mkkeywordhash tool for the native machine: gcc -g -o mkkeywordhash ../sqlite-3.4.2/tool/mkkeywordhash.c I then had run: ./mkkeywordhash >keywordhash.h ... manually before restarting the build. Build completes without errors now, but build tools/code generators are inherently non-cross-compiler friendly when they are part of the main build process and have to be compiled into machine executable form before being run. Any possibility of moving these to an interpreter-based language that doesn't require compiling? #e8e8bd 2632 code review 2007 Sep anonymous 2008 Jan danielk1977 5 2 sqlite3* not available within loadable extension
sqlite3* db = context->s.db;is the required variable for use with sqlite3_prepare, sqlite3_step and sqlite3_finalize etc. (That's my work-around){linebreak} What is needed only for loadable extension developers to stay aligned with new versions is the re-arrangement of existing struct sqlite3_context {Mem s} and struct Mem {sqlite3 *db} to become able to simply *typecast* sqlite3_context* into a sqlite3*; all the other internals of Mem would stay private to vdbe even when version numbers will grow.{linebreak} And now to answer the question "why the hell is he asking for this?" here my explanation by an example of what I made running:{linebreak}
select execSQL("select 'iniTable' as Title;select * from iniTable", '|', X'0D0A', X'0D0A'||'----'||X'0D0A' ); iniTable ---- Common|Integer|4711 Common|Float|3.1416927 Common|Double|3.1416927 Common|Text|Dies ist Text Common|Blob|X'0D0A'Think about an application which has to create a whole InMemory-Database table structure inclusive triggers and views:
select execSQL(stmt) from StatementTable where Usage='InMemory';And all within one and the same transaction. It is really that easy! This could answer also some points in the wishlist of the SQLite3 web site; all to do this is reordering Mem s in sqlite3_context and sqlite3*db in struct Mem to the beginning of the structures.{linebreak}Doing this with the release of version 3.5 is really a good moment, isn't it? _2007-Sep-07 11:54:38 by danielk1977:_ {linebreak} The usual approach is to pass the sqlite3* handle to sqlite3_create_function() as the 5th argument and then retrieve it from within the user function implementation using sqlite3_user_data(). If you depend on the definitions of internal data structures, you're living dangerously. ;) ---- _2007-Sep-07 12:54:56 by anonymous:_ {linebreak} Is then the following correct and will work in the same expected way (same pointer within ld-ex-function)? Even with sqlite_auto_extension called at DLL load time? (Win32 with Amalgamation and BDS2006){linebreak}
#includeThank's for the smilie and your hint. Of cause, I do not want to live dangerous. This was the purpose of my posting to get this fixed officially with the next release. But if you say it is not worth the "small change" of reordering and I'm doing it correct this way, then that's fine. I started similar to your proposal by using a static variable and assigned it from within sqlite3_extension_init but this would not work with more than one db-connection (the last one would win). The other open point for me is the right place and methode to link the same source code as used for the Loadable Extension Functions to become a DLL (for SQlite3.exe .load) as auto-loaded extensions into a specialized made of SQLite3.DLL for my applications. Do you have another trick than modifying the sqlite3_open-functions with compiler switches? This would be great. Even if above statements work, do you think the reordering makes sense for an easier transfer for the sqlite3* into the loadable extension function? The 5th argument would be kept free for other purposes. What is your estimation of likelyhood for that proposed change? What's against it? ---- _2007-Sep-07 14:11:38 by danielk1977:_ {linebreak} The code above is correct as far as I can see. As far as reordering structures to allow tricky casting, there's no particular argument against, other than it makes the code a bit tricky to follow. And limits the ways in which we can change the contents of the currently opaque sqlite3_context structure in the future. Something like: sqlite3 *sqlite3_get_connection(sqlite3_context *p){ return p->s.db; } might be a chance I guess. ---- _2007-Sep-10 18:58:09 by anonymous:_ {linebreak} That is a very good idea. Now I have a case, where I would like to make use of both, the db connection AND the user data:{linebreak} I have two database selections (by a list of keys) and I would like to used one function to compare "equal" and another "unequal". In this case I could use the same function body and use the userdata to distinguish between the cases equal and unequal while using the db connection to query the selection with the given keys.{linebreak} Using 2 functions which are sharing the same body of a 3rd function will be my workaround until this new API function becomes available with the next 3.5.x version. Many thanks. ---- _2007-Sep-10 19:09:51 by anonymous:_ {linebreak} I agree. It is very reasonable to want to use an sqlite3* connection from within a registered function as well. Scenario - perform an SQL query on same connection from within a user-defined function. Setting user data just for this common action is awkward. ---- _2008-Jan-08 14:46:14 by anonymous:_ {linebreak} when will this function become usable? will it come with 3.5.5 ? {linebreak} sqlite3 *sqlite3_get_connection(sqlite3_context *p) #cfe8bd 2631 code fixed 2007 Sep anonymous 2007 Sep 3 3 Comments at end of "CREATE TABLE" cause SQL error. Comments after the semi colon of a "CREATE TABLE" statement that use the double-dash format cause an Incomplete SQL error in the Windows command line program when read in with the ".read" command. The following SQL statement read in from a file through the ".read" command will generate the error: CREATE TABLE test (abc, def); -- this comment causes error With the comment removed the read works correctly. The error doesn't happen if the statement is typed into the console. #c8c8c8 2630 code closed 2007 Sep anonymous 2007 Oct a.rottmann 2 2 sqlite3PagerOpen set a wrong temp file name in pPager->zFilename in pager.c line 1783{linebreak} 1783 rc = sqlite3PagerOpentemp(&fd);{linebreak} 1784 sqlite3OsTempFileName(zTemp);{linebreak} 1785 zFilename = zTemp;{linebreak} 1786 zFullPathname = sqlite3OsFullPathname(zFilename);{linebreak} 1787 if( rc==SQLITE_OK ){{linebreak} 1788 tempFile = 1;{linebreak} .{linebreak} .{linebreak} .{linebreak} 1823 memcpy(pPager->zFilename, zFullPathname, nameLen+1);{linebreak} function sqlite3PagerOpentemp on line 1783 open the fd using e new temp file name{linebreak} but the the function on 1784 get another temp file name and {linebreak}assign it to pPager->zFilename on line 1823 I compile sqlite on POCKETPC and temp file were not delete automatically by the OS{linebreak} so I have uncommented the code that delete temp file in function sqlite3PagerClose{linebreak} /* Temp files are automatically deleted by the OS*/{linebreak} if( pPager->tempFile ){{linebreak} sqlite3OsDelete(pPager->zFilename);{linebreak} }{linebreak} but the sqlite3OsDelete(pPager->zFilename); use the wrong temp file name. I have change the previous code with this one. 1783 // rc = sqlite3PagerOpentemp(&fd);{linebreak} 1784 sqlite3OsTempFileName(zTemp);{linebreak} 1785 zFilename = zTemp;{linebreak} 1786 zFullPathname = sqlite3OsFullPathname(zFilename);{linebreak} {linebreak} 1787 do{{linebreak} 1788 cnt--;{linebreak} 1789 rc = sqlite3OsOpenExclusive(zFullPathname, &fd, 1);{linebreak} 1790 assert( rc!=SQLITE_OK || fd );{linebreak} 1791 }while( cnt>0 && rc!=SQLITE_OK && rc!=SQLITE_NOMEM );{linebreak} 1792{linebreak} 1793 if( rc==SQLITE_OK ){{linebreak} 1794 tempFile = 1;{linebreak} 1795 }{linebreak} and all is fine.{linebreak} I hope this is useful.{linebreak} thanks.{linebreak} by.{linebreak} _2007-Sep-06 23:25:20 by drh:_ {linebreak} The code in question was restructured and rewritten for SQLite version 3.5.0. But looking into this did bring to my attention a bug in the implementation of 3.5.0, for which I am grateful. #f2dcdc 2629 doc active 2007 Sep anonymous 2007 Sep 4 4 typo in os_unix.c for nolockSQLITE_EXTENSION_INIT1 int sqlite3_extension_init( sqlite3 *db, char **pzErrMsg, const sqlite3_api_routines *pApi ) { SQLITE_EXTENSION_INIT2(pApi) sqlite3_create_function(db, "execSQL", -1, SQLITE_UTF8, db, execSqlFunc, 0, 0); ... } static void execSqlFunc( sqlite3_context *context, int argc, sqlite3_value **argv ) { sqlite3 *db = sqlite3_user_data(context ); /* instead of using context's internals ( sqlite3 *db = context->s.db ); or waiting for reordering of them and using typecase ( sqlite3 *db = (sqlite3*)context; ) respective ( sqlite3 *db = *(sqlite3**)context; )? */ ... }
Index: src/os_unix.c =================================================================== RCS file: /sqlite/sqlite/src/os_unix.c,v retrieving revision 1.165 diff -u -3 -p -r1.165 os_unix.c --- src/os_unix.c 5 Sep 2007 13:56:32 -0000 1.165 +++ src/os_unix.c 6 Sep 2007 17:53:47 -0000 @@ -2126,7 +2126,7 @@ static const sqlite3_io_methods sqlite3D /* ** This vector defines all the methods that can operate on an sqlite3_file -** for unix with dotlock style file locking. +** for unix with nolock style file locking. */ static const sqlite3_io_methods sqlite3NolockLockingUnixIoMethod = { 1, /* iVersion */#cfe8bd 2628 code fixed 2007 Sep anonymous 2007 Sep 3 4 Bad error handling for invalid page-size When opening a database which has an invalid page-size, sqlite3BtreeOpen() decides to use default values. This prevents the assert in sqlite3PagerSetPagesize() (which would at least show the cause of the problem) and subsequently brings obscure errors. It would be preferable to return a database error when this is detected. I had foolishly reduced the value of SQLITE_MAX_PAGE_SIZE for use in embedded devices, and noticed this when trying to open a database created with a larger page size. #f2dcdc 2627 code active 2007 Sep anonymous 2007 Sep 3 2 Improper parsing of nested JOIN SQLite has a problem with multiple nested JOINs. The only way to get it workig is to remove the surrounding brackets. Removing the brackets unfortunately do not work in other DB systems such as MS SQL, mysql etc. This does not work: Select ContactPhone.* From (ContactPhone LEFT OUTER JOIN ContactLocation ON ContactPhone.PHNLCT_ID = ContactLocation.LCT_ID) LEFT OUTER JOIN ContactItem ON ContactLocation.LCTITM_ID = ContactItem.ITM_ID (It complains about LCT_ID or similar) This works after removing the brackets: Select ContactPhone.* From ContactPhone LEFT OUTER JOIN ContactLocation ON ContactPhone.PHNLCT_ID = ContactLocation.LCT_ID LEFT OUTER JOIN ContactItem ON ContactLocation.LCTITM_ID = ContactItem.ITM_ID All other major DB systems require the surrounding brackets. Do you think it is possible to fix it? Apart from this little little SQLite is an awesome project. Thank you Jakub Klos _2007-Sep-06 13:07:32 by anonymous:_ {linebreak} I don't have access to MS SQL Server, but MySQL and Oracle have no issue with the query without parentheses:
create table x1(a int, b int); create table x2(c int, d int); create table x3(e int, f int); mysql> select x1.* from x1 left join x2 on x1.a=x2.c left join x3 on x2.d=x3.e; Empty set (0.00 sec)---- _2007-Sep-06 19:03:33 by anonymous:_ {linebreak} MSSQL also has no problems without the parens. As a matter of fact, the only DB that I know of that requires them is MS Access (JET). ---- _2007-Sep-06 20:11:31 by anonymous:_ {linebreak} I guess he had no luck filing a JET bug. ---- _2007-Sep-11 17:22:28 by anonymous:_ {linebreak} True, MS access requires the parens but all other major DBs support the query syntax with the parens. So why SQLite does not like it? It should simply ignore them if possible. Thank you #c8c8c8 2626 code closed 2007 Sep anonymous 2007 Sep 1 1 The Amalgamation compile error when i compile The Amalgamation 3.4.2 with visual studio 2003 in window, i encounted 3 errors : 'sqlite3IsIdChar':unknown size 'sqlite3OpcodeNames':unkown size 'sqlite3UpperToLower':unkown size i think it is caused by following source code in sqlite3.c: #ifndef SQLITE_PRIVATE # define SQLITE_PRIVATE static #endif it should be: #ifndef SQLITE_PRIVATE # define SQLITE_PRIVATE #endif is that right? It looks like vs2003 doesn't like this kind of forward decl: static int aArray[]; ... static int aArray[] = {1, 2, 3}; But is fine with this: int aArray[]; ... int aArray[] = {1, 2, 3}; By defining SQLITE_PRIVATE as an empty string, you are avoiding constructs like the first block above. It's not ideal, but not harmful either. ---- _2007-Sep-06 13:11:51 by anonymous:_ {linebreak} This is fixed in 3.5. #c8c8c8 2625 build closed 2007 Sep anonymous 2007 Sep 2 2 http://www.sqlite.org/cvstrac/wiki?p=HowToCompileWithVsNet has problem when i compile 3.4.2 with visual studio 2003 in windows, i do as http://www.sqlite.org/cvstrac/wiki?p=HowToCompileWithVsNet told me to. but step 8 "Add all the .c and .h files that you unzipped, except for: tclsqlite.c and shell.c.Note: You may add tclsqlite.c and shell.c, but then you have to define the preprocessor-symbol NO_TCL:" won't work. i suggest to modify it to be "Add all the .c and .h files that you unzipped, except for: tclsqlite.c shell.c and icu.c. Note: You may add tclsqlite.c and shell.c, but then you have to define the preprocessor-symbol NO_TCL, you may add icu.c, but then you have to define the preprocessor-symbol SQLITE_CORE". why not supply a .sln file? this will save many people's time. thank you. _2007-Sep-06 07:04:34 by danielk1977:_ {linebreak} What you suggest sounds like an improvement. I'd add the text in myself, but I don't have visual studio, so I can't test it. But the wiki pages are all user editable, so feel free to modify it yourself by clicking the "Edit" link found at the top right-hand side of the page. If you have a *.sln file that you can attach to the page, even better. #cfe8bd 2624 code fixed 2007 Sep anonymous 2007 Sep 1 1 Invalid documentation references in sqlite.h.in Documentation in sqlite.h.in contains invalid references to the following functions and #defines: *: SQLITE_LOCK_READ -> _SHARED *: sqlite3_register_vfs -> vfs_register *: sqlite3_unregister_vfs -> vfs_unregister *: sqlite3_find_vfs -> vfs_find *: sqltie3_blob_size -> sqltie3_blob_bytes *: sqlite3_finalise -> sqlite3_finalize ( s vs. z) *: sqlite_column_text -> missing 3 *: sqltie3_blob_size -> ti mixed up #cfe8bd 2623 code fixed 2007 Sep anonymous 2007 Sep 1 1 zeroblob(-1) returns non-NULL from sqlite3_column_blob() Please consider this psydo-code: sqlite3_prepare ("select zeroblob(-1)", DB, &Stmt); sqlite3_step(Stmt); sqlite3_column_blob(Stmt, 0) == NULL; // Error here! sqlite3_column_blob() used to return NULL for "zeroblob(-1)" as well as any other negative length value for SQLite 3.4.2. After the 3.5.0 changes it returns an arbitrary non-NULL value. sqlite3_column_bytes() returns 0 both before and after the changes. I suppose that the pre 3.5.0 NULL result is correct and meaningful. Or is it in fact undefined? #cfe8bd 2622 doc fixed 2007 Sep anonymous 2007 Sep 3 3 SQLITE_OMIT_MEMORY_ALLOCATION undocumented unresolved functions When SQLITE_OMIT_MEMORY_ALLOCATION is defined, in addition to the functions sqlite3_malloc(), sqlite3_realloc(), and sqlite3_free(), two other functions are also unresolved: sqlite3_memory_alarm() and sqlite3_memory_used(). $ make sqlite3 2>&1 | grep 'undefined reference to' | sed 's/^.*://g' | sort | uniq undefined reference to `sqlite3_free' undefined reference to `sqlite3_malloc' undefined reference to `sqlite3_memory_alarm' undefined reference to `sqlite3_memory_used' undefined reference to `sqlite3_realloc' They are used in sqlite3_soft_heap_limit() in malloc.c. So this issue could either be resolved with a doc fix to the statement "If SQLite is compiled with SQLITE_OMIT_MEMORY_ALLOCATION then no implementation for the sqlite3_malloc(), sqlite3_realloc(), and sqlite3_free() functions is provided." or by using an appropriate #ifdef around sqlite3_memory_used and sqlite3_memory_alarm in sqlite3_soft_heap_limit(). i.e.:
Index: src/malloc.c =================================================================== RCS file: /sqlite/sqlite/src/malloc.c,v retrieving revision 1.13 diff -u -3 -p -r1.13 malloc.c --- src/malloc.c 29 Aug 2007 14:06:23 -0000 1.13 +++ src/malloc.c 4 Sep 2007 04:40:03 -0000 @@ -36,6 +36,7 @@ static void softHeapLimitEnforcer( ** zero or negative value indicates no limit. */ void sqlite3_soft_heap_limit(int n){ +#ifndef SQLITE_OMIT_MEMORY_ALLOCATION sqlite3_uint64 iLimit; int overage; if( n<0 ){ @@ -52,6 +53,7 @@ void sqlite3_soft_heap_limit(int n){ if( overage>0 ){ sqlite3_release_memory(overage); } +#endif } /*I prefer the #ifdef over the doc change, but either is fine. _2007-Sep-04 04:55:41 by anonymous:_ {linebreak} It appears that the next paragraph mentions that sqlite3_memory_alarm must also be provided by the user if SQLITE_OMIT_MEMORY_ALLOCATION is defined. So sqlite3_memory_used is the only one not documented as being required to be supplied by the user. ---- _2007-Sep-04 05:35:13 by anonymous:_ {linebreak} loadext.c also has references to sqlite3_memory_highwater and sqlite3_memory_used when SQLITE_OMIT_MEMORY_ALLOCATION is defined. #cfe8bd 2621 code fixed 2007 Sep anonymous 2007 Sep 1 5 another typo -Application that implement their own OS interface will require modification. +Any application implementing their own OS interface will require modification. #c8c8c8 2620 doc closed 2007 Sep anonymous 2007 Sep 5 5 doc note needed for sqlite3_memory_alarm() In the docs for sqlite3_memory_alarm() it is probably worth noting that the registered callback xCallback may be invoked from any thread where sqlite3_malloc may be invoked and the user must take appropriate measures in writing their callback. _2007-Sep-04 03:32:19 by drh:_ {linebreak} The docs already say:
"Application programs should not attempt to use sqlite3_memory_alarm().... This interface is exposed only so that applications can provide their own alternative implementation...."Isn't that enough? ---- _2007-Sep-04 04:03:54 by anonymous:_ {linebreak} Connection locks may be held. Things could potentially deadlock if you don't know what you're doing. I thought an explicit mention would make this clear, but perhaps you're right. Feel free to close this ticket. #cfe8bd 2619 doc fixed 2007 Sep anonymous 2007 Sep 5 5 typo -This is a large change approximately 1 line of count of out 10 was modified. +This is a large change. Approximately 10 percent of the lines of code was modified. #c8c8c8 2618 code closed 2007 Sep anonymous 2007 Sep linus 4 1 Dabase is Locked I have copied my executables and binaries to my target with Sqlite Database. But When i try to access the Database I get, Error:Database is Locked and memory fault occurs. _2007-Sep-04 03:13:48 by drh:_ {linebreak} SQLite version 2.8.0 dates back to February of 2003. We might consider doing a patch on a branch of version 2.8.17 if someone happens to find a very serious bug and explains the bug clearly. This ticket does neither. Please take further discussion to the mailing list. Tnx. #c8c8c8 2617 code closed 2007 Sep anonymous 2007 Sep drh 2 3 Check in #4357 may lead to bugs in SMP aware systems in windows vfs Hi The method for generating temporary file names in windows vfs driver may cause problems when running on SMP systems, because they could 'collide' on sqliteRandomness() call. A safer and proper fix for this is to use GetTempFileName () windows API (http://msdn2.microsoft.com/en-us/library/aa364991.aspx) and set flags to generate a unique file name. It ensures no-other call could get the same file name. _2007-Sep-03 21:28:15 by drh:_ {linebreak} The chances of a name collision are vanishingly small, so small that the problem can be ignored. ---- _2007-Sep-05 14:49:46 by anonymous:_ {linebreak} Yes. You are correct on this, could never happen, but still exists. Also, after examined the Randomness code for Win32 VFS driver, I found this:
static int winRandomness(sqlite3_vfs *pVfs, int nBuf, char *zBuf){ int n = 0; if( sizeof(LPSYSTEMTIME)<=nBuf-n ){ SYSTEMTIME x; GetSystemTime(&x); memcpy(&zBuf[n], &x, sizeof(x)); n += sizeof(x); } if( sizeof(DWORD)<=nBuf-n ){ DWORD pid = GetCurrentProcessId(); memcpy(&zBuf[n], &pid, sizeof(pid)); n += sizeof(pid); } if( sizeof(DWORD)<=nBuf-n ){ DWORD cnt = GetTickCount(); memcpy(&zBuf[n], &cnt, sizeof(cnt)); n += sizeof(cnt); } if( sizeof(LARGE_INTEGER)<=nBuf-n ){ LARGE_INTEGER i; QueryPerformanceCounter(&i); memcpy(&zBuf[n], &i, sizeof(i)); n += sizeof(i); } return n; }The code could run into a buffer underrun because you check for sizeof(LPSYSTEMTIME) (which is equal to sizeof(void *), because LPSYSTEMTIME is typedef'ed as SYSTEMTIME *, so a underrun could happen here. Also, LPSYSTEMTIME suffer from timer granularity of the kernel, which could be, in some systems, up to 130ms. A better implementation for this routine could be:
static int winRandomness(sqlite3_vfs *pVfs, int nBuf, char *zBuf){ int n = 0; if( sizeof(DWORD)<=nBuf-n ){ DWORD cnt = GetTickCount(); // Ticks from system startup memcpy(&zBuf[n], &cnt, sizeof(cnt)); n += sizeof(cnt); } if( sizeof(LARGE_INTEGER)<=nBuf-n ){ LARGE_INTEGER i; QueryPerformanceCounter(&i); // High-resolution TSC counter memcpy(&zBuf[n], &i, sizeof(i)); n += sizeof(i); } if( sizeof(DWORD)<=nBuf-n ){ DWORD pid = GetCurrentThreadId(); // Current thread id memcpy(&zBuf[n], &pid, sizeof(pid)); n += sizeof(pid); } if( sizeof(DWORD)<=nBuf-n ){ DWORD pid = GetCurrentProcessId(); // Current process id memcpy(&zBuf[n], &pid, sizeof(pid)); n += sizeof(pid); } if( sizeof(SYSTEMTIME)<=nBuf-n ){ SYSTEMTIME x; GetSystemTime(&x); // System date/time memcpy(&zBuf[n], &x, sizeof(x)); n += sizeof(x); } return n; }This code will fill, with less chance of collision, the 20 byte randomness key used to generate the temporary file name, and also will generate more efficiently random numbers, because it includes the thread id on it, which will help in SMP systems, because they cannot get to be the same at same time. There is a chance this could be fixed (SYSTEMTIME issue) and include in pre-3.5.0 code ? #cfe8bd 2616 code fixed 2007 Sep anonymous 2007 Sep 1 1 Flags parameter not honoured in openDatabase() The flags parameter of openDatabase() is not honored. At least I did not find any further reference to it in the function body. As a result, calling sqlite3_open_v2 with the SQLITE_OPEN_READWRITE flag creates a new database file even though, as advertised in the documentation, it should not. #cfe8bd 2615 code fixed 2007 Sep anonymous 2007 Sep 1 1 winRandomness without effect due to < vs. > mixup os_win.c line 1424 checks that if( sizeof(LPSYSTEMTIME)>=nBuf ), supposedly to make sure that the supplied buffer is never overwritten. However, the comparison should be <= instead. The current implementation prevents that the buffer is filled at all because the check always fails. #c8c8c8 2614 code closed 2007 Sep anonymous 2007 Sep 1 1 Custom VFS still calls default xRandomness If a default VFS is registered, any custom VFS's xRandomness callback is never invoked. The only reference to the xRandomness call is hard coded in random.c, line 65: sqlite3OsRandomness(sqlite3_vfs_find(0), 256, k); This makes it impossible to implement custom random seeds for any VFS other than the default. Correct. This fact has now been made explicit in the documentation. #f2dcdc 2613 code active 2007 Sep anonymous 2008 Jan drh 3 3 replace doesn't work with blobs containing \x0, otherwise it does The replace expression function does not work with blobs in case of contained zero terminator character; but it does if there is not this special character included. I expected the function to work similar like substr with blob-safety in case of type is blob only. X'nnnn' is of type blob, so following example should have returned a blob type result X'0102FF0405' in the 2nd and 3rd line. How to get to this result?
SQLite version 3.4.2 Enter ".help" for instructions sqlite> select hex(replace(X'0102030405',X'03',X'FF')); 0102FF0405 sqlite> select hex(replace(X'0102000405',X'00',X'FF')); sqlite> select typeof(replace(X'0102000405',X'00',X'FF')); null sqlite>_2007-Sep-03 04:21:12 by anonymous:_ {linebreak} Replace was designed to work with strings. However, working with blobs would be an interesting extension. ---- _2007-Oct-18 06:13:10 by anonymous:_ {linebreak} I've seen a similar situation where I can't reliably store stings with nulls in the middle of them as TEXT. I can convert them to blobs, in which case length(...) works correctly. I if convert them back to strings, length(...) treats them as C-strings. Is this the expected behavior? I notice the entire column is preserved even when it's has TEXT affinity, I can append data to it as a string, cast back to a blob and see everything (am I explaining this poorly?) This all seems a bit counter intuitive in some ways. Perhaps strings shouldn't treat NULL characters as special? ---- _2007-Oct-27 16:45:41 by anonymous:_ {linebreak} Treatment of length operator is - as fas as I know - dependent on type: {linebreak} As text it is the length number of UTF-8 characters and as blob it is the number of bytes. As long as all the UTF-8 characters out of the lower half ASCII char-set (127 of them), this is identical beside the fact of different 0-terminator interpretation. {linebreak} To append is something different than using the replace operator. My suggestion would be to make the replace operator work with bytes (not UTF-8) in case of all 3 parameters are of type blob. {linebreak} Another suggestion: the UTF aware functions are Private declared and not usable from within a loadable extension dll/so. This should be changed. ---- _2008-Jan-28 19:36:39 by anonymous:_ {linebreak} Will there come a solution for this with the next release? It is really not fair to handle a blob only like text which cannot contain a zero terminator. With this unique useful function a zero-containing blob could be formed into a normal text string without loosing the part behind the zero terminator. It would be really a step forward without too much effort. #cfe8bd 2612 code fixed 2007 Sep anonymous 2007 Sep 1 1 enterMutex() misnamer enterMutex() in mem1.c is wrongly named as enterMem(). This results in linker errors due to missing enterMutex() function. #cfe8bd 2611 code fixed 2007 Sep anonymous 2007 Sep 2 1 AV calling sqlite3_vfs_register() Calling sqlite3_vfs_register() as the first thing in an application raises an AV. Rationale: vfsList in os.c is not initialized until the first call of sqlite3_vfs_find(). Thus calling sqlite3_vfs_find() before sqlite3_vfs_register() functions as a workaround, but can be easily forgotten. #cfe8bd 2610 code fixed 2007 Sep anonymous 2007 Sep drh 4 3 SQLITE_API sqlite3_sleep function header missing in sqlite3ext.h struct sqlite3_api_routines { does not contain sqlite3_sleep }
#includeFor those who want to know: The execSQL function is foreseen to process a list of SQL statements out of a table which keeps the stored SQL statements; Table and Trigger creations and deletions, loading of further extensions, etc. This issue has already been addressed by check-in [4343]. #c8c8c8 2609 code closed 2007 Sep anonymous 2007 Sep 1 1 Is non-thread safe binaries dangerous for multi-threaded app design ? Can accessing two threads ( one is the main app one ) to one connection silently corrupt in-memory data so that next commit to write the wrong ones to the disk, for the non-threadsafe binaries ? _2007-Sep-01 20:45:04 by anonymous:_ {linebreak} Forgot to mention: only the main app thread writes ( calls COMMIT), the other thread only scrolls & reads the data. ---- _2007-Sep-01 20:57:07 by anonymous:_ {linebreak} You have to compile sqlite with -DTHREADSAFE=1 if you are to use it in multiple threads in the same program. This question really belongs on the mailing list, not the bug ticket system. ---- _2007-Sep-01 22:13:40 by anonymous:_ {linebreak} Then this issue should be noted on the top of official site, and there should also be thread-safe binaries in the download area. #cfe8bd 2608 code fixed 2007 Aug anonymous 2007 Sep 3 3 failed allocation leads to unexpected infinite loop I tried to build 3.4.2 on Windows with something else than mingw and I stumbled upon this. The malloc.test hangs in malloc-1.733 so I attached to testfixture.exe with the VC8 debugger and found the following: The function sqlite3WinTempFileName contains an infinite loop where random file names are generated and the loop breaks when one is found which does not exist. That is where it hangs. sqlite3WinTempFileName calls in the infinite loop ( sqlite3WinFileExists, which calls ( convertUtf8Filename, which calls ( utf8ToUnicode, which calls ( sqlite3MallocRaw, which fails, and returns 0 ) and utf8ToUnicode returns 0 ) and convertUtf8Filename returns 0 ) and sqlite3WinFileExists returns SQLITE_NOMEM=7 ) and the test in the loop thinks it has received TRUE and goes on. Essentially this means an existence test seems to return a TRUE answer when the test actually has not happened at all for an obscure technical reason. This can stand improvement. This was probably missed because that allocation should never fail. It is probably failing for reasons to do with my build, which is another matter. _2007-Sep-01 02:16:48 by drh:_ {linebreak} This fix is simply to not check to see if the randomly generated filename already exists. I extended the random section of the filename to be 20 characters long. with 62 possible values at each position, this gives 119 bits of randomness. The chance of such an name colliding is many orders of magnitude smaller than the chance of a random malfunction causing the win32 APIs to return the wrong answer when we ask whether or not the file exists. The flaw in the above argument is that the PRNG in SQLite is poorly seeded on windows. We need to do a better job seeding the PRNG on that platform. It's the winRandomness() function in the os_win.c source file that seeds the PRNG, if any windows hackers would like to suggest a way to improve the seeding. #f2dcdc 2607 event active 2007 Aug anonymous 2007 Sep 2 1 Data loss, continuation to Re: [sqlite] how to flush database to disk? See that mailing list. The originator message : ======================== I've just lost a couple of days' worth of data when my app crashed. (Well, the data wasn't a total loss thanks to backup plans, but the database itself essentially reverted to its state of 2 days ago.) This is despite my app doing a COMMIT after every modification of the DB. It's acting as though all the changes were held in memory, or somehow journaled, and when the crash happened, the changes were all lost or rolled back. What I need is a way to force the database to save its data to disk while my app is running, so that in the event of a crash, I lose little or no data. How can I do this? I presume that closing the database would do the trick, but is there a less heavy-handed way? =========== The exact like that data loss occured at me too. Three times, non-repropucible regualrilly. What common in these losses ? {Editing+committing} in the main thread then {navigating, bof/eof checking, reading data} from within different threads then return to the main thread for {editing+committing}. _2007-Aug-31 19:38:26 by drh:_ {linebreak} The COMMIT does not actually occur until you call sqlite3_reset() and/or sqlite3_finalize() on all your prepared statements. Any prepared statement that has not been reset or finalized is still running, is incomplete, and is thus still holding the transaction open. I'm guessing that you have an unreset and unfinialized statement in your application. I wonder what would happen if we changed the definition of COMMIT so that it returned an error if there were active prepared statements. This is, technically, an incompatibility. But we are coming up on a release with several other minor incompatibilities, so now might be a good time to insert such a change. ---- _2007-Aug-31 19:45:36 by drh:_ {linebreak} I looked in the code, and it turns out we already do this. Perhaps the application is not checking the return code from the COMMIT to see that it is failing? ---- _2007-Aug-31 20:40:47 by anonymous:_ {linebreak} No error reports came from COMMIT. The data loss were noticed after UPDATE & COMMIT after massive reading of results of SQL addressing the same virtual (ATTACHed) tables, from within another thread. ---- _2007-Aug-31 20:53:48 by anonymous:_ {linebreak} Is it possible that a pending transaction survive app shutdown & then OS restart ? Is yes, then any DB error would cause rollback to the data on last BEGIN, isn't ? ---- _2007-Aug-31 21:00:18 by anonymous:_ {linebreak} But me committed each smallest change to dat,a then saw these refreshed data in the tables. And on the next day these data were present. Only reading (with full scrolling ) the affected virtual tables from within another thread then new editing then committing caused the loss. ---- _2007-Sep-01 23:12:31 by anonymous:_ {linebreak} You mention virtual tables. Their data is not maintained by the SQLite engine, but by your own module. If that doesn't implement ACID, you're out of luck. By definition:{linebreak} {quote: A virtual table is an interface to an external storage or computation engine that appears to be a table but does not actually store information in the database file.} references:{linebreak}{link: http://www.sqlite.org/lang_createvtab.html CreateVirtualTable}{linebreak} {wiki: VirtualTables VirtualTables} ---- _2007-Sep-04 04:51:15 by anonymous:_ {linebreak} Me was wrong. These were't true virtual tables. Me used a LEFT OUTER query to several tables residing in different ATTACHed databases. ---- _2007-Sep-04 04:52:01 by anonymous:_ {linebreak} were't => were not, above. #cfe8bd 2606 code fixed 2007 Aug anonymous 2007 Sep 3 3 PATCH - default configure option enable_threadsafe=yes Obvious patch below to enable thread-safe sqlite3 builds by default to match the default thread-safe build of the sqlite3.c amalgamation. Patch included for both configure.ac and configure so you don't have to regenerate the script. I tested it with no options, enable and disable and it works.SQLITE_EXTENSION_INIT1 static void execSQL(sqlite3_context *context, int argc, sqlite3_value **argv) { ... len = strlen(sql); errno = sqlite3_prepare(db, sql, len, &stmt, &t); while( stmt ) { while( (s=sqlite3_step(stmt))==SQLITE_BUSY ) sqlite3_sleep( 100 ); if( s==SQLITE_ROW ) { switch( sqlite3_column_type(stmt, 0 ) ) ... } else { sqlite3_finalize(stmt); stmt=NULL; } } ... int sqlite3_extension_init( sqlite3 *db, char **pzErrMsg, const sqlite3_api_routines *pApi ) { SQLITE_EXTENSION_INIT2(pApi) sqlite3_create_function(db, "execSQL", 1, SQLITE_UTF8, 0, execSQL, 0, 0);
Index: configure =================================================================== RCS file: /sqlite/sqlite/configure,v retrieving revision 1.42 diff -u -3 -p -r1.42 configure --- configure 17 Feb 2007 14:59:18 -0000 1.42 +++ configure 31 Aug 2007 15:43:23 -0000 @@ -18883,7 +18883,7 @@ fi if test "${enable_threadsafe+set}" = set; then enableval=$enable_threadsafe; else - enable_threadsafe=no + enable_threadsafe=yes fi { echo "$as_me:$LINENO: checking whether to support threadsafe operation" >&5 Index: configure.ac =================================================================== RCS file: /sqlite/sqlite/configure.ac,v retrieving revision 1.29 diff -u -3 -p -r1.29 configure.ac --- configure.ac 17 Feb 2007 14:59:18 -0000 1.29 +++ configure.ac 31 Aug 2007 15:43:23 -0000 @@ -178,7 +178,7 @@ AC_SUBST(BUILD_CFLAGS) # Do we want to support multithreaded use of sqlite # AC_ARG_ENABLE(threadsafe, -AC_HELP_STRING([--enable-threadsafe],[Support threadsafe operation]),,enable_threadsafe=no) +AC_HELP_STRING([--enable-threadsafe],[Support threadsafe operation]),,enable_threadsafe=yes) AC_MSG_CHECKING([whether to support threadsafe operation]) if test "$enable_threadsafe" = "no"; then THREADSAFE=0_2007-Aug-31 16:23:18 by drh:_ {linebreak} Thanks for the patch. Here's the thing: Threadsafe builds are currently running about 8% slower than single-threaded builds. (All those calls to pthread_mutex_lock() do not come for free.) So we are going to have to make a decision about whether or not we really want the default build to be threadsafe. ---- _2007-Aug-31 16:29:45 by anonymous:_ {linebreak} The trend these days is multi-core chips and multi-threaded software. Embedded programmers know their way around a compiler and can easily disable thread-safety if they should so choose to get extra speed. Beginners (and apparently one old timer) may forget to configure a threadsafe build which may produce programs that seem to work initially, but fail unexpectedly in the field. Better safe than sorry, I figure. #c8c8c8 2605 build closed 2007 Aug anonymous 2007 Aug 1 3 Amalgamated version 3.4.2 doesn't compile (VC++6) In a project that already used versions 3.3.x through 3.3.17 (successfully) in "amalgamated" source, I substituted sqlite3.c and sqlite3.h from version 3.4.2, but I yield three compilation errors: ----------------------------------------------------------------- Compiling... {linebreak} sqlite3.c {linebreak} c:\lavoro\nonclassificato\resanddev\hcsprog\sqlite\sqlite3.c(6187) : error C2133: 'sqlite3UpperToLower' : unknown size {linebreak} c:\lavoro\nonclassificato\resanddev\hcsprog\sqlite\sqlite3.c(9310) : error C2133: 'sqlite3OpcodeNames' : unknown size {linebreak} c:\lavoro\nonclassificato\resanddev\hcsprog\sqlite\sqlite3.c(47159) : error C2133: 'sqlite3IsIdChar' : unknown size {linebreak} Error executing cl.exe. {linebreak} ----------------------------------------------------------------- They all are related to array declaration missing size in brackets. Duplicate of ticket #2574. Already fixed. #f2dcdc 2604 new active 2007 Aug anonymous 2007 Aug 4 4 CREATE VIRTUAL TABLE does not allow IF NOT EXISTS CREATE VIRTUAL TABLE vt IF NOT EXISTS; would help with development since creating a virtual table that exists returns error 1 - as do several "Real" errors. #c8c8c8 2603 new closed 2007 Aug anonymous 2007 Oct 1 3 B-tree and B+tree allow bidirectional cursor but VDBE does not. B-tree and B+tree allow bidirectional cursor but VDBE does not. We would very much like to have this capability and our fallback is to work with the btree api directly. the closest opcodes are: rewind p1 p2 p3 p1 is p2 is line number if rowcount=p1 and step What we would like to see is something like rstep p1 p1= signed integer for direction of relative motion repos p1 p2 p1=unsigned integer representing low-order 32 bits of 64 bit integer p2=unsigned integer representing hi-order 32 bits of 64 bit integer If this is doable in your opinion then we would be willing to look at implementing it and contributing it to the project. _2007-Oct-12 18:58:46 by drh:_ {linebreak} I'm not really sure what the description above is saying, but my best guess is that it is a request for logic to do a ScrollingCursor. Please see the wiki page for how to accomplish ScrollingCursor in SQLite. #cfe8bd 2602 code fixed 2007 Aug anonymous 2007 Aug 1 1 ATTACH leaks memory Attaching a non-existing file to an empty database leaks memory. The scenario (Win32): *: Create a new empty database. *: sqlite3_prepare_v2 ( "ATTACH DATABASE 'C:\NonExistantFolder\NonExistantFile.db3' AS a;'); returns SQLITE_OK *: sqlite3_step(); returns SQLITE_ERROR *: sqlite3_finalize(); returns SQLITE_ERROR *: Close database and application. This returns a report about a single leak between 77 and 92 bytes in size. FYI: I also receive a 2nd, but smaller (21-28 bytes), leak when executing an FTS1 SELECT on an attached database. If you can confirm the 1st leak I will provide more information on the 2nd, if it still persists. _2007-Aug-30 14:21:25 by drh:_ {linebreak} I am unable to replicate the problem on Linux. My tests show all allocated memory is freed. Both SQLite's own internal accounting reports this, and valgrind too. We observe that the windows OS driver calls malloc() directly, bypassing SQLites internal memory allocator and its accounting procedures. And we do not know of anything like valgrind for windows. So perhaps there is a leak in the windows OS driver someplace. How do you see this leak, Ralf? What tools are you using? Do you get any indication where the leak is occurring? Can you see anything in the windows OS driver that might be causing the problem? ---- _2007-Aug-30 15:37:03 by anonymous:_ {linebreak} Found it: The if block in os_win.c, starting at line 1136, does not free zConverted after the else. The fix is to move the free(zConverted); line up into the 1st if block: if( h==INVALID_HANDLE_VALUE ){ free(zConverted); if( flags & SQLITE_OPEN_READWRITE ){ return winOpen(0, zName, id, ((flags|SQLITE_OPEN_READONLY)&~SQLITE_OPEN_READWRITE), pOutFlags); }else{ return SQLITE_CANTOPEN; } } #c8c8c8 2601 code closed 2007 Aug anonymous 2007 Oct 1 1 Problem with opening database with Unicode filename Problem with opening database with Unicode filename - it doesn't open the database with Unicode filename or path. The description contains insufficient information for us to act on the problem. #cfe8bd 2600 code fixed 2007 Aug anonymous 2007 Aug 1 1 #ifdef SQLITE_MUTEX_WIN32 naming error in mutex_w32.c? Line 23 in mutex_w32.c checks #ifdef SQLITE_MUTEX_WIN32. However, SQLITE_MUTEX_WIN32 is never defined in the SQLite core. Instead, SQLITE_MUTEX_WIN is (notice the missing 32 postfix). mutex_w32.c line 182 #endif /* SQLITE_MUTEX_WIN */ makes me believe that this is just a spelling error? If there are plans for extra support of non-Win32 platforms (Win64, Windows 3.1), it would be reasonable to choose _WIN32 over just _WIN. But then I suppose that this would also be true for the OS_WIN #define as well? #cfe8bd 2599 code fixed 2007 Aug anonymous 2007 Aug 1 1 OS check in mutex.h has no effect OS check in mutex.h (starting from line 34) dependes on OS autodetection in os.h. However, in sqliteInt.h the statement #include "mutex.h" is in line 76, way before #include "os.h" is followed in line 297. OS autodetection is therefore not taken into account for mutex.h and the library is compiled without mutexes, at least in Win32. It did work before mutex.c was split up for different OSes. #cfe8bd 2598 code fixed 2007 Aug anonymous 2007 Oct 3 3 Failure to delete temporary files on WINCE platforms Problem lies in the sqlite3WinOpenExclusive function. Under WINCE platform, winFile.hMutex is set to NULL (line 822 of os_win.c) even though winFile.zDeleteOnClose is set. So when winceDestroyLock (called from winClose) is eventually called, because hMutex is NULL, the file is never deleted. #cfe8bd 2597 code fixed 2007 Aug anonymous 2007 Oct 1 1 tclsqlite "db eval" with array-name doesn't have "*" entry. (tcl8.5a7) This problem was caused on tcl cvs head(tcl8.5a7) only. Tcl8.4 and 8.5a6 doesn't have the problem. package require sqlite3 sqlite db :memory: db eval "create table t (c1, c2, c3)" db eval "PRAGMA empty_result_callbacks = on" db eval "select * from t" val "" puts [array get val] I expected a value "* {c1 c2 c3}", but the val(*) entry didn't exist. It seems to me that "Tcl_DecrRefCount(pStar);" cause the problem... _2007-Aug-30 03:18:14 by anonymous:_ {linebreak} I'm sorry for the bad format. package require sqlite3{linebreak} sqlite db :memory:{linebreak} db eval "create table t (c1, c2, c3)"{linebreak} db eval "PRAGMA empty_result_callbacks = on"{linebreak} db eval "select * from t" val ""{linebreak} puts [array get val]{linebreak} #c8c8c8 2596 code closed 2007 Aug anonymous 2007 Aug 2 2 wrong specifier for extern char arrays In 3.4.2 (compared to 3.4.1) the forward declarations of char arrays sqlite3UpperToLower, sqlite3OpcodeNames and sqlite3IsIdChar are labeled with SQLITE_PRIVATE instead of previous extern. This causes problem with amalgamated file - the compiler (Visual C++ 9.0 beta 2) fails to compile it. Putting extern instead of SQLITE_PRIVATE had fixed the problem. Compile with -DSQLITE_PRIVATE=extern or compile a file like this: /* foo.c */ #define SQLITE_PRIVATE extern #include "sqlite3.c" #f2dcdc 2595 doc active 2007 Aug anonymous 2007 Aug 4 4 sqlite3_commit_hook doc typo src/main.c: -** Register a function to be invoked when a transaction comments. +** Register a function to be invoked when a transaction commits. #c8c8c8 2594 code closed 2007 Aug anonymous 2007 Aug 5 4 ability to replace allocator In order to use my own allocator for the SQLite I reimplemented routines sqlite3GenericMalloc, sqlite3GenericRealloc and sqlite3GenericOsFree. SQLite, however, needs some modification to use these functions: (1) The original functions need to be commented out. (2) Functions sqlite3OsMalloc, sqlite3OsRealloc and sqlite3OsFree (in amalgamated code) use specifier SQLITE_PRIVATE (static) and this needs to be removed too. Perhaps the SQLite code may employ macro SQLITE_I_HAVE_MY_OWN_ALLOCATOR or so to do this all w/o need to touch the code. The change of allocator worked well for me (I use Doug Lea's dlmalloc). _2007-Aug-29 22:34:36 by drh:_ {linebreak} Please check the mailing list archives. This is a planned feature for version 3.5.0. ---- _2007-Aug-29 22:56:18 by anonymous:_ {linebreak} What does OBE stand for? ---- _2007-Aug-29 23:45:10 by drh:_ {linebreak} Overcome by events. Normally that means that a ticket is no longer relevant because things have changed since the ticket was written. That does not apply here, exactly, but it was (in my opinion) the best available option for Resolution. #c8c8c8 2593 code closed 2007 Aug anonymous 2007 Aug 1 1 btshared unlock bug You have want to avoid a race condition by zeroing p->locked within the mutex. Otherwise another thread may acquire the lock, set locked to 1 (while still holding the mutex) and then this thread outside the mutex will unintentionally clear it. @@ -191,6 +193,7 @@ p->wantToLock--; if( p->wantToLock==0 ){ + p->locked = 0; sqlite3_mutex_leave(p->pBt->mutex); - p->locked = 0; } } } Haven't looked at the code, there may be other occurances of this. _2007-Aug-28 17:57:26 by drh:_ {linebreak} The p->locked variable is protected by a separate mutex, specifically the mutex at p->pSqlite->mutex. The database handle, the thing that sqlite3_open() returns, is at p->pSqlite. Each such handle has its own mutex. Associated with each database handle is one or more Btree structures, one for each open database. There are typically two of these, one for the main database and another for the TEMP database. New Btree structures are added for each ATTACH. Each Btree is tied to a single database handle so it is protected by the same mutex as the database handle. "p" is a pointer to a Btree in this instance. The guts of a database are contained in a Btshared structure. Btree points to Btshared. But if shared cache is enabled, two or more Btree structures from different database handles might point to the same Btshared. So Btshared needs its own mutex. This new mutex covers only the Btshared. The Btree structure is covered by the original database handle mutex. See also #2578 #c8c8c8 2592 warn closed 2007 Aug anonymous 2007 Aug 1 1 The Entry C:\Windows\Temp\sqlite_z6onAYdsVGqLG3Q is Invalid Everytime I reboot and check disk I get " The Entry C:\Windows\Temp\sqlite_z6onAYdsVGqLG3Q is Invalid I believe this has something to do with skype and all messengers rebooting my PC everytime I try to enable voice Thank you This is a question for a skype group. #cfe8bd 2591 doc fixed 2007 Aug anonymous 2007 Aug 4 4 sqlite3_open() documentation should mention :memory: I could not find information about in-memory database support in SQLite from API docs. In my opinion, sqlite3_open() function call is the place where it should be mentioned that I can give ":memory:"(?) as filename parameter to open in-memory database. #c8c8c8 2590 todo closed 2007 Aug anonymous 2007 Aug danielk1977 2 2 return value of strftime('%s', 'now') being determined as 'text' Hello. SELECT typeof(strftime('%s', 'now')); results 'text'. Is this correct? As strftime('%s', 'now') returns number of seconds since Unix epoch started. Greatings. In my opinion it is correct. strftime returns a formatted string. If for some reason you want the type of the expression to be an integer you could do: SELECT CAST(strftime('%s', 'now') AS INTEGER); #c8c8c8 2589 doc closed 2007 Aug anonymous 2007 Aug danielk1977 4 4 s/effected/affected/ in PRAGMA locking_mode In the last sentence of the documentation for PRAGMA locking_mode, "effected" should be "affected". thanks. #cfe8bd 2588 code fixed 2007 Aug anonymous 2007 Aug 1 1 bug in sqlite3_mutex_leave Race scenario on same mutex leading to corruption: Thread 1 Thread 2 -------------------------- -------------------------- sqlite3_mutex_enter pthread_mutex_lock ok p->owner = Thread 1 p->nRef = 1; sqlite3_mutex_leave p->nRef--; pthread_mutex_unlock ok sqlite3_mutex_enter sqlite3_mutex_enter pthread_equal is true pthread_equal is false pthread_mutex_lock ok p->owner = Thread 2; p->nRef = 1; p->nRef>0 is true p->nRef++; p->nRef is now 2 and the mutex is "owned" by Thread 2. Any data structure protected by this mutex can now be changed by both threads at the same time. Here's a possible pseudo fix:
Index: src/mutex.c =================================================================== RCS file: /sqlite/sqlite/src/mutex.c,v retrieving revision 1.9 diff -u -3 -p -r1.9 mutex.c --- src/mutex.c 24 Aug 2007 20:46:59 -0000 1.9 +++ src/mutex.c 25 Aug 2007 01:39:10 -0000 @@ -358,6 +358,7 @@ void sqlite3_mutex_leave(sqlite3_mutex * assert( p->nRef>0 ); p->nRef--; if( p->nRef==0 ){ + memset(&p->owner, 0, sizeof(p->owner)); pthread_mutex_unlock(&p->mutex); } }But even this fix is not guaranteed by POSIX standards, as an opaque pthread_t type when memset to zero may indeed be a valid pthread_t for the first (zeroeth) thread. This could lead to the same race condition and corruption. http://mail-archives.apache.org/mod_mbox/apr-dev/200307.mbox/%3c3F285220.7010501@netscape.com%3e This stuff is difficult to get right. Please consult correct recursive mutex implementations from various multi-threaded open source projects. _2007-Aug-25 02:35:35 by anonymous:_ {linebreak} Furthermore, the pthread_equal man page states: "If either t1 or t2 are not valid thread IDs, the behavior is undefined." http://www.opengroup.org/onlinepubs/009695399/functions/pthread_equal.html Only pthread_create and pthread_self can generate a valid pthread_t thread ID. ---- _2007-Aug-25 02:43:32 by anonymous:_ {linebreak} On POSIX systems that support it you should use PTHREAD_MUTEX_RECURSIVE instead of rolling your own. It is more efficient and guaranteed to be correct. http://www.opengroup.org/onlinepubs/000095399/functions/pthread_mutex_lock.html ---- _2007-Aug-25 03:23:56 by anonymous:_ {linebreak} Amusing back story of recursive POSIX mutexes from one of its creators: http://groups.google.com/group/comp.programming.threads/msg/d835f2f6ef8aed99?hl=en ---- _2007-Aug-25 04:13:08 by drh:_ {linebreak} I believe the problem is fixed by doing the p->nRef>0 test first and then the ownership test second. Rationale: 1: If p->nRef>0 it can only mean that some thread is holding the mutex. we don't know which thread, and the mutex might change owners multiple times before the next test. But we do at least know that somebody has held the mutex in the past. And we also know that p->owner has been initialized. 2: If the second pthread_equal() test is true, that means that the current thread claims to own the mutex. No other thread will ever put the current thread's threadid into p->owner. If p->owner==self, then only the current thread put it there. If another thread had claimed the mutex after the current thread, then the other thread would have changed the value of p->owner before raising p->nRef above zero. We know that p->nRef has been above zero, so the current thread must be the last one to have held the mutex. Hence, the current thread must still own the mutex. The above assumes that pthread_equal() is an atomic operation. In other words, it will not fail with a TRUE return if a separate thread changes the value of p->owner while the current thread is testing it. If p->owner is a multi-word value, then pthread_equal() might not be atomic. But pthread_t is an integer on every implementation of pthreads that I am aware of, so I think this is probably safe in most instances. Please double-check my reasoning above and let me know if I've said or done something goofy. Tnx. ---- _2007-Aug-25 04:25:26 by drh:_ {linebreak} Response to Butenhof: Threading in SQLite is not designed to make it more concurrent. It is designed to make the code safer to use by people who insist on writing multithreaded programs when multithreading really isn't necessary. Each database connection has its own mutex. Any thread using the database connection holds that mutex. So only a single thread can use a particular database connection at a time. The mutex is acquired when the API is entered and released when the API exits. But SQLite is reentrant. Top-level APIs can (and are) invoked as callbacks from within other top-level APIs. The only way to allow reentancy for one thread while excluding other threads (the only way that *I* know of, anyhow) is a recursive mutex. If David Butenhof thinks that recursive mutexes can be eliminated from SQLite by do "careful design", I'd like to hear him tell me how. One suspects that Mr. Butenhof has never tried to build a reentrant library for use in a multithreaded environment.... ---- _2007-Aug-25 04:47:52 by anonymous:_ {linebreak} My first impression is that the fix is not correct. Take a look at this post and the accompanying thread: http://groups.google.com/group/comp.programming.threads/msg/329ba20fe934b5e1?hl=en& pthread_equal() is probably not atomic on HPUX where pthread_t is a struct (of 4 int's as I recall). Nothing in the pthread_equal man page suggests that pthread_equal is guaranteed to be atomic, although on platforms where pthread_t is an int it would obviously be fine. Also, there is no concept of an official invalid pthread_t value - and even if there was, the pthread_equal man page is pretty clear that the result of such a comparison with an "invalid" pthread ID is undefined. For what it is worth, Apache Portable Runtime seems to have given up trying to roll their own recursive mutex on POSIX platforms that do not support PTHREAD_MUTEX_RECURSIVE. I appreciate that your use of recursive mutexes is for correctness and ease of API use. An alternate scheme would be to change the SQLite API to simply pass messages via work queues to a background thread that performs the database operations on your behalf. Much like your previous single thread "server" model. But then you have the headache of making every single SQLite API function work asynchronously - and you still would not improve concurrency on the same database. ---- _2007-Aug-25 05:05:17 by anonymous:_ {linebreak} This is somewhat related. When you work with SMP hardware and various optimizing compilers some things are not obvious. Take a look at the seemingly logical double-checked lock mis-pattern: http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html http://en.wikipedia.org/wiki/Double-checked_locking ---- _2007-Aug-25 14:45:21 by drh:_ {linebreak} Well, it turns out that SuSE Linux does support recursive mutexes after all. But in order to get it to work you have to have a line #define _XOPEN_SOURCE 500 prior to your #include
+ if( !isInit ){ + pthread_mutex_lock(&initMutex); + if( !isInit ){ + pthread_mutexattr_init(&recursiveAttr); + pthread_mutexattr_settype(&recursiveAttr, PTHREAD_MUTEX_RECURSIVE); + } + isInit = 1; + pthread_mutex_unlock(&initMutex); + }Although the liklihood of a problem occurring is rare, it is still present. Since SQLite does not create many recursive mutexes I don't think it is worth to try to optimize this case. The calls to pthread_mutexattr_init and pthread_mutexattr_settype are typically very fast. Please consider using this simpler code instead: case SQLITE_MUTEX_RECURSIVE: { p = sqlite3MallocZero( sizeof(*p) ); if( p ){ pthread_mutexattr_t recursiveAttr; pthread_mutexattr_init(&recursiveAttr); pthread_mutexattr_settype(&recursiveAttr, PTHREAD_MUTEX_RECURSIVE); pthread_mutex_init(&p->mutex, &recursiveAttr); p->id = iType; } break; } ---- _2007-Aug-25 16:26:01 by anonymous:_ {linebreak} Forgot the pthread_mutexattr_destroy... case SQLITE_MUTEX_RECURSIVE: { p = sqlite3MallocZero( sizeof(*p) ); if( p ){ pthread_mutexattr_t recursiveAttr; pthread_mutexattr_init(&recursiveAttr); pthread_mutexattr_settype(&recursiveAttr, PTHREAD_MUTEX_RECURSIVE); pthread_mutex_init(&p->mutex, &recursiveAttr); pthread_mutexattr_destroy(&recursiveAttr); p->id = iType; } break; } ---- _2007-Aug-25 16:34:59 by drh:_ {linebreak} You will notice that, unlike the examples cited previously, my double-lock was in fact correct. Nevertheless, your observation that recursive mutexes are rarely allocated (only at sqlite3_open()) and that the initialization and destruction of a pthread_mutexattr_t is relatively cheap are valid. And the code is simpler without the double-lock. So the double-lock has now been removed. ---- _2007-Aug-25 16:48:30 by anonymous:_ {linebreak} Well, I disagree about your double lock being correct. There is a window of failure on SMP machines on some hardware with certain optimizing compilers. http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html Nonetheless, the recursive mutex code as of the latest checkin is fine. The speed of millions of lock/unlock calls of native POSIX recursive mutexes more than makes up for the few cycles of initializing the attribute. Some older OSes will undoubtedly have trouble with _XOPEN_SOURCE, but this is easy enough for the odd port to fix. #f2dcdc 2587 build active 2007 Aug anonymous 2007 Aug 3 4 Build problem when using the SQLITE_OMIT_FLOATING_POINT define. I apologize in advance if the values I chose above are not appropriate. If I define SQLITE_OMIT_FLOATING_POINT=1 and try to build a Windows DLL, I get two errors in loadext.c, line 116 and 192. "error C4028: formal para meter 3 different from declaration" I believe you want to change the include order at the top of loadext.c from: #include "sqlite3ext.h" #include "sqliteInt.h" to: #include "sqliteInt.h" #include "sqlite3ext.h" Reversing the order of include fixes my build. Yes, I know there is no real reason to disable floating point for the Windows DLL. I'm actually porting SqLite for use in an NT kernel mode driver and avoiding floating point operations will save a lot of time if I don't really need them and I don't. So I made sure this was a problem with a supported platform like the Windows DLL and griped about that instead of my insanity. ;-) You can email questons to mspiegel@vipmail.com. If you want to discuss this over the phone, shoot me an email and I'll send you phone numbers. #c8c8c8 2586 code closed 2007 Aug anonymous 2007 Aug 3 3 INSERT of large number fails on Windows When using the windows version of the sqlite3.dll, trying to insert a number larger then what fits in a 64 bits int (so larger then: 9223372036854775807) will give a "Invalid floating point operation" exception. The following code will reproduce this: create table t(a float);{linebreak} insert into t values(2.49190E+26); The command-line tool does not have this problem however. The exception seems to be thrown at this line:{linebreak} pMem->u.i = pMem->r;{linebreak} in the "sqlite3VdbeIntegerAffinity" function in vdbemem.c{linebreak} At least, that is what Visual Studio 8 says. _2007-Aug-24 13:19:25 by anonymous:_ {linebreak} You've changed the floating point accuracy to single precision, probably because you're writing 3D software. Google for "controlfp". http://msdn2.microsoft.com/en-us/library/e9b52ceh(VS.80).aspx #c8c8c8 2585 code closed 2007 Aug anonymous 2007 Oct anonymous 1 1 Error Function GetAtomNameA The Procedure entry point GetAtomNameA could not be located in the dynamic link library sqlite3.dll Using CodeBlock _2007-Oct-05 15:11:29 by drh:_ {linebreak} Nothing in SQLite invokes GetAtomNameA(). This must be a problem with some other library. #c8c8c8 2584 new closed 2007 Aug anonymous 2007 Aug 1 2 On Windows function sqlite3WinSleep needs to dispatch messages When sqlite is busy, it spins in the loop, re-trying call every 100 ms. On Windows if the caller thread runs a message loop (for example a GUI application of COM call from an appartment), the message loop stops and the application hangs. As a workaround, sqlite function sqlite3WinSleep needs to call PeekMessage with subsequent calls TranslateMessage/DispatchMessage to continue message processing as following:{linebreak} /*{linebreak} ** Sleep for a little while. Return the amount of time slept.{linebreak} */{linebreak} int sqlite3WinSleep(int ms){linebreak} {{linebreak} //----------------------------------------------------------------{linebreak} //The fix starts{linebreak} MSG msg;{linebreak} while (PeekMessage(&msg, NULL, 0, 0, PM_REMOVE)){linebreak} {{linebreak} TranslateMessage(&msg);{linebreak} DispatchMessage (&msg);{linebreak} if (msg.message == WM_QUIT) //Note - need to handle WM_QUIT,posted by PostQuitMessage{linebreak} break;{linebreak} }{linebreak} //The fix ends{linebreak} //-----------------------------------------------------------------{linebreak} Sleep(ms);{linebreak} return ms;{linebreak} }{linebreak} _2007-Aug-23 16:08:43 by drh:_ {linebreak} Processing GUI events is not the business of a database engine. If you need to process GUI events while waiting on a lock, then use the sqlite3_busy_handler() API to register your own busy handler (perhaps using the code shown in the patch) instead of using the built-in sqlite3_busy_timeout(). ---- _2007-Aug-23 17:54:06 by anonymous:_ {linebreak} You should wrap in a thread your db calls if you wan't to process using GetMessage() calls #cfe8bd 2583 build fixed 2007 Aug anonymous 2007 Aug 2 2 3.4.2 fails to build on Solaris due to conflict for B_FALSE and B_TRUE While attempting to build SQLite 3.4.2 on Solaris 10/x86 with gcc 3.4.3, I get this error: gcc -D_REENTRANT -g -O2 -o lemon ./tool/lemon.c ./tool/lemon.c:111: error: conflicting types for 'B_FALSE' /usr/local/lib/gcc/i386-pc-solaris2.10/3.4.3/include/sys/types.h:193: error: previous definition of 'B_FALSE' was here ./tool/lemon.c:111: error: conflicting types for 'B_TRUE' /usr/local/lib/gcc/i386-pc-solaris2.10/3.4.3/include/sys/types.h:193: error: previous definition of 'B_TRUE' was here Here is the conflicting line in /sqlite/tool/lemon.c 111 typedef enum {B_FALSE=0, B_TRUE} Boolean; And the conflict in the gcc version of =sys/types.h=: 190 #if defined(__XOPEN_OR_POSIX) 191 typedef enum { _B_FALSE, _B_TRUE } boolean_t; 192 #else 193 typedef enum { B_FALSE, B_TRUE } boolean_t; 194 #endif /* defined(__XOPEN_OR_POSIX) */ Oddly enough, 3.4.1 built successfully despite having the same code in lemon.c. Possibly some header files have changed and are causing __XOPEN_OR_POSIX no longer to be defined. Workaround: rename B_FALSE to BOOL_FALSE and B_TRUE to BOOL_TRUE throughout lemon.c. #f2dcdc 2582 doc active 2007 Aug anonymous 2007 Aug anonymous 5 5 documenation clarification docs for topic `Set A Busy Timeout` int sqlite3_busy_timeout(sqlite3*, int ms); http://sqlite.org/capi3ref.html#sqlite3_busy_timeout the wording "The handler will sleep multiple times until at least "ms" milliseconds of sleeping have been done" implies it will wait the total amount regardless of the lock status, it should perhaps indicate in the same sentence that it will exit early if the lock becomes available. #c8c8c8 2581 code closed 2007 Aug anonymous 2007 Aug 1 1 ./configure && make broken: new source files: mem[12].c, mutex.c, etc Please apply the diff given by this command to Makefile.in: cvs diff -D'2007-08-14' main.mk _2007-Aug-21 20:31:44 by drh:_ {linebreak} We are in the middle of a major reimplementation of large sections of SQLite. We could fix the makefile, but the build still wouldn't work. So what's the point, really. I promise to fix the makefile before the next release.... #f2dcdc 2580 code active 2007 Aug anonymous 2007 Aug anonymous 1 2 Can't open a query if text to search is Greek for example: SELECT * FROM mytable WHERE mycolumn LIKE '%some greek text%' I get wrong results, using the 3.4.2 version. No problem instead using other earlier version. I tested only in Windows. #cfe8bd 2579 code fixed 2007 Aug anonymous 2007 Oct 4 3 the third parameter of substr should be optional Please make the 3rd Parameter of SUBSTR optional as in most other programming languages. It is better and simpler for migrations to SQLite #c8c8c8 2578 code closed 2007 Aug anonymous 2007 Aug 1 1 sqlite3BtreeEnter, sqlite3BtreeLeave threading issues p->wantToLock++ is not an atomic operation and must be done within a mutex lock, otherwise its result is undefined. Same goes for "variable--". You need an architecture-specific operations to do that safely between threads. See http://www.mozilla.org/projects/nspr/reference/html/pratom.html and http://lxr.mozilla.org/nspr/source/nsprpub/pr/include/pratom.h#60 for an example.
void sqlite3BtreeEnter(Btree *p){ Btree *pLater; /* Some basic sanity checking on the Btree. The list of Btrees ** connected by pNext and pPrev should be in sorted order by ** Btree.pBt value. All elements of the list should belong to ** the same connection. Only shared Btrees are on the list. */ assert( p->pNext==0 || p->pNext->pBt>p->pBt ); assert( p->pPrev==0 || p->pPrev->pBtThis needs fixing:pBt ); assert( p->pNext==0 || p->pNext->pSqlite==p->pSqlite ); assert( p->pPrev==0 || p->pPrev->pSqlite==p->pSqlite ); assert( p->sharable || (p->pNext==0 && p->pPrev==0) ); /* Check for locking consistency */ assert( !p->locked || p->wantToLock>0 ); assert( p->sharable || p->wantToLock==0 ); if( !p->sharable ) return; p->wantToLock++; if( p->locked ) return; /* In most cases, we should be able to acquire the lock we ** want without having to go throught the ascending lock ** procedure that follows. Just be sure not to block. */ if( sqlite3_mutex_try(p->pBt->mutex)==SQLITE_OK ){ p->locked = 1; return; }
Index: src/btree.c =================================================================== RCS file: /sqlite/sqlite/src/btree.c,v retrieving revision 1.401 diff -u -3 -p -r1.401 btree.c --- src/btree.c 17 Aug 2007 16:50:38 -0000 1.401 +++ src/btree.c 17 Aug 2007 20:42:49 -0000 @@ -1403,8 +1403,8 @@ void sqlite3BtreeEnter(Btree *p){ assert( pLater->pNext==0 || pLater->pNext->pBt>pLater->pBt ); assert( !pLater->locked || pLater->wantToLock>0 ); if( pLater->locked ){ - sqlite3_mutex_leave(pLater->pBt->mutex); pLater->locked = 0; + sqlite3_mutex_leave(pLater->pBt->mutex); } } sqlite3_mutex_enter(p->pBt->mutex); @@ -1427,8 +1427,8 @@ void sqlite3BtreeLeave(Btree *p){ p->wantToLock--; if( p->wantToLock==0 ){ assert( p->locked ); - sqlite3_mutex_leave(p->pBt->mutex); p->locked = 0; + sqlite3_mutex_leave(p->pBt->mutex); } } }Otherwise there would be corruption if there was a context switch just after sqlite3_mutex_leave to "if( p->locked ) return;" in sqlite3BtreeEnter() in another thread. You might want to replace the u8's below with int's if you intend to read or write to them outside of a mutex:
struct Btree { sqlite3 *pSqlite; /* The database connection holding this btree */ BtShared *pBt; /* Sharable content of this btree */ u8 inTrans; /* TRANS_NONE, TRANS_READ or TRANS_WRITE */ u8 sharable; /* True if we can share pBt with other pSqlite */ u8 locked; /* True if pSqlite currently has pBt locked */ int wantToLock; /* Number of nested calls to sqlite3BtreeEnter() */ Btree *pNext; /* List of Btrees with the same pSqlite value */ Btree *pPrev; /* Back pointer of the same list */ };Reading from and writing to a non-machine word (i.e., not an int) variable outside of a mutex is not atomic or threadsafe. MIPS, for example, can only manipulate 32 bit values at a time and must use bit operations and shifting to get at individual characters. If you read a char value at the wrong time it may contain 0, 255 or some other value. _2007-Aug-17 21:32:11 by anonymous:_ {linebreak} These 3 values will likely be stored in the same 32 bit integer: u8 inTrans; /* TRANS_NONE, TRANS_READ or TRANS_WRITE */ u8 sharable; /* True if we can share pBt with other pSqlite */ u8 locked; /* True if pSqlite currently has pBt locked */ Writing to any one of them outside of a mutex in a multithreaded situation can scramble the others' values. ---- _2007-Aug-20 00:49:28 by drh:_ {linebreak} Not a bug. The Btree.wantToLock values are accessed under the sqlite3.pMutex lock - or at least they will be once we get the new threading implementation half way completed. ---- _2007-Aug-20 05:17:47 by anonymous:_ {linebreak} Helgrind is useful to find these sorts of multi-thread race conditions, as well as mutex deadlocks. http://valgrind.org/docs/manual/hg-manual.html #c8c8c8 2577 code closed 2007 Aug anonymous 2007 Aug 5 5 Initilizing pointers to 0 instead of NULL. Is it better to change all initilizations an comparisions of pointers to NULL instead of 0 ? It is only cosmetic change but the code is then more readable. Bartosz _2007-Aug-17 18:15:16 by anonymous:_ Why the ticket was rejected ? If we have eg.: .... *pId = 0; .. what is *pId ? Integer value or pointer to pointer ? I know that name suggest that it is one level pointer but why we can not make reading easier ? Bartosz. ---- _2007-Aug-17 19:48:01 by anonymous:_ {linebreak} Changing 0 to NULL adds no value to the code. 0 is one character. NULL is four characters. Both are valid, but 0 is consistent and predictable on all platforms. NULL requires an #include to work, 0 does not. #c8c8c8 2576 code closed 2007 Aug anonymous 2007 Aug 1 1 Check-in [4239] mutex race conditions Nevermind. I missed seeing SQLITE_MUTEX_STATIC_MASTER. #c8c8c8 2575 code closed 2007 Aug anonymous 2007 Aug 1 1 sqlite3_mutex_enter missing return I know this is experimental #if'd out code, but nonetheless, sqlite3_mutex_enter is missing a return for the final else clause. #cfe8bd 2574 build fixed 2007 Aug anonymous 2007 Aug anonymous 3 3 amalgamation 3.4.2 compile errors The new amalgamation 3.4.2 fails with the following compile errors using Visual C++ (2003 and 2005): sqlite3.c(6187) : error C2133: 'sqlite3UpperToLower' : unknown size sqlite3.c(9310) : error C2133: 'sqlite3OpcodeNames' : unknown size sqlite3.c(47159) : error C2133: 'sqlite3IsIdChar' : unknown size Cause: The 'extern' keyword was replaced by 'SQLITE_PRIVATE' I suspect this should have been the new macro 'SQLITE_EXTERN' Changing from SQLITE_PRIVATE to SQLITE_EXTERN results in a successfull build _2007-Aug-15 10:35:23 by drh:_ {linebreak} The problem is that there is a declaration of the constant: SQLITE_PRIVATE const unsigned char sqlite3UpperToLower[]; Then later the constant is defined with its content: SQLITE_PRIVATE const unsigned char sqlite3UpperToLower[] = {...}; How does VC++ expect you to declare a static constant with file scope before it is defined? GCC is quite happy with the construct above. ---- _2007-Aug-15 12:56:30 by anonymous:_ {linebreak} VC++ is only happy when you declare it as: SQLITE_PRIVATE const unsigned char sqlite3UpperToLower[256]; It seems it has to know the size upfront. I've also tried various compiler switches, but no postive results. ---- _2007-Aug-15 13:29:05 by anonymous:_ {linebreak} C0x and C++0x seem to allow the construct of forward declaring an unknown sized array of static (file scope) types. I wonder if gcc in --pedantic mode might also complain about it. This compilation set of settings is what perl uses to help guarantee cross-platform portability: gcc -g -O -pedantic -Wall -W -Wshadow -Wpointer-arith -Wbad-function-cast -Wcast-qual -Wcast-align -Wwrite-strings -Wstrict-prototypes -Wmissing-prototypes -Wnested-externs -Wno-long-long ---- _2007-Aug-15 13:29:41 by anonymous:_ {linebreak} Does VC++ allow the same forward declaration without const? SQLITE_PRIVATE unsigned char sqlite3UpperToLower[]; If not, putting the size in advance for the array forward declaration is acceptable. ---- _2007-Aug-15 13:36:57 by anonymous:_ {linebreak} SQLITE_PRIVATE unsigned char sqlite3UpperToLower[]; This produces the same error and also another one: error C2133: 'sqlite3UpperToLower' : unknown size error C2373: 'sqlite3UpperToLower' : redefinition; different type modifiers The second error was to be expected ---- _2007-Aug-15 13:42:36 by anonymous:_ {linebreak} A function could be created to simply return the pointer to the array in question. Seemingly inefficient, but any modern optimizing compiler in the last 5 years would inline this function call away if in the same translation unit, as in the amalgamation. ---- _2007-Aug-15 13:54:05 by anonymous:_ {linebreak} Just change the forward declaration to a pointer: SQLITE_PRIVATE const unsigned char *sqlite3UpperToLower; as well as the implementation: SQLITE_PRIVATE const unsigned char *sqlite3UpperToLower = { ... }; ---- _2007-Aug-15 14:08:11 by anonymous:_ {linebreak} Does VC++ like this? static char* arr; static char* arr; static char arr_impl[] = { 'f', 'o', 'o' }; static char *arr = arr_impl; ---- _2007-Aug-15 15:17:45 by anonymous:_ {linebreak} No, it does not. It gives redefinition warnings on the initializer. I think part of the problem is that 'static TYPE var;' is default initialized when it is seen, and SPACE IS ALLOCATED FOR IT in any compilation unit that sees it. It is not given external scope. What I think I see is that every part of the individual .c files were scanned for file-scope definitions, which were then placed into an internals.h file for inclusion across all things that needed it. What I think you'll see if you look at the .o files for each .c file compiled separately, is a definition for each of the static variables that you were trying to predeclare. I think they'll need extern scope if you want to gain access to them in separate compilation units. And I think it might be better to declare and use the opcodes list, the UpperToLower table, and the character map table as pointers rather than arrays. Or bite the bullet of using extern instead of static. If you want to avoid overrunning the length of the tables you could provide an accessor function when printing the vdbe opcodes which could do range checking. Similarly, you could use something that is C-compiler safe for doing compile-time assertions that the length of the character map and lower-case tables are exactly 256 entries long when initialized. static bool must_be_256_long[sizeof(table)==256]; ---- _2007-Aug-15 15:28:56 by anonymous:_ {linebreak} The compile assert is not portable. static int must_be_256_long[0]; gcc does not complain about this unless the -pedantic flag is used. ---- _2007-Aug-15 15:33:51 by anonymous:_ {linebreak} All the external variables in question should be moved to a seperate vars.h and vars.c file. The amalgamation would have this vars.c file added before all other sqlite3 .h/.c files, and the vars.h file need not be added to the amalgamation at all. vars.h would only be used for non-amalgamation builds. ---- _2007-Aug-15 15:34:14 by anonymous:_ {linebreak} Since it's explicitly defined later, why not just mark the forward declaration as "extern"? This involves removing the SQLITE_PRIVATE in front. VS2005 let it compile for me no problems. ---- _2007-Aug-15 18:05:49 by anonymous:_ {linebreak} gcc will not allow an extern forward declaration followed by a static declaration. For background of why things were done the way they are see Ticket #2554 "Many symbols not marked private/static or SQLITE_API in amalgamation" ---- _2007-Aug-16 13:57:34 by anonymous:_ {linebreak} It seems to me that there are only the three symbols in concern: *: the vdbe opcode-to-string table, which is used in one place and auto-generated into another. *Perhaps* the place that it is used within could #include the auto-generated file, and exclude the auto generated file from the remainder of the compilation. (by naming it with .h, if necessary.) *Alternatively*, it could be encapsulated in an accessor sub-function instead of directly referred to by its array-based name. *: the character map for converting ebcdic to lower-cased ascii, or the similar translation when converting ascii to lower-cased ascii. It's used by one function when converting from a TOKEN to a token ID. It could also be just IN the file marked as SQLITE_PRIVATE and not forward declared at all. *: the upper to lower converter is used in a few places :) . No easy/quick solution for that that doesn't involve a possible decrease in performance based on non-inlined function calls in non-amalgamation compiled libraries. #cfe8bd 2573 doc fixed 2007 Aug anonymous 2007 Aug 3 3 sqlite3_interrupt() + threads sqlite3_interrupt() documentation says that "It is safe to call this routine from a thread different from the thread that is currently running the database operation". It is not true. 1st problem: db may be freed in sqlite3_close() during call to sqlite3_interrupt() - this may cause a crash. 2nd problem: after (db->magic==SQLITE_MAGIC_OPEN || db->magic==SQLITE_MAGIC_BUSY) and before db->u1.isInterrupted = 1; a lot of things can happen. Possibly another statement may be executed - this will interrupt a different statement than was expected. #c8c8c8 2572 warn closed 2007 Aug anonymous 2007 Aug 1 1 Sqlitw error unrecognize token Sqlite error unrecognize token The "is human" test needs to be improved. ---- _2007-Aug-14 21:20:44 by anonymous:_ {linebreak} wa? #c8c8c8 2571 todo closed 2007 Aug anonymous 2007 Aug 1 1 SQLite opening in Access I am trying to open SQLite database through ODBC connections in Microsoft Access 2000. Some tables in Database have Primary key defined . Steps followed:- System DSN TestSQLite is created using Data sources(ODBC) 1)Open Blank access database named db1.mdb 2)Click on FIle->Get External data->Link tables 3)Select - ODBC databases in Files of type in Link Dialog 4)Select Machine data source TestSQLite=20 5)Select All Tables and click Ok 6)If table does not have Primary key ,getting dialog to select unique record identifier. 7)if Primary key is defined in table ,getting error Reserved error (-7748) There is no msg for this error . =20 I can open tables which does not have Primary Key and cannot use MSAccess to open SQLite database through ODBC (if the SQLite database has primary key defined) . Thanks for your help. _2007-Aug-13 21:23:54 by anonymous:_ {linebreak} Install this sqlite odbc driver. http://www.ch-werner.de/sqliteodbc/ #cfe8bd 2570 code fixed 2007 Aug anonymous 2007 Aug 3 3 unportable assumptions in sqlite3AtoF() / cast.test The sqlite3 selftests 3.14, 3.18 and 3.24 fail for me with Expected: [9223372036854774784] Got: [9223372036854773760] Looking at the binary representations one finds that one bit of accuracy is missing (expected 53 bits, got 52 bits). The reason is within sqlite3AtoF(): It uses "long double" internally, and the (simplistic) algorithm does only deliver the expected result if "long double" uses a longer mantissa than "double". This might work on Linux/i386, but on other platforms a "long double" is identical to "double". #c8c8c8 2569 code closed 2007 Aug anonymous 2007 Aug 3 2 Executing a statement causes an AccessViolation Exception Executing follwowing statement causes an AccessViolation Exception:
select being.bid, bname, bvalue, bsyncstatus, bvalid from being where being.bid in ( select bid from searchidx_Campaign where ( campaign_status_id = 2 or campaign_status_id = 3 ) and id in (select campaign_id from searchidx_CampaignUserStatus where user_id = 'b37dd383-3c82-4fe1-9285-d4c2bd6e4ca3' and locked = 0))I tracked the problem down to the 'campaign_status_id = 2 or campaign_status_id = 3' part of the statement. if an 'and' instead of the 'or' is used the Exception doesn't occur ('campaign_status_id = 2 AND campaign_status_id = 3'). It also doesn't matter if there is data in any of the used tables. The exception raises also with empty tables. Used configuration: Windows XP Professional SP2 -> all MS updates installed .NET Framework 2.0 (2.0.50727.832) using c# _2007-Aug-13 10:17:56 by anonymous:_ {linebreak} It would be easier to reproduce if you provided the schema for the tables you use. ---- _2007-Aug-13 14:59:11 by drh:_ {linebreak} I created a schema that allowed the SQL statement to compile and then ran the SQL on version 3.3.4 without any problems. Unable to reproduce. #f2dcdc 2568 new active 2007 Aug anonymous 2007 Aug 3 3 TEMP_STORE is ignored in some cases It seems that sometimes TEMP_STORE is ignored. I've tried to force SQLite to always use memory by setting TEMP_STORE=3, but some etilqs_* temp files are still being created. The call stack that's causing these file to be created is: sqlite3PagerOpentemp(OsFile * *) sqlite3PagerStmtBegin(Pager *) sqlite3BtreeBeginStmt(Btree *) sqlite3VdbeExec(Vdbe *) sqlite3Step(Vdbe *) sqlite3_step(sqlite3_stmt *) It looks like the temp files are being used to store information for undoing earlier parts of a transaction if a later part fails. I'm assuming the fact this part of the code ignores TEMP_STORE is an over site? _2007-Aug-13 15:03:19 by drh:_ {linebreak} The TEMP_STORE compile-time option only changes the storage for temporary database files. The statement journal is not a databaes file and thus does not come under the control of TEMP_STORE. There is currently no mechanism to force the statement journal into memory instead of onto disk. I will reclassify this ticket as a "feature request". ---- _2007-Aug-22 10:42:50 by anonymous:_ {linebreak} Okay, thank you. #f2dcdc 2567 build active 2007 Aug anonymous 2007 Aug 3 2 Build fails to install I compile under MinGW with Msys. A build error occurs during 'make install'. After checking the makefile. The 'install' target depends on 'sqlite3', when it should be 'sqlite3$(TEXE)'. The workaround is, after configure, edit makefile for target install, and replace 'sqlite3' with 'sqlite3${TEXE}' where needed. I did not have this problem with 3.3.17. I assume this could be fixed just by fixing the configure to produce correct makefile. _2007-Aug-12 04:41:12 by anonymous:_ {linebreak} Do you have a patch? #f2dcdc 2566 build active 2007 Aug anonymous 2007 Aug 2 1 fts2 broken after vacuum Hi there, I'm testing your database and I'm having problems with fts2: --------- sqlite> select * from distB where distB match "MARIANO"; Assertion failed: *pData!='\0', file fts2amal.c, line 16790 This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information. --------- Steps: 1) Create a new .db 2) Import data in new distA table 3) Import data in new distB table 4) Create a new distC virtual table (dts2) 5) insert into distC (rowid, f1, f2, f3) select rowid, f1, f2, f3 from DistB Everything working like a charm until here!!! The fts2 works very well, but after 6) vacuum; the fts seems broken... doing a select throws the error I paste at the post of the topic If you want the .db file I can send it to you (607MB) Thanks.- #cfe8bd 2565 code fixed 2007 Aug drh 2007 Aug 1 1 Database corruption following ROLLBACK when soft_heap_limit set With {link: /pragma.html#pragma_auto_vacuum autovacuum mode} turned on low {link: /capi3ref.html#sqlite3_soft_heap_limit sqlite3_soft_heap_limit} database corruption results following a {link: /lang_transaction.html ROLLBACK} of a CREATE TABLE. The following script exhibits the problem: -- set sqlite3_soft_heap_limit to 5000 PRAGMA auto_vacuum=1; CREATE TABLE t1(x); INSERT INTO t1 VALUES(hex(randomblob(5000))); BEGIN; CREATE TABLE t2 AS SELECT * FROM t1; ROLLBACK; PRAGMA integrity_check; This problem was discovered following the enhanced sqlite_soft_heap_limit testing facility added by check-in [4202]. As of this writing, it has not been observed in the wild. _2007-Aug-11 11:55:04 by drh:_ {linebreak} **Background** The rollback journal file consists of a header followed by zero or more page records. Each page record contains original database content which can be used to restore the database back to its original state during a ROLLBACK or when the database is being recovered after a power-loss or crash. The header contains, among other things, the "nRec" field which is a count of the number of pages to follows. Whenever changes are flushed to the database file, the size of the journal is padded out to the next multiple of the disk sector size and a new header is written. (This is a defense against certain kinds of file corruption that might occur during a power failure.) A single journal can contain any number of instances of header followed by zero or more pages. While the journal is being constructed, the nRec value is 0. Then, just prior to writing changes back to the database file, the nRec value is changed to the correct number of records in the journal, the journal file is synced to disk, and only then is it safe to write to the database file. This all works great. **The Problem** The problem is that the ROLLBACK command also uses the journal file to restore changes. But the ROLLBACK command treats the nRec==0 value specially. When the ROLLBACK command sees an nRec of 0, it figures that it was just in the middle of doing some changes that haven't yet by written to the database file, so it tries to recompute the "correct" nRec value based on the current file size. The problem arises when a header contains an nRec value which really is suppose to be zero (no pages written into the journal) but that header is followed by one or more additional headers that contain pages that do need to be rolled back. When a journal is being rolled back during recovery, the recovery process sees the zero nRec, skips to the next header, and every thing works peachy. But during the ROLLBACK command, the zero nRec is converted into some positive value based on the journal file size. The new positive nRec is wrong. The ROLLBACK quickly discovers this and ignores the rest of the journal file under the assumption that it corrupted. This results in an incomplete ROLLBACK and database file corruption. **The Fix** The fix in check-in [4208] is to change the ROLLBACK process so that it only applies the special interpretation to an nRec==0 header if it is the very last header in the journal. **Why We Have Not Seen This Before** The problem only comes up when a header is written into the journal with nRec==0 and subsequent headers are written afterwards. This can only happen when the pager is having great difficulty finding new buffers into which new pages can be loaded and is having to spill dirty pages back to the disk. The minimum cache size is 10, and as it turns out a cache size of 10 still gives a handful of buffered pages before a cache spill occurs, even under the most severe circumstances. So even with a very small cache, the nRec field was always ending up larger than 0. But, if you set the soft heap limit to a ridiculously small number - so small that the pager is under continuous pressure to spill pages back to the database file as soon as they are unpinned - then there are cases where the pager will write non-terminal headers to the journal with nRec==0. Hence, as far as we are aware, you cannot hit this problem as long as you leave the soft heap limit disabled (which is the default configuration) or you set the soft heap limit large enough so that there are always enough memory for the pager to hold 10 or more pages in memory at the same time. It is difficult to imagine a real-world system that would do something different, so even though this problem has serious consequences, we cannot see how it might actually come up in practice. The problem was only noticed when we started stress-testing the soft heap limit mechanism by setting very low soft heap limits and running full regression tests. #c8c8c8 2564 event closed 2007 Aug anonymous 2007 Aug anonymous 4 4 in C:\windows\temp are 50 files like sqlite_fMyj88gwaTTNgUF and 0 kb in C:\windows\temp are 50 files like sqlite_fMyj88gwaTTNgUF and 0 kb every day are more files , which program use these files i cannot delete them with windows, clean up service i deleted them in safe mode they keep coming back and adding more files i use windows xp home edition and mcafee told me, it was not mcafee but a windows program thank you for your time jessica _2007-Aug-10 12:18:32 by anonymous:_ {linebreak} http://www.sqlite.org/cvstrac/wiki?p=McafeeProblem #c8c8c8 2563 code closed 2007 Aug anonymous 2007 Aug 1 1 sqlite3_column_database_name16() is broken =sqlite3_column_database_name16()= does not work at all. However I form my query, regardless of how many fields and in what order are returned in result, it always returns NULL for every single column. I'm using this code and it never returned anything but NULL: if ( pStatement) { UINT32 cols = sqlite3_column_count( pStatement); for ( UINT16 currCol = 0; !isDatabaseNameRetrieved && ( currCol < cols ); ++currCol ) { const WCHAR *databaseName = static_cast
1. 1st thread: open db3-file 2. 2nd thread: open the same (second connection handle) 3. 1st thread: begin transaction 4. 1st thread: run multiple 'insert' statement 5. 2nd thread: begin transaction (successful return from call) 6. 2nd thread: want to 'insert' but blocked (as expected), start waiting on busy flag 7. 1st thread: last 'insert' was successfull 8. 1st thread: want to 'end' transaction, but blocked (sqlite3_step returns infinite SQLITE_BUSY, WHY?!) 9. 2nd thread: still blocked on first insert attempt 10. So, both threads can't continue execution.Workaround: remove 'begin' and 'end' SQL statements from the code - and this works OK, but too slow. _2007-Aug-10 04:13:37 by anonymous:_ {linebreak} You might try BEGIN EXCLUSIVE. You lose parallelism, but it won't deadlock, and the code is simpler. See also: http://www.sqlite.org/capi3ref.html#sqlite3_busy_handler "If SQLite determines that invoking the busy handler could result in a deadlock, it will return SQLITE_BUSY instead. Consider a scenario where one process is holding a read lock that it is trying to promote to a reserved lock and a second process is holding a reserved lock that it is trying to promote to an exclusive lock. The first process cannot proceed because it is blocked by the second and the second process cannot proceed because it is blocked by the first. If both processes invoke the busy handlers, neither will make any progress. Therefore, SQLite returns SQLITE_BUSY for the first process, hoping that this will induce the first process to release its read lock and allow the second process to proceed." ---- _2007-Aug-10 11:36:32 by anonymous:_ {linebreak} Yes, with 'begin exclusive' it works. #cfe8bd 2561 code fixed 2007 Aug anonymous 2007 Dec 1 1 Parser broken -- ORDER BY fails in UNION between two databases To reproduce: 1. Create two simple databases, each with only one table "Table1" containing two fields -- ID (integer primary key) and Name (string); 2. Fill databases with a few records each of sample data; 3. Open one database and attach the other one as "aux"; 4. Try this query: SELECT Table1.Name, Table1.ID FROM main.Table1 UNION SELECT Table1.Name, Table1.ID FROM aux.Table1 ORDER BY Table1.ID; 5. Query fails with message: "SQL error: ORDER BY term number 1 does not match any result column". _2007-Aug-09 05:27:20 by anonymous:_ {linebreak} This doesn't work either: SELECT * FROM main.Table1 UNION SELECT * FROM aux.Table1 ORDER BY Table1.ID ---- _2007-Aug-09 05:28:16 by anonymous:_ {linebreak} WORKAROUND!!! This works: SELECT Table1.name, Table1.ID AS alias FROM main.Table1 UNION SELECT Table1.name, Table1.ID AS alias FROM aux.Table1 ORDER BY alias ---- _2007-Aug-09 05:31:12 by anonymous:_ {linebreak} SELECT Table1.Name, Table1.ID FROM main.Table1 UNION SELECT Table1.Name, Table1.ID FROM aux.Table1 ORDER BY 2; ---- _2007-Aug-10 01:55:58 by anonymous:_ {linebreak} Real problem for me is that this is not working, as previously mentioned: SELECT * FROM main.Table1 UNION SELECT * FROM aux.Table1 ORDER BY Table1.ID Very inconvenient. ---- _2007-Sep-14 07:22:10 by anonymous:_ {linebreak} I also have this problem and had to implement a bit of a dirty workaround to get aliases on the order by column as suggested before. Would be nice if a minor update would come out that fixes this (dont want to install 3.5 ALPHA for a possible future fix) ;-) ---- _2007-Sep-14 08:51:23 by anonymous:_ {linebreak} AUX is in the DOS/Windows world reserved name so maybe it is the source of problems ? ---- _2007-Dec-13 18:58:37 by drh:_ {linebreak} See ticket #2822. #cfe8bd 2560 doc fixed 2007 Aug anonymous 2007 Aug 5 4 Documentation typos Normally, I don't log typos, but I draw the line when the product name (sqlite) is in error. I'm joking, right? No, it's true. If you look at the blob example at http://sqlite.org/lang_expr.html it shows: X'53514697465' However, that is an odd number of digits up there, considering that the part between the quotes is supposed to be hex. In fact, it seems to want to spell 'SQLite'. However, that L is having problems. Presumably, there should be a 'C' after the '4'. In http://www.sqlite.org/lang_insert.html the second occurence of 'not' should be 'no'. In http://www.sqlite.org/limits.html the first two occurrences of 'where' should be 'were'. Csaba Gabor from Vienna _2007-Aug-08 23:39:54 by anonymous:_ {linebreak} I reported blob literal issue a couple of months ago. See ticket #2168. I wonder if somebody is going to do something with it. #f2dcdc 2559 code active 2007 Aug anonymous 2007 Aug 4 4 "make clean" does not delete sqlite3.c and tsrc/
Index: Makefile.in =================================================================== RCS file: /sqlite/sqlite/Makefile.in,v retrieving revision 1.179 diff -u -3 -p -r1.179 Makefile.in --- Makefile.in 27 Aug 2007 23:38:43 -0000 1.179 +++ Makefile.in 28 Aug 2007 01:25:55 -0000 @@ -724,7 +724,7 @@ clean: rm -f testfixture$(TEXE) test.db rm -rf doc rm -f common.tcl - rm -f sqlite3.dll sqlite3.lib sqlite3.def + rm -rf sqlite3.dll sqlite3.lib sqlite3.def sqlite3.c tsrc distclean: clean rm -f config.log config.status libtool Makefile config.h sqlite3.pc#f2dcdc 2558 code active 2007 Aug anonymous 2007 Aug 2 3 Multiple JOIN USING() gives incorrect results I'm having a problem joining multiple tables with USING. It appears to work, but the results are incorrect. Here is an example to illustrate the problem. I believe the three SELECT statements should be equivalent, but they produce three different results. .header on .mode column CREATE TABLE Main (pk INTEGER PRIMARY KEY, name VARCHAR); CREATE TABLE OptA (pk INTEGER PRIMARY KEY, alpha VARCHAR); CREATE TABLE OptB (pk INTEGER PRIMARY KEY, beta VARCHAR); INSERT INTO Main VALUES (1, 'One'); INSERT INTO Main VALUES (2, 'Two'); INSERT INTO Main VALUES (3, 'Three'); INSERT INTO Main VALUES (4, 'Four'); INSERT INTO OptA VALUES (1, 'Alpha1'); INSERT INTO OptA VALUES (4, 'Alpha4'); INSERT INTO OptB VALUES (2, 'Beta2'); INSERT INTO OptB VALUES (4, 'Beta4'); SELECT * FROM Main LEFT JOIN OptA USING (pk) LEFT JOIN OptB USING (pk); SELECT * FROM Main LEFT JOIN OptB USING (pk) LEFT JOIN OptA USING (pk); SELECT Main.pk, name, alpha, beta FROM Main LEFT JOIN OptA ON Main.pk = OptA.pk LEFT JOIN OptB ON Main.pk = OptB.pk; Joining Main, OptA, and OptB omits Beta2: pk name alpha beta ---------- ---------- ---------- ---------- 1 One Alpha1 2 Two 3 Three 4 Four Alpha4 Beta4 Joining Main, OptB, and OptA omits Alpha1: pk name beta alpha ---------- ---------- ---------- ---------- 1 One 2 Two Beta2 3 Three 4 Four Beta4 Alpha4 Only by using ON instead of USING do we get the correct results: pk name alpha beta ---------- ---------- ---------- ---------- 1 One Alpha1 2 Two Beta2 3 Three 4 Four Alpha4 Beta4 I think this is basically the same issue as ticket #1637, but it's a more serious example. In that one, the query simply failed to compile. In this case, it seems to work, but gives you the wrong results. I've also tried this script in PostgreSQL 8.0.13. All three queries give (the same) correct results. _2007-Aug-08 17:34:27 by anonymous:_ {linebreak} The problem is that SQLite is transforming SELECT * FROM Main LEFT JOIN OptA USING (pk) LEFT JOIN OptB USING (pk); into SELECT Main.pk, name, alpha, beta FROM Main LEFT JOIN OptA ON Main.pk = OptA.pk LEFT JOIN OptB ON OptA.pk = OptB.pk; Here is a workaround to this bug that makes use of a subquery: select * from (SELECT * FROM Main LEFT JOIN OptA USING (pk)) LEFT JOIN OptB USING (pk); Conceivably all LEFT JOIN chains could be transformed into the above form, but that would decrease performance due to the intermediate result set of the subquery. Having it work without the subquery is tricky since sqlite must deduce that the last USING (pk) is equivalent to the first pk in the chain of joined tables, namely Main.pk, and not OptA.pk. Joe Wilson #c8c8c8 2557 build closed 2007 Aug anonymous 2007 Aug 3 3 make test errors on Rhel4 x86_64 Hello there, While building sqlite 3.4.1 rpms for Rhel4 on i386 and x86_64, I came across some inconsistent behaviour with "make test". On Rhel4.5 i386, "make test" works fine. However, on Rhel4.5 x86_64, "make test" has two errors which terminate the rpm build process. rpmbuild -bb --with tcl sqlite.spec 2>&1 | tee build.log{linebreak} .{linebreak} .{linebreak} printf-7.5... Ok{linebreak} printf-8.1...{linebreak} Error: integer value too large to represent{linebreak} printf-8.2...{linebreak} Error: integer value too large to represent{linebreak} printf-8.3... Ok{linebreak} .{linebreak} .{linebreak} Thread-specific data deallocated properly{linebreak} 2 errors out of 29903 tests{linebreak} Failures on these tests: printf-8.1 printf-8.2{linebreak} make: *** [test] Error 1{linebreak} error: Bad exit status from /var/tmp/rpm-tmp.97578 (%check){linebreak} My current workaround is to apply a patch which is only applied on x86_64 hardware. Again, thanks for your time and great software. _2007-Aug-07 17:17:39 by drh:_ {linebreak} Neither test is particularly important and both failures problem result from bugs in either TCL or somewhere else in RH, not in the SQLite core. If you want me to look into this, please consider publishing your patch. I have no way to reproduce the problem otherwise. #cfe8bd 2556 todo fixed 2007 Aug anonymous 2007 Aug 5 5 tabs instead of space in some source files There isn't a ticket priority low enough for this item. A few source files use tabs instead of spaces. btree.c expr.c os_unix.c pager.c sqliteInt.h tclsqlite.c test_md5.c trigger.c vdbeaux.c #f2dcdc 2555 new active 2007 Aug anonymous 2007 Aug 1 1 FTS index without original text Is it possible to build FTS index without storing original text? I want to use fts index without features of snippets etc. I just want to find ID of the record not the content of indexed phrase. I suppose that the table myname_content stores this content. I have tried to update all columns of myname_content and set its values to “xyz” (without one column in which I store ID of the record). After this operation FTS search works good, but unfortunately the table isn’t smaller (I cant’t use vacuum on FTS tables). Is there any other way to have pure text indexes without source level changes? #cfe8bd 2554 code fixed 2007 Aug anonymous 2007 Aug 3 3 Many symbols not marked private/static or SQLITE_API in amalgamation I compiled the amalgamation with SQLITE_API=static and then ran "nm -D" on the resulting output which shows undefined (imported symbols) as well as all symbols that are publicly exported. There are numerous sqlite3 symbols that didn't get marked as SQLITE_API or SQLITE_private. Note that my ultimate goal is for two different versions of SQLite to coexist in the same process. This is especially important on the Mac where many of Apple's own components cause the system SQLite library to be loaded, but is also happening on other operating systems where SQLite is used by system libraries. Here is a real world example of the problems: http://lists.initd.org/pipermail/pysqlite/2007-June/001074.html Here is the begining of the list of public symbols from nm -D: 0000000000017460 T sqlite3BinaryCompareCollSeq 000000000005be20 R sqlite3IsIdChar 000000000026b040 D sqlite3OpcodeNames 000000000005bd20 R sqlite3UpperToLower 000000000005bce0 R sqlite3UtfTrans1 000000000000f0b0 T sqlite3_aggregate_context 000000000000b650 T sqlite3_aggregate_count 000000000026bf60 D sqlite3_apis 000000000001d380 T sqlite3_auto_extension 0000000000016cb0 T sqlite3_bind_blob 0000000000016660 T sqlite3_bind_double 0000000000016650 T sqlite3_bind_int 00000000000165e0 T sqlite3_bind_int64 0000000000016590 T sqlite3_bind_null 000000000000b690 T sqlite3_bind_parameter_count 0000000000020b30 T sqlite3_bind_parameter_index 000000000000b700 T sqlite3_bind_parameter_name 0000000000016ca0 T sqlite3_bind_text 0000000000016c80 T sqlite3_bind_text16 0000000000016520 T sqlite3_bind_value 00000000000164b0 T sqlite3_bind_zeroblob 000000000000b7a0 T sqlite3_blob_bytes 000000000002b790 T sqlite3_blob_close 00000000000478f0 T sqlite3_blob_open 000000000002b870 T sqlite3_blob_read 000000000002b860 T sqlite3_blob_write 000000000000d940 T sqlite3_busy_handler 000000000000da30 T sqlite3_busy_timeout 00000000000165a0 T sqlite3_clear_bindings 000000000000db60 T sqlite3_collation_needed 000000000000dbc0 T sqlite3_collation_needed16 0000000000017a30 T sqlite3_column_blob 00000000000180a0 T sqlite3_column_bytes 0000000000017ad0 T sqlite3_column_bytes16 000000000000b660 T sqlite3_column_count 0000000000016d60 T sqlite3_column_decltype 0000000000016d40 T sqlite3_column_decltype16 _2007-Aug-06 12:06:09 by anonymous:_ {linebreak} Just curious, what do you hope to do with the public API symbols? e.g., sqlite3_column_count. Rename them? ---- _2007-Aug-06 15:59:19 by anonymous:_ {linebreak} The sqlite3.c file is #included into my file that does all the database interfacing. There is no other file in my application that needs access to any sqlite code or data. If all the sqlite3 api had SQLITE_API in front of them (which I define to static) then that would work, but as it stands only some of them have SQLITE_API. ---- _2007-Aug-06 16:09:09 by anonymous:_ {linebreak} You could also load your 2nd sqlite3 code via the dlopen option RTLD_LOCAL. There must be a MacOS equivalent. ---- _2007-Aug-06 18:54:12 by anonymous:_ {linebreak} I am trying to solve this properly for all platforms and in the long term. Remember that I have no control over other libraries in the same process and how they link (and in any case they can link to another library that links to another that finally loads SQLite). SQLite is most of the way there to being totally self contained. It just needs SQLITE_API in front of all api functions, not just some of them and private internal sqlite data should really be marked as such. ---- _2007-Aug-06 23:32:37 by anonymous:_ {linebreak} Apply the patch below to the latest CVS sources, and run "./configure && make clean && make sqlite3.c". This will handle most of the public sqlite3 symbols. This patch addresses a bug in mksqlite3c.tcl that prevented putting SQLITE_API prefixes before the sqlite3_* functions which had more than one underscore in their name (such as sqlite3_aggregate_count), or had a '*' in an unexpected place in the return type (sqlite3BinaryCompareCollSeq). After the patch, these are the only public symbols in the library exposed: sqlite3IsIdChar sqlite3one sqlite3OpcodeNames sqlite3UpperToLower sqlite3_version sqlite3_io_trace sqlite3_mallocHasFailed sqlite3_os_trace sqlite3_temp_directory To prevent collision with another sqlite3 library, in your own C module start out your code as follows, changing 'foo' to a unique name: #define sqlite3IsIdChar foo_sqlite3IsIdChar #define sqlite3one foo_sqlite3one #define sqlite3OpcodeNames foo_sqlite3OpcodeNames #define sqlite3UpperToLower foo_sqlite3UpperToLower #define sqlite3_version foo_sqlite3_version #define sqlite3_io_trace foo_sqlite3_io_trace #define sqlite3_mallocHasFailed foo_sqlite3_mallocHasFailed #define sqlite3_os_trace foo_sqlite3_os_trace #define sqlite3_temp_directory foo_sqlite3_temp_directory #include "sqlite3.c" With this technique you can embed several different version of the sqlite3 library in the same program, assuming you compile your module with: -DSQLITE_API=static -DSQLITE_PRIVATE=static The sqlite3 sources could be reorganized to make these last few sqlite3 symbols private as well, but it would be a pain as these variables are used across several .c files. Patch below ran "make test" without error on Linux. Joe Wilson
Index: src/loadext.c =================================================================== RCS file: /sqlite/sqlite/src/loadext.c,v retrieving revision 1.23 diff -u -3 -p -r1.23 loadext.c --- src/loadext.c 20 Jul 2007 10:33:59 -0000 1.23 +++ src/loadext.c 6 Aug 2007 23:10:38 -0000 @@ -109,7 +109,7 @@ ** also check to make sure that the pointer to the function is ** not NULL before calling it. */ -const sqlite3_api_routines sqlite3_apis = { +static const sqlite3_api_routines sqlite3_apis = { sqlite3_aggregate_context, sqlite3_aggregate_count, sqlite3_bind_blob, Index: src/utf.c =================================================================== RCS file: /sqlite/sqlite/src/utf.c,v retrieving revision 1.52 diff -u -3 -p -r1.52 utf.c --- src/utf.c 23 Jul 2007 19:12:42 -0000 1.52 +++ src/utf.c 6 Aug 2007 23:10:39 -0000 @@ -49,7 +49,7 @@ const int sqlite3one = 1; ** This lookup table is used to help decode the first byte of ** a multi-byte UTF8 character. */ -const unsigned char sqlite3UtfTrans1[] = { +static const unsigned char sqlite3UtfTrans1[] = { 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, Index: tool/mksqlite3c.tcl =================================================================== RCS file: /sqlite/sqlite/tool/mksqlite3c.tcl,v retrieving revision 1.14 diff -u -3 -p -r1.14 mksqlite3c.tcl --- tool/mksqlite3c.tcl 26 Jun 2007 00:52:40 -0000 1.14 +++ tool/mksqlite3c.tcl 6 Aug 2007 23:10:39 -0000 @@ -129,10 +129,12 @@ proc copy_file {filename} { set tail [file tail $filename] section_comment "Begin file $tail" set in [open $filename r] + set declpattern \ + {([a-zA-Z][_a-zA-Z0-9* ]+)+(sqlite3[_A-Z][_a-zA-Z0-9]+)\(} if {[file extension $filename]==".h"} { - set declpattern {^ *[a-zA-Z][a-zA-Z_0-9 ]+ \*?(sqlite3[_A-Z][a-zA-Z0-9]+)\(} + set declpattern "^ *$declpattern" } else { - set declpattern {^[a-zA-Z][a-zA-Z_0-9 ]+ \*?(sqlite3[_A-Z][a-zA-Z0-9]+)\(} + set declpattern "^$declpattern" } while {![eof $in]} { set line [gets $in] @@ -154,7 +156,7 @@ proc copy_file {filename} { puts $out "#if 0" } elseif {[regexp {^#line} $line]} { # Skip #line directives. - } elseif {$addstatic && [regexp $declpattern $line all funcname] + } elseif {$addstatic && [regexp $declpattern $line all rtype funcname] && ![regexp {^static} $line]} { # Add the SQLITE_PRIVATE or SQLITE_API keyword before functions. # so that linkage can be modified at compile-time.---- _2007-Aug-07 20:51:20 by anonymous:_ {linebreak} The following symbols remain public after Check-in [4197]. missed SQLITE_PRIVATE symbols:
00003900 R sqlite3_apis 00000008 B sqlite3_io_trace 00000000 B sqlite3_mallocHasFailed 00000004 B sqlite3_os_trace 0000000c B sqlite3_temp_directory 00007c1b R sqlite3_versionmissed SQLITE_API symbols:
0001d6fb T sqlite3_aggregate_context 0001d8ed T sqlite3_aggregate_count 0003c3e3 T sqlite3_auto_extension 0001ded0 T sqlite3_bind_blob 0001df07 T sqlite3_bind_double ...etc... 96 more...This patch addresses the SQLITE_API lines above:
Index: tool/mksqlite3c.tcl =================================================================== RCS file: /sqlite/sqlite/tool/mksqlite3c.tcl,v retrieving revision 1.15 diff -u -3 -p -r1.15 mksqlite3c.tcl --- tool/mksqlite3c.tcl 7 Aug 2007 17:05:00 -0000 1.15 +++ tool/mksqlite3c.tcl 7 Aug 2007 20:06:34 -0000 @@ -130,10 +130,11 @@ proc copy_file {filename} { section_comment "Begin file $tail" set in [open $filename r] set varpattern {^[a-zA-Z][a-zA-Z_0-9 *]+ \*?(sqlite3[a-zA-Z0-9]+)([[;]| =)} + set declpattern {[a-zA-Z][a-zA-Z_0-9 ]+ \*?(sqlite3[_A-Z][_a-zA-Z0-9]+)\(} if {[file extension $filename]==".h"} { - set declpattern {^ *[a-zA-Z][a-zA-Z_0-9 ]+ \*?(sqlite3[_A-Z][a-zA-Z0-9]+)\(} + set declpattern "^ *$declpattern" } else { - set declpattern {^[a-zA-Z][a-zA-Z_0-9 ]+ \*?(sqlite3[_A-Z][a-zA-Z0-9]+)\(} + set declpattern "^$declpattern" } while {![eof $in]} { set line [gets $in]---- _2007-Aug-08 00:44:19 by drh:_ {linebreak} Symbols sqlite3_apis, sqlite3_temp_directory, and sqlite3_version are part of the published API and cannot be made private without breaking things. ---- _2007-Aug-08 01:05:24 by drh:_ {linebreak} Turns out sqlite3_io_trace is used externally too. I have checked in fixes for the rest. ---- _2007-Aug-08 01:41:14 by anonymous:_ {linebreak} If the test of the success of these qualifiers is to link different versions of the sqlite3 library into the same binary, then every function and variable must be marked as static, SQLITE_API or SQLITE_PRIVATE (or some other designation like SQLITE_API_VARIABLE, if you like). No function or variable can be without such a qualifier, i.e., public. If sqlite3_apis, sqlite3_temp_directory, and sqlite3_version are part of the published API then they should be marked as SQLITE_API, even though they are not functions. sqlite3_io_trace could be marked as either SQLITE_API or SQLITE_PRIVATE. I don't know which one is more appropriate. void (*sqlite3_io_trace)(const char*, ...) = 0; On Linux, with an updated CVS tree after Check-in [4199] we see that we have 4 public symbols remaining: make distclean ./configure make sqlite3.c gcc -DSQLITE_API=static -DSQLITE_PRIVATE=static -c sqlite3.c nm sqlite3.o | grep -v ' [trUbd] ' 00003900 R sqlite3_apis 00000000 B sqlite3_io_trace 00000004 B sqlite3_temp_directory 00007c1b R sqlite3_version ---- _2007-Aug-08 12:12:06 by drh:_ {linebreak} You now must also do -DSQLITE_EXTERN=static. #c8c8c8 2553 code closed 2007 Aug anonymous 2007 Aug 3 3 AUTOINCREMENT not working when primary key has multiple columns There is one thing which I really miss in SQLite from mySQL, and that is that AUTOINCREMENT-ing does not work when the primary key has multiple columns. AUTO_INCREMENT-ing should work when the last column of the PRIMARY KEY is of type INTEGER. It should work as described in the second example here: http://dev.mysql.com/doc/refman/5.0/en/example-auto-increment.html In essence, the final column of the primary key gets a value of 1 greater than the largest value in that column where all the other values of the primary key's columns match those of the newly entered row. Why is the useful? As an example, consider that bug reports are being made. The reports are updated. It would make sense for each update to be incrementally numbered within the particular bug that it belongs to. Thanks for considering this, Csaba Gabor from Vienna _2007-Aug-05 17:38:39 by drh:_ {linebreak} AUTOINCREMENT in SQLite works as it is documented to work in SQLite. SQLite is not MySQL. #c8c8c8 2552 code closed 2007 Aug anonymous 2007 Aug 1 3 sqlite3_column_double causes invalid floating point operation We have a table that has values inserted by a Delphi database provider. Using one tool, we can see values in the column as: 0 NULL 0.0 When using a different tool, or our .NET provider, conversion errors are generated. The conversion errors occur on the row that is stored as 0.0. The call stack seems to indicate that sqlite3_column_double is having a problem dealing with the value stored as 0.0. Some of my report is based on third party information and I haven't personally confirmed all details. Let me know if you have further questions. _2007-Aug-03 16:37:35 by anonymous:_ {linebreak} It's likely a bug in the sqlite wrapper library you are using. Post a small standalone C program that only calls the sqlite3 API directly to demonstrate the issue and re-open this ticket. ---- _2007-Aug-03 20:19:20 by anonymous:_ {linebreak} When viewing the data in the sqlite command line utility, it comes across as NaN. It appears that the problem is somehow on the input side. What's not clear is if the NaN was created before being bound to the database or if the NaN value was assigned by the database. #c8c8c8 2551 code closed 2007 Aug anonymous 2007 Aug 3 3 sqlite3 shell gets into bad state with unexpected input The sqlite3 shell can get into a state that you cannot recover from without aborting the process. This happens fairly often on platforms without readline support. 1: build sqlite3 without readline support. 2: run sqlite3 in an xterm (or KDE konsole, probably other terminal programs) 3: at the prompt, press cursor up (probably other 'illegal' characters would suffice as well). 4: hit return Nothing you do at this point will work except for Ctrl-\ to kill the process.
SQLite version 3.4.1 Enter ".help" for instructions sqlite> ^[[A ...> .q ...> ; ...> select 1; ...> .q ...> .h ...> ; ...> ;_2007-Aug-03 15:00:30 by anonymous:_ {linebreak} Try CTRL+D, worked for me. ---- _2007-Aug-03 21:19:31 by drh:_ {linebreak} The "[A" is the beginning of a quoted identifier. The shell continues to issue the continuation prompt because it is waiting on the "]" to terminate the identifier. Your workaround is to type "];". Works as designed. #cfe8bd 2550 code fixed 2007 Aug anonymous 2007 Oct drh 3 3 Lemon parser may accept input before EOF is seen A lemon parser with table compression may accept input before EOF is seen. This can happen even without any syntax errors being reported. Consider this little lemon grammar: %parse_accept{printf("parse accept\n");} %syntax_error{printf("syntax error\n");} %parse_failure{printf("parse failure\n");} start ::= foobar. foobar ::= FOO. foobar ::= foobar BAR. This grammer describes input that matches the regular language FOO BAR*, i.e. one FOO followed by zero or more BARs. This grammar shows how lemon breaks: When this grammar is fed the input FOO BAR FOO FOO BAR, it will accept twice: the second FOO acts as the EOF terminal after the initial FOO BAR and gets eaten by the parser without error. The parser then accepts and resets itself for new input. The remainder of the input is again a valid FOO BAR, giving rise to the second accept. More precisely, lemon has the following logic error: The accept action is triggered at the end of a reduce action of a start rule (that has the start symbol at the left hand side), when that start symbol is about to be pushed onto the stack. This (normally) works because the start symbol may not be used on the right hand side of a rule, so EOF can be the only follower terminal of a start rule. Unfortunately, in the presence of default rules, a start rule can be used as the default rule, and it can be reduced even when the lookahead symbol is not EOF. I see two ways of fixing this: 1:Before accepting, make sure the lookahead symbol is really EOF. 1:Refuse to reduce a start rule without the lookahead symbol actually being EOF. Option 1 seems nice, but is not really user friendly with lemon as it has been set up now: the accept action is not really usable as the final indication that all is well, because it has no access to the parse tree in the minor value of the start symbol for returning to the calling program (through the extra parameter). With option 2, reduction of a start rule is really the last action of the parser (alright, except for the parse_accept action). Normally, I'd say a user uses the start rules for returning the parse tree to the lemon caller, and this is the more naturaly way to do it. Option 2 has an additional advantage that, in combination with judicious placement of error symbols in the start rule, it remains possible to parse remaining input. This would not be possible if a start rule were reduced prematurely as under option 1, because then the parser would have to go to a state where it discards all remaining input (or it would have to call parse_failure) without actually parsing the remaining input. Ciao. Vincent. _2007-Sep-27 08:42:25 by anonymous:_ {linebreak} YES!!! I have this error too :-(( Please ... fix it. Thanks, Alexander. #c8c8c8 2549 build closed 2007 Aug anonymous 2007 Aug 1 3 3.4.1 fails to upgrade from 3.3.1.7 on freeBSD I use DesktopBSD (freebsd), and I was running the regular software upgrade using the Package Manager. This is the result: ./.libs/libsqlite3.so: undefined reference to `sqlite3Fts2InitHashTable' gmake: *** [sqlite3] Error 1 *** Error code 2 Stop in /usr/ports/databases/sqlite3. *** Error code 1 Stop in /usr/ports/databases/sqlite3. ** Command failed [exit code 1]: /usr/bin/script -qa /tmp/portupgrade.64421.1 env DIALOG=/usr/local/bin/kdialog-ports-wrapper UPGRADE_TOOL=portupgrade UPGRADE_PORT=sqlite3-3.3.17 UPGRADE_PORT_VER=3.3.17 make ** Fix the problem and try again. Regards, Arthur This is a bug for the FreeBSD ports maintainers. #cfe8bd 2548 code fixed 2007 Aug anonymous 2007 Oct anonymous 4 4 Pelles C compiler for Windows ANSI prototype / K&R declaration errors Compiling for Windows using the PellesC compiler suite. Mismatches between ANSI style prototypes and K&R style function declarations are reported as errors. All the error reports are of the form: ...\main.c(1235): error #2120: Redeclaration of 'sqlite3_global_recover' previously declared at ...\sqlite3.h(1693): found 'void _cdecl function' expected 'void _stdcall function(void)'. 1. In main.c at line 1235 need to change int sqlite3_global_recover(){ to int sqlite3_global_recover(void){ 2. In malloc.c at line 809 need to change int sqlite3FailedMalloc(){ to int sqlite3FailedMalloc(void){ 3. In os_win.c at line 1613 need to change int sqlite3WinEnterMutex(){ to int sqlite3WinEnterMutex(void){ 4. In os_win.c at line 1630 need to change int sqlite3WinLeaveMutex(){ to int sqlite3WinLeaveMutex(void){ 5. In util.c at line 706 need to change int sqlite3ThreadData(){ to int sqlite3ThreadData(void){ 6. In util.c at line 719 need to change int sqlite3ThreadDataReadOnly(){ to int sqlite3ThreadDataReadOnly(void){ 7. In util.c at line 730 need to change int sqlite3ReleaseThreadData(){ to int sqlite3ReleaseThreadData(void){ ----end of report---- #f2dcdc 2547 code active 2007 Aug danielk1977 2007 Aug 5 3 Changing db encoding of an attached db can confuse shared cache mode. This is quite obscure, but in shared-cache mode: 1) Open db A, attach empty db B. 2) Using another connection from the same thread, set the encoding of B to be different from that of A. Add some data to B. 3) Using the original connection, access database B. It assumes the encoding of A (and therefore mangling any text data). The correct response is to return an error - "attached databases must use the same text encoding as main database". #cfe8bd 2546 code fixed 2007 Jul anonymous 2007 Oct 4 1 Comments are not parsed correctly When I try to insert a new record in a table with the .read command , I cannot load data whose line terminate with a comment. I always get an "Incomplete SQL" error. file content: INSERT INTO _users_parameters VALUES (1,1); -- temperature sqlite> .read smartdas.sql Incomplete SQL: INSERT INTO _users_parameters VALUES (1,3); -- temperature file content: INSERT INTO _users_parameters VALUES (1,1); /* temperature*/ sqlite> .read smartdas.sql Incomplete SQL: INSERT INTO _users_parameters VALUES (1,3); /* temperature */ If I remove the comment, then the row is inserted as expected: file content: INSERT INTO _users_parameters VALUES (1,1); sqlite> .read smartdas.sql sqlite> select * from _users_parameters; 1|1 Thanks, Paolo Saudin #f2dcdc 2545 code active 2007 Jul anonymous 2007 Jul 1 4 Group by returns table name with field name imaginate a table:
create table test (
id INTEGER PRIMARY KEY,
name varchar(50) not null,
age integer not null
);
Then: insert into test (name,age) values ('foo',22);
insert into test (name,age) values ('foo',23);
insert into test (name,age) values ('bar',22);
insert into test (name,age) values ('bar',35);
insert into test (name,age) values ('bar',72);
Now try this; sqlite> .headers on
sqlite> select test.name, test.age from test order by name;
name|age
bar|22
bar|35
bar|72
foo|22
foo|23
sqlite> select test.name, test.age from test group by name;
test.name|test.age
bar|72
foo|23
You see ? if i use "GROUP BY", the field name contains tablename. Because i use "SELECT test.name" and not "SELECT name". If i set an alias, i get alias, that's ok. The trouble appears to be very high on Copix (http://wwW.copix.org). We create some DAO (Data Access Objects) automatically. The "groupBy" method doesn't works with SQLite... Is this normal ? Mysql, PostgreSql, Oracle... doesn't need to create alias. _2007-Jul-31 15:54:52 by anonymous:_ {linebreak} There's probably 4 other tickets reporting this. I don't think it will get fixed. The workaround is to use aliases (AS "whatever") for the selected columns. ---- _2007-Jul-31 23:02:19 by anonymous:_ {linebreak} Ok, we have created a special support for SQLite. PS: I love this database :) Simple, nice, usefull, quick and easy Regards
#cfe8bd 2544 code fixed 2007 Jul rse 2007 Jul 4 3 "sqlite3 foo.db .dump | sqlite3 bar.db" fail because of verbose output A simple "sqlite3 foo.db .dump | sqlite3 bar.db" for upgrading an old database file failed because of the verbose output "Loading resources from [...]/.sqliterc". Although one can use the workaround "sqlite3 -batch foo.db .dump", this is annoying as one stumbles too easily and without any real need over this unimportant verbose output message. I propose two solutions: Either add a similar istty(3) check to shell.c which is already present for stdin or simply drop the verbose message at all. As I cannot decide what is wished here, I will append a patch at least for the istty(3) solution. _2007-Jul-30 19:02:51 by anonymous:_ {linebreak} I agree the warning message is useless. Rather than breaking sqlite3 on platforms without isatty support, please do one of the following instead: *: Get rid of the line altogether *: -- put a comment in front of the line *: if -batch is specified on the command-line, do not print out this line A workaround: sqlite3 -batch -init ignore foo.db .dump | sqlite3 bar.db ---- _2007-Jul-30 19:26:20 by rse:_ {linebreak} The istty(3) is already used by SQLite, so the additional use would make no problem. -- rse
#f2dcdc 2543 code active 2007 Jul anonymous 2007 Jul 1 1 Chinese charset not support?? when i create a table. the table name is " " (chinese) after this "alter table add column aaa text null" error why??/ thank you
#c8c8c8 2542 code closed 2007 Jul anonymous 2007 Jul 2 1 select 3/2 * avg(x) gives wrong result Try this. It gives 2 but obviously the answer is 3 since the average of 1,2,3 is 2 and multiplying that by 3/2 gives 3. C:\tmp2>sqlite3 SQLite version 3.4.0 Enter ".help" for instructions sqlite> sqlite> create table tab (x num); sqlite> insert into tab values (1); sqlite> insert into tab values (2); sqlite> insert into tab values (3); sqlite> select * from tab; 1 2 3 sqlite> select sqlite_version(*); 3.4.0 sqlite> sqlite> select 3/2 * avg(x) from tab; 2.0_2007-Jul-30 04:10:09 by anonymous:_ {linebreak} Sorry, please ignore this one. I just released its doing integer arithmetic. #c8c8c8 2541 doc closed 2007 Jul anonymous 2007 Jul danielk1977 4 4 ALTER TABLE
static int winClose(OsFile **pId){ winFile *pFile; int rc = 1; if( pId && (pFile = (winFile*)*pId)!=0 ){ int rc, cnt = 0; OSTRACE2("CLOSE %d\n", pFile->h); do{ rc = CloseHandle(pFile->h); }while( rc==0 && cnt++ < MX_CLOSE_ATTEMPT && (Sleep(100), 1) ); #if OS_WINCE winceDestroyLock(pFile); // fix begin if( pFile->zDeleteOnClose ){ DeleteFileW(pFile->zDeleteOnClose); sqliteFree(pFile->zDeleteOnClose); } // fix end #endif OpenCounter(-1); sqliteFree(pFile); *pId = 0; } return rc ? SQLITE_OK : SQLITE_IOERR; }Thanks, Fedor _2007-Jul-28 16:41:41 by anonymous:_ {linebreak} The solution is to revert checkin 3836 and re-open ticket #2294.
Looking at the wince locking mechanism, the only time we ever use the zDeleteOnClose flag is when we've opened a database for exclusive access in sqlite3WinOpenExclusive. To save time and resources (and because its not necessary) we never bother creating a locking mechanism for exclusively-opened files. So pFile->hMutex is NULL when hitting winceDestroyLock(), and the file is never deleted.
Is it possible that the original poster of #2294 was trying to close the same connection on multiple threads at the same time? ---- _2007-Jul-31 05:32:39 by anonymous:_ {linebreak} This is actually a duplicate of #2533 ---- _2007-Sep-21 14:20:05 by anonymous:_ {linebreak} So when the fix of [3836] was applied, the code to delete the file was only put in the section that is called when we have a mutex. I wonder, if the deletion of the file should also take place if there was no mutex. Works for me at least: static void winceDestroyLock(winFile *pFile){ if (pFile->hMutex){ /* Acquire the mutex */ winceMutexAcquire(pFile->hMutex); /* The following blocks should probably assert in debug mode, but they are to cleanup in case any locks remained open */ if (pFile->local.nReaders){ pFile->shared->nReaders --; } if (pFile->local.bReserved){ pFile->shared->bReserved = FALSE; } if (pFile->local.bPending){ pFile->shared->bPending = FALSE; } if (pFile->local.bExclusive){ pFile->shared->bExclusive = FALSE; } /* De-reference and close our copy of the shared memory handle */ UnmapViewOfFile(pFile->shared); CloseHandle(pFile->hShared); * if( pFile->zDeleteOnClose ){ * DeleteFileW(pFile->zDeleteOnClose); * sqliteFree(pFile->zDeleteOnClose); * pFile->zDeleteOnClose = 0; * } /* Done with the mutex */ winceMutexRelease(pFile->hMutex); CloseHandle(pFile->hMutex); pFile->hMutex = NULL; } + else + { + if( pFile->zDeleteOnClose ){ + DeleteFileW(pFile->zDeleteOnClose); + sqliteFree(pFile->zDeleteOnClose); + pFile->zDeleteOnClose = 0; + } + } } The code marked with * was put there in
#c8c8c8 2538 code closed 2007 Jul anonymous 2007 Jul 1 1 calling sqlite3_interrupt() randomly leaves db->ul.isInterrupted = 1 Using a sequence as follows, all invoked with sqlite3_exec(): BEGIN; ... long running script with many UPDATE/INSERT statements (with triggers firing, etc)... sqlite3_interrupt() is called Often (always?) the next sqlite3_exec call will also return SQLITE_INTERRUPT instead of SQLITE_OK. Apparently there is at least one point in sqlite execution that can be interrupted with sqlite3_interrupt() where db->ul.isInterrupted is not reset to 0, so the next sqlite3_exec() call also to returns SQLITE_INTERRUPT. Since the documentation states that calling sqlite3_interrupt() will rollback any EXPLICIT transactions (why specifically EXPLICIT ?? - I would assume auto [IMPLICIT] transactions would obviously also rollback with a sqlite3_interrupt call), any subsequent action done on the sqlite instance should be SQLITE_OK (or the error specific to that action since the previous call has been fully aborted). This behavior is definitely different than 3.2.8 (the version I am attempting to upgrade from), which works as expected. After much additional testing, I have concluded that this may actually have been due to my error (sqlite3_interrupt() being called again) and not due to sqlite. Sorry for the wasted bandwidth. Please consider the issue closed.
#c8c8c8 2537 code closed 2007 Jul anonymous 2007 Jul 4 4 sqlite3_column_type() function problem. Hi, -I prepare a sql query statement ex: Select * from Table1. -I want read some metadata from all fields of the result-set before fetch the first row with the function sqlite3_step(...). -sqlite3_column_name(...), sqlite3_column_origin_name(...), sqlite3_column_table_name(...) are all Ok but sqlite3_column_type(...) returns for all columns -> 5 (NULL). After the first sqlite3_step(...) the type is correct for all columns. _2007-Jul-25 23:07:25 by drh:_ {linebreak} The type can vary from one row to the next. Works as designed.
#c8c8c8 2536 code closed 2007 Jul anonymous 2007 Aug 2 3 Possible non-ID index corruption when max_page_count exceeded Using 3.4.0 or 3.4.1 compiled for windows with -DMULTITHREAD, i seem to have a problem where sqlite3_step() will return SQLITE_ROW even when there are, in fact, no rows that meet the WHERE clause criteria. I'm currently attempting a reversion to 3.3.17 to see how it behaved. create table file_data ( rod blob not null primary key /* always 20 bytes long based on a collision-resistant hash function */ , file_size integer not null , compression_method integer not null , the_data blob not null ); sqlite3_prepare(..., "select file_size, compression_method from file_data where rod = ?"); sqlite3_reset(statement); sqlite3_bind_blob(...,0, ptr, len); sqlite3_step(...) sqlite3_step returned SQLITE_ROW, which was my indication that the result set would be non-empty. So I construct a query_result(statement, bool at_end=false), and then execute sqlite3_get_int64(..., first_column, null=-1) and I get back -1. Before I was passing in null=-1 I was passing in =0, which was a permissible value for file_size. ** keep in mind that my exact sqlite3_api calls might be slightly different, since I have a C++ framework that isolates me from having to continually remember the same API calls. Suffice it to say this code worked (so far as I knew) under 3.3.17. --andy On further research, it appears to be related to max_page_count being exceeded during insertion of a row, leaving behind a primary key reference in the (primary key) index on the BLOB instead of being cleaned up during rollback. _2007-Jul-25 17:38:38 by drh:_ {linebreak} Unable to reproduce. I am unable to make a lot of sense of the description above. I have tried various tests setting a low max_page_count and inserting BLOB-keyed entries into a table until the number of pages overflow, then run sqlite3_step to see what happens. Everything works correctly. I tried lots of different combinations but never could get a failure. ---- _2007-Jul-25 19:17:19 by anonymous:_ {linebreak} MULTITHREAD is the wrong macro, but sqlite compilation defaults to threadsafe anyway. How many threads are you running when this thing happens and what's your relationship of sqlite connections to threads?
#c8c8c8 2534 code closed 2007 Jul anonymous 2007 Jul danielk1977 2 3 sqlite3_set_auxdata may cause memory to be leaked The routine _sqlite3_set_auxdata()_ may cause memory allocated by the caller to be leaked. Specifically, if the routine fails to allocate memory for its internal array of struct AuxData then the new argument data will not be stored. If the caller included an xDelete callback, this callback will not be stored or called in this circumstance. Given that _sqlite3_set_auxdata()_ does not return an indicator of success or failure, if the routine fails and a xDelete callback has been supplied then the callback routine should be executed before returning to the caller. Makes sense to me.
#cfe8bd 2533 code fixed 2007 Jul anonymous 2007 Oct 1 2 temporary files are not deleted on WinCe temporary files are not deleted on WinCe if they are not locked. hMutex is false in struct pFile. function: static void winceDestroyLock(winFile *pFile) line: 16490 in sqlite3.c from sqlite-amalgamation-3_4_1.zip problem: delete temporary files only if pFile->hMutex is true if (pFile->hMutex){ ... if( pFile->zDeleteOnClose ){ DeleteFileW(pFile->zDeleteOnClose); sqliteFree(pFile->zDeleteOnClose); pFile->zDeleteOnClose = 0; } ... } solution: delete temporary files not dependend if pFile->Mutex is true or not if (pFile->hMutex){ ... ... } if( pFile->zDeleteOnClose ){ DeleteFileW(pFile->zDeleteOnClose); sqliteFree(pFile->zDeleteOnClose); pFile->zDeleteOnClose = 0; } regards Juergen Wolters Temporary files never create a lock object, so the hMutex member is always null. Since there is no mutex to cleanup, winceDestroyLock() is never called and temporary files are never deleted. Checkin [3836] needs to be reverted and the original ticket which triggered it re-opened.
#c8c8c8 2532 code closed 2007 Jul anonymous 2007 Jul 3 4 Endless loop in Parser? After pasting the statement: " create trigger brandattribute_adtr after delete on brand_attribute for each row begin delete from customattributetype where customattributetype.customattributetype_id == old.customattributetype_id end; " the Sqlite command prompt remains in ...> level. I don't know any solution to solve it except typing PRAGMA auto_vacuum=incremental
setting is not persistent.{linebreak}{link: http://www.sqlite.org/changes.html#version_3_4_1 Changes page} claims the opposite.
#c8c8c8 2527 doc closed 2007 Jul anonymous 2007 Jul 4 2 sqlite3_create_module() API documentation missing Documentation for sqlite3_create_module() is missing on page {link: http://www.sqlite.org/capi3ref.html C/C++ API Reference}. The same goes for new sqlite3_create_module_v2(). _2007-Jul-24 10:06:33 by drh:_ {linebreak} This is because the whole virtual table mechanism is still considered highly experimental and subject to change. By writing up an interface in the C-API documentation, we promise to support it from now on. No such promises are made with virtual tables.
#cfe8bd 2526 code fixed 2007 Jul anonymous 2007 Jul 2 4 Core dump when issuing a select statement with special having clause When executing the attached script, I get a core dump regarding freeing an invalid pointer. Steps to reproduce: python breaks.py # breaks.py is found below Comments: The error is probably related to the fact that having a having clause with "having cn < max(count(author))" breaks with "missuse of aggregate function count()", but aliasing the column count(author as cn) breaks the checks, and instead the program crashes. I submitted the bug here instead of to pysqlite because the breakage also happens with the command line version of sqlite3. _2007-Jul-23 22:14:11 by drh:_ {linebreak} The following script demonstrates the problem. No need for the python complication: create table t1 (a text, b text, c primary key); insert into t1 values ('x', 'y', NULL); insert into t1 values ('x', 'z', NULL); select count(a) as cn from t1 group by a having cn < max(cn);
#cfe8bd 2525 code fixed 2007 Jul anonymous 2007 Jul 1 3 Constraint violations fail silently with "insert or replace" The following code (file "correls.sql") correctly raises a constraint violation: create table if not exists correls ( sym1 varchar, sym2 varchar, correl real, primary key (sym1, sym2), check (correl >= -1), check(correl <= 1)); insert into correls values ("a", "b", -2); Here is the output: # sqlite3 correls.db < correls.sql SQL error near line 5: constraint failed But running the same code with "insert or replace" instead of a plain "insert" will fail silently. The data won't be inserted or updated so integrity is still maintained, but the user will not know that the transaction wasn't successful.
#cfe8bd 2524 code fixed 2007 Jul anonymous 2007 Jul 4 4 Missing error check in call to sqlite3PagerWrite() in autoVacuumCommit The Coverity checker found this minor issue. Most of the calls to sqlite3PagerWrite() have the return value checked and appropriate action taken if the call fails. In btree.c, line 2012, there's a call to sqlite3PagerWrite() without a check. Is this a case where the call can't fail, or are we really missing error handling code here?
#cfe8bd 2523 code fixed 2007 Jul anonymous 2007 Jul 3 4 Leaked memory on error return from sqlite3VdbeMemTranslate In sqlite3VdbeMemTranslate in utf.c, line 214 has an allocation of memory for zOut. Later, at line 265, there's an early return if memory allocation fails for zExtra. If that return is taken, the allocation for zOut is lost, since it's not been assigned to an outbound variable. This does seem like a rare case, but the fix is easy -- just add the right deallocation here if zOut != zShort.
#cfe8bd 2522 code fixed 2007 Jul anonymous 2007 Jul 4 4 wrong sizeof in vdbe.c Harmless, but this can waste memory on platforms where sizeof(int*) > sizeof(int). vdbe.c: - aRoot = sqliteMallocRaw( sizeof(int*)*(nRoot+1) ); + aRoot = sqliteMallocRaw( sizeof(int)*(nRoot+1) );
#cfe8bd 2521 code fixed 2007 Jul anonymous 2007 Jul 2 4 Misplaced prototypes When compiling the amalgamated sqlite3.c, the Digital Mars compiler produces the error sqlite3.c(67876) : Error: no definition for static 'sqlite3ParserFree' Moving the function prototypes at sqlite3.c lines 66302 and 66303 SQLITE_PRIVATE void *sqlite3ParserAlloc(void*(*)(size_t)); SQLITE_PRIVATE void sqlite3ParserFree(void*, void(*)(void*)); to a position before the respective procedures eliminates the problem. Robert Wishlaw
#f2dcdc 2520 new active 2007 Jul anonymous 2007 Jul 4 1 User defined aggregate functions are not reentrant When an aggregate function is defined using sqlite3_create_function, it is not possible to execute any sql statement inside the step part or the finalizer. This is due to the fact that aggregate functions are not reentrant. _2007-Jul-20 02:32:36 by anonymous:_ {linebreak} related: Ticket #2242: sqlite3 authorizer is not reentrant
#c8c8c8 2519 new closed 2007 Jul anonymous 2007 Oct 1 1 Page cache and schema should be shared between all connections/threads The current shared cache scheme is too restrictive to use easily in a multi-threaded application. The page cache should be shared between all connections/threads, as the database file itself is a global resource. If the schema is changed in one thread connection, all other thread connections should pick up the change automatically. Mutexes should be used for coordination as appropriate. Advantages of thread-agnostic shared cache/schema approach: *: reduced memory use due to cache sharing in a multi-threaded/multi-connection program *: simpler usage model for multiple connections so that the programmer need not be concerned with what thread created the connection, or using starting a dummy transaction with an insert into a dummy table in order to {link: http://developer.mozilla.org/en/docs/Storage:Performance#Keeping_the_cache_between_transactions keep the cache between transactions} *: eliminate the schema/thread/connection mismatch problem #2486 *: fewer system resources - use just a single file descriptor for multiple connections to the same database file, since it is a global resource. This would avoid file descriptor problems related to the Linux 2.4 kernel. *: improved concurrency for connections in multiple threads _2007-Jul-26 18:39:20 by anonymous:_ {linebreak} the database file is a global resource, the sqlite handle is not. imagine if you start a transaction in a thread then another thread start another transaction... if you use two sqlite handles, this is ok, because you can lock a file handle for the transactions. since sqlite doesn't provide nested transactions, you should avoid thinking a sqlite handle object is a global resource. this is explicit well documented. ---- _2007-Jul-26 19:49:57 by anonymous:_ {linebreak} That's the problem with the current implementation. There is no need for multiple file descriptors to the same database file, since it is a global resource. There should be only a single file handle shared between all threads and connections. Access to this common file handle should be coordinated via proper mutexes between threads. ---- _2007-Jul-26 20:11:09 by anonymous:_ {linebreak} right again, neither ADO / SQL Server provider in ADO / Microsoft world suport a shared handle (connection object) properly. nobody implements this just because tx control should be done by program, not by database layer driver. ---- _2007-Jul-26 20:19:24 by anonymous:_ {linebreak} Whether there is 1 connection or 50 connections to the same database, the *database file page cache*, a global resource, must be the same in all cases. Don't confuse the page cache with a journal file. ---- _2007-Jul-26 20:45:01 by anonymous:_ {linebreak} suppose you have a main program and a shared library. the two linked static against a sqlite amalgamation, with different versions (e.g. 3.3.14 main program and 3.4.1 shared library). if the pager cache internal structures changed against versions (which could happen), sqlite is going to be crazy and doing worst. also, not all os platforms works like other in memory management. they could have different memory virtual address (in windows, dll memory is protected from program memory if it uses the malloc allocator, with globalalloc things performs differently). since sqlite is multi-platform, it's not cool to put a per-platform feature. a 'possible' (?) workaround is to use a per-process shared memory to implement this, but not all os'es sqlite compile and run may support it to implement this design. ---- _2007-Jul-26 20:53:27 by anonymous:_ {linebreak} The proposed in-process shared database page cache has nothing to do with virtual shared memory or shared libraries. If the in-memory database change counter matches that of the locked database file, we can assume that the in-memory database page cache is current - regardless of whatever sqlite3 connection or thread we're on. ---- _2007-Aug-10 20:33:49 by anonymous:_ {linebreak} using the same file descriptor should not work. locking engine in sqlite is file-locked base. testing in a process within various threads if you could lock a region of the file will not work if sqlite is changed this way. again, not usual and you should realize you can use various sqlite versions inside the same program using amalgamations. ---- _2007-Aug-10 21:25:12 by anonymous:_ {linebreak} Of course using a single file descriptor would work. There is only 1 underlying database file - why on earth do you need more than 1 descriptor to access it? That's what mutexes are for. Regarding putting multiple sqlite3 engines in the same binary, no one has ever done this before, as it was not even possible until 2 days ago (Ticket #2554). But if you wanted to support the old style multi-descriptor file locking, it could be done with an appropriate compile flag -DOMIT_SINGLE_FILE_DESCRIPTOR like every other feature in sqlite. ---- _2007-Aug-10 23:36:34 by anonymous:_ {linebreak} you didn't get the issue. suppose (in my case), I have two DLL´s (shared libraries) that are linked statically against sqlite. if I build a updated DLL with different versions of SQLite, e.g., 3.2.5 (on old dll´s) and 3.4.1 (on the updated dll), sqlite in-memory structures would not be the same layout (new fields, features, etc). this is very dangerous in my point of view. I preffer paying the penalty reading from disk and keep things going smoothly. ---- _2007-Aug-11 00:46:16 by anonymous:_ {linebreak} Okay, keep it running smoothly. ---- _2007-Oct-04 01:02:32 by anonymous:_ {linebreak} Must manually enable shared cache via: http://www.sqlite.org/capi3ref.html#sqlite3_enable_shared_cache
#cfe8bd 2518 code fixed 2007 Jul anonymous 2007 Jul 2 1 Complex paging problem We are experiencing a very strange and complex problem with SQLite which is very difficult to reproduce. Basically 1. We run some bits to set up the DB. 2. Populate the data base with a load of information on one thread. 3. Then running a query on a different thread: SELECT DISTINCT("SessionID") FROM "mfForm_Welcome" WHERE UID IN (-1, 1) AND "D3_TXTCODE" LIKE "1" ORDER BY SessionID DESC; If the query fails in other words there is no matching data in the table the problem happens if it succeeds the problem doesn't seem to happen. 4. Then we run the function to populating the DB with a load of information. Again on a different thread. As 2. 5. Then running the query again which should succeed fails. SELECT DISTINCT("SessionID") FROM "mfForm_Welcome" WHERE UID IN (-1, 1) AND "D3_TXTCODE" LIKE "1234" ORDER BY SessionID DESC; However if we do this: INSERT INTO mfForm_Welcome (SessionID) VALUES (NULL); DELETE FROM mfForm_Welcome WHERE SessionID IS NULL; Immediately prior to running the query for the second time it succeeds. If we run a query that succeeds at step 2. then everything seem to work ok from then on. It also seems to matter if there are other tables in the DB prior to the one the query is run on. I have been debugging in SQLite and looks to me that the paging system hasn't re-got the data or is in some other way incorrect. In other words the paging stuff isn't reflecting what's on the DB, Possibly it isn't getting invalidated or something. I also got a trace of all the SQL run by the app up to this point. Running this SQL in isolation works OK (Doesn’t give the problem) so probably something to do with having two threads. Sorry if this is a bit hard to follow but we would much appreciate help. SQL Trace: Step 1. This is on thread 1 CREATE TABLE "mfSysUsers" ("ID" numeric PRIMARY KEY , "Username" text (20) , "Password" text (20) ) CREATE TABLE "mfSysUserPermissions" ("UserID" numeric PRIMARY KEY , "StartUpForm" text (128) , "Flags" numeric ) CREATE TABLE "mfSysUsersToForms" ("PermissionID" numeric PRIMARY KEY , "UserID" numeric , "FormTitle" text (128) , "Flags" numeric ) CREATE TABLE mfSysSyncLog (ID integer IDENTITY PRIMARY KEY, UID integer, Sync nvarchar(19), Type integer, Records integer, FormName nvarchar(32), FailureReason integer) CREATE TABLE mfSysTrackingLog (ID integer IDENTITY PRIMARY KEY, UID integer, LogIn nvarchar(19), LogOut nvarchar(19), FormName nvarchar(32), Session integer) CREATE TABLE "mfForm_Welcome" ("SessionID" numeric PRIMARY KEY , "UID" numeric , "Flags" numeric , "Expiry" numeric , "Spare" numeric , "D3_txtAccessCode" text , "D3_txtMsgDay" text , "D3_txtUser" text , "D3_txtDesc" text , "D3_txtH" text , "D3_txtErr" text , "D3_txtDontSend" text , "D3_DEVICE" text , "D3_txtCode" text ) CREATE TABLE "mFSysVersion" (asdasd CHAR) CREATE INDEX 'IndexUserName_mfSysUsers' ON 'mfSysUsers' (Username ASC ) CREATE INDEX 'IndexUserID_mfSysUsers' ON 'mfSysUsers' (ID DESC ) CREATE INDEX [IdxTrackingLog] ON [mfSysTrackingLog] (ID) SELECT Username FROM mfSysUsers; SELECT Username FROM mfSysUsers; SELECT Username FROM mfSysUsers WHERE Username='Guest' SELECT Username FROM mfSysUsers WHERE Username='Guest' SELECT Password FROM mfSysUsers WHERE Username='Guest' SELECT Password FROM mfSysUsers WHERE Username='Guest' SELECT ID FROM mfSysUsers WHERE Username='Guest' SELECT ID FROM mfSysUsers WHERE Username='Guest' SELECT StartUpForm, Flags FROM mfSysUserPermissions WHERE UserID = 1; SELECT StartUpForm, Flags FROM mfSysUserPermissions WHERE UserID = 1; CREATE TABLE "mfForm_Welcome" ("SessionID" numeric PRIMARY KEY , "UID" numeric , "Flags" numeric , "Expiry" numeric , "Spare" numeric , "D3_txtAccessCode" text , "D3_txtMsgDay" text , "D3_txtUser" text , "D3_txtDesc" text , "D3_txtH" text , "D3_txtErr" text , "D3_txtDontSend" text , "D3_DEVICE" text , "D3_txtCode" text ); SELECT MAX(SessionID) FROM "mfForm_Welcome"; SELECT MAX(SessionID) FROM "mfForm_Welcome"; INSERT INTO mfSysTrackingLog (UID, Login, Session, FormName) VALUES (1, '2007-07-17 11:39:25', 1, 'Welcome'); SELECT MAX(ID) FROM mfSysTrackingLog; SELECT MAX(ID) FROM mfSysTrackingLog;
Step 2. This is on thread 2 BEGIN; SELECT SessionID FROM "mfForm_PartFields" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_PartFields" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_testaa" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_testaa" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_testbb" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_testbb" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Issue Photos" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Issue Photos" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Inspections Guide" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Inspections Guide" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Street Inspections" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Street Inspections" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Street Issues" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Street Issues" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Street Inspections List" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Street Inspections List" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Inspection Photos" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Inspection Photos" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Menu" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Menu" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_IssuePCN" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_IssuePCN" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_IssuePCNStartUp" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_IssuePCNStartUp" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_PCNNoGenerator" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_PCNNoGenerator" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_PCNShift" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_PCNShift" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_DogFouling" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_DogFouling" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_OffenceList" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_OffenceList" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_PCNShiftDetails" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_PCNShiftDetails" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_IssuePCNMain" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_IssuePCNMain" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_AuditsToClaim" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_AuditsToClaim" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Check Item" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Check Item" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Audit" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Audit" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_AuditClaim" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_AuditClaim" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_AIMS" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_AIMS" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Main" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Main" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_CheckItemActions" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_CheckItemActions" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_HRA" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_HRA" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_TEST LISTS2" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_TEST LISTS2" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_OrderList_OL" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_OrderList_OL" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Statement_ST" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Statement_ST" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_LastOrderList" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_LastOrderList" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_StatementData_SD" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_StatementData_SD" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Receipt_R" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Receipt_R" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_ReceiptReport_RR" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_ReceiptReport_RR" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Form 1aa" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Form 1aa" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_ReceiptReportData_RRD" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_ReceiptReportData_RRD" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_testpush" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_testpush" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_pushlist" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_pushlist" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_eeeee" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_eeeee" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_PushTestData" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_PushTestData" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_PushTestUpdate" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_PushTestUpdate" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_JobDet" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_JobDet" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_userdet" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_userdet" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_VisitDetails" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_VisitDetails" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Welcome" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Welcome" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Photo" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Photo" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_JobFields" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_JobFields" WHERE Flags IN (2,3) GROUP BY SessionID; DROP TABLE 'mfForm_TEST LISTS'; DROP TABLE 'mfForm_Debug'; DROP TABLE 'mfForm_asfsd'; DROP TABLE 'mfForm_Street Issues List'; DROP TABLE 'mfForm_Check Groups'; DROP TABLE 'mfForm_OrderEntry_OE'; DROP TABLE 'mfForm_PCNList'; DROP TABLE 'mfForm_ProductEntry_PE'; DROP TABLE 'mfForm_PushTest'; DROP TABLE 'mfForm_NullTest'; DROP TABLE 'mfForm_Order details'; DROP TABLE 'mfForm_test1'; DROP TABLE 'mfForm_Form 1'; DROP TABLE 'mfForm_Form 3'; DROP TABLE 'mfForm_Form 2'; DROP TABLE 'mfForm_Options'; DROP TABLE 'mfForm_PartDetails'; DROP TABLE 'mfForm_Jobs'; DROP TABLE 'mfForm_Confirm'; DROP TABLE 'mfForm_Unfinished'; DROP TABLE 'mfForm_Errors'; DROP TABLE 'mfForm_JobOptions'; SELECT * FROM "mfSysSyncLog"; SELECT * FROM "mfSysSyncLog"; SELECT * FROM "mfSysTrackingLog"; SELECT * FROM "mfSysTrackingLog"; DELETE FROM mfSysTrackingLog; COMMIT; VACUUM;
Step 3. This is on Thread 1 SELECT DISTINCT("SessionID") FROM "mfForm_Welcome" WHERE UID IN (-1, 1) AND "D3_TXTCODE" LIKE "1" ORDER BY SessionID DESC;
Step 4. This is on thread 2 BEGIN; SELECT SessionID FROM "mfForm_PartFields" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT name, rootpage, sql FROM 'main'.sqlite_master CREATE TABLE "mfSysUsers" ("ID" numeric PRIMARY KEY , "Username" text (20) , "Password" text (20) ) CREATE TABLE "mfSysUserPermissions" ("UserID" numeric PRIMARY KEY , "StartUpForm" text (128) , "Flags" numeric ) CREATE TABLE "mfSysUsersToForms" ("PermissionID" numeric PRIMARY KEY , "UserID" numeric , "FormTitle" text (128) , "Flags" numeric ) CREATE TABLE mfSysSyncLog (ID integer IDENTITY PRIMARY KEY, UID integer, Sync nvarchar(19), Type integer, Records integer, FormName nvarchar(32), FailureReason integer) CREATE TABLE mfSysTrackingLog (ID integer IDENTITY PRIMARY KEY, UID integer, LogIn nvarchar(19), LogOut nvarchar(19), FormName nvarchar(32), Session integer) CREATE TABLE "mfForm_Welcome" ("SessionID" numeric PRIMARY KEY , "UID" numeric , "Flags" numeric , "Expiry" numeric , "Spare" numeric , "D3_txtAccessCode" text , "D3_txtMsgDay" text , "D3_txtUser" text , "D3_txtDesc" text , "D3_txtH" text , "D3_txtErr" text , "D3_txtDontSend" text , "D3_DEVICE" text , "D3_txtCode" text ) CREATE TABLE "mFSysVersion" (asdasd CHAR) CREATE INDEX 'IndexUserName_mfSysUsers' ON 'mfSysUsers' (Username ASC ) CREATE INDEX 'IndexUserID_mfSysUsers' ON 'mfSysUsers' (ID DESC ) CREATE INDEX [IdxTrackingLog] ON [mfSysTrackingLog] (ID) SELECT SessionID FROM "mfForm_PartFields" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_testaa" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_testaa" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_testbb" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_testbb" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Issue Photos" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Issue Photos" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Inspections Guide" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Inspections Guide" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Street Inspections" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Street Inspections" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Street Issues" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Street Issues" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Street Inspections List" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Street Inspections List" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Inspection Photos" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Inspection Photos" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Menu" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Menu" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_IssuePCN" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_IssuePCN" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_IssuePCNStartUp" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_IssuePCNStartUp" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_PCNNoGenerator" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_PCNNoGenerator" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_PCNShift" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_PCNShift" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_DogFouling" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_DogFouling" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_OffenceList" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_OffenceList" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_PCNShiftDetails" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_PCNShiftDetails" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_IssuePCNMain" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_IssuePCNMain" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_AuditsToClaim" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_AuditsToClaim" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Check Item" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Check Item" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Audit" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Audit" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_AuditClaim" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_AuditClaim" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_AIMS" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_AIMS" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Main" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Main" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_CheckItemActions" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_CheckItemActions" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_HRA" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_HRA" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_TEST LISTS2" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_TEST LISTS2" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_OrderList_OL" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_OrderList_OL" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Statement_ST" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Statement_ST" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_LastOrderList" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_LastOrderList" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_StatementData_SD" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_StatementData_SD" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Receipt_R" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Receipt_R" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_ReceiptReport_RR" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_ReceiptReport_RR" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Form 1aa" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Form 1aa" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_ReceiptReportData_RRD" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_ReceiptReportData_RRD" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_testpush" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_testpush" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_pushlist" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_pushlist" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_eeeee" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_eeeee" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_PushTestData" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_PushTestData" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_PushTestUpdate" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_PushTestUpdate" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_JobDet" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_JobDet" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_userdet" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_userdet" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_VisitDetails" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_VisitDetails" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Welcome" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Welcome" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Photo" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_Photo" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_JobFields" WHERE Flags IN (2,3) GROUP BY SessionID; SELECT SessionID FROM "mfForm_JobFields" WHERE Flags IN (2,3) GROUP BY SessionID; DROP TABLE 'mfForm_TEST LISTS'; DROP TABLE 'mfForm_Debug'; DROP TABLE 'mfForm_asfsd'; DROP TABLE 'mfForm_Street Issues List'; DROP TABLE 'mfForm_Check Groups'; DROP TABLE 'mfForm_OrderEntry_OE'; DROP TABLE 'mfForm_PCNList'; DROP TABLE 'mfForm_ProductEntry_PE'; DROP TABLE 'mfForm_PushTest'; DROP TABLE 'mfForm_NullTest'; DROP TABLE 'mfForm_Order details'; DROP TABLE 'mfForm_test1'; DROP TABLE 'mfForm_Form 1'; DROP TABLE 'mfForm_Form 3'; DROP TABLE 'mfForm_Form 2'; DROP TABLE 'mfForm_Options'; DROP TABLE 'mfForm_PartDetails'; DROP TABLE 'mfForm_Jobs'; DROP TABLE 'mfForm_Confirm'; DROP TABLE 'mfForm_Unfinished'; DROP TABLE 'mfForm_Errors'; DROP TABLE 'mfForm_JobOptions'; SELECT * FROM "mfSysSyncLog"; SELECT * FROM "mfSysSyncLog"; SELECT * FROM "mfSysTrackingLog"; SELECT * FROM "mfSysTrackingLog"; SELECT MAX(SessionID) FROM "mfForm_Welcome"; SELECT MAX(SessionID) FROM "mfForm_Welcome"; CREATE TABLE "mfForm_Welcome" ("SessionID" numeric PRIMARY KEY , "UID" numeric , "Flags" numeric , "Expiry" numeric , "Spare" numeric , "D3_txtUser" text , "D3_DEVICE" text , "D3_txtDesc" text , "D3_txtMsgDay" text , "D3_txtCode" text ); INSERT INTO "mfForm_Welcome" (SessionID, UID, Flags, Expiry, Spare, "D3_txtUser", "D3_DEVICE", "D3_txtDesc", "D3_txtMsgDay", "D3_txtCode") VALUES (1,-1,1,0,0,"1", "OEYQB-8LSG1-Y1GRW-RDG9X", "John Smith", "Welcome to the Integral mForms ExpressHaulage sample from Integral Mobile Data", "1234"); COMMIT; VACUUM;
Step 5. This is on thread 1 and the query that fails but should work! SELECT DISTINCT("SessionID") FROM "mfForm_Welcome" WHERE UID IN (-1, 1) AND "D3_TXTCODE" LIKE "1234" ORDER BY SessionID DESC;
_2007-Jul-19 15:48:30 by anonymous:_ {linebreak} See Ticket #2486: pragma index_list does not detect index created on 2nd connection ---- _2007-Jul-19 16:32:00 by anonymous:_ {linebreak} With multi-threaded programs becoming more prevalent due to multi-core chips, having one thread not know the schema of the database is becoming a problem. ---- _2007-Jul-19 16:37:44 by drh:_ {linebreak} This problem has nothing to do with schema loading. The error is that the change counter in the header of the database file was not being incremented following a VACUUM. ---- _2007-Jul-20 10:48:38 by anonymous:_ {linebreak} Many thanks that seems to have sorted it.
#f2dcdc 2517 code active 2007 Jul anonymous 2007 Jul dflam 1 1 exception on reading text in vista but not xp My companies sqlite 3.1 db works perfectly on Win XP but when we moved to Vista (I'm using Vista 64)it is trowing an exception when I access a text field that contains this data: 'A/C Pressure Sensor, raw1 = A/C on, 0 = A/C off (A/C status determines which IACTx cell is used)' Interestingly when I view data I've inserted using sqliteman3 it has unprintable characters added to it. (A/C status determines which IACTx cell is used)9 If I define the field as Char[512] this artifact goes away. But reading your literature this isn't supposed to make a difference because everything is char. I've changed the values in the error column, but the error seems to be depending on length rather than value. Any help appreciatied! Jim _2007-Jul-19 15:13:41 by drh:_ {linebreak} We will be better able to help you with your problem on the SQLite mailing list. See http://www.sqlite.org/support.html for instructions on joining the mailing list.
#c8c8c8 2516 code closed 2007 Jul anonymous 2007 Jul 3 3 REGRESSION:sqlite3_prepare ignores nBytes parameter Bug described as in Ticket #1554 reappeared in recent versions of sqlite (tested debian 3.3.8, and 3.4.0 built from release tarball). Code snippet : int rc = sqlite3_prepare( sqlite_ , "insert into JOB_TODO values (24, 1, 4, 'C', 'p', NULL, NULL, 'respawn', '/usr/bin/moma/mbx_selection 2 /tmp/select_pipe_ED2 300 600 \0', NULL, NULL );" , 151, &stmt, NULL ) ;
Fails with error unrecognized token: "'/usr/bin/moma/mbx_selection 2 /tmp/select_pipe_ED2 300 600 "
Looking at the code, we can find the related [3040] CVS commit in prepare.c (as described in ticket #1554), but the bug still occurs _2007-Jul-19 11:11:36 by anonymous:_ {linebreak} sqlite_prepare_v2 has the same problem ---- _2007-Jul-19 12:12:54 by drh:_ {linebreak} The nBytes parameter is working fine. The problem is that SQLite does not accept strings that contain embedded \0 characters. And this is not going to change. If you want to insert a string that contains an embedded \0, use ? for the string in sqlite3_prepare() then bind the string using {link: /capi3ref.html#sqlite3_bind_text sqlite3_bind_text()}.
#c8c8c8 2515 code closed 2007 Jul anonymous 2007 Jul 1 1 Entered to ASCII zeros as a text string was return a single zero. I have created a Schema that defines all but one column as a string. When I insert data that is defined as '00', '06', '05', etc. I receive no error for the insertion. But, when I retrieve the data it returns the data with the preceeding zero removed. This should not occur when the data is defined as a string and not defined to be of a numeric type. This does not occur for any other type string data. _2007-Jul-18 21:47:55 by drh:_ {linebreak} Here's what I get: CREATE TABLE t1(x TEXT); INSERT INTO t1 VALUES('00'); SELECT * FROM t1; 00 Unable to reproduce the problem.
#cfe8bd 2514 code fixed 2007 Jul anonymous 2007 Jul 3 3 max() and min() is wrong when DESC index exists testcase: sqlite> PRAGMA legacy_file_format=0; sqlite> CREATE TABLE t(a); sqlite> INSERT INTO t VALUES(2); sqlite> INSERT INTO t VALUES(1); sqlite> CREATE INDEX t_a ON t(a DESC); sqlite> SELECT min(a) FROM t; 2 sqlite> SELECT max(a) FROM t; 1 Proposed fix: --- src/select.c~ 2007-06-16 00:31:50.000000000 +0900 +++ src/select.c 2007-07-18 23:48:07.680080000 +0900 @@ -2485,7 +2485,9 @@ static int simpleMinMaxQuery(Parse *pPar if( seekOp==OP_Rewind ){ sqlite3VdbeAddOp(v, OP_Null, 0, 0); sqlite3VdbeAddOp(v, OP_MakeRecord, 1, 0); - seekOp = OP_MoveGt; + seekOp = pIdx->aSortOrder[0]==SQLITE_SO_ASC ? OP_MoveGt : OP_MoveLt; + }else if( pIdx->aSortOrder[0]==SQLITE_SO_DESC ){ + seekOp = OP_Rewind; } sqlite3VdbeAddOp(v, seekOp, iIdx, 0); sqlite3VdbeAddOp(v, OP_IdxRowid, iIdx, 0);
#c8c8c8 2513 code closed 2007 Jul anonymous 2007 Jul 1 2 Internal compiler error tclsqlite.c Using make with Windows XP with MSYS and MinGW 3.4.5 after ./configure --with-tcl=C:/dev/Tcl/lib produces this error ./src/tclsqlite.c: In function 'Sqlite3_Init': ./src/tclsqlite.c:2421: internal compiler error: in rest_of_handle_final, at toplev.c:2067 Robert Wishlaw _2007-Jul-17 20:30:28 by drh:_ {linebreak} Seem like an "internal compiler error" is the type of thing you should report to the compiler vendor, not to the makers of the code you are trying to compile....
#f2dcdc 2512 code active 2007 Jul shess 2007 Jul 1 1 FTS virtual table name quoting problem All table names should be quoted in the FTS module code. with TRACE enabled in ext/fts2/fts2.c: sqlite> create virtual table "a b c" using fts2 (t); FTS2 Create FTS2 sql: CREATE TABLE main.a b c_content(c0t) SQL error: vtable constructor failed: a b c
_2007-Jul-18 06:44:21 by anonymous:_ {linebreak} A similar problem shows if a FTS column has the same name as the FTS table: CREATE VIRTUAL TABLE a USING fts2 (a); Returns "vtable constructor failed: a.".
#f2dcdc 2511 code active 2007 Jul anonymous 2007 Jul drh 3 2 Inconsistent Pragma output Pragma output is inconsistent when setting the value. Most do not generate any output and silently set the value, while others generate a singleton row with the set value. Here is a list of pragmas that generate output while setting the values: sqlite> PRAGMA locking_mode = NORMAL; normal sqlite> PRAGMA max_page_count = 100000; 100000 The following do not generate any output upon query: PRAGMA case_sensitive_like; PRAGMA incremental_vacuum; Sqlite was built from almagamation using the following configuration flags: --enable-threadsafe --disable-tcl --enable-tempstore
#f2dcdc 2510 code active 2007 Jul anonymous 2007 Jul 1 1 Vacuum modified FTS2 rowids VACUUM modifies FTS2 rowids. Here is the test: drop table if exists a; create virtual table a using fts2 (t); insert into a (t) values ('one'); insert into a (t) values ('two'); insert into a (t) values ('three'); select rowid, * from a; delete from a where t = 'two'; vacuum; select rowid, * from a; Unfortunately there is no workaround since table a is auto-generated by the FTS2 module. _2007-Jul-17 14:05:58 by anonymous:_ {linebreak} http://www.sqlite.org/cvstrac/chngview?cn=4157 ---- _2007-Jul-17 14:24:29 by anonymous:_ {linebreak} Yes, this behavior has been recently documented, but there is no user workaround like PRIMARY KEY for FTS2 rowids. Therefore I consider this as a bug which should be fixed in fts2.c. ---- _2007-Jul-17 14:55:57 by anonymous:_ {linebreak} Should virtual tables be VACUUMable? What exactly is being vacuumed here - an internal table? ---- _2007-Jul-17 16:34:55 by shess:_ {linebreak} I agree, I think this is a bug. Rather severe, too, the entire fts system implicitely depends on rowids not changing, this means that vacuum will break fts tables (fts1 or fts2). drop table if exists t; create virtual table t using fts2; insert into t (content) values ('This is a test'); insert into t (content) values ('This is a string'); insert into t (content) values ('That was a test'); insert into t (content) values ('A random string'); select content from t where t MATCH 'test'; delete from t where content = 'This is a string'; vacuum; select content from t where t MATCH 'test';
The first select outputs 'This is a test' and 'That was a test'. The second select outputs 'This is a test', and 'A random string'. ---- _2007-Jul-17 17:27:21 by anonymous:_ {linebreak} This patch seems to address the FTS2 VACUUM problem and passes all fts2 tests. It adds an INTEGER PRIMARY KEY docid column to the hidden %_content table. Note: this new table format is not backwards compatible with existing FTS2 databases. -Joe Wilson Index: ext/fts2/fts2.c =================================================================== RCS file: /sqlite/sqlite/ext/fts2/fts2.c,v retrieving revision 1.40 diff -u -3 -p -r1.40 fts2.c --- ext/fts2/fts2.c 2 Jul 2007 10:16:50 -0000 1.40 +++ ext/fts2/fts2.c 17 Jul 2007 17:19:49 -0000 @@ -1769,9 +1769,9 @@ typedef enum fulltext_statement { */ static const char *const fulltext_zStatement[MAX_STMT] = { /* CONTENT_INSERT */ NULL, /* generated in contentInsertStatement() */ - /* CONTENT_SELECT */ "select * from %_content where rowid = ?", + /* CONTENT_SELECT */ "select * from %_content where docid = ?", /* CONTENT_UPDATE */ NULL, /* generated in contentUpdateStatement() */ - /* CONTENT_DELETE */ "delete from %_content where rowid = ?", + /* CONTENT_DELETE */ "delete from %_content where docid = ?", /* BLOCK_INSERT */ "insert into %_segments values (?)", /* BLOCK_SELECT */ "select block from %_segments where rowid = ?", @@ -1860,14 +1860,14 @@ static struct fulltext_vtab *cursor_vtab static const sqlite3_module fts2Module; /* forward declaration */ /* Return a dynamically generated statement of the form - * insert into %_content (rowid, ...) values (?, ...) + * insert into %_content (docid, ...) values (?, ...) */ static const char *contentInsertStatement(fulltext_vtab *v){ StringBuffer sb; int i; initStringBuffer(&sb); - append(&sb, "insert into %_content (rowid, "); + append(&sb, "insert into %_content (docid, "); appendList(&sb, v->nColumn, v->azContentColumn); append(&sb, ") values (?"); for(i=0; i
---- _2007-Jul-18 00:13:56 by shess:_ {linebreak} BTW, AFAICT this only happens for sqlite3.4. Older versions don't seem to have the problem. ---- _2007-Jul-18 01:31:49 by anonymous:_ {linebreak} The rowid changing after VACUUM predates 3.4.0... SQLite version 3.3.7 Enter ".help" for instructions sqlite> CREATE TABLE t(a); sqlite> INSERT INTO "t" VALUES('one'); sqlite> INSERT INTO "t" VALUES('two'); sqlite> INSERT INTO "t" VALUES('three'); sqlite> select rowid, * from t; 1|one 2|two 3|three sqlite> delete from t where a = 'one'; sqlite> select rowid, * from t; 2|two 3|three sqlite> vacuum; sqlite> select rowid, * from t; 1|two 2|three SQLite version 3.2.0 Enter ".help" for instructions sqlite> CREATE TABLE t(a); sqlite> INSERT INTO "t" VALUES('one'); sqlite> INSERT INTO "t" VALUES('two'); sqlite> INSERT INTO "t" VALUES('three'); sqlite> select rowid, * from t; 1|one 2|two 3|three sqlite> delete from t where a = 'one'; sqlite> select rowid, * from t; 2|two 3|three sqlite> vacuum; sqlite> select rowid, * from t; 1|two 2|three
---- _2007-Jul-18 15:59:24 by anonymous:_ {linebreak} As you may know, INTEGER PRIMARY KEY indexes are the ROWID, so I must supect they would change after a VACUUM. The best workaround is to put docid as INTEGER, then adding a PRIMARY KEY index for the docid column.
#f2dcdc 2509 code active 2007 Jul anonymous 2007 Jul 1 1 SQLITE_DATE SELECT CAST(MyDate AS DATE), CAST(MyTime AS TIME) FROM MyData I hope, it will result/return DATE, TIME. Please support to SQLITE_DATE and SQLITE_TIME. Thanks.
#f2dcdc 2508 code active 2007 Jul anonymous 2007 Dec 1 1 utf8ToUnicode() does not work on some WinCE devices On some WinCE devices first call to =MultiByteToWideChar()= in =utf8ToUnicode()= always fails. Tried calling =GetLastError()= after it fails and it returns error code 87 -- =ERROR_INVALID_PARAMETER=. To fix this had to change code page from =CP_UTF8= to =CP_ACP= -- no idea why this works. Original =utf8ToUnicode()= ---- static WCHAR *utf8ToUnicode(const char *zFilename) { int nChar; WCHAR *zWideFilename; nChar = MultiByteToWideChar(CP_UTF8, 0, zFilename, -1, NULL, 0); zWideFilename = sqliteMalloc( nChar*sizeof(zWideFilename[0]) ); if( zWideFilename==0 ){ return 0; } nChar = MultiByteToWideChar(CP_UTF8, 0, zFilename, -1, zWideFilename, nChar); if( nChar==0 ){ sqliteFree(zWideFilename); zWideFilename = 0; } return zWideFilename; } ---- Fixed =utf8ToUnicode()= ---- static WCHAR *utf8ToUnicode(const char *zFilename) { int nChar; WCHAR *zWideFilename; nChar = MultiByteToWideChar(CP_ACP, 0, zFilename, -1, NULL, 0); if( nChar == 0 ) { DWORD dwError = GetLastError(); OSTRACE2("MultiByteToWideChar() failed, last error: %d\n", dwError); return 0; } zWideFilename = sqliteMalloc( nChar*sizeof(zWideFilename[0]) ); if( zWideFilename==0 ){ return 0; } nChar = MultiByteToWideChar(CP_ACP, 0, zFilename, -1, zWideFilename, nChar); if( nChar==0 ){ sqliteFree(zWideFilename); zWideFilename = 0; } return zWideFilename; } ---- _2007-Jul-17 23:56:10 by anonymous:_ {linebreak} =unicodeToUtf8()= needs to be fixed the same way. Before: ---- static char *unicodeToUtf8(const WCHAR *zWideFilename){ int nByte; char *zFilename; nByte = WideCharToMultiByte(CP_UTF8, 0, zWideFilename, -1, 0, 0, 0, 0); zFilename = sqliteMalloc( nByte ); if( zFilename==0 ){ return 0; } nByte = WideCharToMultiByte(CP_UTF8, 0, zWideFilename, -1, zFilename, nByte, 0, 0); if( nByte == 0 ){ sqliteFree(zFilename); zFilename = 0; } return zFilename; } ---- After: ---- static char *unicodeToUtf8(const WCHAR *zWideFilename){ int nByte; char *zFilename; nByte = WideCharToMultiByte(CP_ACP, 0, zWideFilename, -1, NULL, 0, NULL, NULL); if ( nByte == 0 ) { DWORD dwError = GetLastError(); OSTRACE2("WideCharToMultiByte() failed, last error = %d\n", dwError); return 0; } zFilename = sqliteMalloc( nByte ); if( zFilename==0 ){ return 0; } nByte = WideCharToMultiByte(CP_ACP, 0, zWideFilename, -1, zFilename, nByte, 0, 0); if( nByte == 0 ){ sqliteFree(zFilename); zFilename = 0; } return zFilename; } ---- Note that while original code with =CP_UTF8= works on Windows and SOME WinCE devices, this modified code works well and Windows and all WinCE devices I've tested so far. ---- _2007-Jul-18 16:01:21 by anonymous:_ {linebreak} Why not using the conversions from SQLite internals ? It can change a UTF-16 to UTF-8 and vice-versa. Or using UTF-16 variants in windows ce should be the best case. ---- _2007-Aug-09 20:47:04 by anonymous:_ Why not using the conversions from SQLite internals ? It can change a UTF-16 to UTF-8 and vice-versa. Or using UTF-16 variants in windows ce should be the best case. Not so simple. =unicodeToUtf8()= is used a lot internally regardless of what whether you use UTF-16 or UTF-8 yourself. For example, =unicodeToUtf8()= is used by =sqlite3WinTempFileName()= which is in turn used by =sqlite3PagerOpentemp()= -- I think you get the idea. ---- _2007-Dec-20 00:29:33 by anonymous:_ {linebreak} We've found that using CP_UTF8 fails on WinCE kernels that don't include SYSGEN_CORELOC (http://msdn2.microsoft.com/en-us/library/ms903883.aspx). To make the code handle any device it should be changed to: static WCHAR *utf8ToUnicode(const char *zFilename) { int nChar; WCHAR *zWideFilename; nChar = MultiByteToWideChar(CP_UTF8, 0, zFilename, -1, NULL, 0); if( nChar == 0 ) { nChar = MultiByteToWideChar(CP_ACP, 0, zFilename, -1, NULL, 0); if( nChar == 0 ) { DWORD dwError = GetLastError(); OSTRACE2("MultiByteToWideChar() failed, last error: %d\n", dwError); return 0; } } zWideFilename = sqliteMalloc( nChar*sizeof(zWideFilename[0]) ); if( zWideFilename==0 ) { return 0; } nChar = MultiByteToWideChar(CP_UTF8, 0, zFilename, -1, zWideFilename, nChar); if( nChar==0 ) { nChar = MultiByteToWideChar(CP_ACP, 0, zFilename, -1, zWideFilename, nChar); if( nChar==0 ) { sqliteFree(zWideFilename); zWideFilename = 0; } } return zWideFilename; }
#cfe8bd 2507 code fixed 2007 Jul anonymous 2007 Jul 1 1 test_expr expr-1.104 fails (64-bit int handling) make test fails with: expr-1.104...make: *** [test] Floating point exception System info: gcc-4.1.2, glibc-2.5-r4, 2.6.22.1 x86_64 _2007-Jul-16 18:56:02 by drh:_ {linebreak} Already fixed. See [4130] and [4131].
#f2dcdc 2506 new active 2007 Jul anonymous 2008 Jan 3 2 New API to retrieve ROWID from SQLite3_stmt structire Is it too much trouble to allow an API to retrieve ROWID for non-aggeregate queries directly from SQLite3_stmt structire? It would be very useful to create updatable non aggregate query results for situations when actually internal PK (ROWID) is not gived explicitly in SQL statement nor actual table's PK (if any). SELECT queries that join two or more tables together would be a problem also. ---- _2007-Jul-16 16:51:18 by anonymous:_ {linebreak} It's more of a multi-step process. First you have to enumerate the open cursors on the sqlite3_stmt object. Then you need to resolve the table each cursor goes to, and then fetch the rowid for each active cursor. Of course this may get confusing when you've joined a table onto itself. ---- _2008-Jan-19 10:32:59 by anonymous:_ {linebreak} This is far to old active ticket. Is it in consideration to be implemented in the near future?
#c8c8c8 2505 code closed 2007 Jul anonymous 2007 Jul 1 1 does not return error when use sqlite_open to open an non-sqlite db I use sqlite_open api to open an non sqlite database, for example an exe file.The result of sqlite_open is not an error code but sqlite_ok. Sqlite does not actually read the database file until the first SQL query is executed. So the sqlite3_open succeeds but the first call to sqlite3_prepare_v2() or sqlite3_exec() will return an error.
#c8c8c8 2504 build closed 2007 Jul anonymous 2007 Jul 1 1 sqlite 3.4.0 fails to build FreeBSD 6.2 64-bit system Attached is the configure and make output. As this is a FreeBSD 6.2 AMD 64-bit box, several "warning: cast from pointer to integer of different size" in func.c, table.c, and vdbemem.c concern me as potential bugs. Added to the fact the the undefined reference to __isnanl. The warnings also appear in 3.3.17 version of SQLite, which I hadn't noticed before, but the there was no undefined reference preventing the build. _2007-Jul-13 17:13:47 by anonymous:_ {linebreak} submitter: anthony howe, achowe@snert.com ---- _2007-Jul-13 17:34:33 by drh:_ {linebreak} Already fixed by check-in [4103].
#f2dcdc 2503 code active 2007 Jul anonymous 2007 Jul 3 4 sqlite3PagerReleaseMemory does not decrement page count When cached pages are released by sqlite3PagerReleaseMemory the number of pages (pPager->nPage) is not decremented. This also subsequently affects the maximum value at pPager->nMaxPage. This does not affect the operation of sqlite (but does upset my statistic gathering). Although only tested with 3.3.8, the problem does not appear to have been corrected in version 3.4.0 Actually this will affect the handling of cache_size as the pager will think there are more pages cached than is the case, and may unnecessarily release some. ---- _2007-Jul-18 15:54:29 by anonymous:_ {linebreak} Do you have a simple test case that demonstrates this? Put a print statement in pager.c if necessary.
#cfe8bd 2502 code fixed 2007 Jul anonymous 2007 Jul 4 4 Digital Mars static library runtime problem When I attempt to use a SQLite 3.4.0 static library built with Digital Mars C/C++, I get these errors sqlite340dm.lib(sqlite340dm) Error 42: Symbol Undefined _sqlite3ParserFree sqlite340dm.lib(sqlite340dm) Error 42: Symbol Undefined _sqlite3Parser sqlite340dm.lib(sqlite340dm) Error 42: Symbol Undefined _sqlite3ParserAlloc If I comment out lines 65930-65932 in sqlite3.c extern void *sqlite3ParserAlloc(void*(*)(size_t)); extern void sqlite3ParserFree(void*, void(*)(void*)); extern void sqlite3Parser(void*, int, Token, Parse*); then the static library works as expected. Robert Wishlaw
#c8c8c8 2501 build closed 2007 Jul anonymous 2007 Jul 2 3 limits.h confilct with visual studio limits.h When I include sqlite3.h in my code, and provide the sqlite headers folder to complier include path, limits.h is picked up from sqlite headers even when it must be picked from vc/include, so I get errors compiling other files that we compiling fine. Current workaround is I provide the parent folder of sqlite headers as include path for compiler and instead of #include "sqlite3.h", I do a #include "sqlite-3.4.0/sqlite3.h" _2007-Jul-12 20:39:41 by drh:_ {linebreak} See tickets #2428 and #2495.
#c8c8c8 2500 code closed 2007 Jul anonymous 2007 Jul 2 4 Possible memory leak on avr32 arch I am using sqlite in a C written application running on an embedded device based on an avr32 (the 32AP7000) from atmel. It seams that executing some insertion cause a memory leak. The simpliest leaking code is something like this : (The complete code include error check, error messages freeing, and non-infinite loop.) int main(int ac, char **av) { sqlite3 *db; char *zErrMsg = 0; sqlite3_open(DB_PATH, &db)) while (42) sqlite3_exec(db, "INSERT INTO high (io1) VALUES (1);" NULL, NULL, &zErrMsg)); sqlite3_close(db); return 0; } I've tested with both sqlite3.4.0 and 3.3.6-94 binaries. Based on the tests i have done the amount of memory leaking depends on the number of row in the table and the size of the sql statement It also seems that a 'SELECT' sql statement does not leak at all. It also appear that some of the leaking memory is never returned to the system. I suppose that the problem is somewhere between the sqlite memory management system and the device's one. I still can not tell for sure if the problem lie on the sqlite or the board but i have successfuly ported some heavy memory consumer without experiencing such problems. Maybe the software shiped with my board don't have default behavior for some memory managment tasks and cause sqlite not to act as it should ? I am actually working on getting some more informations on how and where it leak. (porting valgrind, dmalloc or something) So i will try to provide some more information as soon as i can. Maybe you can look at the contents of zErrMsg first? ---- _2007-Jul-13 08:16:37 by anonymous:_ {linebreak} I've already checked zErrMsg and freed it if necessary. ---- _2007-Jul-13 08:57:59 by anonymous:_ {linebreak} Sorry i didn't understand i am focussed on "where is my memory" :). If you meens to check what error happen, then i don't get any errors everything goes fine and return is SQLITE_OK. The comportement is good exept that the data segment of a program doing multiple inertions keep growing. ---- _2007-Jul-13 13:37:06 by anonymous:_ {linebreak} Some fresh news. First the memory leaked seems to be correctly freed when the process done, it was my misunderstanding. Next doing a quick wrapper around sqlite3GenericFree and sqlite3GenericMalloc i found something interristing : Each time i am doing an insert some of the memory allocated wont be freed untill calling sqlite3_close, which cause the process to grow. Since the sqlite malloc mecanism is pretty dark to me i still don't understand how such a behavior happen. ---- _2007-Jul-16 13:50:17 by danielk1977:_ {linebreak} Because sqlite3_close() does free the possibly leaked allocations, we can probably assume this is just the pager cache growing normally. See the documentation for the "cache_size" pragma for details.
#cacae5 2499 build defer 2007 Jul anonymous 2007 Aug drh 1 1 undefined reference to `_WinMain@16' when attempting to build I get the error: /usr/lib/gcc/i686-pc-cygwin/3.4.4/../../../libcygwin.a(libcmain.o):(.text+0xab): undefined reference to `_WinMain@16' _2007-Jul-11 18:16:28 by anonymous:_ ls Makefile.in config.sub main.mk sqlite.pc.in Makefile.linux-gcc configure mkdll.sh sqlite3.1 README configure.ac mkopcodec.awk sqlite3.pc.in VERSION contrib mkopcodeh.awk src aclocal.m4 doc mkso.sh tclinstaller.tcl addopcodes.awk ext notes test art install-sh publish.sh tool config.guess ltmain.sh spec.template www ./configure && make install checking build system type... i686-pc-cygwin checking host system type... i686-pc-cygwin checking for gcc... gcc checking for C compiler default output file name... a.exe checking whether the C compiler works... yes checking whether we are cross compiling... no checking for suffix of executables... .exe checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking for a sed that does not truncate output... /usr/bin/sed checking for grep that handles long lines and -e... /usr/bin/grep checking for egrep... /usr/bin/grep -E checking for ld used by gcc... /usr/i686-pc-cygwin/bin/ld.exe checking if the linker (/usr/i686-pc-cygwin/bin/ld.exe) is GNU ld... yes checking for /usr/i686-pc-cygwin/bin/ld.exe option to reload object files... -r checking for BSD-compatible nm... /usr/bin/nm -B checking whether ln -s works... yes checking how to recognise dependent libraries... file_magic ^x86 archive import|^x86 DLL checking how to run the C preprocessor... gcc -E checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... yes checking dlfcn.h usability... yes checking dlfcn.h presence... yes checking for dlfcn.h... yes checking for g++... g++ checking whether we are using the GNU C++ compiler... yes checking whether g++ accepts -g... yes checking how to run the C++ preprocessor... g++ -E checking for g77... no checking for xlf... no checking for f77... no checking for frt... no checking for pgf77... no checking for cf77... no checking for fort77... no checking for fl32... no checking for af77... no checking for xlf90... no checking for f90... no checking for pgf90... no checking for pghpf... no checking for epcf90... no checking for gfortran... no checking for g95... no checking for xlf95... no checking for f95... no checking for fort... no checking for ifort... no checking for ifc... no checking for efc... no checking for pgf95... no checking for lf95... no checking for ftn... no checking whether we are using the GNU Fortran 77 compiler... no checking whether accepts -g... no checking the maximum length of command line arguments... 8192 checking command to parse /usr/bin/nm -B output from gcc object... ok checking for objdir... .libs checking for ar... ar checking for ranlib... ranlib checking for strip... strip checking for correct ltmain.sh version... yes checking if gcc supports -fno-rtti -fno-exceptions... no checking for gcc option to produce PIC... checking if gcc static flag -static works... yes checking if gcc supports -c -o file.o... yes checking whether the gcc linker (/usr/i686-pc-cygwin/bin/ld.exe) supports shared libraries... yes checking whether -lc should be explicitly linked in... yes checking dynamic linker characteristics... Win32 ld.exe checking how to hardcode library paths into programs... immediate checking whether stripping libraries is possible... yes checking if libtool supports shared libraries... yes checking whether to build shared libraries... yes checking whether to build static libraries... yes configure: creating libtool appending configuration tag "CXX" to libtool checking for ld used by g++... /usr/i686-pc-cygwin/bin/ld.exe checking if the linker (/usr/i686-pc-cygwin/bin/ld.exe) is GNU ld... yes checking whether the g++ linker (/usr/i686-pc-cygwin/bin/ld.exe) supports shared libraries... yes checking for g++ option to produce PIC... checking if g++ static flag -static works... yes checking if g++ supports -c -o file.o... yes checking whether the g++ linker (/usr/i686-pc-cygwin/bin/ld.exe) supports shared libraries... yes checking dynamic linker characteristics... Win32 ld.exe checking how to hardcode library paths into programs... immediate appending configuration tag "F77" to libtool checking for a BSD-compatible install... /usr/bin/install -c checking for gawk... gawk Version set to 3.4 Release set to 3.4.0 Version number set to 3004000 checking whether to support threadsafe operation... no checking whether to allow connections to be shared across threads... no checking whether threads can override each others locks... no checking whether to support shared library linked as release mode or not... no checking whether to use an in-ram database for temporary tables... no checking if executables have the .exe suffix... unknown checking host system type... (cached) i686-pc-cygwin checking for Tcl configuration... found /usr/lib/tclConfig.sh checking for existence of /usr/lib/tclConfig.sh... loading checking for library containing tgetent... -ltermcap checking for readline in -lreadline... no checking readline.h usability... no checking readline.h presence... no checking for readline.h... no checking for /usr/include/readline.h... no checking for /usr/include/readline/readline.h... no checking for /usr/local/include/readline.h... no checking for /usr/local/include/readline/readline.h... no checking for /usr/local/readline/include/readline.h... no checking for /usr/local/readline/include/readline/readline.h... no checking for /usr/contrib/include/readline.h... no checking for /usr/contrib/include/readline/readline.h... no checking for /mingw/include/readline.h... no checking for /mingw/include/readline/readline.h... no checking for library containing fdatasync... none required checking for usleep... yes checking for fdatasync... yes configure: creating ./config.status config.status: creating Makefile config.status: creating sqlite3.pc gcc -g -O2 -o lemon.exe ./tool/lemon.c cp ./tool/lempar.c . cp ./src/parse.y . ./lemon.exe parse.y mv parse.h parse.h.temp awk -f ./addopcodes.awk parse.h.temp >parse.h cat parse.h ./src/vdbe.c | gawk -f ./mkopcodeh.awk >opcodes.h sort -n -b -k 3 opcodes.h | gawk -f ./mkopcodec.awk >opcodes.c gcc -g -O2 -o mkkeywordhash.exe ./tool/mkkeywordhash.c ./mkkeywordhash.exe >keywordhash.h sed -e s/--VERS--/3.4.0/ ./src/sqlite.h.in | \ sed -e s/--VERSION-NUMBER--/3004000/ >sqlite3.h rm -rf tsrc mkdir -p tsrc cp ./src/alter.c ./src/analyze.c ./src/attach.c ./src/auth.c ./src/btree.c ./src/btree.h ./src/build.c ./src/callback.c ./src/complete.c ./src/date.c ./src/delete.c ./src/expr.c ./src/func.c ./src/hash.c ./src/hash.h ./src/insert.c ./src/legacy.c ./src/loadext.c ./src/main.c ./src/malloc.c ./src/os.c ./src/os_unix.c ./src/os_win.c ./src/os_os2.c ./src/pager.c ./src/pager.h ./src/parse.y ./src/pragma.c ./src/prepare.c ./src/printf.c ./src/random.c ./src/select.c ./src/shell.c ./src/sqlite.h.in ./src/sqliteInt.h ./src/table.c ./src/tclsqlite.c ./src/tokenize.c ./src/trigger.c ./src/utf.c ./src/update.c ./src/util.c ./src/vacuum.c ./src/vdbe.c ./src/vdbe.h ./src/vdbeapi.c ./src/vdbeaux.c ./src/vdbeblob.c ./src/vdbefifo.c ./src/vdbemem.c ./src/vdbeInt.h ./src/vtab.c ./src/where.c ./ext/fts1/fts1.c ./ext/fts1/fts1.h ./ext/fts1/fts1_hash.c ./ext/fts1/fts1_hash.h ./ext/fts1/fts1_porter.c ./ext/fts1/fts1_tokenizer.h ./ext/fts1/fts1_tokenizer1.c sqlite3.h ./src/btree.h ./src/btreeInt.h ./src/hash.h ./src/limits.h opcodes.h ./src/os.h ./src/os_common.h ./src/sqlite3ext.h ./src/sqliteInt.h ./src/vdbe.h parse.h ./ext/fts1/fts1.h ./ext/fts1/fts1_hash.h ./ext/fts1/fts1_tokenizer.h ./src/vdbeInt.h tsrc cp: warning: source file `./src/btree.h' specified more than once cp: warning: source file `./src/hash.h' specified more than once cp: warning: source file `./src/sqliteInt.h' specified more than once cp: warning: source file `./src/vdbe.h' specified more than once cp: warning: source file `./ext/fts1/fts1.h' specified more than once cp: warning: source file `./ext/fts1/fts1_hash.h' specified more than once cp: warning: source file `./ext/fts1/fts1_tokenizer.h' specified more than once cp: warning: source file `./src/vdbeInt.h' specified more than once rm tsrc/sqlite.h.in tsrc/parse.y cp parse.c opcodes.c keywordhash.h tsrc tclsh ./tool/mksqlite3c.tcl cc sqlite3.c -o sqlite3 /usr/lib/gcc/i686-pc-cygwin/3.4.4/../../../libcygwin.a(libcmain.o):(.text+0xab): undefined reference to `_WinMain@16' collect2: ld returned 1 exit status make: *** [sqlite3] Error 1 ---- _2007-Jul-11 18:17:17 by anonymous:_ {linebreak} when initially filing this bug, i was told to not paste large screen dumps. i was also promised that i would be able to add an attachment later. however, that option was not given, so I have attached a screen transcript. ---- _2007-Jul-11 18:18:33 by anonymous:_ {linebreak} my email is metaperl at gmail.com ---- _2007-Jul-11 18:38:53 by drh:_ {linebreak} The [Attach] link in the upper-left is what you use to add attachments. The sqlite3.c source file is a library, not a program. If you want to compile a command-line shell, add the "shell.c" source file: cc -o sqlite3 sqlite3.c shell.c ---- _2007-Jul-23 20:35:06 by anonymous:_ {linebreak} I believe this bug report should be reopened because this error comes when you try to configure and compile USING THE DEFAULTS. When you try to do a standard configure and make, it should either try to compile shell.c as is needed or not compile sqlite.c. It shouldn't put in a command that fails the compile by default. Sorry to bother you about this, but I don't see how this isn't a bug or at least a glitch. It's a problem in the configuration and makefile system. ---- _2007-Aug-15 03:26:05 by anonymous:_ {linebreak} Compile and link passed by following patch (at least for me). diff -ru sqlite-3.4.2/Makefile.in sqlite-3.4.2-new/Makefile.in --- sqlite-3.4.2/Makefile.in 2007-08-15 10:37:18.921875000 +0900 +++ sqlite-3.4.2-new/Makefile.in 2007-08-15 10:36:08.750000000 +0900 @@ -681,7 +681,7 @@ mkdir -p doc mv $(DOC) doc -install: sqlite3 libsqlite3.la sqlite3.h ${HAVE_TCL:1=tcl_install} +install: sqlite3$(TEXE) libsqlite3.la sqlite3.h ${HAVE_TCL:1=tcl_install} $(INSTALL) -d $(DESTDIR)$(libdir) $(LTINSTALL) libsqlite3.la $(DESTDIR)$(libdir) $(INSTALL) -d $(DESTDIR)$(exec_prefix)/bin ---- _2007-Aug-22 12:33:17 by anonymous:_ {linebreak} Why is this bug closes and marked "not a bug"? sqlite fails on make install on Cygwin (as in untar/configure/make/make install), which is *clearly* a bug. The "fix" posted above does not work, make install fails instantly with tclsh ../sqlite-3.4.2-n/tclinstaller.tcl 3.4 can't read "env(DESTDIR)": no such variable while executing "set LIBDIR $env(DESTDIR)$env(TCLLIBDIR)" (file "../sqlite-3.4.2-n/tclinstaller.tcl" line 11) make: *** [tcl_install] Error 1 ---- _2007-Aug-22 12:53:45 by drh:_ {linebreak} I misread the problem. Cygwin is not a supported platform. So in that sense, this is *not* clearly a bug in SQLite. This is an imcompatibility with Cygwin. We will take the problem under advisement, but since Cygwin is not officially support and because we have many much more serious problems to work on, progress on this is likely to be slow. You can expedite a fix by posting a patch.
#f2dcdc 2498 code active 2007 Jul anonymous 2007 Jul 3 2 sqlite memory org on linux (related ticket #2473)... he sample programme that I run(wrote) in tty1 and there I operate the command of ps at tty2, there seems two items from the programme of ps command. This error was not at the version 3.3.13 but now it is happening at sqlite versions although i change nothing from the programme, If I turn to old versions, there is seen only one item again. When I upgrade to version 3.3.13 or later, there is seen two items again Is it normal or there is any mistake? (excuse my poor english) _2007-Jul-11 16:44:22 by anonymous:_ {linebreak} So you are seeing 2 processes instead of 1 on Linux? Linux 2.4 and earlier kernels show threads as seperate processes with unique process IDs. Is your program creating any threads? The only place where SQLite creates threads is the function below - but it joins with the thread right away. /* ** This procedure attempts to determine whether or not threads ** can override each others locks then sets the ** threadsOverrideEachOthersLocks variable appropriately. */ static void testThreadLockingBehavior(int fd_orig){ int fd; struct threadTestData d[2]; pthread_t t[2]; fd = dup(fd_orig); if( fd<0 ) return; memset(d, 0, sizeof(d)); d[0].fd = fd; d[0].lock.l_type = F_RDLCK; d[0].lock.l_len = 1; d[0].lock.l_start = 0; d[0].lock.l_whence = SEEK_SET; d[1] = d[0]; d[1].lock.l_type = F_WRLCK; pthread_create(&t[0], 0, threadLockingTest, &d[0]); pthread_create(&t[1], 0, threadLockingTest, &d[1]); pthread_join(t[0], 0); pthread_join(t[1], 0); close(fd); threadsOverrideEachOthersLocks = d[0].result==0 && d[1].result==0; }
If you post a small C program demonstrating what you're seeing, someone may be able to offer a suggestion. ---- _2007-Jul-11 16:47:10 by anonymous:_ {linebreak} I suppose it's not inconceivable that the join failed. Perhaps these pthread_join calls' return codes should be examined for errors. ---- _2007-Jul-11 18:53:36 by anonymous:_ {linebreak} If you're playing games with tty's and you've got an early Linux 2.6 kernel, it's possible that processes are dying because of http://lkml.org/lkml/2004/10/21/119. It was, last I checked, fixed in 2.6.10. The SIGHUP being generated might also interfer with a =pthread_join()=, although =pthread_join()= doesn't say anything about ever generating =EINTR=... c. ---- _2007-Jul-12 06:12:28 by anonymous:_ {linebreak} my example program is very simple, i not use threading-multithreading structure... If I turn to old versions of sqlite, there is seen only one item again, when I upgrade to version 3.3.13 or later, there is seen two items again Is it. note: /lib/libpthread.so.0 linked to /lib/libpthread-0.10.so (size 55468 byte) ---- _2007-Jul-12 11:36:37 by anonymous:_ {linebreak} Your description of the problem isn't clear enough, so the answers you're getting are just guesses. You may have more luck by describing the problem (with as much detail as possible) in your native language and hoping someone in the SQLite community can add a translation. I know you're doing your best with the english you speak, but it's not working well enough for someone to help with your problem. Adding code samples and command-line output would also help considerably, since that sort of this is mostly language independent.
#c8c8c8 2497 code closed 2007 Jul anonymous 2007 Jul 4 2 sqlite3_errcode does not return the proper error code sqlite3_errcode does not return the same result as the last api call when using a progress handler that aborts the connection. sqlite3_step returns SQLITE_ABORT, but sqlite3_errcode is returning SQLITE_ERROR. A test case can be found here: https://bugzilla.mozilla.org/attachment.cgi?id=271753 Downstream bug: https://bugzilla.mozilla.org/show_bug.cgi?id=387609 -sdwilsh _2007-Jul-11 06:37:52 by danielk1977:_ {linebreak} When it comes to setting the database error code, sqlite3_step() behaving the same way regardless of whether sqlite3_prepare() or sqlite3_prepare_v2() is used to prepare the query. So when an error is returned from sqlite3_step() on a statement prepared with _v2(), the "real" error code is returned but the database error code is being set to SQLITE_ERROR. I'm thinking we should change to set the database error code and message to the "real" error code in this case. Also, note that this test-case is incompatible with version 3.4.0. In prior versions, if the progress-handler returns non-zero, the statement is halted and SQLITE_ABORT is returned. In 3.4.0, this error code is now SQLITE_INTERRUPT. ---- _2007-Jul-12 16:35:37 by anonymous:_ {linebreak} Thanks for the heads up. -sdwilsh
#f2dcdc 2496 code active 2007 Jul anonymous 2007 Jul 5 4 "No such column" error should include table information It'd be nice if the "no such column" error included the table/view that SQLite was searching for the column. no such column: ChecklistID Thanks, Sam
#c8c8c8 2495 code closed 2007 Jul anonymous 2007 Jul 4 4 standard C library Naming conflict - limits.h "limits.h" You should really avoid using filenames conflicting with the standard C library. It took me a few minutes to figure out why my code was not compiling. Then I realized that configure:19220: checking for Tcl configuration configure:19302: result: found /usr/lib/tclConfig.sh configure:19305: checking for existence of /usr/lib/tclConfig.sh ac_cv_c_tclconfig=/usr/lib HAVE_TCL='1' TCL_BIN_DIR='/usr/lib' TCL_INCLUDE_SPEC='-I/usr/include/tcl8.4.13' TCL_LIBS='-ldl -lpthread -lieee -lm' TCL_LIB_FILE='libtcl8.4.so' TCL_LIB_FLAG='-ltcl8.4' TCL_LIB_SPEC='-L/usr/lib -ltcl8.4' TCL_SRC_DIR='/usr/include/tcl8.4.13' TCL_STUB_LIB_FILE='libtclstub8.4.a' TCL_STUB_LIB_FLAG='-ltclstub8.4' TCL_STUB_LIB_SPEC='-L/usr/lib -ltclstub8.4' TCL_VERSION='8.4'
---- _2007-Jul-07 03:49:21 by anonymous:_ {linebreak} Ah I see now. I'm using a 64 bit machine so tcl lives in the 64 bit directory hierarchy. Here's how I got most of the tests to run: export CFLAGS='-pipe -O3 -g -DSQLITE_DISABLE_DIRSYNC=1 -Wall' make distclean cvs -d :pserver:anonymous@www.sqlite.org:/sqlite -r update . ./configure --prefix=/usr/local/pkgs/sqlite-3.4.0.1 --enable-tcl --with-tcl=/usr/lib64 This ticket can close. I'll do some further testing and open another ticket about the single thread test that fails. ---- _2007-Jul-07 11:43:05 by drh:_ {linebreak} Please note that this is an open ticketing system. Anybody can close a ticket. You do not need to ask me or Dan to do it.
#f2dcdc 2488 new active 2007 Jul anonymous 2007 Jul 5 4 autosize on column output mode in sqlite3 program It would be nice if sqlite3 program has a autosizecolumn mode for displaying queries, because it truncates values, and to calculate and use .width size for each column is tedious. _2007-Jul-07 11:42:03 by drh:_ {linebreak} In order to do this, we would have to either run the query twice or load the entire result set into memory. Otherwise, there would be no way to determine the longest element of each column. Neither approach seems attractive for large and complex queries. ---- _2007-Jul-28 07:05:56 by anonymous:_ {linebreak} every query should internally detect the longest column sizes and a new command should enable the user to set these values for any repetition or similar queries. At the end the queries run twice or more but only the first trial would have cause irritations on output. And this solution should be easy enough to implement it. {linebreak} Even the core of sqlite could calculate the maximum length of each column and a new API function could make this available. It would be really nice to get such an enhancement! ---- _2007-Jul-28 18:37:36 by anonymous:_ {linebreak} while you don't move to next row, sqlite doesn't know the contents, so it will be impossible to do, only if you cache the text entirely in memory, but this is ugly, imagine a 1GB recordset into RAM... ---- have you tried .mode tabs ? here, i have correct column width
#f2dcdc 2487 code active 2007 Jul anonymous 2007 Jul 1 1 SQLite database locked error on NFS mounted home dir I have a c program using the provided API. My home directory is NFS mounted, Im using SQLite 3.3.17. I open a new database using "sqlite3_open", then strcpy () a SQL command to create a table, and run "sqlite3_exec" with this string. I get a return code of 5=database locked. I then tried to manually (command line using sqlite3) create a table within a database in my home dir, that fails too. =========== x@y> sqlite3 db2 SQLite version 3.3.17 Enter ".help" for instructions sqlite> create table test (Lastname varchar); SQL error: database is locked sqlite> ============== If I try this on my local machine (a Mac), it works fine, but I need it to work in my home directory mounted via NFS as that is where the output of our program goes _2007-Jul-06 19:04:15 by anonymous:_ {linebreak} If you're using a Mac, compile sqlite with SQLITE_ENABLE_LOCKING_STYLE in os_unix.c ---- _2007-Jul-07 11:51:10 by drh:_ {linebreak} This is a problem with your NFS implementation - it does not appear to support posix advisory locking. There is nothing much that SQLite can do about this. Anonymous above suggests making use of the dot-locking mechanism contributed by Apple. This might be an effective work-around. But remember that there is performance impact. Also remember that an SQLite database that uses dot-locking is subtly imcompatible with a standard SQLite database. The file format itself is the same, but if two processes try to access the database file at the same time and one uses dot-locks and the other uses posix advisory locks, you will end up with corruption. ---- _2007-Jul-07 12:44:09 by anonymous:_ {linebreak} It's very odd that Apple does not fix their Mac OSX POSIX locks for NFS given their resources.
#c8c8c8 2486 code closed 2007 Jul anonymous 2007 Sep 2 3 pragma index_list does not detect index created on 2nd connection Open two connections to the same database. On the first connection, create a table "test" and an index on that table. On the second, run "pragma index_list( test )". Expected result: pragma detects the previously-created index. Actual result: no index is detected. Test case to follow. _2007-Jul-05 20:02:25 by anonymous:_ {linebreak} #include "sqlite3.h" #include
---- _2007-Jul-05 20:18:10 by anonymous:_ {linebreak} Can reproduce with 3.4.0 using two sqlite3 processes. I found that if you execute "select * from sqlite_master" from the second sqlite3 commandline shell, then pragma index_list on the 2nd connection will correctly show the indexes for some reason. ---- _2007-Jul-05 21:12:32 by drh:_ {linebreak} The first connection does not know that an index has been created on the second connection until it attempts to access the database. ---- _2007-Jul-05 21:17:36 by anonymous:_ {linebreak} Same problem with connection 1 executing ALTER TABLE ADD COLUMN and connection 2 executing PRAGMA table_info(table-name). I suspect the following PRAGMAs must always read() the database's file change counter and if it changed then reset the cached schema before proceeding: PRAGMA foreign_key_list(table-name); PRAGMA index_info(index-name); PRAGMA index_list(table-name); PRAGMA table_info(table-name); ---- _2007-Jul-05 21:37:12 by anonymous:_ {linebreak} Does the comment on 2007-Jul-05 21:12:32 imply this behavior is by design? ---- _2007-Jul-06 11:31:55 by drh:_ {linebreak} Yes. ---- _2007-Jul-06 15:38:07 by anonymous:_ {linebreak} That's understandable. This ticket can be fixed with documentation. Can you recommend the most efficient/quickest way to force SQLite to reload the schema in memory from a library user's point of view? SELECT 0 from _dummy_table_ SELECT 0 from sqlite_master where 1=0 ? ---- _2007-Jul-06 15:41:37 by anonymous:_ {linebreak} Would a new pragma for this purpose be out of the question? PRAGMA force_schema_reload; ---- _2007-Jul-06 16:57:56 by anonymous:_ {linebreak} I can appreciate seperate processes' in-memory schema not being automatically updated when another process changes the schema or adds a new index. But it does not make sense that another connection in the *same process* is not flagged to automatically update their in-memory schema prior to executing one of these PRAGMAs being issued. It's just a matter of checking an integer change counter in memory for each connection on the same database. ---- _2007-Jul-06 18:22:18 by anonymous:_ {linebreak} =SELECT 0 from sqlite_master where 1=0= doesn't work. =SELECT 0 from sqlite_master= does. Can you elaborate on the reason for this design decision? ---- _2007-Jul-06 18:55:04 by anonymous:_ {linebreak} I was just looking for the fastest executing SELECT statement I could find that would force a reload of all schema from the file. Perhaps something like this? select null from sqlite_master limit 1 I'm pretty sure trying to query a non-existant table also has the side-effect of reloading the entire schema. -- force a re-load, and ignore the error SELECT a from _non_existant_table_ I just don't know which one's faster. It would be better to have a standard way of doing this. ---- _2007-Jul-11 21:08:14 by anonymous:_ {linebreak} Doing a select against a non-existent table will fail without reloading the schema. At least in my independent discovery of this problem that didn't resolve things... ---- _2007-Jul-20 06:25:14 by anonymous:_ {linebreak} Would a schema change (add index) in one connection affect the other connection as it goes to insert into the table without knowing that the index has been added? And result in incomplete index contents? ---- _2007-Jul-26 18:42:06 by anonymous:_ {linebreak} the greatest way to solve this is reloading the db metadata when you issue a pragma command, like pragma index_list or other. i thought this will be great. also, database header has a version to detect SQLITE_SCHEMA changes. why pragma commands doesn't honor this ? ---- _2007-Aug-22 19:09:28 by anonymous:_ {linebreak} related ticket: #2519 ---- _2007-Sep-04 20:10:24 by anonymous:_ {linebreak} Fixed in 3.5.0 (*) #include "sqlite3.h" #include ssehic@dev-5-i386:/profense/devel$ ls -l /profense/log/logdata.db -rwxrwxr-x 1 _admd profense 12106752 Jul 1 23:15 /profense/log/logdata.db ssehic@dev-5-i386:/profense/devel$ sqlite3 /profense/log/logdata.db "select count(*) from log" 156000 ssehic@dev-5-i386:/profense/devel/sqlmem SQLite version 3.3.13 Usage (bytes) initial, rssize=487424, dsize=241664 Usage (bytes) after SELECT COUNT(*), rssize=1417216, dsize=704512 Usage (bytes) after sqlite3_close, rssize=1318912, dsize=618496 ssehic@dev-5-i386:/profense/devel/sqlmem SQLite version 3.4.0 Usage (bytes) initial, rssize=495616, dsize=241664 Usage (bytes) after SELECT COUNT(*), rssize=22118400, dsize=21331968 Usage (bytes) after sqlite3_close, rssize=1351680, dsize=618496
I'm sure something like this should not go unnoticed by the developers. However, I can't believe this is specific to OpenBSD. Something changed after 3.3.13 which broke this in a subtle way. Any hints? _2007-Jul-01 23:03:42 by drh:_ {linebreak} Your test program appears to be OpenBSD-specific. I cannot compile it on Linux. So I cannot replicate your problem. We do test SQLite using both its own internal memory leak detection logic and valgrind, and both report no memory leaks over the entire test suite. ---- _2007-Jul-02 07:31:57 by anonymous:_ {linebreak} Yes. The test program is OpenBSD specific, but only the memory reporting stuff (print_memory_usage()). I'll see if I can provide a Linux variant and complete details on how to reproduce the problem. More to come. ---- _2007-Jul-02 09:28:42 by anonymous:_ {linebreak} I think I figured this one out. The problem is only reproducible if the database is created using sqlite3 prior to 3.3.14 and queries are later run on the same database using 3.4.0. This is weird, since the database format should be backwards compatible between releases. However, lots of "performance improvements" went in after 3.3.13. This could have something to do with it. At http://insecure.dk/bin/logdata.db.gz is a test database created with 3.3.12. If you run a "SELECT COUNT(*) FROM log" on that using 3.4.0, you should see the memory usage increase greatly using sqlite3 utility. ---- _2007-Jul-02 10:38:12 by anonymous:_ {linebreak} This ticket can be closed. Increased memory usage is due to my test database (created with 3.3.12) used PRAGMA default_cache_size = 10000; Apparently versions > 3.3.14 allocate all the memory on the first database query. That's why I saw the jump in memory usage. Sorry for the noise. ---- _2007-Jul-02 10:38:41 by drh:_ {linebreak} I downloaded logdata.db from the link above and ran "SELECT count(*) FROM log" against it using the latest version of SQLite from CVS (which i closer to 3.4.1 than to 3.4.0, I suspect). I did this using valgrind: valgrind --leak-check=full --show-reachable=yes \ ./sqlite3 logdata.db 'select count(*) FROM log' Valgrind tells me that 156 bytes of memory were leaked in 11 allocations. All of the leaked memory was allocated by the getpwuid_r() function in glibc - not from SQLite. The total memory allocated was 13,515,680 bytes, which considering that the logdata.db database specifies a cache size of 10,000 pages and you are reading (and thus attempting to cache) the entire 50MB database, this is about what one would expect. I am unable to reproduce the problem. ---- _2007-Jul-02 10:50:40 by anonymous:_ {linebreak} That was my conclusion as well. Thanks again for the great work and unbelievably speedy responses to "bug" reports.
#c8c8c8 2480 code closed 2007 Jun anonymous 2007 Aug 4 4 typo in faq.tcl --- a/www/faq.tcl +++ b/www/faq.tcl @@ -202,8 +202,8 @@ faq { threads. The sqlite3 pointer must only be used in the same thread in which it was created by sqlite3_open(). If you - break the rules and use an sqlite3 in more than one thread - under these circumstances, then you will likely corrupt some + break the rules and use an sqlite3 pointer in more than one + thread under these circumstances, then you will likely corrupt some internal data structures resulting in a crash.
Under UNIX, you should not carry an open SQLite database across #f2dcdc 2479 code active 2007 Jun anonymous 2007 Jun 1 1 WinCE regression on some systems. Any db open fails. Because Windows CE is a modular system, meaning many parts of it can be optionally ommited by the system builder, some don't include the CP_UTF8 conversion algorithms for MultiByteToWideChar and family. I believe Windows 95 and early 98 systems can also lack this encoding if not updated with a later Internet Explorer version. Solution is to just use the sqlite internal functions that already know how to do the same thing. Attached is an untested patch to os_win.c (I don't have a windows machine nor a cross-compiler set up) to show where the problem is and a possible (sub-optimal) solution. I believe the right thing to do would be to just drop the utf8ToUnicode and unicodeToUtf8 functions, add the sqlite3Utf8to16 equivalent to utf.c and use them instead. ~Nuno Lucas _2007-Jun-29 14:54:11 by anonymous:_ {linebreak} The title is wrong. It should say "Any db open using the UTF-8 API", as using the open16 API will work. #c8c8c8 2478 code closed 2007 Jun anonymous 2007 Jun 2 2 LIMIT and OFFSET abbreviation to LIMIT x,y bug! Statement taken from Apress: The Definitive Guide to SQLite: SELECT * FROM foods WHERE name LIKE 'B%' ORDER BY type_id DESC, name LIMIT 1 OFFSET 2; can be expressed equivalently with SELECT * FROM foods WHERE name LIKE 'B%' ORDER BY type_id DESC, name LIMIT 1,2; SQLite3 3.4.0 bug - The parameters after LIMIT have been reversed in the implementation of this version of SQLite in error!... 2 tuples from the database are returned (LIMIT) and starting from the OFFSET 1 instead of the other way around, 1 tuple starting at offset 2. From the documentation for SELECT: Note that if the OFFSET keyword is used in the LIMIT clause, then the limit is the first number and the offset is the second number. If a comma is used instead of the OFFSET keyword, then the offset is the first number and the limit is the second number. This seeming contradition is intentional - it maximizes compatibility with legacy SQL database systems. SQLite has always been this way. It's a bug in the book. ---- _2007-Jun-29 11:49:02 by drh:_ {linebreak} As an historical curiosity, see ticket #245. #f2dcdc 2476 todo active 2007 Jun anonymous 2007 Jun 4 3 SQLite3 ignores ORDER BY clause when performing SELECT ... GROUP BY I found that sqlite3 ignores the ORDER BY clause when performing SELECT ... GROUP BY ... ORDER BY ... Table schema: CREATE TABLE events ( id integer not null primary key, title integer ); Data: |id|title| |1|hello| |2|hello| Query: SELECT title, id FROM events GROUP BY title ORDER BY id ASC; Result: |title|id| |hello|2| Expected result: |title|id| |hello|1| Note: I don't think this should even work in the first place, because id is not a grouped column, but MySQL and SQLite doesn't seem to have a problem with it. Oracle complains. _2007-Jun-29 07:46:00 by danielk1977:_ {linebreak} In SQL, the sorting specified by the ORDER BY clause is performed (logically) after the grouping specified by the GROUP BY clause. In this case a single row - the ORDER BY clause is redundant. So the problem is that SQLite and MySQL are implementing the non-standard SQL extension of allowing an expression that is neither an aggregate or a part of the GROUP BY clause in the result-set of the SELECT in a different way. #cfe8bd 2475 code fixed 2007 Jun anonymous 2007 Jul 1 3 Handles being left open on unix On unix, I'm seeing processes that have been clone/exec'd inheriting file handles to sqlite databases that were opened in the parent process (and hence subsequently don't/can't get closed in the child process). Which ultimately means that you can't unmount a filesystem. The solution is the FD_CLOEXEC fcntl on the open filehandle, but I'd like to get confirmation from elsewhere/highlight it to DRH/the list to make them aware of it. Tested and fixed by the below patch (against 3.3.6). ==== //depot/qt/4.2/src/3rdparty/sqlite/os_unix.c#2 - /home/bking/depot/qt/4.2/src/3rdparty/sqlite/os_unix.c ==== @@ -701,7 +701,7 @@ OsFile **pId, int *pReadonly ){ - int rc; + int rc, oldflags; unixFile f; CRASH_TEST_OVERRIDE(sqlite3CrashOpenReadWrite, zFilename, pId, pReadonly); @@ -730,6 +730,12 @@ return SQLITE_NOMEM; } TRACE3("OPEN %-3d %s\n", f.h, zFilename); + oldflags = fcntl (f.h, F_GETFD, 0); + if (oldflags >= 0) + { + oldflags |= FD_CLOEXEC; + fcntl (f.h, F_SETFD, oldflags); + } return allocateUnixFile(&f, pId); } @@ -749,7 +755,7 @@ ** On failure, return SQLITE_CANTOPEN. */ int sqlite3UnixOpenExclusive(const char *zFilename, OsFile **pId, int delFlag){ - int rc; + int rc, oldflags; unixFile f; CRASH_TEST_OVERRIDE(sqlite3CrashOpenExclusive, zFilename, pId, delFlag); @@ -772,6 +778,12 @@ unlink(zFilename); } TRACE3("OPEN-EX %-3d %s\n", f.h, zFilename); + oldflags = fcntl (f.h, F_GETFD, 0); + if (oldflags >= 0) + { + oldflags |= FD_CLOEXEC; + fcntl (f.h, F_SETFD, oldflags); + } return allocateUnixFile(&f, pId); } @@ -783,7 +795,7 @@ ** On failure, return SQLITE_CANTOPEN. */ int sqlite3UnixOpenReadOnly(const char *zFilename, OsFile **pId){ - int rc; + int rc, oldflags; unixFile f; CRASH_TEST_OVERRIDE(sqlite3CrashOpenReadOnly, zFilename, pId, 0); @@ -800,6 +812,12 @@ return SQLITE_NOMEM; } TRACE3("OPEN-RO %-3d %s\n", f.h, zFilename); + oldflags = fcntl (f.h, F_GETFD, 0); + if (oldflags >= 0) + { + oldflags |= FD_CLOEXEC; + fcntl (f.h, F_SETFD, oldflags); + } return allocateUnixFile(&f, pId); } @@ -823,6 +841,7 @@ OsFile *id, const char *zDirname ){ + int oldflags; unixFile *pFile = (unixFile*)id; if( pFile==0 ){ /* Do not open the directory if the corresponding file is not already @@ -836,6 +855,12 @@ return SQLITE_CANTOPEN; } TRACE3("OPENDIR %-3d %s\n", pFile->dirfd, zDirname); + oldflags = fcntl (pFile->dirfd, F_GETFD, 0); + if (oldflags >= 0) + { + oldflags |= FD_CLOEXEC; + fcntl (pFile->dirfd, F_SETFD, oldflags); + } return SQLITE_OK; } _2007-Jun-28 23:07:33 by anonymous:_ {linebreak} You'd need #ifdef FD_CLOEXEC around the code at the very least, since some UNIX-like OSes lack this feature. ---- _2007-Jun-29 00:12:29 by anonymous:_ {linebreak} it's not right to fork() and keep using sqlite3 objects handle if I don't misread the documentation. ---- _2007-Jul-02 00:42:22 by anonymous:_ {linebreak} *it's not right to fork() and keep using sqlite3 objects handle if I don't misread the documentation.* That's true, but sqlite without this change is still leaking filehandles (and keeping files open) on a standard practice fork and exec. I discovered it because applications that weren't even using sqlite were holding files open because the parent process had open handles at the point they were created. ---- _2007-Jul-02 21:04:18 by anonymous:_ {linebreak} Which UNIX-like OSes lack this feature? FD_CLOEXEC part of the Single UNIX Specification, Version 2, and POSIX: http://www.opengroup.org/onlinepubs/007908775/xsh/fcntl.h.html #f2dcdc 2474 new active 2007 Jun anonymous 2007 Jun 5 4 Multiple-record comma-delineated INSERT command I believe that both MySQL and DB2 support this feature. Instead of using separate commands for multiple INSERTS, you could use one command, and delineate the separate INSERT data with commas. Having support for this type of INSERT would make migrating MySQL or DB2 files to SQLite easier. Regular INSERT method: INSERT INTO foo VALUES ('Title1',26,NULL); INSERT INTO foo VALUES ('Title2',24,NULL); INSERT INTO foo VALUES ('Title3',12,NULL); Delineated INSERT method: INSERT INTO foo VALUES ('Title1',26,NULL), ('Title2',24,NULL), ('Title3',12,NULL); _2007-Jun-29 12:32:45 by anonymous:_ {linebreak} From the parsing point of view this is a bit interesting. Imagine the multi-insert statement is 100,000 lines long. Do you parse the entire statement for correctness first and hold this entire parsed tree in memory? Or do you begin a transaction, and try to process each sub-insert row by row and rollback if there's any error? I'd think the latter would be better from both a time and memory point of view. Actually, this multi-insert statement could be optimized to work around the sqlite slow bulk insert issue with multiple keys. #f2dcdc 2473 warn active 2007 Jun anonymous 2007 Jun anonymous 3 2 sqlite memory org. on linux i use sqlite on linux (2.4.35) and on FreePascal... i write an simple example code on linux_&_sqlite_&_freepascal. when working my example code on tty1, i run ps command on tty2; multiple process id showing about my example program... my example code is very simple... before v3.3.13 : no problem; but v3.3.17 and later is wrong... (excuse my poor english) _2007-Jun-28 13:10:10 by anonymous:_ {linebreak} There is no bug mentioned in this ticket. ---- _2007-Jun-28 14:09:36 by drh:_ {linebreak} I think I see a hint of a bug report in the description where it says "before v3.3.13: no problem; but v3.3.17 and later is wrong". But that is not anything near enough information to isolate and fix the problem. halukduman09: Please find a friend or coworker who speaks good English and ask that person to translate for you. We can only help you if we can understand what you are saying. Thanks. #c8c8c8 2472 code closed 2007 Jun anonymous 2007 Jun 1 1 smart quotes turning question marks while storing in oracle 9i DB whenever end-user copy & paste the statement from the word which contains smart quotes into the java application and submit, it stores in the oracle 9i database. But the smart quotes are getting converted into question mark (?) / reverse question mark symbol. Kindly tell me the solution. Environment: Java 2.0,Jrun 4.0, Oracle 9.2.0.1.0 Where does SQLite figure into the system? #c8c8c8 2471 secure closed 2007 Jun anonymous 2007 Jun 1 1 a bug for 64 bit driver under operation system Firstly,I can't find a 64bit driver just like a.so file under the specail system Unix/Linxu. Secondly,I find a drvier writen by java,but it has many problem while multi threads operate the database. Sometimes I lost all of my data. Can you offer a 64bit driver under Unix/Linux in your website? This is probably better discussed on the mailing list. Any bugs in a java wrapper (which one) can be figured out there too. #cfe8bd 2470 code fixed 2007 Jun anonymous 2007 Jun 1 3 subselects causing reproducable SIGSEGV After upgrade to 3.4.0, I started receiving SIGSEGV on a particular sub-select query. This did *not* happen on 3.3.12. I have not tested versions 3.3.13 - 3.3.17. I've trimmed the original statement down as much as I could. The following statement shows the problem:
DROP TABLE table_1; DROP TABLE table_2; CREATE TABLE table_1 (col_10); CREATE TABLE table_2 ( col_1, col_2, col_3, col_4, col_5, col_6, col_7, col_8, col_9, col_10 ); SELECT col_10 FROM (SELECT table_1.col_10 AS col_10 FROM table_1), (SELECT table_1.col_10, table_2.col_9 AS qcol_9 FROM table_1, table_2 GROUP BY table_1.col_10, qcol_9 );What I've noticed is that it takes at least 10 columns in table_2 before the bug is triggered. Remove one (unused) column and it goes away. Also, removing the second parameter in GROUP BY (eg. qcol_9) makes the bug go away. This is on OpenBSD 4.1 (i386). I'm not familiar with SQLite's internals, but below is a backtrace. Another thing I noticed is that bug is much harder to trigger (like 1 out of 10 times) without any malloc debugging options turned on. See http://www.openbsd.org/cgi-bin/man.cgi?query=malloc.conf&apropos=0&sektion=0&manpath=OpenBSD+Current&arch=i386&format=html With MALLOC_OPTIONS "AFGJP", the bug is triggered each time.
ssehic@dev-5-i386:/home/ssehic/devel$ sqlite3 test.db SQLite version 3.4.0 Enter ".help" for instructions sqlite> .read sigsegv.sql Segmentation fault (core dumped) ssehic@dev-5-i386:/home/ssehic/devel$ gdb /usr/local/bin/sqlite3 sqlite3.core GNU gdb 6.3 Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-unknown-openbsd4.1"... Core was generated by `sqlite3'. Program terminated with signal 11, Segmentation fault. Reading symbols from /usr/local/lib/libsqlite3.so.9.0...done. Loaded symbols for /usr/local/lib/libsqlite3.so.9.0 Reading symbols from /usr/lib/libreadline.so.3.0...done. Loaded symbols for /usr/lib/libreadline.so.3.0 Reading symbols from /usr/lib/libncurses.so.10.0...done. Loaded symbols for /usr/lib/libncurses.so.10.0 Reading symbols from /usr/lib/libc.so.40.3...done. Loaded symbols for /usr/lib/libc.so.40.3 Reading symbols from /usr/libexec/ld.so...done. Loaded symbols for /usr/libexec/ld.so #0 sqlite3ValueFromExpr (pExpr=0xd0d0d0d0, enc=1 '\001', affinity=208 'Ð', ppVal=0xcfbf1618) at src/vdbemem.c:913 913 op = pExpr->op; (gdb) bt #0 sqlite3ValueFromExpr (pExpr=0xd0d0d0d0, enc=1 '\001', affinity=208 'Ð', ppVal=0xcfbf1618) at src/vdbemem.c:913 #1 0x07f47ef0 in sqlite3ColumnDefault (v=0x7e949000, pTab=0x854d2c80, i=8) at src/update.c:62 #2 0x07f2de40 in sqlite3ExprCodeGetColumn (v=0x7e949000, pTab=0x854d2c80, iColumn=8, iTable=2132004048) at src/expr.c:1683 #3 0x07f2df17 in sqlite3ExprCode (pParse=0xcfbf1a90, pExpr=0x7c2e7880) at src/expr.c:1733 #4 0x07f2e782 in sqlite3ExprCodeExprList (pParse=0xcfbf1a90, pList=0x7f13ccd0) at src/expr.c:2121 #5 0x07f45980 in sqlite3Select (pParse=0xcfbf1a90, p=0x7c2e7d00, eDest=8, iParm=2, pParent=0x7c2e7980, parentTab=1, pParentAgg=0xcfbf18cc, aff=0x0) at src/select.c:3218 #6 0x07f45cbd in sqlite3Select (pParse=0xcfbf1a90, p=0x7c2e7980, eDest=4, iParm=0, pParent=0x0, parentTab=0, pParentAgg=0x0, aff=0x0) at src/select.c:2961 #7 0x07f3b2ef in yy_reduce (yypParser=0x7fc1d800, yyruleno=104) at parse.y:369 #8 0x07f3ce04 in sqlite3Parser (yyp=0x7fc1d800, yymajor=1, yyminor={z = 0x1 "", dyn = 0, n = 1066002024}, pParse=0x1) at parse.c:3417 #9 0x07f469a5 in sqlite3RunParser (pParse=0xcfbf1a90, zSql=0x7f13c600 "SELECT\r\n col_10\r\nFROM\r\n (SELECT table_1.col_10 AS col_10 FROM table_1),\r\n (\r\n SELECT table_1.col_10, table_2.col_9 AS qcol_9\r\n FROM table_1, table_2\r\n GROUP BY table_1.co"..., pzErrMsg=0xcfbf1a8c) at src/tokenize.c:452 #10 0x07f3f5de in sqlite3Prepare (db=0x8693ce00, zSql=0x7f13c600 "SELECT\r\n col_10\r\nFROM\r\n (SELECT table_1.col_10 AS col_10 FROM table_1),\r\n (\r\n SELECT table_1.col_10, table_2.col_9 AS qcol_9\r\n FROM table_1, table_2\r\n GROUP BY table_1.co"..., nBytes=-1, saveSqlFlag=0, ppStmt=0xcfbf1bf4, pzTail=0xcfbf1bf8) at src/prepare.c:504 #11 0x07f3f6c9 in sqlite3_prepare (db=0x8693ce00, zSql=0x7f13c600 "SELECT\r\n col_10\r\nFROM\r\n (SELECT table_1.col_10 AS col_10 FROM table_1),\r\n (\r\n SELECT table_1.col_10, table_2.col_9 AS qcol_9\r\n FROM table_1, table_2\r\n GROUP BY table_1.co"..., nBytes=-1, ppStmt=0xcfbf1bf4, pzTail=0xcfbf1bf8) at src/prepare.c:616 #12 0x07f59ced in sqlite3_exec (db=0x8693ce00, zSql=0x7f13c600 "SELECT\r\n col_10\r\nFROM\r\n (SELECT table_1.col_10 AS col_10 FROM table_1),\r\n (\r\n SELECT table_1.col_10, table_2.col_9 AS qcol_9\r\n FROM table_1, table_2\r\n GROUP BY table_1.co"..., xCallback=0x1c001a24Looks like a NULL pointer deref. Hope that is enough information. Please let me know if you need anything else. Thanks. _2007-Jun-27 21:46:10 by drh:_ {linebreak} Though you would never know it to read the description, this turns out to be the same problem as #2445. It has already been fixed by [4122]. ---- _2007-Jun-27 22:10:22 by anonymous:_ {linebreak} Even though the bug was fixed, can you add this test case for this anyway, as Dan's test case in Check-in [4122] for this bug failed to crash or give any valgrind error for me with the sqlite 3.4.0 sources. -- no SEGV, nor any valgrind error for 3.4.0 release CREATE TEMP TABLE t1 (a, b, c, d, e); CREATE TEMP TABLE t2 (f); SELECT t1.e AS alias FROM t2, t1 WHERE alias = 1 ; -- SEGV on 3.4.0 release CREATE TABLE table_1 (col_10); CREATE TABLE table_2 ( col_1, col_2, col_3, col_4, col_5, col_6, col_7, col_8, col_9, col_10 ); SELECT col_10 FROM (SELECT table_1.col_10 AS col_10 FROM table_1), (SELECT table_1.col_10, table_2.col_9 AS qcol_9 FROM table_1, table_2 GROUP BY table_1.col_10, qcol_9 ); ---- _2007-Jun-28 20:03:38 by anonymous:_ {linebreak} Verified to fix the reported problem on OpenBSD 4.1 with the patch backported to 3.4.0. Thanks guys. You certainly rock. #f2dcdc 2469 build active 2007 Jun anonymous 2007 Jul 1 1 test fails on Solaris I have a problem running the test suite on Solaris 9. Build was done using gcc 4.2.0. The build completes without error but many tests fail. I've created my own minimal test that exhibits the problem:, pArg=0xcfbf3af4, pzErrMsg=0xcfbf1c4c) at src/legacy.c:55 #13 0x1c0046a1 in process_input (p=0xcfbf3af4, in=0x24e8f7a0) at src/shell.c:1656 #14 0x1c003286 in do_meta_command (zLine=0x3c000272 "", p=0x7d2a9145) at src/shell.c:1363 #15 0x1c004851 in process_input (p=0xcfbf3af4, in=0x0) at src/shell.c:1616 #16 0x1c004d9f in main (argc=2, argv=0xcfbf4480) at src/shell.c:1991 (gdb) p pExpr->op Cannot access memory at address 0xd0d0d0d0 (gdb) p pExpr $2 = (Expr *) 0xd0d0d0d0
set testdir [file dirname $argv0] source $testdir/tester.tcl db close file delete -force test.db test.db-journal sqlite db test.db do_test tdb-1 { execsql { PRAGMA auto_vacuum = 1; BEGIN; CREATE TABLE t1(a, b); } execsql { COMMIT; } } {} integrity_check tdb-2 finish_testWhen running this test I get the following output:
tdb-1... Ok tdb-2... Expected: [ok] Got: [{*** in database main *** List of tree roots: 2nd reference to page 1 Page 3 is never used}] Thread-specific data deallocated properly 1 errors out of 3 tests Failures on these tests: tdb-2This error happens on lots of, but not all, tests. I'm happy to do whatever is necessary to help debug this. Thanks, Tim. _2007-Jun-27 10:44:14 by anonymous:_ {linebreak} Further to this, it appears to be related to gcc 4.2.0. It works fine with gcc 3.4.6. ---- _2007-Jun-28 09:54:35 by anonymous:_ {linebreak} Further more, it doesn't appear to be specific to Solaris. The same problem occurs on Linux with gcc 4.2.0. So I guess the subject of this ticket should be changed to "build/test problems with gcc 4.2.0". This is probably a significant problem - the build completes find but the resultant code is broken. People may not notice this until it's too late. ---- _2007-Jun-28 12:24:05 by drh:_ {linebreak} I installed gcc 4.2.0 on my SuSE linux i686 desktop and built test harnesses under three different configurations: gcc420 -g -O0 -Wall -fstrict-aliasing gcc420 -g -O3 -Wall gcc420 -g -O3 -fstrict-aliasing -fomit-frame-pointer The first two configurations used separate source files. The third configuration was built using the amalgamation. I ran the "quick" test under all configurations. All tests ran to completion with no errors. ---- _2007-Jun-28 13:22:20 by anonymous:_ {linebreak} Two ideas: 1. Compile with gcc 4.2.0 using -O0 instead of -O2 and see what happens. Disable any other optimizations you may have. 2. Run truss with full read/write buffer display on the gcc 3.4.6 compiled testfixture running your simple test case and compare its output to the gcc 4.2.0 compiled test case. ---- _2007-Jul-01 19:00:40 by anonymous:_ {linebreak} I've done tests with optimisation, and this appears to tickle the problem. With no optimisation, -O, -O0, -O1 and -03 it works. With -O2 and -Os it's broken. I was compiling with -O2 when I submitted the initial report. Tim. ---- _2007-Jul-01 19:54:54 by drh:_ {linebreak} I can reproduce the problem now on Linux when compiling as follows: gcc420 -g -O2 -Wall ---- _2007-Jul-01 21:50:42 by drh:_ {linebreak} This appears to be a bug in GCC 4.3.0. A work-around is to compile with the -fno-tree-vrp option. GCC appears to miscompile a single loop within the logic that implements the integrity_check PRAGMA. The code that gets miscompiled is in the file vdbe.c (lines numbers added): 4308 for(j=0; j
RCS file: /sqlite/sqlite/src/vdbe.c,v retrieving revision 1.636 diff -u -3 -p -r1.636 vdbe.c --- src/vdbe.c 1 Jul 2007 21:18:40 -0000 1.636 +++ src/vdbe.c 21 Jul 2007 19:10:13 -0000 @@ -4306,7 +4306,8 @@ case OP_IntegrityCk: { pnErr = &p->aMem[j]; assert( (pnErr->flags & MEM_Int)!=0 ); for(j=0; ju.i; } aRoot[j] = 0; popStack(&pTos, nRoot);