Small. Fast. Reliable.
Choose any three.
Key: Active Review Fixed Tested Deferred Closed
# Type Status Created By Subsys Changed Assigned Svr Pri Title  
Description
Remarks
 
2917 code active 2008 Feb anonymous   2008 Feb   4 4 Tcl interface - busy callback confusion script/procedure edit
In the Tcl interface the "busy" method doesn't work if a script is supplied instead of a procedure:

  %  package req sqlite3
  3.5.1
  % sqlite3 db hl7journal.db
  %  db busy {puts busy; expr 0}
  %  db eval {select count(*) from incoming}
  busy
  database is locked

Here the callback script is only invoked once, even though it returns 0. If we put this code in a procedure it works as desired/expected:

  % proc b args {puts busy; after 1000; expr 0}
  % db  busy b
  % db eval {select count(*) from incoming}
  busy
  busy
  busy
  ^C
2008-Feb-01 12:31:45 by anonymous:
After researching this a little it appears this happens because the busy callback is invoked with an extra argument. The extra argument leads to an error but that error is only visible through errorInfo, not the result.

I humbly suggest the following changes:

* mention the extra argument in the documentation of the Tcl interface

* forward the error from the busy callback to Tcl (replacing the "database is locked")

* enhance errorInfo to make the invokation of the busy callback apparent.

Currently, I'm getting this errorInfo:

  % db busy {puts hi; after 1000; return 0}
  %  db eval "select count(*) from incoming"
  hi
  database is locked
  %  set errorInfo
  bad option "0": must be -code, -errorcode, or -errorinfo
      while executing
  "return 0 0"
      invoked from within
  "db eval "select count(*) from incoming""

It would be nicer to get something like

  bad option "0": must be -code, -errorcode, or -errorinfo
      while executing
  "return 0 0"
      from busy callback
  "puts hi; after 1000; return 0 0"
      invoked from within
  "db eval "select count(*) from incoming""
 
2916 code active 2008 Feb anonymous   2008 Feb   1 1 sqlitedll-3_5_5.zip is older 3.5.4 binary edit
sqlitedll-3_5_5.zip in download section is same with old 3.5.4 binary.
2008-Feb-01 12:13:04 by anonymous:
Yes , I can confirm it
 
2915 build active 2008 Feb pweilbacher   2008 Feb   4 3 fix Makefile for platforms that need .exe extension edit
There are some targets that need a $(TEXE) added in Makefile.in. Otherwise a platform that needs a .exe extension cannot run e.g. tests.
 
2914 code active 2008 Jan anonymous   2008 Jan   3 3 ATTACH returns SQLITE_ERROR when it means SQLITE_BUSY edit
I'm seeing the same behavior as in #2096, with SQLite 3.5.4. ATTACH DATABASE fails with SQLITE_ERROR rather than SQLITE_BUSY when the database to be attached, or the main database of the connection being attached to, is EXCLUSIVE-locked by another database connection. For added confusion, sqlite3_errmsg() says "database is locked" when the ATTACH is done via sqlite3_exec(), but "SQL logic error or missing database" when the ATTACH is done via sqlite3_step().

As a result of this bug, it is difficult to distinguish between fatal and transient ATTACH errors, particularly when sqlite3_step() is used.

I am attaching a test program that demonstrates the problem.

 
2913 code fixed 2008 Jan anonymous   2008 Jan   3 3 vtab regression failures edit
FYI:

    vtab6-2.2...
    Expected: [1 2 3 {} 2 3 4 1 3 4 5 2]
	 Got: [1 2 3 {} 2 3 4 {} 3 4 5 {}]
    vtab6-2.3... Ok
    vtab6-2.4...
    Expected: [1 2 3 {} {} {} 2 3 4 {} {} {} 3 4 5 1 2 3]
	 Got: [1 2 3 {} {} {} 2 3 4 {} {} {} 3 4 5 {} {} {}]
    vtab6-2.5...
    Expected: [2 3 4 {} {} {} 3 4 5 1 2 3]
	 Got: [2 3 4 {} {} {} 3 4 5 {} {} {}]
    vtab6-2.6...
    Expected: [1 2 3 {} {} {} 2 3 4 {} {} {}]
	 Got: [1 2 3 {} {} {} 2 3 4 {} {} {} 3 4 5 {} {} {}]
    vtab6-3.1... Ok
    vtab6-3.2... Ok
    vtab6-3.3... Ok
    vtab6-3.4... Ok
    vtab6-3.5... Ok
    vtab6-3.6... Ok
    vtab6-3.7... Ok
    vtab6-3.7... Ok
    vtab6-4.1... Ok
    vtab6-4.2... Ok
    vtab6-4.3... Ok
    vtab6-4.4... Ok
    vtab6-4.6... Ok
    vtab6-4.7... Ok
    vtab6-4.8... Ok
    vtab6-4.9... Ok
    vtab6-4.10... Ok
    vtab6-7.1...
    Expected: [1 999 999 2 131 130 999]
	 Got: [1 999 999 999 999 999 999]
    vtab6-8.1... Ok
    vtab6-8.2... Ok
    vtab6-8.3... Ok
    vtab6-9.1... Ok
    vtab6-9.1.1...
    Expected: []
	 Got: [2 22 {}]
    vtab6-9.2...
    Expected: []
	 Got: [2 22 {}]
    vtab6-10.1... Ok
 
2912 new active 2008 Jan anonymous   2008 Jan   5 4 merge join edit
Would it be possible to implement merge joins?
 
2911 code active 2008 Jan anonymous   2008 Jan   2 2 Adding parentheses to a FROM clause edit
Hi,

Parentheses in a FROM statement seem to mess with the ability to use table aliases in the "what" part. Here is an example:

Start SQLite:

    $ sqlite3 employee.db
    SQLite version 3.5.4
    Enter ".help" for instructions

create a couple of tables and populate them with test data:

    sqlite> create table person (id integer, name text, employerid integer);
    sqlite> create table employer (id integer, name text);
    sqlite> insert into person (id, name, employerid) values (1, "Dave", 1);
    sqlite> insert into employer (id, name) values (1, "ACME");

Run a simple query with no parentheses in the FROM statement:

    sqlite> select b.id from person as a inner join employer as b on a.employerid = b.id;
    1

Everything works as expected. Now, repeat that query with parentheses:

    sqlite> select b.id from (person as a inner join employer as b on a.employerid = b.id);
    SQL error: no such column: b.id

There you have it.

This may be related to ticket #1822, although that ticket deals with aliases and subqueries. This problem seems to be more fundamental.

Many thanks,

-- Dave

 
2910 new closed 2008 Jan anonymous   2008 Jan   1 2 Number of rows in result set - new API edit
Performance problem to retrieve number of records in result set after executed any query using prepare/step is evident. On DB around 200MB using one table with 15.000 records having around 10 fieds, among it's one blob field with data less than 10KB in each record, executed query with 'where' clausule using prepare/step (have no blob field listed) last around 30s to only count records in result set. SQLIte3_Get_Table need only 3s to returning all data as well. I I'm aware that many asserts may slow down the whole process dramatically, however not 10 times.

Since there is no an API to retrieve number of records in result set and as prepare/step is far to slow, suggestion is to create one API, with maximum optimization for the purpose.

2008-Jan-31 02:00:07 by anonymous:
To discover the number of rows, the VDBE engine should iterate over every record of the result. This is the same as calling sqlite_step() until EOF.

But you can't do that, because after each sqlite_step(), you lost in memory data fields fetched for each row, so the way is counting rows after iteration.

Also, tickets should be used for reporting bugs only.


2008-Jan-31 04:24:29 by danielk1977:
Internally, sqlite3_get_table() uses sqlite3_exec() which uses sqlite3_prepare()/step()/finalize(). So you should look for another reason why get_table() is 10 times faster than prepare()/step()/finalize().

You could also try a "SELECT count(*) ..." query to determine the expected size of a result set.

As the above poster points out, there is no way to efficiently add this capability to SQLite (at least, no more efficiently than evaluating a count(*) query before the "real" query).

Dan.


2008-Jan-31 08:19:05 by anonymous:
Anonymous: As allowed during ticket adding, I propose new feature.

Dan: Since this feature should speed-up performance of a SQLite wrapper, "SELECT count(*) ..." is a real problem (require careful parsing to redesine gived query). "SELECT count(*) ..." itself is reasonably fast.


2008-Jan-31 08:54:56 by anonymous:
In addition to upper. "SELECT count(*)..." exactly last around 3s, Get_Table last as well around 3s, prepare/step/finalyze to only count numbers of records last 30s. This clearly indicate radically performance lost in prepare/step/finalyze.
 
2909 build active 2008 Jan aswift   2008 Jan aswift 4 4 Omit the _XOPEN_SOURCE 500 define on Mac OS-X (check for __APPLE__) edit
sqliteInt.h attempts to avoid setting _XOPEN_SOURCE on MacOSX by checking if __DARWIN__ is defined, however on Leopard (as of Mac OS X 10.5) this test fails and should be enhanced by testing for both __DARWIN__ and __APPLE__

Building sqlite succeeds with _XOPEN_SOURCE defined, but it causes _POSIX_C_SOURCE to be defined which leads to failing to define F_FULLFSYNC which prevents use of the fullfsync pragma.

 
2908 code active 2008 Jan anonymous   2008 Jan   3 3 Add support to examine whether statements modify the database edit
Currently there is no way to check whether a compiled statement will modify the database when being executed. Of course, there is the work-around of misusing the authorizer callback for this purpose, but this is kinda error prone and causes quite some overhead for such a simple purpose.
 
2907 code active 2008 Jan anonymous   2008 Jan   1 1 Issues of sqlite3 with Windows Mobile 5/6 edit
hi. we are currently using sqlite3 for our mobile application. it has been running without a hitch on pocket pc 2003 and previous versions. come windows mobile 5 and 6 we have been getting errors, although not consistent yet. one example is 'EXCEPTION_DATATYPE_MISALIGNMENT'. another is 'SELECT STATMENTS TO THE LEFT AND RIGHT OF UNION ARE NOT EQUAL'. i was wondering if you have any known compatibility issues of your product with this version of windows mobile.

thanks in advance.

2008-Jan-28 13:26:26 by anonymous:
EXCEPTION_DATATYPE_MISALIGNMENT is thrown when you try to use and Odd pointer address. I wrote a custom allocator for WinCE/ARM platform, and I have to take care about memory alignment (I used to align at 2 bytes, and at that time it solved the problem)
 
2906 code closed 2008 Jan anonymous   2008 Jan anonymous 2 2 Error in MSVC v.6.0 (optimization: Maximize Speed mode) edit
Hello, I am Vyacheslav Pindura (Kiev, Ukraine).

I will detect error in MSVC v.6.0 (optimization: Maximize Speed mode). Look accessPayload() function in amalgamation project (the same in btree.c in separeted project).

This is fragment from sqlite.cod (listing generated by MSVC compiler). Look my comment "//SLA: ..."

<<<<<< BEGIN LISTING
;	COMDAT _accessPayload
_TEXT	SEGMENT
_pCur$ = 8
_offset$ = 12
_amt$ = 16
_pBuf$ = 20
_skipKey$ = 24
_eOp$ = 28
_nKey$ = -12
_iIdx$ = -8
_pPage$ = -4
_pBt$ = -4
_ovflSize$62039 = 24
_nextPage$62040 = 12
_pDbPage$62056 = -12
_accessPayload PROC NEAR				; COMDAT

; 29385: ){

  00000	83 ec 0c	 sub	 esp, 12			; 0000000cH
  00003	53		 push	 ebx
  00004	55		 push	 ebp
  00005	56		 push	 esi

; 29386:   unsigned char *aPayload;
; 29387:   int rc = SQLITE_OK;
; 29388:   u32 nKey;
; 29389:   int iIdx = 0;
; 29390:   MemPage *pPage = pCur->pPage;     /* Btree page of current cursor entry */

  00006	8b 74 24 1c	 mov	 esi, DWORD PTR _pCur$[esp+20]
  0000a	57		 push	 edi
  0000b	33 db		 xor	 ebx, ebx
  0000d	8b 7e 1c	 mov	 edi, DWORD PTR [esi+28]

; 29391:   BtShared *pBt;                   /* Btree this cursor belongs to */
; 29392:
; 29393:   assert( pPage );
; 29394:   assert( pCur->eState==CURSOR_VALID );
; 29395:   assert( pCur->idx>=0 && pCur->idx<pPage->nCell );
; 29396:   assert( offset>=0 );
; 29397:   assert( cursorHoldsMutex(pCur) );
; 29398:
; 29399:   getCellInfo(pCur);

  00010	56		 push	 esi
  00011	89 5c 24 18	 mov	 DWORD PTR _iIdx$[esp+32], ebx
  00015	89 7c 24 1c	 mov	 DWORD PTR _pPage$[esp+32], edi
  00019	e8 00 00 00 00	 call	 _getCellInfo

; 29400:   aPayload = pCur->info.pCell + pCur->info.nHeader;

  0001e	8b 4e 24	 mov	 ecx, DWORD PTR [esi+36]

; 29401:   nKey = (pPage->intKey ? 0 : pCur->info.nKey);

  00021	8a 47 03	 mov	 al, BYTE PTR [edi+3]
  00024	33 ed		 xor	 ebp, ebp
  00026	83 c4 04	 add	 esp, 4
  00029	66 8b 6e 38	 mov	 bp, WORD PTR [esi+56]
  0002d	03 e9		 add	 ebp, ecx
  0002f	84 c0		 test	 al, al
  00031	74 08		 je	 SHORT $L80642
  00033	33 c0		 xor	 eax, eax
  00035	89 44 24 10	 mov	 DWORD PTR _nKey$[esp+28], eax
  00039	eb 0a		 jmp	 SHORT $L80643
$L80642:
  0003b	8b 4e 2c	 mov	 ecx, DWORD PTR [esi+44]
  0003e	8b 4e 28	 mov	 ecx, DWORD PTR [esi+40]
  00041	89 4c 24 10	 mov	 DWORD PTR _nKey$[esp+28], ecx
$L80643:

; 29402:
; 29403:   if( skipKey ){

  00045	8b 4c 24 30	 mov	 ecx, DWORD PTR _skipKey$[esp+24]	//SLA:  ecx:=skipKey

; 29404:     offset += nKey;

  00049	8b 7c 24 24	 mov	 edi, DWORD PTR _offset$[esp+24]	//SLA:  edi:=offset
  0004d	3b cb		 cmp	 ecx, ebx
  0004f	74 02		 je	 SHORT $L62032
  00051	03 f9		 add	 edi, ecx				//SLA:  offset += skipKey   !!!
$L62032:

; 29405:   }
; 29406:   if( offset+amt > nKey+pCur->info.nData ){

  00053	8b 54 24 28	 mov	 edx, DWORD PTR _amt$[esp+24]
  00057	8b 46 30	 mov	 eax, DWORD PTR [esi+48]
  0005a	03 44 24 10	 add	 eax, DWORD PTR _nKey$[esp+28]
  0005e	8d 0c 17	 lea	 ecx, DWORD PTR [edi+edx]		//SLA:  edi have (offset+skipKey) !!!
  00061	3b c8		 cmp	 ecx, eax
  00063	76 0d		 jbe	 SHORT $L62033
  00065	5f		 pop	 edi
  00066	5e		 pop	 esi
  00067	5d		 pop	 ebp

; 29407:     /* Trying to read or write past the end of the data is an error */
; 29408:     return SQLITE_ERROR;

  00068	b8 01 00 00 00	 mov	 eax, 1
  0006d	5b		 pop	 ebx

; 29510: }
<<<<<<< END LISTING

So, optimization imposible !!! We need disable optimization in MSVC project :(

Best regards, Vyacheslav.

2008-Jan-27 12:51:32 by drh:
See also ticket #2657. There have been other reports of problems with MSCV 6.0 and SQLite, some of which result in an "Internal Compiler Error" and others which just cause incorrect code to be generated.

At this point we recommend that you not use MSVC 6.0 to build SQLite or any other project for that matter. If you must use MSVC 6.0, at least disable optimization.

To be fair to MSVC, other compilers also have bugs. Ticket #2469 is an example of an optimizer bug in GCC which has since been fixed.

Thank you for reporting this problem. But this really is a bug in MSVC, not in SQLite, so there is nothing that we can do about it.

 
2905 code active 2008 Jan anonymous   2008 Jan pweilbacher 2 2 mutex_os2.c - incorrect mutex implementation edit
The OS/2 version of sqlite3_mutex_alloc() is badly broken.

It creates named mutexes which, by design, are global rather than process-specific as intended. This might be minimally acceptable except that the function reuses the same name every time it attempts to create a SQLITE_MUTEX_FAST or SQLITE_MUTEX_RECURSIVE.

The result is that every call returns the exact same semaphore to every thread in every process using sqlite3. Once this mutex is owned by one process, other processes calling sqlite3_mutex_enter() will be blocked. Much the same is true for the static mutexes. Every process ends up using the exact same SQLITE_MUTEX_STATIC_MASTER, SQLITE_MUTEX_STATIC_MEM, etc.

There's another flaw that is fairly minor compared with the above: in an attempt to avoid concurrency when creating the static mutexes, this function uses an API call that is thoroughly deprecated.

The attached patch remedies all of these issues. Since the logic that protects the creation of the static mutexes may not be self-evident, here's an explanation:

The existence (or non-existence) of a given named mutex is itself a semaphore. If the isInit flag is false, the code attempts to create a mutex whose name is unique to that process. If the attempt is successful, there are two possibilities: (1) either the current thread is the first to reach this code & may proceed; (2) or while the current thread was making its preparations, another thread created the mutex, did the init, then closed the mutex. Testing isInit immediately after creating the mutex determines which possibility is valid.

If mutex creation fails due to a duplicate name, then another thread is currently performing the init. In this case, the current thread simply has to wait a while until the other thread is done & isInit becomes true.

Submitted by Rich Walsh (richATe-vertise.com)

 
2904 code fixed 2008 Jan anonymous   2008 Feb pweilbacher 3 3 os_os2.c - 2 dysfunctional functions edit
The attached patch fixes os2Truncate() and os2FullPathname().

Currently, os2Truncate() does nothing useful. It moves the pointer associated with a file handle but has no effect on the file's size. The patch provides code that will change the actual size of a file on the disk.

os2FullPathname() is overly complex and fails in marginal cases. E.g. for file "c:\programs\myapp\mydata.db", it fails to return a fully-qualified path when given "c:mydata.db". The patch replaces existing code with a single API call that always returns a valid result.

Submitted by Rich Walsh (richATe-vertise.com)

 
2903 code active 2008 Jan anonymous   2008 Jan   2 2 tclinstaller.tcl fails on path and permissions issue edit
When compiling using custom PREFIX, pointing to private directory, tclinstaller.tcl fails, because it tries to remove contents from /usr/share/tcl8.4/sqlite3.

./configure --prefix=/my/private/sqlite/sqlite-3.5.4
... # success
make
... # success
make install
...
tclsh ./tclinstaller.tcl 3.5
error deleting "/usr/share/tcl8.4/sqlite3": not owner
    while executing
"file delete -force $LIBDIR/sqlite3"
    (file "./tclinstaller.tcl" line 17)
make: *** [tcl_install] Error 1
I've found two work-arounds:

  1. If you run make install as root.
  2. If you use ./configure --disable-tcl
2008-Jan-28 17:47:39 by anonymous:
I also ran into this problem.

make install as root will end up copying files into the system's library directory and is almost certainly not what you want if you specified your own --prefix.

 
2902 code active 2008 Jan anonymous   2008 Jan drh 3 3 Add watch support to SQLite edit
SQLite currently provides only TRIGGERs and the update_hook() as a way for applications to stay informed about changes to the database. But both of these alternatives do not provide enough details about the actual changes to the underlying database file(s).

We've prepared a patch for SQLite 3.5.x to allow applications to install a watch_hook into the database, that will be invoked everytime the database is changed with exact details about the change that was performed.

2008-Jan-22 16:06:59 by anonymous:
Great idea and nice job. This functionality is very useful.


2008-Jan-31 18:27:16 by anonymous:
Any chance to get this committed for the next release (i.e. 3.5.6)?


2008-Jan-31 19:38:35 by drh:
Unlikely, for two reasons:

  1. I am unconvinced that this patch solves a problem that needs solving. It is vitally important to a project like SQLite that we work to avoid clutter and cruft. That means that any change must have a compelling rational or else it is rejected.

  2. The patches are against version 3.5.4. There were many changes to the core for 3.5.5 and the patches no longer work.


2008-Jan-31 20:58:55 by anonymous:
We can (and will) port the changes to 3.5.5, so the second point will be done. First the first point, I'm not sure how many projects will actually need this functionality, but I guess there are quite a lot of projects that would benefit, and for the others, there's zero overhead due to this patch.


2008-Jan-31 21:21:30 by drh:
There is a lot of overhead for me because if I accept this patch, that means I have to maintain it forever. Most of the work is in maintenance, not coming up with the original patch.
 
2901 code active 2008 Jan anonymous   2008 Jan   3 3 ROLLBACK and COMMIT statements should not expire edit
Currently, whenever a statement changes the schema of the database, all prepared statements will be expired, no matter whether they actually need to be prepared again or not. This is especially problematic for ROLLBACK statements in a multi-statement transaction. Currently there is no way to guaranty that a multi-statement transaction can at least be rolled back in case of an error, because one has to (re)prepare the ROLLBACK statement to roll back the transaction, which can fail because of OOM (in a multi-threaded application).
 
2900 code closed 2008 Jan anonymous   2008 Jan   3 4 Can't ESCAPE in GLOB edit
If you apply an ESCAPE clause to the GLOB operator, you get the message:

SQL error: wrong number of arguments to function glob()

You can reproduce this on Mac OS X 10.5 with the following:

$ sqlite3 test.db
SQLite version 3.4.0
Enter ".help" for instructions
sqlite> create table Identifiers ( ident text );
sqlite> insert into Identifiers values ( "foo?" );
sqlite> insert into Identifiers values ( "foo" );
sqlite> insert into Identifiers values ( "foo_" );
sqlite> select * from Identifiers where ident like "foo\_" escape "\";
foo_
sqlite> select * from Identifiers where ident glob "foo\?" escape "?";
SQL error: wrong number of arguments to function glob()
I believe that this is because sqlite3RegisterLikeFunctions (in "func.c") registers the "like" function with either 2 or 3 arguments, but only registers the "glob" function with 2 arguments.
2008-Jan-22 13:04:28 by drh:
Escape characters in a GLOB as you would in a shell glob, which is to put them inside of a [...]. Example:

   select * from identifiers where ident glob 'foo[?]';

Note also the use of single-quotes instead of double-quotes around the string literal. SQL requires single-quotes.

 
2899 code active 2008 Jan anonymous   2008 Jan   4 4 sqlite3_reset() after exec() takes > 100 ms to complete edit
I'm not entirely sure whether this is a bug or not... I'm not familiar enough with SQLite to know if this is way too unconventional, but I noticed today that running exec() in conjunction with a prepared statement really kills the performance of sqlite3_reset(), if it's called after exec():

// init sqlite3_prepare_v2( db,
"SELECT [file_id],[file_name],[file_mime],[file_type],"
"[file_size],[date_created],[date_accessed],[data] "
"FROM file_cache WHERE [file_id] = ? LIMIT 1;",
-1, &sqQueryStatement, &unused );

// query function -- called repeatedly (usually second or third run starts to cause big delays)

/* begin function */

sqlite3_reset( sqQueryStatement );
sqlite3_bind_int( sqQuerystatement, 1, 292 );
sqlite3_step( sqQueryStatement );

bla = sqlite3_column_int( sqQueryStatement, 0 );

/* ... */

char *erm;

sqlite3_exec( db, "UPDATE file_cache SET [date_accessed] = DATETIME( 'NOW', 'LOCALTIME' ) WHERE [file_id] = 292", NULL, NULL, &erm );

/* ... */

sqlite3_clear_bindings( sqQueryStatement );

/* end function */

If sqlite3_exec() is commented out, consecutive calls to the function run in less than a millisecond. With sqlite3_exec() included, sqlite3_reset() call in this function takes > 100 ms to complete.

Tested: Windows Vista Ultimate 32 bit, Visual Studio 2005 8.0.50727.876

 
2898 code active 2008 Jan anonymous   2008 Jan   1 1 Latest CVS for 3.5.4 fails to build test1.c edit
gcc -pipe -O3 -g -Wall -DSQLITE_DISABLE_DIRSYNC=1 -I. -I../src -DNDEBUG -I/usr/include -DSQLITE_THREADSAFE=1 -DSQLITE_THREAD_OVERRIDE_LOCK=1 -DSQLITE_OMIT_LOAD_EXTENSION=1 -DTCLSH=1 -DSQLITE_TEST=1 -DSQLITE_CRASH_TEST=1 -DSQLITE_NO_SYNC=1 -DTEMP_STORE=1 -o .libs/testfixture ../src/attach.c ../src/btree.c ../src/build.c ../src/date.c ../src/expr.c ../src/func.c ../src/insert.c ../src/malloc.c ../src/os.c ../src/os_os2.c ../src/os_unix.c ../src/os_win.c ../src/pager.c ../src/pragma.c ../src/prepare.c ../src/printf.c ../src/select.c ../src/test1.c ../src/test2.c ../src/test3.c ../src/test4.c ../src/test5.c ../src/test6.c ../src/test7.c ../src/test8.c ../src/test9.c ../src/test_autoext.c ../src/test_async.c ../src/test_btree.c ../src/test_config.c ../src/test_hexio.c ../src/test_malloc.c ../src/test_md5.c ../src/test_onefile.c ../src/test_schema.c ../src/test_server.c ../src/test_tclvar.c ../src/test_thread.c ../src/tokenize.c ../src/utf.c ../src/util.c ../src/vdbe.c ../src/vdbeapi.c ../src/vdbeaux.c ../src/vdbemem.c ../src/where.c parse.c ../src/tclsqlite.c ./.libs/libsqlite3.so -L/usr/lib64 -ltcl8.4 -ldl -lpthread -lieee -lm -Wl,--rpath -Wl,/common/pkgs/sqlite-3.5.4.3/lib ../src/build.c: In function 'sqlite3RefillIndex': ../src/build.c:2275: warning: cast to pointer from integer of different size ../src/func.c: In function 'trimFunc': ../src/func.c:919: warning: cast from pointer to integer of different size ../src/func.c: In function 'sqlite3RegisterBuiltinFunctions': ../src/func.c:1464: warning: cast to pointer from integer of different size ../src/func.c:1483: warning: cast to pointer from integer of different size ../src/insert.c: In function 'sqlite3GenerateConstraintChecks': ../src/insert.c:1200: warning: cast to pointer from integer of different size ../src/insert.c:1034: warning: 'j2' may be used uninitialized in this function ../src/insert.c: In function 'sqlite3Insert': ../src/insert.c:373: warning: 'regFromSelect' may be used uninitialized in this function ../src/test1.c: In function 'test_collate_func': ../src/test1.c:2085: warning: cast from pointer to integer of different size ../src/test1.c: In function 'test_collate_needed_cb': ../src/test1.c:2209: warning: cast to pointer from integer of different size ../src/test1.c: In function 'alignmentCollFunc': ../src/test1.c:2258: warning: cast from pointer to integer of different size ../src/test1.c:2259: warning: cast from pointer to integer of different size ../src/test8.c: In function 'echoBestIndex': ../src/test8.c:722: warning: 'nRow' may be used uninitialized in this function ../src/vdbe.c: In function 'sqlite3VdbeExec': ../src/vdbe.c:502: warning: 'pOut' may be used uninitialized in this function ../src/vdbe.c:501: warning: 'pIn3' may be used uninitialized in this function ../src/vdbe.c:501: warning: 'pIn2' may be used uninitialized in this function ../src/vdbe.c:501: warning: 'pIn1' may be used uninitialized in this function ../src/vdbeaux.c: In function 'sqlite3VdbeChangeP4': ../src/vdbeaux.c:529: warning: cast from pointer to integer of different size ../src/vdbemem.c: In function 'sqlite3ValueText': ../src/vdbemem.c:911: warning: cast from pointer to integer of different size /tmp/ccsuOeus.o: In function `reset_prng_state': /build/work/sqlite-3.5.4.3/bld/../src/test1.c:4280: undefined reference to `sqlite3ResetPrngState' /tmp/ccsuOeus.o: In function `restore_prng_state': /build/work/sqlite-3.5.4.3/bld/../src/test1.c:4267: undefined reference to `sqlite3RestorePrngState' /tmp/ccsuOeus.o: In function `save_prng_state': /build/work/sqlite-3.5.4.3/bld/../src/test1.c:4254: undefined reference to `sqlite3SavePrngState' collect2: ld returned 1 exit status make: *** [testfixture] Error 1
2008-Jan-17 23:54:58 by anonymous:
Problem appears to be here in libsqlite.3.so.0.8.6 as shown by:

   nm -A .libs/libsqlite3.so.0.8.6 | grep sqlite3ResetPrngState

which shows no entry point. And:

   nm -A .libs/random.o  | grep sqlite3ResetPrngState

which also shows no entry point.


2008-Jan-17 23:56:55 by anonymous:
Ah... It appears -DSQLITE_TEST should be passed when building test1.c and left off when building prior to install.


2008-Jan-21 20:16:00 by anonymous:
In the makefile the right flag appears to be set, it's just not making it through to the compile for some reason.


2008-Jan-21 20:16:24 by anonymous:
Still fails the same based on today's cvs update.


2008-Jan-23 03:14:49 by anonymous:
This bug fixed as of latest cvs pull
 
2897 code active 2008 Jan anonymous   2008 Jan   1 1 String or BLOB exceed size limit edit
This error was shown after attemp to read script from SQLite 3.5.4 shell in order to recreate old DB. Details:

1. Database was created with SQLite 3.3.4. Around 20 standard fieds and one BLOB.

2. The only one existed table was dumped with shell of SQLite 3.5.4. SQL script seems to be coorrect.

3. Opened SQLite 3.5.4 and read script in new DB. The error "String or BLOB exceed size limit" are sown for several lines. Many records missing.

4. Attempted to dump table with shell of version 3.3.6 (have no more 3.3.4 shell) and read into new DB with 3.5.4 shell The same errors are shown.

The same steps was attempted with 3.3.6. shell only. All seems to be correct.

2008-Jan-17 20:23:25 by drh:
This size limit on BLOBs in SQLite version 3.5.4 is 1GB. How big is your blob, exactly?


2008-Jan-17 22:22:24 by anonymous:
BLOB in each record is no more than few MB. Mostly it is few KB (e-client and news application). Whole DB have around 200MB.


2008-Jan-18 02:28:11 by drh:
This issue is probably resolved by check-in [4636] , then.


2008-Jan-18 14:28:13 by anonymous:
If directive SQLITE_MAX_SQL_LENGTH is not defined it is set to 1,000,000 (10^6) in amalgamation code of 3.5.4.
 
2896 code closed 2008 Jan anonymous   2008 Jan   1 1 FTS3 misses document, but finds non-matching ones instead edit
The following script presents a severe bug in FTS3 where it does not find a document even though it is in the index. Instead it finds other documents which do not match the search criteria.

The log below was generated by the original SQLite3.exe 3.5.4 downloaded from the web-site. It shows that the entry 'A-123' is present in the table but is not matched by the query. It also demonstrates that non-matching entries are found instead.

  DROP TABLE IF EXISTS fts3;
  CREATE VIRTUAL TABLE fts3 using fts3 (a);

  INSERT INTO fts3 VALUES ('A-123');
  SELECT * FROM fts3;
  A-123
  SELECT * FROM fts3 WHERE fts3 MATCH 'A-123';

  INSERT INTO fts3 VALUES ('A-12');
  SELECT * FROM fts3 WHERE fts3 MATCH 'A-123';
  A-12

  INSERT INTO fts3 VALUES ('A-1');
  SELECT * FROM fts3 WHERE fts3 MATCH 'A-123';
  A-12
  A-1

Here is the SQL only (without output, for easy reproduction):

  DROP TABLE IF EXISTS fts3;
  CREATE VIRTUAL TABLE fts3 using fts3 (a);

  INSERT INTO fts3 VALUES ('A-123');
  SELECT * FROM fts3;
  SELECT * FROM fts3 WHERE fts3 MATCH 'A-123';

  INSERT INTO fts3 VALUES ('A-12');
  SELECT * FROM fts3 WHERE fts3 MATCH 'A-123';

  INSERT INTO fts3 VALUES ('A-1');
  SELECT * FROM fts3 WHERE fts3 MATCH 'A-123';
2008-Jan-17 19:36:18 by anonymous:
Sorry, I must have been dreaming: This is of course not a bug but the expected behavior. The '-' character is supposed to exclude the word following it.

My sincere apologies if this has caused doubt or uncertainties to anyone!

 
2895 build closed 2008 Jan anonymous   2008 Jan   3 1 Please describe how to run an indvidual test out of the test suite edit
Please describe how to run an individual test out of the test suite? Thanks.

For example, how to run the test incrvacuum-ioerr-1.31.4

2008-Jan-17 17:14:12 by drh:
Tickets are for reporting bugs, not asking questions about how to compile or operate SQLite. For questions of that nature, please post comments on the SQLite mailing list. See http://www.sqlite.org/support.html.

PS: You cannot run individual tests. You have to run an entire script.

 
2894 code fixed 2008 Jan anonymous   2008 Jan   1 1 virtual table calls Next() and Column() after Eof() returns true edit
I saw this problem in my virtual table implementation. When I had a query that used a left outer join on a column that could be NULL, Next() and Column() were called after Eof() had returned true. I tried to reproduce the problem using test_schema from the source distribution which resulted in a segmentation fault, and I assume it's the same defect. Here is a sample script:

.load test_schema.so

create virtual table x using schema;

-- this produces results

select * from x where dflt_value is null;

-- this produces a segmentation fault

select * from x lhs left outer join x rhs on lhs.dflt_value = rhs.dflt_value;

 
2893 code active 2008 Jan anonymous   2008 Jan   1 1 incorrect integer range tests edit
recently a function that performs integer range tests was added to the cvs (check-in [4706] ), but if i am correct there is a problem in the return value of the function in the file vdbemem.c:

  static i64 doubleToInt64(double r){
    ...
    if( r<(double)minInt ){
      return minInt;
    }else if( r>(double)maxInt ){
      return minInt;     <-- is this correct, shouldn't it be maxInt?
    }else{
      return (i64)r;
    }
  }
2008-Jan-16 17:33:56 by drh:
See the remarks on ticket #2280. The code duplicates the behavior of the FPU on x86.


2008-Jan-16 18:21:28 by anonymous:
did you mean ticket #2880? didn't read that ticket before, but since there was no comment regarding that behavior in the function it seemed (to my eyes) that it was a mistake.

maybe adding a small comment in there would clarify this issue


2008-Jan-16 18:39:42 by anonymous:
Just because the double to int overflow behavior happens to be that way with GCC on x86, is it desirable?
 
2892 new active 2008 Jan anonymous   2008 Jan   5 5 There should be a way in the api to read more precise error message. edit
Currently all errors in sql queries issued via C,C++ api result in error code SQLITE_ERROR = 1 and message "SQL error or missing database".

That leaves user completely clueless what mistake in his sql he actually made.

Some evidence of confusion can be found here: http://bugs.php.net/bug.php?id=33117

2008-Jan-16 17:41:00 by drh:
SQLite gives detailed error information for SQL syntax or logic errors. (Try, for example, entering invalid SQL into the CLI.) I think perhaps that PHP is simply failing to to access those errors and is instead picking up some other error indication from someplace else.


2008-Jan-16 21:59:17 by anonymous:
Well I shall retest it, but I got the same error message upon a duplicate key in Delphi. Although I'm aware of an enhanced api to get the errormessage.


2008-Jan-17 00:05:58 by drh:
What error message do you get from the CLI?
 
2891 code closed 2008 Jan anonymous   2008 Jan   2 3 col with "NOT NULL", no default: works on table create, not col add edit
  1. WORKS: CREATE TABLE my_table ("my_column" integer NOT NULL);
  2. GIVES ERROR: ALTER TABLE my_table ADD "my_other_column" integer NOT NULL;

(2) gives the error: "SQL error: Cannot add a NOT NULL column with default value NULL"

This is inconsistent, either both cases should work, or both should give an error.

2008-Jan-15 19:59:03 by anonymous:
This is default in all SQL database managers.

If a column can't have a default NULL value, how it will be added to a table with existing rows ?

How that columns will have the default column value is there&#180;s no default and has the NOT NULL constraint ?


2008-Jan-16 00:15:50 by drh:
Anonymous poster above is correct. This is not a bug.
 
2890 build closed 2008 Jan anonymous   2008 Jan   1 1 Assistance request fixing six 3.5.4 i/o test failures edit
Failures include exclusive-ioerr-2.280.4 exclusive-ioerr-2.281.4 exclusive-ioerr-2.282.4 incrvacuum-ioerr-1.31.4 io-4.1 io-4.2.3

Please let me know how to run individual tests, so I can debug these and submit fix suggestions.

Thanks.

2008-Jan-15 21:50:52 by drh:
These errors do not appear when the tests are run individually. They are only reproducible when you run the complete test suite.


2008-Jan-16 03:15:50 by anonymous:
Please give an example of how to run one individual test.

Thanks.

 
2889 build closed 2008 Jan anonymous   2008 Jan   3 2 Cannot disable TCL edit
Slackware 12

TCL/TK is not installed, as I don't need it, and don't really want to install it

Sequence of commands to reproduce :

tar -xzvf sqlite-3.5.4.tar.gz

cd sqlite-3.5.4

mkdir bld

cd bld

../configure --disable-tcl --without-tcl

make

The last command dies with


rm tsrc/sqlite.h.in tsrc/parse.y

cp parse.c opcodes.c keywordhash.h tsrc

tclsh ../tool/mksqlite3c.tcl

make: tclsh: Command not found

make: *** [sqlite3.c] Error 127

root@slack12:~/install/sqlite-3.5.4/bld# ls ..


Thanks in advance

Roman

Same as #2845, #2346, #2871.
 
2888 code closed 2008 Jan anonymous   2008 Jan   3 4 statement not commited edit
code:

#include <stdio.h>

#include "sqlite3.h"

int main() {

  sqlite3 *db;

  sqlite3_stmt *stmt;

  sqlite3_open("test.db", &db);

  sqlite3_prepare(db, "SELECT 1", -1, &stmt, NULL);

  sqlite3_step(stmt);

  sqlite3_exec(db, "CREATE TABLE t(a INT)", NULL, NULL, NULL);

  printf("%s\n", sqlite3_errmsg(db));

  return -1;

}

CREATE TABLE isn't commited. There is no BEGIN, CREATE TABLE doesn't return an error, so I think table should be created.

2008-Jan-15 12:28:21 by drh:
The pending SELECT statement must complete before a COMMIT will occur. See the documentation.
 
2887 code closed 2008 Jan anonymous   2008 Jan anonymous 5 5 || not outputing correctly with empty fields edit
While using || to concatenate fields for output, the rows with a NULL field are not printed.

Example:

-all rows:

sqlite> select vend_city,vend_state,vend_zip from vendors;

vend_city vend_state vend_zip

---------- ---------- ----------

Bear Town MI 44444

Anytown OH 44333

Dollsville CA 99999

New York NY 11111

London N16 6PS

Paris 45678


-Using ||, the 2 last rows with NULL states do not show on the output:


sqlite> select vend_city ||','|| vend_state ||','|| vend_zip from vendors;

vend_city ||','|| vend_state ||','|| vend_zip


Bear Town,MI,44444

Anytown,OH,44333

Dollsville,CA,99999

New York,NY,11111

2008-Jan-15 12:30:30 by drh:
The expression

   NULL || <expr>

evaluates to NULL, regardless of the value of <expr>. This is in accordance with the SQL standard.

Use the coalesce() or ifnull() functions to work around this.

 
2886 code active 2008 Jan anonymous   2008 Jan   3 3 testfixture: -fPIC needed when building extension(s) edit
(this fix/change is probably needed in older versions too, i meant to send this in earlier)

-fPIC is needed when building extensions (some platforms don't need this or don't care --- x86-64 does)

  diff --git a/test/loadext.test b/test/loadext.test
  index 81e152f..2a7fa2e 100644
  --- a/test/loadext.test
  +++ b/test/loadext.test
  @@ -64,7 +64,7 @@ if {![file exists $testextension]} {
     set srcdir [file dir $testdir]/src
     set testextsrc $srcdir/test_loadext.c
     if {[catch {
  -    exec gcc -Wall -I$srcdir -I. -g -shared $testextsrc -o $testextension
  +    exec gcc -Wall -fPIC -I$srcdir -I. -g -shared $testextsrc -o $testextension
     } msg]} {
       puts "Skipping loadext tests: Test extension not built..."
       puts $msg
 
2885 code active 2008 Jan anonymous   2008 Jan   4 4 (minor) fulltest failures edit
Minor noise in fulltest on a 64-bit machine (everything seems to work otherwise though):

  4 errors out of 61527 tests
  Failures on these tests: exclusive-ioerr-2.280.4
  exclusive-ioerr-2.281.4 exclusive-ioerr-2.282.4 incrvacuum-ioerr-1.31.4
  All memory allocations freed - no leaks

details:

  exclusive-ioerr-2.280.3... Ok
  exclusive-ioerr-2.280.4...
  Expected: [8ee59fe0b5bc391ecc5002539379b063]
       Got: [35e56178ad878809d4789c5797265a23]
  exclusive-ioerr-2.281.1... Ok
  exclusive-ioerr-2.281.2... Ok
  exclusive-ioerr-2.281.3... Ok
  exclusive-ioerr-2.281.4...
  Expected: [95762fb35ef83ddba65f681325346ef2]
       Got: [1c20c63d975ad85b395a5e7701d785c2]
  exclusive-ioerr-2.282.1... Ok
  exclusive-ioerr-2.282.2... Ok
  exclusive-ioerr-2.282.3... Ok
  exclusive-ioerr-2.282.4...
  Expected: [2c5ea7db8424f38d17cdff41056da0e0]
       Got: [c02841c40d3e6e0579922745bd9c0260]
  exclusive-ioerr-2.283.1... Ok

[...]

    incrvacuum-ioerr-1.31.4...
    Expected: [ea55042a449c3b4759730a882e8271a0]
	 Got: [9951814984696d0811cacbe862060af8]
    incrvacuum-ioerr-1.32.1... Ok
2008-Jan-19 00:56:30 by anonymous:
A test from 20 minutes ago passes cleanly. This could be closed.
 
2884 new active 2008 Jan anonymous   2008 Jan   4 4 Way to find out limits edit
The page http://www.sqlite.org/limits.html shows some limits that SQLite has. They are consts in the code, and defaults are listed, but there doesn't seem to be any way to find out what values a particular binary was built with.

For example, I've gotten SQLite as part of my OS or another program, and I don't know what value of SQLITE_MAX_COLUMN was used to compile this SQLite. I'm trying to track down a bug in my code, and if this value was too small, that would explain it. (In truth, it's probably not the cause, but I'd like to rule it out.)

I'd like a way to print the value of these limits at runtime. It doesn't have to have a stable API -- it would mostly be useful for debugging. If the command-line client had a command ".limits" that printed all of these values, that would be super.

 
2883 code fixed 2008 Jan anonymous   2008 Jan   1 1 compile error - missing data edit
I have original sqlite 3.2.8 source code in external SVN and a dashboard has encountered the following compile error win gcc 4.x:

"shell.c:364: error: format not a string literal and no format arguments"

See http://mail.kde.org/pipermail/kde-dashboard/2008-January/008494.html

Any SQLite newer is addected as well. The patch below fixes misused fprintf.

Index: shell.c
===================================================================
--- shell.c (revision 761314)
+++ shell.c (working copy)
@@ -361,7 +361,7 @@
output_c_string(p->out, z);
}
if( bSep ){
- fprintf(p->out, p->separator);
+ fprintf(p->out, "%s", p->separator);
}
}

 
2882 code active 2008 Jan anonymous   2008 Jan   3 3 fulltest failure: ./testfixture: wrong # args: should be "cksum db" edit
  exclusive-ioerr-2.2.1... Ok

  ./testfixture: wrong # args: should be "cksum db"
      while executing
  "ifcapable vacuum {
    do_ioerr_test ioerr-2 -cksum true -sqlprep {
      BEGIN;
      CREATE TABLE t1(a, b, c);
      INSERT INTO t1 VALUES(1, randstr(50,..."
      (file "../test/ioerr.test" line 58)
      invoked from within
  "source $testdir/ioerr.test"
      (file "../test/exclusive3.test" line 50)
      invoked from within
  "source $testfile"
      ("foreach" body line 5)
      invoked from within
  "foreach testfile [lsort -dictionary [glob $testdir/*.test]] {
      set tail [file tail $testfile]
      if {[lsearch -exact $EXCLUDE $tail]>=0} continue
  ..."
      ("for" body line 7)
      invoked from within
  "for {set Counter 0} {$Counter<$COUNT && $nErr==0} {incr Counter} {
    if {$Counter%2} {
      set ::SETUP_SQL {PRAGMA default_synchronous=off;}
    } else ..."
      (file "..//test/all.test" line 85)
2008-Jan-14 23:25:49 by anonymous:
The latest code seems to have fixed this. I would close this but I don't see how to do that.
 
2881 build active 2008 Jan anonymous   2008 Jan   1 1 Latest sqlite-3.5.4 build fail on latest Fedora 2.6.23.12-52.fc7 edit
Two test cases fail.

  io-4.1...
  Expected: [3]
     Got: [2]
  io-4.2.1... Ok
  io-4.2.2... Ok
  io-4.2.3...
  Expected: [3]
     Got: [2]
  io-4.3.1... Ok

Let me know how to run individual test cases and how this might be fixed.

Here's how I built sqlite using latest CVS. If something is wrong here, let me know and I'll rebuild/retest.

I'm building on latest Fedora fc7.

Thanks.

_______

  net1#uname -a
  Linux net1.coolsurf.com 2.6.23.12-52.fc7 #1 SMP Tue Dec 18 20:27:10 EST 2007 x86_64 x86_64 x86_64 GNU/Linux

  net1#build_sqlite
  mkdir -p /build/work/sqlite-3.5.4
  cd /build/work/sqlite-3.5.4
  unset CDPATH
  export CFLAGS='-pipe -O3 -g -DSQLITE_DISABLE_DIRSYNC=1 -Wall'
  rm -rf bld
  cvs -d :pserver:anonymous@www.sqlite.org:/sqlite -r update .
  mkdir bld
  cd bld
  ../configure --prefix=/common/pkgs/sqlite-3.5.4 --enable-tcl --with-tcl=/usr/lib64 --enable-threadsafe --enable-threads-override-locks
  make
  groupadd vuser || /bin/true
  useradd -M -g vuser -d /vhost/davidfavor.com/users/david -s /bin/zsh   david  || /bin/true
  useradd -M -g vuser -d /vhost/livefeast.com/users/yemiah -s /bin/zsh yemiah || /bin/true
  chown david:vuser -R ..
  su -c "make test" david
2008-Jan-11 17:18:52 by anonymous:
Same tests still fail with CVS of today around 11AM CST.


2008-Jan-11 17:41:55 by drh:
FWIW, both those test cases pass on SuSE 10.1. I do not understand why they are failing on Fedora. But in any event, the tests in question are verifying logic that implements an optimization that is not used on Fedora, ever. So the failures are of no consequence. If those are the only two tests that fail, then you can safely use the build for whatever it is you are trying to do.


2008-Jan-11 19:16:19 by anonymous:
Failures when 'make fulltest' built with CFLAGS of

  '-pipe -O3 -g -Wall -DSQLITE_DISABLE_DIRSYNC=1 -DSQLITE_MEMDEBUG'

   exclusive-malloc-1.transient.746...make: *** [fulltest]
      Segmentation fault

Failures when 'make fulltest' built with CFLAGS of

   '-pipe -O3 -g -Wall -DSQLITE_DISABLE_DIRSYNC=1'

   Skipping malloc tests: not compiled with -DSQLITE_MEMDEBUG...
   6 errors out of 61998 tests
   Failures on these tests: exclusive-ioerr-2.280.4
      exclusive-ioerr-2.281.4 exclusive-ioerr-2.282.4
      incrvacuum-ioerr-1.31.4 io-4.1 io-4.2.3
   All memory allocations freed - no leaks
   Maximum memory usage: 14376554 bytes

Pre 3.5.x builds work fine on Fedora. If you're open to debugging all these, I'd like to go through and resolve all these one by one, so Fedora has a clean build/fulltest.

Please let me know how to run each test individually and I'll try to figure out the problem with each.

Thanks.

 
2880 code fixed 2008 Jan anonymous   2008 Jan   1 1 problem with reals bigger then 1e18 edit
We've been reported (and confirmed) that there might be an isue using the Windows sqlite3 dll on large numbers. When defining fields as real then inserting data like 1e18 goes well, but 1e19 generates an error about being not a valid float. If you enter the data using sqlite3.exe there's no problem, retrieving the data is also working well. The problem arises when calling sqlite3step, the prepare does not give any error.

We've checked and the same error appears using sqlite administrator (which is a Delphi development too, but using ZEOS). Since it goes wrong within sqlite3step this might be a sqlite isue?

albert drent aducom software

2008-Jan-09 13:56:01 by drh:
You write: "If you enter the data using sqlite3.exe there's no problem." But sqlite3.exe uses the sqlite3_prepare() and sqlite3_step() interface the same as anything else. So if sqlite3.exe is working, that suggests the problem is in your program, and not in the SQLite core.

If you can provide a specific example of what is not working we can investigate this problem further. But based on the information provide, we are unable to reproduce the problem and suspect the problem is in the application, not in SQLite.


2008-Jan-09 15:48:49 by anonymous:
Delphi has been known to futz with the FPU registers. It's possible that Delphi is changing the FPU accuracy and causing this problem.


2008-Jan-09 19:49:57 by anonymous:
I don't know where the problem lies and will investigate further. It's not a fpu isue since it's a ansistring containing the string, being prepared and executed. There hardly any Delphi involved here. I'll create a small app in Delphi showing the problem and report this issue on Delphi forums too.

The string is somewhat like 'insert into demo (myval) value (1e19)'

Will come back here. Albert


2008-Jan-10 13:47:28 by anonymous:
Thanks to sasa: he found out the following

It appears to be a problem in used C compiler. Following script is tested in sqlite 3-5-4 shell:

create table t1 (ID integer primary key, v1 double); insert into t1(v1) values(1e18); insert into t1(v1) values(1e19);

This works correctly on linux compiled with GNU compiler 4.1.2. On windows with MinGW 3.2.0 works correctly too, however it fails when shell is compiled with BCC 5.5.1. Without detail check I can only assume there is a problem in representing and using 8 and 6 bytes floats together in SQLite without explicit casting BCC may require. Or this may be a limitation of BCC 5.5.1 or a bug. Without detail check those are most probably causes.


2008-Jan-10 14:01:30 by drh:
SQLite does not use any "6 byte floats". I'm not sure what a 6 byte float is. The code in SQLite is suppose to be ANSI C and casts that are not required by ANSI C are generally omitted. If you find an exception to this please let us know. If BCC requires casts above and beyond what is required by ANSI C, then that is a bug in BCC.

I assume then, that this is a bug in BCC. I do not have access to BCC so there is nothing I can do to fix it. If you can suggest patches that will work around the BCC bug (for example, some unnecessary explicit casts) we will consider adding them to the core. But I am afraid it will be up to you to supply those patches, since we do not have BCC available to experiment with and SQLite works perfectly for every compiler we have access to.


2008-Jan-10 18:34:56 by anonymous:
Funny thing is that if the dll is created with a gnu compiler it seems to work. Can you tell me with what compiler the dll (w.o. tcl binding) has been build?


2008-Jan-10 20:16:12 by anonymous:
This is the reply of sasa, it's way ahead of me...

Unfortunately, I have no more time to test any DLL since I do not using any I did not compiled myself. As I understand vac, that one from official SQLite site have this bug. In the signature of DLL can be seen which compiler was used for compilation.

BCC 5.5.1 is free and can freely be downloaded from official Borland site by anyone. This is I prepared to add to the ticket, but you can add yourself. I have spent all my spare time for now. Since it do not break anything it will be probably included in official version:

/*
** The MEM structure is already a MEM_Real. Try to also make it a
** MEM_Int if we can.
*/
SQLITE_PRIVATE void sqlite3VdbeIntegerAffinity(Mem *pMem){
  assert( pMem->flags & MEM_Real );
  assert( pMem->db==0 || sqlite3_mutex_held(pMem->db->mutex) );
  pMem->u.i = pMem->r; // <- This causing termination for values >= 1e19
  if( ((double)pMem->u.i)==pMem->r ){
    pMem->flags |= MEM_Int;
  }
}
This conversion line causing BCC 5.5.1 run-time to raise a conversion error if values are >= 1e19. It is necessary a clean way to convert double to integer/int64. If nothing else, this will solve the problem (code is not essential for correct functionality):

/*
** The MEM structure is already a MEM_Real. Try to also make it a
** MEM_Int if we can.
*/
SQLITE_PRIVATE void sqlite3VdbeIntegerAffinity(Mem *pMem){
  assert( pMem->flags & MEM_Real );
  assert( pMem->db==0 || sqlite3_mutex_held(pMem->db->mutex) );

#ifndef __BORLANDC__
  pMem->u.i = pMem->r;
  if( ((double)pMem->u.i)==pMem->r ){
    pMem->flags |= MEM_Int;
  }
#endif
}
[ Bijgewerkt do jan 10 2008, 08:49 ] SZUTILS - link


2008-Jan-11 10:23:37 by anonymous:
In "sqlite - Check-in [4705] " you have tryed to "attempt to work around this bug in the Borland compiler + ** by moving the value into a temporary variable first so that if + ** the assignment into the integer really does corrupt the right-hand + ** side value, it will corrupt a temporary variable that we do not + ** care about."

First of all, thank you for attempt to solve the issue. I will first write several facts:

1. Statically linked SQLite 3.5.4 commandline shell (aka sqlite3.exe) is compiled and SQL script worked fine on Windows with MinGW 3.2.3, however not with BCC 5.5.1 (without any special compiler options).

2. Library (aka sqlite3.dll) compiled with MinGW 2.95 (from official SQLite site), MinGW 3.2.3, BCC 5.5.1 and probably other compilers for Windows does FAIL executing provided SQL script only with large floats >=1e19. With MinGW 3.2.3, library was created from official SQLite 3.5.4 distribution with following script:

sh configure

make dll

3. Both (integer and double) have 8 byte size in BCC. BCC is ANSI C complient compiler.

4. It is not the fact that BCC corrupt right-hand side value. It fail trying to convert values >=1e19 and assign it to int64 variable. This will raise invalid floating-point operation exception for most C compilers.

5. Since front-end application use only DLL (compiled by several different C compilers, including provided from offical SQLite site) to execute an SQL command by sqlite_exec(), it is nothing which connect the bug and the application itself (made in Delphi or C).

Current attempt in 4705 will not work from two reasons:

1. The same conversion logic is used.

2. Variable definition in the middle of the function in not allowed by ANSI C. This will fail not only on BCC, but on any ANSI C compiler.

According to upper, conclusion is clear that this is not a bug in BCC, but in all C compiles for Windows - or in conversion double to integer currently used.

Thank you for your time.

Sasa Zeman


2008-Jan-11 13:22:27 by drh:
The purpose of the sqlite3VdbeIntegerAffinity() function is to convert a floating point value into a 64-bit integer value if and only if that conversion can be done without loss of precision. Perhaps Sasa Zeman or somebody else can suggest a way to do that successfully on BCC 5.5.1. (The current implementation appears to work on every other compiler.)

Or perhaps this is not a compiler issue at all, but an instance of Delphi messing with FPU registers, as was suggested by the anonymous comment on 2008-Jan-09.

Please note that I do not own a machine that runs windows, and I am unwilling to purchase one for the purpose of fixing this issue. So I cannot download and install BCC 5.5.1 or Delphi. And thus, I cannot reproduce the problem here. So if you want this problem resolved, you are going to need to suggest code for resolving it. Completely disabling the sqlite3VdbeIntegerAffinity() function is not an acceptable resolution because that breaks quite a few other things.


2008-Jan-11 14:00:05 by anonymous:
AFAIK it is a general compiler isue on Windows. Reading Sasa's comment this is NOT a Delphi isue since the problem lies in SQLite3.dll. In fact, the tests who are shown here are all done in C++. What I would like to know is how the Windows dll on this website and which DOES have this problem, is created and with what compiler. Appearantly you haven't created this one since you do not own a Windows compiler (?). Then we are in pursuit of an answer and solution. Ignoring the problem is not an option IMHO and would be bad news to all on the Windows community. The Borland C++ compiler with the patch here (SASA does this work?) is completely FREE and doesn't need to be purchased. But if this isue cannot be resolved, and I'm not a C++ guru, then there should be a note here that on Windows the precision is limited to 10e18??? There are now three people busy with this, sure there must be a solution in conjunction with all of us? It's in everyones benefit if this isue can be resolved in stead of ignored.


2008-Jan-11 14:36:11 by drh:
The windows binaries on the SQLite website are cross-compiled on a Linux box using Mingw-GCC.


2008-Jan-11 17:49:26 by anonymous:
Which steps do I need to take to reproduce the problem? It seems to work just fine with sqlite3.exe 3.5.0 on Windows XP:

  sqlite> create table t1 (ID integer primary key, v1 double);
  sqlite> insert into t1(v1) values(1e18);
  sqlite> insert into t1(v1) values(1e19);
  sqlite> select * from t1;
  1|1.0e+18
  2|1.0e+19

Could someone please post a little C-code which exhibits the problem? Thanks!


2008-Jan-11 18:10:29 by anonymous:
Thank you for your respond. In order to reproduce the problem, you can create very simple C program using the DLL, which execute upper script by each line, using sqlite3_exec() command.

Thank you for your time.

Sasa


2008-Jan-11 19:05:51 by anonymous:
The following:

  +  static const i64 maxInt = (((i64)0x7fffffff)<<32)|0xffffffff;
  +  static const i64 minInt = ((i64)0x80000000)<<32;
  +
  +  if( r<(double)minInt ){
  +    return minInt;
  +  }else if( r>(double)maxInt ){

runs faster on my compiler if you do this:

  +  static const double maxInt = (((i64)0x7fffffff)<<32)|0xffffffff;
  +  static const double minInt = ((i64)0x80000000)<<32;
  +
  +  if( r<minInt ){
  +    return minInt;
  +  }else if( r>maxInt ){


2008-Jan-12 11:26:41 by anonymous:
I asked for code to reproduce the problem and would like to report back that I was able to do so on WinXP using Delphi. Here are my findings:

  • The sqlite3.exe console provided on the SQLite web-site is not affected by the problem, it executes the SQL above with no errors.
  • The precompiled sqlite3.dll from the SQLite web-site does show the problem (version 3.5.4 tested) when executing the SQL through sqlite3_exec().
  • The DISQLite3 Delphi library does not exhibit the problem (same tests as with sqlite3.dll). DISQLite3 is compiled with Borland's C Compiler, so I conclude that it is not a BCC problem.

Cause of the problem with sqlite3.dll is a FPU exception, probably generated by converting 1e19 to a 64-bit integer in the line identified above. AFAIK, Windows -- contrary to Linux -- has FPU exceptions turned on by default, which would explain why Linux users do not see it.

A solution is to disable FPU exceptions for Win applications which use sqlite3.dll. I'd generally recommended to do so, since there are other parts in the SQLite code which depend on the FPU not raising exceptions. SQLite is, by the way, not the odd one out with this recommendation: The Delphi help recommends to do so when using OpenGL as well.

In Delphi, this call disables all FPU exceptions. Run it before calling any SQLite functions:

  Set8087CW($133F); { Disable all FPU exceptions. }

BCC has these available:

  unsigned int _control87(unsigned int newcw, unsigned int mask);
  unsigned int _controlfp(unsigned int newcw, unsigned int mask); /* For MS compatability. */

With FPU exceptions disabled, the SQL executes correctly using sqlite3.dll 3.5.4.

Since I can not recompile sqlite3.dll exactly as the downloaded version, I can not test if changes in [4706] still require to disable FPU exceptions.


2008-Jan-12 19:34:01 by anonymous:
Thank you for your respond. I suppose it is all clear now.

That explains why GNU compiled SQLite shell works correctly on Windows too, but not its DLL - GNU force disabling FPU exceptions at program beginning to preserve multi-platfom compatibility. Changes in [4706] solve the issue in this case. Compiled DLL from latest CVS works correctly. If exist similar coversion in the SQLite code, that should be solved as well.

Current souluton in [4706] is suggested to be permanent since SQLite is DB engine, not a math computation library. Thus enabling/disabling FPU exceptions ($133F/$1332) can be a bit of problem if front-end application is math oriented and take explicit care of FPU exceptions. And second reason is connected - not to pressing user of SQLite to set unnecesary FPU settings which is highly depent on his compiler/RAD. And last and most important - preserve multi-platform compatibility.

Thank you once again. As well, thanks to author(s) of SQLite for solving the issue.

Sasa


2008-Jan-13 12:56:31 by anonymous:
I'm glad that the isue has resolved finally. Thanks to everybody working on this. Albert Drent aducom software


2008-Jan-13 17:37:46 by anonymous:
The patch is incorrect:

  +  if( r<(double)minInt ){
  +    return minInt;
  +  }else if( r>(double)maxInt ){
  +    return minInt;  <============= should return maxInt here

also consider making minInt and maxInt const doubles instead of const ints.


2008-Jan-13 18:20:13 by drh:
Actually, the patch is correct as written. Attempting to convert a out-of-range double into an integer yields 0x8000000000000000 regardless of whether or not the double is positive or negative. At least this is what intel hardware does.

I did consider making the minInt and maxInt constants doubles, but that would mean that the conversion from integer to double would occur on the build machine, which might do the conversion differently than the target machine. It seems safer to do the conversion at runtime. We might reconsider this in the future.


2008-Jan-13 19:38:50 by anonymous:
In C, casting an out of range double to int is undefined, as was SQLite's previous behavior by extension. GCC on x86 happens to yield 0x8000000000000000 for positive overflow, a negative value. This is not guaranteed for other compilers even on x86. I can understand why you'd want to be compatible with x86 GCC, but you have a chance for this database operation to at least preserve the sign of the number even though you lose the magnitude. This has value.
 
2879 code active 2008 Jan anonymous   2008 Jan anonymous 4 3 VACUUM enters temporary sequence numbers in sqlite_sequence edit
I have two TEMPORARY tables added to a database, that both contain an INTEGER PRIMARY KEY AUTOINCREMENT field. After issuing a VACUUM command, the maximum value of these fields is added to the sqlite_sequence table. And stay there after the connection closes.
 
2878 code active 2008 Jan anonymous   2008 Jan   1 1 Memory leaks with latest CVS [4693] edit
This SQL leaks memory with CVS [4693] :

  CREATE TABLE x(id integer primary key, a TEXT NULL);
  INSERT INTO x (a) VALUES ('first');
  CREATE TABLE tempx(id integer primary key, a TEXT NULL);
  INSERT INTO tempx (a) VALUES ('t-first');
  CREATE VIEW tv1 AS SELECT x.id, tx.id FROM x JOIN tempx tx ON tx.id=x.id;

One leak is caused by "CREATE TABLE tempx", a second one by "CREATE VIEW tv1".

The above SQL is a digest of select7.test, select7-2.1.

2008-Jan-14 17:51:11 by anonymous:
I have tested again with CVS [4711] and it does no longer show the original leaks.

I therefore consider this issue fixed and I will now close this ticket.


2008-Jan-14 23:56:34 by anonymous:
Doing a fulltest with -MSQLITE_MEMDEUG I see reports of memory leaks. I assume these are of little or minimal interest at present because of the amount of code flux.

If you do want details, please let me know (I'll recheck this ticket tomorrow I guess).


2008-Jan-15 08:23:56 by anonymous:
Thanks for the follow-up. I am not running the original test-suite but have have ported a great number of them to Delphi.

If you could just let me know which tests caused the leaks you fixed in [4712] I'd be more than glad the port these test as well and let you know my findings.


2008-Jan-19 00:56:17 by anonymous:
A test from 20 minutes ago passes cleanly. This could be closed.
 
2877 code fixed 2008 Jan anonymous   2008 Jan   1 1 Unescaped quotation marks in mkopcodeh.awk (CVS [4693]) edit
mkopcodeh.awk line 146 is missing escapes in front of the "case" quotation marks:

  print "** comments following the "case" for each opcode in the vdbe.c"

Works for me with:

  print "** comments following the \"case\" for each opcode in the vdbe.c"
 
2876 build closed 2008 Jan anonymous   2008 Jan   1 1 3.5.4 fails to build on OpenBSD... edit
$ ./configure --prefix=/home/achowe/Projects/org/sqlite --disable-tcl --enable-threadsafe $ make ... rm tsrc/sqlite.h.in tsrc/parse.y cp parse.c opcodes.c keywordhash.h tsrc tclsh ./tool/mksqlite3c.tcl tclsh: not found *** Error code 1

There appears to be a TCL dependency even though I've specified --disable-tcl. Stock OpenBSD does not supply TCL and I have no desire to install it.

Duplicate of ticket #2845.
 
2875 code active 2008 Jan anonymous   2008 Jan   3 2 LIKE does not work with lowercase swedish characters edit
Swedish letters &aring;,&auml;,&ouml; is not supported by the LIKE statement. When trying to perform a query like

SELECT * FROM table WHERE name LIKE "&aring;%"

we will not get a match for names starting on &Aring; (which is uppercase for &aring;).

To recreate:


SQLite version 3.5.4 Enter ".help" for instructions sqlite> CREATE TABLE TestingTable(Name varchar(20)); sqlite> INSERT INTO TestingTable values ('Sweden'); sqlite> INSERT INTO TestingTable values ('sweden'); sqlite> INSERT INTO TestingTable values ('&Aring;land'); sqlite> INSERT INTO TestingTable values ('&aring;land'); sqlite> SELECT * FROM TestingTable; Sweden sweden &Aring;land &aring;land sqlite> SELECT * FROM TestingTable WHERE Name LIKE "swe%"; Sweden sweden sqlite> SELECT * FROM TestingTable WHERE Name LIKE &aring;la%"; &aring;land
 
2874 code active 2008 Jan anonymous   2008 Jan   1 1 THREADSAFE #define HAVE_LOCALTIME_R, HAVE_GMTIME_R in os_unix.c edit
The precompiled shared sqlite3 library for Linux on sqlite.org which appears to be built with pthread support is using localtime and gmtime which are not threadsafe.

For THREADSAFE builds could either configure be changed to detect the functions gmtime_r and localtime_r or change os_unix.c to explicitly #define HAVE_LOCALTIME_R and HAVE_GMTIME_R?

 
2873 code active 2008 Jan anonymous   2008 Jan   1 1 HAVE_USLEEP, HAVE_FDATASYNC=1 detected but not used by configure; make edit
I noticed that a couple of open source projects were not picking up usleep() for recent sqlite builds and used the coarser grained sleep() instead. Around 11 months ago something changed in sqlite's build process.

It seems that both -DHAVE_USLEEP=1, -DHAVE_FDATASYNC=1 and -DOS_UNIX=1 are detected correctly by configure but not used by the generated Makefile. As result, UNIX builds of sqlite3 via ./configure do not use usleep() and fdatasync() and do not define OS_UNIX.

I don't know whether the lack of fdatasync() versus the default fsync() affects anyone.

Please apply the following patch which corrects the problem with "./configure && make". Thank you.

Index: configure
===================================================================
RCS file: /sqlite/sqlite/configure,v
retrieving revision 1.45
diff -u -3 -p -r1.45 configure
--- configure	27 Nov 2007 14:50:07 -0000	1.45
+++ configure	5 Jan 2008 07:41:00 -0000
@@ -18520,9 +18520,9 @@ if test "$TARGET_EXEEXT" = ".exe"; then
     OS_UNIX=0
     OS_WIN=0
     OS_OS2=1
-    TARGET_CFLAGS="$TARGET_CFLAGS -DOS_OS2=1"
+    CFLAGS="$CFLAGS -DOS_OS2=1"
     if test "$ac_compiler_gnu" == "yes" ; then
-      TARGET_CFLAGS="$TARGET_CFLAGS -Zomf -Zexe -Zmap"
+      CFLAGS="$CFLAGS -Zomf -Zexe -Zmap"
       BUILD_CFLAGS="$BUILD_CFLAGS -Zomf -Zexe"
     fi
   else
@@ -18530,14 +18530,14 @@ if test "$TARGET_EXEEXT" = ".exe"; then
     OS_WIN=1
     OS_OS2=0
     tclsubdir=win
-    TARGET_CFLAGS="$TARGET_CFLAGS -DOS_WIN=1"
+    CFLAGS="$CFLAGS -DOS_WIN=1"
   fi
 else
   OS_UNIX=1
   OS_WIN=0
   OS_OS2=0
   tclsubdir=unix
-  TARGET_CFLAGS="$TARGET_CFLAGS -DOS_UNIX=1"
+  CFLAGS="$CFLAGS -DOS_UNIX=1"
 fi


@@ -19392,7 +19392,7 @@ fi
 echo "$as_me:$LINENO: result: $ac_cv_func_usleep" >&5
 echo "${ECHO_T}$ac_cv_func_usleep" >&6
 if test $ac_cv_func_usleep = yes; then
-  TARGET_CFLAGS="$TARGET_CFLAGS -DHAVE_USLEEP=1"
+  CFLAGS="$CFLAGS -DHAVE_USLEEP=1"
 fi


@@ -19491,7 +19491,7 @@ fi
 echo "$as_me:$LINENO: result: $ac_cv_func_fdatasync" >&5
 echo "${ECHO_T}$ac_cv_func_fdatasync" >&6
 if test $ac_cv_func_fdatasync = yes; then
-  TARGET_CFLAGS="$TARGET_CFLAGS -DHAVE_FDATASYNC=1"
+  CFLAGS="$CFLAGS -DHAVE_FDATASYNC=1"
 fi


Index: configure.ac
===================================================================
RCS file: /sqlite/sqlite/configure.ac,v
retrieving revision 1.31
diff -u -3 -p -r1.31 configure.ac
--- configure.ac	27 Nov 2007 14:50:07 -0000	1.31
+++ configure.ac	5 Jan 2008 07:41:00 -0000
@@ -310,9 +310,9 @@ if test "$TARGET_EXEEXT" = ".exe"; then
     OS_UNIX=0
     OS_WIN=0
     OS_OS2=1
-    TARGET_CFLAGS="$TARGET_CFLAGS -DOS_OS2=1"
+    CFLAGS="$CFLAGS -DOS_OS2=1"
     if test "$ac_compiler_gnu" == "yes" ; then
-      TARGET_CFLAGS="$TARGET_CFLAGS -Zomf -Zexe -Zmap"
+      CFLAGS="$CFLAGS -Zomf -Zexe -Zmap"
       BUILD_CFLAGS="$BUILD_CFLAGS -Zomf -Zexe"
     fi
   else
@@ -320,14 +320,14 @@ if test "$TARGET_EXEEXT" = ".exe"; then
     OS_WIN=1
     OS_OS2=0
     tclsubdir=win
-    TARGET_CFLAGS="$TARGET_CFLAGS -DOS_WIN=1"
+    CFLAGS="$CFLAGS -DOS_WIN=1"
   fi
 else
   OS_UNIX=1
   OS_WIN=0
   OS_OS2=0
   tclsubdir=unix
-  TARGET_CFLAGS="$TARGET_CFLAGS -DOS_UNIX=1"
+  CFLAGS="$CFLAGS -DOS_UNIX=1"
 fi

 AC_SUBST(BUILD_EXEEXT)
@@ -565,13 +565,13 @@ AC_SUBST(TARGET_DEBUG)
 #########
 # Figure out whether or not we have a "usleep()" function.
 #
-AC_CHECK_FUNC(usleep, [TARGET_CFLAGS="$TARGET_CFLAGS -DHAVE_USLEEP=1"])
+AC_CHECK_FUNC(usleep, [CFLAGS="$CFLAGS -DHAVE_USLEEP=1"])

 #--------------------------------------------------------------------
 # Redefine fdatasync as fsync on systems that lack fdatasync
 #--------------------------------------------------------------------

-AC_CHECK_FUNC(fdatasync, [TARGET_CFLAGS="$TARGET_CFLAGS -DHAVE_FDATASYNC=1"])
+AC_CHECK_FUNC(fdatasync, [CFLAGS="$CFLAGS -DHAVE_FDATASYNC=1"])

 #########
 # Generate the output files.
2008-Jan-05 08:24:29 by anonymous:
It appears that http://www.sqlite.org/sqlite3-3.5.4.bin.gz and http://www.sqlite.org/sqlite-3.5.4.so.gz use sleep and fsync even though usleep and fdatasync are available on Linux.

On the Linux man page, it claims that fdatasync is more efficient than fsync:

"Unfortunately, fsync() will always initiate two write operations: one for the newly written data and another one in order to update the modification time stored in the inode. If the modification time is not a part of the transaction concept fdatasync() can be used to avoid unnecessary inode disk write operations."

 
2872 code active 2008 Jan anonymous   2008 Jan   4 4 Some table scan operations could use an index for better data density edit
If a suitable index covers a column being during a table scan, it makes sense to use the index for IO not the table pages themselves for speed.

As (contrived) example:

  CREATE TABLE t1 ( c1 blob, c2 integer, c3 integer );
  CREATE INDEX i1 on t1(c2,c3);
  CREATE INDEX i2 on t1(c3);

T1 populated with 4096 rows, c1 being 64K random blobs (to make c2, c3 access slower in this case), c2 and c3 being small random integers.

Now:

  sqlite> select sum(c3) from t1;
  sum(c3)
  ----------
  519895

is very slow (several seconds). A table scan is done.

Forcing use of an index, in a way that I know all rows will be included:

  sqlite> select sum(c3) from t1 where c2<1000;
  sum(c3)
  ----------
  519895

This is instantaneous.

It seems to me that if a table scan has to be performance, it makes sense to grab the data from an index whenever possible. ideally the most densely packed index.

(BTW; this is contrived, I know putting the blob as the last column will greatly speed things up).

 
2871 build closed 2008 Jan anonymous   2008 Jan   3 3 SQLite 3.5.4 build process does not listen to --disable-tcl edit
I build sqlite without TCL support, with an incantation along the lines of: ../sqlite-3.5.4/configure --disable-tcl; make. This works fine in 3.5.3 but 3.5.4 fails late in the compilation, looking for tclsh: tclsh ../sqlite-3.5.4/tool/mksqlite3c.tcl is the command it's trying to run.

I'm building on Solaris 10u4 and the full incantation for configure is as follows (most of this is for readline): ../sqlite-3.5.4/configure --prefix=/home/tfb/packages/sqlite-3.5.4 --enable-static=no --with-readline-lib="-L/opt/sfw/lib -R/opt/sfw/lib -lreadline -lcurses" --with-readline-inc=-I/opt/sfw/include --disable-tcl. I use gmake (which is GNU make, in /usr/sfw/bin= on Solaris 10) to compile.

This is not urgent for me - I will just stick with 3.5.3 for now - but it would be nice to avoid mandatory dependence on TCL I think.

Thanks

--tim

Duplicate of ticket #2845
 
2870 code closed 2008 Jan anonymous   2008 Jan danielk1977 5 5 Comment error edit
  In the #4666 checkin comment is:

  +** Change the value of the P3 operand for a specific instruction.
  +*/
  +void sqlite3VdbeChangeP5(Vdbe *p, int addr, u8 val){

  but should be: .... P5 .... operand
Thanks.
 
2869 new active 2008 Jan anonymous   2008 Jan   5 5 add "sqlite3_open16_v2" to the C API edit
I'm using UTF-16, and if the database file does not exist, "sqlite3_open16" will create a new one, but i wish it fails in such conditions. I notice that there's a "sqlite3_open_v2", but it doesn't support UTF-16, although i can implement a "sqlite3_open16_v2" myself, I think it should exists in the offical releases.
 
2868 code closed 2008 Jan anonymous   2008 Jan   1 1 Encryption user can't link latest SQLite edit
I have licensed the encryption package and have it working with an earlier version of SQLite. I downloaded the current stable release, copied it over the previous version and compiled it. It compiled fine, but now the pager has unresolved externals, it seems many of the function names have changed.

Please advise.

Thanks Tim

2008-Jan-03 14:00:20 by drh:
The license is perpetual with unlimited free updates. You just have to send us private email and we will send you back the latest code.
 
2867 code active 2008 Jan anonymous   2008 Jan   2 2 doesn't build on Cygwin - wrong sqlite3 exe suffix edit
The new Makefile used $(EXE), which doesn't seem to be defined (typo?)
2008-Jan-02 11:12:39 by anonymous:
Same on mingw:

Following patch fixes things:

--- sqlite-3.5.4/Makefile.in    Thu Dec 13 19:17:42 2007
+++ sqlite-3.5.4-mingw-fix/Makefile.in  Wed Jan  2 11:37:50 2008
@@ -322,7 +322,7 @@
                -o $@ $(TOP)/src/shell.c libsqlite3.la \
                $(LIBREADLINE) $(TLIBS)

-sqlite3$(EXE): $(TOP)/src/shell.c sqlite3.c sqlite3.h
+sqlite3$(TEXE):        $(TOP)/src/shell.c sqlite3.c sqlite3.h
        $(LTLINK) $(READLINE_FLAGS) -o $@                          \
                -DSQLITE_MAX_SQL_LENGTH=1000000000                  \
                -USQLITE_THREADSAFE -DSQLITE_THREADSAFE=0           \
@@ -577,7 +577,7 @@
          -e 's,$$,\\n",' \
          $(TOP)/tool/spaceanal.tcl >spaceanal_tcl.h
        $(LTLINK) -DTCLSH=2 -DSQLITE_TEST=1 $(TEMP_STORE)\
-                -o sqlite3_analyzer$(EXE) $(TESTSRC) $(TOP)/src/tclsqlite.c \
+                -o sqlite3_analyzer$(TEXE) $(TESTSRC) $(TOP)/src/tclsqlite.c \
                libtclsqlite3.la $(LIBTCL)
 
2866 build active 2008 Jan anonymous   2008 Jan   1 3 Problems building Windows native in cygwin/mingw environment edit
Trying to build Windows native version using the Cygwin build environment.

$ gcc -v Reading specs from /usr/lib/gcc/i686-pc-cygwin/3.4.4/specs Configured with: /usr/build/package/orig/test.respin/gcc-3.4.4-3/configure --verbose --prefix=/usr --exec-prefix=/usr --sysconfdir=/etc --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --enable-languages=c,ada,c++,d,f77,pascal,java,objc --enable-nls --without-included-gettext --enable-version-specific-runtime-libs --without-x --enable-libgcj --disable-java-awt --with-system-zlib --enable-interpreter --disable-libgcj-debug --enable-threads=posix --enable-java-gc=boehm --disable-win32-registry --enable-sjlj-exceptions --enable-hash-synchronization --enable-libstdcxx-debug Thread model: posix gcc version 3.4.4 (cygming special, gdc 0.12, using dmd 0.125) $

$ CFLAGS=-mno-cygwin ./configure --disable-tcl --enable-threadsafe

$ make

A) The make appears to build sqlite3.exe just fine, without errors or warnings. This binary does work from cmd.exe, BUT not from within the bash cygwin shell for some reason, unlike other Windows native binaries I've built.

Next...

$ make install

B) The

cc sqlite3.c -o sqlite3

fails to rebuild sqlite3.exe correctly with the -mno-cygwin option. The output follows:

rm -rf tsrc mkdir -p tsrc cp ./src/alter.c ./src/analyze.c ./src/attach.c ./src/auth.c ./src/btmutex.c ./src/btree.c ./src/btree.h ./src/build.c ./src/callback.c ./src/complete.c ./src/date.c ./src/delete.c ./src/expr.c ./src/func.c ./src/hash.c ./src/hash.h ./src/insert.c ./src/journal.c ./src/legacy.c ./src/loadext.c ./src/main.c ./src/malloc.c ./src/mem1.c ./src/mem2.c ./src/mem3.c ./src/mutex.c ./src/mutex_os2.c ./src/mutex_unix.c ./src/mutex_w32.c ./src/os.c ./src/os_unix.c ./src/os_win.c ./src/os_os2.c ./src/pager.c ./src/pager.h ./src/parse.y ./src/pragma.c ./src/prepare.c ./src/printf.c ./src/random.c ./src/select.c ./src/shell.c ./src/sqlite.h.in ./src/sqliteInt.h ./src/table.c ./src/tclsqlite.c ./src/tokenize.c ./src/trigger.c ./src/utf.c ./src/update.c ./src/util.c ./src/vacuum.c ./src/vdbe.c ./src/vdbe.h ./src/vdbeapi.c ./src/vdbeaux.c ./src/vdbeblob.c ./src/vdbefifo.c ./src/vdbemem.c ./src/vdbeInt.h ./src/vtab.c ./src/where.c ./ext/fts1/fts1.c ./ext/fts1/fts1.h ./ext/fts1/fts1_hash.c ./ext/fts1/fts1_hash.h ./ext/fts1/fts1_porter.c ./ext/fts1/fts1_tokenizer.h ./ext/fts1/fts1_tokenizer1.c sqlite3.h ./src/btree.h ./src/btreeInt.h ./src/hash.h ./src/sqliteLimit.h ./src/mutex.h opcodes.h ./src/os.h ./src/os_common.h ./src/sqlite3ext.h ./src/sqliteInt.h ./src/vdbe.h parse.h ./ext/fts1/fts1.h ./ext/fts1/fts1_hash.h ./ext/fts1/fts1_tokenizer.h ./src/vdbeInt.h tsrc cp: warning: source file `./src/btree.h' specified more than once cp: warning: source file `./src/hash.h' specified more than once cp: warning: source file `./src/sqliteInt.h' specified more than once cp: warning: source file `./src/vdbe.h' specified more than once cp: warning: source file `./ext/fts1/fts1.h' specified more than once cp: warning: source file `./ext/fts1/fts1_hash.h' specified more than once cp: warning: source file `./ext/fts1/fts1_tokenizer.h' specified more than once cp: warning: source file `./src/vdbeInt.h' specified more than once rm tsrc/sqlite.h.in tsrc/parse.y cp parse.c opcodes.c keywordhash.h tsrc tclsh ./tool/mksqlite3c.tcl cc sqlite3.c -o sqlite3 /usr/lib/gcc/i686-pc-cygwin/3.4.4/../../../libcygwin.a(libcmain.o):(.text+0xab): undefined reference to `_WinMain@16' collect2: ld returned 1 exit status make: *** [sqlite3] Error 1 $

 
2865 code active 2007 Dec anonymous   2007 Dec   1 2 FTS3 does not build with amalgamation in CVS edit
Grab the latest CVS sources, then run:

  ./configure
  make sqlite3.c

  grep sqlite3Fts3Init sqlite3.c

    extern int sqlite3Fts3Init(sqlite3*);
    rc = sqlite3Fts3Init(db);

If you compile sqlite3.c with -DSQLITE_ENABLE_FTS3, then sqlite3Fts3Init is unresolved.

For some reason, sqlite3Fts3Init and fts3.c was not included in the sqlite3.c amalg.

It used to work correctly in 3.5.4.

2007-Dec-30 18:17:57 by anonymous:
Nevermind, "make sqlite3.c" has never built with the fts3 sources in 3.5.4 or before.

You have to run ext/fts3/mkfts3amal.tcl


2007-Dec-30 18:20:56 by anonymous:
It seems that the sqlite3+fts3 amalg can only be built from main.mk, not Makefile.
 
2864 doc active 2007 Dec anonymous   2007 Dec   5 3 ext/fts3/README.txt edit
File ext/fts3/README.txt reads: This folder contains source code to the second full-text search [...]

Shouldn't that be: This folder contains source code to the third full-text search [...]

2007-Dec-30 18:27:08 by anonymous:
Oh, after Googl'ing a little bit, I found that fts3 really is fts2-with-rowid-fixed.

If both fts2 and fts3 are considered to be the "second full-text search extension for SQLite", the README files could maybe explain the situation.

 
2863 code active 2007 Dec anonymous   2007 Dec   2 3 test cast-3.14, cast-3.18 and cast-3.24 fail edit
  test cast-3.{14,18,24} fail on freebsd-6.3-PRERELEASE2:
  cast-3.14...^M
  Expected: [9223372036854774784]^M
       Got: [9223372036854773760]^M
  cast-3.18...^M
  Expected: [-9223372036854774784]^M
       Got: [-9223372036854773760]^M
  cast-3.24...^M
  Expected: [9223372036854774784]^M
       Got: [9223372036854773760]^M
  I used tcl8.4 from ports with no threads and here was the config line:
  ../sqlite-3.5.4/configure --prefix=/home/marc/local --with-tcl=/usr/local/lib/tcl8.4/

This was built on an ibm t30 laptop

 
2862 warn closed 2007 Dec anonymous   2008 Jan   4 3 Small warning fix edit
Hi, There is a harmless warning in btree.c about szNext being used uninitialized, with a comment in the code saying that adding an extra initialization to silence it would slow it down. But I've done it another way which will probably make a tiny speed-up instead (only when failing some step though, I think)

Diff:
http://megasource.no-ip.org/PTstuff/btree.c.morefix
http://megasource.no-ip.org/PTstuff/btree.c.fix (the original fix, just to silence the warning)

2008-Jan-01 06:19:11 by danielk1977:
Thanks.
 
2861 doc closed 2007 Dec anonymous   2008 Jan   1 1 How do I connect to Crystal Reports - I am using vs2003 & the Finisar edit
Not sure of the right venue for this question. My apologies if this is not the right place. I am developing a simple accounting application for a non-profit organisation. I am using vs2003 and programming in c#. To date I have used Access (Jet) database files and the Crystal Report Generator built in to VS2003. I thought SQLite would be a better choice than the Access database - simple to change my c# code using the Finisar data provider, but I am at a loss to see how I can use this provider in Crystal. Is there any way to get this provider onto the list of ADO DB providers in Crystal - or any other approach I could use. If I cannot use Crystal is there an alternative for producing reports which can link to SQLite. Any help much appreciated. Roger Lovelock
2007-Dec-29 13:58:55 by anonymous:
Not a bug. Better ask this on the mailing list.

To subscribe to the list, send a message to: <sqlite-users-subscribe@sqlite.org>

For help and a description of available commands, send a message to: <sqlite-users-help@sqlite.org>


2008-Jan-01 06:07:34 by danielk1977:
This question would have a good chance of success if put to the mailing list. Suggest trying it there.
 
2860 todo active 2007 Dec anonymous   2007 Dec   3 1 Database file fragmentation edit
Adding data in database file increases file fragmentation. for example my file which size is 1G, consists of 20000 pieces. (NTFS)

This happens because truncation of '-journal' file. I see some ways to reduce fragmentaion: 1. Increase database file size by greater pieces (not by PAGESIZE). 2. SQLite can save '-journal' file in another folder(logical disc). 3. Preallocation of database file(must increase INSERT speed).

 
2859 code active 2007 Dec anonymous   2007 Dec drh 3 2 Inconsistent column names with DISTINCT edit
Given the following SQL:

  CREATE TABLE foo(a,b);
  INSERT INTO foo (a, b) VALUES (1,2);

SQLite returns inconsistent column names when using the DISTINCT clause:

  SELECT DISTINCT foo.A, foo.B FROM foo;
  foo.A|foo.B
  1|2

  SELECT DISTINCT a, b FROM foo;
  a|b
  1|2

  SELECT DISTINCT * FROM foo;
  a|b
  1|2

  SELECT DISTINCT foo.* FROM foo;
  a|b
  1|2

Compared with SELECT without DISTINCT:

  SELECT foo.A, foo.B FROM foo;
  a|b
  1|2

  SELECT a, b FROM foo;
  a|b
  1|2

  SELECT * FROM foo;
  a|b
  1|2

  SELECT foo.* FROM foo;
  a|b
  1|2
 
2858 build closed 2007 Dec anonymous   2008 Jan danielk1977 3 3 amalgamated sqlite3.c requires sqlite3ext.h edit
Staring with SQLite 3.5.3, the "amalgamted" source file sqlite3.c requires sqlite3ext.h to compile. This is because sqlite3.c attempts to effectively include sqlite3ext.h instead of simply copying/pasting its contents:

$ grep sqlite3ext.h sqlite3.c #define SQLITE_CORE 1 /* Disable the API redefinition in sqlite3ext.h / /************** Include sqlite3ext.h in the middle of loadext.c **************/ /************** Begin file sqlite3ext.h **************************************/ ** @(#) $Id: sqlite3ext.h,v 1.17 2007/08/31 16:11:36 drh Exp $ /************** End of sqlite3ext.h ******************************************/ #include "sqlite3ext.h" /************** Include sqlite3ext.h in the middle of fts3_tokenizer.c *******/ /************** Begin file sqlite3ext.h **************************************/ ** @(#) $Id: sqlite3ext.h,v 1.17 2007/08/31 16:11:36 drh Exp $ /************** Include sqlite3.h in the middle of sqlite3ext.h **************/ /************** Continuing where we left off in sqlite3ext.h *****************/ /************** End of sqlite3ext.h ******************************************/

Compare to earlier versions such as SQLite 3.5.2:

#define SQLITE_CORE 1 /* Disable the API redefinition in sqlite3ext.h / /************** Include sqlite3ext.h in the middle of loadext.c **************/ /************** Begin file sqlite3ext.h **************************************/ ** @(#) $Id: sqlite3ext.h,v 1.17 2007/08/31 16:11:36 drh Exp $ /************** End of sqlite3ext.h ******************************************/

If you look at it in context you can see that sqlite3ext.h will never be #included since SQLITE_CORE is defined.

  #ifndef SQLITE_CORE
    #include "sqlite3ext.h"
    SQLITE_EXTENSION_INIT1
  #endif


2007-Dec-29 10:16:59 by anonymous:
And yet SQLite 3.5.2 doesn't build here, maybe because SQLITE_CORE is not defined here. Where and how should SQLITE_CORE be defined?


2007-Dec-29 10:24:58 by anonymous:
We compile SQLite with SQLITE_OMIT_LOAD_EXTENSION defined.


2007-Dec-30 13:10:01 by anonymous:
OK, I now understand the problem much better. Below is a short explanation. My apologies for the initial bug report, it was a bit misleading and not detailed enough.

You've added fts3.c to the amalgamated source file of SQLite, starting with version 3.5.3.

Now "sqlite3ext.h" is needed by two of the amalgamated files:

  • loadext.c: It includes "sqlite3ext.h" and SQLITE_CORE must be defined first.
  • fts3.c: It includes "sqlite3ext.h" and I think SQLITE_CORE must be defined first since the FTS3 module is being built into the core of SQLite and not as an extension.

There are actually many issues:

  1. I think the amalgamated source file should never include "sqlite3ext.h". Instead its should copy/paste its contents when needed. This is done for loadext.c but not for fts3.c, resulting in the following error when building sqlite3.c without "sqlite3ext.h":
    sqlite3.c:80297:26: error: sqlite3ext.h: No such file or directory
    sqlite3.c:80335: error: expected ',', ';', 'asm' or '__attribute__' before 'static'
    sqlite3.c:86299: error: expected ';', ',' or ')' before '*' token
    I solved this by making sure SQLITE_CORE is defined when processing the contents of fts3.c, whether SQLITE_OMIT_LOAD_EXTENSION is defined or not, by moving:
    #ifndef SQLITE_OMIT_LOAD_EXTENSION
    after the block starting with:
    #define SQLITE_CORE 1 /* Disable the API redefinition in sqlite3ext.h */
    /************** Include sqlite3ext.h in the middle of loadext.c **************/

  2. The FTS3 module in the amalgamated source file is built into the resulting library when SQLITE_CORE is not defined, whether SQLITE_ENABLE_FTS3 is defined or not:
    #if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3)

    #if defined(SQLITE_ENABLE_FTS3) && !defined(SQLITE_CORE)
    # define SQLITE_CORE 1
    #endif
    [...]
    I'm not sure that's on purpose. Do you want the FTS3 module always built into the resulting SQLite library? If so, I think it would be more readable not to trigger this by defining SQLITE_CORE. Or at the very least it could be triggered by defining SQLITE_ENABLE_FTS3 if SQLITE_CORE is defined in a single place - not all over the FTS3 source files.

Is there any chance this ticket is opened again? Should I open a new ticket instead ?


2007-Dec-30 17:17:07 by anonymous:
SQLITE_CORE is defined previously in sqlite3.c. You can define it yourself on the commandline if you like.

What compiler are you using? Will it #include a file even when it is not wanted, as in:

  #if 0
  #include "foo.h"
  #endif

or

  #ifdef SOME_UNDEFINED_MACRO
  #include "bar.h"
  #endif


2007-Dec-30 17:53:42 by anonymous:
I can recreate your problem. If I use http://sqlite.org/sqlite-amalgamation-3_5_4.zip, unzip it, delete sqlite3ext.h and grab shell.c from CVS:

$ gcc -I. -DSQLITE_OMIT_LOAD_EXTENSION shell.c sqlite3.c -o shell -lpthread
sqlite3.c:81710:26: error: sqlite3ext.h: No such file or directory
sqlite3.c:81748: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘static’
sqlite3.c:87815: error: expected ‘;’, ‘,’ or ‘)’ before ‘*’ token
You must use both -DSQLITE_OMIT_LOAD_EXTENSION -DSQLITE_CORE on the compile line:

  gcc -I. -DSQLITE_OMIT_LOAD_EXTENSION -DSQLITE_CORE shell.c sqlite3.c -o shell -lpthread

That's a bug. The workaround is simple, though - always compile with -DSQLITE_CORE

As you've found out, if you choose to build with FTS3 support you do not have to #define SQLITE_CORE on the compile line, as it defines it for you:

  gcc -I. -DSQLITE_OMIT_LOAD_EXTENSION -DSQLITE_ENABLE_FTS3 shell.c sqlite3.c -o shell -lpthread


2007-Dec-30 17:55:25 by anonymous:
I'm using GCC, Visual Studio, Intel C++, HP ANSI C/aC++, SGI MIPSpro, Sun Studio, IBM XL C and other compilers. They will certainly not #include a file even when it is not wanted.

How is your remark related to the use case exposed above?


2007-Dec-30 18:08:50 by anonymous:
Ignore the #include thing - it was just an idea.

The sqlite bug is explained in detail in the 2007-Dec-30 17:53:42 comment.


2007-Dec-30 18:09:36 by anonymous:
Indeed, SQLITE_CORE is usually defined by the loadext.c source code. Unfortunately this will not happen if SQLITE_OMIT_LOAD_EXTENSION is defined.

The fts3.c source code in the amalgamated source file needs to be compiled with SQLITE_CORE if we want it:

  • not to include "sqlite3ext.h",
  • and not to build the FTS3 module unless SQLITE_OMIT_LOAD_EXTENSION is defined.

So yes, the solution is to define SQLITE_CORE when building the amalgamated source file:

  • either from the command line, in which case this requirement needs to be documented,
  • or within the amalgamated source file itself, in which case
    #define SQLITE_CORE 1
    should be added before any reference to _sqlite3ext.h".


2008-Jan-01 05:51:17 by danielk1977:
Thanks.
 
2857 code active 2007 Dec anonymous   2007 Dec   2 2 GROUP BY cost estimate wrong with WHERE clause edit
There seems to be an issue with the sqlite cost heuristic with an INDEX present on GROUP BY with certain types of WHERE clauses.

Given the database formed by running these statements:

  create table stuff(a,b,c,d);
  insert into stuff values(1,2,3,4);
  create temp view v1 as select random()%100,
    random()%100, random()%1000, random()%10000
     from stuff x, stuff y;
  insert into stuff select * from v1;
  insert into stuff select * from v1;
  insert into stuff select * from v1;
  insert into stuff select * from v1;
  insert into stuff select * from v1;
  create index stuff_b on stuff(b);
  create index stuff_c on stuff(c);
  create index stuff_d on stuff(d);
  analyze;

Using sqlite.org's sqlite3-3.5.4.bin, this query takes 47 seconds:

  select c from stuff where a=23 group by c;

while this query takes just 2 seconds:

  select c from stuff where a=23 group by +c;

It is more efficient in this case to do a full table scan instead of using the INDEX on column c.

2007-Dec-23 23:14:06 by anonymous:
The queries above both run in a couple of seconds with this naive patch:

--- src/where.c 12 Dec 2007 17:42:53 -0000      1.266
+++ src/where.c 23 Dec 2007 22:48:37 -0000
@@ -1514,6 +1514,12 @@ static double bestIndex(
     flags = 0;
   }

+  if( pWC && pWC->nTerm>0 && pOrderBy ){
+    /* Reduce cost if both an ORDER/GROUP BY exists with a WHERE. */
+    cost /= 100; /* A very rough guess. */
+    WHERETRACE(("... WHERE + ORDER BY decreases cost to: %.9g\n", cost));
+  }
+
   /* If the table scan does not satisfy the ORDER BY clause, increase
   ** the cost by NlogN to cover the expense of sorting. */
   if( pOrderBy ){
But it has not been tested on queries with more than one table. Its logic could be flawed.


2007-Dec-24 00:09:00 by drh:
The complaint is centered around these two queries:

    /* 1 */  SELECT c FROM stuff WHERE a=23 GROUP BY c;
    /* 2 */  SELECT c FROM stuff WHERE a=23 GROUP BY +c;

Query 1 runs in about 40 seconds and query 2 in about 1.5 seconds on my macbook. But with the patch, both queries run in about 1.5 seconds. Fair enough. But now consider these two queries:

    /* 3 */  SELECT c FROM stuff WHERE a!=23 GROUP BY c;
    /* 4 */  SELECT c FROM stuff WHERE a!=23 GROUP BY +c;

In this case, query 3 runs in 42 seconds on an unpatched version of 3.5.4 and query 4 runs in about 109 seconds. So in cases where the WHERE clause is not particularly selective, the first version is faster than the second by a good margin. On a patched version of 3.5.4, both queries 3 and 4 run in about 110 seconds.

So it seems to me that the patch is robbing Peter to pay Paul. It makes ORDER BY queries with very selective WHERE clauses run faster but at the expense of making queries with unselective WHERE clauses running slower.

But notice this: in the current (unpatched) implementation, the programmer at least has the ability to select a different algorithm by the judicious placement of a "+" sign. After the patch, this is no longer possible. The patch forces the second algorithm to be used in all cases, even cases where it is slower.

It seems to me to be better to leave things as they are since the current approach at least allows the programmer to override SQLite's algorithm selection if SQLite chooses incorrectly.

The only way, it seems to me, to automatically choose the correct algorithm is to devise some test that will determine (in advance) whether or not the WHERE clause weeds out many or few rows from the result set. I'm thinking that determination is going to be very hard (or impossible) to do without first doing a full table scan.


2007-Dec-24 05:40:47 by anonymous:
It think it would be surprising to average users that adding an index (on column C in this case) may significantly decrease query performance for some queries. It was surprising to me, at least. In my opinion, a query being 20 times slower in a default bad guess situation is worse than a query only being 2.5 times slower with a default bad guess in a worst case scenario. It's a question of relative magnitude of the difference. This is why I think that the database should err on the side of the WHERE clause having a more selective bias.

(Side note: the query timings difference is less pronounced if you use PRAGMA temp_store=memory, in which case query 3 running on an unpatched 3.5.4 takes just 50% more time to run than query 4 on my machine.)

But you raise a good point in that if there's a wrong guess in the selectivity bias it would be nice to be able to manually override it.

How much do you hate this type of syntax that some other databases use?

  select c from stuff where a!=23 group by /*+stuff_c*/ c;

SQLite does not currently offer a way to pick a specific index. I think it would be quite useful.


2007-Dec-24 17:05:16 by anonymous:
Another option is to collect WHERE clause statistics in a table like

  create table sqlite_stat2(
    where_clause_md5 BLOB primary key,
    where_clause     TEXT,
    rows_examined    INT,
    rows_true        INT
  );

where the last 2 columns are cumulative for each query. The statistics option could be enabled/disabled via a PRAGMA sqlite_collect_statistics.

The where_clause column could be a string generated fairly easily from the walking the parse tree of the resolved Select statement's pWhere. This way the where_clause is normalized and a single query with many subselects could generate more than 1 where_clause, and different queries that happen to use the same normalized where clause would update the same entry in the stat2 table. where_clause normalization would strip off aliases and only refer to the original table and column names.

For example the 2 queries below:

  -- CREATE TABLE t1(a, b);
  -- CREATE TABLE t2(b, c);
  SELECT t1.a*c as AC, t2.b-a as BA FROM t1, t2 WHERE AC>BA;
  SELECT *, t1.a Foo FROM t2, t1 WHERE Foo*c > t2.b - t1.a;

would generate the same normalized where_clause string "(T1.A*T2.C)>(T2.B-T1.A)". The table information is already encoded within it.

The generated VDBE code would have to generate Mem counters that would be incremented by each WHERE test, and lazily updated at the end of transactions or periodically written to the stat2 table to minimize disk use, as this information is not critical.

One could also manually set the stat2 table with statistical values they would like their queries to use even if PRAGMA sqlite_collect_statistics=off;

Any time the schema is changed, the entire sqlite_stat2 table would be cleared.

 
2856 doc active 2007 Dec anonymous   2007 Dec anonymous 4 3 SQLite Documentation - Tcl API - Link broken edit
On the page "The Tcl Interface to the SQLite library", the link "enable load extension" does not work.
 
2855 code closed 2007 Dec anonymous   2007 Dec drh 2 3 Get an error "11:database disk image is malformed" during queries edit
I don't know what has happened to my database and when. I followed every new sqlite version as soon as it got released. So now I am with version 3.5.4 (WinXP) and my database has grown up to a size of about 1.74 GB. It contains only three tables and neither views nor triggers.

When I run a query on all records of one table I got the a.m. error message and no results. I was able to query all details by RowID up to and above a small range of only 105 records (out of 345767 ). No problem below and none above. Then I renamed the table, created another table and "insert into .. select from .." one into the other in two steps. ("where RowID between .. and .. or RowID between .. and .." popped up the error message again but running two queries one after the other worked fine.

That's my workaround.

It is a pitty that I cannot send you an example but I do not know how to shrink the database to only remain with the route cause of trouble. I dropped the other two tables and deleted all records before and after the malformed part but now I cannot vacuum without getting again that message.

  1. How may I got this malformed records into my database? How to avoid repetition?
  2. How can I eliminate a malformed part out of the database without having to recreate almost everything from scrap?

As soon as the malformed records are touched every command stops.

Isn't there a possibility to put such records into quarantine ?! This would really help to isolate problems and would hopefully give me a chance to send you the damaged records for your analysis to find out what may have happened. The isolation of records may be related to "check integrity" of the database. Another good idea may be to give sqlite.exe a chance to extract isolated part under quaratine from one into another database. - I keep my damaged database for such a case for a while, expecting such a command with the next release.

2007-Dec-21 12:52:51 by drh:
Information on how to corrupt an SQLite database file can be found at

To restore data as much as you can from a corrupt database file, do this:

   sqlite3 corrupt.db .dump | sqlite3 restored.db

This ticket reads more like a support request than a bug report. Or perhaps it is a feature request - requesting that corruption be "quarantined". Quarantining bad records merely masks errors and gives no new capabilities so that is not something we are interested in doing. For technical support see


2007-Dec-21 13:06:46 by anonymous:
Thank you for your quick response and hints about more info on causes for db corruptions. However, "sqlite3.exe corrupt.db .dump | sqlite3 restored.db" did not work because of the malformed records, which interrupted the process after a while of work. - For me it would be enough if "delete from tbl where RowID between nn and mm" would work without failing because of using the malformed space. This would be good for healing a db file if the problem could be isolated like in my case. Sorry, if dealing with such problems isn't the right place here. (By the way: the first 5 fields of that table are forming the primary index; there are no other indexes on that table.) I wish you a merry christmas and a happy and successfull new year.


2007-Dec-21 13:08:59 by drh:
The ".dump" command attempts to work around corruption. If you have some kind of corruption that it is unable to work-around, then append an example database to this ticket and we will reopen the ticket and work on the problem. Without having access to a database that demonstrates the problem, I am afraid there is not much we can do.


2007-Dec-21 14:03:59 by anonymous:
Well. Didn't know about .dump is working around corruption and tried to do this with my database where I eliminated good content around the malformed records. Your recommendation worked without raising the aborting error message. It will not solve the appearance but is a great help to heal the database for continued work. Thank you for this additional information!
 
2854 code closed 2007 Dec danielk1977   2007 Dec danielk1977 1 1 BEGIN EXCLUSIVE does not prevent other shared-cache users from reading edit
Subject says it all. Executing a "BEGIN EXCLUSIVE" does not prevent other shared cache users from reading the database.
2007-Dec-20 17:53:32 by danielk1977:
The attached test file demonstrates the problem.


2007-Dec-20 19:28:29 by anonymous:
Could this cause unexpected deadlocks?
 
2853 new active 2007 Dec anonymous   2008 Jan   2 3 optimizer fails to use an index on MAX subquery edit
i have these 2 identical queries with an index on (place_id,visit_date), the second query is about 2x fast the the first, while i'd expect that the MAX is faster than a limited order by clause... It's like the index is mis-used with MAX

  SELECT h.id, h.url, h.title, h.rev_host, h.visit_count,
  (SELECT MAX(visit_date) FROM moz_historyvisits WHERE place_id = h.id
  AND visit_type NOT IN(0,4)),
  f.url, null, null
  FROM moz_places h
  LEFT OUTER JOIN moz_favicons f ON h.favicon_id = f.id
  WHERE h.id IN
  (SELECT DISTINCT p.id
          FROM moz_places p
          JOIN moz_historyvisits ON place_id = p.id
          WHERE hidden <> 1 AND visit_type NOT IN (0,4)
          ORDER BY visit_date DESC
          LIMIT 10)
  ORDER BY 6 DESC;

  SELECT h.id, h.url, h.title, h.rev_host, h.visit_count,
  (SELECT visit_date FROM moz_historyvisits WHERE place_id = h.id
   AND visit_type NOT IN(0,4) ORDER BY visit_date DESC LIMIT 1),
  f.url, null, null
  FROM moz_places h
  LEFT OUTER JOIN moz_favicons f ON h.favicon_id = f.id
  WHERE h.id IN
  (SELECT DISTINCT p.id
          FROM moz_places p
          JOIN moz_historyvisits ON place_id = p.id
          WHERE hidden <> 1 AND visit_type NOT IN (0,4)
          ORDER BY visit_date DESC
          LIMIT 10)
  ORDER BY 6 DESC;
2007-Dec-20 04:13:44 by anonymous:
Can you supply the schema of these tables and all related indexes?


2007-Dec-20 10:55:29 by danielk1977:
Correct. SQLite does not use the index to optimize the max() in the first query. But it does use it to optimize ORDER BY in the second.


2007-Dec-21 00:54:10 by anonymous:
the schema is that of mozilla firefox 3 places.sqlite, with an added index on moz_historyvisits(place_id,visit_date), this is extracted from a i bug i was working on. Do you need that i append the full schema here?

Will this be fixed to use the index with max or is that the expected behaviour?


2007-Dec-21 15:59:58 by drh:
We will be enhancing the min()/max() optimization to be able to cover the case you describe above. But this will take some time to get right. If you are in a hurry, it seems like modifying your SQL is the easiest way to go.

The current optimizer recognizes queries of the form:

    SELECT min(x) FROM table;
    SELECT max(x) FROM table;

And it causes special VDBE code to be generated for these cases. The presence of a WHERE clause defeats the current optimization. We need to modify the optimizer to recognize queries of the form:

    SELECT min(x) FROM table WHERE expr

And convert them into:

    SELECT x FROM table WHERE expr AND x NOT NULL
     ORDER BY x LIMIT 1

But we only want to do this optimization if the ORDER BY can be evaluated using an index. If we have to accumulate the entire result set, sort it, then take the largest entry, that is more work than just keeping track of the largest entry as we loop through the result set.

Notice the addition of the "x NOT NULL" term in the WHERE clause. This is critical to preserving correct semantics. The query optimizer currently does not know how to deal with a NOT NULL. It just evaluates all rows, tests them individually for NULL, and throws out those that match. This would work in most cases, but would slow down in a min() where there were many NULL entries. The current implemention of the min() optimization knows to skip over the NULL entries in a single operation. The WHERE generator part of the optimizer will need to be enhanced to recognize NOT NULL constraints and skip over them.

All of this is a substantial change. The min()/max() optimization function will be rewritten from scratch. Significant and non-trivial modifications will need to be made to the code that looks for indices and generates the WHERE clause constraints. There are many opportunities to make coding mistakes, so we will need to spend a lot of time in testing before we put these changes into production.

So, we will be working the problem. But do not expect an overnight fix.

I suppose that while we are redesigning the min/max optimizer, we might as well also fix it so that

   SELECT arbitrary_expr(max(x)) FROM table WHERE expr;

gets converted into

   SELECT arbitrary_expr(x) FROM table WHERE expr AND x NOT NULL
    ORDER BY x DESC LIMIT 1;


2007-Dec-31 06:41:03 by anonymous:
Regarding:

  SELECT min(x) from table WHERE expr

being converted to:

  SELECT x FROM table WHERE expr AND x NOT NULL ORDER BY x LIMIT 1

There's no need for the "x NOT NULL" condition considering NULL is returned by min() (or max for that matter) when no rows match.

  sqlite> .nullvalue <null>

  sqlite> select min(a) from (select 123 as a) where a=7;
  <null>

  sqlite> select min(a) from (select NULL as a) where a=7;
  <null>

Even with this in mind, you can see that the rewritten query still does not return the same result in this case:

  sqlite> select a from (select 123 as a) where a=7 order by 1 limit 1;
  -- no rows returned

Logic would have to be added to return a NULL row in the event the WHERE clause matches no rows.


2007-Dec-31 07:06:22 by anonymous:
Given:

  SELECT min(x) from table WHERE expr

If column x or the WHERE expr can make use of an index, then the min query should be converted to:

  SELECT x from (
    SELECT x FROM table WHERE expr
      ORDER BY x LIMIT 1
  ) UNION ALL SELECT NULL LIMIT 1

which ought to cover all the corner cases, even if the WHERE matches no rows.


2008-Jan-01 19:21:51 by anonymous:
This ticket is related to WHERE cost estimation in ticket #2857.

Given:

  create table stuff(a,b,c,d);
  insert into stuff values(1,2,3,4);
  create temp view v1 as select random()%100,
    random()%100, random()%1000, random()%10000
     from stuff x, stuff y;
  insert into stuff select * from v1;
  insert into stuff select * from v1;
  insert into stuff select * from v1;
  insert into stuff select * from v1;
  insert into stuff select * from v1;
  create index stuff_b on stuff(b);
  create index stuff_c on stuff(c);
  create index stuff_d on stuff(d);
  analyze;

In the following example the existing min() implementation which uses a full table scan:

  sqlite> explain query plan select min(c) from stuff where a=12345;
  0|0|TABLE stuff

  sqlite> select min(c) from stuff where a=12345;
  (null)
  CPU Time: user 1.388087 sys 0.216014

is faster than performing the select min->order by/limit transform which uses an index:

  sqlite> explain query plan select c from stuff where a=12345 order by 1 limit 1;
  0|0|TABLE stuff WITH INDEX stuff_c ORDER BY

  sqlite> explain query plan select c from (select c from stuff where a=12345 order by c limit 1) union all select null limit 1;
  0|0|TABLE stuff WITH INDEX stuff_c ORDER BY
  0|0|TABLE  AS sqlite_subquery_82F83F8_
  CPU Time: user 0.000000 sys 0.000000

  sqlite> select c from (select c from stuff where a=12345 order by c limit 1) union all select null limit 1;
  (null)
  CPU Time: user 15.880993 sys 15.400962

Note that a=12345 is always false in this data set.

Had a=23 been used instead, we can see that the transformed select would be much faster than the original min() select:

  sqlite> select min(c) from stuff where a=23;
  -999
  CPU Time: user 1.552097 sys 0.148009

  sqlite> select c from (select c from stuff where a=23 order by c limit 1) union all select null limit 1;
  -999
  CPU Time: user 0.004001 sys 0.000000

I don't think WHERE clause cost estimation can be done accurately without using a statistical method based on historical WHERE clause hit percentages.


2008-Jan-05 18:15:39 by anonymous:
I appreciate that you have more refinements pending, but this query in particular has regressed from sqlite 3.5.4 using the schema mentioned above:

  -- sqlite 3.5.4
  sqlite> select min(c) from stuff where a=12345;
  (null)
  CPU Time: user 1.532095 sys 0.132008

  -- as of Check-in [4687]
  sqlite> select min(c) from stuff where a=12345;
  (null)
  CPU Time: user 23.041440 sys 16.369023
 
2852 code fixed 2007 Dec anonymous   2007 Dec   4 4 sqlite library is sometimes 40K bigger when built on OpenBSD edit
lhsStart not being initialized leads to non-deterministic behavior in parser code generation. The generated code code for parse.c will run correctly, but will be 40K larger if lhsStart is not set to zero after rule struct memory is malloc'ed. See #2835 for details and fix.
 
2851 code fixed 2007 Dec anonymous   2007 Dec   1 1 make test double free or corruption error edit
Fresh CVS checkout, ./configure && make && make test

capi3-13-4... Ok
capi3-13-5...*** glibc detected ***
capi3-14.1-misuse... Ok
capi3-15.1...
Error: error code SQLITE_TOOBIG (18) does not match sqlite3_errcode SQLITE_OK (0)
capi3-15.2...
Expected: [2]
     Got: [0]
capi3-15.3...
Expected: [SQLITE_OK]
     Got: [SQLITE_MISUSE]
capi3-15.4...
Error: (21) library routine called out of sequence
capi3-15.5...
Error: (21) library routine called out of sequence
capi3-15.6...
Error: (21) library routine called out of sequence
capi3-15.7...
Error: (21) library routine called out of sequence
capi3-15.8...
Error: (21) library routine called out of sequence
capi3-16.1...
Error: (21) library routine called out of sequence
capi3-16.2...
Error: (21) library routine called out of sequence
capi3-16.3...
Error: (21) library routine called out of sequence
capi3-16.4...
Error: (21) library routine called out of sequence
capi3-17.1...
Error: (21) library routine called out of sequence
capi3-17.2... Ok
capi3-17.3...*** glibc detected *** ./testfixture: double free or corruption (!prev): 0x0812d650 ***
 
2850 code fixed 2007 Dec anonymous   2007 Dec   2 2 CSV output missing quotes if contains equals character edit
Here's yet another bug in the CSV output using the sqlite3 command line tool.

Here's a sample of the new bug:

.mode csv select 'a=1,234', 'b=5';

gives: a=1,234,b=5

but should give: "a=1,234",b=5

I'm using sqlite 3.4.0 on Mac OS X 10.5.1.

CSV (comma separated values) output should show value1,value2,value3 etc. If a value contains a comma, then csv should encapsulate that value in quotes: "value,1",value2,value3. And if a value contains a quote, csv should also encapsulate that value in quotes and show the quote as a double quote, eg "value,1","value ""2""",value3

Sqlite3 seems to ignore the need for quotes in a value that contains an equals "=". I don't know if it also fails with other characters.

Needless to say, this messes up the output considerably, placing output in the wrong column.

 
2849 doc fixed 2007 Dec anonymous   2007 Dec   5 5 unary operator ! edit
From documentation:

"Supported unary prefix operators are these:

- + ! ~ NOT"

There is no ! operator in sqlite

 
2848 code closed 2007 Dec anonymous   2007 Dec   4 4 windows CRLF is incorrectly handled by .dump edit
If I insert a string containing standard windows CRLF (0x0D, 0x0A) into the table, and then try to .dump the table, instead of original CRLF, I get a 'CRCRLF' (0x0D, 0x0D, 0x0A). It is probably not helpful to mention that the column does not use any special collation and that the database is in UTF-8 mode. I am on windows xp, and use original build of sqlite3.exe 3.5.4.

I am sorry if this problem already has its own ticket, or if I missed some configuration in sqlite3/dump, but I could not find any such.

2007-Dec-17 14:06:58 by anonymous:
I can't reproduce this on MS Windows Vista with SQLite v3.5.4.

Snippet from .dump output:

sqlite3 test.db3 .dump | od  --address-radix=x --format=x1a

000060 0d 0a 49 4e 53 45 52 54 20 49 4e 54 4f 20 22 74
        cr  nl   I   N   S   E   R   T  sp   I   N   T   O  sp   "   t
000070 65 73 74 31 22 20 56 41 4c 55 45 53 28 31 2c 27
         e   s   t   1   "  sp   V   A   L   U   E   S   (   1   ,   '
000080 61 6c 0d 0a 70 68 61 27 29 3b 0d 0a 43 4f 4d 4d
         a   l  cr  nl   p   h   a   '   )   ;  cr  nl   C   O   M   M
000090 49 54 3b 0d 0a
         I   T   ;  cr  nl

Perhaps the extra control character was accidently loaded into the database?


2007-Dec-17 15:18:53 by drh:
Unable to reproduce on WinXP.


2007-Dec-17 22:12:27 by anonymous:
Thanks for fast response. The extra 0x0D was surely not in the database. I reviewed my situation over and it looks like a windows shell (cmd.exe) is adding the extra 0x0D as I was redirecting stdout from dump like this: cmd /c sqlite3 test.db3 .dump > test.txt Sorry to blame you.
 
2847 new active 2007 Dec anonymous   2008 Jan   5 4 Include major, minor, and patch version numbers in sqlite.h edit
Hi, I'm working on a project where sqlite is being compiled into a DLL. Currently, sqlite.h makes the version number available as both an x.y.z string and an integer value in the form of x*1000000 + y*1000 + z.

Unfortunately, neither of these options works particularly well when trying to create a resource file so that the DLL can display the proper version information within Windows. I've tried many different ways of disassembling the integer version number, but limitations in the resource compiler unfortunately prevent them from working.

As a result, I've been forced for the time being to define SQLITE_VERSION_MAJOR, SQLITE_VERSION_MINOR, and SQLITE_VERSION_PATCH with manually-given values of x, y, and z respectively in order to accomplish this task. It would be really nice if these could be generated automatically for sqlite.h when running configure in the same way that VERSION and VERSION_NUMBER are so that setting the values manually wouldn't be required for the future.

Would you be willing to do that? Thanks in advance.

2007-Dec-17 13:30:22 by anonymous:
You could invoke this awk script in your make file:
#
## extrvers.awk
## Extract verison parts from sqlite3.h
#
## Usage:
#  %GNU_AWK% -f extrvers.awk sqlite3.h >sqlite3.h.new
# 	rm / del sqlite3.h
# 	mv / ren sqlite3.h.new sqlite3.h
#
#
## Ignore any previous defines
/^#define SQLITE_VERSION_(MAJOR|MINOR|PATCH)/{
	next
}
## generate extra #define MAJOR/MINOR/PATH lines
/^#define[[:blank:]]+SQLITE_VERSION[[:blank:]]/{
	split(substr($3,2,length($3) - 2),tmp,".")
	print "#define SQLITE_VERSION_MAJOR " tmp[1]
	print "#define SQLITE_VERSION_MINOR " tmp[2]
	print "#define SQLITE_VERSION_PATCH " tmp[3]
}
## Repeat all other lines untouched
{
	print
}


2007-Dec-18 00:06:53 by anonymous:
Original poster here. Thanks for the useful script! I'm assuming you're granting a license for this (or a modified version of it) to be included in the source tree if the powers that be are willing to accept it? This would be for the Mozilla project.


2007-Dec-18 00:21:52 by drh:
First off, I didn't post the script. I don't know who "anonymous" is.

Secondly, if you are working for Mozilla, you will get much faster service if you identify yourself as such.


2007-Dec-18 00:39:38 by anonymous:
I just want to make sure I'm not running afoul of anybody by using their work without proper permission.

To whoever posted it, you can contact me at ryanvm [at] gmail [dot] com. Thanks again for help!


2007-Dec-26 19:06:27 by anonymous:
I'm the poster of the script. Of course I don't mind it being used. For the peace of mind of anyone using it: please prepend the code with:
# Copyright (C) 2007 by Kees Nuyt, Rotterdam, Netherlands
# The author of this code dedicates any and all copyright
# interest in this code to the public domain. I make this
# dedication for the benefit of the public at large and
# to the detriment of my heirs and successors. I intend
# this dedication to be an overt act of relinquishment in
# perpetuity of all present and future rights to this
# code under copyright law.
#

Cheers! "Kees Nuyt" <k [dot] nuyt [at] zonnet [dot] nl>


2007-Dec-27 02:32:44 by anonymous:
Declaring a copyright and putting it into the public domain is not compatible.


2008-Jan-11 04:55:22 by anonymous:
Original poster here: FYI, I ended up writing a Python script to parse sqlite3.h for what we needed. The Mozilla bug for the work was

You can see the final script here: http://mxr.mozilla.org/mozilla/source/db/sqlite3/src/sqlite-version.py

At this point, you can close the bug if you want. I'll leave that up to you to decide.

 
2846 build fixed 2007 Dec anonymous   2007 Dec   1 1 --disable-tcl does not work edit
rm -rf tsrc mkdir -p tsrc cp ./src/alter.c ./src/analyze.c ./src/attach.c ./src/auth.c ./src/btmutex.c ./src/btree.c ./src/btree.h ./src/build.c ./src/callback.c ./src/complete.c ./src/date.c ./src/delete.c ./src/expr.c ./src/func.c ./src/hash.c ./src/hash.h ./src/insert.c ./src/journal.c ./src/legacy.c ./src/loadext.c ./src/main.c ./src/malloc.c ./src/mem1.c ./src/mem2.c ./src/mem3.c ./src/mem4.c ./src/mutex.c ./src/mutex_os2.c ./src/mutex_unix.c ./src/mutex_w32.c ./src/os.c ./src/os_unix.c ./src/os_win.c ./src/os_os2.c ./src/pager.c ./src/pager.h ./src/parse.y ./src/pragma.c ./src/prepare.c ./src/printf.c ./src/random.c ./src/select.c ./src/shell.c ./src/sqlite.h.in ./src/sqliteInt.h ./src/table.c ./src/tclsqlite.c ./src/tokenize.c ./src/trigger.c ./src/utf.c ./src/update.c ./src/util.c ./src/vacuum.c ./src/vdbe.c ./src/vdbe.h ./src/vdbeapi.c ./src/vdbeaux.c ./src/vdbeblob.c ./src/vdbefifo.c ./src/vdbemem.c ./src/vdbeInt.h ./src/vtab.c ./src/where.c ./ext/fts1/fts1.c ./ext/fts1/fts1.h ./ext/fts1/fts1_hash.c ./ext/fts1/fts1_hash.h ./ext/fts1/fts1_porter.c ./ext/fts1/fts1_tokenizer.h ./ext/fts1/fts1_tokenizer1.c sqlite3.h ./src/btree.h ./src/btreeInt.h ./src/hash.h ./src/sqliteLimit.h ./src/mutex.h opcodes.h ./src/os.h ./src/os_common.h ./src/sqlite3ext.h ./src/sqliteInt.h ./src/vdbe.h parse.h ./ext/fts1/fts1.h ./ext/fts1/fts1_hash.h ./ext/fts1/fts1_tokenizer.h ./src/vdbeInt.h tsrc cp: warning: source file `./src/btree.h' specified more than once cp: warning: source file `./src/hash.h' specified more than once cp: warning: source file `./src/sqliteInt.h' specified more than once cp: warning: source file `./src/vdbe.h' specified more than once cp: warning: source file `./ext/fts1/fts1.h' specified more than once cp: warning: source file `./ext/fts1/fts1_hash.h' specified more than once cp: warning: source file `./ext/fts1/fts1_tokenizer.h' specified more than once cp: warning: source file `./src/vdbeInt.h' specified more than once rm tsrc/sqlite.h.in tsrc/parse.y cp parse.c opcodes.c keywordhash.h tsrc tclsh ./tool/mksqlite3c.tcl make: tclsh: Command not found make: *** [sqlite3.c] Error 127
2007-Dec-17 00:45:26 by anonymous:
Seems the same problem as #2845
 
2845 code fixed 2007 Dec rse   2007 Dec   3 3 SQLite 3.5.4 requires TCL during built-time edit
SQLite 3.5.4 requires TCL during build-time as the generated sqlite3.c is not part of the distribution tarball. This is IMHO a major drawback as SQLite on the one hand is a small light-weight RDBMS without any external dependencies (so even can be used for bootstrapping other tools) and on the other hand SQLite's TCL dependency now requires one to build the TCL interpreter. I strongly recommend to either rewrite the mksqlite3c.tcl in /bin/sh, AWK or C or at least distribute a pre-generated version of sqlite3.c in the distribution tarball with SQLite 3.5.5 and higher. The newly introduced extra dependency to TCL is at least a nasty issue for packagers...
2007-Dec-15 21:10:10 by anonymous:
There's a "testcli" target added (see Ticket #2838), although I prefer the original way sqlite was built prior to Check-in [4609] for the reasons described in the that ticket.
 
2844 build active 2007 Dec anonymous   2007 Dec   4 1 lemon is being built without respecting LDFLAGS edit
lemon is being built without respecting LDFLAGS. I'm attaching a patch which fixes this bug.

In other words, why should we fix this? What problem is it causing?

2007-Dec-17 16:22:19 by drh:
Why is this important? What LDFLAGS settings might a user want to carry through into lemon?


2007-Dec-17 18:00:59 by anonymous:
> Why is this important?

It is considered to be be good practice to respect user's LDFLAGS. A user might want to have all executables and libraries built with identical LDFLAGS.

> What LDFLAGS settings might a user want to carry through into lemon?

A user might have LDFLAGS="-Wl,-O1,--hash-style=gnu,--sort-common"

You can read http://lwn.net/Articles/192082/.

Users can also use some other flags.

> In other words, why should we fix this? What problem is it causing?

It slightly increases the size of lemon executable and it slightly decreases performance.


2007-Dec-17 18:04:31 by drh:
lemon is used as an intermediate build tool in part of the SQLite build process. It is not a deliverable. If it runs a little slower or uses a little more memory, nobody cares. We only care if it gets the wrong answer.

Is it ever possible that the lack of LDFLAGS support might result in lemon getting the wrong answer?


2007-Dec-17 18:27:33 by anonymous:
Can you comment on Lemon bug in #2835? It produces 2 different sqlite3.c files depending on your malloc implementation.


2007-Dec-17 19:19:01 by anonymous:
> lemon is used as an intermediate build tool in part of the

> SQLite build process. It is not a deliverable. If it runs a

> little slower or uses a little more memory, nobody cares.

CFLAGS are respected when lemon is being built, so for consistency LDFLAGS also should be respected.

(The comment above was not created by me.)

 
2843 code fixed 2007 Dec anonymous   2007 Dec   3 4 Low memory crash in sqlite3ExprCodeAndCache edit
The function sqlite3ExprCodeAndCache() in expr.c includes the four lines:

  addr1 = sqlite3VdbeCurrentAddr(v);
  sqlite3ExprCode(pParse, pExpr);
  addr2 = sqlite3VdbeCurrentAddr(v);
  if( addr2>addr1+1 || sqlite3VdbeGetOp(v, addr1)->opcode==OP_Function ){

sqlite3VdbeGetOp() can return a NULL pointer (if there have been allocation failures AND (0 <= addr1 < v->nOp) does not hold).

This can occur if there are memory allocation failures which prevent sqlite3ExprCode() from adding any vdbe operations (so addr2 is the same as addr1 which is equal to v->nOp).

The solution is probably to include a test for allocation failures in the if condition, ie:

  addr1 = sqlite3VdbeCurrentAddr(v);
  sqlite3ExprCode(pParse, pExpr);
  addr2 = sqlite3VdbeCurrentAddr(v);
  if( !v->db->mallocFailed && (addr2>addr1+1 ||
                               sqlite3VdbeGetOp(v, addr1)->opcode==OP_Function ) ){
 
2842 code active 2007 Dec anonymous   2007 Dec   1 1 .import does not recongnise NULL values edit
.import function fails to see NULL values in csv files as NULL values...instead they are treated as the string "NULL".

This is with .mode list and separator , But behaves similarly for .mode csv

Also if one outputs a table with NULL values to a file, then re-imports that file, again .import does not recognise the values as NULL, but as "NULL".

Everything here also applies to empty strings in files, e.g. instead of "NULL" using nothing...

This is a showstopper for us since we want to import a large amount of data with many tables containing NULL values. I can't see any valid reason for .import not to recognise the same syntax as the command line.

Note that something like:

sqlite3 my.db insert into MY_TABLE values (1,"foo","bar",NULL)

..works fine. It is just .import that appears to be broken.

2007-Dec-14 16:39:51 by rdc:
.import only inserts string values into database tables. If your column has a declared type that changes the columns affinity to numeric or integer, then those strings will be converted to numeric values by the SQLIte library.

The workaround is to simply insert a unique string where ever you want a NULL value, and then run an update that replaces those strings with real NULL values. If you inserted the string 'NULL' then do this after the .import

update t set field = null where field = 'NULL';

You will have to repeat this for each field in your table that might contain the 'NULL' string.

 
2841 todo active 2007 Dec anonymous   2007 Dec   1 1 The sqlite mailing list has become overrun by trolls edit
The sqlite mailing list is very useful. The S/N is at times a little high but nonetheless quite manageable.

Recently (see the DeviceSQL thread) it got really bad. Would moderation be unacceptable during these periods of time where people feel the need to protect their ego's?

The sqlite mailing list is primarily about sqlite (well, and lemon), not a marketing vector for other products? Surely they have their own lists and resources for that?

 
2840 doc fixed 2007 Dec anonymous   2007 Dec   5 4 Punctuation errors in comment edit
The comment relating to the declaration of function sqlite3_reset in the sqlite/src/sqlite.h header has two punctuation errors:

** The sqlite3_reset() function is called to reset a ** [sqlite3_stmt | compiled SQL statement] object. ** back to it's initial state, ready to be re-executed.

Should be:

** The sqlite3_reset() function is called to reset a ** [sqlite3_stmt | compiled SQL statement] object ** back to its initial state, ready to be re-executed.

If in doubt, see http://www.bristol.ac.uk/arts/skills/grammar/grammar_tutorial/page_13.htm

2007-Dec-13 21:14:37 by drh:
I observe that "it's" is used where "its" is correct in many comments. But Christopher, if you post one more ticket about this I promise I will ban you from this website forever. One ticket is enough!


2007-Dec-13 21:16:24 by anonymous:
lol - sorry.
 
2839 new defer 2007 Dec anonymous   2007 Dec   4 4 Deleting row from sqlite_stat1 will not take effect until next connect edit
I noticed if sqlite_stat1 is purged or altered, the query plans remain the same until you disconnect and reconnect to the database. Is this by design?
2007-Dec-13 17:54:02 by drh:
If you manually mess around with the sqlite3_stat1 table, I don't think it is too much to ask that you restart the connection to make those changes go into effect. To add code for this specialized corner case seems unwarranted.
 
2838 code fixed 2007 Dec anonymous   2007 Dec   3 3 Request: use traditional way to build CLI edit
This is a request to revert the recent Check-in [4609] (but keep the large sql max len value) to use the traditional .o files and library. It's more difficult to debug changes when compile errors and gdb line numbers are from the amalgamation sqlite3.c, not the original .c source files. Incremental compile times take longer with the amalgamation as well.

Using the amalgamation can also mask missing #include files from .c files which cause compile errors when compiling in the "traditional" .c->.o manner. Such missing header files in certain .c source files may work in the amalgamation due to some other pre-pended .c file that just happened to have included the header file.

Also, on some platforms the C compiler cannot compile the sqlite3.c amalgamation with -g (debug) because it is too big.

2007-Dec-13 07:57:04 by anonymous:
As a compromise, perhaps a new make target "make release" might build an optimized sqlite3 CLI with the sqlite3.c amalgamation, while the default "make" builds via the traditional .c->.o means.


Added the "testcli" target for this purpose.
 
2837 code closed 2007 Dec anonymous   2007 Dec   2 2 can't build testfixture in latest CVS edit
./configure && make clean && make testfixture

 gcc -g -O2 -I. -I./src -DNDEBUG -I/usr/include/tcl8.4.13 -DSQLITE_THREADSAFE=1 -DSQLITE_THREAD_OVERRIDE_LOCK=-1 -DSQLITE_OMIT_LOAD_EXTENSION=1 -c ./src/alter.c -o alter.o
./src/alter.c: In function ‘renameTableFunc’:
./src/alter.c:77: error: ‘TK_SPACE’ undeclared (first use in this function)
./src/alter.c:77: error: (Each undeclared identifier is reported only once
./src/alter.c:77: error: for each function it appears in.)
./src/alter.c: In function ‘renameTriggerFunc’:
./src/alter.c:135: error: ‘TK_SPACE’ undeclared (first use in this function)
make: *** [alter.lo] Error 1
2007-Dec-13 13:22:42 by drh:
Works for me.

Try running configure in a directory separate from the source tree:

   ../sqlite/configure; make testfixture


2007-Dec-13 16:16:04 by anonymous:
When I "cvs up -A; cvs diff" the sources show no changes at all. So I tried the following:

  make clean
  make distclean
  ./configure --disable-shared
  make test

But I get the same errors.

So I run the old tree and the new tree's alter.c through gcc -E and here's the difference:

--- ../../sqlite/alter.old      2007-12-13 11:04:15.000000000 -0500
+++ alter.new   2007-12-13 11:04:25.000000000 -0500
@@ -599,7 +599,7 @@
 HashElem *sqlite3HashFindElem(const Hash*, const void *pKey, int nKey);
 void sqlite3HashClear(Hash*);
 # 154 "./src/sqliteInt.h" 2
-# 1 "./src/parse.h" 1
+# 1 "./parse.h" 1
 # 155 "./src/sqliteInt.h" 2
 # 1 "/usr/include/stdio.h" 1 3 4
 # 28 "/usr/include/stdio.h" 3 4
@@ -3532,7 +3532,7 @@
       do {
         zCsr += len;
         len = sqlite3GetToken(zCsr, &token);
-      } while( token==TK_SPACE );
+      } while( token==145 );
       ((void) (0));
     } while( token!=19 && token!=115 );

@@ -3582,7 +3582,7 @@
       do {
         zCsr += len;
         len = sqlite3GetToken(zCsr, &token);
-      }while( token==TK_SPACE );
+      }while( token==145 );
       ((void) (0));
 # 147 "./src/alter.c"
       dist++;
So somehow an old parse.h got copied to to src/ which was the cause of the problem.

If I remove src/parse.h as is well and "make test" now builds.

Documented in this ticket in case someone else happens to have the same problem in the future.


2007-Dec-13 16:18:16 by anonymous:
I must have been testing lemon on src/parse.y some time in the past to cause this problem.
 
2836 build closed 2007 Dec anonymous   2007 Dec   1 1 Undefined symbol 'OP_StackDepth' edit
There is a error on current cvs version(12/12/07)

with "Undefined symbol 'OP_StackDepth'" message.

opcodes.h and opcodes.c havn't 'OP_StackDepth' definition currently.

Sorry for my short English.

2007-Dec-13 07:41:07 by danielk1977:
opcodes.c and opcodes.h are auto-generated files (generated based on vdbe.c). Likely your copies have not been regenerated since yesterdays modifications.

Try running "make clean" before rebuilding. If that fails, manually remove opcodes.c and opcodes.h from your build directory.

 
2835 code fixed 2007 Dec anonymous   2007 Dec   1 1 Lemon memory read/write errors edit
$ cat lemonbug.y
phrase ::= foo AND A B C foo.
phrase ::= foo AND A B C bar.
foo ::= BIRD | CAT | DOG.
bar ::= CAT | DOG.

$ ./lemon lemonbug.y
1 parsing conflicts.

$ head -23 lemonbug.out
State 0:
          phrase ::= * foo AND A B C foo
          phrase ::= * foo AND A B C bar
          foo ::= * BIRD| CAT| DOG

                          BIRD shift  2
                               shift  2
                        phrase accept
                           foo shift  5

State 1:
          phrase ::= foo AND A B C * foo
          phrase ::= foo AND A B C * bar
          foo ::= * BIRD| CAT| DOG
          bar ::= * CAT| DOG

                          BIRD shift  2
                               shift  2
                               reduce 134589256 ** Parsing conflict **
                           CAT shift  4
                           foo shift  9
                           bar shift  10

valgrind output:

==11245== Invalid read of size 1
==11245==    at 0x8049109: SetAdd (lemon.c:4080)
==11245==    by 0x804B083: FindFirstSets (lemon.c:649)
==11245==    by 0x8051919: main (lemon.c:1464)
==11245==  Address 0x416980F is 0 bytes after a block of size 7 alloc'd
==11245==    at 0x401F396: malloc (in /usr/lib/valgrind/x86-linux/vgpreload_memcheck.so)
==11245==    by 0x804AEF3: SetNew (lemon.c:4057)
==11245==    by 0x804AF7D: FindFirstSets (lemon.c:617)
==11245==    by 0x8051919: main (lemon.c:1464)
==11245==
==11245== Invalid write of size 1
==11245==    at 0x804910C: SetAdd (lemon.c:4081)
==11245==    by 0x804B083: FindFirstSets (lemon.c:649)
==11245==    by 0x8051919: main (lemon.c:1464)
==11245==  Address 0x416980F is 0 bytes after a block of size 7 alloc'd
==11245==    at 0x401F396: malloc (in /usr/lib/valgrind/x86-linux/vgpreload_memcheck.so)
==11245==    by 0x804AEF3: SetNew (lemon.c:4057)
==11245==    by 0x804AF7D: FindFirstSets (lemon.c:617)
==11245==    by 0x8051919: main (lemon.c:1464)
==11245==
==11245== Invalid read of size 4
==11245==    at 0x8048BF4: actioncmp (lemon.c:365)
==11245==    by 0x8048DBC: merge (lemon.c:1577)
==11245==    by 0x8048E6D: msort (lemon.c:1623)
==11245==    by 0x80512C1: FindActions (lemon.c:966)
==11245==    by 0x805194E: main (lemon.c:1479)
==11245==  Address 0x416B298 is 4 bytes after a block of size 36 alloc'd
==11245==    at 0x401F396: malloc (in /usr/lib/valgrind/x86-linux/vgpreload_memcheck.so)
==11245==    by 0x804ABF2: State_new (lemon.c:4503)
==11245==    by 0x8051015: getstate (lemon.c:759)
==11245==    by 0x805113A: buildshifts (lemon.c:829)
==11245==    by 0x805104E: getstate (lemon.c:766)
==11245==    by 0x805113A: buildshifts (lemon.c:829)
==11245==    by 0x805104E: getstate (lemon.c:766)
==11245==    by 0x805113A: buildshifts (lemon.c:829)
==11245==    by 0x805104E: getstate (lemon.c:766)
==11245==    by 0x805113A: buildshifts (lemon.c:829)
==11245==    by 0x805104E: getstate (lemon.c:766)
==11245==    by 0x805113A: buildshifts (lemon.c:829)
==11245==
2007-Dec-12 18:00:29 by anonymous:
Another grammar with parsing conflicts, different valgrind error:

  article ::= blocks FIN.
  blocks ::= block.
  blocks ::= blocks block.
  block ::= NEWLINE.
  block ::= stanza.
  block ::= heading.
  block ::= stanza heading.
  heading ::= HEADING_START text HEADING_END.
  stanza ::= text.
  stanza ::= stanza NEWLINE text.
  text ::= textpiece.
  text ::= text textpiece.
  textpiece ::= TEXT.
  textpiece ::= LINK.

==11727== Conditional jump or move depends on uninitialised value(s)
==11727==    at 0x804ADBB: CompressTables (lemon.c:3947)
==11727==    by 0x8051A2C: main (lemon.c:1482)

3925 void CompressTables(lemp)
3926 struct lemon *lemp;
3927 {
3928   struct state *stp;
3929   struct action *ap, *ap2;
3930   struct rule *rp, *rp2, *rbest;
3931   int nbest, n;
3932   int i;
3933   int usesWildcard;
3934
3935   for(i=0; i<lemp->nstate; i++){
3936     stp = lemp->sorted[i];
3937     nbest = 0;
3938     rbest = 0;
3939     usesWildcard = 0;
3940
3941     for(ap=stp->ap; ap; ap=ap->next){
3942       if( ap->type==SHIFT && ap->sp==lemp->wildcard ){
3943         usesWildcard = 1;
3944       }
3945       if( ap->type!=REDUCE ) continue;
3946       rp = ap->x.rp;
3947       if( rp->lhsStart ) continue;

2007-Dec-12 18:14:11 by anonymous:
"./lemon src/parse.y" also produces:

  ==12904== Conditional jump or move depends on uninitialised value(s)
  ==12904==    at 0x80523AF: CompressTables (lemon.c:3947)
  ==12904==    by 0x804B015: main (lemon.c:1482)


2007-Dec-12 19:00:41 by anonymous:
This appears to remedy the Conditional jump error for parse.y. The read/write errors in first grammar remains.

--- tool/lemon.c        5 Oct 2007 16:16:36 -0000       1.51
+++ tool/lemon.c        12 Dec 2007 19:02:37 -0000
@@ -2100,6 +2100,7 @@ to follow the previous rule.");
           psp->prevrule = 0;
        }else{
           int i;
+          rp->lhsStart = 0;
           rp->ruleline = psp->tokenlineno;
           rp->rhs = (struct symbol**)&rp[1];
           rp->rhsalias = (char**)&(rp->rhs[psp->nrhs]);

2007-Dec-13 23:44:45 by anonymous:
I originally thought this reading the unitialized memory for rp->lhsStart lemon parser bug was completely inconsequential, but now I see if I apply the following patch and run lemon against parse.y it generates a completely different parse.c file.

If the lemon is built with the following patch and your run parse.y against it, lemon will generate a parse.c that is 275868 bytes in size:

diff -u -3 -p -r1.51 lemon.c
--- tool/lemon.c        5 Oct 2007 16:16:36 -0000       1.51
+++ tool/lemon.c        13 Dec 2007 23:37:39 -0000
@@ -2100,6 +2100,7 @@ to follow the previous rule.");
           psp->prevrule = 0;
        }else{
           int i;
+          rp->lhsStart = 0xffff; // *** change to zero and rerun
           rp->ruleline = psp->tokenlineno;
           rp->rhs = (struct symbol**)&rp[1];
           rp->rhsalias = (char**)&(rp->rhs[psp->nrhs]);
While if the rp->lhsStart value is set to 0, it will produce a completely different parse.c file of 122960 bytes.
 
2834 event closed 2007 Dec anonymous   2007 Dec   2 1 Is Version 3.1.3 not compatible with 3.5.3 ? edit
I created this table "CREATE TABLE CALCULATION (sceneid INTEGER, calcid INTEGER, data BLOB, timestamp INTEGER, PRIMARY KEY (sceneid, calcid))" with the sqlite version 3.1.3. And now I have updated to version 3.5.3 and it is not possible to read the data from table CALCULATION. But if I create the table only with one primary key (Primary key (sceneid)), than it is possible to read the data. Is it a bug in version 3.5.3 oder in 3.1.3? How can I read the data with a double primary key?
2007-Dec-11 18:43:51 by drh:
All versions of SQLite are compatible back through version 3.0.0. If you think otherwise, please provide details and we will investigate.

I generate a database using your schema and SQLite version 3.1.3 and it works just fine with SQLite version 3.5.3. Unable to recreate the problem...

 
2833 doc fixed 2007 Dec anonymous   2007 Dec   5 1 typo on SQLite home page edit
As pointed out by silvestru@molddata.md on the mailing list the phrase "This the homepage for SQLite - ..." should be changed to "This is the homepage for SQLite - ...".
We fixed this yesterday. And the homepage is going to be completely replaced tomorrow morning, at which point the issue will become moot.
 
2832 code fixed 2007 Dec anonymous   2007 Dec   1 1 Data corruption with UPDATE or DELETE clause ... edit
On any UPDATE or DELETE operation, if a side effect causes other rows in the same table to be deleted, a stack leak can occur. In rare circumstances, database corruption can result. Side effect deletes might be caused by an OR REPLACE clause or by triggers.

This bug appears to be in all prior versions of SQLite 3.

The original problem description follows:


My scenario is rather simple - I have products table, and details table. Producs have unique column called code. Then I have transform table to renumber my products table. I used update or replace constraint, and my data got corrupted. Here's example, I will try to rewrite column names into english, sorry if I make some typo here:

sql "create table products (code text primary key, former_code text, typy text, name text, warehouse text)" sql "create table transform (code text, name text, new_code text, former_code text, type text)"

no indices (there are in newer version of my code, but simpler the case the better)

And now sql code, which corrupts my data (contains a bit of REBOL code):

sql trim/lines {
 update or replace products
   set code = (select new_code from transform where transform.code=products.code),
        type = (select type from transform where transform.code=products.code)
 where exists
  (select code from transform where transform.code=products.code)
}
So - I just wonder - I am not fluent in SQL, so I am not sure I should update primary key in products that way? I expect this code would either update codes found for transformation, or it would fail, because such code to translate to would be already present on table, so that would be caught by "or replace" constraint, and such item would be replaced.

So - what you suggest now? Is that a bug? Or am I behaving really badly to my data? :-)

Tried with 3.5.2, 3.5.3, Windows Vista 32bit ....

PS: This text is from my original post to ml - tried to subscribe to ml two times, no luck receiving any message, so posting directly to bugbase. Sorry if the bug is already known ...

2007-Dec-11 14:34:41 by drh:
What makes you think your data is being corrupted, as opposed to merely transformed in exactly the way you have requested? In other words, why do you think this is a bug in SQLite?


2007-Dec-12 08:22:19 by anonymous:
If you can see that instead of product code you get characters you even can't write on your keyboard probably, then I think data is being corrupted :-)

Here's a screenshot:

corrupted data?

Cheers, Petr


2007-Dec-12 08:35:14 by danielk1977:
Can you post a database file and the SQL statement that is causing the corruption?

This is likely an encoding problem. Are you inserting UTF-8 text into the database? And handling the returned data as UTF-8?


2007-Dec-12 08:56:10 by anonymous:
OK, before I post some data, I will try to describe what I did:

In products table I have product numbers starting with "9", which are going to be translated into other numbers:

9xxxx.xx ---> 1xxxx.xx

9xxxx.xxA --> 1xxxx.xxA

9xxxx.xxBZ--> 9xxxx.xx

I import all data from Excel .csv files. Encoding should be OK imo, as data live in one file. However, I noticed "bug" in my logic. Simply put, to make my life easier, I imported 9xxxx.xxBZ codes into my transform table, thinking that when those codes are at the end of the table, those are OK. But - my update clause (posted in my first request), does not "join data" that safe way, so actually what happens is - if the clause reaches 9xxxx.xxBZ code, it updates it to 9xxxx.xx code, but this code is then once again translated to 1xxxx.xx later in the process. If I don't import 9xxxx.xxBZ codes to the end of the table, my data remain being OK.

Where can I send my database? I don't want it being public. Thanks a lot ...

Petr


2007-Dec-12 10:25:21 by anonymous:
OK, here is a reduced recordset without sensitive data.

corruption data example

Just open the database and apply query as stored in query.txt


2007-Dec-12 11:11:11 by anonymous:
I can confirm the database corruption as provided in the sample. Before running the SQL in query.txt, pragma integrity_check returns 'ok', afterwards it returns these errors:

  rowid 156 missing from index sqlite_autoindex_products_1
  rowid 159 missing from index sqlite_autoindex_products_1
  rowid 160 missing from index sqlite_autoindex_products_1
  rowid 162 missing from index sqlite_autoindex_products_1
  rowid 163 missing from index sqlite_autoindex_products_1

Tested with CVS [4603] .


2007-Dec-12 15:41:33 by anonymous:
Regarding the test case in Check-in [4614] , it produces the same expected result for both 3.5.3 and the latest CVS. No sign of database corruption or valgrind error for 3.5.3 on Linux. Are you certain this isolates the problem?


2007-Dec-12 15:56:02 by drh:
tkt2832.test provokes the problem if you use the OP_StackDepth opcode from check-in [4612] . The change in [4612] , together with additional enhancements that will go in as part of this fix, is designed to head off this kind of bug in the future by making the VDBE very unforgiving to the kinds of code generator errors that caused this problem.
 
2831 new active 2007 Dec anonymous   2007 Dec   3 4 alter view edit
View can't be used after ALTER RENAME TO:

SQLite version 3.5.3
Enter ".help" for istructions
sqlite> create table t(a);
sqlite> create view v1 as select * from t;
sqlite> alter table v1 rename to v2;
sqlite> select * from v2;
SQL error: no such table: v2
sqlite> select * from v1;
SQL error: no such table: v1
sqlite> .schema
CREATE TABLE t(a);
CREATE VIEW v1 as select * from t;
sqlite> select * from sqlite_master;
table|t|t|2|CREATE TABLE t(a)
view|v1|v1|0|CREATE VIEW v1 as select * from t
This is a feature request, not a bug.


2007-Dec-11 18:40:17 by anonymous:
Notice that alter table doesn't return an error. After the command neither v1 nor v2 can be used.


2007-Dec-13 08:18:16 by danielk1977:
[4623] improves the situation by returning an error when the user attempts to rename a view.

One reason this feature (renaming views) is not a high priority is because a view can be dropped and recreated with a different name efficiently. This was not the case with tables.

 
2830 code closed 2007 Dec anonymous   2007 Dec   5 5 tkt2822.test refers to tkt2820 ? edit
typo?

  sqlite/test/tkt2822.test -> 1.1

  do_test tkt2820-1.1 {
  do_test tkt2820-1.3 {
  do_test tkt2820-1.3 {
Thanks.
 
2829 code fixed 2007 Dec anonymous   2007 Dec   1 1 Win32: temp files left over in temp area edit
Seems that temp tables generated on the fly for large group by or order by queries do not have their files removed on CloseHandle. This is because the CreateFile is not done with the right flag. Result is that a lot of files are left over in temp directory, even when queries are run properly.

This patch seems to fix it (added: SQLITE_OPEN_DELETEONCLOSE):

   if( flags & (SQLITE_OPEN_TEMP_DB | SQLITE_OPEN_TEMP_JOURNAL
-                    | SQLITE_OPEN_SUBJOURNAL) ){
+                    | SQLITE_OPEN_SUBJOURNAL | SQLITE_OPEN_DELETEONCLOSE) ){
2007-Dec-10 19:54:25 by drh:
Yikes! I just checked and I have 174260 SQLite temp files in my /var/tmp folder! And cleaning up such a mess is challenging. You cannot do "rm etilqs_*" because the argument list is too long. Even "ls etilqs_*" doesn't work. Here is what I'm doing:

    ls -U | grep etliqs_ | while read i; do rm $i; done

That will have to run for 20 minutes or so to clear out all of the dead files.

This problem is serious enough to warrant a new release once we get it fixed.


2007-Dec-10 20:06:32 by anonymous:

I don't see these temp file on UNIX - which OS is affected by this bug and under what circumstance?


2007-Dec-10 20:50:20 by drh:
Hmmm. I finally got my /var/tmp folder cleaned up. But now when I rerun the regression suite, no new temp files appear. Perhaps those 174260 temp files came from some older bug that has now been fixed.


2007-Dec-10 21:24:27 by anonymous:
Did you by chance catch a glimpse of the date of the most recent undeleted tmp files on UNIX?


2007-Dec-11 03:31:38 by drh:
Sadly, I did not observe the mtimes on any of the files in /var/tmp. Recall that ls was not working very well for me (probably due to the large number of files). I was having to use the -U open (to prevent sorting) in order to get ls to work at all. And I did not have the presence of mind to add a -l to that. Oh well....


2007-Dec-11 05:26:45 by anonymous:
find is your friend:

  find /var/tmp -type f -iname "etliqs_*" -print0 | xargs -r0 rm

(yes, you can simplify this a whole lot if you make some assumptions but why bother be unsafe?)

 
2828 code closed 2007 Dec anonymous   2007 Dec   1 2 SQLITE_OMIT_SUBQUERY compile error: sqlite3CodeSubselect() missing. edit
The new sqlite3FindInIndex() calls sqlite3CodeSubselect(), but this is kept out if SQLITE_OMIT_SUBQUERY is defined.

As it stands now, SQLite does not compiled with SQLITE_OMIT_SUBQUERY defined.

 
2827 code fixed 2007 Dec anonymous   2007 Dec   1 1 make test: vacuum2.test did not close all files: 1 edit
Latest unmodified CVS source tree. Run "make test" to see the error.

vacuum2-1.1... Ok
vacuum2-2.1... Ok
vacuum2-2.1... Ok
vacuum2-3.1... Ok
vacuum2-3.2... Ok
vacuum2-3.3... Ok
vacuum2-3.4... Ok
vacuum2-3.5... Ok
vacuum2-3.6... Ok
vacuum2-3.7... Ok
vacuum2-3.13... Ok
vacuum2-3.14... Ok
vacuum2-3.15... Ok
vacuum2-3.16... Ok
vacuum2-3.17... Ok
vacuum2.test did not close all files: 1

...

1 errors out of 38209 tests
Failures on these tests: vacuum2.test
All memory allocations freed - no leaks
Maximum memory usage: 14164890 bytes
make: *** [test] Error 1
Note: this does not happen if you only run vacuum2.test directory:

$ ./testfixture test/vacuum2.test
vacuum2-1.1... Ok
vacuum2-2.1... Ok
vacuum2-2.1... Ok
vacuum2-3.1... Ok
vacuum2-3.2... Ok
vacuum2-3.3... Ok
vacuum2-3.4... Ok
vacuum2-3.5... Ok
vacuum2-3.6... Ok
vacuum2-3.7... Ok
vacuum2-3.13... Ok
vacuum2-3.14... Ok
vacuum2-3.15... Ok
vacuum2-3.16... Ok
vacuum2-3.17... Ok
0 errors out of 16 tests
All memory allocations freed - no leaks
Maximum memory usage: 85541 bytes
 
2826 build fixed 2007 Dec anonymous   2007 Dec   4 4 test_thread.c won't build if !SQLITE_THREADSAFE || !TCL_THREADS edit
Trying a quick build of the test sources I discovered that test_thread.c won't build the null-test, since it requires tcl.h for a Tcl typedef for the function signature but only includes tcl.h if it thinks it's doing something useful. The following patch fixes the problem:

diff -cr unpack-orig/sqlite-3.5.3/src/test_thread.c sqlite-3.5.3/src/test_thread.c
*** unpack-orig/sqlite-3.5.3/src/test_thread.c	Mon Sep 10 11:53:02 2007
--- sqlite-3.5.3/src/test_thread.c	Thu Dec  6 14:26:16 2007
***************
*** 18,27 ****
  */

  #include "sqliteInt.h"

  #if SQLITE_THREADSAFE && defined(TCL_THREADS)

- #include <tcl.h>
  #include <errno.h>
  #include <unistd.h>

--- 18,27 ----
  */

  #include "sqliteInt.h"
+ #include <tcl.h>

  #if SQLITE_THREADSAFE && defined(TCL_THREADS)

  #include <errno.h>
  #include <unistd.h>
 
2825 code active 2007 Dec anonymous   2007 Dec   3 3 FormatMessage (win32) should use extra flag and convert from Unicode edit
The call to FormatMessageA in the win32 source code needs to have the flags changed from: FORMAT_MESSAGE_FROM_SYSTEM to FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS

This ensures that any system messages that expect arguments do not try to grab the argument from some random memory location.

ref: http://blogs.msdn.com/oldnewthing/archive/2007/11/28/6564257.aspx

2007-Dec-06 14:07:53 by anonymous:
I also noticed that the result is NOT converted to UTF-8.

FormatMessageA returns the text in the local ANSI codepage. FormatMessageW should be used on NT systems, and either result should be converted to the SQLite UTF-8 default.


2007-Dec-11 00:34:37 by anonymous:
to simplify what is meant even more...

http://www.sqlite.org/cvstrac/fileview?f=sqlite/src/os_win.c&v=1.118

Search for FormatMessageA (only 1 instance)

  -    FORMAT_MESSAGE_FROM_SYSTEM,
  +    FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS,

No breakage, ensures that no crashes with some messages (e.g. filesystem errors). The encoding issue should be addressed separately.


2007-Dec-11 01:27:07 by anonymous:
The function should be changed to the following to correctly handle the conversion from Unicode/MBCS.

static void winDlError(sqlite3_vfs *pVfs, int nBuf, char *zBufOut){
  int error = GetLastError();
#if OS_WINCE
  if( error>0x7FFFFFF ){
    sqlite3_snprintf(nBuf, zBufOut, "OsError 0x%x", error);
  }else{
    sqlite3_snprintf(nBuf, zBufOut, "OsError %d", error);
  }
#else
  if( isNT() ){
    LPWSTR zWinTemp = NULL;
    DWORD dwLen = FormatMessageW(
      FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS,
      NULL,
      error,
      0,
      (LPWSTR) &zWinTemp,
      0,
      0
    );
    if (dwLen > 0) {
      char * zOut = unicodeToUtf8(zWinTemp);
      LocalFree(zWinTemp);
      sqlite3_snprintf(nBuf, zBufOut, "%s", zOut);
      free(zOut);
    }
  }else{
    LPSTR zWinTemp = NULL;
    DWORD dwLen = FormatMessageA(
      FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS,
      NULL,
      error,
      0,
      (LPSTR) &zWinTemp,
      0,
      0
    );
    if (dwLen > 0) {
      char * zOut = mbcsToUtf8(zWinTemp);
      LocalFree(zWinTemp);
      sqlite3_snprintf(nBuf, zBufOut, "%s", zOut);
      free(zOut);
    }
  }
#endif
}
 
2824 code closed 2007 Dec anonymous   2007 Dec   1 1 enable shared cache problem edit
System info: -Celeron M. -Windows Xp. -Single process. -No thread.

Code:

int WINAPI WinMain(HINSTANCE h1, HINSTANCE h2, LPSTR lpCMD, int nSCMD) {

sqlite3* db; sqlite3_stmt * stm;

sqlite3_enable_shared_cache(1); sqlite3_open("db", &db); sqlite3_prepare_v2(db, "ATTACH DATABASE db2 AS d", 24, &stm, 0); sqlite3_step(stm); }

The problem:

the sqlite3_step(stm) execute for an infinite time.

MA

2007-Dec-06 07:56:12 by danielk1977:
I just tried the following program with the cvs version:

  #include "sqlite3.h"

  int main (int argc, char **argv){
    sqlite3* db;
    sqlite3_stmt *stm;

    sqlite3_enable_shared_cache(1);
    sqlite3_open("db", &db);
    sqlite3_prepare_v2(db, "ATTACH DATABASE db2 AS d", 24, &stm, 0);
    sqlite3_step(stm);

    return 0;
  }

And the sqlite3_step() did not block for any amount of time. Can you confirm that the code above produces the bug in your environment?

Thanks.


2007-Dec-07 13:59:02 by anonymous:
I have ran more test but the problem is never arised. I think it was generated from code or configuration properties from my system, outside sqlite.

MA

 
2823 code closed 2007 Dec anonymous   2007 Dec   1 1 query not return values edit
I have a database with this description:

CREATE TABLE [coletores] (
  [coletor] INTEGER NOT NULL ON CONFLICT ABORT PRIMARY KEY,
  [senha] NVARCHAR2(20) NOT NULL ON CONFLICT ABORT);
CREATE UNIQUE INDEX [SenhaUnica] ON [coletores] ([senha]);
and contain this data:

coletor   |   senha
   1         43043553
When i run the query: "select coletor, senha from coletores where senha = '43043553'"

query not return values.

Works when I try it.

Perhaps you have some whitespace on one side of the '43043553' value in the database. Whitespace is significant to SQLite.

 
2822 code fixed 2007 Dec anonymous   2007 Dec   3 3 ORDER BY term number 1 does not match any result column in select.c edit
The original problem description is preserved below. This text is summary of what has changed. After this change the algorithm for matching ORDER BY terms is as follows:

  1. If an ORDER BY term is a constant integer k then sort by the k-th column of the result set.

  2. If an ORDER BY term is a simple identifer (like "x", not "x.y" and not "x.y.z") and if there if the k-th column uses that same identifer as an AS alias, the sort by the k-th column.

  3. Otherwise, evaluate the expression which is in the ORDER BY term and sort by the resulting value.

For a compound query, the expression in step (3) must exactly match one of the result columns. Also, the three steps are attempted first on the left-most SELECT. If there is a match, the process stops there. If no match is found, the next SELECT to the right is tried. This repeats as necessary until a match is found or until you run out of SELECT statement in which case there is an error.

This algorithm differs from all prior versions of SQLite (1.0.0 through 3.5.3) by the addition of step (2). Adding step (2) brings SQLite much closer to the SQL standard. I believe that SQLite is now a superset of the SQL standard. In a compound query, SQL only looks at the left-most SELECT and does not fail over to SELECT statements to the right looking for a match. The difference in SQLite can be considered an extension.

The revised algorithm is mostly compatible with the way SQLite has always operated before. But there are a few obscure corner cases where there is a difference. An example of the difference is the following:

    CREATE TABLE a(x,y);
    INSERT INTO a VALUES(1,8);
    INSERT INTO a VALUES(9,2);

    SELECT x AS y FROM a ORDER BY y;

In older versions of SQLite, the SELECT statement above would return 9, 1 since the ORDER BY term evaluated to the expression a.y by rule (3) In the next release, because of the addition of rule (2) above, the result will be 1, 9.


Original Problem Description

Starting from version 3.4.2 I receive errors with queries like:

  SELECT a.field FROM a UNION ALL SELECT b.field FROM b ORDER BY a.field;

or even

  SELECT a.field FROM a UNION ALL SELECT a.field FROM a ORDER BY a.field;

error is:

  ORDER BY term number 1 does not match any result column

Tables are created by:

  CREATE TABLE a (field);
  CREATE TABLE b (field);

Please note that the above queries worked fine with sqlite 3.2.x or 3.3.x.

it's odd that this one doesn't work as well:

  create table t1(a);
  create table t2(b);

  select t1.a from t1 union all select t2.b from t2 order by a;

  SQL error: ORDER BY term number 1 does not match any result column
2007-Dec-04 04:35:48 by anonymous:
Analysis from mailing list: http://www.mail-archive.com/sqlite-users%40sqlite.org/msg29701.html

It appears as though the /src/select.c (Line1499) changed from:

   if( iCol<0 && mustComplete ){

to:

    }else if( mustComplete ){

in version 1.336 of this file - http://www.sqlite.org/cvstrac/filediff?f=sqlite/src/select.c&v1=1.335&v2=1.336

And this change results in this bug.


2007-Dec-04 04:38:57 by anonymous:
Related to Check-in [3842] : Match ORDER BY terms to columns using names in compound queries. Make sure this works for subqueries, especially in the right-hand side of an IN operator. Ticket #2296.


2007-Dec-04 16:27:41 by anonymous:
Other databases produce results for the following that differ from sqlite:

  create table x1(a INT, b INT, c INT);
  insert into x1 values(1, 2, 3);
  create table x2(a INT, b INT, c INT);
  insert into x2 values(9, 0, 4);
  SELECT x1.b, a FROM x1 UNION SELECT a, b FROM x2 ORDER BY b;

SQLite version 3.5.3

  sqlite> SELECT x1.b, a FROM x1 UNION SELECT a, b FROM x2 ORDER BY b;
  x1.b|a
  9|0
  2|1

versus:

  mysql> SELECT x1.b, a FROM x1 UNION SELECT a, b FROM x2 ORDER BY b;
  +------+------+
  | b    | a    |
  +------+------+
  |    2 |    1 |
  |    9 |    0 |
  +------+------+

Also, for this SQL:

  create table x1(a INT, b INT, c INT);
  insert into x1 values(1, 2, 3);
  create table g2(x INT, y INT, z INT);
  insert into g2 values(9, 0, 4);

sqlite 3.5.3 produces:

  sqlite> SELECT x1.b, a FROM x1 UNION SELECT x, y FROM g2 ORDER BY y;
  x1.b|a
  9|0
  2|1

compared to an error on other databases that only use the first SELECT to determine ORDER BY/GROUP BY column names:

  mysql> SELECT x1.b, a FROM x1 UNION SELECT x, y FROM g2 ORDER BY y;
  ERROR 1054 (42S22): Unknown column 'y' in 'order clause'


2007-Dec-05 05:14:38 by anonymous:
Another related issue in 3.5.3+ CVS:

  CREATE TABLE foo(b);

These are fine:

  SELECT 3 b union select b from foo order by b;
  3

  SELECT b from foo union SELECT 3 b order by b;
  3

These have issues:

  SELECT b from foo union SELECT 3 b order by +b;
  SQL error: ORDER BY term number 1 does not match any result column

  SELECT 3 b union select b from foo order by +b;
  SQL error: ORDER BY term number 1 does not match any result column


2007-Dec-05 05:34:23 by anonymous:
A problem with expressions in an ORDER BY of a compound statement:

  CREATE TABLE t1(b);
  INSERT INTO t1 VALUES(-3);
  INSERT INTO t1 VALUES(2);

  -- works, of course
  select b from t1 order by b*b;
  2
  -3

  -- expression does not work in compound SELECT
  select b from t1 union all select b from t1 order by b*b;
  SQL error: ORDER BY term number 1 does not match any result column

  -- an alias does not help

  select b as b from t1 union all select b from t1 order by b*b;
  SQL error: ORDER BY term number 1 does not match any result column

  select b as b from t1 union all select b as b from t1 order by b*b;
  SQL error: ORDER BY term number 1 does not match any result column


2007-Dec-08 04:08:53 by anonymous:
The column heading names in the result set of a compound query are not consistent with regular queries:

SQLite version 3.5.3
Enter ".help" for instructions
sqlite> create table foo(a);
sqlite> insert into foo values(1);
sqlite> .header on

sqlite> select foo.a from foo;
a
1

sqlite> select foo.a from foo union all select foo.a from foo;
a
1
1

sqlite> select foo.a from foo union all select foo.a from foo order by 1;
foo.a    <=====
1
1

sqlite> select foo.a from foo union all select foo.a from foo group by 1;
a
1
1

2007-Dec-08 17:43:29 by anonymous:
Possible fix:

http://www.mail-archive.com/sqlite-users%40sqlite.org/msg29790.html

http://marc.info/?l=sqlite-users&m=119708950929249&q=p3


2007-Dec-10 19:35:33 by anonymous:
What should the following return?

  create table t1(a INT, b INT, c INT);
  insert into t1 values(1, 2, 4);
  insert into t1 values(2, -1000, 5);
  select a, a+b AS c from t1 order by c;

sqlite 3.5.3:

  a|c
  1|3
  2|-998

mysql and postgres:

   a |  c
  ---+------
   2 | -998
   1 |    3


2007-Dec-13 08:13:24 by anonymous:
It's unfortunate that the expression support in compound ORDER BY which existed briefly in CVS for a couple of days was removed:

sqlite/test/tkt2822.test 1.2 -> 1.3

  -    SELECT a+1, b+1 FROM t1 UNION ALL SELECT a, c FROM t2 ORDER BY a+1;
  +    SELECT a, b, c FROM t1 UNION ALL SELECT a AS x, b, c FROM t2 ORDER BY x;

It was a useful extension.


2007-Dec-13 16:05:10 by drh:
The support is still there. We just removed the test case. I'm not sure why. Probably we should put it back...


2007-Dec-13 16:22:42 by anonymous:
Are you sure?

From latest CVS:

  sqlite> select 3 as A, 6 as B union select 1 as B, 5 as A order by A;
  1|5
  3|6

  sqlite> select 3 as A, 6 as B union select 1 as B, 5 as A order by +A;
  SQL error: 1st ORDER BY term does not match any column in the result set

  sqlite> select 3 as A, 6 as B union select 1 as B, 5 as A order by A+1;
  SQL error: 1st ORDER BY term does not match any column in the result set

  sqlite> select 3 as A, 6 as B union select 1 as B, 5 as A order by 9*A-A*A;
  SQL error: 1st ORDER BY term does not match any column in the result set

  select * from (select 3 as A, 6 as B union select 1 as B, 5 as A) order by 9*A-A*A;
  1|5
  3|6


2007-Dec-13 17:01:22 by drh:
In a compound SELECT, the ORDER BY terms must match an expression in the result set exactly. "A" does not exactly match "+A". But "a+1" does exactly match "a+1". To keep things relatively simple, we do not allow equivalent expressions. "1+a" does not match "a+1". The expressions must be identical (from the point of view of the parser - differences in whitespace don't matter of course since that is all stripped out by the tokenizer.)

In a simple SELECT, if an ORDER BY term does not exactly match a result set expression, we can add an additional "virtual" result column to contain the ORDER BY term. But we cannot do that in a compound SELECT because we would have to add such "virtual" columns to all SELECTs in the compound and the ORDER BY expression typically only makes sense for one of them.


2007-Dec-13 17:28:33 by anonymous:
Some popular databases do in fact add "virtual" columns you describe and support ORDER BY expressions not explicitly mentioned in any compound SELECT.

  mysql> select 3 as A, 6 as B union select 1 as B, 5 as A order by -A*A;
  +---+---+
  | A | B |
  +---+---+
  | 3 | 6 |
  | 1 | 5 |
  +---+---+

It's a useful extension, but not a show stopper. One could manually do to the transformation identified above to make the compound query a subquery and move the ORDER BY to the parent. I don't know if it is any less efficient.

 
2821 new active 2007 Dec anonymous   2007 Dec   3 4 hashtable indicies edit
It would be nice to implement non btree indices. I.e. CREATE INDEX ON table(rowid) AS HASH. Using a hashtable's O(1) properties, you could use the index for very quick lookups when one result is expected. This does have the tradeoff that a hashtable index has no ordering properties (can not be used for sorts or non-equality searching). However, it would be a huge win when you have 250,000 rowids in memory, and you want to go fetch another column in the database for each one of those rowids (SELECT * FROM table WHERE rowid=?).
2007-Dec-03 21:58:01 by anonymous:
For 250,000 rows I doubt you would see that much of an improvement (try it.)

You'll almost certainly find log_n is going to be fairly fast (especially for large n.)

I personally would prefer some sort of 'virtual' index though, that could be a hash or actually from a user-supplied function so that I can index large blobs by some function (i.e. a hash).

And yes, this would be an incompatible file-format change and it's not clear how to update an index when the function isn't loaded (i.e. db reopened with that function.) Perhaps mark the index as 'stale' and ignore it until the function loads then you can do the updates.

Of course this starts to get quite complicated.


2007-Dec-03 22:12:17 by anonymous:
Everything in sqlite depends on btree indexes. You're talking a major rewrite if you support hash-based or other indexing.
 
2820 code fixed 2007 Dec anonymous   2007 Dec   1 1 rollback doesn't work edit
Code:

#include <stdio.h>
#include "sqlite3.h"

sqlite3 *db;
static int callback(void *NotUsed, int argc, char **argv, char **azColName){
  sqlite3_exec(db, "DROP TABLE t", 0, 0, 0);
  printf("%s\n", sqlite3_errmsg(db));
  return 0;
}

int main() {
  sqlite3_open("test.db", &db);
  sqlite3_exec(db, "CREATE TABLE IF NOT EXISTS t(a INT)", 0, 0, 0);
  sqlite3_exec(db, "select name, type from sqlite_master", &callback, 0, 0);
  sqlite3_close(db);
  return 0;
}
prints:

database table is locked

In sqlite rolback is called (I'v checked in older version), but in test.db there is no table 't'.

Tested on win2k.

2007-Dec-03 16:31:50 by anonymous:
What is the desired outcome?


2007-Dec-03 18:28:38 by anonymous:
No error. Or if error is reported during delete, nothing should be deleted.
 
2819 code closed 2007 Dec anonymous   2007 Dec   2 1 Blocking and crashing edit
First it was blocking when using perl lib DBI, i tried to update the packages and finally reinstall them and now i get this:

mush@server:~$ /usr/bin/sqlite3 ./Bureau/lexfix/lexFix/test.dat
SQLite version 3.3.10
Enter ".help" for instructions
sqlite> .schema
/usr/bin/sqlite3: symbol lookup error: /usr/bin/sqlite3: undefined symbol: sqlite3_enable_load_extension
mush@server:~$ /usr/bin/sqlite3 ./Bureau/lexfix/lexFix/test.dat
/usr/bin/sqlite3: symbol lookup error: /usr/bin/sqlite3: undefined symbol: sqlite3_enable_load_extension
I think this is a support request, not a bug report. Please take your comments to the SQLite mailing list.
 
2818 code closed 2007 Dec anonymous   2007 Dec   2 4 Comparison of 8 bytes integer numbers is truncated edit
sqlite> select -9223372036854775808>=-9223372036854775808; 1 sqlite> select -9223372036854775809>=-9223372036854775808; 1 sqlite> select -9223372036854875808>=-9223372036854775808; 0
   sqlite>  select -9223372036854775808>=-9223372036854775808;
   1

This is the correct answer.

   sqlite>  select -9223372036854775809>=-9223372036854775808;
   1

-9223372036854775809 is not an 8-byte integer. The smallest possible 8-byte integer is -9223372036854775808. The -9..809 value is a 64-bit IEEE floating point number: -9.22337203685488e+18. In order to perform the comparison, SQLite converts the number on the right to floating point value. Due to the limited precision of floating point numbers, both values are the same and so the values compare equal to one another. Hence the result shown above is correct.

   sqlite>  select -9223372036854875808>=-9223372036854775808;
   0

This is the correct answer.

No bugs seen here.

 
2817 code closed 2007 Dec danielk1977   2007 Dec danielk1977 1 1 Temp-tables with the same name as database tables can cause corruption edit
Suspect a problem in the CREATE INDEX statement here:

  CREATE TEMP TABLE abc(a, b, c);
  CREATE TABLE main.abc(a, b, c);
  CREATE INDEX main.abc_i ON abc(a, b, c);

The attached script demonstrates the problem.

 
2816 code closed 2007 Dec anonymous   2007 Dec drh 4 3 .timer missing from CLI edit
.timer not valid for 3.5.2/3.5.3
It works when I try it.


2007-Dec-04 14:48:14 by anonymous:
No .timer support on Windows.


2007-Dec-04 15:31:30 by drh:
That is correct - there is no timer support on windows. But on the other hand, we never claimed there was. The ".timer" command does not show up in the help menu on windows.
 
2815 doc fixed 2007 Dec anonymous   2007 Dec   2 3 sqlite3_blob_close closes blob even when reporting error! edit
As part of my testing, I open a blob read only and then do some writing to it. On calling sqlite3_blob_close, I get back error 8 (SQLITE_READONLY), but the blob is closed anyway.

Normal practise for file like apis is that if close returns an error, the handle remains open. Please either make SQLite match other apis, or update the documentation to mention that the blob handle is always closed no matter what the return.

It is probably also a good to mention that the return of close must be checked since that is where errors are returned (ie view it as a commit).

The Linux close(2) man page contains advise that translates to sqlite3_blob_close as well http://linux.die.net/man/2/close

 
2814 code active 2007 Nov anonymous   2007 Dec   3 3 _XOPEN_SOURCE again edit
Ideally setting _XOPEN_SOURCE should be an opt-in detected by configure, rather than a hardcoded opt-out as it is now. I find you create more problems in setting it than just leaving it out on modern platforms.

Can you please give users the option of not defining _XOPEN_SOURCE at all?

+#ifndef SQLITE_DONT_DEFINE_XOPEN_SOURCE
 #if !defined(_XOPEN_SOURCE) && !defined(__DARWIN__) && SQLITE_THREADSAFE
 #  define _XOPEN_SOURCE 500  /* Needed to enable pthread recursive mutexes */
 #endif
+#endif
2007-Dec-01 09:23:15 by anonymous:
Also when using Python, it sets _XOPEN_SOURCE to 600. No idea what the 500 vs 600 difference is about.


2007-Dec-01 15:58:28 by anonymous:
I've used a couple of different Linux OSes and _XOPEN_SOURCE is not needed. Maybe it's for OSes more than 5 years old. Recursive mutexes are pretty much standard these days since the popularity of Java which uses them extensively.


2007-Dec-01 17:21:05 by drh:
See also tickets #2673, #2681, and #2741.


2007-Dec-02 02:08:26 by anonymous:
On Linux, PTHREAD_MUTEX_RECURSIVE is the same as PTHREAD_MUTEX_RECURSIVE_NP:

    PTHREAD_MUTEX_RECURSIVE = PTHREAD_MUTEX_RECURSIVE_NP,

Since PTHREAD_MUTEX_RECURSIVE_NP is always available, you could avoid defining _XOPEN_SOURCE and use this code instead:

-        pthread_mutexattr_settype(&recursiveAttr, PTHREAD_MUTEX_RECURSIVE);
+        pthread_mutexattr_settype(&recursiveAttr,
+#ifdef linux
+          PTHREAD_MUTEX_RECURSIVE_NP
+#else
+          PTHREAD_MUTEX_RECURSIVE
+#endif
+        );

2007-Dec-02 02:17:22 by anonymous:
A quick google search reveals how various projects deal with this recursive mutex declaration problem (in no particular order):

  • #define _XOPEN_SOURCE 500 and use PTHREAD_MUTEX_RECURSIVE
  • #define _XOPEN_SOURCE 600 and use PTHREAD_MUTEX_RECURSIVE
  • #define _GNU_SOURCE and use PTHREAD_MUTEX_RECURSIVE
  • don't define anything and use PTHREAD_MUTEX_RECURSIVE_NP on linux, and PTHREAD_MUTEX_RECURSIVE elsewhere.

Unfortunately, since PTHREAD_MUTEX_RECURSIVE is an enum on Linux, so you can't use the #ifdef PTHREAD_MUTEX_RECURSIVE compile-time technique.

 
2813 build active 2007 Nov anonymous   2007 Nov   1 1 compile error on Windows CE edit
environment: visual c++ 2005 window ce 6.0 customize sdk sqlite-amalgamation-3_5_3

I get error:

Error 27 error C2040: 'localtime' : 'tm *(const time_t *)' differs in levels of indirection from 'int ()' d:\SubProjects\Sqlite\sqlite3.c 18574

but if I add code in line 7095:

struct tm *__cdecl localtime(const time_t *t);

then Success!

 
2812 new closed 2007 Nov anonymous   2007 Dec   3 4 SQLITE_SECURE_DELETE overwrites with 0, but should use random data edit
Hello Dr. Hipp,

SQLITE_SECURE_DELETE overwrites with 0, but should use random data.

I'm not an expert in this area, but it also sounds like secure delete should involve random data then writing to disk several times in a row.

For the firefox bug that relates to this, please see https://bugzilla.mozilla.org/show_bug.cgi?id=405925

Thanks,

-Seth

2007-Nov-29 18:40:32 by anonymous:
This is a questionable enhancement to security because you cannot ensure where the disk controller will actually write the data on the media. The old fragment may exist elsewhere on the disk. Flash memory cards try to even out where blocks are written to on the media because of the finite number of writes allowed, for example.

It's better to encrypt the whole database file.


2007-Nov-29 21:19:34 by anonymous:
dr. hipp writes:

I don't think writing random bits or writing multiple times will help any. There are too many layers (operating system, file system, disk driver, and disk controller) in between SQLite and the oxide for SQLite to really have much control over what is truly stored on media. SQLite might write the same page multiple times, for example, but the OS would likely coalese the writes into a single write. Perhaps we could force multiple writes using fsync() but that would be a huge performance penalty. And even if we did, just because we write to the same offset in some file does not mean that the information is written to the same place on mass storage. Some filesystems (YAFFS) intentionally spread the writes around to different places for the purpose of wear leveling. And even if the operating system doesn't do this, then on-drive disk controller might.

What problem, exactly, are you trying to solve? The existing SQLITE_SECURE_DELETE is adequate to prevent casual snooping. It is not, as you observe, adequate to prevent snooping from major governments. Against a determined adversary, the only sure defense is to physically shred the disk platter.


2007-Nov-29 21:20:33 by anonymous:
in https://bugzilla.mozilla.org/show_bug.cgi?id=405925#c4, jesse writes:

I don't think it's worth the effort and perf hit to attempt to protect Firefox profile data from forensic data recovery. It's really hard to get right, especially at the app level, and we don't want to give users a false sense of security by trying.

Any given data is likely to appear in many physical locations on the disk due to swapping, journaling, automated backups, and continuous defragmentation [1] .

When you try to overwrite part of a file, you find yourself fighting against optimizations in operating systems and in disks themselves [2] . Data also appears in RAM, which can survive power cycling [2] and is even harder to control.

Zeroing sounds reasonable; it will protect against casual snooping, and depending on the file system, might protect against forensic software. But I don't think we should go beyond zeroing (e.g. multiple overwrites) in an attempt to defeat forensic analysis of the physical hard drive and RAM.

Extremely paranoid users can use operating system features to encrypt an entire hard drive or automatically do multiple overwrites every time data on disk as it is changed.

[1] http://en.wikipedia.org/wiki/Disk_wipe [2] http://en.wikipedia.org/wiki/Data_remanence


2007-Nov-29 21:21:02 by anonymous:
based on these responses, this seems like a "wontfix"

thanks,

-Seth

 
2811 code fixed 2007 Nov anonymous   2007 Nov   5 4 "appropriate" misspelled "approriate" on SQLite Documentation page edit
In the description of the "Appropriate Uses For SQLite" document on the SQLite Documentation page at http://www.sqlite.org/docs.html, "appropriate" is misspelled "approriate".
 
2810 code active 2007 Nov anonymous   2007 Nov   1 1 Unregistered collation problems with simple subselects edit
As discussed on the mailing-list:

Imagine that a SQLite3 database opened in a custom application with a registered a collation sequence named "unknown" has created the following table:

  CREATE TABLE a (b COLLATE unknown);

Now open this table in the default SQLite3 CLI. Up to here, everything works as expected. Simple queries like "SELECT * FROM a;" work fine.

But subselects, in their most basic form and with no sorting or comparisons, result in an error:

  sqlite> INSERT INTO a VALUES ('one');

  sqlite> SELECT * FROM a, (SELECT * FROM a);
  SQL error: no such collation sequence: unknown

  sqlite> SELECT * FROM (SELECT * FROM a);
  SQL error: no such collation sequence: unknown

  sqlite> SELECT *, * FROM a;
  one|one

This is surprising because the collation sequence should not matter to the queries. In fact, the union without the subselect works just fine and without errors.

To demonstrate, here is the explain output of a table with a registered collation sequence. No mention of the collation name here:

  sqlite> CREATE TABLE b (b collate nocase);
  sqlite> EXPLAIN SELECT * FROM b, (SELECT * FROM b);
  0|Goto|0|17|
  1|Integer|0|0|
  2|OpenRead|0|3|
  3|SetNumColumns|0|1|
  4|Integer|0|0|
  5|OpenRead|2|3|
  6|SetNumColumns|2|1|
  7|Rewind|0|14|
  8|Rewind|2|13|
  9|Column|0|0|
  10|Column|2|0|
  11|Callback|2|0|
  12|Next|2|9|
  13|Next|0|8|
  14|Close|0|0|
  15|Close|2|0|
  16|Halt|0|0|
  17|Transaction|0|0|
  18|VerifyCookie|0|4|
  19|TableLock|0|3|b
  20|Goto|0|1|
  21|Noop|0|0|
 
2809 code active 2007 Nov anonymous   2007 Nov   1 1 PRAGMA collation_list shows unregistered collations edit
As presented on the mailing list:

Imagine that a SQLite3 database opened in a custom application with a registered a collation sequence named "unknown" has created the following table:

  CREATE TABLE a (b COLLATE unknown);

Now open this table in the default SQLite3 CLI. Up to here, everything works as expected.

Next issue "PRAGMA collation_list;" and notice that "unknown" lists next to the other registered collations, even though "unknown" is not registered with the default SQLite3 CLI:

  sqlite> PRAGMA collation_list;
  0|unknown
  1|NOCASE
  2|BINARY

Responses from the mailing list indicate that this is not the expected behaviour. "PRAGMA collation_list;" should list registered collations only.

2007-Nov-28 16:12:17 by anonymous:
I don't think this is a bug. If the CLI is not aware of the collation, it should not process the query that makes use of the collation because it would certainly be wrong if it simply ignored the collation. This is not unlike a user-registered SQL function that does not exist in the CLI. I would not expect or want the sqlite3 CLI to ignore the unknown function, nor would I want the CLI to process queries ignoring the custom collation.
 
2808 doc active 2007 Nov anonymous   2007 Nov   4 2 Documentation GIF images take up too much space edit
The GIF format used for the documentation images takes up too much space. I believe that this is in strict contrast to the "small" feature of SQLite.

Converting the GIFs to PNG images saves up to 181 KB (64%) of the documentation storage space:

  Documentation images as is (mostly GIF): 381 KB == 100%
  Documentation images as PNG:             292 KB ==  77%
  Documentation images as PNG 16 colors:   137 KB ==  36%

16 colors are more than plenty -- the diagrams show no visibility degradation. You might even cut it down to 8 colors to save even more ...

 
2807 doc fixed 2007 Nov anonymous   2007 Nov   4 3 Link errors in Documentation edit
There are link errors in these documentation files:

atomiccommit.html:

<p>SQLite allows a single <a href="c3ref/sqlite3.html">database connection</a> to talk to two or more database files simultaneously through the use of the <a href="/lang_attach.html">ATTACH DATABASE</a> command.

---

and <a href="#section_3.10">step 3.10</a>.

---

During recovery at <a href=section_4_2>step 4.2</a> SQLite locates

lang_insert.html:

---

single keyword <a href=lange_replace.html>REPLACE</a> as an

 
2806 code fixed 2007 Nov anonymous   2007 Nov   3 4 Low memory assert in insertCell() edit
The call to allocateSpace() at line 4511 of btree.c can fail, returning zero if the nested call to defragmentPage() fails to allocate its temp buffer. This then triggers the following assert.

    idx = allocateSpace(pPage, sz);
    assert( idx>0 );

For the older versions (ours is derived from 3.3.8) the solution is quite simple:

    idx = allocateSpace(pPage, sz);
    /* Check for memory allocation errors - pre v 3.5.0 */
    if (sqlite3MallocFailed()) return SQLITE_NOMEM;
    assert( idx>0 );

For later versions you probably need something more like:

    if (idx == 0) return SQLITE_NOMEM;

replacing the assert.

This was found running test join-4.4 (from 3.3.8).

2007-Nov-28 15:47:39 by anonymous:
Out of curiosity, which sqlite memory allocator are you using to find these low memory problems? The default malloc one, or the heap-in-an-array one?


2007-Nov-28 17:03:19 by anonymous:
Sorry, it's a proprietary allocator.

But we do try to share the 'benefits'!

 
2805 code fixed 2007 Nov anonymous   2007 Nov   1 1 Checkin [4573] Home grown recursive mutexes edit
Checkin [4573] would only work on uniprocessors, not SMP hardware. You cannot implement recursive mutexes without some sort of memory barrier or atomic "test and set" instruction. The reason being the each SMP CPU has its own memory image that is not synchronized with main memory until there is a memory barrier or lock/spinlock.

Take a look at NPTL or the previous incarnation, LinuxThreads, to see how difficult it is to implement a reliable and efficient recursive mutex on different CPU architectures. It simply can't be done without dropping into architecture-specific assembler.

2007-Nov-28 17:46:31 by anonymous:
Solaris 2.6 was mentioned in the original homegrown recursive mutex implementation. Be aware that some Sparc machines offer cache coherency, while others do not. So it's a run-time hardware-specific thing, not an OS thing or a compile-time thing.

Here is a fairly comprehensive list of cache coherency info (circa 2003) for various CPUs:

http://linux.derkeiler.com/Mailing-Lists/Kernel/2003-09/3187.html

 
2804 code closed 2007 Nov anonymous   2007 Dec   1 1 PRAGMA legacy_file_format=OFF setting lost with VACUUM edit
It seems that the PRAGMA legacy_file_format=OFF and the ability to use DESC indexes is lost after two VACUUMs and reconnects.

$ rm -f foo.db

$ ./sqlite3-3.5.3.bin foo.db
SQLite version 3.5.3
Enter ".help" for instructions
sqlite> PRAGMA legacy_file_format;
1
sqlite> PRAGMA legacy_file_format=OFF;
sqlite> PRAGMA legacy_file_format;
0
sqlite> CREATE TABLE abc(a,b,c);
sqlite> CREATE INDEX abc_i on abc(b desc, c asc, a desc);
sqlite> explain query plan select * from abc order by b desc, c asc, a desc;
0|0|TABLE abc WITH INDEX abc_i ORDER BY
sqlite> vacuum;
sqlite> .q

In the next connection we see that the legacy file format reverted back to 1, but the DESC index is still picked up...

$ ./sqlite3-3.5.3.bin foo.db
SQLite version 3.5.3
Enter ".help" for instructions
sqlite> PRAGMA legacy_file_format;
1
sqlite> explain query plan select * from abc order by b desc, c asc, a desc;
0|0|TABLE abc WITH INDEX abc_i ORDER BY
sqlite> vacuum;
sqlite> .q

But if connected to another time, the DESC index is not picked up...


$ ./sqlite3-3.5.3.bin foo.db
SQLite version 3.5.3
Enter ".help" for instructions
sqlite> PRAGMA legacy_file_format;
1
sqlite> explain query plan select * from abc order by b desc, c asc, a desc;
0|0|TABLE abc
Actually, this works correctly, though the documentation could be clearer. The legacy_file_format pragma does not tell you the file format of the current database. Rather, it tells you what file format will be used when creating new databases (for example, using ATTACH). The behavior shown above is correct. There is no pragma to tell you what the file format of the current database is.

The documentation has been clarified. Two new test cases have been inserted into the test suite to verify that the operation is correct.

 
2803 code fixed 2007 Nov anonymous   2007 Nov   4 3 test: io-4.2.2 fails on 64-bit --- signed extension edit
io-4.2.2 expects 0xfffffff but gets 0xffffffffffffffff

  format 0x%X $res

is this a 64-bit sign-extension quirk wrt to tcl?

2007-Nov-27 20:48:24 by anonymous:
Suggested fix:

  --- a/test/io.test
  +++ b/test/io.test
  @@ -449,6 +449,7 @@ if {$::tcl_platform(platform)=="unix"} {
       set blob [read $fd 4]
       close $fd
       binary scan $blob i res
  +    set res [expr $res & 0xFFFFFFFF]
       format 0x%X $res
     } {0xFFFFFFFF}
   }
 
2802 code closed 2007 Nov anonymous   2007 Dec   1 1 SELECT UNION and EXCEPT seem to have a memory allocation bug edit
It seems that SELECT UNION and EXCEPT operators work wrong. It looks like memory access overflow or integer overflow.

Well, I have a large text file 'text2.text' (~1.5mg), which contains:

20091208
20216510
20149599
20250484
20211910
20199980
........
20183528
20151994
20179184
20077607

182500 lines in total. The meaning of these numbers is a date in YYYYMMDD format. Numbers are random and unsorted at all.

The next file is 'test1.text' is the same as 'test2.text' but without the last 1000 lines. The 'diff' between these files shows the difference in last 1000 lines.

Than I execute script:

@echo off
if exist test.db del test.db
rem
rem Load data into table 'a'
rem It prints 181500
sqlite3 test.db "create table a (i1 text)"
sqlite3 test.db ".import test1.text a"
sqlite3 test.db "select count(*) from a"
rem
rem Load data into table 'b'
rem It prints 182500
sqlite3 test.db "create table b (i1 text)"
sqlite3 test.db ".import test2.text b"
sqlite3 test.db "select count(*) from b"
rem
rem PROBLEM IS HERE
rem Now I am trying to merge the tables 'a' and 'b' via UNION.
rem It should print '182500' but it prints '16384' !!!
sqlite3 test.db "create table c (i1 text)"
sqlite3 test.db "insert into c select * from b union select * from a"
sqlite3 test.db "select count(*) from c"
rem
rem Just a check that EXCEPT also has the same bug.
rem It should print 1000, but it prints '0' !!!
sqlite3 test.db "select count(*) from (select * from c except select * from b)"

I suggest that UNION and EXCEPT has some kind of memory overflow. And also it depends on how strong the numbers in the source text file are shuffled and how wide is the range of the numbers.

I can attach my files 'test1.text' and 'test2.text' if needed.

My system: CPU Intel Core 2 Duo, 1.06GHz, 2GB RAM.

Operating system: Windows XP SP2 (Version 5.1.2600)

2007-Nov-27 23:45:42 by drh:
UNION and EXCEPT both imply DISTINCT. Since you are dealing with dates, it seems unlikely that you really have 182500 distinct dates. That would span 500 years and your sample data seems to all be from the early part of the 21st century. 16384 distinct dates would span 44 years, which seems more reasonable, given your sample data. The fact that the number is a power of two probably results from a poor-quality PRNG used to generate the same data.

You can use UNION ALL to to get the union of two tables without the implied DISTINCT. There is no way to omit the implied DISTINCT from EXCEPT. You will probably need to refactor your query to use the NOT IN operator.


2007-Dec-05 20:10:08 by anonymous:
Actually, in the real data, when I've got this situation on EXCEPT and UNION, there are some extra fields in the database. Except 'date', there were text fields: 'amount', 'owner', 'descr', 'labels' (it's personal cash accouting database). When the test data was generatated, all fields could have duplicate values, but 'amount' field was generated like:

$amount = sprintf( "%d.%02d", rand( 1, 10000 ), rand( 0, 99 ) );

so, it can take 9999999 possible values, which is more than 16384 for sure.

And the behaviour of the EXCEPT was completely the same as I've described the the original message of this bug report. That's why I've decided that the key of the problem is just in the amount of data, not in values, and I've reduced the database structure for the bug report to make it more clear (I left only one field 'date'), but the 'bug' (I mean this strange behaviour of the EXCEPT and UNION) still happens in the reduced database structure (with only one field 'date'). So DISTINCT function will not merge all values of 'amount' field into 16384 unique values.

What do you think?

 
2801 code fixed 2007 Nov anonymous   2007 Nov   5 5 superfluous -lpthread for CLI in publish_osx.sh edit
-lpthread not needed in publish_osx.sh:

  + CFLAGS="-Os -DSQLITE_ENABLE_FTS3=1 -DSQLITE_THREADSAFE=0"
  + NAME=sqlite3-$VERS-osx-x86.bin
  + echo '***** '"COMPILING $NAME..."
  + gcc $CFLAGS -Itsrc sqlite3.c tsrc/shell.c -o $NAME -ldl -lpthread
 
2800 build fixed 2007 Nov anonymous   2007 Nov   5 5 -lpthread not needed for publish.sh for CLI binary edit
-lpthread not needed by publish.sh for CLI binary since not threadsafe build.

  +CFLAGS="-Os -DSQLITE_ENABLE_FTS3=1 -DSQLITE_THREADSAFE=0"
  +echo '***** '"COMPILING sqlite3-$VERS.bin..."
  +gcc $CFLAGS -Itsrc sqlite3.c tsrc/shell.c -o sqlite3 -ldl -lpthread

ldd ./sqlite3-3.5.3.bin

        linux-gate.so.1 =>  (0xb7f1d000)
        libdl.so.2 => /lib/libdl.so.2 (0xb7f05000)
        libpthread.so.0 => /lib/i686/libpthread.so.0 (0xb7ef2000)
        libc.so.6 => /lib/i686/libc.so.6 (0xb7dc5000)
        /lib/ld-linux.so.2 (0xb7f1e000)

Unrelated - why are the shared libs and binary chmod'ed to be not executable?

  chmod 644 sqlite3-$VERS.bin.gz
  chmod 644 tclsqlite3.so
  chmod 644 sqlite3.so
The webserver used by www.sqlite.org treats any file with execute permission as a CGI program and tries to run it. So we have to remove the execution permission bit from files which are intended to be served intact.


2007-Nov-27 19:00:03 by anonymous:
Instead of gzipping the files directly, can you wrap the individual files in a tar.gz instead? This way they could retain the executable permission. UNIX novices may not know about chmod.
 
2799 build fixed 2007 Nov anonymous   2007 Nov   3 3 'configure ' cannot locate VERSION file edit
'configure' diff: 18437c18437 < ALLOWRELEASE="-release `cat VERSION`" --- > ALLOWRELEASE="-release `cat $srcdir/VERSION`"
 
2798 build closed 2007 Nov anonymous   2007 Nov   3 4 Wrong nm used when cross-compiling edit
When cross-compiling, the build platforms nm is used, instead of the one for the target platform.

I included a patch for Makefile.in and configure.ac to start using the detected nm instead of the hardcoded 'nm'.

2007-Nov-27 14:53:06 by drh:
The configure script is a mess and I am disinclined to make it a bigger mess by trying to fix cross-compilation.

A better approch to cross-compiling is to use the amalgamation. Build the amalgamation using:

    make sqlite3.c

Then compile sqlite3.c for your target system using whatever cross-compiler you want.

 
2797 code fixed 2007 Nov anonymous   2007 Nov   3 3 Low memory SEGV in whereClauseClear() edit
The function whereClauseClear() in where.c dereferences 'a' which may be null following an allocation error in whereClauseInsert().

It looks as if this allocation error will also result in leaking the old WhereTerm array addressed through pWC->a (if this was not static).

The correction is probably to re-instate the old value on error before returning from whereClauseInsert(), ie to add the line:

  pWC->a = pOld;

before the return 0.

 
2796 code fixed 2007 Nov anonymous   2007 Nov   3 3 Another low memory SEGV in exprAnalyze() edit
The first statement after declaring local variables in function exprAnalyze() in where.c is:

  if( db->mallocFailed ) return;

Unfortunately initialising some of the local variables can already reference null pointers:

  WhereTerm *pTerm = &pWC->a[idxTerm];
  ExprMaskSet *pMaskSet = pWC->pMaskSet;
  Expr *pExpr = pTerm->pExpr;

If pWC->a is NULL the third of these initialisations causes problems.

This happened here in the two calls:

    whereSplit(&sOr, pExpr, TK_OR);
    exprAnalyzeAll(pSrc, &sOr);

from exprAnalyze(), early in the section beginning:

  /* Attempt to convert OR-connected terms into an IN operator so that
  ** they can make use of indices.  Example:

where the allocation in whereClauseInsert() (called by 5 depths of whereSplit()) failed while processing "SELECT y FROM t9\nWHERE x=(SELECT x FROM t9 WHERE y=1)\n OR x=(SELECT x FROM t9 WHERE y=2)\n OR x=(SELECT x FROM t9 WHERE y=3)\n OR x=(SELECT x FROM t9 WHERE y=4)\n OR x=(SELECT x FROM t9 WHERE y=5".

The fix is to delay initialising the variables until after the check for mallocFailed.

 
2795 code fixed 2007 Nov anonymous   2007 Nov   3 3 Low memory SEGV in exprAnalyze() edit
Function exprAnalyze() in where.c includes the following code:

  /* Add constraints to reduce the search space on a LIKE or GLOB
  ** operator.
  ...
    pStr2 = sqlite3ExprDup(db, pStr1);
    if( pStr2 ){
      assert( pStr2->token.dyn );
      ++*(u8*)&pStr2->token.z[nPattern-1];
    }

A memory allocation error (in sqlite3StrNDup() called from sqlite3ExprDup()) can leave pStr2->token.z as NULL. Dereferencing this to increment an element brings a segment violation.

sqlite3ExprDup() currently has no way of returning an error.

 
2794 code fixed 2007 Nov anonymous   2007 Nov   3 3 Low memory SEGV in isLikeOrGlob() edit
Function isLikeOrGlob() in where.c includes (towards the end of the function) the lines:

  sqlite3DequoteExpr(db, pRight);
  z = (char *)pRight->token.z;
  for(cnt=0; (c=z[cnt])!=0 && c!=wc[0] && c!=wc[1] && c!=wc[2]; cnt++){}

A memory allocation error (in sqlite3StrNDup(0 called from sqlite3TokenCopy() called from sqlite3DequoteExpr()) can leave pRight->token.z as NULL. Dereferencing z in the for statement brings a segment violation.

 
2793 code active 2007 Nov anonymous   2007 Nov   3 3 fts3 lacks scoping edit
It would be nice if the fts3 symbols could optionally be made private/static as the rest of the sqlite3 library. Not sure why sqlite3_api becomes public when used with the amalgamation, for that matter.

  make TOP=`pwd` BCC=gcc TCC=gcc AR=ar RANLIB=echo NAWK=gawk -f \
    main.mk sqlite3.h sqlite3.c fts3amal.c
  cat fts3amal.c >> sqlite3.c
  gcc -DSQLITE_THREADSAFE -DSQLITE_API=static -DSQLITE_PRIVATE=static \
    -DSQLITE_EXTERN=static -DSQLITE_ENABLE_FTS3 -c sqlite3.c
  nm sqlite3.o | grep -v ' [trUbd] '

  00000004 C sqlite3_api
  00064da2 T sqlite3Fts3HashClear
  000652a4 T sqlite3Fts3HashFind
  00064d60 T sqlite3Fts3HashInit
  0006533b T sqlite3Fts3HashInsert
  00064b4c T sqlite3Fts3Init
  00066b34 T sqlite3Fts3InitHashTable
  000669bd T sqlite3Fts3PorterTokenizerModule
  0006702d T sqlite3Fts3SimpleTokenizerModule
 
2792 code fixed 2007 Nov anonymous   2007 Nov   3 3 invalidateCursorsOnModifiedBtrees has no scoping edit
The function invalidateCursorsOnModifiedBtrees is not scoped ala #2554. It should presumably be SQLITE_PRIVATE.

May I suggest this as part of the release/test process:

  gcc --pthread -DNDEBUG -DSQLITE_THREADSAFE -DEXPERIMENTAL -DSQLITE_API=static -DSQLITE_PRIVATE=static -DSQLITE_EXTERN=static -c sqlite3.c
  nm sqlite3.o | grep -v ' [trUbd] '

If there is any output from the nm/grep then those symbols have not been scoped.

2007-Nov-24 05:54:27 by anonymous:
I agree it should be part of the release check.

But I don't see EXPERIMENTAL #defined in the code anywhere. Also, --pthread does not work for gcc on Linux.

  $ gcc -DSQLITE_THREADSAFE -DSQLITE_API=static -DSQLITE_PRIVATE=static -DSQLITE_EXTERN=static -c sqlite3.c
  $ nm sqlite3.o | grep -v ' [trUbd] '
  0001d964 T invalidateCursorsOnModifiedBtrees


2007-Nov-24 07:55:20 by anonymous:
It should have been -pthread (one dash not two). And yes experimental should also be defined.


2007-Nov-24 18:38:47 by anonymous:
The macro EXPERIMENTAL is used by what exactly?


2007-Nov-24 19:55:44 by anonymous:
Several of the SQLite C APIs are marked as EXPERIMENTAL meaning that they could change between releases. This is noted on the documentation pages as appropriate. For example see http://sqlite.org/c3ref/commit_hook.html


2007-Nov-24 22:27:15 by anonymous:
These are just comments, not #ifdefs:

$ grep EXPERIMENTAL sqlite3.[ch]
sqlite3.c:****** EXPERIMENTAL - subject to change without notice **************
sqlite3.c:****** EXPERIMENTAL - subject to change without notice **************
sqlite3.c:****** EXPERIMENTAL - subject to change without notice **************
sqlite3.c:** This is an EXPERIMENTAL api and is subject to change or removal.
sqlite3.c:** EXPERIMENTAL - This is not an official function.  The interface may
sqlite3.c:/*** EXPERIMENTAL ***
sqlite3.c:******* THIS IS AN EXPERIMENTAL API AND IS SUBJECT TO CHANGE ******
sqlite3.c:****** EXPERIMENTAL - subject to change without notice **************
sqlite3.c:****** EXPERIMENTAL - subject to change without notice **************
 
2791 code active 2007 Nov anonymous   2007 Nov   1 1 Allow building FTS[123] as part of sqlite library with configure edit
See attached patch.
 
2790 code closed 2007 Nov anonymous   2007 Nov danielk1977 2 2 sqlite3_compile returns NULL for empty statements edit
There was apparently a backwards-incompatible behaviour change in 3.5 and later versions.

sqlite3_compile() for an empty statement, like "" now returns a NULL pointer for the statement, but the function returns SQLITE_OK. In my applications I always assumed that if sqlite3_compile() returns SQLITE_OK, then there will be a valid statement.

Now if you sqlite3_step() on the NULL pointer, you get a segfault.

This has been fixed so that passing a NULL pointer to sqlite3_step() returns SQLITE_MISUSE, as it did for earlier versions. Strictly speaking, applications are not supposed to pass NULL to sqlite3_step().

See also #2773.

 
2789 code fixed 2007 Nov anonymous   2007 Nov   3 3 Low memory double free from addWhereTerm edit
The end of the function addWhereTerm() in select.c contains the following:

  pE = sqlite3ExprAnd(pParse->db,*ppExpr, pE);
  if( pE ){
    *ppExpr = pE;
  }

The function sqlite3Expr() called by sqlite3ExprAnd() deletes both Expr parameters if it fails to create its result Expr.

Here the caller still has a pointer to the now freed Expr structure.

In our system it currently gets deleted again later, in a call from clearSelect() called from sqlite3SelectDelete() resulting in double free errors.

However, it is possible that this does not occur on current systems as ours is still based on version 3.3.8 (which still includes the bad (and no longer required) sqlite3ExprOrFree()).

The problem looks as if it slipped in when sqlite3Expr() was changed to delete its parameters on error.

The solution is to clear *ppExpr in this case, and this is most easily done by simply copying pE into it in all cases, even when NULL, ie make the line "*ppExpr = pE;" unconditional.

 
2788 build fixed 2007 Nov anonymous   2007 Nov drh 3 3 FormatMessageA not defined for Windows Mobile edit
Compiling SQLite 3.5.2 with embedded Visual C++ 4 produces the following link error:

os_win.obj : error LNK2019: unresolved external symbol FormatMessageA referenced in function winDlError

The only occurence of FormatMessageA is in os_win.c. Windows Mobile doesn't support ASCII, only wide characters. Since LoadLibrary isn't supported in winDlOpen on Windows Mobile, I suggest the following patch for winDlError:

  #if OS_WINCE
  	if (nBuf > 0)
  		zBufOut[0] = '\0';
  #else
    FormatMessageA(
      FORMAT_MESSAGE_FROM_SYSTEM,
      NULL,
      GetLastError(),
      0,
      zBufOut,
      nBuf-1,
      0
    );
  #endif
Already fixed by check-in [4541] .
 
2787 build active 2007 Nov anonymous   2007 Nov   4 5 sqlite3.pc is not remade edit
the subject says it all, there is no make rule to rebuild the sqlite3.pc from the .in file. it is only possible by hand (./config.status sqlite3.pc)
2007-Nov-22 12:17:50 by drh:
I don't know what the sqlite3.pc file does. I certainly do not use it for any of my builds on Linux, Mac OSX, or windows. Why should I leave it in the source tree? Isn't the best solution to this problem to simply delete the file?


2007-Nov-22 17:14:27 by anonymous:
It is indirectly used by pkgconfig. Here's some info on pkgconfig:

http://pkg-config.freedesktop.org/wiki/


2007-Nov-23 15:22:32 by drh:
Could somebody who understands what the sqlite3.pc file is used for suggest a makefile rule for rebuilding it?
 
2786 code fixed 2007 Nov anonymous   2007 Nov   4 4 sqlite3.pc does not mention -lpthread when used with --static flag edit
I'm trying to statically link sqlite3 on Linux. I'm using pkg-config to get the settings. However, when I use "pkg-config sqlite3 --libs --static", it does not specify "-lpthreads", which is necesary. The fix I use is to add the following line to sqlite3.pc.in:

  Libs.private: -lpthread
 
2785 doc fixed 2007 Nov anonymous   2007 Nov   3 3 Broken Anchors in C API Reference edit
All of the links at the top of the page are linked to #link rather than the correct section within the documentation.

http://www.sqlite.org/capi3ref.html

Fixed by http://www.sqlite.org/docsrc/vinfo/32573eb907ac4faea29a91677a69cd667d198239
 
2784 code fixed 2007 Nov anonymous   2007 Nov   1 3 Low memory double free and SEGV from flattenSubquery() edit
Low memory double free and SEGV from flattenSubquery()

sqlite3SrcListAppend() in build.c can call sqlite3DbRealloc() which can fail under low-memory conditions. In this case it returns zero.

It is called by flattenSubquery() in select.c in the section following:

  /* Move all of the FROM elements of the subquery into the
  ** the FROM clause of the outer query.

Immediately (3 lines) after this call (after the loop) the return value is dereferenced in the start of the next loop:

        pSrc = sqlite3SrcListAppend(db, pSrc, 0, 0);
      }
      p->pSrc = pSrc;
      for(i=pSrc->nSrc-1; i-extra>=iFrom; i--){

I am also getting a 'double free' error during the cleanup in sqlite3SrcListAppend(), where the old (unextended) structure is freed by sqlite3SrcListDelete(). Apparently one of the pItem->zName strings has already been freed by the lines:

    sqlite3DeleteTable(pSubitem->pTab);
    sqlite3_free(pSubitem->zDatabase);
    sqlite3_free(pSubitem->zName);
    sqlite3_free(pSubitem->zAlias);

immediately before the call in this section of flattenSubquery(). Clearing these pointers after freeing gets rid of the double free error.

I can't offer a simple solution to the SEGV

The SQL statement is "SELECT * FROM v10_11 LEFT JOIN t9 ON( a=x );" from the join-8.3 runtest.

 
2783 doc fixed 2007 Nov anonymous   2007 Nov   5 4 Broken links into the website edit
Hi,

there are some broken links in the new website: Documentation -> SQL Syntax -> PRAGMA

and

http://www.sqlite.org/lang_droptable.html try to click on "CREATE TABLE" and "VACUUM".

Fixed by http://www.sqlite.org/docsrc/vinfo/4f5adc7af26df28e61a1a5bacde935573394e7e2
 
2782 code closed 2007 Nov anonymous   2007 Nov   1 1 bind_text(...) into transaction edit
In Transaction i insert some values, using sqlite3_bind_text(...) The first row has undef value(some times it's empty), but other rows have normal values.
2007-Nov-21 13:50:25 by drh:
If you can describe some kind of problem with SQLite, we will be happy to look into the matter. Your description above sounds like a bug in your own code though. What about this problem makes you think it is a bug in SQLite?
 
2781 doc fixed 2007 Nov anonymous   2007 Nov   5 4 Make documentation available offline (i.e. pdf, chm, etc) edit
It would be nice if all of the documentation under "Documentation" was available offline in conjunction with the online format. The resulting pdf, chm, etc. could have the documentation segments merged or keep each segment separate.

For example: SQL Syntax segment could be its own pdf or could be merged into a pdf with the c++ API Reference.

Added a link to the download page containing the complete static text of this website.
 
2780 build fixed 2007 Nov anonymous   2007 Nov   3 3 sqliteint.h: test to disable _XOPEN_SOURCE uses wrong macro for OSX edit
sqliteint.h tests for __MACOS__ for whether to define _XOPEN_SOURCE on OSX. But this is not the correct macro (I think it's an old OS9 macro). __APPLE__would be a better choice.
 
2779 code closed 2007 Nov anonymous   2007 Nov   2 2 Performance optimizations edit
Performance optimizations

)) PRAGMA: Does anyone have experience and good results optimizing sqlite performance using PRAGMA's? )) Other techniques: Any success stories on sqlite optimization methods of any type would be appreciated.

Here is a list of the PRAGMA's and example use from the sqlite documentation:

PRAGMA auto_vacuum; PRAGMA auto_vacuum = 0 | none | 1 | full | 2 | incremental;

PRAGMA cache_size; PRAGMA cache_size = Number-of-pages;

PRAGMA case_sensitive_like; PRAGMA case_sensitive_like = 0 | 1;

PRAGMA count_changes; PRAGMA count_changes = 0 | 1;

PRAGMA default_cache_size; PRAGMA default_cache_size = Number-of-pages;

PRAGMA default_synchronous;

PRAGMA empty_result_callbacks; PRAGMA empty_result_callbacks = 0 | 1;

PRAGMA encoding; PRAGMA encoding = "UTF-8"; PRAGMA encoding = "UTF-16"; PRAGMA encoding = "UTF-16le"; PRAGMA encoding = "UTF-16be";

PRAGMA full_column_names; PRAGMA full_column_names = 0 | 1;

PRAGMA fullfsync PRAGMA fullfsync = 0 | 1;

PRAGMA incremental_vacuum(N);

PRAGMA legacy_file_format; PRAGMA legacy_file_format = ON | OFF

PRAGMA locking_mode; PRAGMA locking_mode = NORMAL | EXCLUSIVE PRAGMA main.locking_mode=EXCLUSIVE;

PRAGMA page_size; PRAGMA page_size = bytes;

PRAGMA max_page_count; PRAGMA max_page_count = N;

PRAGMA read_uncommitted; PRAGMA read_uncommitted = 0 | 1;

PRAGMA short_column_names; PRAGMA short_column_names = 0 | 1;

PRAGMA synchronous; PRAGMA synchronous = FULL; (2) PRAGMA synchronous = NORMAL; (1) PRAGMA synchronous = OFF; (0)

PRAGMA temp_store; PRAGMA temp_store = DEFAULT; (0) PRAGMA temp_store = FILE; (1) PRAGMA temp_store = MEMORY; (2)

PRAGMA temp_store_directory; PRAGMA temp_store_directory = 'directory-name';

This is not a bug report. Ask any questions on the mailing list.
 
2778 code closed 2007 Nov anonymous   2007 Nov adixon 1 1 Error 0 edit
CREATE PROCEDURE Driver_Daily_Activity {

	@comp_code 		varchar(8),
	@begin_date  		datetime,
	@end_date     		datetime,
	@emp_short_name 	varchar(8),
	@emp_id 		int,
	@driver_time_in 		output,
	@driver_time_out 	output,
	@driver_beaktime_in 	output,
	@driver_beaktime_out 	output,
	@driver_lunchtime_in 	output,
	@driver_lunchtime_out 	output,
	@total_break_time   	output

}

AS

select @driver_time_in =driver_datetime_in,@driver_time_out =driver_datetime_out from driver_daily_log with (nolock) where company_code=@comp_code and emp_id=@emp_id and driver_datetime_in >=@begin_date and driver_datetime_out<@end_date and driver_datetime_out is not null

select @driver_breaktime_in=substring((CAST(break_time_in AS varchar(20))),12,8) ,@driver_breaktime_out=substring((CAST(break_time_out AS varchar(20))),12,8) from driver_daily_break_log with (nolock) where company_code=@comp_code and emp_id=@emp_id and driver_status='OB'.

select @driver_lunchtime_in=substring((CAST(break_time_in AS varchar(20))),12,8) ,@driver_lunchtime_out=substring((CAST(break_time_out AS varchar(20))),12,8) from driver_daily_break_log with (nolock) where company_code=@comp_code and emp_id=@emp_id and driver_status='OL'.

select @total_break_time =sum(total_break_time) from driver_daily_break_log with (nolock) where company_code=@comp_code and emp_id=@emp_id.

This procedure is giving me the following error :

SQL State 42000: Error 0 : Syntax error or access violation

SQL identifiers may not contain the "@" character.
 
2777 code closed 2007 Nov anonymous   2007 Nov shess 1 1 FTS[23] compile bug edit
You need to #include "sqlite3.h" to pick up the sqlite3_malloc() declaration:

  ./ext/fts3/fts3_hash.c: In function "fts3HashMalloc":
  ./ext/fts3/fts3_hash.c:38: warning: initialization makes pointer from integer without a cast

Also, can you move the following:

  #ifndef SQLITE_ENABLE_BROKEN_FTS2
  #error fts2 has a design flaw and has been deprecated.
  #endif

to be after this line:

  #if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS2)

in fts2.c?

That way you can build fts3 while having fts2 objects in the Makefile without defining -DSQLITE_ENABLE_BROKEN_FTS2 for an fts3-only build.

2007-Nov-23 18:21:04 by anonymous:
This error still exists in FTS2 and FTS3:

  ./ext/fts3/fts3_hash.c: In function "fts3HashMalloc":
  ./ext/fts3/fts3_hash.c:38: warning: initialization makes pointer from integer without a cast

On some 64-bit systems this will cause FTS to crash because an int is smaller than a pointer.

Nevermind - it was fixed.

 
2776 warn active 2007 Nov anonymous   2007 Nov   5 1 mailing list edit
sqlite-users-digest doesn't work. It remembers registered addresses but doesn't send any emails.
 
2775 code closed 2007 Nov anonymous   2007 Nov danielk1977 3 4 Low-memory SEGV from sqlite3CodeRowTrigger() edit
I have hit a low memory segment violation in sqlite3CodeRowTrigger() after /* code the WHEN clause */.

sqlite3CodeRowTrigger() calls sqlite3ExprDup() which fails to deep copy the string p->pRight->token.z (and therefore everything thereafter).

sqlite3CodeRowTrigger() continues to call sqlite3ExprResolveNames()

sqlite3ExprResolveNames() calls walkExprTree() for the expression, which recursively calls walkExprTree() for pExpr->pRight which in turn calls nameResolverStep() which then tries to dereference pExpr->token.z[0] (for case TK_STRING) generating the SEGV.

The solution appears to be checking for allocation failures before calling nameResolverStep(), using the same early exit as when this fails. ie replacing:

      if( sqlite3ExprResolveNames(&sNC, whenExpr) ){

with:

      if( pParse->db->mallocFailed ||
          sqlite3ExprResolveNames(&sNC, whenExpr) ){
2007-Nov-16 14:36:10 by anonymous:
This change will also prevent an assertion if the previous allocation is the first to fail, or a SEGV if the following one is the first.


Thanks.
 
2774 code fixed 2007 Nov shess   2007 Nov shess 1 1 Crash merging empty OR queries in fts3. edit
  create virtual table test using fts2(a);
  insert into test (a) values ('alpha');
  insert into test (a) values ('beta');
  select a from test where test match ('a:gamma* OR a:delta*');

sqlite3: ../sqlite/ext/fts3/fts3.c:478: dataBufferAppend: Assertion `nSource>0 && pSource!=((void *)0)' failed.

"Workaround" is to compile with -DNDEBUG=1, but the person who reported this to me was using the fts2 dll from sqlite.org.

Fix on the way.

 
2773 code closed 2007 Nov danielk1977   2007 Nov   2 1 sqlite crashes in sqlite3_step when compiled with SQLITE_THREADS=1 edit
When running the self-test of pysqlite2 [1] , it crashes.

This only happens when sqlite3 is compiled with the --enable-threads option, but not with --disable-threads.

  #0  0xbb987124 in sqlite3_step () from /usr/pkg/lib/libsqlite3.so.0
  #1  0xbb9a7307 in _sqlite_step_with_busyhandler ()
     from /usr/pkg/lib/python2.4/site-packages/pysqlite2/_sqlite.so
  #2  0xbb9a437b in pysqlite_cursor_executescript ()
     from /usr/pkg/lib/python2.4/site-packages/pysqlite2/_sqlite.so
  #3  0xbbb45baa in PyCFunction_Call () from /usr/pkg/lib/libpython2.4.so.1.0

The reason for the crash is that v->db is NULL. In the non-threaded case, it all works.

[1] <http://initd.org/pub/software/pysqlite/releases/2.3/2.3.5/pysqlite-2.3.5.tar.gz>

Python is incorrectly passing NULL to sqlite3_step(). SQLite tests for this and returns SQLITE_MISUSE, but strictly speaking the behaviour in this case is unspecified.

Have changed things so that SQLITE_MISUSE is returned whether the build is threadsafe or not. But this should also be fixed in Python.


2007-Nov-23 13:24:36 by anonymous:
This is now also fixed in pysqlite.
 
2772 code closed 2007 Nov anonymous   2007 Nov danielk1977 3 3 Low memory assert in analyze edit
Function sqlite3Analyze() in analyze.c includes the following two lines:

  ...
      z = sqlite3NameFromToken(db, pName1);
      pTab = sqlite3LocateTable(pParse, z, 0);
  ...

In a low memory situation z can be returned as NULL.
sqlite3LocateTable() calls sqlite3FindTable() which promptly asserts on:

  assert( zName!=0 );

I expect the solution is to conditionalise some of the code in sqlite3Analyze():

  ...
      z = sqlite3NameFromToken(db, pName1);
      if (z){
        pTab = sqlite3LocateTable(pParse, z, 0);
        sqlite3_free(z);
        if( pTab ){
          analyzeTable(pParse, pTab);
        }
      }
  ...
Thanks for the report.
 
2771 code active 2007 Nov anonymous   2007 Nov   4 4 Lemon: Generated parser needs stdlib.h (not in default template) edit
I tested a simple do-nothing parser just to get lemon output, and this doesn't compile (if warnings treated as errors) because of non declaration of the memset() function for the following statment:

memset(&yygotominor, 0, sizeof(yygotominor));

(added for the resolution of SQLite ticket #2172).

The lempar.c just include stdio.h, it would suffice to add stdlib.h to get the memset() declaration (even if all real parsers must include stdlib.h to get something really working).

2007-Nov-14 21:02:25 by anonymous:
Sorry, the needed header is string.h, not stdlib.h :-)
 
2770 code active 2007 Nov anonymous   2007 Nov   1 1 Problem with BLOB in 3.5.x ? edit
After I've switched from 3.3.18 to 3.5.2, selecting from table which contains BLOB LONGER THAN ABOUT 990 BYTES returns error "SQL logic error or missing database" after call to _sqlite3_step().

I'm using preprocessed sources downloaded from here. DEBUG build of preprocessed sources works correctly, problem is only in RELEASE build. I'm using VC6.0 to compile.

Any idea what could be wrong? Thank you!

Can you try to reproduce this with the sqlite shell tool? Thanks.

Large blobs work for me with both release and debug builds (not msvc though, gcc/linux).


2007-Nov-12 18:41:37 by anonymous:
sqlite3.exe provided here works with the database. Problem is only with release build (static library linked into test application). Here is test app which exits with "Error 1" in release build:

  int main(int argc, char* argv[])
  {
	int rc;
	sqlite3* db;
	sqlite3_stmt* stmt;

	rc = sqlite3_open("n2.db3", &db);

	rc = sqlite3_prepare(db, "CREATE TABLE [ttt] ([bbb] BLOB)", -1, &stmt, 0 );
	rc = sqlite3_step(stmt);
	rc = sqlite3_reset(stmt);

	char text[10000],query[20000];
	strnset(text,'a',sizeof(text)-1);
	sprintf(query,"insert into [ttt] values (?)");

	rc = sqlite3_prepare(db, query, -1, &stmt, 0 );

	rc = sqlite3_bind_blob(stmt,1,text,sizeof(text), SQLITE_TRANSIENT);

	rc = sqlite3_step(stmt);
	rc = sqlite3_reset(stmt);

	rc = sqlite3_prepare(db, "select * from ttt", -1, &stmt, 0 );

	rc = sqlite3_step(stmt);

	if (rc == SQLITE_ROW)
	{
		printf("%s: OK",sqlite3_column_text(stmt,1));
	}
	else
	if (rc == SQLITE_DONE)
	{
		printf("DONE");
	}
	else
	{
		printf("Error %d",rc);
	}

	return 0;
}


2007-Nov-12 18:56:24 by drh:
You should be using sqlite3_finalize() instead of sqlite3_reset(). You are leaking memory. Also, you should use sqlite3_prepare_v2() to avoid problems with changing schemas.

But even without those fixes, I cannot reproduce the problem on Linux.


2007-Nov-12 19:39:31 by anonymous:
Suggested fixes didn't help.

I've tried to debug it. It fails in btree.c, line 3056:

if( offset+amt > nKey+pCur->info.nData ){ /* Trying to read or write past the end of the data is an error */ return SQLITE_ERROR; }

there seems to be different values in release mode. My debugger does not show values of variables in release mode, so I can be wrong, but it seems in release offset is 5 and in debug it is 4. There can be something wrong with compilation, I'll try to figure this out tomorrow.

BTW compilation of static libraty in VC6.0 gives 185 warnings. I don't know if it is ok, it haven't caused problems in older sqlite


2007-Nov-13 08:47:28 by anonymous:
I've turned off "Maximize Speed" option - this is causing the problem. No optimizations and optimize for size seems to be working. But it still makes me nervous :(( I really don't need corrupted database and now I hope it won't slow down too much. Unfortunately old library does not implement replace function so I don't want to switch back. This could be warning to others, I'm using VC++ 6.0 SP 6.

Thank you for your time.


2007-Nov-22 17:20:31 by anonymous:
I have exactly the same problem here (win XP, vc6 SP2) when I link against my sqlite static or dynamic library in release.

I have also used boundschecker to check sqlite, and it detects many dangling pointers ! But the strange thing is that I cannot find why these pointers are dangling, here an example:

  In prepare.c@188
  pTab = sqlite3FindTable(db, zMasterName, db->aDb[iDb].zName);

Boundchecker say that zMasterName is a dangling pointer, previously released here:

  in build.c@711:
  void sqlite3StartTable(
    Parse *pParse,   /* Parser context */
    Token *pName1,   /* First part of the name of the table or view */
    Token *pName2,   /* Second part of the name of the table or view */
    int isTemp,      /* True if this is a TEMP table */
    int isView,      /* True if this is a VIEW */
    int isVirtual,   /* True if this is a VIRTUAL table */
    int noErr        /* Do nothing if table already exists */
  ){
  }

It does not make sens for me, maybe it a false positive from boundchecker, but it is weird.

I don't know if these "errors" are related to the "blob" bug in release mode. I will try to debug these error with some "printf" in release mode.

Note: The provided dll (the one from the sqlite site) does not have this "bug".


2007-Nov-22 18:27:35 by anonymous:
More info: It seems that there is a bug in the VC6 (SP6) compiler.

In btree.c, line 3056:

  if( offset+amt > nKey+pCur->info.nData ){
    /* Trying to read or write past the end of the data is an error */
    return SQLITE_ERROR;
  }

After adding some printf around, It seems that the "speed optimization" compilation flag of VC6 changes the code order in a way that the offset variable is miss incremented !!

Two remarks:

  • I've traced the calling function, sqlite3BtreeData, and the it call accessPayload with the good offset value
  • VC6 produces an internal error: "fatal error C1001: INTERNAL COMPILER ERROR" in the accessPayload function, if I try to access the offset value before this line:

  aPayload = pCur->info.pCell + pCur->info.nHeader;

A dirty workaround could be to change the code order or the local var usage. I'm trying ....

 
2769 new closed 2007 Nov anonymous   2007 Nov danielk1977 3 3 Enable access to the sql string used to prepare a statement edit
If would be nice if it was possible to get access to the sql string used to prepare a sqlite3_stmt*, if it was prepared using sqlite3_prepare_v2. In code terms this would probably mean a public wrapper around sqlite3VdbeGetSql. Possible uses are error messages and statement cache.
 
2768 code closed 2007 Nov anonymous   2007 Nov danielk1977 3 3 asserts in wrong order edit
If pC is not set, produce a nice assert error message instead of a segfault:

diff -u -3 -p -r1.653 vdbe.c
--- src/vdbe.c  23 Oct 2007 15:39:45 -0000      1.653
+++ src/vdbe.c  11 Nov 2007 19:38:11 -0000
@@ -2005,10 +2005,10 @@ case OP_Column: {
   ** which is the number of records.
   */
   pC = p->apCsr[p1];
+  assert( pC!=0 );
 #ifndef SQLITE_OMIT_VIRTUALTABLE
   assert( pC->pVtabCursor==0 );
 #endif
-  assert( pC!=0 );
   if( pC->pCursor!=0 ){
     /* The record is stored in a B-Tree */
     rc = sqlite3VdbeCursorMoveto(pC);
Ok. Thanks.
 
2767 code fixed 2007 Nov anonymous   2007 Nov   2 1 UPDATE causes ./src/vdbe.c:503: sqlite3VdbeExec: Assertion `pTos<=&p-> edit
This might be hard to pin down, but since I don't know the code base all that well I would put this out there in case someone else might understand this better or would be willing to help me track this bug down. With that said, here is what is going on:

I get the following assertion error:

./src/vdbe.c:503: sqlite3VdbeExec: Assertion `pTos<=&p->aStack[pc]' failed.

with the attached "script".

2007-Nov-11 07:41:40 by anonymous:
Version 3.5.2 does not cause this to fail and runs as expected.


2007-Nov-11 09:30:49 by drh:
I beg to differ. I'm seeing the problem in 3.5.2 and in CVS head. I am able to reproduce the problem with the following test script:

   CREATE TABLE t1(x);
   CREATE TRIGGER r1 BEFORE UPDATE ON t1 BEGIN
      SELECT raise(ignore);
   END;
   INSERT INTO t1 VALUES(1);
   INSERT INTO t1 VALUES(2);
   INSERT INTO t1 SELECT x+2 FROM t1;
   INSERT INTO t1 SELECT x+4 FROM t1;
   UPDATE t1 SET x=x+1;

The problem is that the code generator is generating VDBE code that fails to clear values from the VDBE stack. After a few iterations of the trigger, the VDBE stack overflows, triggering the assertion fault.

It may be that the person posting the previous comment did not compile their version 3.5.2 with -DSQLITE_DEBUG. Hence assert() statements were disabled. The assert() statement is correct, however. This is a real problem.


2007-Nov-11 13:01:54 by drh:
It appears that operations such as UPDATE push a few values onto the stack prior to calling BEFORE triggers. Those values are normally consumed after the triggers run. But if a trigger runs RAISE(IGNORE) the later part of the processing is skipped and those values pushed on the stack are never consumed.

Should be a relatively easy fix. Either defer pushing the extra values until after the BEFORE triggers run. Or else pass an extra parameter into sqlite3CodeRowTrigger() to tell it how many values to pop from the stack when a RAISE(IGNORE) is encountered. Preferably the first.

I have long wanted to switch the VDBE from being a stack-based VM to being a register based VM. Notice that were the VDBE register based, this bug would have never come up....


2007-Nov-11 20:24:35 by anonymous:
Thank you for the amazingly fast turn around. (Yes, I forgot to compile 3.5.2 with -DSQLITE_DEBUG.)
 
2766 code active 2007 Nov drh   2007 Nov   1 1 TCL transaction started from within a query does not commit edit
This is a problem with the TCL interface. Consider the following TCL script:

   file delete -force test.db test.db-journal
   sqlite3 db test.db
   db eval {
     CREATE TABLE t1(x,y);
     INSERT INTO t1 VALUES(1,2);
     CREATE TABLE t2(a,b);
     INSERT INTO t2 VALUES(8,9);
   }
   db eval {SELECT * FROM t1} {
     db transaction {
       db eval {UPDATE t2 SET a=a*2}
     }
   }

The [db transaction] statement starts a transaction and it is suppose to commit the tranaction at the end of the code block. But because the transaction started while a query was active, the tranaction is unable to commit. The TCL interface never commits the tranaction nor does it give any kind of error indication.

It is unclear if an error should be returned or if the commit should be deferred until outer query finishes.

If the code within the [db transaction] block throws an error, we really need the transaction to rollback right away. Perhaps there should be a new API that cancels all pending queries. Perhaps a call to sqlite3_interrupt() would suffice for this. Need to investigate further....

 
2765 code closed 2007 Nov anonymous   2007 Nov   1 3 bad rollback if temporary table created during transaction edit
Hi, trying to resolve a test failure in an application using pysqlite2 (2.3.5) / sqlite 3.42 for its test suite, I've been able to produce a minimal script which shows that the problem is actually in sqlite... I don't think it comes from the python wrapper (if you think it is tell me and I'll redirect it to the appropriate tracker).

Anyway here is the script :

  import pysqlite2.dbapi2 as sqlite
  cnx = sqlite.connect('testdb')
  cursor = cnx.cursor()
  cursor.execute('SELECT eid from Trinfo')
  nb = len(cursor.fetchall())
  cursor.execute("INSERT INTO Trinfo ( from_state, to_state, wf_info_for, eid, creation_date, modification_date ) VALUES ( 734, 735, 5, 758, 1, 1)")
  # comment this line and the final assertion is ok
  cursor.execute("CREATE TEMPORARY TABLE T58d712f777a7170cb8ce7d94cef119ce (C0 integer,C1 timestamp);")
  # get the same result with or without dropping the temp table
  #cursor.execute('DROP TABLE T58d712f777a7170cb8ce7d94cef119ce')
  cnx.rollback()
  cursor.execute('SELECT eid from Trinfo')
  nb3 = len(cursor.fetchall())
  assert nb3 == nb # FAIL

Notice that if you don't create the temporary table, everything run fine, else the insertion is not actually rollbacked.

2007-Nov-09 00:23:38 by drh:
I tested using the following TCL script:

    file delete -force test.db test.db-journal
    sqlite3 db test.db
    db eval {
      CREATE TABLE t1(a,b,c);
      INSERT INTO t1 VALUES(1,2,3);
      INSERT INTO t1 VALUES(4,5,6);
      INSERT INTO t1 SELECT a*2, b*2, c*2 FROM t1;
      INSERT INTO t1 SELECT a*4, b*4, c*4 FROM t1;
      INSERT INTO t1 SELECT a*8, b*8, c*8 FROM t1;
    }
    puts "begin=[md5 [db eval {SELECT * FROM t1}]]"
    db eval {BEGIN}
    db eval {INSERT INTO t1 VALUES('hello','out','there');}
    db eval {CREATE TEMPORARY TABLE T58d712f777(C0 integer,C1 timestamp)}
    puts "  mid=[md5 [db eval {SELECT * FROM t1}]]"
    db eval {ROLLBACK}
    puts "  end=[md5 [db eval {SELECT * FROM t1}]]"

The results are:

    begin=c17bc52be49269f8f2cd3a4df3caee6e
      mid=f86cc8ad5b925962323edc041bfaa09d
      end=c17bc52be49269f8f2cd3a4df3caee6e

Begin and end are equal so it appears to work using the TCL interface. I also observe that there are countless tests in the SQLite test suite that verify that this sort of thing works correctly.

Perhaps this is something to do with python after all.


2007-Nov-11 20:48:52 by anonymous:
See: http://www.initd.org/pub/software/pysqlite/doc/usage-guide.html#controlling-transactions
 
2764 todo closed 2007 Nov anonymous   2007 Nov   2 2 Since I installed Leopard I keep getting sqlite3 data bases in trash edit
Every night I seem to have a bunch of database folders in my trash with sqlite doc inside. I can't seem to delete them all even after quitting my applications.

(I realize this is a developer's page, but this is a real annoyance and I have no idea what to do about it!) Thanks, Liz

2007-Nov-08 02:50:51 by drh:
I am told by engineers at Apple that this is an issue with Leopard and that it has nothing to do with SQLite. Apple is working on the problem and will fix it.
 
2763 build active 2007 Nov anonymous   2007 Nov   4 4 AIX build failure due to explicit _XOPEN_SOURCE definition edit
Our AIX machine has started having problems building the new sqlite 3.5.2 source. I'm no AIX expert, but I did some digging and I think it's due to sqliteInt.h defining _XOPEN_SOURCE without also defining _ALL_SOURCE.

AIX system header files usually define this themselves (in /usr/include/ standards.h) but only if _XOPEN_SOURCE was previously undefined.

Has anyone else come across this bug? I'm willing to believe it's our build system if not; we can work around it by setting _ALL_SOURCE on the command-line, but I thought it might be useful to raise a bug anyway.

See also tickets #2673, #2681, and #2741.

I do not have access to an AIX machine and so have no ability to debug this problem. If you have suggested patches we will consider them. Otherwise, there is not much we can do about this ticket.

 
2762 code fixed 2007 Nov anonymous   2007 Nov   3 3 fts3 is cavalier about malloc vs sqlite3_malloc edit
The fts3 code (and, from a brief look, the fts2 and fts1 also) are inconsistent with the memory upcalls they use; in some cases they use malloc/realloc/free/calloc(..., 1), and in some they use the sqlite3_ equivalents. I have a patch that makes them use only the sqlite3_ equivalents, but was there a specific reason they were written without originally?

And another problem: trying to build fts3_hash.c gives warnings about unknown parameters for sqlite3_malloc (used from fts3HashMalloc) because it never finds sqlite3.h from anywhere. On at least one 64-bit Solaris platform this has caused a segfault on execution, because the compiler guess for return type (int) was 32 bits, whereas the pointer it was trying to return was 64 bits.

2007-Nov-20 03:41:52 by anonymous:
Indeed, I also have segfaults with FTS3 memory allocation with a 64bit build on OSX 10.5. It works fine on a 32bit build on OS X 10.5. This is the thread trace:

  Thread 0 Crashed:
  0   libSystem.B.dylib             	0x00007fffffe00637 __bzero + 55
  1   org.sqlite.sqlite3            	0x0000000100051fc3 fts3HashMalloc + 46
  2   org.sqlite.sqlite3            	0x0000000100052465 sqlite3Fts3HashInsert + 316
  3   org.sqlite.sqlite3            	0x000000010004bb0e sqlite3Fts3Init + 126
  4   org.sqlite.sqlite3            	0x000000010002537c openDatabase + 808
  5   sqlite3                       	0x000000010000115c open_db + 37
  6   sqlite3                       	0x0000000100003af9 process_input + 938
  7   sqlite3                       	0x00000001000043a8 main + 1696
  8   sqlite3                       	0x000000010000099c start + 52

I didn't try the idea of changing malloc & friends to sqlite equivalents.


2007-Nov-20 03:56:07 by anonymous:
All calls should use the new sqlite3_ alloc/free calls instead of malloc/free. Because fts is not a part of the sqlite core, this code receives less attention.

Also, Makefile.in should be updated so that fts2 and fts3 can build out of the box on all architectures without requiring them loaded as external modules.


2007-Nov-23 18:00:39 by anonymous:
Those patches seem to work on OSX 10.5 64bit. No compile errors. No run errors in a simple test. I don't build with TclTk (no 64bit tcl on OSX anyways), so I can't run the full tests.
 
2761 code active 2007 Nov anonymous   2007 Dec   3 3 CLI (shell.c) should be bundled with amalgamation edit
The CLI (shell.c) should be bundled with the amalgamation for database administrative purposes without downloading the matching shell.c from the full source tree.
I second that! Qt ships with the amalgamated source files, but we also ship shell.c, whch we have to retrieve from the non-amalgamated source files.


2007-Dec-26 15:20:04 by anonymous:
I also agree. It is inconvenient to retrieve the matching shell.c from the source tree.
 
2760 new active 2007 Nov anonymous   2007 Nov   5 4 request: sqlite3_unlink() to delete db files. edit
Hi!

Today i came across a use case where i would like client code to be able to delete an underlying sqlite3 db, but that code doesn't have immediate access to the file name of that db (without refactoring the db wrapper code).

An interesting feature addition would, IMO, be:

int sqlite3_unlink( sqlite3 * db, bool closeTheFile );

Unlinks the file associated with the given database. It does not alter the database in any way (thus is it a no-op on a :memory: database). The closeTheFile flag specifies whether the file handle associated with db should also be closed (and thus db must also be closed), or just unlinked (e.g., as temporary databases are unlinked right after creation but kept open).

After browsing through the VFS API a bit, i see that there is an xDelete function, but i'm not sure if its semantics require that the underlying file handle be closed. i don't see an extra xClose member of VFS, so i assume that xDelete also handles closing the file handle. If these were split into two features, sqlite3_unlink() could be implemented very easily.

:)

 
2759 code closed 2007 Nov anonymous   2007 Nov danielk1977 1 2 crash due to 'select * from virtualTable where x is null' edit
For virtual table T1:

select * from t1 where y is NULL;

causes:

1301 assert( pTerm->eOperator & (WO_EQ|WO_LT|WO_LE|WO_GT|WO_GE|WO_MATCH) ); (gdb) bt

#0 0x005e17a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2

#1 0x006217d5 in raise () from /lib/tls/libc.so.6

#2 0x00623149 in abort () from /lib/tls/libc.so.6

#3 0x0061adb1 in __assert_fail () from /lib/tls/libc.so.6

#4 0x00849534 in bestVirtualIndex (pParse=0xbfe2d750, pWC=0xbfe2d1d0, pSrc=0x9a23d2c, notReady=18446744073709551615, pOrderBy=0x0, orderByUsable=1, ppIdxInfo=0x9a372bc) at ../../../../Source/sqlite-3.5.1-hp/src/where.c:1301

#5 0x0084a989 in sqlite3WhereBegin (pParse=0xbfe2d750, pTabList=0x9a23d28, pWhere=0x9a28ed8, ppOrderBy=0xbfe2d59c) at ../../../../Source/sqlite-3.5.1-hp/src/where.c:2074

#6 0x0082c6c6 in sqlite3Select (pParse=0xbfe2d750, p=0x9a28f28, eDest=4, iParm=0, pParent=0x0, parentTab=0, pParentAgg=0x0, aff=0x0) at ../../../../Source/sqlite-3.5.1-hp/src/select.c:3118

#7 0x0081f16c in yy_reduce (yypParser=0x9a33aa8, yyruleno=104) at parse.y:368

#8 0x00820f65 in sqlite3Parser (yyp=0x9a33aa8, yymajor=1, yyminor= {z = 0x9a2e120 ";", dyn = 0, n = 1}, pParse=0xbfe2d750) at parse.c:3419

#9 0x0082e877 in sqlite3RunParser (pParse=0xbfe2d750, zSql=0x9a2e100 "select * from t1 where y is NULL;", pzErrMsg=0xbfe2d74c) at ../../../../Source/sqlite-3.5.1-hp/src/tokenize.c:449

#10 0x008246a0 in sqlite3Prepare (db=0x9920010, zSql=0x9a2e100 "select * from t1 where y is NULL;", nBytes=-1, saveSqlFlag=0, ppStmt=0xbfe2d8e0, pzTail=0xbfe2d8e4) at ../../../../Source/sqlite-3.5.1-hp/src/prepare.c:525

#11 0x00824a2c in sqlite3LockAndPrepare (db=0x9920010, zSql=0x9a2e100 "select * from t1 where y is NULL;", nBytes=-1, saveSqlFlag=0, ppStmt=0xbfe2d8e0, pzTail=0xbfe2d8e4) at ../../../../Source/sqlite-3.5.1-hp/src/prepare.c:604

#12 0x00824bf9 in sqlite3_prepare (db=0x9920010, zSql=0x9a2e100 "select * from t1 where y is NULL;", nBytes=-1, ppStmt=0xbfe2d8e0, pzTail=0xbfe2d8e4) at ../../../../Source/sqlite-3.5.1-hp/src/prepare.c:658

#13 0x0084d862 in sqlite3_exec (db=0x9920010, zSql=0x9a2e100 "select * from t1 where y is NULL;", xCallback=0x8049b23 <callback>, pArg=0xbfe2da20, pzErrMsg=0xbfe2d9ac) at ../../../../Source/sqlite-3.5.1-hp/src/legacy.c:56

#14 0x0804d2e9 in process_input (p=0xbfe2da20, in=0x0) at ../../../../Source/sqlite-3.5.1-hp/src/shell.c:1663

#15 0x0804de80 in main (argc=5, argv=0xbfe2efd4) at ../../../../Source/sqlite-3.5.1-hp/src/shell.c:2001

Fails to accommodate WO_ISNULL flag.

2007-Nov-05 05:13:22 by danielk1977:
Thanks for the report.
 
2758 code closed 2007 Nov anonymous   2007 Dec   1 1 Why this? edit
sqlite3* s1; sqlite3* s2;

sqlite3_stmt* stm;

sqlite3_open("db", &s1); sqlite3_open("db", &s2);

sqlite3_prepare_v2(s2, "SELECT * FROM sqlite_master", 27,&stm, NULL); sqlite3_step(stm); sqlite3_finalize(stm);

sqlite3_prepare_v2(s1, "CREATE TABLE C (c1)", 19, &stm, NULL); sqlite3_step(stm); sqlite3_finalize(stm);

sqlite3_prepare_v2(s2, "SELECT * FROM C", 15, &stm, NULL);

Last row return ERROR n.17 SQLITE_SCHEMA, But All previous statements are finalized.

2007-Nov-05 05:17:04 by danielk1977:
It's not obvious.

Which call is returning SQLITE_SCHEMA? sqite3_prepare_v2() or sqlite3_step()?

Are any other processes accessing the same database?

Which version of SQLite are you using?

Can you post a complete program that demonstrates this?


2007-Nov-05 16:00:54 by anonymous:
I can reproduce the original poster's findings.

For me, the final sqlite3_prepare_v2() returns 17 (SQLITE_SCHEMA). Up to this everything works as documented.

I run the posted code unchanged in a single process on Windows XP as main(). The database file does not exist prior to execution.


2007-Dec-13 19:50:15 by rdc:
This was closed as "can not reproduce", but I think the reported behavior is exactly as expected from SQLite.

This program opens two connections to the database file. Runs a query in one which causes SQLite to read the schema. It then adds a table using the second connection. This changes the schema. Finally it tries to execute a query that references the new table using the first connection. This generates an error.

I was informed by DanK on the mailing list that this is the normal behavior of SQLite. I thought that the new prepare_v2 API was supposed check if the schema cache was stale, and reload it before generating a schema error, but he said it didn't. Apparently the poster thinks this behavior is strange as well.

I think this should be reopened, or re-closed as "works as designed". It should be reproducible (I suspect the test was done using a single connection) and maybe shouldn't have been closed the way it was.

 
2757 doc closed 2007 Nov anonymous   2007 Nov   1 1 Special characters handling in SQLite Like Query edit
If I have data as follows, they wont show up when I try to search for them.

[[[ ]]]

`~!@#$%^&*

()_+|{}:<>?

-=\[];',./

@@@ %%%

### ^^^

___---

Ho_mer

`~!@#$%^&*()_+|`{}:<>?-=\[];,./

_Storage-Room

@ # $

_ - +

% ^ &

asdf_qwerty

As all the data contains special charcters ,I am replacing them with SearchS = Trim(SearchS)

SearchS = Replace(SearchS, "'", "''")

SearchS = Replace(SearchS, "[", "[[]")

SearchS = Replace(SearchS, "%", "[%]")

SearchS = Replace(SearchS, "^", "[^]")

SearchS = Replace(SearchS, "#", "[#]")

SearchS = Replace(SearchS, "_", "[_]")

My SQL Query looks like this select * from table_name where name LIKE '%" & SearchS & "%'

eg . If I am trying to search for '%%%' my query will be

select * from table_name where name LIKE '% [%][%][%]%'

Which is returning zero results.

so how should I replace special characters for SQlite.

Thanks

2007-Nov-03 00:11:35 by drh:
The description appears to be a question on the proper way to use SQL, not a bug report. Please use tickets for reporting bugs only. For help with using SQLite, send a message the the SQLite mailing list.
 
2756 new active 2007 Nov anonymous   2007 Nov   1 1 allow vacuum to change pragma setting instead of using existing ones edit
we've got databases created with page_size of 1k, and we'd like to change that setting to 4k. vacuum creates a temporary db, attach it to the current connection, creates the tables (based on what's in the old db), and then selects from the old db and inserts into the new one.

vacuum does exactly what we need (creating a new db from an old one), but it re-uses the existing pragmas for page size, auto vacuum and reserved page size.

from sqlite.c, see sqlite3RunVacuum()

  sqlite3BtreeSetPageSize(pTemp, sqlite3BtreeGetPageSize(pMain),
     sqlite3BtreeGetReserve(pMain));

dr hipp points out that the the operands to vacuum are unused. from sqlite.c:

sqlite3VdbeAddOp(v, OP_Vacuum, 0, 0);

one solution would be to allow the user to specify the page size, reserve page size, and autovacuum as optional params to vacuum.

he had an idea of using the signedness of the first operand to represent the autovacuum setting (since after a table is created, you can change the setting from auto to incremental, but you can't change it from none to auto (or none to incremental)

 
2755 code active 2007 Nov anonymous   2007 Nov   3 3 trace interfere with transaction Tcl interface edit
When using the transaction method of the Tcl interface to the SQLite with a registered "trace" function, the stack trace is lost in case an error occurs inside the transaction.

As an example I provide two outputs, the first one without a registered trace function and the second one with one (in which it cannot be seen where the exception cames from):


First:

   > ./a.tcl
vorher
BUMMM
    while executing
"a"
    invoked from within
"db transaction {
        puts "vorher"
        a
        puts "nachher"
    }"
    ("uplevel" body line 1)
    invoked from within
"uplevel 1 [list db transaction {
        puts "vorher"
        a
        puts "nachher"
    }]"
    (procedure "b" line 2)
    invoked from within
"b"
    (file "./a.tcl" line 28)


Second:

   > ./a.tcl
BEGIN
vorher
ROLLBACK

    while executing
"db transaction {
        puts "vorher"
        a
        puts "nachher"
    }"
    ("uplevel" body line 1)
    invoked from within
"uplevel 1 [list db transaction {
        puts "vorher"
        a
        puts "nachher"
    }]"
    (procedure "b" line 2)
    invoked from within
"b"
    (file "./a.tcl" line 28)

********

A scritp that demostrates this behaviour is attached. The only workaround is not to trace.

Thanks

 
2753 code active 2007 Nov anonymous   2007 Nov drh 3 3 Master journal files sometimes not deleted edit
In the 3.4.1 amalgamation, in vdbeCommit, the master journal file is created, and deleted at the end or if there is an error. But it looks like there is one case where it gets closed but not deleted. The code is:

    for(i=0; rc==SQLITE_OK && i<db->nDb; i++){
      Btree *pBt = db->aDb[i].pBt;
      if( pBt && sqlite3BtreeIsInTrans(pBt) ){
        rc = sqlite3BtreeCommitPhaseOne(pBt, zMaster);
      }
    }
    sqlite3OsClose(&master);
    if( rc!=SQLITE_OK ){
      sqliteFree(zMaster);
      return rc;
    }

It seems like that last bit should be:

    if( rc!=SQLITE_OK ){
      sqlite3OsDelete(zMaster);
      sqliteFree(zMaster);
      return rc;
    }
2007-Nov-01 19:26:28 by anonymous:
Should have read the comment above that code segment closer. Looks like this was by design. Trying to figure out what to do with the client that has hundreds of orphaned master journal files...
 
2752 code closed 2007 Nov anonymous   2007 Nov   5 5 redundant assignment? edit
In unixLock()

  struct lockInfo *pLock = pFile->pLock;

and later in the function

  pLock = pFile->pLock;
2007-Nov-01 05:56:16 by danielk1977:
The second assignment is required as the value of pFile->pLock might be changed within the call to transferOwnership() above the second assignment.

In non-threadsafe builds (where transferOwnership is a no-op) it is redundant, but it's not hurting anybody.

 
2751 build closed 2007 Oct anonymous   2007 Nov   2 3 Regression - Solaris/sparc throws BUS ERROR on count()/sum()/etc. edit
A regression has crept into sqlite 3.5 (both 3.5.0 and 3.5.1) running on Solaris/sparc when built with GCC (both 3.4.2 and 4.2.2 tested) which leads to a BUS ERROR when simple count()/sum() queries are run. A workaround is to build with Solaris CC.

Example session:

  # ./sqlite3
  SQLite version 3.5.1
  Enter ".help" for instructions
  sqlite> create table foo(a);
  sqlite> insert into foo values(1);
  sqlite> select count(a) from foo;
  Bus error (core dumped)
2007-Oct-31 18:19:12 by danielk1977:
Can you check that applying this clears the problem? Thanks.

   diff -u -r1.110 vdbeapi.c
   --- src/vdbeapi.c       17 Oct 2007 01:44:21 -0000      1.110
   +++ src/vdbeapi.c       31 Oct 2007 18:15:44 -0000
   @@ -446,7 +446,7 @@
          pMem->flags = MEM_Agg;
          pMem->xDel = sqlite3_free;
          pMem->u.pDef = p->pFunc;
   -      if( nByte<=NBFS ){
   +      if( nByte<=NBFS && 0 ){
            pMem->z = pMem->zShort;
            memset(pMem->z, 0, nByte);
          }else{


2007-Nov-02 09:06:57 by anonymous:
I can confirm that your patch prevents the crash. Thank you.


2007-Nov-05 05:51:35 by anonymous:
Do you attribute this error to be a compiler bug or a problem in SQLite?


2007-Nov-05 06:20:42 by danielk1977:
Sorry, I missed your post on November 2.

I think it's an alignment problem. Without the patch, sqlite3_aggregate_context() is returning a pointer to a pre-allocated buffer. The problem is (I suspect) that the start of this buffer is not aligned to an 8-byte boundary as required by your architecture.

So I figure an SQLite bug, not a compiler problem.

Will fix shortly.


2007-Nov-05 14:41:25 by anonymous:
Perhaps zShort should be moved after a double in the struct for alignment?

diff -u -3 -p -r1.130 vdbeInt.h
--- src/vdbeInt.h       28 Aug 2007 23:28:08 -0000      1.130
+++ src/vdbeInt.h       5 Nov 2007 14:39:04 -0000
@@ -123,6 +123,7 @@ struct Mem {
     FuncDef *pDef;      /* Used only when flags==MEM_Agg */
   } u;
   double r;           /* Real value */
+  char zShort[NBFS];  /* Space for short strings */
   sqlite3 *db;        /* The associated database connection */
   char *z;            /* String or BLOB value */
   int n;              /* Number of characters in string value, including '\0' */
@@ -130,7 +131,6 @@ struct Mem {
   u8  type;           /* One of SQLITE_NULL, SQLITE_TEXT, SQLITE_INTEGER, etc */
   u8  enc;            /* SQLITE_UTF8, SQLITE_UTF16BE, SQLITE_UTF16LE */
   void (*xDel)(void *);  /* If not null, call this function to delete Mem.z */
-  char zShort[NBFS];  /* Space for short strings */
 };
 typedef struct Mem Mem;

 
2750 code closed 2007 Oct anonymous   2007 Oct   2 1 for the new.somefield value returns the old value of it edit
For the update trigger on a table containing n columns when retrieving the new value of filed the value of which was not changed at update (so the new value is null)it returns its old value.
2007-Oct-31 18:21:32 by danielk1977:
That's how it's documented to work.
 
2749 warn active 2007 Oct anonymous   2007 Oct   3 4 SQLITE_OMIT_FLOATING_POINT, int constants too large edit
Hiya!

i'm working from the 3.5.1 (CVS version) of the amalgamation. The code was pulled from CVS sometime around Oct 27th 2007.

Platform: Kubuntu Linux 7.04, i386-32, gcc 4.1.2.

The compiler output says it all:

  gcc -c -DSQLITE_OMIT_FLOATING_POINT sqlite3.c
  sqlite3.c: In function 'bestVirtualIndex':
  sqlite3.c:65531: warning: integer constant is too large for 'long' type
  sqlite3.c: In function 'bestIndex':
  sqlite3.c:65600: warning: integer constant is too large for 'long' type
  sqlite3.c: In function 'sqlite3WhereBegin':
  sqlite3.c:66223: warning: integer constant is too large for 'long' type
  sqlite3.c:66248: warning: integer constant is too large for 'long' type
  sqlite3.c:66254: warning: integer constant is too large for 'long' type

These warnings would seem to indicate potentially serious problems, though i admittedly have not investigated whether overflows are really fatal in the affected contexts.

 
2748 warn active 2007 Oct anonymous   2007 Oct   4 4 amalgamation: SQLITE_OMIT_ALTERTABLE warnings edit
Hiya!

i'm working from the 3.5.1 (CVS version) of the amalgamation. The code was pulled from CVS sometime around Oct 27th 2007.

The compiler output says it all:

  gcc -c -DSQLITE_OMIT_ALTERTABLE sqlite3.c
  sqlite3.c:6909: warning: 'sqlite3AlterRenameTable' used but never defined
  sqlite3.c:6916: warning: 'sqlite3AlterFinishAddColumn' used but never defined
  sqlite3.c:6917: warning: 'sqlite3AlterBeginAddColumn' used but never defined

The relevant code is:

  sed -ne '6909p;6916p;6917p' sqlite3.c
  SQLITE_PRIVATE void sqlite3AlterRenameTable(Parse*, SrcList*, Token*);
  SQLITE_PRIVATE void sqlite3AlterFinishAddColumn(Parse *, Token *);
  SQLITE_PRIVATE void sqlite3AlterBeginAddColumn(Parse *, SrcList *);

Platform: Kubuntu Linux 7.04, i386, gcc 4.1.2.

 
2747 code closed 2007 Oct anonymous   2007 Nov maria.khomenko 1 1 How to use Composite Key in Sqlite edit
How to use Composite Key in Sqlite
Tickets are for reporting bugs. Please ask general questions such as this on the SQLite mailing list.


2007-Nov-01 11:34:53 by anonymous:
but kindly Maria Give me the Way how can i use


2007-Nov-02 07:15:59 by anonymous:
How and where Can i do that


2007-Nov-02 07:18:06 by anonymous:
If you have some Articles regarding that mail me at rizwan_aman007@yahoo.co.in


2007-Nov-05 05:39:22 by anonymous:
I did't get any resolution. plz help me


2007-Nov-08 12:07:18 by anonymous:
`
 
2746 code closed 2007 Oct anonymous   2007 Oct   1 1 PRIMARY KEY does not treat NULL as unique edit
The below inserts 2 records. One record would be expected.

CREATE TABLE test(Colour VARCHAR COLLATE NOCASE,SubType VARCHAR COLLATE NOCASE,weight DOUBLE, PRIMARY KEY (Colour,SubType));

BEGIN TRANSACTION; INSERT OR REPLACE INTO test VALUES('red', NULL, 100); INSERT OR REPLACE INTO test VALUES('red', NULL, 100); COMMIT

I understand that the NULL behaviour is hazy but this seems to not agree with the doc where it says: "nulls are distinct in a UNIQUE column".

2007-Oct-28 23:35:17 by drh:
"nulls are distinct" means that every NULL is considered to be different from every other NULL. Hence, if two rows contain only NULL values, they are considered UNIQUE because they are not equal to one another.
 
2745 code closed 2007 Oct anonymous   2007 Oct shazow 1 1 Sharing File between two processes edit
Sharing File between two processes
 
2744 code closed 2007 Oct drh   2007 Nov danielk1977 1 1 Cannot quote collating sequence name edit
The following statement gives an error:

   CREATE TABLE demo(x TEXT COLLATE "binary");

It is using the quote characters as part of the name. The quotes ought to be removed before searching for the collating sequence.

 
2743 doc closed 2007 Oct anonymous   2007 Oct   1 1 Comparison of SQLite against McObject Fusion edit
Can you please provide a comparison of SQLite against McObhect Fusion. Thanks.
2007-Oct-24 11:07:16 by drh:
I've never heard of McObject Fusion before.
 
2742 code closed 2007 Oct anonymous   2007 Oct   1 1 codeInteger and codeReal edit
In the latest CVS I noticed that codeInteger and codeReal ignores the argument `n'.

  static void codeReal(Vdbe *v, const char *z, int n, int negateFlag){
    assert( z || v==0 || sqlite3VdbeDb(v)->mallocFailed );
    if( z ){
      double value;
      char *zV;
      sqlite3AtoF(z, &value); //***** can this accidentally go beyond n bytes?
      if( negateFlag ) value = -value;
      zV = dup8bytes(v, (char*)&value);
      sqlite3VdbeOp3(v, OP_Real, 0, 0, zV, P3_REAL);
    }
  }

I was under the impression that both sqlite3AtoF and sqlite3Atoi64 required null terminated strings. Is `z' guaranteed to be null terminated in all cases? Could a random byte follow the number string buffer that happens to be a digit and cause the parsed number to be incorrect?

So at best you could drop the 'n' arg from codeReal and codeInteger, at worst, you'd have to take n bytes into consideration for the sqlite3Ato* functions or less desirable - allocate a buffer and copy the string to it and nil terminate it prior to calling the sqlite3Ato* functions.

2007-Oct-23 19:23:24 by anonymous:
You forgot to assert z[n] is not '.', 'e', and 'E' for codeReal.

I'm not comfortable with this not-a-valid-number-character assumption, but am too lazy to look for a counter example that might break it.

If you're confident that the z[n] byte always exists and reading that byte in sqlite3Ato* will not segfault or cause valgrind to complain, why not do this instead:

   char zn = z[n];
   z[n] = 0;
   ... do sqlite3Ato* stuff
   z[n] = zn;

It's fast, guaranteed to be safe and is more resiliant in the face of future possibly incompatible code changes in the parser.


2007-Oct-23 19:26:05 by anonymous:
As you well know, the assert only works in debug mode, not in production. The debug runs pad memory allocations at the end with a byte or two, do they not?


2007-Oct-23 19:52:10 by anonymous:
The saving/restoring z[n] hack above won't work if the original parse string on which the tokens are based is stored in read-only memory or if the original string is being parsed simultaneously in 2 or more threads. But if the original parse string is guaranteed to be copied before parsing begins, then sure, it could work.


2007-Oct-23 21:15:15 by drh:
The code is correct as written. If you think otherwise, show me a counter-example.

Could the code benefit from be refactored to be cleaner. Probably so. But that would require hundreds of lines of carefully optimized code to be redesigned and rewritten. I have more important fish to fry right now and so I am not going to waste several days modifying code that works. It is what it is. Either show me a case where it fails or else learn to live with it.

 
2741 code fixed 2007 Oct anonymous   2007 Oct   3 3 _XOPEN_SOURCE may be necessary for pread/pwrite as well as threads edit
In 3.5.1, _XOPEN_SOURCE is only defined if sqlite is configured to be thread-safe. However, this definition may also be necessary to make prototypes visible for pread, pwrite, fdatasync, and usleep, all of which are conditionally used by os_unix.c. This is only a major problem for pread and pwrite, both of which take arguments that may be 64 bits wide and require a special calling convention on systems with 32-bit "long".
2007-Oct-23 01:09:32 by anonymous:
But what value should it be set to in order to be portable across all POSIX machines?


2007-Oct-23 14:52:57 by anonymous:
_XOPEN_SOURCE is so ill defined and poorly implemented, the user should add -D_XOPEN_SOURCE=whatever to their own makefile's compile flags if they need 64 bit pread/pwrite support.


2007-Oct-23 16:01:32 by drh:
I have added comments to the code to warn about this situtation. But in accordance with the suggestions on ticket #2681, we do not want to define _OPEN_SOURCE by default.
 
2740 code fixed 2007 Oct anonymous   2007 Oct   4 4 incorrect assertions in vdbe.c edit
/sqlite/src/vdbe.c contains two assertions of the form

  assert( SQLITE_MAX_SQL_LENGTH < SQLITE_MAX_LENGTH );

as part of the implementations of the OP_String8 and OP_HexBlob opcodes. This assertion is wrong, because the default definition of SQLITE_MAX_SQL_LENGTH is equal to SQLITE_MAX_LENGTH. It should be

  assert( SQLITE_MAX_SQL_LENGTH <= SQLITE_MAX_LENGTH );
 
2739 code fixed 2007 Oct anonymous   2007 Oct   2 3 large-file enable macros should be defined before first #include edit
Currently (3.4.2 and it appears 3.5.1 as well) /sqlite/src/os_unix.c tries to enable support for large files by defining _LARGEFILE_SOURCE and/or _FILE_OFFSET_BITS=64. Unfortunately, it does this after including sqliteInt.h and os.h, which include several system headers, which means that on some platforms the macros have no effect.

Contrast the behavior of these two test files:

testA.c

  #define _FILE_OFFSET_BITS 64
  #include <stdio.h>
  #include <unistd.h>

testB.c

  #include <stdio.h>
  #define _FILE_OFFSET_BITS 64
  #include <unistd.h>

When preprocessed, testA.c defines a 64-bit off_t, testB.c doesn't:

  $ gcc -E testA.c | grep -E 'typedef .* (__)?off(64)?_t;'
  __extension__ typedef long int __off_t;
  __extension__ typedef __quad_t __off64_t;
  typedef __off64_t off_t;

  $ gcc -E testB.c | grep -E 'typedef .* (__)?off(64)?_t;'
  __extension__ typedef long int __off_t;
  __extension__ typedef __quad_t __off64_t;
  typedef __off_t off_t;

this on powerpc-linux (GNU libc 2.6.1). In general the standards (POSIX, SUS) specify that "feature test" macros are only effective if defined before the very first C-library header is included.

 
2738 code fixed 2007 Oct anonymous   2007 Oct   2 2 vfsUnlink breaks when building with OS_OTHER=1 edit
I'm building with OS_OTHER=1 defined. That means there is no default vfs implemented. When I register my first vfs, sqlite3_vfs_register() calls vfsUnlink(). As a result of the OS_OTHER=1 define, vfsList is NULL. This causes a NULL pointer to be dereferenced. Note that I believe the same problem would happen if I were to unregister all my vfs nodes for some reason, then register a new one. From vfsUnlink():

  if( vfsList==pVfs ){
    vfsList = pVfs->pNext;
  }else{
    sqlite3_vfs *p = vfsList;
    while( p->pNext && p->pNext!=pVfs ){
      p = p->pNext;
    }
    if( p->pNext==pVfs ){
      p->pNext = pVfs->pNext;
    }
  }

I believe "}else{" should be "}else if ( vfsList != NULL ){"

2007-Oct-22 20:53:13 by anonymous:
On a related note, sqlite3_vfs_unregister() asserts that vfsList is non-NULL before exiting. If built with OS_OTHER=1, it may be possible that a client such as myself unregisters all the vfs objects at some point, before registering a new one. (This should not be a problem if no new databases are opened while no vfs objects are registered.) It seems that the assert should only be used if OS_OTHER is not defined.
 
2737 code fixed 2007 Oct anonymous   2007 Oct   4 4 save 35 bytes in VACUUM with new 2 argument substr() edit
Index: src/vacuum.c
===================================================================
RCS file: /sqlite/sqlite/src/vacuum.c,v
retrieving revision 1.73
diff -u -3 -p -r1.73 vacuum.c
--- src/vacuum.c        29 Aug 2007 12:31:28 -0000      1.73
+++ src/vacuum.c        20 Oct 2007 17:32:10 -0000
@@ -136,18 +136,18 @@ int sqlite3RunVacuum(char **pzErrMsg, sq
   ** in the temporary database.
   */
   rc = execExecSql(db,
-      "SELECT 'CREATE TABLE vacuum_db.' || substr(sql,14,100000000) "
-      "  FROM sqlite_master WHERE type='table' AND name!='sqlite_sequence'"
-      "   AND rootpage>0"
+      "SELECT 'CREATE TABLE vacuum_db.' || substr(sql,14) "
+      "FROM sqlite_master WHERE type='table' AND name!='sqlite_sequence' "
+      "AND rootpage>0"
   );
   if( rc!=SQLITE_OK ) goto end_of_vacuum;
   rc = execExecSql(db,
-      "SELECT 'CREATE INDEX vacuum_db.' || substr(sql,14,100000000)"
-      "  FROM sqlite_master WHERE sql LIKE 'CREATE INDEX %' ");
+      "SELECT 'CREATE INDEX vacuum_db.' || substr(sql,14) "
+      "FROM sqlite_master WHERE sql LIKE 'CREATE INDEX %'");
   if( rc!=SQLITE_OK ) goto end_of_vacuum;
   rc = execExecSql(db,
-      "SELECT 'CREATE UNIQUE INDEX vacuum_db.' || substr(sql,21,100000000) "
-      "  FROM sqlite_master WHERE sql LIKE 'CREATE UNIQUE INDEX %'");
+      "SELECT 'CREATE UNIQUE INDEX vacuum_db.' || substr(sql,21) "
+      "FROM sqlite_master WHERE sql LIKE 'CREATE UNIQUE INDEX %'");
   if( rc!=SQLITE_OK ) goto end_of_vacuum;

   /* Loop through the tables in the main database. For each, do
 
2736 build active 2007 Oct anonymous   2007 Oct   2 2 build problems on freebsd edit
on freebsd:

--disable-threads does not work. it is accepted as a valid option but no defs are added to the makefile -lgcc needs to be included in SHLIB_LD_LIBS pkgIndex.tcl is not built when -DSQLITE_THREADSAFE=0 is added manually, this causes the install target to fail.

2007-Oct-17 21:16:23 by anonymous:
I forgot to mention this is with the TEA version
 
2735 code closed 2007 Oct anonymous   2007 Oct nmadhav 3 1 Error : Unrecognized Token edit
Hi, I am trying to Create a table in the sqlite DB. The table name is 9I0_STATIC_101607 and I get an error Unrecognized token : 9I0_STATIC_101607 But if I create a table with name STATIC_9I0_101607, my code works fine. It is something to do with the table name starting with number. Is there any kind of restriction with table name starting with a digit??

Thanks for your help. Regards, Mayura.

2007-Oct-17 17:07:43 by drh:
This is not a bug report. This is a support request. Please see

   http://www.sqlite.org/support.html

for information on how to receive support for SQLite. Please use these ticketing system to report bugs only. Thanks.

 
2734 code closed 2007 Oct anonymous   2007 Oct   3 1 2 bug under the wince edit
the test version is 3.5.1  thank you!
1. in os_win.c line:1165
	   /*
	   samsting_xie change the last line: zFilename to zName.
	   the zFilename is not declared, so i change it to zName,
	   maybe it's write error? the old last line is:
			&& !winceCreateLock(zFilename, pFile)
	   */
2. in os_win.c line:1341
  return SQLITE_OK;
  /*
  samsting_xie add the last line: return SQLITE_OK;
  if OS_WINCE defined 1, then this function is not return value.
  it's maybe return a unknown value.
  */
Duplicate. See, for example, #2718, #2711, #2710, #2702, #2700, #2683.
 
2733 code fixed 2007 Oct anonymous   2007 Oct   1 1 OP_Real and OP_Int64 inefficient for constants edit
Let's say I have a table with 50 million rows:

  CREATE TABLE t1(a real);

If I perform a query like:

  select -3.14159265 * a from t1;

then the string "-3.14159265" will be converted to a double 50 million times via OP_Real -> sqlite3VdbeRealValue -> sqlite3AtoF.

  explain select -3.14 * a from t1;
  ...
  5|Real|0|0|-3.14    -- converts string to double each loop pass!
  6|Column|0|0|# t1.a
  7|RealAffinity|0|0|
  8|Multiply|0|0|
  9|Callback|1|0|
  10|Next|0|5|
  ...

This is actually quite computationally expensive and unnecessary. If OP_Real and OP_Int64 would store the double/int64 values into P1/P2 at expression parse time then they would not have to be reparsed to a double/int64 each pass in the loop at vdbe runtime.

2007-Oct-18 03:37:47 by anonymous:
Proof of concept patch:

http://www.mail-archive.com/sqlite-users%40sqlite.org/msg28390.html

http://marc.info/?l=sqlite-users&m=119266402408308&q=p3

 
2732 code fixed 2007 Oct anonymous   2007 Oct   2 2 sqlite3Step crashes if called with a null argument (easy fix) edit
With the reorganisation of the code in 3.5 to use a memory manager accessed via the db structure, sqlite3Step in vdbeapi.c can now cause an access violation if called with a null Vdbe structure.

Unfortunately pysqlite quite often calls this routine with a null statement and so can crash if you execute an empty query.

The obvious fix is to move the two marked lines below the null check.

static int sqlite3Step(Vdbe *p){ sqlite3 *db; int rc;

  /* Assert that malloc() has not failed */
->  db = p->db;
->  assert( !db->mallocFailed );

  if( p==0 || p->magic!=VDBE_MAGIC_RUN ){
    return SQLITE_MISUSE;
  }
2007-Oct-17 01:38:03 by drh:
This is really a bug in pysqlite. Have you filed a bug report there?


2007-Oct-17 22:47:06 by anonymous:
Just filed a bug there. I had to verify a few things first.

In hindsight, the check for a null statement should perhaps be made in sqlite3_step rather than sqlite3Step. That would be more useful, as it appears to be the case that the mutex check causes an exception (I built the version I first observed this in without mutexes, and then fixed the bug in my copy of pysqlite)

 
2731 code closed 2007 Oct anonymous   2007 Oct   1 1 sqlite3ExprCode TK_UMINUS inefficient with large negative floats edit
The follow change in expr.c makes my program (which deals with large negative floating point numbers) run 14 times faster.

14 seconds originally to just 1 second after this patch.

Assuming the fix is correct, can you please apply it? make test runs with 0 errors with this.

Thanks.

Index: src/expr.c
===================================================================
RCS file: /sqlite/sqlite/src/expr.c,v
retrieving revision 1.313
diff -u -3 -p -r1.313 expr.c
--- src/expr.c  18 Sep 2007 15:55:07 -0000      1.313
+++ src/expr.c  16 Oct 2007 23:54:17 -0000
@@ -1879,7 +1879,10 @@ void sqlite3ExprCode(Parse *pParse, Expr
       assert( pLeft );
       if( pLeft->op==TK_FLOAT || pLeft->op==TK_INTEGER ){
         Token *p = &pLeft->token;
-        char *z = sqlite3MPrintf(pParse->db, "-%.*s", p->n, p->z);
+        char *z = sqlite3_malloc(p->n+2);
+        z[0] = '-';
+        memcpy(z+1, p->z, p->n);
+        z[p->n+1] = 0;
         if( pLeft->op==TK_FLOAT ){
           sqlite3VdbeOp3(v, OP_Real, 0, 0, z, p->n+1);
         }else{
2007-Oct-17 01:34:39 by drh:
FWIW, the patch is not correct. It does not work if sqlite3_malloc fails and returns a NULL pointer. But it isn't too hard to fix.

I'm curious, though, what kind of program you are running where this makes a 14x speed improvement. One wonders if you couldn't do 100x faster using sqlite3_bind_double() instead. We could go through and make patches like this one all over the place, in order to make this or that special case run a little faster. But that would make the library footprint larger. The sqlite3MPrintf() call in this context is there for reasons of space efficiency, not CPU time. Some years ago, we when through this huge effort to reduce the footprint of the library by replacing the expanded string construction code like you are inserting with instances of sqlite3MPrintf(). I am reluctant to undo that effort for a single special case that could probably be handled more efficiently using sqlite3_bind_double().


2007-Oct-17 02:21:17 by anonymous:
I have a script to generate ASCII SQL statements which I use in conjunction with the sqlite3 shell and MySQL to populate huge tables.

I use the multi-insert patch on the mailing list for tables that hold millions of rows of negative numbers, with anywhere from 3 to 20 columns.

http://www.mail-archive.com/sqlite-users%40sqlite.org/msg28337.html

  INSERT INTO BIGTABLE VALUES
   (-334712687065.09, -334712687065.12, -334712687065.13),
   (-334712687065.09, -334712687065.12, -334712687065.13),
   ... 10 thousand rows ...
   (-334712687065.09, -334712687065.12, -334712687065.13),
   (-334712687065.09, -334712687065.12, -334712687065.13);

With the mulit-insert patch and the patch in this ticket, it is 25% faster for inserting than using the equivalent INSERT commands within a transaction:

  BEGIN
  INSERT INTO BIGTABLE VALUES(-334712687065.09, -334712687065.12, -334712687065.13);
  ... 10 thousand rows ...
  INSERT INTO BIGTABLE VALUES(-334712687065.09, -334712687065.12, -334712687065.13);
  COMMIT;

I have to have no more than 10,000 rows per insert because it blows the stack due to sqlite3Select -> multiSelect recursion. I had to bump a few other sqliteLimits as well to get more than a few thousand rows at a time.

I suppose the replacement of the sqlite3MPrintf could be useful in other scenarios as well.


2007-Oct-17 02:27:30 by anonymous:
The bind double approach is not 100X faster, by the way. It's only around 1.3X faster than the individual ASCII inserts in my tests. But it doesn't matter for my purposes, because it's not MySQL compatible anyway.


2007-Oct-17 03:04:11 by anonymous:
If the string were not pre-prepended with a '-' char, but instead parsed into a double, and the double simply negated it would be much faster and avoid a needless memory allocation/deallocation. To facilitate this, OP_Real would have to be changed or a new OP code created to accommodate an 8 byte double argument - possibly via straddling P1 and P2.


2007-Oct-23 16:00:39 by anonymous:
Fixed by Check-in [4507]
 
2730 code closed 2007 Oct anonymous   2007 Oct anonymous 3 2 Can't compile for WindowsCE edit
There is a compilation bug in os_win.c line 1165 (then compile for WindowsCE) "sqlite3.5_\os_win.c(1165) : error C2065: 'zFilename' : undeclared identifier"

the code #if OS_WINCE if( (flags & (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB)) == (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB) && !winceCreateLock(zFilename, pFile)

seems to be

#if OS_WINCE if( (flags & (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB)) == (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB) && !winceCreateLock(zName, pFile)

Best regards, Yuri Noyanov

Duplicate of #2648, #2700, #2702, and #2718.
 
2729 doc active 2007 Oct anonymous   2007 Oct   1 1 Lemon: %fallback, %wildcard, and @X uncodumented edit
I noticed that the lemon documentation does not mention the %fallback and %wildcard directives. Both are in the code and are apparently doing useful work in SQLite's parse.y. Can other users benefit from them as well?

The symbol @X is also undocumented. From a source code comment I read that it "If the argument is of the form @X then substituted the token number of X, not the value of X". A short documentation example would help to understand where and how it can be useful to apply this syntax.

Are there other nice but undocumented Lemon goodies lacking documentation?

 
2728 code active 2007 Oct anonymous   2007 Oct   4 4 Some indexes could contain pointers not the data edit
It would be nice if there was a class of index(-column) that would contain a reference to the data in (in the table) rather than a copy of the data.

This is useful where you have a need to index large strings or blobs and the size penalty of having 2+ copies ends up being very expensive.

Consider the cost of the implied index on:

create table t1 ( c1 BLOB UNIQUE);

and store 100s of 1MB objects. The raw file ends up being twice as large as you ideally would like it to be.

WHEN to use such indexes isn't entirely clear to me, not is the syntax for doing it in general, 30s of thought I can come up with:

  create table t1 ( c1 BLOB &UNIQUE);

and say

  create index idx1 on t2(c1, &c2, c3);

(meaning c2 would be done via reference, c1 & c3 as copies of the data). It's FAR from clear what other databases do here (I didn't look).

This would also be useful in cases where I have multiple indexes on tables like email (headers) and values which can be very large (100s of bytes in some cases (Subject:) and I might have 5 or more indexes, means a 100-byte column will take over 600 bytes (+ padding)). Allowing for references would be slower in some cases but faster in others because of the smaller footprint and much grater CPU cache utilization.

2007-Oct-14 15:27:12 by anonymous:
This enhacement has been discussed on the mailing list before, and would break sqlite3 file format compatability.


2007-Oct-15 02:42:02 by anonymous:
This is non standard sql.

Also, the random access to pages to check indexes constraints (unique indexes, also primary keys) will trash the entire database performance.

If you need small index disk usage, consider using hashes to your data keys. it&#180;s the best that you could do to solve your problem.


2007-Oct-18 06:07:58 by anonymous:
Using a hash is what I've tried doing in a couple of cases. It's not very ideal. For one thing is ordering is messed up.

I wonder about a virtual index concept, where you can define a function that takes the column value and returns something to actually store in the index?

I'm told oracle has this feature, so whilst it's non-standard there is some precedent (albeit in a very different space).

 
2727 build active 2007 Oct anonymous   2007 Oct   4 4 building with -malign-double causes strange behavior edit
sqlite 3.5.1 amalgamation has problems when enabling a wide set of compiler features on gcc (GCC) 4.1.2 (Ubuntu 4.1.2-0ubuntu4) on linux/i686 /w glibc-2.5 strange behavior occurs.

typical strangeness is that SQLITE_FULL is returned from sqlite3_prepare_v2() (called immediately after a statement was created with sqlite3_mprintf). This happens even though there is a reasonable amount of disk space free (6G).

the compile line looks like:
cc -g -pg -O1 -march=i686 -msse2 -malign-double -m128bit-long-double -momit-leaf-frame-pointer -minline-all-stringops -D_XOPEN_SOURCE=520 '-DVERSION="0.10r276M"' -Isqlite/ -DSQLITE_OMIT_LOAD_EXTENSION -DTHREADSAFE=0 -DSQLITE_OMIT_EXPLAIN -DSQLITE_ENABLE_COLUMN_METADATA -c -o sqlite/sqlite3.o sqlite/sqlite3.c

and valgrind reports:
==15323== Use of uninitialised value of size 4
==15323== at 0x80585B7: insertElement (sqlite3.c:13072)
==15323== by 0x807366E: sqlite3HashInsert (sqlite3.c:13290)
==15323== by 0x809FE72: unixOpen (sqlite3.c:15403)
==15323== by 0x80579A7: sqlite3OsOpen (sqlite3.c:8210)
==15323== by 0x806B12F: sqlite3BtreeFactory (sqlite3.c:21317)
==15323== by 0x80767BA: openDatabase (sqlite3.c:71237)
==15323== by 0x8076BD1: sqlite3_open (sqlite3.c:71337)
==15323== by 0x804BEAE: db_init (dbmgr.c:81)
==15323== by 0x804CE64: main (main.c:59)

my dbmgr.c:81:db_init() is: res=sqlite3_open(filename, &system_db);

and system_db=NULL and filename="zomg.sqlite" (string literal). so the parameters seem normal. if I turn off -malign-double everything works fine. (this "bug" also seems to be on 3.4.1 amalgamation)

2007-Oct-14 04:35:16 by anonymous:
This is a compiler issue. -malign-double creates problems for most programs. What are you trying to accomplish?
 
2726 code closed 2007 Oct anonymous   2007 Oct   4 3 solaris 10 compilation B_FALSE, B_TRUE clash from sys/types.h edit
% make
386-pc-solaris2.10-gcc -O3 -pipe -o lemon ./tool/lemon.c
./tool/lemon.c:111: error: redeclaration of enumerator &#8216;B_FALSE&#8217;
/usr/include/sys/types.h:176: error: previous definition of &#8216;B_FALSE&#8217; was here
./tool/lemon.c:111: error: redeclaration of enumerator &#8216;B_TRUE&#8217;
/usr/include/sys/types.h:176: error: previous definition of &#8216;B_TRUE&#8217; was here
make: *** [lemon] Error 1

% sed -e 176\!d /usr/include/sys/types.h{linebreak} typedef enum { B_FALSE, B_TRUE } boolean_t;

% sed -e 111\!d tool/lemon.c
typedef enum {B_FALSE=0, B_TRUE} Boolean;

even when the enum is only set when it is not yet defined, I guess this results in conflicts in the code because Boolean may not equal boolean_t.

I suggest to rename B_FALSE and B_TRUE to something which makes it unique for the sqlite case.

Duplicate of #2583
 
2725 code active 2007 Oct anonymous   2007 Oct   1 1 memory leak in sqlite3_open_v2() when it fails edit
only happens with

flags = SQLITE_OPEN_READWRITE;

and when

  res = sqlite3_open_v2(sourcename, &conn, flags, NULL);

seems to leak 674 bytes per call

2007-Oct-15 07:07:07 by danielk1977:
Are you calling sqlite3_close(conn) after the error occurs?

All calls to sqlite3_open_v2() need to be matched by a call to sqlite3_close(), even if an error occurs.

 
2724 code closed 2007 Oct anonymous   2007 Dec drh 2 2 Load Ext api missing error_toobig function. edit
Function call for sqlite3_error_toobig is not in the sqlite3Ext.h

void (result_error_toobig)(sqlite3_context);

#define sqlite3_result_error_toobig sqlite3_api->result_error_toobig

Without these, externally loadded libs that would like to use error_toobig will not load into sqlite

2007-Dec-13 18:51:28 by drh:
See ticket #2610.
 
2723 new closed 2007 Oct anonymous   2007 Oct anonymous 5 4 After execution of SQL, the Data Returned should display the col.name edit
Hi! After executing a SQL command, the program shows the results as a table with columns. The first row ist allways grey but empty. Couldn't the program display the selected entities (or their declared AS-alias) as column headers in the result table? Yours sincerely M. Heinrich, M&#252;nchen
2007-Oct-12 18:19:38 by drh:
Do you mean in the CLI? Just type

   .header on

And the headers will appear.

 
2722 code fixed 2007 Oct anonymous   2007 Oct   1 1 data mis-alignment in UTF16 collation callback arguments edit
We're using a Unicode build of SQLite, and found that internally, the 'NOCASE' collation was not Unicode aware -- it would convert any wide strings to UTF-8 when needed, and compare those. For performance, we would like to implement a WNOCASE for wide strings.

However, upon implementing this, we found that the pointers passed to our collation function are not wide character (short-) aligned. On our platform we receive data mis-alignment errors when comparing the strings.

We have already implemented similiar sqlite wide string functions (WLIKE, WCONTAINS), and we never receive mis-aligned strings there (are those guaranteed to be aligned); it would be helpful if this restriction could be applied to collation callbacks.

 
2721 code active 2007 Oct anonymous   2007 Dec   2 1 if db file is in a folder with non-ansi character some functions fail edit
If database file is located in directory with some non-ANSI characters (in my case with a Russian subdirectory c:\&#1052;&#1086;&#1080; &#1076;&#1086;&#1082;&#1091;&#1084;&#1077;&#1085;&#1090;&#1099;\Data_Jobs), or it's name is non-ansi. Some functions fail to execute sql.

For example (with defined UNICODE):

TCHAR sql[512];
_stprintf(sql, _T("INSERT INTO tab_SurveyedPoints (name, comment, code,")
		_T("coordinatetype, b, l, h, solutiontype, sigmah, sigmav)")
		_T(" VALUES ('%s','%s','%s',0,%lf,%lf,%lf,0,%lf,%lf);"),
		point.m_name.c_str(), point.m_description.c_str(), point.m_code.c_str(),
		point.m_coordinates.b, point.m_coordinates.l, point.m_coordinates.h,
		point.m_sigmah, point.m_sigmav);
int rc1 = sqlite3_prepare16(m_db, sqlfmt, -1, &stmt, (const void**)&pszTail);
rc != SQLITE_OK

But if I move the file to c:\My documents\Data_Jobs this works ok.

It's improbable behaviour, but I can't work around yet. Although, prepare() functions work ok as well in both cases.

Yuri Noyanov.

2007-Oct-11 19:33:34 by drh:
All string arguments to SQLite, and especially filename arguments, must be UTF-8 or UTF-16 (depending on the function). If you use string parameters which are not UTF-8 or UTF-16 (as appropriate) then the behavior of SQLite is undefined and probably not what you want.


2007-Oct-12 04:25:56 by anonymous:
but ALL programs to handle SQLite DBs (SQLIteBrowser, SQLite Control) fail to handle the files as well. Till I move the file to different directory !!!


2007-Oct-12 04:27:54 by anonymous:
Also I must note, that I CAN open the database, I CAN execute some SQLs with sqlite_prepare function OK. But sqlite_prepare16 FAILS if I just rename my database !!!


2007-Oct-12 04:31:46 by anonymous:
Also note to make my issue clearer:

sqlite_prepare16() with the same code either works OK either doesn't work. depends on database filename or folder path. The database is opened OK in both cases (I used utf8 conversion). sql_prepare() works ok in both cases.


2007-Oct-13 06:37:43 by anonymous:
That appears to be only with INSERT sql statement. Both SELECT and UPDATE work fine with sqlite_prepare16.
 
2720 code closed 2007 Oct anonymous   2007 Oct   1 1 WinCE - Can't write edit
Application which was working correctly with one of the last sqlite 3.3.16 fails on 3.5.1 to write anything to the database - no create tables and no inserts if the db existed beforehand. It seems to happen always, with either UTF-8 or UTF-16 encoding.
2007-Oct-11 07:26:49 by anonymous:
I meant, "...one of the last 3.3.x (I think 3.3.16)..."


2007-Oct-11 12:20:13 by drh:
The developers have no ability to compile or run on winCE, much less any ability to debug on winCE. All winCE support is community provided.

There have been many patches to the winCE code since 3.5.1 was released. The writer of this ticket previously reported two other problems that have already been fixed. Several other problems have also been fixed since 3.5.1. We are guessing that this "Can't write" problem will probably go away if the writer will simply get the latest code out of CVS. If not, then there is nothing really we can do about it unless we have patches to fix the problem.

 
2719 code closed 2007 Oct anonymous   2007 Oct   3 1 WinCE winFullPathName doesn't return a value edit
#if OS_WINCE

  /* WinCE has no concept of a relative pathname, or so I am told. */

  sqlite3_snprintf(pVfs->mxPathname, zFull, "%s", zRelative);<br>

#endif

Duplicate of #2702
 
2718 build closed 2007 Oct anonymous   2007 Oct   2 1 WinCE compile error edit
On function winOpen

#if OS_WINCE
  if( (flags & (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB)) ==
               (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB)
       && !winceCreateLock(zFilename, pFile)
  ){
    CloseHandle(h);
    free(zConverted);
    return SQLITE_CANTOPEN;
  }
  if( dwFlagsAndAttributes & FILE_FLAG_DELETE_ON_CLOSE ){
    pFile->zDeleteOnClose = zConverted;
  }else
#endif
That zFilename of the winceCreateLock call doesn't exist. I guess it should be zName instead
Duplicate of #2700
 
2717 doc closed 2007 Oct anonymous   2007 Dec anonymous 5 5 Document clarification around SQLITE_OMIT_MEMORY_ALLOCATION edit
In the " Moving From SQLite 3.4.2 to 3.5.0" document, section 3.0 it states that "These routines are provided for use by the application. SQLite never invokes them itself. So if the application is providing its own memory allocation subsystem, it can omit these interfaces if desired."

This doesn't seem to be correct unless SQLITE_OMIT_LOAD_EXTENSION is defined. If this value is not defined, then the function pointers are assigned in a structure (sqlite3_apis). As stated, they are not used by SQLite proper, but must be defined. In this case, I am unable to link without either defining the functions or building with SQLITE_OMIT_LOAD_EXTENSION defined.

2007-Oct-11 19:45:36 by anonymous:
It appears that the sqlite3_memory_used() function is required in sqlite3_soft_heap_limit() so my previous comments were not entirely correct. I'm still having a link problem even with the define.


2007-Dec-13 18:48:21 by drh:
SQLITE_OMIT_MEMORY_ALLOCATION is no longer supported.
 
2716 new active 2007 Oct anonymous   2007 Oct   5 1 Create Clear Command edit
I want a command caled "clear" like in MySQL. This command should erase the screen and then put the sqlite pointer on top of the screen
2007-Oct-11 07:41:45 by anonymous:
How about a cookie instead?


2007-Oct-30 08:02:04 by anonymous:
Clearing the screen and moving the cursor are platform-dependent operations. On Unix they are not only platform-dependent, but also terminal-dependent. Thus such a feature does not really belong in the cross-platform and minimalistic sqlite3 shell (in my opinion).
 
2715 code active 2007 Oct anonymous   2007 Oct   1 1 no authorization needed to remove authorizer edit
there should be a new auth code created and the auth function should be consulted for permission for removal.
2007-Oct-10 01:08:48 by drh:
I'm assuming that this feature request comes from RockShox and that the development language is Tcl.

No. If your adversary has the ability to invoke the interface that removes an authorizer, then you system is already pwned.

What you really need is the ability to [interp alias] the eval method into a safe interpreter. That way you can:

  • Open the database in the main interpreter
  • Set up the authorizer in the main interpreter to invoke a script in the main interpreter
  • Set up the [interp alias] so that the safe interpreter can do [db eval ...] but not [db auth ...]

It seems like an "-interp" option on the "eval" method of the database connection object would likely be the right interface. Or perhaps there should be separate "safeeval" method. Either way, it has been years and years since I have done anything with safe interpreters so I will have to look into what needs to be done to make that happen.


2007-Oct-17 20:11:23 by anonymous:
ok i think i agree with that.

currently you cannot use an interp alias since the target command runs in the target interp and all your variables and commands are in the wrong scope. this means one needs to load sqlite again in the new interp, and sqlite will not load in a safe interp so a regular interp is required.

to be useful, a -interp flag would need to execute in the current scope of the interp and not the global scope.

 
2714 code active 2007 Oct anonymous   2007 Oct drh 3 4 Shell cannot import large files greater than 2 GB [patch] edit
If I issue an .import command within the SQLite shell and the file size is larger than 2 GB, the shell gives the error message "cannot open file: foo.txt".

Putting #define _FILE_OFFSET_BITS 64 before the #include statements in shell.c fixes this problem under Linux.

I tested this solution with a 2.6 GB file under Ubuntu Feisty Fawn 7.04 with Linux kernel 2.6.20-16-generic.

2007-Oct-12 18:48:35 by drh:
I'm not convinced SQLite should be able to import files larger than 2GiB. Does anybody really ever need to read more than 2GiB of SQL text? That is a lot of SQL, don't you think? Is this really a desirable feature? Can you not split it up into two or more smaller files?


2007-Oct-14 09:25:42 by anonymous:
I'm not 100% certain about this but:

  #define _FILE_OFFSET_BITS 64

isn't necessarily correct, better to tweak the makefile(s):

  CFLAGS += $(shell getconf LFS_CLFAGS)

(similar for LDFLAGS) should work on 32/64 bit machines and different platforms.

For shell.c I really wouldn't bother and think this should be left alone. Using .import for load more than 2GB makes me things the shell should be fleshed out and extended for this sort of abuse, handy as it is right now, it's not a full SQL shell but a minimalist shell and a useful example).


2007-Oct-16 14:08:09 by anonymous:
I am the original poster. I use the shell to import CSV files in the 3-4 GiB range on a daily basis. I don't see the point of splitting the files or writing my own importer when the shell does a perfectly fine job once large file support has been activated.

With all due respect, I don't understand the reluctance to enable a useful feature. Shouldn't the user decide whether it is reasonable for his application to import big files? I work in the health care industry processing insurance claims for several very large companies. There's nothing broken or wrong with my application -- big files are just the nature of the data I'm handling.

Neither of you has proposed any disadvantage to enabling LFS in the shell. If there are no drawbacks, why not turn it on? If the core library can open large database files, why shouldn't the shell be able to import large ones?


2007-Oct-16 16:24:22 by anonymous:
For reasons also unknown to me, the authors appear to regard the shell as a largely unused sample program. Based on my personal experience in talking to various sqlite library users, it is the primary administrative front-end for sqlite. In particular, doing backups and merging databases.
 
2713 build fixed 2007 Oct anonymous   2007 Oct   1 5 build fails for 3.5.1 on win32/cygwin - duplicate opcode labels edit
Problem: build fails on compiling vdbe.c =>

src/vdbe.c: In function `sqlite3VdbeExec': src/vdbe.c:1136: error: duplicate case value src/vdbe.c:921: error: previously used here src/vdbe.c:1655: error: duplicate case value src/vdbe.c:900: error: previously used here ...

Seems like my generated opcodes.h has lots of duplicates:

#define OP_Multiply 80 /* same as TK_STAR / #define OP_Pull 80 #define OP_Dup 71 #define OP_Lt 71 / same as TK_LT */

I'm running on XP with a c:\cygwin library available.

NB. i'm not a regular SQLLite user and was just trying the library out of curiosity so if no one else reports this then don't sweat it... it may be down to my environment screwing up the build process.

2007-Oct-09 17:15:24 by drh:
Why don't you try compiling the amalgamation instead?


2007-Oct-09 18:06:22 by anonymous:
Original poster - just run "make clean" and try again.


(OP) OK. tracked it down to the initialisation of the tk array in mkopcodeh.awk which is getting strings like "22 " (not sure if this is a space or a CR). the following patch fixes it for me:

   --- mkopcodeh.awk-orig  2007-03-29 19:39:30.000000000 +0100
   +++ mkopcodeh.awk       2007-10-12 15:14:46.980260800 +0100
   @@ -41,7 +41,9 @@

    # Remember the TK_ values from the parse.h file
    /^#define TK_/ {
   -  tk[$2] = $3
   +  num = $3
   +  num += 0 # convert "22 " -> 22
   +  tk[$2] = num
    }

    # Scan for "case OP_aaaa:" lines in the vdbe.c file

ie. ensure that the keys are numeric.

With this patch, the build completes ok.

Thanks.


2007-Oct-12 20:08:41 by anonymous:
It's not an AWK bug in Cygwin. The only way to convert strings to numbers in AWK is to add zero.

Cygwin uses the same version of GNU gawk as Linux. It's the exact same code.


2007-Oct-12 20:43:55 by drh:
Seems like if it were exactly the same code it would work exactly the same. But clearly it does not. I don't know what is going on. Probably some strange DOS \r line ending problems. Presumably the change has fixed the problem.
 
2712 code closed 2007 Oct anonymous   2007 Oct   5 4 Expose a function to change the journal filename location edit
Being able to change the pager journal filename using a nice function ie: sqlite3PagerSetJournal it maybe the following prototype: int sqlite3PagerSetJournal(sqlite3* db, const char *zJournal);

It is intended to maximize disk IOs by splitting IO concerning the journal and the IO concerning the main DB storage. Typically we dedicate a specific disk channel for the journal, which may be a RAM disk like iRAM from Gigabyte, or a SATA2 disk with 16MB cache dedicated, or using a RAID controller with 256MB RAM with Backup Unit. The main DB storage resides on a different channel on SAS disks configured as RAID10. Separating journal and main data file is typically a feature available in common SQL servers : MySQL, MSSQL, ...

2007-Oct-09 15:38:50 by drh:
Changing the name of the rollback journal and especially moving it to a different volume can lead to database corruption following a power failure. See http://www.sqlite.org/ac/atomiccommit.html for additional information.
 
2711 code fixed 2007 Oct anonymous   2007 Oct   1 1 WinCE: flags incorrectly checked edit
One last fix for WinCE: checkin 4479 is correct in the omission of certain flags for WinCE, but the flags are still being checked for later on in winOpen:

if( dwFlagsAndAttributes & FILE_FLAG_DELETE_ON_CLOSE ){
  pFile->zDeleteOnClose = zConverted;
}else ...
Since that flag is no longer set, it needs to read:

if( flags & (SQLITE_OPEN_TEMP_DB | SQLITE_OPEN_TEMP_JOURNAL | SQLITE_OPEN_SUBJOURNAL) ){
  pFile->zDeleteOnClose = zConverted;
}else ...
Finally, on WinCE, since FILE_FLAG_RANDOM_ACCESS is always set, it doesn't make sense to set FILE_FLAG_SEQUENTIAL_SCAN (for journals). At least on CE, it hampered performance in our tests.
 
2710 code fixed 2007 Oct anonymous   2007 Oct   1 1 Typo in code specific to WinCE (checkin 4479) edit
Checkin [4479] introduces a typo in code specific to WiNCE, returning SQLTIE_OK instead of SQLITE_OK
2007-Oct-09 15:21:50 by anonymous:
Also, checkin 4479 is correct in the omission of certain flags for WinCE, but the flags are still being checked for later on in winOpen:

if( dwFlagsAndAttributes & FILE_FLAG_DELETE_ON_CLOSE ){ pFile->zDeleteOnClose = zConverted; }else ...

Since that flag is no longer set, it needs to read:

if( flags & (SQLITE_OPEN_TEMP_DB | SQLITE_OPEN_TEMP_JOURNAL | SQLITE_OPEN_SUBJOURNAL) ){ pFile->zDeleteOnClose = zConverted; }else ...

 
2709 code closed 2007 Oct anonymous   2007 Oct   2 3 Some SQLITE_OMIT_* keys break the sqlite3.c amalgamation compilation edit
I found that SQLITE_OMIT_TRIGGER, SQLITE_OMIT_REINDEX and SQLITE_OMIT_ALTERTABLE keys break the compilation, when defined.
Following patch solved this problem for me.

sqlite-amalgamation-3_5_1.define.patch

--- ./sqlite3.c.orig	Wed Oct  3 18:08:44 2007
+++ ./sqlite3.c	Tue Oct  9 14:52:09 2007
@@ -53246,6 +53266,7 @@
   /* Run the BEFORE and INSTEAD OF triggers, if there are any
   */
   endOfLoop = sqlite3VdbeMakeLabel(v);
+#ifndef SQLITE_OMIT_TRIGGER
   if( triggers_exist & TRIGGER_BEFORE ){

     /* build the NEW.* reference row.  Note that if there is an INTEGER
@@ -53309,6 +53330,7 @@
       goto insert_cleanup;
     }
   }
+#endif

   /* If any triggers exists, the opening of tables and indices is deferred
   ** until now.
@@ -62064,6 +62086,7 @@
     sqlite3VdbeAddOp(v, OP_MemInt, 0, memCnt);
   }

+#ifndef SQLITE_OMIT_TRIGGER
   if( triggers_exist ){
     /* Create pseudo-tables for NEW and OLD
     */
@@ -62129,6 +62152,7 @@
       goto update_cleanup;
     }
   }
+#endif

   if( !isView && !IsVirtual(pTab) ){
     /*
@@ -67517,10 +67541,12 @@
     case 214:
 {sqlite3IdListDelete((yypminor->yy432));}
       break;
+#ifndef SQLITE_OMIT_TRIGGER
     case 231:
     case 236:
 {sqlite3DeleteTriggerStep((yypminor->yy243));}
       break;
+#endif
     case 233:
 {sqlite3IdListDelete((yypminor->yy370).b);}
       break;
@@ -68378,11 +68404,13 @@
   sqlite3DropTable(pParse, yymsp[0].minor.yy373, 0, yymsp[-1].minor.yy46);
 }
         break;
+#ifndef SQLITE_OMIT_VIEW
       case 102:
 {
   sqlite3CreateView(pParse, &yymsp[-7].minor.yy0, &yymsp[-3].minor.yy410, &yymsp[-2].minor.yy410, yymsp[0].minor.yy219, yymsp[-6].minor.yy46, yymsp[-4].minor.yy46);
 }
         break;
+#endif
       case 103:
 {
   sqlite3DropTable(pParse, yymsp[0].minor.yy373, 1, yymsp[-1].minor.yy46);
@@ -68921,6 +68949,7 @@
       case 250:
 {sqlite3Pragma(pParse,&yymsp[-1].minor.yy410,&yymsp[0].minor.yy410,0,0);}
         break;
+#ifndef SQLITE_OMIT_TRIGGER
       case 258:
 {
   Token all;
@@ -68935,6 +68964,7 @@
   yygotominor.yy410 = (yymsp[-6].minor.yy410.n==0?yymsp[-7].minor.yy410:yymsp[-6].minor.yy410);
 }
         break;
+#endif
       case 260:
       case 263:
 { yygotominor.yy46 = TK_BEFORE; }
@@ -68972,6 +69002,7 @@
       case 272:
 { yygotominor.yy243 = 0; }
         break;
+#ifndef SQLITE_OMIT_TRIGGER
       case 273:
 { yygotominor.yy243 = sqlite3TriggerUpdateStep(pParse->db, &yymsp[-3].minor.yy410, yymsp[-1].minor.yy174, yymsp[0].minor.yy172, yymsp[-4].minor.yy46); }
         break;
@@ -68987,6 +69018,7 @@
       case 277:
 {yygotominor.yy243 = sqlite3TriggerSelectStep(pParse->db, yymsp[0].minor.yy219); }
         break;
+#endif
       case 278:
 {
   yygotominor.yy172 = sqlite3PExpr(pParse, TK_RAISE, 0, 0, 0);
@@ -69011,11 +69043,13 @@
       case 282:
 {yygotominor.yy46 = OE_Fail;}
         break;
+#ifndef SQLITE_OMIT_TRIGGER
       case 283:
 {
   sqlite3DropTrigger(pParse,yymsp[0].minor.yy373,yymsp[-1].minor.yy46);
 }
         break;
+#endif
       case 284:
 {
   sqlite3Attach(pParse, yymsp[-3].minor.yy172, yymsp[-1].minor.yy172, yymsp[0].minor.yy386);
@@ -69032,19 +69066,22 @@
       case 287:
 { yygotominor.yy386 = yymsp[0].minor.yy172; }
         break;
+#ifndef SQLITE_OMIT_REINDEX
       case 290:
 {sqlite3Reindex(pParse, 0, 0);}
         break;
       case 291:
 {sqlite3Reindex(pParse, &yymsp[-1].minor.yy410, &yymsp[0].minor.yy410);}
         break;
+#endif
       case 292:
 {sqlite3Analyze(pParse, 0, 0);}
         break;
       case 293:
 {sqlite3Analyze(pParse, &yymsp[-1].minor.yy410, &yymsp[0].minor.yy410);}
         break;
-      case 294:
+#ifndef SQLITE_OMIT_ALTERTABLE
+	  case 294:
 {
   sqlite3AlterRenameTable(pParse,yymsp[-3].minor.yy373,&yymsp[0].minor.yy410);
 }
@@ -69059,6 +69096,7 @@
   sqlite3AlterBeginAddColumn(pParse, yymsp[0].minor.yy373);
 }
         break;
+#endif
       case 299:
 {sqlite3VtabFinishParse(pParse,0);}
         break;

Thank you!
2007-Oct-09 13:22:03 by anonymous:
This is because parse.c has to be regenerated from parse.y. Then, with an altered Makefile with the appropriate options, you have to run "make clean && make sqlite3.c".

Could Lemon be altered to emit the %if/%ifndef/%else/%endif's as their C preprocessor equivalents instead of just not generating the if'd out parse code? This way the amalgamation could work with all OMITs.


2007-Oct-09 15:13:49 by drh:
Changing Lemon to generate "universal" code that works with any combination of OMIT macros is not practical. Any change to the grammer rules typically results in massiver reorganization of the parser tables.

If you want to use OMIT macros, then you need to compile from the canonical source code - not the amalgamation.


2007-Oct-10 15:30:31 by anonymous:
I had a issue like that in previous sqlite 3.1x versions, which I want to omit alter table statement but need to generate a parser from lemon, but makefile did not supported it. there&#180;s a way to configure --d(omit something) to omit some features of sqlite ? I know there are few on it, but not all are supported on sqlite configure script
 
2708 code active 2007 Oct anonymous   2007 Oct   4 2 SQL error:disk I/O error edit
I cross-compile sqlite to embedded Linux,but after I insert data to the table ,it failed.the warning is "SQL error:disk I/O error".
2007-Oct-09 05:12:28 by anonymous:

  Why do you think it is SQLite error ??


2007-Oct-09 05:46:06 by danielk1977:
We'll need a bit more data than that to figure this out. Did earlier SQLite versions work? Can you post the entire output of the compile process so that we can see if there are any clues there? Can you run strace so that we can see if there really is an IO error, or at least when SQLite believes there to be one?
 
2707 code closed 2007 Oct anonymous   2007 Oct appledev 1 3 SQLite on Mac OS X seems to have a 4GB database limit edit
This is on Mac OX 10.4 using Apple's Core Data: I run into a strange error, always when I try to save a managed- object-context and the file-size is already at the 4GB limit:

An error saving the context: NSError "An error occurred while saving." Domain=NSCocoaError Domain Code=134030 UserInfo={NSUnderlyingException = disk I/O error; }

2007-Oct-09 05:12:47 by anonymous:

    Why do you think it is SQLite error ??


2007-Oct-09 06:52:50 by danielk1977:
What kind of file system are you using for the database?

I used the attached script to create a 5.5 GB database on HFS+, then checked it using the integrity check and everything seemed Ok.

If it's not a simple file-system limitation, is is possible for you to post some code that demonstrates the issue? Thanks.


2007-Oct-12 18:46:31 by drh:
Unable to reproduce. Submitter does not respond.
 
2706 warn closed 2007 Oct anonymous   2007 Oct danielk1977 4 4 make test aborts when built with SQLITE_OMIT_ATTACH edit
testfixture aborts during make test when it is compiled with -DSQLITE_OMIT_ATTACH=1

attach-4.5...
Error: no such table: db2.t3
./testfixture: near "DETACH": syntax error
    while executing
"db eval {
  DETACH db2;
}"
    ("uplevel" body line 1)
    invoked from within
"uplevel [list $db eval $sql]"
    (procedure "execsql" line 3)
    invoked from within
"execsql {
  DETACH db2;
}"
    (file "./test/attach.test" line 505)
    invoked from within
"source $testfile"
    ("foreach" body line 5)
    invoked from within
"foreach testfile [lsort -dictionary [glob $testdir/*.test]] {
  set tail [file tail $testfile]
  if {[lsearch -exact $EXCLUDE $tail]>=0} continue
  if..."
    (file "./test/quick.test" line 93)
make: *** [test] Error 1
 
2705 code active 2007 Oct anonymous   2007 Oct   4 4 testfixture unresolved externals with SQLITE_OMIT_GET_TABLE edit
Cannot build/run "make test" with -DSQLITE_OMIT_GET_TABLE due to testfixture link error:

In function `test_get_table_printf':
./src/test1.c:526: undefined reference to `sqlite3_get_table'
./src/test1.c:541: undefined reference to `sqlite3_free_table'
collect2: ld returned 1 exit status
make: *** [testfixture] Error 1
 
2704 code active 2007 Oct anonymous   2007 Oct   4 4 "make test" aborts before completion with SQLITE_OMIT_BLOB_LITERAL edit
When compiled with -DSQLITE_OMIT_BLOB_LITERAL make test aborts with this error:

substr-2.5.2... Ok
./testfixture: near "'61626364656667'": syntax error
    while executing
"db eval "
    DELETE FROM t1;
    INSERT INTO t1(b) VALUES(x'$hex')
  ""
    (procedure "subblob-test" line 2)
    invoked from within
"subblob-test 3.1 61626364656667 1 1 61"
    (file "./test/substr.test" line 86)
    invoked from within
"source $testfile"
    ("foreach" body line 5)
    invoked from within
"foreach testfile [lsort -dictionary [glob $testdir/*.test]] {
  set tail [file tail $testfile]
  if {[lsearch -exact $EXCLUDE $tail]>=0} continue
  if..."
    (file "./test/quick.test" line 93)
make: *** [test] Error 1
 
2703 code active 2007 Oct anonymous   2007 Oct   3 4 make test does not work with SQLITE_OMIT_FLOATING_POINT edit
make test cannot run with SQLITE_OMIT_FLOATING_POINT. Shouldn't these tests be skipped?

./src/test1.c:1255: warning: passing argument 3 of ‘Tcl_GetDouble’ from incompatible pointer type
./src/test1.c: In function ‘sqlite3_mprintf_scaled’:
./src/test1.c:1284: warning: passing argument 3 of ‘Tcl_GetDouble’ from incompatible pointer type
./src/test1.c: In function ‘test_bind_double’:
./src/test1.c:2607: warning: passing argument 3 of ‘Tcl_GetDoubleFromObj’ from incompatible pointer type
./src/tclsqlite.c: In function ‘tclSqlFunc’:
./src/tclsqlite.c:728: warning: passing argument 3 of ‘Tcl_GetDoubleFromObj’ from incompatible pointer type
./src/tclsqlite.c: In function ‘DbObjCmd’:
./src/tclsqlite.c:1609: warning: passing argument 3 of ‘Tcl_GetDoubleFromObj’ from incompatible pointer type

...

$ ./testfixture test/select1.test
select1-1.1... Ok
select1-1.2... Ok
select1-1.3... Ok
select1-1.4... Ok
select1-1.5... Ok
select1-1.6... Ok
select1-1.7... Ok
select1-1.8... Ok
select1-1.8.1... Ok
select1-1.8.2... Ok
select1-1.8.3... Ok
./testfixture: near ".": syntax error
    while executing
"db eval {INSERT INTO test2(r1,r2) VALUES(1.1,2.2)}"
    ("uplevel" body line 1)
    invoked from within
"uplevel [list $db eval $sql]"
    (procedure "execsql" line 3)
    invoked from within
"execsql {INSERT INTO test2(r1,r2) VALUES(1.1,2.2)}"
    (file "test/select1.test" line 69)
 
2702 code fixed 2007 Oct anonymous   2007 Oct   1 1 Errors in CreateFileW function use on WinCE edit
CreateFileW function in WindowsCE uses only subspace of flags of desktop windows' version and reports errors, when passed flag that can't be dispatched. For 3.5.1 this means that any attempt to open/create temporary file on WCE fails.
Also there is bad argument name passed in winOpen to winceCreateLock.
Also, there is 'return' missing in winFullPathname when building for WCE, so any attempt to open file fails.

Here is the patch solving all(for me) filesystem errors on WindowsCE.

sqlite-amalgamation-3_5_1.ce.patch
----------------------------------
--- ./sqlite3.c.orig	Wed Oct  3 18:08:44 2007
+++ ./sqlite3.c	Mon Oct  8 17:04:16 2007
@@ -18730,17 +18730,24 @@
   }
   if( flags & (SQLITE_OPEN_TEMP_DB | SQLITE_OPEN_TEMP_JOURNAL
                     | SQLITE_OPEN_SUBJOURNAL) ){
+#if OS_WINCE
+    dwFlagsAndAttributes = FILE_ATTRIBUTE_HIDDEN;
+#else
     dwFlagsAndAttributes = FILE_ATTRIBUTE_TEMPORARY
                                | FILE_ATTRIBUTE_HIDDEN
                                | FILE_FLAG_DELETE_ON_CLOSE;
+#endif
   }else{
     dwFlagsAndAttributes = FILE_ATTRIBUTE_NORMAL;
   }
   if( flags & (SQLITE_OPEN_MAIN_DB | SQLITE_OPEN_TEMP_DB)){
     dwFlagsAndAttributes |= FILE_FLAG_RANDOM_ACCESS;
-  }else{
+  }
+#if !OS_WINCE
+  else{
     dwFlagsAndAttributes |= FILE_FLAG_SEQUENTIAL_SCAN;
   }
+#endif
   if( isNT() ){
     h = CreateFileW((WCHAR*)zConverted,
        dwDesiredAccess,
@@ -18786,7 +18793,7 @@
 #if OS_WINCE
   if( (flags & (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB)) ==
                (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB)
-       && !winceCreateLock(zFilename, pFile)
+       && !winceCreateLock(zName, pFile)
   ){
     CloseHandle(h);
     free(zConverted);
@@ -18957,6 +18964,7 @@
 #if OS_WINCE
   /* WinCE has no concept of a relative pathname, or so I am told. */
   sqlite3_snprintf(pVfs->mxPathname, zFull, "%s", zRelative);
+  return SQLITE_OK;
 #endif

 #if !OS_WINCE && !defined(__CYGWIN__)

Thank you!
 
2701 new active 2007 Oct anonymous   2007 Oct   5 5 Make INSERT-ing multiple rows MySQL-compatible edit
SQLite syntax allows to insert only one row with
    insert into test (a, b, c) values (1, 2, 3);

MySQL allows to insert multiple with

    insert into test (a, b, c) values (1, 2, 3), (4, 5, 6), (7, 8, 9) -- etc

But SQLite is also capable of inserting multiple by using INSERT...SELECT:

    insert into test (a, b, c) select 1, 2, 3 union select 4, 5, 6 -- etc

It would be nice to make INSERT statement syntactically compatible with MySQL, allowing to insert multiple rows with VALUES clause.

It can be implemented by simply translating multiple 'VALUES ()()()' to 'select union' - no serious change required at all.

2007-Oct-08 21:45:05 by anonymous:
You mean "UNION ALL", not "UNION".

UNION would remove duplicate rows, and create an ephemeral table that you don't want because it's less efficient.

Your idea is a good one and could be implemented largely in the parser. The number of VDBE opcodes would be quite large for such a statement. I wonder if that would present a problem.


2007-Oct-14 08:12:11 by anonymous:
Patch implementing multi-row INSERT statements against 3.5.1 source tree:

http://www.mail-archive.com/sqlite-users%40sqlite.org/msg28337.html

 
2700 code fixed 2007 Oct anonymous   2007 Oct   1 1 invalid name in winOpen when compiling under WinCE edit
in os_win.c:1165, WinCE specific code to create lock still refers to former zFilename variable whereas argument is now named zName.

  --- os_win.c-orig       2007-10-07 22:04:02.000000000 +0200
  +++ os_win.c    2007-10-07 23:32:51.000000000 +0200
  @@ -1162,7 +1162,7 @@
   #if OS_WINCE
     if( (flags & (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB)) ==
                  (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB)
  -       && !winceCreateLock(zFilename, pFile)
  +       && !winceCreateLock(zName, pFile)
     ){
       CloseHandle(h);
       free(zConverted);
 
2699 code fixed 2007 Oct anonymous   2007 Oct   2 2 Performance issues in 3.5.1 (Windows) edit
As reported on mailing list: http://www.mail-archive.com/sqlite-users%40sqlite.org/msg28164.html

"Forcibly adding FILE_FLAG_RANDOM_ACCESS to the flags in os_win.c's winOpen fixed it. It looks like in 3.4.2, that flag was set on all files (journals and db's), while in 3.5, it (was) only being set on db's. Adding it to journals too improves performance almost tenfold. This may also help performance on desktop Windows."

 
2698 code fixed 2007 Oct anonymous   2007 Oct   3 4 memory leak in WIN CE because tempfilename is not freed at WinClose edit
in a build for Windows CE ( OS_WIN is defined ) temporary files have to be deleted after winclose because the operating systems does not delete themself. At the first sqlite3_get_table() somewhere in the code in function sqlite3WinOpenExclusive() the call to convertUtf8Filename() returns a newly allocated name for the tempfile. This is afterwards assigned to f.DeleteOnClose but is never deleted or freed in WinClose, when hMutex is not used.
2007-Oct-08 04:49:56 by anonymous:
This was introduced in checkin [3836] and subsequently reported in bugs #2533 and #2598.
 
2697 code fixed 2007 Oct anonymous   2007 Oct   1 3 sqlite3_open_v2 fails then filename includes non-english chars edit
Can't create database with new sqlite3_open_v2 function if filename is international (non English)

char filename[MAX_PATH]; memset(filename, 0, sizeof(filename)); ::WideCharToMultiByte(CP_ACP, 0, jobfilename.c_str(), -1, filename, sizeof(filename)-1, 0, 0); int dbcreateres = sqlite3_open_v2(filename, &m_db, SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, NULL);

dbcreateres == 14

By the way, why we just don't have sqlite3_open16_v2 function !!! It would be best solution for me !

changing the code to ::WideCharToMultiByte(CP_UTF8, ... fixes the issue... sorry for your time...

 
2696 code fixed 2007 Oct anonymous   2007 Oct   3 3 sqlite3.c with SQLITE_OMIT_FLOATING_POINT cannot compile on Linux edit
Unmodified compile of sqlite3.c with SQLITE_OMIT_FLOATING_POINT will generate these errors:

In file included from /usr/include/math.h:141,
                 from sqlite3.c:10227:
/usr/include/bits/mathcalls.h:55: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘acosl’
/usr/include/bits/mathcalls.h:55: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘__acosl’
/usr/include/bits/mathcalls.h:57: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘asinl’
/usr/include/bits/mathcalls.h:57: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘__asinl’
/usr/include/bits/mathcalls.h:59: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘atanl’
/usr/include/bits/mathcalls.h:59: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘__atanl’
/usr/include/bits/mathcalls.h:61: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘atan2l’
etc...
#include <math.h> must be moved to just after the line #define SQLITE3_H in the sqlite3.c amalgamation for it to compile with gcc. It probably has to do with the place where double is redefined.
 
2695 new closed 2007 Oct anonymous   2007 Oct drh 3 3 sqlite3 CLI unresolved externals with SQLITE_OMIT_GET_TABLE edit
sqlite3_free_table and sqlite3_get_table are unresolved if sqlite3 is compiled with -DSQLITE_OMIT_GET_TABLE=1

Required fix below.

Index: src/shell.c
===================================================================
RCS file: /sqlite/sqlite/src/shell.c,v
retrieving revision 1.167
diff -u -3 -p -r1.167 shell.c
--- src/shell.c 7 Sep 2007 01:12:32 -0000       1.167
+++ src/shell.c 5 Oct 2007 06:19:58 -0000
@@ -1458,6 +1458,7 @@ static int do_meta_command(char *zLine,
     fprintf(p->out,"\n");
   }else

+#ifndef SQLITE_OMIT_GET_TABLE
   if( c=='t' && n>1 && strncmp(azArg[0], "tables", n)==0 ){
     char **azResult;
     int nRow, rc;
@@ -1514,6 +1515,7 @@ static int do_meta_command(char *zLine,
     }
     sqlite3_free_table(azResult);
   }else
+#endif

   if( c=='t' && n>1 && strncmp(azArg[0], "timeout", n)==0 && nArg>=2 ){
     open_db(p);
2007-Oct-05 14:34:39 by drh:
I would imagine that there are many SQLITE_OMIT_... macros that will break the CLI. The CLI is a separate program which is distinct from the SQLite library and is not expected to work with every possible configuration of the SQLite library.

So this really should be a feature request and not a bug report.

My dilemma now is whether to accept or reject this feature request. The changes are relatively minor. But on the other hand, the requested feature is very obscure. Why is it that you want to have a CLI that omits the sqlite3_get_table() API? As I sit here pondering, I cannot think of a single reason why anybody would want to do such a thing, and so for that reason I am rejecting this feature request. If somebody (the original poster or anybody else) can give me a legitimate reason for wanting to build the CLI without the sqlite3_get_table() interface, I will reconsider the rejection.


2007-Oct-05 15:49:01 by anonymous:
Why? To test all the OMIT macros on a representative program already in the source tree. I found other OMIT compile problems outside of shell.c when performing this exercise. I'll file a seperate ticket SQLITE_OMIT_FLOATING_POINT.

No one is saying anyone would #define this particular OMIT. It is just there for completeness. It gives programmers confidence in using the sqlite library without investing too much time in it. For that matter, it would be nice to test an OMIT-filled sqlite3 library against the exactly same configured corresponding sqlite3 shell.

Also, you define it if you wanted to build a minimalistic shell for a severely memory-constrained embedded device. Truth be told, the .table command is a nice but rarely used command whose info can be gleaned with a simple query on sqlite_master anyway.


2007-Oct-05 16:08:23 by anonymous:
By the way, only a couple of other OMITs broke shell.c:

  SQLITE_OMIT_COMPLETE
  SQLITE_OMIT_DISKIO

You did a better job than you realized. ;-)


2007-Oct-05 16:26:04 by anonymous:
Was able to successfully compile/link/run shell.c with:

  THREADSAFE=0
  SQLITE_OMIT_ALTERTABLE
  SQLITE_OMIT_ANALYZE
  SQLITE_OMIT_ATTACH
  SQLITE_OMIT_AUTHORIZATION
  SQLITE_OMIT_AUTOINCREMENT
  SQLITE_OMIT_AUTOVACUUM
  SQLITE_OMIT_BETWEEN_OPTIMIZATION
  SQLITE_OMIT_BLOB_LITERAL
  SQLITE_OMIT_CAST
  SQLITE_OMIT_CHECK
  SQLITE_OMIT_COMPOUND_SELECT
  SQLITE_OMIT_CONFLICT_CLAUSE
  SQLITE_OMIT_DATETIME_FUNCS
  SQLITE_OMIT_EXPLAIN
  SQLITE_OMIT_FLAG_PRAGMAS
  SQLITE_OMIT_FLOATING_POINT    (see Ticket #2696)
  SQLITE_OMIT_FOREIGN_KEY
  SQLITE_OMIT_GET_TABLE         (with above patch)
  SQLITE_OMIT_GLOBALRECOVER
  SQLITE_OMIT_INCRBLOB
  SQLITE_OMIT_INTEGRITY_CHECK
  SQLITE_OMIT_LIKE_OPTIMIZATION
  SQLITE_OMIT_LOAD_EXTENSION
  SQLITE_OMIT_MEMORYDB
  SQLITE_OMIT_OR_OPTIMIZATION
  SQLITE_OMIT_PAGER_PRAGMAS
  SQLITE_OMIT_PROGRESS_CALLBACK
  SQLITE_OMIT_QUICKBALANCE
  SQLITE_OMIT_REINDEX
  SQLITE_OMIT_SCHEMA_PRAGMAS
  SQLITE_OMIT_SCHEMA_VERSION_PRAGMAS
  SQLITE_OMIT_SHARED_CACHE
  SQLITE_OMIT_SUBQUERY
  SQLITE_OMIT_TCL_VARIABLE
  SQLITE_OMIT_TEMPDB
  SQLITE_OMIT_TRACE
  SQLITE_OMIT_TRIGGER
  SQLITE_OMIT_UTF16
  SQLITE_OMIT_VACUUM
  SQLITE_OMIT_VIEW
  SQLITE_OMIT_VIRTUALTABLE
  SQLITE_OMIT_XFER_OPT


2007-Oct-05 16:28:11 by drh:
If you are building a minimalist shell, you should start from scratch and not try to use shell.c. There is a lot of cruft in shell.c (above and beyond ".table") that a minimalist shell does not want. So this is not a valid reason for wanting shell.c to support SQLITE_OMIT_GET_TABLE.

Likewise, there a lot of ways to test to make sure SQLite work with SQLITE_OMIT_GET_TABLE without compiling shell.c.

I remain unconvinced. Feature request still rejected.


2007-Oct-05 16:33:33 by anonymous:
Wow, you won't check in an trivially obvious perfectly harmless 2 line patch despite a half dozen reasons why it's useful.

A lot of people use shell.c for testing and in production - probably more people use the CLI than do the Tcl API.

 
2694 code fixed 2007 Oct anonymous   2007 Oct drh 2 2 Makefile.in is missing invokation of mksqlite3internalh.tcl for a10n edit
The amalgamation should produce both sqlite3.c and sqlite3internal.h which is correct in the master main.mk but missing in Makefile.in; the following patch corrects Makefile.in:

  --- sqlite3.orig/Makefile.in    2007-09-04 00:11:33.000000000 +0200
  +++ sqlite3/Makefile.in 2007-10-04 22:24:00.000000000 +0200
  @@ -333,6 +333,7 @@

   sqlite3.c:     target_source $(TOP)/tool/mksqlite3c.tcl
          tclsh $(TOP)/tool/mksqlite3c.tcl
  +       tclsh $(TOP)/tool/mksqlite3internalh.tcl

   # Rules to build the LEMON compiler generator
   #
2007-Oct-05 14:24:38 by drh:
See my comment on ticket #2693.
 
2693 code fixed 2007 Oct anonymous   2007 Oct drh 2 2 mutex.h missing in a10n of sqlite3internal.h edit
mutex.h is still included in amalgamatized sqlite3internal.h; maybe that patch fixes it.

  --- tool/mksqlite3internalh.tcl.orig    2007-08-14 15:21:27.000000000 +0200
  +++ tool/mksqlite3internalh.tcl 2007-10-04 22:52:46.000000000 +0200
  @@ -57,6 +57,7 @@
      btreeInt.h
      hash.h
      keywordhash.h
  +   mutex.h
      opcodes.h
      os_common.h
      os.h
2007-Oct-05 14:23:58 by drh:
Our intent is to eliminate sqlite3internal.h. As of 3.5.0 there should never be a need to access the internal definitions of SQLite. Everything that you need to use SQLite properly is contained in sqlite3.h. If you need definitions found in the internal include files, that indicates that you are using SQLite in a way that is unsupported and likely to break from one release to the next.
 
2692 build closed 2007 Oct anonymous   2007 Oct   1 1 COmpilation of Sqlite amalgamation v3.4.2. failed edit
Version 3.4.2 (amalgamation) could not be compiled under MS DevStudio 6

the compilation errors are: sqlite3.c(6187) : error C2133: 'sqlite3UpperToLower' : unknown size sqlite3.c(9310) : error C2133: 'sqlite3OpcodeNames' : unknown size sqlite3.c(47159) : error C2133: 'sqlite3IsIdChar' : unknown size

2007-Oct-04 18:19:44 by drh:
Duplicate of #2574.
 
2691 code closed 2007 Oct anonymous   2007 Oct   1 1 SQLITE_OMIT_AUTHORIZATION compilation error edit
sqlite3.Int.h does not compile with SQLITE_OMIT_AUTHORIZATION.

Rationale: sqlite3Int.h, line 1768:

  # define sqlite3AuthRead(a,b,c)

is missing a 4th parameter like this:

  # define sqlite3AuthRead(a,b,c,d)
thanks.
 
2690 code closed 2007 Oct anonymous   2007 Oct   3 4 "PRAGMA synchronous = OFF;" is more dangerous than documented edit
The "PRAGMA synchronous = OFF;" is much more dangerous than implied in the documentation (at http://www.sqlite.org/pragma.html).

On our embedded system written data may remain in a middleware buffer (without being passed to the OS) until a subsequent read, write, seek or close call occurs.

In particular, the last write to the database file may remain buffered semi-locally for quite a while after the lock is released. This will inevitably cause problems if the database is read through a different handle meanwhile - even by the same thread!

I have seen this occur (fairly regularly), when a subsequent SELECT returns SQLITE_CORRUPT. Four pages are read through another handle before the error is raised. The write didn't complete until the file was closed as part of the application shutdown more than 10 seconds later.

It might reasonably be argued that our middleware is broken in this case, but I'm sure that there are other systems with similar problems.

The problem is still present in version 3.4.1 where the sqlite3OsSync call at pager.c:3954 should not be conditional on !pPager->noSync.

Another solution may be to change the documentation to highlight this risk.

2007-Oct-04 14:26:18 by drh:
From the description, this seems definitely to be a bug in your alternative OS interface, not in the delivered SQLite code. Perhaps you should modify your middleware so that it flushes pending writes prior to unlocking.
 
2689 build closed 2007 Oct anonymous   2007 Oct   3 4 make test fails in 3.5.1 - unresolved externals in testfixture edit
tar zxvf sqlite......

cd sqlite-3.5.1

mkdir bld

cd bld

../configure

make

make test

./libtool --mode=link gcc -g -O2 -I. -I../src -DNDEBUG -I/System/Library/Frameworks/Tcl.framework/Versions/8.4/Headers -DSQLITE_THREADSAFE=1 -DSQLITE_THREAD_OVERRIDE_LOCK=-1 -DSQLITE_OMIT_LOAD_EXTENSION=1 -DTCLSH=1 -DSQLITE_TEST=1 -DSQLITE_CRASH_TEST=1 \ -DSQLITE_NO_SYNC=1 -DTEMP_STORE=1 \ -o testfixture ../src/attach.c ../src/btree.c ../src/build.c ../src/date.c ../src/expr.c ../src/func.c ../src/insert.c ../src/malloc.c ../src/os.c ../src/os_os2.c ../src/os_unix.c ../src/os_win.c ../src/pager.c ../src/pragma.c ../src/prepare.c ../src/printf.c ../src/select.c ../src/test1.c ../src/test2.c ../src/test3.c ../src/test4.c ../src/test5.c ../src/test6.c ../src/test7.c ../src/test8.c ../src/test9.c ../src/test_autoext.c ../src/test_async.c ../src/test_btree.c ../src/test_config.c ../src/test_hexio.c ../src/test_malloc.c ../src/test_md5.c ../src/test_schema.c ../src/test_server.c ../src/test_tclvar.c ../src/tokenize.c ../src/utf.c ../src/util.c ../src/vdbe.c ../src/vdbeapi.c ../src/vdbeaux.c ../src/vdbemem.c ../src/where.c parse.c ../src/tclsqlite.c \ libsqlite3.la -framework Tcl -lpthread -framework CoreFoundation

gcc -g -O2 -I. -I../src -DNDEBUG -I/System/Library/Frameworks/Tcl.framework/Versions/8.4/Headers -DSQLITE_THREADSAFE=1 -DSQLITE_THREAD_OVERRIDE_LOCK=-1 -DSQLITE_OMIT_LOAD_EXTENSION=1 -DTCLSH=1 -DSQLITE_TEST=1 -DSQLITE_CRASH_TEST=1 -DSQLITE_NO_SYNC=1 -DTEMP_STORE=1 -o .libs/testfixture ../src/attach.c ../src/btree.c ../src/build.c ../src/date.c ../src/expr.c ../src/func.c ../src/insert.c ../src/malloc.c ../src/os.c ../src/os_os2.c ../src/os_unix.c ../src/os_win.c ../src/pager.c ../src/pragma.c ../src/prepare.c ../src/printf.c ../src/select.c ../src/test1.c ../src/test2.c ../src/test3.c ../src/test4.c ../src/test5.c ../src/test6.c ../src/test7.c ../src/test8.c ../src/test9.c ../src/test_autoext.c ../src/test_async.c ../src/test_btree.c ../src/test_config.c ../src/test_hexio.c ../src/test_malloc.c ../src/test_md5.c ../src/test_schema.c ../src/test_server.c ../src/test_tclvar.c ../src/tokenize.c ../src/utf.c ../src/util.c ../src/vdbe.c ../src/vdbeapi.c ../src/vdbeaux.c ../src/vdbemem.c ../src/where.c parse.c ../src/tclsqlite.c -framework Tcl -framework CoreFoundation ./.libs/libsqlite3.dylib -lpthread

/usr/bin/ld: Undefined symbols:

_Sqlitetest7_Init

_SqlitetestOnefile_Init

_SqlitetestThread_Init

collect2: ld returned 1 exit status

make: *** [testfixture] Error 1

2007-Oct-04 11:21:47 by anonymous:
This is mac os x


2007-Oct-04 15:40:13 by anonymous:
Same unresolved externals for testfixture on Linux.

Fix below. Do a "make distclean" and re-run configure/make test.

Index: Makefile.in
===================================================================
RCS file: /sqlite/sqlite/Makefile.in,v
retrieving revision 1.183
diff -u -3 -p -r1.183 Makefile.in
--- Makefile.in 3 Sep 2007 22:15:45 -0000       1.183
+++ Makefile.in 4 Oct 2007 15:37:57 -0000
@@ -212,6 +212,8 @@ SRC += \
 # Source code to the test files.
 #
 TESTSRC = \
+  $(TOP)/src/test_onefile.c \
+  $(TOP)/src/test_thread.c \
   $(TOP)/src/attach.c \
   $(TOP)/src/btree.c \
   $(TOP)/src/build.c \
Index: src/test7.c
===================================================================
RCS file: /sqlite/sqlite/src/test7.c,v
retrieving revision 1.9
diff -u -3 -p -r1.9 test7.c
--- src/test7.c 12 Sep 2007 17:01:45 -0000      1.9
+++ src/test7.c 4 Oct 2007 15:37:57 -0000
@@ -21,8 +21,8 @@
 ** This test only works on UNIX with a SQLITE_THREADSAFE build that includes
 ** the SQLITE_SERVER option.
 */
-#if defined(SQLITE_SERVER) && !defined(SQLITE_OMIT_SHARED_CACHE)
-#if defined(OS_UNIX) && OS_UNIX && SQLITE_THREADSAFE
+#if defined(SQLITE_SERVER) && !defined(SQLITE_OMIT_SHARED_CACHE) && \
+    defined(OS_UNIX) && OS_UNIX && SQLITE_THREADSAFE

 #include <stdlib.h>
 #include <string.h>
@@ -720,5 +720,4 @@ int Sqlitetest7_Init(Tcl_Interp *interp)
 }
 #else
 int Sqlitetest7_Init(Tcl_Interp *interp){ return TCL_OK; }
-#endif /* OS_UNIX */
 #endif

2007-Oct-04 15:43:15 by anonymous:
Output of make test after patch:

  0 errors out of 31801 tests
  All memory allocations freed - no leaks
  Maximum memory usage: 14160983 bytes


2007-Oct-04 19:09:05 by anonymous:
I am the original reporter - that fix worked for me, thanks.

My results (OS X)

  0 errors out of 31207 tests
  All memory allocations freed - no leaks
  Maximum memory usage: 14161431 bytes
 
2688 build closed 2007 Oct anonymous   2007 Oct   3 1 SQLITE_OMIT_AUTHORIZATION build failure edit
Version 3.5.1 does not compile when specifying -DSQLITE_OMIT_AUTHORIZATION. The problem lies on line 6872 (of the amalgamation):

# define sqlite3AuthRead(a,b,c)

should be:

# define sqlite3AuthRead(a,b,c,d)

thanks.
 
2687 code fixed 2007 Oct anonymous   2007 Nov   3 2 Building with _UNICODE fails in Win32 (3.5.1) edit
The function winDlError uses the unadorned function FormatMessage() which gets turned into FormatMessageW() in a Unicode build. Because the output buffer is a char*, it needs to be specified as FormatMessageA() to avoid the compile time error.

static void winDlError(sqlite3_vfs *pVfs, int nBuf, char *zBufOut){
  FormatMessageA(
    FORMAT_MESSAGE_FROM_SYSTEM,
    NULL,
    GetLastError(),
    0,
    zBufOut,
    nBuf-1,
    0
  );
}
Note that if the output message is supposed to be in UTF-8 then extra processing needs to be added to this function to convert the message into that character encoding.
2007-Oct-04 03:59:59 by anonymous:
In summary:
-    FormatMessage(
+    FormatMessageA(


2007-Nov-12 18:37:43 by anonymous:
Please reopen this ticket: Windows Mobile / CE does not necessarily provide the ANSI version FormatMessageA(). This was observed with the arm-wince-mingw32ce GCC cross compiler.

Maybe the following replacement could be added to os_win.c

  #ifdef UNDER_CE
  int FormatMessageA(int dwFlags, void *lpSource, int dwMessageId,
    int dwLangId, char *lpBuffer, int nSize, va_list *args){
    if( dwFlags & 256 ){
      char *buf = (char *)LocalAlloc(64, 126);
      *(char **)lpBuffer = buf;
      if( dwMessageId > 0x7FFFFFF ){
        sprintf(buf, "OsError 0x%x", dwMessageId, dwMessageId);
      }else{
        sprintf(buf, "OsError %d", dwMessageId);
      }
      return strlen(buf);
    }
    if( dwMessageId > 0x7FFFFFF ){
      sprintf((char *)lpBuffer, "OsError 0x%x", dwMessageId);
    }else{
      sprintf((char *)lpBuffer, "OsError %d", dwMessageId);
    }
    return strlen((char *)lpBuffer);
  }
  #endif
 
2686 code fixed 2007 Oct anonymous   2007 Oct   3 3 Integrity check fails if you hit max_page_count and dont rollback edit
If you set PRAGMA max_page_count to reachable value, fill database until SQLITE_FULL is returned, and don't do a rollback immediately (say, because your program quits or crashes before rollback), the database will be left in damaged state (PRAGMA integrity_check will detect errors).

This happens for me for both 3.4.2 and 3.5.0.

This is serious defect which breaks A.C.I.D. ideology, although it is not a show stopper and will not happen often to users.

Furthermore, one time this ended in "database is malformed" error, although I could not reproduce this again. Try modifying the test below to not stop after integrity check failure, let it cycle hundreds times, and you will get malformed database error sooner or later.

This should not happen, database integrity must stay o.k. whenever program crashes before doing rollback, or simply never does a rollback.

The following code reproduces the problem. Notice the commented out conn.rollback() line, if you uncomment it, bug will disappear. You may consider adding this test to other SQLite unit-tests.

#!/usr/bin/python
import sqlite3, random, os, os.path, sys
dbfile = '''db.tmp'''
if os.path.exists(dbfile): os.unlink(dbfile)
conn = sqlite3.connect(dbfile)
cur = conn.cursor()
cur.execute('''PRAGMA page_size=1024''') # just to make sure
cur.execute('''PRAGMA max_page_count=5120''') # 5 megabytes
cur.execute('''PRAGMA auto_vacuum=0''')
cur.execute('''CREATE TABLE filler (fill)''')

while 1:
    try:
        cur.execute('''INSERT INTO filler (fill) VALUES (randomblob(%d))''' % int(random.uniform(10240, 102400)))
        conn.commit()
        sys.stdout.write('.')
    except sqlite3.OperationalError, e:
        if e.message != "database or disk is full":
            break
        sys.stdout.write('*')
        #conn.rollback() #uncomment this and error will disappear
        cur.execute('''DELETE FROM filler WHERE rowid <= (SELECT MAX(rowid) FROM filler LIMIT 20)''')
        cur.execute('''PRAGMA integrity_check''')
        r = cur.fetchall()
        s = str(r[0][0])
        if s != 'ok':
            print "Something is wrong! Integrity check returned", s
            print r
            break
</pre>
 
2685 code fixed 2007 Oct anonymous   2007 Oct anonymous 2 1 Compilation under MSVC gives warning on TryEnterCriticalSection edit
If SQLite is compiled under MSVC 8.0 for instance it gives following warning while compiling mutex_w32.c: src\mutex_w32.c(147) : warning C4013: 'TryEnterCriticalSection' undefined; assuming extern returning int

This warning shall be treated as error, if correct behavior of try mutex is exected.

Solution: to declare #define _WIN32_WINNT 0x0400 before # include <windows.h> in os.h. Also it worth adding WIN32_LEAN_AND_MEAN to exclude unneeded stuff.

Workaround: I propose adding 2 lines of macros starting from line 72 in os.h as:

  #define _WIN32_WINNT 0x0400
  #define WIN32_LEAN_AND_MEAN

Code view:

  #if OS_WIN
  #define _WIN32_WINNT 0x0400
  #define WIN32_LEAN_AND_MEAN
  # include <windows.h>
  # define SQLITE_TEMPNAME_SIZE (MAX_PATH+50)

Sincerely yours, Dmitry Kostjuchenko.

2007-Oct-02 21:40:15 by anonymous:
This assumes SQLite will not run under non-NT systems. Should backward compatibility still existing ? I'm not sure...


2007-Oct-30 20:33:20 by anonymous:
Why not enable the code when _WIN32_WINNT is actually defined to a proper value (i.e. >= 0x0400):

  #if _WIN32_WINNT >= 0x0400
     // code that uses TryEnterCriticalSection in here
  #endif

That way, if people want this improvement, they can define _WIN32_WINNT in their make process.

 
2684 code active 2007 Oct anonymous   2007 Oct   1 1 Accessing sqlite from an NT service will lock the complete databse. edit
Accessing sqlite from a NT service (application 1) will lock the complete database. Any other process trying to open an sqlite db (application 2) will get error "80004005 unable to lock database"

If application 1 runs as normal application, started by local user, this problem doesnt occur and both applications can open the db.

2007-Oct-02 15:48:05 by anonymous:
SQLite has no knowledge of Windows services. How do you propose to work around this Windows anachronism?


2007-Oct-02 17:20:38 by anonymous:
Suggesion: Try running the service in the same account as the other program that needs to access the database.

Anachronism? Service is just another word for daemon.

-knu-


2007-Oct-02 17:33:56 by anonymous:
Re: Anachronism, the OP suggested there was something fundamentally different about file access using a service. You've pointed out that it's just a file permissions issue.


2007-Oct-05 14:45:07 by drh:
Two points:

  1. The error message "80004005 unable to lock database" is not generated by SQLite. There must be some middleware someplace that is producing this message. The problem might be in that middleware and not in SQLite.

  2. None of the SQLite developers run windows. Consequently any fixes for this problem will need to come from the community. Please append patches to this ticket if you find a fix. Or close the ticket if you discover that the problem is outside of SQLite.
 
2683 code fixed 2007 Oct anonymous   2007 Oct   1 1 Windows CE won't compile, also a problem with sqlite3OsFullPath on CE edit
With some modifications the program will compile, but opening a database returns (sometimes) unknown error, and (sometimes) attempt to write a read-only database. This strange behaviour is due to a missing return statement that should be "return SQLITE_OK;". See the patch.
2007-Oct-16 19:02:55 by anonymous:
I built version for WindowsCE and tested. Now it could open and create databases !

The only difference is in

@@ -1162,13 +1168,13 @@
 #if OS_WINCE
   if( (flags & (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB)) ==
                (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB)
-       && !winceCreateLock(zFilename, &f)
+       && !winceCreateLock(zName, pFile)
 
2682 code closed 2007 Oct anonymous   2007 Oct   1 1 SQLite Text Problem edit
I am using SQLite 3 as backend database in my asp application. I am using SQLite3 ODBC driver (latest version).

Table Design


CREATE TABLE log ( id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, from_value TEXT NULL, to_value TEXT NULL );

INSERT 10 RECORDS IN TO DATABASE


Const adOpenStatic = 3 Const adLockOptimistic = 3 Dim SQLString ,objConnection Dim fromVal,toval Set objConnection = CreateObject("ADODB.Connection") for a = 1 to 10 objConnection.Open "DSN=Test;uid=;pwd=;" fromVal = "FROM Value : "& a toVal = "To Value : "& a SQLString = "INSERT INTO log(from_value, to_value) VALUES ('"& fromVal & "' ,'"&toVal& "')" objConnection.Execute SQLString objConnection.Close next

Access records in the log table


Dim LogRs,SQLStmt SQLStmt = "SELECT * FROM log " LogRs.ActiveConnection = "DSN=Test;uid=;pwd=;" LogRs.Source= SQLStmt LogRs.CursorType = 3'ADOPENSTATIC LogRs.CursorLocation = 2 'ADUSESERVER LogRs.LockType = 3 'ADLOCKOPTIMISTIC LogRs.Open() RecordCount = LogRs.RecordCount

If I use datatype of from_value/to_value as TEXT ,I get RecordCount as zero which is wrong. If I use datatype of from_value/to_value as Varchar(1600) ,I get RecordCount as 10 which is correct. If I use datatype of from_value/to_value as Varchar(1600) and tried to put more than 1600 charcters in it ,SQLite stores data in DB. While retrieving the data I get the error Arguments are of the wrong type,are out of acceptable range,or are in conflict with one another. I enabled ODBC trace log ,but its empty after running my application.So I am not sure where it is ODBC problem or SQLite ODBC driver error? If I open table (database) in SQLite Studio I could see data without any error.

If I use MS Access with same table design and same code, I dont get any error . The code works fine as expected .

2007-Oct-01 22:04:18 by drh:
There are multiple ODBC drivers for SQLite, none of which are support by this site.

Your problem appears to be in the ODBC driver. If you find evidence to the contrary, please reopen this ticket.

 
2681 code fixed 2007 Oct anonymous   2007 Oct   3 3 #ifndef _XOPEN_SOURCE needed around #define _XOPEN_SOURCE 500 edit
_XOPEN_SOURCE is really poorly defined/implemented in general. Some platforms need _XOPEN_SOURCE to be set to 600, or a value other than 500. Could you please put an #ifdef wrapper around this, as follows?

  #ifndef _XOPEN_SOURCE
  # ifndef __MACOS__
  #  define _XOPEN_SOURCE 500  /* Needed to enable pthread recursive mutexes */
  # endif
  #endif
2007-Oct-01 16:47:50 by anonymous:
Furthermore, can you avoid defining _XOPEN_SOURCE if it is not a threadsafe build? (i.e., define _XOPEN_SOURCE only if threadsafe build requested).
 
2680 code fixed 2007 Sep anonymous   2007 Oct   2 2 sqlite_int64 vs sqlite3_int64 type confusion in headers and docs edit
Header file sqlite3.h (3.4.2) defines sqlite_int64 but not sqlite3_int64.

3.5.0 defines both.

I discovered this used the 3.5.0 documentation (on the site) but building against a local installation of 3.4.2.

The documentation and examples used BOTH types in a somewhat ad-hoc manner.

Could someone please clarify which is the preferred type? If sqlite3_int64 is preferred, then should the header files for 3.4.x be updated to reflect this?

2007-Oct-03 20:21:49 by anonymous:
sqlite 3.4.2 doesn't define sqlite3_int though, only sqlite_int

this means i can't use common code for both sqlite3.4.2 and sqlite3.5.0


2007-Oct-03 20:29:51 by anonymous:
Just use sqlite_int then. They are typedef'd to the same thing in 3.5.
 
2679 code closed 2007 Sep anonymous   2007 Oct   2 3 locking errors lead to disk i/o failure with bulk operations on winxp edit
First got lost cursor and disk i/o errors with bulk imports to sqlite database using sqliteodbc driver with OpenOffice under win xp sp 2. Then used sqlite3.exe to reproduce and found:

if sqlite database is on an ext2 volume accessed through ext2ifs driver, and I .read a sql command file with 348 times INSERT INTO ... VALUES('...','...'); to a table with two text columns, after some (hundred or more) rows without error I receive the following error: SQL error near line xxx: disk I/O error for all following lines; likewise, when I replace the INSERT by DELETE FROM ... WHERE col='...', I will receive the same error after some ten lines. The database cannot be accessed any more as long as the sqlite3 executable is running; closing and restarting it restores access (same with odbc driver - no access unless connection closed and reopened). I can omit these errors by surrounding the statements with a BEGIN TRANSACTION / COMMIT clause (not available with OOo and the odbc driver, unfortunately). Likewise, when the sqlite database is on a ntfs volume, the error is gone.

I first thought there might be an error with the ext2 ifs driver, but I use it heavily for all kinds of data without problems so far. Then I took a closer look at the sqlite3 executable by monitoring its activity using sysinternals' procmon. And this revealed that there were no failing reads or writes as the error message would suggest, but I noticed RANGE NOT LOCKED results with UnlockFileSingle operations (corresponding to UnlockFileEx calls in the source) when the error surfaces. Examing the sequence of LockFile and UnlockFileSingle on locations 1gb (1), 1gb+1 (1) and 1gb+2 (510) I found that sqlite3 indeed tried to unlock locations already unlocked previously - the reported errors (and the file locking implementation in the ext2 driver) were correct. It seems that sqlite looses track with its locks after multiple lock/unlock sequences.

I append the procmon log of locking activities of the sqlite3 executable during 'mass' inserts in csv format.

2007-Sep-28 23:22:07 by anonymous:
Could you clarify a few things?

How many SQLite processes are there in your scenario?

What is the actual schema and SQL statements issued by each process?


2007-Sep-29 18:25:50 by anonymous:
could be a ext2fs ifs driver issue. did you tested id in NTFS / FAT32 filesystems ?


2007-Sep-29 23:32:09 by anonymous:
ok, first to clarify: there is a single sqlite3 process, absolutly no concurrency; and a simple table used, like

  CREATE TABLE "WhoIsWho" ("Nummer" varchar(50) NOT NULL, "Teilnehmer" varchar(50), PRIMARY KEY ("Nummer"));

then I feed a text file with 348 statements like

  INSERT INTO WhoIsWho VALUES('01805312111','conrad electronic hotline');

with different values to a sqlite3-executable, either by issuing a .read statement at the sqlite3 prompt or using input redirection on the commandline. likewise, with full table to delete the entries I use a file with statements like

  DELETE FROM WhoIsWho WHERE Nummer=='01805220111';

this should make the setup clear - but what do you mean with schema?

and about ext2ifs issue and testing on ntfs: see above ;-) and read on

but now: I digged a lot deeper in this issue: first of all, there is a big difference in the sequence of operations as seen by procmon between ext2 and ntfs. on ntfs, there are 2 (!) "CloseFile C:\" between all other operations, and a lot of additional "CloseFile <current dir>"; this way each INSERT operation takes much longer, 142 +/- 23 ms on ntfs vs. 43 +/- 12 ms on ext2 on my system.

every INSERT line is handled similar: first a check for a left over journal file, then writes to the journal, then writes to the database, then mark the journal for deletion and close it

on the first failing INSERT the check for a left over journal finds it still there - although marked for deletion. the first time this happens the time between the last Close and the new CreateFile is very short, 4 ms in my case, so the journal is not yet deleted. now sqlite issues 2 identical successive QueryStandardInformationFile calls on the journal file, the result is

  AllocationSize: 49'152, EndOfFile: 6'184, NumberOfLinks: 1, DeletePending: True, Directory: False

ignoring the DeletePending: True result sqlite takes it for a valid journal file and does a rollback assuming the last operation had failed - this I consider a bug: a file marked for deletion has to be looked at as not valid any more.

after rollback it marks the file again for deletion, and then creates it new with OverwriteIf flag set without checking if deletion actually occured in this case. then it restarts the current INSERT again, this time succeeding - but the next INSERT cycle will rollback this line also. prior to restarting file locks are released, but this time sqlite has lost record of which locks have been set: it releases 2 locks that are not locked, resulting in the errors reported above. this I consider the second bug. Besides, I feel the locking during rollback is not what it should be: it is done with locks at 1gb and at 1gb+1 released, and the lock on 1gb+2 is released and set to exclusive state without any of the 1gb and 1gb+1 locks protecting it. I think there is a UnlockFileSingle to 1gb+1 where a LockFile at 1gb should have been.

why it works with ntfs and not with ext2: i think this is a question of timing and those massive "CloseFile C:\" inserted in case of ntfs. they appear like a q&d workaround for similar issues on ntfs.


2007-Sep-30 00:13:23 by anonymous:
Interesting. You seem to have uncovered a race condition in the Windows version of SQLite.

Just for the heck of it, what happens if you change the os_win.c code to Sleep(1000) after every single CloseFile() call?


2007-Sep-30 00:22:31 by anonymous:
Also, can you increase MX_DELETION_ATTEMPTS from 3 to 30, and Sleep(100) to Sleep(1000) and see what happens?

#define MX_DELETION_ATTEMPTS 3
static int winDelete(
  sqlite3_vfs *pVfs,          /* Not used on win32 */
  const char *zFilename,      /* Name of file to delete */
  int syncDir                 /* Not used on win32 */
){
  int cnt = 0;
  int rc;
  void *zConverted = convertUtf8Filename(zFilename);
  if( zConverted==0 ){
    return SQLITE_NOMEM;
  }
  SimulateIOError(return SQLITE_IOERR_DELETE);
  if( isNT() ){
    do{
      rc = DeleteFileW(zConverted);
    }while( rc==0 && GetFileAttributesW(zConverted)!=0xffffffff
            && cnt++ < MX_DELETION_ATTEMPTS && (Sleep(100), 1) );
  }else{
#if OS_WINCE
    return SQLITE_NOMEM;
#else
    do{
      rc = DeleteFileA(zConverted);
    }while( rc==0 && GetFileAttributesA(zConverted)!=0xffffffff
            && cnt++ < MX_DELETION_ATTEMPTS && (Sleep(100), 1) );
#endif
  }
  free(zConverted);
  OSTRACE2("DELETE \"%s\"\n", zFilename);
  return rc!=0 ? SQLITE_OK : SQLITE_IOERR;
}

2007-Sep-30 12:35:32 by anonymous:
Hmm, I think your suggestions point to the wrong place. If there were multiple attempts to delete the file, I would have seen them in the ProcMon log. But this is not the case. The DeleteFileW succeeds immediately (rc!=0), so the rest of the while clause is never executed. But, as the documentation says, a successful DeleteFile call does not mean the file is deleted, it only means the file has been marked for deletion, and the OS is free to schedule the actual deletion anytime after the CloseFile call.

We should look at the code that checks for a left over journal file. This code already reads the file attributes, so it should be easy to check for the DeletePending flag: if it is set it should treat the journal file as if it were already deleted.

Nevertheless, there is a fault in the fs driver or the OS: the MS documentation says that a CreateFile call on a file marked for deletion fails with ERROR_ACCESS_DENIED - but in this case it succeeds!

The second error - loosing track of locks during rollback - still has to be addressed.


2007-Sep-30 15:59:26 by anonymous:
What happens when you insert the Sleep calls after CloseFile?


2007-Oct-01 12:47:25 by anonymous:
If I would place a Sleep(1000) after every CloseFile (I didn't) then the errror would shurely vanish - and my 348 inserts would last 6 minutes! That's not the way to go. I suggested to look for the code to check for the journal file still there, but I see this has already been accounted for in the code to exclusively open a file on windows, here:

/*
...
** Sometimes if we have just deleted a prior journal file, windows
** will fail to open a new one because there is a "pending delete".
** To work around this bug, we pause for 100 milliseconds and attempt
** a second open after the first one fails.  The whole operation only
** fails if both open attempts are unsuccessful.
*/
int sqlite3WinOpenExclusive(const char *zFilename, OsFile **pId, int delFlag){
...
    do{
      h = CreateFileW((WCHAR*)zConverted,
         GENERIC_READ | GENERIC_WRITE,
         0,
         NULL,
         CREATE_ALWAYS,
         fileflags,
         NULL
      );
    }while( h==INVALID_HANDLE_VALUE && cnt++ < 2 && (Sleep(100), 1) );
...
}
This requires the CreateFileW call to fail and return an invalid handle - which is what Microsoft specifies for a file marked for deletion but what the file system driver does not acknowledge: it allows the file marked for deletion to be successfully opened.

So it's again the ext2ifs driver developer's turn: I have already contacted him ...


2007-Oct-01 14:46:18 by anonymous:
Adding the Sleep would simply confirm your theory. No one is suggesting it be added permanently.


2007-Oct-07 23:21:26 by anonymous:
I finally tracked down also the unlock errors: they are there by design. When a hot journal is found, the immediate rollback is purposely only with an EXCLUSIVE lock, but without a RESERVED or PENDING lock. Nevertheless, the unlock code assumes that whenever there is an EXCLUSIVE lock, then there might also be a RESERVED and a PENDING lock, and both are unlocked too. That's nothing to bother with, as it only provokes error returns from the OS, the handling itself is correct. See:
static int winUnlock(OsFile *id, int locktype){
...
  if( type>=RESERVED_LOCK ){
    UnlockFile(pFile->h, RESERVED_BYTE, 0, 1, 0);
  }
...
  if( type>=PENDING_LOCK ){
    UnlockFile(pFile->h, PENDING_BYTE, 0, 1, 0);
  }
...
}


2007-Oct-12 18:50:28 by drh:
Summary:

This turns out to be a bug in the ext2 driver for winNT, not in SQLite. The locking problem is not a real issue.


2007-Oct-14 16:27:15 by anonymous:
Just to approve: the ext2ifs driver developer sent me a patched driver that returns a "DELETE PENDING" error when a journal file already marked for deletion tried to be opened, and with it the errors are gone. I suppose the corrected driver will soon be general available.
 
2678 code fixed 2007 Sep anonymous   2007 Oct   1 1 amalgamation zip file should include sqlite3ext.h edit
It is not possible to build portable external loadable modules using just sqlite-amalgamation-3_5_0.zip - instead you need the entire sqlite3 source tree. If sqlite3ext.h were bundled with the amalgamation, this would not be necessary.
2007-Sep-28 19:44:00 by anonymous:
Actually, a better solution would be to get rid of sqlite3ext.h altogether, fold its functionality into sqlite3.h and when people go to compile a module they #define whether they wish to compile an external library or not.
 
2677 new active 2007 Sep anonymous   2007 Sep danielk1977 5 3 CREATE TRIGGER .. {databaseevent ON ... }+ for multiple events edit
I have to write triggers which react on either INSERT or UPDATE of a specific field in a table. In some SQL syntax descriptions I found a possibility to combine the trigger events for more than one condition but not in SQLite3. Isn't there a possibility to use one trigger definition for more than one event or didn't I find the trick? Do I have to copy the whole definition for each event; even if the body text is exactly the same?
Suggestion: On INSERT event use the OLD.fieldnames filled with NULL content to avoid problems with UPDATE event. - If not possible please explain why not for my better understanding. Thank you.
 
2676 code closed 2007 Sep anonymous   2007 Sep   1 1 TEMP_STORE does not appear to be set by the compile time switch edit
The compile time switch -DTEMP_STORE=2 does not appear to change TEMP_STORE.

When a database is created after compiling the library I tried to view temp_store by using PRAGMA temp_store which returns the value 0. If this is true that would force all temp storage to a file. I want temp storage to be in memory.

It's more complex than that. See the table at:

   http://www.sqlite.org/pragma.html#pragma_temp_store
 
2675 new closed 2007 Sep anonymous   2007 Sep   1 1 TEMP_STORE does not appear to be set by the compile time switch edit
The compile time switch -DTEMP_STORE=2 does not appear to change TEMP_STORE.

When a database is created after compiling the library I tried to view temp_store by using PRAGMA temp_store which returns the value 0. If this is true that would force all temp storage to a file. I want temp storage to be in memory.

See #2676
 
2674 code active 2007 Sep anonymous   2007 Sep   3 4 NFS fails without lock manager edit
Problem:

SQLite fails entirely if the NFS lock manager is not running on the share hosting the DB -- even when all access to the DB is serialized.

Under these circumstances, it becomes difficult to create applications that are capable of running in a multitude of environments because restrictions are now imposed upon the storage location.


Reproduction:

  • setup an NFS share
  • disable the lock manager
  • attempt to perform any transactions on a db on that share
  • start the lock manager
  • perform the same operations


Request:

  • Make a change to the error handling, as necessary, to allow processes to access a DB over an NFS share without use of a lock manager.

  • Make a big bold flashing sign in the FAQ about this failure-mode. The wording of the current FAQ led us to believe that the transaction would go through, but protection from other processes was not guaranteed.


Shameless Fanmail:

Love the product!


Screenshot:

  dev-srs08 ~ # <cmd>
  [__db_init <file>.c:384] Statement:
  CREATE TABLE
  versions (
    id       INTEGER PRIMARY KEY AUTOINCREMENT,
    version  INTEGER,
    tbl      CHAR(256)
  );
  Failed with error:
  database is locked
  [main <file>.c:1096] Unable to init output DB: /mnt/nfs/o
  dev-srs08 ~ # /etc/init.d/nfslock start
  Starting NFS statd:                                        [  OK  ]
  dev-srs08 ~ # <cmd>
  dev-srs08 ~ #
2007-Sep-28 04:35:00 by anonymous:
Yet another reason to avoid using SQLite on remote shares.


2007-Sep-29 18:04:19 by anonymous:
Make a change to the error handling, as necessary, to allow processes to access a DB over an NFS share without use of a lock manager.

By design the SQLite library guarantees ACID. It can't provide that without file locks. In my opinion a non-ACID version would be a custom version, which you can (should) build yourself. The source is available, or there might already be a compilation option to accomplish that.

Make a big bold flashing sign in the FAQ about this failure-mode. The wording of the current FAQ led us to believe that the transaction would go through, but protection from other processes was not guaranteed.

FAQ (5) "When any process wants to write, it must lock the entire database file for the duration of its update." seems to be quite clear.

-knu-

 
2673 build fixed 2007 Sep anonymous   2007 Oct   4 4 3.5.0 amalgamation on OS X, "#define _XOPEN_SOURCE 500" fails build edit
In the first amalgamation release for 3.5.0, the following #define causes a build problem on Mac OS X 10.4.

  #define _XOPEN_SOURCE 500  /* Needed to enable pthread recursive mutexes */

This is not needed on OS X and, instead, causes _POSIX_C_SOURCE to be #defined around line 280 in <sys/cdefs.h> which in turn causes the fsync() and localtime_r() functions to not be declared in <unistd.h> and <time.h>, respectively. This causes a compile warning or an error if you have -Werror-implicit-function-declaration enabled. More importantly, it causes the compiler to create an implicit declaration for those two functions which is somewhat dangerous.

Thanks!

 
2672 doc fixed 2007 Sep anonymous   2007 Oct   1 3 icu ext init failure: sqlite3_create_function returns SQLITE_BUSY edit
icu extenstion init fails b/c sqlite3_create_function returns 5 (SQLITE_BUSY)

in the init code for icu i print the function name and the result code from sqlite3_create_function, i see:

  sqlite> select load_extension('./icu.so');
  create function for regexp gave 0
  create function for lower gave 0
  create function for lower gave 0
  create function for upper gave 0
  create function for upper gave 0
  create function for lower gave 5

i,e. the first lower with SQLITE_UTF8 fails (i didn't check why yet, i'm wondering was there some API change here though wrt to having both _UST16 and _UTF8?)

also, i notice the code on failure doesn't cleanup, so it will leave functions registered even though init failed (recompile & retry probably will crash sooner or later)

This is a bug.

SQLite does not allow you to change or delete a user function while there are SQL statements running (could cause a seg-fault if the running statement users the function being deleted/modified). Adding new functions is Ok though.

The reason this is failing is that SQLite is counting "SELECT load_extension(...)" as a running statement. The first couple of create function calls succeeded because these did not override existing user functions.

The easiest workaround would be to use the sqlite3_load_extension() API directly, not via "SELECT load_extension()". In the shell, try this:

  sqlite> .load icu.so

Alternatively, you could try linking the extension staticly with SQLite. To do this, add ext/icu/icu.c to the list of C files to be compiled and add -DSQLITE_ENABLE_ICU to the compiler command line (for all C files, not just icu.c).


2007-Sep-28 17:34:36 by anonymous:
Ah right!

Testing using ".load ./icu.so" does indeed work correctly.

However, there still remains the bug that when it does fail to initialize some functions are left registered and accessing those will usually seg-fault.

Given that sqlite3_create_function cannot be undone, perhaps there should be some way to detect (inside the extension) if it's being loaded from inside a statement and bail out early?


2007-Sep-28 17:40:14 by anonymous:
To fix this bug, sqlite need only defer the loading of the extension module until after any pending SQL call is complete.
 
2671 code active 2007 Sep shess   2007 Sep shess 2 4 Fts field-based queries are not correctly case-insensitive. edit
  CREATE VIRTUAL TABLE t USING fts2(A, B);
  -- At the SQL level, things are case-insensitive:
  INSERT INTO t (A, b) VALUES ('Test', 'Columns');
  INSERT INTO T (a, B) VALUES ('Second', 'Test');
  -- Unfortunately, fts cannot do field-level queries:
  SELECT rowid FROM t WHERE t MATCH 'test';  -- works
  SELECT rowid FROM t WHERE b MATCH 'test';  -- works
  SELECT rowid FROM t WHERE t MATCH 'b:test';  -- no results
  SELECT rowid FROM t WHERE t MATCH 'B:test';  -- no results

It doesn't work because fts is keeping the column name as 'B', but the query parsing uses the results from the tokenizer, which are case-folded, and 'b' != 'B'.

I'm thinking on the solution. A quick fix would be to make the azColumn storage be lowercase, but the core problem is that field names probably shouldn't be run through the tokenizer in the first place.

 
2670 doc closed 2007 Sep anonymous   2007 Oct   5 4 Document for EXPLAIN doesn't mention using EXPLAIN QUERY PLAN edit
I was looking for documentation for determining what indicies a select statement uses. I found "EXPLAIN" in the documentation, but couldn't figure anything out from the output. Eventually I found "EXPLAIN QUERY PLAN" on the idxchk wiki page.
2007-Oct-01 13:37:02 by drh:
EXPLAIN QUERY PLAN is unsupported. If we document it we will have to support it forever. That is something we do not want to do.
 
2669 code closed 2007 Sep anonymous   2007 Sep anonymous 2 3 INSERT of Large REAL value fails with Floating Point Exception edit
When inserting a large number as a REAL value, a Floating Point Invalid Operation exception is thrown. This occurs in the "sqlite3VdbeIntegerAffinity" function in the "vdbemem.c" source, on the line "pMem->i = pMem->r". This exception is only thrown when the value in pMem->r is outside the range of a signed I64 value.

The following is a work-around that seems to suppress the exception and produce no adverse affects on the content of the database:

	/*
	**  @date	Added 9/26/07 - ESK -- Include needed for the _I64_MIN and _I64_MAX defines
	*/
	#include <limits.h>

	/*
	** The MEM structure is already a MEM_Real.  Try to also make it a
	** MEM_Int if we can.
	*/
	void sqlite3VdbeIntegerAffinity(Mem *pMem){
	  assert( pMem->flags & MEM_Real );
	  /*
	  **  @date	9/26/07 - ESK -- Added conditional to check for a double value that is
	  **  outside the range of the __int64 value (when the value is out of the range it
	  **  throws a invalid floating point operation exception).
	  */
	  if (( pMem->r > _I64_MAX ) || ( pMem->r < _I64_MIN )){
		 pMem->i = 0;
	  }else{
		 pMem->i = pMem->r;
	  }
	  if( ((double)pMem->i)==pMem->r ){
		 pMem->flags |= MEM_Int;
	  }
	}

At the time of the exception, pMem->r is equal to 3.8E+49. The library was compiled under Borland C++ Builder 5 on the Windows Platform (2k and XP).

This issue has only occurred for us when pulling data out of a database (that originally accepted the real value that was inserted) and inserting the data into another database. The problem occurs when calling sqlite3_step with a sqlite3_stmt struct, after binding the REAL value to the structure.

I'm not sure if this floating point exception will present in any other location, since this is the only location we've seen it in so far.

This issue may or may not be related to Ticket #2586.

2007-Sep-26 15:10:55 by anonymous:
Please provide a few line standalone SQL example with schema that reproduces what you're seeing.


2007-Sep-26 15:26:59 by anonymous:
To be more specific, please provide a few line standalone SQL example with schema that reproduces what you're seeing using the sqlite3 commandline shell program.

http://www.sqlite.org/sqlite-3_5_0.zip


2007-Sep-26 23:34:56 by anonymous:
We cannot replicate the issue using the sqlite3.exe utility. However the following code sample does demonstrate how to reproduce the exception (the exception occurs within the call to sqlite3_step):

	#include "sqlite3.h"
	int main(int argc, char* argv[])
	{
		sqlite3        *pDB;
		int            ret;
		char           *errmsg = NULL;
		sqlite3_stmt   *pStmt;

		ret = sqlite3_open ("database.db", &pDB);

		ret = sqlite3_exec (pDB, "CREATE TABLE POS_NUMERIC (MAIN_ID INTEGER, NUMERIC_VALUE REAL )", 0, 0, &errmsg);
		if ( errmsg ){
			free (errmsg);
		}

		ret = sqlite3_prepare (pDB, "INSERT INTO POS_NUMERIC (MAIN_ID, NUMERIC_VALUE) VALUES (?, ?)", -1, &pStmt, 0);
		sqlite3_bind_text (pStmt, 1, "1234", strlen ("1234"), SQLITE_TRANSIENT);
		sqlite3_bind_text (pStmt, 2, "3.8E49", strlen ("3.8E49"), SQLITE_TRANSIENT);
		ret = sqlite3_step (pStmt);

		sqlite3_finalize (pStmt);
		pStmt = NULL;

		ret = sqlite3_close (pDB);

		return 0;
	}


2007-Sep-27 01:07:15 by anonymous:
Your program works fine here with gcc, and the database has the correct values.

What compile flags did you use for Borland C++?

Can you run this program?

  #include <stdio.h>
  #include <stdlib.h>
  #include "sqlite3.h"
  void p(char* z) {
    puts(z);
    sqlite3_free(z);
  }
  int main() {
    sqlite_int64 i;
    double r = atof("3.8E49");
    printf("sizeof(sqlite_int64) = %d\n", sizeof(sqlite_int64));
    p(sqlite3_mprintf("r = %f", r));
    i = r;
    if ((double)i == r)
      puts("can fit into sqlite_int64");
    else
      puts("can not fit into sqlite_int64");
    return 0;
  }

It should produce:

  sizeof(sqlite_int64) = 8
  r = 38000000000000000000000000000000000000000000000000.000000
  can not fit into sqlite_int64


2007-Sep-27 01:14:21 by anonymous:
It's a Borland bug. You have to set the FPU control word as follows to avoid these floating point exceptions.

    _control87(MCW_EM, MCW_EM);

See:

http://www.virtualdub.org/blog/pivot/entry.php?id=53

http://homepages.borland.com/ccalvert/TechPapers/FloatingPoint.html

 
2668 code closed 2007 Sep anonymous   2007 Oct   3 4 Build failure due to syntax error under Cygwin edit
Missing semi-colon at end of line. The patch below fixes it.

    --- os_win.c.orig       2007-09-25 16:55:35.053064600 -0400
    +++ os_win.c    2007-09-25 16:52:46.146226100 -0400
    @@ -1326,7 +1326,7 @@

     #if defined(__CYGWIN__)
       cygwin_conv_to_full_win32_path(zRelative, zFull);
    -  return SQLITE_OK
    +  return SQLITE_OK;
     #endif

     #if OS_WINCE
2007-Sep-25 21:22:40 by anonymous:
4th time this was reported. Cygwin appears to be a popular platform.


Duplicate of #2642
 
2667 code closed 2007 Sep anonymous   2007 Oct   1 1 no such function: hex no such function: regexp edit
I am using SQLite 3 with pysqlite. I have few triggers that uses regexp and hex functions. on linux it worked fine. Once I switch to Windows I am getting no such functions errors.
2007-Oct-01 13:35:49 by drh:
The regexp function is not supported by SQLite. hex() is supported but we are unable to find anything wrong with it.
 
2666 doc fixed 2007 Sep anonymous   2007 Oct   4 4 Typos/spelling errors in sqlite3.h edit
I don't want to be a pain in the backside, but here are some spelling errors I've spotted in sqlite3. They're all from sqlite3.h, and are all present as of the 3.5 alpha.

** The sqlite3_errmsg() and sqlite3_errmsg16() return English-langauge

int *pAutoinc /* OUTPUT: True if colums is auto-increment */
Note: The intention here was probably column, not columns.

** When the virtual-table mechanism stablizes, we will declare the
Note: This error occurs twice.

** The optimizer automatically inverts terms of the form "expr OP column"
** and makes other simplificatinos to the WHERE clause in an attempt to


** be taylored to the specific needs of the module implementation. The

 
2665 code closed 2007 Sep anonymous   2007 Sep   3 1 Dissapearing exclusive lock when DB is copied... edit
In our application, we occasionally need to make copies of a running database. To insure consistancy, the program does something like this (in python):

# we're about to lock the db
dbcursor.execute('begin exclusive')
# the db is now locked
shutil.copy('database.db','other-name.db')
# WHOAH, what happened to our lock on database.db??????

the comments indicate the locking status at each time. I've got a workaround, which amounts to:

# we're about to lock the db
dbcursor.execute('begin exclusive')
# the db is now locked
os.system('cp database.db other-name.db')
# database.db is still locked, but other-name.db is not locked

which is more or less what I'd expect.

Is this copying of databases totally evil? Why does a copy made by a process that has an exclusive lock on the origin of the copy destroy the exclusive lock? Why doesn't a copy by a different process destroy it?

2007-Sep-25 18:30:59 by anonymous:
The problem is

  shutil.copy('database.db','other-name.db')

must be unlocking the file or doing something altogether wrong.

Run strace/truss to confirm.


2007-Sep-25 18:32:06 by drh:
On unix, when you close a file, all locks on that file held by the same process are cleared - even locks that were created by separate file descriptors. You are probably closing the file at the conclusion of your copy, which is clearing the locks.

This "feature" of unix is brought to you by The Open Group. After a while, you learn to clearly see the differences between those parts of unix that were designed by Thompson and Richie and those parts developed by committee. Would that there was more of the former and less of the latter...


2007-Sep-25 18:52:38 by anonymous:
How can we get the UNIX file descriptor for the sqlite3* connection from the sqlite3 API so that we do not have to reopen and ultimately close the file, thus clearing the locks?


2007-Sep-25 19:12:05 by anonymous:
I've run strace, and the results are exactly what you'd expect; the shutil.copy() command opens the FD for creating. opens the destination FD, and writes out a copy.

Also; the bug dissapears if you replace the shutil.copy() with something like:

os.system('python -e "shutil.copy(...."')

so it's not the mechanics of shutil.copy, but instead is because of the same processness of doing the copy. Sorry for not including that earlier!


2007-Sep-26 12:09:00 by anonymous:
Maybe you could consider introducing new API - sqlite3_backup(sqlite3*, const char*), or new SQL command BACKUP(FileName) which would backup main database to specified file using (exclusive?) lock. I don't see far to SQLite internals, but I hope this change could also provide functionality for saving in-memory databases to disk.
 
2664 code active 2007 Sep danielk1977   2007 Sep   1 1 attaching the same db twice in shared-cache mode fails edit
The following SQL script can cause an assert() to fail in shared-cache mode.

  ATTACH 'db' AS aux1;
  ATTACH 'db' AS aux2;
  CREATE TABLE aux1.abc(a, b, c);
  CREATE TABLE aux2.abc(a, b, c);
See also #2653
 
2663 code closed 2007 Sep anonymous   2007 Oct   3 3 Adding a column with default value not reflected in view edit
When adding a column to a table using alter table any default value for the column is not returned by a select using a view even if the view is recreated.

Adding a single integer column to a table with a default value of 0. Any existing record correctly appears to have a value of 0 for the new column. However, if this table is used in a view, then the column appears to be NULL (the view has a select a.*, b.x from a, b where a.i = b.i; definition and we add a column to a). So, I then dropped the view and re-created it again because the schema of a has changed. However, the new column still appears to be NULL in the view whereas 0 is correctly displayed when a select from the table is done.

This error occurs using sqlite3 and no code of ours.

2007-Sep-24 14:40:49 by anonymous:
# sqlite3 f.db sqlite> create table a (a int); sqlite> create table b (b int); sqlite> create view ab as select a.*, b.* from a,b where a.a = b.b; sqlite> insert into a values (1); sqlite> insert into a values (2); sqlite> insert into a values (3); sqlite> insert into b values (2); sqlite> select * from ab; 2|2 sqlite> alter table a add column c int default 0; sqlite> select * from a; 1|0 2|0 3|0 sqlite> select * from ab; 2| sqlite> drop view ab; sqlite> create view ab as select a.*, b.* from a,b where a.a = b.b; sqlite> select * from ab; 2||2 sqlite>


2007-Sep-24 14:43:26 by anonymous:
Sorry, that should have read (and I also see that this is 3.3.5)

  $ sqlite3 f.db
  SQLite version 3.3.5
  Enter ".help" for instructions
  sqlite> create table a (a int);
  sqlite> create table b (b int);
  sqlite> create view ab as select a.*, b.* from a,b where a.a = b.b;
  sqlite> insert into a values (1);
  sqlite> insert into a values (2);
  sqlite> insert into a values (3);
  sqlite> insert into b values (2);
  sqlite> select * from ab;
  2|2
  sqlite> alter table a add column c int default 0;
  sqlite> select * from a;
  1|0
  2|0
  3|0
  sqlite> select * from ab;
  2|
  sqlite> drop view ab;
  sqlite> create view ab as select a.*, b.* from a,b where a.a = b.b;
  sqlite> select * from ab;
  2||2
  sqlite>


2007-Oct-01 13:14:13 by drh:
Unable to reproduce in 3.5.0. This problem appears to have been fixed already.


2007-Oct-22 09:58:35 by anonymous:
FYI: This problem was not present in 3.4.1 either
 
2662 code closed 2007 Sep anonymous   2007 Sep   1 1 Serious Performance Degradation from 3.3.4 to 3.3.5 & latest versions edit
Starting from 3.3.5, SQLite suffers a serious performance degradation for the following schema and query. 3.3.4 returnes results immedialtely, all later versions take considerable time (multiple seconds) to execute the query. The database is about 6 MB in size.

Schema:

  CREATE TABLE MTV20030105AdLink     (AdvertisementID     Numeric,
                                      AdRelationShipID    Numeric,
                                      AgencyID            Numeric,
                                      AdvertiserID        Numeric,
                                      ProductID           Numeric);
  CREATE TABLE MTV20030105AdType     (ADTypeID            Numeric,
                                      AdTypeName          char(50));
  CREATE TABLE MTV20030105Adjacency  (AdInstID            Numeric,
                                      ProductID           Numeric);
  CREATE TABLE MTV20030105Advertiser (AdvertiserID        Numeric,
                                      AdvertiserName      char(50));
  CREATE TABLE MTV20030105Agency     (AgencyID           Numeric,
                                      AgencyName          char(50));
  CREATE TABLE MTV20030105Products   (ProductID           Numeric,
                                      ProductName         char(50));
  CREATE TABLE MTV20030105Programs   (ProgramID           Numeric,
                                      ProgramName         char(50));
  CREATE TABLE MTV20030105Spots      (AdInstID            Numeric,
                                      AdvertisementID     Numeric,
                                      KeyNumber           char(15),
                                      StationID           Numeric,
                                      SpotDateTime        DateTime,
                                      Duration            Numeric,
                                      BreakNumber         Numeric,
                                      NumAdsInBreak       Numeric,
                                      PositionInBreak     Numeric,
                                      PromoBefore         Numeric,
                                      PromoAfter          Numeric,
                                      AdTypeID            Numeric,
                                      CommentID           Numeric,
                                      ProgramID           Numeric,
                                      MarketInd           Numeric);
  CREATE TABLE MTV20030105Stations   (StationID           Numeric,
                                      NetworkID           Numeric,
                                      NetworkName         char(50),
                                      CallSign            char(50),
                                      LeadMarketInd       Numeric);
  CREATE INDEX MTV20030105AdLink_i1 ON MTV20030105AdLink(ProductID);
  CREATE INDEX MTV20030105Spots_i1 ON MTV20030105Spots (AdvertisementID);

Query:

  select SpotDateTime,NetworkName,CallSign,KeyNumber,Duration,ProgramName
  from MTV20030105Spots,MTV20030105Stations,MTV20030105Programs
  where MTV20030105Spots.AdvertisementID
  in (select AdvertisementID from MTV20030105AdLink where productid=90887)
  and MTV20030105Stations.StationID=MTV20030105Spots.StationID
  and MTV20030105Programs.ProgramID=MTV20030105Spots.ProgramID;

"ANALYZE;" was not executed with any version of SQLite, and running it does not speed up the query with recent versions.

I can provide the test database if required, just let me know.

2007-Sep-24 14:49:29 by anonymous:
This is not a bug. This question belongs on the mailing list.

If you run "EXPLAIN QUERY PLAN" on your SELECT you can see that your schema is not optimal, and results in full table scans for stations and programs:

  0|1|TABLE MTV20030105Stations
  1|2|TABLE MTV20030105Programs
  2|0|TABLE MTV20030105Spots WITH INDEX MTV20030105Spots_i1
  0|0|TABLE MTV20030105AdLink WITH INDEX MTV20030105AdLink_i1

If you change your schema as follows, it will use the indexes effectively:

  CREATE TABLE MTV20030105AdLink     (AdvertisementID     Numeric,
                                      AdRelationShipID    Numeric,
                                      AgencyID            Numeric,
                                      AdvertiserID        Numeric,
                                      ProductID           Numeric);
  CREATE TABLE MTV20030105AdType     (ADTypeID            INTEGER PRIMARY KEY,
                                      AdTypeName          char(50));
  CREATE TABLE MTV20030105Adjacency  (AdInstID            INTEGER PRIMARY KEY,
                                      ProductID           Numeric);
  CREATE TABLE MTV20030105Advertiser (AdvertiserID        INTEGER PRIMARY KEY,
                                      AdvertiserName      char(50));
  CREATE TABLE MTV20030105Agency     (AgencyID            INTEGER PRIMARY KEY,
                                      AgencyName          char(50));
  CREATE TABLE MTV20030105Products   (ProductID           INTEGER PRIMARY KEY,
                                      ProductName         char(50));
  CREATE TABLE MTV20030105Programs   (ProgramID           INTEGER PRIMARY KEY,
                                      ProgramName         char(50));
  CREATE TABLE MTV20030105Spots      (AdInstID            Numeric,
                                      AdvertisementID     Numeric,
                                      KeyNumber           char(15),
                                      StationID           Numeric,
                                      SpotDateTime        DateTime,
                                      Duration            Numeric,
                                      BreakNumber         Numeric,
                                      NumAdsInBreak       Numeric,
                                      PositionInBreak     Numeric,
                                      PromoBefore         Numeric,
                                      PromoAfter          Numeric,
                                      AdTypeID            Numeric,
                                      CommentID           Numeric,
                                      ProgramID           Numeric,
                                      MarketInd           Numeric);
  -- assumes that StationID is a unique identifier
  CREATE TABLE MTV20030105Stations   (StationID           INTEGER PRIMARY KEY,
                                      NetworkID           Numeric,
                                      NetworkName         char(50),
                                      CallSign            char(50),
                                      LeadMarketInd       Numeric);
  CREATE INDEX MTV20030105AdLink_i1 ON MTV20030105AdLink(ProductID);
  CREATE INDEX MTV20030105Spots_i1 ON MTV20030105Spots (AdvertisementID);

  explain query plan
  select SpotDateTime,NetworkName,CallSign,KeyNumber,Duration,ProgramName
  from MTV20030105Spots,MTV20030105Stations,MTV20030105Programs
  where MTV20030105Spots.AdvertisementID
  in (select AdvertisementID from MTV20030105AdLink where productid=90887)
  and MTV20030105Stations.StationID=MTV20030105Spots.StationID
  and MTV20030105Programs.ProgramID=MTV20030105Spots.ProgramID;

  0|0|TABLE MTV20030105Spots WITH INDEX MTV20030105Spots_i1
  1|1|TABLE MTV20030105Stations USING PRIMARY KEY
  2|2|TABLE MTV20030105Programs USING PRIMARY KEY
  0|0|TABLE MTV20030105AdLink WITH INDEX MTV20030105AdLink_i1


2007-Sep-24 15:12:03 by anonymous:
This query is faster:

  explain query plan
  select SpotDateTime,NetworkName,CallSign,KeyNumber,Duration,ProgramName
  from (select AdvertisementID from MTV20030105AdLink where productid=90887) ads,
       MTV20030105Spots,
       MTV20030105Stations,
       MTV20030105Programs
  where MTV20030105Spots.AdvertisementID = ads.AdvertisementID
  and MTV20030105Stations.StationID=MTV20030105Spots.StationID
  and MTV20030105Programs.ProgramID=MTV20030105Spots.ProgramID;

  0|0|TABLE MTV20030105AdLink WITH INDEX MTV20030105AdLink_i1
  1|1|TABLE MTV20030105Spots WITH INDEX MTV20030105Spots_i1
  2|2|TABLE MTV20030105Stations USING PRIMARY KEY
  3|3|TABLE MTV20030105Programs USING PRIMARY KEY


2007-Sep-24 15:45:58 by anonymous:
Point well taken ;-)

For completeness, here is the query plan produced by SQLite 3.3.4:

  0|0|TABLE MTV20030105Spots WITH INDEX MTV20030105Spots_i1
  1|1|TABLE MTV20030105Stations
  2|2|TABLE MTV20030105Programs
  0|0|TABLE MTV20030105AdLink WITH INDEX MTV20030105AdLink_i1

The table scan order is different which results in much faster execution times.

If anyone knows if this change was intentional in later versions, I'd be glad to read about it on the mailing list.

Also, out of plain curiosity: While I realize that the presented DB schema is not defined adequately, would anyone still consider that the new scan order behavior breaks backwards compatibility? For very large tables, the query becomes unusable after 3.3.4.

Meet you at the mailing list ...


2007-Sep-24 18:18:28 by anonymous:
The table/index scan order has changed many times from release to release, and has never been guaranteed. When the query optimizer was first introduced many queries relying on the join order specified in the FROM clause were slower and had to be reworked. Your previous defined schema was just lucky in this regard. If you have proper indexing you would not have had this performance degradation. For that matter, it would not hurt if you used ANALYZE as well.
 
2661 code closed 2007 Sep anonymous   2007 Sep   2 3 Single line, multi inserts edit
Using a wrapper for the DLL in Delphi, when doing single SQL statements, each row being inserted is successfully inserted as it should be.

When doing multiple inserts in one call though, only the first insert seems to be processed.

When using SQLITE3.EXE and I do the multi insert on one line, the transaction adds two rows to the DB.

In detail;

insert into tblfiles ([Name], fkid) values ('.bash_history',1); insert into tblfiles ([Name], fkid) values ('.rnd',1);

When I run this via SqlDB.ExecSQL in Delphi, only .bash_history shows up in tblfiles. When I run that statement in SQLITE3.EXE, both show up. The wrapper directly calls the DLL.

I'm pounding the DB pretty hard with a bulk inserts (45,000 inserts easy). One at a time takes a VERY long time.

Apparantly it was the wrapper that was causing the issue. Found another one and multi-inserts work just fine.
 
2660 code closed 2007 Sep anonymous   2007 Sep   4 4 Spelling error in comment for SQLITE_CONSTRAINT edit
"constraint" is mispelled as "contraint" once in sqlite3.h and once in sqliteInt.h.

In sqlite3.h

#define SQLITE_CONSTRAINT 19 /* Abort due to contraint violation */

In sqliteInt.h

/*
** SQLite supports many different ways to resolve a contraint
** error. ROLLBACK processing means that a constraint violation

Thanks.
 
2659 code fixed 2007 Sep rse   2007 Sep danielk1977 2 2 openDirectory function broken edit
IMHO the recently changed/introduced function src/os_unix.c:openDirectory() is broken in case the "zFilename" is just a filename without any path information in it (e.g. "foo.db"). In this case "ii" is 0 and as a result an uninitialized value "fd" is passed back and because of "fd" is uninitialized even the return code could be arbitrarily "SQLITE_OK" or "SQLITE_CANOPEN". The fix IMHO is to open "." in case no path information is present in "zFilename" and pass back an "fd" only if open(2) really was successful. Same for the return code. I append a patch which fixes this. But as I don't know what the original intentions were when this function was adjusted recently the patch should be reviewed by someone and then rejected or applied accordingly.
 
2658 code fixed 2007 Sep anonymous   2007 Sep   2 3 sqlite3VtabOverloadFunction broken edit
Triggered by a compiler warning ("rc" not initialized) I reviewed the function src/vtab.c:sqlite3VtabOverloadFunction() and came to the conclusion that it is broken at all:

1. The compiler is right that "rc" is not initialized before it is used for comparison.

2. If sqlite3DbStrDup fails a the uninitialized "xFunc" function pointer is passed back in the result structure.

I do not understand this piece of code in detail, so I can only guess what it should do. IMHO the function has to be fixed this way: if sqlite3DbStrDup fails one has to short-circuit the processing and immediately return/pass-through "pDef" as is. Then use xFindFunction and this way correctly initialize "rc" and then return "pDef" again if "rc" is now zero. I append a patch for fixing the code this way. I'm rather sure this is the way the function was intended to work, but as I'm not 100% sure please review it and apply or reject accordingly.

 
2657 build closed 2007 Sep anonymous   2007 Oct anonymous 1 1 fail to compile edit
I fail to compile.

It is a "win32 static library" project.

I simply create an empty project with "not using MFC" option and then add all files included in "sqlite-source-3_5_0.zip" into the project.

Debug configuration is fine, but I cannot compile release.

Information:

...sqlite-source-3_5_0\btree.c(3111) : fatal error C1001: INTERNAL COMPILER ERROR

  (compiler file 'F:\9782\vc98\p2\src\P2\regasg.c', line 646)

    Please choose the Technical Support command on the Visual C++

    Help menu, or open the Technical Support help file for more information

Error executing cl.exe.

2007-Sep-21 00:05:42 by anonymous:
I have the same problem. I am using the preprocessed source code in sqlite-source-3_5_0.zip with Visual Studio 6.0 SP6. Compiler version:

  Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 12.00.8804 for 80x86

The offending line of btree.c is line 3111, which looks fairly innocent to me:

  pCur->aOverflow[iIdx] = nextPage;

After investigating, it looks like this error is only emitted when global optimizations (/Og) are enabled, which is one of the switches set by optimizing for maximum speed.

As a workaround, you can insert the line

	#pragma optimize("g", off)

before line 3026 and the line

	#pragma optimize( "", on)

after line 3156, which are the beginning and ending of the function

	static int accessPayload(BtCursor,int,int,unsigned char,int,int)

This will disable global optimizations for the function, but leave them enabled for the rest of the file.

MSDN has more information about error C1001, but it is not quite the same as what the VS6 docs say:

This error is most often generated [due to] ... Failure of the code generator to find a way to generate correct code for a construct. This is most often caused by the interaction of an expression and an optimization option. The optimization has generated a tree which the compiler does not know how to handle. Such a problem can often be fixed by removing one or more optimization options when compiling the particular function containing the line indicated in the error message.


2007-Oct-12 18:53:47 by drh:
In other words, there is a bug in MSVC++ that prevents it from correctly compiling SQLite.

Shouldn't this be a problem reported to Microsoft? What exactly can we (the SQLite developers) do about this? Is there some simple code change that we can make to work around the bug in MSVC++?

 
2656 doc closed 2007 Sep anonymous   2007 Oct   2 1 missing documentation: END TRANSACTION edit
The documentation lists END [TRANSACTION] as a legal command, but does not describe it. It should be documented that is is the same as COMMIT [TRANSACTION].

Reference: http://www.mail-archive.com/sqlite-users@sqlite.org/msg03000.html

2007-Oct-01 13:32:34 by drh:
The END TRANSACTION syntax works but it is unsupported. If I document it that means I will have to begin supporting it, which is something I do not want to do.


2007-Oct-01 17:41:52 by anonymous:
The END DOCUMENTATION syntax is documented - but it is not described what it does.

While this might be intentional, I know about a novice user who was confused if he had to call END DOCUMENTATION after either COMMIT or ROLLBACK.

If you decide not to document END DOCUMENTATION, you might want to think about not documenting it at all, in other words, remove the syntax declaration.


2007-Oct-01 17:46:07 by drh:
OK. We have added documentation for END TRANSACTION.


2007-Oct-02 06:30:17 by anonymous:
Thanks!
 
2655 code closed 2007 Sep anonymous   2007 Sep   2 3 Reuse prepared statement after locked(5) gives called out of seq(21) edit
Reusing a prepared statement after locked(5) error gives library routine called out of sequence(21)

  Create a test db:
  sqlite3 test.db
  CREATE TABLE test (id INTEGER PRIMARY KEY AUTOINCREMENT, val INTEGER);
  go

  Lock db by running a statement in a transaction:
  begin transaction
  go
  delete from test where id = 99
  go

Try to run a prepared statement on the table via Perl DBI will give database is locked(5) at the first try. On following tries library routine called out of sequence(21) will be returned.

Only works if statement handle is undef:ed and prepared again.

If this is a problem in the SQLite DBD or in the SQLite library I don't know.

  Environment:
  FreeBSD 6.1
  Perl 5.8.8
  DBI 1.50
  SQLite DBD 1.13
  SQLite 3.4.2

  Test script:
  #!/usr/local/bin/perl
  use DBI;
  my $dbh = DBI->connect("dbi:SQLite:dbname=test.db","","",{RaiseError => 0}) or die "Couldn't open dbfile.";
  $dbh->func(1000, 'busy_timeout');
  my $sql = "DELETE FROM test WHERE id = ?";
  my $sth = $dbh->prepare($sql);
  die "Couldn't prepare" unless defined $sth;
  my $ret;
  while(! defined $ret){
        sleep 1;
        print "Execute sth\n";
        $ret = $sth->execute(452);
        print "ERROR: " . $sth->err . $sth->errstr . "\n" if $ret;
        $sth->finish;
  }
  if($ret){
        print "Failure\n";
  }
  else{
        print "Success\n";
  }

DBI->trace(5) is attached.

2007-Sep-19 11:47:29 by drh:
You appear to be calling sqlite3_finalize() (I'm guessing that is what the "finish" method does) on the statement after the error. sqlite3_finalize() is the destructor for a statement. Then you try to use the object after it has been destroyed. Most systems would segfault under those circumstances. SQLite is nice and gives you an SQLITE_MISUSE error.


2007-Sep-19 14:33:56 by anonymous:
How is the library supposed to be used in order to reuse the already prepared statement?

(According to perldoc DBI->finish is supposed to finish collection of data from an execution. It is not supposed to be destructive on the statement handle. So I guess that is a misuse of the library from SQLite DBD.)

If I leave out DBI->finish I get the same error.

I added the trace when not running finish.

 
2654 code closed 2007 Sep anonymous   2007 Oct jadams 1 1 sqlite3_prepare return 1(fail) edit
   result = sqlite3_exec( db, "create table MyTable_1sadsd ( ID integer primary key autoincrement, name nvarchar(32),comp nvarchar(32), file_content  blob )",NULL, NULL, errmsg );//ok
   result=sqlite3_prepare(db,"insert into MyTable_1sadsd ( name,comp,file_content ) values ( 'aaaa','aaaaaa1',? )",-1,&stat,&zLeftover );//fail

   //the result==1 so fail;

   //but

   result = sqlite3_exec( db, "create table MyTable( ID integer primary key autoincrement, name nvarchar(32),comp nvarchar(32), file_content  blob )",NULL, NULL, errmsg );//ok
   result=sqlite3_prepare(db,"insert into MyTable ( name,comp,file_content ) values ( 'aaaa','aaaaaa1',? )",-1,&stat,&zLeftover );//ok

   //is ok  ,because MyTable is shorter than MyTable_1sadsd ;
   //why ? tks
It's not clear from the report. There's nothing wrong with the SQL or th e C code shown. Maybe some other process is changing the schema or locking the dataabase or something. If you can post a complete program that demonstrates the problem, we will be able to help you.
 
2653 code active 2007 Sep anonymous   2007 Sep   3 4 Exclusive Transactions do not work with a Database File attached twice edit
Regarding the docs, it is possible to attach the same database file multiple times.

After doing so, I wanted to begin an exclusive transaction. Unfortunately, this fails ("database is locked") and surprises me as I did not find any notice on this particular situation and possible side-effects neither in the attach nor in the transaction/locking documentation.

If this behaviour is seen as an error, It would be useful to have it error fixed, because if one reads a list of (possibly duplicate) database files but with unique identifiers, it would be helpful to use these defined identifiers when accessing the databases (and that in an exclusive transaction).

Real-world case: Each "module" uses its own database file. Some modules share a database file. So there is a list of module -> database file assignments. Now an update process gets some database update scripts from the modules. Every module wants its changes to be done in the right database, so it relies on having its own database attached with an unique identifier - the module's name. On the other hand, the update process need exclusive access to the databases and starts a transaction -- bummer.

 
2652 code active 2007 Sep drh   2007 Sep   1 1 Aggregate function cannot be used from within a subquery edit
The following SQL fails:

   CREATE TABLE t1(x,y);
   INSERT INTO t1 VALUES(1,2);
   CREATE TABLE t2(z);
   INSERT INTO t2 VALUES(1);
   SELECT (SELECT y FROM t1 WHERE x=min(z)) FROM t2;

Problem reported on the mailing list.

2007-Sep-23 16:01:09 by anonymous:
Your syntax appears to be incorrect.

  SQLite v3.4.2
  CREATE TABLE t1(x,y);
  CREATE TABLE t2(z);
  INSERT INTO t1 VALUES(1,21);
  INSERT INTO t1 VALUES(2,22);
  INSERT INTO t1 VALUES(3,23);
  INSERT INTO t2 VALUES(3);
  INSERT INTO t2 VALUES(2);
  INSERT INTO t2 VALUES(1);

What you wanted to do:

  SELECT y FROM t1 WHERE x=(SELECT min(z) FROM t2);
  21    -- works as expected

What you did:

  SELECT (SELECT y FROM t1 WHERE x=min(z)) FROM t2;
  SQL error near line []: misuse of aggregate function min()
 
2651 new active 2007 Sep anonymous   2007 Sep   5 4 Add support for overriding home directory location edit
Currently, the history file's and the rc file's location is hard-wired to $HOME. It would be nice if this could be overridden. One way is to look for a SQLITE_HOME environment variable that points to the location to use. This can be achieved by a simple addition to find_home_dir() in src/shell.c.
 
2650 build closed 2007 Sep anonymous   2007 Sep   1 1 lack ; in os_win.c:1329 edit
Hi,

I want just to inform you that it misses one; in the file os_win.c (line 1329).

in winFullPathname, #if defined(__CYGWIN__) cygwin_conv_to_full_win32_path(zRelative, zFull); return SQLITE_OK <--- lack ';' #endif

Regards

YoCarBo

Duplicate of #2642 and #2647
 
2649 new active 2007 Sep anonymous   2007 Sep   4 4 Add an "--enable-extensions" (default=no) to the configure script edit
The attached patch adds "--enable-extensions" to the configure script, but is disabled by default (because of the security considerations of having it enabled).
 
2648 build fixed 2007 Sep anonymous   2007 Sep   1 1 compilation error for PocketPC/WindowsCE edit
function winOpen has some type errors under OS_WINCE preprocessor

#if OS_WINCE
  if( (flags & (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB)) ==
               (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB)
       && !winceCreateLock(zFileName, &f)
  ){
    CloseHandle(h);
    free(zConverted);
    return SQLITE_CANTOPEN;
  }
  if( dwFlagsAndAttributes & FILE_FLAG_DELETE_ON_CLOSE ){
    pFile->zDeleteOnClose = zConverted;
  }else
#endif
The line 18511 should be:

       && !winceCreateLock(zName, pFile)

Also it's have mistake in typing FILE_FLAG_DELETE_ON_CLOSE (with last underlines - FILE_FLAG_DELETEONCLOSE in line 18517).

Please change the sources to be compiled for WindowsCE.

BR. Yuri Noyanov

 
2647 code closed 2007 Sep anonymous   2007 Sep   1 1 sqlite-3.5.0 Cygwin: Compilation fails on src/os_win.c - ';' missing edit
There is a ';' symbol missing on line 1329 in file os_win.c (winFullPathname function) after return statement:
#if defined(__CYGWIN__)
  cygwin_conv_to_full_win32_path(zRelative, zFull);
  return SQLITE_OK
#endif
Duplicate of Ticket #2642
 
2646 code closed 2007 Sep anonymous   2007 Sep a.rottmann 4 1 sqlite3_get_table loses one table row edit
sqlite3_get_table seems to loose one row.

this patch fixes the problem for me. If it is wrong, please fix the bug in a right way.

--- sqlite-3.4.2/src/table.c 2007-06-12 15:18:01.000000000 +0300

+++ sqlite-3.4.2-x/src/table.c 2007-09-14 13:55:32.000000000 +0300

@@ -77,6 +77,7 @@ static int sqlite3_get_table_cb(void *pA

       }

       p->azResult[p->nData++] = z;

     }

+ p->nRow++;

   }else if( p->nColumn!=nCol ){

     sqlite3SetString(&p->zErrMsg,

        "sqlite3_get_table() called with two or more incompatible
queries",
Can you take a look at the documentation here and confirm that you are interpreting it correctly?

   http://www.sqlite.org/capi3ref.html#sqlite3_free_table

In particular, this bit:

"Notice that there is an extra row of data containing the column headers. But the *nrow return value is still 3. *ncolumn is set to 2. In general, the number of values inserted into azResult will be ((*nrow) + 1)*(*ncolumn)."

The code looks like it matches this description to me. If you find that it does not, please re-open this ticket. Thanks.

 
2645 build active 2007 Sep anonymous   2007 Sep   5 4 Conflict between tclConfig.sh and tclinstaller.tcl edit
I'm using a non-standard location for Tcl:

  --with-tcl=/home/scott/lib

The build process finds and uses the tclConfig.sh file from /home/scott/lib just fine, but tclinstaller.tcl is called without an explicit path and uses the system's tclsh instead of the one in /home/scott/bin and thus tries to install that portion of code into the system's area.

While I can change my path to resolve this, I think it makes more sense for tclinstaller.tcl to use the path information that's embedded in tclConfig.sh to be consistent.

/s.

 
2644 code closed 2007 Sep anonymous   2007 Sep   1 1 sqlite edit
Will you tell me why sqlite program is running on my laptop computer? Is it part of another program. Your answer and time is really appreciated. Thank you.
2007-Sep-13 18:00:08 by drh:
SQLite is used by Mcaffee Anti-virus, AOL Mail, Skype, Adobe Lightroom, and countless other programs, most of which we are completely unaware of.

This website is for reporting bugs against SQLite, not for getting help with figuring out what it is or why it is running on your computer. Please use the SQLite mailing list for such help.

 
2643 code fixed 2007 Sep anonymous   2007 Sep   1 1 bestIndex: Assertion `flags!=0' failed after index creation edit
I get an assertion crash on a database using sqlite 3.3.12 and also 3.4.2:

sqlite> create index idx_media_items_p1 on media_items (p1);
sqlite> select count(1) from media_items where p1 is not null;
lt-sqlite3: ./src/where.c:1486: bestIndex: Assertion `flags!=0' failed.
Aborted (core dumped)

Program received signal SIGABRT, Aborted.
[Switching to Thread 47454430964672 (LWP 24045)]
0x00002b28d839247b in raise () from /lib/libc.so.6
(gdb) bt
#0  0x00002b28d839247b in raise () from /lib/libc.so.6
#1  0x00002b28d8393da0 in abort () from /lib/libc.so.6
#2  0x00002b28d838bbf6 in __assert_fail () from /lib/libc.so.6
#3  0x00002b28d7fd3360 in bestIndex () from /home/steve/dev/sqlite-3.4.2/.libs/libsqlite3-3.4.2.so.0
#4  0x00002b28d7fd3f7c in sqlite3WhereBegin () from /home/steve/dev/sqlite-3.4.2/.libs/libsqlite3-3.4.2.so.0
#5  0x00002b28d7fb3bdc in sqlite3Select () from /home/steve/dev/sqlite-3.4.2/.libs/libsqlite3-3.4.2.so.0
#6  0x00002b28d7fa2568 in yy_reduce () from /home/steve/dev/sqlite-3.4.2/.libs/libsqlite3-3.4.2.so.0
#7  0x00002b28d7fa4e52 in sqlite3Parser () from /home/steve/dev/sqlite-3.4.2/.libs/libsqlite3-3.4.2.so.0
#8  0x00002b28d7fb5719 in sqlite3RunParser () from /home/steve/dev/sqlite-3.4.2/.libs/libsqlite3-3.4.2.so.0
#9  0x00002b28d7fa8e1d in sqlite3Prepare () from /home/steve/dev/sqlite-3.4.2/.libs/libsqlite3-3.4.2.so.0
#10 0x00002b28d7fa92f6 in sqlite3_prepare () from /home/steve/dev/sqlite-3.4.2/.libs/libsqlite3-3.4.2.so.0
#11 0x00002b28d7fd70df in sqlite3_exec () from /home/steve/dev/sqlite-3.4.2/.libs/libsqlite3-3.4.2.so.0
#12 0x0000000000406a74 in process_input ()
#13 0x0000000000407bc2 in main ()
core: http://skrul.com/core.gz

I can send the db file upon request.

2007-Sep-13 17:09:29 by anonymous:
Two additional notes: This does not crash on 3.3.4, plus this particular database was created using the "attach database" command from another database.


2007-Sep-13 17:18:42 by drh:
I need to know the schema for the media_items table.


2007-Sep-13 17:22:13 by anonymous:
steve@steve-linux:~/dev/sqlite-3.4.2/.libs$ ./sqlite3 /home/steve/dev/songbird/db/flat2.db
SQLite version 3.4.2
Enter ".help" for instructions
sqlite> .schema
CREATE TABLE media_items (
  media_item_id integer primary key autoincrement,
  guid text unique not null,
  created integer not null,
  updated integer not null,
  content_url text not null,
  content_mime_type text,
  content_length integer,
  hidden integer not null check(hidden in (0, 1)),
  media_list_type_id integer,
  p1 text,
  p1_sort text,
  p2 text,
  p2_sort text,
  p3 text,
  p3_sort text,
  p4 text,
  p4_sort text,
  p5 text,
  p5_sort text,
  p6 text,
  p6_sort text,
  p7 text,
  p7_sort text,
  p8 text,
  p8_sort text,
  p9 text,
  p9_sort text,
  p10 text,
  p10_sort text
, p11 text);
CREATE INDEX idx_media_items_p1 on media_items (p1);
CREATE INDEX idx_media_items_p11 on media_items (p11);
CREATE INDEX idx_media_items_p2 on media_items (p2);
sqlite>


2007-Sep-13 17:24:37 by drh:
I also want to see a ".dump" of the sqlite_stat1 table.


2007-Sep-13 17:25:52 by anonymous:
sqlite> .dump sqlite_stat1
BEGIN TRANSACTION;
ANALYZE sqlite_master;
INSERT INTO "sqlite_stat1" VALUES('media_items','idx_media_items_p11','61799 1');
INSERT INTO "sqlite_stat1" VALUES('media_items','sqlite_autoindex_media_items_1','61799 1');
COMMIT;
sqlite>


2007-Sep-13 18:03:19 by anonymous:
Great! I applied those changes to my sqlite 3.4.2 where.c and it fixed the problem. Thanks for the unbelievably fast fix!
 
2642 code closed 2007 Sep anonymous   2007 Sep   5 5 CYGWIN compile error due to missing semicolon edit
Missing semi-colon near line 1329 of os_win.c after SQLITE_OK.

   #if defined(__CYGWIN__)
     cygwin_conv_to_full_win32_path(zRelative, zFull);
     return SQLITE_OK
   #endif
Thanks.
 
2641 code closed 2007 Sep anonymous   2007 Sep   2 3 sqlite - Ticket #2638 continued as it was arbitrarily closed unsolved edit
to continue the thread from 2638. I tried as a blob and it was identical behaviour, this was not a workaround.

behaviour is different in different versions for test or blobs.

used to work, is now broken

Indeed, the behaviour has changed. SQLite is now less tolerant of non utf-8 encoded strings being handled as text. It is very likely that that is the reason your shiftjis data is being mangled. It should not of happened if the data was being treated as a blob.

Suggest going to the mailing-list for clarification. There are many clever people there who understand this kind of issue. If it does turn out to be a bug, we can reopen this ticket.

 
2640 code fixed 2007 Sep anonymous   2007 Sep   1 1 sqlite doesn't treat global identifiers correctly in a subquery edit
Apparently, sqlite doesn't treat global identifiers correctly in a subquery; in particular, with my database filling, the following query gives a wrong result (3 rows) whereas changing the innermost p.person_id into d.person_id (which are equal by the preceding condition) gives the correct result (56 rows).

  select distinct p.name
  from persons p, directors d
  where d.person_id=p.person_id and
      not exists
      ( select * from directors d1 where d1.person_id=p.person_id
       except
        select * from writers w
      )
2007-Sep-12 11:36:42 by danielk1977:
Can you provide a sample populated database? This would be a lot quicker to look into that way...

If it's too large to attach to this ticket, you can mail it to me: dan@sqlite.org


2007-Sep-12 14:30:33 by anonymous:
CREATE TABLE persons(person_id, name);
INSERT INTO "persons" VALUES(1,'fred');
INSERT INTO "persons" VALUES(2,'barney');
INSERT INTO "persons" VALUES(3,'wilma');
INSERT INTO "persons" VALUES(4,'pebbles');
INSERT INTO "persons" VALUES(5,'bambam');
CREATE TABLE directors(person_id);
INSERT INTO "directors" VALUES(5);
INSERT INTO "directors" VALUES(3);
CREATE TABLE writers(person_id);
INSERT INTO "writers" VALUES(2);
INSERT INTO "writers" VALUES(3);
INSERT INTO "writers" VALUES(4);

 select distinct p.name
  from persons p, directors d
  where d.person_id=p.person_id and
      not exists
      ( select * from directors d1 where d1.person_id=p.person_id
       except
        select * from writers w
      );
wilma


 select distinct p.name
  from persons p, directors d
  where d.person_id=p.person_id and
      not exists
      ( select * from directors d1 where d1.person_id=d.person_id
       except
        select * from writers w
      );
wilma
bambam
 
2639 code closed 2007 Sep anonymous   2007 Oct   1 2 Russian charset edit
At implementation of query, Russian charset is inserted like a "&#1113794;&#1113838;&#1113835;"
2007-Sep-12 02:29:24 by drh:
I do not understand the complaint.

But a good guess is that you are trying to use a codepage of some kind in a TEXT entry. SQLite understands only Unicode - either UTF8 or UTF16. Translate all of your strings into one of these encodings first, before inserting them into the database, if you want them to be compared correctly.

If you do not want to use unicode, insert your data as BLOBs instead of a strings.

 
2638 code closed 2007 Sep anonymous   2007 Sep   2 3 selects on japanese text okay in 3.3.5 bad in later versions edit
I have a database that was created in version 3.3.5 that contains japanese shift-jis text fields (a kanji lookup db). It works fine with the 3.3.5 dll. When I put the 3.4.2 or alpha 3.5 dll in a select of a japanese character does not match characters in the DB correctly, instead of returning 300 or so matching entries it returns 2000 non matching ones. I have a workaround in the sense that I can keep the current 3.3.5 dll in the app and it works but I'd like to move forward to new versions soon.

It's a database of japanese characters with japanese pronounciation and english meanings. format is code page 932. User puts a character in and it matches to a character in the system. replacing the dll and putting the same character in exactly (ie, paste current clipboard in both cases to be sure input codex is the same) and results are completely different.

2007-Sep-12 01:09:08 by drh:
SQLite uses Unicode, either UTF8 or UTF16. You can put in string that are not well-formed UTF8 or UTF16 and sometimes it might work. But later versions have become noticably less tolerate of illformed UTF8. This is due to security concerns. See, for example, ticket #2029.

If you want to store strings of codepage 932, I suggest you store them as BLOBs and not as text strings.

 
2637 code closed 2007 Sep anonymous   2007 Sep   1 1 Segfault in pthread_getspecific () edit
Hi,

I am trying to run Trac 0.11-dev with Apache 2.2.3-3.2ubuntu0.1/mod_python and libsqlite3 3.4.2. Accessing Trac over Apache causes segfault in libsqlite3. This has something to do with threading (models), I assume. I am not sure if this is an configuration issue, but I couldn't find any help after hard googling.

I tried also libqlite 3.3.x with the same result. I marked the bug with severity 1, since there is no known workaround AFAIK.

I compiled libsqlite3 to get the symbolic traceback:

#0  0xb7c521fa in pthread_getspecific () from /lib/tls/i686/cmov/libpthread.so.0
#1  0xb7d07f01 in sqlite3Insert (pParse=0x0, pTabList=0xb7d39bc8, pList=0xbfe16c98, pSelect=0xb7cebe8c,
    pColumn=0xb7cdf298, onError=1) at ./src/insert.c:1564
#2  0xb7d1e40e in sqlite3_get_table_cb (pArg=0xb7cdf298, nCol=1, argv=0xa582000, colv=0xb7d39bc8)
    at ./src/table.c:107
#3  0xb7cebe8c in sqlite3AuthRead (pParse=0xb7cebe8c, pExpr=0xbfe16c98, pTabList=0xb7d39bc8)
    at ./src/auth.c:118
#4  0xb7cebf18 in sqlite3AuthRead (pParse=0xb7d39bc8, pExpr=0xa582000, pTabList=0x1) at ./src/auth.c:140
#5  0xb7d118b6 in sqlite3Parser (yyp=0xa6ff558, yymajor=<value optimized out>, yyminor=
      {z = 0xffffffff <Address 0xffffffff out of bounds>, dyn = 0, n = 0}, pParse=0xbfe16e34)
    at parse.y:468
#6  0xb7d11fc1 in sqlite3Parser (yyp=0xa6ff558, yymajor=<value optimized out>, yyminor=
      {z = 0xffffffff <Address 0xffffffff out of bounds>, dyn = 0, n = 1609611034}, pParse=0xbfe16e38)
    at parse.y:651
#7  0xb7d2f625 in getMask (pMaskSet=0x1, iCursor=1) at ./src/where.c:289
#8  0xb7d12243 in sqlite3Parser (yyp=0xbfe16ef8, yymajor=<value optimized out>, yyminor=
      {z = 0xbfe16f08 "\222\023ӷ\232cӷ\200dӷ", dyn = 0, n = 0}, pParse=0x0) at parse.y:706
#9  0xb7d12347 in sqlite3Parser (yyp=0xb7cdffdc, yymajor=<value optimized out>, yyminor=
      {z = 0xbfe1734c "X�o\n", dyn = 0, n = 0}, pParse=0xbfe16f78) at parse.y:735
#10 0xb7d1279c in sqlite3Parser (yyp=0xa6ff558, yymajor=<value optimized out>, yyminor=
      {z = 0xa43a300 "system", dyn = 0, n = 86104312}, pParse=0xb7d39bc8) at parse.y:768
#11 0xb7d1288f in sqlite3Parser (yyp=0xbfe1734c, yymajor=<value optimized out>, yyminor=
      {z = 0xa760dc8 "�>u\no", dyn = 0, n = 86104336}, pParse=0xbfe17048) at parse.y:812
#12 0xb7cf46da in sqlite3BtreeClose (p=0x0) at ./src/btree.c:1236
#13 0xb7d16c56 in base_vprintf (xRealloc=0, useInternal=-1210868792,
    zInitBuf=0x1 <Address 0x1 out of bounds>, nInitBuf=-1211228996, zFormat=0xa760dc8 "�>u\no",
    ap=0xb7d36480 "CREATE TABLE sqlite_master(\n  type text,\n  name text,\n  tbl_name text,\n  rootpage integer,\n  sql text\n)") at ./src/printf.c:785
#14 0xb7d173ce in base_vprintf (xRealloc=0xbfe16f28, useInternal=-1210882944,
    zInitBuf=<value optimized out>, nInitBuf=-1075743924, zFormat=<value optimized out>,
---Type <return> to continue, or q <return> to quit---
    ap=0xe9e1744a <Address 0xe9e1744a out of bounds>) at ./src/printf.c:589
#15 0xb7d179d6 in base_vprintf (xRealloc=0xbfe17038, useInternal=0, zInitBuf=<value optimized out>,
    nInitBuf=-1075743924, zFormat=<value optimized out>, ap=0xbfe17090 "\030r��") at ./src/printf.c:548
#16 0xb7d0d75d in pager_playback (pPager=0x68, isHot=16777221) at ./src/pager.c:1474
#17 0xb7d1b000 in prepSelectStmt (pParse=0x5cc2c, p=0xb7cdd000) at ./src/select.c:1368
#18 0xb7d1191f in sqlite3Parser (yyp=0xa6ff558, yymajor=<value optimized out>, yyminor=
      {z = 0xffffffff <Address 0xffffffff out of bounds>, dyn = 0, n = 0}, pParse=0xa73708c) at parse.y:480
#19 0xb7d11fc1 in sqlite3Parser (yyp=0xa6ff558, yymajor=<value optimized out>, yyminor=
      {z = 0xffffffff <Address 0xffffffff out of bounds>, dyn = 0, n = 87668806}, pParse=0xbfe17488)
    at parse.y:651
#20 0xb6810cef in statement_create () from /usr/lib/python2.5/site-packages/pysqlite2/_sqlite.so
#21 0xb680c7ad in connection_call () from /usr/lib/python2.5/site-packages/pysqlite2/_sqlite.so
#22 0xb7547987 in PyObject_Call () from /usr/lib/libpython2.5.so.1.0
#23 0xb7547c03 in ?? () from /usr/lib/libpython2.5.so.1.0
#24 0x0a681a70 in ?? ()
#25 0x0a4154ec in ?? ()
#26 0x00000000 in ?? ()
2007-Sep-11 14:47:34 by anonymous:
Also, Trac works fine if started using tracd standalone daemon tool


2007-Sep-11 14:47:34 by anonymous:
Could you try sqlite 3.5? The threading code has been completely redone.


2007-Sep-11 14:57:17 by anonymous:
No dice. libsqlite 3.5.0:

#0  0xb7c521fa in pthread_getspecific () from /lib/tls/i686/cmov/libpthread.so.0
#1  0xb7d07f01 in sqlite3Insert (pParse=0x0, pTabList=0xb7d39bc8, pList=0xbfe16c98, pSelect=0xb7cebe8c,
    pColumn=0xb7cdf298, onError=1) at ./src/insert.c:1465
#2  0xb7d1e40e in sqlite3Select (pParse=0xb7cdf298, p=0x1, eDest=171204008, iParm=-1210868792,
    pParent=0x0, parentTab=0, pParentAgg=0xbfe16ca8, aff=0xb7cebf18 "�\205�\017\225�\017���\215�&")
    at ./src/select.c:2915
#3  0xb7cebe8c in attachFunc (context=0x0, argc=0, argv=0x0) at ./src/attach.c:82
#4  0xb7cebf18 in attachFunc (context=0x0, argc=-1211232372, argv=0x1) at ./src/attach.c:114
#5  0xb7d118b6 in sqlite3PagerWrite (pDbPage=0xa8fa470) at ./src/pager.c:4156
#6  0xb7d11fc1 in sqlite3PagerOpen (pVfs=0xa8fa470, ppPager=0xb7d36480,
    zFilename=0xffffffff <Address 0xffffffff out of bounds>, nExtra=-1075745228, flags=-1075745224,
    vfsFlags=1) at ./src/pager.c:2110
#7  0xb7d2f625 in sqlite3VdbeHalt (p=0xa8fa470) at ./src/vdbeaux.c:1142
#8  0xb7d12243 in yy_find_shift_action (pParser=0x0, iLookAhead=1 '\001') at parse.c:1442
#9  0xb7d12347 in yy_destructor (yymajor=<value optimized out>, yypminor=0x1) at parse.c:1337
#10 0xb7d1279c in sqlite3Parser (yyp=0xa8fa470, yymajor=<value optimized out>, yyminor=
      {z = 0xa8fec48 "system", dyn = 0, n = 86973048}, pParse=0xb7d39bc8) at parse.c:3268
#11 0xb7d1288f in sqlite3Parser (yyp=0xbfe1734c, yymajor=-1075744936, yyminor=
      {z = 0xa906428 "�l\212\no", dyn = 0, n = 86973072}, pParse=0xbfe17048) at parse.c:3477
#12 0xb7cf46da in ?? () at ./src/btree.c:2498 from /usr/lib/libsqlite3.so.0
#13 0xbfe1734c in ?? ()
#14 0xb7d39bc8 in ?? () from /usr/lib/libsqlite3.so.0
#15 0x0a906428 in ?? ()
#16 0x0a5e3520 in ?? ()
#17 0xbfe17048 in ?? ()
#18 0xb7f18300 in ?? () from /lib/ld-linux.so.2
#19 0xb7d16c56 in sqlite3Pragma (pParse=0xbfe1734c, pId1=0xa8fec48, pId2=0x0, pValue=0xb757a700,
    minusFlag=173792084) at ./src/pragma.c:601

2007-Sep-11 14:58:54 by anonymous:
Just a shot in the dark - do you have the correct system header files for your version of GLIBC?


2007-Sep-11 15:02:42 by anonymous:
sqlite 3.5 does not call pthread_getspecific


2007-Sep-11 15:59:51 by anonymous:
After a little fight, I rollbacked everything on my system to older versions. I'll see whether this problem reappears when Trac 0.11 is released. Meanwhile you can close the ticket, since I cannot provide anymore further info.

It must be some sort of configuration issue. Glibc headers might be a good guess. Maybe the latest 3.5.0 was not installed correctly.


2007-Sep-11 16:17:03 by drh:
Anonymous users can close tickets in CVSTrac. You could have closed this ticket yourself.
 
2636 code closed 2007 Sep anonymous   2007 Sep   1 1 Bug in sqlite3_prepare_v2 edit
If we prepare two statements using sqlite3_prepare_v2, lets say

DELETE FROM table WHERE ID = ?;
and
UPDATE table SET ID = ID - ? WHERE ID > ?;

at the same time, bind values to it

Then execute each of the statements, they get executed, but the changes to the database from the second (UPDATE) statement does not show up in the database.

2007-Sep-11 07:21:12 by danielk1977:
Can you post a short program to demonstrate the problem? From the report, I'm not sure exactly how to reproduce it.

2007-Sep-12: by drh:
Unable to reproduce and original submitter is not responding.

 
2635 code closed 2007 Sep anonymous   2007 Sep   3 2 problem with calculation of juliand date edit
in the calculation of the function computeYMD (in date.c) the date corresponding of the julian day 77528 is 29 february 300 (which should not exist) instead of 1 march 300, as matter of fact the next day is 2 march 300. This problem is present every 400 years backward. Thanks.
2007-Sep-08 21:46:10 by drh:
The date and time functions in SQLite render all dates according to the gregorian calendar. This is by design.
 
2634 code active 2007 Sep anonymous   2007 Sep   3 3 .schema uses incorrect ORDER BY giving wrong dependency order edit
When the schema is exported, views are sorted by name instead of by dependency. If there are nested views, the schema may be invalid when used to re-create the database.

  sqlite3
  create table t ( f text );
  create view v2 as select f from t;
  create view v1 as select f from v2;
  .output test.txt
  .schema
  .exit

  sqlite3
  .read test.txt
  SQL error near line 2: no such table: main.v2
2007-Sep-07 15:33:06 by anonymous:
Use .dump instead as a workaround. Unlike .schema, .dump does not use ORDER BY in its queries on sqlite_master and it outputs its rows in order of entry.

SQLite version 3.5.0
Enter ".help" for instructions
sqlite> create table t ( f text );
sqlite> create view v2 as select f from t;
sqlite> create view v1 as select f from v2;
sqlite>
sqlite> .schema
CREATE TABLE t ( f text );
CREATE VIEW v1 as select f from v2;
CREATE VIEW v2 as select f from t;
sqlite>
sqlite> .dump
BEGIN TRANSACTION;
CREATE TABLE t ( f text );
CREATE VIEW v2 as select f from t;
CREATE VIEW v1 as select f from v2;
COMMIT;
Suggested patch:

Index: src/shell.c
===================================================================
RCS file: /sqlite/sqlite/src/shell.c,v
retrieving revision 1.167
diff -u -3 -p -r1.167 shell.c
--- src/shell.c 7 Sep 2007 01:12:32 -0000       1.167
+++ src/shell.c 7 Sep 2007 15:28:24 -0000
@@ -1411,8 +1411,7 @@ static int do_meta_command(char *zLine,
           "SELECT sql FROM "
           "  (SELECT * FROM sqlite_master UNION ALL"
           "   SELECT * FROM sqlite_temp_master) "
-          "WHERE tbl_name LIKE shellstatic() AND type!='meta' AND sql NOTNULL "
-          "ORDER BY substr(type,2,1), name",
+          "WHERE tbl_name LIKE shellstatic() AND type!='meta' AND sql NOTNULL",
           callback, &data, &zErrMsg);
         zShellStatic = 0;
       }
@@ -1421,8 +1420,7 @@ static int do_meta_command(char *zLine,
          "SELECT sql FROM "
          "  (SELECT * FROM sqlite_master UNION ALL"
          "   SELECT * FROM sqlite_temp_master) "
-         "WHERE type!='meta' AND sql NOTNULL AND name NOT LIKE 'sqlite_%'"
-         "ORDER BY substr(type,2,1), name",
+         "WHERE type!='meta' AND sql NOTNULL AND name NOT LIKE 'sqlite_%'",
          callback, &data, &zErrMsg
       );
     }
after patch:

SQLite version 3.5.0
Enter ".help" for instructions
sqlite>  create table t ( f text );
sqlite>   create view v2 as select f from t;
sqlite>   create view v1 as select f from v2;
sqlite> .schema
CREATE TABLE t ( f text );
CREATE VIEW v2 as select f from t;
CREATE VIEW v1 as select f from v2;
sqlite> .q
 
2633 build fixed 2007 Sep anonymous   2007 Sep   1 1 building with gcc3.3.4 - undefined reference to `sqlite3KeywordCode' edit
I'm using gcc 3.3.4 with glibc 2.3.2 trying to cross compile for arm.

My configure command line:

../sqlite-3.4.2/configure --disable-readline --disable-tcl --host=arm-linux --prefix=/usr/local/opt/crosstool/arm-linux/gcc-3.3.4-glibc-2.3.2/arm-linux/

During the build I have to compile lemon and mkkeywordhash for my native system and restart the build. Otherwise the build goes smoothly until I get here:

... ranlib .libs/libsqlite3.a creating libsqlite3.la (cd .libs && rm -f libsqlite3.la && ln -s ../libsqlite3.la libsqlite3.la) ./libtool --mode=link gcc -I/usr/local/opt/crosstool/arm-linux/gcc-3.3.4-glibc-2.3.2/arm-linux/include -I. -I../sqlite-3.4.2/src -DNDEBUG -DTHREADSAFE=0 -DSQLITE_THREAD_OVERRIDE_LOCK=-1 -DSQLITE_OMIT_LOAD_EXTENSION=1 -L/usr/local/opt/crosstool/arm-linux/gcc-3.3.4-glibc-2.3.2/arm-linux/lib -DHAVE_READLINE=0 \ -o sqlite3 ../sqlite-3.4.2/src/shell.c libsqlite3.la \

gcc -I/usr/local/opt/crosstool/arm-linux/gcc-3.3.4-glibc-2.3.2/arm-linux/include -I. -I../sqlite-3.4.2/src -DNDEBUG -DTHREADSAFE=0 -DSQLITE_THREAD_OVERRIDE_LOCK=-1 -DSQLITE_OMIT_LOAD_EXTENSION=1 -DHAVE_READLINE=0 -o .libs/sqlite3 ../sqlite-3.4.2/src/shell.c -L/usr/local/opt/crosstool/arm-linux/gcc-3.3.4-glibc-2.3.2/arm-linux/lib ./.libs/libsqlite3.so -Wl,--rpath -Wl,/usr/local/opt/crosstool/arm-linux/gcc-3.3.4-glibc-2.3.2/arm-linux//lib ./.libs/libsqlite3.so: undefined reference to `sqlite3KeywordCode' ./.libs/libsqlite3.so: undefined reference to `keywordCode' collect2: ld returned 1 exit status make: *** [sqlite3] Error 1

I'm trying to build the shared libraries for use on an arm system. I'd just build the sqlite3.c amalgamation into my apps, but I had a grocery list of build errors in gcc 4.1.2 on my native system and I didn't think my gcc3.3.4 cross-compiler would fair much better. So I'd like to at least get the shared libs working.

Any help would be appreciated.

I figured out that the build does not warn you if keywordhash.h is not created. Perhaps it should or the build will fail. So after compiling the mkkeywordhash tool for the native machine:

gcc -g -o mkkeywordhash ../sqlite-3.4.2/tool/mkkeywordhash.c

I then had run: ./mkkeywordhash >keywordhash.h

... manually before restarting the build.

Build completes without errors now, but build tools/code generators are inherently non-cross-compiler friendly when they are part of the main build process and have to be compiled into machine executable form before being run. Any possibility of moving these to an interpreter-based language that doesn't require compiling?

 
2632 code review 2007 Sep anonymous   2008 Jan danielk1977 5 2 sqlite3* not available within loadable extension <sqlite3ext.h> edit
It is not really a bug but it may be the right moment to introduce a valuable improvement for developers of loadable extensions, which are really great (may be for all and in general) with the migration to version 3.5 branch.
By standard usage of sqlite3ext.h within loadable extensions source code there is common to all functions the 1st parameter as sqlite3_context* context, as you know. Unfortunately there is no database context direct accessible from this. However, if you take the definitions of Mem and sqlite3_context out of vdbeInt.h and remove type by space holders, then
sqlite3* db = context->s.db;
is the required variable for use with sqlite3_prepare, sqlite3_step and sqlite3_finalize etc. (That's my work-around)
What is needed only for loadable extension developers to stay aligned with new versions is the re-arrangement of existing
struct sqlite3_context {Mem s} and struct Mem {sqlite3 *db}
to become able to simply typecast sqlite3_context* into a sqlite3*; all the other internals of Mem would stay private to vdbe even when version numbers will grow.
And now to answer the question "why the hell is he asking for this?" here my explanation by an example of what I made running:
select execSQL("select 'iniTable' as Title;select * from iniTable", '|', X'0D0A', X'0D0A'||'----'||X'0D0A' );

iniTable
----
Common|Integer|4711
Common|Float|3.1416927
Common|Double|3.1416927
Common|Text|Dies ist Text
Common|Blob|X'0D0A'

Think about an application which has to create a whole InMemory-Database table structure inclusive triggers and views:

select execSQL(stmt) from StatementTable where Usage='InMemory';
And all within one and the same transaction. It is really that easy! This could answer also some points in the wishlist of the SQLite3 web site; all to do this is reordering Mem s in sqlite3_context and sqlite3*db in struct Mem to the beginning of the structures.
Doing this with the release of version 3.5 is really a good moment, isn't it?
2007-Sep-07 11:54:38 by danielk1977:
The usual approach is to pass the sqlite3* handle to sqlite3_create_function() as the 5th argument and then retrieve it from within the user function implementation using sqlite3_user_data().

If you depend on the definitions of internal data structures, you're living dangerously. ;)


2007-Sep-07 12:54:56 by anonymous:
Is then the following correct and will work in the same expected way (same pointer within ld-ex-function)? Even with sqlite_auto_extension called at DLL load time? (Win32 with Amalgamation and BDS2006)
#include <sqlite3ext.h>
SQLITE_EXTENSION_INIT1

int sqlite3_extension_init( sqlite3 *db, char **pzErrMsg, const sqlite3_api_routines *pApi )
{
  SQLITE_EXTENSION_INIT2(pApi)
  sqlite3_create_function(db, "execSQL",  -1, SQLITE_UTF8, db, execSqlFunc, 0, 0);
...
}

static void execSqlFunc(
  sqlite3_context *context, int argc, sqlite3_value **argv )
{
  sqlite3 *db = sqlite3_user_data(context );
/*
  instead of using context's internals ( sqlite3 *db = context->s.db );
  or waiting for reordering of them and using typecase ( sqlite3 *db = (sqlite3*)context; )
  respective ( sqlite3 *db = *(sqlite3**)context; )?
*/
...
}

Thank's for the smilie and your hint. Of cause, I do not want to live dangerous. This was the purpose of my posting to get this fixed officially with the next release. But if you say it is not worth the "small change" of reordering and I'm doing it correct this way, then that's fine. I started similar to your proposal by using a static variable and assigned it from within sqlite3_extension_init but this would not work with more than one db-connection (the last one would win).

The other open point for me is the right place and methode to link the same source code as used for the Loadable Extension Functions to become a DLL (for SQlite3.exe .load) as auto-loaded extensions into a specialized made of SQLite3.DLL for my applications. Do you have another trick than modifying the sqlite3_open-functions with compiler switches? This would be great. Even if above statements work, do you think the reordering makes sense for an easier transfer for the sqlite3* into the loadable extension function? The 5th argument would be kept free for other purposes. What is your estimation of likelyhood for that proposed change? What's against it?


2007-Sep-07 14:11:38 by danielk1977:
The code above is correct as far as I can see.

As far as reordering structures to allow tricky casting, there's no particular argument against, other than it makes the code a bit tricky to follow. And limits the ways in which we can change the contents of the currently opaque sqlite3_context structure in the future. Something like:

  sqlite3 *sqlite3_get_connection(sqlite3_context *p){
    return p->s.db;
  }

might be a chance I guess.


2007-Sep-10 18:58:09 by anonymous:
That is a very good idea. Now I have a case, where I would like to make use of both, the db connection AND the user data:
I have two database selections (by a list of keys) and I would like to used one function to compare "equal" and another "unequal". In this case I could use the same function body and use the userdata to distinguish between the cases equal and unequal while using the db connection to query the selection with the given keys.
Using 2 functions which are sharing the same body of a 3rd function will be my workaround until this new API function becomes available with the next 3.5.x version. Many thanks.


2007-Sep-10 19:09:51 by anonymous:
I agree. It is very reasonable to want to use an sqlite3* connection from within a registered function as well. Scenario - perform an SQL query on same connection from within a user-defined function. Setting user data just for this common action is awkward.


2008-Jan-08 14:46:14 by anonymous:
when will this function become usable? will it come with 3.5.5 ?
sqlite3 *sqlite3_get_connection(sqlite3_context *p)
 
2631 code fixed 2007 Sep anonymous   2007 Sep   3 3 Comments at end of "CREATE TABLE" cause SQL error. edit
Comments after the semi colon of a "CREATE TABLE" statement that use the double-dash format cause an Incomplete SQL error in the Windows command line program when read in with the ".read" command.

The following SQL statement read in from a file through the ".read" command will generate the error:

CREATE TABLE test (abc, def); -- this comment causes error

With the comment removed the read works correctly. The error doesn't happen if the statement is typed into the console.

 
2630 code closed 2007 Sep anonymous   2007 Oct a.rottmann 2 2 sqlite3PagerOpen set a wrong temp file name in pPager->zFilename edit
in pager.c line 1783
1783 rc = sqlite3PagerOpentemp(&fd);
1784 sqlite3OsTempFileName(zTemp);
1785 zFilename = zTemp;
1786 zFullPathname = sqlite3OsFullPathname(zFilename);
1787 if( rc==SQLITE_OK ){
1788 tempFile = 1;
.
.
.
1823 memcpy(pPager->zFilename, zFullPathname, nameLen+1);

function sqlite3PagerOpentemp on line 1783 open the fd using e new temp file name
but the the function on 1784 get another temp file name and
assign it to pPager->zFilename on line 1823

I compile sqlite on POCKETPC and temp file were not delete automatically by the OS
so I have uncommented the code that delete temp file in function sqlite3PagerClose
/* Temp files are automatically deleted by the OS*/
if( pPager->tempFile ){
sqlite3OsDelete(pPager->zFilename);
}

but the sqlite3OsDelete(pPager->zFilename); use the wrong temp file name.

I have change the previous code with this one.

1783 // rc = sqlite3PagerOpentemp(&fd);
1784 sqlite3OsTempFileName(zTemp);
1785 zFilename = zTemp;
1786 zFullPathname = sqlite3OsFullPathname(zFilename);

1787 do{
1788 cnt--;
1789 rc = sqlite3OsOpenExclusive(zFullPathname, &fd, 1);
1790 assert( rc!=SQLITE_OK || fd );
1791 }while( cnt>0 && rc!=SQLITE_OK && rc!=SQLITE_NOMEM );
1792
1793 if( rc==SQLITE_OK ){
1794 tempFile = 1;
1795 }

and all is fine.

I hope this is useful.
thanks.
by.

2007-Sep-06 23:25:20 by drh:
The code in question was restructured and rewritten for SQLite version 3.5.0.

But looking into this did bring to my attention a bug in the implementation of 3.5.0, for which I am grateful.

 
2629 doc active 2007 Sep anonymous   2007 Sep   4 4 typo in os_unix.c for nolock edit
Index: src/os_unix.c
===================================================================
RCS file: /sqlite/sqlite/src/os_unix.c,v
retrieving revision 1.165
diff -u -3 -p -r1.165 os_unix.c
--- src/os_unix.c       5 Sep 2007 13:56:32 -0000       1.165
+++ src/os_unix.c       6 Sep 2007 17:53:47 -0000
@@ -2126,7 +2126,7 @@ static const sqlite3_io_methods sqlite3D

 /*
 ** This vector defines all the methods that can operate on an sqlite3_file
-** for unix with dotlock style file locking.
+** for unix with nolock style file locking.
 */
 static const sqlite3_io_methods sqlite3NolockLockingUnixIoMethod = {
   1,                        /* iVersion */
 
2628 code fixed 2007 Sep anonymous   2007 Sep   3 4 Bad error handling for invalid page-size edit
When opening a database which has an invalid page-size, sqlite3BtreeOpen() decides to use default values.

This prevents the assert in sqlite3PagerSetPagesize() (which would at least show the cause of the problem) and subsequently brings obscure errors.

It would be preferable to return a database error when this is detected.

I had foolishly reduced the value of SQLITE_MAX_PAGE_SIZE for use in embedded devices, and noticed this when trying to open a database created with a larger page size.

 
2627 code active 2007 Sep anonymous   2007 Sep   3 2 Improper parsing of nested JOIN edit
SQLite has a problem with multiple nested JOINs. The only way to get it workig is to remove the surrounding brackets. Removing the brackets unfortunately do not work in other DB systems such as MS SQL, mysql etc.

This does not work:

Select ContactPhone.* From (ContactPhone LEFT OUTER JOIN ContactLocation ON ContactPhone.PHNLCT_ID = ContactLocation.LCT_ID) LEFT OUTER JOIN ContactItem ON ContactLocation.LCTITM_ID = ContactItem.ITM_ID

(It complains about LCT_ID or similar)

This works after removing the brackets:

Select ContactPhone.* From ContactPhone LEFT OUTER JOIN ContactLocation ON ContactPhone.PHNLCT_ID = ContactLocation.LCT_ID LEFT OUTER JOIN ContactItem ON ContactLocation.LCTITM_ID = ContactItem.ITM_ID

All other major DB systems require the surrounding brackets. Do you think it is possible to fix it? Apart from this little little SQLite is an awesome project.

Thank you Jakub Klos

2007-Sep-06 13:07:32 by anonymous:
I don't have access to MS SQL Server, but MySQL and Oracle have no issue with the query without parentheses:

create table x1(a int, b int);
create table x2(c int, d int);
create table x3(e int, f int);

mysql> select x1.* from x1 left join x2 on x1.a=x2.c left join x3 on x2.d=x3.e;
Empty set (0.00 sec)

2007-Sep-06 19:03:33 by anonymous:
MSSQL also has no problems without the parens. As a matter of fact, the only DB that I know of that requires them is MS Access (JET).


2007-Sep-06 20:11:31 by anonymous:
I guess he had no luck filing a JET bug.


2007-Sep-11 17:22:28 by anonymous:
True, MS access requires the parens but all other major DBs support the query syntax with the parens. So why SQLite does not like it? It should simply ignore them if possible. Thank you
 
2626 code closed 2007 Sep anonymous   2007 Sep   1 1 The Amalgamation compile error edit
when i compile The Amalgamation 3.4.2 with visual studio 2003 in window, i encounted 3 errors :

   'sqlite3IsIdChar':unknown size
   'sqlite3OpcodeNames':unkown size
   'sqlite3UpperToLower':unkown size

i think it is caused by following source code in sqlite3.c: #ifndef SQLITE_PRIVATE # define SQLITE_PRIVATE static #endif

it should be:

   #ifndef SQLITE_PRIVATE
     # define SQLITE_PRIVATE
   #endif

is that right?

It looks like vs2003 doesn't like this kind of forward decl:

   static int aArray[];
   ...
   static int aArray[] = {1, 2, 3};

But is fine with this:

   int aArray[];
   ...
   int aArray[] = {1, 2, 3};

By defining SQLITE_PRIVATE as an empty string, you are avoiding constructs like the first block above. It's not ideal, but not harmful either.


2007-Sep-06 13:11:51 by anonymous:
This is fixed in 3.5.
 
2625 build closed 2007 Sep anonymous   2007 Sep   2 2 http://www.sqlite.org/cvstrac/wiki?p=HowToCompileWithVsNet has problem edit
when i compile 3.4.2 with visual studio 2003 in windows, i do as http://www.sqlite.org/cvstrac/wiki?p=HowToCompileWithVsNet told me to. but step 8 "Add all the .c and .h files that you unzipped, except for: tclsqlite.c and shell.c.Note: You may add tclsqlite.c and shell.c, but then you have to define the preprocessor-symbol NO_TCL:" won't work. i suggest to modify it to be "Add all the .c and .h files that you unzipped, except for: tclsqlite.c shell.c and icu.c. Note: You may add tclsqlite.c and shell.c, but then you have to define the preprocessor-symbol NO_TCL, you may add icu.c, but then you have to define the preprocessor-symbol SQLITE_CORE". why not supply a .sln file? this will save many people's time. thank you.
2007-Sep-06 07:04:34 by danielk1977:
What you suggest sounds like an improvement. I'd add the text in myself, but I don't have visual studio, so I can't test it.

But the wiki pages are all user editable, so feel free to modify it yourself by clicking the "Edit" link found at the top right-hand side of the page.

If you have a *.sln file that you can attach to the page, even better.

 
2624 code fixed 2007 Sep anonymous   2007 Sep   1 1 Invalid documentation references in sqlite.h.in edit
Documentation in sqlite.h.in contains invalid references to the following functions and #defines:

  • SQLITE_LOCK_READ -> _SHARED
  • sqlite3_register_vfs -> vfs_register
  • sqlite3_unregister_vfs -> vfs_unregister
  • sqlite3_find_vfs -> vfs_find
  • sqltie3_blob_size -> sqltie3_blob_bytes
  • sqlite3_finalise -> sqlite3_finalize ( s vs. z)
  • sqlite_column_text -> missing 3
  • sqltie3_blob_size -> ti mixed up
 
2623 code fixed 2007 Sep anonymous   2007 Sep   1 1 zeroblob(-1) returns non-NULL from sqlite3_column_blob() edit
Please consider this psydo-code:

  sqlite3_prepare ("select zeroblob(-1)", DB, &Stmt);
  sqlite3_step(Stmt);
  sqlite3_column_blob(Stmt, 0) == NULL; // Error here!

sqlite3_column_blob() used to return NULL for "zeroblob(-1)" as well as any other negative length value for SQLite 3.4.2.

After the 3.5.0 changes it returns an arbitrary non-NULL value. sqlite3_column_bytes() returns 0 both before and after the changes.

I suppose that the pre 3.5.0 NULL result is correct and meaningful. Or is it in fact undefined?

 
2622 doc fixed 2007 Sep anonymous   2007 Sep   3 3 SQLITE_OMIT_MEMORY_ALLOCATION undocumented unresolved functions edit
When SQLITE_OMIT_MEMORY_ALLOCATION is defined, in addition to the functions sqlite3_malloc(), sqlite3_realloc(), and sqlite3_free(), two other functions are also unresolved: sqlite3_memory_alarm() and sqlite3_memory_used().

  $ make sqlite3 2>&1 | grep 'undefined reference to' | sed 's/^.*://g' | sort | uniq
  undefined reference to `sqlite3_free'
  undefined reference to `sqlite3_malloc'
  undefined reference to `sqlite3_memory_alarm'
  undefined reference to `sqlite3_memory_used'
  undefined reference to `sqlite3_realloc'

They are used in sqlite3_soft_heap_limit() in malloc.c.

So this issue could either be resolved with a doc fix to the statement "If SQLite is compiled with SQLITE_OMIT_MEMORY_ALLOCATION then no implementation for the sqlite3_malloc(), sqlite3_realloc(), and sqlite3_free() functions is provided." or by using an appropriate #ifdef around sqlite3_memory_used and sqlite3_memory_alarm in sqlite3_soft_heap_limit(). i.e.:

Index: src/malloc.c
===================================================================
RCS file: /sqlite/sqlite/src/malloc.c,v
retrieving revision 1.13
diff -u -3 -p -r1.13 malloc.c
--- src/malloc.c        29 Aug 2007 14:06:23 -0000      1.13
+++ src/malloc.c        4 Sep 2007 04:40:03 -0000
@@ -36,6 +36,7 @@ static void softHeapLimitEnforcer(
 ** zero or negative value indicates no limit.
 */
 void sqlite3_soft_heap_limit(int n){
+#ifndef SQLITE_OMIT_MEMORY_ALLOCATION
   sqlite3_uint64 iLimit;
   int overage;
   if( n<0 ){
@@ -52,6 +53,7 @@ void sqlite3_soft_heap_limit(int n){
   if( overage>0 ){
     sqlite3_release_memory(overage);
   }
+#endif
 }

 /*
I prefer the #ifdef over the doc change, but either is fine.
2007-Sep-04 04:55:41 by anonymous:
It appears that the next paragraph mentions that sqlite3_memory_alarm must also be provided by the user if SQLITE_OMIT_MEMORY_ALLOCATION is defined. So sqlite3_memory_used is the only one not documented as being required to be supplied by the user.


2007-Sep-04 05:35:13 by anonymous:
loadext.c also has references to sqlite3_memory_highwater and sqlite3_memory_used when SQLITE_OMIT_MEMORY_ALLOCATION is defined.
 
2621 code fixed 2007 Sep anonymous   2007 Sep   1 5 another typo edit
  -Application that implement their own OS interface will require modification.
  +Any application implementing their own OS interface will require modification.
 
2620 doc closed 2007 Sep anonymous   2007 Sep   5 5 doc note needed for sqlite3_memory_alarm() edit
In the docs for sqlite3_memory_alarm() it is probably worth noting that the registered callback xCallback may be invoked from any thread where sqlite3_malloc may be invoked and the user must take appropriate measures in writing their callback.
2007-Sep-04 03:32:19 by drh:
The docs already say:

"Application programs should not attempt to use sqlite3_memory_alarm().... This interface is exposed only so that applications can provide their own alternative implementation...."
Isn't that enough?


2007-Sep-04 04:03:54 by anonymous:
Connection locks may be held. Things could potentially deadlock if you don't know what you're doing. I thought an explicit mention would make this clear, but perhaps you're right. Feel free to close this ticket.
 
2619 doc fixed 2007 Sep anonymous   2007 Sep   5 5 typo edit
  -This is a large change approximately 1 line of count of out 10 was modified.
  +This is a large change. Approximately 10 percent of the lines of code was modified.
 
2618 code closed 2007 Sep anonymous   2007 Sep linus 4 1 Dabase is Locked edit
I have copied my executables and binaries to my target with Sqlite Database. But When i try to access the Database I get, Error:Database is Locked and memory fault occurs.
2007-Sep-04 03:13:48 by drh:
SQLite version 2.8.0 dates back to February of 2003. We might consider doing a patch on a branch of version 2.8.17 if someone happens to find a very serious bug and explains the bug clearly. This ticket does neither. Please take further discussion to the mailing list.

Tnx.

 
2617 code closed 2007 Sep anonymous   2007 Sep drh 2 3 Check in #4357 may lead to bugs in SMP aware systems in windows vfs edit
Hi

The method for generating temporary file names in windows vfs driver may cause problems when running on SMP systems, because they could 'collide' on sqliteRandomness() call.

A safer and proper fix for this is to use GetTempFileName () windows API (http://msdn2.microsoft.com/en-us/library/aa364991.aspx) and set flags to generate a unique file name. It ensures no-other call could get the same file name.

2007-Sep-03 21:28:15 by drh:
The chances of a name collision are vanishingly small, so small that the problem can be ignored.


2007-Sep-05 14:49:46 by anonymous:
Yes. You are correct on this, could never happen, but still exists. Also, after examined the Randomness code for Win32 VFS driver, I found this:

static int winRandomness(sqlite3_vfs *pVfs, int nBuf, char *zBuf){
  int n = 0;
  if( sizeof(LPSYSTEMTIME)<=nBuf-n ){
    SYSTEMTIME x;
    GetSystemTime(&x);
    memcpy(&zBuf[n], &x, sizeof(x));
    n += sizeof(x);
  }
  if( sizeof(DWORD)<=nBuf-n ){
    DWORD pid = GetCurrentProcessId();
    memcpy(&zBuf[n], &pid, sizeof(pid));
    n += sizeof(pid);
  }
  if( sizeof(DWORD)<=nBuf-n ){
    DWORD cnt = GetTickCount();
    memcpy(&zBuf[n], &cnt, sizeof(cnt));
    n += sizeof(cnt);
  }
  if( sizeof(LARGE_INTEGER)<=nBuf-n ){
    LARGE_INTEGER i;
    QueryPerformanceCounter(&i);
    memcpy(&zBuf[n], &i, sizeof(i));
    n += sizeof(i);
  }
  return n;
}
The code could run into a buffer underrun because you check for sizeof(LPSYSTEMTIME) (which is equal to sizeof(void *), because LPSYSTEMTIME is typedef'ed as SYSTEMTIME *, so a underrun could happen here. Also, LPSYSTEMTIME suffer from timer granularity of the kernel, which could be, in some systems, up to 130ms. A better implementation for this routine could be:
static int winRandomness(sqlite3_vfs *pVfs, int nBuf, char *zBuf){
  int n = 0;
  if( sizeof(DWORD)<=nBuf-n ){
    DWORD cnt = GetTickCount(); // Ticks from system startup
    memcpy(&zBuf[n], &cnt, sizeof(cnt));
    n += sizeof(cnt);
  }
  if( sizeof(LARGE_INTEGER)<=nBuf-n ){
    LARGE_INTEGER i;
    QueryPerformanceCounter(&i); // High-resolution TSC counter
    memcpy(&zBuf[n], &i, sizeof(i));
    n += sizeof(i);
  }
  if( sizeof(DWORD)<=nBuf-n ){
    DWORD pid = GetCurrentThreadId(); // Current thread id
    memcpy(&zBuf[n], &pid, sizeof(pid));
    n += sizeof(pid);
  }
  if( sizeof(DWORD)<=nBuf-n ){
    DWORD pid = GetCurrentProcessId(); // Current process id
    memcpy(&zBuf[n], &pid, sizeof(pid));
    n += sizeof(pid);
  }
  if( sizeof(SYSTEMTIME)<=nBuf-n ){
    SYSTEMTIME x;
    GetSystemTime(&x); // System date/time
    memcpy(&zBuf[n], &x, sizeof(x));
    n += sizeof(x);
  }
  return n;
}

This code will fill, with less chance of collision, the 20 byte randomness key used to generate the temporary file name, and also will generate more efficiently random numbers, because it includes the thread id on it, which will help in SMP systems, because they cannot get to be the same at same time.

There is a chance this could be fixed (SYSTEMTIME issue) and include in pre-3.5.0 code ?

 
2616 code fixed 2007 Sep anonymous   2007 Sep   1 1 Flags parameter not honoured in openDatabase() edit
The flags parameter of openDatabase() is not honored. At least I did not find any further reference to it in the function body. As a result, calling sqlite3_open_v2 with the SQLITE_OPEN_READWRITE flag creates a new database file even though, as advertised in the documentation, it should not.
 
2615 code fixed 2007 Sep anonymous   2007 Sep   1 1 winRandomness without effect due to < vs. > mixup edit
os_win.c line 1424 checks that if( sizeof(LPSYSTEMTIME)>=nBuf ), supposedly to make sure that the supplied buffer is never overwritten. However, the comparison should be <= instead. The current implementation prevents that the buffer is filled at all because the check always fails.
 
2614 code closed 2007 Sep anonymous   2007 Sep   1 1 Custom VFS still calls default xRandomness edit
If a default VFS is registered, any custom VFS's xRandomness callback is never invoked. The only reference to the xRandomness call is hard coded in random.c, line 65: sqlite3OsRandomness(sqlite3_vfs_find(0), 256, k); This makes it impossible to implement custom random seeds for any VFS other than the default.
Correct. This fact has now been made explicit in the documentation.
 
2613 code active 2007 Sep anonymous   2008 Jan drh 3 3 replace doesn't work with blobs containing \x0, otherwise it does edit
The replace expression function does not work with blobs in case of contained zero terminator character; but it does if there is not this special character included. I expected the function to work similar like substr with blob-safety in case of type is blob only. X'nnnn' is of type blob, so following example should have returned a blob type result X'0102FF0405' in the 2nd and 3rd line. How to get to this result?
SQLite version 3.4.2
Enter ".help" for instructions
sqlite> select hex(replace(X'0102030405',X'03',X'FF'));
0102FF0405
sqlite> select hex(replace(X'0102000405',X'00',X'FF'));

sqlite> select typeof(replace(X'0102000405',X'00',X'FF'));
null
sqlite>
2007-Sep-03 04:21:12 by anonymous:
Replace was designed to work with strings. However, working with blobs would be an interesting extension.


2007-Oct-18 06:13:10 by anonymous:
I've seen a similar situation where I can't reliably store stings with nulls in the middle of them as TEXT.

I can convert them to blobs, in which case length(...) works correctly. I if convert them back to strings, length(...) treats them as C-strings.

Is this the expected behavior? I notice the entire column is preserved even when it's has TEXT affinity, I can append data to it as a string, cast back to a blob and see everything (am I explaining this poorly?)

This all seems a bit counter intuitive in some ways. Perhaps strings shouldn't treat NULL characters as special?


2007-Oct-27 16:45:41 by anonymous:
Treatment of length operator is - as fas as I know - dependent on type:
As text it is the length number of UTF-8 characters and as blob it is the number of bytes. As long as all the UTF-8 characters out of the lower half ASCII char-set (127 of them), this is identical beside the fact of different 0-terminator interpretation.
To append is something different than using the replace operator. My suggestion would be to make the replace operator work with bytes (not UTF-8) in case of all 3 parameters are of type blob.
Another suggestion: the UTF aware functions are Private declared and not usable from within a loadable extension dll/so. This should be changed.


2008-Jan-28 19:36:39 by anonymous:
Will there come a solution for this with the next release? It is really not fair to handle a blob only like text which cannot contain a zero terminator. With this unique useful function a zero-containing blob could be formed into a normal text string without loosing the part behind the zero terminator. It would be really a step forward without too much effort.
 
2612 code fixed 2007 Sep anonymous   2007 Sep   1 1 enterMutex() misnamer edit
enterMutex() in mem1.c is wrongly named as enterMem(). This results in linker errors due to missing enterMutex() function.
 
2611 code fixed 2007 Sep anonymous   2007 Sep   2 1 AV calling sqlite3_vfs_register() edit
Calling sqlite3_vfs_register() as the first thing in an application raises an AV.

Rationale: vfsList in os.c is not initialized until the first call of sqlite3_vfs_find(). Thus calling sqlite3_vfs_find() before sqlite3_vfs_register() functions as a workaround, but can be easily forgotten.

 
2610 code fixed 2007 Sep anonymous   2007 Sep drh 4 3 SQLITE_API sqlite3_sleep function header missing in sqlite3ext.h edit
struct sqlite3_api_routines { does not contain sqlite3_sleep }
I am using the amalgamation for Windows to write loadable extension functions and was going to use sqlite3_sleep in cases where trials to access with sqlite3_step do not succeed.
There may be many more SQLITE_API declared functions, which are not yet part of the interface. How can I use SQLITE_API declared functions also in my extension dll's ? Why aren't they in the struct?
Is there another and better way to retry sqlite3_step on SQLITE_BUSY error for a maximum of time (like .timeout in sqlite3.exe)? My example:
#include <sqlite3ext.h>
SQLITE_EXTENSION_INIT1

static void execSQL(sqlite3_context *context, int argc, sqlite3_value **argv)
{
...
  len = strlen(sql);
  errno = sqlite3_prepare(db, sql, len, &stmt, &t);
  while( stmt )
  {
    while( (s=sqlite3_step(stmt))==SQLITE_BUSY )
      sqlite3_sleep( 100 );
    if( s==SQLITE_ROW )
    {
      switch( sqlite3_column_type(stmt, 0 ) )
...
    }
    else
    {
      sqlite3_finalize(stmt);
      stmt=NULL;
    }
  }
...
int sqlite3_extension_init( sqlite3 *db, char **pzErrMsg, const sqlite3_api_routines *pApi )
{
  SQLITE_EXTENSION_INIT2(pApi)
  sqlite3_create_function(db, "execSQL", 1, SQLITE_UTF8, 0, execSQL,  0, 0);
For those who want to know: The execSQL function is foreseen to process a list of SQL statements out of a table which keeps the stored SQL statements; Table and Trigger creations and deletions, loading of further extensions, etc.
This issue has already been addressed by check-in [4343] .
 
2609 code closed 2007 Sep anonymous   2007 Sep   1 1 Is non-thread safe binaries dangerous for multi-threaded app design ? edit
Can accessing two threads ( one is the main app one ) to one connection silently corrupt in-memory data so that next commit to write the wrong ones to the disk, for the non-threadsafe binaries ?
2007-Sep-01 20:45:04 by anonymous:
Forgot to mention:

only the main app thread writes ( calls COMMIT), the other thread only scrolls & reads the data.


2007-Sep-01 20:57:07 by anonymous:
You have to compile sqlite with -DTHREADSAFE=1 if you are to use it in multiple threads in the same program.

This question really belongs on the mailing list, not the bug ticket system.


2007-Sep-01 22:13:40 by anonymous:
Then this issue should be noted on the top of official site, and there should also be thread-safe binaries in the download area.
 
2608 code fixed 2007 Aug anonymous   2007 Sep   3 3 failed allocation leads to unexpected infinite loop edit
I tried to build 3.4.2 on Windows with something else than mingw and I stumbled upon this. The malloc.test hangs in malloc-1.733 so I attached to testfixture.exe with the VC8 debugger and found the following:

The function sqlite3WinTempFileName contains an infinite loop where random file names are generated and the loop breaks when one is found which does not exist. That is where it hangs.

sqlite3WinTempFileName calls in the infinite loop ( sqlite3WinFileExists, which calls ( convertUtf8Filename, which calls ( utf8ToUnicode, which calls ( sqlite3MallocRaw, which fails, and returns 0 ) and utf8ToUnicode returns 0 ) and convertUtf8Filename returns 0 ) and sqlite3WinFileExists returns SQLITE_NOMEM=7 ) and the test in the loop thinks it has received TRUE and goes on.

Essentially this means an existence test seems to return a TRUE answer when the test actually has not happened at all for an obscure technical reason. This can stand improvement.

This was probably missed because that allocation should never fail. It is probably failing for reasons to do with my build, which is another matter.

2007-Sep-01 02:16:48 by drh:
This fix is simply to not check to see if the randomly generated filename already exists. I extended the random section of the filename to be 20 characters long. with 62 possible values at each position, this gives 119 bits of randomness. The chance of such an name colliding is many orders of magnitude smaller than the chance of a random malfunction causing the win32 APIs to return the wrong answer when we ask whether or not the file exists.

The flaw in the above argument is that the PRNG in SQLite is poorly seeded on windows. We need to do a better job seeding the PRNG on that platform. It's the winRandomness() function in the os_win.c source file that seeds the PRNG, if any windows hackers would like to suggest a way to improve the seeding.

 
2607 event active 2007 Aug anonymous   2007 Sep   2 1 Data loss, continuation to Re: [sqlite] how to flush database to disk? edit
See that mailing list.

The originator message :


I've just lost a couple of days' worth of data when my app crashed. (Well, the data wasn't a total loss thanks to backup plans, but the database itself essentially reverted to its state of 2 days ago.) This is despite my app doing a COMMIT after every modification of the DB.

It's acting as though all the changes were held in memory, or somehow journaled, and when the crash happened, the changes were all lost or rolled back. What I need is a way to force the database to save its data to disk while my app is running, so that in the event of a crash, I lose little or no data. How can I do this? I presume that closing the database would do the trick, but is there a less heavy-handed way?


The exact like that data loss occured at me too. Three times, non-repropucible regualrilly.

What common in these losses ?

{Editing+committing} in the main thread then {navigating, bof/eof checking, reading data} from within different threads then return to the main thread for {editing+committing}.

2007-Aug-31 19:38:26 by drh:
The COMMIT does not actually occur until you call sqlite3_reset() and/or sqlite3_finalize() on all your prepared statements. Any prepared statement that has not been reset or finalized is still running, is incomplete, and is thus still holding the transaction open. I'm guessing that you have an unreset and unfinialized statement in your application.

I wonder what would happen if we changed the definition of COMMIT so that it returned an error if there were active prepared statements. This is, technically, an incompatibility. But we are coming up on a release with several other minor incompatibilities, so now might be a good time to insert such a change.


2007-Aug-31 19:45:36 by drh:
I looked in the code, and it turns out we already do this. Perhaps the application is not checking the return code from the COMMIT to see that it is failing?


2007-Aug-31 20:40:47 by anonymous:
No error reports came from COMMIT. The data loss were noticed after UPDATE & COMMIT after massive reading of results of SQL addressing the same virtual (ATTACHed) tables, from within another thread.


2007-Aug-31 20:53:48 by anonymous:
Is it possible that a pending transaction survive app shutdown & then OS restart ? Is yes, then any DB error would cause rollback to the data on last BEGIN, isn't ?


2007-Aug-31 21:00:18 by anonymous:
But me committed each smallest change to dat,a then saw these refreshed data in the tables. And on the next day these data were present. Only reading (with full scrolling ) the affected virtual tables from within another thread then new editing then committing caused the loss.


2007-Sep-01 23:12:31 by anonymous:
You mention virtual tables. Their data is not maintained by the SQLite engine, but by your own module. If that doesn't implement ACID, you're out of luck.

By definition:
A virtual table is an interface to an external storage or computation engine that appears to be a table but does not actually store information in the database file.

references:
CreateVirtualTable
VirtualTables


2007-Sep-04 04:51:15 by anonymous:
Me was wrong. These were't true virtual tables. Me used a LEFT OUTER query to several tables residing in different ATTACHed databases.


2007-Sep-04 04:52:01 by anonymous:
were't => were not, above.
 
2606 code fixed 2007 Aug anonymous   2007 Sep   3 3 PATCH - default configure option enable_threadsafe=yes edit
Obvious patch below to enable thread-safe sqlite3 builds by default to match the default thread-safe build of the sqlite3.c amalgamation.

Patch included for both configure.ac and configure so you don't have to regenerate the script. I tested it with no options, enable and disable and it works.

Index: configure
===================================================================
RCS file: /sqlite/sqlite/configure,v
retrieving revision 1.42
diff -u -3 -p -r1.42 configure
--- configure   17 Feb 2007 14:59:18 -0000      1.42
+++ configure   31 Aug 2007 15:43:23 -0000
@@ -18883,7 +18883,7 @@ fi
 if test "${enable_threadsafe+set}" = set; then
   enableval=$enable_threadsafe;
 else
-  enable_threadsafe=no
+  enable_threadsafe=yes
 fi

 { echo "$as_me:$LINENO: checking whether to support threadsafe operation" >&5
Index: configure.ac
===================================================================
RCS file: /sqlite/sqlite/configure.ac,v
retrieving revision 1.29
diff -u -3 -p -r1.29 configure.ac
--- configure.ac        17 Feb 2007 14:59:18 -0000      1.29
+++ configure.ac        31 Aug 2007 15:43:23 -0000
@@ -178,7 +178,7 @@ AC_SUBST(BUILD_CFLAGS)
 # Do we want to support multithreaded use of sqlite
 #
 AC_ARG_ENABLE(threadsafe,
-AC_HELP_STRING([--enable-threadsafe],[Support threadsafe operation]),,enable_threadsafe=no)
+AC_HELP_STRING([--enable-threadsafe],[Support threadsafe operation]),,enable_threadsafe=yes)
 AC_MSG_CHECKING([whether to support threadsafe operation])
 if test "$enable_threadsafe" = "no"; then
   THREADSAFE=0
2007-Aug-31 16:23:18 by drh:
Thanks for the patch.

Here's the thing: Threadsafe builds are currently running about 8% slower than single-threaded builds. (All those calls to pthread_mutex_lock() do not come for free.) So we are going to have to make a decision about whether or not we really want the default build to be threadsafe.


2007-Aug-31 16:29:45 by anonymous:
The trend these days is multi-core chips and multi-threaded software. Embedded programmers know their way around a compiler and can easily disable thread-safety if they should so choose to get extra speed. Beginners (and apparently one old timer) may forget to configure a threadsafe build which may produce programs that seem to work initially, but fail unexpectedly in the field. Better safe than sorry, I figure.
 
2605 build closed 2007 Aug anonymous   2007 Aug   1 3 Amalgamated version 3.4.2 doesn't compile (VC++6) edit
In a project that already used versions 3.3.x through 3.3.17 (successfully) in "amalgamated" source, I substituted sqlite3.c and sqlite3.h from version 3.4.2, but I yield three compilation errors:


Compiling...
sqlite3.c
c:\lavoro\nonclassificato\resanddev\hcsprog\sqlite\sqlite3.c(6187) : error C2133: 'sqlite3UpperToLower' : unknown size
c:\lavoro\nonclassificato\resanddev\hcsprog\sqlite\sqlite3.c(9310) : error C2133: 'sqlite3OpcodeNames' : unknown size
c:\lavoro\nonclassificato\resanddev\hcsprog\sqlite\sqlite3.c(47159) : error C2133: 'sqlite3IsIdChar' : unknown size
Error executing cl.exe.

They all are related to array declaration missing size in brackets.
Duplicate of ticket #2574. Already fixed.
 
2604 new active 2007 Aug anonymous   2007 Aug   4 4 CREATE VIRTUAL TABLE does not allow IF NOT EXISTS edit
CREATE VIRTUAL TABLE vt IF NOT EXISTS; would help with development since creating a virtual table that exists returns error 1 - as do several "Real" errors.
 
2603 new closed 2007 Aug anonymous   2007 Oct   1 3 B-tree and B+tree allow bidirectional cursor but VDBE does not. edit
B-tree and B+tree allow bidirectional cursor but VDBE does not. We would very much like to have this capability and our fallback is to work with the btree api directly.

the closest opcodes are: rewind p1 p2 p3 p1 is p2 is line number if rowcount=p1

and step

What we would like to see is something like

rstep p1 p1= signed integer for direction of relative motion

repos p1 p2 p1=unsigned integer representing low-order 32 bits of 64 bit integer p2=unsigned integer representing hi-order 32 bits of 64 bit integer

If this is doable in your opinion then we would be willing to look at implementing it and contributing it to the project.

2007-Oct-12 18:58:46 by drh:
I'm not really sure what the description above is saying, but my best guess is that it is a request for logic to do a ScrollingCursor. Please see the wiki page for how to accomplish ScrollingCursor in SQLite.
 
2602 code fixed 2007 Aug anonymous   2007 Aug   1 1 ATTACH leaks memory edit
Attaching a non-existing file to an empty database leaks memory.

The scenario (Win32):

  • Create a new empty database.
  • sqlite3_prepare_v2 ( "ATTACH DATABASE 'C:\NonExistantFolder\NonExistantFile.db3' AS a;'); returns SQLITE_OK
  • sqlite3_step(); returns SQLITE_ERROR
  • sqlite3_finalize(); returns SQLITE_ERROR
  • Close database and application.

This returns a report about a single leak between 77 and 92 bytes in size.

FYI: I also receive a 2nd, but smaller (21-28 bytes), leak when executing an FTS1 SELECT on an attached database. If you can confirm the 1st leak I will provide more information on the 2nd, if it still persists.

2007-Aug-30 14:21:25 by drh:
I am unable to replicate the problem on Linux. My tests show all allocated memory is freed. Both SQLite's own internal accounting reports this, and valgrind too.

We observe that the windows OS driver calls malloc() directly, bypassing SQLites internal memory allocator and its accounting procedures. And we do not know of anything like valgrind for windows. So perhaps there is a leak in the windows OS driver someplace. How do you see this leak, Ralf? What tools are you using? Do you get any indication where the leak is occurring? Can you see anything in the windows OS driver that might be causing the problem?


2007-Aug-30 15:37:03 by anonymous:
Found it: The if block in os_win.c, starting at line 1136, does not free zConverted after the else. The fix is to move the free(zConverted); line up into the 1st if block:

  if( h==INVALID_HANDLE_VALUE ){
    free(zConverted);
    if( flags & SQLITE_OPEN_READWRITE ){
      return winOpen(0, zName, id,
             ((flags|SQLITE_OPEN_READONLY)&~SQLITE_OPEN_READWRITE), pOutFlags);
    }else{
      return SQLITE_CANTOPEN;
    }
  }
 
2601 code closed 2007 Aug anonymous   2007 Oct   1 1 Problem with opening database with Unicode filename edit
Problem with opening database with Unicode filename - it doesn't open the database with Unicode filename or path.
The description contains insufficient information for us to act on the problem.
 
2600 code fixed 2007 Aug anonymous   2007 Aug   1 1 #ifdef SQLITE_MUTEX_WIN32 naming error in mutex_w32.c? edit
Line 23 in mutex_w32.c checks #ifdef SQLITE_MUTEX_WIN32. However, SQLITE_MUTEX_WIN32 is never defined in the SQLite core. Instead, SQLITE_MUTEX_WIN is (notice the missing 32 postfix). mutex_w32.c line 182 #endif /* SQLITE_MUTEX_WIN */ makes me believe that this is just a spelling error?

If there are plans for extra support of non-Win32 platforms (Win64, Windows 3.1), it would be reasonable to choose _WIN32 over just _WIN. But then I suppose that this would also be true for the OS_WIN #define as well?

 
2599 code fixed 2007 Aug anonymous   2007 Aug   1 1 OS check in mutex.h has no effect edit
OS check in mutex.h (starting from line 34) dependes on OS autodetection in os.h. However, in sqliteInt.h the statement #include "mutex.h" is in line 76, way before #include "os.h" is followed in line 297. OS autodetection is therefore not taken into account for mutex.h and the library is compiled without mutexes, at least in Win32. It did work before mutex.c was split up for different OSes.
 
2598 code fixed 2007 Aug anonymous   2007 Oct   3 3 Failure to delete temporary files on WINCE platforms edit
Problem lies in the sqlite3WinOpenExclusive function. Under WINCE platform, winFile.hMutex is set to NULL (line 822 of os_win.c) even though winFile.zDeleteOnClose is set. So when winceDestroyLock (called from winClose) is eventually called, because hMutex is NULL, the file is never deleted.
 
2597 code fixed 2007 Aug anonymous   2007 Oct   1 1 tclsqlite "db eval" with array-name doesn't have "*" entry. (tcl8.5a7) edit
This problem was caused on tcl cvs head(tcl8.5a7) only. Tcl8.4 and 8.5a6 doesn't have the problem.

package require sqlite3 sqlite db :memory: db eval "create table t (c1, c2, c3)" db eval "PRAGMA empty_result_callbacks = on" db eval "select * from t" val "" puts [array get val]

I expected a value "* {c1 c2 c3}", but the val(*) entry didn't exist.

It seems to me that "Tcl_DecrRefCount(pStar);" cause the problem...

2007-Aug-30 03:18:14 by anonymous:
I'm sorry for the bad format.

package require sqlite3
sqlite db :memory:
db eval "create table t (c1, c2, c3)"
db eval "PRAGMA empty_result_callbacks = on"
db eval "select * from t" val ""
puts [array get val]

 
2596 code closed 2007 Aug anonymous   2007 Aug   2 2 wrong specifier for extern char arrays edit
In 3.4.2 (compared to 3.4.1) the forward declarations of char arrays sqlite3UpperToLower, sqlite3OpcodeNames and sqlite3IsIdChar are labeled with SQLITE_PRIVATE instead of previous extern. This causes problem with amalgamated file - the compiler (Visual C++ 9.0 beta 2) fails to compile it.

Putting extern instead of SQLITE_PRIVATE had fixed the problem.

Compile with -DSQLITE_PRIVATE=extern or compile a file like this:

  /* foo.c */
  #define SQLITE_PRIVATE extern
  #include "sqlite3.c"
 
2595 doc active 2007 Aug anonymous   2007 Aug   4 4 sqlite3_commit_hook doc typo edit
src/main.c:

  -** Register a function to be invoked when a transaction comments.
  +** Register a function to be invoked when a transaction commits.
 
2594 code closed 2007 Aug anonymous   2007 Aug   5 4 ability to replace allocator edit
In order to use my own allocator for the SQLite I reimplemented routines sqlite3GenericMalloc, sqlite3GenericRealloc and sqlite3GenericOsFree.

SQLite, however, needs some modification to use these functions: (1) The original functions need to be commented out. (2) Functions sqlite3OsMalloc, sqlite3OsRealloc and sqlite3OsFree (in amalgamated code) use specifier SQLITE_PRIVATE (static) and this needs to be removed too.

Perhaps the SQLite code may employ macro SQLITE_I_HAVE_MY_OWN_ALLOCATOR or so to do this all w/o need to touch the code.

The change of allocator worked well for me (I use Doug Lea's dlmalloc).

2007-Aug-29 22:34:36 by drh:
Please check the mailing list archives. This is a planned feature for version 3.5.0.


2007-Aug-29 22:56:18 by anonymous:
What does OBE stand for?


2007-Aug-29 23:45:10 by drh:
Overcome by events.

Normally that means that a ticket is no longer relevant because things have changed since the ticket was written. That does not apply here, exactly, but it was (in my opinion) the best available option for Resolution.

 
2593 code closed 2007 Aug anonymous   2007 Aug   1 1 btshared unlock bug edit
You have want to avoid a race condition by zeroing p->locked within the mutex. Otherwise another thread may acquire the lock, set locked to 1 (while still holding the mutex) and then this thread outside the mutex will unintentionally clear it.

  @@ -191,6 +193,7 @@
       p->wantToLock--;
       if( p->wantToLock==0 ){
  +      p->locked = 0;
         sqlite3_mutex_leave(p->pBt->mutex);
  -      p->locked = 0;
       }
     }
   }

Haven't looked at the code, there may be other occurances of this.

2007-Aug-28 17:57:26 by drh:
The p->locked variable is protected by a separate mutex, specifically the mutex at p->pSqlite->mutex.

The database handle, the thing that sqlite3_open() returns, is at p->pSqlite. Each such handle has its own mutex. Associated with each database handle is one or more Btree structures, one for each open database. There are typically two of these, one for the main database and another for the TEMP database. New Btree structures are added for each ATTACH. Each Btree is tied to a single database handle so it is protected by the same mutex as the database handle. "p" is a pointer to a Btree in this instance.

The guts of a database are contained in a Btshared structure. Btree points to Btshared. But if shared cache is enabled, two or more Btree structures from different database handles might point to the same Btshared. So Btshared needs its own mutex. This new mutex covers only the Btshared. The Btree structure is covered by the original database handle mutex.

See also #2578

 
2592 warn closed 2007 Aug anonymous   2007 Aug   1 1 The Entry C:\Windows\Temp\sqlite_z6onAYdsVGqLG3Q is Invalid edit
Everytime I reboot and check disk I get " The Entry C:\Windows\Temp\sqlite_z6onAYdsVGqLG3Q is Invalid I believe this has something to do with skype and all messengers rebooting my PC everytime I try to enable voice Thank you
This is a question for a skype group.
 
2591 doc fixed 2007 Aug anonymous   2007 Aug   4 4 sqlite3_open() documentation should mention :memory: edit
I could not find information about in-memory database support in SQLite from API docs. In my opinion, sqlite3_open() function call is the place where it should be mentioned that I can give ":memory:"(?) as filename parameter to open in-memory database.
 
2590 todo closed 2007 Aug anonymous   2007 Aug danielk1977 2 2 return value of strftime('%s', 'now') being determined as 'text' edit
Hello.

SELECT typeof(strftime('%s', 'now'));

results 'text'. Is this correct? As strftime('%s', 'now') returns number of seconds since Unix epoch started.

Greatings.

In my opinion it is correct. strftime returns a formatted string. If for some reason you want the type of the expression to be an integer you could do:

  SELECT CAST(strftime('%s', 'now') AS INTEGER);
 
2589 doc closed 2007 Aug anonymous   2007 Aug danielk1977 4 4 s/effected/affected/ in PRAGMA locking_mode edit
In the last sentence of the documentation for PRAGMA locking_mode, "effected" should be "affected".
thanks.
 
2588 code fixed 2007 Aug anonymous   2007 Aug   1 1 bug in sqlite3_mutex_leave edit
Race scenario on same mutex leading to corruption:

  Thread 1                    Thread 2
  --------------------------  --------------------------
  sqlite3_mutex_enter
    pthread_mutex_lock ok
    p->owner = Thread 1
    p->nRef = 1;
  sqlite3_mutex_leave
    p->nRef--;
    pthread_mutex_unlock ok
  sqlite3_mutex_enter         sqlite3_mutex_enter
    pthread_equal is true       pthread_equal is false
                                pthread_mutex_lock ok
                                p->owner = Thread 2;
                                p->nRef = 1;
    p->nRef>0 is true
    p->nRef++;

p->nRef is now 2 and the mutex is "owned" by Thread 2. Any data structure protected by this mutex can now be changed by both threads at the same time.

Here's a possible pseudo fix:

Index: src/mutex.c
===================================================================
RCS file: /sqlite/sqlite/src/mutex.c,v
retrieving revision 1.9
diff -u -3 -p -r1.9 mutex.c
--- src/mutex.c 24 Aug 2007 20:46:59 -0000      1.9
+++ src/mutex.c 25 Aug 2007 01:39:10 -0000
@@ -358,6 +358,7 @@ void sqlite3_mutex_leave(sqlite3_mutex *
   assert( p->nRef>0 );
   p->nRef--;
   if( p->nRef==0 ){
+    memset(&p->owner, 0, sizeof(p->owner));
     pthread_mutex_unlock(&p->mutex);
   }
 }
But even this fix is not guaranteed by POSIX standards, as an opaque pthread_t type when memset to zero may indeed be a valid pthread_t for the first (zeroeth) thread. This could lead to the same race condition and corruption.

http://mail-archives.apache.org/mod_mbox/apr-dev/200307.mbox/%3c3F285220.7010501@netscape.com%3e

This stuff is difficult to get right. Please consult correct recursive mutex implementations from various multi-threaded open source projects.

2007-Aug-25 02:35:35 by anonymous:
Furthermore, the pthread_equal man page states:

"If either t1 or t2 are not valid thread IDs, the behavior is undefined."

http://www.opengroup.org/onlinepubs/009695399/functions/pthread_equal.html

Only pthread_create and pthread_self can generate a valid pthread_t thread ID.


2007-Aug-25 02:43:32 by anonymous:
On POSIX systems that support it you should use PTHREAD_MUTEX_RECURSIVE instead of rolling your own. It is more efficient and guaranteed to be correct.

http://www.opengroup.org/onlinepubs/000095399/functions/pthread_mutex_lock.html


2007-Aug-25 03:23:56 by anonymous:
Amusing back story of recursive POSIX mutexes from one of its creators:

http://groups.google.com/group/comp.programming.threads/msg/d835f2f6ef8aed99?hl=en


2007-Aug-25 04:13:08 by drh:
I believe the problem is fixed by doing the p->nRef>0 test first and then the ownership test second. Rationale:

  1. If p->nRef>0 it can only mean that some thread is holding the mutex. we don't know which thread, and the mutex might change owners multiple times before the next test. But we do at least know that somebody has held the mutex in the past. And we also know that p->owner has been initialized.

  2. If the second pthread_equal() test is true, that means that the current thread claims to own the mutex. No other thread will ever put the current thread's threadid into p->owner. If p->owner==self, then only the current thread put it there. If another thread had claimed the mutex after the current thread, then the other thread would have changed the value of p->owner before raising p->nRef above zero. We know that p->nRef has been above zero, so the current thread must be the last one to have held the mutex. Hence, the current thread must still own the mutex.

The above assumes that pthread_equal() is an atomic operation. In other words, it will not fail with a TRUE return if a separate thread changes the value of p->owner while the current thread is testing it. If p->owner is a multi-word value, then pthread_equal() might not be atomic. But pthread_t is an integer on every implementation of pthreads that I am aware of, so I think this is probably safe in most instances.

Please double-check my reasoning above and let me know if I've said or done something goofy. Tnx.


2007-Aug-25 04:25:26 by drh:
Response to Butenhof:

Threading in SQLite is not designed to make it more concurrent. It is designed to make the code safer to use by people who insist on writing multithreaded programs when multithreading really isn't necessary. Each database connection has its own mutex. Any thread using the database connection holds that mutex. So only a single thread can use a particular database connection at a time. The mutex is acquired when the API is entered and released when the API exits.

But SQLite is reentrant. Top-level APIs can (and are) invoked as callbacks from within other top-level APIs. The only way to allow reentancy for one thread while excluding other threads (the only way that I know of, anyhow) is a recursive mutex. If David Butenhof thinks that recursive mutexes can be eliminated from SQLite by do "careful design", I'd like to hear him tell me how. One suspects that Mr. Butenhof has never tried to build a reentrant library for use in a multithreaded environment....


2007-Aug-25 04:47:52 by anonymous:
My first impression is that the fix is not correct.

Take a look at this post and the accompanying thread:

http://groups.google.com/group/comp.programming.threads/msg/329ba20fe934b5e1?hl=en&

pthread_equal() is probably not atomic on HPUX where pthread_t is a struct (of 4 int's as I recall). Nothing in the pthread_equal man page suggests that pthread_equal is guaranteed to be atomic, although on platforms where pthread_t is an int it would obviously be fine.

Also, there is no concept of an official invalid pthread_t value - and even if there was, the pthread_equal man page is pretty clear that the result of such a comparison with an "invalid" pthread ID is undefined.

For what it is worth, Apache Portable Runtime seems to have given up trying to roll their own recursive mutex on POSIX platforms that do not support PTHREAD_MUTEX_RECURSIVE.

I appreciate that your use of recursive mutexes is for correctness and ease of API use. An alternate scheme would be to change the SQLite API to simply pass messages via work queues to a background thread that performs the database operations on your behalf. Much like your previous single thread "server" model. But then you have the headache of making every single SQLite API function work asynchronously - and you still would not improve concurrency on the same database.


2007-Aug-25 05:05:17 by anonymous:
This is somewhat related. When you work with SMP hardware and various optimizing compilers some things are not obvious. Take a look at the seemingly logical double-checked lock mis-pattern:

http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html

http://en.wikipedia.org/wiki/Double-checked_locking


2007-Aug-25 14:45:21 by drh:
Well, it turns out that SuSE Linux does support recursive mutexes after all. But in order to get it to work you have to have a line

    #define _XOPEN_SOURCE 500

prior to your #include <pthread.h>. The value must be exactly 500. What this means is unclear. It is certainly undocumented as far as I can tell. A line such as the above was added to sqliteInt.h and the mutex.c sources were converted to use native recursive mutexes.

SQLite now requires support for PTHREAD_RECURSIVE_MUTEX in order to build with SQLITE_THREADSAFE=1 on unix. If you do not have recursive mutexes, then you have to build with SQLITE_THREADSAFE=0.


2007-Aug-25 15:22:39 by anonymous:
The latest check-in uses the double-checked lock mis-pattern mentioned in an earlier comment.

+      if( !isInit ){
+        pthread_mutex_lock(&initMutex);
+        if( !isInit ){
+          pthread_mutexattr_init(&recursiveAttr);
+          pthread_mutexattr_settype(&recursiveAttr, PTHREAD_MUTEX_RECURSIVE);
+        }
+        isInit = 1;
+        pthread_mutex_unlock(&initMutex);
+      }
Although the liklihood of a problem occurring is rare, it is still present. Since SQLite does not create many recursive mutexes I don't think it is worth to try to optimize this case. The calls to pthread_mutexattr_init and pthread_mutexattr_settype are typically very fast. Please consider using this simpler code instead:

    case SQLITE_MUTEX_RECURSIVE: {
      p = sqlite3MallocZero( sizeof(*p) );
      if( p ){
        pthread_mutexattr_t recursiveAttr;
        pthread_mutexattr_init(&recursiveAttr);
        pthread_mutexattr_settype(&recursiveAttr, PTHREAD_MUTEX_RECURSIVE);
        pthread_mutex_init(&p->mutex, &recursiveAttr);
        p->id = iType;
      }
      break;
    }


2007-Aug-25 16:26:01 by anonymous:
Forgot the pthread_mutexattr_destroy...

    case SQLITE_MUTEX_RECURSIVE: {
      p = sqlite3MallocZero( sizeof(*p) );
      if( p ){
        pthread_mutexattr_t recursiveAttr;
        pthread_mutexattr_init(&recursiveAttr);
        pthread_mutexattr_settype(&recursiveAttr, PTHREAD_MUTEX_RECURSIVE);
        pthread_mutex_init(&p->mutex, &recursiveAttr);
        pthread_mutexattr_destroy(&recursiveAttr);
        p->id = iType;
      }
      break;
    }


2007-Aug-25 16:34:59 by drh:
You will notice that, unlike the examples cited previously, my double-lock was in fact correct.

Nevertheless, your observation that recursive mutexes are rarely allocated (only at sqlite3_open()) and that the initialization and destruction of a pthread_mutexattr_t is relatively cheap are valid. And the code is simpler without the double-lock. So the double-lock has now been removed.


2007-Aug-25 16:48:30 by anonymous:
Well, I disagree about your double lock being correct. There is a window of failure on SMP machines on some hardware with certain optimizing compilers.

http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html

Nonetheless, the recursive mutex code as of the latest checkin is fine. The speed of millions of lock/unlock calls of native POSIX recursive mutexes more than makes up for the few cycles of initializing the attribute.

Some older OSes will undoubtedly have trouble with _XOPEN_SOURCE, but this is easy enough for the odd port to fix.

 
2587 build active 2007 Aug anonymous   2007 Aug   3 4 Build problem when using the SQLITE_OMIT_FLOATING_POINT define. edit
I apologize in advance if the values I chose above are not appropriate.

If I define SQLITE_OMIT_FLOATING_POINT=1 and try to build a Windows DLL, I get two errors in loadext.c, line 116 and 192. "error C4028: formal para meter 3 different from declaration"

I believe you want to change the include order at the top of loadext.c from:

#include "sqlite3ext.h"

#include "sqliteInt.h"

to:

#include "sqliteInt.h"

#include "sqlite3ext.h"

Reversing the order of include fixes my build.

Yes, I know there is no real reason to disable floating point for the Windows DLL. I'm actually porting SqLite for use in an NT kernel mode driver and avoiding floating point operations will save a lot of time if I don't really need them and I don't. So I made sure this was a problem with a supported platform like the Windows DLL and griped about that instead of my insanity. ;-)

You can email questons to mspiegel@vipmail.com. If you want to discuss this over the phone, shoot me an email and I'll send you phone numbers.

 
2586 code closed 2007 Aug anonymous   2007 Aug   3 3 INSERT of large number fails on Windows edit
When using the windows version of the sqlite3.dll, trying to insert a number larger then what fits in a 64 bits int (so larger then: 9223372036854775807) will give a "Invalid floating point operation" exception. The following code will reproduce this:

create table t(a float);
insert into t values(2.49190E+26);

The command-line tool does not have this problem however.

The exception seems to be thrown at this line:
pMem->u.i = pMem->r;
in the "sqlite3VdbeIntegerAffinity" function in vdbemem.c
At least, that is what Visual Studio 8 says.

2007-Aug-24 13:19:25 by anonymous:
You've changed the floating point accuracy to single precision, probably because you're writing 3D software. Google for "controlfp".

http://msdn2.microsoft.com/en-us/library/e9b52ceh(VS.80).aspx

 
2585 code closed 2007 Aug anonymous   2007 Oct anonymous 1 1 Error Function GetAtomNameA edit
The Procedure entry point GetAtomNameA could not be located in the dynamic link library sqlite3.dll

Using CodeBlock

2007-Oct-05 15:11:29 by drh:
Nothing in SQLite invokes GetAtomNameA(). This must be a problem with some other library.
 
2584 new closed 2007 Aug anonymous   2007 Aug   1 2 On Windows function sqlite3WinSleep needs to dispatch messages edit
When sqlite is busy, it spins in the loop, re-trying call every 100 ms. On Windows if the caller thread runs a message loop (for example a GUI application of COM call from an appartment), the message loop stops and the application hangs. As a workaround, sqlite function sqlite3WinSleep needs to call PeekMessage with subsequent calls TranslateMessage/DispatchMessage to continue message processing as following:
/*
** Sleep for a little while. Return the amount of time slept.
*/
int sqlite3WinSleep(int ms)
{
//----------------------------------------------------------------{linebreak} //The fix starts
MSG msg;
while (PeekMessage(&msg, NULL, 0, 0, PM_REMOVE))
{
TranslateMessage(&msg);
DispatchMessage (&msg);
if (msg.message == WM_QUIT) //Note - need to handle WM_QUIT,posted by PostQuitMessage
break;
}
//The fix ends
//-----------------------------------------------------------------{linebreak} Sleep(ms);
return ms;
}
2007-Aug-23 16:08:43 by drh:
Processing GUI events is not the business of a database engine. If you need to process GUI events while waiting on a lock, then use the sqlite3_busy_handler() API to register your own busy handler (perhaps using the code shown in the patch) instead of using the built-in sqlite3_busy_timeout().


2007-Aug-23 17:54:06 by anonymous:
You should wrap in a thread your db calls if you wan't to process using GetMessage() calls
 
2583 build fixed 2007 Aug anonymous   2007 Aug   2 2 3.4.2 fails to build on Solaris due to conflict for B_FALSE and B_TRUE edit
While attempting to build SQLite 3.4.2 on Solaris 10/x86 with gcc 3.4.3, I get this error:

   gcc -D_REENTRANT -g -O2 -o lemon ./tool/lemon.c
   ./tool/lemon.c:111: error: conflicting types for 'B_FALSE'
   /usr/local/lib/gcc/i386-pc-solaris2.10/3.4.3/include/sys/types.h:193: error: previous definition of 'B_FALSE' was here
   ./tool/lemon.c:111: error: conflicting types for 'B_TRUE'
   /usr/local/lib/gcc/i386-pc-solaris2.10/3.4.3/include/sys/types.h:193: error: previous definition of 'B_TRUE' was here

Here is the conflicting line in /sqlite/tool/lemon.c

    111 typedef enum {B_FALSE=0, B_TRUE} Boolean;

And the conflict in the gcc version of sys/types.h:

    190 #if defined(__XOPEN_OR_POSIX)
    191 typedef enum { _B_FALSE, _B_TRUE } boolean_t;
    192 #else
    193 typedef enum { B_FALSE, B_TRUE } boolean_t;
    194 #endif /* defined(__XOPEN_OR_POSIX) */

Oddly enough, 3.4.1 built successfully despite having the same code in lemon.c. Possibly some header files have changed and are causing __XOPEN_OR_POSIX no longer to be defined.

Workaround: rename B_FALSE to BOOL_FALSE and B_TRUE to BOOL_TRUE throughout lemon.c.

 
2582 doc active 2007 Aug anonymous   2007 Aug anonymous 5 5 documenation clarification edit
docs for topic `Set A Busy Timeout` int sqlite3_busy_timeout(sqlite3*, int ms); http://sqlite.org/capi3ref.html#sqlite3_busy_timeout

the wording "The handler will sleep multiple times until at least "ms" milliseconds of sleeping have been done" implies it will wait the total amount regardless of the lock status, it should perhaps indicate in the same sentence that it will exit early if the lock becomes available.

 
2581 code closed 2007 Aug anonymous   2007 Aug   1 1 ./configure && make broken: new source files: mem[12].c, mutex.c, etc edit
Please apply the diff given by this command to Makefile.in:

  cvs diff -D'2007-08-14' main.mk
2007-Aug-21 20:31:44 by drh:
We are in the middle of a major reimplementation of large sections of SQLite. We could fix the makefile, but the build still wouldn't work. So what's the point, really. I promise to fix the makefile before the next release....
 
2580 code active 2007 Aug anonymous   2007 Aug anonymous 1 2 Can't open a query if text to search is Greek edit
for example:

SELECT * FROM mytable WHERE mycolumn LIKE '%some greek text%'

I get wrong results, using the 3.4.2 version. No problem instead using other earlier version. I tested only in Windows.

 
2579 code fixed 2007 Aug anonymous   2007 Oct   4 3 the third parameter of substr should be optional edit
Please make the 3rd Parameter of SUBSTR optional as in most other programming languages. It is better and simpler for migrations to SQLite
 
2578 code closed 2007 Aug anonymous   2007 Aug   1 1 sqlite3BtreeEnter, sqlite3BtreeLeave threading issues edit
p->wantToLock++ is not an atomic operation and must be done within a mutex lock, otherwise its result is undefined. Same goes for "variable--". You need an architecture-specific operations to do that safely between threads. See http://www.mozilla.org/projects/nspr/reference/html/pratom.html and http://lxr.mozilla.org/nspr/source/nsprpub/pr/include/pratom.h#60 for an example.

void sqlite3BtreeEnter(Btree *p){
  Btree *pLater;

  /* Some basic sanity checking on the Btree.  The list of Btrees
  ** connected by pNext and pPrev should be in sorted order by
  ** Btree.pBt value. All elements of the list should belong to
  ** the same connection. Only shared Btrees are on the list. */
  assert( p->pNext==0 || p->pNext->pBt>p->pBt );
  assert( p->pPrev==0 || p->pPrev->pBtpBt );
  assert( p->pNext==0 || p->pNext->pSqlite==p->pSqlite );
  assert( p->pPrev==0 || p->pPrev->pSqlite==p->pSqlite );
  assert( p->sharable || (p->pNext==0 && p->pPrev==0) );

  /* Check for locking consistency */
  assert( !p->locked || p->wantToLock>0 );
  assert( p->sharable || p->wantToLock==0 );

  if( !p->sharable ) return;
  p->wantToLock++;
  if( p->locked ) return;

  /* In most cases, we should be able to acquire the lock we
  ** want without having to go throught the ascending lock
  ** procedure that follows.  Just be sure not to block.
  */
  if( sqlite3_mutex_try(p->pBt->mutex)==SQLITE_OK ){
    p->locked = 1;
    return;
  }
This needs fixing:

Index: src/btree.c
===================================================================
RCS file: /sqlite/sqlite/src/btree.c,v
retrieving revision 1.401
diff -u -3 -p -r1.401 btree.c
--- src/btree.c 17 Aug 2007 16:50:38 -0000      1.401
+++ src/btree.c 17 Aug 2007 20:42:49 -0000
@@ -1403,8 +1403,8 @@ void sqlite3BtreeEnter(Btree *p){
     assert( pLater->pNext==0 || pLater->pNext->pBt>pLater->pBt );
     assert( !pLater->locked || pLater->wantToLock>0 );
     if( pLater->locked ){
-      sqlite3_mutex_leave(pLater->pBt->mutex);
       pLater->locked = 0;
+      sqlite3_mutex_leave(pLater->pBt->mutex);
     }
   }
   sqlite3_mutex_enter(p->pBt->mutex);
@@ -1427,8 +1427,8 @@ void sqlite3BtreeLeave(Btree *p){
     p->wantToLock--;
     if( p->wantToLock==0 ){
       assert( p->locked );
-      sqlite3_mutex_leave(p->pBt->mutex);
       p->locked = 0;
+      sqlite3_mutex_leave(p->pBt->mutex);
     }
   }
 }
Otherwise there would be corruption if there was a context switch just after sqlite3_mutex_leave to "if( p->locked ) return;" in sqlite3BtreeEnter() in another thread.

You might want to replace the u8's below with int's if you intend to read or write to them outside of a mutex:

struct Btree {
  sqlite3 *pSqlite;  /* The database connection holding this btree */
  BtShared *pBt;     /* Sharable content of this btree */
  u8 inTrans;        /* TRANS_NONE, TRANS_READ or TRANS_WRITE */
  u8 sharable;       /* True if we can share pBt with other pSqlite */
  u8 locked;         /* True if pSqlite currently has pBt locked */
  int wantToLock;    /* Number of nested calls to sqlite3BtreeEnter() */
  Btree *pNext;      /* List of Btrees with the same pSqlite value */
  Btree *pPrev;      /* Back pointer of the same list */
};
Reading from and writing to a non-machine word (i.e., not an int) variable outside of a mutex is not atomic or threadsafe. MIPS, for example, can only manipulate 32 bit values at a time and must use bit operations and shifting to get at individual characters. If you read a char value at the wrong time it may contain 0, 255 or some other value.
2007-Aug-17 21:32:11 by anonymous:
These 3 values will likely be stored in the same 32 bit integer:

  u8 inTrans;        /* TRANS_NONE, TRANS_READ or TRANS_WRITE */
  u8 sharable;       /* True if we can share pBt with other pSqlite */
  u8 locked;         /* True if pSqlite currently has pBt locked */

Writing to any one of them outside of a mutex in a multithreaded situation can scramble the others' values.


2007-Aug-20 00:49:28 by drh:
Not a bug.

The Btree.wantToLock values are accessed under the sqlite3.pMutex lock - or at least they will be once we get the new threading implementation half way completed.


2007-Aug-20 05:17:47 by anonymous:
Helgrind is useful to find these sorts of multi-thread race conditions, as well as mutex deadlocks.

http://valgrind.org/docs/manual/hg-manual.html

 
2577 code closed 2007 Aug anonymous   2007 Aug   5 5 Initilizing pointers to 0 instead of NULL. edit
  Is it better to change all initilizations an comparisions of pointers
  to NULL instead of 0 ? It is only cosmetic change but the code is then
  more readable.

  Bartosz
2007-Aug-17 18:15:16 by anonymous:

  Why the ticket was rejected ?

  If we have eg.:
    ....
    *pId = 0;
    ..

  what is *pId ?
  Integer value or pointer to pointer ?
  I know that name suggest that it is one level pointer
  but why we can not make reading easier ?

  Bartosz.


2007-Aug-17 19:48:01 by anonymous:
Changing 0 to NULL adds no value to the code. 0 is one character. NULL is four characters. Both are valid, but 0 is consistent and predictable on all platforms. NULL requires an #include to work, 0 does not.
 
2576 code closed 2007 Aug anonymous   2007 Aug   1 1 Check-in [4239] mutex race conditions edit
Nevermind. I missed seeing SQLITE_MUTEX_STATIC_MASTER.
 
2575 code closed 2007 Aug anonymous   2007 Aug   1 1 sqlite3_mutex_enter missing return edit
I know this is experimental #if'd out code, but nonetheless, sqlite3_mutex_enter is missing a return for the final else clause.
 
2574 build fixed 2007 Aug anonymous   2007 Aug anonymous 3 3 amalgamation 3.4.2 compile errors edit
The new amalgamation 3.4.2 fails with the following compile errors using Visual C++ (2003 and 2005):

sqlite3.c(6187) : error C2133: 'sqlite3UpperToLower' : unknown size sqlite3.c(9310) : error C2133: 'sqlite3OpcodeNames' : unknown size sqlite3.c(47159) : error C2133: 'sqlite3IsIdChar' : unknown size

Cause: The 'extern' keyword was replaced by 'SQLITE_PRIVATE' I suspect this should have been the new macro 'SQLITE_EXTERN'

Changing from SQLITE_PRIVATE to SQLITE_EXTERN results in a successfull build

2007-Aug-15 10:35:23 by drh:
The problem is that there is a declaration of the constant:

   SQLITE_PRIVATE const unsigned char sqlite3UpperToLower[];

Then later the constant is defined with its content:

   SQLITE_PRIVATE const unsigned char sqlite3UpperToLower[] = {...};

How does VC++ expect you to declare a static constant with file scope before it is defined? GCC is quite happy with the construct above.


2007-Aug-15 12:56:30 by anonymous:
VC++ is only happy when you declare it as:

SQLITE_PRIVATE const unsigned char sqlite3UpperToLower[256];

It seems it has to know the size upfront. I've also tried various compiler switches, but no postive results.


2007-Aug-15 13:29:05 by anonymous:
C0x and C++0x seem to allow the construct of forward declaring an unknown sized array of static (file scope) types. I wonder if gcc in --pedantic mode might also complain about it.

This compilation set of settings is what perl uses to help guarantee cross-platform portability:

gcc -g -O -pedantic -Wall -W -Wshadow -Wpointer-arith -Wbad-function-cast -Wcast-qual -Wcast-align -Wwrite-strings -Wstrict-prototypes -Wmissing-prototypes -Wnested-externs -Wno-long-long


2007-Aug-15 13:29:41 by anonymous:
Does VC++ allow the same forward declaration without const?

  SQLITE_PRIVATE unsigned char sqlite3UpperToLower[];

If not, putting the size in advance for the array forward declaration is acceptable.


2007-Aug-15 13:36:57 by anonymous:
SQLITE_PRIVATE unsigned char sqlite3UpperToLower[];

This produces the same error and also another one:

error C2133: 'sqlite3UpperToLower' : unknown size

error C2373: 'sqlite3UpperToLower' : redefinition; different type modifiers

The second error was to be expected


2007-Aug-15 13:42:36 by anonymous:
A function could be created to simply return the pointer to the array in question. Seemingly inefficient, but any modern optimizing compiler in the last 5 years would inline this function call away if in the same translation unit, as in the amalgamation.


2007-Aug-15 13:54:05 by anonymous:
Just change the forward declaration to a pointer:

  SQLITE_PRIVATE const unsigned char *sqlite3UpperToLower;

as well as the implementation:

  SQLITE_PRIVATE const unsigned char *sqlite3UpperToLower = { ... };


2007-Aug-15 14:08:11 by anonymous:
Does VC++ like this?

  static char* arr;
  static char* arr;
  static char arr_impl[] = { 'f', 'o', 'o' };
  static char *arr = arr_impl;


2007-Aug-15 15:17:45 by anonymous:
No, it does not. It gives redefinition warnings on the initializer.

I think part of the problem is that 'static TYPE var;' is default initialized when it is seen, and SPACE IS ALLOCATED FOR IT in any compilation unit that sees it. It is not given external scope.

What I think I see is that every part of the individual .c files were scanned for file-scope definitions, which were then placed into an internals.h file for inclusion across all things that needed it.

What I think you'll see if you look at the .o files for each .c file compiled separately, is a definition for each of the static variables that you were trying to predeclare.

I think they'll need extern scope if you want to gain access to them in separate compilation units. And I think it might be better to declare and use the opcodes list, the UpperToLower table, and the character map table as pointers rather than arrays. Or bite the bullet of using extern instead of static.

If you want to avoid overrunning the length of the tables you could provide an accessor function when printing the vdbe opcodes which could do range checking. Similarly, you could use something that is C-compiler safe for doing compile-time assertions that the length of the character map and lower-case tables are exactly 256 entries long when initialized.

static bool must_be_256_long[sizeof(table)==256];


2007-Aug-15 15:28:56 by anonymous:
The compile assert is not portable.

  static int must_be_256_long[0];

gcc does not complain about this unless the -pedantic flag is used.


2007-Aug-15 15:33:51 by anonymous:
All the external variables in question should be moved to a seperate vars.h and vars.c file. The amalgamation would have this vars.c file added before all other sqlite3 .h/.c files, and the vars.h file need not be added to the amalgamation at all. vars.h would only be used for non-amalgamation builds.


2007-Aug-15 15:34:14 by anonymous:
Since it's explicitly defined later, why not just mark the forward declaration as "extern"? This involves removing the SQLITE_PRIVATE in front. VS2005 let it compile for me no problems.


2007-Aug-15 18:05:49 by anonymous:
gcc will not allow an extern forward declaration followed by a static declaration.

For background of why things were done the way they are see Ticket #2554 "Many symbols not marked private/static or SQLITE_API in amalgamation"


2007-Aug-16 13:57:34 by anonymous:
It seems to me that there are only the three symbols in concern:

  • the vdbe opcode-to-string table, which is used in one place and auto-generated into another. Perhaps the place that it is used within could #include the auto-generated file, and exclude the auto generated file from the remainder of the compilation. (by naming it with .h, if necessary.) Alternatively, it could be encapsulated in an accessor sub-function instead of directly referred to by its array-based name.

  • the character map for converting ebcdic to lower-cased ascii, or the similar translation when converting ascii to lower-cased ascii. It's used by one function when converting from a TOKEN to a token ID. It could also be just IN the file marked as SQLITE_PRIVATE and not forward declared at all.

  • the upper to lower converter is used in a few places :) . No easy/quick solution for that that doesn't involve a possible decrease in performance based on non-inlined function calls in non-amalgamation compiled libraries.
 
2573 doc fixed 2007 Aug anonymous   2007 Aug   3 3 sqlite3_interrupt() + threads edit
sqlite3_interrupt() documentation says that "It is safe to call this routine from a thread different from the thread that is currently running the database operation".

It is not true.

1st problem: db may be freed in sqlite3_close() during call to sqlite3_interrupt() - this may cause a crash.

2nd problem: after

(db->magic==SQLITE_MAGIC_OPEN || db->magic==SQLITE_MAGIC_BUSY)

and before

db->u1.isInterrupted = 1;

a lot of things can happen. Possibly another statement may be executed - this will interrupt a different statement than was expected.

 
2572 warn closed 2007 Aug anonymous   2007 Aug   1 1 Sqlitw error unrecognize token edit
Sqlite error unrecognize token
The "is human" test needs to be improved.


2007-Aug-14 21:20:44 by anonymous:
wa?
 
2571 todo closed 2007 Aug anonymous   2007 Aug   1 1 SQLite opening in Access edit
I am trying to open SQLite database through ODBC connections in Microsoft Access 2000. Some tables in Database have Primary key defined . Steps followed:- System DSN TestSQLite is created using Data sources(ODBC) 1)Open Blank access database named db1.mdb 2)Click on FIle->Get External data->Link tables 3)Select - ODBC databases in Files of type in Link Dialog 4)Select Machine data source TestSQLite=20 5)Select All Tables and click Ok 6)If table does not have Primary key ,getting dialog to select unique record identifier. 7)if Primary key is defined in table ,getting error Reserved error (-7748) There is no msg for this error . =20 I can open tables which does not have Primary Key and cannot use MSAccess to open SQLite database through ODBC (if the SQLite database has primary key defined) . Thanks for your help.
2007-Aug-13 21:23:54 by anonymous:
Install this sqlite odbc driver.

http://www.ch-werner.de/sqliteodbc/

 
2570 code fixed 2007 Aug anonymous   2007 Aug   3 3 unportable assumptions in sqlite3AtoF() / cast.test edit
The sqlite3 selftests 3.14, 3.18 and 3.24 fail for me with Expected: [9223372036854774784] Got: [9223372036854773760]

Looking at the binary representations one finds that one bit of accuracy is missing (expected 53 bits, got 52 bits). The reason is within sqlite3AtoF(): It uses "long double" internally, and the (simplistic) algorithm does only deliver the expected result if "long double" uses a longer mantissa than "double". This might work on Linux/i386, but on other platforms a "long double" is identical to "double".

 
2569 code closed 2007 Aug anonymous   2007 Aug   3 2 Executing a statement causes an AccessViolation Exception edit
Executing follwowing statement causes an AccessViolation Exception:

select being.bid, bname, bvalue, bsyncstatus, bvalid from being
where being.bid in (
select bid from searchidx_Campaign
where
(
  campaign_status_id = 2 or campaign_status_id = 3
)
and id in (select campaign_id from searchidx_CampaignUserStatus
            where user_id = 'b37dd383-3c82-4fe1-9285-d4c2bd6e4ca3'
              and locked = 0))
I tracked the problem down to the 'campaign_status_id = 2 or campaign_status_id = 3' part of the statement. if an 'and' instead of the 'or' is used the Exception doesn't occur ('campaign_status_id = 2 AND campaign_status_id = 3').

It also doesn't matter if there is data in any of the used tables. The exception raises also with empty tables.

Used configuration: Windows XP Professional SP2 -> all MS updates installed .NET Framework 2.0 (2.0.50727.832) using c#

2007-Aug-13 10:17:56 by anonymous:
It would be easier to reproduce if you provided the schema for the tables you use.


2007-Aug-13 14:59:11 by drh:
I created a schema that allowed the SQL statement to compile and then ran the SQL on version 3.3.4 without any problems.

Unable to reproduce.

 
2568 new active 2007 Aug anonymous   2007 Aug   3 3 TEMP_STORE is ignored in some cases edit
It seems that sometimes TEMP_STORE is ignored. I've tried to force SQLite to always use memory by setting TEMP_STORE=3, but some etilqs_* temp files are still being created. The call stack that's causing these file to be created is:

  sqlite3PagerOpentemp(OsFile * *)
  sqlite3PagerStmtBegin(Pager *)
  sqlite3BtreeBeginStmt(Btree *)
  sqlite3VdbeExec(Vdbe *)
  sqlite3Step(Vdbe *)
  sqlite3_step(sqlite3_stmt *)

It looks like the temp files are being used to store information for undoing earlier parts of a transaction if a later part fails. I'm assuming the fact this part of the code ignores TEMP_STORE is an over site?

2007-Aug-13 15:03:19 by drh:
The TEMP_STORE compile-time option only changes the storage for temporary database files. The statement journal is not a databaes file and thus does not come under the control of TEMP_STORE. There is currently no mechanism to force the statement journal into memory instead of onto disk.

I will reclassify this ticket as a "feature request".


2007-Aug-22 10:42:50 by anonymous:
Okay, thank you.
 
2567 build active 2007 Aug anonymous   2007 Aug   3 2 Build fails to install edit
I compile under MinGW with Msys. A build error occurs during 'make install'. After checking the makefile. The 'install' target depends on 'sqlite3', when it should be 'sqlite3$(TEXE)'.

The workaround is, after configure, edit makefile for target install, and replace 'sqlite3' with 'sqlite3${TEXE}' where needed. I did not have this problem with 3.3.17. I assume this could be fixed just by fixing the configure to produce correct makefile.

2007-Aug-12 04:41:12 by anonymous:
Do you have a patch?
 
2566 build active 2007 Aug anonymous   2007 Aug   2 1 fts2 broken after vacuum edit
Hi there, I'm testing your database and I'm having problems with fts2:
sqlite> select * from distB where distB match "MARIANO"; Assertion failed: *pData!='\0', file fts2amal.c, line 16790

This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information.


Steps: 1) Create a new .db 2) Import data in new distA table 3) Import data in new distB table 4) Create a new distC virtual table (dts2) 5) insert into distC (rowid, f1, f2, f3) select rowid, f1, f2, f3 from DistB

Everything working like a charm until here!!! The fts2 works very well, but after

6) vacuum;

the fts seems broken... doing a select throws the error I paste at the post of the topic

If you want the .db file I can send it to you (607MB)

Thanks.-

 
2565 code fixed 2007 Aug drh   2007 Aug   1 1 Database corruption following ROLLBACK when soft_heap_limit set edit
With autovacuum mode turned on low sqlite3_soft_heap_limit database corruption results following a ROLLBACK of a CREATE TABLE. The following script exhibits the problem:

    -- set sqlite3_soft_heap_limit to 5000
    PRAGMA auto_vacuum=1;
    CREATE TABLE t1(x);
    INSERT INTO t1 VALUES(hex(randomblob(5000)));
    BEGIN;
    CREATE TABLE t2 AS SELECT * FROM t1;
    ROLLBACK;
    PRAGMA integrity_check;

This problem was discovered following the enhanced sqlite_soft_heap_limit testing facility added by check-in [4202] . As of this writing, it has not been observed in the wild.

2007-Aug-11 11:55:04 by drh:
Background

The rollback journal file consists of a header followed by zero or more page records. Each page record contains original database content which can be used to restore the database back to its original state during a ROLLBACK or when the database is being recovered after a power-loss or crash. The header contains, among other things, the "nRec" field which is a count of the number of pages to follows.

Whenever changes are flushed to the database file, the size of the journal is padded out to the next multiple of the disk sector size and a new header is written. (This is a defense against certain kinds of file corruption that might occur during a power failure.) A single journal can contain any number of instances of header followed by zero or more pages.

While the journal is being constructed, the nRec value is 0. Then, just prior to writing changes back to the database file, the nRec value is changed to the correct number of records in the journal, the journal file is synced to disk, and only then is it safe to write to the database file. This all works great.

The Problem

The problem is that the ROLLBACK command also uses the journal file to restore changes. But the ROLLBACK command treats the nRec==0 value specially. When the ROLLBACK command sees an nRec of 0, it figures that it was just in the middle of doing some changes that haven't yet by written to the database file, so it tries to recompute the "correct" nRec value based on the current file size.

The problem arises when a header contains an nRec value which really is suppose to be zero (no pages written into the journal) but that header is followed by one or more additional headers that contain pages that do need to be rolled back. When a journal is being rolled back during recovery, the recovery process sees the zero nRec, skips to the next header, and every thing works peachy. But during the ROLLBACK command, the zero nRec is converted into some positive value based on the journal file size. The new positive nRec is wrong. The ROLLBACK quickly discovers this and ignores the rest of the journal file under the assumption that it corrupted. This results in an incomplete ROLLBACK and database file corruption.

The Fix

The fix in check-in [4208] is to change the ROLLBACK process so that it only applies the special interpretation to an nRec==0 header if it is the very last header in the journal.

Why We Have Not Seen This Before

The problem only comes up when a header is written into the journal with nRec==0 and subsequent headers are written afterwards. This can only happen when the pager is having great difficulty finding new buffers into which new pages can be loaded and is having to spill dirty pages back to the disk. The minimum cache size is 10, and as it turns out a cache size of 10 still gives a handful of buffered pages before a cache spill occurs, even under the most severe circumstances. So even with a very small cache, the nRec field was always ending up larger than 0. But, if you set the soft heap limit to a ridiculously small number - so small that the pager is under continuous pressure to spill pages back to the database file as soon as they are unpinned - then there are cases where the pager will write non-terminal headers to the journal with nRec==0.

Hence, as far as we are aware, you cannot hit this problem as long as you leave the soft heap limit disabled (which is the default configuration) or you set the soft heap limit large enough so that there are always enough memory for the pager to hold 10 or more pages in memory at the same time. It is difficult to imagine a real-world system that would do something different, so even though this problem has serious consequences, we cannot see how it might actually come up in practice.

The problem was only noticed when we started stress-testing the soft heap limit mechanism by setting very low soft heap limits and running full regression tests.

 
2564 event closed 2007 Aug anonymous   2007 Aug anonymous 4 4 in C:\windows\temp are 50 files like sqlite_fMyj88gwaTTNgUF and 0 kb edit
in C:\windows\temp are 50 files like sqlite_fMyj88gwaTTNgUF and 0 kb every day are more files , which program use these files i cannot delete them with windows, clean up service i deleted them in safe mode they keep coming back and adding more files

i use windows xp home edition and mcafee told me, it was not mcafee but a windows program

thank you for your time jessica

2007-Aug-10 12:18:32 by anonymous:
http://www.sqlite.org/cvstrac/wiki?p=McafeeProblem
 
2563 code closed 2007 Aug anonymous   2007 Aug   1 1 sqlite3_column_database_name16() is broken edit
sqlite3_column_database_name16() does not work at all.

However I form my query, regardless of how many fields and in what order are returned in result, it always returns NULL for every single column. I'm using this code and it never returned anything but NULL:

    if ( pStatement)
    {
      UINT32 cols = sqlite3_column_count( pStatement);
      for ( UINT16 currCol = 0; !isDatabaseNameRetrieved && ( currCol < cols ); ++currCol )
      {
        const WCHAR *databaseName = static_cast<const WCHAR*>( sqlite3_column_database_name16( pStatement, currCol ) );
        if ( databaseName && *databaseName )
        {
          *dbName = databaseName;
          isDatabaseNameRetrieved = true;
        }
      }
    }

When I stepped through the code I noticed that MEM_Null flag check in sqlite3ValueText() always fails (flag is always set):

  static const void *sqlite3ValueText(sqlite3_value* pVal, u8 enc){
    if( !pVal ) return 0;

    assert( (enc&3)==(enc&~SQLITE_UTF16_ALIGNED) );
    if( pVal->flags&MEM_Null ){                      //<--- THIS CHECK IS ALWAYS TRUE
      return 0;
    }
2007-Aug-11 16:07:41 by drh:
The 35 separate test cases in capi3.test and capi3c.test that call this routine all seem to function correction and return non-NULL results.

Unable to reproduce.


2007-Aug-14 20:54:58 by anonymous:
Hmmm. It NEVER worked for me. If I have just main database opened it will return "main" but as soon as I attach another one it starts returning consistently returning NULL. I had the same problem with 3.3.17 and 3.4.0.
 
2562 code closed 2007 Aug anonymous   2007 Aug   2 3 Multithreaded transactions deadlock (Windows) edit
Two threads connected to the same database/table and use transactions:

1. 1st thread: open db3-file
2. 2nd thread: open the same (second connection handle)
3. 1st thread: begin transaction
4. 1st thread: run multiple 'insert' statement
5. 2nd thread: begin transaction (successful return from call)
6. 2nd thread: want to 'insert' but blocked (as expected), start waiting on busy flag
7. 1st thread: last 'insert' was successfull
8. 1st thread: want to 'end' transaction, but blocked (sqlite3_step returns infinite SQLITE_BUSY, WHY?!)
9. 2nd thread: still blocked on first insert attempt
10. So, both threads can't continue execution.
Workaround: remove 'begin' and 'end' SQL statements from the code - and this works OK, but too slow.
2007-Aug-10 04:13:37 by anonymous:
You might try BEGIN EXCLUSIVE. You lose parallelism, but it won't deadlock, and the code is simpler.

See also:

http://www.sqlite.org/capi3ref.html#sqlite3_busy_handler

"If SQLite determines that invoking the busy handler could result in a deadlock, it will return SQLITE_BUSY instead. Consider a scenario where one process is holding a read lock that it is trying to promote to a reserved lock and a second process is holding a reserved lock that it is trying to promote to an exclusive lock. The first process cannot proceed because it is blocked by the second and the second process cannot proceed because it is blocked by the first. If both processes invoke the busy handlers, neither will make any progress. Therefore, SQLite returns SQLITE_BUSY for the first process, hoping that this will induce the first process to release its read lock and allow the second process to proceed."


2007-Aug-10 11:36:32 by anonymous:
Yes, with 'begin exclusive' it works.
 
2561 code fixed 2007 Aug anonymous   2007 Dec   1 1 Parser broken -- ORDER BY fails in UNION between two databases edit
To reproduce:

1. Create two simple databases, each with only one table "Table1" containing two fields -- ID (integer primary key) and Name (string);

2. Fill databases with a few records each of sample data;

3. Open one database and attach the other one as "aux";

4. Try this query: SELECT Table1.Name, Table1.ID FROM main.Table1 UNION SELECT Table1.Name, Table1.ID FROM aux.Table1 ORDER BY Table1.ID;

5. Query fails with message: "SQL error: ORDER BY term number 1 does not match any result column".

2007-Aug-09 05:27:20 by anonymous:
This doesn't work either: SELECT * FROM main.Table1 UNION SELECT * FROM aux.Table1 ORDER BY Table1.ID


2007-Aug-09 05:28:16 by anonymous:
WORKAROUND!!!

This works:

SELECT Table1.name, Table1.ID AS alias FROM main.Table1 UNION SELECT Table1.name, Table1.ID AS alias FROM aux.Table1 ORDER BY alias


2007-Aug-09 05:31:12 by anonymous:
SELECT Table1.Name, Table1.ID FROM main.Table1 UNION SELECT Table1.Name, Table1.ID FROM aux.Table1 ORDER BY 2;


2007-Aug-10 01:55:58 by anonymous:
Real problem for me is that this is not working, as previously mentioned:

  SELECT * FROM main.Table1 UNION SELECT * FROM aux.Table1 ORDER BY Table1.ID

Very inconvenient.


2007-Sep-14 07:22:10 by anonymous:
I also have this problem and had to implement a bit of a dirty workaround to get aliases on the order by column as suggested before.

Would be nice if a minor update would come out that fixes this (dont want to install 3.5 ALPHA for a possible future fix) ;-)


2007-Sep-14 08:51:23 by anonymous:
AUX is in the DOS/Windows world reserved name so maybe it is the source of problems ?


2007-Dec-13 18:58:37 by drh:
See ticket #2822.
 
2560 doc fixed 2007 Aug anonymous   2007 Aug   5 4 Documentation typos edit
Normally, I don't log typos, but I draw the line when the product name (sqlite) is in error. I'm joking, right? No, it's true. If you look at the blob example at http://sqlite.org/lang_expr.html it shows: X'53514697465'

However, that is an odd number of digits up there, considering that the part between the quotes is supposed to be hex. In fact, it seems to want to spell 'SQLite'. However, that L is having problems. Presumably, there should be a 'C' after the '4'.

In http://www.sqlite.org/lang_insert.html the second occurence of 'not' should be 'no'.

In http://www.sqlite.org/limits.html the first two occurrences of 'where' should be 'were'.

Csaba Gabor from Vienna

2007-Aug-08 23:39:54 by anonymous:
I reported blob literal issue a couple of months ago. See ticket #2168. I wonder if somebody is going to do something with it.
 
2559 code active 2007 Aug anonymous   2007 Aug   4 4 "make clean" does not delete sqlite3.c and tsrc/ edit
Index: Makefile.in
===================================================================
RCS file: /sqlite/sqlite/Makefile.in,v
retrieving revision 1.179
diff -u -3 -p -r1.179 Makefile.in
--- Makefile.in 27 Aug 2007 23:38:43 -0000      1.179
+++ Makefile.in 28 Aug 2007 01:25:55 -0000
@@ -724,7 +724,7 @@ clean:
        rm -f testfixture$(TEXE) test.db
        rm -rf doc
        rm -f common.tcl
-       rm -f sqlite3.dll sqlite3.lib sqlite3.def
+       rm -rf sqlite3.dll sqlite3.lib sqlite3.def sqlite3.c tsrc

 distclean:     clean
        rm -f config.log config.status libtool Makefile config.h sqlite3.pc
 
2558 code active 2007 Aug anonymous   2007 Aug   2 3 Multiple JOIN USING() gives incorrect results edit
I'm having a problem joining multiple tables with USING. It appears to work, but the results are incorrect. Here is an example to illustrate the problem. I believe the three SELECT statements should be equivalent, but they produce three different results.

  .header on
  .mode column

  CREATE TABLE Main (pk INTEGER PRIMARY KEY, name VARCHAR);
  CREATE TABLE OptA (pk INTEGER PRIMARY KEY, alpha VARCHAR);
  CREATE TABLE OptB (pk INTEGER PRIMARY KEY, beta VARCHAR);

  INSERT INTO Main VALUES (1, 'One');
  INSERT INTO Main VALUES (2, 'Two');
  INSERT INTO Main VALUES (3, 'Three');
  INSERT INTO Main VALUES (4, 'Four');

  INSERT INTO OptA VALUES (1, 'Alpha1');
  INSERT INTO OptA VALUES (4, 'Alpha4');

  INSERT INTO OptB VALUES (2, 'Beta2');
  INSERT INTO OptB VALUES (4, 'Beta4');

  SELECT * FROM Main LEFT JOIN OptA USING (pk) LEFT JOIN OptB USING (pk);

  SELECT * FROM Main LEFT JOIN OptB USING (pk) LEFT JOIN OptA USING (pk);

  SELECT Main.pk, name, alpha, beta
    FROM Main LEFT JOIN OptA ON Main.pk = OptA.pk
              LEFT JOIN OptB ON Main.pk = OptB.pk;

Joining Main, OptA, and OptB omits Beta2:

  pk          name        alpha       beta
  ----------  ----------  ----------  ----------
  1           One         Alpha1
  2           Two
  3           Three
  4           Four        Alpha4      Beta4

Joining Main, OptB, and OptA omits Alpha1:

  pk          name        beta        alpha
  ----------  ----------  ----------  ----------
  1           One
  2           Two         Beta2
  3           Three
  4           Four        Beta4       Alpha4

Only by using ON instead of USING do we get the correct results:

  pk          name        alpha       beta
  ----------  ----------  ----------  ----------
  1           One         Alpha1
  2           Two                     Beta2
  3           Three
  4           Four        Alpha4      Beta4

I think this is basically the same issue as ticket #1637, but it's a more serious example. In that one, the query simply failed to compile. In this case, it seems to work, but gives you the wrong results.

I've also tried this script in PostgreSQL 8.0.13. All three queries give (the same) correct results.

2007-Aug-08 17:34:27 by anonymous:
The problem is that SQLite is transforming

  SELECT * FROM Main LEFT JOIN OptA USING (pk) LEFT JOIN OptB USING (pk);

into

  SELECT Main.pk, name, alpha, beta
    FROM Main LEFT JOIN OptA ON Main.pk = OptA.pk
              LEFT JOIN OptB ON OptA.pk = OptB.pk;

Here is a workaround to this bug that makes use of a subquery:

  select * from (SELECT * FROM Main LEFT JOIN OptA USING (pk))
    LEFT JOIN OptB USING (pk);

Conceivably all LEFT JOIN chains could be transformed into the above form, but that would decrease performance due to the intermediate result set of the subquery. Having it work without the subquery is tricky since sqlite must deduce that the last USING (pk) is equivalent to the first pk in the chain of joined tables, namely Main.pk, and not OptA.pk.

Joe Wilson

 
2557 build closed 2007 Aug anonymous   2007 Aug   3 3 make test errors on Rhel4 x86_64 edit
Hello there,

While building sqlite 3.4.1 rpms for Rhel4 on i386 and x86_64, I came across some inconsistent behaviour with "make test".

On Rhel4.5 i386, "make test" works fine.

However, on Rhel4.5 x86_64, "make test" has two errors which terminate the rpm build process.

rpmbuild -bb --with tcl sqlite.spec 2>&1 | tee build.log
.
.
printf-7.5... Ok
printf-8.1...
Error: integer value too large to represent
printf-8.2...
Error: integer value too large to represent
printf-8.3... Ok
.
.
Thread-specific data deallocated properly
2 errors out of 29903 tests
Failures on these tests: printf-8.1 printf-8.2
make: *** [test] Error 1
error: Bad exit status from /var/tmp/rpm-tmp.97578 (%check)

My current workaround is to apply a patch which is only applied on x86_64 hardware. Again, thanks for your time and great software.

2007-Aug-07 17:17:39 by drh:
Neither test is particularly important and both failures problem result from bugs in either TCL or somewhere else in RH, not in the SQLite core.

If you want me to look into this, please consider publishing your patch. I have no way to reproduce the problem otherwise.

 
2556 todo fixed 2007 Aug anonymous   2007 Aug   5 5 tabs instead of space in some source files edit
There isn't a ticket priority low enough for this item. A few source files use tabs instead of spaces.

  btree.c
  expr.c
  os_unix.c
  pager.c
  sqliteInt.h
  tclsqlite.c
  test_md5.c
  trigger.c
  vdbeaux.c
 
2555 new active 2007 Aug anonymous   2007 Aug   1 1 FTS index without original text edit
Is it possible to build FTS index without storing original text? I want to use fts index without features of snippets etc. I just want to find ID of the record not the content of indexed phrase. I suppose that the table myname_content stores this content. I have tried to update all columns of myname_content and set its values to &#8220;xyz&#8221; (without one column in which I store ID of the record). After this operation FTS search works good, but unfortunately the table isn&#8217;t smaller (I cant&#8217;t use vacuum on FTS tables). Is there any other way to have pure text indexes without source level changes?
 
2554 code fixed 2007 Aug anonymous   2007 Aug   3 3 Many symbols not marked private/static or SQLITE_API in amalgamation edit
I compiled the amalgamation with SQLITE_API=static and then ran "nm -D" on the resulting output which shows undefined (imported symbols) as well as all symbols that are publicly exported.

There are numerous sqlite3 symbols that didn't get marked as SQLITE_API or SQLITE_private. Note that my ultimate goal is for two different versions of SQLite to coexist in the same process. This is especially important on the Mac where many of Apple's own components cause the system SQLite library to be loaded, but is also happening on other operating systems where SQLite is used by system libraries. Here is a real world example of the problems:

http://lists.initd.org/pipermail/pysqlite/2007-June/001074.html

Here is the begining of the list of public symbols from nm -D:

  0000000000017460 T sqlite3BinaryCompareCollSeq
  000000000005be20 R sqlite3IsIdChar
  000000000026b040 D sqlite3OpcodeNames
  000000000005bd20 R sqlite3UpperToLower
  000000000005bce0 R sqlite3UtfTrans1
  000000000000f0b0 T sqlite3_aggregate_context
  000000000000b650 T sqlite3_aggregate_count
  000000000026bf60 D sqlite3_apis
  000000000001d380 T sqlite3_auto_extension
  0000000000016cb0 T sqlite3_bind_blob
  0000000000016660 T sqlite3_bind_double
  0000000000016650 T sqlite3_bind_int
  00000000000165e0 T sqlite3_bind_int64
  0000000000016590 T sqlite3_bind_null
  000000000000b690 T sqlite3_bind_parameter_count
  0000000000020b30 T sqlite3_bind_parameter_index
  000000000000b700 T sqlite3_bind_parameter_name
  0000000000016ca0 T sqlite3_bind_text
  0000000000016c80 T sqlite3_bind_text16
  0000000000016520 T sqlite3_bind_value
  00000000000164b0 T sqlite3_bind_zeroblob
  000000000000b7a0 T sqlite3_blob_bytes
  000000000002b790 T sqlite3_blob_close
  00000000000478f0 T sqlite3_blob_open
  000000000002b870 T sqlite3_blob_read
  000000000002b860 T sqlite3_blob_write
  000000000000d940 T sqlite3_busy_handler
  000000000000da30 T sqlite3_busy_timeout
  00000000000165a0 T sqlite3_clear_bindings
  000000000000db60 T sqlite3_collation_needed
  000000000000dbc0 T sqlite3_collation_needed16
  0000000000017a30 T sqlite3_column_blob
  00000000000180a0 T sqlite3_column_bytes
  0000000000017ad0 T sqlite3_column_bytes16
  000000000000b660 T sqlite3_column_count
  0000000000016d60 T sqlite3_column_decltype
  0000000000016d40 T sqlite3_column_decltype16
2007-Aug-06 12:06:09 by anonymous:
Just curious, what do you hope to do with the public API symbols? e.g., sqlite3_column_count. Rename them?


2007-Aug-06 15:59:19 by anonymous:
The sqlite3.c file is #included into my file that does all the database interfacing. There is no other file in my application that needs access to any sqlite code or data. If all the sqlite3 api had SQLITE_API in front of them (which I define to static) then that would work, but as it stands only some of them have SQLITE_API.


2007-Aug-06 16:09:09 by anonymous:
You could also load your 2nd sqlite3 code via the dlopen option RTLD_LOCAL. There must be a MacOS equivalent.


2007-Aug-06 18:54:12 by anonymous:
I am trying to solve this properly for all platforms and in the long term. Remember that I have no control over other libraries in the same process and how they link (and in any case they can link to another library that links to another that finally loads SQLite).

SQLite is most of the way there to being totally self contained. It just needs SQLITE_API in front of all api functions, not just some of them and private internal sqlite data should really be marked as such.


2007-Aug-06 23:32:37 by anonymous:
Apply the patch below to the latest CVS sources, and run "./configure && make clean && make sqlite3.c". This will handle most of the public sqlite3 symbols.

This patch addresses a bug in mksqlite3c.tcl that prevented putting SQLITE_API prefixes before the sqlite3_* functions which had more than one underscore in their name (such as sqlite3_aggregate_count), or had a '*' in an unexpected place in the return type (sqlite3BinaryCompareCollSeq).

After the patch, these are the only public symbols in the library exposed:

   sqlite3IsIdChar
   sqlite3one
   sqlite3OpcodeNames
   sqlite3UpperToLower
   sqlite3_version
   sqlite3_io_trace
   sqlite3_mallocHasFailed
   sqlite3_os_trace
   sqlite3_temp_directory

To prevent collision with another sqlite3 library, in your own C module start out your code as follows, changing 'foo' to a unique name:

  #define sqlite3IsIdChar         foo_sqlite3IsIdChar
  #define sqlite3one              foo_sqlite3one
  #define sqlite3OpcodeNames      foo_sqlite3OpcodeNames
  #define sqlite3UpperToLower     foo_sqlite3UpperToLower
  #define sqlite3_version         foo_sqlite3_version
  #define sqlite3_io_trace        foo_sqlite3_io_trace
  #define sqlite3_mallocHasFailed foo_sqlite3_mallocHasFailed
  #define sqlite3_os_trace        foo_sqlite3_os_trace
  #define sqlite3_temp_directory  foo_sqlite3_temp_directory
  #include "sqlite3.c"

With this technique you can embed several different version of the sqlite3 library in the same program, assuming you compile your module with:

  -DSQLITE_API=static -DSQLITE_PRIVATE=static

The sqlite3 sources could be reorganized to make these last few sqlite3 symbols private as well, but it would be a pain as these variables are used across several .c files.

Patch below ran "make test" without error on Linux.

Joe Wilson

Index: src/loadext.c
===================================================================
RCS file: /sqlite/sqlite/src/loadext.c,v
retrieving revision 1.23
diff -u -3 -p -r1.23 loadext.c
--- src/loadext.c       20 Jul 2007 10:33:59 -0000      1.23
+++ src/loadext.c       6 Aug 2007 23:10:38 -0000
@@ -109,7 +109,7 @@
 ** also check to make sure that the pointer to the function is
 ** not NULL before calling it.
 */
-const sqlite3_api_routines sqlite3_apis = {
+static const sqlite3_api_routines sqlite3_apis = {
   sqlite3_aggregate_context,
   sqlite3_aggregate_count,
   sqlite3_bind_blob,
Index: src/utf.c
===================================================================
RCS file: /sqlite/sqlite/src/utf.c,v
retrieving revision 1.52
diff -u -3 -p -r1.52 utf.c
--- src/utf.c   23 Jul 2007 19:12:42 -0000      1.52
+++ src/utf.c   6 Aug 2007 23:10:39 -0000
@@ -49,7 +49,7 @@ const int sqlite3one = 1;
 ** This lookup table is used to help decode the first byte of
 ** a multi-byte UTF8 character.
 */
-const unsigned char sqlite3UtfTrans1[] = {
+static const unsigned char sqlite3UtfTrans1[] = {
   0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07,
   0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f,
   0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17,
Index: tool/mksqlite3c.tcl
===================================================================
RCS file: /sqlite/sqlite/tool/mksqlite3c.tcl,v
retrieving revision 1.14
diff -u -3 -p -r1.14 mksqlite3c.tcl
--- tool/mksqlite3c.tcl 26 Jun 2007 00:52:40 -0000      1.14
+++ tool/mksqlite3c.tcl 6 Aug 2007 23:10:39 -0000
@@ -129,10 +129,12 @@ proc copy_file {filename} {
   set tail [file tail $filename]
   section_comment "Begin file $tail"
   set in [open $filename r]
+  set declpattern \
+    {([a-zA-Z][_a-zA-Z0-9* ]+)+(sqlite3[_A-Z][_a-zA-Z0-9]+)\(}
   if {[file extension $filename]==".h"} {
-    set declpattern {^ *[a-zA-Z][a-zA-Z_0-9 ]+ \*?(sqlite3[_A-Z][a-zA-Z0-9]+)\(}
+    set declpattern "^ *$declpattern"
   } else {
-    set declpattern {^[a-zA-Z][a-zA-Z_0-9 ]+ \*?(sqlite3[_A-Z][a-zA-Z0-9]+)\(}
+    set declpattern "^$declpattern"
   }
   while {![eof $in]} {
     set line [gets $in]
@@ -154,7 +156,7 @@ proc copy_file {filename} {
       puts $out "#if 0"
     } elseif {[regexp {^#line} $line]} {
       # Skip #line directives.
-    } elseif {$addstatic && [regexp $declpattern $line all funcname]
+    } elseif {$addstatic && [regexp $declpattern $line all rtype funcname]
                   && ![regexp {^static} $line]} {
       # Add the SQLITE_PRIVATE or SQLITE_API keyword before functions.
       # so that linkage can be modified at compile-time.

2007-Aug-07 20:51:20 by anonymous:
The following symbols remain public after Check-in [4197] .

missed SQLITE_PRIVATE symbols:

00003900 R sqlite3_apis
00000008 B sqlite3_io_trace
00000000 B sqlite3_mallocHasFailed
00000004 B sqlite3_os_trace
0000000c B sqlite3_temp_directory
00007c1b R sqlite3_version
missed SQLITE_API symbols:

0001d6fb T sqlite3_aggregate_context
0001d8ed T sqlite3_aggregate_count
0003c3e3 T sqlite3_auto_extension
0001ded0 T sqlite3_bind_blob
0001df07 T sqlite3_bind_double
...etc... 96 more...
This patch addresses the SQLITE_API lines above:

Index: tool/mksqlite3c.tcl
===================================================================
RCS file: /sqlite/sqlite/tool/mksqlite3c.tcl,v
retrieving revision 1.15
diff -u -3 -p -r1.15 mksqlite3c.tcl
--- tool/mksqlite3c.tcl 7 Aug 2007 17:05:00 -0000       1.15
+++ tool/mksqlite3c.tcl 7 Aug 2007 20:06:34 -0000
@@ -130,10 +130,11 @@ proc copy_file {filename} {
   section_comment "Begin file $tail"
   set in [open $filename r]
   set varpattern {^[a-zA-Z][a-zA-Z_0-9 *]+ \*?(sqlite3[a-zA-Z0-9]+)([[;]| =)}
+  set declpattern {[a-zA-Z][a-zA-Z_0-9 ]+ \*?(sqlite3[_A-Z][_a-zA-Z0-9]+)\(}
   if {[file extension $filename]==".h"} {
-    set declpattern {^ *[a-zA-Z][a-zA-Z_0-9 ]+ \*?(sqlite3[_A-Z][a-zA-Z0-9]+)\(}
+    set declpattern "^ *$declpattern"
   } else {
-    set declpattern {^[a-zA-Z][a-zA-Z_0-9 ]+ \*?(sqlite3[_A-Z][a-zA-Z0-9]+)\(}
+    set declpattern "^$declpattern"
   }
   while {![eof $in]} {
     set line [gets $in]

2007-Aug-08 00:44:19 by drh:
Symbols sqlite3_apis, sqlite3_temp_directory, and sqlite3_version are part of the published API and cannot be made private without breaking things.


2007-Aug-08 01:05:24 by drh:
Turns out sqlite3_io_trace is used externally too. I have checked in fixes for the rest.


2007-Aug-08 01:41:14 by anonymous:
If the test of the success of these qualifiers is to link different versions of the sqlite3 library into the same binary, then every function and variable must be marked as static, SQLITE_API or SQLITE_PRIVATE (or some other designation like SQLITE_API_VARIABLE, if you like). No function or variable can be without such a qualifier, i.e., public.

If sqlite3_apis, sqlite3_temp_directory, and sqlite3_version are part of the published API then they should be marked as SQLITE_API, even though they are not functions.

sqlite3_io_trace could be marked as either SQLITE_API or SQLITE_PRIVATE. I don't know which one is more appropriate.

  void (*sqlite3_io_trace)(const char*, ...) = 0;

On Linux, with an updated CVS tree after Check-in [4199] we see that we have 4 public symbols remaining:

  make distclean
  ./configure
  make sqlite3.c
  gcc -DSQLITE_API=static -DSQLITE_PRIVATE=static -c sqlite3.c
  nm sqlite3.o | grep -v ' [trUbd] '
  00003900 R sqlite3_apis
  00000000 B sqlite3_io_trace
  00000004 B sqlite3_temp_directory
  00007c1b R sqlite3_version


2007-Aug-08 12:12:06 by drh:
You now must also do -DSQLITE_EXTERN=static.
 
2553 code closed 2007 Aug anonymous   2007 Aug   3 3 AUTOINCREMENT not working when primary key has multiple columns edit
There is one thing which I really miss in SQLite from mySQL, and that is that AUTOINCREMENT-ing does not work when the primary key has multiple columns. AUTO_INCREMENT-ing should work when the last column of the PRIMARY KEY is of type INTEGER. It should work as described in the second example here: http://dev.mysql.com/doc/refman/5.0/en/example-auto-increment.html

In essence, the final column of the primary key gets a value of 1 greater than the largest value in that column where all the other values of the primary key's columns match those of the newly entered row.

Why is the useful? As an example, consider that bug reports are being made. The reports are updated. It would make sense for each update to be incrementally numbered within the particular bug that it belongs to.

Thanks for considering this, Csaba Gabor from Vienna

2007-Aug-05 17:38:39 by drh:
AUTOINCREMENT in SQLite works as it is documented to work in SQLite. SQLite is not MySQL.
 
2552 code closed 2007 Aug anonymous   2007 Aug   1 3 sqlite3_column_double causes invalid floating point operation edit
We have a table that has values inserted by a Delphi database provider. Using one tool, we can see values in the column as:

   0
   NULL
   0.0

When using a different tool, or our .NET provider, conversion errors are generated. The conversion errors occur on the row that is stored as 0.0. The call stack seems to indicate that sqlite3_column_double is having a problem dealing with the value stored as 0.0.

Some of my report is based on third party information and I haven't personally confirmed all details. Let me know if you have further questions.

2007-Aug-03 16:37:35 by anonymous:
It's likely a bug in the sqlite wrapper library you are using.

Post a small standalone C program that only calls the sqlite3 API directly to demonstrate the issue and re-open this ticket.


2007-Aug-03 20:19:20 by anonymous:
When viewing the data in the sqlite command line utility, it comes across as NaN. It appears that the problem is somehow on the input side. What's not clear is if the NaN was created before being bound to the database or if the NaN value was assigned by the database.
 
2551 code closed 2007 Aug anonymous   2007 Aug   3 3 sqlite3 shell gets into bad state with unexpected input edit
The sqlite3 shell can get into a state that you cannot recover from without aborting the process. This happens fairly often on platforms without readline support.

  1. build sqlite3 without readline support.
  2. run sqlite3 in an xterm (or KDE konsole, probably other terminal programs)
  3. at the prompt, press cursor up (probably other 'illegal' characters would suffice as well).
  4. hit return

Nothing you do at this point will work except for Ctrl-\ to kill the process.

SQLite version 3.4.1
Enter ".help" for instructions
sqlite> ^[[A
   ...> .q
   ...> ;
   ...> select 1;
   ...> .q
   ...> .h
   ...> ;
   ...> ;
2007-Aug-03 15:00:30 by anonymous:
Try CTRL+D, worked for me.


2007-Aug-03 21:19:31 by drh:
The "[A" is the beginning of a quoted identifier. The shell continues to issue the continuation prompt because it is waiting on the "]" to terminate the identifier.

Your workaround is to type "];".

Works as designed.

 
2550 code fixed 2007 Aug anonymous   2007 Oct drh 3 3 Lemon parser may accept input before EOF is seen edit
A lemon parser with table compression may accept input before EOF is seen. This can happen even without any syntax errors being reported.

Consider this little lemon grammar:

  %parse_accept{printf("parse accept\n");}
  %syntax_error{printf("syntax error\n");}
  %parse_failure{printf("parse failure\n");}
  start ::= foobar.
  foobar ::= FOO.
  foobar ::= foobar BAR.

This grammer describes input that matches the regular language FOO BAR*, i.e. one FOO followed by zero or more BARs.

This grammar shows how lemon breaks:

When this grammar is fed the input FOO BAR FOO FOO BAR, it will accept twice: the second FOO acts as the EOF terminal after the initial FOO BAR and gets eaten by the parser without error. The parser then accepts and resets itself for new input. The remainder of the input is again a valid FOO BAR, giving rise to the second accept.

More precisely, lemon has the following logic error:

The accept action is triggered at the end of a reduce action of a start rule (that has the start symbol at the left hand side), when that start symbol is about to be pushed onto the stack. This (normally) works because the start symbol may not be used on the right hand side of a rule, so EOF can be the only follower terminal of a start rule.

Unfortunately, in the presence of default rules, a start rule can be used as the default rule, and it can be reduced even when the lookahead symbol is not EOF.

I see two ways of fixing this:

  1. Before accepting, make sure the lookahead symbol is really EOF.

  2. Refuse to reduce a start rule without the lookahead symbol actually being EOF.

Option 1 seems nice, but is not really user friendly with lemon as it has been set up now: the accept action is not really usable as the final indication that all is well, because it has no access to the parse tree in the minor value of the start symbol for returning to the calling program (through the extra parameter).

With option 2, reduction of a start rule is really the last action of the parser (alright, except for the parse_accept action). Normally, I'd say a user uses the start rules for returning the parse tree to the lemon caller, and this is the more naturaly way to do it.

Option 2 has an additional advantage that, in combination with judicious placement of error symbols in the start rule, it remains possible to parse remaining input. This would not be possible if a start rule were reduced prematurely as under option 1, because then the parser would have to go to a state where it discards all remaining input (or it would have to call parse_failure) without actually parsing the remaining input.

Ciao. Vincent.

2007-Sep-27 08:42:25 by anonymous:
YES!!!

I have this error too :-((

Please ... fix it.

Thanks, Alexander.

 
2549 build closed 2007 Aug anonymous   2007 Aug   1 3 3.4.1 fails to upgrade from 3.3.1.7 on freeBSD edit
I use DesktopBSD (freebsd), and I was running the regular software upgrade using the Package Manager. This is the result:

./.libs/libsqlite3.so: undefined reference to `sqlite3Fts2InitHashTable' gmake: *** [sqlite3] Error 1 *** Error code 2

Stop in /usr/ports/databases/sqlite3. *** Error code 1

Stop in /usr/ports/databases/sqlite3. ** Command failed [exit code 1]: /usr/bin/script -qa /tmp/portupgrade.64421.1 env DIALOG=/usr/local/bin/kdialog-ports-wrapper UPGRADE_TOOL=portupgrade UPGRADE_PORT=sqlite3-3.3.17 UPGRADE_PORT_VER=3.3.17 make ** Fix the problem and try again.

Regards,

Arthur

This is a bug for the FreeBSD ports maintainers.
 
2548 code fixed 2007 Aug anonymous   2007 Oct anonymous 4 4 Pelles C compiler for Windows ANSI prototype / K&R declaration errors edit
Compiling for Windows using the PellesC compiler suite.

Mismatches between ANSI style prototypes and K&R style function declarations are reported as errors.

All the error reports are of the form:

...\main.c(1235): error #2120: Redeclaration of 'sqlite3_global_recover' previously declared at ...\sqlite3.h(1693): found 'void _cdecl function' expected 'void _stdcall function(void)'.

1. In main.c at line 1235 need to change int sqlite3_global_recover(){ to int sqlite3_global_recover(void){

2. In malloc.c at line 809 need to change int sqlite3FailedMalloc(){ to int sqlite3FailedMalloc(void){

3. In os_win.c at line 1613 need to change int sqlite3WinEnterMutex(){ to int sqlite3WinEnterMutex(void){

4. In os_win.c at line 1630 need to change int sqlite3WinLeaveMutex(){ to int sqlite3WinLeaveMutex(void){

5. In util.c at line 706 need to change int sqlite3ThreadData(){ to int sqlite3ThreadData(void){

6. In util.c at line 719 need to change int sqlite3ThreadDataReadOnly(){ to int sqlite3ThreadDataReadOnly(void){

7. In util.c at line 730 need to change int sqlite3ReleaseThreadData(){ to int sqlite3ReleaseThreadData(void){

----end of report----

 
2547 code active 2007 Aug danielk1977   2007 Aug   5 3 Changing db encoding of an attached db can confuse shared cache mode. edit
This is quite obscure, but in shared-cache mode:

  1) Open db A, attach empty db B.
  2) Using another connection from the same thread, set the
     encoding of B to be different from that of A. Add some
     data to B.
  3) Using the original connection, access database B. It assumes
     the encoding of A (and therefore mangling any text data).

The correct response is to return an error - "attached databases must use the same text encoding as main database".

 
2546 code fixed 2007 Jul anonymous   2007 Oct   4 1 Comments are not parsed correctly edit
When I try to insert a new record in a table with the .read command , I cannot load data whose line terminate with a comment. I always get an "Incomplete SQL" error.

file content: INSERT INTO _users_parameters VALUES (1,1); -- temperature

sqlite> .read smartdas.sql Incomplete SQL: INSERT INTO _users_parameters VALUES (1,3); -- temperature

file content: INSERT INTO _users_parameters VALUES (1,1); /* temperature*/

sqlite> .read smartdas.sql Incomplete SQL: INSERT INTO _users_parameters VALUES (1,3); /* temperature */

If I remove the comment, then the row is inserted as expected: file content: INSERT INTO _users_parameters VALUES (1,1);

sqlite> .read smartdas.sql sqlite> select * from _users_parameters; 1|1

Thanks, Paolo Saudin

 
2545 code active 2007 Jul anonymous   2007 Jul   1 4 Group by returns table name with field name edit
imaginate a table:
create table test (
id INTEGER PRIMARY KEY,
name varchar(50) not null,
age integer not null
);

Then:

insert into test (name,age) values ('foo',22);
insert into test (name,age) values ('foo',23);
insert into test (name,age) values ('bar',22);
insert into test (name,age) values ('bar',35);
insert into test (name,age) values ('bar',72);

Now try this;

sqlite> .headers on
sqlite> select test.name, test.age from test order by name;
name|age
bar|22
bar|35
bar|72
foo|22
foo|23
sqlite> select test.name, test.age from test group by name;
test.name|test.age
bar|72
foo|23

You see ? if i use "GROUP BY", the field name contains tablename. Because i use "SELECT test.name" and not "SELECT name".

If i set an alias, i get alias, that's ok.

The trouble appears to be very high on Copix (http://wwW.copix.org). We create some DAO (Data Access Objects) automatically. The "groupBy" method doesn't works with SQLite...

Is this normal ? Mysql, PostgreSql, Oracle... doesn't need to create alias.

2007-Jul-31 15:54:52 by anonymous:
There's probably 4 other tickets reporting this. I don't think it will get fixed. The workaround is to use aliases (AS "whatever") for the selected columns.


2007-Jul-31 23:02:19 by anonymous:
Ok, we have created a special support for SQLite.

PS: I love this database :) Simple, nice, usefull, quick and easy

Regards

 
2544 code fixed 2007 Jul rse   2007 Jul   4 3 "sqlite3 foo.db .dump | sqlite3 bar.db" fail because of verbose output edit
A simple "sqlite3 foo.db .dump | sqlite3 bar.db" for upgrading an old database file failed because of the verbose output "Loading resources from [...]/.sqliterc". Although one can use the workaround "sqlite3 -batch foo.db .dump", this is annoying as one stumbles too easily and without any real need over this unimportant verbose output message.

I propose two solutions: Either add a similar istty(3) check to shell.c which is already present for stdin or simply drop the verbose message at all. As I cannot decide what is wished here, I will append a patch at least for the istty(3) solution.

2007-Jul-30 19:02:51 by anonymous:
I agree the warning message is useless. Rather than breaking sqlite3 on platforms without isatty support, please do one of the following instead:

  • Get rid of the line altogether
  • -- put a comment in front of the line
  • if -batch is specified on the command-line, do not print out this line

A workaround:

  sqlite3 -batch -init ignore foo.db .dump | sqlite3 bar.db


2007-Jul-30 19:26:20 by rse:
The istty(3) is already used by SQLite, so the additional use would make no problem. -- rse
 
2543 code active 2007 Jul anonymous   2007 Jul   1 1 Chinese charset not support?? edit
when i create a table. the table name is " " (chinese) after this "alter table   add column aaa text null" error why??/ thank you
 
2542 code closed 2007 Jul anonymous   2007 Jul   2 1 select 3/2 * avg(x) gives wrong result edit
Try this.  It gives 2 but obviously the answer is 3 since the
average of 1,2,3 is 2 and multiplying that by 3/2 gives 3.

C:\tmp2>sqlite3
SQLite version 3.4.0
Enter ".help" for instructions
sqlite>
sqlite> create table tab (x num);
sqlite> insert into tab values (1);
sqlite> insert into tab values (2);
sqlite> insert into tab values (3);
sqlite> select * from tab;
1
2
3
sqlite> select sqlite_version(*);
3.4.0
sqlite>
sqlite> select 3/2 * avg(x) from tab;
2.0

2007-Jul-30 04:10:09 by anonymous:
Sorry, please ignore this one. I just released its doing integer arithmetic.
 
2541 doc closed 2007 Jul anonymous   2007 Jul danielk1977 4 4 ALTER TABLE <foo> RENAME TO <bar> doesn't update (all) trigger schema edit
(This might be by design, I didn't see anything in the notes to this effect though).

ALTER TABLE <foo> RENAME TO <bar> doesn't update trigger schema correctly, for example:

  sqlite> .sc
  CREATE TABLE t1 ( c1 );
  CREATE TABLE t2 ( c1 );
  CREATE TRIGGER trg_1 after INSERT ON t1 begin insert into t2 select c1   from t1 where rowid=new.rowid ; end;

then

  sqlite> alter table t1 rename to table1;
  sqlite> .sc
  [...]
  CREATE TRIGGER trg_1 after INSERT ON 'table1' begin insert into t2 select c1 from t1 where rowid=new.rowid ; end;

So you can get errors like:

  sqlite> insert into t1(c1) values('one');
  SQL error: no such table: t1
2007-Jul-28 01:00:05 by anonymous:
That's not easy to fix. SQLite stores triggers, views, etc as strings in sqlite_master - not in its tokenized form.

You basically have to drop the trigger and recreate it.


2007-Jul-28 01:31:35 by anonymous:
I think you meant to write the following command after the table name was changed:

  sqlite> insert into table1 values('one');
  SQL error: no such table: main.t1


2007-Jul-30 02:52:13 by danielk1977:
You need to edit (i.e. drop and recreate) views and statements executed by triggers externally. See the third paragraph here ("If the table being renamed..."):

   http://www.sqlite.org/lang_altertable.html
 
2540 code active 2007 Jul anonymous   2007 Jul   5 4 Display dlopen() errors for errors when loading modules on Unix system edit
If there is an error loading a module the message "unable to open shared library..." is displayed. In the Unix world the dlopen() error could be display to help diagnose the problem (e.g. missing external refs, etc). I have implemented it in our install of SQLite, I'm sure there is a Windows analog. Here is the patch for loadext.c.

- Chris - Christopher Hailey Sr. Software Engineer Maxim Systems

   ::::::::::::::
   loadext.c.patch
   ::::::::::::::
   *** ./src/loadext.c.orig        2007-07-22 00:11:35.000000000 -0700
   --- ./src/loadext.c     2007-07-22 00:12:07.000000000 -0700
   ***************
   *** 292,298 ****
       handle = sqlite3OsDlopen(zFile);
       if( handle==0 ){
         if( pzErrMsg ){
   !       *pzErrMsg = sqlite3_mprintf("unable to open shared library [%s]", zFile);
         }
         return SQLITE_ERROR;
       }
   --- 292,298 ----
       handle = sqlite3OsDlopen(zFile);
       if( handle==0 ){
         if( pzErrMsg ){
   !       *pzErrMsg = sqlite3_mprintf("unable to open shared library [%s]: %s", zFile,dlerror());
         }
         return SQLITE_ERROR;
 
2539 code active 2007 Jul anonymous   2007 Sep   2 2 WinCE: Temporary etilqs_ files are not removed from temporary folder edit
Hi, when temporary etilqs_* files are created during SQLite work on Windows CE devices, they are not removed at all.

Temporary folder at CE devices: /Application Data/Volatile

I've research that it winClose(os_win.c) function has been changed at do not remove this file, assuming it to be removed at winceDestroyLock(os_win.c), so if no lock was happened then files will stay here forever.

Has fixed it in my local copy, with hope that it will be fixed when new cool versions of SQLite will be available.

My fix at os_win.c:

static int winClose(OsFile **pId){
  winFile *pFile;
  int rc = 1;
  if( pId && (pFile = (winFile*)*pId)!=0 ){
    int rc, cnt = 0;
    OSTRACE2("CLOSE %d\n", pFile->h);
    do{
      rc = CloseHandle(pFile->h);
    }while( rc==0 && cnt++ < MX_CLOSE_ATTEMPT && (Sleep(100), 1) );
#if OS_WINCE
    winceDestroyLock(pFile);
// fix begin
	if( pFile->zDeleteOnClose ){
		DeleteFileW(pFile->zDeleteOnClose);
		sqliteFree(pFile->zDeleteOnClose);
	}
// fix end
#endif
    OpenCounter(-1);
    sqliteFree(pFile);
    *pId = 0;
  }
  return rc ? SQLITE_OK : SQLITE_IOERR;
}
Thanks,

Fedor

2007-Jul-28 16:41:41 by anonymous:
The solution is to revert checkin 3836 and re-open ticket #2294.

Looking at the wince locking mechanism, the only time we ever use the zDeleteOnClose flag is when we've opened a database for exclusive access in sqlite3WinOpenExclusive. To save time and resources (and because its not necessary) we never bother creating a locking mechanism for exclusively-opened files. So pFile->hMutex is NULL when hitting winceDestroyLock(), and the file is never deleted.

Is it possible that the original poster of #2294 was trying to close the same connection on multiple threads at the same time?


2007-Jul-31 05:32:39 by anonymous:
This is actually a duplicate of #2533


2007-Sep-21 14:20:05 by anonymous:
So when the fix of [3836] was applied, the code to delete the file was only put in the section that is called when we have a mutex. I wonder, if the deletion of the file should also take place if there was no mutex. Works for me at least:

	static void winceDestroyLock(winFile *pFile){
	  if (pFile->hMutex){
	    /* Acquire the mutex */
	    winceMutexAcquire(pFile->hMutex);

	    /* The following blocks should probably assert in debug mode, but they
	       are to cleanup in case any locks remained open */
 	   if (pFile->local.nReaders){
 	     pFile->shared->nReaders --;
 	   }
 	   if (pFile->local.bReserved){
 	     pFile->shared->bReserved = FALSE;
 	   }
 	   if (pFile->local.bPending){
 	     pFile->shared->bPending = FALSE;
 	   }
 	   if (pFile->local.bExclusive){
 	     pFile->shared->bExclusive = FALSE;
 	   }

	    /* De-reference and close our copy of the shared memory handle */
	    UnmapViewOfFile(pFile->shared);
	    CloseHandle(pFile->hShared);

	*    if( pFile->zDeleteOnClose ){
 	*     DeleteFileW(pFile->zDeleteOnClose);
 	*     sqliteFree(pFile->zDeleteOnClose);
 	*     pFile->zDeleteOnClose = 0;
 	*   }

	    /* Done with the mutex */
	    winceMutexRelease(pFile->hMutex);
	    CloseHandle(pFile->hMutex);
	    pFile->hMutex = NULL;
	  }
	+ else
	+ {
	+   if( pFile->zDeleteOnClose ){
	+      DeleteFileW(pFile->zDeleteOnClose);
	+      sqliteFree(pFile->zDeleteOnClose);
	+      pFile->zDeleteOnClose = 0;
	+    }
	+  }
	}

The code marked with * was put there in

 
2538 code closed 2007 Jul anonymous   2007 Jul   1 1 calling sqlite3_interrupt() randomly leaves db->ul.isInterrupted = 1 edit
Using a sequence as follows, all invoked with sqlite3_exec():

  BEGIN;
  ... long running script with many UPDATE/INSERT statements (with triggers firing, etc)...
  sqlite3_interrupt() is called

Often (always?) the next sqlite3_exec call will also return SQLITE_INTERRUPT instead of SQLITE_OK. Apparently there is at least one point in sqlite execution that can be interrupted with sqlite3_interrupt() where db->ul.isInterrupted is not reset to 0, so the next sqlite3_exec() call also to returns SQLITE_INTERRUPT.

Since the documentation states that calling sqlite3_interrupt() will rollback any EXPLICIT transactions (why specifically EXPLICIT ?? - I would assume auto [IMPLICIT] transactions would obviously also rollback with a sqlite3_interrupt call), any subsequent action done on the sqlite instance should be SQLITE_OK (or the error specific to that action since the previous call has been fully aborted).

This behavior is definitely different than 3.2.8 (the version I am attempting to upgrade from), which works as expected.

After much additional testing, I have concluded that this may actually have been due to my error (sqlite3_interrupt() being called again) and not due to sqlite. Sorry for the wasted bandwidth. Please consider the issue closed.
 
2537 code closed 2007 Jul anonymous   2007 Jul   4 4 sqlite3_column_type() function problem. edit
Hi,

-I prepare a sql query statement ex: Select * from Table1. -I want read some metadata from all fields of the result-set before fetch the first row with the function sqlite3_step(...). -sqlite3_column_name(...), sqlite3_column_origin_name(...), sqlite3_column_table_name(...) are all Ok but sqlite3_column_type(...) returns for all columns -> 5 (NULL). After the first sqlite3_step(...) the type is correct for all columns.

2007-Jul-25 23:07:25 by drh:
The type can vary from one row to the next.

Works as designed.

 
2536 code closed 2007 Jul anonymous   2007 Aug   2 3 Possible non-ID index corruption when max_page_count exceeded edit
Using 3.4.0 or 3.4.1 compiled for windows with -DMULTITHREAD, i seem to have a problem where sqlite3_step() will return SQLITE_ROW even when there are, in fact, no rows that meet the WHERE clause criteria. I'm currently attempting a reversion to 3.3.17 to see how it behaved.

    create table file_data
    ( rod blob not null primary key /* always 20 bytes long based on a collision-resistant hash function */
    , file_size integer not null
    , compression_method integer not null
    , the_data blob not null
    );

    sqlite3_prepare(..., "select file_size, compression_method from file_data where rod = ?");
    sqlite3_reset(statement);
    sqlite3_bind_blob(...,0, ptr, len);
    sqlite3_step(...)

sqlite3_step returned SQLITE_ROW, which was my indication that the result set would be non-empty. So I construct a query_result(statement, bool at_end=false), and then execute

    sqlite3_get_int64(..., first_column, null=-1)

and I get back -1. Before I was passing in null=-1 I was passing in =0, which was a permissible value for file_size.

** keep in mind that my exact sqlite3_api calls might be slightly different, since I have a C++ framework that isolates me from having to continually remember the same API calls. Suffice it to say this code worked (so far as I knew) under 3.3.17.

--andy

On further research, it appears to be related to max_page_count being exceeded during insertion of a row, leaving behind a primary key reference in the (primary key) index on the BLOB instead of being cleaned up during rollback.

2007-Jul-25 17:38:38 by drh:
Unable to reproduce.

I am unable to make a lot of sense of the description above. I have tried various tests setting a low max_page_count and inserting BLOB-keyed entries into a table until the number of pages overflow, then run sqlite3_step to see what happens. Everything works correctly. I tried lots of different combinations but never could get a failure.


2007-Jul-25 19:17:19 by anonymous:
MULTITHREAD is the wrong macro, but sqlite compilation defaults to threadsafe anyway.

How many threads are you running when this thing happens and what's your relationship of sqlite connections to threads?

 
2534 code closed 2007 Jul anonymous   2007 Jul danielk1977 2 3 sqlite3_set_auxdata may cause memory to be leaked edit
The routine sqlite3_set_auxdata() may cause memory allocated by the caller to be leaked.

Specifically, if the routine fails to allocate memory for its internal array of struct AuxData then the new argument data will not be stored. If the caller included an xDelete callback, this callback will not be stored or called in this circumstance.

Given that sqlite3_set_auxdata() does not return an indicator of success or failure, if the routine fails and a xDelete callback has been supplied then the callback routine should be executed before returning to the caller.

Makes sense to me.
 
2533 code fixed 2007 Jul anonymous   2007 Oct   1 2 temporary files are not deleted on WinCe edit
temporary files are not deleted on WinCe if they are not locked.

hMutex is false in struct pFile.

function: static void winceDestroyLock(winFile *pFile)

line: 16490 in sqlite3.c from sqlite-amalgamation-3_4_1.zip

problem: delete temporary files only if pFile->hMutex is true

           if (pFile->hMutex){
             ...
             if( pFile->zDeleteOnClose ){
               DeleteFileW(pFile->zDeleteOnClose);
               sqliteFree(pFile->zDeleteOnClose);
               pFile->zDeleteOnClose = 0;
             }
             ...
           }

solution: delete temporary files not dependend if pFile->Mutex is true or not

           if (pFile->hMutex){
             ...
             ...
           }

           if( pFile->zDeleteOnClose ){
             DeleteFileW(pFile->zDeleteOnClose);
             sqliteFree(pFile->zDeleteOnClose);
             pFile->zDeleteOnClose = 0;
           }

regards

Juergen Wolters

Temporary files never create a lock object, so the hMutex member is always null. Since there is no mutex to cleanup, winceDestroyLock() is never called and temporary files are never deleted.

Checkin [3836] needs to be reverted and the original ticket which triggered it re-opened.

 
2532 code closed 2007 Jul anonymous   2007 Jul   3 4 Endless loop in Parser? edit
After pasting the statement: " create trigger brandattribute_adtr after delete on brand_attribute for each row begin delete from customattributetype where customattributetype.customattributetype_id == old.customattributetype_id end; " the Sqlite command prompt remains in ...> level. I don't know any solution to solve it except typing <Ctrl>-C. It seems to be an endless loop in the parser.
You need a semi-colon after the DELETE statement. Before "end;".
 
2531 code closed 2007 Jul anonymous   2007 Jul a.rottmann 1 1 Can't find head files in icu.c edit
/* Include ICU headers */ #include <unicode/utypes.h> #include <unicode/uregex.h> #include <unicode/ustring.h> #include <unicode/ucol.h>

where can I find them?

thanks.

2007-Jul-25 05:14:37 by danielk1977:
They are part of the ICU library distribution. You need ICU installed to build the sqlite ICU extension.

  http://icu-project.org/
 
2530 code active 2007 Jul anonymous   2007 Jul   2 3 Unable to write to windows share, even with exclusive lock edit
It has been mentioned that the file locking does not work on windows shared network drives (Samba or SMB drives from Windows or Linux). It seems that an exclusive lock should be a workaround for this problem if you need to write to a shared drive. Currently a more complicated locking is being attempted and failing on network drives.

With an exclusive lock, SQLite could resort to simply holding a open write or append enabled file handle to the database as a more primitive locking system that is more likely to work on network drive. No other process could open the database but that would be expected with an exclusive lock. The following case should then function:

  grudy@gamma:~$ mount | grep Files
  //winserver/FileDump on /mnt/Files type cifs (rw,mand,noexec,nosuid,nodev)
  grudy@gamma:~$ touch /mnt/Files/i_have_write_permissions.txt; rm /mnt/Files/i_have_write_permissions.txt
  grudy@gamma:~$ sqlite3 /mnt/Files/foo.sqlite
  SQLite version 3.3.17
  Enter ".help" for instructions
  sqlite> PRAGMA locking_mode = EXCLUSIVE;
  exclusive
  sqlite> create table bar (foobar);
  SQL error: database is locked
  sqlite>
 
2529 code closed 2007 Jul anonymous   2007 Jul anonymous 1 1 Bug in sqlite3_prepare_v2( ) function edit
In Win 32 console All queries are working which include "SELECT * FROM A WHERE col1 NOT IN ( -1, 6);" "SELECT * FROM A WHERE col2 IN ( 0, 1, 2, 3 );"

where A is the table name, col1 is the column label, table values are entered correctly ( checked and verified )

These queries are working perfectly in sqlite3 command prompt and Win32 VC++ console

But are not working on Qualcomm chipsets, with REX OS The position where it is returning error is sqlite3_prepare_v2( )

2007-Jul-24 10:20:40 by drh:
OK, Vineethev, let's think this through. You admit that everything is working prefectly on Win32 using the sqlite3 command prompt and Win32 VC++ console. You are reporting against SQLite version 3.3.13 which has been used by thousands of thousands of people on diverse systems without problems.

You further observe that the queries do not work on Qualcomm chipsets on the REX operating system (which is an OS I have never before heard of and is certainly not directly supported by the SQLite core team.)

In other words, everything appears to be working fine everywhere except on Qualcomm chips and REX OS?

So what is it that make you think this is a bug in SQLite in the sqlite3_prepare_v2() function and not a bug in either the Qualcomm chips or REX OS or your proprietary port of SQLite to that platform?

I am not making fun of you or dismissing your complaint. My question is serious. Why do you think this is a bug in SQLite and not someplace else? Think carefully about that question and answer it as fully and carefully as you can. Because unless you answer that question, there is very little that I or anyone else can do to help you.

 
2528 doc fixed 2007 Jul anonymous   2007 Jul   3 2 PRAGMA auto_vacuum documentation mismatch edit
Pragma documentation claims that
PRAGMA auto_vacuum=incremental
setting is not persistent.
Changes page claims the opposite.
 
2527 doc closed 2007 Jul anonymous   2007 Jul   4 2 sqlite3_create_module() API documentation missing edit
Documentation for sqlite3_create_module() is missing on page C/C++ API Reference. The same goes for new sqlite3_create_module_v2().
2007-Jul-24 10:06:33 by drh:
This is because the whole virtual table mechanism is still considered highly experimental and subject to change. By writing up an interface in the C-API documentation, we promise to support it from now on. No such promises are made with virtual tables.
 
2526 code fixed 2007 Jul anonymous   2007 Jul   2 4 Core dump when issuing a select statement with special having clause edit
When executing the attached script, I get a core dump regarding freeing an invalid pointer.

Steps to reproduce: python breaks.py # breaks.py is found below

Comments: The error is probably related to the fact that having a having clause with "having cn < max(count(author))" breaks with "missuse of aggregate function count()", but aliasing the column count(author as cn) breaks the checks, and instead the program crashes.

I submitted the bug here instead of to pysqlite because the breakage also happens with the command line version of sqlite3.

2007-Jul-23 22:14:11 by drh:
The following script demonstrates the problem. No need for the python complication:

    create table t1 (a text, b text, c primary key);
    insert into t1 values ('x', 'y', NULL);
    insert into t1 values ('x', 'z', NULL);
    select count(a) as cn from t1 group by a having cn < max(cn);
 
2525 code fixed 2007 Jul anonymous   2007 Jul   1 3 Constraint violations fail silently with "insert or replace" edit
The following code (file "correls.sql") correctly raises a constraint violation:

  create table if not exists correls (
    sym1 varchar, sym2 varchar, correl real,
    primary key (sym1, sym2),
    check (correl >= -1), check(correl <= 1));
  insert into correls values ("a", "b", -2);

Here is the output:

  # sqlite3 correls.db < correls.sql
  SQL error near line 5: constraint failed

But running the same code with "insert or replace" instead of a plain "insert" will fail silently. The data won't be inserted or updated so integrity is still maintained, but the user will not know that the transaction wasn't successful.

 
2524 code fixed 2007 Jul anonymous   2007 Jul   4 4 Missing error check in call to sqlite3PagerWrite() in autoVacuumCommit edit
The Coverity checker found this minor issue. Most of the calls to sqlite3PagerWrite() have the return value checked and appropriate action taken if the call fails. In btree.c, line 2012, there's a call to sqlite3PagerWrite() without a check. Is this a case where the call can't fail, or are we really missing error handling code here?
 
2523 code fixed 2007 Jul anonymous   2007 Jul   3 4 Leaked memory on error return from sqlite3VdbeMemTranslate edit
In sqlite3VdbeMemTranslate in utf.c, line 214 has an allocation of memory for zOut. Later, at line 265, there's an early return if memory allocation fails for zExtra. If that return is taken, the allocation for zOut is lost, since it's not been assigned to an outbound variable.

This does seem like a rare case, but the fix is easy -- just add the right deallocation here if zOut != zShort.

 
2522 code fixed 2007 Jul anonymous   2007 Jul   4 4 wrong sizeof in vdbe.c edit
Harmless, but this can waste memory on platforms where sizeof(int*) > sizeof(int).

vdbe.c:

  -  aRoot = sqliteMallocRaw( sizeof(int*)*(nRoot+1) );
  +  aRoot = sqliteMallocRaw( sizeof(int)*(nRoot+1) );
 
2521 code fixed 2007 Jul anonymous   2007 Jul   2 4 Misplaced prototypes edit
When compiling the amalgamated sqlite3.c, the Digital Mars compiler produces the error

sqlite3.c(67876) : Error: no definition for static 'sqlite3ParserFree'

Moving the function prototypes at sqlite3.c lines 66302 and 66303

SQLITE_PRIVATE void sqlite3ParserAlloc(void*(*)(size_t)); SQLITE_PRIVATE void sqlite3ParserFree(void, void(*)(void*));

to a position before the respective procedures eliminates the problem.

Robert Wishlaw

 
2520 new active 2007 Jul anonymous   2007 Jul   4 1 User defined aggregate functions are not reentrant edit
When an aggregate function is defined using sqlite3_create_function, it is not possible to execute any sql statement inside the step part or the finalizer. This is due to the fact that aggregate functions are not reentrant.
2007-Jul-20 02:32:36 by anonymous:
related: Ticket #2242: sqlite3 authorizer is not reentrant
 
2519 new closed 2007 Jul anonymous   2007 Oct   1 1 Page cache and schema should be shared between all connections/threads edit
The current shared cache scheme is too restrictive to use easily in a multi-threaded application. The page cache should be shared between all connections/threads, as the database file itself is a global resource. If the schema is changed in one thread connection, all other thread connections should pick up the change automatically. Mutexes should be used for coordination as appropriate.

Advantages of thread-agnostic shared cache/schema approach:

  • reduced memory use due to cache sharing in a multi-threaded/multi-connection program
  • simpler usage model for multiple connections so that the programmer need not be concerned with what thread created the connection, or using starting a dummy transaction with an insert into a dummy table in order to keep the cache between transactions
  • eliminate the schema/thread/connection mismatch problem #2486
  • fewer system resources - use just a single file descriptor for multiple connections to the same database file, since it is a global resource. This would avoid file descriptor problems related to the Linux 2.4 kernel.
  • improved concurrency for connections in multiple threads
2007-Jul-26 18:39:20 by anonymous:
the database file is a global resource, the sqlite handle is not.

imagine if you start a transaction in a thread then another thread start another transaction... if you use two sqlite handles, this is ok, because you can lock a file handle for the transactions.

since sqlite doesn't provide nested transactions, you should avoid thinking a sqlite handle object is a global resource. this is explicit well documented.


2007-Jul-26 19:49:57 by anonymous:
That's the problem with the current implementation. There is no need for multiple file descriptors to the same database file, since it is a global resource. There should be only a single file handle shared between all threads and connections. Access to this common file handle should be coordinated via proper mutexes between threads.


2007-Jul-26 20:11:09 by anonymous:
right again, neither ADO / SQL Server provider in ADO / Microsoft world suport a shared handle (connection object) properly. nobody implements this just because tx control should be done by program, not by database layer driver.


2007-Jul-26 20:19:24 by anonymous:
Whether there is 1 connection or 50 connections to the same database, the database file page cache, a global resource, must be the same in all cases. Don't confuse the page cache with a journal file.


2007-Jul-26 20:45:01 by anonymous:
suppose you have a main program and a shared library. the two linked static against a sqlite amalgamation, with different versions (e.g. 3.3.14 main program and 3.4.1 shared library).

if the pager cache internal structures changed against versions (which could happen), sqlite is going to be crazy and doing worst.

also, not all os platforms works like other in memory management. they could have different memory virtual address (in windows, dll memory is protected from program memory if it uses the malloc allocator, with globalalloc things performs differently). since sqlite is multi-platform, it's not cool to put a per-platform feature.

a 'possible' (?) workaround is to use a per-process shared memory to implement this, but not all os'es sqlite compile and run may support it to implement this design.


2007-Jul-26 20:53:27 by anonymous:
The proposed in-process shared database page cache has nothing to do with virtual shared memory or shared libraries.

If the in-memory database change counter matches that of the locked database file, we can assume that the in-memory database page cache is current - regardless of whatever sqlite3 connection or thread we're on.


2007-Aug-10 20:33:49 by anonymous:
using the same file descriptor should not work. locking engine in sqlite is file-locked base. testing in a process within various threads if you could lock a region of the file will not work if sqlite is changed this way.

again, not usual and you should realize you can use various sqlite versions inside the same program using amalgamations.


2007-Aug-10 21:25:12 by anonymous:
Of course using a single file descriptor would work. There is only 1 underlying database file - why on earth do you need more than 1 descriptor to access it? That's what mutexes are for. Regarding putting multiple sqlite3 engines in the same binary, no one has ever done this before, as it was not even possible until 2 days ago (Ticket #2554). But if you wanted to support the old style multi-descriptor file locking, it could be done with an appropriate compile flag -DOMIT_SINGLE_FILE_DESCRIPTOR like every other feature in sqlite.


2007-Aug-10 23:36:34 by anonymous:
you didn't get the issue.

suppose (in my case), I have two DLL&#180;s (shared libraries) that are linked statically against sqlite.

if I build a updated DLL with different versions of SQLite, e.g., 3.2.5 (on old dll&#180;s) and 3.4.1 (on the updated dll), sqlite in-memory structures would not be the same layout (new fields, features, etc).

this is very dangerous in my point of view. I preffer paying the penalty reading from disk and keep things going smoothly.


2007-Aug-11 00:46:16 by anonymous:
Okay, keep it running smoothly.


2007-Oct-04 01:02:32 by anonymous:
Must manually enable shared cache via:

http://www.sqlite.org/capi3ref.html#sqlite3_enable_shared_cache

 
2518 code fixed 2007 Jul anonymous   2007 Jul   2 1 Complex paging problem edit
We are experiencing a very strange and complex problem with SQLite which is very difficult to reproduce.

Basically

1. We run some bits to set up the DB.

2. Populate the data base with a load of information on one thread.

3. Then running a query on a different thread:

  SELECT DISTINCT("SessionID")
    FROM "mfForm_Welcome"
   WHERE UID IN (-1, 1)
     AND "D3_TXTCODE" LIKE "1"
   ORDER BY SessionID DESC;

If the query fails in other words there is no matching data in the table the problem happens if it succeeds the problem doesn't seem to happen.

4. Then we run the function to populating the DB with a load of information. Again on a different thread. As 2.

5. Then running the query again which should succeed fails.

  SELECT DISTINCT("SessionID")
    FROM "mfForm_Welcome"
   WHERE UID IN (-1, 1)
     AND "D3_TXTCODE" LIKE "1234"
   ORDER BY SessionID DESC;

However if we do this:

  INSERT INTO mfForm_Welcome (SessionID) VALUES (NULL);
  DELETE FROM mfForm_Welcome WHERE SessionID IS NULL;

Immediately prior to running the query for the second time it succeeds.

If we run a query that succeeds at step 2. then everything seem to work ok from then on.

It also seems to matter if there are other tables in the DB prior to the one the query is run on.

I have been debugging in SQLite and looks to me that the paging system hasn't re-got the data or is in some other way incorrect. In other words the paging stuff isn't reflecting what's on the DB, Possibly it isn't getting invalidated or something.

I also got a trace of all the SQL run by the app up to this point. Running this SQL in isolation works OK (Doesn&#8217;t give the problem) so probably something to do with having two threads.

Sorry if this is a bit hard to follow but we would much appreciate help.

SQL Trace:

Step 1. This is on thread 1

CREATE TABLE "mfSysUsers" ("ID" numeric PRIMARY KEY , "Username" text (20) , "Password" text (20) )
CREATE TABLE "mfSysUserPermissions" ("UserID" numeric PRIMARY KEY , "StartUpForm" text (128) , "Flags" numeric )
CREATE TABLE "mfSysUsersToForms" ("PermissionID" numeric PRIMARY KEY , "UserID" numeric , "FormTitle" text (128) , "Flags" numeric )
CREATE TABLE mfSysSyncLog (ID integer IDENTITY PRIMARY KEY, UID integer, Sync nvarchar(19), Type integer, Records integer, FormName nvarchar(32), FailureReason integer)
CREATE TABLE mfSysTrackingLog (ID integer IDENTITY PRIMARY KEY, UID integer, LogIn nvarchar(19), LogOut nvarchar(19), FormName nvarchar(32), Session integer)
CREATE TABLE "mfForm_Welcome" ("SessionID" numeric PRIMARY KEY , "UID" numeric , "Flags" numeric , "Expiry" numeric , "Spare" numeric , "D3_txtAccessCode" text , "D3_txtMsgDay" text , "D3_txtUser" text , "D3_txtDesc" text , "D3_txtH" text , "D3_txtErr" text , "D3_txtDontSend" text , "D3_DEVICE" text , "D3_txtCode" text )
CREATE TABLE "mFSysVersion" (asdasd CHAR)
CREATE INDEX 'IndexUserName_mfSysUsers' ON 'mfSysUsers' (Username ASC )
CREATE INDEX 'IndexUserID_mfSysUsers' ON 'mfSysUsers' (ID DESC )
CREATE INDEX [IdxTrackingLog] ON [mfSysTrackingLog] (ID)

SELECT Username FROM mfSysUsers;
SELECT Username FROM mfSysUsers;
SELECT Username FROM mfSysUsers WHERE Username='Guest'
SELECT Username FROM mfSysUsers WHERE Username='Guest'
SELECT Password FROM mfSysUsers WHERE Username='Guest'
SELECT Password FROM mfSysUsers WHERE Username='Guest'
SELECT ID FROM mfSysUsers WHERE Username='Guest'
SELECT ID FROM mfSysUsers WHERE Username='Guest'
SELECT StartUpForm, Flags FROM mfSysUserPermissions WHERE UserID = 1;
SELECT StartUpForm, Flags FROM mfSysUserPermissions WHERE UserID = 1;

CREATE TABLE "mfForm_Welcome" ("SessionID" numeric PRIMARY KEY , "UID" numeric , "Flags" numeric , "Expiry" numeric , "Spare" numeric , "D3_txtAccessCode" text , "D3_txtMsgDay" text , "D3_txtUser" text , "D3_txtDesc" text , "D3_txtH" text , "D3_txtErr" text , "D3_txtDontSend" text , "D3_DEVICE" text , "D3_txtCode" text );
SELECT MAX(SessionID) FROM "mfForm_Welcome";
SELECT MAX(SessionID) FROM "mfForm_Welcome";
INSERT INTO mfSysTrackingLog (UID, Login, Session, FormName) VALUES (1, '2007-07-17 11:39:25', 1, 'Welcome');
SELECT MAX(ID) FROM mfSysTrackingLog;
SELECT MAX(ID) FROM mfSysTrackingLog;
Step 2. This is on thread 2

BEGIN;
SELECT SessionID FROM "mfForm_PartFields" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_PartFields" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_testaa" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_testaa" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_testbb" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_testbb" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Issue Photos" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Issue Photos" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Inspections Guide" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Inspections Guide" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Street Inspections" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Street Inspections" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Street Issues" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Street Issues" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Street Inspections List" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Street Inspections List" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Inspection Photos" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Inspection Photos" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Menu" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Menu" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_IssuePCN" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_IssuePCN" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_IssuePCNStartUp" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_IssuePCNStartUp" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_PCNNoGenerator" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_PCNNoGenerator" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_PCNShift" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_PCNShift" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_DogFouling" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_DogFouling" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_OffenceList" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_OffenceList" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_PCNShiftDetails" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_PCNShiftDetails" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_IssuePCNMain" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_IssuePCNMain" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_AuditsToClaim" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_AuditsToClaim" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Check Item" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Check Item" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Audit" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Audit" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_AuditClaim" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_AuditClaim" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_AIMS" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_AIMS" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Main" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Main" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_CheckItemActions" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_CheckItemActions" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_HRA" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_HRA" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_TEST LISTS2" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_TEST LISTS2" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_OrderList_OL" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_OrderList_OL" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Statement_ST" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Statement_ST" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_LastOrderList" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_LastOrderList" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_StatementData_SD" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_StatementData_SD" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Receipt_R" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Receipt_R" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_ReceiptReport_RR" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_ReceiptReport_RR" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Form 1aa" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Form 1aa" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_ReceiptReportData_RRD" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_ReceiptReportData_RRD" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_testpush" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_testpush" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_pushlist" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_pushlist" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_eeeee" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_eeeee" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_PushTestData" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_PushTestData" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_PushTestUpdate" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_PushTestUpdate" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_JobDet" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_JobDet" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_userdet" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_userdet" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_VisitDetails" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_VisitDetails" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Welcome" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Welcome" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Photo" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Photo" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_JobFields" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_JobFields" WHERE Flags IN (2,3) GROUP BY SessionID;
DROP TABLE 'mfForm_TEST LISTS';
DROP TABLE 'mfForm_Debug';
DROP TABLE 'mfForm_asfsd';
DROP TABLE 'mfForm_Street Issues List';
DROP TABLE 'mfForm_Check Groups';
DROP TABLE 'mfForm_OrderEntry_OE';
DROP TABLE 'mfForm_PCNList';
DROP TABLE 'mfForm_ProductEntry_PE';
DROP TABLE 'mfForm_PushTest';
DROP TABLE 'mfForm_NullTest';
DROP TABLE 'mfForm_Order details';
DROP TABLE 'mfForm_test1';
DROP TABLE 'mfForm_Form 1';
DROP TABLE 'mfForm_Form 3';
DROP TABLE 'mfForm_Form 2';
DROP TABLE 'mfForm_Options';
DROP TABLE 'mfForm_PartDetails';
DROP TABLE 'mfForm_Jobs';
DROP TABLE 'mfForm_Confirm';
DROP TABLE 'mfForm_Unfinished';
DROP TABLE 'mfForm_Errors';
DROP TABLE 'mfForm_JobOptions';
SELECT * FROM "mfSysSyncLog";
SELECT * FROM "mfSysSyncLog";
SELECT * FROM "mfSysTrackingLog";
SELECT * FROM "mfSysTrackingLog";
DELETE FROM mfSysTrackingLog;
COMMIT;
VACUUM;
Step 3. This is on Thread 1

SELECT DISTINCT("SessionID") FROM "mfForm_Welcome" WHERE UID IN (-1, 1) AND "D3_TXTCODE" LIKE "1" ORDER BY SessionID DESC;
Step 4. This is on thread 2

BEGIN;
SELECT SessionID FROM "mfForm_PartFields" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT name, rootpage, sql FROM 'main'.sqlite_master
CREATE TABLE "mfSysUsers" ("ID" numeric PRIMARY KEY , "Username" text (20) , "Password" text (20) )
CREATE TABLE "mfSysUserPermissions" ("UserID" numeric PRIMARY KEY , "StartUpForm" text (128) , "Flags" numeric )
CREATE TABLE "mfSysUsersToForms" ("PermissionID" numeric PRIMARY KEY , "UserID" numeric , "FormTitle" text (128) , "Flags" numeric )
CREATE TABLE mfSysSyncLog (ID integer IDENTITY PRIMARY KEY, UID integer, Sync nvarchar(19), Type integer, Records integer, FormName nvarchar(32), FailureReason integer)
CREATE TABLE mfSysTrackingLog (ID integer IDENTITY PRIMARY KEY, UID integer, LogIn nvarchar(19), LogOut nvarchar(19), FormName nvarchar(32), Session integer)
CREATE TABLE "mfForm_Welcome" ("SessionID" numeric PRIMARY KEY , "UID" numeric , "Flags" numeric , "Expiry" numeric , "Spare" numeric , "D3_txtAccessCode" text , "D3_txtMsgDay" text , "D3_txtUser" text , "D3_txtDesc" text , "D3_txtH" text , "D3_txtErr" text , "D3_txtDontSend" text , "D3_DEVICE" text , "D3_txtCode" text )
CREATE TABLE "mFSysVersion" (asdasd CHAR)
CREATE INDEX 'IndexUserName_mfSysUsers' ON 'mfSysUsers' (Username ASC )
CREATE INDEX 'IndexUserID_mfSysUsers' ON 'mfSysUsers' (ID DESC )
CREATE INDEX [IdxTrackingLog] ON [mfSysTrackingLog] (ID)
SELECT SessionID FROM "mfForm_PartFields" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_testaa" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_testaa" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_testbb" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_testbb" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Issue Photos" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Issue Photos" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Inspections Guide" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Inspections Guide" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Street Inspections" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Street Inspections" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Street Issues" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Street Issues" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Street Inspections List" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Street Inspections List" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Inspection Photos" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Inspection Photos" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Menu" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Menu" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_IssuePCN" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_IssuePCN" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_IssuePCNStartUp" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_IssuePCNStartUp" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_PCNNoGenerator" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_PCNNoGenerator" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_PCNShift" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_PCNShift" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_DogFouling" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_DogFouling" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_OffenceList" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_OffenceList" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_PCNShiftDetails" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_PCNShiftDetails" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_IssuePCNMain" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_IssuePCNMain" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_AuditsToClaim" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_AuditsToClaim" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Check Item" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Check Item" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Audit" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Audit" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_AuditClaim" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_AuditClaim" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_AIMS" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_AIMS" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Main" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Main" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_CheckItemActions" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_CheckItemActions" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_HRA" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_HRA" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_TEST LISTS2" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_TEST LISTS2" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_OrderList_OL" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_OrderList_OL" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Statement_ST" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Statement_ST" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_LastOrderList" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_LastOrderList" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_StatementData_SD" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_StatementData_SD" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Receipt_R" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Receipt_R" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_ReceiptReport_RR" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_ReceiptReport_RR" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Form 1aa" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Form 1aa" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_ReceiptReportData_RRD" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_ReceiptReportData_RRD" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_testpush" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_testpush" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_pushlist" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_pushlist" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_eeeee" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_eeeee" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_PushTestData" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_PushTestData" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_PushTestUpdate" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_PushTestUpdate" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_JobDet" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_JobDet" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_userdet" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_userdet" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_VisitDetails" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_VisitDetails" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Welcome" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Welcome" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Photo" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_Photo" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_JobFields" WHERE Flags IN (2,3) GROUP BY SessionID;
SELECT SessionID FROM "mfForm_JobFields" WHERE Flags IN (2,3) GROUP BY SessionID;
DROP TABLE 'mfForm_TEST LISTS';
DROP TABLE 'mfForm_Debug';
DROP TABLE 'mfForm_asfsd';
DROP TABLE 'mfForm_Street Issues List';
DROP TABLE 'mfForm_Check Groups';
DROP TABLE 'mfForm_OrderEntry_OE';
DROP TABLE 'mfForm_PCNList';
DROP TABLE 'mfForm_ProductEntry_PE';
DROP TABLE 'mfForm_PushTest';
DROP TABLE 'mfForm_NullTest';
DROP TABLE 'mfForm_Order details';
DROP TABLE 'mfForm_test1';
DROP TABLE 'mfForm_Form 1';
DROP TABLE 'mfForm_Form 3';
DROP TABLE 'mfForm_Form 2';
DROP TABLE 'mfForm_Options';
DROP TABLE 'mfForm_PartDetails';
DROP TABLE 'mfForm_Jobs';
DROP TABLE 'mfForm_Confirm';
DROP TABLE 'mfForm_Unfinished';
DROP TABLE 'mfForm_Errors';
DROP TABLE 'mfForm_JobOptions';
SELECT * FROM "mfSysSyncLog";
SELECT * FROM "mfSysSyncLog";
SELECT * FROM "mfSysTrackingLog";
SELECT * FROM "mfSysTrackingLog";
SELECT MAX(SessionID) FROM "mfForm_Welcome";
SELECT MAX(SessionID) FROM "mfForm_Welcome";
CREATE TABLE "mfForm_Welcome" ("SessionID" numeric PRIMARY KEY , "UID" numeric , "Flags" numeric , "Expiry" numeric , "Spare" numeric , "D3_txtUser" text , "D3_DEVICE" text , "D3_txtDesc" text , "D3_txtMsgDay" text , "D3_txtCode" text );
INSERT INTO "mfForm_Welcome" (SessionID, UID, Flags, Expiry, Spare, "D3_txtUser", "D3_DEVICE", "D3_txtDesc", "D3_txtMsgDay", "D3_txtCode") VALUES (1,-1,1,0,0,"1", "OEYQB-8LSG1-Y1GRW-RDG9X", "John Smith", "Welcome to the Integral mForms ExpressHaulage sample from Integral Mobile Data", "1234");
COMMIT;
VACUUM;
Step 5. This is on thread 1 and the query that fails but should work!

SELECT DISTINCT("SessionID") FROM "mfForm_Welcome" WHERE UID IN (-1, 1) AND "D3_TXTCODE" LIKE "1234" ORDER BY SessionID DESC;
2007-Jul-19 15:48:30 by anonymous:
See Ticket #2486: pragma index_list does not detect index created on 2nd connection


2007-Jul-19 16:32:00 by anonymous:
With multi-threaded programs becoming more prevalent due to multi-core chips, having one thread not know the schema of the database is becoming a problem.


2007-Jul-19 16:37:44 by drh:
This problem has nothing to do with schema loading. The error is that the change counter in the header of the database file was not being incremented following a VACUUM.


2007-Jul-20 10:48:38 by anonymous:
Many thanks that seems to have sorted it.
 
2517 code active 2007 Jul anonymous   2007 Jul dflam 1 1 exception on reading text in vista but not xp edit
My companies sqlite 3.1 db works perfectly on Win XP but when we moved to Vista (I'm using Vista 64)it is trowing an exception when I access a text field that contains this data:

'A/C Pressure Sensor, raw1 = A/C on, 0 = A/C off (A/C status determines which IACTx cell is used)'

Interestingly when I view data I've inserted using sqliteman3 it has unprintable characters added to it.

(A/C status determines which IACTx cell is used)9

If I define the field as Char[512] this artifact goes away. But reading your literature this isn't supposed to make a difference because everything is char.

I've changed the values in the error column, but the error seems to be depending on length rather than value.

Any help appreciatied!

Jim

2007-Jul-19 15:13:41 by drh:
We will be better able to help you with your problem on the SQLite mailing list. See http://www.sqlite.org/support.html for instructions on joining the mailing list.
 
2516 code closed 2007 Jul anonymous   2007 Jul   3 3 REGRESSION:sqlite3_prepare ignores nBytes parameter edit
Bug described as in Ticket #1554 reappeared in recent versions of sqlite (tested debian 3.3.8, and 3.4.0 built from release tarball).

Code snippet :

int rc = sqlite3_prepare( sqlite_
                        , "insert  into JOB_TODO values (24, 1, 4,  'C', 'p', NULL, NULL, 'respawn', '/usr/bin/moma/mbx_selection 2 /tmp/select_pipe_ED2 300 600 \0', NULL, NULL );"
                        , 151, &stmt, NULL ) ;

Fails with error

unrecognized token: "'/usr/bin/moma/mbx_selection 2 /tmp/select_pipe_ED2 300 600 "
Looking at the code, we can find the related [3040] CVS commit in prepare.c (as described in ticket #1554), but the bug still occurs
2007-Jul-19 11:11:36 by anonymous:
sqlite_prepare_v2 has the same problem


2007-Jul-19 12:12:54 by drh:
The nBytes parameter is working fine. The problem is that SQLite does not accept strings that contain embedded \0 characters. And this is not going to change.

If you want to insert a string that contains an embedded \0, use ? for the string in sqlite3_prepare() then bind the string using sqlite3_bind_text().

 
2515 code closed 2007 Jul anonymous   2007 Jul   1 1 Entered to ASCII zeros as a text string was return a single zero. edit
I have created a Schema that defines all but one column as a string. When I insert data that is defined as '00', '06', '05', etc. I receive no error for the insertion. But, when I retrieve the data it returns the data with the preceeding zero removed. This should not occur when the data is defined as a string and not defined to be of a numeric type.

This does not occur for any other type string data.

2007-Jul-18 21:47:55 by drh:
Here's what I get:

   CREATE TABLE t1(x TEXT);
   INSERT INTO t1 VALUES('00');
   SELECT * FROM t1;

   00

Unable to reproduce the problem.

 
2514 code fixed 2007 Jul anonymous   2007 Jul   3 3 max() and min() is wrong when DESC index exists edit
testcase:

  sqlite> PRAGMA legacy_file_format=0;
  sqlite> CREATE TABLE t(a);
  sqlite> INSERT INTO t VALUES(2);
  sqlite> INSERT INTO t VALUES(1);
  sqlite> CREATE INDEX t_a ON t(a DESC);
  sqlite> SELECT min(a) FROM t;
  2
  sqlite> SELECT max(a) FROM t;
  1

Proposed fix:

  --- src/select.c~	2007-06-16 00:31:50.000000000 +0900
  +++ src/select.c	2007-07-18 23:48:07.680080000 +0900
  @@ -2485,7 +2485,9 @@ static int simpleMinMaxQuery(Parse *pPar
       if( seekOp==OP_Rewind ){
         sqlite3VdbeAddOp(v, OP_Null, 0, 0);
         sqlite3VdbeAddOp(v, OP_MakeRecord, 1, 0);
  -      seekOp = OP_MoveGt;
  +      seekOp = pIdx->aSortOrder[0]==SQLITE_SO_ASC ? OP_MoveGt : OP_MoveLt;
  +    }else if( pIdx->aSortOrder[0]==SQLITE_SO_DESC ){
  +      seekOp = OP_Rewind;
       }
       sqlite3VdbeAddOp(v, seekOp, iIdx, 0);
       sqlite3VdbeAddOp(v, OP_IdxRowid, iIdx, 0);
 
2513 code closed 2007 Jul anonymous   2007 Jul   1 2 Internal compiler error tclsqlite.c edit
Using make with Windows XP with MSYS and MinGW 3.4.5 after

./configure --with-tcl=C:/dev/Tcl/lib

produces this error

./src/tclsqlite.c: In function 'Sqlite3_Init': ./src/tclsqlite.c:2421: internal compiler error: in rest_of_handle_final, at toplev.c:2067

Robert Wishlaw

2007-Jul-17 20:30:28 by drh:
Seem like an "internal compiler error" is the type of thing you should report to the compiler vendor, not to the makers of the code you are trying to compile....
 
2512 code active 2007 Jul shess   2007 Jul   1 1 FTS virtual table name quoting problem edit
All table names should be quoted in the FTS module code.

with TRACE enabled in ext/fts2/fts2.c:

sqlite> create virtual table "a b c" using fts2 (t);
FTS2 Create
FTS2 sql: CREATE TABLE main.a b c_content(c0t)
SQL error: vtable constructor failed: a b c
2007-Jul-18 06:44:21 by anonymous:
A similar problem shows if a FTS column has the same name as the FTS table:

  CREATE VIRTUAL TABLE a USING fts2 (a);

Returns "vtable constructor failed: a.".

 
2511 code active 2007 Jul anonymous   2007 Jul drh 3 2 Inconsistent Pragma output edit
Pragma output is inconsistent when setting the value. Most do not generate any output and silently set the value, while others generate a singleton row with the set value. Here is a list of pragmas that generate output while setting the values:

sqlite> PRAGMA locking_mode = NORMAL; normal sqlite> PRAGMA max_page_count = 100000; 100000

The following do not generate any output upon query: PRAGMA case_sensitive_like; PRAGMA incremental_vacuum;

Sqlite was built from almagamation using the following configuration flags: --enable-threadsafe --disable-tcl --enable-tempstore

 
2510 code active 2007 Jul anonymous   2007 Jul   1 1 Vacuum modified FTS2 rowids edit
VACUUM modifies FTS2 rowids. Here is the test:

  drop table if exists a;

  create virtual table a using fts2 (t);

  insert into a (t) values ('one');
  insert into a (t) values ('two');
  insert into a (t) values ('three');

  select rowid, * from a;

  delete from a where t = 'two';
  vacuum;

  select rowid, * from a;

Unfortunately there is no workaround since table a is auto-generated by the FTS2 module.

2007-Jul-17 14:05:58 by anonymous:
http://www.sqlite.org/cvstrac/chngview?cn=4157


2007-Jul-17 14:24:29 by anonymous:
Yes, this behavior has been recently documented, but there is no user workaround like PRIMARY KEY for FTS2 rowids. Therefore I consider this as a bug which should be fixed in fts2.c.


2007-Jul-17 14:55:57 by anonymous:
Should virtual tables be VACUUMable? What exactly is being vacuumed here - an internal table?


2007-Jul-17 16:34:55 by shess:
I agree, I think this is a bug. Rather severe, too, the entire fts system implicitely depends on rowids not changing, this means that vacuum will break fts tables (fts1 or fts2).

drop table if exists t;
create virtual table t using fts2;
insert into t (content) values ('This is a test');
insert into t (content) values ('This is a string');
insert into t (content) values ('That was a test');
insert into t (content) values ('A random string');
select content from t where t MATCH 'test';
delete from t where content = 'This is a string';
vacuum;
select content from t where t MATCH 'test';
The first select outputs 'This is a test' and 'That was a test'. The second select outputs 'This is a test', and 'A random string'.


2007-Jul-17 17:27:21 by anonymous:
This patch seems to address the FTS2 VACUUM problem and passes all fts2 tests.

It adds an INTEGER PRIMARY KEY docid column to the hidden %_content table.

Note: this new table format is not backwards compatible with existing FTS2 databases.

-Joe Wilson

Index: ext/fts2/fts2.c
===================================================================
RCS file: /sqlite/sqlite/ext/fts2/fts2.c,v
retrieving revision 1.40
diff -u -3 -p -r1.40 fts2.c
--- ext/fts2/fts2.c	2 Jul 2007 10:16:50 -0000	1.40
+++ ext/fts2/fts2.c	17 Jul 2007 17:19:49 -0000
@@ -1769,9 +1769,9 @@ typedef enum fulltext_statement {
 */
 static const char *const fulltext_zStatement[MAX_STMT] = {
   /* CONTENT_INSERT */ NULL,  /* generated in contentInsertStatement() */
-  /* CONTENT_SELECT */ "select * from %_content where rowid = ?",
+  /* CONTENT_SELECT */ "select * from %_content where docid = ?",
   /* CONTENT_UPDATE */ NULL,  /* generated in contentUpdateStatement() */
-  /* CONTENT_DELETE */ "delete from %_content where rowid = ?",
+  /* CONTENT_DELETE */ "delete from %_content where docid = ?",

   /* BLOCK_INSERT */ "insert into %_segments values (?)",
   /* BLOCK_SELECT */ "select block from %_segments where rowid = ?",
@@ -1860,14 +1860,14 @@ static struct fulltext_vtab *cursor_vtab
 static const sqlite3_module fts2Module;   /* forward declaration */

 /* Return a dynamically generated statement of the form
- *   insert into %_content (rowid, ...) values (?, ...)
+ *   insert into %_content (docid, ...) values (?, ...)
  */
 static const char *contentInsertStatement(fulltext_vtab *v){
   StringBuffer sb;
   int i;

   initStringBuffer(&sb);
-  append(&sb, "insert into %_content (rowid, ");
+  append(&sb, "insert into %_content (docid, ");
   appendList(&sb, v->nColumn, v->azContentColumn);
   append(&sb, ") values (?");
   for(i=0; i<v->nColumn; ++i)
@@ -1878,7 +1878,7 @@ static const char *contentInsertStatemen

 /* Return a dynamically generated statement of the form
  *   update %_content set [col_0] = ?, [col_1] = ?, ...
- *                    where rowid = ?
+ *                    where docid = ?
  */
 static const char *contentUpdateStatement(fulltext_vtab *v){
   StringBuffer sb;
@@ -1893,7 +1893,7 @@ static const char *contentUpdateStatemen
     append(&sb, v->azContentColumn[i]);
     append(&sb, " = ?");
   }
-  append(&sb, " where rowid = ?");
+  append(&sb, " where docid = ?");
   return stringBufferData(&sb);
 }

@@ -2027,15 +2027,15 @@ static int sql_step_leaf_statement(fullt
   return rc;
 }

-/* insert into %_content (rowid, ...) values ([rowid], [pValues]) */
-static int content_insert(fulltext_vtab *v, sqlite3_value *rowid,
+/* insert into %_content (docid, ...) values ([docid], [pValues]) */
+static int content_insert(fulltext_vtab *v, sqlite3_value *docid,
                           sqlite3_value **pValues){
   sqlite3_stmt *s;
   int i;
   int rc = sql_get_statement(v, CONTENT_INSERT_STMT, &s);
   if( rc!=SQLITE_OK ) return rc;

-  rc = sqlite3_bind_value(s, 1, rowid);
+  rc = sqlite3_bind_value(s, 1, docid);
   if( rc!=SQLITE_OK ) return rc;

   for(i=0; i<v->nColumn; ++i){
@@ -2047,7 +2047,7 @@ static int content_insert(fulltext_vtab
 }

 /* update %_content set col0 = pValues[0], col1 = pValues[1], ...
- *                  where rowid = [iRowid] */
+ *                  where docid = [iRowid] */
 static int content_update(fulltext_vtab *v, sqlite3_value **pValues,
                           sqlite_int64 iRowid){
   sqlite3_stmt *s;
@@ -2075,7 +2075,7 @@ static void freeStringArray(int nString,
   free((void *) pString);
 }

-/* select * from %_content where rowid = [iRow]
+/* select * from %_content where docid = [iRow]
  * The caller must delete the returned array and all strings in it.
  * null fields will be NULL in the returned array.
  *
@@ -2101,10 +2101,10 @@ static int content_select(fulltext_vtab

   values = (const char **) malloc(v->nColumn * sizeof(const char *));
   for(i=0; i<v->nColumn; ++i){
-    if( sqlite3_column_type(s, i)==SQLITE_NULL ){
+    if( sqlite3_column_type(s, i+1)==SQLITE_NULL ){
       values[i] = NULL;
     }else{
-      values[i] = string_dup((char*)sqlite3_column_text(s, i));
+      values[i] = string_dup((char*)sqlite3_column_text(s, i+1));
     }
   }

@@ -2120,7 +2120,7 @@ static int content_select(fulltext_vtab
   return rc;
 }

-/* delete from %_content where rowid = [iRow ] */
+/* delete from %_content where docid = [iRow ] */
 static int content_delete(fulltext_vtab *v, sqlite_int64 iRow){
   sqlite3_stmt *s;
   int rc = sql_get_statement(v, CONTENT_DELETE_STMT, &s);
@@ -2870,7 +2870,7 @@ static int fulltextCreate(sqlite3 *db, v
   if( rc!=SQLITE_OK ) return rc;

   initStringBuffer(&schema);
-  append(&schema, "CREATE TABLE %_content(");
+  append(&schema, "CREATE TABLE %_content(docid INTEGER PRIMARY KEY, ");
   appendList(&schema, spec.nColumn, spec.azContentColumn);
   append(&schema, ")");
   rc = sql_exec(db, spec.zDb, spec.zName, stringBufferData(&schema));
@@ -3731,8 +3731,8 @@ static int fulltextFilter(

   TRACE(("FTS2 Filter %p\n",pCursor));

-  zSql = sqlite3_mprintf("select rowid, * from %%_content %s",
-                          idxNum==QUERY_GENERIC ? "" : "where rowid=?");
+  zSql = sqlite3_mprintf("select * from %%_content %s",
+                          idxNum==QUERY_GENERIC ? "" : "where docid=?");
   sqlite3_finalize(c->pStmt);
   rc = sql_prepare(v->db, v->zDb, v->zName, &c->pStmt, zSql);
   sqlite3_free(zSql);

2007-Jul-18 00:13:56 by shess:
BTW, AFAICT this only happens for sqlite3.4. Older versions don't seem to have the problem.


2007-Jul-18 01:31:49 by anonymous:
The rowid changing after VACUUM predates 3.4.0...

SQLite version 3.3.7
Enter ".help" for instructions
sqlite> CREATE TABLE t(a);
sqlite> INSERT INTO "t" VALUES('one');
sqlite> INSERT INTO "t" VALUES('two');
sqlite> INSERT INTO "t" VALUES('three');
sqlite> select rowid, * from t;
1|one
2|two
3|three
sqlite> delete from t where a = 'one';
sqlite> select rowid, * from t;
2|two
3|three
sqlite> vacuum;
sqlite> select rowid, * from t;
1|two
2|three

SQLite version 3.2.0
Enter ".help" for instructions
sqlite> CREATE TABLE t(a);
sqlite> INSERT INTO "t" VALUES('one');
sqlite> INSERT INTO "t" VALUES('two');
sqlite> INSERT INTO "t" VALUES('three');
sqlite> select rowid, * from t;
1|one
2|two
3|three
sqlite> delete from t where a = 'one';
sqlite> select rowid, * from t;
2|two
3|three
sqlite> vacuum;
sqlite> select rowid, * from t;
1|two
2|three

2007-Jul-18 15:59:24 by anonymous:
As you may know, INTEGER PRIMARY KEY indexes are the ROWID, so I must supect they would change after a VACUUM.

The best workaround is to put docid as INTEGER, then adding a PRIMARY KEY index for the docid column.

 
2509 code active 2007 Jul anonymous   2007 Jul   1 1 SQLITE_DATE edit
SELECT CAST(MyDate AS DATE), CAST(MyTime AS TIME) FROM MyData

I hope, it will result/return DATE, TIME. Please support to SQLITE_DATE and SQLITE_TIME. Thanks.

 
2508 code active 2007 Jul anonymous   2007 Dec   1 1 utf8ToUnicode() does not work on some WinCE devices edit
On some WinCE devices first call to MultiByteToWideChar() in utf8ToUnicode() always fails. Tried calling GetLastError() after it fails and it returns error code 87 -- ERROR_INVALID_PARAMETER.

To fix this had to change code page from CP_UTF8 to CP_ACP -- no idea why this works.

Original utf8ToUnicode()


static WCHAR *utf8ToUnicode(const char *zFilename) { int nChar; WCHAR *zWideFilename;

  nChar = MultiByteToWideChar(CP_UTF8, 0, zFilename, -1, NULL, 0);
  zWideFilename = sqliteMalloc( nChar*sizeof(zWideFilename[0]) );
  if( zWideFilename==0 ){
    return 0;
  }
  nChar = MultiByteToWideChar(CP_UTF8, 0, zFilename, -1, zWideFilename, nChar);
  if( nChar==0 ){
    sqliteFree(zWideFilename);
    zWideFilename = 0;
  }
  return zWideFilename;
}


Fixed utf8ToUnicode()
static WCHAR *utf8ToUnicode(const char *zFilename) { int nChar; WCHAR *zWideFilename;

  nChar = MultiByteToWideChar(CP_ACP, 0, zFilename, -1, NULL, 0);
  if( nChar == 0 )
  {
    DWORD dwError = GetLastError();
    OSTRACE2("MultiByteToWideChar() failed, last error: %d\n", dwError);
    return 0;
  }
  zWideFilename = sqliteMalloc( nChar*sizeof(zWideFilename[0]) );
  if( zWideFilename==0 ){
    return 0;
  }
  nChar = MultiByteToWideChar(CP_ACP, 0, zFilename, -1, zWideFilename, nChar);
  if( nChar==0 ){
    sqliteFree(zWideFilename);
    zWideFilename = 0;
  }
  return zWideFilename;
}


2007-Jul-17 23:56:10 by anonymous:
unicodeToUtf8() needs to be fixed the same way.

Before:


static char *unicodeToUtf8(const WCHAR *zWideFilename){ int nByte; char *zFilename;

  nByte = WideCharToMultiByte(CP_UTF8, 0, zWideFilename, -1, 0, 0, 0, 0);
  zFilename = sqliteMalloc( nByte );
  if( zFilename==0 ){
    return 0;
  }
  nByte = WideCharToMultiByte(CP_UTF8, 0, zWideFilename, -1, zFilename, nByte,
                              0, 0);
  if( nByte == 0 ){
    sqliteFree(zFilename);
    zFilename = 0;
  }
  return zFilename;
}


After:
static char *unicodeToUtf8(const WCHAR *zWideFilename){ int nByte; char *zFilename;

  nByte = WideCharToMultiByte(CP_ACP, 0, zWideFilename, -1, NULL, 0, NULL, NULL);
  if ( nByte == 0 )
  {
    DWORD dwError = GetLastError();
    OSTRACE2("WideCharToMultiByte() failed, last error = %d\n", dwError);
    return 0;
  }
  zFilename = sqliteMalloc( nByte );
  if( zFilename==0 ){
    return 0;
  }
  nByte = WideCharToMultiByte(CP_ACP, 0, zWideFilename, -1, zFilename, nByte,
                              0, 0);
  if( nByte == 0 ){
    sqliteFree(zFilename);
    zFilename = 0;
  }
  return zFilename;
}


Note that while original code with CP_UTF8 works on Windows and SOME WinCE devices, this modified code works well and Windows and all WinCE devices I've tested so far.


2007-Jul-18 16:01:21 by anonymous:
Why not using the conversions from SQLite internals ? It can change a UTF-16 to UTF-8 and vice-versa.

Or using UTF-16 variants in windows ce should be the best case.


2007-Aug-09 20:47:04 by anonymous:

  Why not using the conversions from SQLite internals ? It can change a UTF-16 to UTF-8 and vice-versa.

  Or using UTF-16 variants in windows ce should be the best case.

Not so simple. unicodeToUtf8() is used a lot internally regardless of what whether you use UTF-16 or UTF-8 yourself. For example, unicodeToUtf8() is used by sqlite3WinTempFileName() which is in turn used by sqlite3PagerOpentemp() -- I think you get the idea.


2007-Dec-20 00:29:33 by anonymous:
We've found that using CP_UTF8 fails on WinCE kernels that don't include SYSGEN_CORELOC (http://msdn2.microsoft.com/en-us/library/ms903883.aspx). To make the code handle any device it should be changed to:

  static WCHAR *utf8ToUnicode(const char *zFilename)
  {
    int nChar;
    WCHAR *zWideFilename;

    nChar = MultiByteToWideChar(CP_UTF8, 0, zFilename, -1, NULL, 0);
    if( nChar == 0 )
    {
      nChar = MultiByteToWideChar(CP_ACP, 0, zFilename, -1, NULL, 0);
      if( nChar == 0 )
      {
        DWORD dwError = GetLastError();
        OSTRACE2("MultiByteToWideChar() failed, last error: %d\n", dwError);
        return 0;
      }
    }
    zWideFilename = sqliteMalloc( nChar*sizeof(zWideFilename[0]) );
    if( zWideFilename==0 )
    {
      return 0;
    }
    nChar = MultiByteToWideChar(CP_UTF8, 0, zFilename, -1, zWideFilename, nChar);
    if( nChar==0 )
    {
      nChar = MultiByteToWideChar(CP_ACP, 0, zFilename, -1, zWideFilename, nChar);
      if( nChar==0 )
      {
        sqliteFree(zWideFilename);
        zWideFilename = 0;
      }
    }
    return zWideFilename;
  }
 
2507 code fixed 2007 Jul anonymous   2007 Jul   1 1 test_expr expr-1.104 fails (64-bit int handling) edit
make test fails with: expr-1.104...make: *** [test] Floating point exception

System info: gcc-4.1.2, glibc-2.5-r4, 2.6.22.1 x86_64

2007-Jul-16 18:56:02 by drh:
Already fixed. See [4130] and [4131] .
 
2506 new active 2007 Jul anonymous   2008 Jan   3 2 New API to retrieve ROWID from SQLite3_stmt structire edit
Is it too much trouble to allow an API to retrieve ROWID for non-aggeregate queries directly from SQLite3_stmt structire? It would be very useful to create updatable non aggregate query results for situations when actually internal PK (ROWID) is not gived explicitly in SQL statement nor actual table's PK (if any).
SELECT queries that join two or more tables together would be a problem also.


2007-Jul-16 16:51:18 by anonymous:
It's more of a multi-step process. First you have to enumerate the open cursors on the sqlite3_stmt object. Then you need to resolve the table each cursor goes to, and then fetch the rowid for each active cursor. Of course this may get confusing when you've joined a table onto itself.


2008-Jan-19 10:32:59 by anonymous:
This is far to old active ticket. Is it in consideration to be implemented in the near future?
 
2505 code closed 2007 Jul anonymous   2007 Jul   1 1 does not return error when use sqlite_open to open an non-sqlite db edit
I use sqlite_open api to open an non sqlite database, for example an exe file.The result of sqlite_open is not an error code but sqlite_ok.
Sqlite does not actually read the database file until the first SQL query is executed. So the sqlite3_open succeeds but the first call to sqlite3_prepare_v2() or sqlite3_exec() will return an error.
 
2504 build closed 2007 Jul anonymous   2007 Jul   1 1 sqlite 3.4.0 fails to build FreeBSD 6.2 64-bit system edit
Attached is the configure and make output. As this is a FreeBSD 6.2 AMD 64-bit box, several "warning: cast from pointer to integer of different size" in func.c, table.c, and vdbemem.c concern me as potential bugs. Added to the fact the the undefined reference to __isnanl.

The warnings also appear in 3.3.17 version of SQLite, which I hadn't noticed before, but the there was no undefined reference preventing the build.

2007-Jul-13 17:13:47 by anonymous:
submitter: anthony howe, achowe@snert.com


2007-Jul-13 17:34:33 by drh:
Already fixed by check-in [4103] .
 
2503 code active 2007 Jul anonymous   2007 Jul   3 4 sqlite3PagerReleaseMemory does not decrement page count edit
When cached pages are released by sqlite3PagerReleaseMemory the number of pages (pPager->nPage) is not decremented. This also subsequently affects the maximum value at pPager->nMaxPage.

This does not affect the operation of sqlite (but does upset my statistic gathering).

Although only tested with 3.3.8, the problem does not appear to have been corrected in version 3.4.0

Actually this will affect the handling of cache_size as the pager will think there are more pages cached than is the case, and may unnecessarily release some.


2007-Jul-18 15:54:29 by anonymous:
Do you have a simple test case that demonstrates this? Put a print statement in pager.c if necessary.
 
2502 code fixed 2007 Jul anonymous   2007 Jul   4 4 Digital Mars static library runtime problem edit
When I attempt to use a SQLite 3.4.0 static library built with Digital Mars C/C++, I get these errors

sqlite340dm.lib(sqlite340dm) Error 42: Symbol Undefined _sqlite3ParserFree sqlite340dm.lib(sqlite340dm) Error 42: Symbol Undefined _sqlite3Parser sqlite340dm.lib(sqlite340dm) Error 42: Symbol Undefined _sqlite3ParserAlloc

If I comment out lines 65930-65932 in sqlite3.c

  extern void *sqlite3ParserAlloc(void*(*)(size_t));
  extern void sqlite3ParserFree(void*, void(*)(void*));
  extern void sqlite3Parser(void*, int, Token, Parse*);

then the static library works as expected.

Robert Wishlaw

 
2501 build closed 2007 Jul anonymous   2007 Jul   2 3 limits.h confilct with visual studio limits.h edit
When I include sqlite3.h in my code, and provide the sqlite headers folder to complier include path, limits.h is picked up from sqlite headers even when it must be picked from vc/include, so I get errors compiling other files that we compiling fine.

Current workaround is I provide the parent folder of sqlite headers as include path for compiler and instead of #include "sqlite3.h", I do a #include "sqlite-3.4.0/sqlite3.h"

2007-Jul-12 20:39:41 by drh:
See tickets #2428 and #2495.
 
2500 code closed 2007 Jul anonymous   2007 Jul   2 4 Possible memory leak on avr32 arch edit
I am using sqlite in a C written application running on an embedded device based on an avr32 (the 32AP7000) from atmel.

It seams that executing some insertion cause a memory leak.

The simpliest leaking code is something like this : (The complete code include error check, error messages freeing, and non-infinite loop.)

  int   main(int ac, char **av)
  {
    sqlite3 *db;
    char *zErrMsg = 0;

    sqlite3_open(DB_PATH, &db))
    while (42)
      sqlite3_exec(db, "INSERT INTO high (io1) VALUES (1);"
                   NULL, NULL, &zErrMsg));
    sqlite3_close(db);
    return 0;
  }

I've tested with both sqlite3.4.0 and 3.3.6-94 binaries.

Based on the tests i have done the amount of memory leaking depends on the number of row in the table and the size of the sql statement It also seems that a 'SELECT' sql statement does not leak at all.

It also appear that some of the leaking memory is never returned to the system.

I suppose that the problem is somewhere between the sqlite memory management system and the device's one.

I still can not tell for sure if the problem lie on the sqlite or the board but i have successfuly ported some heavy memory consumer without experiencing such problems. Maybe the software shiped with my board don't have default behavior for some memory managment tasks and cause sqlite not to act as it should ?

I am actually working on getting some more informations on how and where it leak. (porting valgrind, dmalloc or something) So i will try to provide some more information as soon as i can.

Maybe you can look at the contents of zErrMsg first?


2007-Jul-13 08:16:37 by anonymous:
I've already checked zErrMsg and freed it if necessary.


2007-Jul-13 08:57:59 by anonymous:
Sorry i didn't understand i am focussed on "where is my memory" :). If you meens to check what error happen, then i don't get any errors everything goes fine and return is SQLITE_OK. The comportement is good exept that the data segment of a program doing multiple inertions keep growing.


2007-Jul-13 13:37:06 by anonymous:
Some fresh news. First the memory leaked seems to be correctly freed when the process done, it was my misunderstanding. Next doing a quick wrapper around sqlite3GenericFree and sqlite3GenericMalloc i found something interristing : Each time i am doing an insert some of the memory allocated wont be freed untill calling sqlite3_close, which cause the process to grow. Since the sqlite malloc mecanism is pretty dark to me i still don't understand how such a behavior happen.


2007-Jul-16 13:50:17 by danielk1977:
Because sqlite3_close() does free the possibly leaked allocations, we can probably assume this is just the pager cache growing normally. See the documentation for the "cache_size" pragma for details.
 
2499 build defer 2007 Jul anonymous   2007 Aug drh 1 1 undefined reference to `_WinMain@16' edit
when attempting to build I get the error:

/usr/lib/gcc/i686-pc-cygwin/3.4.4/../../../libcygwin.a(libcmain.o):(.text+0xab): undefined reference to `_WinMain@16'

2007-Jul-11 18:16:28 by anonymous:

  <!-- ~/mydocs/dl/sqlite-3.4.0 --> ls
Makefile.in	    config.sub	  main.mk	 sqlite.pc.in
Makefile.linux-gcc  configure	  mkdll.sh	 sqlite3.1
README		    configure.ac  mkopcodec.awk  sqlite3.pc.in
VERSION		    contrib	  mkopcodeh.awk  src
aclocal.m4	    doc		  mkso.sh	 tclinstaller.tcl
addopcodes.awk	    ext		  notes		 test
art		    install-sh	  publish.sh	 tool
config.guess	    ltmain.sh	  spec.template  www
  <!-- ~/mydocs/dl/sqlite-3.4.0 --> ./configure && make install
checking build system type... i686-pc-cygwin
checking host system type... i686-pc-cygwin
checking for gcc... gcc
checking for C compiler default output file name... a.exe
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables... .exe
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for a sed that does not truncate output... /usr/bin/sed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for ld used by gcc... /usr/i686-pc-cygwin/bin/ld.exe
checking if the linker (/usr/i686-pc-cygwin/bin/ld.exe) is GNU ld... yes
checking for /usr/i686-pc-cygwin/bin/ld.exe option to reload object files... -r
checking for BSD-compatible nm... /usr/bin/nm -B
checking whether ln -s works... yes
checking how to recognise dependent libraries... file_magic ^x86 archive import|^x86 DLL
checking how to run the C preprocessor... gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking dlfcn.h usability... yes
checking dlfcn.h presence... yes
checking for dlfcn.h... yes
checking for g++... g++
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking how to run the C++ preprocessor... g++ -E
checking for g77... no
checking for xlf... no
checking for f77... no
checking for frt... no
checking for pgf77... no
checking for cf77... no
checking for fort77... no
checking for fl32... no
checking for af77... no
checking for xlf90... no
checking for f90... no
checking for pgf90... no
checking for pghpf... no
checking for epcf90... no
checking for gfortran... no
checking for g95... no
checking for xlf95... no
checking for f95... no
checking for fort... no
checking for ifort... no
checking for ifc... no
checking for efc... no
checking for pgf95... no
checking for lf95... no
checking for ftn... no
checking whether we are using the GNU Fortran 77 compiler... no
checking whether  accepts -g... no
checking the maximum length of command line arguments... 8192
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for objdir... .libs
checking for ar... ar
checking for ranlib... ranlib
checking for strip... strip
checking for correct ltmain.sh version... yes
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC...
checking if gcc static flag -static works... yes
checking if gcc supports -c -o file.o... yes
checking whether the gcc linker (/usr/i686-pc-cygwin/bin/ld.exe) supports shared libraries... yes
checking whether -lc should be explicitly linked in... yes
checking dynamic linker characteristics... Win32 ld.exe
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
configure: creating libtool
appending configuration tag "CXX" to libtool
checking for ld used by g++... /usr/i686-pc-cygwin/bin/ld.exe
checking if the linker (/usr/i686-pc-cygwin/bin/ld.exe) is GNU ld... yes
checking whether the g++ linker (/usr/i686-pc-cygwin/bin/ld.exe) supports shared libraries... yes
checking for g++ option to produce PIC...
checking if g++ static flag -static works... yes
checking if g++ supports -c -o file.o... yes
checking whether the g++ linker (/usr/i686-pc-cygwin/bin/ld.exe) supports shared libraries... yes
checking dynamic linker characteristics... Win32 ld.exe
checking how to hardcode library paths into programs... immediate
appending configuration tag "F77" to libtool
checking for a BSD-compatible install... /usr/bin/install -c
checking for gawk... gawk
Version set to 3.4
Release set to 3.4.0
Version number set to 3004000
checking whether to support threadsafe operation... no
checking whether to allow connections to be shared across threads... no
checking whether threads can override each others locks... no
checking whether to support shared library linked as release mode or not... no
checking whether to use an in-ram database for temporary tables... no
checking if executables have the .exe suffix... unknown
checking host system type... (cached) i686-pc-cygwin
checking for Tcl configuration... found /usr/lib/tclConfig.sh
checking for existence of /usr/lib/tclConfig.sh... loading
checking for library containing tgetent... -ltermcap
checking for readline in -lreadline... no
checking readline.h usability... no
checking readline.h presence... no
checking for readline.h... no
checking for /usr/include/readline.h... no
checking for /usr/include/readline/readline.h... no
checking for /usr/local/include/readline.h... no
checking for /usr/local/include/readline/readline.h... no
checking for /usr/local/readline/include/readline.h... no
checking for /usr/local/readline/include/readline/readline.h... no
checking for /usr/contrib/include/readline.h... no
checking for /usr/contrib/include/readline/readline.h... no
checking for /mingw/include/readline.h... no
checking for /mingw/include/readline/readline.h... no
checking for library containing fdatasync... none required
checking for usleep... yes
checking for fdatasync... yes
configure: creating ./config.status
config.status: creating Makefile
config.status: creating sqlite3.pc
gcc -g -O2 -o lemon.exe ./tool/lemon.c
cp ./tool/lempar.c .
cp ./src/parse.y .
./lemon.exe  parse.y
mv parse.h parse.h.temp
awk -f ./addopcodes.awk parse.h.temp >parse.h
cat parse.h ./src/vdbe.c | gawk -f ./mkopcodeh.awk >opcodes.h
sort -n -b -k 3 opcodes.h | gawk -f ./mkopcodec.awk >opcodes.c
gcc -g -O2 -o mkkeywordhash.exe  ./tool/mkkeywordhash.c
./mkkeywordhash.exe >keywordhash.h
sed -e s/--VERS--/3.4.0/ ./src/sqlite.h.in | \
	sed -e s/--VERSION-NUMBER--/3004000/ >sqlite3.h
rm -rf tsrc
mkdir -p tsrc
cp ./src/alter.c ./src/analyze.c ./src/attach.c ./src/auth.c ./src/btree.c ./src/btree.h ./src/build.c ./src/callback.c ./src/complete.c ./src/date.c ./src/delete.c ./src/expr.c ./src/func.c ./src/hash.c ./src/hash.h ./src/insert.c ./src/legacy.c ./src/loadext.c ./src/main.c ./src/malloc.c ./src/os.c ./src/os_unix.c ./src/os_win.c ./src/os_os2.c ./src/pager.c ./src/pager.h ./src/parse.y ./src/pragma.c ./src/prepare.c ./src/printf.c ./src/random.c ./src/select.c ./src/shell.c ./src/sqlite.h.in ./src/sqliteInt.h ./src/table.c ./src/tclsqlite.c ./src/tokenize.c ./src/trigger.c ./src/utf.c ./src/update.c ./src/util.c ./src/vacuum.c ./src/vdbe.c ./src/vdbe.h ./src/vdbeapi.c ./src/vdbeaux.c ./src/vdbeblob.c ./src/vdbefifo.c ./src/vdbemem.c ./src/vdbeInt.h ./src/vtab.c ./src/where.c ./ext/fts1/fts1.c ./ext/fts1/fts1.h ./ext/fts1/fts1_hash.c ./ext/fts1/fts1_hash.h ./ext/fts1/fts1_porter.c ./ext/fts1/fts1_tokenizer.h ./ext/fts1/fts1_tokenizer1.c sqlite3.h ./src/btree.h ./src/btreeInt.h ./src/hash.h ./src/limits.h opcodes.h ./src/os.h ./src/os_common.h ./src/sqlite3ext.h ./src/sqliteInt.h ./src/vdbe.h parse.h ./ext/fts1/fts1.h ./ext/fts1/fts1_hash.h ./ext/fts1/fts1_tokenizer.h ./src/vdbeInt.h tsrc
cp: warning: source file `./src/btree.h' specified more than once
cp: warning: source file `./src/hash.h' specified more than once
cp: warning: source file `./src/sqliteInt.h' specified more than once
cp: warning: source file `./src/vdbe.h' specified more than once
cp: warning: source file `./ext/fts1/fts1.h' specified more than once
cp: warning: source file `./ext/fts1/fts1_hash.h' specified more than once
cp: warning: source file `./ext/fts1/fts1_tokenizer.h' specified more than once
cp: warning: source file `./src/vdbeInt.h' specified more than once
rm tsrc/sqlite.h.in tsrc/parse.y
cp parse.c opcodes.c keywordhash.h tsrc
tclsh ./tool/mksqlite3c.tcl
cc     sqlite3.c   -o sqlite3
/usr/lib/gcc/i686-pc-cygwin/3.4.4/../../../libcygwin.a(libcmain.o):(.text+0xab): undefined reference to `_WinMain@16'
collect2: ld returned 1 exit status
make: *** [sqlite3] Error 1
  <!-- ~/mydocs/dl/sqlite-3.4.0 -->


2007-Jul-11 18:17:17 by anonymous:
when initially filing this bug, i was told to not paste large screen dumps.

i was also promised that i would be able to add an attachment later. however, that option was not given, so I have attached a screen transcript.


2007-Jul-11 18:18:33 by anonymous:
my email is metaperl at gmail.com


2007-Jul-11 18:38:53 by drh:
The [Attach] link in the upper-left is what you use to add attachments.

The sqlite3.c source file is a library, not a program. If you want to compile a command-line shell, add the "shell.c" source file:

   cc -o sqlite3 sqlite3.c shell.c


2007-Jul-23 20:35:06 by anonymous:
I believe this bug report should be reopened because this error comes when you try to configure and compile USING THE DEFAULTS. When you try to do a standard configure and make, it should either try to compile shell.c as is needed or not compile sqlite.c. It shouldn't put in a command that fails the compile by default.

Sorry to bother you about this, but I don't see how this isn't a bug or at least a glitch. It's a problem in the configuration and makefile system.


2007-Aug-15 03:26:05 by anonymous:
Compile and link passed by following patch (at least for me).

  diff -ru sqlite-3.4.2/Makefile.in sqlite-3.4.2-new/Makefile.in
  --- sqlite-3.4.2/Makefile.in    2007-08-15 10:37:18.921875000 +0900
  +++ sqlite-3.4.2-new/Makefile.in        2007-08-15 10:36:08.750000000 +0900
  @@ -681,7 +681,7 @@
          mkdir -p doc
          mv $(DOC) doc

  -install:       sqlite3 libsqlite3.la sqlite3.h ${HAVE_TCL:1=tcl_install}
  +install:       sqlite3$(TEXE) libsqlite3.la sqlite3.h ${HAVE_TCL:1=tcl_install}
          $(INSTALL) -d $(DESTDIR)$(libdir)
          $(LTINSTALL) libsqlite3.la $(DESTDIR)$(libdir)
          $(INSTALL) -d $(DESTDIR)$(exec_prefix)/bin


2007-Aug-22 12:33:17 by anonymous:
Why is this bug closes and marked "not a bug"? sqlite fails on make install on Cygwin (as in untar/configure/make/make install), which is clearly a bug. The "fix" posted above does not work, make install fails instantly with

tclsh ../sqlite-3.4.2-n/tclinstaller.tcl 3.4 can't read "env(DESTDIR)": no such variable while executing "set LIBDIR $env(DESTDIR)$env(TCLLIBDIR)" (file "../sqlite-3.4.2-n/tclinstaller.tcl" line 11) make: *** [tcl_install] Error 1


2007-Aug-22 12:53:45 by drh:
I misread the problem.

Cygwin is not a supported platform. So in that sense, this is not clearly a bug in SQLite. This is an imcompatibility with Cygwin.

We will take the problem under advisement, but since Cygwin is not officially support and because we have many much more serious problems to work on, progress on this is likely to be slow. You can expedite a fix by posting a patch.

 
2498 code active 2007 Jul anonymous   2007 Jul   3 2 sqlite memory org on linux edit
(related ticket #2473)... he sample programme that I run(wrote) in tty1 and there I operate the command of ps at tty2, there seems two items from the programme of ps command. This error was not at the version 3.3.13 but now it is happening at sqlite versions although i change nothing from the programme, If I turn to old versions, there is seen only one item again. When I upgrade to version 3.3.13 or later, there is seen two items again Is it normal or there is any mistake? (excuse my poor english)
2007-Jul-11 16:44:22 by anonymous:
So you are seeing 2 processes instead of 1 on Linux? Linux 2.4 and earlier kernels show threads as seperate processes with unique process IDs. Is your program creating any threads?

The only place where SQLite creates threads is the function below - but it joins with the thread right away.

/*
** This procedure attempts to determine whether or not threads
** can override each others locks then sets the
** threadsOverrideEachOthersLocks variable appropriately.
*/
static void testThreadLockingBehavior(int fd_orig){
  int fd;
  struct threadTestData d[2];
  pthread_t t[2];

  fd = dup(fd_orig);
  if( fd<0 ) return;
  memset(d, 0, sizeof(d));
  d[0].fd = fd;
  d[0].lock.l_type = F_RDLCK;
  d[0].lock.l_len = 1;
  d[0].lock.l_start = 0;
  d[0].lock.l_whence = SEEK_SET;
  d[1] = d[0];
  d[1].lock.l_type = F_WRLCK;
  pthread_create(&t[0], 0, threadLockingTest, &d[0]);
  pthread_create(&t[1], 0, threadLockingTest, &d[1]);
  pthread_join(t[0], 0);
  pthread_join(t[1], 0);
  close(fd);
  threadsOverrideEachOthersLocks =  d[0].result==0 && d[1].result==0;
}
If you post a small C program demonstrating what you're seeing, someone may be able to offer a suggestion.


2007-Jul-11 16:47:10 by anonymous:
I suppose it's not inconceivable that the join failed. Perhaps these pthread_join calls' return codes should be examined for errors.


2007-Jul-11 18:53:36 by anonymous:
If you're playing games with tty's and you've got an early Linux 2.6 kernel, it's possible that processes are dying because of http://lkml.org/lkml/2004/10/21/119. It was, last I checked, fixed in 2.6.10.

The SIGHUP being generated might also interfer with a pthread_join(), although pthread_join() doesn't say anything about ever generating EINTR...

c.


2007-Jul-12 06:12:28 by anonymous:
my example program is very simple, i not use threading-multithreading structure... If I turn to old versions of sqlite, there is seen only one item again, when I upgrade to version 3.3.13 or later, there is seen two items again Is it.

note: /lib/libpthread.so.0 linked to /lib/libpthread-0.10.so (size 55468 byte)


2007-Jul-12 11:36:37 by anonymous:
Your description of the problem isn't clear enough, so the answers you're getting are just guesses.

You may have more luck by describing the problem (with as much detail as possible) in your native language and hoping someone in the SQLite community can add a translation. I know you're doing your best with the english you speak, but it's not working well enough for someone to help with your problem.

Adding code samples and command-line output would also help considerably, since that sort of this is mostly language independent.

 
2497 code closed 2007 Jul anonymous   2007 Jul   4 2 sqlite3_errcode does not return the proper error code edit
sqlite3_errcode does not return the same result as the last api call when using a progress handler that aborts the connection. sqlite3_step returns SQLITE_ABORT, but sqlite3_errcode is returning SQLITE_ERROR.

A test case can be found here:

https://bugzilla.mozilla.org/attachment.cgi?id=271753

Downstream bug:

https://bugzilla.mozilla.org/show_bug.cgi?id=387609

-sdwilsh

2007-Jul-11 06:37:52 by danielk1977:
When it comes to setting the database error code, sqlite3_step() behaving the same way regardless of whether sqlite3_prepare() or sqlite3_prepare_v2() is used to prepare the query.

So when an error is returned from sqlite3_step() on a statement prepared with _v2(), the "real" error code is returned but the database error code is being set to SQLITE_ERROR. I'm thinking we should change to set the database error code and message to the "real" error code in this case.

Also, note that this test-case is incompatible with version 3.4.0. In prior versions, if the progress-handler returns non-zero, the statement is halted and SQLITE_ABORT is returned. In 3.4.0, this error code is now SQLITE_INTERRUPT.


2007-Jul-12 16:35:37 by anonymous:
Thanks for the heads up.

-sdwilsh

 
2496 code active 2007 Jul anonymous   2007 Jul   5 4 "No such column" error should include table information edit
It'd be nice if the "no such column" error included the table/view that SQLite was searching for the column.

no such column: ChecklistID

Thanks,

Sam

 
2495 code closed 2007 Jul anonymous   2007 Jul   4 4 standard C library Naming conflict - limits.h edit
"limits.h"

You should really avoid using filenames conflicting with the standard C library. It took me a few minutes to figure out why my code was not compiling. Then I realized that <limits.h> was resolving to the SQLITE header.

Best Wishes, -David Delaune Software Engineer Marine Technologies, LLC

oops, I just found this issue was corrected in ticket 2428. Closing the ticket.

-David Delaune

 
2494 code closed 2007 Jul anonymous   2007 Jul   1 1 DRow["ID"] = reader.GetString(reader.GetOrdinal("ID")); fails edit
DRow["ID"] = reader.GetString(reader.GetOrdinal("ID")); This fails under Vista Ultimate, succeeds under XP Pro. It is Sqlite 3.1, will upgrading help?

Thanks! Jim

Please post your question to the author of the sqlite wrapper for whatever language you're using.
 
2493 build closed 2007 Jul anonymous   2007 Jul   2 3 Fails to build under GCC-4.1.3 on FreeBSD-6.2 edit
OS: FreeBSD-6.2

When using the build-in gcc 3.4.6 [FreeBSD] 20060305, sqlite3, version 3.3.17 builds fine.

However, when using gcc 4.1.3 20070618, sqlite3 version 3.3.17 fails to build.

A copy of the failed build attempt is attached.

2007-Jul-09 13:23:38 by anonymous:

  ===>  Applying FreeBSD patches for sqlite3-3.3.17

Your SQLite sources have apparently been modified for FreeBSD. This is a question for the BSD port system maintainers or simply google "unable to infer tagged configuration" where you will find many workarounds for your operating system.

  libtool: link: unable to infer tagged configuration
  libtool: link: specify a tag with `--tag'
  gmake: *** [libsqlite3.la] Error 1
  *** Error code 2

  Stop in /usr/ports/databases/sqlite3.


2007-Jul-21 19:58:11 by anonymous:
This is a libtool bug that happens when the C compiler you're using is not named 'gcc'.

As a workaround in your case:

  CC=/usr/local/bin/gcc41 ./configure
  make

Above command assumes bash shell. Use appropriate construct for other shells.

 
2492 code closed 2007 Jul anonymous   2007 Jul   4 4 Incorrect protoype for function sqlite3_result_zeroblob edit
Line 1859 in sqlite3.h and line 1882 in sqlite3.c

void sqlite3_result_zeroblob(sqlite3_context*, int n);

should read

void sqlite3_result_zeroblob(sqlite3_context*, int);

 
2491 code active 2007 Jul anonymous   2007 Jul   1 1 Mingw Warnings w/ 3.4.0 Amalgamation edit
When compiling the 3.4.0 amalgamation sqlite3.c file w/ no defines, you get the following warnings:

sqlite3/sqlite3.c: In function `sqlite3BtreeFindCell':
sqlite3/sqlite3.c:23249: warning: unused variable `data'
sqlite3/sqlite3.c: In function `vxprintf':
sqlite3/sqlite3.c:8488: warning: 'xtype' might be used uninitialized in this function
sqlite3/sqlite3.c: In function `sqlite3BtreeOpen':
sqlite3/sqlite3.c:19488: warning: 'nameLen' might be used uninitialized in this function
sqlite3/sqlite3.c: In function `getOverflowPage':
sqlite3/sqlite3.c:25386: warning: 'rc' might be used uninitialized in this function
sqlite3/sqlite3.c: In function `sqlite3Select':
sqlite3/sqlite3.c:56300: warning: 'pEList' might be used uninitialized in this function
sqlite3/sqlite3.c:56301: warning: 'pTabList' might be used uninitialized in this function
sqlite3/sqlite3.c: At top level:
sqlite3/sqlite3.c:16020: warning: 'sqlite3GenericAllocationSize' defined but not used
sqlite3/sqlite3.c:6188: warning: 'sqlite3Utf16Substr' declared `static' but never defined
sqlite3/sqlite3.c:6307: warning: 'sqlite3Get2byte' declared `static' but never defined
sqlite3/sqlite3.c:6309: warning: 'sqlite3Put2byte' declared `static' but never defined
sqlite3/sqlite3.c:23248: warning: 'sqlite3BtreeFindCell' defined but not used
sqlite3/sqlite3.c:63547: warning: 'sqlite3ParserAlloc' defined but not used
sqlite3/sqlite3.c:63673: warning: 'sqlite3ParserFree' defined but not used
sqlite3/sqlite3.c:65286: warning: 'sqlite3Parser' defined but not used

I know the uninitialized warnings are false warnings but the defined functions that aren't used seem to be an error in building the amalgamation.

 
2490 code closed 2007 Jul anonymous   2007 Oct   1 1 Latest cvs 3.4.0 seg fault edit
This problem is the same with 3.3.1.13, Fedora's 3.4.0 and the debug version I just built.

Using latest Fedora and perl modules:

   net1#r mx
   mxaddrs ...
   perl        : 5.8.8
   DBI         : 1.58
   DBD::SQLite : 1.13
   SQLite::DBMS: 3.4.0.1

Code is simple db create and inserts. Problem only occurs when a DBI sth->prepare() is used with multiple execute inserts of duplicate data. Instead of returning an error seg fault occurs. If required I can try and put together a small code sample.

2007-Jul-07 06:25:54 by anonymous:

  Why do you think its pure SQLite problem ?


2007-Jul-07 11:46:55 by drh:
To amplify the remark by anonymous above: what makes you think this is a problem with SQLite and not a problem with perl or the DBD::SQLite module? We have run literally millions of test cases through the core SQLite without hitting any segfaults, yet you say that a simple test case is sufficient to cause the problem. That would seem to implicate DBD::SQLite rather than the SQLite core, would it not? Please tell us why you think this is an SQLite problem. And please also provide us with a reproducible test case, keeping in mind that the core SQLite developers do not normally have DBD::SQLite installed.


2007-Aug-10 15:18:32 by anonymous:
This is a DBD::SQLite bug. See http://rt.cpan.org/Public/Bug/Display.html?id=28757 for a test case and a patch.

Mike

 
2489 build closed 2007 Jul anonymous   2007 Jul   1 1 Latest cvs 3.4.0 make test fails related to TCL edit
Here's how I'm building.

I've tried --disable-tcl and --enable-tcl with no joy.

I'm using latest Fedora 7 with tcl-8.4.13 installed.

  mkdir -p /build/work/sqlite-3.4.0.1
  cd /build/work/sqlite-3.4.0.1
  unset CDPATH
  export CFLAGS='-pipe -O3 -g'
  make distclean
  cvs -d :pserver:anonymous@www.sqlite.org:/sqlite -r update .
  ./configure --prefix=/usr/local/pkgs/sqlite-3.4.0.1 --disable-tcl
  make
  groupadd vuser || /bin/true
  useradd -M -g vuser -d /vhost/davidfavor.com/users/david -s /bin/zsh david  || /bin/true
  useradd -M -g vuser -d /vhost/livefeast.com/users/yemiah -s /bin/zsh yemiah || /bin/true
  chown david:vuser -R .
  su -c "make test" david
  /build/work/sqlite-3.4.0.1/./src/tclsqlite.c:2416: undefined reference to `Tcl_CreateObjCommand'
  /build/work/sqlite-3.4.0.1/./src/tclsqlite.c:2417: undefined reference to `Tcl_PkgProvide'
  /build/work/sqlite-3.4.0.1/./src/tclsqlite.c:2418: undefined reference to `Tcl_CreateObjCommand'
  ... hundreds of errors deleted ...
"make test" requires Tcl to function.

Post the result of "grep -i tcl config.log" after you run ./configure with no arguments. The key to it working is finding tclConfig.sh

configure:19220: checking for Tcl configuration
configure:19302: result: found /usr/lib/tclConfig.sh
configure:19305: checking for existence of /usr/lib/tclConfig.sh
ac_cv_c_tclconfig=/usr/lib
HAVE_TCL='1'
TCL_BIN_DIR='/usr/lib'
TCL_INCLUDE_SPEC='-I/usr/include/tcl8.4.13'
TCL_LIBS='-ldl  -lpthread -lieee -lm'
TCL_LIB_FILE='libtcl8.4.so'
TCL_LIB_FLAG='-ltcl8.4'
TCL_LIB_SPEC='-L/usr/lib -ltcl8.4'
TCL_SRC_DIR='/usr/include/tcl8.4.13'
TCL_STUB_LIB_FILE='libtclstub8.4.a'
TCL_STUB_LIB_FLAG='-ltclstub8.4'
TCL_STUB_LIB_SPEC='-L/usr/lib -ltclstub8.4'
TCL_VERSION='8.4'

2007-Jul-07 03:49:21 by anonymous:
Ah I see now. I'm using a 64 bit machine so tcl lives in the 64 bit directory hierarchy.

Here's how I got most of the tests to run:

   export CFLAGS='-pipe -O3 -g -DSQLITE_DISABLE_DIRSYNC=1 -Wall'
   make distclean
   cvs -d :pserver:anonymous@www.sqlite.org:/sqlite -r update .
   ./configure --prefix=/usr/local/pkgs/sqlite-3.4.0.1 --enable-tcl --with-tcl=/usr/lib64

This ticket can close. I'll do some further testing and open another ticket about the single thread test that fails.


2007-Jul-07 11:43:05 by drh:
Please note that this is an open ticketing system. Anybody can close a ticket. You do not need to ask me or Dan to do it.
 
2488 new active 2007 Jul anonymous   2007 Jul   5 4 autosize on column output mode in sqlite3 program edit
It would be nice if sqlite3 program has a autosizecolumn mode for displaying queries, because it truncates values, and to calculate and use .width size for each column is tedious.
2007-Jul-07 11:42:03 by drh:
In order to do this, we would have to either run the query twice or load the entire result set into memory. Otherwise, there would be no way to determine the longest element of each column. Neither approach seems attractive for large and complex queries.


2007-Jul-28 07:05:56 by anonymous:
every query should internally detect the longest column sizes and a new command should enable the user to set these values for any repetition or similar queries. At the end the queries run twice or more but only the first trial would have cause irritations on output. And this solution should be easy enough to implement it.
Even the core of sqlite could calculate the maximum length of each column and a new API function could make this available. It would be really nice to get such an enhancement!


2007-Jul-28 18:37:36 by anonymous:
while you don't move to next row, sqlite doesn't know the contents, so it will be impossible to do, only if you cache the text entirely in memory, but this is ugly, imagine a 1GB recordset into RAM...


have you tried .mode tabs ? here, i have correct column width
 
2487 code active 2007 Jul anonymous   2007 Jul   1 1 SQLite database locked error on NFS mounted home dir edit
I have a c program using the provided API. My home directory is NFS mounted, Im using SQLite 3.3.17. I open a new database using "sqlite3_open", then strcpy () a SQL command to create a table, and run "sqlite3_exec" with this string. I get a return code of 5=database locked. I then tried to manually (command line using sqlite3) create a table within a database in my home dir, that fails too.

   ===========
   x@y> sqlite3 db2
   SQLite version 3.3.17
   Enter ".help" for instructions
   sqlite> create table test (Lastname varchar);
   SQL error: database is locked
   sqlite>
   ==============

If I try this on my local machine (a Mac), it works fine, but I need it to work in my home directory mounted via NFS as that is where the output of our program goes

2007-Jul-06 19:04:15 by anonymous:
If you're using a Mac, compile sqlite with SQLITE_ENABLE_LOCKING_STYLE in os_unix.c


2007-Jul-07 11:51:10 by drh:
This is a problem with your NFS implementation - it does not appear to support posix advisory locking. There is nothing much that SQLite can do about this.

Anonymous above suggests making use of the dot-locking mechanism contributed by Apple. This might be an effective work-around. But remember that there is performance impact. Also remember that an SQLite database that uses dot-locking is subtly imcompatible with a standard SQLite database. The file format itself is the same, but if two processes try to access the database file at the same time and one uses dot-locks and the other uses posix advisory locks, you will end up with corruption.


2007-Jul-07 12:44:09 by anonymous:
It's very odd that Apple does not fix their Mac OSX POSIX locks for NFS given their resources.
 
2486 code closed 2007 Jul anonymous   2007 Sep   2 3 pragma index_list does not detect index created on 2nd connection edit
Open two connections to the same database. On the first connection, create a table "test" and an index on that table. On the second, run "pragma index_list( test )".

Expected result: pragma detects the previously-created index. Actual result: no index is detected.

Test case to follow.

2007-Jul-05 20:02:25 by anonymous:
#include "sqlite3.h"

#include <boost/test/auto_unit_test.hpp>

extern "C" int IsIndexCreatedCallback( void* p, int, char**, char** )
{
	*(bool*)p = true;
	return 0;
}

BOOST_AUTO_TEST_CASE( SQLiteTest )
{
	sqlite3* sqlite1;
	int result = sqlite3_open( "test", &sqlite1 );

	sqlite3* sqlite2;
	result = sqlite3_open( "test", &sqlite2 );

	result = sqlite3_exec( sqlite1, "create table test ( id integer )", 0, 0, 0 );

	bool exists = false;
	result = sqlite3_exec( sqlite2, "pragma index_list( test )", IsIndexCreatedCallback, &exists, 0 );
	BOOST_CHECK( !exists );

	result = sqlite3_exec( sqlite1, "create index ix on test ( id )", 0, 0, 0 );

	exists = false;
	result = sqlite3_exec( sqlite2, "pragma index_list( test )", IsIndexCreatedCallback, &exists, 0 );
	BOOST_CHECK( exists );
}


2007-Jul-05 20:18:10 by anonymous:
Can reproduce with 3.4.0 using two sqlite3 processes.

I found that if you execute "select * from sqlite_master" from the second sqlite3 commandline shell, then pragma index_list on the 2nd connection will correctly show the indexes for some reason.


2007-Jul-05 21:12:32 by drh:
The first connection does not know that an index has been created on the second connection until it attempts to access the database.


2007-Jul-05 21:17:36 by anonymous:
Same problem with connection 1 executing ALTER TABLE ADD COLUMN and connection 2 executing PRAGMA table_info(table-name). I suspect the following PRAGMAs must always read() the database's file change counter and if it changed then reset the cached schema before proceeding:

  PRAGMA foreign_key_list(table-name);
  PRAGMA index_info(index-name);
  PRAGMA index_list(table-name);
  PRAGMA table_info(table-name);


2007-Jul-05 21:37:12 by anonymous:
Does the comment on 2007-Jul-05 21:12:32 imply this behavior is by design?


2007-Jul-06 11:31:55 by drh:
Yes.


2007-Jul-06 15:38:07 by anonymous:
That's understandable. This ticket can be fixed with documentation. Can you recommend the most efficient/quickest way to force SQLite to reload the schema in memory from a library user's point of view?

  SELECT 0 from _dummy_table_

  SELECT 0 from sqlite_master where 1=0

?


2007-Jul-06 15:41:37 by anonymous:
Would a new pragma for this purpose be out of the question?

  PRAGMA force_schema_reload;


2007-Jul-06 16:57:56 by anonymous:
I can appreciate seperate processes' in-memory schema not being automatically updated when another process changes the schema or adds a new index. But it does not make sense that another connection in the same process is not flagged to automatically update their in-memory schema prior to executing one of these PRAGMAs being issued. It's just a matter of checking an integer change counter in memory for each connection on the same database.


2007-Jul-06 18:22:18 by anonymous:
SELECT 0 from sqlite_master where 1=0 doesn't work. SELECT 0 from sqlite_master does.

Can you elaborate on the reason for this design decision?


2007-Jul-06 18:55:04 by anonymous:
I was just looking for the fastest executing SELECT statement I could find that would force a reload of all schema from the file. Perhaps something like this?

  select null from sqlite_master limit 1

I'm pretty sure trying to query a non-existant table also has the side-effect of reloading the entire schema.

  -- force a re-load, and ignore the error
  SELECT a from _non_existant_table_

I just don't know which one's faster. It would be better to have a standard way of doing this.


2007-Jul-11 21:08:14 by anonymous:
Doing a select against a non-existent table will fail without reloading the schema. At least in my independent discovery of this problem that didn't resolve things...


2007-Jul-20 06:25:14 by anonymous:
Would a schema change (add index) in one connection affect the other connection as it goes to insert into the table without knowing that the index has been added? And result in incomplete index contents?


2007-Jul-26 18:42:06 by anonymous:
the greatest way to solve this is reloading the db metadata when you issue a pragma command, like pragma index_list or other.

i thought this will be great.

also, database header has a version to detect SQLITE_SCHEMA changes. why pragma commands doesn't honor this ?


2007-Aug-22 19:09:28 by anonymous:
related ticket: #2519


2007-Sep-04 20:10:24 by anonymous:
Fixed in 3.5.0 (*)

  #include "sqlite3.h"
  #include <stdio.h>
  #include <assert.h>
  #include <unistd.h>
  int mycb(void* p, int nCol, char** azVals, char** azCols) {
    int i;
    *(int*)p = 1;
    printf("mycb: ");
    for (i = 0; i < nCol; i++) {
      printf("%s = %s, ", azCols[i], azVals[i]);
    }
    printf("\n");
    return 0;
  }
  int main() {
    const char* dbname = "schemachange.db";
    sqlite3* sqlite1;
    sqlite3* sqlite2;
    int exists;
    int result;
    sqlite3_enable_shared_cache(1); /***** line necessary for it to work ****/
    unlink(dbname);
    result = sqlite3_open(dbname, &sqlite1);
    assert(result == SQLITE_OK);
    result = sqlite3_open(dbname, &sqlite2);
    assert(result == SQLITE_OK);
    result = sqlite3_exec(sqlite1, "create table t(id integer)", 0, 0, 0);
    assert(result == SQLITE_OK);
    exists = 0;
    result = sqlite3_exec(sqlite2, "pragma index_list(t)", mycb, &exists, 0);
    assert(result == SQLITE_OK);
    result = sqlite3_exec(sqlite1, "create index ix on t(id)", 0, 0, 0);
    assert(result == SQLITE_OK);
    exists = 0;
    result = sqlite3_exec(sqlite2, "pragma index_list(t)", mycb, &exists, 0);
    assert(result == SQLITE_OK);
    assert(exists);
    return 0;
  }

(*) It seems that sqlite3_enable_shared_cache(1) is required.

What's the downside in making this the default behavior? It seems like a win/win from a memory usage and schema change detection point of view.

 
2485 code closed 2007 Jul anonymous   2007 Jul   1 2 SQLite + Python segfaults at various places in the code edit
I've been getting segfaults while running unit tests on my code base (http://drproject.org). It is written in Python and uses SQLAlchemy for database abstraction. I believe the Python module is PySQLite2. After taking some core dumps, they all seem to point to SQLite code (but different parts each time). Another guy I'm working with is having similar problems with some code he is working on. I have run memtest over night (turning up nothing), so I don't think it's my machine.

I am not sure what would be most helpful for you, so before I start throwing around 20 meg core dumps, I thought I'd check and see what you would prefer.

2007-Jul-04 17:05:32 by anonymous:

  Try the mailing list as your problem is not pure SQLite problem.


2007-Jul-05 06:25:40 by danielk1977:
If it's different places in the code each time, this really sounds like a corrupted heap. It may be that it is SQLite corrupting the heap, or it may be the wrapper code. Posting to the mailing list is probably the best way to progress this.


2007-Jul-05 13:02:35 by anonymous:
Setting up and learning the inner workings of another computer language and third party wrappers requires a lot of effort. You greatly increase your odds of finding the source of your bug if you make a small standalone C or Tcl program that demonstrates the problem using only the SQLite API. This eliminates the possibility that the host language or wrapper is corrupting memory or doing something else wrong. Often in the process of creating such a standalone test case, people find their own mistake.
 
2484 new active 2007 Jul anonymous   2007 Jul   5 4 Support for RETURNING edit
I was recently trying to get HTSQL (http://htsql.org) to work with SQLite, especially since it'd be nice to work out-of-the-box with Python. One of the hiccups was the lack of a RETURNING clause, this is especially important once you have auto-incremented keys. For example..

  INSERT INTO TABLE some_table (a_column) values ('value')
    RETURNING (serial_column);

This acts like a SELECT following the INSERT returning the requested columns on the affected rows. It is quite helpful for cases like UPDATE or DELETE when more than one row is affected. While this feature isn't critical for SQLite, it reduces client-side code significantly. Thank you for your kind consideration.

 
2483 code closed 2007 Jul anonymous   2007 Jul   1 3 like clause not working edit
while using the Trac SCM tool I've experienced some issues with the database usage of the like clause in certain select statements.

with values for name such as 'TracIni', 'TracBrowser', 'TracFqastCGI' the following quert does not catch any of the above names:

SELECT name FROM wiki WHERE name like 'Trac';

seems to be that this should catch it.

2007-Jul-04 03:56:32 by anonymous:
To catch the terms you are looking for, wouldn't you need this query:

SELECT name FROM wiki WHERE name like 'Trac%'

 
2482 code closed 2007 Jul anonymous   2007 Jul   3 4 Possible optimizer bug edit
The versions I have tested are 3.3.5 and 3.4.0. Here you see the same query four times with different table ordering and sometimes an additional OR clause. The queries are executed on an ARM device, but I observed the same behavior on x86. I know that the table ordering makes a difference in query execution time but I'm rather puzzled by the behavior I get here. The "OR 0" doesn't make much sense to me, it was coincidence that I discovered it.

  1. The query as intended, this takes a rather long time:
    "select pr_value from property join item_prop on ip_prop=pr_id where ip_item=1049545 and ip_visible=1 and pr_class=4"
    Executed once: 127 ms
    Executed 10 times: 264 ms

  2. The same as above but I added a "OR 0" which shouldn't change anything. The same query is now much faster:
    "select pr_value from property join item_prop on ip_prop=pr_id where ip_item=1049545 and ip_visible=1 and (pr_class=4 OR 0)"
    Executed once: 4 ms
    Executed 10 times: 30 ms

  3. The same query as 1. but with exchanged table order:
    "select pr_value from item_prop join property on ip_prop=pr_id where ip_item=1049545 and ip_visible=1 and pr_class=4"
    Executed once: 3 ms
    Executed 10 times: 28 ms

  4. Added an additional "OR 0" to 3. This is now slower than the previous:
    "select pr_value from item_prop join property on ip_prop=pr_id where ip_item=1049545 and ip_visible=1 and (pr_class=4 OR 0)"
    Executed once: 3 ms
    Executed 10 times: 32 ms

The tables used:

  create table property (
  	pr_id		integer primary key,
  	pr_value	text,
  	pr_class	int
  );
  create index pr_idx1 ON property(pr_id);
  create index pr_idx2 ON property(pr_value COLLATE NOCASE);
  create index pr_idx3 ON property(pr_class);

  create table item_prop (
  	ip_item		int,
  	ip_prop		int,
  	ip_visible  int default 1
  );
  create index ip_idx2 ON item_prop(ip_item);
  create index ip_idx3 ON item_prop(ip_prop);
2007-Jul-02 16:01:24 by drh:
Thank you for taking the time to learn enough about the wiki formatting rules to format the problem description so that it is easily readable!

We do not normally consider a problem to be a "bug" if it gets the correct answer, even if the answer takes longer than you want to obtain. My understanding is that all of the queries above return the correct answer and you just want them to go faster. This is not a bug. For help in optimizing your queries, please post on the sqlite mailing list. Instructions for subscribing to the mailing list can be found at

You will probably get better results in this case if you do one or both of the following:

  • Drop the pr_idx3 index.
  • Run the ANALYZE command.


2007-Jul-02 20:55:18 by anonymous:
Yes, they actually work as expected. Thanks for the hints. The actual reason why I posted this ticket is the weird improve in performance by adding a "OR 0" which doesn't change anything on the query itself. Everything is the same except for this no-op condition.

If the optimizer would do such a thing by default (whatever it is that it does) then this could possibly improve the sqlite performance in other scenarios as well.

 
2481 code closed 2007 Jul anonymous   2007 Jul   1 3 sqlite leaks memory (after 3.3.13) edit
After upgrade from 3.3.12 to 3.4.0 I noticed an huge negative impact on memory usage of my SQLite-based programs. I suspected a bug in my code. After some debugging, I created a simple program and narrowed this down to the following:

  1. this started from 3.3.14 and is still relevant for 3.4.0
  2. 3.3.13 was the last version which "worked" as expected

Attached is a sample program which demonstrates the problem. sqlite3 command utility exhibits the same increase and the same memory usage patterns.

Basically, what happens is that memory allocated by SQLite is first released after call to sqlite3_close(). It seems like the memory usage will eventually "stabilize" and only increase again in smaller chunks.

This is on OpenBSD 4.1/i386.

ssehic@dev-5-i386:/profense/devel$ ls -l /profense/log/logdata.db
-rwxrwxr-x  1 _admd  profense  12106752 Jul  1 23:15 /profense/log/logdata.db

ssehic@dev-5-i386:/profense/devel$ sqlite3 /profense/log/logdata.db "select count(*) from log"
156000

ssehic@dev-5-i386:/profense/devel/sqlmem
SQLite version 3.3.13
Usage (bytes)               initial, rssize=487424, dsize=241664
Usage (bytes) after SELECT COUNT(*), rssize=1417216, dsize=704512
Usage (bytes)   after sqlite3_close, rssize=1318912, dsize=618496

ssehic@dev-5-i386:/profense/devel/sqlmem
SQLite version 3.4.0
Usage (bytes)               initial, rssize=495616, dsize=241664
Usage (bytes) after SELECT COUNT(*), rssize=22118400, dsize=21331968
Usage (bytes)   after sqlite3_close, rssize=1351680, dsize=618496
I'm sure something like this should not go unnoticed by the developers. However, I can't believe this is specific to OpenBSD. Something changed after 3.3.13 which broke this in a subtle way.

Any hints?

2007-Jul-01 23:03:42 by drh:
Your test program appears to be OpenBSD-specific. I cannot compile it on Linux. So I cannot replicate your problem.

We do test SQLite using both its own internal memory leak detection logic and valgrind, and both report no memory leaks over the entire test suite.


2007-Jul-02 07:31:57 by anonymous:
Yes. The test program is OpenBSD specific, but only the memory reporting stuff (print_memory_usage()). I'll see if I can provide a Linux variant and complete details on how to reproduce the problem.

More to come.


2007-Jul-02 09:28:42 by anonymous:
I think I figured this one out. The problem is only reproducible if the database is created using sqlite3 prior to 3.3.14 and queries are later run on the same database using 3.4.0.

This is weird, since the database format should be backwards compatible between releases. However, lots of "performance improvements" went in after 3.3.13. This could have something to do with it.

At http://insecure.dk/bin/logdata.db.gz is a test database created with 3.3.12. If you run a "SELECT COUNT(*) FROM log" on that using 3.4.0, you should see the memory usage increase greatly using sqlite3 utility.


2007-Jul-02 10:38:12 by anonymous:
This ticket can be closed. Increased memory usage is due to my test database (created with 3.3.12) used PRAGMA default_cache_size = 10000;

Apparently versions > 3.3.14 allocate all the memory on the first database query. That's why I saw the jump in memory usage.

Sorry for the noise.


2007-Jul-02 10:38:41 by drh:
I downloaded logdata.db from the link above and ran "SELECT count(*) FROM log" against it using the latest version of SQLite from CVS (which i closer to 3.4.1 than to 3.4.0, I suspect). I did this using valgrind:

   valgrind --leak-check=full --show-reachable=yes \
      ./sqlite3 logdata.db 'select count(*) FROM log'

Valgrind tells me that 156 bytes of memory were leaked in 11 allocations. All of the leaked memory was allocated by the getpwuid_r() function in glibc - not from SQLite.

The total memory allocated was 13,515,680 bytes, which considering that the logdata.db database specifies a cache size of 10,000 pages and you are reading (and thus attempting to cache) the entire 50MB database, this is about what one would expect.

I am unable to reproduce the problem.


2007-Jul-02 10:50:40 by anonymous:
That was my conclusion as well. Thanks again for the great work and unbelievably speedy responses to "bug" reports.
 
2480 code closed 2007 Jun anonymous   2007 Aug   4 4 typo in faq.tcl edit
  --- a/www/faq.tcl
  +++ b/www/faq.tcl
  @@ -202,8 +202,8 @@ faq {
     threads.  The <b>sqlite3</b> pointer must only be used in the same
     thread in which it was created by
     <a href="capi3ref.html#sqlite3_open">sqlite3_open()</a>.  If you
  -  break the rules and use an <b>sqlite3</b> in more than one thread
  -  under these circumstances, then you will likely corrupt some
  +  break the rules and use an <b>sqlite3</b> pointer in more than one
  +  thread under these circumstances, then you will likely corrupt some
     internal data structures resulting in a crash.</p>

     <p>Under UNIX, you should not carry an open SQLite database across
 
2479 code active 2007 Jun anonymous   2007 Jun   1 1 WinCE regression on some systems. Any db open fails. edit
Because Windows CE is a modular system, meaning many parts of it can be optionally ommited by the system builder, some don't include the CP_UTF8 conversion algorithms for MultiByteToWideChar and family. I believe Windows 95 and early 98 systems can also lack this encoding if not updated with a later Internet Explorer version.

Solution is to just use the sqlite internal functions that already know how to do the same thing.

Attached is an untested patch to os_win.c (I don't have a windows machine nor a cross-compiler set up) to show where the problem is and a possible (sub-optimal) solution.

I believe the right thing to do would be to just drop the utf8ToUnicode and unicodeToUtf8 functions, add the sqlite3Utf8to16 equivalent to utf.c and use them instead.

~Nuno Lucas

2007-Jun-29 14:54:11 by anonymous:
The title is wrong. It should say "Any db open using the UTF-8 API", as using the open16 API will work.
 
2478 code closed 2007 Jun anonymous   2007 Jun   2 2 LIMIT and OFFSET abbreviation to LIMIT x,y bug! edit
Statement taken from Apress: The Definitive Guide to SQLite:

SELECT * FROM foods WHERE name LIKE 'B%' ORDER BY type_id DESC, name LIMIT 1 OFFSET 2;

can be expressed equivalently with

SELECT * FROM foods WHERE name LIKE 'B%' ORDER BY type_id DESC, name LIMIT 1,2;

SQLite3 3.4.0 bug - The parameters after LIMIT have been reversed in the implementation of this version of SQLite in error!... 2 tuples from the database are returned (LIMIT) and starting from the OFFSET 1 instead of the other way around, 1 tuple starting at offset 2.

From the documentation for SELECT:

   Note that if the OFFSET keyword is used in the LIMIT clause, then
   the limit is the first number and the offset is the second number.
   If a comma is used instead of the OFFSET keyword, then the offset
   is the first number and the limit is the second number. This
   seeming contradition is intentional - it maximizes compatibility
   with legacy SQL database systems.

SQLite has always been this way. It's a bug in the book.


2007-Jun-29 11:49:02 by drh:
As an historical curiosity, see ticket #245.
 
2476 todo active 2007 Jun anonymous   2007 Jun   4 3 SQLite3 ignores ORDER BY clause when performing SELECT ... GROUP BY edit
I found that sqlite3 ignores the ORDER BY clause when performing SELECT ... GROUP BY ... ORDER BY ...

Table schema:

  CREATE TABLE events (
    id integer not null primary key,
    title integer
  );

Data:

id title
1 hello
2 hello

Query: SELECT title, id FROM events GROUP BY title ORDER BY id ASC;

Result:

title id
hello 2

Expected result:

title id
hello 1

Note: I don't think this should even work in the first place, because id is not a grouped column, but MySQL and SQLite doesn't seem to have a problem with it. Oracle complains.

2007-Jun-29 07:46:00 by danielk1977:
In SQL, the sorting specified by the ORDER BY clause is performed (logically) after the grouping specified by the GROUP BY clause. In this case a single row - the ORDER BY clause is redundant.

So the problem is that SQLite and MySQL are implementing the non-standard SQL extension of allowing an expression that is neither an aggregate or a part of the GROUP BY clause in the result-set of the SELECT in a different way.

 
2475 code fixed 2007 Jun anonymous   2007 Jul   1 3 Handles being left open on unix edit
On unix, I'm seeing processes that have been clone/exec'd inheriting file handles to sqlite databases that were opened in the parent process (and hence subsequently don't/can't get closed in the child process). Which ultimately means that you can't unmount a filesystem.

The solution is the FD_CLOEXEC fcntl on the open filehandle, but I'd like to get confirmation from elsewhere/highlight it to DRH/the list to make them aware of it.

Tested and fixed by the below patch (against 3.3.6).

  ==== //depot/qt/4.2/src/3rdparty/sqlite/os_unix.c#2 - /home/bking/depot/qt/4.2/src/3rdparty/sqlite/os_unix.c ====
  @@ -701,7 +701,7 @@
     OsFile **pId,
     int *pReadonly
   ){
  -  int rc;
  +  int rc, oldflags;
     unixFile f;

     CRASH_TEST_OVERRIDE(sqlite3CrashOpenReadWrite, zFilename, pId, pReadonly);
  @@ -730,6 +730,12 @@
       return SQLITE_NOMEM;
     }
     TRACE3("OPEN    %-3d %s\n", f.h, zFilename);
  +  oldflags = fcntl (f.h, F_GETFD, 0);
  +  if (oldflags >= 0)
  +  {
  +    oldflags |= FD_CLOEXEC;
  +    fcntl (f.h, F_SETFD, oldflags);
  +  }
     return allocateUnixFile(&f, pId);
   }

  @@ -749,7 +755,7 @@
   ** On failure, return SQLITE_CANTOPEN.
   */
   int sqlite3UnixOpenExclusive(const char *zFilename, OsFile **pId, int delFlag){
  -  int rc;
  +  int rc, oldflags;
     unixFile f;

     CRASH_TEST_OVERRIDE(sqlite3CrashOpenExclusive, zFilename, pId, delFlag);
  @@ -772,6 +778,12 @@
       unlink(zFilename);
     }
     TRACE3("OPEN-EX %-3d %s\n", f.h, zFilename);
  +  oldflags = fcntl (f.h, F_GETFD, 0);
  +  if (oldflags >= 0)
  +  {
  +    oldflags |= FD_CLOEXEC;
  +    fcntl (f.h, F_SETFD, oldflags);
  +  }
     return allocateUnixFile(&f, pId);
   }

  @@ -783,7 +795,7 @@
   ** On failure, return SQLITE_CANTOPEN.
   */
   int sqlite3UnixOpenReadOnly(const char *zFilename, OsFile **pId){
  -  int rc;
  +  int rc, oldflags;
     unixFile f;

     CRASH_TEST_OVERRIDE(sqlite3CrashOpenReadOnly, zFilename, pId, 0);
  @@ -800,6 +812,12 @@
       return SQLITE_NOMEM;
     }
     TRACE3("OPEN-RO %-3d %s\n", f.h, zFilename);
  +  oldflags = fcntl (f.h, F_GETFD, 0);
  +  if (oldflags >= 0)
  +  {
  +    oldflags |= FD_CLOEXEC;
  +    fcntl (f.h, F_SETFD, oldflags);
  +  }
     return allocateUnixFile(&f, pId);
   }

  @@ -823,6 +841,7 @@
     OsFile *id,
     const char *zDirname
   ){
  +  int oldflags;
     unixFile *pFile = (unixFile*)id;
     if( pFile==0 ){
       /* Do not open the directory if the corresponding file is not already
  @@ -836,6 +855,12 @@
       return SQLITE_CANTOPEN;
     }
     TRACE3("OPENDIR %-3d %s\n", pFile->dirfd, zDirname);
  +  oldflags = fcntl (pFile->dirfd, F_GETFD, 0);
  +  if (oldflags >= 0)
  +  {
  +    oldflags |= FD_CLOEXEC;
  +    fcntl (pFile->dirfd, F_SETFD, oldflags);
  +  }
     return SQLITE_OK;
   }
2007-Jun-28 23:07:33 by anonymous:
You'd need #ifdef FD_CLOEXEC around the code at the very least, since some UNIX-like OSes lack this feature.


2007-Jun-29 00:12:29 by anonymous:
it's not right to fork() and keep using sqlite3 objects handle if I don't misread the documentation.


2007-Jul-02 00:42:22 by anonymous:
it's not right to fork() and keep using sqlite3 objects handle if I don't misread the documentation.

That's true, but sqlite without this change is still leaking filehandles (and keeping files open) on a standard practice fork and exec. I discovered it because applications that weren't even using sqlite were holding files open because the parent process had open handles at the point they were created.


2007-Jul-02 21:04:18 by anonymous:
Which UNIX-like OSes lack this feature?

FD_CLOEXEC part of the Single UNIX Specification, Version 2, and POSIX: http://www.opengroup.org/onlinepubs/007908775/xsh/fcntl.h.html

 
2474 new active 2007 Jun anonymous   2007 Jun   5 4 Multiple-record comma-delineated INSERT command edit
I believe that both MySQL and DB2 support this feature. Instead of using separate commands for multiple INSERTS, you could use one command, and delineate the separate INSERT data with commas. Having support for this type of INSERT would make migrating MySQL or DB2 files to SQLite easier.

Regular INSERT method:

  INSERT INTO foo VALUES ('Title1',26,NULL);
  INSERT INTO foo VALUES ('Title2',24,NULL);
  INSERT INTO foo VALUES ('Title3',12,NULL);

Delineated INSERT method:

  INSERT INTO foo VALUES
  ('Title1',26,NULL),
  ('Title2',24,NULL),
  ('Title3',12,NULL);
2007-Jun-29 12:32:45 by anonymous:
From the parsing point of view this is a bit interesting. Imagine the multi-insert statement is 100,000 lines long. Do you parse the entire statement for correctness first and hold this entire parsed tree in memory? Or do you begin a transaction, and try to process each sub-insert row by row and rollback if there's any error? I'd think the latter would be better from both a time and memory point of view.

Actually, this multi-insert statement could be optimized to work around the sqlite slow bulk insert issue with multiple keys.

 
2473 warn active 2007 Jun anonymous   2007 Jun anonymous 3 2 sqlite memory org. on linux edit
i use sqlite on linux (2.4.35) and on FreePascal... i write an simple example code on linux_&_sqlite_&_freepascal. when working my example code on tty1, i run ps command on tty2; multiple process id showing about my example program... my example code is very simple... before v3.3.13 : no problem; but v3.3.17 and later is wrong... (excuse my poor english)
2007-Jun-28 13:10:10 by anonymous:
There is no bug mentioned in this ticket.


2007-Jun-28 14:09:36 by drh:
I think I see a hint of a bug report in the description where it says "before v3.3.13: no problem; but v3.3.17 and later is wrong". But that is not anything near enough information to isolate and fix the problem.

halukduman09: Please find a friend or coworker who speaks good English and ask that person to translate for you. We can only help you if we can understand what you are saying. Thanks.

 
2472 code closed 2007 Jun anonymous   2007 Jun   1 1 smart quotes turning question marks while storing in oracle 9i DB edit
whenever end-user copy & paste the statement from the word which contains smart quotes into the java application and submit, it stores in the oracle 9i database. But the smart quotes are getting converted into question mark (?) / reverse question mark symbol. Kindly tell me the solution. Environment: Java 2.0,Jrun 4.0, Oracle 9.2.0.1.0
Where does SQLite figure into the system?
 
2471 secure closed 2007 Jun anonymous   2007 Jun   1 1 a bug for 64 bit driver under operation system edit
Firstly,I can't find a 64bit driver just like a.so file under the specail system Unix/Linxu. Secondly,I find a drvier writen by java,but it has many problem while multi threads operate the database. Sometimes I lost all of my data.

Can you offer a 64bit driver under Unix/Linux in your website?

This is probably better discussed on the mailing list.

Any bugs in a java wrapper (which one) can be figured out there too.

 
2470 code fixed 2007 Jun anonymous   2007 Jun   1 3 subselects causing reproducable SIGSEGV edit
After upgrade to 3.4.0, I started receiving SIGSEGV on a particular sub-select query. This did not happen on 3.3.12. I have not tested versions 3.3.13 - 3.3.17.

I've trimmed the original statement down as much as I could. The following statement shows the problem:

DROP TABLE table_1; DROP TABLE table_2;
CREATE TABLE table_1 (col_10);
CREATE TABLE table_2 (
col_1, col_2, col_3, col_4, col_5,
col_6, col_7, col_8, col_9, col_10
);
SELECT col_10
FROM
(SELECT table_1.col_10 AS col_10 FROM table_1),
(SELECT table_1.col_10, table_2.col_9 AS qcol_9
 FROM table_1, table_2
 GROUP BY table_1.col_10, qcol_9
);
What I've noticed is that it takes at least 10 columns in table_2 before the bug is triggered. Remove one (unused) column and it goes away. Also, removing the second parameter in GROUP BY (eg. qcol_9) makes the bug go away.

This is on OpenBSD 4.1 (i386).

I'm not familiar with SQLite's internals, but below is a backtrace. Another thing I noticed is that bug is much harder to trigger (like 1 out of 10 times) without any malloc debugging options turned on.

See http://www.openbsd.org/cgi-bin/man.cgi?query=malloc.conf&apropos=0&sektion=0&manpath=OpenBSD+Current&arch=i386&format=html

With MALLOC_OPTIONS "AFGJP", the bug is triggered each time.

ssehic@dev-5-i386:/home/ssehic/devel$ sqlite3 test.db
SQLite version 3.4.0
Enter ".help" for instructions
sqlite> .read sigsegv.sql
Segmentation fault (core dumped)

ssehic@dev-5-i386:/home/ssehic/devel$ gdb /usr/local/bin/sqlite3 sqlite3.core
GNU gdb 6.3
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-unknown-openbsd4.1"...
Core was generated by `sqlite3'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /usr/local/lib/libsqlite3.so.9.0...done.
Loaded symbols for /usr/local/lib/libsqlite3.so.9.0
Reading symbols from /usr/lib/libreadline.so.3.0...done.
Loaded symbols for /usr/lib/libreadline.so.3.0
Reading symbols from /usr/lib/libncurses.so.10.0...done.
Loaded symbols for /usr/lib/libncurses.so.10.0
Reading symbols from /usr/lib/libc.so.40.3...done.
Loaded symbols for /usr/lib/libc.so.40.3
Reading symbols from /usr/libexec/ld.so...done.
Loaded symbols for /usr/libexec/ld.so
#0  sqlite3ValueFromExpr (pExpr=0xd0d0d0d0, enc=1 '\001', affinity=208 'Ð', ppVal=0xcfbf1618) at src/vdbemem.c:913
913       op = pExpr->op;
(gdb) bt
#0  sqlite3ValueFromExpr (pExpr=0xd0d0d0d0, enc=1 '\001', affinity=208 'Ð', ppVal=0xcfbf1618) at src/vdbemem.c:913
#1  0x07f47ef0 in sqlite3ColumnDefault (v=0x7e949000, pTab=0x854d2c80, i=8) at src/update.c:62
#2  0x07f2de40 in sqlite3ExprCodeGetColumn (v=0x7e949000, pTab=0x854d2c80, iColumn=8, iTable=2132004048) at src/expr.c:1683
#3  0x07f2df17 in sqlite3ExprCode (pParse=0xcfbf1a90, pExpr=0x7c2e7880) at src/expr.c:1733
#4  0x07f2e782 in sqlite3ExprCodeExprList (pParse=0xcfbf1a90, pList=0x7f13ccd0) at src/expr.c:2121
#5  0x07f45980 in sqlite3Select (pParse=0xcfbf1a90, p=0x7c2e7d00, eDest=8, iParm=2, pParent=0x7c2e7980, parentTab=1, pParentAgg=0xcfbf18cc,
    aff=0x0) at src/select.c:3218
#6  0x07f45cbd in sqlite3Select (pParse=0xcfbf1a90, p=0x7c2e7980, eDest=4, iParm=0, pParent=0x0, parentTab=0, pParentAgg=0x0, aff=0x0)
    at src/select.c:2961
#7  0x07f3b2ef in yy_reduce (yypParser=0x7fc1d800, yyruleno=104) at parse.y:369
#8  0x07f3ce04 in sqlite3Parser (yyp=0x7fc1d800, yymajor=1, yyminor={z = 0x1 "", dyn = 0, n = 1066002024}, pParse=0x1) at parse.c:3417
#9  0x07f469a5 in sqlite3RunParser (pParse=0xcfbf1a90,
    zSql=0x7f13c600 "SELECT\r\n    col_10\r\nFROM\r\n    (SELECT table_1.col_10 AS col_10 FROM table_1),\r\n    (\r\n        SELECT table_1.col_10, table_2.col_9 AS qcol_9\r\n        FROM table_1, table_2\r\n        GROUP BY table_1.co"..., pzErrMsg=0xcfbf1a8c) at src/tokenize.c:452
#10 0x07f3f5de in sqlite3Prepare (db=0x8693ce00,
    zSql=0x7f13c600 "SELECT\r\n    col_10\r\nFROM\r\n    (SELECT table_1.col_10 AS col_10 FROM table_1),\r\n    (\r\n        SELECT table_1.col_10, table_2.col_9 AS qcol_9\r\n        FROM table_1, table_2\r\n        GROUP BY table_1.co"..., nBytes=-1, saveSqlFlag=0,
    ppStmt=0xcfbf1bf4, pzTail=0xcfbf1bf8) at src/prepare.c:504
#11 0x07f3f6c9 in sqlite3_prepare (db=0x8693ce00,
    zSql=0x7f13c600 "SELECT\r\n    col_10\r\nFROM\r\n    (SELECT table_1.col_10 AS col_10 FROM table_1),\r\n    (\r\n        SELECT table_1.col_10, table_2.col_9 AS qcol_9\r\n        FROM table_1, table_2\r\n        GROUP BY table_1.co"..., nBytes=-1, ppStmt=0xcfbf1bf4,
    pzTail=0xcfbf1bf8) at src/prepare.c:616
#12 0x07f59ced in sqlite3_exec (db=0x8693ce00,
    zSql=0x7f13c600 "SELECT\r\n    col_10\r\nFROM\r\n    (SELECT table_1.col_10 AS col_10 FROM table_1),\r\n    (\r\n        SELECT table_1.col_10, table_2.col_9 AS qcol_9\r\n        FROM table_1, table_2\r\n        GROUP BY table_1.co"..., xCallback=0x1c001a24 <callback>,
    pArg=0xcfbf3af4, pzErrMsg=0xcfbf1c4c) at src/legacy.c:55
#13 0x1c0046a1 in process_input (p=0xcfbf3af4, in=0x24e8f7a0) at src/shell.c:1656
#14 0x1c003286 in do_meta_command (zLine=0x3c000272 "", p=0x7d2a9145) at src/shell.c:1363
#15 0x1c004851 in process_input (p=0xcfbf3af4, in=0x0) at src/shell.c:1616
#16 0x1c004d9f in main (argc=2, argv=0xcfbf4480) at src/shell.c:1991
(gdb) p pExpr->op
Cannot access memory at address 0xd0d0d0d0
(gdb) p pExpr
$2 = (Expr *) 0xd0d0d0d0
Looks like a NULL pointer deref. Hope that is enough information. Please let me know if you need anything else.

Thanks.

2007-Jun-27 21:46:10 by drh:
Though you would never know it to read the description, this turns out to be the same problem as #2445. It has already been fixed by [4122] .


2007-Jun-27 22:10:22 by anonymous:
Even though the bug was fixed, can you add this test case for this anyway, as Dan's test case in Check-in [4122] for this bug failed to crash or give any valgrind error for me with the sqlite 3.4.0 sources.

    -- no SEGV, nor any valgrind error for 3.4.0 release
    CREATE TEMP TABLE t1 (a, b, c, d, e);
    CREATE TEMP TABLE t2 (f);
    SELECT t1.e AS alias FROM t2, t1 WHERE alias = 1 ;

  -- SEGV on 3.4.0 release
  CREATE TABLE table_1 (col_10);
  CREATE TABLE table_2 (
  col_1, col_2, col_3, col_4, col_5,
  col_6, col_7, col_8, col_9, col_10
  );
  SELECT col_10
  FROM
  (SELECT table_1.col_10 AS col_10 FROM table_1),
  (SELECT table_1.col_10, table_2.col_9 AS qcol_9
   FROM table_1, table_2
   GROUP BY table_1.col_10, qcol_9
  );


2007-Jun-28 20:03:38 by anonymous:
Verified to fix the reported problem on OpenBSD 4.1 with the patch backported to 3.4.0.

Thanks guys. You certainly rock.

 
2469 build active 2007 Jun anonymous   2007 Jul   1 1 test fails on Solaris edit
I have a problem running the test suite on Solaris 9. Build was done using gcc 4.2.0. The build completes without error but many tests fail. I've created my own minimal test that exhibits the problem:

set testdir [file dirname $argv0]
source $testdir/tester.tcl

db close
file delete -force test.db test.db-journal
sqlite db test.db

do_test tdb-1 {
  execsql {
    PRAGMA auto_vacuum = 1;
    BEGIN;
    CREATE TABLE t1(a, b);
  }
  execsql {
    COMMIT;
  }
} {}
integrity_check tdb-2

finish_test
When running this test I get the following output:

tdb-1... Ok
tdb-2...
Expected: [ok]
     Got: [{*** in database main ***
List of tree roots: 2nd reference to page 1
Page 3 is never used}]
Thread-specific data deallocated properly
1 errors out of 3 tests
Failures on these tests: tdb-2
This error happens on lots of, but not all, tests.

I'm happy to do whatever is necessary to help debug this.

Thanks, Tim.

2007-Jun-27 10:44:14 by anonymous:
Further to this, it appears to be related to gcc 4.2.0. It works fine with gcc 3.4.6.


2007-Jun-28 09:54:35 by anonymous:
Further more, it doesn't appear to be specific to Solaris. The same problem occurs on Linux with gcc 4.2.0.

So I guess the subject of this ticket should be changed to "build/test problems with gcc 4.2.0".

This is probably a significant problem - the build completes find but the resultant code is broken. People may not notice this until it's too late.


2007-Jun-28 12:24:05 by drh:
I installed gcc 4.2.0 on my SuSE linux i686 desktop and built test harnesses under three different configurations:

     gcc420 -g -O0 -Wall -fstrict-aliasing
     gcc420 -g -O3 -Wall
     gcc420 -g -O3 -fstrict-aliasing -fomit-frame-pointer

The first two configurations used separate source files. The third configuration was built using the amalgamation. I ran the "quick" test under all configurations. All tests ran to completion with no errors.


2007-Jun-28 13:22:20 by anonymous:
Two ideas:

1. Compile with gcc 4.2.0 using -O0 instead of -O2 and see what happens. Disable any other optimizations you may have.

2. Run truss with full read/write buffer display on the gcc 3.4.6 compiled testfixture running your simple test case and compare its output to the gcc 4.2.0 compiled test case.


2007-Jul-01 19:00:40 by anonymous:
I've done tests with optimisation, and this appears to tickle the problem.

With no optimisation, -O, -O0, -O1 and -03 it works.

With -O2 and -Os it's broken.

I was compiling with -O2 when I submitted the initial report.

Tim.


2007-Jul-01 19:54:54 by drh:
I can reproduce the problem now on Linux when compiling as follows:

    gcc420 -g -O2 -Wall


2007-Jul-01 21:50:42 by drh:
This appears to be a bug in GCC 4.3.0. A work-around is to compile with the -fno-tree-vrp option.

GCC appears to miscompile a single loop within the logic that implements the integrity_check PRAGMA. The code that gets miscompiled is in the file vdbe.c (lines numbers added):

   4308    for(j=0; j<nRoot; j++){
   4309      aRoot[j] = pTos[-j].u.i;
   4310    }
   4311    aRoot[j] = 0;

It is line 4309 that appears to be miscompiled. The pTos pointer points to the top of the stack in virtual machine. The loop attempts to load the array with integer values that have been pushed onto the stack.

Running this code in GDB and setting a breakpoint on line 4311, we get the following output:

    Breakpoint 1, sqlite3VdbeExec (p=0x80ca708) at ../sqlite/src/vdbe.c:4311
    4311      aRoot[j] = 0;
    (gdb) print nRoot
    $1 = 2
    (gdb) print aRoot[0]@2
    $2 = {1, 1}
    (gdb) print pTos[0].u.i
    $3 = 1
    (gdb) print pTos[-1].u.i
    $4 = 2

As you can see above, two values have been pushed onto the virtual machine stack. (nRoot==2). But instead of loading each value into aRoot[], the top-most value is loaded multiple times. aRoot[] ends up holding {1, 1} instead of the correct values of {1, 2}.

I will submit a bug report to GCC shortly...


2007-Jul-01 22:09:23 by drh:
Bug reported filed with GCC:

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=32575


2007-Jul-05 13:33:09 by anonymous:
Could the test case at the top of this ticket be checked in with a comment pointing to this ticket? Other people may be experiencing this problem and not realizing it.


2007-Jul-21 10:00:56 by anonymous:
I can confirm this is still broken with gcc 4.2.1 and sqlite 3.4.1.

Tim.


2007-Jul-21 12:53:40 by drh:
The bug is in GCC, not SQLite The work-around is to compile without optimization. See the comments above.


2007-Jul-21 14:10:43 by anonymous:
The problem is that this is a compiler in widespread use, and you'll likely see randomly corrupted databases in the wild as result of this.

Are you able to create a smaller test case so that GCC will still exhibit this bug? (Assuming it is verified to be a compiler bug).


2007-Jul-21 16:12:42 by anonymous:
Yup - I know it's a GCC bug. I just wanted to note that the recent gcc 4.2.1 release doesn't fix the issue.


2007-Jul-21 18:55:33 by anonymous:
Simple way to detect if your copy of sqlite3 is broken:

  echo "CREATE TABLE t(x); pragma integrity_check;" | ./sqlite3

If it is broken it will output:

  *** in database main ***
  List of tree roots: 2nd reference to page 1
  Page 2 is never used

If fine:

  ok


2007-Jul-21 19:20:07 by anonymous:
This patch works around the problem with gcc-4.2.1 and sqlite 3.4.1 and allows "make test" to run cleanly. I'd appreciate if you'd apply it as gcc 4.2.x is a widely distributed compiler, and the default sqlite3 ./configure build will result in this bug.

Joe

RCS file: /sqlite/sqlite/src/vdbe.c,v
retrieving revision 1.636
diff -u -3 -p -r1.636 vdbe.c
--- src/vdbe.c  1 Jul 2007 21:18:40 -0000       1.636
+++ src/vdbe.c  21 Jul 2007 19:10:13 -0000
@@ -4306,7 +4306,8 @@ case OP_IntegrityCk: {
   pnErr = &p->aMem[j];
   assert( (pnErr->flags & MEM_Int)!=0 );
   for(j=0; j<nRoot; j++){
-    aRoot[j] = pTos[-j].u.i;
+    /* See Ticket #2469. Was: aRoot[j] = pTos[-j].u.i; */
+    aRoot[j] = (pTos-j)->u.i;
   }
   aRoot[j] = 0;
   popStack(&pTos, nRoot);

2007-Jul-21 20:31:01 by anonymous:
Not verified, but these debug trace lines follow the same pattern and may be problematic under gcc 4.2.x -O2.

  src/vdbe.c:          fprintf(p->trace, " si:%lld", pTos[i].u.i);
  src/vdbe.c:          fprintf(p->trace, " i:%lld", pTos[i].u.i);

This line's code appears to be generated okay under -O2:

  src/vdbe.c:  nArg = pTos[-1].u.i;


2007-Jul-23 13:33:55 by anonymous:
Two other GCC 4.2.1 -O2 bug workarounds are shown below which may help in the search for the optimization bug.

Joe.

(1)

  int ZZZ = 0; // must be a global variable

...

   for(j=0; j<nRoot; j++){
     aRoot[j] = pTos[ZZZ-j].u.i;
   }

(2)

  int ZZZ = 0; // must be a global variable

...

   for(j=ZZZ; j<nRoot; j++){
     aRoot[j] = pTos[-j].u.i;
   }
 
2468 doc fixed 2007 Jun anonymous   2007 Jun   5 5 Spelling error in documentation edit
http://www.sqlite.org/capi3ref.html#sqlite3_changes

In the last paragraph:

(This is much faster than going through and deleting individual elements form the table.)

2007-Jun-27 23:58:19 by anonymous:
When I click on the link that I originally provided, I still see the error. Will it take some time for the change to go through?


2007-Jun-28 00:16:47 by drh:
The documentation is built automatically and the website is updated when we do a release. If it were a serious error we could manually update the offending document, but this is a simple typo. I think it can wait.


2007-Jun-28 01:37:23 by anonymous:
Ah, I sort of thought that was the case. Like you said, it's not a big deal, I was just curious how things worked.
 
2467 code active 2007 Jun anonymous   2007 Jun   4 4 changes() not reporting correctly after DELETE FROM table edit
Similar to the problem reported in #2459, this was originally reported by someone else in the PHP manual for sqlite_changes(). I looked into the description, and produced a live example of the bug here:

The original bug description is here:

EDIT: I found this "bug" noted on the site while browsing random documentation. It seems that this is, in fact, just how the method was designed. Please do correct me if I am wrong. Also, does specifying "WHERE 1" in the "DELETE FROM" statement cause it to delete records individually? Documentation noting this is here:

2007-Jun-27 20:32:31 by anonymous:
yeah. delete from [table_name] drops and reconstructs the table again.
 
2466 code fixed 2007 Jun anonymous   2007 Jun   5 5 ENCODING not used in Makefile.linux-gcc edit
ENCODING isn't used (and if it was I would argue ISO8859 is a horrible default under Linux).

Patch to be attached.

 
2465 code fixed 2007 Jun anonymous   2007 Jun   4 4 64-bit/tcl regression failure with types3-1.3 edit
  types3-1.3...
  Expected: [wideInt integer]
       Got: [int integer]

from:

  do_test types3-1.3 {
    set V [expr {1+123456789012345}]
    concat [tcl_variable_type V] [execsql {SELECT typeof(:V)}]
  } {wideInt integer}

so I guess tcl on (this) 64-bit platform has 64-bit 'integers'?

(drh/dan, contact me directly if you want shell access to a 64-bit x86-64 machine)

 
2464 code fixed 2007 Jun anonymous   2007 Jun   5 4 error message not set by sqlite3_blob_write edit
Calling sqlite3_blob_write on a readonly blob returns an errorcode as expected, but does not set the error message that is retrieved by sqlite3_errmsg. "not an error" is the return value.
 
2463 todo closed 2007 Jun anonymous   2007 Jun anonymous 1 1 Crash after calling sqlite_close() if SQLITE_ENABLE_MEMORY_MANAGEMENT edit
Hi

I am using SQLITE amalgamated code to build sqlite static lib which is linked to my code.
All the sqlite queries are handled in separate thread. So when the application shuts down, sqlite_close() is called.

Following section is executed in sqlite_close()

for(j=0; j<db->nDb; j++){
    struct Db *pDb = &db->aDb[j];
    if( pDb->pBt ){
      sqlite3BtreeClose(pDb->pBt);
      pDb->pBt = 0;
      if( j!=1 ){
        pDb->pSchema = 0;
      }
    }
  }
Finally the crash appears in function

int sqlite3PagerClose(Pager *pPager){
#ifdef SQLITE_ENABLE_MEMORY_MANAGEMENT
  /* Remove the pager from the linked list of pagers starting at
  ** ThreadData.pPager if memory-management is enabled.
  */
  if( pPager==pTsd->pPager ){
    pTsd->pPager = pPager->pNext;
  }else{
    Pager *pTmp;
    for(pTmp = pTsd->pPager; pTmp->pNext!=pPager; pTmp=pTmp->pNext){}
    pTmp->pNext = pPager->pNext;
  }
#endif
Here pTsd appears invalid so assigning to pTmp creates crash. VS Debugger doesnt gives any value for pTsd, just shows it as invalid data.

Please have a look at it and let me know if there is some solution.Let me know if further information is needed

Thanks
anand

2007-Jun-26 18:16:53 by drh:
When using SQLITE_ENABLE_MEMORY_MANAGEMENT, All interaction with a single sqlite3* handle, including calling sqlite3_close() on that handle, must occur in the same thread that sqlite3_open() was called in.


2007-Jun-26 18:22:35 by anonymous:
sqlite_close() is being called on same thread from where sqlite_open() is called.
I guess this is same issue as raised in Ticket #2357


2007-Jun-26 23:46:12 by drh:
When SQLITE_ENABLE_MEMORY_MANAGEMENT is turned on, the sqlite3* handle may not be used in more than one thread. That means if you call sqlite3_open() in thread A and then call sqlite3_prepare() in thread B you have broken the rules and are likely to get this problem.

It is not a matter of calling sqlite3_close() in the same thread as sqlite3_open(). You have to call all sqlite3_*() routines in the same thread as sqlite3_open().


2007-Jun-27 00:09:57 by drh:
The FAQ have been updated to description the restriction on SQLITE_ENABLE_MEMORY_MANAGEMENT and threads. Since SQLITE_ENABLE_MEMORY_MANAGEMENT is otherwise undocumented, this is the only place I could find to put the information.
 
2462 todo closed 2007 Jun anonymous   2007 Jun   5 5 www.sqlite.org/cvstrac/tktnew suggestion edit
In an effort to increase the value of bug tickets you might consider putting a brief blurb at the top of the http://www.sqlite.org/cvstrac/tktnew html page stressing the value of reproducability of bugs in getting them fixed. Sort of like a bug report check list. For example, a large percentage of bug reports fail to include the schema, valid SQL, or neglect to mention the platform and media on which the database is stored.
2007-Jun-26 22:07:16 by drh:
This would be an enhancement request for the CVSTrac project. Please go to http://www.cvstrac.org/cvstrac/ and enter your request there.
 
2461 code closed 2007 Jun anonymous   2007 Jun   2 3 BLOB retrieval produces garbled data in UTF-16 edit
I came across this when using SQLiteJDBC; see http://groups.google.com/group/sqlitejdbc/t/e4ea318dea31c741 for the original description.

From a SQLite3 DB with UTF-16 encoding, retrieving a BLOB which contains a PNG image with the 3.2.7 CLI produces the correct content (displays

ëPNG...

for the head of the BLOB) and correct length(). In contrast the 3.4.0 CLI shows garbled binary content, but still returns the same length. SQLiteJDBC 0.3.4 now retrieves garbled content of an incorrect (greater) length. Setting up the DB with the same data, but encoding UTF-8 solves this issue, both CLI and SQLiteJDBC are now able to retrieve the BLOB as a valid PNG. This happens on a Win2k system, using the provided SQLite binaries resp. SQLiteJDBC v034 with nested (pure Java) DB engine.

2007-Jun-26 13:47:58 by danielk1977:
Can you upload a database file that exhibits the problem?

Thanks.


2007-Jun-26 14:24:30 by anonymous:
What does .dump produce for the BLOBs from the native (not nestedvm) sqlite3 commandline shell?


2007-Jun-29 12:50:35 by drh:
I'm not sure what problem is being reported here.

The APIs for retrieving BLOBs from a database appear to work fine, both in our countless tests and on the sample database in the attachments. This is clearly seen in the CLI if you do

    SELECT hex(avatar), length(avatar) FROM avatars;

When the CLI displays a value, it first converts that value to UTF-8 text. If the value is originally a BLOB in a UTF-16 database, then the BLOB is first cast as UTF-16 text then that text is converted to UTF-8. That whole conversion process has changed to become more robust and to fix bugs in between version 3.2.7 and 3.4.0. Are you saying that the current UTF16->UTF8 conversion process is incorrect? If so, can you please provide us with a well-formed UTF16 byte sequence (not a PNG image that is ill-formed UTF16 text) that exhibits the problem?

I cannot speak to the problem with SQLiteJDBC since that is outside the scope of this project.


2007-Jun-29 13:12:33 by drh:
I think I see the problem now.

Back in version 3.2.7, if you requested a BLOB as a UTF8 string out of a UTF16 database then SQLite would return the content of that blob directly. No conversions occurred. This is, arguably, incorrect behavior. In version 3.4.0, when you request a BLOB as a UTF8 string out of a UTF16 database, the BLOB is first cast to a UTF16 string, then there is a UTF16->UTF8 conversion. We believe the current behavior is correct.

Summary: The 3.2.7 behavior was bug. The 3.4.0 behavior is correct.


2007-Jun-29 13:18:34 by anonymous:
The commandline sqlite3 shell compiled with NestedVM shows the correct blob value with .dump for the database in question.

http://www.sqlite.org/contrib/download/sqlite-java-shell-3.4.0.zip?get=19

When spaces are adjusted for the new .dump INSERT format, it produces the same values as the natively compiled sqlite 3.2.2 shell. So either you're not calling the correct sequence of JDBC statements, or this SQLiteJDBC driver should use the same sequence of sqlite API instructions that sqlite/src/shell.c uses for .dump to process binary values.

 
2460 code closed 2007 Jun anonymous   2007 Jun   1 1 VC6 Compile Error edit
When 3.4.0 amalgamation codes are compiled by using MS-VC6 under "Release" Settings, the following error is reported:

sqlite3.c(25593) : fatal error C1001: INTERNAL COMPILER ERROR

Under "Debug" settings, there is no this error. The previous versions don't have this problem also.

2007-Jun-26 09:47:45 by drh:
This appears to be a duplicate of #2457.
 
2459 code active 2007 Jun anonymous   2007 Jun   4 4 changes() not reporting correctly after "failed" multiline INSERT edit
I have described this bug at http://www.php.net/manual/en/function.sqlite-exec.php#75962.
2007-Jun-25 23:37:22 by drh:
The bug reports says "If you run a multiline ... and there is a SQL error in any of the lines..." but in the accompanying example code, there are no errors. So I do not understand what the problem is.

I do not know what SQLite function the PDO method changes() is bound to. Let us assume that it is bound to sqlite3_changes(). In that case, it returns the number rows that changed in the most recent SQL statement - the last line of the multiline input. This is what SQLite is suppose to do. If you want to know the total number of changes across all the lines in your multiline input, then you have to call sqlite3_total_changes() before and after the multiline SQL and take the difference.


2007-Jun-26 00:10:08 by anonymous:
The error is in the $ins_query variable. The third INSERT is incorrectly spelled as "INSECT". This will cause an error. If you correct the spelling and run the code, changes() will return the integer result "3" to indicate that 3 records have been changed.


2007-Jun-26 00:29:56 by drh:
OK. I generated the following test case:

   CREATE TABLE t1(x INTEGER PRIMARY KEY, name CHAR(255));
   SELECT total_changes();
   INSERT INTO t1(name) VALUES('Ilia1');
   INSERT INTO t1(name) VALUES('Ilia2');
   INSECT INTO t1(name) VALUES('Ilia3');
   SELECT total_changes(), changes();
   SELECT * from t1;

Notice the changes of R->C in the last INSERT. This script appears to do the correct thing when handed in batch to sqlite3_exec() by the command-line shell.

   0
   SQL error near line 5: near "INSECT": syntax error
   2|1
   1|Ilia1
   2|Ilia2

Can you suggest revisions to this script that might induce the error? Do you know how PHP implements its "changes()" method? What is the "changes()" method suppose to show? Do you know that the "changes()" method is implemented correctly in PHP?


2007-Jun-26 05:52:57 by anonymous:
I am not using a command-line interface, and I am not using SQLite 3 (so would sqlite3_exec and my exec operate differently?). I am running these commands from PHP on a web server.

phpinfo() returns this information:

PHP Version: 5.1.6
System: SunOS snail 5.9 sun4u
Server API: Apache

I have created a working version of my initial example, as well as ported your example to my environment:

EDIT: I am aware that my port of your example produces a warning for "no such function". What I do not know is why; I was hoping that you could tell me if total_changes() is something supported only in a command line (TCL, C, C++, or other) environment, if it is not supported in SQLite 2.8.17, or if there is another reason you may be aware of.

EDIT2: I have provided an additional live example of what the server I'm using does with changes() when there is not an error. To clarify, if a multiline query contains no errors, changes() will show that multiple records have been changed, not just the one record affected by the last line of the statement. What I had expected is that, in this scenario, either changes() would report that there HAVE been rows changed, or that a multiline query with an error would not run, or that sqlite would undo the statements that DID execute in a multiline statement once it encountered an error. If there are ways that this CAN be done in SQLite 2.8.17, let me know. Otherwise, I will probably write up some logic in PHP to create the same functionality.


2007-Jun-27 00:25:10 by drh:
I did not notice before that you were talking about 2.8.17.

That version of SQLite is in maintenance mode. We will leave this ticket open for reference, but because 2.8.17 is so very very old and because this is not a serious bug, the problem will likely not be fixed anytime soon. Had this been a bug that was a security vulnerability or could lead to database corruption, we would look into it. But the policy with 2.8.17 is that minor things like this go unfixed. We feel that it is better to focus our efforts on the 3.x series.

You are encouraged to find a work-around. In this case, a good work-around seems to be to not enter invalid SQL...


2007-Jun-27 04:02:33 by anonymous:
Naturally, not entering invalid SQL would be a great idea, but I do not expect everyone to enter perfect SQL statements every time, so I want to try to anticipate possible errors and write in logic that would handle such errors. However, I will find some way to get this to work without changes() working as I desired it to.

Thank you for your help.


2007-Jun-27 11:04:11 by anonymous:
"What I had expected is that, in this scenario, either changes() would report that there HAVE been rows changed, or that a multiline query with an error would not run, or that sqlite would undo the statements that DID execute in a multiline statement once it encountered an error. If there are ways that this CAN be done in SQLite 2.8.17, let me know. Otherwise, I will probably write up some logic in PHP to create the same functionality."

I have determined that I can prevent a multiline query with an error from having its correct statements prior to the one with the error affecting the database by using "BEGIN TRANSACTION", "COMMIT TRANSACTION", and "ROLLBACK TRANSACTION" statements manually in combination with the queryExec method and PHP IF statements. There is a live example with source here:

I could also produce a way to calculate rows changed in such a transaction, by storing the number of records before and after the transaction and doing a simple subtraction. However, this would require me to not use ROLLBACK, so I do not see a benefit to writing such a function.


2007-Jun-30 07:45:12 by anonymous:
Oh, I just realized that I never mentioned what I was wanting this functionality for. I was writing an SQL command-line interface in PHP so that I could work with SQLite that was installed on a remote server. I needed a way to execute SQL commands or batch files without having to put it in a PHP page just to get an error and need to reupload everything.
 
2458 code closed 2007 Jun anonymous   2007 Jun   1 1 sqlite caching results between calls to select on an open database edit
When I open a database and I do a refresh after dropping a table and inserting the same number of row before the drop I am always returned cached data, I don't see the new inserted data?

Step to reproduce:

you will most likely have to write c/c++ code

start with an empty table / database

1. open database 2. create a new table 3. insert 5 rows 4. do a select operation 5. drop the table 6. create a new table 7. insert 5 different rows 8. do a select operation

results: I see the old cached rows!

expected results: I should see the new insert rows

This only happens if you insert the same number of rows you insert before the drop table command.

Basically my refresh functionality is broken.

If I do all the same 8 steps but on step 7 change the row count up or down by 1 or more I alwasy see the new data, as it should be.

2007-Jun-25 20:45:50 by drh:
Unable to reproduce.

Why do you think that C/C++ code is necessary to reproduce this? Have you left out some steps. Based on your description above, it seems like the following SQL script would suffice:

   CREATE TABLE t1(a);
   INSERT INTO t1 VALUES(1);
   INSERT INTO t1 VALUES(2);
   INSERT INTO t1 VALUES(3);
   INSERT INTO t1 VALUES(4);
   INSERT INTO t1 VALUES(5);
   SELECT * FROM t1;
   DROP TABLE t1;
   CREATE TABLE t2(b);
   INSERT INTO t2 VALUES(11);
   INSERT INTO t2 VALUES(12);
   INSERT INTO t2 VALUES(13);
   INSERT INTO t2 VALUES(14);
   INSERT INTO t2 VALUES(15);
   SELECT * FROM t2;


2007-Jun-25 20:55:08 by anonymous:
I have 2 processes: i) viewer gui app and ii) a console test logger.

The viewer app opens the database table then does selects from the database table each time the refresh button is hit. I do not open and close the database between selects calls in the viewer. The sql statement is however finalized and released and prepared each time using the new sqlite3_prepare16_v2 API.

In the test logger app that generates entries, I have a switch that allows me to drop the table before the inserts are done.

If the number so rows inserted between drops is different I see fresh data, otherwise I see cached (old) data. Also if I don't drop the table between inserts, then I see fresh data.


2007-Jun-25 20:59:31 by anonymous:
c/c++ code is necessary because if you use some sqlite shell tool to enter sql statements, the database might be getting open and closed between the steps.

If I force the database to open and close between refresh I see fresh data, if I don't I see cached data.


2007-Jun-25 21:04:05 by anonymous:
Please also use the same table name!


2007-Jun-25 21:13:20 by drh:
My next test uses two processes.

Process 1:

   create table t1(x);
   insert into t1 values(1);
   insert into t1 values(2);
   insert into t1 values(3);
   insert into t1 values(4);
   insert into t1 values(5);

Process 2:

   select * from t1;

Process 1:

   drop table t1;
   create table t1(x);
   insert into t1 values(6);
   insert into t1 values(7);
   insert into t1 values(8);
   insert into t1 values(9);
   insert into t1 values(10);

Process 2:

   select * from t1;

Process 2 sees 1,2,3,4,5 on the first SELECT and 6,7,8,9,10 on the second SELECT.

Still unable to reproduce. What else should I try?

Note that the command-line shell does not close or reopen the database between commands.


2007-Jun-25 21:59:20 by drh:
Thanks you for providing screenshots of the problem in your application, Rajinder. However, those screenshots are no help in isolating this problem. I do not have access to your code and so the screenshoots are useless to me in reproducing the bug. And if I cannot reproduce the problem, I cannot fix it.

Recognize that this is not necessarily a error in SQLite. There could be any number of bugs in your application which would give the same symptoms. And so until I have some kind of problem description or test code with which I can reproduce the symptoms here, I am going to assume that the bug is outside of SQLite and leave this ticket closed.


2007-Jun-25 22:10:06 by anonymous:
I understand, I'll keep you posted if I figure out how you can reproduce this problem in some other way without the benefit of having my source code at your disposal.
 
2457 build active 2007 Jun anonymous   2007 Jun drh 1 2 Build fails with internal compiler error (Windows, MS VC++ 6.0) edit
I can't say much. Using the TEA tarball for sqlite 3.4.0 and trying to build this on a windows box using MS VC++ 6.0 the compilation aborts with an 'INTERNAL COMPILER ERROR'.

I will attach the log as a remark. There are lots of warnings as well, about double/int conversion, argument int size mismatches. Most worrying is a series of warnings with negative line numbers!. Given that I am actually not even sure if the location where it fails is correct.

... Yes, when I tried to exclude the reported line (25589, amalgamation) via -DSQLITE_OMIT_INCRBLOB the compiler still crashes, but now in line 25555 of the amalgamation.

Note: v3.3.17 builds fine (again tea tarball, amalgamation).

I wonder ... Is it possible to provide a TEA tarball without amalgamation ? I would like to see if that compiles, maybe the amalgamation has become so large that it is hitting some compiler limit.

2007-Jun-25 20:04:23 by anonymous:
Ok, reducing the size of the amalgamation by actually stripping out comments does not help. While the negative line numbers in the warnings go away, ditto the warnings about terminating line number generation, it still runs into the Internal Error.


2007-Jun-25 20:22:41 by anonymous:
Even a version of gcc chokes on the amalgamation when I compile with -g -O3. Try compiling without debug information. If that still fails, you have to build sqlite directly from the sources.


2007-Jun-25 21:16:07 by anonymous:
Which version of gcc is failing? It seems to work fine here:

  gcc -I. -g -O6 -c ./sqlite3.c
  gcc -I. -g -O6 -DHAVE_READLINE=1 -c ../src/shell.c
  gcc -I. -o amalg-sqlite -g ./shell.o ./sqlite3.o -ldl -lpthread -lreadline

works fine. It might be worth filing a gcc bug if you see something radically different with gcc.


2007-Jun-25 21:18:52 by anonymous:
AK Regarding GCC. We are here not using GCC on Windows, so the comment regarding 'gcc -O3' does not apply.


2007-Jun-25 21:20:47 by drh:
The comment above about being unable to compile using gcc -g -O3 is not from the originator of this problem report. I do not have any problem compiling with gcc -g -O3 here on SuSE Linux with gcc 4.1.0. Will the person who reports problems compiling the amalgamation with gcc -g -O3 please add details, such as the operating system and the version of gcc being used?


2007-Jun-25 21:22:31 by anonymous:
AK

Using the exploded sources the compiler error was still present, in the file btree.c. Same code line as in the amalgamation, just different line numbers.

So the thinking that this is a problem of the amalgamation is a red herring. It is something deeper.

Getting all the revisions of btree.c between releases 3.3.17 and 3.40, and bisecting I find that the trouble starts for me with revision 1.382 of that file. Some functions where replaced by macros.

Creating a revers patch and applying it to revision 3.88 (in release 3.4.0) gets this revisions to compile as well.

Which means I now have a workaround, sa a variant of the revers patch can be applied to the amalgamation as well.


2007-Jun-25 21:26:36 by drh:
Reversing patch [4015] results in a performance hit. I am unwilling to fold in a performance hit for all platforms in order to work around a bug in MSVC++ 6.0. Can anybody suggest a better fix?


2007-Jun-25 21:35:55 by anonymous:
AK

My specific built of MSVC++ 6.0 is

Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 12.00.8168 for 80x86


2007-Jun-25 21:57:04 by anonymous:
Install Service pack 6 of Visual Studio 6.0. The KB article that I'm pretty sure covers this bug is:

http://support.microsoft.com/kb/890892


2007-Jun-25 22:16:32 by anonymous:
After hunting down and installing VC6 ServicePack 6 the compiler reports:

Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 12.00.8804 for 80x86

So, it has changed. Even so, the internal compiler error is still present.


2007-Jun-25 22:33:38 by anonymous:
Reboot of the machine is no help.


2007-Jun-25 22:38:18 by anonymous:
VC 6.0 - what compile flags are you using?


2007-Jun-25 23:11:59 by anonymous:
I thought that cygwin gcc-3.4.4 fails with -O3 -g, but it's just 1800 warning messages. No error, as it turns out.

$ gcc -I`pwd` -O3 -g sqlite3.c src/shell.c -o sqlite3

  /cygdrive/c/tmp/cc2D7Vgb.s: Assembler messages:
  /cygdrive/c/tmp/cc2D7Vgb.s:30139: Warning: .stabn: description field '103ff' too big, try a different debug format
  /cygdrive/c/tmp/cc2D7Vgb.s:30145: Warning: .stabn: description field '103fa' too big, try a different debug format
  /cygdrive/c/tmp/cc2D7Vgb.s:30170: Warning: .stabn: description field '103fa' too big, try a different debug format
  /cygdrive/c/tmp/cc2D7Vgb.s:30174: Warning: .stabn: description field '103fc' too big, try a different debug format
  /cygdrive/c/tmp/cc2D7Vgb.s:30184: Warning: .stabn: description field '103fd' too big, try a different debug format
  /cygdrive/c/tmp/cc2D7Vgb.s:30190: Warning: .stabn: description field '103fe' too big, try a different debug format
  /cygdrive/c/tmp/cc2D7Vgb.s:30194: Warning: .stabn: description field
  ...1800 more lines of the same...


2007-Jun-25 23:25:41 by anonymous:
AK Compile Flags: -O2 -W2 -MD.

According to a poster on the sqlite ML the btree.c can be compiled using -Ow is instead of O2.

Went through the options and O1, O2, Ox, Og all run into the error, the others don't.


2007-Jun-26 02:00:23 by anonymous:
FYI, MinGW gcc version 3.4.2 also produces the same 1800 warnings when you compile with -O3 -g. Still produces an object file okay, it's just annoying. No warning when only -O3 used.

  gcc -c -I. -I.. -g -O3 sqlite3.c

  c:\tmp/ccuuaaaa.s: Assembler messages:
c:\tmp/ccuuaaaa.s:31474: Warning: .stabn: description field '1001c' too big, try a different debug format
c:\tmp/ccuuaaaa.s:31480: Warning: .stabn: description field '10017' too big, try a different debug format
c:\tmp/ccuuaaaa.s:31505: Warning: .stabn: description field '10017' too big, try a different debug format
c:\tmp/ccuuaaaa.s:31509: Warning: .stabn: description field '10019' too big, try a different debug format
c:\tmp/ccuuaaaa.s:31519: Warning: .stabn: description field '1001a' too big, try a different debug format
c:\tmp/ccuuaaaa.s:31525: Warning: .stabn: description field '1001b' too big, try a different debug format
c:\tmp/ccuuaaaa.s:31529: Warning: .stabn: description field '1001e' too big, try a different debug format
c:\tmp/ccuuaaaa.s:31615: Warning: .stabs: description field '10017' too big, try a different debug format
...1800 lines of this...
 
2456 new active 2007 Jun anonymous   2007 Jun   5 5 REQ: use index where applicable instead of full table scan edit
There are times when you need information from a table that's held entirely in an index. Would it not make sense to scan over the index in these cases as those are likely to be more densely packed and therefore faster and more cache friendly?

  sqlite> .sc
  CREATE TABLE t1 ( c1 integer, c2 text, c3 text );
  CREATE INDEX idx1 on t1(c1);

so here SELECT SUM(c1) from t1 could be satisfied by scanning over idx1, which might be a lot smaller & more dense than t1.

 
2455 build fixed 2007 Jun anonymous   2007 Jun drh 2 3 aix linking fails due to missing library edit
When trying to build sqlite 3.4.0 (TEA tarball) on our AIX build box configure was ok, compilation was ok, and then linking of the shared library failed with error:

/home/andreask/dbn/lba/night/builds/aix-rs6000/tcl/unix/ldAix /bin/ld -bhalt:4 -bM:SRE -bE:lib.exp -H512 -T512 -bnoentry -o libsqlite3.4.0.so tclsqlite3.o fts2amal.o -lpthread -lc -L/home/andreask/dbn/lba/night/builds/aix-rs6000/tcl/unix/aix-rs6000 -ltclstub8.4

noDotA="libsqlite3.4.0.so"

ld: 0711-317 ERROR: Undefined symbol: .isnan

I was able to fix this problem by adding -lm to the definition of LIBS in the Makefile (hence only severity 2, and this workaround goes into our buildsystem).

Note: I do not know how/where to puts this in the TEA buildsystem (configure.in, *.m4, etc.). For that Jeff is more likely to know the answer for.

2007-Jun-25 17:25:51 by drh:
Check-in [4103] should have already fixed this problem.
 
2454 code fixed 2007 Jun drh   2007 Jun   1 1 Leading zeros cause large integers to be convert to floating point. edit
SQLite understands this number as an integer:

   9223372036854775807

But the following numbers are understood as floating point values since the number of digits is greater than 19. The leading zeros are not ignored:

   09223372036854775807
   00000000000000000001
 
2453 code fixed 2007 Jun anonymous   2007 Jun   4 3 Can the amalgamation be made entirely static edit
I would like all the functions in the amalgamation to all be declared static as I #include sqlite3.c into my main file (as I do with my other code) which is all loaded as a shared library.

In terms of implementation, my preference would be for each currently public function to be prefixed with SQLITE_API or a similar #define that by default is empty. I would #define it to static.

The benefits of this are:

  • Even better optimization since the compiler knows the functions can't be called from anywhere else
  • Unused functions can be removed by the compiler making my dll smaller
  • It would allow multiple versions of SQLite to be used in the same process at the same time. With more and more programs and libraries using SQLite this increasingly becomes a problem. (Will the tls stuff still work correctly with more than one SQLite version in the same process?)
2007-Jun-25 11:47:28 by drh:
The amalgamation can be made completely static by applying the patch shown at the end of this comment. But that is not something we are interested in doing in the standard release at this time.

--- mksqlite3c.tcl      19 Jun 2007 15:23:48 -0000      1.12
+++ mksqlite3c.tcl      25 Jun 2007 11:43:47 -0000
@@ -121,9 +121,9 @@
   section_comment "Begin file $tail"
   set in [open $filename r]
   if {[file extension $filename]==".h"} {
-    set declpattern {^ *[a-zA-Z][a-zA-Z_0-9 ]+ \*?sqlite3[A-Z][a-zA-Z0-9]+\(}
+    set declpattern {^ *[a-zA-Z][a-zA-Z_0-9 ]+ \*?sqlite3[_A-Z][a-zA-Z0-9]+\(}
   } else {
-    set declpattern {^[a-zA-Z][a-zA-Z_0-9 ]+ \*?sqlite3[A-Z][a-zA-Z0-9]+\(}
+    set declpattern {^[a-zA-Z][a-zA-Z_0-9 ]+ \*?sqlite3[_A-Z][a-zA-Z0-9]+\(}
   }
   while {![eof $in]} {
     set line [gets $in]

2007-Jun-26 00:51:42 by drh:
Here's another idea. I have modified the amalgamation generator to put the macro SQLITE_API in front of every API call. The SQLITE_API macro is normally defined to be nothing. But you can redefine it on the compiler command line:

   -DSQLITE_API=static

In order to make the amalgamation completely static if you want.

 
2452 code closed 2007 Jun anonymous   2007 Jun drh 1 1 In Memory Query Performance edit
Hi,

When the DB is opened in Memory mode,performance of query does not improve. For table which has 10 columns of type Text and each column having 128bytes data and having a total of 10000 records. Performance is around 2400 records/sec. Any ways to improve it.

Thanks in advance.

regards ragha

This is not a bug report. Please ask questions on the mailing list.


2007-Jun-25 02:39:44 by anonymous:
Thanks.
 
2451 code closed 2007 Jun anonymous   2007 Jun   5 5 vdbe.c comment typo edit
"a P1 parameter of -1" is repeated twice. One of them is wrong.

  ** database accessed is ((P1+1)*-1). For example, a P1 parameter of -1
  ** corresponds to database 0 ("main"), a P1 of -1 is database 1 ("temp").
Thanks.
 
2450 code closed 2007 Jun anonymous   2007 Jun   1 1 Crashes and random wrong results with certain column names edit
As reported on mailing list

http://www.mail-archive.com/sqlite-users%40sqlite.org/msg25614.html

  CREATE TABLE "t a" (&q