bgcolor # Type Status Created By Subsys Changed Assigned Svr Pri Title _Description _Remarks
#e8e8bd 1905 new active 2006 Aug anonymous VDBE 2006 Aug 5 4 Get bound param as sqlite3_value? It doesn't seem to be possible to get the values, that have been bound to a prepared statement, _after_ binding them. What's the use you ask? In complex scenarios the code might do complex binds (binding a NULL or an int/text depending on application logic is a simple example). A wrapper around sqlite3_step can use the requested new function to iterate over all parameters and dump them to the debug log on errors. Looking at the vdbe stuff, this seems to be a straight forward function, that will just return a pointer to vdbe->aVar[i-1] after range checking.
#f2dcdc 1816 code active 2006 May anonymous VDBE 2006 May 1 2 Database corruption with pragma auto_vacuum We had a database created with PRAGMA auto_vacuum=1, that started returning the following message on a DELETE statement. SQL error: database disk image is malformed Running the VACUUM command and running the same DELETE statement succeeds. Running PRAGMA integrity_check on the database (before the VACUUM command is issued) results in the following output: sqlite> PRAGMA integrity_check; *** in database main *** Page 3393 is never used Page 3398 is never used Page 3400 is never used Page 3401 is never used Page 3402 is never used Page 3405 is never used Page 3406 is never used sqlite> VACUUM; sqlite> PRAGMA integrity_check; ok We tried as a temporary workaround, running PRAGMA integrity_check and, based on the result, deciding whether or not to run VACUUM, but this can consume too much time. If needed, I can send a small database that exhibits this problem. _2006-May-22 21:45:47 by drh:_ {linebreak} The database is probably not helpful. What I need to know is: *: What sequence of SQL statements do you issue to cause this to occur? *: What operating system you are using. *: Is the application multi-threaded? *: Is the problem reproducible? *: Are you using a precompiled binary or did you compile it yourself? *: Does the problem go away if you turn off autovacuum? ---- _2006-May-22 22:11:09 by anonymous:_ {linebreak} *: What sequence of SQL statements do you issue to cause this to occur? It is unknown exactly what all of the the statements are leading up to the corruption. I can send the possible statements via private e-mail. *: What operating system you are using. Windows XP Professional w/ Service Pack 2. *: Is the application multi-threaded? Yes. *: Is the problem reproducible? The corruption happens on occasion -- so far it is not known to be easily reproducable in a finite number of steps. *: Are you using a precompiled binary or did you compile it yourself? Self-compiled library. When we use the database in our application, it is contained in abstracted classes with concurrency control. *: Does the problem go away if you turn off autovacuum? We have not seen database corruption if auto_vacuum is off when the database is initially created. Is it possible to turn off auto vacuum after the database tables have been created (no when using pragma auto_vacuum, according to the docs)? ---- _2006-May-22 22:28:46 by anonymous:_ {linebreak} Rather than relying on trial and error to reproduce the bug, one technique the bug reporter might try to reproduce the problem is to take a snapshot of the database when it is in a known good state and save it somewhere and then have every process that comes into contact with the database file log every SQLite command (and pragma) complete with millisecond-resolution timestamp and process/thread ID as follows: SELECT * FROM WHATEVER; -- 2006-05-23 14:44:45.237 PID 345 Thread 0 insert into blah values(3,4,5); -- 2006-05-23 14:50:15.345 PID 345 Thread 0 update foo set v=5 where y>4; -- 2006-05-23 15:05:12.930 PID 239 Thread 0 Should the problem happen again, each command could easily be replayed in an appropriate thread in the same order from the last known "good" state, greatly increasing the chances of repeating the bug. If repeating these commands does not lead to database corruption, it is fairly likely that the bug is in your multithreaded code, and not in SQLite. Perhaps SQLite already has such a command tracing facility already. I don't know. ---- _2006-May-22 22:42:04 by anonymous:_ sqlite3_trace(); It passes all the caller-generated SQL statements to a callback (although it doesn't fill in bindings). It also outputs a lot of "internal" SQL statements (VACUUM, for example, is a collection of operations on a temp table), but you should be able to recognize that stuff as something your app would never generate.
#f2dcdc 1633 code active 2006 Jan anonymous VDBE 2006 Apr 3 3 sqlite3_step returns SQLITE_ERROR when interrupted When interrupting an execution of sqlite3_step, sqlite3_step will return a generic SQLITE_ERROR instead of the SQLITE_INTERRUPT error code I'd expect. This is because although in vdbe.c:4427 the SQLITE_INTERRUPT result code is set to the internal 'rc' field of the database connection, a plain SQLITE_ERROR is returned: vdbe_halt: if( rc ){ p->rc = rc; rc = SQLITE_ERROR; }else{ rc = SQLITE_DONE; } The SQLITE_ERROR returned is then also stored as the errCode inside the db handle by the calling function, so there's no way to find out whether the error occured due to an interrupt or some other error (since sqlite3_error() will return SQLITE_ERROR as well). I'd like to ignore interrupt errors where I call sqlite3_step, since those are not of importance to my users, but with the current scheme, I have no way of finding out whether an sqlite3_interrupt causes the error or whether it's a serious error... You get a more specific error-code (for example SQLITE_INTERRUPT) when you call sqlite3_finalize() or sqlite3_reset() on the statement. There shouldn't be any reason not to call one of these functions after sqlite3_step() returns SQLITE_ERROR, as you cannot do anything else with the statement handle at that point anyway. ---- _2006-Apr-26 11:58:01 by anonymous:_ {linebreak} Well, I get SQLITE_ERROR in sqlite3_finalize(), too after I interrupted the query, so I can't even find out at finalize time. However, it might be interesting to see the real error when stepping: in our C++ wrapper, the finalize occurs very late and we usually raise database exceptions when errors occur while stepping. I would really like to raise the proper exception (some sqlite_execution_interrupted exception) when the query was interrupted instead of raising some generic "sql error" exception... ---- _2006-Apr-26 12:24:02 by anonymous:_ {linebreak} Sorry, my previous comment was a bit too quick: I have to admit that _most_ of the time, I do in fact get SQLITE_INTERRUPTED as result from sqlite3_finalize, but every now and then, I do get SQL_ERROR. Maybe this happens when I try to interrupt just before the actual statement execution has started or when execution has just finished? ---- _2006-Jul-26 13:26:19 by anonymous:_ {linebreak} This might, of course, be related to my incorrect usage of =sqlite3_interrupt= from a different thread (Ticket #1897) ?
#e8e8bd 1741 warn active 2006 Mar anonymous VDBE 2006 Mar 5 4 unused variable with SQLITE_OMIT_UTF16 defined vdbemem.c, function sqlite3VdbeChangeEncoding(): int sqlite3VdbeChangeEncoding(Mem *pMem, int desiredEnc){
If SQLITE_OMIT_UTF16 is defined then the "rc" variable is unused and compiler (Windows Intel 7.0) emits useless warning.
#f2dcdc 1733 code active 2006 Mar anonymous VDBE 2006 Mar drh 4 3 Unaligned Access on ia64: aggregate_context ptr isn't 16-bytes aligned There is a problem on ia64 with pointer returned by sqlite3_aggregate_context function. If the size requested is less than NBFS bytes, then the pointer returned is 8 bytes aligned while every pointer returned by allocator function must be 16-bytes aligned (the specification requires that the pointer is aligned so that every basic typed can be stored there and long double is 16 bytes on Itanium). So if a user allocates, say, 24 bytes for his context, and the first member in his context happens to be a long double, he will get unaligned access exception. This will lead to performance hit on Linux and to crash on HP-UX, since no default SIGBUS handler is present on HP-UX (IIRC). _2006-Mar-27 10:37:37 by anonymous:_ {linebreak} Additional details can be found in this mailing list thread: http://thread.gmane.org/gmane.comp.db.sqlite.general/18144
#e8e8bd 1655 new active 2006 Feb anonymous VDBE 2006 Mar 4 4 Every function can have their private data like agreagates Is it possible to modify the way functions are handled in sqlite ? My idea is to allow functions to have their own private data space to save data from row to row like the agregates have, with that we can have functions that remember last row values, create counters and totalizers that return their updated values for each row. Ex:
int rc;
if( !(pMem->flags&MEM_Str) || pMem->enc==desiredEnc ){
return SQLITE_OK;
}
#ifdef SQLITE_OMIT_UTF16
return SQLITE_ERROR;
#else
...
select increment(1),* from my_table; 1|car|rose|3 2|sea|bike|7 3|flower|water|33 select sum_and_return_row_by_row(row_value_to_sum),* from my_table; 3|car|rose|3 10|sea|bike|7 43|flower|water|33 select current_row_value + last_row_value(current_row_value),* from my_table; 3|car|rose|3 10|sea|bike|7 40|flower|water|33The structure for that is already there, in fact is the same used by agregates, I was scratching the code but I could not find easily where to introduce code to push the context and pop it for functions that aren't agregates, someone know how to do that ? _2006-Feb-04 20:45:32 by anonymous:_ {linebreak}
/* ** Implementation of the increment() function */ static void incrementFunc(sqlite3_context *context, int argc, sqlite3_value **argv){ assert( argc==1 ); switch( sqlite3_value_type(argv[0]) ){ case SQLITE_INTEGER: { i64 iVal = sqlite3_value_int64(argv[0]); i64 *pi = (i64*)&context->s.zShort; *pi = *pi + iVal; sqlite3_result_int64(context, *pi); break; } } }I've tried that but context are not saved row to row, of course here I was using a hidden member (s.zShort) of the context structure that seems not to be used or damaged, ideally should have a function like "void *sqlite3_set_presistent_data(sqlite3_context *context, void* value, int size_to_be_allocated)" or something like it that will allocate memory and return the actual value stored before if any. #e8e8bd 1566 new active 2005 Dec ghaering VDBE 2005 Dec 5 4 Implement sqlite3_clone() It would be nice to have an API function int sqlite3_clone(sqlite3_stmt *src, sqlite3_stmt **dest) which efficiently creates a new clone of the src statement. This is useful when you need to run the same query with different parameters on the same database simultaneously, like when walking through a tree-like table:
create table node ( id integer primary key not null, parent integer, data );All in all, this is not needed very often, but I reckon implementing the API function would be relatively trivial. _2005-Dec-19 16:04:54 by anonymous:_ {linebreak} If you want to re-execute the same query with different parameters, you should just use sqlite3_reset() instead of sqlite3_finalize(). This will reset the query so that it is ready to be executed again. Then issue the sqlite3_bind_...() calls to bind new values to the parameters that you need to change. ---- _2005-Dec-19 16:06:47 by anonymous:_ {linebreak} Oops, sorry I missed the "simultaneously" in your post the first time I read it. #e8e8bd 1558 new active 2005 Dec anonymous VDBE 2005 Dec 3 4 Request more debugging info when SQL statements in progress I am sporadically receiving the error message "Can't commit transaction - SQL statements in progress." I believe my own code is at fault, not anything in SQLite, but I would appreciate help in diagnosing the problem in my code. SQLite obviously knows that I've lost track of one or more prepared statements that haven't run to completion, it isn't telling me *what* statements those are. I'm wondering if there is any way of getting that information. Armed with that knowledge, I can probably fix my code fairly quickly. Note: DRH did provide a workaround that involved recompiling with extra debugging information, which will probably help me in the short run. This feature request is aimed at making that debugging process easier for future developers. :-) #f2dcdc 1441 code active 2005 Sep anonymous VDBE 2005 Sep 4 4 sqlite3_clear_bindings routine missing Implementation of sqlite3_clear_bindings routine is missing. I fix it this way, but the question is if that is correct... in vdbeapi.c :
/* ** The following routine unbind (set to NULL) binded parameters */ int sqlite3_clear_bindings(sqlite3_stmt *pStmt){ Vdbe *p = (Vdbe*)pStmt; int i; for(i = 1; i <= p->nVar; i++) vdbeUnbind(p, i); return SQLITE_OK; }_2005-Sep-19 10:06:12 by anonymous:_ {linebreak} The function is in experimental.c. check the source file. #f2dcdc 1327 code active 2005 Jul ghaering VDBE 2005 Jul 2 3 UNION causes sqlite3_column_decltype to always return NULL create table test(foo bar); select foo from test union select foo from test; Normally, sqlite3_column_decltype() will return 'bar', but with the UNION, it will always return NULL. This is quite annoying for pysqlite users, because it renders the type detection useless for UNION queries (and EXCEPT and INTERSECT). #e8e8bd 1262 new active 2005 May anonymous VDBE 2005 May 5 3 5-15% performance increase for slow joins Attaching an optimization to sqlite3VdbeRecordCompare() that gives a speedup of 5-15% for certain SELECTs with join. The optimization does comparisons directly on serialized data instead of deserializing and then comparing. It also inlines sqlite3GetVarint32 partly. The way the __inline keyword is used needs to be made portable. _2005-May-22 12:52:08 by anonymous:_ {linebreak} I don't have the neccessary infrastructure to run the unit tests, so it's likely that unit tests fail since I only have done limited testing of the code. As long as it doesn't contain any fundamental logic flaws any bugs should be possible to fix. ---- _2005-May-22 19:56:29 by anonymous:_ {linebreak} Looks like negative doubles don't compare as easy as I thought. Please wait for a fix before comitting. ---- _2005-May-22 21:25:49 by anonymous:_ {linebreak} Attaching file that passes the unit tests. #f2dcdc 1158 code active 2005 Mar anonymous VDBE 2005 Mar 1 1 core dump on solaris, some sqlite functions core dumped on solaris. I tracked this back to a byte alignment problem in Vdbeint.h. I added the __attribute__ ((__aligned__(16))) to the zshort character string.
/* ** A single level of the stack or a single memory cell ** is an instance of the following structure. */ struct Mem { int i; /* Integer value */ int n; /* Number of characters in string value, including '\0' */ int flags; /* Some combination of MEM_Null, MEM_Str, MEM_Dyn, etc. */ double r; /* Real value */ char *z ; /* String value */ char zShort[NBFS] __attribute__ ((__aligned__(16))); /* Space for short strings */ }; typedef struct Mem Mem;_2005-Mar-09 22:06:47 by anonymous:_ {linebreak} I think this is a duplicate of bug #700.... simply adding the attribute seems to reslove the bug... select the sum of an integer column, to reproduce.... sqlite> select sum(seqval) from tbl; Bus Error(coredump) #e8e8bd 965 new active 2004 Oct anonymous VDBE 2004 Oct 3 3 Busy Handler not invoked for SELECT et.al. in SQLite <= 2.8.15 There exist very rare cases when sqlite_exec() can return SQLITE_BUSY but the user-provided busy handler although set with sqlite_busy_handler() was never invoked. Please consider the following patch against vdbe.c. It primarily affects SELECT statements. For PRAGMA similar code snippets might be necessary for the VDBE opcodes ReadCookie/SetCookie. --- sqlite.orig/src/vdbe.c Mon Jul 19 21:30:50 2004 +++ sqlite/src/vdbe.c Tue Oct 19 14:45:11 2004 @@ -2325,8 +2325,11 @@ */ case OP_VerifyCookie: { int aMeta[SQLITE_N_BTREE_META]; + int busy = 1; assert( pOp->p1>=0 && pOp->p1
... Program terminated with signal 10, Bus Error. ... #0 0xff33d4a4 in sumStep (context=0x2a158, argc=203556, argv=0x3ac08) at src/func.c:421 421 p->sum += sqliteAtoF(argv[0], 0); ... (gdb) list 416 static void sumStep(sqlite_func *context, int argc, const char **argv){ 417 SumCtx *p; 418 if( argc<1 ) return; 419 p = sqlite_aggregate_context(context, sizeof(*p)); 420 if( p && argv[0] ){ 421 p->sum += sqliteAtoF(argv[0], 0); 422 p->cnt++; 423 } 424 } .... (gdb) print &p.sum $5 = (double *) 0x31b24Now my basic maths (a hex 16-byte aligned number should end in 0, right?) says that somehow p.sum has become misaligned by 4 bytes. sqlite_aggregate_sum simply assigns p->pAgg to p->s.z and returns p->pAgg, which means that p->s.z is misaligned also. Further investigation makes more worrying reading:
(gdb) print *context $18 = {pFunc = 0x75736572, s = {i = 0, n = 0, flags = 16, z = 0x0, r = 3.6586602629506839e-309, zShort = '\000'Note that pAgg is 0x10 (!?) and s.z is 0. Something seriously unhappy going on there: I think it's likely there's some corruption going on due to some specific set of events which was being run. I'll rerun the exact scenario and try again. ---- _2004-May-03 02:27:25 by anonymous:_ {linebreak} i have seen this exact same problem on sparc/solaris. my core looks exactly the same. it really does look like an alignment issue. ---- _2004-Jul-12 13:09:26 by anonymous:_ {linebreak} I had this exact problem on solaris 8 with gcc3.3.4 and well, every version of sqlite over 2.5. Heres my solution, hopefully it will help someone with more time and IQ points to figure out the real problem After forcing PTR_FMT to %x in test1.c (so i can run all the tests) I changed src/vbdeInt.h #define NBFS from 32 to 15 (one less than a long double on a sparc) thus forcing all long doubles to be malloced. This allowed me to run all the tests (and my application) bus error free. 5 of the tests failed, which looks like a precision problem and seems harmless in my applications. date-1.19... Expected: [2451545.00000116] Got: [2451545.00000] date-1.20... Expected: [2451545.00000012] Got: [2451545.00000] date-1.21... Expected: [2451545.00000001] Got: [2451545.00000] expr-2.4... Expected: [0.525641025641026] Got: [0.52564102564] expr-2.5... Expected: [1.90243902439024] Got: [1.90243902439] ---- _2004-Jul-17 12:26:31 by anonymous:_ {linebreak} I found a better solution than my changing NBFS solution. from what little info i found, doubles in a structure are aligned to 8. so align zShort to 8 and it works with NBFS as 32. change PTR_FMT to %X instead of %x and it passes all the tests. #f2dcdc 575 code active 2004 Jan anonymous VDBE 2004 Mar drh 3 3 pragma (default_)temp_store implementations seems incomplete This problem is a conflict between documented behaviour and actual behaviour, and could fall in the 'Documentation' category as well. There seems to be a problem with 'pragma default_temp_store'. In pragma.c code exists to handle it, and that code stores the provided value in Cookie 5 (as VDBE instruction argument; that is the _sixth_ metadata integer, and would correspond to meta[6] in sqliteInitOne() in main.c). However, the code loading a database (the aforementioned sqliteInitOne() in main.c) never looks at that value, and the setting is ignored. (Also, Vacuum doesn't seem to copy it.) A related problem is that using the default_temp_store or temp_store pragma's doesn't work as advertised, at least not in the precompiled commandline tool sqlite.exe: you will always get the following error, even if you use the pragma at init time: SQL error: The temporary database already exists - its location cannot now be changed Trying to set the flag (the value at offset 0x50 in a database file) to 2 (attempting to force an in-memory database for temp tables) with a hex editor has only partial success: the handcrafted value is reported by _pragma default_temp_store;_ but typing _.databases_ still shows a file name for the temporary data base, and using the filemon tool (a windows file activity monitor, downloadable from www.sysinternals.com) shows that the temp file is actually accessed when giving a 'create temp table' command (not surprising, if there is no code to actually ever initialize the db->temp_store from the Cookie). If for some reason it is infeasible to circumvent the issue that the temp table will always be open before executing the pragma, I suggest changing the semantics of _pragma default_temp_store_ to _only_ change the default (as stored in the file), but _not_ change the current value. This would allow executing _pragma default_temp_store_ even while a temp table is open (though its effect will only be visible when the database is opened again). Note that this issue has a few documentation issues: *: lang.html suggests that _pragma default_temp_store_ and _pragma temp_store_ are currently working. At least in the commandline tool they aren't (I didn't make a dedicated test program to see if the problem already exists at the C-API level) *: fileformat.html doesn't document the location where the temp_store flag is stored. In fact, I consider the fact that the fifth meta value (meta[5] a.k.a. Cookie 4) is seemingly not used anywhere slightly suspicious. *: the number of metadata values is documented inconsistently in fileformat.html: in one place it mentions there are 6 values including the two leading values (which makes 4 metavalues), a bit later 9 metavalues are mentioned... #e8e8bd 611 new active 2004 Feb anonymous VDBE 2004 Feb 1 3 Remove giant SWITCH statement in VM for better portability This is not really a bug but a major drawback to the platform portability of SQLite. Specifically, this relates to the ease of support for segmented architectures such as Palm OS. The problem is that the enormous SWITCH statement in the sqliteVdbeExec() function contains more than 64K of code. In a 64K segmented architecture this exceeds the code segment limit and the ONLY workaround is to break up the code into subfunctions. This is not an easy task. At present our "workaround" involves creating a "context" data structure that holds necessary state for execution (essentially the locals of sqliteVdbeExec()) then running an elaborate AWK script that parses each CASE in the switch statement and generates a separate function for the CASE that receives the execution context structure (functions are indexed by opcode for speed). This, followed by some hand tuning is our process of porting whenever we need to pick up a new release of SQLite. Over time this has proven to be the easiest way to integrate new releases, however, it is far from optimial and because of the non-automated steps involved, bug prone. After making this change the SQLite engine runs fine on the Palm OS segmented architecture and suffers no visible performance drawbacks. If a similar approach were to be integrated directly into the SQLite codebase it would GREATLY improve the portability of the engine. _2004-Feb-16 23:05:07 by drh:_ {linebreak} I experimented with using separate functions for each opcode and found the results to be about 20% slower than using a switch on GCC for i386. So changing the code to use functions only is clearly not an option for most users. Modifying the code so that it could be compiled either way would complicate the code and it would eliminate the opportunity to use persistent local variables. If absolutely necessary, this could be tolerated. But is it really necessary? The sqliteVdbeExec() function is only about 21K in size when compiled using GCC for i386 - less than a third of the maximum size allowed by PalmOS. Why is it so much bigger on Palm? Wouldn't a better approch to this problem be to fix the compiler on the Palm so that it generated binaries of a more modest size? Surely the Palm does not require more than three times the code space as an i386. Is there a GCC implementation available that targets palm? Have you tried compiling using it? ---- _2004-Mar-24 11:22:01 by anonymous:_ {linebreak} _>Why is it so much bigger on Palm?_ The reason is not the compiler, but processor architecture. Palm's processor is a RISC (reduced instruction set computer) processor (in opposite to x86 CICS (complex instruction set compute)). That means that in general RISC code is much bigger, because action performed by one CISC instruction, takes several RISC instructions to do the same. ---- _2006-Jun-06 18:38:46 by anonymous:_ {linebreak} Perhaps the "workaround" could be implemented as a Palm-specific build script, augmented by some source-level "hints" (ala lint comments) to automate the tasks currently hand-tuned. #e8e8bd 539 new active 2003 Dec anonymous VDBE 2003 Dec 5 4 Addition of engine 'Suspension' interface For my use of SQLite I require that the engine be suspended temporarily such that other tasks can be performed. At present SQLite has the ability to be interrupted - stopping the current VM in its tracks and aborting whatever it was doing. This isn't what I required - I just need it to stop doing what it was doing and return to me. As the system is implemented as a Virtual Machine and is already capable of being suspended by in order to return its results, the suspension of the engine is relatively simple. The implementation I have used will return the code 'SQLITE_SUSPEND' from an sqlite_step() call if the operation has been suspended. The function sqlite_suspend(sqlite_vm *) is provided to set a flag which will cause the suspend to take effect. The intended use is within an environment where a co-operative environment where SQLite can run for a period before and interrupt triggers the sqlite_suspend() function. Thus the caller can know that the operation is progressing but no results have been returned before the timeout. I can create diffs, or supply other information about the information if necessary. Sadly I've only tried this with the sqlite_step() interface, because that's all I've needed it with. My examination of the code implies that it is safe to perform this suspension in this manner, and it has worked very well within my application. I don't know if it would be of use to others, but I can at least offer it back to anyone who might find a use for it. #e8e8bd 472 new active 2003 Oct anonymous VDBE 2003 Nov 5 5 [Patch] Using localtime instead UTC time to 'now' --- os.c.orig Mon Oct 6 09:23:03 2003 +++ os.c Mon Oct 6 09:29:16 2003 @@ -1605,20 +1605,35 @@ */ int sqliteOsCurrentTime(double *prNow){ #if OS_UNIX - time_t t; + time_t t, lot; + struct tm ltm; + int ltmsec; time(&t); - *prNow = t/86400.0 + 2440587.5; + /* + * to use localtime, I used localtime - gmtime. + * localtime_r, gmtime_r are thread safe. + * by iosephATuriDATsarangDATnet + */ + localtime_r(&t,<m); + lot = mktime(<m); + gmtime_r(&t, <m); + ltmsec = lot - mktime(<m); + *prNow = (t + ltmsec)/86400.0 + 2440587.5; return 0; #endif #if OS_WIN - FILETIME ft; + FILETIME ft,ft2; /* FILETIME structure is a 64-bit value representing the number of 100-nanosecond intervals since January 1, 1601 (= JD 2305813.5). */ double now; GetSystemTimeAsFileTime( &ft ); - now = ((double)ft.dwHighDateTime) * 4294967296.0; - *prNow = (now + ft.dwLowDateTime)/864000000000.0 + 2305813.5; + /* to use localtime, I used this API. + * by iosephATuriDATsarangDATnet + */ + FileTimeToLocalFileTime(&ft, &ft2); + now = ((double)ft2.dwHighDateTime) * 4294967296.0; + *prNow = (now + ft2.dwLowDateTime)/864000000000.0 + 2305813.5; #endif return 0; -- EOF -- ----------------------- This patch is local patch. I patched both OS_UNIX and OS_WIN to be thread safe. #f2dcdc 414 code active 2003 Jul anonymous VDBE 2003 Nov 3 4 sqlite_compile() and database schema changes If you compile a query using sqlite_compile(), and then change the database schema using the same db handle, SQLite doesn't detect this. The following program demonstrates: void corrupt_query(char * filename) { sqlite *db; sqlite_vm *vm; int rc; db = sqlite_open(filename, 0, 0); assert(db); sqlite_exec(db, "DROP TABLE tbl;", 0, 0, 0); sqlite_exec(db, "CREATE TABLE tbl(a, b);", 0, 0, 0); sqlite_compile(db, "SELECT * FROM tbl;", 0, &vm, 0); sqlite_exec(db, "DROP TABLE tbl;", 0, 0, 0); /* This next line causes an assert() in the memory db or an SQLITE_CORRUPT * with the real db. This is because the program stored in vm tries to * open a cursor to a table that no longer exists. */ rc = sqlite_step(vm, 0, 0, 0); printf("sqlite_step returned %d, \"%s\"\n", rc, sqlite_error_string(rc)); sqlite_close(db); } int main( int argc, char ** argv ) { corrupt_query("cookie_test.db"); corrupt_query(":memory:"); return 0; } It seems unlikely to happen in practice, but I imagine there is a series of operations you could perform using this loophole that would corrupt the database file. äääüüüü""&5 #e8e8bd 490 new active 2003 Nov anonymous VDBE 2003 Nov xdong 4 3 a few api improvements for recordset operations hi friends, i'm working on a visual user interface for sqlite under dos. sqlite works great and i will post a patch for short file name support. there are a few api enhancements needed for automatic synchronization of data in a recordset with the database. 1. please add a new format in printf (like %k) to convert a string to a valid sql identifier. (like rename field "from" to "[from]"). 2. additional information on a selection result will be great also: add the source table name for each column, and the rowid of each field coming from a table. a timestamp for each such field will also be of great importance, though i understand it is not present in the current code. this information will be repeated of course when there are several fields of the same table, and other way to have this information will be welcome too (such like querying the vdbe for each column of interest, every row). 3. please provide a structured way to query the fields information for a table, index, view or view order part. 4. the database dump facility should be provided as an atomic routine in the utils library. best regards, alex #e8e8bd 486 new active 2003 Oct anonymous VDBE 2003 Oct 5 4 index on expressions (general solution to case-insensitive problem) Hi. Just wanted to throw a suggestion for this very useful database: It would be very useful to be able to have indexes that use expressions rather than column names (and have the query optimizer be able to use them when it sees those expressions) Oracle has this capability and it's surprisingly general-purpose and useful for a variety of problems, so I'd like to ask that this be considered for a sqlite wish-list (not that I want to try to make sqlite into oracle or use 'oracle has this' as a justification for any feature:-). For example, a frequent request or requirement is to search fields case-insensitively. However, most such requirements also require that the field be displayed in the original case. CREATE INDEX upper_ix ON employees (UPPER(last_name)); makes for an easy solution (though one might want a FOLD() function to correctly deal with internationalization, but that's another subject) Thanks. Mark. #e8e8bd 425 new active 2003 Aug anonymous VDBE 2003 Aug 5 3 Datetime enhancements: fractional seconds, addmonth This code adds support for fractional seconds in the datetime functions. From 0 to 6 digits are supported on creation of timestamp or time strings; up to 12 digits are read from them. YYYY-MM-DD HH:MM:SS.SSSSSSS HH:MM:SS.SSSSSSS The functions TIMESTAMP TIME and SECOND have been modified to accept an optional second argument to specify the number of fractional digits to be produced in the string result. The JD is rounded for the specified precision. A new function ADDMONTHS is also included. It takes a datetime and an integer, and increments the datetime by the number of months specified. The number can be negative, and can be greater than 12. To add years, do ADDMONTHS(, "\020\000\000\000\000\000\002¡\230", '\000' }, pAgg = 0x10, isError = 0 '\000', isStep = 0 '\000', cnt = 172464}
biblioteca=> CREATE TABLE test ( Name VARCHAR (40), UID INTEGER, DeviceID VARCHAR (64), PRIMARY KEY (Name, UID, DeviceID) ); NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "test_pkey" for table "test" CREATE TABLE biblioteca=> insert into test(Name, DeviceID) values ('Michael', 'test'); ERROR: null value in column "uid" violates not-null constraintMySQL:
mysql> CREATE TABLE test ( Name VARCHAR (40), UID INTEGER, DeviceID VARCHAR (64), PRIMARY KEY (Name, UID, DeviceID) ); Query OK, 0 rows affected (0.07 sec) mysql> insert into test(Name, DeviceID) values ('Michael', 'test'); Query OK, 1 row affected (0.11 sec) mysql> insert into test(Name, DeviceID) values ('Michael', 'test'); ERROR 1062 (23000): Duplicate entry 'Michael-0-test' for key 1that is: it MySQL is also not allowing NULL primary keys and is silently replacing NULL values with 0. Oracle:
CREATE TABLE test ( Name VARCHAR (40), UniqueID INTEGER, DeviceID VARCHAR (64), PRIMARY KEY (Name, UniqueID, DeviceID) ); insert into test(Name, DeviceID) values ('Michael', 'test'); ORA-01400: impossibile inserire NULL in ("IBOTEST"."TEST"."UNIQUEID")I hope no-one will disagree if I reopen the ticket. ---- _2006-Aug-29 11:51:00 by anonymous:_ {linebreak} behaviour with the patch: sqlite> CREATE TABLE User ( Name VARCHAR (40), ...> UID INTEGER, ...> DeviceID VARCHAR (64), ...> PRIMARY KEY (Name, UID, DeviceID) ); sqlite> INSERT INTO User (Name,DeviceID) VALUES ('Michael','Test'); SQL error: User.UID may not be NULL sqlite> INSERT INTO User (Name,DeviceID) VALUES ('Michael','Test'); SQL error: User.UID may not be NULL ---- _2006-Aug-29 13:10:37 by drh:_ {linebreak} To my great surprise, I discovered that SQLite has allowed NULL values in PRIMARY KEY columns for a very long time. This is a bug. But the bug has been in the code for so long, that we fear fixing it might break legacy applications. So for now we have chosen to merely document the fact that the bug exists and to warn developers not to depend on it since it might be fixed at some unspecified point in the future. ---- _2006-Aug-29 19:54:12 by anonymous:_ {linebreak} Can you put a SQLITE_ENABLE_NOTNULL_PRIMARYKEYS #ifdef and apply this patch to the official tree? This may solve the problems with legacy code and solve the SQL issue #e8e8bd 1939 event active 2006 Aug anonymous Unknown 2006 Aug 2 1 SQLite hangs with WHERE condition in an SELECT with joined table I have an database with 7 Tables. I wanted to make an SELECT in which i join the other Tables with an LEFT OUTER JOIN. That works but when i add an WHERE clause on an Joined Table my Application hangs. The sqlite3.exe hangs at the same query. My Database: CREATE TABLE MMBundesland (ID INTEGER, Bezeichnung varchar(255), Kennung INTEGER); CREATE TABLE MMCoords (ID INTEGER, Laenge DOUBLE, Breite DOUBLE, System varchar(50)); CREATE TABLE MMKFZ (ID INTEGER, Bezeichnung varchar(255)); CREATE TABLE MMKategorie (ID INTEGER,Bezeichnung varchar(255)); CREATE TABLE MMKreis (ID INTEGER, Bezeichnung varchar(255), Kennung INTEGER); CREATE TABLE MMOrte (ID INTEGER, CoordID INTEGER, Name varchar(255), KategorieID INTEGER, KreisID INTEGER, BundeslandID INTEGER, KfzID INTEGER, DefPLZID INTEGER, Kennung INTEGER); CREATE TABLE MMPLZ (ID INTEGER, PLZ varchar(10), CoordID INTEGER, OrtID INTEGER); The Query was: SELECT o.ID, o.Name,p.PLZ as DefPLZ,k.Bezeichnung,b.Bezeichnung as Bundesland, p.ID AS PLZID FROM MMOrte o LEFT OUTER JOIN MMPLZ p ON o.DefPLZID = p.ID OR o.ID = p.OrtID LEFT OUTER JOIN MMKategorie k ON o.KategorieID = k.ID LEFT OUTER JOIN MMBundesland b ON o.BundeslandID = b.ID LEFT OUTER JOIN MMPLZ p2 ON o.ID = p2.OrtID WHERE (p.PLZ like '72141%') ORDER BY o.Name Wenn I use: SELECT o.ID, o.Name,p.PLZ as DefPLZ,k.Bezeichnung,b.Bezeichnung as Bundesland, p.ID AS PLZID FROM *MMPLZ p* LEFT OUTER JOIN *MMOrte o* ON o.DefPLZID = p.ID OR o.ID = p.OrtID LEFT OUTER JOIN MMKategorie k ON o.KategorieID = k.ID LEFT OUTER JOIN MMBundesland b ON o.BundeslandID = b.ID LEFT OUTER JOIN MMPLZ p2 ON o.ID = p2.OrtID WHERE (p.PLZ like '72141%') ORDER BY o.Name The Query Works. It seems to work if i use in WHERE Clause one Column from the Table neer the FROM keyword. The main Problem is that SQLite hangs... I will generate my Query so that it works... _2006-Aug-25 13:32:21 by anonymous:_ {linebreak} "Hanging" is not very descriptive. Your query is working but now it either makes poor use of indexes or is doing full table scans, or possibly an unintended cross join. ---- _2007-Jan-10 16:31:41 by anonymous:_ {linebreak} I am seeing the same problem. It works on a limit of 1 and then fails on a limit of 2. ---- _2007-Jan-10 16:49:09 by drh:_ {linebreak} If you would like me to work on this, please send a reproducible test case. #e8e8bd 1906 new active 2006 Aug anonymous Unknown 2006 Aug 1 2 date.c line 417 uses non-thread safe localtime() call The date.c file uses a call to localtime() that is not threadsafe. Could it be replaced with localtime_r where supported. _2006-Aug-04 16:17:22 by anonymous:_ {linebreak} Please apply this patch to make SQLite threadsafe: Index: configure.ac =================================================================== RCS file: /sqlite/sqlite/configure.ac,v retrieving revision 1.26 diff -u -r1.26 configure.ac --- configure.ac 3 Jun 2006 18:02:18 -0000 1.26 +++ configure.ac 4 Aug 2006 16:05:21 -0000 @@ -669,6 +669,11 @@ # AC_CHECK_FUNC(usleep, [TARGET_CFLAGS="$TARGET_CFLAGS -DHAVE_USLEEP=1"]) +######### +# Figure out whether or not we have a "localtime_r()" function. +# +AC_CHECK_FUNC(localtime_r, [TARGET_CFLAGS="$TARGET_CFLAGS -DHAVE_LOCALTIME_R=1"]) + #-------------------------------------------------------------------- # Redefine fdatasync as fsync on systems that lack fdatasync #-------------------------------------------------------------------- Index: src/date.c =================================================================== RCS file: /sqlite/sqlite/src/date.c,v retrieving revision 1.54 diff -u -r1.54 date.c --- src/date.c 31 Jan 2006 20:49:13 -0000 1.54 +++ src/date.c 4 Aug 2006 16:05:21 -0000 @@ -393,7 +393,7 @@ static double localtimeOffset(DateTime *p){ DateTime x, y; time_t t; - struct tm *pTm; + struct tm tmLocaltime; x = *p; computeYMD_HMS(&x); if( x.Y<1971 || x.Y>=2038 ){ @@ -411,15 +411,20 @@ x.validJD = 0; computeJD(&x); t = (x.rJD-2440587.5)*86400.0 + 0.5; - sqlite3OsEnterMutex(); - pTm = localtime(&t); - y.Y = pTm->tm_year + 1900; - y.M = pTm->tm_mon + 1; - y.D = pTm->tm_mday; - y.h = pTm->tm_hour; - y.m = pTm->tm_min; - y.s = pTm->tm_sec; - sqlite3OsLeaveMutex(); +//#define HAVE_LOCALTIME_R +#ifdef HAVE_LOCALTIME_R + localtime_r(&t, &tmLocaltime); +#else + /* sqlite3OsEnterMutex(); not needed due to HAVE_LOCALTIME_R */ + tmLocaltime = *localtime(&t); + /* sqlite3OsLeaveMutex(); not needed due to HAVE_LOCALTIME_R */ +#endif + y.Y = tmLocaltime.tm_year + 1900; + y.M = tmLocaltime.tm_mon + 1; + y.D = tmLocaltime.tm_mday; + y.h = tmLocaltime.tm_hour; + y.m = tmLocaltime.tm_min; + y.s = tmLocaltime.tm_sec; y.validYMD = 1; y.validHMS = 1; y.validJD = 0; ---- _2006-Aug-04 17:48:38 by anonymous:_ {linebreak} The same type of autoconfig and src/date.c code changes have to be done for the not-threadsafe gmtime() function. The current implementation (even with the mutex) is not threadsafe in all cases. #f2dcdc 1901 code active 2006 Jul anonymous Unknown 2006 Jul adamd 2 2 problem in select request with a alias table I have a table with 3 columns : c0, c1 and c2 My request is: select * from (select *, 'test' as new_col from table) as tmp inner join (select 'test' as new_col) as tmp1 on tmp.new_col = tmp1.new_col; The column's name as a result of this request (sqlite 3-3.3.6) is: |tmp.table.c0|tmp.table.c1|tmp.table.c2|tmp.new_col|tmp1.new_col In sqlite 3-3.2.7, the column's name is: |c0|c1|c2|collected|new_col|new_col Before this version, my request ran on mysql, postgresql and sqlite. Now I don't have the possibility of using this request with the new sqlite version. _2006-Jul-31 10:11:59 by anonymous:_ {linebreak} sorry in In sqlite 3-3.2.7, the column's name is: |c0|c1|c2|new_col|new_col ---- _2007-Jan-08 14:52:43 by anonymous:_ {linebreak} I had a similar problem with SQLite in PHP, see my bug report here: http://bugs.php.net/bug.php?id=40064 #f2dcdc 1900 code active 2006 Jul anonymous Unknown 2006 Jul a.rottmann 1 1 CURRENT_TIMESTAMP keyword not inserting UTC date in column This is the schema for my table. create table char (player varchar(64) NOT NULL default '~', name varchar(64) NOT NULL default '~', date timestamp NOT NULL default current_timestamp) Whenever an insert is made to the table the column 'date' does get a UTC timestamp, it gets a string value 'current_timestamp'. Is my schema wrong? _2006-Jul-30 22:31:06 by anonymous:_ {linebreak} *doesnt get a UTC timestap ---- _2006-Jul-31 00:38:49 by anonymous:_ {linebreak} Works fine for me. What's the exact syntax of your INSERT statement? #f2dcdc 1890 code active 2006 Jul anonymous Unknown 2006 Jul 3 4 double quotes ("") in a query are ambiguous sqlite> SELECT "uuid" FROM objects LIMIT 1;{linebreak} b43c9cdc-0dc8-11db-9475-080020a846a9{linebreak} sqlite> SELECT "uuidx" FROM objects LIMIT 1;{linebreak} uuidx{linebreak} The objects table has a column named uuid; it does not have a column named uuidx. The behaviour of "" depends on whether the contents are a valid column name, and I cannot see when this is desirable behaviour. (I know that `` and '' are the right quotes to use, by the way - I'm just pointing out that "" can surprise people a lot and should probably be fixed or removed) _2006-Jul-13 13:39:37 by anonymous:_ {linebreak} The current behaviour is an SQL standard thing, so SQLite implements it for the sake of compatibility. Certainly most people agree with you that it's a bad thing. ---- _2006-Jul-13 16:56:05 by anonymous:_ {linebreak} Perhaps what we need then is a big fat warning in the manual :) - Peter ---- _2006-Jul-17 00:06:41 by drh:_ {linebreak} Please suggest a specific location in the documentation where I should put a warning about the use of " instead of ' and I will add it. ---- _2006-Jul-27 10:30:12 by anonymous:_ {linebreak} lang_expr.html seems like an obvious choice; perhaps also a sidenote on FAQ question 16 - Peter #e8e8bd 1869 doc active 2006 Jun anonymous Unknown 2006 Jun anonymous 4 4 Website typo http://www.sqlite.org/capi3ref.html#sqlite3_exec has this: "As an example, suppose the query result where this table:" Instead of "where," "were" should have been used. #f2dcdc 1851 code active 2006 Jun anonymous Unknown 2006 Jun 2 1 USE "ORDER BY" error on uClinux when I use "ORDER BY" function in a "select" on uClinux 2.4.24, I get a error:"SQL error or missing database" but the same program run on windows or Linux OK. _2006-Jun-16 11:20:15 by drh:_ {linebreak} This is certainly a strange error. Combined with #1850, it suggests a problem with your build, not a problem in SQLite. I have no ability to use or run uCLinux. So if the error cannot be reproduced on a desktop system, there is not much I can do to address the problem. I am afraid you are on your own on this one. ---- _2006-Jun-19 03:12:31 by anonymous:_ {linebreak} I'm just wondering that SQLite 3.2.8 runs on this uClinux system OK but SQLite 3.3.5 is error. #f2dcdc 1850 code active 2006 Jun anonymous Unknown 2006 Jun 2 1 NUMERIC data type ERROE when read on uClinux I have update some data of a tables's NUMERIC TYPE column on Windows or Linux,but when I use "select *......." to read on uClinux 2.4.24,I get the wrong value,example:the date I've written is 12.5,but readback is 2.3534826093695e -18.5(use the sqlite3_column_text API). I tried to get the value used "sqlite3_column_double" API,but the result is also wrong; But when I update some data with this column on uClinux,I can read the data right! _2006-Jun-16 01:53:24 by drh:_ {linebreak} What CPU is this happening on? SQLite assumes that floating point values are stored as IEEE 64-bit floats in the same byte order as a 64-bit integer. If your chip does not match this expectation, then floating point won't work. #e8e8bd 1833 doc active 2006 Jun anonymous Unknown 2006 Jun drh 4 3 PRAGMA legacy_file_format not documented [2922] introduces the *legacy_file_format* pragma, but it's not documented anywhere. At the very least, it should be mentioned in /sqlite/www/pragma.tcl, unless there's some better way to have a SQLite 3.3.5 (or so) generate databases usable by older versions of SQLite 3 (Debian stable, for example, ships with 3.2.1). #f2dcdc 1804 code active 2006 May anonymous Unknown 2006 May 4 3 Inconsistent value type returned by SUM when using a GROUP BY Using a schema with test table: CREATE TABLE Tbl1 (Key1 INTEGER, Num1 REAL) And test data: INSERT INTO Tbl1 (Key1,Num1) VALUES (1,5.0) The query: SELECT SUM(Tbl1.Num1) AS Num1Sum FROM Tbl1 Returns a column with the value type correctly reported as FLOAT (2). However, the query: SELECT Tbl1.Key1, SUM(Tbl1.Num1) AS Num1Sum FROM Tbl1 GROUP BY Tbl1.Key1 Returns two columns with value types INT (1) and INT (1). The SUM function is returning a different value type for these two queries when both should return FLOAT (2). This problem does not occur when any SUMmed value is not a whole number in which case, both queries return a value type of FLOAT for the SUM column. I have applied the patch from Check In 3169 (relating to #1726 and #1755) to select.c but this does not resolve the problem. _2006-May-10 09:34:11 by anonymous:_ {linebreak} I should have added that this problem was seen in a Windows CE build running on Pocket PC 2003 and built using eMbedded Visual C++ 4.0. ---- _2006-May-10 11:24:47 by anonymous:_ {linebreak} I can confirm that exactly the same behaviour is exhibited when built under Windows XP (32-bit). ---- _2006-May-10 12:40:21 by drh:_ {linebreak} The answer you are getting back is exactly correct. Why do you care what its datatype is? If you don't like the datatype, cast it. ---- _2006-May-10 13:42:14 by anonymous:_ {linebreak} For maximum compatibility with other SQL databases, both SELECT SUM(field2) FROM table and SELECT field1, SUM(field2) FROM table GROUP BY field1 should return the same data type for the SUM column. All other databases I have worked with do this. I understand that SQLite uses manifest typing but believe that it should be consistent. The problem I have is that in my query function (which takes an SQL string and returns a page of results as a 2D array of objects), I don't know whether to use sqlite3_column_double or sqlite3_column_int because I don't know what the calling function requires this column to be returned as. I am currently using the sqlite3_column_decltype call to switch which sqlite3_column_* function I use (and falling back on sqlite3_column_type when the declared type is not known e.g. for aggregate functions like SUM). If the return type of SUM is unpredictable, my calling functions can't assume returned values will be of the same type as the field that is being SUMmed (as is the case with other SQL databases). If you don't consider this to be a problem with SQLite, then I think my only option will be for calling functions to pass in an array of return types so that I always return objects of the correct type. #f2dcdc 1791 code active 2006 May anonymous Unknown 2006 May 1 1 Native threads support for BeOS BeOS ports lacks native thread support. BeOS has very powerful but lightweight threading system, being throughout multithreaded, but it differs from posix-thread ideology, thus our pthreads implementation atm looks more like flacky workaround. Ideally will be to have separate implementation for thread-support, like for Win16/32 versions. At the moment this problem caused bustage of BeOS Mozilla port, https://bugzilla.mozilla.org/show_bug.cgi?id=330340 nearest workaround might be pthreads usage, inspite its flackyness, but it also causes mess for Mozilla build/configure system, because for other parts in Mozilla we use nspr-threads, which, for BeOS, use native version _2006-Oct-27 05:48:51 by anonymous:_ {linebreak} BeOS locking extensions (using native bthreads) have been written and are included in the SQLite3 built into Mozilla Firefox. Is there some process wherein these changes might be incorporated into the SQLite tree? ---- _2006-Oct-27 12:48:11 by anonymous:_ {linebreak} Follow the example of OS/2 and propose a patch against the latest SQLite CVS that has proper #ifdef's around BeOS code so it won't break other platforms. Since you're probably the only one interested in this patch, you'll have to do the diffing/merging/testing work yourself. ---- _2006-Nov-07 03:55:36 by anonymous:_ {linebreak} Thanks for the advice. We've completed updates to code so it works with the sqlite 3.3.8 patches proposed for Firefox. Current implementation has a parallel os-specific file (os_beos.c). However, with the latest round of locking enhancements to os_unix.c, we're now wondering if it makes more sense to simply enhance this file to support BeOS locking. (yes, we. surprisingly, there is more than one BeOS user left on the planet.) :) #e8e8bd 1789 doc active 2006 May anonymous Unknown 2006 May drh 4 4 sqlite3_result_error() not adequately documented Documentation for =sqlite3_result_error()= and friends says "operation of these routines is very similar to the operation of sqlite3_bind_blob() and its cousins", but none of the bind routines are really that similar. So it's not obvious from existing documentation whether the =int= argument is a string length or, say, a =SQLITE_= error code, or a static/transient flag. A glance at /sqlite/src/func.c _suggests_ that it's a static/transient flag, but it's not entirely clear (and if it is, why isn't the signature similar to =sqlite3_result_text()=?) =sqlite3.h= and =capi3.html= should probably have a little more discussion about returning error situations from user-defined functions. c. #f2dcdc 1783 code active 2006 Apr anonymous Unknown 2006 Apr 3 2 insert times increase with growing table size (when indexed) The time needed to insert (or update) entries in a table with an index on one of the fields increases with the size of the table. For large databases inserts become very slow (which I suppose is likely the problem in ticket #1547). sqlite2 does not have this scaling problem on inserts. (Some of our queries do not scale on sqlite2 however, making its use also impossible.) ----- example code ----- package require dbi package require dbi_sqlite3 dbi_sqlite3 db db create /tmp/test.db db open /tmp/test.db db exec {create table "region" ( "id" integer not null primary key, "start" integer, "end" integer )} db exec {create index "region_index" on "region"("start")} set num 1 for {set j 1} {$j < 20} {incr j} { puts [lindex [time { db begin for {set i 1} {$i < 100000} {incr i} { set s [expr {round(rand()*1000000)}] set e [expr {round(rand()*1000000)}] db exec { insert into "region"("id","start","end") values(?,?,?) } $num $s $e incr num } db commit }] 0] } ----- timings ----- 5712186 6621934 9492997 13234978 14881322 19119044 25296162 26670866 35378986 35877042 44383517 54576510 53317621 63516664 76587973 73791188 88460462 101650099 #f2dcdc 1782 code active 2006 Apr anonymous Unknown 2006 Apr 1 1 journal file exclusion I have a long running process which opens a connection to a sqlite3 database, let's called it a.rdb. The connection is never closed during the life time of the process, and it's set to auto commit mode. Now, if I delete a.rdb, and re-create it again(the long running process is still holding a fd to the deleted file at this point), the long running process is still creating a.rdb-journal from time to time. To make it worse, if at that time, I use the sqlite3 command line to modify the database when the file a.rdb-journal exists, the file in a.rdb-journal is also played back into the new a.rdb file, which doesn't seem to be the correct behavior. Is it the intended design? Thanks, John #f2dcdc 1775 code active 2006 Apr anonymous Unknown 2006 Apr 1 1 strftime() not working in Windows Mobile 2005 The strftime() is not working in windows mobile 2005 pocket pc. I am using the beta version of visual studio 2005. _2006-Apr-17 13:10:20 by anonymous:_ {linebreak} Is it a compile/link problem or a runtime problem? ---- _2006-Apr-18 07:26:08 by anonymous:_ {linebreak} It's a runtime problem. I am not able to get the dates formated using strftime(). datetime(), date() and time() are working properly. #e8e8bd 1769 new active 2006 Apr anonymous Unknown 2006 Apr drh 5 5 Is there any djgpp port for 3-3-5 yeah, us got 3.0.7 port. while 3.3.5 is far far away from that. me have tried port meself, dispointed. did any lord get spare? using 2.8.16 port #e8e8bd 1762 build active 2006 Apr anonymous Unknown 2006 Apr 3 4 sqlite3_clear_bindings not exported in windows dll While documented sqlite3_clear_bindings is not exported in the def file for the windows dll. Therefore it is unable to link. #e8e8bd 1747 new active 2006 Apr anonymous Unknown 2006 Apr 4 4 Non descriptive error message When operating on closed db handle, sqlite3_exec returns SQLITE_MISUSE, which IMO is quite fair name. But when using: if( rc!=SQLITE_OK ){ fprintf(stderr, "SQL error: (%d) %s\n",rc, zErrMsg); sqlite3_close(db); exit (1); } Message printed is as follows: SQL error: (21) library routine called out of sequence This is a bit misleadig when starting debuging SQLite application. #f2dcdc 1742 code active 2006 Mar anonymous Unknown 2006 Mar drh 2 3 ORDER BY on more than one column causes a big slowdown Put simply, any query which contains an ORDER BY clause that sorts on more than one column incurs a strange slowdown. Running SQLite 3.3.4 on WindowsXPSP2 and on OS X 1.4.5, the behavior is similar; if the ORDER BY clause contains one column, the query is very fast; on two or more columns, it is terribly slow. _2006-Mar-28 23:58:15 by anonymous:_ {linebreak} Also worth noting that this behavior seems to start with SQLite 3.3.x; earlier versions of SQLite handle multiple ORDER BY columns much faster. ---- _2006-Mar-29 01:22:11 by anonymous:_ {linebreak} Note also that this behavior is being exhibited when sorting on *indexed* columns ---- _2006-Mar-29 01:50:38 by drh:_ {linebreak} Some examples would be helpful. ---- _2006-Mar-29 18:11:43 by anonymous:_ {linebreak} Most definitely! I will attach a sample 3.3.4 database dump, that displays this behavior. #f2dcdc 1735 code active 2006 Mar anonymous Unknown 2006 Mar 1 3 Encoding problem I use latin2 (iso-8859-2) encoding in my system. When operating on sqlite 3 I can insert data that contains national characters into a database (for example using sqlite3 console) and then when I select them back, I am given the proper result. But when I use sqlite driver from Qt4, which uses sqlite3_column_text16() to fetch data from the database, I don't get the expected result (meaning the conversion to UTF-16 probably messed things up). Now the problem can be in one of two places -- either sqlite3 console application doesn't use a proper conversion to convert from my locale encoding into its internal encoding or the database internal mechanisms mess some things up. In short: sqlite3(somelatin2string) ==> SQLITE DMBS ==> sqlite3_column_text16() ==> garbage != somelatin2string At first I thought this was Qt problem as data stored through sqlite console and retrieved from it was correct and data stored by Qt and retrieved by Qt was also correct whereas data stored by Qt and retrieved by sqlite3 console or stored by the console and retrieved by Qt was not correct. I contacted Qt support guys @ trolltech and talked about it and it looks like Qt side if fine -- it expects a UTF-16 encoded data (because it uses the function mentioned earlier) and it converts from UTF-16 to whatever encoding it needs (and vice versa). So the error is probably somewhere in the line between the console and the database itself or in the database internally. It could be that sqlite3 expects UTF-8 (or UTF-16) encoded data on input but is given ISO-8859-2 data (entered manually by me at the console). _2006-Mar-27 16:36:26 by anonymous:_ {linebreak} The console app doesn't convert from your local code page to UTF-8 (or UTF-16). ---- _2006-Mar-27 22:45:21 by anonymous:_ {linebreak} It probably should, in the documentation of sqlite a suggested method of converting databases between versions 2 and 3 is: sqlite OLD.DB .dump | sqlite3 NEW.DB Now =sqlite= outputs the data in "local" format and if =sqlite3= doesn't encode it properly, such a conversion will be invalid because the incoming data won't be utf encoded. A solution could be to do: sqlite OLD.DB .dump | iconv -f
.mode columns .headers on select 'Create two tables, with nasty column names.' as remark; create table t_a (c_a integer); create table t_b (c_a integer); select 'Create two views which each alias the column names of the above tables.' as remark; create view v_a as select c_a as pretty from t_a; create view v_b as select c_a as pretty from t_b; select 'Insert some data' as remark; insert into t_a values (1); insert into t_b values (2); select 'Notice that the views work fine by themselves.' as remark; select 'The column names are both as we asked.' as remark; select pretty from v_a; select pretty from v_b; select 'Notice that used in concert, with a join, the column name is now wrong.' as remark; select pretty from v_a union select pretty from v_b; select 'Aliasing the name of the column in the first half of the join is no help.' as remark; select pretty as pretty from v_a union select pretty from v_b; select 'Alias the name of the column in the second half of the join "fixes" the result.' as remark; select pretty from v_a union select pretty as pretty from v_b;_2005-Oct-25 03:05:27 by anonymous:_ {linebreak} same as ticket 1228 ---- _2005-Oct-26 07:59:17 by anonymous:_ {linebreak} see also #1327 #f2dcdc 1500 code active 2005 Oct anonymous Unknown 2005 Oct 4 4 MIssing file listing permission causes DB opening/creation to fail When using SQLite in PHP 5.0.5, trying to create or open a SQLite database in a directory to which the path does not have file listing (r) permissions all the way, the opening or creation fails. Please see PHP bug report #34868:
http://bugs.php.net/bug.php?id=34868
The PHP developers say that this is phenomenon is caused by the SQLite code, and suggested that I should forward this to you for possible investigation, as they merely bundle the lib with their PHP releases (and have some kind of minimal wrapper around it). It is worth noting that this problem seem to appear in other PHP situations as well (then mostly connected to the usage of PHP's "Safe Mode" or "open_basedir" restrictions), but should appear with just about any application using getcwd() under IBM AIX. The underlying reason is that getcwd() under IBM AIX fails if file listing (r) permissions are missing somewhere along the path. However, I am not sure if there are other, equal, means of determining the current working directory without using getcwd(). Best regards, Bjorn Wiberg #f2dcdc 1485 code active 2005 Oct anonymous Unknown 2005 Oct jshen 3 1 cyrilic problem(suppose Unicode as a whole, for PPC) I think there is a problem in the utf.c file. _2005-Oct-14 06:18:45 by anonymous:_ {linebreak} Pocket PC support isn't provided in the default SQLite provider. The code necessary to support the Pocket PC is at http://sourceforge.net/projects/sqlite-wince and its there that a bug report should be filed. #f2dcdc 1484 code active 2005 Oct anonymous Unknown 2005 Oct 4 4 Solaris8 awk problem /usr/bin/awk in Solaris8 doesn't have sub() function. To avoid this problem, use @AWK@ in Makefile and add AC_PROG_AWK in configure.ac. #e8e8bd 1483 doc active 2005 Oct anonymous Unknown 2005 Oct xdong 1 3 expression document:describe ERROR. http://www.sqlite.org/lang_expr.html about the "CASE" expression,"CASE [expr] ( WHEN expr THEN expr )+ [ELSE expr] END " does not work. May the right is "CASE [expr] WHEN condition THEN expr [ELSE expr] END " others:all the documents about SQLite are scattered, someone should organize them for the user.It is very useful,sometime is important than the software itself's development. infree at Beijing #e8e8bd 1480 event active 2005 Oct anonymous Unknown 2005 Oct 1 3 Wrong SQL statement crashes PHP+ApacheFaulting application Apache.exe, version 2.0.53.0, faulting module php5ts.dll, version 5.0.5.5, fault address 0x000cd0d5.
Firing a SQL statement like
SELECT A.f1, A.f2, B.f3, B.f4 FROM t1 AS INNER JOIN t2 ON A.f1 = B.f1 ORDER BY A.f1 ASC, B.f4 DESC;where the alias for t1 is missing, crashes both PHP and Apache.
<?php ob_end_flush(); $goodSql = ''; $goodSql .= 'SELECT'; $goodSql .= ' A.categoryCode,'; $goodSql .= ' A.categoryStatus,'; $goodSql .= ' A.categoryName,'; $goodSql .= ' A.categoryDescription,'; $goodSql .= ' B.documentCode,'; $goodSql .= ' B.documentStatus,'; $goodSql .= ' B.documentFile,'; $goodSql .= ' B.documentCRC32,'; $goodSql .= ' B.documentSize,'; $goodSql .= ' B.documentTitle,'; $goodSql .= ' B.documentComments,'; $goodSql .= ' B.documentMimeType,'; $goodSql .= ' B.documentAddedOn,'; $goodSql .= ' B.documentAddedBy'; $goodSql .= ' FROM categories AS A INNER JOIN documents AS B'; $goodSql .= ' ON A.categoryCode = B.categoryCode'; $goodSql .= ' WHERE A.categoryStatus = "active" AND B.documentStatus = "active"'; $goodSql .= ' ORDER BY A.categoryCode, B.documentAddedOn DESC;'; $wrongSql = ''; $wrongSql .= 'SELECT'; $wrongSql .= ' A.categoryCode,'; $wrongSql .= ' A.categoryStatus,'; $wrongSql .= ' A.categoryName,'; $wrongSql .= ' A.categoryDescription,'; $wrongSql .= ' B.documentCode,'; $wrongSql .= ' B.documentStatus,'; $wrongSql .= ' B.documentFile,'; $wrongSql .= ' B.documentCRC32,'; $wrongSql .= ' B.documentSize,'; $wrongSql .= ' B.documentTitle,'; $wrongSql .= ' B.documentComments,'; $wrongSql .= ' B.documentMimeType,'; $wrongSql .= ' B.documentAddedOn,'; $wrongSql .= ' B.documentAddedBy'; $wrongSql .= ' FROM categories AS INNER JOIN documents AS B'; $wrongSql .= ' ON A.categoryCode = B.categoryCode'; $wrongSql .= ' WHERE A.categoryStatus = "active" AND B.documentStatus = "active"'; $wrongSql .= ' ORDER BY A.categoryCode, B.documentAddedOn DESC;'; echo '<h2>Running good SQL</h2>'; flush(); test('./db/sagnedb.dat', $goodSql); flush(); echo '<h2>Running wrong SQL in 20 seconds...</h2>'; flush(); for ($i = 20; $i > 0; $i--) { echo "<small>$i...</small>"; echo str_repeat(' ', 4096); flush(); sleep(1); } test('./db/sagnedb.dat', $wrongSql); echo '<h1>It will never print this!!!!!!!!!!!!!!!!!!!!!<h2>'; flush(); function test($dbFilename, $sql) { // ini_set('sqlite.assoc_case', 2); if ($dbConn = sqlite_open($dbFilename, 0666, $dbError)) { $sql = sqlite_escape_string($sql); if ($queryResult = @sqlite_query($dbConn, $sql)) { $n = sqlite_num_fields($queryResult); echo 'cols: ' . $n . '<br />'; echo 'rows: ' . sqlite_num_rows($queryResult) . '<br />'; for ($i = 0; $i < $n; $i++) { echo 'field[' . $i . ']: ' . sqlite_field_name($queryResult, $i) . '<br />'; } echo '<br />data:<pre>'; $data = sqlite_fetch_all($queryResult, SQLITE_ASSOC); print_r($data); echo '</pre>'; } sqlite_close($dbConn); } } ?>
Sergio
_2005-Oct-11 22:19:24 by anonymous:_ {linebreak} I had no idea to whom I'd have assigned this. Sorry about that. Sergio ---- _2005-Oct-12 02:58:37 by anonymous:_ {linebreak} This is the wrong place to file this bug. Report this to the PHP folks; Unless you can reproduce the issue with sqlite alone. ---- _2005-Oct-12 15:05:42 by anonymous:_ {linebreak} The alias for t1 is not missing, it's "INNER". This is a perfectly well formed SQL statement. The problem is that SQLite is not reporting the erroneous use of the reserved word INNER as an identifier when it should. It does not do this for all reserved words as shown below. sqlite> select * from t as join; SQL error: near "join": syntax error sqlite> select * from t as "join"; sqlite> select * from t as inner; sqlite> However, I suspect that the error reported for the first select is due to the right hand table being omitted in the join, not because it recognizes that the alias (what the standard calls a correlation name) is a reserved word. The second select does not produce an error when it should, because the alias is still a reserved word even though it is delimited by double quotes. The third select does not produce an error either. ---- _2005-Oct-12 15:09:36 by anonymous:_ {linebreak} Of course this lack of error reporting is not what is causing Apache and/or PHP to crash. The erroneous queries that are accepted by SQLite execute correctly. sqlite> select * from t as inner; 1|2 2|4 sqlite> select * from t as "join"; 1|2 2|4 #f2dcdc 1461 code active 2005 Sep anonymous Unknown 2005 Sep drh 1 2 3.2.7 DLL can not deal File paths with international characters My platform:winxpsp2 chinese version,my database file under a path with chinese charater,with the dll(3.2.1) sqliteexplore works fine,if I change the dll to 3.2.7,it show sqlite error 14:can't open the file,and then I change the path fully english,it can work fine again,so I think mybe it relate to Check-in:2656 _2005-Sep-28 15:31:34 by anonymous:_ {linebreak} and more,the source version has no such problem. ---- _2005-Sep-28 16:09:04 by anonymous:_ {linebreak} I notice in os_win.c, function "sqlite3OsFileExists" use GetFileAttributesA and GetFileAttributesW,but I tried GetFileAttributes works ok. #e8e8bd 1426 event active 2005 Sep anonymous Unknown 2005 Sep drh 2 2 Problem with DETACH in 2.8.16 sqlite1.txt: create table documents (a); create index i on documents(a); sqlite2.txt: attach 'x1.dbx' as d1; attach 'x2.dbx' as d2; detach d1; Commands: sqlite x1.dbxUPDATE group_article SET parent=null WHERE group_id=?; UPDATE group_article SET parent= ( SELECT article.id FROM refs ,article WHERE refs.article_id=group_article.article_id AND reference=hash AND EXISTS (SELECT id FROM group_article WHERE group_id=?1 AND article_id=article.id) ORDER BY refs.id DESC LIMIT 1 ) WHERE group_id=?1Version 2:{linebreak}
CREATE TEMP TABLE thrd(aid UNIQUE, parent); INSERT INTO thrd(aid) SELECT article_id FROM group_article WHERE group_id=?; UPDATE thrd SET parent= ( SELECT article.id FROM refs ,article WHERE refs.article_id=thrd.aid AND reference=hash AND EXISTS (SELECT aid FROM thrd WHERE aid=article.id) ORDER BY refs.id DESC LIMIT 1 ); UPDATE group_article SET parent= ( SELECT parent FROM thrd WHERE aid=article_id ) WHERE group_id=? ;Here are partial table defs. {linebreak}
table group_article (id PRIMARY, group_id INT, article_id INT); UNIQUE index on (group_id, article_id); table refs (id PRIMARY, article_id int, reference BLOB(15)); index on (article_id) table article (id PRIMARY, hash BLOB(15)); UNIQUE index on (hash)Almost forgot, I'm running this on windows xp, compiled with mingw gcc 3.4.4. The appears to be an enhancment request - you are wanting better optimization in the SQLite. It seems to get the correct answer, you just want to get the answer faster. So I have changed this ticket to an enhancment request and set the priority very low. If you want help optimizing your query, the proper place to ask is on the mailing list. ---- _2005-Jul-24 03:50:31 by anonymous:_ {linebreak} After taking a day away from the code I finnaly found the problem. Declaring a column as UNIQUE without giving it a type means that either the index is not created or it's not used. A full table scan was being used each time in the second version. Adding INT as the type on thrd.aid fixed the performance problem. Assuming I didn't miss this in the docs, this is either a bug or a note is needed so others don't make the same mistake. #f2dcdc 1325 code active 2005 Jul anonymous Unknown 2005 Jul 1 4 sqlite3_bind_value not implemented function sqlite3_bind_value not implemented #e8e8bd 1302 event active 2005 Jun anonymous Unknown 2005 Jun 1 4 malloc error after dumping table I dumped a table then quit SQLite and received the following error while using the shell program. sqlite> .output "/participants.csv" sqlite> .dump part_info sqlite> .quit sqlite3(1593,0xa000ef98) malloc: *** Deallocation of a pointer not malloced: 0x726f7365; This could be a double free(), or free() called with the middle of an allocated block; Try setting environment variable MallocHelp to see tools to help debug Segmentation fault #e8e8bd 1300 event active 2005 Jun anonymous Unknown 2005 Jun 2 2 Apache.exe has encountered a problem and needs to close. I converted a MySql DB to SWLite. They are running on the same system and have identical data. Works on MySQL and not SQLite. SQLite gives Apache problem above. It's a very simple SQL that finds the last WPO (Wine Process Order) for each storage tank. Here is the SQL -------------------------- SELECT s.storage_id, d.from_to, d.leg_num, d.wpo_num, w.wpo_datetime, s.descr FROM wpo_dtl AS d, wpo AS w, storage AS s WHERE d.wpo_num = w.wpo_num AND d.leg_num = w.leg_num AND s.storage_id = d.storage_id AND w.wpo_datetime = ( SELECT max(wa.wpo_datetime) FROM wpo as wa, wpo_dtl as da WHERE da.wpo_num = wa.wpo_num AND da.leg_num = wa.leg_num AND da.storage_id = s.storage_id) -------------------------------------- WPO has; wpo_num int,leg_num int , wpo_datetime datetime ... WPO_DTL has; wpo_num int,leg_num int, storage_id char ... Storage has; storage_id char ... ---------------------------- Thank you for your time.. Have a great day. Dan _2005-Jun-23 19:31:49 by drh:_ {linebreak} Please include the database schema so that we can have at least an outside chance of reproducing the problem. Please also note that version 3.2.2 is current and version 2.8.16 is current in the legacy version 2 series. Your bug has likely already been fixed. #e8e8bd 1295 event active 2005 Jun anonymous Unknown 2005 Jun 1 1 SQL query involving subqueries causes reproduceable segfault in 2.8.14 The query is SELECT COUNT(g.gid) FROM kestassd_games g WHERE ( ( SELECT COUNT(m1.gid) FROM kestassd_memberships m1 WHERE m1.gid=g.gid ) = ( SELECT COUNT(m2.gid) FROM kestassd_memberships m2 WHERE m2.gid=g.gid AND m2.finalized=1 ) OR (".time()." > g.nextproc) ) AND NOT (g.mode = 0) When executed Apache 2 logs a segfault, and you disconnect with no error message. The error which Apache reports is "[Tue Jun 21 15:39:05 2005] [notice] child pid 1107 exit signal Segmentation fault (11)" Here's a zip/tar.gz of the database file which causes the segfault to occur if you run the query, along with a PHP script which will connect and execute the query: http://www.kuliukas.com/segfault.zip or http://www.kuliukas.com/segfault.tar.gz Thanks for the help, please e-mail me if you need more info. _2005-Jun-21 09:28:35 by drh:_ {linebreak} I will look at this. But because there is no easily reproducible script and the bug is against an older version of SQLite (a version to which many bug fixes have already been published), this problem will be investigated at a lower priority. I've attached a copy of the segfault.tar.gz file so that if the files on the http://www.kuliukas.com/ website go away we will still have a copy available. ---- _2005-Jun-21 14:27:27 by anonymous:_ {linebreak} Bear in mind that this is the version used in PHP 5 currently, it's quite a commonly used version I imagine, but I see your point. #f2dcdc 1111 code active 2005 Feb anonymous Unknown 2005 Jun 3 1 no such column error with a subselect as a table Execute the following script. In 3.0.8, you get the following results: line 0 line 2 in 3.1.1 beta, you get the following error: SQL error: no such column: bb.a SCRIPT: create table test1 (a int); insert into test1 values (0); insert into test1 select max(a)+1 from test1; insert into test1 select max(a)+1 from test1; create table test2 (a int, b text); insert into test2 select a,'line ' || a from test1; select test2.b from (select test1.a from test1 where a%2 = 0) as bb join test2 on bb.a = test2.a; _2005-Feb-21 17:48:20 by anonymous:_ {linebreak} Tested in 3.1.3, issue still exists ---- _2005-Jun-12 22:09:08 by drh:_ {linebreak} Workaround: select test2.b from (select test1.a as a from test1 where a%2=0) as bb join test2 on bb.a=test2.a; #f2dcdc 1264 code active 2005 May anonymous Unknown 2005 May 3 3 access() undefined on MSVC shell.c(1705) : warning C4013: 'access' undefined; assuming extern returning int Add this: #if defined(_WIN32) && defined(_MSC_VER) # include
#!/bin/sh RPM_SOURCE_DIR="/home/jms1/rpm/SOURCES" RPM_BUILD_DIR="/home/jms1/rpm/BUILD" RPM_OPT_FLAGS="-O2 -g -march=i386 -mcpu=i686" RPM_ARCH="i386" RPM_OS="linux" export RPM_SOURCE_DIR RPM_BUILD_DIR RPM_OPT_FLAGS RPM_ARCH RPM_OS RPM_DOC_DIR="/usr/share/doc" export RPM_DOC_DIR RPM_PACKAGE_NAME="sqlite" RPM_PACKAGE_VERSION="3.2.1" RPM_PACKAGE_RELEASE="1" export RPM_PACKAGE_NAME RPM_PACKAGE_VERSION RPM_PACKAGE_RELEASE RPM_BUILD_ROOT="/var/tmp/sqlite-3.2.1-root" export RPM_BUILD_ROOT set -x umask 022 cd /home/jms1/rpm/BUILD cd sqlite-3.2.1 install -d $RPM_BUILD_ROOT//usr install -d $RPM_BUILD_ROOT//usr/bin install -d $RPM_BUILD_ROOT//usr/include install -d $RPM_BUILD_ROOT//usr/lib make install prefix=$RPM_BUILD_ROOT//usr /usr/lib/rpm/brp-compress /usr/lib/rpm/brp-strip /usr/lib/rpm/brp-strip-static-archive /usr/lib/rpm/brp-strip-comment-noteand when it runs, it generates the following output:
Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.11845 + umask 022 + cd /home/jms1/rpm/BUILD + cd sqlite-3.2.1 + install -d /var/tmp/sqlite-3.2.1-root//usr + install -d /var/tmp/sqlite-3.2.1-root//usr/bin + install -d /var/tmp/sqlite-3.2.1-root//usr/include + install -d /var/tmp/sqlite-3.2.1-root//usr/lib + make install prefix=/var/tmp/sqlite-3.2.1-root//usr tclsh ./tclinstaller.tcl 3.2 can't create directory "/usr/share/tcl8.4/sqlite3": permission denied while executing "file mkdir $LIBDIR/sqlite3" (file "./tclinstaller.tcl" line 15) make: *** [tcl_install] Error 1 error: Bad exit status from /var/tmp/rpm-tmp.11845 (%install)looking at tclinstaller.tcl, it looks like it sets DESTDIR to an empty string if the variable doesn't already exist (i will admit i'm guessing here, i'm not an expert with tcl) which causes LIBDIR to point to "/usr/share/tcl8.4" (instead of "/var/tmp/sqlite-3.2.1-root/usr/share/tcl8.4", which is what one would expect.) the "rpmbuild" process, running as non-root, doesn't have permission to create this system directory, hence the error message. i don't know enough about tcl to figure out how to make DESTDIR carry into the script correctly, or even how to debug it to make sure that's what's actually going on. #f2dcdc 1214 code active 2005 Apr anonymous Unknown 2005 Apr 2 3 sqlite3_column_bytes returns 0 on p3 column with EXPLAINed selects Both functions sqlite3_column_bytes() sqlite3_column_bytes16() do not return the correct length for the p3 text column of EXPLAIN select queries. Both functions always return 0, even if the p3 column contains text. The bug can be easily reproduced with the following query: EXPLAIN SELECT 'text'; The p3 column, row 2, contains the word 'text', but the functions return 0 regardles. I have not seen this bug with non EXPLAIN queries, but it breaks code which relies on the fact that sqlite3_column_bytes always retun the correct length of the text and needs to preallocate memory accordingly. #e8e8bd 1209 doc active 2005 Apr anonymous Unknown 2005 Apr anonymous 3 5 SQLite3_Free_Table returns not #SQLITE_OK if suceeds Using the function SQLITE3_Get_Table I get back a pointer to memory where the data table is stored. After reading this table I try to free the memory under use of command SQLITE3_Free_Table (using Windows DLL V3.2.1). I expect that, if freeing succeds, the answer "#SQLITE_OK". But instead I get mostly #SQLITE_ERROR, but sometimes very big numbers. In fact I checked the memory usage of my program. When I use SQLITE_Free_Table with a wrong pointer, the answer of DLL is #SQLITE_OK while the memory usage of my program increases. Whe I use the correct pointer the answer of SQLITE3_Free_Table us #SQLITE_ERROR, but the memory usage of my program remains the same as before the SQLITE3_Get_Table command. For me it seems as if the answer for the "free" command is not correct. #e8e8bd 1208 new active 2005 Apr anonymous Unknown 2005 Apr 4 4 no version info provided in precompiled sqlite3.dll sometimes i get confused which version i'm using. it would be nice to have an easy way to view the sqlite version. _2005-Jul-14 12:43:28 by anonymous:_ {linebreak} Yes, would be great if the version information could be put in the resources of the dll and not only be accessible through the exported function! Tools like the popular Installer "InnoSetup" could recognize the version of the sqlite3.dll then and check if a newer version is already installed... #f2dcdc 1205 code active 2005 Apr anonymous Unknown 2005 Apr 3 1 accent ecu etc. in connectionstring Hi Whenever I use a accent grave or accent ecu or whatever accented characters in the connectionstring sqlite turns this into weird characters (in the file name of the database. Since my database name is dependent on user input I can't eliminate this situation... bart@arlanet.com _2005-Apr-13 12:16:08 by drh:_ {linebreak} What version of SQLite? What operating system? ---- _2005-Apr-14 12:22:32 by anonymous:_ {linebreak} Windows 2000 (dutch version) Sqlite version 3. the word privé in the connectionstring becomes privé.db3 on the disk, which causes errors afterwards when opening the database ---- _2005-Apr-14 12:23:06 by anonymous:_ {linebreak} I see something goes wrong with the accents as well on the previous remark ---- _2005-Apr-14 12:29:41 by anonymous:_ {linebreak} It really sounds to me like you're taking an accented character encoded in ISO8859-1 or some similar single-byte encoding and feeding it to something that's expecting a UTF-8 encoded string. In UTF-8, any byte that has its high bit set *must* be part of a multi-byte sequence. ---- _2005-Apr-14 14:00:56 by anonymous:_ {linebreak} After converting my string to an UTF8 encoded string, it still does the same thing.... ---- _2005-Apr-14 14:06:41 by anonymous:_ {linebreak} using sqlite.net #e8e8bd 1040 new active 2004 Dec anonymous Unknown 2005 Apr 4 4 pragma needed to enable flexibility of insert statements (esp. for awk-like filter use) In order to work around http://www.sqlite.org/cvstrac/tktview?tn=1032,3 it is possible to create insert statements manually using sed. However, those insert statements are not as flexible as they are documented to be. An error is generated if there are more data than columns. It is too slow to count the number of columns in each row to work around it. This makes the use of sqlite as an awk-like filter impossible, since you can't just define columns f1 through f9 and insert rows that have fewer columns into it. Defaulting is not done. This is inconsistent with the sqlite2 copy command, which is flexible. I propose a pragma to allow more flexibility. There is no fast workaround, but I am not assigning that severity, because in principle it is possible to much more slowly create a disk db and use .import for most purposes. #e8e8bd 1190 new active 2005 Mar anonymous Unknown 2005 Mar drh 4 4 strftime() does not Support Years with 3 or 5 digits It looks like strftime() doesn't support years before 1000 CE or after 9999 CE: sqlite> select strftime('%Y', '2004-01-01T02:34:56'); strftime('%Y', '2004-01-01T02:34:56') ------------------------------------- 2004 sqlite> select strftime('%Y', '200-01-01T02:34:56'); strftime('%Y', '200-01-01T02:34:56') ------------------------------------ [null] sqlite> select strftime('%Y', '20000-01-01T02:34:56'); strftime('%Y', '20000-01-01T02:34:56') -------------------------------------- [null] The same applies to years BCE: strftime('%Y', '-2000-01-01T02:34:56') -------------------------------------- -2000 sqlite> select strftime('%Y', '-20000-01-01T02:34:56'); strftime('%Y', '-20000-01-01T02:34:56') --------------------------------------- [null] sqlite> select strftime('%Y', '-200-01-01T02:34:56'); strftime('%Y', '-200-01-01T02:34:56') ------------------------------------- [null] One can get around the requirement for pre-1000 by using a preceding 0 (and the ISO-8601 spec may well require this; I'm not sure), but the lack of support for five or more digits in the year has no workaround. So much for my science fiction database! ;-) _2005-Mar-30 23:45:53 by anonymous:_ {linebreak} Ah, found this link, which says that years must be represented by at least four digits. So 0200 is correct and works, while 200 is not. Sorry 'bout that. Ignore that part of the report. http://www.absoluteastronomy.com/encyclopedia/I/IS/ISO_86012.htm But it still would be nice to support more than four digits for the year, so that astronomy folks can use SQLite. :-) ---- _2005-Mar-31 18:24:53 by drh:_ {linebreak} I think this is an enhancement request, not a bug report. I deliberately limited the span of years that SQLite would handle to be 1000 through 9999 as a means of detecting faulty input. I reasoned that astronomers and others who were doing date calculations outside of this range are likely using their on date/time library anyhow and so the limited range of dates that SQLite would handle was not seen as a serious handicap. Even if an astronomer wanted to use SQLite, the changes to the SQLite code base to enable a larger span of years are minor and could be done on a case by case basis. I'm not sure this is something that needs to be in the standard release. ---- _2005-Mar-31 18:32:36 by anonymous:_ {linebreak} I certainly understand the desire to detect faulty input, but since dates are simply stored in TEXT columns in SQLite (I think that's right--please correct me if I'm wrong), there isn't actually any fault input detection going on until someon uses strftime() or another date/time function. Furthermore, 10000-12-19 is not faulty input; it's perfectly valid. So is -10001-02-28 (for the archaeologists). It may be easy to patch SQLite to support such dates, but that's relatively out of the hands of mere mortals. Now if there was a compile directive to enable such support, that might be a decent compromise. Then, you'd still have the validation support, but at the same time, those who need astronomical or archaeological dates could easily get them without having to learn any C or the SQLite source code. ---- _2005-Mar-31 18:55:48 by drh:_ {linebreak} I will consider the request to support 5- or 6-digit dates. But the date algorithms used in SQLite doe not work correctly for negative Julian days (that is, dates prior to -4713-11-24T12:00:00). In fact, the date algorithms always assume the Gregorian calendar, which wasn't invented until the 16th century, was not in wide use until the 18th century. So any really old dates are suspect. Due to changing standards and politics, date/time computations can be amazingly complex, especially when you start to consider things like daylight savings time. It is not the mission of SQLite to provide a complete date/time management system. The date/time functions provided are intended to give a baseline of functionality that meets 90% of the need. Users who need more can add their own code using the sqlite3_create_function() API. The date/time functions are already one of the largest modules within SQLite and I am very hesitant to go make them even larger. ---- _2005-Mar-31 19:08:30 by anonymous:_ {linebreak} Understood, thanks for the consideration. #f2dcdc 1191 code active 2005 Mar anonymous Unknown 2005 Mar 2 3 last_insert_id() Does Not Work after Insert on View If I have a view with an INSERT trigger attached to it, and then use the view to insert a record, last_insert_rowid() does not return the ID inserted. For example, given this DDL: CREATE TABLE _simple ( id INTEGER NOT NULL PRIMARY KEY, name TEXT ); CREATE VIEW simple AS SELECT _simple.id AS id, _simple.name AS name FROM _simple; CREATE TRIGGER insert_simple INSTEAD OF INSERT ON simple FOR EACH ROW BEGIN INSERT INTO _simple (name) VALUES (NEW.name); END; This is what I get when I use the view to insert: sqlite> insert into simple (name) values ('foo'); sqlite> select last_insert_rowid(); last_insert_rowid() ------------------- 0 It does work if I insert directly into the table, of course: sqlite> insert into _simple (name) values ('foo'); sqlite> select last_insert_rowid(); last_insert_rowid() ------------------- 2 _2005-Mar-31 06:04:59 by anonymous:_ VIEWs are read-only so you can not INSERT into them. See: http://www.sqlite.org/omitted.html Regards, Bartosz. ---- _2005-Mar-31 18:12:10 by anonymous:_ {linebreak} Bartosz, Please note that triggers have been applied to the view, so you can actually insert into them. I do it all the time. For example, given my previous examples, this insert will work: sqlite> insert into simple (id, ame) values (1, 'foo'); Pretty coole, eh? See: http://www.sqlite.org/lang_createtrigger.html --Theory ---- _2005-Mar-31 18:21:08 by drh:_ {linebreak} The last_insert_rowid() routine *does* return the correct insert rowid while you are still within the trigger. Once you leave the trigger, last_insert_rowid() returns the rowid of the most recently inserted row outside of any trigger. This is by design. If last_insert_rowid() were to be responsive to inserts done by triggers, then any AFTER INSERT trigger that happened to update a logfile would overwrite the last_insert_rowid() from the actual INSERT. One could argue, I suppose that last_insert_rowid() should work for inserts performed by an INSTEAD OF trigger but not by other kinds of triggers. I will ponder that notion and might implement it if I cannot think of any objections. ---- _2005-Mar-31 18:25:14 by anonymous:_ {linebreak} Allowing it to persist past the trigger in an INSTEAD OF trigger would certainly do what I need. I think that'd make a lot of sense. All of this should probably be well-documented somewhere. I'd be happy to add a page to the wiki once you've decided how to proceed with this issue. Thanks! Theory ---- _2005-Apr-27 02:21:21 by anonymous:_ {linebreak} Have you had a chance to think more on this issue? Thanks, Theory ---- _2005-Nov-08 03:06:51 by anonymous:_ {linebreak} Just checking in on this issue again. Do you think that it's something that will be resolved, one way or the other, soon? Thanks, Theory ---- _2005-Nov-08 17:44:14 by anonymous:_ {linebreak} The instead of insert trigger can't, in general, update the last_insert_rowid value automatically and do it correctly, since a trigger may do multiple inserts into multiple tables. SQLite has no idea which rowid should be reported back outside the trigger. This is left to the user (i.e. the author of the trigger). You can get the last_insert_rowid after the appropriate insert in the trigger and then save that value into an auxillary table that is visible outside the trigger. This is especially important when your view is created from entries in several joined tables. Your instead of insert trigger must do inserts into the individual tables, and may need to use the last_insert_rowid function to link the records in the tables together correctly. It then needs to return the "master" rowid for the table. SQL doesn't have any syntax to specify which value is returned as the rowid of the instead of trigger, so SQLite doesn't do anything automatically. You need to create a sperate last_rowid table, and inside the trigger you update the value of a row in that table (possibly the only row) with the value you want to return. Then the code that does the insert into the view needs to get the value of that row instead of calling last_insert_rowid. This type of behavior is needed since there is no limit to the number of nested instead of triggers that could be executed by a single SQL statement (i.e. one instead of trigger could do an insert into another view... and so on). The current behavior isn't quite as convenient in the simplest case, but it works correctly in the more complicated general case, where simply updating the last_insert_rowid value would be wrong. ---- _2005-Nov-08 22:51:56 by anonymous:_ {linebreak} I think that there's an argument to be made that last_insert_rowid() should return the last inserted row ID, even if a trigger inserted a bunch. It should simply return the last one entered. However, I agree with your analysis. Could there perhaps be a set_insert_id() function, or some such, that the trigger could use to tell last_insert_row_id() what to return? #e8e8bd 1176 new active 2005 Mar anonymous Unknown 2005 Mar 4 3 Blocking Writes I would be nice if there was an option where a write query would block when the database was locked by another process/thread. I would prefer this over returning SQLITE_BUSY and looping...Is this planned at some point? #f2dcdc 709 code active 2004 Apr anonymous Unknown 2005 Jan drh 1 1 Unable to unregister or replace functions I believe this started with 2.8.13, as it did work previously. There appears to be a show-stopper with unregistering or replacing existing functions. Specifically, if one tries to replace (or remove by passing nulls) one of the built-in functions, for example "like" or "upper", the function does not get replaced, and is in fact still called and available. The odd thing is that if you try to replace one of the functions with an underscore, such as "change_count", it works fine! This is causing problems as we replace a lot of the existing functions, and allow users to add and replace their own functions, which are now failing. _2004-Apr-26 20:56:19 by anonymous:_ {linebreak} Alright, turns out this is due to mismatch in argument count when unregistering functions. It would be useful if we could unregister all instances of a function name, irregardless of argument counts. ---- _2004-Apr-26 23:11:43 by anonymous:_ {linebreak} Turns out there really is a bug... the problem is in the sqliteFindFunction function's matching of inexact argument counts, when being called from sqliteExprCheck with >0 argument count. This is the scenario. I override the "upper" function with my own, but first removing the old "upper", specifying 1 argument and null for the function. I then register a new "upper" with -1 for the argument count, and a valid function. When sqliteFindFunction attempts to locate the upper function, It locates the new function, but because it is registered with -1, it tries to find a better match. It then runs into the original one, and because it has a null function pointer, it fails. This causes sqliteExprCheck to try again with -1 as the count, and since that matches, it reports an error of wrong_num_args. Unfortunately this means there is no way to override an existing method (it would be good if we could just delete them, rather than override them, although I still think this behaviour with -1 is wrong). ---- _2004-Apr-26 23:29:23 by anonymous:_ {linebreak} I've applied the following patch in the sqliteFindFunction function, that I believe addresses the problem:
/* Change this if( p && !createFlag && p->xFunc==0 && p->xStep==0 ){ return 0; } */ /* To this */ if( p && !createFlag && p->xFunc==0 && p->xStep==0 ){ return pMaybe;By returning pMaybe we provide a function that will work, while returning 0 if no variable argument function was found. #e8e8bd 1059 doc active 2005 Jan anonymous Unknown 2005 Jan 5 4 Full support for IEEE 754 floating points Current situation:{linebreak} IEEE standard 754 allows for the following special cases of double values:{linebreak} _:Zero. Denoted as an exponent field of zero and a fraction field of zero{linebreak} _:Denormalized. Denoted as an all 0 exponent field with a non zero fraction field{linebreak} _:Infinity. Denoted as an all 1 exponent field and an all zero fraction field(sign bit indicates +infinity or -infinity){linebreak} _:NaN. Denoted with an all 1 exponent field and a non zero fraction field. The most significant bit fraction indicates type of NaN. SQLite converts these values to 0. Request option to store these special values. Current workaround: If only one type of special value is to be used, replace it with NULL. If multiple types are used add a type column to each floating point column indicating what is stored in the linked floating point column. _2005-Jul-25 01:46:58 by anonymous:_ {linebreak} Here is some _partial_ support for Infinity and NaN given C99 support. Both quantities are input and output correctly, and arithmetic works. I haven't defined what happens when storing these quantities on non-REAL columns. I also haven't added tests; I'm allergic to tcl. (note: gcc needs -std=gnu9x or -std=c99 to fully support Inf and NaN.) The float i/o needs some serious work; are there any non-licensing constraints on what someone can replace it with? If the patch is mangled, mail me. Jason Riedy
#include#e8e8bd 717 new active 2004 May anonymous Unknown 2004 May drh 2 2 minor fixes and port to dos attached is a diff for sqlite 2.8.13 that allows work in dos gnu environment (32 bit djgpp), even without support of long file names. it also contains minor adjacent bugfixes, in the file names handling area, and for the case of missing libreadline. please take a look, and do apply to the main source tree. i do not ask for anything, even not for personal credits. it is a very small contribution, to make your wonderful work even more popular. best regards, alex _2004-Jul-09 02:22:15 by anonymous:_ {linebreak} after feedback, mainly from hans-juergen taenzer, i have reviewed the port to dos and made better relative path support, applying to unix also. for any questions or remarks, please feel free to contact me, alex, alexbodn@012.net.il. #e8e8bd 714 new active 2004 Apr anonymous Unknown 2004 Apr drh 3 3 Change sqlite_decode_binary to return buffer size It would be nice to be able to have sqlite_decode_binary return the required buffer size, like the encode routine does. The reason is that in managed systems (we're using this in .NET) it avoids a needless allocation and then a copy. For example, without it, we need to allocate a buffer as big as the incoming one, decode, allocate one of the proper size, then copy the bytes. Otherwise, the array we return is incorrectly sized. I've made the following change... // TODO: Mark this up... was... out[i++] = c + e; if (out) out[i++] = c + e; else i++; By checking the out argument, we still get the increment count, but we don't write anywhere. #e8e8bd 712 new active 2004 Apr anonymous Unknown 2004 Apr drh 3 3 Allow temporary databases to be created in specified folder Right now temporary databases are created in a default folder, for example the Temp folder under Windows. It should be possible to customize this location by passing the "context" of the current database to the temporary OS.C function. If that function received the current database, it could parse the folder and optionally create the temporary file in the same location. #e8e8bd 710 new active 2004 Apr anonymous Unknown 2004 Apr drh 3 4 Provide the ability to pass a user-data value during sqlite_compile It would be very useful to be able to pass a user-data value during the sqlite_compile command, that could then be retrieved in a user function. This would allow context-sensitive responses by the functions, by allowing them to know what "command" is executing them. I can provide further details if required. #e8e8bd 707 new active 2004 Apr anonymous Unknown 2004 Apr drh 3 3 how to use internal functions? to name a few: 1: "SELECT @@CHECK_INTEGRITY", it will return the integrity of the database. 2: "SELECT @@DATABASE_VERSION", it will return the version of the database. 3: "SELECT @@ENGINE_VERSION", it will return the current version of the sqlite engine 4:"SELECT @@ENGINE_ENCODING", it will return the encoding of the sqlite engine. etc. #f2dcdc 698 code active 2004 Apr anonymous Unknown 2004 Apr 1 3 .mode list - not going to next line To create a comma delimited output file:{linebreak} ------------------------------------------------{linebreak} C:\SQLite>sqlite locate.db{linebreak} SQLite version 2.8.13{linebreak} Enter ".help" for instructions{linebreak} sqlite> .mode list{linebreak} sqlite> .separator ", "{linebreak} sqlite> .output data.cdf{linebreak} sqlite> select * from parts;{linebreak} sqlite> .quit{linebreak} {linebreak} That should create a text file of something like this:{linebreak} 1st rec field 1, 1st rec field 2, 1st rec field 3, 1st rec field 4{linebreak} 2nd rec field 1, 2nd rec field 2, 2nd rec field 3, 2nd rec field 4{linebreak} 3rd rec field 1, 3rd rec field 2, 3rd rec field 3, 3rd rec field 4{linebreak} {linebreak} but it does not provide a line break after each record, so the output looks like this:{linebreak} {linebreak} 1st rec field 1, 1st rec field 2, 1st rec field 3, 1st rec field 42nd rec field 1, 2nd rec field 2, 2nd rec field 3, 2nd rec field 43rd rec field 1, 3rd rec field 2, 3rd rec field 3, 3rd rec field 4{linebreak} {linebreak} Each record is butted up against the previous record, without even a space. This is inconsitant with the instruction on how it is supposed to work, via this page:{linebreak} {linebreak} http://www.sqlite.org/sqlite.html {linebreak} {linebreak} Also, can you please refer me to somewhere that would explain how I can use SQLite with a batchfile, EG: using a batchfile to add a record, delete a record, query, Etc... {linebreak} {linebreak} Thanks,{linebreak} Tom #e8e8bd 169 new active 2002 Oct anonymous Unknown 2004 Apr anonymous 5 5 CE version I've successfully ported sqlite to windows CE 3.0 using eVC3.5maybe it was already done ?) I have only modified os.c to take into account the unicode interface. I attach my modified file. Best regards Noel Frankinet Can you attach the full modified source code for me ? (.zip) My personal e-mail is hensel@al.furb.br Thanks ... Andri #f2dcdc 691 code active 2004 Apr anonymous Unknown 2004 Apr drh 1 1 OS X File Sharing Hello Sir: This ticket may be considered a duplicate of ticket #301. I am unable to access SQLite databases from HFS or SMB network shares when using Mac OS X (10.3.3) as a client. The more technical aspects of the problem are explained well in ticket #301. I am using SQLabs SQLite plugin for RealBasic 5.5, and would like to use SQLite exclusively as my DB.#include int main() { sqlite *d1, *d2; const char *tail1, *tail2; const char **col_data1, **col_names1; const char **col_data2, **col_names2; sqlite_vm *v1, *v2; int i, j; int pN; char del_msg[255]; char *err; d1 = sqlite_open( "test", 0, &err ); if( err != 0 ) printf( "Open error: %s\n", err ); d2 = sqlite_open( "test", 0, &err ); if( err != 0 ) printf( "Open error: %s\n", err ); /* Select one row */ printf( " allocated: %d %d\n", (int)d1, (int)d2 ); i = sqlite_compile( d1, "select * from test_table", &tail1, &v1, &err ); sqlite_step( v1, &pN, &col_data1, &col_names1 ); if( err != 0 ) printf( "Select error: %s\n", err ); printf( "nCols: %d\n", pN ); sprintf( del_msg, "delete from test_table where a=\"%s\"", col_data1[0]); printf( "%s\n", del_msg ); /* // UNCOMMENT FOR THIS TO WORK * * sqlite_finalize( v1, &err ); * if( err != 0 ) printf( "Finalize error: %s\n", err ); * */ /* Delete that row */ j = sqlite_compile( d2, del_msg, &tail2, &v2, &err ); if( err != 0 ) printf( "Delete error: %s\n", err ); sqlite_step( v2, &pN, &col_data2, &col_names2 ); sqlite_finalize( v2, &err ); if( err != 0 ) printf( "Finalize error: %s\n", err ); /* COMMENT FOR THIS TO WORK */ sqlite_finalize( v1, &err ); if( err != 0 ) printf( "Finalize error: %s\n", err ); /****************************/ sqlite_close( d1 ); sqlite_close( d2 ); return 0; }
#-----------> load ./lib/tclsqlite-3.0.8.so sqlite3 puts [info patchlevel] sqlite3 db :memory: db eval "create table t1(a,b);" puts "before 3.0.8 select, no pragma" db eval "select * from t1;" x { puts "x(*) = $x(*)" } db eval "PRAGMA empty_result_callbacks=1" puts "before 3.0.8 select, yes pragma" db eval "select * from t1;" x { puts "x(*) = $x(*)" } db close load ./lib/tclsqlite-2.8.15.so Tclsqlite sqlite db2 :memory: db2 eval "create table t1(a,b);" puts "before 2.8.15 select, no pragma" db2 eval "select * from t1;" x { puts "x(*) = $x(*)" } db2 eval "PRAGMA empty_result_callbacks=1" puts "before 2.8.15 select, yes pragma" db2 eval "select * from t1;" x { puts "x(*) = $x(*)" } db2 close puts "done" # <-------------------and the results:
$ tclsh test_sqlite.tcl 8.4.3 before 3.0.8 select, no pragma before 3.0.8 select, yes pragma before 2.8.15 select, no pragma before 2.8.15 select, yes pragma x(*) = a b done_2006-May-16 18:29:44 by anonymous:_ {linebreak} This is still a problem in the 3.3.5 version of the tclsqlite library. The tclsqlite.c code never calls the callback code on empty results when PRAGMA empty_result_callbacks=1 is set. #e8e8bd 1034 build active 2004 Dec anonymous TclLib 2004 Dec 2 4 configure generated Makefile doesn't use TCL on MinGW on Windows When using MinGW/MSYS on Windows XP, the Makefile generated by the configure script fails while building the testfixture. The makefile is generated using ../sqlite/configure --with-tcl=/c/mingw/lib The configure script correctly locates the tclConfig.sh file in this directory and loads it. When I try to run the test suite, the build fails while linking the testfixture as shown below: $ make test ./libtool --mode=link gcc -g -O2 -DOS_WIN=1 -I. -I../sqlite/src -DNDEBUG -I/mingw/include -DSQLITE_OMIT_CURSOR -DTCLSH=1 -DSQLITE_TEST=1\ -DTHREADSAFE=0 -DTEMP_STORE=2\ -o testfixture ../sqlite/src/btree.c ../sqlite/src/date.c ../sqlite/src/func.c ../sqlite/src/os_mac.c ../sqlite/src/os_unix.c ../sqlite/src/os_win.c ../sqlite/src/pager.c ../sqlite/src/pragma.c ../sqlite/src/printf.c ../sqlite/src/test1.c ../sqlite/src/test2.c ../sqlite/src/test3.c ../sqlite/src/test4.c ../sqlite/src/test5.c ../sqlite/src/utf.c ../sqlite/src/util.c ../sqlite/src/vdbe.c ../sqlite/src/md5.c ../sqlite/src/tclsqlite.c \ libtclsqlite3.la gcc -g -O2 -DOS_WIN=1 -I. -I../sqlite/src -DNDEBUG -I/mingw/include -DSQLITE_OMIT_CURSOR -DTCLSH=1 -DSQLITE_TEST=1 -DTHREADSAFE=0 -DTEMP_STORE=2 -o testfixture ../sqlite/src/btree.c ../sqlite/src/date.c ../sqlite/src/func.c ../sqlite/src/os_mac.c ../sqlite/src/os_unix.c ../sqlite/src/os_win.c ../sqlite/src/pager.c ../sqlite/src/pragma.c ../sqlite/src/printf.c ../sqlite/src/test1.c ../sqlite/src/test2.c ../sqlite/src/test3.c ../sqlite/src/test4.c ../sqlite/src/test5.c ../sqlite/src/utf.c ../sqlite/src/util.c ../sqlite/src/vdbe.c ../sqlite/src/md5.c ../sqlite/src/tclsqlite.c ./.libs/libtclsqlite3.a -L/mingw/lib -ltclstub84 C:/DOCUME~1/DennisC/LOCALS~1/Temp/cc2taaaa.o(.text+0x511): In function `sqlite3TestErrCode': c:/SQLite/SQLiteV3/build2/../sqlite/src/test1.c:77: undefined reference to `_imp__Tcl_ResetResult' It looks like the code is being linked against the tclstub84 library which doesn't contain the needed code. To work around this problem, I needed to modify the Makefile that configure generated. I modified the line: LIBTCL = to use the correct library like this: LIBTCL = -L/mingw/lib -ltcl84 Now the testfixture builds correctly and I can run the test suite. Someone more familiar with the automake tools will need to correct this problem on a more permanent basis. _2004-Dec-10 02:15:25 by drh:_ {linebreak} You can help us to troubleshoot this by attaching the following files to the ticket: *: tclConfig.sh *: the complete output of configure and make ---- _2004-Dec-10 16:08:05 by anonymous:_ {linebreak} I have attached the requested files. Note, these files were generated from the latest CVS source after checkin 2162. The tclConfig.sh file is exactly as is was installed by the MinGW tcltk binary package in the last stable version of MinGW/MSYS (I just checked and they have released a newer stable version of MinGW/MSYS since I installed mine. It seems the only package which is newer than my installation is the gcc compiler. I have the current version of the tcl/tk package.) #e8e8bd 755 new active 2004 Jun anonymous TclLib 2004 Jun drh 5 2 Adding aggregate function to the tcl binding The diff against tclsqlite.c to solve the problem can be also submitted by email #e8e8bd 734 build active 2004 May anonymous TclLib 2004 May 4 4 libtclsqlite.xxx not built by default make all target Not a big deal but the Quickstart documentation assumes that the libtclsqllite.so (or dylib (macosx) in my case) is made & installed but the default make all target doesn't make it. You have to explicitly specify that to be made. Although, the 'Building from Source' documentation page does say that make all makes just sqlite and the basic libsqlite.{a, so, whatever} in the comment. Just a bit confusing. #e8e8bd 428 new active 2003 Aug anonymous TclLib 2003 Aug drh 5 5 missing prepare/fetch functions to be compatible with other DBs To have a compatible behaviour to other database engines when fetching single rows from a database table, it would be nice to be able to have a single fetch function instead of the standard "db eval ..." ability using a callback code. I've found three functions in the file test1.c which comes with the source files and implemented them into the TCL interface (file tclsqlite.c proc DbObjCmd). oops deleted attach. please post it again. #e8e8bd 1648 new active 2006 Jan anonymous Shell 2007 Dec 4 3 meaningful error message: constraint failed create table emp( id text unique, sex text check( sex in 'm' or sex in 'f' ); insert into emp values( '1','x' ); SQL error: constraint failed This error message could be better. If there are several constraints, which constraint failed? So I named the constraint create table emp( id text unique, sex text constraint chk_sex check( sex in 'm' or sex in 'f' ); insert into emp values( '1','x' ); SQL error: constraint failed Still no joy . . . It would be nice if the error message were more specific. _2006-Jan-30 16:22:58 by anonymous:_ {linebreak} actually my testing was better than my typing, I used: check (sex = 'm' or sex = 'f' ) ---- _2007-Oct-25 09:47:14 by anonymous:_ {linebreak} This is a really big deal for me and for many others I suspect. If this is not a priority, could you at least throw out some hints about implementing it? I browsed through the code but can't seem to find where this would even go. ---- _2007-Oct-25 10:10:36 by anonymous:_ {linebreak} Hm, ok, the check constraints are stored in the table structure as a single expression which is the AND of all of them. This alone suggests that the task at hand is not simple... ---- _2007-Oct-25 10:17:36 by anonymous:_ {linebreak} Perhaps a new Check type could be created which could basically be Expr plus an extra pointer, which could then be used to make a list of them, similar to how the triggers seem to be stored. I'll keep snooping around, but I thought I'd post what I've found thus far in case anyone else looks at this. ---- _2007-Oct-25 19:40:37 by anonymous:_ {linebreak} I have attached a patch that implements this. I've only tested it lightly by hand. (The test suite failed to run and gave me some strange linking errors) ---- _2007-Dec-20 11:38:03 by anonymous:_ {linebreak} Although this is tagged as shell, the error message comes from the sqlite core. My single biggest problem besides the lack of detail (some of my tables have 5 constraints) is that it also prevents me from localizing the error messages. If I have the constraint name then at least I can look it up in a translation table and tell non-english speakers something meaningful. #f2dcdc 1242 code active 2005 May anonymous Shell 2007 Aug 3 4 EXPLAIN causes segmentation fault on OSX (and linux) Under Mac OS X, EXPLAIN causes a segmentation fault: [jacob@046] ~$ sqlite3 foo.db SQLite version 3.2.1 Enter ".help" for instructions sqlite> CREATE TABLE test (a int, b int); sqlite> EXPLAIN SELECT * FROM test; Segmentation fault The crash dump follows: Host Name: jacobian Date/Time: 2005-05-13 09:17:04.860 -0500 OS Version: 10.4 (Build 8A428) Report Version: 3 Command: sqlite3 Path: /usr/local/bin/sqlite3 Parent: bash [15421] Version: ??? (???) PID: 15544 Thread: 0 Exception: EXC_BAD_ACCESS (0x0001) Codes: KERN_INVALID_ADDRESS (0x0001) at 0x1400fffc Thread 0 Crashed: 0 libSystem.B.dylib 0x90003228 strlen + 8 1 libsqlite3.0.dylib 0x002387c8 sqlite3VdbeList + 284 (vdbeaux.c:609) 2 libsqlite3.0.dylib 0x002376e0 sqlite3_step + 312 (vdbeapi.c:207) 3 libsqlite3.0.dylib 0x0023e5d8 sqlite3_exec + 260 (legacy.c:82) 4 sqlite3 0x00005b64 process_input + 808 (shell.c:1504) 5 sqlite3 0x000062bc main + 1528 (shell.c:1790) 6 sqlite3 0x00001db4 _start + 348 (crt.c:272) 7 sqlite3 0x00001c54 start + 60 Thread 0 crashed with PPC Thread State: srr0: 0x90003228 srr1: 0x0000d030 vrsave: 0x00000000 cr: 0x22444428 xer: 0x00000006 lr: 0x002387c8 ctr: 0x90003220 r0: 0x002387c8 r1: 0xbfffef40 r2: 0x00249a00 r3: 0x1400fffe r4: 0x00000028 r5: 0x00000000 r6: 0x00000001 r7: 0xffffffff r8: 0x00000001 r9: 0x1400fffc r10: 0x00000086 r11: 0x00249180 r12: 0x90003220 r13: 0x00000000 r14: 0x00000000 r15: 0x00000000 r16: 0x00000000 r17: 0xbffff0f8 r18: 0x00000000 r19: 0xbffff17c r20: 0x00000000 r21: 0x000036d0 r22: 0x00303d90 r23: 0x00303d74 r24: 0x01805700 r25: 0x01807e00 r26: 0x00000001 r27: 0x00000004 r28: 0x01805640 r29: 0x01805600 r30: 0x01805200 r31: 0x002386bc Binary Images Description: 0x1000 - 0x7fff sqlite3 /usr/local/bin/sqlite3 0x205000 - 0x248fff libsqlite3.0.dylib /usr/local/lib/libsqlite3.0.dylib 0x8fe00000 - 0x8fe50fff dyld 43 /usr/lib/dyld 0x90000000 - 0x901a6fff libSystem.B.dylib /usr/lib/libSystem.B.dylib 0x901fe000 - 0x90202fff libmathCommon.A.dylib /usr/lib/system/libmathCommon.A.dylib 0x91d33000 - 0x91d53fff libmx.A.dylib /usr/lib/libmx.A.dylib 0x9680c000 - 0x9683afff libncurses.5.4.dylib /usr/lib/libncurses.5.4.dylib 0x969a3000 - 0x969b9fff libedit.2.dylib /usr/lib/libedit.2.dylib Happening to me as well on FC6 sqlite3 version 3.3.6 ---- _2007-Aug-21 17:09:34 by anonymous:_ {linebreak} Try to upgrade to 3.4.2. #f2dcdc 1948 code active 2006 Aug anonymous Shell 2006 Aug 2 3 Double quotes are not escaped in csv mode If text is exported using "csv" mode, double quotes in strings are not escaped. Generally double-quotes in a quoted field in CSV should be escaped by repeate. I.e., 'This is a "test".' could be about as "This is a ""test""." This doesn't appear to be the behavior SQLite uses, so, in the meantime, I'll have to export my data using another method and then transform that data into CSV for my import script. #f2dcdc 1947 code active 2006 Aug anonymous Shell 2006 Aug 3 3 ".mode insert" works bad with BLOBs .mode insert displays BLOBs as strings, which isn't very good for embedded NULs. Having output more like the one from .dump would be better, IMO. sqlite> select * from t; INSERT INTO table VALUES(''); sqlite> .dump BEGIN TRANSACTION; CREATE TABLE t(f BLOB); INSERT INTO "t" VALUES(X'0041'); COMMIT; #f2dcdc 1885 code active 2006 Jul anonymous Shell 2006 Jul 2 3 sqlite3 .mode insert and .dump do not list column names for selects In sqlite3 .mode insert does not list column names for selects - it should. This makes dumping selected columns from tables when intending to add or delete columns problematic. .dump doesn't list column names either, IMHO it should. Consider sqlite> .mode tabs{linebreak} sqlite> select * from users;{linebreak} ed 2006-07-05 52{linebreak} sqlite> .mode insert{linebreak} sqlite> select abs_tgt from users;{linebreak} INSERT INTO table VALUES(52);{linebreak} sqlite> Obviously the workaround is to hand edit the output SQL _2006-Jul-11 10:20:08 by anonymous:_ {linebreak} I've just noticed it doesn't include the table name in the INSERT statements either. #e8e8bd 1785 new active 2006 Apr anonymous Shell 2006 Apr 4 3 rl_readline_name not set Hi, I have a conditionnal construct into my .inputrc: $if sqlite "S ": "select " "L ": "like " "II ": "insert into " "U ": "update " "V ": "view " "(V ": "values (( " "C ": "create " "DR ": "drop " "DI ": "distinct " "F ": "from " "W ": "where " $endif http://tiswww.tis.case.edu/~chet/readline/readline.html#SEC11 but it doesn't work with sqlite because rl_readline_name is not set in the source. I don't know about readline so i can't submit a patch. regards #e8e8bd 1636 todo active 2006 Jan anonymous Shell 2006 Jan drh 4 3 stdev does not work When trying to calculate standard deviation I get the following error message: SQL error: no such function: stdev How does SQLite support statistical function stdev? What is the correct name of the function? The same with sqr and sqrt. What are the names for square and square-root? Is there any other way to use SQLite for statistical calculations? #f2dcdc 1573 code active 2005 Dec anonymous Shell 2005 Dec 3 3 Bad CSV output when data contains double quotes SQLite 3.2.7 emits invalid CSV when a field value contains double quotes (or at least, it's CSV that Gnumeric cannot parse). Here's an example: sqlite> create table foo (a text, b text); sqlite> insert into foo values ("hello", "world"); sqlite> insert into foo values ("mr.", "o'reilly"); sqlite> insert into foo values ('12" EP', 'blah'); sqlite> .mode csv sqlite> .output foo.csv sqlite> select * from foo; Here's what foo.csv looks like: $ cat foo.csv "hello","world" "mr.","o'reilly" "12" EP","blah" Note the ambiguous quoting on line 3. If I load this file into Gnumeric, it parses the first two lines just fine. But the last line confuses it. It appears that doubling the quote works -- at least for Gnumeric's CSV parser. That is, if I edit foo.csv to "hello","world" "mr.","o'reilly" "12"" EP","blah" then it's OK. This is basically a duplicate of [1312] and related shell problem reports, though it provides a better test case than the previous reports. #e8e8bd 1506 new active 2005 Nov anonymous Shell 2005 Nov 1 4 Add some flags to .import command I think it would be very useful to be able to instruct .import to use either "INSERT INTO" or "INSERT OR REPLACE INTO" via some flag.{linebreak} Something along the lines of: .import FILE TABLE REPLACE One other extension of .import that I'd find useful is to be able to instruct .import not to error out when malformed line of input is found, but instead just print the error message as it does now and continue on with next line.{linebreak} Why is this useful? Sometimes I need to import large files with several million lines into SQLite for some statistical analysis and some of those lines are malformed sometimes. As it is now, sqlite3.exe will error out at that line and then I have to fix or remove that line from input file and restart the import. If there are other malformed lines I need to restart import again. Needless to say this is very time consuming and losing a few lines out of few millions wouldn't change my results (statistics) a bit. Perhaps both above mentioned flags could be combined into one with 4 possible values. _2005-Nov-02 14:43:52 by anonymous:_ {linebreak} I've attached a patch that does what I proposed in this ticket. It uses MODE argument to set two proposed flags. Other flags can be added easily, but it's kind of cumbersome to document all posible values of MODE argument. #e8e8bd 1491 new active 2005 Oct anonymous Shell 2005 Oct 4 4 syntax error with " .mode column: when I key in ".mode column" without any space, the command is work well. If I key in "^.mode column" with a space(the ^ means a space) , it will show the following error message : ===================================================== SQL error: near "syntax": syntax error ===================================================== Would you like fix it on next version, please ? Have a nice day! John Wei #f2dcdc 1298 code active 2005 Jun anonymous Shell 2005 Oct drh 4 3 sqlite3.exe not recognizing newline as comment terminator The sqlite3.exe program does not recognize comments that end in a newline if there is valid SQL before them on the same line. The statement: {linebreak} "SELECT 1; -- this is a comment" {linebreak} should be valid SQL, but sqlite3.exe still requires a semicolon after the comment. Please look at the test below: C:\Temp\SQLite>sqlite3 SQLite version 3.2.1 Enter ".help" for instructions sqlite> select 1; 1 sqlite> select 1; -- test comment ...> ; 1 sqlite> select 1; -- test; 1 sqlite> _2005-Jun-22 18:20:37 by anonymous:_ {linebreak} The string "SELECT 1; -- this is a comment" is not valid SQL. SQL does not use the semicolon character inside SQL statements except to seperate the constituent statements of a trigger, stored procedure, or an embedded program. The semicolon is used by sqlite3.exe (and standard SQL) as a end of statement marker that triggers it to parse and execute the input up to that point. From the SQL 2003 standard: 21 Direct invocation of SQL 21.1
sqlite> CREATE TABLE a(b); sqlite> INSERT INTO a VALUES (X'41424300500051'); sqlite> .dump BEGIN TRANSACTION; CREATE TABLE a(b); INSERT INTO "a" VALUES(X'41424300500051'); COMMIT; sqlite> .mode insert sqlite> SELECT * FROM a; INSERT INTO table VALUES('ABC');It would be nice for ".mode insert" to print a command that would actually re-create the same data, the same as ".dump" (the obvious difference is that .dump can't filter data in any way, it just dumps it all) or, at least, it would be very nice if the already existing function that "prints binary data as X'-encoded-string" were reachable from SQL, so that one could use something like:
SELECT xencode(b) FROM a;and obtain
X'41424300500051'_2005-Sep-25 17:03:40 by anonymous:_ {linebreak} I am not the original ticket poster, but I noticed that this feature request is related to {link: http://lists.gnu.org/archive/html/monotone-devel/2005-09/msg00294.html Monotone} migrating away from Base64 encoding to using straight Sqlite blobs. ---- _2005-Sep-26 01:49:33 by drh:_ {linebreak} The built-in quote() function converts BLOBs into ascii BLOB literals. Will it not server for the requested xencode() function? SELECT quote(b) FROM a ---- _2005-Sep-26 08:57:47 by anonymous:_ {linebreak} Yes, I guess it can perfectly do. What about ".mode insert" output? Is it supposed to print raw data? #f2dcdc 1401 code active 2005 Aug anonymous Shell 2005 Aug 4 4 concatenating html in select list create table test( f1 text, f2 text ); insert into table values( 'hello', 'world' ); .mode html select f1 || '
Row1 try1 Row2 try2 Row3 try3Step 2: In sqlite3, type the following:
CREATE TABLE test (col1 char(4), col2 char(4)); .separator \t .import test.txt test .mode csv SELECT * FROM test;The following results are observed:
"Row1","try1\r" "Row2","try2\r" "Row3","try3"I'm new to sqlite and I'm not sure whom shall I assign this ticket to. Please help me to re-assign to the right person if I've set it wrong. Thanks. Amanda _2005-Mar-24 02:53:08 by anonymous:_ {linebreak} Seems this is also related to ticket #939 ... there it says it's been solved, but as far as I'm using 3.20, it seems that it's not been fixed completely. Please note that Windows use BOTH \r and \n for every new line break. #e8e8bd 1128 event active 2005 Feb anonymous Shell 2005 Feb 2 3 sqlite3-3.1.2.bin segfaults Linux 2.4.26 (Knoppix 3.4 on hard drive) download: http://sqlite.org/sqlite3-3.1.2.bin.gz attached: strace log In the strace log I notice that the segfault happens just after /etc/nsswitch.conf has been read, if that's of any consequence. #e8e8bd 1061 new active 2005 Jan anonymous Shell 2005 Jan 4 4 shell.c .import from stdin e.g.: in = ( 0 == strcmp( "-", zFile ) ) ? stdin : fopen(zFile, "rb"); #f2dcdc 1056 code active 2004 Dec anonymous Shell 2004 Dec 3 3 test pragma-9.4 fails during second pass in "make fulltest" During a "make fulltest" run, the pragma tests appear to run twice. On the first run, pragma-9.4 runs properly. On the second run, it gives an error: pragma-9.4... Expected: [] Got: [/Volumes/Local/Users/sqlite/test/bld] (where the path listed is the build directory for this build of sqlite). The pragma-9.4 test is a recent addition to sqlite. This is currently the only failure I'm seeing in a "make fulltest" of the current cvs tree on Mac OS X when the build/test directory is on a hard drive. 1 errors out of 68411 tests Failures on these tests: pragma-9.4 make: *** [fulltest] Error 1 #e8e8bd 944 new active 2004 Oct anonymous Shell 2004 Oct 4 3 Command to abort current expression. It would be convenient if, at the sqlite prompt, it was possible to abort the current expression by entering e.g. a singel dot on a new line. I.e.:
sqlite> SELECT * FROM ...> . sqlite>#e8e8bd 883 new active 2004 Sep anonymous Shell 2004 Sep 1 4 Choose appropriate column widths for column output Currently, when you use a select statement and have .mode set to column, it truncates the data based on the values in .width. It can be a problem to set .width properly because sometimes you are only selecting some columns and sometimes all of them. This means you really have to set .width before each select statement. Even when you do, sometimes the width is too wide and space is wasted unless you know in advance the width of the widest value being printed. What I would like is another mode or some option to have SQLite figure out the optimal width of the columns. It would simply set the width of each column to the width of the widest value in the column. MySQL does this. _2004-Sep-02 15:18:20 by drh:_ {linebreak} There are two ways of doing this: 1: Run the query twice. Record the maximum column witdh on the first run then display the results on the second run. 2: Store the entire result set in memory. Compute the maximum column widths prior to displaying anything. Either approach requires a significant increase in resources to do the query. ---- _2005-Jan-21 20:56:53 by anonymous:_ {linebreak} How about instead of, or in addition to this feature, a mode where each column has a default width which corresponds to the data type? E.g. if the type is CHAR(50) the column would be 50 wide. That would be faster and would be good enough for my purposes. ---- _2005-Apr-09 19:41:59 by anonymous:_ {linebreak} I don't need this feature after all, since I wrote a filtering program to convert the tab-separated data from sqlite to produce the column-separated data I wanted. In case someone wants to use the code to implement this feature in sqlite, or in case someone wanting this functionality wants to use it directly, you can download the code for free from: http://personal.nbnet.nb.ca/aerichar/tab2col/ #f2dcdc 754 code active 2004 Jun anonymous Shell 2004 Jun drh 2 3 problem opening a dbfile in the upper directory (../dbname) there is a problem in the calculation of a full path name based on a relative path name in an uproot location (../). i have fixed this during my porting to dos, and the relevant diff is at http://www.sqlite.org/cvstrac/tktview?tn=524. best regards alex #e8e8bd 737 new active 2004 May anonymous Shell 2004 May anonymous 3 3 xml export Export a sql statement to an oracle-style export xml.
CREATE TABLE T1(T1Id, Name); INSERT INTO T1(T1Id, Name) VALUES (0,"titi"); INSERT INTO T1(T1Id, Name) VALUES (1,"toto"); INSERT INTO T1(T1Id, Name) VALUES (2,"tutu"); INSERT INTO T1(T1Id, Name) VALUES (3,"hat"); INSERT INTO T1(T1Id, Name) VALUES (4,"socks"); CREATE TABLE T2(T2Id, Name); INSERT INTO T2(T2Id, Name) VALUES(0,"Black"); INSERT INTO T2(T2Id, Name) VALUES(1,"Red"); INSERT INTO T2(T2Id, Name) VALUES(2,"Blue"); INSERT INTO T2(T2Id, Name) VALUES(3,"Green"); INSERT INTO T2(T2Id, Name) VALUES(4,"Yellow"); INSERT INTO T2(T2Id, Name) VALUES(5,"Brown"); INSERT INTO T2(T2Id, Name) VALUES(6,"White"); CREATE TABLE T3(T3Id,T1Id,T2Id,Number); INSERT INTO T3(T3Id,T1Id,T2Id,Number) VALUES (1,4,0,5); INSERT INTO T3(T3Id,T1Id,T2Id,Number) VALUES (2,4,1,4); INSERT INTO T3(T3Id,T1Id,T2Id,Number) VALUES (3,4,2,3); INSERT INTO T3(T3Id,T1Id,T2Id,Number) VALUES (4,4,3,2); INSERT INTO T3(T3Id,T1Id,T2Id,Number) VALUES (5,4,4,1); INSERT INTO T3(T3Id,T1Id,T2Id,Number) VALUES (6,4,5,0); INSERT INTO T3(T3Id,T1Id,T2Id,Number) VALUES (7,3,0,10); SELECT main.Number AS Number, prod.Name AS product, col.Name AS color, Main.T3Id AS Id FROM T3 AS main LEFT JOIN T1 AS prod USING(T1Id) LEFT JOIN T2 AS col USING(T2Id);The second USING doesn't work. _2006-Jan-23 16:03:17 by drh:_ {linebreak} SQLite only looks for columns to satisfy the USING clause in the two tables immediately tot he left and right of the join. In the example above, it is trying to resolve "USING(t2id)" by looking for columns t1.t2id and t2.t2id. ---- _2006-Jan-23 23:12:32 by anonymous:_ {linebreak} I'm not sure if you are saying that is the way SQLite should work, or just explaining the source of the error. I believe that JOINS are supposed to combine two tables to produce a third table in a left associative manner. The sample query should join T3 and T1 using T1Id to produce an intermediate table, lets call it simply T. Then T should be joined with T2 using T2Id to produce the final result table. As shown below, T does in fact have a column named T2Id, and this column should be used for the second join. sqlite> CREATE table t AS SELECT * ...> FROM T3 AS main ...> LEFT JOIN T1 AS prod USING(T1Id); sqlite> sqlite> PRAGMA table_info(t); 0|T3Id|numeric|0||0 1|T1Id|numeric|0||0 2|T2Id|numeric|0||0 3|Number|numeric|0||0 4|Name|numeric|0||0 sqlite> sqlite> SELECT T2ID from t; 0 1 2 3 4 5 0 One area where SQLite may have a standards compliance problem is with the name of the result set columns with column name joins (i.e. joins with a USING() clause). The standard says that the result set column that was used to join the two tables can not have a qualifier. That means it is NOT the column from either of the two input tables, and can't be qualified using either table name. The USING clause "projects out" the columns from both of the input tables and replaces them with a single column with the same name. The two columns could have different types for example, and the result set column could be yet another type. The first two otuput columns below should generate an error. Only the third is legal according to the SQL standard. sqlite> SELECT T3.T2Id, T2.T2Id, T2Id from T3 LEFT JOIN T2 USING(T2Id); T2Id|T2Id|T2Id 0|0|0 1|1|1 2|2|2 3|3|3 4|4|4 5|5|5 0|0|0 The type differences between the different columns are displayed in the following example. In one table the values of column a are integers and in the other they are text. The type of the result column depends upon the order of the tables in the join. sqlite> select a from tt1 join tt2 using(a); a 1 2 sqlite> select typeof(a) from tt1 join tt2 using(a); typeof(a) integer integer sqlite> select typeof(a), typeof(tt1.a), typeof(tt2.a) from tt1 join tt2 using(a); typeof(a)|typeof(tt1.a)|typeof(tt2.a) integer|integer|text integer|integer|text sqlite> select * from tt1 join tt2 using(a); a|b|c 1|1|-1 2|4|-2 sqlite> select typeof(a), typeof(tt1.a), typeof(tt2.a) from tt2 join tt1 using(a); typeof(a)|typeof(tt1.a)|typeof(tt2.a) text|integer|text text|integer|text The same rules also apply to NATURAL joins. The result columns cannot legally be qualified by either table name. ---- _2006-Jan-23 23:45:04 by drh:_ {linebreak} My previous remark is describe what SQLite does. You are probably right in pointing out that what it currently does is not correct and ought to be fixed. I merely observet that SQLite has never worked that way and has (up until now) caused no serious concern. So, I'm not going to consider this a high-priority bug. I will get to it as I am able. But I need to get 3.3.x out the door first. #f2dcdc 1493 code active 2005 Oct anonymous Parser 2005 Oct 3 3 lemon: pathsearch uses wrong directory separator under Win32 The pathsearch function in lemon.c uses a semicolon (;) to separate the directories in the path. Under Win32 systems this should be a colon (:). _2005-Oct-18 09:37:10 by anonymous:_ {linebreak} --- lemon.c.orig 2005-10-18 11:27:55.753467000 +0200 +++ lemon.c 2005-10-18 11:29:11.897825400 +0200 @@ -2791,13 +2791,16 @@ { char *pathlist; char *path,*cp; + char ds; char c; extern int access(); #ifdef __WIN32__ cp = strrchr(argv0,'\\'); + ds = ';'; #else cp = strrchr(argv0,'/'); + ds = ':'; #endif if( cp ){ c = *cp; @@ -2812,7 +2815,7 @@ path = (char *)malloc( strlen(pathlist)+strlen(name)+2 ); if( path!=0 ){ while( *pathlist ){ - cp = strchr(pathlist,':'); + cp = strchr(pathlist,ds); if( cp==0 ) cp = &pathlist[strlen(pathlist)]; c = *cp; *cp = 0; ---- _2005-Oct-18 09:39:00 by anonymous:_ {linebreak} More readable version of the patch: --- lemon.c.orig 2005-10-18 11:27:55.753467000 +0200 +++ lemon.c 2005-10-18 11:29:11.897825400 +0200 @@ -2791,13 +2791,16 @@ { char *pathlist; char *path,*cp; + char ds; char c; extern int access(); #ifdef __WIN32__ cp = strrchr(argv0,'\\'); + ds = ';'; #else cp = strrchr(argv0,'/'); + ds = ':'; #endif if( cp ){ c = *cp; @@ -2812,7 +2815,7 @@ path = (char *)malloc( strlen(pathlist)+strlen(name)+2 ); if( path!=0 ){ while( *pathlist ){ - cp = strchr(pathlist,':'); + cp = strchr(pathlist,ds); if( cp==0 ) cp = &pathlist[strlen(pathlist)]; c = *cp; *cp = 0; ---- _2005-Oct-19 14:05:27 by anonymous:_ {linebreak} Cygwin and other Posix emulation layers on Windows require ':' for path separators, so you cannot blindly rely on __WIN32__ to make this determination. #e8e8bd 1475 new active 2005 Oct anonymous Parser 2005 Oct anonymous 5 5 sytax error near autoincrement Consider following sql syntax. create table t1 (id integer primary key autoincrement); // no error create table t2 (id integer autoincrement primary key); // error create table t3 (id integer autoincrement, primary key(id)); // error _2005-Oct-07 03:15:23 by anonymous:_ {linebreak} Your last two examples don't follow the syntax shown in the documentation. AUTOINCREMENT is a modifier for a PRIMARY KEY constraint and can only come after one. #e8e8bd 1469 event active 2005 Oct anonymous Parser 2005 Oct 1 1 Having problems with subselects that work on mySQL I'm converting from mySQL. However I've run across several sub-selects that work on mySQL but not SQLite. Apache gives me a error. If I can't get them fixed, I will be forced back to mySQL. Not sure what I'm doing wrong. Basically trying to find the last variety record against a Wine Process Order (wpo). Program is in PHP5 script as follows: ------------------------------------------------- $sql = " SELECT * FROM wpo W, wpo_dtl as D, wpo_varietal V, storage S, variety VM WHERE S.stop_trace <> 'Y' AND D.storage_id = S.storage_id AND D.from_to = 'T' AND D.wpo_num = W.wpo_num AND D.leg_num = W.leg_num AND V.storage_id = S.storage_id AND V.from_to = 'T' AND V.wpo_num = D.wpo_num AND V.leg_num = D.leg_num AND VM.variety_num = V.variety_num AND VM.color = 'R' AND W.wpo_datetime = (SELECT MAX(W1.wpo_datetime) FROM wpo as W1,wpo_varietal as V1 WHERE V1.storage_id = S.storage_id AND V1.from_to = 'T' AND W1.wpo_num = V1.wpo_num AND W1.leg_num = V1.leg_num) ORDER BY S.storage_id"; $query = sqlite_query($link_id, $sql); $result = sqlite_fetch_all($query, SQLITE_ASSOC); foreach ($result as $qd) { ............... ------------------------------------------ _2005-Oct-05 20:08:11 by anonymous:_ {linebreak} You've said it "gives you an error" and "doesn't work" but you don't tell anybody *what* error message it gives you or *how* it fails. Nobody here can help you solve your problem unless you provide that information. #e8e8bd 1468 new active 2005 Oct anonymous Parser 2005 Oct 5 5 Added Mysql function CONCAT
static void concatFunc( sqlite3_context *context, int argc, sqlite3_value **argv ){ unsigned char *z; int i; int len = 0; for(i=0;iThen:_2005-Oct-05 14:27:29 by drh:_ {linebreak} Seems kind of pointless since SQLite already supports the SQL-standard || operator. ---- _2005-Oct-09 13:24:49 by anonymous:_ {linebreak} sorry :-) #e8e8bd 229 new active 2003 Jan anonymous Parser 2005 Sep 5 4 DECLARE ... CURSOR FOR ... would be really nice Cursors can be a tremendous performance aid in GUI-driven applications with large data sets, and it's pretty hard to emulate them if your application is written with the concept in mind. I would really be thrilled if cursors found their way into SQLite! Thanks, b.g. OK, this is now Jan 2005 and "cursors" are not there yet. Any plans? what would be an easy work around. ---- _2005-Sep-14 06:31:57 by anonymous:_ {linebreak} Yes Cursor would be a added advantage so that we dont have to move to and for from application to sqlite for smiple calculation ---- _2005-Sep-14 20:03:40 by anonymous:_ {linebreak} SQLite3's cursors are called sqlite3_prepare(), sqlite3_step(), sqlite3_reset() and sqlite3_finalize() Seriously, the purpose of a cursor is to leave the data on the database server and retrieve only the data and columns you need from a resultset managed by the server. SQLite is an embedded file-based database engine, not a database server -- therefore cursors make no sense. Asking for cursors in sqlite is akin to asking the filesystem to give you a cursor on a textfile so you can only read one line at a time -- when the reality is, you just open the file and read the lines yourself. sqlite3_step() *is* your cursor, it is reading the row from the file, one row at a time. #e8e8bd 1381 new active 2005 Aug anonymous Parser 2005 Aug 5 4 missing important builtin function stdev I really miss function *stdev* for support of statistical usage of SQLite. Also missed *sqr* and *sqrt* for square and square root calculations. This would help me very much to simplify my applications if they could be integrated also into the command line tool. An *avg* without *stdev* is of no useful meaning. Another wish is the *support of date and time stamp calculations*. Some few functions to convert literal expressions into float and backwards would be sufficient for me. No time zone consideration required. _2005-Aug-25 11:35:15 by anonymous:_ {linebreak} Proposal for powerful enhancements:{linebreak} Introduce alias scripting to combine built in functions to create new function names. Similar to C language macros. SQR and SQRT to become available as builtin functions.{linebreak}Example for creation:{linebreak} *PRAGMA FUNCTION newfunctionname( {, +})= * {linebreak}Example for an application: {linebreak}PRAGMA FUNCTION{linebreak} stdev(xsum,xsumcubed,n)=sqrt((n*xsumcubed-sqr(xsum))/(n*(n-1))); {linebreak}SELECT avg(t1.c1) as c1mean,{linebreak} stdev(sum(t1.c1),sum(sqr(t1.c1)),count(t1.c1)) as c1sigma,{linebreak} avg(t1.c2) as c2mean,{linebreak} stdev(sum(t1.c2),sum(sqr(t1.c2)),count(t1.c2)) as c2sigma{linebreak} from t1;{linebreak} This would reduce typing and enables to simulate non built in functions which can be created on basis of built in functions. Should be well supported by expression optimization to gain most benefit. ---- _2005-Aug-29 16:00:39 by anonymous:_ {linebreak} I also lack of sqrt, sqr and stdev (beside some other basics like sin, cos, tan, atan2, deg, rad, ... ). I like the idea of macros, too, because they make simplifications in statements easy. While sqr could be written like MACRO sqr(x)= ((x)*(x)) it is very difficult to live without the sqrt. Even stdev could become a macro but without sqrt the live stays difficult. If macros become part of SQLite3 then there should also be a command PROBE to see the expanded statement as executed by the parser; similar to EXPLAIN command. Priority to this ticket should be higher to become realized. #f2dcdc 1213 code active 2005 Apr anonymous Parser 2005 Apr 3 2 Problem with alias columns in subqueries The following query create an SQL error while it seems to me to be SQL complient (in fact it works fine with a lot database servers): sqlite> select i from (select 0 as i union all select 1) as tmp; SQL error: no such column: i I found a workaround writting the following SQL query (it works well if you explicits the implicit variable i) : select i from (select 0 as i union all select 1 as i) as tmp; Thanks for your help Jerome _2005-Apr-17 11:03:52 by anonymous:_ {linebreak} SQLite appears to take the column names from the last clause in a UNION (your originally query works if you reverse the order of the clauses). IIRC, the standard says that the column names in a UNION of clauses with different column names is DBMS-dependent and while many DBMSs take them from the first clause, this is not something to count on. ---- _2005-Apr-18 19:53:09 by anonymous:_ {linebreak} You're perfectly right, but it seems to me that it should be better that sqlite do respect the "de facto" standard. #e8e8bd 1189 new active 2005 Mar anonymous Parser 2005 Mar 1 1 Make where-clause item count limit dynamic If you enter a query with more then 100 'where clause items' (pieces of a where clause joined by AND or OR) then you get an error saying you must limited yourself to 100 or less items. This is fine for 99.9% of the time, but I ran into a problem when trying to implement ad-hoc bitmap indexes -- in other words, hundreds of columns in a table and therefor hundreds of items in the where clause. I was able to work around this, at the sqlite level, by simply changing a constant in where.c from 101 to 1001 (437 was what was needed for this specific application). The requested I'd like to see filled is that this be more-or-less a dynamic value. Perhaps it reads from some environment variable to determine this, and defaults to 100(+1). This would likely result in a bit of code rewrite as to how where.c works, but I don't think it would be a terribly huge amount. #e8e8bd 1187 build active 2005 Mar anonymous Parser 2005 Mar drh 3 3 OMIT_TRIGGER OMIT_VIEW do not compile properly SQLITE_OMIT_TRIGGER and SQLITE_OMIT_VIEW are needed by lemon to process parse.y{linebreak} They are not passed from CFLAGS to Makefile.in OPTS. #f2dcdc 1142 code active 2005 Feb anonymous Parser 2005 Feb danielk1977 1 1 Column names create table a (id, x); create table b (id, y); insert into a values (1,1); insert into b values (1,2); select * from a inner join b; column names returned: id,x,id,y How am I supposed to use such column names? Ouwey. _2005-Mar-15 07:28:43 by anonymous:_ {linebreak} This bug breaks existing applications where 'SELECT rowid, * FROM table' was used to open any table and then precompiled statements were used to update the changed record by using 'UPDATE table SET =?, =? ... WHERE rowid=?'. We cannot update to sqlite v3.1.x because of this. ---- _2005-Mar-15 08:36:31 by drh:_ {linebreak} There are lots of people telling me the current behavior is wrong. But nobody has yet suggested what the correct behavior should be. Until I know what the correct behavior should be, there is little I can do to fix the problem. Rather than just telling me the current behavior is wrong, please offer an explanation of what the correct behavior should be. What do Oracle and PostgreSQL do in the same situation? ---- _2005-Mar-16 09:04:30 by anonymous:_ {linebreak} Dr. Hipp, very sorry for the confusion I've caused! My comments were actually targeted at bug #1141! I've included the same remarks there. As for this bug: I can only tell you how SQL Server reacts: select * from a inner join b Line 1: Incorrect syntax near 'b'. select * from a inner join b on a.id = b.id | id | x | id | y | ------------------- | 1 | 1 | 1 | 2 | So SQL Server behaves like sqlite in returning column names, although it doesn't accept the syntax 'select * from a inner join b' I guess this behavior is normal and it's the programmer's resposability to use aliases in sql statement. Just my personal oppinion, though. #e8e8bd 1014 doc new 2004 Nov anonymous Parser 2004 Nov 5 5 Can't have column name "first" any more... I have an application where the main table is like: create table people ( id integer primary key, first varchar(20), last varchar(50), ...etc... This has never caused problems. But I just updated after the 'cursor' stuff was added, and my existing db can get a 'bad schema' error. After investigating, I found that sqlite was complaining about having a column 'first'. It doesn't mind 'last', which I find curious. _2004-Nov-24 22:48:14 by anonymous:_ {linebreak} I might point out that the documenation does *not* indicate 'first' is a reserved word. ---- _2004-Nov-24 23:57:11 by drh:_ {linebreak} I'd like to point out that the "cursor stuff" has not been released yet. When you use unstable code out of CVS you have to be prepared to encounter problems. #e8e8bd 853 doc active 2004 Aug anonymous Parser 2004 Aug 4 3 syntax of DEFAULT in a CREATE TABLE statement not as documented The syntax (as found under the syntax link on the website) indicates that a column default in a create table statement is like a column constraint definition. However, explicitly naming the column default fails. Consider: CREATE TABLE FOO( BAR VARCHAR(50) CONSTRAINT DEF_1 DEFAULT '(default)' ) yields SQL error: near "DEFAULT": syntax error #e8e8bd 851 doc active 2004 Aug anonymous Parser 2004 Aug 5 2 SQLite allows Zero length table column names SQLite allows zero length identifiers, that is, a statement like: CREATE TABLE [] ( "" not null , name ) does in fact create a (nameless) table with two (2) columns, be it that the first one ("", or []) is nameless. Although everything is still consistent (SELECT [] FROM "", or SELECT "" from [] does project the nameless column) it can be quit confusing. Consider this: INSERT INTO ""(name) VALUES ('value') generates SQL error: . may not be NULL At least the syntax documentation should state that it is in fact allowed to use zero length identifiers. _2004-Aug-10 16:03:02 by anonymous:_ {linebreak} C:\database>sqlite3.exe temp.db---- _2004-Aug-10 16:04:50 by anonymous:_ {linebreak} we can say sqlite allows zero length table & column name. #e8e8bd 797 new active 2004 Jul anonymous Parser 2004 Jul 4 3 require basic named subquery / WITH sql support This ticket, derived from a recent discussion list posting, is a request for *simple* named subquery / WITH sql support. The level of support that I'm looking for should only require and update to the "Parser" subsystem, or possibly "CodeGen" too. The fundamental implementation of how SQLite handles subqueries is not changed at all. You still execute the subqueries exactly once, prior to the main queries, as you do now. The subset would be compliant with the SQL-1999 standard. As an additional reference, you can see SQL-2003 Foundation, 7.13 "
SQLite version 3.0.4
Enter ".help" for instructions
sqlite> create table [] ( "" int,name);
sqlite> insert into [] values (1,'t');
sqlite> select * from [];
1|t
sqlite> .mode column
sqlite> .header on
sqlite> select * from [];
name
---------- ----------
1 t
sqlite>
sqlite> create table "" (i int);
SQL error: table "" already exists
sqlite>" (p351). What I want is to be able to make a query like this: WITH first_subq AS ( SELECT id FROM foo WHERE name LIKE '%zoo%' ), second_subq AS ( SELECT a, b, COUNT(c) AS d FROM bar GROUP BY a, b ) SELECT * FROM second_subq AS s INNER JOIN baz AS z ON z.a = s.a AND z.b = s.b WHERE (z.opt1 IN first_subq OR z.opt2 IN first_subq) AND s.d >= z.boo ORDER BY s.d DESC One of the main advantages I cite is that SQL code is a lot cleaner and easier to understand. You can do within complex selects the same thing you can do in complex routines, which is akin to breaking out blocks into named subroutines. This advantage is particularly seen where the same subquery would be getting invoked multiple times within the main query (see example). It now does not have to be declared multiple times, leading to shorter and easier to parse SQL, and the code runs faster, because the subquery only has to be run once. As such, developers can also reduce some use of explicit temporary tables. This ticket is not requesting support for passing arguments to the named subqueries, as SQL-1999 allows, nor having support for recursive named subqueries (those that invoke themselves). Those would be nice some day, but would require more substantial changes to SQLite, such as invoking subqueries during the main query, and multiple times, rather than in advance. So I am not requesting those today. (Note that, should you ever decide to support recursive subqueries later, it should be done the SQL-1999 way, and not Oracle 8's way of start-with connect-by.) In summary, I propose that in the long run, named subqueries should be a lot more useful than inlined subqueries, both for programmer efficiency, and because SQLite itself has less work to do when parsing or optimizing sql queries; the programmer can tell you ways to optimize by reducing redundancy. Thank you. -- Darren Duncan P.S. On the list, Dennis Cote also showed a desire for the same features, saying: I also believe support for named subqueries would be a valuable addition to SQLite. I have previously advocated for this feature on the list as well, though I called it a WITH clause rather than a named subquery. It seems to me to be a fairly simple extension to SQLite that would allow the user to manually perform common sub-expression elimination optimizations. These optimizations are done automatically by other, not so lite, database engines. I also think they make the resulting SQL select statements easier to read. #e8e8bd 726 new active 2004 May anonymous Parser 2004 May anonymous 5 5 As to single quote Hello when we insert single quote(') into the query, currently it should be added a more single quote. I think it is not good for most of program espcially in database system uses backslash(\) not single quote. To be appliable to other program, would you change this? _2004-May-07 15:31:06 by anonymous:_ {linebreak} if you don't have any plan, I think putting an escaping option will be great. thanks. #f2dcdc 637 code active 2004 Mar anonymous Parser 2004 Mar 5 4 union uses non-standard column names for other dbms, such as postgresql, mysql, etc, issuing a select x as name, y as value from first_table union select x, y from second_table will produce a resultset with column names of 'name' and 'value' (the first resultset). in sqlite it will produce a resultset with column names of 'x' and 'y' (the last resultset). Code causing this appears to be in multiSelect function in src/select.c. #e8e8bd 590 new active 2004 Jan anonymous Parser 2004 Jan 5 4 Treat ` like spaces outside of quoted strings. Mysql allows the usage of ` for quoting field names in queries. SQLite interrupts with "unrecognized token". To improve compatibility with mysql ` should either be handled correctly or simply ignored. For ignoring ` the following code can be used: case ' ': case '\t': case '\n': case '\f': case '\r': case '`': { for(i=1; isspace(z[i]) || z[i] == '`'; i++){} starting at line 224 in tokenize.c I know that this is a dirty enhancement but the structure of tokenize.c is rather unknown to me... #e8e8bd 563 new active 2004 Jan anonymous Parser 2004 Jan 4 3 Support for autoincrement type "SERIAL" (from PostgreSQL) I am porting my application from PostgreSQL to SQLite but I want to keep it backwards compatible. With PostgreSQL autoincrement fields are created with type SERIAL. For example: _:CREATE TABLE t1( a SERIAL PRIMARY KEY, b INTEGER); And statements like: _:INSERT INTO t1 VALUES(default,123); are used to create rows with automatically increased ids. #e8e8bd 517 new active 2003 Dec anonymous Parser 2003 Dec drh 1 1 \' sequence unrecognized token error enhancement request per an email exchange with Richard on 12/6/03, am requesting an enhancement to handle a situation where a sequence of \' in a SQL INSERT INTO data stream causes an unrecognized token error. This occurs when attempting to reload data that has been dumped with the phpMyAdmin application. There are a number of allowed escape sequences in MySQL namely: \0 => NUL (ASCII 0) \' \" \b \n \r \t \\ \z #e8e8bd 268 new active 2003 Mar anonymous Parser 2003 Nov 1 3 columns of subselects containing joins are not recognized The following is a transcript of a minimal sqlite session exhibiting the problem: SQLite version 2.8.0 Enter ".help" for instructions sqlite> create table A(x); sqlite> create table B(y); sqlite> select * from (select * from A,B) where x = 0; SQL error: no such column: x The problem does not appear to be with the statement _select * from (select * from A,B)_ itself, because that produces the correct results, but with the parser being unable to resolve the column name. Using an alias does not help: sqlite> select * from (select * from A,B) as t1 where t1.x = 0; SQL error: no such column: t1.x Obviously the simpler _select * from A,B where x = 0_ would work, but that's not a workaround in this case - I ran into this because I'm working with a relational algebra to SQL translator, that doesn't know how to simplify the expressions to more idiomatic SQL. #e8e8bd 478 new active 2003 Oct anonymous Parser 2003 Nov drh 1 3 select into is lacking hi, thanks for sqlite. i accidentally tryed a "select into" statement and found it is not parsed by sqlite. best regards, alex #e8e8bd 479 new active 2003 Oct anonymous Parser 2003 Oct drh 4 3 Sequences Adding sequences to sqlite would be very useful... allowing to do several things that are not possible today. But I'm not sure if default column values must be implemented to take full use of sequences. _2004-Mar-03 13:14:00 by anonymous:_ {linebreak} Any plans to add sequences? #f2dcdc 154 code active 2002 Sep drh Pager 2007 Jun drh 3 3 Prohibit links on database files. If a database file is aliased using either hard or symbolic links, it can happen that an aborted transaction will not roll back correctly. Consider this scenario. The database file is named both a.db and a.db. Application one opens a.db and starts to make a change. This creates a journal file a.db-journal. But application one crashes without completing the transaction. Later, application two attempts to open the database as b.db. App two looks for a journal file to rollback, but it thinks the journal should be named b.db-journal. So it fails to see the a.db-journal that app one left and fails to rollback the transaction. The only way I can think of to prevent this kind of thing it to refuse to open any database file that contains two or more hard links and to refuse to open a file through a symbolic link. _2004-Mar-16 20:46:17 by anonymous:_ {linebreak} What if the journal file name wasn't based on the database name, but instead was based on the starting inode of the database file? For instance, "journal-10293" would be used if the starting inode for the associated database file was 10293. ---- _2004-Mar-20 17:17:41 by anonymous:_ {linebreak} Using Inode-numbers to solve this problem is a dangerous proposition, as disk defragmenters can alter the inode the db starts at in between a crash and a subsequent roll-back attempt. ---- _2007-Jun-05 03:57:17 by anonymous:_ {linebreak} On unix, you can use ftok() to solve this problem. It guarantees to return the same key for all paths to the same file, including symbolic and hard links. I have to experience programming in Windows, but have no doubt a similar function call exists in that API. #f2dcdc 1861 code active 2006 Jun anonymous Pager 2006 Jun 1 1 Problem in using Triggers and multithreading I am using SQLite3 database with triggers . This database is used by my processing engine which is having 10 threads accessing the same database. Trigger is used to updata and insert records in a table and that very table is also updated by threads. Processing engine crashes whenever a trigger updates or inserts a record in the table. Can you tell me how to configure my existing engine to avoid crashing? Is it safe to use trigger? #e8e8bd 1860 doc active 2006 Jun anonymous Pager 2006 Jun danielk1977 1 1 Problem in using SQLite3 with trigger and multithreading I am using SQLite3 database with triggers . This database is used by my processing engine which is having 10 threads accessing the same database. Trigger is used to updata and insert records in a table and that very table is also updated by threads. Processing engine crashes whenever a trigger updates or inserts a record in the table. #e8e8bd 1837 new active 2006 Jun anonymous Pager 2006 Jun 4 4 Deadlock detection would be best reported using explicit error code. As discussed in email (*), an explicit deadlock error code might be beneficial for an application to detect and recover from deadlock. Deadlock is different to SQLITE_BUSY, and should be notified as such (IMHO). (*) http://www.mail-archive.com/sqlite-users%40sqlite.org/msg15979.html _2006-Jun-12 11:23:49 by drh:_ {linebreak} Changing the integer return code would be not be a backwards compatible change, so that approach must be rejected. But we will consider some mechanism to make it easier for programmers to figure out that they are dealing with deadlock and not a temporary delay. ---- _2006-Jun-12 15:43:59 by anonymous:_ {linebreak} Is there a fatal return code? If so, perhaps it could be overloaded with an additional function to determine if this 'fatal' error is actually a deadlock. Deadlocks are a common return code from almost every database I've ever programmed for. I am not sure how you can work around it otherwise. #e8e8bd 1801 new active 2006 May anonymous Pager 2006 May owensmk 5 4 Include Support for User-defined Lock Synchronization SQLite uses a busy-wait model for locking. For high concurrency applications this can be become inefficient. I have written a patch for =pager.c= which introduces two hooks - =lock()= and =unlock()= - whereby the application can participate in locking/sychronization of connections. This can in some cases increase overall concurrency by an order of magnitude. The callback implementation is very similar to the busy handler. All of the synchronization is implemented by the application. SQLite simply calls the hooks at the appropriate times, if they are registered. A full description of the patch is available at {link: http://www.gintana.com/sqlite/}. #f2dcdc 1799 code active 2006 May anonymous Pager 2006 May 2 3 temp_store=MEMORY slower than FILE for large intermediate result sets (This ticket was split off from #1790 because that ticket was becoming too broad.) When temp_store=MEMORY it can negatively effect the performance of queries with large intermediate result sets generated from SELECTs of either file-based tables or memory-based tables. This is true when sufficient RAM is available to the SQLite process to completely hold the intermediate results in memory without swapping to disk. In the example below, "big" is a file-based table in foo.db with 10 million rows. It was created with "create table big(x,y)". # unmodified stock SQLite built from May 5 2006 CVS (after check-in [3178]) # compiled with default settings for SQLITE_DEFAULT_PAGE_SIZE and N_PG_HASH $ time ./may5-sqlite/sqlite3 foo.db "PRAGMA temp_store = MEMORY; select x, y from big order by y, x" >/dev/null real 13m23.828s user 13m18.452s sys 0m0.811s # SQLite built from May 5 2006 CVS, but compiled with proposed change of # SQLITE_DEFAULT_PAGE_SIZE set to 4096, and N_PG_HASH set to 16384 $ time ./may5-sqlite-hash-opt/sqlite3 foo.db "PRAGMA temp_store = MEMORY; select x, y from big order by y, x" >/dev/null real 6m16.031s user 6m13.108s sys 0m0.811s Compiling with SQLITE_DEFAULT_PAGE_SIZE = 1024, and N_PG_HASH = 32768 resulted in the same timing as the may5-sqlite-hash-opt test run above. If temp_store=FILE (with default SQLite values for SQLITE_DEFAULT_PAGE_SIZE and N_PG_HASH), the timings are comparable to temp_store=MEMORY with SQLITE_DEFAULT_PAGE_SIZE=4096, and N_PG_HASH=16384. Large intermediate results sets can cause SQLite to spend more than half of its CPU time in the function pager_lookup(). By increasing the value of N_PG_HASH and SQLITE_DEFAULT_PAGE_SIZE, the time spent in pager_lookup can be reduced to near zero, thus doubling performance in such cases. % cumulative self self total time seconds seconds calls ms/call ms/call name 51.97 118.31 118.31 119658386 0.00 0.00 pager_lookup 4.36 128.25 9.94 4000009 0.00 0.06 sqlite3VdbeExec 3.06 135.21 6.96 315629923 0.00 0.00 parseCellPtr 3.05 142.16 6.95 171797186 0.00 0.00 sqlite3VdbeRecordCompare 2.67 148.22 6.07 12000005 0.00 0.01 sqlite3BtreeMoveto 2.14 153.10 4.88 343594380 0.00 0.00 sqlite3VdbeSerialGet 1.68 156.93 3.83 171797188 0.00 0.00 sqlite3MemCompare 1.63 160.65 3.72 77995781 0.00 0.00 sqlite3pager_get 1.60 164.29 3.65 169734946 0.00 0.00 sqlite3pager_unref 1.58 167.88 3.59 654100795 0.00 0.00 get2byte _2006-May-07 18:37:50 by anonymous:_ {linebreak} Timings on same Windows machine with check-in [3180] applied: # FILE $ time ./may7-sqlite/sqlite3 foo.db "PRAGMA temp_store = FILE; select x, y from big order by y, x" >/dev/null real 5m7.157s user 4m19.905s sys 0m20.827s # MEMORY $ time ./may7-sqlite/sqlite3 foo.db "PRAGMA temp_store = MEMORY; select x, y from big order by y, x" >/dev/null real 5m12.328s user 5m9.781s sys 0m0.984s Much better. temp_store=MEMORY is now competitive with FILE, although temp_store=FILE (when the OS is able to cache the file entirely in memory) is marginally faster. I still think the MEMORY time can be reduced further by another 20 seconds judging by the sys time of 20.827s in the FILE test. The MEMORY subsystem of SQLite ought to have an advantage over the FILE subsystem because it does not incur any system call overhead. I'll see if a profile turns up anything obvious. #f2dcdc 1790 code active 2006 May anonymous Pager 2006 May 3 3 :memory: performance difference between v2 and v3 Please see the following link for details: http://www.mail-archive.com/sqlite-users%40sqlite.org/msg14937.html Possible fix? RCS file: /sqlite/sqlite/src/pager.c,v retrieving revision 1.266 diff -u -r1.266 pager.c --- pager.c 7 Apr 2006 13:54:47 -0000 1.266 +++ pager.c 3 May 2006 19:02:17 -0000 @@ -1663,7 +1663,7 @@ pPager->memDb = memDb; pPager->readOnly = readOnly; /* pPager->needSync = 0; */ - pPager->noSync = pPager->tempFile || !useJournal; + pPager->noSync = pPager->tempFile || !pPager->useJournal; pPager->fullSync = (pPager->noSync?0:1); /* pPager->pFirst = 0; */ /* pPager->pFirstSynced = 0; */ _2006-May-03 19:32:12 by drh:_ {linebreak} The suggested change makes no difference in performance when I try it. ---- _2006-May-03 21:41:24 by anonymous:_ {linebreak} If transactions are not used, 85% of the time of this memory database benchmark is spent in pager_get_all_dirty_pages(). Each sample counts as 0.01 seconds. % cumulative self self total time seconds seconds calls ms/call ms/call name 85.25 31.20 31.20 100002 0.31 0.31 pager_get_all_dirty_pages 1.39 31.71 0.51 100011 0.01 0.20 sqlite3VdbeExec 1.17 32.14 0.43 10487713 0.00 0.00 parseCellPtr 0.63 32.37 0.23 12943618 0.00 0.00 sqlite3VdbeSerialGet 0.61 32.59 0.23 3432951 0.00 0.00 pager_lookup 0.52 32.78 0.19 4849544 0.00 0.00 sqlite3VdbeRecordCompare 0.44 32.95 0.16 400006 0.00 0.00 sqlite3BtreeMoveto 0.41 33.09 0.15 2064924 0.00 0.00 sqlite3pager_get 0.40 33.24 0.14 6471807 0.00 0.00 sqlite3MemCompare 0.06 31.25 100002/100002 sqlite3BtreeCommit [4] [5] 85.6 0.06 31.25 100002 sqlite3pager_commit [5] 31.20 0.00 100002/100002 pager_get_all_dirty_pages [6] 0.05 0.00 389365/389365 clearHistory [65] ----------------------------------------------- 31.20 0.00 100002/100002 sqlite3pager_commit [5] [6] 85.2 31.20 0.00 100002 pager_get_all_dirty_pages [6] ---- _2006-May-03 21:51:30 by anonymous:_ {linebreak} Stats with BEGIN/COMMIT enabled: Each sample counts as 0.01 seconds. % cumulative self self total time seconds seconds calls ms/call ms/call name 11.88 0.34 0.34 4849544 0.00 0.00 sqlite3VdbeRecordCompare 8.16 0.56 0.23 10487713 0.00 0.00 parseCellPtr 7.80 0.79 0.22 12943618 0.00 0.00 sqlite3VdbeSerialGet 6.38 0.96 0.18 100013 0.00 0.03 sqlite3VdbeExec 4.26 1.08 0.12 29816 0.00 0.02 balance_nonroot 3.90 1.20 0.11 6471807 0.00 0.00 sqlite3MemCompare 3.19 1.28 0.09 1964925 0.00 0.00 sqlite3pager_get 3.19 1.38 0.09 400006 0.00 0.00 sqlite3BtreeMoveto 2.84 1.46 0.08 19170231 0.00 0.00 get2byte 2.66 1.53 0.07 700015 0.00 0.00 sqlite3VdbeSerialPut 2.13 1.59 0.06 600993 0.00 0.00 sqlite3Malloc 1.77 1.64 0.05 4400155 0.00 0.00 sqlite3pager_unref 1.77 1.69 0.05 3332952 0.00 0.00 pager_lookup 1.77 1.74 0.05 1418379 0.00 0.00 decodeFlags 1.77 1.79 0.05 1332302 0.00 0.00 initPage 1.60 1.83 0.04 5270826 0.00 0.00 findOverflowCell 1.42 1.88 0.04 12181181 0.00 0.00 findCell 1.42 1.92 0.04 4849549 0.00 0.00 fetchPayload 1.42 1.96 0.04 359548 0.00 0.00 insertCell 1.24 1.99 0.04 4896877 0.00 0.00 parseCell 1.06 2.02 0.03 5284245 0.00 0.00 cellSizePtr 1.06 2.05 0.03 3227291 0.00 0.00 binCollFunc 1.06 2.08 0.03 2616113 0.00 0.00 _page_ref 1.06 2.11 0.03 1368027 0.00 0.00 reparentPage 1.06 2.14 0.03 934205 0.00 0.00 sqlite3GenericMalloc 1.06 2.17 0.03 300010 0.00 0.00 sqlite3BtreeCursor 0.89 2.19 0.03 2536689 0.00 0.00 get4byte 0.89 2.22 0.03 1864920 0.00 0.00 getPage ... 0.00 2.82 0.00 3 0.00 0.00 pager_get_all_dirty_pages 0.00 0.00 3/3 sqlite3BtreeCommit [116] [119] 0.0 0.00 0.00 3 sqlite3pager_commit [119] 0.00 0.00 6551/6551 clearHistory [118] 0.00 0.00 3/3 pager_get_all_dirty_pages [370] ----------------------------------------------- 0.00 0.00 3/3 sqlite3pager_commit [119] [370] 0.0 0.00 0.00 3 pager_get_all_dirty_pages [370] ---- _2006-May-03 22:27:35 by anonymous:_ {linebreak} with the outer BEGIN/COMMIT disabled, the memory database benchmark stats: static PgHdr *pager_get_all_dirty_pages(Pager *pPager){ // this point is reached 100,002 times PgHdr *p, *pList; pList = 0; for(p=pPager->pAll; p; p=p->pNextAll){ // this point is reached 322,956,271 times if( p->dirty ){ // this point is reached 389,365 times p->pDirty = pList; pList = p; } } return pList; } ---- _2006-May-04 05:23:08 by anonymous:_ {linebreak} This patch makes the test (with transaction) run 7% faster for gcc 3.4.4 with -O2. At -O3, gcc performs the inlining of these functions even without the inline hint, so this patch has no effect. RCS file: /sqlite/sqlite/src/btree.c,v retrieving revision 1.324 diff -u -3 -p -r1.324 btree.c --- btree.c 4 Apr 2006 01:54:55 -0000 1.324 +++ btree.c 4 May 2006 05:12:35 -0000 @@ -439,17 +439,17 @@ static int checkReadLocks(BtShared*,Pgno /* ** Read or write a two- and four-byte big-endian integer values. */ -static u32 get2byte(unsigned char *p){ +inline static u32 get2byte(unsigned char *p){ return (p[0]<<8) | p[1]; } -static u32 get4byte(unsigned char *p){ +inline static u32 get4byte(unsigned char *p){ return (p[0]<<24) | (p[1]<<16) | (p[2]<<8) | p[3]; } -static void put2byte(unsigned char *p, u32 v){ +inline static void put2byte(unsigned char *p, u32 v){ p[0] = v>>8; p[1] = v; } -static void put4byte(unsigned char *p, u32 v){ +inline static void put4byte(unsigned char *p, u32 v){ p[0] = v>>24; p[1] = v>>16; p[2] = v>>8; ---- _2006-May-04 19:44:57 by anonymous:_ {linebreak} I just want to confirm that a _file database is faster_ than a memory database for 3.3.5+. Are these numbers correct? 43,478 inserts/second best case for file for 3.3.5+ and 40,000 inserts/second best case for memory? Even with the OS caching the entire database file entirely in RAM, this finding is quite surprising. Test DB IDX TX RC 3.3.5+ 3.3.5 2.8.17 1 mem n y 1000000 40000 33333 76923 2 mem y y 1000000 27027 22727 58824 3 mem n n 1000000 35714 5263 83333 4 mem y n 1000000 24390 2778 62500 5 file n y 1000000 43478 35714 40000 6 file y y 1000000 28571 24390 23256 7 file n n 1000 11 11 13 8 file y n 1000 9 10 13 http://www.sqlite.org/cvstrac/attach_get/256/sqlite_speed.txt ---- _2006-May-04 20:19:18 by anonymous:_ {linebreak} I'm seeing slightly different results. The memory database using a transaction is (slightly) faster than the file-based database using a transaction. Timings on 3.3.5+ on Windows XP, gcc 3.4.4 -O3 -fomit-frame-pointer IDX TX # inserts wall time inserts/sec --- --- --------- ---------- ----------- mem no no 100,000 4.8s 20,833 mem no yes 100,000 4.3s 23,255 file no yes 100,000 4.7s 21,276 file no no 1,000 99.8s 10 ...things get worse for :memory: as you increase the number of inserts, while the file database numbers remain constant: IDX TX # inserts wall time inserts/sec --- --- --------- ---------- ----------- mem no yes 1,000,000 48.5s 20,638 mem no yes 2,000,000 118.6s 16,863 mem no yes 4,000,000 364.7s 10,967 file no yes 1,000,000 46.8s 21,354 file no yes 2,000,000 93.8s 21,321 file no yes 4,000,000 187.5s 21,333 Do Linux users get similar results? Considering I have 512K CPU L2 cache, I wonder if there's some CPU cache effect going on here with the way the :memory: db is allocated. ---- _2006-May-04 21:35:07 by anonymous:_ {linebreak} It seems there is some quadratic behavior in pager_lookup (latest CVS). 52% of the time is spent in that function. Profile data from :memory: db, TX on, no IDX, 4 million inserts: /* ** Find a page in the hash table given its page number. Return ** a pointer to the page or NULL if not found. */ static PgHdr *pager_lookup(Pager *pPager, Pgno pgno){ PgHdr *p = pPager->aHash[pager_hash(pgno)]; while( p && p->pgno!=pgno ){ p = p->pNextHash; } return p; } % cumulative self self total time seconds seconds calls ms/call ms/call name 51.97 118.31 118.31 119658386 0.00 0.00 pager_lookup 4.36 128.25 9.94 4000009 0.00 0.06 sqlite3VdbeExec 3.06 135.21 6.96 315629923 0.00 0.00 parseCellPtr 3.05 142.16 6.95 171797186 0.00 0.00 sqlite3VdbeRecordCompare 2.67 148.22 6.07 12000005 0.00 0.01 sqlite3BtreeMoveto 2.14 153.10 4.88 343594380 0.00 0.00 sqlite3VdbeSerialGet 1.68 156.93 3.83 171797188 0.00 0.00 sqlite3MemCompare 1.63 160.65 3.72 77995781 0.00 0.00 sqlite3pager_get 1.60 164.29 3.65 169734946 0.00 0.00 sqlite3pager_unref 1.58 167.88 3.59 654100795 0.00 0.00 get2byte 1.30 170.84 2.95 973877 0.00 0.07 balance_nonroot 1.27 173.74 2.90 56939555 0.00 0.00 initPage 1.24 176.56 2.83 171797188 0.00 0.00 binCollFunc 0.93 178.69 2.12 386371475 0.00 0.00 findCell 0.86 180.65 1.96 96207437 0.00 0.00 pageDestructor 0.83 182.53 1.89 95976540 0.00 0.00 _page_ref 0.80 184.36 1.83 2708031 0.00 0.00 assemblePage 0.80 186.19 1.82 41662605 0.00 0.00 reparentPage 0.74 187.88 1.70 171797188 0.00 0.00 fetchPayload 0.73 189.55 1.67 73995778 0.00 0.00 getPage 0.67 191.07 1.52 59647596 0.00 0.00 decodeFlags 0.63 192.51 1.44 132945443 0.00 0.00 findOverflowCell 0.62 193.93 1.41 40148167 0.00 0.00 sqlite3PutVarint 0.59 195.27 1.34 134687272 0.00 0.00 releasePage 0.59 196.62 1.34 73764879 0.00 0.00 getAndInitPage 0.54 197.84 1.22 8000003 0.00 0.02 sqlite3BtreeInsert 0.52 199.01 1.18 60000030 0.00 0.00 sqlite3VdbeSerialType 0.52 200.19 1.18 24000011 0.00 0.00 moveToRoot 0.49 201.30 1.10 179797130 0.00 0.00 getCellInfo 0.43 202.28 0.98 9882306 0.00 0.00 insertCell 0.42 203.22 0.94 47288132 0.00 0.00 moveToChild 0.40 204.15 0.92 173434760 0.00 0.00 parseCell 0.40 205.06 0.91 95806930 0.00 0.00 get4byte 0.34 205.83 0.78 41662605 0.00 0.00 sqlite3pager_lookup 0.33 206.57 0.74 165099370 0.00 0.00 sqlite3MallocFailed 0.32 207.31 0.73 20000010 0.00 0.00 sqlite3VdbeSerialPut 0.31 208.02 0.71 8000015 0.00 0.00 sqlite3VdbeHalt 0.30 208.70 0.68 27052986 0.00 0.00 sqlite3GetVarint 0.28 209.33 0.63 174637767 0.00 0.00 put2byte 0.27 209.96 0.62 8000006 0.00 0.00 sqlite3BtreeCursor 0.26 210.54 0.59 8148152 0.00 0.00 fillInCell 0.25 211.12 0.57 3385610 0.00 0.01 reparentChildPages 0.25 211.69 0.57 16000006 0.00 0.00 checkReadLocks 0.22 212.19 0.51 48000093 0.00 0.00 sqlite3VdbeFreeCursor 0.22 212.69 0.50 133898861 0.00 0.00 cellSizePtr 0.22 213.19 0.50 24000010 0.00 0.00 popStack 0.20 213.65 0.46 50076560 0.00 0.00 sqlite3pager_ref 0.20 214.10 0.45 pager_reset 0.19 214.54 0.44 8000024 0.00 0.00 closeAllCursors 0.19 214.97 0.42 12000024 0.00 0.00 sqlite3VdbeMemMakeWriteable 0.18 215.38 0.41 32000052 0.00 0.00 sqlite3VdbeMemSetStr 0.18 215.78 0.40 11616158 0.00 0.00 allocateSpace 0.17 216.16 0.39 8000000 0.00 0.00 bindText 0.16 216.51 0.35 25098767 0.00 0.00 sqlite3MallocRaw 0.16 216.87 0.35 8000005 0.00 0.00 sqlite3BtreeCloseCursor 0.15 217.22 0.34 45560699 0.00 0.00 sqlite3FreeX 0.15 217.56 0.34 36000014 0.00 0.00 sqlite3VarintLen 0.15 217.90 0.34 36000009 0.00 0.00 sqlite3VdbeMemShallowCopy 0.14 218.22 0.33 47999969 0.00 0.00 sqlite3VdbeSerialTypeLen 0.14 218.54 0.33 4000008 0.00 0.00 sqlite3VdbeMakeReady ----------------------------------------------- 41.19 0.00 41662605/119658386 sqlite3pager_lookup [15] 77.12 0.00 77995781/119658386 sqlite3pager_get [8] [5] 52.0 118.31 0.00 119658386 pager_lookup [5] ----------------------------------------------- 0.19 4.02 4000003/77995781 sqlite3BtreeGetMeta [28] 3.53 74.30 73995778/77995781 getPage [9] [8] 36.0 3.72 78.31 77995781 sqlite3pager_get [8] 77.12 0.00 77995781/119658386 pager_lookup [5] 1.12 0.00 56939550/95976540 _page_ref [40] 0.03 0.00 230897/230897 page_remove_from_stmt_list [139] 0.03 0.00 230897/230897 makeClean [138] 0.01 0.00 230897/461804 sqlite3pager_pagecount [150] 0.00 0.00 230897/25098767 sqlite3MallocRaw [58] ----------------------------------------------- 1.82 44.94 41662605/41662605 reparentChildPages [13] [14] 20.5 1.82 44.94 41662605 reparentPage [14] 0.78 41.96 41662605/41662605 sqlite3pager_lookup [15] 2.21 0.00 41672966/131189801 sqlite3pager_unref [31] 0.00 0.00 93099/50076560 sqlite3pager_ref [75] ----------------------------------------------- 0.78 41.96 41662605/41662605 reparentPage [14] [15] 18.8 0.78 41.96 41662605 sqlite3pager_lookup [15] 41.19 0.00 41662605/119658386 pager_lookup [5] 0.77 0.00 39036990/95976540 _page_ref [40] ----------------------------------------------- 0.77 0.00 39036990/95976540 sqlite3pager_lookup [15] 1.12 0.00 56939550/95976540 sqlite3pager_get [8] [40] 0.8 1.89 0.00 95976540 _page_ref [40] ---- _2006-May-04 21:41:37 by anonymous:_ {linebreak} I guess increasing this array size is in order: PgHdr *aHash[N_PG_HASH]; /* Hash table to map page number to PgHdr */ Too many hash collisions leading to growing linked lists in buckets. Or perhaps pager_hash has to be replaced with a better hash function. ---- _2006-May-04 22:04:47 by anonymous:_ {linebreak} Increasing the size of N_PG_HASH to 8192 seems to help the "4 million insert in a transaction into a memory database" benchmark. It now runs in 203.5 seconds (19656 inserts/sec), as opposed to 364.7 seconds (10967 inserts/sec) previously. This is closer to the 187.5 seconds for the file-based database timing. ---- _2006-May-04 22:13:16 by anonymous:_ {linebreak} Increasing N_PG_HASH to 16384 yields 21,052 inserts/second for a 4 million insert single-transaction :memory: database no-index run. This is very close to the file database figure of 21,333 inserts/second. ---- _2006-May-04 22:23:19 by anonymous:_ {linebreak} Setting N_PG_HASH to 32768 yields 21,621 inserts/second in the 4M insert s in a single-transaction in a memory db test. This is marginally faster than the file based database timing. Increasing N_PG_HASH has diminishing returns after 16384. ---- _2006-May-05 15:33:31 by anonymous:_ {linebreak} You should get the same effect if you increase the page size instead of increasing the size of the hash table. With a larger page size there will be fewer pages to be managed by the hash table. This might be a better solution for many applications. A hash table with 32K entries occupies 128K of RAM, whether it is used or not. ---- _2006-May-05 19:37:51 by anonymous:_ {linebreak} 128K of RAM when dealing with a 230M :memory: database is not terribly significant. Here's the timings for various N_PG_HASH and SQLITE_DEFAULT_PAGE_SIZE values for 4 million inserts into a :memory: database in a single transaction: N_PG_HASH SQLITE_DEFAULT_PAGE_SIZE inserts/sec --------- ------------------------ ----------- 16384 4096 21,622 32768 1024 21,621 8192 8192 20,513 4096 4096 20,101 4096 8192 19,417 2048 4096 16,878 2048 8192 16,598 2048 16384 15,038 2048 32768 13,937 2048 1024 10,782 So it seems the default values of N_PG_HASH and SQLITE_DEFAULT_PAGE_SIZE should be raised. ---- _2006-May-05 21:34:01 by anonymous:_ {linebreak} My point was that most users do not have 230 MB memory databases, so having a large hash table which is fixed at that size may be a burden. 128K for the hash table is a lot if you only have 128K in your memmory database. I agree that increasing these values would seem to provide a substantial performance increase at little cost. I would suggest using the 4K hash table and the 4K page size. These values are close to the current values. Many users have reported a general speed improvement using a page size of 4K which matches the value used by WinXP (and think many other Os's as well) for disk I/O blocks. These values nearly double the insert rate over the current default values. The fixed size hash table only takes twice the space. ---- _2006-May-06 14:54:32 by anonymous:_ {linebreak} Memory page speed should be as fast as possible as it effects the general performance of SQLite. Perhaps a static hash table is not the best data structure here. Don't temp tables and intermediate select results on file-based tables use memory-based pages? Making memory page speed as fast as possible will improve overall SQLite performance whether you are using a file or memory based database. For example, when ordering result sets from a file-based database select this routine is used to generate the code: static void pushOntoSorter( Parse *pParse, /* Parser context */ ExprList *pOrderBy, /* The ORDER BY clause */ Select *pSelect /* The whole SELECT statement */ ){ Vdbe *v = pParse->pVdbe; sqlite3ExprCodeExprList(pParse, pOrderBy); sqlite3VdbeAddOp(v, OP_Sequence, pOrderBy->iECursor, 0); sqlite3VdbeAddOp(v, OP_Pull, pOrderBy->nExpr + 1, 0); sqlite3VdbeAddOp(v, OP_MakeRecord, pOrderBy->nExpr + 2, 0); sqlite3VdbeAddOp(v, OP_IdxInsert, pOrderBy->iECursor, 0); For those of us who have very complicated nested sub-selects of file-based tables in many queries or even ORDER BYs on huge result sets, speeding up the memory page performance should be a performance win for SQLite in general. ---- _2006-May-06 17:37:32 by anonymous:_ {linebreak} The following test demonstrates that this memory page issue can greatly effect the performance of queries against file-based tables if temp_store is set to MEMORY. "big" is a file-based table in foo.db with 10 million rows. It was created with "create table big(x,y)". # unmodified stock SQLite built from May 5 2006 CVS (after check-in [3178]) # compiled with default settings for SQLITE_DEFAULT_PAGE_SIZE and N_PG_HASH $ time ./may5-sqlite/sqlite3 foo.db "PRAGMA temp_store = MEMORY; select x, y from big order by y, x" >/dev/null real 13m23.828s user 13m18.452s sys 0m0.811s # SQLite built from May 5 2006 CVS, but compiled with proposed change of # SQLITE_DEFAULT_PAGE_SIZE set to 4096, and N_PG_HASH set to 16384 $ time ./may5-sqlite-hash-opt/sqlite3 foo.db "PRAGMA temp_store = MEMORY; select x, y from big order by y, x" >/dev/null real 6m16.031s user 6m13.108s sys 0m0.811s This is not even what I would consider to be a big table. I should mention that compiling with SQLITE_DEFAULT_PAGE_SIZE = 1024, and N_PG_HASH = 32768 resulted in same timing as the may5-sqlite-hash-opt test run above. A pretty good return for an extra 126K. ---- _2006-May-08 04:07:25 by anonymous:_ {linebreak} You now get 20,725 inserts/second as of the latest check-in [3180] for 4 million inserts into a :memory: database in a single transaction (using the default SQLITE_DEFAULT_PAGE_SIZE of 1K). This is nearly twice as fast as SQLite prior to the check-in [3180] (10,782 inserts/second). However, it is 4% slower than the best timing prior to [3180] when compiled with N_PG_HASH=32768 and SQLITE_DEFAULT_PAGE_SIZE=1024 which got 21,622 inserts/second (see table above). Increasing the size of SQLITE_DEFAULT_PAGE_SIZE with the latest CVS either has no effect or makes the memory insert benchmark timings slightly worse. #f2dcdc 1754 code active 2006 Apr anonymous Pager 2006 Apr anonymous 2 2 Version 3.3.5 error if SQLITE_OMIT_MEMORYDB is defined My solution is to move the following code to de line the the syncJournal is "forwarded" at line 1809. #ifndef SQLITE_OMIT_MEMORYDB /* ** Clear a PgHistory block */ static void clearHistory(PgHistory *pHist){ sqliteFree(pHist->pOrig); sqliteFree(pHist->pStmt); pHist->pOrig = 0; pHist->pStmt = 0; } #else #define clearHistory(x) #endif#e8e8bd 1703 new active 2006 Mar anonymous Pager 2006 Mar 2 3 Second parameter to gettimeofday() in os_unix.c should be NULL in os_unix.c, function sqlite3UnixCurrentTime(): the second argument to gettimeofday() should be NULL and the declaration of sTz should be removed. struct timezone seems to cause trouble on Linux systems. #f2dcdc 1504 code active 2005 Nov anonymous Pager 2005 Nov 1 3 Multithreaded DB lock not released using Begin/Commit between threads When using transaction-based insertion of rows in v3.2.7 in a multi-threaded environment, one thread appears not to release the database lock for any competing threads to be able to issue an "INSERT" statement (the thread issuing the "COMMIT" apparently). This problem does not appear in v2.8.16. The problem also appears in Linux (RH-9) as well. I'm attaching "testsqlite.c", a test application (compiled in a WIN32 environment) that will duplicate the issue (Define SQLITE_2_8_16 or SQLITE_3_2_7 depending on the version of SQLite library to test against). Direct any questions to Erik -> lonepenguin@hotmail.com Thank You. #f2dcdc 1403 code active 2005 Sep anonymous Pager 2005 Sep 4 4 Newly initialized pages aren't completely cleared Rational Purify reports many, many UMRs within SQLite, all originating from the following code path: sqlite3OsWrite [os_win.c:296]{linebreak} pager_write_pagelist [pager.c:2209]{linebreak} sqlite3pager_get [pager.c:2445]{linebreak} getPage [btree.c:1114]{linebreak} allocatePage [btree.c:3153]{linebreak} At a glance, it appears that the problem lies in sqlite3pager_get, which does appear to fully clear the newly allocated page. A portion of the Purify-generated backtrace is attached. _2005-Sep-01 15:41:50 by drh:_ {linebreak} SQLite does not clear "don't-care" sections of the page. Doing so would use CPU cycles unnecessarily and hence slow things down. That unused sections of a database page are never initialized is a feature of SQLite, not a bug. ---- _2005-Sep-01 16:17:17 by anonymous:_ {linebreak} Would it be possible to clear the entire page when NDEBUG is not defined? Admittedly, not clearing the page has no ill effects; it does clutter the output from Purify quite a bit though, which will inevitably lead to missing a real error amidst the noise. ---- _2005-Sep-01 17:59:04 by anonymous:_ {linebreak} Actually, it looks like I was off the mark as to what's causing this. It isn't writing a newly allocated page that's causing the warning, but reuse of an existing page (from pPager->pFirstSynced). So I'm not sure what needs to be cleared to avoid the warning; some range within the page's memory area, obviously, but it's not as simple as just memset'ing the entire memory area like it would be when the page is freshly allocated earlier in sqlite3pager_get. #e8e8bd 1254 new active 2005 May anonymous Pager 2005 May 3 3 better pager_cksum pager_cksum currently reads one byte every 200 bytes in a page and computes a checksum from this. Suggest changing this to reading 32-bits every 200 bytes. This will run a the same speed, but the likelyhood of detecting errors is larger, since 400% more data is used in the checksumming. #e8e8bd 1250 new active 2005 May anonymous Pager 2005 May 1 3 FlushFileBuffers makes Windows sqlite dead slow compared to Linux On Windows, FlushFileBuffers is used to commit changes to the disk. This makes Windows sqlite very slow in comparison with sqlite on Linux, because fsync() doesn't flush the disk controller's cache like FlushFileBuffers does. When opening the database file and journal file, there should be an option to use the FILE_FLAG_WRITE_THROUGH file flag. This will make sure that all writes go through the OS cache directly to the disk before the write function returns. This will give a small slowdown for HUGE commits, but the speedup for small commits is substantial. FILE_FLAG_WRITE_THROUGH is 5-10x faster than FlushFileBuffers for all commits that don't write to a huge number of pages. For commits in the size range of 5-10 megabytes, FILE_FLAG_WRITE_THROUGH is around 2x slower. _2005-May-18 17:49:22 by anonymous:_ {linebreak} Attaching sqlite.patch that adds a global variable that controls the write through status. This is not ideal, but it allows you to benchmark the differences easily. ---- _2006-Jun-06 18:34:18 by anonymous:_ {linebreak} FILE_FLAG_WRITE_THROUGH is the norm for Microsoft's SQL Server. http://support.microsoft.com/default.aspx?scid=kb;en-us;234656 #e8e8bd 1240 new active 2005 May anonymous Pager 2005 May drh 1 3 Need integration of Apple's file locking callbacks improvement Now that OSX 10.4 has been released to the public; Apple has dedicated their changes to improve and support locking on the OSX platform (as well as others) to the public domain. These changes will also make it possible to use sqlite on prior versions of OSX. They externalize the necessary file locking into a callback system which greatly simplifies maintaining this area of persistent headaches. Apple has also included support for F_FULLFSYNC as described here: http://lists.apple.com/archives/darwin-dev/2005/Feb/msg00072.html Apple's changes are to a base distro of sqlite 3.1.3 The sources can be found at: http://www.opensource.apple.com/darwinsource/tarballs/other/SQLite-28.tar.gz #f2dcdc 1053 code active 2004 Dec anonymous Pager 2004 Dec 3 3 SQLITE_IOERR and strange rollback when db is busy Environment on which bug was found:{linebreak} Windows XP, both SP1 and SP2, on different computers. The SQLite library was built using the precompiled source from the download page (as static library). Description of bug scenario: One process performs very long reads from a db (multiple joins, so the cartesian product is *very* large, and the reader needs a while to complete). Another process performs a _BEGIN TRANSACTION_ , then executes lots of _INSERT INTO ... VALUES_ .{linebreak} At some point, this process will end up in sqlite3pager_get, when it tries to read some page from the database file (the main file, not a temp file or a journal). It detects that the page is not in the page cache (it ends up in the 'else' branch of _if( pPg==0 )_ ). It runs down to the block of code covered by the following comment: /* Write the page to the database file if it is dirty. */ In this block, pager_write_pagelist( pPg ) returns with SQLITE_BUSY. As a consequence, the changes are rolled back and SQLITE_IOERR is returned. And here seems to be the problem: First, the database file is locked, so I don't understand why the SQLITE_BUSY value isn't propagated back to the caller. If SQLITE_BUSY would be returned, then the application could restart the command. Seconds, sqlite3VdbeHalt decides to perform a sqlite3BtreeRollbackStmt, so only the last command should be rolled back. However, this is not what happens! In fact, all commands back to the beginning of the transaction are rolled back; the transaction, however is not closed. Doesn't this violate the default rollback behaviour (roll back last command, keep transaction open)? As a consequence, even if the application would get SQLITE_BUSY, it couldn't properly react on it. There are other places in sqlite3pager_get where SQLITE_IOERR are returned; I've not checked whether these can also be triggered by the db being locked or if they indicate serious problem. I will attach the code I used to reproduce and track down the problem, together with a Visual Studio 2003 project. If you extract the archive, on toplevel you will find the following: *: Reader: the directory containing the source for the reader *: Writer: the directory contaiing the source for the writer *: SQLite: A directory in which to place the precompiled source for windows users, which is used to build the library. If you want to use the provided project file with Visual Studio, just copy the source in there and everything will build with a single mouse click. *: BugDemo.sln: The Visual Studio project file. *: bugdemo.sql: The SQL statements used to create the test database. How to reproduce: *: Create a database using bugdemo.sql *: Adapt reader.cpp and writer.cpp to include the sqlite3 headers, and set the define at the top of the files to the path of the test database. *: Compile everything. *: Start the reader. *: Start the writer, and wait until it reports an error (for me, it takes < 30 seconds). I tried to keep the source portable, so it shouldn't be too hard to make it compile on Unix. #e8e8bd 646 warn active 2004 Mar anonymous Pager 2004 Mar 5 4 Pager.c compilation warnings Compilation of pager.c with OpenWatcom 1.3 reports d:\!progs\sqlite\src\pager.c(360): Warning! W201: Unreachable code d:\!progs\sqlite\src\pager.c(1674): Warning! W201: Unreachable code Bartosz Polednia #e8e8bd 175 new active 2002 Oct anonymous Pager 2002 Oct 5 3 journal file should be kept open In windows at least, the journal file used to manage transactions is being created and deleted for every transaction. this makes transactions slow in the following case: if a virus scanner is active, it will attempt to scan the file being created *every time* , and it will add 50-100 ms to each transaction. While starting an explicit transaction makes the problem smaller, in an environment where many atomic transactions with a single INSERT or UPDATE must be executed, this slows things down quite a lot. I understand that the peculiarity of the virus scanner actually creates the problem, but , is it a good design decision to keep creating and droping the file ? of course, there is the issue of multiple process accessing the same database, in which case multiple journals would exist, but this should/would be resolved by the locking mechanism anyway... #e8e8bd 139 new active 2002 Aug anonymous Pager 2002 Aug 3 3 Named transactions are misleading as documented As documented it appears as though you can assign meaningful names to transactions, but in practice the names are just a throwaway for SQL92 compatibility. Nested transactions could be implemented if names were used -- perhaps by using markers in the journal file. This method would work well and provide for unlimited nesting of transactions given that sqlite does not support concurrent transactions. Example nested transaction: begin transaction t1; -- creates journal: mydb-journal ... inserts, updates, deletes, etc... ... begin transaction t1a; -- adds a marker (e.g. 'begin t1a') to the journal ... inserts, updates, deletes, etc... ... commit transaction t1a; -- adds a marker (e.g. 'commit t1a') to the journal ... begin transaction t1b; -- adds a marker (e.g. 'begin t1b') to the journal ... ... begin transaction t1b1; -- adds a marker (e.g. 'begin t1b1') to the journal ... commit transaction t1b1; ... rollback transaction t1b; -- rolls back the database to journal marker 'begin t1b' rollback transaction t1; -- rolls back to the beginning of the journal, undoing -- the commit performed in transaction t1a. #f2dcdc 1878 code active 2006 Jun anonymous CodeGen 2006 Jun 2 3 No index used when specifying alias name in ORDER BY clause Using an alias name in the ORDER BY clause prevents indices from being used in the query for sorting purposes: For this schema: CREATE TABLE t1 (c1, c2); CREATE TABLE t2 (c3, c4); CREATE INDEX t1_idx ON t1(c2); the following select query: EXPLAIN QUERY PLAN SELECT t1.c2 AS col2, t2.c4 AS col4 FROM t1 LEFT JOIN t2 ON t1.c1=t2.c3 ORDER BY t1.c2; will indeed use index t1_idx: sqlite> EXPLAIN QUERY PLAN SELECT t1.c2 AS col2, t2.c4 AS col4 FROM t1 LEFT JOIN t2 ON t1.c1=t2.c3 ORDER BY t1.c2; 0|0|TABLE t1 WITH INDEX t1_idx 1|1|TABLE t2 However, when using the alias name =col2= in the =ORDER BY= clause, the index won't be used: sqlite> EXPLAIN QUERY PLAN SELECT t1.c2 AS col2, t2.c4 AS col4 FROM t1 LEFT JOIN t2 ON t1.c1=t2.c3 ORDER BY col2; 0|0|TABLE t1 1|1|TABLE t2 IMHO, the same index should be used in both queries? _2006-Jun-30 13:54:10 by anonymous:_ {linebreak} Not sure whether it's a different issue, but when using a second column in the ORDER BY clause, also no index will be used: sqlite> EXPLAIN QUERY PLAN SELECT t1.c2 AS col2, t2.c4 AS col4 FROM t1 LEFT JOIN t2 ON t1.c1=t2.c3 ORDER BY t1.c2, t2.c4; 0|0|TABLE t1 1|1|TABLE t2 Personally, I'd expect sqlite to use the =t1_idx= index as well to fulfill the primary ordering? ---- _2006-Jun-30 16:04:31 by anonymous:_ {linebreak} As a workaround try "ORDER BY 1" ---- _2006-Jul-03 08:41:01 by anonymous:_ {linebreak} Sorry, I'm not sure how "ORDER BY 1" would be a workaround, when I really need the results to be sorted by table column data... (I don't want to start a discussion in the bug tracker, so you're welcome to take any suggestions/answers to the sqlite-user mailing list, which I also monitor.) ---- _2006-Jul-03 15:51:11 by anonymous:_ {linebreak} I'm not the poster of previous comment, but ORDER BY (n) order by result column index. In your case, using ORDER BY 1, it will be ordered by the first column. ---- _2006-Jul-04 07:33:30 by anonymous:_ {linebreak} Thanks for the clarification. This would be a workaround for the first problem mentioned, but when sorting by two columns, still no index will be used, even if using =ORDER BY 1,2= ---- _2006-Jul-04 21:34:24 by anonymous:_ {linebreak} SQLite really needs a way to explicitly state which index(es) to use. Perhaps something similar to Oracle's comment hints. #e8e8bd 1874 new active 2006 Jun anonymous CodeGen 2006 Jun 2 3 IN is much slower than making separate queries I have a 500,000-row table with the following schema: CREATE TABLE foo(bar INTEGER NOT NULL, baz INTEGER NOT NULL, biz INTEGER NULL, buzz INTEGER NULL); CREATE INDEX biz ON foo (bar, baz, biz); CREATE INDEX buzz ON foo (bar, baz, buzz); I'm performing the query: SELECT * FROM foo WHERE bar IN (0,1) AND baz IN (0,1) AND (biz IN (0,1) OR buzz IN (0,1)); On both Apple's 3.1.3 and a stock 3.3.6 on Mac OS X 10.4.6 PowerPC, this query consistently takes 3 seconds to execute. However, if I unroll the query: SELECT * FROM foo WHERE bar=0 AND baz=0 AND biz=0; SELECT * FROM foo WHERE bar=0 AND baz=0 AND buzz=0; ...and so on for the other values of bar and baz... it takes 0.2 seconds. I was able to reproduce this with the attached scripts by doing: ./mkdb > mkdb.sql sqlite3 testdb < mkdb.sql time sqlite3 testdb < q1 > /dev/null time sqlite3 testdb < q2 > /dev/null The database was recreated between the 3.1.3 and the 3.3.6 testing. EXPLAIN on the IN query segfaults on both 3.1.3 and 3.3.6, otherwise I'd attach that output. I'll write up a separate bug for that :) _2006-Jun-27 20:52:35 by anonymous:_ {linebreak} [3315] fixed the EXPLAIN crash, so I've attached EXPLAIN output for the IN query. ---- _2006-Jun-27 22:01:31 by drh:_ {linebreak} On SuSE Linux 10.0 running on a Dell Latitude D600 laptop and using the latest code from CVS, I'm getting times of 1.2s and 0.45s. If I create an additional index: CREATE INDEX i2 ON foo(bar,baz,biz,buzz); then the time for the first query drops to 0.7s. Note, however, the q1 and q2 are very different queries. In particular q2 omits half the rows. The (rough) equivalent of q2 is this: SELECT * FROM foo WHERE bar IN (0,1) AND baz IN (0,1) AND (biz=0 or buzz=0); If I modify q2 so that it includes the biz=1 and buzz=1 cases, its query time increases to 0.7s, the same as q1 with the added index. Further note that q1 and q2 are still not exactly the same. Q2 includes multiple copies of rows where biz IN (0,1) AND buzz IN (0,1) where q1 only includes such lines once. There are only 120 such lines in the database, but it still a difference. I will recast this ticket as a request for performance enhancements on queries using the IN operator on a fixed list of values. ---- _2006-Jun-28 01:52:32 by drh:_ {linebreak} Note to self: The expression +x IN (1,2,3,...) appears to be faster than +x=1 OR +x=2 OR ... when there are 6 or more terms. With 5 terms or fewer, a string of ORs is faster. #f2dcdc 1809 code active 2006 May anonymous CodeGen 2006 May 1 3 Huge slowdown/increased memory use when using GROUP BY on big dataset This seemingly nonsensical query is a greatly reduced test case taken from several queries I use with SQLite 3.2.1. The real example joins various huge tables and much more complicated views. I'd like to upgrade beyond SQLite 3.2.1, but this is a showstopper. It takes 13 seconds to run on SQLite 3.2.1 and uses just 1.2M of memory. With 3.3.5+ from CVS it takes 185 seconds and uses 230M of memory. PRAGMA temp_store=MEMORY; CREATE TABLE n1(a integer primary key); INSERT INTO "n1" VALUES(1); INSERT INTO "n1" VALUES(2); INSERT INTO "n1" VALUES(3); INSERT INTO "n1" VALUES(4); INSERT INTO "n1" VALUES(5); INSERT INTO "n1" VALUES(6); INSERT INTO "n1" VALUES(7); INSERT INTO "n1" VALUES(8); INSERT INTO "n1" VALUES(9); INSERT INTO "n1" VALUES(10); INSERT INTO "n1" VALUES(11); INSERT INTO "n1" VALUES(12); INSERT INTO "n1" VALUES(13); INSERT INTO "n1" VALUES(14); INSERT INTO "n1" VALUES(15); CREATE VIEW vu as select v3.a a, v5.a-v2.a*v7.a b from n1 v1,n1 v2,n1 v3,n1 v4,n1 v5,n1 v6,n1 v7; select a a, sum(b) T from vu where a=7 group by a; It seems that SQLite 3.2.1 had a much more efficient GROUP BY algorithm that discarded unnecessary data as the view was traversed. _2006-May-13 03:01:28 by anonymous:_ {linebreak} Seeing as this ticket concerns the GROUP BY statement it would make more sense to have an example like this: select a a, sum(b) T from vu where a<4 group by a; But both queries exhibit the same slowdown and memory increase, in any event. ---- _2006-May-13 15:09:39 by anonymous:_ {linebreak} This GROUP BY slowdown/memory increase is not specific to VIEWs. I repeated the test against a comparably sized table with the same results. You'll see this effect for any SELECT operating on a large number of rows using GROUP BY. ---- _2006-May-13 16:44:04 by anonymous:_ {linebreak} The slowdown first appears in SQLite 3.2.6 in check-in [2662]. ---- _2006-May-24 13:19:29 by anonymous:_ {linebreak} Here's an example to show an actual real-life use of GROUP BY in SQLite <= 3.2.5... Imagine performing mathematical operations on every combination of rows in several large tables for statistical analysis. The GROUP BY algorithm change in 3.2.6 now makes using GROUP BY on huge cross joins not usable for this purpose because it creates an intermediate result set of the product of all cross joins - several times larger than the size of the (already huge) database itself. Indexing is not useful in this case because there is nothing to index by design. All table rows must be traversed. Older versions of SQLite performed this operation extremely efficiently because grouping took place in the main traversal loop. I would think that the old algorithm could be used, but instead of keeping the intermediate results in memory, an index and a table in temp store could be used. #f2dcdc 1714 code active 2006 Mar anonymous CodeGen 2006 Mar 4 4 Slow query when tables in 2 different files This slow query seems to be sped up by using an explicit CROSS JOIN. ANALYZE apparently does not help. The tables span 2 different database files. Taken from the SQLite mailing list: -- table1.schema (file 1) ATTACH DATABASE './table1.db' AS t1 ; CREATE TABLE t1.table1 ( i_id INT4, b_id INT4, d_id INT4, c_id INT2, data_in REAL, data_out REAL ); CREATE INDEX t1.ix_table1_b_id ON table1( b_id ); DETACH DATABASE t1 ; -- table2.schema (file 2) ATTACH DATABASE './table2.db' AS t2 ; CREATE TABLE t2.table2 ( d_id INT4 PRIMARY KEY, r_id INT2, m_id INT2, i TEXT, ct TEXT, cc TEXT, type TEXT, notes TEXT ); DETACH DATABASE t2 ; -- the slow query (does not use indexes on both tables?) select t1.b_id, t1.c_id, t2.r_id, t2.m_id, sum( t1.data_in ) as data_in, sum( t1.data_out ) as data_out from table1 t1 join table2 t2 on t2.d_id = t1.d_id and t1.b_id >= 100 and t1.b_id < 200 group by t1.b_id, t1.c_id, t2.m_id, t2.r_id; -- the fast query (seems to use both tables' indices) select t1.b_id, t1.c_id, t2.r_id, t2.m_id, sum( t1.data_in ) as data_in, sum( t1.data_out ) as data_out from table1 t1 cross join table2 t2 where t2.d_id = t1.d_id and t1.b_id >= 100 and t1.b_id < 200 group by t1.b_id, t1.c_id, t2.m_id, t2.r_id; More information can be found here: http://www.mail-archive.com/sqlite-users%40sqlite.org/msg13648.html #e8e8bd 1653 new active 2006 Feb anonymous CodeGen 2006 Feb 4 5 SQLITE_OMIT_COST_REORDERING ? Embedded SQLite installations could benefit from conditionally compiling out the logic of the cost-based reordering of tables (i.e., always assume cross join). It might save a few kilobytes of code when the join order is always known in advance. #f2dcdc 1465 code active 2005 Oct anonymous CodeGen 2005 Oct 1 1 fdatasync not available and not yet fixed fdatasync is still there ... i downloaded the current configure files which should check for it. do i have to use something else too? ./libtool --mode=link cc -g -DOS_UNIX=1 -DHAVE_USLEEP=1 -I. -I./src -DNDEBUG -DTHREADSAFE=0 -DSQLITE_OMIT_CURSOR -DHAVE_READLINE=0 \ -o sqlite3 ./src/shell.c libsqlite3.la -lcurses cc -g -DOS_UNIX=1 -DHAVE_USLEEP=1 -I. -I./src -DNDEBUG -DTHREADSAFE=0 -DSQLITE_OMIT_CURSOR -DHAVE_READLINE=0 -o .libs/sqlite3 ./src/shell.c ./.libs/libsqlite3.so -lcurses "./src/shell.c", line 355: warning: argument #1 is incompatible with prototype: prototype: pointer to const unsigned char : "./src/shell.c", line 84 argument : pointer to const char "./src/shell.c", line 523: warning: argument #1 is incompatible with prototype: prototype: pointer to const unsigned char : "./src/shell.c", line 84 argument : pointer to char "./src/shell.c", line 694: warning: argument #2 is incompatible with prototype: prototype: pointer to const char : "./src/shell.c", line 583 argument : pointer to const unsigned char Undefined first referenced symbol in file fdatasync ./.libs/libsqlite3.so ld: fatal: Symbol referencing errors. No output written to .libs/sqlite3 make: *** [sqlite3] Error 1 _2005-Oct-04 21:42:26 by drh:_ {linebreak} Clear out your build directory (or start a new one) and rerun configure from scratch. Save the output of configure. Then rerun make. If you still have a problem, attach the output of configure to this ticket. #f2dcdc 1201 code active 2005 Apr anonymous CodeGen 2005 Apr danielk1977 1 1 erro defined type UINT8_TYPE In file sqliteInt.h line 203 typedef UINT8_TYPE i8; /* 1-byte signed integer */ it should be INT8_TYPE. #e8e8bd 1184 todo active 2005 Mar danielk1977 CodeGen 2005 Mar 4 3 UTF-8 and UTF-16 substr() functions are subtly different If you pass 0 as the second argument of substr() it is interpreted differently by the two different substr() functions. The documentation doesn't say what the correct result should be. SQLite version 3.2.0 Enter ".help" for instructions sqlite> select substr('ABCDE', 0, 2); AB sqlite> pragma encoding = "utf-16"; sqlite> select substr('ABCDE', 0, 2); sqlite> #e8e8bd 1070 build active 2005 Jan anonymous CodeGen 2005 Jan 1 1 error by compiling windows vcc / vc#not found Cannot build sqlite.lib. reason sqlite.h not found #e8e8bd 746 warn active 2004 May anonymous CodeGen 2004 May anonymous 5 5 How can I make sqlite support serial term? How can I make sqlite support "serial" term? I just put some code into build.c and this is the cord that I changed. void sqlite3AddPrimaryKey(Parse *pParse, IdList *pList, int onError){
...
if( zType && ( sqlite3StrICmp(zType, "INTEGER")==0 ||
sqlite3StrICmp(zType, "SERIAL")==0 )
){
...
void sqlite3AddColumnType(Parse *pParse, Token *pFirst, Token *pLast){
Table *p;
int iCol, i, j;
int n;
char *z, **pz, *t=0;
Column *pCol;
if( (p = pParse->pNewTable)==0 ) return;
iCol = p->nCol-1;
if( iCol<0 ) return;
pCol = &p->aCol[iCol];
pz = &pCol->zType;
n = pLast->n + Addr(pLast->z) - Addr(pFirst->z);
sqliteSetNString(pz, pFirst->z, n, 0);
z = *pz;
if( z==0 ) return;
for(i=j=0; z[i]; i++){
int c = z[i];
if( isspace(c) ) continue;
z[j++] = c;
}
z[j] = 0;
if( pParse->db->file_format>=4 ){
pCol->sortOrder = sqliteCollateType(z, n);
}else{
pCol->sortOrder = SQLITE_SO_NUM;
}
t = p->aCol[iCol].zType;
if( t && sqliteStrICmp(t, "SERIAL")==0 ){
if( p->iPKey == -1 ) p->iPKey = i;
else {
/* it has already "INTEGER PRIMARY KEY" or "SERIAL" */
sqliteErrorMsg(pParse,
"table \"%s\" has more than one serial type column", p->zName);
}
}
}
the result is "serial primary key" is OK, but " serial" has a bug. the program exits abnormally. I just want to change it for my own. anybody give me some advice. thanks. _2004-May-24 12:30:21 by anonymous:_ {linebreak} 1.Add following code to the end of sqliteAddColumnType function in build.c
if( z && sqliteStrICmp(z, "SERIAL")==0 ){
if( p->iPKey == -1 ) p->iPKey = iCol;
else {
/* it has already "INTEGER PRIMARY KEY" or "SERIAL" */
sqliteErrorMsg(pParse,
"table \"%s\" has more than one serial type column", p->zName);
}
}
2. Change some code of sqliteAddPrimaryKey function so that it looks like this
if( pParse->db->file_format>=1 &&
zType && ( sqliteStrICmp(zType, "INTEGER")==0 ||
sqliteStrICmp(zType, "SERIAL")==0 )
){
---- _2004-May-24 13:16:08 by anonymous:_ {linebreak} Ups... the second code above has a bug.
correct one is this:
if( pParse->db->file_format>=1 &&
zType && ( sqliteStrICmp(zType, "INTEGER")==0 ||
sqliteStrICmp(zType, "SERIAL")==0 )
){
/* If there is already a serial */
if( pTab->iPKey>=0 && sqliteStrICmp(zType, "INTEGER")==0 ){
sqliteErrorMsg(pParse,
"table \"%s\" has more than one serial column type", pTab->zName);
goto primary_key_exit;
}
the result :
C:\test>sqlite.exe test.db
SQLite version 2.8.13
Enter ".help" for instructions
sqlite> create table t1 (id integer primary key, t text, i int);
sqlite> insert into t1 values (null, 'a', 1);
sqlite> insert into t1 values (null, 'b', 2);
sqlite> select * from t1;
1|a|1
2|b|2
sqlite> create table t2 (id serial primary key, t text, i int);
sqlite> insert into t2 values (null, 'a', 1);
sqlite> insert into t2 values (null, 'b', 2);
sqlite> select * from t2;
1|a|1
2|b|2
sqlite> create table t3 (id serial, t text, i int);
sqlite> insert into t3 values (null, 'a', 1);
sqlite> insert into t3 values (null, 'b', 2);
sqlite> select * from t3;
1|a|1
2|b|2
sqlite> create table t4 (id serial, t text primary key, i int);
sqlite> insert into t4 values (null, 'a', 1);
sqlite> insert into t4 values (null, 'b', 2);
sqlite> insert into t4 values (null, 'b', 2);
SQL error: column t is not unique
sqlite> select * from t4;
1|a|1
2|b|2
sqlite> create table t5 (id serial, t text, i integer primary key);
SQL error: table "t5" has more than one serial column type
sqlite> create table t5 (id serial, t text, i serial primary key);
SQL error: table "t5" has more than one serial type column
sqlite> create table t5 (id serial, t text, i serial);
SQL error: table "t5" has more than one serial type column
sqlite> create table t5 (id integer primary key, t text, i serial);
SQL error: table "t5" has more than one serial type column
sqlite>
#e8e8bd 741 new active 2004 May anonymous CodeGen 2004 May anonymous 5 5 command tool does not support history if command tool support history, it would be great. windows supports its own history but on linux system command tool doesn't support history. _2004-May-23 18:07:31 by anonymous:_ {linebreak} The sqlite command on linux and cygwin have history (internal to sqlite at least). This capability comes from readline if it is installed on your systems. For some more details including libedit and OSX support see ReadLine #f2dcdc 685 code active 2004 Apr anonymous CodeGen 2004 Apr 1 3 SELECT from a VIEW with GROUP BY When you SELECT from a VIEW (which is having a GROUP BY statement) and try to apply another GROUP BY statement you get: $ sqlite ../../db/main.db SQLite version 2.8.13 Enter ".help" for instructions sqlite> .dump prod_elem_totals BEGIN TRANSACTION; CREATE VIEW prod_elem_totals AS SELECT pe.elem_id AS elem_id, p.prod_id AS prod_id, e.name AS name, p.name AS p_name, pe.count AS count, SUM(b.count) / pe.count AS p_max, SUM(b.count) AS total, SUM(b.price * b.count) / SUM(b.count) AS price, e.min AS min FROM products AS p, elements AS e, batches AS b, prod_elems AS pe WHERE p.prod_id = pe.prod_id AND pe.elem_id = b.elem_id AND pe.elem_id = e.elem_id GROUP BY p.prod_id, pe.elem_id ORDER BY e.name; COMMIT; sqlite> SELECT * FROM prod_elem_totals GROUP BY elem_id; sqlite: src/select.c:1775: flattenSubquery: Assertion `p->pGroupBy==0' failed. Aborted It seams it doesn't matter which column I GROUP BY. I can prepare a full test case if needed. Maybe somehow connected with #678. After further investigation I found that when I add a aggregate function like "SUM (count * 10) AS min" it works... #e8e8bd 259 new active 2003 Mar anonymous CodeGen 2003 Nov 1 2 ORDER BY queries including an integer primary key don't use index G'day, I have a table and index with the following specification: CREATE TABLE Disturbance (Entry INTEGER PRIMARY KEY,Input,Value,Time,HmiQuality,Quality); CREATE INDEX DisturbanceIndex ON Disturbance ( Input, Time, Entry ); When I do a query where the order by includes Input and Time, but not Entry no sorting is required as sqlite uses the index. When I include Entry at the end of the order by the index is not used for sorting. I've traced this to the findSortingIndex() function in where.c. The first two order-by entries pass the appropriate tests in the part of the function beginning at pMatch = 0. The third colum fails the test. On the line if( pOrderBy->a[i+j].pExpr->iColumn!=pIdx->aiColumn[i+nEqCol] ) break; the break is triggered because pOrderBy->a[i+j].pExpr->iColumn = -1, but pIdx->aiColumn[i+nEqCol] = 0. It looks to me like it's the order by version that's wrong, but I didn't want to delve any deeper than this, because there's probably a reason why the value is -1, and not 0 :) I'll leave it for more experienced hands to look at from this point onwards. Any work-around suggestions would also be appreciated as this is software that will be going to a customer in a week :) Benjamin. Yes, the query optimizer, and especially the code that tries to avoid sorting by using an index, needs some work. That has been on the to-do list for some time. Here is a temporary work-around: In the schema you define above, "ORDER BY Input, Time, Entry" and "ORDER BY Input, Time" will always generate the same order. So you should use the second ORDER BY form. Or, consider making Entry an "INT PRIMARY KEY" instead of "INTEGER PRIMARY KEY" to avoid invoking the primary key magic of SQLite. BTW, a value of -1 for a column number means that the column is the key of the record. #e8e8bd 273 new active 2003 Mar drh CodeGen 2003 Mar drh 1 1 Make NULL_ALWAYS_DISTINCT a pragma rather than a compile-time option NULL_ALWAYS_DISTINCT is a #define currently set to 0. If set to 1, then SQLite will treat NULLs a distinct in SELECT DISTINCT statements and in UNION operators. This is what the SQL standard calls for. But no database (other than Ocelot) works this way, so SQLite leaves NULL_ALWAYS_DISTINCT set to 0 to enhance compatibility. See http://www.sqlite.org/nulls.html for additional information. This compile-time options should become a run-time option. It should still be off by default, but a pragma should be available to turn it on for those who want it. #e8e8bd 263 new active 2003 Mar anonymous CodeGen 2003 Mar 1 4 sqlite locking strategy can lead to deadlock G'day, I'm currently on a customer site, and as one does when a big upgrade is coming up the next day I was thinking about problems I've encountered over the last few weeks. I realise now that a problem I'd considered not very imporant really should be reported to you: The locking strategy of sqlite when doing operations that modify the database is to set a read lock while data is being analysed, then upgrade that read lock to a write lock when it comes time to modify the data. This strategy can cause deadlock on a UNIX-based system if two processes are executing sql code to modify data simultaneously. The following interleaving results in deadlock: Process 1: Lock file for read (ok) Process 2: Lock file for read (ok) Process 1: Upgrade lock to write lock (go into wait state for process 2 to release read lock) Process 2: Upgrade lock to write lock (deadlock waiting for process 1 to release read lock: fail) I have a slightly modified version of sqlite that uses blocking locks instead of your usual non-blocking lock with delay backoff strategy. I have seen this bug in operation in my build, and although I think it is more likely to occur in my version it is still possible in the vanilla sqlite version. Since most sql statements that can modify data probably do eventually end up modifying data, my suggested fix to the locking strategy is to decide based on the kind of overall statement whether to set a read lock or a write lock. In that scenario there is no interleaving which can cause a problem because every time a lock is obtained the relevant sqlite process is guaranteed to complete it's operation and then release the lock before trying to obtain another lock (unless the user application is trying to do something tricky, say run sqlite statements from two sqlite instances at the same time and in the same thread). For the BEGIN TRANSACTION command's lock I would also recommend a write lock be obtained, since most use of this statement would involve non-const operations during the transaction (otherwise the transaction would have no value!). Benjamin. The scenario described above is not a problem for the standard SQLite build since locks are non-blocking there. But I will consider how locking might be changed to better accomodate blocking locks. This is not a high priority, however. And since the problem described above does not occur for the standard SQLite build with non-blocking locks, I'm changing this ticket to an enhancement request. Further remarks by bug reporter: The locking strategy can have implications for the non-blocking sqlite version as well. Consider the following interleaving: Process 1: Obtain read lock (ok) Process 2: Obtain read lock (ok) Process 1: Upgrade read lock to write lock (failed, read lock is active) This means that even though Process 1 obtained a lock before process 2 the operation fails. If a write lock had been obtained instead of a read lock then it would have been Process 2 that had it's operation fail. This would unfairly balance the contention between a reader and a writer process in favour of the reader in very tightly contentious environments. On the other hand, the non-blocking mechanism is not one that deals with high contention between readers and writers very well anyway ;) That's why I ended up making the locks blocking in the first place for my version. Benjamin. #e8e8bd 164 event active 2002 Oct anonymous CodeGen 2002 Oct anonymous 3 4 Compiler warnings with MS Visual C++ 6.0 Hi, when I incorporated the "sqlite_source.zip" source code into a Microsoft Visual C++6.0 sample application, I got 24 warnings, mostly due to some signed/unsigned mismatch. It looks as it still works fine (no difference as when I used the DLL), but you never know if this holds true alltimes. Not showing up warnings would improve convidence into the product. Regards, Louis Schneider Deleting intermediate files and output files for project 'GENERIC - Win32 Release'. --------------------Configuration: GENERIC - Win32 Release-------------------- Compiling resources... Compiling... GENERIC.C where.c build.c delete.c expr.c C:\samples\techart\tech\win32\generic3\expr.c(1461) : warning C4018: '!=' : signed/unsigned mismatch func.c hash.c insert.c main.c opcodes.c os.c C:\samples\techart\tech\win32\generic3\os.c(495) : warning C4018: '==' : signed/unsigned mismatch pager.c C:\samples\techart\tech\win32\generic3\pager.c(346) : warning C4018: '>' : signed/unsigned mismatch C:\samples\techart\tech\win32\generic3\pager.c(1008) : warning C4018: '>=' : signed/unsigned mismatch parse.c printf.c random.c select.c C:\samples\techart\tech\win32\generic3\select.c(99) : warning C4018: '==' : signed/unsigned mismatch shell.c table.c tokenize.c trigger.c Generating Code... parse.c(6771) : warning C4761: integral size mismatch in argument; conversion supplied parse.c(6782) : warning C4761: integral size mismatch in argument; conversion supplied Compiling... update.c util.c vdbe.c C:\samples\techart\tech\win32\generic3\vdbe.c(2069) : warning C4244: 'initializing' : conversion from 'double ' to 'int ', possible loss of data btree.c C:\samples\techart\tech\win32\generic3\btree.c(2306) : warning C4018: '<' : signed/unsigned mismatch Generating Code... C:\samples\techart\tech\win32\generic3\btree.c(629) : warning C4761: integral size mismatch in argument; conversion supplied C:\samples\techart\tech\win32\generic3\btree.c(1762) : warning C4761: integral size mismatch in argument; conversion supplied C:\samples\techart\tech\win32\generic3\btree.c(1764) : warning C4761: integral size mismatch in argument; conversion supplied C:\samples\techart\tech\win32\generic3\btree.c(516) : warning C4761: integral size mismatch in argument; conversion supplied C:\samples\techart\tech\win32\generic3\btree.c(520) : warning C4761: integral size mismatch in argument; conversion supplied C:\samples\techart\tech\win32\generic3\btree.c(534) : warning C4761: integral size mismatch in argument; conversion supplied C:\samples\techart\tech\win32\generic3\btree.c(538) : warning C4761: integral size mismatch in argument; conversion supplied C:\samples\techart\tech\win32\generic3\btree.c(541) : warning C4761: integral size mismatch in argument; conversion supplied C:\samples\techart\tech\win32\generic3\btree.c(482) : warning C4761: integral size mismatch in argument; conversion supplied C:\samples\techart\tech\win32\generic3\btree.c(483) : warning C4761: integral size mismatch in argument; conversion supplied C:\samples\techart\tech\win32\generic3\btree.c(410) : warning C4761: integral size mismatch in argument; conversion supplied C:\samples\techart\tech\win32\generic3\btree.c(421) : warning C4761: integral size mismatch in argument; conversion supplied C:\samples\techart\tech\win32\generic3\btree.c(432) : warning C4761: integral size mismatch in argument; conversion supplied C:\samples\techart\tech\win32\generic3\btree.c(434) : warning C4761: integral size mismatch in argument; conversion supplied C:\samples\techart\tech\win32\generic3\btree.c(1924) : warning C4761: integral size mismatch in argument; conversion supplied Linking... Creating library GENERIC.lib and object GENERIC.exp GENERIC.exe - 0 error(s), 24 warning(s) #f2dcdc 1867 code active 2006 Jun anonymous BTree 2006 Jun 1 3 Access Violation after set a new page_size An access violation occured on W2K when I try to create a new table in the empty database. There was a following sequence of SQL commands select count(*)==2 as cnt from sqlite_master where type='table' and tbl_name in ('tbl1', 'tbl2'); so if cnt is equal 0 then I execute command pragma page_size=4096; and then create a new table. I gess that some of internal structures by this time have been initialized and so when I try to create new table the page_size is lower then needed. we overwrite memory in the function zeroPage in instruction: memset(&data[hdr], 0, pBt->usableSize - hdr); Size of structure data less then pBt->usableSize Below result after memset 0:000> dt MemPage 004c3cf0
+0x000 isInit : 0 ''
+0x001 idxShift : 0 ''
+0x002 nOverflow : 0 ''
+0x003 intKey : 0x1 ''
+0x004 leaf : 0x1 ''
+0x005 zeroData : 0 ''
+0x006 leafData : 0x1 ''
+0x007 hasData : 0 ''
+0x008 hdrOffset : 0 ''
+0x009 childPtrSize : 0 ''
+0x00a maxLocal : 0
+0x00c minLocal : 0
+0x00e cellOffset : 0
+0x010 idxParent : 0
+0x012 nFree : 0xf94
+0x014 nCell : 0
+0x018 aOvfl : [5] _OvflCell
+0x040 pBt : (null)
+0x044 aData : (null)
+0x048 pgno : 0
+0x04c pParent : (null)
0012ea50 10006861 004c3cf0 0000000d 00000064 dblited!decodeFlags+0x80 [D:\sqllite\sqlite-3.3.6\btree.c @ 1349]
0012ea70 10006710 004c3cf0 0000000d 004c3cf0 dblited!zeroPage+0xd0 [D:\sqllite\sqlite-3.3.6\btree.c @ 1466]
0012ea8c 10006215 002fd390 002fd390 00000000 dblited!newDatabase+0xf9 [D:\sqllite\sqlite-3.3.6\btree.c @ 2061]
0012eaa0 10052ba0 002f7c30 00000001 0012f0e4 dblited!sqlite3BtreeBeginTrans+0xd6 [D:\sqllite\sqlite-3.3.6\btree.c @ 2141]
0012f0a4 10057cf5 004c3d80 0012f13c 0012f478 dblited!sqlite3VdbeExec+0x2c6d [D:\sqllite\sqlite-3.3.6\vdbe.c @ 2386]
0012f0e4 00412801 004c3d80 0012f1d4 0012f478 dblited!sqlite3_step+0x1db [D:\sqllite\sqlite-3.3.6\vdbeapi.c @ 223]
#e8e8bd 1555 doc active 2005 Dec anonymous BTree 2005 Dec appledev 2 4 sqlite+Crystal reports connection I have problem of sqlite and crystal report connection #f2dcdc 1508 code active 2005 Nov anonymous BTree 2005 Nov 1 3 sqlite 2.8.16 crashes on 64-bit / strict memory alignment archs A few months ago, sqlite3 was fixed on 64-bit / strict memory alignment architectures. Would it be possible for those fixes to be backported to the version_2 code? I have an OpenBSD/sparc64 machine which I can provide ssh access to (as I did before to drh). I know sqlite2 is mostly unsupported, but as php5 uses sqlite2, it would be nice to have these fixes backported. _2006-Jan-05 02:28:23 by anonymous:_ {linebreak} with the attached patch, sqlite 2.8.17 passes all the regressions tests on openbsd/amd64 and openbsd/sparc64. #f2dcdc 1487 code active 2005 Oct anonymous BTree 2005 Oct 1 3 Corrupt database causes indefinite loop in sqlite3_step() I had a database become corrupt (no idea why, 57 other databases of similiar information are fine). When attempting to work with the database and execute a SELECT query the application froze in an endless loop inside sqlite3_step(). Upon further investigation (which is when I found the db was corrupt) it seems to be stuck inside the btree code (as reported by Sample). I tested the same query and alternates from sqlite3 CLI and got the same results. The exact query causes an infinite loop. Leaving off part of the WHERE statement (and making it broader) or removing one of the reporting columns simply causes a corruption error. I have the original database as-is and the SQL query that can be run to cause the problem. OS: Mac OS X 10.4.2 #f2dcdc 1447 code active 2005 Sep anonymous BTree 2005 Sep 1 1 Abnormal program termination in src/btree.c line 1339 In some circumstances (after having used wxgrid ..) a call to sqlite gives a strange : Assertion Failed: pCur->idx>=0 && pCur->idx < pCur->pPage->nCell, file src/btree line 1339 abnormal program termination there seems to be non way of making a trace back ... any idea? Thanx Doriaqn Tessore _2005-Sep-23 14:32:27 by drh:_ {linebreak} Not much to go on. What version of SQLite is being used? ("SQLite 2" is kind of vague.) #f2dcdc 1058 code active 2005 Jan anonymous BTree 2005 Jan 3 3 btree.c pageSize -> usableSize Check-in [2125] was a fix for Ticket #1010, but left out some of the fixes proposed in the ticket. I'm not sure whether this was an oversight or an intentional omission. I tried to re-open the ticket, but those edits didn't persist (I'm not sure why). Here are the remaining instances of pageSize that I believe should be changed to usableSize in btree.c: --- sqlite/src/btree.c.ORIG Wed Nov 24 18:54:30 2004 +++ sqlite/src/btree.c Wed Nov 24 20:29:21 2004 @@ -220,13 +220,13 @@ /* The following value is the maximum cell size assuming a maximum page ** size give above. */ -#define MX_CELL_SIZE(pBt) (pBt->pageSize-8) +#define MX_CELL_SIZE(pBt) (pBt->usableSize-8) /* The maximum number of cells on a single page of the database. This ** assumes a minimum cell size of 3 bytes. Such small cells will be ** exceedingly rare, but they are possible. */ -#define MX_CELL(pBt) ((pBt->pageSize-8)/3) +#define MX_CELL(pBt) ((pBt->usableSize-8)/3) /* Forward declarations */ typedef struct MemPage MemPage; @@ -1745,7 +1745,7 @@ Pgno finSize; /* Pages in the database file after truncation */ int rc; /* Return code */ u8 eType; - int pgsz = pBt->pageSize; /* Page size for this database */ + int pgsz = pBt->usableSize;/* Usable bytes on each page */ Pgno iDbPage; /* The database page to move */ MemPage *pDbMemPage = 0; /* "" */ Pgno iPtrPage; /* The page that contains a pointer to iDbPage */ #f2dcdc 744 code active 2004 May anonymous BTree 2004 May anonymous 2 2 make test seg faults on x86_64 Linux I'm running the 64 bit version of Gentoo Linux on an AMD Opteron system. Ordinarily I'd install software with "emerge" but "emerge sqlite" only gives me version 2.8.11. I downloaded the 2.8.13 source and did the usual ./configure; make; make test. The configure and make steps went OK but make test fails half way through: bind-1.99... Ok btree-1.1... Ok btree-1.1.1... Ok btree-1.2... Ok btree-1.3... Ok btree-1.4... Ok btree-1.4.1... Ok btree-1.5... Ok btree-1.6...make: *** [test] Segmentation fault The code was built with GCC 3.3.3. As sqlite is a known 'emerge' option for 64 bit Gentoo I'm guessing sqlite is known to work on 64 bit platforms? I didn't mark this as a severe error because in theory I should be able to create a statically linked executable on a 32 bit linux system and run this on the Opteron box. Haven't been successful at that yet however. _2004-May-25 04:35:32 by anonymous:_ {linebreak} It's pretty clear that sqlite has never been compiled on a 64-bit system, much less run. The test problems are fatal bugs caused by type conversions between 64-bit and 32-bit values, including truncating pointers and other sins. The fixes look quite involved. #e8e8bd 209 new active 2002 Dec anonymous BTree 2003 Nov 1 1 update libtool The version of libtool shipped with sqlite is very out of date (well over a year old). It does not properly detect the ability to build shared libraries on darwin, for example, which is bad. You need to update to the latest version of gnu libtool. #e8e8bd 485 new active 2003 Oct anonymous BTree 2003 Oct 5 4 Comparing strings: user defined callback, strcoll? I think it would be useful to specify the "C" locale character set (like strcoll), at database level, so strings are compared based on that locale. Or at least some callback routine so one could perform those string comparisons. #f2dcdc 2917 code active 2008 Feb anonymous 2008 Feb 4 4 Tcl interface - busy callback confusion script/procedure In the Tcl interface the "busy" method doesn't work if a script is supplied instead of a procedure: % package req sqlite3 3.5.1 % sqlite3 db hl7journal.db % db busy {puts busy; expr 0} % db eval {select count(*) from incoming} busy database is locked Here the callback script is only invoked once, even though it returns 0. If we put this code in a procedure it works as desired/expected: % proc b args {puts busy; after 1000; expr 0} % db busy b % db eval {select count(*) from incoming} busy busy busy ^C _2008-Feb-01 12:31:45 by anonymous:_ {linebreak} After researching this a little it appears this happens because the busy callback is invoked with an extra argument. The extra argument leads to an error but that error is only visible through errorInfo, not the result. I humbly suggest the following changes: * mention the extra argument in the documentation of the Tcl interface * forward the error from the busy callback to Tcl (replacing the "database is locked") * enhance errorInfo to make the invokation of the busy callback apparent. Currently, I'm getting this errorInfo: % db busy {puts hi; after 1000; return 0} % db eval "select count(*) from incoming" hi database is locked % set errorInfo bad option "0": must be -code, -errorcode, or -errorinfo while executing "return 0 0" invoked from within "db eval "select count(*) from incoming"" It would be nicer to get something like bad option "0": must be -code, -errorcode, or -errorinfo while executing "return 0 0" from busy callback "puts hi; after 1000; return 0 0" invoked from within "db eval "select count(*) from incoming"" #f2dcdc 2916 code active 2008 Feb anonymous 2008 Feb 1 1 sqlitedll-3_5_5.zip is older 3.5.4 binary sqlitedll-3_5_5.zip in download section is same with old 3.5.4 binary. _2008-Feb-01 12:13:04 by anonymous:_ {linebreak} Yes , I can confirm it #e8e8bd 2228 new active 2007 Feb anonymous 2008 Feb 3 4 horizontal partitioning I would love to see a feature in the next version of sqlite that includes horizontal table partitioning. I see it currently in MySQL Version 5.1 beta. It would be awesome to see it in SQLite _2008-Feb-01 02:46:29 by anonymous:_ {linebreak} Well it would be great for us - won't need to emulate it. Any thoughts about that? That's not only about large data sets - you can also make it for some sort of horizontal ATTACH of small separate pieces. #e8e8bd 2915 build active 2008 Feb pweilbacher 2008 Feb 4 3 fix Makefile for platforms that need .exe extension There are some targets that need a $(TEXE) added in Makefile.in. Otherwise a platform that needs a .exe extension cannot run e.g. tests. #f2dcdc 2914 code active 2008 Jan anonymous 2008 Jan 3 3 ATTACH returns SQLITE_ERROR when it means SQLITE_BUSY I'm seeing the same behavior as in #2096, with SQLite 3.5.4. ATTACH DATABASE fails with SQLITE_ERROR rather than SQLITE_BUSY when the database to be attached, or the main database of the connection being attached to, is EXCLUSIVE-locked by another database connection. For added confusion, sqlite3_errmsg() says "database is locked" when the ATTACH is done via sqlite3_exec(), but "SQL logic error or missing database" when the ATTACH is done via sqlite3_step(). As a result of this bug, it is difficult to distinguish between fatal and transient ATTACH errors, particularly when sqlite3_step() is used. I am attaching a test program that demonstrates the problem. #f2dcdc 2905 code active 2008 Jan anonymous 2008 Jan pweilbacher 2 2 mutex_os2.c - incorrect mutex implementation The OS/2 version of sqlite3_mutex_alloc() is badly broken. It creates named mutexes which, by design, are global rather than process-specific as intended. This might be minimally acceptable except that the function reuses the same name every time it attempts to create a SQLITE_MUTEX_FAST or SQLITE_MUTEX_RECURSIVE. The result is that every call returns the exact same semaphore to every thread in every process using sqlite3. Once this mutex is owned by one process, other processes calling sqlite3_mutex_enter() will be blocked. Much the same is true for the static mutexes. Every process ends up using the exact same SQLITE_MUTEX_STATIC_MASTER, SQLITE_MUTEX_STATIC_MEM, etc. There's another flaw that is fairly minor compared with the above: in an attempt to avoid concurrency when creating the static mutexes, this function uses an API call that is thoroughly deprecated. The attached patch remedies all of these issues. Since the logic that protects the creation of the static mutexes may not be self-evident, here's an explanation: The existence (or non-existence) of a given named mutex is itself a semaphore. If the isInit flag is false, the code attempts to create a mutex whose name is unique to that process. If the attempt is successful, there are two possibilities: (1) either the current thread is the first to reach this code & may proceed; (2) or while the current thread was making its preparations, another thread created the mutex, did the init, then closed the mutex. Testing isInit immediately after creating the mutex determines which possibility is valid. If mutex creation fails due to a duplicate name, then another thread is currently performing the init. In this case, the current thread simply has to wait a while until the other thread is done & isInit becomes true. Submitted by Rich Walsh (richATe-vertise.com) #f2dcdc 2902 code active 2008 Jan anonymous 2008 Jan drh 3 3 Add watch support to SQLite SQLite currently provides only TRIGGERs and the update_hook() as a way for applications to stay informed about changes to the database. But both of these alternatives do not provide enough details about the actual changes to the underlying database file(s). We've prepared a patch for SQLite 3.5.x to allow applications to install a watch_hook into the database, that will be invoked everytime the database is changed with exact details about the change that was performed. _2008-Jan-22 16:06:59 by anonymous:_ {linebreak} Great idea and nice job. This functionality is very useful. ---- _2008-Jan-31 18:27:16 by anonymous:_ {linebreak} Any chance to get this committed for the next release (i.e. 3.5.6)? ---- _2008-Jan-31 19:38:35 by drh:_ {linebreak} Unlikely, for two reasons: 1: I am unconvinced that this patch solves a problem that needs solving. It is vitally important to a project like SQLite that we work to avoid clutter and cruft. That means that any change must have a compelling rational or else it is rejected. 2: The patches are against version 3.5.4. There were many changes to the core for 3.5.5 and the patches no longer work. ---- _2008-Jan-31 20:58:55 by anonymous:_ {linebreak} We can (and will) port the changes to 3.5.5, so the second point will be done. First the first point, I'm not sure how many projects will actually need this functionality, but I guess there are quite a lot of projects that would benefit, and for the others, there's zero overhead due to this patch. ---- _2008-Jan-31 21:21:30 by drh:_ {linebreak} There is a lot of overhead for me because if I accept this patch, that means I have to maintain it forever. Most of the work is in maintenance, not coming up with the original patch. #e8e8bd 2912 new active 2008 Jan anonymous 2008 Jan 5 4 merge join Would it be possible to implement merge joins? #f2dcdc 2903 code active 2008 Jan anonymous 2008 Jan 2 2 tclinstaller.tcl fails on path and permissions issue When compiling using custom PREFIX, pointing to private directory, tclinstaller.tcl fails, because it tries to remove contents from /usr/share/tcl8.4/sqlite3. ./configure --prefix=/my/private/sqlite/sqlite-3.5.4 ... # success make ... # success make install ... tclsh ./tclinstaller.tcl 3.5 error deleting "/usr/share/tcl8.4/sqlite3": not owner while executing "file delete -force $LIBDIR/sqlite3" (file "./tclinstaller.tcl" line 17) make: *** [tcl_install] Error 1I've found two work-arounds: 1: If you run make install as root. 2: If you use ./configure --disable-tcl _2008-Jan-28 17:47:39 by anonymous:_ {linebreak} I also ran into this problem. make install as root will end up copying files into the system's library directory and is almost certainly not what you want if you specified your own --prefix. #f2dcdc 2911 code active 2008 Jan anonymous 2008 Jan 2 2 Adding parentheses to a FROM clause Hi, Parentheses in a FROM statement seem to mess with the ability to use table aliases in the "what" part. Here is an example: Start SQLite: $ sqlite3 employee.db SQLite version 3.5.4 Enter ".help" for instructions create a couple of tables and populate them with test data: sqlite> create table person (id integer, name text, employerid integer); sqlite> create table employer (id integer, name text); sqlite> insert into person (id, name, employerid) values (1, "Dave", 1); sqlite> insert into employer (id, name) values (1, "ACME"); Run a simple query with *no parentheses* in the FROM statement: sqlite> select b.id from person as a inner join employer as b on a.employerid = b.id; 1 Everything works as expected. Now, repeat that query *with parentheses*: sqlite> select b.id from (person as a inner join employer as b on a.employerid = b.id); SQL error: no such column: b.id There you have it. This may be related to ticket #1822, although that ticket deals with aliases and subqueries. This problem seems to be more fundamental. Many thanks, -- Dave #e8e8bd 2909 build active 2008 Jan aswift 2008 Jan aswift 4 4 Omit the _XOPEN_SOURCE 500 define on Mac OS-X (check for __APPLE__) sqliteInt.h attempts to avoid setting _XOPEN_SOURCE on MacOSX by checking if __DARWIN__ is defined, however on Leopard (as of Mac OS X 10.5) this test fails and should be enhanced by testing for both __DARWIN__ and __APPLE__ Building sqlite succeeds with _XOPEN_SOURCE defined, but it causes _POSIX_C_SOURCE to be defined which leads to failing to define F_FULLFSYNC which prevents use of the fullfsync pragma. #f2dcdc 2613 code active 2007 Sep anonymous 2008 Jan drh 3 3 replace doesn't work with blobs containing \x0, otherwise it does The replace expression function does not work with blobs in case of contained zero terminator character; but it does if there is not this special character included. I expected the function to work similar like substr with blob-safety in case of type is blob only. X'nnnn' is of type blob, so following example should have returned a blob type result X'0102FF0405' in the 2nd and 3rd line. How to get to this result?SQLite version 3.4.2 Enter ".help" for instructions sqlite> select hex(replace(X'0102030405',X'03',X'FF')); 0102FF0405 sqlite> select hex(replace(X'0102000405',X'00',X'FF')); sqlite> select typeof(replace(X'0102000405',X'00',X'FF')); null sqlite>_2007-Sep-03 04:21:12 by anonymous:_ {linebreak} Replace was designed to work with strings. However, working with blobs would be an interesting extension. ---- _2007-Oct-18 06:13:10 by anonymous:_ {linebreak} I've seen a similar situation where I can't reliably store stings with nulls in the middle of them as TEXT. I can convert them to blobs, in which case length(...) works correctly. I if convert them back to strings, length(...) treats them as C-strings. Is this the expected behavior? I notice the entire column is preserved even when it's has TEXT affinity, I can append data to it as a string, cast back to a blob and see everything (am I explaining this poorly?) This all seems a bit counter intuitive in some ways. Perhaps strings shouldn't treat NULL characters as special? ---- _2007-Oct-27 16:45:41 by anonymous:_ {linebreak} Treatment of length operator is - as fas as I know - dependent on type: {linebreak} As text it is the length number of UTF-8 characters and as blob it is the number of bytes. As long as all the UTF-8 characters out of the lower half ASCII char-set (127 of them), this is identical beside the fact of different 0-terminator interpretation. {linebreak} To append is something different than using the replace operator. My suggestion would be to make the replace operator work with bytes (not UTF-8) in case of all 3 parameters are of type blob. {linebreak} Another suggestion: the UTF aware functions are Private declared and not usable from within a loadable extension dll/so. This should be changed. ---- _2008-Jan-28 19:36:39 by anonymous:_ {linebreak} Will there come a solution for this with the next release? It is really not fair to handle a blob only like text which cannot contain a zero terminator. With this unique useful function a zero-containing blob could be formed into a normal text string without loosing the part behind the zero terminator. It would be really a step forward without too much effort. #f2dcdc 2907 code active 2008 Jan anonymous 2008 Jan 1 1 Issues of sqlite3 with Windows Mobile 5/6 hi. we are currently using sqlite3 for our mobile application. it has been running without a hitch on pocket pc 2003 and previous versions. come windows mobile 5 and 6 we have been getting errors, although not consistent yet. one example is 'EXCEPTION_DATATYPE_MISALIGNMENT'. another is 'SELECT STATMENTS TO THE LEFT AND RIGHT OF UNION ARE NOT EQUAL'. i was wondering if you have any known compatibility issues of your product with this version of windows mobile. thanks in advance. _2008-Jan-28 13:26:26 by anonymous:_ {linebreak} EXCEPTION_DATATYPE_MISALIGNMENT is thrown when you try to use and Odd pointer address. I wrote a custom allocator for WinCE/ARM platform, and I have to take care about memory alignment (I used to align at 2 bytes, and at that time it solved the problem) #f2dcdc 2908 code active 2008 Jan anonymous 2008 Jan 3 3 Add support to examine whether statements modify the database Currently there is no way to check whether a compiled statement will modify the database when being executed. Of course, there is the work-around of misusing the authorizer callback for this purpose, but this is kinda error prone and causes quite some overhead for such a simple purpose. #f2dcdc 2898 code active 2008 Jan anonymous 2008 Jan 1 1 Latest CVS for 3.5.4 fails to build test1.c gcc -pipe -O3 -g -Wall -DSQLITE_DISABLE_DIRSYNC=1 -I. -I../src -DNDEBUG -I/usr/include -DSQLITE_THREADSAFE=1 -DSQLITE_THREAD_OVERRIDE_LOCK=1 -DSQLITE_OMIT_LOAD_EXTENSION=1 -DTCLSH=1 -DSQLITE_TEST=1 -DSQLITE_CRASH_TEST=1 -DSQLITE_NO_SYNC=1 -DTEMP_STORE=1 -o .libs/testfixture ../src/attach.c ../src/btree.c ../src/build.c ../src/date.c ../src/expr.c ../src/func.c ../src/insert.c ../src/malloc.c ../src/os.c ../src/os_os2.c ../src/os_unix.c ../src/os_win.c ../src/pager.c ../src/pragma.c ../src/prepare.c ../src/printf.c ../src/select.c ../src/test1.c ../src/test2.c ../src/test3.c ../src/test4.c ../src/test5.c ../src/test6.c ../src/test7.c ../src/test8.c ../src/test9.c ../src/test_autoext.c ../src/test_async.c ../src/test_btree.c ../src/test_config.c ../src/test_hexio.c ../src/test_malloc.c ../src/test_md5.c ../src/test_onefile.c ../src/test_schema.c ../src/test_server.c ../src/test_tclvar.c ../src/test_thread.c ../src/tokenize.c ../src/utf.c ../src/util.c ../src/vdbe.c ../src/vdbeapi.c ../src/vdbeaux.c ../src/vdbemem.c ../src/where.c parse.c ../src/tclsqlite.c ./.libs/libsqlite3.so -L/usr/lib64 -ltcl8.4 -ldl -lpthread -lieee -lm -Wl,--rpath -Wl,/common/pkgs/sqlite-3.5.4.3/lib ../src/build.c: In function 'sqlite3RefillIndex': ../src/build.c:2275: warning: cast to pointer from integer of different size ../src/func.c: In function 'trimFunc': ../src/func.c:919: warning: cast from pointer to integer of different size ../src/func.c: In function 'sqlite3RegisterBuiltinFunctions': ../src/func.c:1464: warning: cast to pointer from integer of different size ../src/func.c:1483: warning: cast to pointer from integer of different size ../src/insert.c: In function 'sqlite3GenerateConstraintChecks': ../src/insert.c:1200: warning: cast to pointer from integer of different size ../src/insert.c:1034: warning: 'j2' may be used uninitialized in this function ../src/insert.c: In function 'sqlite3Insert': ../src/insert.c:373: warning: 'regFromSelect' may be used uninitialized in this function ../src/test1.c: In function 'test_collate_func': ../src/test1.c:2085: warning: cast from pointer to integer of different size ../src/test1.c: In function 'test_collate_needed_cb': ../src/test1.c:2209: warning: cast to pointer from integer of different size ../src/test1.c: In function 'alignmentCollFunc': ../src/test1.c:2258: warning: cast from pointer to integer of different size ../src/test1.c:2259: warning: cast from pointer to integer of different size ../src/test8.c: In function 'echoBestIndex': ../src/test8.c:722: warning: 'nRow' may be used uninitialized in this function ../src/vdbe.c: In function 'sqlite3VdbeExec': ../src/vdbe.c:502: warning: 'pOut' may be used uninitialized in this function ../src/vdbe.c:501: warning: 'pIn3' may be used uninitialized in this function ../src/vdbe.c:501: warning: 'pIn2' may be used uninitialized in this function ../src/vdbe.c:501: warning: 'pIn1' may be used uninitialized in this function ../src/vdbeaux.c: In function 'sqlite3VdbeChangeP4': ../src/vdbeaux.c:529: warning: cast from pointer to integer of different size ../src/vdbemem.c: In function 'sqlite3ValueText': ../src/vdbemem.c:911: warning: cast from pointer to integer of different size /tmp/ccsuOeus.o: In function `reset_prng_state': /build/work/sqlite-3.5.4.3/bld/../src/test1.c:4280: undefined reference to `sqlite3ResetPrngState' /tmp/ccsuOeus.o: In function `restore_prng_state': /build/work/sqlite-3.5.4.3/bld/../src/test1.c:4267: undefined reference to `sqlite3RestorePrngState' /tmp/ccsuOeus.o: In function `save_prng_state': /build/work/sqlite-3.5.4.3/bld/../src/test1.c:4254: undefined reference to `sqlite3SavePrngState' collect2: ld returned 1 exit status make: *** [testfixture] Error 1 _2008-Jan-17 23:54:58 by anonymous:_ {linebreak} Problem appears to be here in libsqlite.3.so.0.8.6 as shown by: nm -A .libs/libsqlite3.so.0.8.6 | grep sqlite3ResetPrngState which shows no entry point. And: nm -A .libs/random.o | grep sqlite3ResetPrngState which also shows no entry point. ---- _2008-Jan-17 23:56:55 by anonymous:_ {linebreak} Ah... It appears -DSQLITE_TEST should be passed when building test1.c and left off when building prior to install. ---- _2008-Jan-21 20:16:00 by anonymous:_ {linebreak} In the makefile the right flag appears to be set, it's just not making it through to the compile for some reason. ---- _2008-Jan-21 20:16:24 by anonymous:_ {linebreak} Still fails the same based on today's cvs update. ---- _2008-Jan-23 03:14:49 by anonymous:_ {linebreak} This bug fixed as of latest cvs pull #f2dcdc 2901 code active 2008 Jan anonymous 2008 Jan 3 3 ROLLBACK and COMMIT statements should not expire Currently, whenever a statement changes the schema of the database, all prepared statements will be expired, no matter whether they actually need to be prepared again or not. This is especially problematic for ROLLBACK statements in a multi-statement transaction. Currently there is no way to guaranty that a multi-statement transaction can at least be rolled back in case of an error, because one has to (re)prepare the ROLLBACK statement to roll back the transaction, which can fail because of OOM (in a multi-threaded application). #e8e8bd 2506 new active 2007 Jul anonymous 2008 Jan 3 2 New API to retrieve ROWID from SQLite3_stmt structire Is it too much trouble to allow an API to retrieve ROWID for non-aggeregate queries directly from SQLite3_stmt structire? It would be very useful to create updatable non aggregate query results for situations when actually internal PK (ROWID) is not gived explicitly in SQL statement nor actual table's PK (if any). SELECT queries that join two or more tables together would be a problem also. ---- _2007-Jul-16 16:51:18 by anonymous:_ {linebreak} It's more of a multi-step process. First you have to enumerate the open cursors on the sqlite3_stmt object. Then you need to resolve the table each cursor goes to, and then fetch the rowid for each active cursor. Of course this may get confusing when you've joined a table onto itself. ---- _2008-Jan-19 10:32:59 by anonymous:_ {linebreak} This is far to old active ticket. Is it in consideration to be implemented in the near future? #f2dcdc 2899 code active 2008 Jan anonymous 2008 Jan 4 4 sqlite3_reset() after exec() takes > 100 ms to complete I'm not entirely sure whether this is a bug or not... I'm not familiar enough with SQLite to know if this is way too unconventional, but I noticed today that running exec() in conjunction with a prepared statement really kills the performance of sqlite3_reset(), if it's called after exec(): // init sqlite3_prepare_v2( db,{linebreak} "SELECT [file_id],[file_name],[file_mime],[file_type],"{linebreak} "[file_size],[date_created],[date_accessed],[data] "{linebreak} "FROM file_cache WHERE [file_id] = ? LIMIT 1;",{linebreak} -1, &sqQueryStatement, &unused ); // query function -- called repeatedly (usually second or third run starts to cause big delays) /* begin function */ sqlite3_reset( sqQueryStatement );{linebreak} sqlite3_bind_int( sqQuerystatement, 1, 292 );{linebreak} sqlite3_step( sqQueryStatement );{linebreak} bla = sqlite3_column_int( sqQueryStatement, 0 ); /* ... */ char *erm; sqlite3_exec( db, "UPDATE file_cache SET [date_accessed] = DATETIME( 'NOW', 'LOCALTIME' ) WHERE [file_id] = 292", NULL, NULL, &erm ); /* ... */ sqlite3_clear_bindings( sqQueryStatement ); /* end function */ If sqlite3_exec() is commented out, consecutive calls to the function run in less than a millisecond. With sqlite3_exec() included, sqlite3_reset() call in this function takes > 100 ms to complete. Tested: Windows Vista Ultimate 32 bit, Visual Studio 2005 8.0.50727.876 #f2dcdc 2885 code active 2008 Jan anonymous 2008 Jan 4 4 (minor) fulltest failures Minor noise in fulltest on a 64-bit machine (everything seems to work otherwise though): 4 errors out of 61527 tests Failures on these tests: exclusive-ioerr-2.280.4 exclusive-ioerr-2.281.4 exclusive-ioerr-2.282.4 incrvacuum-ioerr-1.31.4 All memory allocations freed - no leaks details: exclusive-ioerr-2.280.3... Ok exclusive-ioerr-2.280.4... Expected: [8ee59fe0b5bc391ecc5002539379b063] Got: [35e56178ad878809d4789c5797265a23] exclusive-ioerr-2.281.1... Ok exclusive-ioerr-2.281.2... Ok exclusive-ioerr-2.281.3... Ok exclusive-ioerr-2.281.4... Expected: [95762fb35ef83ddba65f681325346ef2] Got: [1c20c63d975ad85b395a5e7701d785c2] exclusive-ioerr-2.282.1... Ok exclusive-ioerr-2.282.2... Ok exclusive-ioerr-2.282.3... Ok exclusive-ioerr-2.282.4... Expected: [2c5ea7db8424f38d17cdff41056da0e0] Got: [c02841c40d3e6e0579922745bd9c0260] exclusive-ioerr-2.283.1... Ok [...] incrvacuum-ioerr-1.31.4... Expected: [ea55042a449c3b4759730a882e8271a0] Got: [9951814984696d0811cacbe862060af8] incrvacuum-ioerr-1.32.1... Ok _2008-Jan-19 00:56:30 by anonymous:_ {linebreak} A test from 20 minutes ago passes cleanly. This could be closed. #f2dcdc 2878 code active 2008 Jan anonymous 2008 Jan 1 1 Memory leaks with latest CVS [4693] This SQL leaks memory with CVS [4693]: CREATE TABLE x(id integer primary key, a TEXT NULL); INSERT INTO x (a) VALUES ('first'); CREATE TABLE tempx(id integer primary key, a TEXT NULL); INSERT INTO tempx (a) VALUES ('t-first'); CREATE VIEW tv1 AS SELECT x.id, tx.id FROM x JOIN tempx tx ON tx.id=x.id; One leak is caused by "CREATE TABLE tempx", a second one by "CREATE VIEW tv1". The above SQL is a digest of select7.test, select7-2.1. _2008-Jan-14 17:51:11 by anonymous:_ {linebreak} I have tested again with CVS [4711] and it does no longer show the original leaks. I therefore consider this issue fixed and I will now close this ticket. ---- _2008-Jan-14 23:56:34 by anonymous:_ {linebreak} Doing a fulltest with -MSQLITE_MEMDEUG I see reports of memory leaks. I assume these are of little or minimal interest at present because of the amount of code flux. If you do want details, please let me know (I'll recheck this ticket tomorrow I guess). ---- _2008-Jan-15 08:23:56 by anonymous:_ {linebreak} Thanks for the follow-up. I am not running the original test-suite but have have ported a great number of them to Delphi. If you could just let me know which tests caused the leaks you fixed in [4712] I'd be more than glad the port these test as well and let you know my findings. ---- _2008-Jan-19 00:56:17 by anonymous:_ {linebreak} A test from 20 minutes ago passes cleanly. This could be closed. #f2dcdc 2897 code active 2008 Jan anonymous 2008 Jan 1 1 String or BLOB exceed size limit This error was shown after attemp to read script from SQLite 3.5.4 shell in order to recreate old DB. Details: 1. Database was created with SQLite 3.3.4. Around 20 standard fieds and one BLOB. 2. The only one existed table was dumped with shell of SQLite 3.5.4. SQL script seems to be coorrect. 3. Opened SQLite 3.5.4 and read script in new DB. The error "String or BLOB exceed size limit" are sown for several lines. Many records missing. 4. Attempted to dump table with shell of version 3.3.6 (have no more 3.3.4 shell) and read into new DB with 3.5.4 shell The same errors are shown. The same steps was attempted with 3.3.6. shell only. All seems to be correct. _2008-Jan-17 20:23:25 by drh:_ {linebreak} This size limit on BLOBs in SQLite version 3.5.4 is 1GB. How big is your blob, exactly? ---- _2008-Jan-17 22:22:24 by anonymous:_ {linebreak} BLOB in each record is no more than few MB. Mostly it is few KB (e-client and news application). Whole DB have around 200MB. ---- _2008-Jan-18 02:28:11 by drh:_ {linebreak} This issue is probably resolved by check-in [4636], then. ---- _2008-Jan-18 14:28:13 by anonymous:_ {linebreak} If directive SQLITE_MAX_SQL_LENGTH is not defined it is set to 1,000,000 (10^6) in amalgamation code of 3.5.4. #e8e8bd 2892 new active 2008 Jan anonymous 2008 Jan 5 5 There should be a way in the api to read more precise error message. Currently all errors in sql queries issued via C,C++ api result in error code SQLITE_ERROR = 1 and message "SQL error or missing database". That leaves user completely clueless what mistake in his sql he actually made. Some evidence of confusion can be found here: http://bugs.php.net/bug.php?id=33117 _2008-Jan-16 17:41:00 by drh:_ {linebreak} SQLite gives detailed error information for SQL syntax or logic errors. (Try, for example, entering invalid SQL into the CLI.) I think perhaps that PHP is simply failing to to access those errors and is instead picking up some other error indication from someplace else. ---- _2008-Jan-16 21:59:17 by anonymous:_ {linebreak} Well I shall retest it, but I got the same error message upon a duplicate key in Delphi. Although I'm aware of an enhanced api to get the errormessage. ---- _2008-Jan-17 00:05:58 by drh:_ {linebreak} What error message do you get from the CLI? #f2dcdc 2893 code active 2008 Jan anonymous 2008 Jan 1 1 incorrect integer range tests recently a function that performs integer range tests was added to the cvs (check-in [4706]), but if i am correct there is a problem in the return value of the function in the file vdbemem.c: static i64 doubleToInt64(double r){ ... if( r<(double)minInt ){ return minInt; }else if( r>(double)maxInt ){ return minInt; <-- is this correct, shouldn't it be maxInt? }else{ return (i64)r; } } _2008-Jan-16 17:33:56 by drh:_ {linebreak} See the remarks on ticket #2280. The code duplicates the behavior of the FPU on x86. ---- _2008-Jan-16 18:21:28 by anonymous:_ {linebreak} did you mean ticket #2880? didn't read that ticket before, but since there was no comment regarding that behavior in the function it seemed (to my eyes) that it was a mistake. maybe adding a small comment in there would clarify this issue ---- _2008-Jan-16 18:39:42 by anonymous:_ {linebreak} Just because the double to int overflow behavior happens to be that way with GCC on x86, is it desirable? #f2dcdc 2886 code active 2008 Jan anonymous 2008 Jan 3 3 testfixture: -fPIC needed when building extension(s) (this fix/change is probably needed in older versions too, i meant to send this in earlier) -fPIC is needed when building extensions (some platforms don't need this or don't care --- x86-64 does) diff --git a/test/loadext.test b/test/loadext.test index 81e152f..2a7fa2e 100644 --- a/test/loadext.test +++ b/test/loadext.test @@ -64,7 +64,7 @@ if {![file exists $testextension]} { set srcdir [file dir $testdir]/src set testextsrc $srcdir/test_loadext.c if {[catch { - exec gcc -Wall -I$srcdir -I. -g -shared $testextsrc -o $testextension + exec gcc -Wall -fPIC -I$srcdir -I. -g -shared $testextsrc -o $testextension } msg]} { puts "Skipping loadext tests: Test extension not built..." puts $msg #f2dcdc 2882 code active 2008 Jan anonymous 2008 Jan 3 3 fulltest failure: ./testfixture: wrong # args: should be "cksum db" exclusive-ioerr-2.2.1... Ok ./testfixture: wrong # args: should be "cksum db" while executing "ifcapable vacuum { do_ioerr_test ioerr-2 -cksum true -sqlprep { BEGIN; CREATE TABLE t1(a, b, c); INSERT INTO t1 VALUES(1, randstr(50,..." (file "../test/ioerr.test" line 58) invoked from within "source $testdir/ioerr.test" (file "../test/exclusive3.test" line 50) invoked from within "source $testfile" ("foreach" body line 5) invoked from within "foreach testfile [lsort -dictionary [glob $testdir/*.test]] { set tail [file tail $testfile] if {[lsearch -exact $EXCLUDE $tail]>=0} continue ..." ("for" body line 7) invoked from within "for {set Counter 0} {$Counter<$COUNT && $nErr==0} {incr Counter} { if {$Counter%2} { set ::SETUP_SQL {PRAGMA default_synchronous=off;} } else ..." (file "..//test/all.test" line 85) _2008-Jan-14 23:25:49 by anonymous:_ {linebreak} The latest code seems to have fixed this. I would close this but I don't see how to do that. #e8e8bd 2884 new active 2008 Jan anonymous 2008 Jan 4 4 Way to find out limits The page http://www.sqlite.org/limits.html shows some limits that SQLite has. They are consts in the code, and defaults are listed, but there doesn't seem to be any way to find out what values a particular binary was built with. For example, I've gotten SQLite as part of my OS or another program, and I don't know what value of SQLITE_MAX_COLUMN was used to compile this SQLite. I'm trying to track down a bug in my code, and if this value was too small, that would explain it. (In truth, it's probably not the cause, but I'd like to rule it out.) I'd like a way to print the value of these limits at runtime. It doesn't have to have a stable API -- it would mostly be useful for debugging. If the command-line client had a command ".limits" that printed all of these values, that would be super. #e8e8bd 489 new active 2003 Nov anonymous 2008 Jan 4 4 DLL exports suggestion Just a suggestion: I'm building SQLite using MS C++, and an easily maintained alternative to a .def file for the DLL exports is to incorporate the following into the sqlite.h header... ---- #ifdef _MSC_VER #ifdef SQLITE_EXPORTS #define SQLITE_API __declspec(dllexport) #else #define SQLITE_API __declspec(dllimport) #endif #else #define SQLITE_API #endif // example function declaration SQLITE_API void sqlite_close(sqlite *); ---- SQLITE_EXPORTS is defined when building the library, but not when building client applications. I don't know if other compilers have similar methods of defining library exports, but if so, this could possibly be extended to support them. _2008-Jan-11 01:58:27 by anonymous:_ {linebreak} There is a good reason for adding this feature in some form other than merely avoiding the manual import library creation step. When __declspec(dllimport) is used, it is a hint to MSVC++ to produce more efficient code. This allows the function in the DLL to be called in a single call. Otherwise there is an extra jmp (2 in Debug mode). So those that want maximum performance out of C and DLL combo should take note. I patched this into my copy of the source. But the code given in the ticket assumes that the library is used as a DLL and not statically linked in which case dllimport shouldn't be used. -- Bz ---- _2008-Jan-11 13:28:27 by drh:_ {linebreak} The prefix SQLITE_API appears in front of all interfaces in the {link: /cvstrac/wiki?p=TheAmalgamation amalgamation}. So it seems like this problem could be solved by adding -DSQLITE_API=__declspec(dllexport) to the compiler command line. No? ---- _2008-Jan-12 07:16:44 by anonymous:_ {linebreak} Yes -DSQLITE_API will do, and also it is necessary to define SQLITE_EXTERN because some data symbols: sqlite3_version[] sqlite3_temp_directory sqlite3_io_trace are forward declared with SQLITE_EXTERN but defined with SQLITE_API and these must match. But like I mentioned, this is only half of the issue. The other half is the small performance boost from __declspec(dllimport). It would be nice to have SQLITE_API prefixes in sqlite3.h for users of the DLL. It's low priority, but I thought I would point this out. Another thing to note is that using __declspec(dllexport) instead of DEF file along with the (non default) stdcall convention (callee cleans up the stack) will result in function names in DLL export table being mangled with numeric suffixes. If one links through the auto generated import lib file, this isn't an issue, but it affects looking up a function name with GetProcAddress(). So the DEF file (the original intent of the ticket before I hijacked it) has a benefit in this case. But to use the stdcall convention properly there would have to be yet another prefix for the benefit of clients using the default cdecl convention: #define SQLITE_CALL #define SQLITE_CALL __stdcall SQLITE_API int SQLITE_CALL sqlite3_open( const char *filename, sqlite3 **ppDb ); The 2 prefixes is what MS does with their API functions. I assume there is some small performance boost from __stdcall. -- Bz ---- _2008-Jan-13 02:02:38 by sdwilsh:_ {linebreak} See also Ticket #2448 #e8e8bd 2881 build active 2008 Jan anonymous 2008 Jan 1 1 Latest sqlite-3.5.4 build fail on latest Fedora 2.6.23.12-52.fc7 Two test cases fail. io-4.1... Expected: [3] Got: [2] io-4.2.1... Ok io-4.2.2... Ok io-4.2.3... Expected: [3] Got: [2] io-4.3.1... Ok Let me know how to run individual test cases and how this might be fixed. Here's how I built sqlite using latest CVS. If something is wrong here, let me know and I'll rebuild/retest. I'm building on latest Fedora fc7. Thanks. _______ net1#uname -a Linux net1.coolsurf.com 2.6.23.12-52.fc7 #1 SMP Tue Dec 18 20:27:10 EST 2007 x86_64 x86_64 x86_64 GNU/Linux net1#build_sqlite mkdir -p /build/work/sqlite-3.5.4 cd /build/work/sqlite-3.5.4 unset CDPATH export CFLAGS='-pipe -O3 -g -DSQLITE_DISABLE_DIRSYNC=1 -Wall' rm -rf bld cvs -d :pserver:anonymous@www.sqlite.org:/sqlite -r update . mkdir bld cd bld ../configure --prefix=/common/pkgs/sqlite-3.5.4 --enable-tcl --with-tcl=/usr/lib64 --enable-threadsafe --enable-threads-override-locks make groupadd vuser || /bin/true useradd -M -g vuser -d /vhost/davidfavor.com/users/david -s /bin/zsh david || /bin/true useradd -M -g vuser -d /vhost/livefeast.com/users/yemiah -s /bin/zsh yemiah || /bin/true chown david:vuser -R .. su -c "make test" david _2008-Jan-11 17:18:52 by anonymous:_ {linebreak} Same tests still fail with CVS of today around 11AM CST. ---- _2008-Jan-11 17:41:55 by drh:_ {linebreak} FWIW, both those test cases pass on SuSE 10.1. I do not understand why they are failing on Fedora. But in any event, the tests in question are verifying logic that implements an optimization that is not used on Fedora, ever. So the failures are of no consequence. If those are the only two tests that fail, then you can safely use the build for whatever it is you are trying to do. ---- _2008-Jan-11 19:16:19 by anonymous:_ {linebreak} Failures when 'make fulltest' built with CFLAGS of '-pipe -O3 -g -Wall -DSQLITE_DISABLE_DIRSYNC=1 -DSQLITE_MEMDEBUG' exclusive-malloc-1.transient.746...make: *** [fulltest] Segmentation fault Failures when 'make fulltest' built with CFLAGS of '-pipe -O3 -g -Wall -DSQLITE_DISABLE_DIRSYNC=1' Skipping malloc tests: not compiled with -DSQLITE_MEMDEBUG... 6 errors out of 61998 tests Failures on these tests: exclusive-ioerr-2.280.4 exclusive-ioerr-2.281.4 exclusive-ioerr-2.282.4 incrvacuum-ioerr-1.31.4 io-4.1 io-4.2.3 All memory allocations freed - no leaks Maximum memory usage: 14376554 bytes Pre 3.5.x builds work fine on Fedora. If you're open to debugging all these, I'd like to go through and resolve all these one by one, so Fedora has a clean build/fulltest. Please let me know how to run each test individually and I'll try to figure out the problem with each. Thanks. #e8e8bd 2847 new active 2007 Dec anonymous 2008 Jan 5 4 Include major, minor, and patch version numbers in sqlite.h Hi, I'm working on a project where sqlite is being compiled into a DLL. Currently, sqlite.h makes the version number available as both an x.y.z string and an integer value in the form of x*1000000 + y*1000 + z. Unfortunately, neither of these options works particularly well when trying to create a resource file so that the DLL can display the proper version information within Windows. I've tried many different ways of disassembling the integer version number, but limitations in the resource compiler unfortunately prevent them from working. As a result, I've been forced for the time being to define SQLITE_VERSION_MAJOR, SQLITE_VERSION_MINOR, and SQLITE_VERSION_PATCH with manually-given values of x, y, and z respectively in order to accomplish this task. It would be really nice if these could be generated automatically for sqlite.h when running configure in the same way that VERSION and VERSION_NUMBER are so that setting the values manually wouldn't be required for the future. Would you be willing to do that? Thanks in advance. _2007-Dec-17 13:30:22 by anonymous:_ {linebreak} You could invoke this awk script in your make file:# ## extrvers.awk ## Extract verison parts from sqlite3.h # ## Usage: # %GNU_AWK% -f extrvers.awk sqlite3.h >sqlite3.h.new # rm / del sqlite3.h # mv / ren sqlite3.h.new sqlite3.h # # ## Ignore any previous defines /^#define SQLITE_VERSION_(MAJOR|MINOR|PATCH)/{ next } ## generate extra #define MAJOR/MINOR/PATH lines /^#define[[:blank:]]+SQLITE_VERSION[[:blank:]]/{ split(substr($3,2,length($3) - 2),tmp,".") print "#define SQLITE_VERSION_MAJOR " tmp[1] print "#define SQLITE_VERSION_MINOR " tmp[2] print "#define SQLITE_VERSION_PATCH " tmp[3] } ## Repeat all other lines untouched { print }---- _2007-Dec-18 00:06:53 by anonymous:_ {linebreak} Original poster here. Thanks for the useful script! I'm assuming you're granting a license for this (or a modified version of it) to be included in the source tree if the powers that be are willing to accept it? This would be for the Mozilla project. ---- _2007-Dec-18 00:21:52 by drh:_ {linebreak} First off, I didn't post the script. I don't know who "anonymous" is. Secondly, if you are working for Mozilla, you will get *much* faster service if you identify yourself as such. ---- _2007-Dec-18 00:39:38 by anonymous:_ {linebreak} I just want to make sure I'm not running afoul of anybody by using their work without proper permission. To whoever posted it, you can contact me at ryanvm [at] gmail [dot] com. Thanks again for help! ---- _2007-Dec-26 19:06:27 by anonymous:_ {linebreak} I'm the poster of the script. Of course I don't mind it being used. For the peace of mind of anyone using it: please prepend the code with:# Copyright (C) 2007 by Kees Nuyt, Rotterdam, Netherlands # The author of this code dedicates any and all copyright # interest in this code to the public domain. I make this # dedication for the benefit of the public at large and # to the detriment of my heirs and successors. I intend # this dedication to be an overt act of relinquishment in # perpetuity of all present and future rights to this # code under copyright law. #Cheers! "Kees Nuyt"---- _2007-Dec-27 02:32:44 by anonymous:_ {linebreak} Declaring a copyright and putting it into the public domain is not compatible. ---- _2008-Jan-11 04:55:22 by anonymous:_ {linebreak} Original poster here: FYI, I ended up writing a Python script to parse sqlite3.h for what we needed. The Mozilla bug for the work was bug 408603. You can see the final script here: http://mxr.mozilla.org/mozilla/source/db/sqlite3/src/sqlite-version.py At this point, you can close the bug if you want. I'll leave that up to you to decide. #f2dcdc 2879 code active 2008 Jan anonymous 2008 Jan anonymous 4 3 VACUUM enters temporary sequence numbers in sqlite_sequence I have two TEMPORARY tables added to a database, that both contain an INTEGER PRIMARY KEY AUTOINCREMENT field. After issuing a VACUUM command, the maximum value of these fields is added to the sqlite_sequence table. And stay there after the connection closes. #e8e8bd 2336 warn active 2007 May anonymous 2008 Jan 4 4 Compiler warnings on strcpy(), sprintf(), et al. OpenBSD's gcc compiler reports link warnings concerning potential insecure use of certain functions that have a history of abuse, such as strcpy, strcat, sprintf. It would be nice if SQLite could use the more secure version when possible. /home/achowe/Projects/org/sqlite/lib/libsqlite3.a(util.o)(.text+0x184): In function `sqlite3StrDup': : warning: strcpy() is almost always misused, please use strlcpy() /home/achowe/Projects/org/sqlite/lib/libsqlite3.a(os_unix.o)(.text+0x822): In function `sqlite3UnixTempFileName': : warning: sprintf() is often misused, please use snprintf() There was no misuse of these functions. But we'll get rid of them to avoid the library dependency. For those that think that the use of strcpy() is a security problem: if you will check the diffs for this check-in you will see that by avoiding the use of strcpy() we have made the code more obtuse and more likely to contain a security bug, not the other way around. But at least now the compiler will not give warnings about strcpy()... ---- _2007-May-04 15:43:08 by anonymous:_ {linebreak} Some are still left in fts1.c and fts2.c. ---- _2008-Jan-07 10:54:43 by anonymous:_ {linebreak} SQLite 3.5.4 now generates new warnings on OpenBSD concerning unsafe functions. gcc -g -O2 -o lemon ./tool/lemon.c /tmp//cco18706.o(.text+0x14df): In function `ErrorMsg': tool/lemon.c:1327: warning: vsprintf() is often misused, please use vsnprintf() /tmp//cco18706.o(.text+0x1655): In function `handle_D_option': tool/lemon.c:1378: warning: strcpy() is almost always misused, please use strlcpy() /tmp//cco18706.o(.text+0x14a2): In function `ErrorMsg': tool/lemon.c:1319: warning: sprintf() is often misused, please use snprintf() /tmp//cco18706.o(.text+0x3918): In function `file_makename': tool/lemon.c:2678: warning: strcat() is almost always misused, please use strlcat() #f2dcdc 2875 code active 2008 Jan anonymous 2008 Jan 3 2 LIKE does not work with lowercase swedish characters Swedish letters å,ä,ö is not supported by the LIKE statement. When trying to perform a query like SELECT * FROM table WHERE name LIKE "å%" we will not get a match for names starting on Å (which is uppercase for å). To recreate: ============================================== SQLite version 3.5.4 Enter ".help" for instructions sqlite> CREATE TABLE TestingTable(Name varchar(20)); sqlite> INSERT INTO TestingTable values ('Sweden'); sqlite> INSERT INTO TestingTable values ('sweden'); sqlite> INSERT INTO TestingTable values ('Åland'); sqlite> INSERT INTO TestingTable values ('åland'); sqlite> SELECT * FROM TestingTable; Sweden sweden Åland åland sqlite> SELECT * FROM TestingTable WHERE Name LIKE "swe%"; Sweden sweden sqlite> SELECT * FROM TestingTable WHERE Name LIKE åla%"; åland ============================================================ #e8e8bd 2853 new active 2007 Dec anonymous 2008 Jan 2 3 optimizer fails to use an index on MAX subquery i have these 2 identical queries with an index on (place_id,visit_date), the second query is about 2x fast the the first, while i'd expect that the MAX is faster than a limited order by clause... It's like the index is mis-used with MAX SELECT h.id, h.url, h.title, h.rev_host, h.visit_count, (SELECT MAX(visit_date) FROM moz_historyvisits WHERE place_id = h.id AND visit_type NOT IN(0,4)), f.url, null, null FROM moz_places h LEFT OUTER JOIN moz_favicons f ON h.favicon_id = f.id WHERE h.id IN (SELECT DISTINCT p.id FROM moz_places p JOIN moz_historyvisits ON place_id = p.id WHERE hidden <> 1 AND visit_type NOT IN (0,4) ORDER BY visit_date DESC LIMIT 10) ORDER BY 6 DESC; SELECT h.id, h.url, h.title, h.rev_host, h.visit_count, (SELECT visit_date FROM moz_historyvisits WHERE place_id = h.id AND visit_type NOT IN(0,4) ORDER BY visit_date DESC LIMIT 1), f.url, null, null FROM moz_places h LEFT OUTER JOIN moz_favicons f ON h.favicon_id = f.id WHERE h.id IN (SELECT DISTINCT p.id FROM moz_places p JOIN moz_historyvisits ON place_id = p.id WHERE hidden <> 1 AND visit_type NOT IN (0,4) ORDER BY visit_date DESC LIMIT 10) ORDER BY 6 DESC; _2007-Dec-20 04:13:44 by anonymous:_ {linebreak} Can you supply the schema of these tables and all related indexes? ---- _2007-Dec-20 10:55:29 by danielk1977:_ {linebreak} Correct. SQLite does not use the index to optimize the max() in the first query. But it does use it to optimize ORDER BY in the second. ---- _2007-Dec-21 00:54:10 by anonymous:_ {linebreak} the schema is that of mozilla firefox 3 places.sqlite, with an added index on moz_historyvisits(place_id,visit_date), this is extracted from a i bug i was working on. Do you need that i append the full schema here? Will this be fixed to use the index with max or is that the expected behaviour? ---- _2007-Dec-21 15:59:58 by drh:_ {linebreak} We will be enhancing the min()/max() optimization to be able to cover the case you describe above. But this will take some time to get right. If you are in a hurry, it seems like modifying your SQL is the easiest way to go. The current optimizer recognizes queries of the form: SELECT min(x) FROM table; SELECT max(x) FROM table; And it causes special VDBE code to be generated for these cases. The presence of a WHERE clause defeats the current optimization. We need to modify the optimizer to recognize queries of the form: SELECT min(x) FROM table WHERE expr And convert them into: SELECT x FROM table WHERE expr AND x NOT NULL ORDER BY x LIMIT 1 But we only want to do this optimization if the ORDER BY can be evaluated using an index. If we have to accumulate the entire result set, sort it, then take the largest entry, that is more work than just keeping track of the largest entry as we loop through the result set. Notice the addition of the "x NOT NULL" term in the WHERE clause. This is critical to preserving correct semantics. The query optimizer currently does not know how to deal with a NOT NULL. It just evaluates all rows, tests them individually for NULL, and throws out those that match. This would work in most cases, but would slow down in a min() where there were many NULL entries. The current implemention of the min() optimization knows to skip over the NULL entries in a single operation. The WHERE generator part of the optimizer will need to be enhanced to recognize NOT NULL constraints and skip over them. All of this is a substantial change. The min()/max() optimization function will be rewritten from scratch. Significant and non-trivial modifications will need to be made to the code that looks for indices and generates the WHERE clause constraints. There are many opportunities to make coding mistakes, so we will need to spend a lot of time in testing before we put these changes into production. So, we will be working the problem. But do not expect an overnight fix. I suppose that while we are redesigning the min/max optimizer, we might as well also fix it so that SELECT arbitrary_expr(max(x)) FROM table WHERE expr; gets converted into SELECT arbitrary_expr(x) FROM table WHERE expr AND x NOT NULL ORDER BY x DESC LIMIT 1; ---- _2007-Dec-31 06:41:03 by anonymous:_ {linebreak} Regarding: SELECT min(x) from table WHERE expr being converted to: SELECT x FROM table WHERE expr AND x NOT NULL ORDER BY x LIMIT 1 There's no need for the "x NOT NULL" condition considering NULL is returned by min() (or max for that matter) when no rows match. sqlite> .nullvalue sqlite> select min(a) from (select 123 as a) where a=7; sqlite> select min(a) from (select NULL as a) where a=7; Even with this in mind, you can see that the rewritten query still does not return the same result in this case: sqlite> select a from (select 123 as a) where a=7 order by 1 limit 1; -- no rows returned Logic would have to be added to return a NULL row in the event the WHERE clause matches no rows. ---- _2007-Dec-31 07:06:22 by anonymous:_ {linebreak} Given: SELECT min(x) from table WHERE expr If column *x* or the *WHERE expr* can make use of an index, then the min query should be converted to: SELECT x from ( SELECT x FROM table WHERE expr ORDER BY x LIMIT 1 ) UNION ALL SELECT NULL LIMIT 1 which ought to cover all the corner cases, even if the WHERE matches no rows. ---- _2008-Jan-01 19:21:51 by anonymous:_ {linebreak} This ticket is related to WHERE cost estimation in ticket #2857. Given: create table stuff(a,b,c,d); insert into stuff values(1,2,3,4); create temp view v1 as select random()%100, random()%100, random()%1000, random()%10000 from stuff x, stuff y; insert into stuff select * from v1; insert into stuff select * from v1; insert into stuff select * from v1; insert into stuff select * from v1; insert into stuff select * from v1; create index stuff_b on stuff(b); create index stuff_c on stuff(c); create index stuff_d on stuff(d); analyze; In the following example the existing min() implementation which uses a full table scan: sqlite> explain query plan select min(c) from stuff where a=12345; 0|0|TABLE stuff sqlite> select min(c) from stuff where a=12345; (null) CPU Time: user 1.388087 sys 0.216014 is faster than performing the select min->order by/limit transform which uses an index: sqlite> explain query plan select c from stuff where a=12345 order by 1 limit 1; 0|0|TABLE stuff WITH INDEX stuff_c ORDER BY sqlite> explain query plan select c from (select c from stuff where a=12345 order by c limit 1) union all select null limit 1; 0|0|TABLE stuff WITH INDEX stuff_c ORDER BY 0|0|TABLE AS sqlite_subquery_82F83F8_ CPU Time: user 0.000000 sys 0.000000 sqlite> select c from (select c from stuff where a=12345 order by c limit 1) union all select null limit 1; (null) CPU Time: user 15.880993 sys 15.400962 Note that a=12345 is always false in this data set. Had a=23 been used instead, we can see that the transformed select would be much faster than the original min() select: sqlite> select min(c) from stuff where a=23; -999 CPU Time: user 1.552097 sys 0.148009 sqlite> select c from (select c from stuff where a=23 order by c limit 1) union all select null limit 1; -999 CPU Time: user 0.004001 sys 0.000000 I don't think WHERE clause cost estimation can be done accurately without using a statistical method based on historical WHERE clause hit percentages. ---- _2008-Jan-05 18:15:39 by anonymous:_ {linebreak} I appreciate that you have more refinements pending, but this query in particular has regressed from sqlite 3.5.4 using the schema mentioned above: -- sqlite 3.5.4 sqlite> select min(c) from stuff where a=12345; (null) CPU Time: user 1.532095 sys 0.132008 -- as of Check-in [4687] sqlite> select min(c) from stuff where a=12345; (null) CPU Time: user 23.041440 sys 16.369023 #f2dcdc 2874 code active 2008 Jan anonymous 2008 Jan 1 1 THREADSAFE #define HAVE_LOCALTIME_R, HAVE_GMTIME_R in os_unix.c The precompiled shared sqlite3 library for Linux on sqlite.org which appears to be built with pthread support is using localtime and gmtime which are not threadsafe. For THREADSAFE builds could either configure be changed to detect the functions gmtime_r and localtime_r or change os_unix.c to explicitly #define HAVE_LOCALTIME_R and HAVE_GMTIME_R? #f2dcdc 2873 code active 2008 Jan anonymous 2008 Jan 1 1 HAVE_USLEEP, HAVE_FDATASYNC=1 detected but not used by configure; make I noticed that a couple of open source projects were not picking up usleep() for recent sqlite builds and used the coarser grained sleep() instead. Around 11 months ago something changed in sqlite's build process. It seems that both -DHAVE_USLEEP=1, -DHAVE_FDATASYNC=1 and -DOS_UNIX=1 are detected correctly by configure but not used by the generated Makefile. As result, UNIX builds of sqlite3 via ./configure do not use usleep() and fdatasync() and do not define OS_UNIX. I don't know whether the lack of fdatasync() versus the default fsync() affects anyone. Please apply the following patch which corrects the problem with "./configure && make". Thank you. Index: configure =================================================================== RCS file: /sqlite/sqlite/configure,v retrieving revision 1.45 diff -u -3 -p -r1.45 configure --- configure 27 Nov 2007 14:50:07 -0000 1.45 +++ configure 5 Jan 2008 07:41:00 -0000 @@ -18520,9 +18520,9 @@ if test "$TARGET_EXEEXT" = ".exe"; then OS_UNIX=0 OS_WIN=0 OS_OS2=1 - TARGET_CFLAGS="$TARGET_CFLAGS -DOS_OS2=1" + CFLAGS="$CFLAGS -DOS_OS2=1" if test "$ac_compiler_gnu" == "yes" ; then - TARGET_CFLAGS="$TARGET_CFLAGS -Zomf -Zexe -Zmap" + CFLAGS="$CFLAGS -Zomf -Zexe -Zmap" BUILD_CFLAGS="$BUILD_CFLAGS -Zomf -Zexe" fi else @@ -18530,14 +18530,14 @@ if test "$TARGET_EXEEXT" = ".exe"; then OS_WIN=1 OS_OS2=0 tclsubdir=win - TARGET_CFLAGS="$TARGET_CFLAGS -DOS_WIN=1" + CFLAGS="$CFLAGS -DOS_WIN=1" fi else OS_UNIX=1 OS_WIN=0 OS_OS2=0 tclsubdir=unix - TARGET_CFLAGS="$TARGET_CFLAGS -DOS_UNIX=1" + CFLAGS="$CFLAGS -DOS_UNIX=1" fi @@ -19392,7 +19392,7 @@ fi echo "$as_me:$LINENO: result: $ac_cv_func_usleep" >&5 echo "${ECHO_T}$ac_cv_func_usleep" >&6 if test $ac_cv_func_usleep = yes; then - TARGET_CFLAGS="$TARGET_CFLAGS -DHAVE_USLEEP=1" + CFLAGS="$CFLAGS -DHAVE_USLEEP=1" fi @@ -19491,7 +19491,7 @@ fi echo "$as_me:$LINENO: result: $ac_cv_func_fdatasync" >&5 echo "${ECHO_T}$ac_cv_func_fdatasync" >&6 if test $ac_cv_func_fdatasync = yes; then - TARGET_CFLAGS="$TARGET_CFLAGS -DHAVE_FDATASYNC=1" + CFLAGS="$CFLAGS -DHAVE_FDATASYNC=1" fi Index: configure.ac =================================================================== RCS file: /sqlite/sqlite/configure.ac,v retrieving revision 1.31 diff -u -3 -p -r1.31 configure.ac --- configure.ac 27 Nov 2007 14:50:07 -0000 1.31 +++ configure.ac 5 Jan 2008 07:41:00 -0000 @@ -310,9 +310,9 @@ if test "$TARGET_EXEEXT" = ".exe"; then OS_UNIX=0 OS_WIN=0 OS_OS2=1 - TARGET_CFLAGS="$TARGET_CFLAGS -DOS_OS2=1" + CFLAGS="$CFLAGS -DOS_OS2=1" if test "$ac_compiler_gnu" == "yes" ; then - TARGET_CFLAGS="$TARGET_CFLAGS -Zomf -Zexe -Zmap" + CFLAGS="$CFLAGS -Zomf -Zexe -Zmap" BUILD_CFLAGS="$BUILD_CFLAGS -Zomf -Zexe" fi else @@ -320,14 +320,14 @@ if test "$TARGET_EXEEXT" = ".exe"; then OS_WIN=1 OS_OS2=0 tclsubdir=win - TARGET_CFLAGS="$TARGET_CFLAGS -DOS_WIN=1" + CFLAGS="$CFLAGS -DOS_WIN=1" fi else OS_UNIX=1 OS_WIN=0 OS_OS2=0 tclsubdir=unix - TARGET_CFLAGS="$TARGET_CFLAGS -DOS_UNIX=1" + CFLAGS="$CFLAGS -DOS_UNIX=1" fi AC_SUBST(BUILD_EXEEXT) @@ -565,13 +565,13 @@ AC_SUBST(TARGET_DEBUG) ######### # Figure out whether or not we have a "usleep()" function. # -AC_CHECK_FUNC(usleep, [TARGET_CFLAGS="$TARGET_CFLAGS -DHAVE_USLEEP=1"]) +AC_CHECK_FUNC(usleep, [CFLAGS="$CFLAGS -DHAVE_USLEEP=1"]) #-------------------------------------------------------------------- # Redefine fdatasync as fsync on systems that lack fdatasync #-------------------------------------------------------------------- -AC_CHECK_FUNC(fdatasync, [TARGET_CFLAGS="$TARGET_CFLAGS -DHAVE_FDATASYNC=1"]) +AC_CHECK_FUNC(fdatasync, [CFLAGS="$CFLAGS -DHAVE_FDATASYNC=1"]) ######### # Generate the output files._2008-Jan-05 08:24:29 by anonymous:_ {linebreak} It appears that http://www.sqlite.org/sqlite3-3.5.4.bin.gz and http://www.sqlite.org/sqlite-3.5.4.so.gz use sleep and fsync even though usleep and fdatasync are available on Linux. On the Linux man page, it claims that fdatasync is more efficient than fsync: "Unfortunately, fsync() will always initiate two write operations: one for the newly written data and another one in order to update the modification time stored in the inode. If the modification time is not a part of the transaction concept fdatasync() can be used to avoid unnecessary inode disk write operations." #f2dcdc 2872 code active 2008 Jan anonymous 2008 Jan 4 4 Some table scan operations could use an index for better data density If a suitable index covers a column being during a table scan, it makes sense to use the index for IO not the table pages themselves for speed. As (contrived) example: CREATE TABLE t1 ( c1 blob, c2 integer, c3 integer ); CREATE INDEX i1 on t1(c2,c3); CREATE INDEX i2 on t1(c3); T1 populated with 4096 rows, c1 being 64K random blobs (to make c2, c3 access slower in this case), c2 and c3 being small random integers. Now: sqlite> select sum(c3) from t1; sum(c3) ---------- 519895 is very slow (several seconds). A table scan is done. Forcing use of an index, in a way that I know *all* rows will be included: sqlite> select sum(c3) from t1 where c2<1000; sum(c3) ---------- 519895 This is instantaneous. It seems to me that if a table scan has to be performance, it makes sense to grab the data from an index whenever possible. ideally the most densely packed index. (BTW; this is contrived, I know putting the blob as the last column will greatly speed things up). #e8e8bd 2869 new active 2008 Jan anonymous 2008 Jan 5 5 add "sqlite3_open16_v2" to the C API I'm using UTF-16, and if the database file does not exist, "sqlite3_open16" will create a new one, but i wish it fails in such conditions. I notice that there's a "sqlite3_open_v2", but it doesn't support UTF-16, although i can implement a "sqlite3_open16_v2" myself, I think it should exists in the offical releases. #f2dcdc 2867 code active 2008 Jan anonymous 2008 Jan 2 2 doesn't build on Cygwin - wrong sqlite3 exe suffix The new Makefile used $(EXE), which doesn't seem to be defined (typo?) _2008-Jan-02 11:12:39 by anonymous:_ {linebreak} Same on mingw: Following patch fixes things:--- sqlite-3.5.4/Makefile.in Thu Dec 13 19:17:42 2007 +++ sqlite-3.5.4-mingw-fix/Makefile.in Wed Jan 2 11:37:50 2008 @@ -322,7 +322,7 @@ -o $@ $(TOP)/src/shell.c libsqlite3.la \ $(LIBREADLINE) $(TLIBS) -sqlite3$(EXE): $(TOP)/src/shell.c sqlite3.c sqlite3.h +sqlite3$(TEXE): $(TOP)/src/shell.c sqlite3.c sqlite3.h $(LTLINK) $(READLINE_FLAGS) -o $@ \ -DSQLITE_MAX_SQL_LENGTH=1000000000 \ -USQLITE_THREADSAFE -DSQLITE_THREADSAFE=0 \ @@ -577,7 +577,7 @@ -e 's,$$,\\n",' \ $(TOP)/tool/spaceanal.tcl >spaceanal_tcl.h $(LTLINK) -DTCLSH=2 -DSQLITE_TEST=1 $(TEMP_STORE)\ - -o sqlite3_analyzer$(EXE) $(TESTSRC) $(TOP)/src/tclsqlite.c \ + -o sqlite3_analyzer$(TEXE) $(TESTSRC) $(TOP)/src/tclsqlite.c \ libtclsqlite3.la $(LIBTCL)#e8e8bd 2866 build active 2008 Jan anonymous 2008 Jan 1 3 Problems building Windows native in cygwin/mingw environment Trying to build Windows native version using the Cygwin build environment. $ gcc -v Reading specs from /usr/lib/gcc/i686-pc-cygwin/3.4.4/specs Configured with: /usr/build/package/orig/test.respin/gcc-3.4.4-3/configure --verbose --prefix=/usr --exec-prefix=/usr --sysconfdir=/etc --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --enable-languages=c,ada,c++,d,f77,pascal,java,objc --enable-nls --without-included-gettext --enable-version-specific-runtime-libs --without-x --enable-libgcj --disable-java-awt --with-system-zlib --enable-interpreter --disable-libgcj-debug --enable-threads=posix --enable-java-gc=boehm --disable-win32-registry --enable-sjlj-exceptions --enable-hash-synchronization --enable-libstdcxx-debug Thread model: posix gcc version 3.4.4 (cygming special, gdc 0.12, using dmd 0.125) $ $ CFLAGS=-mno-cygwin ./configure --disable-tcl --enable-threadsafe $ make A) The make appears to build sqlite3.exe just fine, without errors or warnings. This binary does work from cmd.exe, BUT not from within the bash cygwin shell for some reason, unlike other Windows native binaries I've built. Next... $ make install B) The cc sqlite3.c -o sqlite3 fails to rebuild sqlite3.exe correctly with the -mno-cygwin option. The output follows: rm -rf tsrc mkdir -p tsrc cp ./src/alter.c ./src/analyze.c ./src/attach.c ./src/auth.c ./src/btmutex.c ./src/btree.c ./src/btree.h ./src/build.c ./src/callback.c ./src/complete.c ./src/date.c ./src/delete.c ./src/expr.c ./src/func.c ./src/hash.c ./src/hash.h ./src/insert.c ./src/journal.c ./src/legacy.c ./src/loadext.c ./src/main.c ./src/malloc.c ./src/mem1.c ./src/mem2.c ./src/mem3.c ./src/mutex.c ./src/mutex_os2.c ./src/mutex_unix.c ./src/mutex_w32.c ./src/os.c ./src/os_unix.c ./src/os_win.c ./src/os_os2.c ./src/pager.c ./src/pager.h ./src/parse.y ./src/pragma.c ./src/prepare.c ./src/printf.c ./src/random.c ./src/select.c ./src/shell.c ./src/sqlite.h.in ./src/sqliteInt.h ./src/table.c ./src/tclsqlite.c ./src/tokenize.c ./src/trigger.c ./src/utf.c ./src/update.c ./src/util.c ./src/vacuum.c ./src/vdbe.c ./src/vdbe.h ./src/vdbeapi.c ./src/vdbeaux.c ./src/vdbeblob.c ./src/vdbefifo.c ./src/vdbemem.c ./src/vdbeInt.h ./src/vtab.c ./src/where.c ./ext/fts1/fts1.c ./ext/fts1/fts1.h ./ext/fts1/fts1_hash.c ./ext/fts1/fts1_hash.h ./ext/fts1/fts1_porter.c ./ext/fts1/fts1_tokenizer.h ./ext/fts1/fts1_tokenizer1.c sqlite3.h ./src/btree.h ./src/btreeInt.h ./src/hash.h ./src/sqliteLimit.h ./src/mutex.h opcodes.h ./src/os.h ./src/os_common.h ./src/sqlite3ext.h ./src/sqliteInt.h ./src/vdbe.h parse.h ./ext/fts1/fts1.h ./ext/fts1/fts1_hash.h ./ext/fts1/fts1_tokenizer.h ./src/vdbeInt.h tsrc cp: warning: source file `./src/btree.h' specified more than once cp: warning: source file `./src/hash.h' specified more than once cp: warning: source file `./src/sqliteInt.h' specified more than once cp: warning: source file `./src/vdbe.h' specified more than once cp: warning: source file `./ext/fts1/fts1.h' specified more than once cp: warning: source file `./ext/fts1/fts1_hash.h' specified more than once cp: warning: source file `./ext/fts1/fts1_tokenizer.h' specified more than once cp: warning: source file `./src/vdbeInt.h' specified more than once rm tsrc/sqlite.h.in tsrc/parse.y cp parse.c opcodes.c keywordhash.h tsrc tclsh ./tool/mksqlite3c.tcl cc sqlite3.c -o sqlite3 /usr/lib/gcc/i686-pc-cygwin/3.4.4/../../../libcygwin.a(libcmain.o):(.text+0xab): undefined reference to `_WinMain@16' collect2: ld returned 1 exit status make: *** [sqlite3] Error 1 $ #e8e8bd 2864 doc active 2007 Dec anonymous 2007 Dec 5 3 ext/fts3/README.txt File ext/fts3/README.txt reads: This folder contains source code to the second full-text search [...] Shouldn't that be: This folder contains source code to the third full-text search [...] _2007-Dec-30 18:27:08 by anonymous:_ {linebreak} Oh, after Googl'ing a little bit, I found that _fts3_ really is _fts2-with-rowid-fixed_. If both _fts2_ and _fts3_ are considered to be the _"second full-text search extension for SQLite"_, the _README_ files could maybe explain the situation. #f2dcdc 2865 code active 2007 Dec anonymous 2007 Dec 1 2 FTS3 does not build with amalgamation in CVS Grab the latest CVS sources, then run: ./configure make sqlite3.c grep sqlite3Fts3Init sqlite3.c extern int sqlite3Fts3Init(sqlite3*); rc = sqlite3Fts3Init(db); If you compile sqlite3.c with -DSQLITE_ENABLE_FTS3, then sqlite3Fts3Init is unresolved. For some reason, sqlite3Fts3Init and fts3.c was not included in the sqlite3.c amalg. It used to work correctly in 3.5.4. _2007-Dec-30 18:17:57 by anonymous:_ {linebreak} Nevermind, "make sqlite3.c" has never built with the fts3 sources in 3.5.4 or before. You have to run ext/fts3/mkfts3amal.tcl ---- _2007-Dec-30 18:20:56 by anonymous:_ {linebreak} It seems that the sqlite3+fts3 amalg can only be built from main.mk, not Makefile. #f2dcdc 2863 code active 2007 Dec anonymous 2007 Dec 2 3 test cast-3.14, cast-3.18 and cast-3.24 fail test cast-3.{14,18,24} fail on freebsd-6.3-PRERELEASE2: cast-3.14...^M Expected: [9223372036854774784]^M Got: [9223372036854773760]^M cast-3.18...^M Expected: [-9223372036854774784]^M Got: [-9223372036854773760]^M cast-3.24...^M Expected: [9223372036854774784]^M Got: [9223372036854773760]^M I used tcl8.4 from ports with no threads and here was the config line: ../sqlite-3.5.4/configure --prefix=/home/marc/local --with-tcl=/usr/local/lib/tcl8.4/ This was built on an ibm t30 laptop #e8e8bd 2860 todo active 2007 Dec anonymous 2007 Dec 3 1 Database file fragmentation Adding data in database file increases file fragmentation. for example my file which size is 1G, consists of 20000 pieces. (NTFS) This happens because truncation of '-journal' file. I see some ways to reduce fragmentaion: 1. Increase database file size by greater pieces (not by PAGESIZE). 2. SQLite can save '-journal' file in another folder(logical disc). 3. Preallocation of database file(must increase INSERT speed). #f2dcdc 2859 code active 2007 Dec anonymous 2007 Dec drh 3 2 Inconsistent column names with DISTINCT Given the following SQL:{linebreak} CREATE TABLE foo(a,b); INSERT INTO foo (a, b) VALUES (1,2); SQLite returns inconsistent column names when using the DISTINCT clause:{linebreak} SELECT DISTINCT foo.A, foo.B FROM foo; foo.A|foo.B 1|2 SELECT DISTINCT a, b FROM foo; a|b 1|2 SELECT DISTINCT * FROM foo; a|b 1|2 SELECT DISTINCT foo.* FROM foo; a|b 1|2 Compared with SELECT without DISTINCT:{linebreak} SELECT foo.A, foo.B FROM foo; a|b 1|2 SELECT a, b FROM foo; a|b 1|2 SELECT * FROM foo; a|b 1|2 SELECT foo.* FROM foo; a|b 1|2 #f2dcdc 2761 code active 2007 Nov anonymous 2007 Dec 3 3 CLI (shell.c) should be bundled with amalgamation The CLI (shell.c) should be bundled with the amalgamation for database administrative purposes without downloading the matching shell.c from the full source tree. I second that! Qt ships with the amalgamated source files, but we also ship shell.c, whch we have to retrieve from the non-amalgamated source files. ---- _2007-Dec-26 15:20:04 by anonymous:_ {linebreak} I also agree. It is inconvenient to retrieve the matching shell.c from the source tree. #f2dcdc 2857 code active 2007 Dec anonymous 2007 Dec 2 2 GROUP BY cost estimate wrong with WHERE clause There seems to be an issue with the sqlite cost heuristic with an INDEX present on GROUP BY with certain types of WHERE clauses. Given the database formed by running these statements: create table stuff(a,b,c,d); insert into stuff values(1,2,3,4); create temp view v1 as select random()%100, random()%100, random()%1000, random()%10000 from stuff x, stuff y; insert into stuff select * from v1; insert into stuff select * from v1; insert into stuff select * from v1; insert into stuff select * from v1; insert into stuff select * from v1; create index stuff_b on stuff(b); create index stuff_c on stuff(c); create index stuff_d on stuff(d); analyze; Using sqlite.org's sqlite3-3.5.4.bin, this query takes 47 seconds: select c from stuff where a=23 group by c; while this query takes just 2 seconds: select c from stuff where a=23 group by +c; It is more efficient in this case to do a full table scan instead of using the INDEX on column c. _2007-Dec-23 23:14:06 by anonymous:_ {linebreak} The queries above both run in a couple of seconds with this naive patch:--- src/where.c 12 Dec 2007 17:42:53 -0000 1.266 +++ src/where.c 23 Dec 2007 22:48:37 -0000 @@ -1514,6 +1514,12 @@ static double bestIndex( flags = 0; } + if( pWC && pWC->nTerm>0 && pOrderBy ){ + /* Reduce cost if both an ORDER/GROUP BY exists with a WHERE. */ + cost /= 100; /* A very rough guess. */ + WHERETRACE(("... WHERE + ORDER BY decreases cost to: %.9g\n", cost)); + } + /* If the table scan does not satisfy the ORDER BY clause, increase ** the cost by NlogN to cover the expense of sorting. */ if( pOrderBy ){But it has not been tested on queries with more than one table. Its logic could be flawed. ---- _2007-Dec-24 00:09:00 by drh:_ {linebreak} The complaint is centered around these two queries: /* 1 */ SELECT c FROM stuff WHERE a=23 GROUP BY c; /* 2 */ SELECT c FROM stuff WHERE a=23 GROUP BY +c; Query 1 runs in about 40 seconds and query 2 in about 1.5 seconds on my macbook. But with the patch, both queries run in about 1.5 seconds. Fair enough. But now consider these two queries: /* 3 */ SELECT c FROM stuff WHERE a!=23 GROUP BY c; /* 4 */ SELECT c FROM stuff WHERE a!=23 GROUP BY +c; In this case, query 3 runs in 42 seconds on an unpatched version of 3.5.4 and query 4 runs in about 109 seconds. So in cases where the WHERE clause is not particularly selective, the first version is faster than the second by a good margin. On a patched version of 3.5.4, both queries 3 and 4 run in about 110 seconds. So it seems to me that the patch is robbing Peter to pay Paul. It makes ORDER BY queries with very selective WHERE clauses run faster but at the expense of making queries with unselective WHERE clauses running slower. But notice this: in the current (unpatched) implementation, the programmer at least has the ability to select a different algorithm by the judicious placement of a "+" sign. After the patch, this is no longer possible. The patch forces the second algorithm to be used in all cases, even cases where it is slower. It seems to me to be better to leave things as they are since the current approach at least allows the programmer to override SQLite's algorithm selection if SQLite chooses incorrectly. The only way, it seems to me, to automatically choose the correct algorithm is to devise some test that will determine (in advance) whether or not the WHERE clause weeds out many or few rows from the result set. I'm thinking that determination is going to be very hard (or impossible) to do without first doing a full table scan. ---- _2007-Dec-24 05:40:47 by anonymous:_ {linebreak} It think it would be surprising to average users that _adding_ an index (on column C in this case) may significantly _decrease_ query performance for some queries. It was surprising to me, at least. In my opinion, a query being 20 times slower in a default bad guess situation is worse than a query only being 2.5 times slower with a default bad guess in a worst case scenario. It's a question of relative magnitude of the difference. This is why I think that the database should err on the side of the WHERE clause having a more selective bias. (Side note: the query timings difference is less pronounced if you use PRAGMA temp_store=memory, in which case query 3 running on an unpatched 3.5.4 takes just 50% more time to run than query 4 on my machine.) But you raise a good point in that if there's a wrong guess in the selectivity bias it would be nice to be able to manually override it. How much do you hate this type of syntax that some other databases use? select c from stuff where a!=23 group by /*+stuff_c*/ c; SQLite does not currently offer a way to pick a specific index. I think it would be quite useful. ---- _2007-Dec-24 17:05:16 by anonymous:_ {linebreak} Another option is to collect WHERE clause statistics in a table like create table sqlite_stat2( where_clause_md5 BLOB primary key, where_clause TEXT, rows_examined INT, rows_true INT ); where the last 2 columns are cumulative for each query. The statistics option could be enabled/disabled via a PRAGMA sqlite_collect_statistics. The where_clause column could be a string generated fairly easily from the walking the parse tree of the resolved Select statement's pWhere. This way the where_clause is normalized and a single query with many subselects could generate more than 1 where_clause, and different queries that happen to use the same normalized where clause would update the same entry in the stat2 table. where_clause normalization would strip off aliases and only refer to the original table and column names. For example the 2 queries below: -- CREATE TABLE t1(a, b); -- CREATE TABLE t2(b, c); SELECT t1.a*c as AC, t2.b-a as BA FROM t1, t2 WHERE AC>BA; SELECT *, t1.a Foo FROM t2, t1 WHERE Foo*c > t2.b - t1.a; would generate the same normalized where_clause string "(T1.A*T2.C)>(T2.B-T1.A)". The table information is already encoded within it. The generated VDBE code would have to generate Mem counters that would be incremented by each WHERE test, and lazily updated at the end of transactions or periodically written to the stat2 table to minimize disk use, as this information is not critical. One could also manually set the stat2 table with statistical values they would like their queries to use even if PRAGMA sqlite_collect_statistics=off; Any time the schema is changed, the entire sqlite_stat2 table would be cleared. #e8e8bd 2856 doc active 2007 Dec anonymous 2007 Dec anonymous 4 3 SQLite Documentation - Tcl API - Link broken On the page "The Tcl Interface to the SQLite library", the link "enable load extension" does not work. #f2dcdc 2508 code active 2007 Jul anonymous 2007 Dec 1 1 utf8ToUnicode() does not work on some WinCE devices On some WinCE devices first call to =MultiByteToWideChar()= in =utf8ToUnicode()= always fails. Tried calling =GetLastError()= after it fails and it returns error code 87 -- =ERROR_INVALID_PARAMETER=. To fix this had to change code page from =CP_UTF8= to =CP_ACP= -- no idea why this works. Original =utf8ToUnicode()= ---- static WCHAR *utf8ToUnicode(const char *zFilename) { int nChar; WCHAR *zWideFilename; nChar = MultiByteToWideChar(CP_UTF8, 0, zFilename, -1, NULL, 0); zWideFilename = sqliteMalloc( nChar*sizeof(zWideFilename[0]) ); if( zWideFilename==0 ){ return 0; } nChar = MultiByteToWideChar(CP_UTF8, 0, zFilename, -1, zWideFilename, nChar); if( nChar==0 ){ sqliteFree(zWideFilename); zWideFilename = 0; } return zWideFilename; } ---- Fixed =utf8ToUnicode()= ---- static WCHAR *utf8ToUnicode(const char *zFilename) { int nChar; WCHAR *zWideFilename; nChar = MultiByteToWideChar(CP_ACP, 0, zFilename, -1, NULL, 0); if( nChar == 0 ) { DWORD dwError = GetLastError(); OSTRACE2("MultiByteToWideChar() failed, last error: %d\n", dwError); return 0; } zWideFilename = sqliteMalloc( nChar*sizeof(zWideFilename[0]) ); if( zWideFilename==0 ){ return 0; } nChar = MultiByteToWideChar(CP_ACP, 0, zFilename, -1, zWideFilename, nChar); if( nChar==0 ){ sqliteFree(zWideFilename); zWideFilename = 0; } return zWideFilename; } ---- _2007-Jul-17 23:56:10 by anonymous:_ {linebreak} =unicodeToUtf8()= needs to be fixed the same way. Before: ---- static char *unicodeToUtf8(const WCHAR *zWideFilename){ int nByte; char *zFilename; nByte = WideCharToMultiByte(CP_UTF8, 0, zWideFilename, -1, 0, 0, 0, 0); zFilename = sqliteMalloc( nByte ); if( zFilename==0 ){ return 0; } nByte = WideCharToMultiByte(CP_UTF8, 0, zWideFilename, -1, zFilename, nByte, 0, 0); if( nByte == 0 ){ sqliteFree(zFilename); zFilename = 0; } return zFilename; } ---- After: ---- static char *unicodeToUtf8(const WCHAR *zWideFilename){ int nByte; char *zFilename; nByte = WideCharToMultiByte(CP_ACP, 0, zWideFilename, -1, NULL, 0, NULL, NULL); if ( nByte == 0 ) { DWORD dwError = GetLastError(); OSTRACE2("WideCharToMultiByte() failed, last error = %d\n", dwError); return 0; } zFilename = sqliteMalloc( nByte ); if( zFilename==0 ){ return 0; } nByte = WideCharToMultiByte(CP_ACP, 0, zWideFilename, -1, zFilename, nByte, 0, 0); if( nByte == 0 ){ sqliteFree(zFilename); zFilename = 0; } return zFilename; } ---- Note that while original code with =CP_UTF8= works on Windows and SOME WinCE devices, this modified code works well and Windows and all WinCE devices I've tested so far. ---- _2007-Jul-18 16:01:21 by anonymous:_ {linebreak} Why not using the conversions from SQLite internals ? It can change a UTF-16 to UTF-8 and vice-versa. Or using UTF-16 variants in windows ce should be the best case. ---- _2007-Aug-09 20:47:04 by anonymous:_ Why not using the conversions from SQLite internals ? It can change a UTF-16 to UTF-8 and vice-versa. Or using UTF-16 variants in windows ce should be the best case. Not so simple. =unicodeToUtf8()= is used a lot internally regardless of what whether you use UTF-16 or UTF-8 yourself. For example, =unicodeToUtf8()= is used by =sqlite3WinTempFileName()= which is in turn used by =sqlite3PagerOpentemp()= -- I think you get the idea. ---- _2007-Dec-20 00:29:33 by anonymous:_ {linebreak} We've found that using CP_UTF8 fails on WinCE kernels that don't include SYSGEN_CORELOC (http://msdn2.microsoft.com/en-us/library/ms903883.aspx). To make the code handle any device it should be changed to: static WCHAR *utf8ToUnicode(const char *zFilename) { int nChar; WCHAR *zWideFilename; nChar = MultiByteToWideChar(CP_UTF8, 0, zFilename, -1, NULL, 0); if( nChar == 0 ) { nChar = MultiByteToWideChar(CP_ACP, 0, zFilename, -1, NULL, 0); if( nChar == 0 ) { DWORD dwError = GetLastError(); OSTRACE2("MultiByteToWideChar() failed, last error: %d\n", dwError); return 0; } } zWideFilename = sqliteMalloc( nChar*sizeof(zWideFilename[0]) ); if( zWideFilename==0 ) { return 0; } nChar = MultiByteToWideChar(CP_UTF8, 0, zFilename, -1, zWideFilename, nChar); if( nChar==0 ) { nChar = MultiByteToWideChar(CP_ACP, 0, zFilename, -1, zWideFilename, nChar); if( nChar==0 ) { sqliteFree(zWideFilename); zWideFilename = 0; } } return zWideFilename; } #e8e8bd 558 build active 2004 Jan anonymous 2007 Dec 4 4 Makefile.in should honor libdir and bindir Please support non-standard installation layouts by honoring configure's --libdir and --bindir flags rather than hard-coding $(exec_prefix)/lib and $(exec_prefix)/bin. (For instance, the layout we often use on Solaris has parallel "lib" and "lib64" directories under a common prefix.) _2007-Dec-18 17:29:26 by anonymous:_ {linebreak} Why is this ticket not solved? The patch is trivial and solves a real problem. Thank you. ---- _2007-Dec-18 17:54:46 by drh:_ {linebreak} The patch does not apply to the current makefile. And I do not understand what the -libdir or -bindir options are for or what they are suppose to do so I do not know how to fix it. #f2dcdc 368 code active 2003 Jun anonymous 2007 Dec 3 4 UPDATE trigger doesn't fire on INSERT OR REPLACE After executing the following SQL, there will be nothing in table T2. I expect to see '1' there: CREATE TABLE T1 ( id, name ); CREATE TABLE T2 ( id ); CREATE TRIGGER T1A AFTER UPDATE ON T1 BEGIN INSERT INTO T2 VALUES( new.id ); END; INSERT INTO T1 VALUES (1, 'Hi'); INSERT INTO T1 VALUES (2, 'There'); INSERT OR REPLACE INTO T1 VALUES (1,'Me'); An INSERT trigger *does* fire on INSERT OR REPLACE if the item exists already -- I would expect an UPDATE trigger. ---- _2004-Sep-21 17:15:37 by anonymous:_ {linebreak} Still repros in 3.0.7 :-( ---- _2007-Dec-17 21:45:03 by anonymous:_ {linebreak} I would say that ON DELETE and ON INSERT better describes what really happens (and not ON UPDATE), because if there would be another columns in the '1' row, their values would not be preserved after INSERT OR REPLACE takes place, as the documentation of the ON REPLACE algorithm states: "When a UNIQUE constraint violation occurs, the pre-existing rows that are causing the constraint violation are removed prior to inserting or updating the current row". However, neither ON UPDATE nor ON DELETE trigger occurs, which still is a bug. Thank you. #e8e8bd 2844 build active 2007 Dec anonymous 2007 Dec 4 1 lemon is being built without respecting LDFLAGS lemon is being built without respecting LDFLAGS. I'm attaching a patch which fixes this bug. In other words, why should we fix this? What problem is it causing? _2007-Dec-17 16:22:19 by drh:_ {linebreak} Why is this important? What LDFLAGS settings might a user want to carry through into lemon? ---- _2007-Dec-17 18:00:59 by anonymous:_ {linebreak} > Why is this important? It is considered to be be good practice to respect user's LDFLAGS. A user might want to have all executables and libraries built with identical LDFLAGS. > What LDFLAGS settings might a user want to carry through into lemon? A user might have LDFLAGS="-Wl,-O1,--hash-style=gnu,--sort-common" You can read http://lwn.net/Articles/192082/. Users can also use some other flags. > In other words, why should we fix this? What problem is it causing? It slightly increases the size of lemon executable and it slightly decreases performance. ---- _2007-Dec-17 18:04:31 by drh:_ {linebreak} lemon is used as an intermediate build tool in part of the SQLite build process. It is not a deliverable. If it runs a little slower or uses a little more memory, nobody cares. We only care if it gets the wrong answer. Is it ever possible that the lack of LDFLAGS support might result in lemon getting the wrong answer? ---- _2007-Dec-17 18:27:33 by anonymous:_ {linebreak} Can you comment on Lemon bug in #2835? It produces 2 different sqlite3.c files depending on your malloc implementation. ---- _2007-Dec-17 19:19:01 by anonymous:_ {linebreak} > lemon is used as an intermediate build tool in part of the > SQLite build process. It is not a deliverable. If it runs a > little slower or uses a little more memory, nobody cares. CFLAGS are respected when lemon is being built, so for consistency LDFLAGS also should be respected. (The comment above was not created by me.) #f2dcdc 2842 code active 2007 Dec anonymous 2007 Dec 1 1 .import does not recongnise NULL values .import function fails to see NULL values in csv files as NULL values...instead they are treated as the string "NULL". This is with .mode list and separator , But behaves similarly for .mode csv Also if one outputs a table with NULL values to a file, then re-imports that file, again .import does not recognise the values as NULL, but as "NULL". Everything here also applies to empty strings in files, e.g. instead of "NULL" using nothing... This is a showstopper for us since we want to import a large amount of data with many tables containing NULL values. I can't see any valid reason for .import not to recognise the same syntax as the command line. Note that something like: sqlite3 my.db insert into MY_TABLE values (1,"foo","bar",NULL) ..works fine. It is just .import that appears to be broken. _2007-Dec-14 16:39:51 by rdc:_ {linebreak} .import only inserts string values into database tables. If your column has a declared type that changes the columns affinity to numeric or integer, then those strings will be converted to numeric values by the SQLIte library. The workaround is to simply insert a unique string where ever you want a NULL value, and then run an update that replaces those strings with real NULL values. If you inserted the string 'NULL' then do this after the .import update t set field = null where field = 'NULL'; You will have to repeat this for each field in your table that might contain the 'NULL' string. #e8e8bd 2841 todo active 2007 Dec anonymous 2007 Dec 1 1 The sqlite mailing list has become overrun by trolls The sqlite mailing list is very useful. The S/N is at times a little high but nonetheless quite manageable. Recently (see the DeviceSQL thread) it got really bad. Would moderation be unacceptable during these periods of time where people feel the need to protect their ego's? The sqlite mailing list is primarily about sqlite (well, and lemon), not a marketing vector for other products? Surely they have their own lists and resources for that? #f2dcdc 2721 code active 2007 Oct anonymous 2007 Dec 2 1 if db file is in a folder with non-ansi character some functions fail If database file is located in directory with some non-ANSI characters (in my case with a Russian subdirectory c:\Мои документы\Data_Jobs), or it's name is non-ansi. Some functions fail to execute sql. For example (with defined UNICODE):TCHAR sql[512]; _stprintf(sql, _T("INSERT INTO tab_SurveyedPoints (name, comment, code,") _T("coordinatetype, b, l, h, solutiontype, sigmah, sigmav)") _T(" VALUES ('%s','%s','%s',0,%lf,%lf,%lf,0,%lf,%lf);"), point.m_name.c_str(), point.m_description.c_str(), point.m_code.c_str(), point.m_coordinates.b, point.m_coordinates.l, point.m_coordinates.h, point.m_sigmah, point.m_sigmav); int rc1 = sqlite3_prepare16(m_db, sqlfmt, -1, &stmt, (const void**)&pszTail); rc != SQLITE_OKBut if I move the file to c:\My documents\Data_Jobs this works ok. It's improbable behaviour, but I can't work around yet. Although, prepare() functions work ok as well in both cases. Yuri Noyanov. _2007-Oct-11 19:33:34 by drh:_ {linebreak} All string arguments to SQLite, and especially filename arguments, must be UTF-8 or UTF-16 (depending on the function). If you use string parameters which are not UTF-8 or UTF-16 (as appropriate) then the behavior of SQLite is undefined and probably not what you want. ---- _2007-Oct-12 04:25:56 by anonymous:_ {linebreak} but ALL programs to handle SQLite DBs (SQLIteBrowser, SQLite Control) fail to handle the files as well. Till I move the file to different directory !!! ---- _2007-Oct-12 04:27:54 by anonymous:_ {linebreak} Also I must note, that I CAN open the database, I CAN execute some SQLs with sqlite_prepare function OK. But sqlite_prepare16 FAILS if I just rename my database !!! ---- _2007-Oct-12 04:31:46 by anonymous:_ {linebreak} Also note to make my issue clearer: sqlite_prepare16() with the same code either works OK either doesn't work. depends on database filename or folder path. The database is opened OK in both cases (I used utf8 conversion). sql_prepare() works ok in both cases. ---- _2007-Oct-13 06:37:43 by anonymous:_ {linebreak} That appears to be only with INSERT sql statement. Both SELECT and UPDATE work fine with sqlite_prepare16. #e8e8bd 2831 new active 2007 Dec anonymous 2007 Dec 3 4 alter view View can't be used after ALTER RENAME TO:SQLite version 3.5.3 Enter ".help" for istructions sqlite> create table t(a); sqlite> create view v1 as select * from t; sqlite> alter table v1 rename to v2; sqlite> select * from v2; SQL error: no such table: v2 sqlite> select * from v1; SQL error: no such table: v1 sqlite> .schema CREATE TABLE t(a); CREATE VIEW v1 as select * from t; sqlite> select * from sqlite_master; table|t|t|2|CREATE TABLE t(a) view|v1|v1|0|CREATE VIEW v1 as select * from tThis is a feature request, not a bug. ---- _2007-Dec-11 18:40:17 by anonymous:_ {linebreak} Notice that alter table doesn't return an error. After the command neither v1 nor v2 can be used. ---- _2007-Dec-13 08:18:16 by danielk1977:_ {linebreak} [4623] improves the situation by returning an error when the user attempts to rename a view. One reason this feature (renaming views) is not a high priority is because a view can be dropped and recreated with a different name efficiently. This was not the case with tables. #f2dcdc 2825 code active 2007 Dec anonymous 2007 Dec 3 3 FormatMessage (win32) should use extra flag and convert from Unicode The call to FormatMessageA in the win32 source code needs to have the flags changed from: FORMAT_MESSAGE_FROM_SYSTEM to FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS This ensures that any system messages that expect arguments do not try to grab the argument from some random memory location. ref: http://blogs.msdn.com/oldnewthing/archive/2007/11/28/6564257.aspx _2007-Dec-06 14:07:53 by anonymous:_ {linebreak} I also noticed that the result is NOT converted to UTF-8. FormatMessageA returns the text in the local ANSI codepage. FormatMessageW should be used on NT systems, and either result should be converted to the SQLite UTF-8 default. ---- _2007-Dec-11 00:34:37 by anonymous:_ {linebreak} to simplify what is meant even more... http://www.sqlite.org/cvstrac/fileview?f=sqlite/src/os_win.c&v=1.118 Search for FormatMessageA (only 1 instance) - FORMAT_MESSAGE_FROM_SYSTEM, + FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS, No breakage, ensures that no crashes with some messages (e.g. filesystem errors). The encoding issue should be addressed separately. ---- _2007-Dec-11 01:27:07 by anonymous:_ {linebreak} The function should be changed to the following to correctly handle the conversion from Unicode/MBCS.static void winDlError(sqlite3_vfs *pVfs, int nBuf, char *zBufOut){ int error = GetLastError(); #if OS_WINCE if( error>0x7FFFFFF ){ sqlite3_snprintf(nBuf, zBufOut, "OsError 0x%x", error); }else{ sqlite3_snprintf(nBuf, zBufOut, "OsError %d", error); } #else if( isNT() ){ LPWSTR zWinTemp = NULL; DWORD dwLen = FormatMessageW( FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS, NULL, error, 0, (LPWSTR) &zWinTemp, 0, 0 ); if (dwLen > 0) { char * zOut = unicodeToUtf8(zWinTemp); LocalFree(zWinTemp); sqlite3_snprintf(nBuf, zBufOut, "%s", zOut); free(zOut); } }else{ LPSTR zWinTemp = NULL; DWORD dwLen = FormatMessageA( FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS, NULL, error, 0, (LPSTR) &zWinTemp, 0, 0 ); if (dwLen > 0) { char * zOut = mbcsToUtf8(zWinTemp); LocalFree(zWinTemp); sqlite3_snprintf(nBuf, zBufOut, "%s", zOut); free(zOut); } } #endif }#e8e8bd 2821 new active 2007 Dec anonymous 2007 Dec 3 4 hashtable indicies It would be nice to implement non btree indices. I.e. CREATE INDEX ON table(rowid) AS HASH. Using a hashtable's O(1) properties, you could use the index for very quick lookups when one result is expected. This does have the tradeoff that a hashtable index has no ordering properties (can not be used for sorts or non-equality searching). However, it would be a *huge* win when you have 250,000 rowids in memory, and you want to go fetch another column in the database for each one of those rowids (SELECT * FROM table WHERE rowid=?). _2007-Dec-03 21:58:01 by anonymous:_ {linebreak} For 250,000 rows I doubt you would see that much of an improvement (try it.) You'll almost certainly find log_n is going to be fairly fast (especially for large n.) I personally would prefer some sort of 'virtual' index though, that could be a hash or actually from a user-supplied function so that I can index large blobs by some function (i.e. a hash). And yes, this would be an incompatible file-format change and it's not clear how to update an index when the function isn't loaded (i.e. db reopened with that function.) Perhaps mark the index as 'stale' and ignore it until the function loads then you can do the updates. Of course this starts to get quite complicated. ---- _2007-Dec-03 22:12:17 by anonymous:_ {linebreak} Everything in sqlite depends on btree indexes. You're talking a major rewrite if you support hash-based or other indexing. #f2dcdc 2814 code active 2007 Nov anonymous 2007 Dec 3 3 _XOPEN_SOURCE again Ideally setting _XOPEN_SOURCE should be an opt-in detected by configure, rather than a hardcoded opt-out as it is now. I find you create more problems in setting it than just leaving it out on modern platforms. Can you please give users the option of not defining _XOPEN_SOURCE at all?+#ifndef SQLITE_DONT_DEFINE_XOPEN_SOURCE #if !defined(_XOPEN_SOURCE) && !defined(__DARWIN__) && SQLITE_THREADSAFE # define _XOPEN_SOURCE 500 /* Needed to enable pthread recursive mutexes */ #endif +#endif_2007-Dec-01 09:23:15 by anonymous:_ {linebreak} Also when using Python, it sets _XOPEN_SOURCE to 600. No idea what the 500 vs 600 difference is about. ---- _2007-Dec-01 15:58:28 by anonymous:_ {linebreak} I've used a couple of different Linux OSes and _XOPEN_SOURCE is not needed. Maybe it's for OSes more than 5 years old. Recursive mutexes are pretty much standard these days since the popularity of Java which uses them extensively. ---- _2007-Dec-01 17:21:05 by drh:_ {linebreak} See also tickets #2673, #2681, and #2741. ---- _2007-Dec-02 02:08:26 by anonymous:_ {linebreak} On Linux, PTHREAD_MUTEX_RECURSIVE is the same as PTHREAD_MUTEX_RECURSIVE_NP: PTHREAD_MUTEX_RECURSIVE = PTHREAD_MUTEX_RECURSIVE_NP, Since PTHREAD_MUTEX_RECURSIVE_NP is always available, you could avoid defining _XOPEN_SOURCE and use this code instead:- pthread_mutexattr_settype(&recursiveAttr, PTHREAD_MUTEX_RECURSIVE); + pthread_mutexattr_settype(&recursiveAttr, +#ifdef linux + PTHREAD_MUTEX_RECURSIVE_NP +#else + PTHREAD_MUTEX_RECURSIVE +#endif + );---- _2007-Dec-02 02:17:22 by anonymous:_ {linebreak} A quick google search reveals how various projects deal with this recursive mutex declaration problem (in no particular order): *: #define _XOPEN_SOURCE 500 and use PTHREAD_MUTEX_RECURSIVE *: #define _XOPEN_SOURCE 600 and use PTHREAD_MUTEX_RECURSIVE *: #define _GNU_SOURCE and use PTHREAD_MUTEX_RECURSIVE *: don't define anything and use PTHREAD_MUTEX_RECURSIVE_NP on linux, and PTHREAD_MUTEX_RECURSIVE elsewhere. Unfortunately, since PTHREAD_MUTEX_RECURSIVE is an enum on Linux, so you can't use the #ifdef PTHREAD_MUTEX_RECURSIVE compile-time technique. #e8e8bd 2813 build active 2007 Nov anonymous 2007 Nov 1 1 compile error on Windows CE environment: visual c++ 2005 window ce 6.0 customize sdk sqlite-amalgamation-3_5_3 I get error: Error 27 error C2040: 'localtime' : 'tm *(const time_t *)' differs in levels of indirection from 'int ()' d:\SubProjects\Sqlite\sqlite3.c 18574 but if I add code in line 7095: struct tm *__cdecl localtime(const time_t *t); then Success! #f2dcdc 2809 code active 2007 Nov anonymous 2007 Nov 1 1 PRAGMA collation_list shows unregistered collations As presented on the mailing list: Imagine that a SQLite3 database opened in a custom application with a registered a collation sequence named "unknown" has created the following table: CREATE TABLE a (b COLLATE unknown); Now open this table in the default SQLite3 CLI. Up to here, everything works as expected. Next issue "PRAGMA collation_list;" and notice that "unknown" lists next to the other registered collations, even though "unknown" is not registered with the default SQLite3 CLI: sqlite> PRAGMA collation_list; 0|unknown 1|NOCASE 2|BINARY Responses from the mailing list indicate that this is not the expected behaviour. "PRAGMA collation_list;" should list registered collations only. _2007-Nov-28 16:12:17 by anonymous:_ {linebreak} I don't think this is a bug. If the CLI is not aware of the collation, it should not process the query that makes use of the collation because it would certainly be wrong if it simply ignored the collation. This is not unlike a user-registered SQL function that does not exist in the CLI. I would not expect or want the sqlite3 CLI to ignore the unknown function, nor would I want the CLI to process queries ignoring the custom collation. #f2dcdc 2810 code active 2007 Nov anonymous 2007 Nov 1 1 Unregistered collation problems with simple subselects As discussed on the mailing-list: Imagine that a SQLite3 database opened in a custom application with a registered a collation sequence named "unknown" has created the following table: CREATE TABLE a (b COLLATE unknown); Now open this table in the default SQLite3 CLI. Up to here, everything works as expected. Simple queries like "SELECT * FROM a;" work fine. But subselects, in their most basic form and with no sorting or comparisons, result in an error: sqlite> INSERT INTO a VALUES ('one'); sqlite> SELECT * FROM a, (SELECT * FROM a); SQL error: no such collation sequence: unknown sqlite> SELECT * FROM (SELECT * FROM a); SQL error: no such collation sequence: unknown sqlite> SELECT *, * FROM a; one|one This is surprising because the collation sequence should not matter to the queries. In fact, the union without the subselect works just fine and without errors. To demonstrate, here is the explain output of a table with a registered collation sequence. No mention of the collation name here: sqlite> CREATE TABLE b (b collate nocase); sqlite> EXPLAIN SELECT * FROM b, (SELECT * FROM b); 0|Goto|0|17| 1|Integer|0|0| 2|OpenRead|0|3| 3|SetNumColumns|0|1| 4|Integer|0|0| 5|OpenRead|2|3| 6|SetNumColumns|2|1| 7|Rewind|0|14| 8|Rewind|2|13| 9|Column|0|0| 10|Column|2|0| 11|Callback|2|0| 12|Next|2|9| 13|Next|0|8| 14|Close|0|0| 15|Close|2|0| 16|Halt|0|0| 17|Transaction|0|0| 18|VerifyCookie|0|4| 19|TableLock|0|3|b 20|Goto|0|1| 21|Noop|0|0| #e8e8bd 2808 doc active 2007 Nov anonymous 2007 Nov 4 2 Documentation GIF images take up too much space The GIF format used for the documentation images takes up too much space. I believe that this is in strict contrast to the "small" feature of SQLite. Converting the GIFs to PNG images saves up to 181 KB (64%) of the documentation storage space: Documentation images as is (mostly GIF): 381 KB == 100% Documentation images as PNG: 292 KB == 77% Documentation images as PNG 16 colors: 137 KB == 36% 16 colors are more than plenty -- the diagrams show no visibility degradation. You might even cut it down to 8 colors to save even more ... #f2dcdc 2223 code active 2007 Feb scouten 2007 Nov 3 3 pragma auto_vacuum doesn't survive .dump & reconstitute When you run sqlite3 path/to/database .dump, it does not contain pragma auto_vacuum even if that option was chosen when creating the source database. _2007-Feb-08 18:13:27 by scouten:_ {linebreak} We wonder if other pragmas are also not being propogated. ---- _2007-Feb-08 18:53:42 by anonymous:_ {linebreak} No pragmas are output from .dump. SQLite should have a .dump_with_pragmas command or equivalent. ---- _2007-Nov-27 02:11:05 by anonymous:_ {linebreak} auto_vacuum is especially important since you need to specify it *before* loading the tables in the dump; if you notice that it's missing after loading a dump, it's too late. I'd also appreciate user_version surviving. Would patches be welcome for these? I would be happy to contribute one: mail glasser@davidglasser.net. #f2dcdc 2793 code active 2007 Nov anonymous 2007 Nov 3 3 fts3 lacks scoping It would be nice if the fts3 symbols could optionally be made private/static as the rest of the sqlite3 library. Not sure why sqlite3_api becomes public when used with the amalgamation, for that matter. make TOP=`pwd` BCC=gcc TCC=gcc AR=ar RANLIB=echo NAWK=gawk -f \ main.mk sqlite3.h sqlite3.c fts3amal.c cat fts3amal.c >> sqlite3.c gcc -DSQLITE_THREADSAFE -DSQLITE_API=static -DSQLITE_PRIVATE=static \ -DSQLITE_EXTERN=static -DSQLITE_ENABLE_FTS3 -c sqlite3.c nm sqlite3.o | grep -v ' [trUbd] ' 00000004 C sqlite3_api 00064da2 T sqlite3Fts3HashClear 000652a4 T sqlite3Fts3HashFind 00064d60 T sqlite3Fts3HashInit 0006533b T sqlite3Fts3HashInsert 00064b4c T sqlite3Fts3Init 00066b34 T sqlite3Fts3InitHashTable 000669bd T sqlite3Fts3PorterTokenizerModule 0006702d T sqlite3Fts3SimpleTokenizerModule #f2dcdc 2791 code active 2007 Nov anonymous 2007 Nov 1 1 Allow building FTS[123] as part of sqlite library with configure See attached patch. #e8e8bd 2787 build active 2007 Nov anonymous 2007 Nov 4 5 sqlite3.pc is not remade the subject says it all, there is no make rule to rebuild the sqlite3.pc from the .in file. it is only possible by hand (./config.status sqlite3.pc) _2007-Nov-22 12:17:50 by drh:_ {linebreak} I don't know what the sqlite3.pc file does. I certainly do not use it for any of my builds on Linux, Mac OSX, or windows. Why should I leave it in the source tree? Isn't the best solution to this problem to simply delete the file? ---- _2007-Nov-22 17:14:27 by anonymous:_ {linebreak} It is indirectly used by pkgconfig. Here's some info on pkgconfig: http://pkg-config.freedesktop.org/wiki/ ---- _2007-Nov-23 15:22:32 by drh:_ {linebreak} Could somebody who understands what the sqlite3.pc file is used for suggest a makefile rule for rebuilding it? #f2dcdc 2770 code active 2007 Nov anonymous 2007 Nov 1 1 Problem with BLOB in 3.5.x ? After I've switched from 3.3.18 to 3.5.2, selecting from table which contains BLOB LONGER THAN ABOUT 990 BYTES returns error "SQL logic error or missing database" after call to _sqlite3_step(). I'm using preprocessed sources downloaded from here. DEBUG build of preprocessed sources works correctly, problem is only in RELEASE build. I'm using VC6.0 to compile. Any idea what could be wrong? Thank you! Can you try to reproduce this with the sqlite shell tool? Thanks. Large blobs work for me with both release and debug builds (not msvc though, gcc/linux). ---- _2007-Nov-12 18:41:37 by anonymous:_ {linebreak} sqlite3.exe provided here works with the database. Problem is only with release build (static library linked into test application). Here is test app which exits with "Error 1" in release build: int main(int argc, char* argv[]) { int rc; sqlite3* db; sqlite3_stmt* stmt; rc = sqlite3_open("n2.db3", &db); rc = sqlite3_prepare(db, "CREATE TABLE [ttt] ([bbb] BLOB)", -1, &stmt, 0 ); rc = sqlite3_step(stmt); rc = sqlite3_reset(stmt); char text[10000],query[20000]; strnset(text,'a',sizeof(text)-1); sprintf(query,"insert into [ttt] values (?)"); rc = sqlite3_prepare(db, query, -1, &stmt, 0 ); rc = sqlite3_bind_blob(stmt,1,text,sizeof(text), SQLITE_TRANSIENT); rc = sqlite3_step(stmt); rc = sqlite3_reset(stmt); rc = sqlite3_prepare(db, "select * from ttt", -1, &stmt, 0 ); rc = sqlite3_step(stmt); if (rc == SQLITE_ROW) { printf("%s: OK",sqlite3_column_text(stmt,1)); } else if (rc == SQLITE_DONE) { printf("DONE"); } else { printf("Error %d",rc); } return 0; } ---- _2007-Nov-12 18:56:24 by drh:_ {linebreak} You should be using sqlite3_finalize() instead of sqlite3_reset(). You are leaking memory. Also, you should use sqlite3_prepare_v2() to avoid problems with changing schemas. But even without those fixes, I cannot reproduce the problem on Linux. ---- _2007-Nov-12 19:39:31 by anonymous:_ {linebreak} Suggested fixes didn't help. I've tried to debug it. It fails in btree.c, line 3056: if( offset+amt > nKey+pCur->info.nData ){ /* Trying to read or write past the end of the data is an error */ return SQLITE_ERROR; } there seems to be different values in release mode. My debugger does not show values of variables in release mode, so I can be wrong, but it seems in release offset is 5 and in debug it is 4. There can be something wrong with compilation, I'll try to figure this out tomorrow. BTW compilation of static libraty in VC6.0 gives 185 warnings. I don't know if it is ok, it haven't caused problems in older sqlite ---- _2007-Nov-13 08:47:28 by anonymous:_ {linebreak} I've turned off "Maximize Speed" option - this is causing the problem. No optimizations and optimize for size seems to be working. But it still makes me nervous :(( I really don't need corrupted database and now I hope it won't slow down too much. Unfortunately old library does not implement replace function so I don't want to switch back. This could be warning to others, I'm using VC++ 6.0 SP 6. Thank you for your time. ---- _2007-Nov-22 17:20:31 by anonymous:_ {linebreak} I have exactly the same problem here (win XP, vc6 SP2) when I link against my sqlite static or dynamic library in release. I have also used boundschecker to check sqlite, and it detects many dangling pointers ! But the strange thing is that I cannot find why these pointers are dangling, here an example: In prepare.c@188 pTab = sqlite3FindTable(db, zMasterName, db->aDb[iDb].zName); Boundchecker say that zMasterName is a dangling pointer, previously released here: in build.c@711: void sqlite3StartTable( Parse *pParse, /* Parser context */ Token *pName1, /* First part of the name of the table or view */ Token *pName2, /* Second part of the name of the table or view */ int isTemp, /* True if this is a TEMP table */ int isView, /* True if this is a VIEW */ int isVirtual, /* True if this is a VIRTUAL table */ int noErr /* Do nothing if table already exists */ ){ } It does not make sens for me, maybe it a false positive from boundchecker, but it is weird. I don't know if these "errors" are related to the "blob" bug in release mode. I will try to debug these error with some "printf" in release mode. Note: The provided dll (the one from the sqlite site) does not have this "bug". ---- _2007-Nov-22 18:27:35 by anonymous:_ {linebreak} More info: It seems that there is a bug in the VC6 (SP6) compiler. In btree.c, line 3056: if( offset+amt > nKey+pCur->info.nData ){ /* Trying to read or write past the end of the data is an error */ return SQLITE_ERROR; } After adding some printf around, It seems that the "speed optimization" compilation flag of VC6 changes the code order in a way that the offset variable is miss incremented !! Two remarks: *: I've traced the calling function, sqlite3BtreeData, and the it call accessPayload with the good offset value *: VC6 produces an internal error: "fatal error C1001: INTERNAL COMPILER ERROR" in the accessPayload function, if I try to access the offset value before this line: aPayload = pCur->info.pCell + pCur->info.nHeader; A dirty workaround could be to change the code order or the local var usage. I'm trying .... #e8e8bd 2776 warn active 2007 Nov anonymous 2007 Nov 5 1 mailing list sqlite-users-digest doesn't work. It remembers registered addresses but doesn't send any emails. #f2dcdc 2771 code active 2007 Nov anonymous 2007 Nov 4 4 Lemon: Generated parser needs stdlib.h (not in default template) I tested a simple do-nothing parser just to get lemon output, and this doesn't compile (if warnings treated as errors) because of non declaration of the =memset()= function for the following statment: =memset(&yygotominor, 0, sizeof(yygotominor));= (added for the resolution of SQLite ticket #2172). The lempar.c just include =stdio.h=, it would suffice to add =stdlib.h= to get the =memset()= declaration (even if all real parsers must include =stdlib.h= to get something really working). _2007-Nov-14 21:02:25 by anonymous:_ {linebreak} Sorry, the needed header is =string.h=, not =stdlib.h= :-) #f2dcdc 2766 code active 2007 Nov drh 2007 Nov 1 1 TCL transaction started from within a query does not commit This is a problem with the TCL interface. Consider the following TCL script: file delete -force test.db test.db-journal sqlite3 db test.db db eval { CREATE TABLE t1(x,y); INSERT INTO t1 VALUES(1,2); CREATE TABLE t2(a,b); INSERT INTO t2 VALUES(8,9); } db eval {SELECT * FROM t1} { db transaction { db eval {UPDATE t2 SET a=a*2} } } The [db transaction] statement starts a transaction and it is suppose to commit the tranaction at the end of the code block. But because the transaction started while a query was active, the tranaction is unable to commit. The TCL interface never commits the tranaction nor does it give any kind of error indication. It is unclear if an error should be returned or if the commit should be deferred until outer query finishes. If the code within the [db transaction] block throws an error, we really need the transaction to rollback right away. Perhaps there should be a new API that cancels all pending queries. Perhaps a call to sqlite3_interrupt() would suffice for this. Need to investigate further.... #e8e8bd 2417 new active 2007 Jun anonymous 2007 Nov drh 3 3 Idea for read write concurrency. This is not a problem, but rather an idea on how to resolve the reader/writer concurrency issues encountered in sqlite. The idea is to allow a reader and writer to work concurrently not blocking each other. Dual writers would of course block. When a write occurs: 1. block level changes are made to the database file. 2. Pre-image of that change is written to the journal. Readers: 1. File I/O on the main file would occur normally. 2. If the block encountered is "new" ie one that was written out by the writer. Then get the original block from the Journal file. In order to determine "NEW" a change number could be put on each block. When a READ (select) begins it would first determine the starting global change number. (maybe on the master block?) When a write occurs it would read the Master blocks change number. (increment this in memory) and use write new blocks with the new value. At commit. The Master block would be updated and the txn journal marked for purge if there are pending reads. -- Drawbacks: Reading becomes dependent upon the txn journal. -- Implementation of BLOCK level versioning may ultimately be a simpler approach. Idea would be for a seperate file conaining versioned blocks. This file could be accessed instead of the txn journal. _2007-Nov-08 15:12:00 by anonymous:_ {linebreak} DRH: Also unaddressed in the proposal is how to locate a particular page within the journal file without having to do (performance killing) sequential scan of the possible very large file. Resolution of page access to avoid sequential scans of Txn Journal. When a writer is making the modification to a page first it writes the original page to the journal. At this point the journal file offset location is known. Save this offset in the "NEW" page being written into the database file. This implements a backwards chaining of pages into the txn journal. The reader upon reading the db file page would recognize (see above) that the page is dirty. Acquire the txn journal offset from the dirty page, Read the page from the journal until the starting page is found. This would eliminate any sequential scanning, but may require more than one read request. #e8e8bd 2763 build active 2007 Nov anonymous 2007 Nov 4 4 AIX build failure due to explicit _XOPEN_SOURCE definition Our AIX machine has started having problems building the new sqlite 3.5.2 source. I'm no AIX expert, but I did some digging and I think it's due to sqliteInt.h defining _XOPEN_SOURCE without also defining _ALL_SOURCE. AIX system header files usually define this themselves (in /usr/include/ standards.h) but only if _XOPEN_SOURCE was previously undefined. Has anyone else come across this bug? I'm willing to believe it's our build system if not; we can work around it by setting _ALL_SOURCE on the command-line, but I thought it might be useful to raise a bug anyway. See also tickets #2673, #2681, and #2741. I do not have access to an AIX machine and so have no ability to debug this problem. If you have suggested patches we will consider them. Otherwise, there is not much we can do about this ticket. #e8e8bd 2760 new active 2007 Nov anonymous 2007 Nov 5 4 request: sqlite3_unlink() to delete db files. Hi! Today i came across a use case where i would like client code to be able to delete an underlying sqlite3 db, but that code doesn't have immediate access to the file name of that db (without refactoring the db wrapper code). An interesting feature addition would, IMO, be: int sqlite3_unlink( sqlite3 * db, bool closeTheFile ); Unlinks the file associated with the given database. It does not alter the database in any way (thus is it a no-op on a :memory: database). The closeTheFile flag specifies whether the file handle associated with db should also be closed (and thus db must also be closed), or just unlinked (e.g., as temporary databases are unlinked right after creation but kept open). After browsing through the VFS API a bit, i see that there is an xDelete function, but i'm not sure if its semantics require that the underlying file handle be closed. i don't see an extra xClose member of VFS, so i assume that xDelete also handles closing the file handle. If these were split into two features, sqlite3_unlink() could be implemented very easily. :) #e8e8bd 2756 new active 2007 Nov anonymous 2007 Nov 1 1 allow vacuum to change pragma setting instead of using existing ones we've got databases created with page_size of 1k, and we'd like to change that setting to 4k. vacuum creates a temporary db, attach it to the current connection, creates the tables (based on what's in the old db), and then selects from the old db and inserts into the new one. vacuum does exactly what we need (creating a new db from an old one), but it re-uses the existing pragmas for page size, auto vacuum and reserved page size. from sqlite.c, see sqlite3RunVacuum() sqlite3BtreeSetPageSize(pTemp, sqlite3BtreeGetPageSize(pMain), sqlite3BtreeGetReserve(pMain)); dr hipp points out that the the operands to vacuum are unused. from sqlite.c: sqlite3VdbeAddOp(v, OP_Vacuum, 0, 0); one solution would be to allow the user to specify the page size, reserve page size, and autovacuum as optional params to vacuum. he had an idea of using the signedness of the first operand to represent the autovacuum setting (since after a table is created, you can change the setting from auto to incremental, but you can't change it from none to auto (or none to incremental) #f2dcdc 2755 code active 2007 Nov anonymous 2007 Nov 3 3 trace interfere with transaction Tcl interface When using the transaction method of the Tcl interface to the SQLite with a registered "trace" function, the stack trace is lost in case an error occurs inside the transaction. As an example I provide two outputs, the first one without a registered trace function and the second one with one (in which it *cannot* be seen where the exception cames from): ========= First: > ./a.tcl vorher BUMMM while executing "a" invoked from within "db transaction { puts "vorher" a puts "nachher" }" ("uplevel" body line 1) invoked from within "uplevel 1 [list db transaction { puts "vorher" a puts "nachher" }]" (procedure "b" line 2) invoked from within "b" (file "./a.tcl" line 28) ========= Second: > ./a.tcl BEGIN vorher ROLLBACK while executing "db transaction { puts "vorher" a puts "nachher" }" ("uplevel" body line 1) invoked from within "uplevel 1 [list db transaction { puts "vorher" a puts "nachher" }]" (procedure "b" line 2) invoked from within "b" (file "./a.tcl" line 28) ******** A scritp that demostrates this behaviour is attached. The only workaround is not to trace. Thanks #f2dcdc 2753 code active 2007 Nov anonymous 2007 Nov drh 3 3 Master journal files sometimes not deleted In the 3.4.1 amalgamation, in vdbeCommit, the master journal file is created, and deleted at the end or if there is an error. But it looks like there is one case where it gets closed but not deleted. The code is: for(i=0; rc==SQLITE_OK && inDb; i++){ Btree *pBt = db->aDb[i].pBt; if( pBt && sqlite3BtreeIsInTrans(pBt) ){ rc = sqlite3BtreeCommitPhaseOne(pBt, zMaster); } } sqlite3OsClose(&master); if( rc!=SQLITE_OK ){ sqliteFree(zMaster); return rc; } It seems like that last bit should be: if( rc!=SQLITE_OK ){ sqlite3OsDelete(zMaster); sqliteFree(zMaster); return rc; } _2007-Nov-01 19:26:28 by anonymous:_ {linebreak} Should have read the comment above that code segment closer. Looks like this was by design. Trying to figure out what to do with the client that has hundreds of orphaned master journal files... #e8e8bd 2716 new active 2007 Oct anonymous 2007 Oct 5 1 Create Clear Command I want a command caled "clear" like in MySQL. This command should erase the screen and then put the sqlite pointer on top of the screen _2007-Oct-11 07:41:45 by anonymous:_ {linebreak} How about a cookie instead? ---- _2007-Oct-30 08:02:04 by anonymous:_ {linebreak} Clearing the screen and moving the cursor are platform-dependent operations. On Unix they are not only platform-dependent, but also terminal-dependent. Thus such a feature does not really belong in the cross-platform and minimalistic sqlite3 shell (in my opinion). #e8e8bd 2749 warn active 2007 Oct anonymous 2007 Oct 3 4 SQLITE_OMIT_FLOATING_POINT, int constants too large Hiya! i'm working from the 3.5.1 (CVS version) of the amalgamation. The code was pulled from CVS sometime around Oct 27th 2007. Platform: Kubuntu Linux 7.04, i386-32, gcc 4.1.2. The compiler output says it all: gcc -c -DSQLITE_OMIT_FLOATING_POINT sqlite3.c sqlite3.c: In function 'bestVirtualIndex': sqlite3.c:65531: warning: integer constant is too large for 'long' type sqlite3.c: In function 'bestIndex': sqlite3.c:65600: warning: integer constant is too large for 'long' type sqlite3.c: In function 'sqlite3WhereBegin': sqlite3.c:66223: warning: integer constant is too large for 'long' type sqlite3.c:66248: warning: integer constant is too large for 'long' type sqlite3.c:66254: warning: integer constant is too large for 'long' type These warnings would seem to indicate potentially serious problems, though i admittedly have not investigated whether overflows are really fatal in the affected contexts. #e8e8bd 2748 warn active 2007 Oct anonymous 2007 Oct 4 4 amalgamation: SQLITE_OMIT_ALTERTABLE warnings Hiya! i'm working from the 3.5.1 (CVS version) of the amalgamation. The code was pulled from CVS sometime around Oct 27th 2007. The compiler output says it all: gcc -c -DSQLITE_OMIT_ALTERTABLE sqlite3.c sqlite3.c:6909: warning: 'sqlite3AlterRenameTable' used but never defined sqlite3.c:6916: warning: 'sqlite3AlterFinishAddColumn' used but never defined sqlite3.c:6917: warning: 'sqlite3AlterBeginAddColumn' used but never defined The relevant code is: sed -ne '6909p;6916p;6917p' sqlite3.c SQLITE_PRIVATE void sqlite3AlterRenameTable(Parse*, SrcList*, Token*); SQLITE_PRIVATE void sqlite3AlterFinishAddColumn(Parse *, Token *); SQLITE_PRIVATE void sqlite3AlterBeginAddColumn(Parse *, SrcList *); Platform: Kubuntu Linux 7.04, i386, gcc 4.1.2. #f2dcdc 2392 code active 2007 May anonymous 2007 Oct 4 4 Reduce MemPage and PgHdr struct sizes. Better page memory utilization. Patch to reduce sizeof(MemPage) below. It saves 8 bytes per cached page or in-memory page on Linux. sizeof(MemPage) on Linux: original: 84 patched: 76 Patched "make test" runs without regressions on Linux and Windows. Timings for "make test" (elapsed): original: 1:20.74 patched: 1:20.22 Size of sqlite3.o when compiled from almalogmation with all sqlite features enabled with gcc flags -O3 -fomit-frame-pointer: original: 586976 bytes patched: 587880 bytes Patched sqlite3.o is 904 bytes larger. Index: src/btreeInt.h =================================================================== RCS file: /sqlite/sqlite/src/btreeInt.h,v retrieving revision 1.4 diff -u -3 -p -r1.4 btreeInt.h --- src/btreeInt.h 16 May 2007 17:28:43 -0000 1.4 +++ src/btreeInt.h 30 May 2007 16:26:03 -0000 @@ -269,15 +269,15 @@ typedef struct BtLock BtLock; */ struct MemPage { u8 isInit; /* True if previously initialized. MUST BE FIRST! */ - u8 idxShift; /* True if Cell indices have changed */ u8 nOverflow; /* Number of overflow cell bodies in aCell[] */ - u8 intKey; /* True if intkey flag is set */ - u8 leaf; /* True if leaf flag is set */ - u8 zeroData; /* True if table stores keys only */ - u8 leafData; /* True if tables stores data on leaves only */ - u8 hasData; /* True if this page stores data */ - u8 hdrOffset; /* 100 for page 1. 0 otherwise */ - u8 childPtrSize; /* 0 if leaf==1. 4 if leaf==0 */ + u8 hdrOffset:7; /* 100 for page 1. 0 otherwise */ + u8 zeroData:1; /* True if table stores keys only */ + u8 childPtrSize:3; /* 0 if leaf==1. 4 if leaf==0 */ + u8 leaf:1; /* True if leaf flag is set */ + u8 idxShift:1; /* True if Cell indices have changed */ + u8 intKey:1; /* True if intkey flag is set */ + u8 leafData:1; /* True if tables stores data on leaves only */ + u8 hasData:1; /* True if this page stores data */ u16 maxLocal; /* Copy of Btree.maxLocal or Btree.maxLeaf */ u16 minLocal; /* Copy of Btree.minLocal or Btree.minLeaf */ u16 cellOffset; /* Index in aData of first cell pointer */ _2007-Sep-07 05:58:45 by anonymous:_ {linebreak} Any word on applying this patch? It saves 8 bytes per MemPage. For the default 2000 page cache it would save 16000 bytes. Larger page caches would save considerably more memory. ---- _2007-Oct-26 21:08:03 by anonymous:_ {linebreak} With this patch sizeof(PgHdr) is reduced by 4 bytes (from 48 bytes to 44 bytes) for gcc. No regressions. "make test" runs in the same amount of time. Index: src/pager.c =================================================================== RCS file: /sqlite/sqlite/src/pager.c,v retrieving revision 1.393 diff -u -3 -p -r1.393 pager.c --- src/pager.c 20 Oct 2007 13:17:55 -0000 1.393 +++ src/pager.c 26 Oct 2007 21:00:33 -0000 @@ -261,11 +261,11 @@ struct PgHdr { PgHdr *pNextHash, *pPrevHash; /* Hash collision chain for PgHdr.pgno */ PagerLruLink free; /* Next and previous free pages */ PgHdr *pNextAll; /* A list of all pages */ - u8 inJournal; /* TRUE if has been written to journal */ - u8 dirty; /* TRUE if we need to write back changes */ - u8 needSync; /* Sync journal before writing this page */ - u8 alwaysRollback; /* Disable DontRollback() for this page */ - u8 needRead; /* Read content if PagerWrite() is called */ + u8 inJournal:1; /* TRUE if has been written to journal */ + u8 dirty:1; /* TRUE if we need to write back changes */ + u8 needSync:1; /* Sync journal before writing this page */ + u8 alwaysRollback:1; /* Disable DontRollback() for this page */ + u8 needRead:1; /* Read content if PagerWrite() is called */ short int nRef; /* Number of users of this page */ PgHdr *pDirty, *pPrevDirty; /* Dirty pages */ #ifdef SQLITE_ENABLE_MEMORY_MANAGEMENT---- _2007-Oct-26 21:20:09 by anonymous:_ {linebreak} How about a compromise. What about some optional setting like? #ifndef OMIT_SQLITE_BITFIELDS # define SQLITE_BITFIELD(X) : X #else # define SQLITE_BITFIELD(X) #endif ... u8 inJournal SQLITE_BITFIELD(1); /* TRUE if has been written to journal */ #f2dcdc 2728 code active 2007 Oct anonymous 2007 Oct 4 4 Some indexes could contain pointers not the data It would be nice if there was a class of index(-column) that would contain a reference to the data in (in the table) rather than a copy of the data. This is useful where you have a need to index large strings or blobs and the size penalty of having 2+ copies ends up being *very* expensive. Consider the cost of the implied index on: create table t1 ( c1 BLOB UNIQUE); and store 100s of 1MB objects. The raw file ends up being twice as large as you ideally would like it to be. *WHEN* to use such indexes isn't entirely clear to me, not is the syntax for doing it in general, 30s of thought I can come up with: create table t1 ( c1 BLOB &UNIQUE); and say create index idx1 on t2(c1, &c2, c3); (meaning c2 would be done via reference, c1 & c3 as copies of the data). It's *FAR* from clear what other databases do here (I didn't look). This would also be useful in cases where I have multiple indexes on tables like email (headers) and values which can be very large (100s of bytes in some cases (Subject:) and I might have 5 or more indexes, means a 100-byte column will take over 600 bytes (+ padding)). Allowing for references would be slower in some cases but faster in others because of the smaller footprint and much grater CPU cache utilization. _2007-Oct-14 15:27:12 by anonymous:_ {linebreak} This enhacement has been discussed on the mailing list before, and would break sqlite3 file format compatability. ---- _2007-Oct-15 02:42:02 by anonymous:_ {linebreak} This is non standard sql. Also, the random access to pages to check indexes constraints (unique indexes, also primary keys) will trash the entire database performance. If you need small index disk usage, consider using hashes to your data keys. it´s the best that you could do to solve your problem. ---- _2007-Oct-18 06:07:58 by anonymous:_ {linebreak} Using a hash is what I've tried doing in a couple of cases. It's not very ideal. For one thing is ordering is messed up. I wonder about a virtual index concept, where you can define a function that takes the column value and returns something to actually store in the index? I'm told oracle has this feature, so whilst it's non-standard there is some precedent (albeit in a very different space). #e8e8bd 2736 build active 2007 Oct anonymous 2007 Oct 2 2 build problems on freebsd on freebsd: --disable-threads does not work. it is accepted as a valid option but no defs are added to the makefile -lgcc needs to be included in SHLIB_LD_LIBS pkgIndex.tcl is not built when -DSQLITE_THREADSAFE=0 is added manually, this causes the install target to fail. _2007-Oct-17 21:16:23 by anonymous:_ {linebreak} I forgot to mention this is with the TEA version #f2dcdc 2715 code active 2007 Oct anonymous 2007 Oct 1 1 no authorization needed to remove authorizer there should be a new auth code created and the auth function should be consulted for permission for removal. _2007-Oct-10 01:08:48 by drh:_ {linebreak} I'm assuming that this feature request comes from {quote: RockShox} and that the development language is Tcl. No. If your adversary has the ability to invoke the interface that removes an authorizer, then you system is already pwned. What you really need is the ability to [interp alias] the eval method into a safe interpreter. That way you can: *: Open the database in the main interpreter *: Set up the authorizer in the main interpreter to invoke a script in the main interpreter *: Set up the [interp alias] so that the safe interpreter can do [db eval ...] but not [db auth ...] It seems like an "-interp" option on the "eval" method of the database connection object would likely be the right interface. Or perhaps there should be separate "safeeval" method. Either way, it has been years and years since I have done anything with safe interpreters so I will have to look into what needs to be done to make that happen. ---- _2007-Oct-17 20:11:23 by anonymous:_ {linebreak} ok i think i agree with that. currently you cannot use an interp alias since the target command runs in the target interp and all your variables and commands are in the wrong scope. this means one needs to load sqlite again in the new interp, and sqlite will not load in a safe interp so a regular interp is required. to be useful, a -interp flag would need to execute in the current scope of the interp and not the global scope. #f2dcdc 2714 code active 2007 Oct anonymous 2007 Oct drh 3 4 Shell cannot import large files greater than 2 GB [patch] If I issue an .import command within the SQLite shell and the file size is larger than 2 GB, the shell gives the error message "cannot open file: foo.txt". Putting #define _FILE_OFFSET_BITS 64 before the #include statements in shell.c fixes this problem under Linux. I tested this solution with a 2.6 GB file under Ubuntu Feisty Fawn 7.04 with Linux kernel 2.6.20-16-generic. _2007-Oct-12 18:48:35 by drh:_ {linebreak} I'm not convinced SQLite _should_ be able to import files larger than 2GiB. Does anybody really ever need to read more than 2GiB of SQL text? That is a lot of SQL, don't you think? Is this really a desirable feature? Can you not split it up into two or more smaller files? ---- _2007-Oct-14 09:25:42 by anonymous:_ {linebreak} I'm not 100% certain about this but: #define _FILE_OFFSET_BITS 64 isn't necessarily correct, better to tweak the makefile(s): CFLAGS += $(shell getconf LFS_CLFAGS) (similar for LDFLAGS) should work on 32/64 bit machines and different platforms. For shell.c I really wouldn't bother and think this should be left alone. Using .import for load more than 2GB makes me things the shell should be fleshed out and extended for this sort of abuse, handy as it is right now, it's not a full SQL shell but a minimalist shell and a useful example). ---- _2007-Oct-16 14:08:09 by anonymous:_ {linebreak} I am the original poster. I use the shell to import CSV files in the 3-4 GiB range on a daily basis. I don't see the point of splitting the files or writing my own importer when the shell does a perfectly fine job once large file support has been activated. With all due respect, I don't understand the reluctance to enable a useful feature. Shouldn't the user decide whether it is reasonable for his application to import big files? I work in the health care industry processing insurance claims for several very large companies. There's nothing broken or wrong with my application -- big files are just the nature of the data I'm handling. Neither of you has proposed any disadvantage to enabling LFS in the shell. If there are no drawbacks, why not turn it on? If the core library can open large database files, why shouldn't the shell be able to import large ones? ---- _2007-Oct-16 16:24:22 by anonymous:_ {linebreak} For reasons also unknown to me, the authors appear to regard the shell as a largely unused sample program. Based on my personal experience in talking to various sqlite library users, it is the primary administrative front-end for sqlite. In particular, doing backups and merging databases. #e8e8bd 2729 doc active 2007 Oct anonymous 2007 Oct 1 1 Lemon: %fallback, %wildcard, and @X uncodumented I noticed that the lemon documentation does not mention the %fallback and %wildcard directives. Both are in the code and are apparently doing useful work in SQLite's parse.y. Can other users benefit from them as well? The symbol @X is also undocumented. From a source code comment I read that it "If the argument is of the form @X then substituted the token number of X, not the value of X". A short documentation example would help to understand where and how it can be useful to apply this syntax. Are there other nice but undocumented Lemon goodies lacking documentation? #f2dcdc 2725 code active 2007 Oct anonymous 2007 Oct 1 1 memory leak in sqlite3_open_v2() when it fails only happens with flags = SQLITE_OPEN_READWRITE; and when res = sqlite3_open_v2(sourcename, &conn, flags, NULL); seems to leak 674 bytes per call _2007-Oct-15 07:07:07 by danielk1977:_ {linebreak} Are you calling sqlite3_close(conn) after the error occurs? All calls to sqlite3_open_v2() need to be matched by a call to sqlite3_close(), even if an error occurs. #e8e8bd 2701 new active 2007 Oct anonymous 2007 Oct 5 5 Make INSERT-ing multiple rows MySQL-compatible SQLite syntax allows to insert only one row withinsert into test (a, b, c) values (1, 2, 3);MySQL allows to insert multiple withinsert into test (a, b, c) values (1, 2, 3), (4, 5, 6), (7, 8, 9) -- etcBut SQLite is also capable of inserting multiple by using INSERT...SELECT:insert into test (a, b, c) select 1, 2, 3 union select 4, 5, 6 -- etcIt would be nice to make INSERT statement syntactically compatible with MySQL, allowing to insert multiple rows with VALUES clause. It can be implemented by simply translating multiple 'VALUES ()()()' to 'select union' - no serious change required at all. _2007-Oct-08 21:45:05 by anonymous:_ {linebreak} You mean "UNION ALL", not "UNION". UNION would remove duplicate rows, and create an ephemeral table that you don't want because it's less efficient. Your idea is a good one and could be implemented largely in the parser. The number of VDBE opcodes would be quite large for such a statement. I wonder if that would present a problem. ---- _2007-Oct-14 08:12:11 by anonymous:_ {linebreak} Patch implementing multi-row INSERT statements against 3.5.1 source tree: http://www.mail-archive.com/sqlite-users%40sqlite.org/msg28337.html #e8e8bd 2727 build active 2007 Oct anonymous 2007 Oct 4 4 building with -malign-double causes strange behavior sqlite 3.5.1 amalgamation has problems when enabling a wide set of compiler features on gcc (GCC) 4.1.2 (Ubuntu 4.1.2-0ubuntu4) on linux/i686 /w glibc-2.5 strange behavior occurs. typical strangeness is that SQLITE_FULL is returned from sqlite3_prepare_v2() (called immediately after a statement was created with sqlite3_mprintf). This happens even though there is a reasonable amount of disk space free (6G). the compile line looks like:{linebreak} cc -g -pg -O1 -march=i686 -msse2 -malign-double -m128bit-long-double -momit-leaf-frame-pointer -minline-all-stringops -D_XOPEN_SOURCE=520 '-DVERSION="0.10r276M"' -Isqlite/ -DSQLITE_OMIT_LOAD_EXTENSION -DTHREADSAFE=0 -DSQLITE_OMIT_EXPLAIN -DSQLITE_ENABLE_COLUMN_METADATA -c -o sqlite/sqlite3.o sqlite/sqlite3.c and valgrind reports:{linebreak} ==15323== Use of uninitialised value of size 4{linebreak} ==15323== at 0x80585B7: insertElement (sqlite3.c:13072){linebreak} ==15323== by 0x807366E: sqlite3HashInsert (sqlite3.c:13290){linebreak} ==15323== by 0x809FE72: unixOpen (sqlite3.c:15403){linebreak} ==15323== by 0x80579A7: sqlite3OsOpen (sqlite3.c:8210){linebreak} ==15323== by 0x806B12F: sqlite3BtreeFactory (sqlite3.c:21317){linebreak} ==15323== by 0x80767BA: openDatabase (sqlite3.c:71237){linebreak} ==15323== by 0x8076BD1: sqlite3_open (sqlite3.c:71337){linebreak} ==15323== by 0x804BEAE: db_init (dbmgr.c:81){linebreak} ==15323== by 0x804CE64: main (main.c:59){linebreak} my dbmgr.c:81:db_init() is: res=sqlite3_open(filename, &system_db); and system_db=NULL and filename="zomg.sqlite" (string literal). so the parameters seem normal. if I turn off -malign-double everything works fine. (this "bug" also seems to be on 3.4.1 amalgamation) _2007-Oct-14 04:35:16 by anonymous:_ {linebreak} This is a compiler issue. -malign-double creates problems for most programs. What are you trying to accomplish? #f2dcdc 2708 code active 2007 Oct anonymous 2007 Oct 4 2 SQL error:disk I/O error I cross-compile sqlite to embedded Linux,but after I insert data to the table ,it failed.the warning is "SQL error:disk I/O error". _2007-Oct-09 05:12:28 by anonymous:_ Why do you think it is SQLite error ?? ---- _2007-Oct-09 05:46:06 by danielk1977:_ {linebreak} We'll need a bit more data than that to figure this out. Did earlier SQLite versions work? Can you post the entire output of the compile process so that we can see if there are any clues there? Can you run strace so that we can see if there really is an IO error, or at least when SQLite believes there to be one? #f2dcdc 2705 code active 2007 Oct anonymous 2007 Oct 4 4 testfixture unresolved externals with SQLITE_OMIT_GET_TABLE Cannot build/run "make test" with -DSQLITE_OMIT_GET_TABLE due to testfixture link error:In function `test_get_table_printf': ./src/test1.c:526: undefined reference to `sqlite3_get_table' ./src/test1.c:541: undefined reference to `sqlite3_free_table' collect2: ld returned 1 exit status make: *** [testfixture] Error 1#f2dcdc 2704 code active 2007 Oct anonymous 2007 Oct 4 4 "make test" aborts before completion with SQLITE_OMIT_BLOB_LITERAL When compiled with -DSQLITE_OMIT_BLOB_LITERAL make test aborts with this error:substr-2.5.2... Ok ./testfixture: near "'61626364656667'": syntax error while executing "db eval " DELETE FROM t1; INSERT INTO t1(b) VALUES(x'$hex') "" (procedure "subblob-test" line 2) invoked from within "subblob-test 3.1 61626364656667 1 1 61" (file "./test/substr.test" line 86) invoked from within "source $testfile" ("foreach" body line 5) invoked from within "foreach testfile [lsort -dictionary [glob $testdir/*.test]] { set tail [file tail $testfile] if {[lsearch -exact $EXCLUDE $tail]>=0} continue if..." (file "./test/quick.test" line 93) make: *** [test] Error 1#f2dcdc 2703 code active 2007 Oct anonymous 2007 Oct 3 4 make test does not work with SQLITE_OMIT_FLOATING_POINT make test cannot run with SQLITE_OMIT_FLOATING_POINT. Shouldn't these tests be skipped?./src/test1.c:1255: warning: passing argument 3 of ‘Tcl_GetDouble’ from incompatible pointer type ./src/test1.c: In function ‘sqlite3_mprintf_scaled’: ./src/test1.c:1284: warning: passing argument 3 of ‘Tcl_GetDouble’ from incompatible pointer type ./src/test1.c: In function ‘test_bind_double’: ./src/test1.c:2607: warning: passing argument 3 of ‘Tcl_GetDoubleFromObj’ from incompatible pointer type ./src/tclsqlite.c: In function ‘tclSqlFunc’: ./src/tclsqlite.c:728: warning: passing argument 3 of ‘Tcl_GetDoubleFromObj’ from incompatible pointer type ./src/tclsqlite.c: In function ‘DbObjCmd’: ./src/tclsqlite.c:1609: warning: passing argument 3 of ‘Tcl_GetDoubleFromObj’ from incompatible pointer type ... $ ./testfixture test/select1.test select1-1.1... Ok select1-1.2... Ok select1-1.3... Ok select1-1.4... Ok select1-1.5... Ok select1-1.6... Ok select1-1.7... Ok select1-1.8... Ok select1-1.8.1... Ok select1-1.8.2... Ok select1-1.8.3... Ok ./testfixture: near ".": syntax error while executing "db eval {INSERT INTO test2(r1,r2) VALUES(1.1,2.2)}" ("uplevel" body line 1) invoked from within "uplevel [list $db eval $sql]" (procedure "execsql" line 3) invoked from within "execsql {INSERT INTO test2(r1,r2) VALUES(1.1,2.2)}" (file "test/select1.test" line 69)#f2dcdc 2684 code active 2007 Oct anonymous 2007 Oct 1 1 Accessing sqlite from an NT service will lock the complete databse. Accessing sqlite from a NT service (application 1) will lock the complete database. Any other process trying to open an sqlite db (application 2) will get error "80004005 unable to lock database" If application 1 runs as normal application, started by local user, this problem doesnt occur and both applications can open the db. _2007-Oct-02 15:48:05 by anonymous:_ {linebreak} SQLite has no knowledge of Windows services. How do you propose to work around this Windows anachronism? ---- _2007-Oct-02 17:20:38 by anonymous:_ {linebreak} Suggesion: Try running the service in the same account as the other program that needs to access the database. Anachronism? Service is just another word for daemon. -knu- ---- _2007-Oct-02 17:33:56 by anonymous:_ {linebreak} Re: Anachronism, the OP suggested there was something fundamentally different about file access using a service. You've pointed out that it's just a file permissions issue. ---- _2007-Oct-05 14:45:07 by drh:_ {linebreak} Two points: 1: The error message "80004005 unable to lock database" is not generated by SQLite. There must be some middleware someplace that is producing this message. The problem might be in that middleware and not in SQLite. 2: None of the SQLite developers run windows. Consequently any fixes for this problem will need to come from the community. Please append patches to this ticket if you find a fix. Or close the ticket if you discover that the problem is outside of SQLite. #e8e8bd 55 new active 2002 Jun anonymous 2007 Oct drh 5 3 instead of triggers for inserts (updates) on regular tablesxx It would be very useful to be able to define a trigger that only executes the trigger code and will prevent the actual insert (or update) on regular tables (eg. specialised autoincrement fields). This could be done by allowing the same "instead of" syntax as used for views. Even better would be the possibility of return codes in a before trigger that can prevents the insert or raise an error. Ideally, this return code could be given conditionally (implementing foreign key and check like functionality). test ---- _2007-Oct-01 23:37:44 by anonymous:_ {linebreak} {link: http://www.sqlite.org/lang_createtrigger.html create trigger syntax} :{linebreak} A special SQL function RAISE() may be used within a trigger-program, with the following syntax: raise-function ::= RAISE ( ABORT, error-message ) | RAISE ( FAIL, error-message ) | RAISE ( ROLLBACK, error-message ) | RAISE ( IGNORE ) #f2dcdc 2674 code active 2007 Sep anonymous 2007 Sep 3 4 NFS fails without lock manager *Problem:* SQLite fails entirely if the NFS lock manager is not running on the share hosting the DB -- even when all access to the DB is serialized. Under these circumstances, it becomes difficult to create applications that are capable of running in a multitude of environments because restrictions are now imposed upon the storage location. ---- *Reproduction:* *:setup an NFS share *:disable the lock manager *:attempt to perform any transactions on a db on that share *:start the lock manager *:perform the same operations ---- *Request:* *:Make a change to the error handling, as necessary, to allow processes to access a DB over an NFS share without use of a lock manager. *:Make a big bold flashing sign in the FAQ about this failure-mode. The wording of the current FAQ led us to believe that the transaction would go through, but protection from other processes was not guaranteed. ---- *Shameless Fanmail:* Love the product! ---- *Screenshot:* dev-srs08 ~ #[__db_init .c:384] Statement: CREATE TABLE versions ( id INTEGER PRIMARY KEY AUTOINCREMENT, version INTEGER, tbl CHAR(256) ); Failed with error: database is locked [main .c:1096] Unable to init output DB: /mnt/nfs/o dev-srs08 ~ # /etc/init.d/nfslock start Starting NFS statd: [ OK ] dev-srs08 ~ # dev-srs08 ~ # _2007-Sep-28 04:35:00 by anonymous:_ {linebreak} Yet another reason to avoid using SQLite on remote shares. ---- _2007-Sep-29 18:04:19 by anonymous:_ {linebreak} _Make a change to the error handling, as necessary, to allow processes to access a DB over an NFS share without use of a lock manager._ By design the SQLite library guarantees ACID. It can't provide that without file locks. In my opinion a non-ACID version would be a custom version, which you can (should) build yourself. The source is available, or there might already be a compilation option to accomplish that. _Make a big bold flashing sign in the FAQ about this failure-mode. The wording of the current FAQ led us to believe that the transaction would go through, but protection from other processes was not guaranteed._ FAQ (5) "When any process wants to write, it *must lock* the entire database file for the duration of its update." seems to be quite clear. -knu- #e8e8bd 2677 new active 2007 Sep anonymous 2007 Sep danielk1977 5 3 CREATE TRIGGER .. {databaseevent ON ... }+ for multiple events I have to write triggers which react on either INSERT or UPDATE of a specific field in a table. In some SQL syntax descriptions I found a possibility to combine the trigger events for more than one condition but not in SQLite3. Isn't there a possibility to use one trigger definition for more than one event or didn't I find the trick? Do I have to copy the whole definition for each event; even if the body text is exactly the same? {linebreak} Suggestion: On INSERT event use the OLD.fieldnames filled with NULL content to avoid problems with UPDATE event. - If not possible please explain why not for my better understanding. Thank you. #f2dcdc 2671 code active 2007 Sep shess 2007 Sep shess 2 4 Fts field-based queries are not correctly case-insensitive. CREATE VIRTUAL TABLE t USING fts2(A, B); -- At the SQL level, things are case-insensitive: INSERT INTO t (A, b) VALUES ('Test', 'Columns'); INSERT INTO T (a, B) VALUES ('Second', 'Test'); -- Unfortunately, fts cannot do field-level queries: SELECT rowid FROM t WHERE t MATCH 'test'; -- works SELECT rowid FROM t WHERE b MATCH 'test'; -- works SELECT rowid FROM t WHERE t MATCH 'b:test'; -- no results SELECT rowid FROM t WHERE t MATCH 'B:test'; -- no results It doesn't work because fts is keeping the column name as 'B', but the query parsing uses the results from the tokenizer, which are case-folded, and 'b' != 'B'. I'm thinking on the solution. A quick fix would be to make the azColumn storage be lowercase, but the core problem is that field names probably shouldn't be run through the tokenizer in the first place. #f2dcdc 2664 code active 2007 Sep danielk1977 2007 Sep 1 1 attaching the same db twice in shared-cache mode fails The following SQL script can cause an assert() to fail in shared-cache mode. ATTACH 'db' AS aux1; ATTACH 'db' AS aux2; CREATE TABLE aux1.abc(a, b, c); CREATE TABLE aux2.abc(a, b, c); See also #2653 #f2dcdc 2294 code active 2007 Apr anonymous 2007 Sep 2 1 segfault when destroying lock on WinCE with threads DestroyLock emulation on WinCE platform releases the zDeleteOnClose file outside the mutex acquire section. This lead to frequent segfault when working with several databases concurrently. Patch simply consists in moving code: if( pFile->zDeleteOnClose ){ DeleteFileW(pFile->zDeleteOnClose); sqliteFree(pFile->zDeleteOnClose); pFile->zDeleteOnClose=NULL; } from winClose() (os_win.c:980) to winceDestroyLock(), inside scope of winceMutexAcquire(pFile->hMutex): /* De-reference and close our copy of the shared memory handle */ UnmapViewOfFile(pFile->shared); CloseHandle(pFile->hShared); + if( pFile->zDeleteOnClose ){ + DeleteFileW(pFile->zDeleteOnClose); + sqliteFree(pFile->zDeleteOnClose); + pFile->zDeleteOnClose=NULL; + } /* Done with the mutex */ winceMutexRelease(pFile->hMutex); CloseHandle(pFile->hMutex); pFile->hMutex = NULL; _2007-Apr-13 00:21:33 by anonymous:_ {linebreak} Check in [3836] fixes it for me. eTcl regression tests, related to running several sqlite database concurrently in several threads, are now passed while they were frequently segfaulting without it. To help being confident with this _blind_ commit, let's mention that exactly same patch has been introduced in eTcl built since a couple of monthes, to fix issue reported by WM2003 users, and all reported a correct fix. However, I did suggest the patch, so testing and feedback from others may help :-) Also, note that [3836] has a typo, requesting feedback in ticket #2249 instead of #2294 ---- _2007-Sep-25 03:24:09 by anonymous:_ {linebreak} The fix doesn't work and should be reverted. Temporary files do not create locks, so when they are closed and hMutex is null, the winceDestroyLock() file is never called and the temporary files are not cleaned up properly. #f2dcdc 2652 code active 2007 Sep drh 2007 Sep 1 1 Aggregate function cannot be used from within a subquery The following SQL fails: CREATE TABLE t1(x,y); INSERT INTO t1 VALUES(1,2); CREATE TABLE t2(z); INSERT INTO t2 VALUES(1); SELECT (SELECT y FROM t1 WHERE x=min(z)) FROM t2; Problem reported on the mailing list. _2007-Sep-23 16:01:09 by anonymous:_ {linebreak} Your syntax appears to be incorrect.{linebreak} SQLite v3.4.2 CREATE TABLE t1(x,y); CREATE TABLE t2(z); INSERT INTO t1 VALUES(1,21); INSERT INTO t1 VALUES(2,22); INSERT INTO t1 VALUES(3,23); INSERT INTO t2 VALUES(3); INSERT INTO t2 VALUES(2); INSERT INTO t2 VALUES(1); What you wanted to do: SELECT y FROM t1 WHERE x=(SELECT min(z) FROM t2); 21 -- works as expected What you did: SELECT (SELECT y FROM t1 WHERE x=min(z)) FROM t2; SQL error near line []: misuse of aggregate function min() #f2dcdc 2539 code active 2007 Jul anonymous 2007 Sep 2 2 WinCE: Temporary etilqs_ files are not removed from temporary folder Hi, when temporary etilqs_* files are created during SQLite work on Windows CE devices, they are not removed at all. Temporary folder at CE devices: /Application Data/Volatile I've research that it winClose(os_win.c) function has been changed at do not remove this file, assuming it to be removed at winceDestroyLock(os_win.c), so if no lock was happened then files will stay here forever. Has fixed it in my local copy, with hope that it will be fixed when new cool versions of SQLite will be available. My fix at os_win.c: static int winClose(OsFile **pId){ winFile *pFile; int rc = 1; if( pId && (pFile = (winFile*)*pId)!=0 ){ int rc, cnt = 0; OSTRACE2("CLOSE %d\n", pFile->h); do{ rc = CloseHandle(pFile->h); }while( rc==0 && cnt++ < MX_CLOSE_ATTEMPT && (Sleep(100), 1) ); #if OS_WINCE winceDestroyLock(pFile); // fix begin if( pFile->zDeleteOnClose ){ DeleteFileW(pFile->zDeleteOnClose); sqliteFree(pFile->zDeleteOnClose); } // fix end #endif OpenCounter(-1); sqliteFree(pFile); *pId = 0; } return rc ? SQLITE_OK : SQLITE_IOERR; }Thanks, Fedor _2007-Jul-28 16:41:41 by anonymous:_ {linebreak} The solution is to revert checkin 3836 and re-open ticket #2294.Looking at the wince locking mechanism, the only time we ever use the zDeleteOnClose flag is when we've opened a database for exclusive access in sqlite3WinOpenExclusive. To save time and resources (and because its not necessary) we never bother creating a locking mechanism for exclusively-opened files. So pFile->hMutex is NULL when hitting winceDestroyLock(), and the file is never deleted.
Is it possible that the original poster of #2294 was trying to close the same connection on multiple threads at the same time? ---- _2007-Jul-31 05:32:39 by anonymous:_ {linebreak} This is actually a duplicate of #2533 ---- _2007-Sep-21 14:20:05 by anonymous:_ {linebreak} So when the fix of [3836] was applied, the code to delete the file was only put in the section that is called when we have a mutex. I wonder, if the deletion of the file should also take place if there was no mutex. Works for me at least: static void winceDestroyLock(winFile *pFile){ if (pFile->hMutex){ /* Acquire the mutex */ winceMutexAcquire(pFile->hMutex); /* The following blocks should probably assert in debug mode, but they are to cleanup in case any locks remained open */ if (pFile->local.nReaders){ pFile->shared->nReaders --; } if (pFile->local.bReserved){ pFile->shared->bReserved = FALSE; } if (pFile->local.bPending){ pFile->shared->bPending = FALSE; } if (pFile->local.bExclusive){ pFile->shared->bExclusive = FALSE; } /* De-reference and close our copy of the shared memory handle */ UnmapViewOfFile(pFile->shared); CloseHandle(pFile->hShared); * if( pFile->zDeleteOnClose ){ * DeleteFileW(pFile->zDeleteOnClose); * sqliteFree(pFile->zDeleteOnClose); * pFile->zDeleteOnClose = 0; * } /* Done with the mutex */ winceMutexRelease(pFile->hMutex); CloseHandle(pFile->hMutex); pFile->hMutex = NULL; } + else + { + if( pFile->zDeleteOnClose ){ + DeleteFileW(pFile->zDeleteOnClose); + sqliteFree(pFile->zDeleteOnClose); + pFile->zDeleteOnClose = 0; + } + } } The code marked with * was put there in #f2dcdc 2653 code active 2007 Sep anonymous 2007 Sep 3 4 Exclusive Transactions do not work with a Database File attached twice Regarding the docs, it is possible to attach the same database file multiple times. After doing so, I wanted to begin an exclusive transaction. Unfortunately, this fails ("database is locked") and surprises me as I did not find any notice on this particular situation and possible side-effects neither in the attach nor in the transaction/locking documentation. If this behaviour is seen as an error, It would be useful to have it error fixed, because if one reads a list of (possibly duplicate) database files but with unique identifiers, it would be helpful to use these defined identifiers when accessing the databases (and that in an exclusive transaction). Real-world case: Each "module" uses its own database file. Some modules share a database file. So there is a list of module -> database file assignments. Now an update process gets some database update scripts from the modules. Every module wants its changes to be done in the right database, so it relies on having its own database attached with an unique identifier - the module's name. On the other hand, the update process need exclusive access to the databases and starts a transaction -- bummer. #e8e8bd 2651 new active 2007 Sep anonymous 2007 Sep 5 4 Add support for overriding home directory location Currently, the history file's and the rc file's location is hard-wired to =$HOME=. It would be nice if this could be overridden. One way is to look for a =SQLITE_HOME= environment variable that points to the location to use. This can be achieved by a simple addition to =find_home_dir()= in =src/shell.c=. #e8e8bd 2649 new active 2007 Sep anonymous 2007 Sep 4 4 Add an "--enable-extensions" (default=no) to the configure script The attached patch adds "--enable-extensions" to the configure script, but is disabled by default (because of the security considerations of having it enabled). #e8e8bd 2645 build active 2007 Sep anonymous 2007 Sep 5 4 Conflict between tclConfig.sh and tclinstaller.tcl I'm using a non-standard location for Tcl: --with-tcl=/home/scott/lib The build process finds and uses the tclConfig.sh file from /home/scott/lib just fine, but tclinstaller.tcl is called without an explicit path and uses the system's tclsh instead of the one in /home/scott/bin and thus tries to install that portion of code into the system's area. While I can change my path to resolve this, I think it makes more sense for tclinstaller.tcl to use the path information that's embedded in tclConfig.sh to be consistent. /s. #f2dcdc 2627 code active 2007 Sep anonymous 2007 Sep 3 2 Improper parsing of nested JOIN SQLite has a problem with multiple nested JOINs. The only way to get it workig is to remove the surrounding brackets. Removing the brackets unfortunately do not work in other DB systems such as MS SQL, mysql etc. This does not work: Select ContactPhone.* From (ContactPhone LEFT OUTER JOIN ContactLocation ON ContactPhone.PHNLCT_ID = ContactLocation.LCT_ID) LEFT OUTER JOIN ContactItem ON ContactLocation.LCTITM_ID = ContactItem.ITM_ID (It complains about LCT_ID or similar) This works after removing the brackets: Select ContactPhone.* From ContactPhone LEFT OUTER JOIN ContactLocation ON ContactPhone.PHNLCT_ID = ContactLocation.LCT_ID LEFT OUTER JOIN ContactItem ON ContactLocation.LCTITM_ID = ContactItem.ITM_ID All other major DB systems require the surrounding brackets. Do you think it is possible to fix it? Apart from this little little SQLite is an awesome project. Thank you Jakub Klos _2007-Sep-06 13:07:32 by anonymous:_ {linebreak} I don't have access to MS SQL Server, but MySQL and Oracle have no issue with the query without parentheses:
create table x1(a int, b int); create table x2(c int, d int); create table x3(e int, f int); mysql> select x1.* from x1 left join x2 on x1.a=x2.c left join x3 on x2.d=x3.e; Empty set (0.00 sec)---- _2007-Sep-06 19:03:33 by anonymous:_ {linebreak} MSSQL also has no problems without the parens. As a matter of fact, the only DB that I know of that requires them is MS Access (JET). ---- _2007-Sep-06 20:11:31 by anonymous:_ {linebreak} I guess he had no luck filing a JET bug. ---- _2007-Sep-11 17:22:28 by anonymous:_ {linebreak} True, MS access requires the parens but all other major DBs support the query syntax with the parens. So why SQLite does not like it? It should simply ignore them if possible. Thank you #f2dcdc 2634 code active 2007 Sep anonymous 2007 Sep 3 3 .schema uses incorrect ORDER BY giving wrong dependency order When the schema is exported, views are sorted by name instead of by dependency. If there are nested views, the schema may be invalid when used to re-create the database. sqlite3 create table t ( f text ); create view v2 as select f from t; create view v1 as select f from v2; .output test.txt .schema .exit sqlite3 .read test.txt SQL error near line 2: no such table: main.v2 _2007-Sep-07 15:33:06 by anonymous:_ {linebreak} Use .dump instead as a workaround. Unlike .schema, .dump does not use ORDER BY in its queries on sqlite_master and it outputs its rows in order of entry.SQLite version 3.5.0 Enter ".help" for instructions sqlite> create table t ( f text ); sqlite> create view v2 as select f from t; sqlite> create view v1 as select f from v2; sqlite> sqlite> .schema CREATE TABLE t ( f text ); CREATE VIEW v1 as select f from v2; CREATE VIEW v2 as select f from t; sqlite> sqlite> .dump BEGIN TRANSACTION; CREATE TABLE t ( f text ); CREATE VIEW v2 as select f from t; CREATE VIEW v1 as select f from v2; COMMIT;Suggested patch:Index: src/shell.c =================================================================== RCS file: /sqlite/sqlite/src/shell.c,v retrieving revision 1.167 diff -u -3 -p -r1.167 shell.c --- src/shell.c 7 Sep 2007 01:12:32 -0000 1.167 +++ src/shell.c 7 Sep 2007 15:28:24 -0000 @@ -1411,8 +1411,7 @@ static int do_meta_command(char *zLine, "SELECT sql FROM " " (SELECT * FROM sqlite_master UNION ALL" " SELECT * FROM sqlite_temp_master) " - "WHERE tbl_name LIKE shellstatic() AND type!='meta' AND sql NOTNULL " - "ORDER BY substr(type,2,1), name", + "WHERE tbl_name LIKE shellstatic() AND type!='meta' AND sql NOTNULL", callback, &data, &zErrMsg); zShellStatic = 0; } @@ -1421,8 +1420,7 @@ static int do_meta_command(char *zLine, "SELECT sql FROM " " (SELECT * FROM sqlite_master UNION ALL" " SELECT * FROM sqlite_temp_master) " - "WHERE type!='meta' AND sql NOTNULL AND name NOT LIKE 'sqlite_%'" - "ORDER BY substr(type,2,1), name", + "WHERE type!='meta' AND sql NOTNULL AND name NOT LIKE 'sqlite_%'", callback, &data, &zErrMsg ); }after patch:SQLite version 3.5.0 Enter ".help" for instructions sqlite> create table t ( f text ); sqlite> create view v2 as select f from t; sqlite> create view v1 as select f from v2; sqlite> .schema CREATE TABLE t ( f text ); CREATE VIEW v2 as select f from t; CREATE VIEW v1 as select f from v2; sqlite> .q#e8e8bd 2629 doc active 2007 Sep anonymous 2007 Sep 4 4 typo in os_unix.c for nolockIndex: src/os_unix.c =================================================================== RCS file: /sqlite/sqlite/src/os_unix.c,v retrieving revision 1.165 diff -u -3 -p -r1.165 os_unix.c --- src/os_unix.c 5 Sep 2007 13:56:32 -0000 1.165 +++ src/os_unix.c 6 Sep 2007 17:53:47 -0000 @@ -2126,7 +2126,7 @@ static const sqlite3_io_methods sqlite3D /* ** This vector defines all the methods that can operate on an sqlite3_file -** for unix with dotlock style file locking. +** for unix with nolock style file locking. */ static const sqlite3_io_methods sqlite3NolockLockingUnixIoMethod = { 1, /* iVersion */#e8e8bd 2607 event active 2007 Aug anonymous 2007 Sep 2 1 Data loss, continuation to Re: [sqlite] how to flush database to disk? See that mailing list. The originator message : ======================== I've just lost a couple of days' worth of data when my app crashed. (Well, the data wasn't a total loss thanks to backup plans, but the database itself essentially reverted to its state of 2 days ago.) This is despite my app doing a COMMIT after every modification of the DB. It's acting as though all the changes were held in memory, or somehow journaled, and when the crash happened, the changes were all lost or rolled back. What I need is a way to force the database to save its data to disk while my app is running, so that in the event of a crash, I lose little or no data. How can I do this? I presume that closing the database would do the trick, but is there a less heavy-handed way? =========== The exact like that data loss occured at me too. Three times, non-repropucible regualrilly. What common in these losses ? {Editing+committing} in the main thread then {navigating, bof/eof checking, reading data} from within different threads then return to the main thread for {editing+committing}. _2007-Aug-31 19:38:26 by drh:_ {linebreak} The COMMIT does not actually occur until you call sqlite3_reset() and/or sqlite3_finalize() on all your prepared statements. Any prepared statement that has not been reset or finalized is still running, is incomplete, and is thus still holding the transaction open. I'm guessing that you have an unreset and unfinialized statement in your application. I wonder what would happen if we changed the definition of COMMIT so that it returned an error if there were active prepared statements. This is, technically, an incompatibility. But we are coming up on a release with several other minor incompatibilities, so now might be a good time to insert such a change. ---- _2007-Aug-31 19:45:36 by drh:_ {linebreak} I looked in the code, and it turns out we already do this. Perhaps the application is not checking the return code from the COMMIT to see that it is failing? ---- _2007-Aug-31 20:40:47 by anonymous:_ {linebreak} No error reports came from COMMIT. The data loss were noticed after UPDATE & COMMIT after massive reading of results of SQL addressing the same virtual (ATTACHed) tables, from within another thread. ---- _2007-Aug-31 20:53:48 by anonymous:_ {linebreak} Is it possible that a pending transaction survive app shutdown & then OS restart ? Is yes, then any DB error would cause rollback to the data on last BEGIN, isn't ? ---- _2007-Aug-31 21:00:18 by anonymous:_ {linebreak} But me committed each smallest change to dat,a then saw these refreshed data in the tables. And on the next day these data were present. Only reading (with full scrolling ) the affected virtual tables from within another thread then new editing then committing caused the loss. ---- _2007-Sep-01 23:12:31 by anonymous:_ {linebreak} You mention virtual tables. Their data is not maintained by the SQLite engine, but by your own module. If that doesn't implement ACID, you're out of luck. By definition:{linebreak} {quote: A virtual table is an interface to an external storage or computation engine that appears to be a table but does not actually store information in the database file.} references:{linebreak}{link: http://www.sqlite.org/lang_createvtab.html CreateVirtualTable}{linebreak} {wiki: VirtualTables VirtualTables} ---- _2007-Sep-04 04:51:15 by anonymous:_ {linebreak} Me was wrong. These were't true virtual tables. Me used a LEFT OUTER query to several tables residing in different ATTACHed databases. ---- _2007-Sep-04 04:52:01 by anonymous:_ {linebreak} were't => were not, above. #e8e8bd 2604 new active 2007 Aug anonymous 2007 Aug 4 4 CREATE VIRTUAL TABLE does not allow IF NOT EXISTS CREATE VIRTUAL TABLE vt IF NOT EXISTS; would help with development since creating a virtual table that exists returns error 1 - as do several "Real" errors. #e8e8bd 2595 doc active 2007 Aug anonymous 2007 Aug 4 4 sqlite3_commit_hook doc typo src/main.c: -** Register a function to be invoked when a transaction comments. +** Register a function to be invoked when a transaction commits. #f2dcdc 2559 code active 2007 Aug anonymous 2007 Aug 4 4 "make clean" does not delete sqlite3.c and tsrc/Index: Makefile.in =================================================================== RCS file: /sqlite/sqlite/Makefile.in,v retrieving revision 1.179 diff -u -3 -p -r1.179 Makefile.in --- Makefile.in 27 Aug 2007 23:38:43 -0000 1.179 +++ Makefile.in 28 Aug 2007 01:25:55 -0000 @@ -724,7 +724,7 @@ clean: rm -f testfixture$(TEXE) test.db rm -rf doc rm -f common.tcl - rm -f sqlite3.dll sqlite3.lib sqlite3.def + rm -rf sqlite3.dll sqlite3.lib sqlite3.def sqlite3.c tsrc distclean: clean rm -f config.log config.status libtool Makefile config.h sqlite3.pc#e8e8bd 2587 build active 2007 Aug anonymous 2007 Aug 3 4 Build problem when using the SQLITE_OMIT_FLOATING_POINT define. I apologize in advance if the values I chose above are not appropriate. If I define SQLITE_OMIT_FLOATING_POINT=1 and try to build a Windows DLL, I get two errors in loadext.c, line 116 and 192. "error C4028: formal para meter 3 different from declaration" I believe you want to change the include order at the top of loadext.c from: #include "sqlite3ext.h" #include "sqliteInt.h" to: #include "sqliteInt.h" #include "sqlite3ext.h" Reversing the order of include fixes my build. Yes, I know there is no real reason to disable floating point for the Windows DLL. I'm actually porting SqLite for use in an NT kernel mode driver and avoiding floating point operations will save a lot of time if I don't really need them and I don't. So I made sure this was a problem with a supported platform like the Windows DLL and griped about that instead of my insanity. ;-) You can email questons to mspiegel@vipmail.com. If you want to discuss this over the phone, shoot me an email and I'll send you phone numbers. #e8e8bd 2568 new active 2007 Aug anonymous 2007 Aug 3 3 TEMP_STORE is ignored in some cases It seems that sometimes TEMP_STORE is ignored. I've tried to force SQLite to always use memory by setting TEMP_STORE=3, but some etilqs_* temp files are still being created. The call stack that's causing these file to be created is: sqlite3PagerOpentemp(OsFile * *) sqlite3PagerStmtBegin(Pager *) sqlite3BtreeBeginStmt(Btree *) sqlite3VdbeExec(Vdbe *) sqlite3Step(Vdbe *) sqlite3_step(sqlite3_stmt *) It looks like the temp files are being used to store information for undoing earlier parts of a transaction if a later part fails. I'm assuming the fact this part of the code ignores TEMP_STORE is an over site? _2007-Aug-13 15:03:19 by drh:_ {linebreak} The TEMP_STORE compile-time option only changes the storage for temporary database files. The statement journal is not a databaes file and thus does not come under the control of TEMP_STORE. There is currently no mechanism to force the statement journal into memory instead of onto disk. I will reclassify this ticket as a "feature request". ---- _2007-Aug-22 10:42:50 by anonymous:_ {linebreak} Okay, thank you. #e8e8bd 2582 doc active 2007 Aug anonymous 2007 Aug anonymous 5 5 documenation clarification docs for topic `Set A Busy Timeout` int sqlite3_busy_timeout(sqlite3*, int ms); http://sqlite.org/capi3ref.html#sqlite3_busy_timeout the wording "The handler will sleep multiple times until at least "ms" milliseconds of sleeping have been done" implies it will wait the total amount regardless of the lock status, it should perhaps indicate in the same sentence that it will exit early if the lock becomes available. #f2dcdc 2580 code active 2007 Aug anonymous 2007 Aug anonymous 1 2 Can't open a query if text to search is Greek for example: SELECT * FROM mytable WHERE mycolumn LIKE '%some greek text%' I get wrong results, using the 3.4.2 version. No problem instead using other earlier version. I tested only in Windows. #e8e8bd 2567 build active 2007 Aug anonymous 2007 Aug 3 2 Build fails to install I compile under MinGW with Msys. A build error occurs during 'make install'. After checking the makefile. The 'install' target depends on 'sqlite3', when it should be 'sqlite3$(TEXE)'. The workaround is, after configure, edit makefile for target install, and replace 'sqlite3' with 'sqlite3${TEXE}' where needed. I did not have this problem with 3.3.17. I assume this could be fixed just by fixing the configure to produce correct makefile. _2007-Aug-12 04:41:12 by anonymous:_ {linebreak} Do you have a patch? #e8e8bd 2566 build active 2007 Aug anonymous 2007 Aug 2 1 fts2 broken after vacuum Hi there, I'm testing your database and I'm having problems with fts2: --------- sqlite> select * from distB where distB match "MARIANO"; Assertion failed: *pData!='\0', file fts2amal.c, line 16790 This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information. --------- Steps: 1) Create a new .db 2) Import data in new distA table 3) Import data in new distB table 4) Create a new distC virtual table (dts2) 5) insert into distC (rowid, f1, f2, f3) select rowid, f1, f2, f3 from DistB Everything working like a charm until here!!! The fts2 works very well, but after 6) vacuum; the fts seems broken... doing a select throws the error I paste at the post of the topic If you want the .db file I can send it to you (607MB) Thanks.- #f2dcdc 2558 code active 2007 Aug anonymous 2007 Aug 2 3 Multiple JOIN USING() gives incorrect results I'm having a problem joining multiple tables with USING. It appears to work, but the results are incorrect. Here is an example to illustrate the problem. I believe the three SELECT statements should be equivalent, but they produce three different results. .header on .mode column CREATE TABLE Main (pk INTEGER PRIMARY KEY, name VARCHAR); CREATE TABLE OptA (pk INTEGER PRIMARY KEY, alpha VARCHAR); CREATE TABLE OptB (pk INTEGER PRIMARY KEY, beta VARCHAR); INSERT INTO Main VALUES (1, 'One'); INSERT INTO Main VALUES (2, 'Two'); INSERT INTO Main VALUES (3, 'Three'); INSERT INTO Main VALUES (4, 'Four'); INSERT INTO OptA VALUES (1, 'Alpha1'); INSERT INTO OptA VALUES (4, 'Alpha4'); INSERT INTO OptB VALUES (2, 'Beta2'); INSERT INTO OptB VALUES (4, 'Beta4'); SELECT * FROM Main LEFT JOIN OptA USING (pk) LEFT JOIN OptB USING (pk); SELECT * FROM Main LEFT JOIN OptB USING (pk) LEFT JOIN OptA USING (pk); SELECT Main.pk, name, alpha, beta FROM Main LEFT JOIN OptA ON Main.pk = OptA.pk LEFT JOIN OptB ON Main.pk = OptB.pk; Joining Main, OptA, and OptB omits Beta2: pk name alpha beta ---------- ---------- ---------- ---------- 1 One Alpha1 2 Two 3 Three 4 Four Alpha4 Beta4 Joining Main, OptB, and OptA omits Alpha1: pk name beta alpha ---------- ---------- ---------- ---------- 1 One 2 Two Beta2 3 Three 4 Four Beta4 Alpha4 Only by using ON instead of USING do we get the correct results: pk name alpha beta ---------- ---------- ---------- ---------- 1 One Alpha1 2 Two Beta2 3 Three 4 Four Alpha4 Beta4 I think this is basically the same issue as ticket #1637, but it's a more serious example. In that one, the query simply failed to compile. In this case, it seems to work, but gives you the wrong results. I've also tried this script in PostgreSQL 8.0.13. All three queries give (the same) correct results. _2007-Aug-08 17:34:27 by anonymous:_ {linebreak} The problem is that SQLite is transforming SELECT * FROM Main LEFT JOIN OptA USING (pk) LEFT JOIN OptB USING (pk); into SELECT Main.pk, name, alpha, beta FROM Main LEFT JOIN OptA ON Main.pk = OptA.pk LEFT JOIN OptB ON OptA.pk = OptB.pk; Here is a workaround to this bug that makes use of a subquery: select * from (SELECT * FROM Main LEFT JOIN OptA USING (pk)) LEFT JOIN OptB USING (pk); Conceivably all LEFT JOIN chains could be transformed into the above form, but that would decrease performance due to the intermediate result set of the subquery. Having it work without the subquery is tricky since sqlite must deduce that the last USING (pk) is equivalent to the first pk in the chain of joined tables, namely Main.pk, and not OptA.pk. Joe Wilson #e8e8bd 2555 new active 2007 Aug anonymous 2007 Aug 1 1 FTS index without original text Is it possible to build FTS index without storing original text? I want to use fts index without features of snippets etc. I just want to find ID of the record not the content of indexed phrase. I suppose that the table myname_content stores this content. I have tried to update all columns of myname_content and set its values to “xyz” (without one column in which I store ID of the record). After this operation FTS search works good, but unfortunately the table isn’t smaller (I cant’t use vacuum on FTS tables). Is there any other way to have pure text indexes without source level changes? #f2dcdc 2547 code active 2007 Aug danielk1977 2007 Aug 5 3 Changing db encoding of an attached db can confuse shared cache mode. This is quite obscure, but in shared-cache mode: 1) Open db A, attach empty db B. 2) Using another connection from the same thread, set the encoding of B to be different from that of A. Add some data to B. 3) Using the original connection, access database B. It assumes the encoding of A (and therefore mangling any text data). The correct response is to return an error - "attached databases must use the same text encoding as main database". #e8e8bd 2488 new active 2007 Jul anonymous 2007 Jul 5 4 autosize on column output mode in sqlite3 program It would be nice if sqlite3 program has a autosizecolumn mode for displaying queries, because it truncates values, and to calculate and use .width size for each column is tedious. _2007-Jul-07 11:42:03 by drh:_ {linebreak} In order to do this, we would have to either run the query twice or load the entire result set into memory. Otherwise, there would be no way to determine the longest element of each column. Neither approach seems attractive for large and complex queries. ---- _2007-Jul-28 07:05:56 by anonymous:_ {linebreak} every query should internally detect the longest column sizes and a new command should enable the user to set these values for any repetition or similar queries. At the end the queries run twice or more but only the first trial would have cause irritations on output. And this solution should be easy enough to implement it. {linebreak} Even the core of sqlite could calculate the maximum length of each column and a new API function could make this available. It would be really nice to get such an enhancement! ---- _2007-Jul-28 18:37:36 by anonymous:_ {linebreak} while you don't move to next row, sqlite doesn't know the contents, so it will be impossible to do, only if you cache the text entirely in memory, but this is ugly, imagine a 1GB recordset into RAM... ---- have you tried .mode tabs ? here, i have correct column width #f2dcdc 2545 code active 2007 Jul anonymous 2007 Jul 1 4 Group by returns table name with field name imaginate a table:create table test (
id INTEGER PRIMARY KEY,
name varchar(50) not null,
age integer not null
);
insert into test (name,age) values ('foo',22);
insert into test (name,age) values ('foo',23);
insert into test (name,age) values ('bar',22);
insert into test (name,age) values ('bar',35);
insert into test (name,age) values ('bar',72);
Now try this; sqlite> .headers on
sqlite> select test.name, test.age from test order by name;
name|age
bar|22
bar|35
bar|72
foo|22
foo|23
sqlite> select test.name, test.age from test group by name;
test.name|test.age
bar|72
foo|23
You see ? if i use "GROUP BY", the field name contains tablename. Because i use "SELECT test.name" and not "SELECT name". If i set an alias, i get alias, that's ok. The trouble appears to be very high on Copix (http://wwW.copix.org). We create some DAO (Data Access Objects) automatically. The "groupBy" method doesn't works with SQLite... Is this normal ? Mysql, PostgreSql, Oracle... doesn't need to create alias. _2007-Jul-31 15:54:52 by anonymous:_ {linebreak} There's probably 4 other tickets reporting this. I don't think it will get fixed. The workaround is to use aliases (AS "whatever") for the selected columns. ---- _2007-Jul-31 23:02:19 by anonymous:_ {linebreak} Ok, we have created a special support for SQLite. PS: I love this database :) Simple, nice, usefull, quick and easy Regards
#f2dcdc 2543 code active 2007 Jul anonymous 2007 Jul 1 1 Chinese charset not support?? when i create a table. the table name is " " (chinese) after this "alter table add column aaa text null" error why??/ thank you
#f2dcdc 2540 code active 2007 Jul anonymous 2007 Jul 5 4 Display dlopen() errors for errors when loading modules on Unix system If there is an error loading a module the message "unable to open shared library..." is displayed. In the Unix world the dlopen() error could be display to help diagnose the problem (e.g. missing external refs, etc). I have implemented it in our install of SQLite, I'm sure there is a Windows analog. Here is the patch for loadext.c. - Chris - Christopher Hailey Sr. Software Engineer Maxim Systems :::::::::::::: loadext.c.patch :::::::::::::: *** ./src/loadext.c.orig 2007-07-22 00:11:35.000000000 -0700 --- ./src/loadext.c 2007-07-22 00:12:07.000000000 -0700 *************** *** 292,298 **** handle = sqlite3OsDlopen(zFile); if( handle==0 ){ if( pzErrMsg ){ ! *pzErrMsg = sqlite3_mprintf("unable to open shared library [%s]", zFile); } return SQLITE_ERROR; } --- 292,298 ---- handle = sqlite3OsDlopen(zFile); if( handle==0 ){ if( pzErrMsg ){ ! *pzErrMsg = sqlite3_mprintf("unable to open shared library [%s]: %s", zFile,dlerror()); } return SQLITE_ERROR;
#f2dcdc 2530 code active 2007 Jul anonymous 2007 Jul 2 3 Unable to write to windows share, even with exclusive lock It has been mentioned that the file locking does not work on windows shared network drives (Samba or SMB drives from Windows or Linux). It seems that an exclusive lock should be a workaround for this problem if you need to write to a shared drive. Currently a more complicated locking is being attempted and failing on network drives. With an exclusive lock, SQLite could resort to simply holding a open write or append enabled file handle to the database as a more primitive locking system that is more likely to work on network drive. No other process could open the database but that would be expected with an exclusive lock. The following case should then function: grudy@gamma:~$ mount | grep Files //winserver/FileDump on /mnt/Files type cifs (rw,mand,noexec,nosuid,nodev) grudy@gamma:~$ touch /mnt/Files/i_have_write_permissions.txt; rm /mnt/Files/i_have_write_permissions.txt grudy@gamma:~$ sqlite3 /mnt/Files/foo.sqlite SQLite version 3.3.17 Enter ".help" for instructions sqlite> PRAGMA locking_mode = EXCLUSIVE; exclusive sqlite> create table bar (foobar); SQL error: database is locked sqlite>
#e8e8bd 2469 build active 2007 Jun anonymous 2007 Jul 1 1 test fails on Solaris I have a problem running the test suite on Solaris 9. Build was done using gcc 4.2.0. The build completes without error but many tests fail. I've created my own minimal test that exhibits the problem: set testdir [file dirname $argv0] source $testdir/tester.tcl db close file delete -force test.db test.db-journal sqlite db test.db do_test tdb-1 { execsql { PRAGMA auto_vacuum = 1; BEGIN; CREATE TABLE t1(a, b); } execsql { COMMIT; } } {} integrity_check tdb-2 finish_testWhen running this test I get the following output:
tdb-1... Ok tdb-2... Expected: [ok] Got: [{*** in database main *** List of tree roots: 2nd reference to page 1 Page 3 is never used}] Thread-specific data deallocated properly 1 errors out of 3 tests Failures on these tests: tdb-2This error happens on lots of, but not all, tests. I'm happy to do whatever is necessary to help debug this. Thanks, Tim. _2007-Jun-27 10:44:14 by anonymous:_ {linebreak} Further to this, it appears to be related to gcc 4.2.0. It works fine with gcc 3.4.6. ---- _2007-Jun-28 09:54:35 by anonymous:_ {linebreak} Further more, it doesn't appear to be specific to Solaris. The same problem occurs on Linux with gcc 4.2.0. So I guess the subject of this ticket should be changed to "build/test problems with gcc 4.2.0". This is probably a significant problem - the build completes find but the resultant code is broken. People may not notice this until it's too late. ---- _2007-Jun-28 12:24:05 by drh:_ {linebreak} I installed gcc 4.2.0 on my SuSE linux i686 desktop and built test harnesses under three different configurations: gcc420 -g -O0 -Wall -fstrict-aliasing gcc420 -g -O3 -Wall gcc420 -g -O3 -fstrict-aliasing -fomit-frame-pointer The first two configurations used separate source files. The third configuration was built using the amalgamation. I ran the "quick" test under all configurations. All tests ran to completion with no errors. ---- _2007-Jun-28 13:22:20 by anonymous:_ {linebreak} Two ideas: 1. Compile with gcc 4.2.0 using -O0 instead of -O2 and see what happens. Disable any other optimizations you may have. 2. Run truss with full read/write buffer display on the gcc 3.4.6 compiled testfixture running your simple test case and compare its output to the gcc 4.2.0 compiled test case. ---- _2007-Jul-01 19:00:40 by anonymous:_ {linebreak} I've done tests with optimisation, and this appears to tickle the problem. With no optimisation, -O, -O0, -O1 and -03 it works. With -O2 and -Os it's broken. I was compiling with -O2 when I submitted the initial report. Tim. ---- _2007-Jul-01 19:54:54 by drh:_ {linebreak} I can reproduce the problem now on Linux when compiling as follows: gcc420 -g -O2 -Wall ---- _2007-Jul-01 21:50:42 by drh:_ {linebreak} This appears to be a bug in GCC 4.3.0. A work-around is to compile with the -fno-tree-vrp option. GCC appears to miscompile a single loop within the logic that implements the integrity_check PRAGMA. The code that gets miscompiled is in the file vdbe.c (lines numbers added): 4308 for(j=0; j
RCS file: /sqlite/sqlite/src/vdbe.c,v retrieving revision 1.636 diff -u -3 -p -r1.636 vdbe.c --- src/vdbe.c 1 Jul 2007 21:18:40 -0000 1.636 +++ src/vdbe.c 21 Jul 2007 19:10:13 -0000 @@ -4306,7 +4306,8 @@ case OP_IntegrityCk: { pnErr = &p->aMem[j]; assert( (pnErr->flags & MEM_Int)!=0 ); for(j=0; ju.i; } aRoot[j] = 0; popStack(&pTos, nRoot);