Small. Fast. Reliable.
Choose any three.
Key: Priority 1 Priority 2 Priority 3+ Under Review
# Status Created By Subsys Due Date SCR Assigned Svr Pri Title  
Description
Remarks
 
154 active 2002 Sep drh Pager     drh 3 3 Prohibit links on database files. edit
If a database file is aliased using either hard or symbolic links, it can happen that an aborted transaction will not roll back correctly.

Consider this scenario. The database file is named both a.db and a.db. Application one opens a.db and starts to make a change. This creates a journal file a.db-journal. But application one crashes without completing the transaction. Later, application two attempts to open the database as b.db. App two looks for a journal file to rollback, but it thinks the journal should be named b.db-journal. So it fails to see the a.db-journal that app one left and fails to rollback the transaction.

The only way I can think of to prevent this kind of thing it to refuse to open any database file that contains two or more hard links and to refuse to open a file through a symbolic link.

2004-Mar-16 20:46:17 by anonymous:
What if the journal file name wasn't based on the database name, but instead was based on the starting inode of the database file? For instance, "journal-10293" would be used if the starting inode for the associated database file was 10293.


2004-Mar-20 17:17:41 by anonymous:
Using Inode-numbers to solve this problem is a dangerous proposition, as disk defragmenters can alter the inode the db starts at in between a crash and a subsequent roll-back attempt.


2007-Jun-05 03:57:17 by anonymous:
On unix, you can use ftok() to solve this problem. It guarantees to return the same key for all paths to the same file, including symbolic and hard links.

I have to experience programming in Windows, but have no doubt a similar function call exists in that API.

 
301 active 2003 Apr anonymous Unknown       2 3 Can't acquire lock for database on Mac OS X AppleShare volume edit
I'm using SQLite on Mac OS X 10.2.5. If I try to do a SELECT from a database that resides on an AppleShare volume (from my code or from sqlite), SQLite says that it's locked, even if no other process is using it. It appears that sqliteOsReadLock always returns SQLITE_BUSY for files on AppleShare volumes. I've temporarily solved the problem here by disabling SQLite's locking code and implementing higher-level protections in my application.

You may find this link helpful: http://developer.apple.com/technotes/tn/tn2037.html

2004-Mar-22 11:21:56 by anonymous:
under OSX fcntl returns ENOTSUP (45) = Operation Not Supported when trying to open a DB on AFP or SMB. Local HD is fine (HFS) as well as on USB devices HFS and UFS. Even with the recommendations from Apple's technote tn2037 the problem persists.
 
369 active 2003 Jun anonymous         3 2 Testsuite fails on btree-1.1.1 (Mac OS X, SQLite 2.8.4) edit
On Mac OS X the testsuite fails:

  btree-1.1.1..../src/btree.c:2687: failed assertion `pPage->isInit'
  make: *** [test] Abort trap  

SQLite version: 2.8.4, OS Version: 10.2.6

Obtained same result. Mac OS X 10.2.6, Developer Tools Dec 2002, SQLite 2.8.5.
Shared libraries are busted on Macs. As far as I can tell, this appears to be Apple's fault. Until a workaround is devised, do not attempt to compile using shared libraries. Add the --disable-shared option to the configure script:

    ../sqlite/configure --disable-shared


On 2.8.5+, this shows up on 2689. Also, configure does not allow the use of --disable-shared (probably requries a fix in the configure scripts). On a G5 in 10.3.2, this error shows up as a Bus Error. Builds work fine otherwise. This issue may be related to the warnings received in src/test1.c thru src/test4.c and in src/tclsqlite.c regarding Tcl_SetVar, Tcl_GetInt, Tcl_GetBoolean, Tcl_GetIndexFromObj. All warnings are regarding promotion of arguments to pointers of invalid type. oso2k/Louis


2004-Feb-11 22:57:17 by anonymous:
I did some quick testing of 2.8.12 on the machines I have available to me. In general, there seems to be more warnings than I remember (I believe I was testing 2.8.9 from cvs before it went live).

  • G3 700MHz/640MB iBook 10.2.8
    Same results as we last spoke. Fails make test at:
    btree-1.1.1

  • Dual G4 800MHz/1.25GB 10.2.8
    Same results as we last spoke. Fails make test at:
    btree-1.1.1

  • G5 1.6GHz/1.25GB 10.3.2
    Something really weird happens here. There is no longer a bus error. Right after make test gets past bigfile-1.1, the machine seems to enter an infinite loop or something.
 
575 active 2004 Jan anonymous VDBE     drh 3 3 pragma (default_)temp_store implementations seems incomplete edit
This problem is a conflict between documented behaviour and actual behaviour, and could fall in the 'Documentation' category as well.

There seems to be a problem with 'pragma default_temp_store'. In pragma.c code exists to handle it, and that code stores the provided value in Cookie 5 (as VDBE instruction argument; that is the sixth metadata integer, and would correspond to meta[6] in sqliteInitOne() in main.c).

However, the code loading a database (the aforementioned sqliteInitOne() in main.c) never looks at that value, and the setting is ignored. (Also, Vacuum doesn't seem to copy it.)

A related problem is that using the default_temp_store or temp_store pragma's doesn't work as advertised, at least not in the precompiled commandline tool sqlite.exe: you will always get the following error, even if you use the pragma at init time:

   SQL error: The temporary database already exists - its location cannot now be changed

Trying to set the flag (the value at offset 0x50 in a database file) to 2 (attempting to force an in-memory database for temp tables) with a hex editor has only partial success: the handcrafted value is reported by pragma default_temp_store; but typing .databases still shows a file name for the temporary data base, and using the filemon tool (a windows file activity monitor, downloadable from www.sysinternals.com) shows that the temp file is actually accessed when giving a 'create temp table' command (not surprising, if there is no code to actually ever initialize the db->temp_store from the Cookie).

If for some reason it is infeasible to circumvent the issue that the temp table will always be open before executing the pragma, I suggest changing the semantics of pragma default_temp_store to only change the default (as stored in the file), but not change the current value. This would allow executing pragma default_temp_store even while a temp table is open (though its effect will only be visible when the database is opened again).

Note that this issue has a few documentation issues:

  • lang.html suggests that pragma default_temp_store and pragma temp_store are currently working. At least in the commandline tool they aren't (I didn't make a dedicated test program to see if the problem already exists at the C-API level)
  • fileformat.html doesn't document the location where the temp_store flag is stored. In fact, I consider the fact that the fifth meta value (meta[5] a.k.a. Cookie 4) is seemingly not used anywhere slightly suspicious.
  • the number of metadata values is documented inconsistently in fileformat.html: in one place it mentions there are 6 values including the two leading values (which makes 4 metavalues), a bit later 9 metavalues are mentioned...
 
608 active 2004 Feb anonymous         3 3 Problem with "pragma show_datatypes = on" and busy timeout edit
When a busy timeout is set, pragma show_datatypes = on and SQLite sleeps some time on the lock, no datatypes are passed to the exec callback function.

The attachment is an archive with a Makefile, a shell script and a program that reproduce the error.

2004-Feb-12 21:05:28 by anonymous:
This problem breaks the auto-typing feature of PySQLite when a busy timeout is used.
 
627 active 2004 Feb anonymous         3 3 sqliteRunVacuum returning wrong code? edit
The last 3 lines of sqliteRunVacuum, as of the version checked in on Feb 12 2004, are:

  if( rc==SQLITE_ABORT ) rc = SQLITE_ERROR;
  if( sVac.rc!=SQLITE_OK ) rc = sVac.rc;
  return sVac.rc;

It seems suspicious to set a local variable, rc, that one is never going to use again. I suspect that the last line should be

return rc;

2004-Feb-27 00:54:03 by anonymous:
The fix by check-in 1271 still doesn't look right to me. If one of the execsql calls returns SQLITE_CANTOPEN (which I have seen happen), then rc will be SQLITE_CANTOPEN and sVac.rc will be 0, and sqliteRunVacuum will return 0.
 
684 active 2004 Apr anonymous Unknown       3 2 Incorrect function result type when using SQLITE_ARGS edit
I registered a function using the SQLITE_ARGS return type. I then execute the statement "select test('sample')". The type information returned from sqlite_step in the pazColName is incorrectly reported as "NUMERIC". If I use a "0", specifying the first column, instead of SQLITE_ARGS when registering the function, the return value is correctly set to "TEXT".
 
685 active 2004 Apr anonymous CodeGen       1 3 SELECT from a VIEW with GROUP BY edit
When you SELECT from a VIEW (which is having a GROUP BY statement) and try to apply another GROUP BY statement you get:

  $ sqlite ../../db/main.db
  SQLite version 2.8.13
  Enter ".help" for instructions
  sqlite> .dump prod_elem_totals
  BEGIN TRANSACTION;
  CREATE VIEW prod_elem_totals AS
    SELECT
      pe.elem_id AS elem_id, p.prod_id AS prod_id, e.name AS name,
      p.name AS p_name, pe.count AS count,
      SUM(b.count) / pe.count AS p_max,
      SUM(b.count) AS total, SUM(b.price * b.count) / SUM(b.count) AS price,
      e.min AS min
    FROM
      products AS p, elements AS e, batches AS b, prod_elems AS pe
    WHERE
      p.prod_id = pe.prod_id AND
      pe.elem_id = b.elem_id AND
      pe.elem_id = e.elem_id
    GROUP BY
      p.prod_id,
      pe.elem_id
    ORDER BY
      e.name;
  COMMIT;
  sqlite> SELECT * FROM prod_elem_totals GROUP BY elem_id;
  sqlite: src/select.c:1775: flattenSubquery: Assertion `p->pGroupBy==0' failed.
  Aborted

It seams it doesn't matter which column I GROUP BY. I can prepare a full test case if needed.

Maybe somehow connected with #678.

After further investigation I found that when I add a aggregate function like "SUM (count * 10) AS min" it works...

 
691 active 2004 Apr anonymous Unknown     drh 1 1 OS X File Sharing edit
Hello Sir:

This ticket may be considered a duplicate of ticket #301. I am unable to access SQLite databases from HFS or SMB network shares when using Mac OS X (10.3.3) as a client.

The more technical aspects of the problem are explained well in ticket #301.

I am using SQLabs SQLite plugin for RealBasic 5.5, and would like to use SQLite exclusively as my DB.

I am concerned that the original ticket was submitted approximately one year ago. So I am submitting this to see if this issue is being addressed, and if there is a timetable set for its resolution.

Thank you,

Tony Dellos Milwaukee WI

 
698 active 2004 Apr anonymous Unknown       1 3 .mode list - not going to next line edit
To create a comma delimited output file:
------------------------------------------------
C:\SQLite>sqlite locate.db
SQLite version 2.8.13
Enter ".help" for instructions
sqlite> .mode list
sqlite> .separator ", "
sqlite> .output data.cdf
sqlite> select * from parts;
sqlite> .quit

That should create a text file of something like this:
1st rec field 1, 1st rec field 2, 1st rec field 3, 1st rec field 4
2nd rec field 1, 2nd rec field 2, 2nd rec field 3, 2nd rec field 4
3rd rec field 1, 3rd rec field 2, 3rd rec field 3, 3rd rec field 4

but it does not provide a line break after each record, so the output looks like this:

1st rec field 1, 1st rec field 2, 1st rec field 3, 1st rec field 42nd rec field 1, 2nd rec field 2, 2nd rec field 3, 2nd rec field 43rd rec field 1, 3rd rec field 2, 3rd rec field 3, 3rd rec field 4

Each record is butted up against the previous record, without even a space. This is inconsitant with the instruction on how it is supposed to work, via this page:

http://www.sqlite.org/sqlite.html

Also, can you please refer me to somewhere that would explain how I can use SQLite with a batchfile, EG: using a batchfile to add a record, delete a record, query, Etc...

Thanks,

Tom

 
700 active 2004 Apr anonymous VDBE       2 3 Solaris-sparc segfaults on sum() edit
On Solaris (sparc) trying to do a sum() (sometimes) SEGVs: this is because the result is placed in a chunk of memory which is allocated as a char * and therefore isn't aligned to 16-byte boundaries (which SPARC-Solaris seems to want).

One fix for this which seems to work for me is to change vdbeInt.h:118 to

  char zShort[NBFS]  __attribute__ ((__aligned__(16)));  /* Space for short strings */

and change sqlite_aggregate_context to assign p->pAgg to zShort rather than z (since z is malloc()ed you can't align it) - I don't know if this would cause problems elsewhere though.

2004-Apr-22 16:10:37 by dougcurrie:
malloc() should always return memory aligned for any purpose; I don't think this is the problem.

Looking at the function sqlite_aggregate_context though, I wonder:

  • where is p->z initialized?
  • where is p->pAgg sqliteFree()d?
  • what happens when sqlite_aggregate_context and sqlite_set_result_string share Mem.zShort?


2004-Apr-23 09:58:03 by anonymous:
> malloc() should always return memory aligned for any purpose; I don't think this is the problem.

I didn't make it clear: I'm getting a Bus Error, not just a normal SEGV.

There are several places mentioned on the web which suggests that solaris' malloc() aligns memory to 8-byte boundaries, while (on 64-bit, I assume) a double is 128 bits... however you're correct, the manpage does insist that all malloc()s are aligned to data large enough for any purpose. I suppose if s.z isn't even assigned at this point (but hasn't been cleared at initialisation) it might contain something completely non-aligned.

However I now can't reproduce the problem, although that's not to say that it means the thing isn't broken... I'll try and break it again and let you know.

As to your other point: the reason I posted was because of exactly this: I don't know the code well enough (I'd never heard of it until yesterday!) to be able to say whether .zShort could be used elsewhere at the same time as a sum() function.


2004-Apr-23 10:35:11 by anonymous:
Aha. A core file lying around may well help.


...
Program terminated with signal 10, Bus Error.

...
#0  0xff33d4a4 in sumStep (context=0x2a158, argc=203556, argv=0x3ac08)
    at src/func.c:421
421         p->sum += sqliteAtoF(argv[0], 0);
...
(gdb) list
416     static void sumStep(sqlite_func *context, int argc, const char **argv){
417       SumCtx *p;
418       if( argc<1 ) return;
419       p = sqlite_aggregate_context(context, sizeof(*p));
420       if( p && argv[0] ){
421         p->sum += sqliteAtoF(argv[0], 0);
422         p->cnt++;
423       }
424     }

....

(gdb) print &p.sum
$5 = (double *) 0x31b24
Now my basic maths (a hex 16-byte aligned number should end in 0, right?) says that somehow p.sum has become misaligned by 4 bytes.

sqlite_aggregate_sum simply assigns p->pAgg to p->s.z and returns p->pAgg, which means that p->s.z is misaligned also.

Further investigation makes more worrying reading:

(gdb) print *context
$18 = {pFunc = 0x75736572, s = {i = 0, n = 0, flags = 16, z = 0x0,
    r = 3.6586602629506839e-309,
    zShort = '\000' <repeats 11 times>, "\020\000\000\000\000\000\002ˇ\230", '\000' <repeats 11 times>}, pAgg = 0x10, isError = 0 '\000', isStep = 0 '\000',
  cnt = 172464}
Note that pAgg is 0x10 (!?) and s.z is 0. Something seriously unhappy going on there: I think it's likely there's some corruption going on due to some specific set of events which was being run.

I'll rerun the exact scenario and try again.


2004-May-03 02:27:25 by anonymous:
i have seen this exact same problem on sparc/solaris. my core looks exactly the same. it really does look like an alignment issue.


2004-Jul-12 13:09:26 by anonymous:
I had this exact problem on solaris 8 with gcc3.3.4 and well, every version of sqlite over 2.5.

Heres my solution, hopefully it will help someone with more time and IQ points to figure out the real problem

After forcing PTR_FMT to %x in test1.c (so i can run all the tests) I changed src/vbdeInt.h #define NBFS from 32 to 15 (one less than a long double on a sparc) thus forcing all long doubles to be malloced.

This allowed me to run all the tests (and my application) bus error free. 5 of the tests failed, which looks like a precision problem and seems harmless in my applications.

date-1.19... Expected: [2451545.00000116] Got: [2451545.00000] date-1.20... Expected: [2451545.00000012] Got: [2451545.00000] date-1.21... Expected: [2451545.00000001] Got: [2451545.00000]

expr-2.4... Expected: [0.525641025641026] Got: [0.52564102564] expr-2.5... Expected: [1.90243902439024] Got: [1.90243902439]


2004-Jul-17 12:26:31 by anonymous:
I found a better solution than my changing NBFS solution. from what little info i found, doubles in a structure are aligned to 8. so align zShort to 8 and it works with NBFS as 32. change PTR_FMT to %X instead of %x and it passes all the tests.
 
709 active 2004 Apr anonymous Unknown     drh 1 1 Unable to unregister or replace functions edit
I believe this started with 2.8.13, as it did work previously. There appears to be a show-stopper with unregistering or replacing existing functions. Specifically, if one tries to replace (or remove by passing nulls) one of the built-in functions, for example "like" or "upper", the function does not get replaced, and is in fact still called and available. The odd thing is that if you try to replace one of the functions with an underscore, such as "change_count", it works fine! This is causing problems as we replace a lot of the existing functions, and allow users to add and replace their own functions, which are now failing.
2004-Apr-26 20:56:19 by anonymous:
Alright, turns out this is due to mismatch in argument count when unregistering functions. It would be useful if we could unregister all instances of a function name, irregardless of argument counts.


2004-Apr-26 23:11:43 by anonymous:
Turns out there really is a bug... the problem is in the sqliteFindFunction function's matching of inexact argument counts, when being called from sqliteExprCheck with >0 argument count. This is the scenario. I override the "upper" function with my own, but first removing the old "upper", specifying 1 argument and null for the function. I then register a new "upper" with -1 for the argument count, and a valid function.

When sqliteFindFunction attempts to locate the upper function, It locates the new function, but because it is registered with -1, it tries to find a better match. It then runs into the original one, and because it has a null function pointer, it fails. This causes sqliteExprCheck to try again with -1 as the count, and since that matches, it reports an error of wrong_num_args.

Unfortunately this means there is no way to override an existing method (it would be good if we could just delete them, rather than override them, although I still think this behaviour with -1 is wrong).


2004-Apr-26 23:29:23 by anonymous:
I've applied the following patch in the sqliteFindFunction function, that I believe addresses the problem:

/* Change this 
  if( p && !createFlag && p->xFunc==0 && p->xStep==0 ){
    return 0;
  }
*/
/* To this */
  if( p && !createFlag && p->xFunc==0 && p->xStep==0 ){
    return pMaybe;
By returning pMaybe we provide a function that will work, while returning 0 if no variable argument function was found.
 
721 active 2004 May anonymous Shell     drh 3 2 empty .databases information edit
in file shell.c around line 570 the callback_data structure needs to be added:

  data.cnt = 0;

so the column widths are correctly used. or else, it will display empty lines.

this problem is visible after executing one sql command

 
735 active 2004 May anonymous Shell       4 3 .sqliterc not processed if running on a driver other than C: edit
In shell.c there is a snippet that reads:

if (!home_dir) { home_dir = getenv("HOMEPATH"); /* Windows? */ }

The HOMEPATH environment variable does not include the drive letter and needs to be concatenated with the HOMEDRIVE environment variable.

2004-May-12 14:43:40 by anonymous:
That should read "drive" in the title, not "driver"
 
744 active 2004 May anonymous BTree     anonymous 2 2 make test seg faults on x86_64 Linux edit
I'm running the 64 bit version of Gentoo Linux on an AMD Opteron system. Ordinarily I'd install software with "emerge <package>" but "emerge sqlite" only gives me version 2.8.11. I downloaded the 2.8.13 source and did the usual ./configure; make; make test. The configure and make steps went OK but make test fails half way through:

bind-1.99... Ok

btree-1.1... Ok

btree-1.1.1... Ok

btree-1.2... Ok

btree-1.3... Ok

btree-1.4... Ok

btree-1.4.1... Ok

btree-1.5... Ok

btree-1.6...make: *** [test] Segmentation fault

The code was built with GCC 3.3.3.

As sqlite is a known 'emerge' option for 64 bit Gentoo I'm guessing sqlite is known to work on 64 bit platforms?

I didn't mark this as a severe error because in theory I should be able to create a statically linked executable on a 32 bit linux system and run this on the Opteron box. Haven't been successful at that yet however.

2004-May-25 04:35:32 by anonymous:
It's pretty clear that sqlite has never been compiled on a 64-bit system, much less run. The test problems are fatal bugs caused by type conversions between 64-bit and 32-bit values, including truncating pointers and other sins. The fixes look quite involved.
 
754 active 2004 Jun anonymous Shell     drh 2 3 problem opening a dbfile in the upper directory (../dbname) edit
there is a problem in the calculation of a full path name based on a relative path name in an uproot location (../).

i have fixed this during my porting to dos, and the relevant diff is at http://www.sqlite.org/cvstrac/tktview?tn=524.

best regards

alex

 
783 active 2004 Jun anonymous Unknown       3 3 Build on MAC with -DSQLITE_DEBUG=1 compile error edit
MacOS 10.3.4 gcc 3.3

Compileing with -DSQLITE_DEBUG=1 gives following error

./libtool --mode=compile gcc -g -O2 -DOS_UNIX=1 -DHAVE_USLEEP=1 -DSQLITE_DEBUG=1 -I. -I../sqlite/src -DTHREADSAFE=0 -c ../sqlite/src/os_unix.c gcc -g -O2 -DOS_UNIX=1 -DHAVE_USLEEP=1 -DSQLITE_DEBUG=1 -I. -I../sqlite/src -DTHREADSAFE=0 -c ../sqlite/src/os_unix.c -fno-common -DPIC -o .libs/os_unix.o ../sqlite/src/os_unix.c: In function `sqlite3OsRead': ../sqlite/src/os_common.h:31: error: inconsistent operand constraints in an `asm' make: *** [os_unix.lo] Error 1

 
798 active 2004 Jul anonymous Unknown       1 1 Unable to run tests on Tru64 bit Linux platform edit
I was able to compile SQLite 3.0.2 on a RedHat 64-bit Linux system; however, when running the tests I would get a segmentation fault when executing a blob test. I was wondering if anyone has attempted to build SQLite for a 64-bit architecture and run all tests successfully. If so I was hoping to get any configuration parameters needed.
 
841 active 2004 Aug anonymous Unknown Pending     3 2 inner group by query isn't honored by outer count(*) aggregate edit
  CREATE TEMP TABLE A(a int NOT NULL, b int NOT NULL, c int NOT NULL);
  INSERT INTO A VALUES (1, 1, 1);
  INSERT INTO A VALUES (1, 2, 1);
  INSERT INTO A VALUES (2, 1, 1);

  -- typical behaviour is for this to behave like the DISTINCT query below
  -- but instead it shows a=1 as having occured twice (but it was grouped in the inner query)
  SELECT a, count(*) FROM (
    SELECT a, c FROM A GROUP BY 1, 2) GROUP BY a;
  Result:
  2|1
  1|2

  -- shows a=1 as having occured once (correctly)
  SELECT a, count(*) FROM (
    SELECT DISTINCT a, c FROM A) GROUP BY a;
  Result:
  2|1
  1|1

  -- the top query performs better, which is why I am reporting this bug
2004-Aug-08 18:42:00 by drh:
SQLite ignores the ORDER BY clause if there are no aggregate functions.
 
923 active 2004 Sep anonymous   Pending   anonymous 3 2 Missing quotes in 2.8.15 .dump cause data loss when loading in sqlite3 edit
When converting a database by means of the command:

  sqlite old.db .dump | sqlite3 new.db

the content of char/varchar fields is dumped by sqlite without quotes (e.g. 00001) and then when reloaded by sqlite3 it looses the heading zeroes (i.e. becomes '1', which is a really different thing for an alphanumeric field).

This could be solved by a new release (sqlite 2.8.16 ?) which add quotes to alphanumeric fields (as sqlite3 does), or by a filter script that adds the quotes to the sqlite2 .dump output (I used a quick and dirty perl script to fix my dump...).

2005-Jul-11 20:08:11 by anonymous:
This does not appear to be solved in sqlite 2.8.16.
 
969 new 2004 Oct anonymous TclLib Pending     3 3 PRAGMA empty_result_callbacks not working in tclsqlite-3.0.8.so part 2 edit
Referencing ticket # 967, I stated that it was the tclsqlite code that was not functioning properly, not the ./sqlite3 executable.

Here is a script that you can use to reproduce the issue. As you can see the results are quite different.

#----------->
load ./lib/tclsqlite-3.0.8.so sqlite3
puts [info patchlevel]
sqlite3 db :memory:
db eval "create table t1(a,b);"
puts "before 3.0.8 select, no pragma"
db eval "select * from t1;" x {
        puts "x(*) = $x(*)"
}
db eval "PRAGMA empty_result_callbacks=1"
puts "before 3.0.8 select, yes pragma"
db eval "select * from t1;" x {
        puts "x(*) = $x(*)"
}
db close
load ./lib/tclsqlite-2.8.15.so Tclsqlite
sqlite db2 :memory:
db2 eval "create table t1(a,b);"
puts "before 2.8.15 select, no pragma"
db2 eval "select * from t1;" x {
        puts "x(*) = $x(*)"
}
db2 eval "PRAGMA empty_result_callbacks=1"
puts "before 2.8.15 select, yes pragma"
db2 eval "select * from t1;" x {
        puts "x(*) = $x(*)"
}
db2 close
puts "done"
# <-------------------

and the results:

$ tclsh test_sqlite.tcl
8.4.3
before 3.0.8 select, no pragma
before 3.0.8 select, yes pragma
before 2.8.15 select, no pragma
before 2.8.15 select, yes pragma
x(*) = a b
done
2006-May-16 18:29:44 by anonymous:
This is still a problem in the 3.3.5 version of the tclsqlite library. The tclsqlite.c code never calls the callback code on empty results when PRAGMA empty_result_callbacks=1 is set.
 
1026 active 2004 Dec anonymous Unknown Pending     3 3 sqlite automake sqlite3.pc file does not have version information edit
When the configure of sqlite has been done, the sqlite3.pc file does not have information in the Version: section. This means there's no way to check for versions in other autogen/configure files concerning the sqlite version in the system, style:

PKG_CHECK_MODULES(SQLITE, sqlite >= 3.0.3, AC_MSG_ERROR([$SQLITE_PKG_ERRORS]))

 
1030 active 2004 Dec anonymous Shell Pending     2 3 impossible to import a file and do other things in the same invocation (strongly affects scripting) edit
The sqlite2 shell can be scripted in 2 ways:

        1.  sqlite mydb 'commands'
        2.  commands | sqlite mydb

The first form has a bug, which I have not submitted, and which also occurs in sqlite3:

        o dot commands are not fully intermixable with sql commands

It will also fail for long command lines.

To work around this, it is necessary to use the second form of scripting. However:

        o for entering data via COPY or .import, a separator is
          necessary to allow sql after the data
        o COPY has \. as a separator

Here is the sqlite3 bug:

        o .import has no separator that I know of

The only possible workaround seems to be to call the executable more than once. However:

        o this makes :memory: databases impossible

Therefore, I said this is major with a workaround. The workaround is to use temporary files instead of memory databases, and:

        o inconveniently kludge the sqlite2 behavior using multiple
          calls to the executable

However, files are far slower than memory, making the use of sqlite as an awk-like filter less attractive than it was with sqlite2.

Here are 2 solutions to fix the problem, both of which are desirable:

        o allow dot commands and sqlite commands to be intermixable on
          the command line such as with "sqlite :memory:
          '.show;.import ...;select ...;.import ...; select ...'" 
        o allow a separator for .import

Here are some related bugs, also not submitted:

        o .separator is overloaded to mean input and output
          separators, but for scripting it would be useful to have
          them separate
        o csv mode and tabs mode are lightly documented.  what are the
          exact syntaxes for them (line continuation, quoting, etc.),
          and what is the difference between tabs mode and .separator
          set to tab?
        o it might also be useful to have a .with command so that you
          can do .with .separator ","; .import ...; .endwith to
          effectively emulate (let ((...)) ...) in lisp.
          i.e. temporarily set a value to something without having to
          know what to set it back to when you are done.

Sqlite rocks.

Thanks.

 
1053 active 2004 Dec anonymous Pager Pending     3 3 SQLITE_IOERR and strange rollback when db is busy edit
Environment on which bug was found:
Windows XP, both SP1 and SP2, on different computers. The SQLite library was built using the precompiled source from the download page (as static library).

Description of bug scenario:

One process performs very long reads from a db (multiple joins, so the cartesian product is very large, and the reader needs a while to complete).

Another process performs a BEGIN TRANSACTION , then executes lots of INSERT INTO ... VALUES .
At some point, this process will end up in sqlite3pager_get, when it tries to read some page from the database file (the main file, not a temp file or a journal). It detects that the page is not in the page cache (it ends up in the 'else' branch of if( pPg==0 ) ). It runs down to the block of code covered by the following comment:

     /* Write the page to the database file if it is dirty.
     */

In this block, pager_write_pagelist( pPg ) returns with SQLITE_BUSY. As a consequence, the changes are rolled back and SQLITE_IOERR is returned.

And here seems to be the problem: First, the database file is locked, so I don't understand why the SQLITE_BUSY value isn't propagated back to the caller. If SQLITE_BUSY would be returned, then the application could restart the command. Seconds, sqlite3VdbeHalt decides to perform a sqlite3BtreeRollbackStmt, so only the last command should be rolled back. However, this is not what happens! In fact, all commands back to the beginning of the transaction are rolled back; the transaction, however is not closed. Doesn't this violate the default rollback behaviour (roll back last command, keep transaction open)? As a consequence, even if the application would get SQLITE_BUSY, it couldn't properly react on it.

There are other places in sqlite3pager_get where SQLITE_IOERR are returned; I've not checked whether these can also be triggered by the db being locked or if they indicate serious problem.

I will attach the code I used to reproduce and track down the problem, together with a Visual Studio 2003 project.

If you extract the archive, on toplevel you will find the following:

  • Reader: the directory containing the source for the reader
  • Writer: the directory contaiing the source for the writer
  • SQLite: A directory in which to place the precompiled source for windows users, which is used to build the library. If you want to use the provided project file with Visual Studio, just copy the source in there and everything will build with a single mouse click.
  • BugDemo.sln: The Visual Studio project file.
  • bugdemo.sql: The SQL statements used to create the test database.

How to reproduce:

  • Create a database using bugdemo.sql
  • Adapt reader.cpp and writer.cpp to include the sqlite3 headers, and set the define at the top of the files to the path of the test database.
  • Compile everything.
  • Start the reader.
  • Start the writer, and wait until it reports an error (for me, it takes < 30 seconds).

I tried to keep the source portable, so it shouldn't be too hard to make it compile on Unix.

 
1056 active 2004 Dec anonymous Shell Pending     3 3 test pragma-9.4 fails during second pass in "make fulltest" edit
During a "make fulltest" run, the pragma tests appear to run twice. On the first run, pragma-9.4 runs properly. On the second run, it gives an error:

  pragma-9.4...
  Expected: []
       Got: [/Volumes/Local/Users/sqlite/test/bld]

(where the path listed is the build directory for this build of sqlite).

The pragma-9.4 test is a recent addition to sqlite.

This is currently the only failure I'm seeing in a "make fulltest" of the current cvs tree on Mac OS X when the build/test directory is on a hard drive.

  1 errors out of 68411 tests
  Failures on these tests: pragma-9.4
  make: *** [fulltest] Error 1
 
1058 active 2005 Jan anonymous BTree Pending     3 3 btree.c pageSize -> usableSize edit
Check-in [2125] was a fix for Ticket #1010, but left out some of the fixes proposed in the ticket. I'm not sure whether this was an oversight or an intentional omission. I tried to re-open the ticket, but those edits didn't persist (I'm not sure why).

Here are the remaining instances of pageSize that I believe should be changed to usableSize in btree.c:

  --- sqlite/src/btree.c.ORIG	Wed Nov 24 18:54:30 2004
  +++ sqlite/src/btree.c	Wed Nov 24 20:29:21 2004
  @@ -220,13 +220,13 @@
   /* The following value is the maximum cell size assuming a maximum page
   ** size give above.
   */
  -#define MX_CELL_SIZE(pBt)  (pBt->pageSize-8)
  +#define MX_CELL_SIZE(pBt)  (pBt->usableSize-8)

   /* The maximum number of cells on a single page of the database.  This
   ** assumes a minimum cell size of 3 bytes.  Such small cells will be
   ** exceedingly rare, but they are possible.
   */
  -#define MX_CELL(pBt) ((pBt->pageSize-8)/3)
  +#define MX_CELL(pBt) ((pBt->usableSize-8)/3)

   /* Forward declarations */
   typedef struct MemPage MemPage;
  @@ -1745,7 +1745,7 @@
     Pgno finSize;   /* Pages in the database file after truncation */
     int rc;           /* Return code */
     u8 eType;
  -  int pgsz = pBt->pageSize;  /* Page size for this database */
  +  int pgsz = pBt->usableSize;/* Usable bytes on each page */
     Pgno iDbPage;              /* The database page to move */
     MemPage *pDbMemPage = 0;   /* "" */
     Pgno iPtrPage;             /* The page that contains a pointer to iDbPage */
 
1063 active 2005 Jan anonymous   Pending     1 3 Lemon bug: Strings in rule code should not be interpreted edit
There are two related bugs in the lemon parser related to processing code snippets defined in rule actions.

Here is a simple grammar that demonstrates the problem:

%include { extern int line_number; extern const char *file_name; }

result(r) ::= TOKEN(s). { printf("BAD: Got a token on line '%d'\n", line_number); printf("BAD: \tFile = '%s'\n", file_name); r = s; }

The first bug is that the "%d" in the first printf is interpreted by the append_str function, when it shouldn't be, producing code that looks like: printf("BAD: Got a token on line '0d'\n", line_number); I believe that the solution is to have append_str() NOT do %d substitution when it is copying the code.

The second bug is that the "s" in the "%s" format is being interpreted as a symbolic name, producing code that looks like: printf("BAD: \tFile = '%yymsp[0].minor.yy0'\n", file_name);

I believe that the solution is to have translate_code() ignore symbolic names inside of quoted strings.

 
1078 active 2005 Jan anonymous   Pending     2 3 Lemon destructor bugs that don't affect sqlite edit
I found a few bugs Lemon's destructor handling code. I don't think that they affect sqlite, but the do affect other grammars.

- The code that collapses cases for default destructors erroneously assumes that all symbols have the same type.

- If a reduction rule doesn't have code, then the RHS symbols will not have their destructors called.

- The default destructor shouldn't be called on the auto-generated "error" symbol

- In the internal function "append_str", zero-length strings may be returned un-terminated.

I have some proposed fixes that I'll try to attach to this ticket.

2005-Jan-14 13:33:52 by drh:
Do you also have some test grammars? That would really be helpful.


2005-Jan-14 17:14:15 by anonymous:
Sure. Here is one grammar that will demonstrate the "Tokens leak when rule has no code" bug:

 
%token_type { char * }
%token_destructor 
{ 
    printf("Deleting token '%s' at %x\n", $$, (int)$$); 
    free($$); 
}
result ::= nt. 
nt ::= FOO BAR.
Running the following code against the grammar should theoretically show 2 allocations and two destructions. It won't though, unless you modify the rule for nt to have an empty body, like:
nt ::= FOO BAR. {}
char *mkStr(const char *s)
{
    printf("Allocating '%s' at 0x%x\n", s, (int)(s));
    return strdup(s);
}

int main(int argc, char **argv)
{
    void *parser = ParseAlloc(malloc);
    Parse(parser, FOO, mkStr("foo"));
    Parse(parser, BAR, mkStr("bar"));
    Parse(parser, 0, 0);
    ParseFree(parser, free);
    return 0;
}



2005-Jan-14 17:50:26 by anonymous:
Here is another test grammar. This one demonstrates (a) default destructors being called on the 'error' symbol, and (b) problems with default destructors being called on the wrong symbol type.

%token_type { char * }
%token_destructor { delete [] $$; }
%default_destructor { delete $$; }

%type result { int }
%destructor result { }
result ::= fooStruct barStruct. { }

%type fooStruct { Foo * }
fooStruct(lhs) ::= FOO(f). { lhs = new Foo(f); }

%type barStruct { Bar * }
barStruct(lhs) ::= BAR(b). { lhs = new Bar(b); }
Here is the code generated by lemon (with comments added & removed for clarity):

typedef union {
  ParseTOKENTYPE yy0;
  int yy4;
  Bar * yy5; 
  Foo * yy7; 
  int yy15;
} YYMINORTYPE;

static const char *const yyTokenName[] = { 
  "$",             "FOO",           "BAR",           "error",       
  "result",        "fooStruct",     "barStruct",   
};


static void yy_destructor(YYCODETYPE yymajor, YYMINORTYPE *yypminor){
  switch( yymajor ){
    case 1:
    case 2:
{ delete [] (yypminor->yy0); }
      break;
    case 3: /* error */
    case 5: /* fooStruct of type "Foo *"  */ 
    case 6: /* barStruct of type "Bar *"  */ 
#line 3 "typeBug.y"
{ delete (yypminor->yy5); }  /* Yikes! yy5 is a "Bar *" */
#line 308 "typeBug.c"
      break;
    case 4:
#line 6 "typeBug.y"
{ }
#line 313 "typeBug.c"
      break;
    default:  break;   /* If no destructor action specified: do nothing */
  }
}
 
1085 active 2005 Jan anonymous   Pending     2 3 pragma full_column_names and short_column_names still broken edit
the following statement :

  SELECT T1.*,D1.*
  FROM test T1,dt D1
  WHERE T1.id=D1.id

does not give "long" column names, even if full_column_names is ON.

But, the following does:

  SELECT T1.ID,D1.NAME
  FROM test T1,dt D1
  WHERE T1.id=D1.id

in other words, tablename prefix is applied only to explicit columns, not to "*" selected columns.

 
1100 active 2005 Feb anonymous   Pending     3 3 make test segfaults at capi2-7.12 on amd64 system edit
System is Gentoo Linux 2004.1 on Opteron processor; gcc v3.3.3, creating 64 bit binaries.

Here is a traceback:

  capi2-7.11... Ok
  capi2-7.11a... Ok
  capi2-7.12...
  Program received signal SIGSEGV, Segmentation fault.
  0x0000002a95b25830 in strlen () from /lib/libc.so.6
  (gdb) where
  #0  0x0000002a95b25830 in strlen () from /lib/libc.so.6
  #1  0x000000000043991c in sqlite3VdbeList (p=0x5aab60) at  src/vdbeaux.c:528
  #2  0x0000000000438737 in sqlite3_step (pStmt=0x13000a6023d0064)
     at src/vdbeapi.c:207
  #3  0x0000000000416a80 in test_step (clientData=0x13000a6023d0064, 
    interp=0x55a450, objc=0, objv=0x4) at src/test1.c:2070
  #4  0x0000002a956978fe in TclEvalObjvInternal () from  /usr/lib/libtcl8.4.so
  #5  0x0000002a956ba181 in TclCompEvalObj () from /usr/lib/libtcl8.4.so
  #6  0x0000002a956b9648 in TclCompEvalObj () from /usr/lib/libtcl8.4.so
  #7  0x0000002a956e4f66 in TclObjInterpProc () from /usr/lib/libtcl8.4.so
  #8  0x0000002a956978fe in TclEvalObjvInternal () from /usr/lib/libtcl8.4.so
  #9  0x0000002a956982d8 in Tcl_EvalEx () from /usr/lib/libtcl8.4.so
  #10 0x0000002a95698097 in Tcl_EvalTokensStandard () from /usr/lib/libtcl8.4.so
  #11 0x0000002a95698273 in Tcl_EvalEx () from /usr/lib/libtcl8.4.so
  #12 0x0000002a95698767 in Tcl_EvalObjEx () from /usr/lib/libtcl8.4.so
  #13 0x0000002a956e4a93 in Tcl_UplevelObjCmd () from /usr/lib/libtcl8.4.so
  #14 0x0000002a956978fe in TclEvalObjvInternal () from /usr/lib/libtcl8.4.so
  #15 0x0000002a956ba181 in TclCompEvalObj () from /usr/lib/libtcl8.4.so
  #16 0x0000002a956b9648 in TclCompEvalObj () from /usr/lib/libtcl8.4.so
  #17 0x0000002a956e4f66 in TclObjInterpProc () from /usr/lib/libtcl8.4.so
  #18 0x0000002a956978fe in TclEvalObjvInternal () from /usr/lib/libtcl8.4.so
  #19 0x0000002a956982d8 in Tcl_EvalEx () from /usr/lib/libtcl8.4.so
  #20 0x0000002a95698767 in Tcl_EvalObjEx () from /usr/lib/libtcl8.4.so
  #21 0x0000002a956e4a93 in Tcl_UplevelObjCmd () from /usr/lib/libtcl8.4.so
  #22 0x0000002a956978fe in TclEvalObjvInternal () from /usr/lib/libtcl8.4.so
  #23 0x0000002a956ba181 in TclCompEvalObj () from /usr/lib/libtcl8.4.so
  #24 0x0000002a956b9648 in TclCompEvalObj () from /usr/lib/libtcl8.4.so
  #25 0x0000002a95698815 in Tcl_EvalObjEx () from /usr/lib/libtcl8.4.so
  #26 0x0000002a9569c486 in Tcl_CatchObjCmd () from /usr/lib/libtcl8.4.so
  #27 0x0000002a956978fe in TclEvalObjvInternal () from  /usr/lib/libtcl8.4.so
  #28 0x0000002a956ba181 in TclCompEvalObj () from /usr/lib/libtcl8.4.so
  #29 0x0000002a956b9648 in TclCompEvalObj () from /usr/lib/libtcl8.4.so
  #30 0x0000002a95698815 in Tcl_EvalObjEx () from /usr/lib/libtcl8.4.so
  #31 0x0000002a9569ef7e in Tcl_IfObjCmd () from /usr/lib/libtcl8.4.so
  #32 0x0000002a956978fe in TclEvalObjvInternal () from /usr/lib/libtcl8.4.so
  #33 0x0000002a956ba181 in TclCompEvalObj () from /usr/lib/libtcl8.4.so
  #34 0x0000002a956b9648 in TclCompEvalObj () from /usr/lib/libtcl8.4.so
  #35 0x0000002a956e4f66 in TclObjInterpProc () from /usr/lib/libtcl8.4.so
  #36 0x0000002a956978fe in TclEvalObjvInternal () from /usr/lib/libtcl8.4.so
  #37 0x0000002a956982d8 in Tcl_EvalEx () from /usr/lib/libtcl8.4.so
  #38 0x0000002a956d1e41 in Tcl_FSEvalFile () from  /usr/lib/libtcl8.4.so
  #39 0x0000002a956d120e in Tcl_EvalFile () from /usr/lib/libtcl8.4.so
  #40 0x0000000000424641 in main (argc=2, argv=0x7fbffff138)
     at src/tclsqlite.c:1744
  (gdb)
2005-Feb-04 23:11:02 by anonymous:
Some more information from a gdb session (I added the printf at line 527 and recompiled; the line numbers below will be off by one from the stack trace above):

  (gdb) up
  #1  0x000000000043996d in sqlite3VdbeList (p=0x5aab60) at src/vdbeaux.c:529
  529         pMem->n = strlen(pMem->z);
  (gdb) list 525,535
  525
  526         pMem->flags = MEM_Static|MEM_Str|MEM_Term;
  527     printf("src/vdbeaux.c sqlite3VdbeList pOp->opcode= %d\n", pOp->opcode);
  528         pMem->z = sqlite3OpcodeNames[pOp->opcode];  /* Opcode */
  529         pMem->n = strlen(pMem->z);
  530         pMem->type = SQLITE_TEXT;
  531         pMem->enc = SQLITE_UTF8;
  532         pMem++;
  533
  534         pMem->flags = MEM_Int;
  535         pMem->i = pOp->p1;                          /* P1 */
  (gdb) p pOp->opcode
  $1 = 40 '('
  (gdb) p pMem->z
  $2 = 0x175002100200173 <Address 0x175002100200173 out of bounds>
 
1111 active 2005 Feb anonymous Unknown Pending     3 1 no such column error with a subselect as a table edit
Execute the following script. In 3.0.8, you get the following results:

  line 0
  line 2

in 3.1.1 beta, you get the following error:

  SQL error: no such column: bb.a

SCRIPT:

  create table test1 (a int);
  insert into test1 values (0);
  insert into test1 select max(a)+1 from test1;
  insert into test1 select max(a)+1 from test1;

  create table test2 (a int, b text);
  insert into test2 select a,'line ' || a from test1;

  select test2.b from (select test1.a from test1 where a%2 = 0) as bb   join test2 on bb.a = test2.a;
2005-Feb-21 17:48:20 by anonymous:
Tested in 3.1.3, issue still exists


2005-Jun-12 22:09:08 by drh:
Workaround:

  select test2.b 
  from (select test1.a as a from test1 where a%2=0) as bb
       join test2 on bb.a=test2.a;
 
1134 active 2005 Feb anonymous   Pending   drh 3 3 select command with a view and a condition gets no result edit
In a view with more than one tables it is no longer possible to use that view within a select command; the columns of the view are not found.
 
1142 active 2005 Feb anonymous Parser Pending   danielk1977 1 1 Column names edit
  create table a (id, x);
  create table b (id, y);
  insert into a values (1,1);
  insert into b values (1,2);
  select * from a inner join b;

column names returned: id,x,id,y How am I supposed to use such column names? Ouwey.

2005-Mar-15 07:28:43 by anonymous:
This bug breaks existing applications where 'SELECT rowid, * FROM table' was used to open any table and then precompiled statements were used to update the changed record by using 'UPDATE table SET <field1>=?, <field2>=? ... WHERE rowid=?'. We cannot update to sqlite v3.1.x because of this.


2005-Mar-15 08:36:31 by drh:
There are lots of people telling me the current behavior is wrong. But nobody has yet suggested what the correct behavior should be. Until I know what the correct behavior should be, there is little I can do to fix the problem.

Rather than just telling me the current behavior is wrong, please offer an explanation of what the correct behavior should be. What do Oracle and PostgreSQL do in the same situation?


2005-Mar-16 09:04:30 by anonymous:
Dr. Hipp, very sorry for the confusion I've caused! My comments were actually targeted at bug #1141! I've included the same remarks there.

As for this bug: I can only tell you how SQL Server reacts:

select * from a inner join b Line 1: Incorrect syntax near 'b'.

select * from a inner join b on a.id = b.id

id x id y

1 1 1 2

So SQL Server behaves like sqlite in returning column names, although it doesn't accept the syntax 'select * from a inner join b'

I guess this behavior is normal and it's the programmer's resposability to use aliases in sql statement. Just my personal oppinion, though.

 
1149 active 2005 Feb anonymous   Pending     2 1 VACUUM, DUMP/RESTORE fail in certain cases edit
When a database has interacting views, such as the following:

CREATE VIEW test1 AS SELECT * FROM tableA;
CREATE VIEW test AS SELECT COUNT(*) FROM test1;
then a dump and restore fails, because the views will be created in the alphabetical order of the table names, rather than the order of their dependence. Thus, the create of table test will fail. Because the script otherwise runs to completion, the restore will usually be adequate except that the views are not recreated.

VACUUM appears to fail the same way, probably for the same reason. In this case, however, the VACUUM fails without clearly alerting the user.

I ran into this problem trying to VACUUM a large file, which produced the otherwise inexplicable error message about the syntax of a dependent view definition.

Users can workaround, once the problem is understood, either by renaming views so that their names have alphabetical sequence consistent with their dependency, or by dropping them prior to a vacuum or dump/restore and then recreating them later.

Of course, the view order can be repaired in the dump file, but this usually requires patience with a powerful text editor, and some capacity to understand the problem.

 
1158 active 2005 Mar anonymous VDBE Pending     1 1 core dump on solaris, edit
some sqlite functions core dumped on solaris. I tracked this back to a byte alignment problem in Vdbeint.h. I added the __attribute__ ((__aligned__(16))) to the zshort character string.

/*
** A single level of the stack or a single memory cell
** is an instance of the following structure.
*/
struct Mem {
  int i;              /* Integer value */
  int n;              /* Number of characters in string value, including '\0' */
  int flags;          /* Some combination of MEM_Null, MEM_Str, MEM_Dyn, etc. */
  double r;           /* Real value */
  char *z   ;         /* String value */
  char zShort[NBFS] __attribute__ ((__aligned__(16)));  /* Space for short strings */
};
typedef struct Mem Mem;
2005-Mar-09 22:06:47 by anonymous:
I think this is a duplicate of bug #700.... simply adding the attribute seems to reslove the bug... select the sum of an integer column, to reproduce....

sqlite> select sum(seqval) from tbl; Bus Error(coredump)

 
1191 active 2005 Mar anonymous Unknown Pending     2 3 last_insert_id() Does Not Work after Insert on View edit
If I have a view with an INSERT trigger attached to it, and then use the view to insert a record, last_insert_rowid() does not return the ID inserted. For example, given this DDL:

  CREATE TABLE _simple (
    id INTEGER NOT NULL PRIMARY KEY,
    name TEXT
  );
  CREATE VIEW simple AS
    SELECT _simple.id AS id, _simple.name AS name
    FROM   _simple;

  CREATE TRIGGER insert_simple
  INSTEAD OF INSERT ON simple
  FOR EACH ROW BEGIN
    INSERT INTO _simple (name)
    VALUES (NEW.name);
  END;

This is what I get when I use the view to insert:

  sqlite> insert into simple (name) values ('foo');
  sqlite> select last_insert_rowid();
  last_insert_rowid()
  -------------------
  0                  

It does work if I insert directly into the table, of course:

  sqlite> insert into _simple (name) values ('foo');
  sqlite> select last_insert_rowid();
  last_insert_rowid()
  -------------------
  2
2005-Mar-31 06:04:59 by anonymous:

  VIEWs are read-only so you can not INSERT into them.
  See: http://www.sqlite.org/omitted.html

  Regards,
  Bartosz.


2005-Mar-31 18:12:10 by anonymous:
Bartosz,

Please note that triggers have been applied to the view, so you can actually insert into them. I do it all the time. For example, given my previous examples, this insert will work:

sqlite> insert into simple (id, ame) values (1, 'foo');

Pretty coole, eh? See:

  http://www.sqlite.org/lang_createtrigger.html

--Theory


2005-Mar-31 18:21:08 by drh:
The last_insert_rowid() routine does return the correct insert rowid while you are still within the trigger. Once you leave the trigger, last_insert_rowid() returns the rowid of the most recently inserted row outside of any trigger. This is by design. If last_insert_rowid() were to be responsive to inserts done by triggers, then any AFTER INSERT trigger that happened to update a logfile would overwrite the last_insert_rowid() from the actual INSERT.

One could argue, I suppose that last_insert_rowid() should work for inserts performed by an INSTEAD OF trigger but not by other kinds of triggers. I will ponder that notion and might implement it if I cannot think of any objections.


2005-Mar-31 18:25:14 by anonymous:
Allowing it to persist past the trigger in an INSTEAD OF trigger would certainly do what I need. I think that'd make a lot of sense.

All of this should probably be well-documented somewhere. I'd be happy to add a page to the wiki once you've decided how to proceed with this issue.

Thanks!

Theory


2005-Apr-27 02:21:21 by anonymous:
Have you had a chance to think more on this issue?

Thanks,

Theory


2005-Nov-08 03:06:51 by anonymous:
Just checking in on this issue again. Do you think that it's something that will be resolved, one way or the other, soon?

Thanks,

Theory


2005-Nov-08 17:44:14 by anonymous:
The instead of insert trigger can't, in general, update the last_insert_rowid value automatically and do it correctly, since a trigger may do multiple inserts into multiple tables. SQLite has no idea which rowid should be reported back outside the trigger. This is left to the user (i.e. the author of the trigger). You can get the last_insert_rowid after the appropriate insert in the trigger and then save that value into an auxillary table that is visible outside the trigger.

This is especially important when your view is created from entries in several joined tables. Your instead of insert trigger must do inserts into the individual tables, and may need to use the last_insert_rowid function to link the records in the tables together correctly. It then needs to return the "master" rowid for the table. SQL doesn't have any syntax to specify which value is returned as the rowid of the instead of trigger, so SQLite doesn't do anything automatically. You need to create a sperate last_rowid table, and inside the trigger you update the value of a row in that table (possibly the only row) with the value you want to return. Then the code that does the insert into the view needs to get the value of that row instead of calling last_insert_rowid.

This type of behavior is needed since there is no limit to the number of nested instead of triggers that could be executed by a single SQL statement (i.e. one instead of trigger could do an insert into another view... and so on).

The current behavior isn't quite as convenient in the simplest case, but it works correctly in the more complicated general case, where simply updating the last_insert_rowid value would be wrong.


2005-Nov-08 22:51:56 by anonymous:
I think that there's an argument to be made that last_insert_rowid() should return the last inserted row ID, even if a trigger inserted a bunch. It should simply return the last one entered.

However, I agree with your analysis. Could there perhaps be a set_insert_id() function, or some such, that the trigger could use to tell last_insert_row_id() what to return?

 
1200 active 2005 Apr anonymous   Pending     2 3 new versions of SQLite return different (incorrect) results. edit
Newer versions of SQLite are returning different, and I believe incorrect, results compared to those returned by older versions.

Using version 3.0.8 the following query returns the correct results given the attached database.

  sqlite> select * from device_property_list where device_property_value = 0;
  1|Station|3|Initial Volume|0
  1|Station|1|Template|0
  2|Station|3|Initial Volume|0
  2|Station|1|Template|0
  3|Station|3|Initial Volume|0
  3|Station|1|Template|0

This same query does not return any results using versions 3.1.6 and 3.2. If the query is changed slightly, by quoting the zero in the where condition, then both of the newer versions return the same set of results as 3.0.8.

  sqlite> select * from device_property_list where device_property_value = '0';
  1|Station|3|Initial Volume|0
  1|Station|1|Template|0
  2|Station|3|Initial Volume|0
  2|Station|1|Template|0
  3|Station|3|Initial Volume|0
  3|Station|1|Template|0

The device_property_list is a view that hides a complex join of several tables, and a long case expression that select the value to return for device_property_value; the field that is being tested by the condition.

I have attached a sample database and the sql script used to create it for testing.

 
1201 active 2005 Apr anonymous CodeGen Pending   danielk1977 1 1 erro defined type UINT8_TYPE edit
In file sqliteInt.h line 203 typedef UINT8_TYPE i8; /* 1-byte signed integer */

it should be INT8_TYPE.

 
1205 active 2005 Apr anonymous Unknown Pending     3 1 accent ecu etc. in connectionstring edit
Hi

Whenever I use a accent grave or accent ecu or whatever accented characters in the connectionstring sqlite turns this into weird characters (in the file name of the database. Since my database name is dependent on user input I can't eliminate this situation... bart@arlanet.com

2005-Apr-13 12:16:08 by drh:
What version of SQLite? What operating system?


2005-Apr-14 12:22:32 by anonymous:
Windows 2000 (dutch version) Sqlite version 3.

the word privé in the connectionstring becomes privĂ©.db3 on the disk, which causes errors afterwards when opening the database


2005-Apr-14 12:23:06 by anonymous:
I see something goes wrong with the accents as well on the previous remark


2005-Apr-14 12:29:41 by anonymous:
It really sounds to me like you're taking an accented character encoded in ISO8859-1 or some similar single-byte encoding and feeding it to something that's expecting a UTF-8 encoded string. In UTF-8, any byte that has its high bit set must be part of a multi-byte sequence.


2005-Apr-14 14:00:56 by anonymous:
After converting my string to an UTF8 encoded string, it still does the same thing....


2005-Apr-14 14:06:41 by anonymous:
using sqlite.net
 
1213 active 2005 Apr anonymous Parser Pending     3 2 Problem with alias columns in subqueries edit
The following query create an SQL error while it seems to me to be SQL complient (in fact it works fine with a lot database servers): sqlite> select i from (select 0 as i union all select 1) as tmp; SQL error: no such column: i

I found a workaround writting the following SQL query (it works well if you explicits the implicit variable i) : select i from (select 0 as i union all select 1 as i) as tmp;

Thanks for your help Jerome

2005-Apr-17 11:03:52 by anonymous:
SQLite appears to take the column names from the last clause in a UNION (your originally query works if you reverse the order of the clauses).

IIRC, the standard says that the column names in a UNION of clauses with different column names is DBMS-dependent and while many DBMSs take them from the first clause, this is not something to count on.


2005-Apr-18 19:53:09 by anonymous:
You're perfectly right, but it seems to me that it should be better that sqlite do respect the "de facto" standard.
 
1214 active 2005 Apr anonymous Unknown Pending     2 3 sqlite3_column_bytes returns 0 on p3 column with EXPLAINed selects edit
Both functions

  sqlite3_column_bytes()
  sqlite3_column_bytes16()

do not return the correct length for the p3 text column of EXPLAIN select queries. Both functions always return 0, even if the p3 column contains text.

The bug can be easily reproduced with the following query:

  EXPLAIN SELECT 'text';

The p3 column, row 2, contains the word 'text', but the functions return 0 regardles.

I have not seen this bug with non EXPLAIN queries, but it breaks code which relies on the fact that sqlite3_column_bytes always retun the correct length of the text and needs to preallocate memory accordingly.

 
1228 active 2005 Apr anonymous Unknown Pending     4 3 problem with select+union on a view with aliased columns edit
    CREATE TABLE tbl1 (col1 VARCHAR PRIMARY KEY);
    INSERT INTO tbl1 VALUES ('1');
    INSERT INTO tbl1 VALUES ('2');
    CREATE VIEW view1 AS SELECT col1 AS col1a FROM tbl1;

This creates a view with one column (col1a). Normal SELECTs on this view return results as expected, but the following:

  SELECT col1a FROM view1 WHERE col1a = 1
  UNION
  SELECT col1a FROM view1 WHERE col1a = 2;

Produces:

  col1
  1
  2

When the column name should in fact be col1a (this is the behaviour in postgres). You can work around this by doing:

  SELECT col1a AS col1a FROM view1 WHERE col1a = 1
  UNION
  SELECT col1a AS col1a FROM view1 WHERE col1a = 2;

But that shouldn't be necessary. Thanks. --Sebastian Kun

 
1235 active 2005 May anonymous Unknown Pending   drh 4 3 inconsistent pragma handling edit
The pragma user_version and schema_version are handled inconsistently in this respect : the result set returned contains a single column that has no name. all other pragmas return named columns. since some high-level languages complain about db fields with no names (most wrappers will gag at this), I suggest that a simulated "column_name" is also generated here, as with all other pragmas.
 
1248 active 2005 May anonymous Unknown Pending     3 1 sqlite3_get_table returns garbage for BLOB data edit
BLOB data returned by sqlite_get_table() is garbage when it contains data bytes holding '0' values.

Code: Windows, 3.2.1, VC 6.0

char data[128];

for(int i = 0;i<128;i++)

{ data[i] = i+1; }

sqlite3_exec(thedb,"CREATE TABLE test (b BLOB)",NULL,NULL,NULL);

result = sqlite3_prepare(thedb,"INSERT INTO test (b) VALUES (?)",-1,&cstate,NULL);

sqlite3_bind_blob(cstate,1,data,128,SQLITE_STATIC); // bind the data to the statement

result = sqlite3_step(cstate);

result = sqlite3_finalize(cstate);

sqlite3_get_table(thedb,"SELECT * FROM test",result,rows,columns,NULL);

this sqlite3_get_table() returns a properly layed out table. However, it looks like sqlite_get_table() is converting blob data to another type of data (string?) as it processes the command. The BLOB data it returns using this call is completely corrupted, due to processing '0' valued bytes as EOS characters. Instead it should be inferring BLOB data where appropriate, and returning a correct data block.

As a workaround I've been using the prepare/step functions instead. However, the sqlite3_get_table() is a neccessity for many users of this library, as it allows a simple and elegant SQL query mechanism, and should be fixed ASAP to support BLOB data properly.

2005-Sep-27 01:44:39 by anonymous:
sqlite3_get_table is considered legacy code, intended to make porting of sqlite2 applications (which never had to deal with BLOBs) easier. Its use in new applications is deprecated. If you need something like it that can handle BLOBs, you're best off writing a wrapper function for the prepare/step interface (you can use the sqlite3_get_table code as a template).
 
1255 active 2005 May anonymous   Pending     4 3 Decrease number of warnings with Microsoft Visual C++ edit
Add this to sqliteInt.h to decrease the number of warnings produced by sqlite:

  #if defined(_MSC_VER)
  #pragma warning (disable: 4018) // signed/unsigned mismatch
  #pragma warning (disable: 4244) // conversion from 'unsigned __int64 ' to 'unsigned char ', possible loss of data
  #pragma warning (disable: 4761) // integral size mismatch in argument; conversion supplied
  #endif
2005-May-20 19:27:50 by drh:
Is there no command-line option on microsoft to disable these warnings?


2005-May-21 08:56:15 by anonymous:
It's possible to lower the warning level, from say 3 to 2 using the command line. But this is not as selective and will remove more warnings than those #pragmas.
 
1264 active 2005 May anonymous Unknown Pending     3 3 access() undefined on MSVC edit
  shell.c(1705) : warning C4013: 'access' undefined; assuming extern returning int

Add this:

  #if defined(_WIN32) && defined(_MSC_VER)
  # include <io.h>
  #endif
 
1278 active 2005 Jun anonymous   Pending     1 2 sqlite3_finalize doesn't clear previous error code or message edit
A call to sqlite3_finalize(), after an error during sqlite3_prepare() of another statement, returns the correct result SQLITE_OK, but does not reset the error code or error message returned by sqlite3_errcode() and sqlite3_errmsg(). The error reporting functions still return the error code and message associated with the error that occurred during the previous prepare.

The attached code demonstrates the problem. One statement is prepared successfully. Then an second statement is prepared. This one fails and returns an error result. The correct error code and message are retrieved using the error reporting API functions. Next, the first statement is finalized, which returns SQLITE_OK. Calling the sqlite_errcode() function at this point still returns the error code from the previous error.

I believe the error code and message should be cleared by the successful call to the sqlite3_finalize() API function.

 
1298 active 2005 Jun anonymous Shell Pending   drh 4 3 sqlite3.exe not recognizing newline as comment terminator edit
The sqlite3.exe program does not recognize comments that end in a newline if there is valid SQL before them on the same line. The statement:
"SELECT 1; -- this is a comment"
should be valid SQL, but sqlite3.exe still requires a semicolon after the comment.

Please look at the test below:

C:\Temp\SQLite>sqlite3

SQLite version 3.2.1

Enter ".help" for instructions

sqlite> select 1;

1

sqlite> select 1; -- test comment

   ...> ;

1

sqlite> select 1; -- test;

1

sqlite>

2005-Jun-22 18:20:37 by anonymous:
The string "SELECT 1; -- this is a comment" is not valid SQL. SQL does not use the semicolon character inside SQL statements except to seperate the constituent statements of a trigger, stored procedure, or an embedded program.

The semicolon is used by sqlite3.exe (and standard SQL) as a end of statement marker that triggers it to parse and execute the input up to that point. From the SQL 2003 standard:

  21 Direct invocation of SQL
  21.1 <direct SQL statement>
  Function
  Specify direct execution of SQL.
  Format
  <direct SQL statement> ::= <directly executable statement> <semicolon>

This SQL statement should be "SELECT 1 --this is a comment" followed by a semicolon to mark the end of the statement. Putting the semicolon in the middle terminates the first statement at that point, executes it, and then starts collecting input for a second statement (which in this case does not have a terminating semicolon).

Note that sqlite3 also accepts the word GO (used by SQL server) or the Oracle compatible character "/" as the end of statement marker, but only when they appear on a line of input by themselves (i.e. at the continuation prompt in the shell).

  sqlite> SELECT 1 --this is a comment
     ...> go
  1
  sqlite> SELECT 1 --this is a comment
     ...> /
  1
  sqlite>
 
1305 active 2005 Jun anonymous TclLib Pending     1 1 Tcl installs pkgIndex with wrong path edit
Correct script below:

  # This script attempts to install SQLite3 so that it can be used
  # by TCL.  Invoke this script with single argument which is the
  # version number of SQLite.  Example:
  #
  #    tclsh tclinstaller.tcl 3.0
  #
  set VERSION [lindex $argv 0]
  set LIBFILE .libs/libtclsqlite3[info sharedlibextension]
  if { ![info exists env(DESTDIR)] } { set DESTDIR "" } else { set    DESTDIR $env(DESTDIR) }
  set LIBDIR [lindex $auto_path 0]
  set LIBNAME [file tail $LIBFILE]
  set LIB $LIBDIR/sqlite3/$LIBNAME

  file delete -force $DESTDIR$LIBDIR/sqlite3
  file mkdir $DESTDIR$LIBDIR/sqlite3
  set fd [open $DESTDIR$LIBDIR/sqlite3/pkgIndex.tcl w]
  puts $fd "package ifneeded sqlite3 $VERSION \[list load $LIB   sqlite3\]"
  close $fd

  # We cannot use [file copy] because that will just make a copy of
  # a symbolic link.  We have to open and copy the file for ourselves.
  #
  set in [open $LIBFILE]
  fconfigure $in -translation binary
  set out [open $DESTDIR$LIB w]
  fconfigure $out -translation binary
  puts -nonewline $out [read $in]
  close $in
  close $out
 
1312 active 2005 Jun anonymous Shell Pending     1 1 CSV file import / export is all wrong! edit
  Four problems:
    Importing a proper CSV file (which delimits strings within
    double-quotes) is impossible. The '.import' command just
    treats the double-quotes as ordinary text characters. So
    (1) Error "expected 2 columns of data but found 3" can occur
        if one of the strings contains a comma
    (2) the delimiting double-quotes are NOT stripped off before
        inserting the data into the table as it should
    (3) it doesn't understand the standard convention that to
        represent a double-quote character within a double-quoted
        string you use TWO double-quote (eg. "3.5"" Floppy Drive")
    (4) outputting data in CSV mode also doesn't use this standard
        convention

  --------------------------------------------
  _Product.csv contains:

  "A001","McVities"
  "B001","Heinz"
  "C001","Callard,Bowser"

  sqlite> .mode csv
  sqlite> .import _Product.csv Product
  _Product.csv line 3: expected 2 columns of data but found 3
2005-Aug-12 23:13:54 by anonymous:
Commas are also not accounted for when using the sqlite3_mprintf() function(s). I'm guessing the %q flag should also escape these.
 
1323 active 2005 Jul anonymous   Pending     1 1 misuse-4.4...gmake: *** [test] Segmentation Fault (core dumped) edit
misuse-4.4...gmake: *** [test] Segmentation Fault (core dumped)

on both solaris 8 and 9 build env

export CPPFLAGS="-I/tps/include"

export LDFLAGS="-L/tps/lib -R/tps/lib"

export PKG_CONFIG_PATH=/tps/lib/pkgconfig

CC=/tps/bin/gcc

CXX=/tps/bin/g++

LD_LIBRARY_PATH=/tps/lib:/tps/lib/sparcv9:/lib:/usr/lib:/usr/local/lib:\ /usr/ccs/lib:/usr/dt/lib:/usr/ucblib:/usr/openwin/lib

PATH=/tps/bin:/tps/java/bin:/dsw/source/bin:/dsw/depot-5.13/bin:\ /usr/ccs/bin:/usr/bin:/usr/openwin/bin:/bin:/usr/local/bin:/sbin:\ /usr/sbin:/usr/ucb:/etc:.:/sfoc/bin:/usr/dt/bin:\ /dsw/source/harvest/bin:/usr/afsws/bin:/dsw/pgp-2.6.2s/bin

export CC CXX LD_LIBRARY_PATH PATH

where /tps is my version of /usr/local where I put all the configuration controlled open source and licensed s/w for my network.

<pray@fir:/export/build/sqlite-3.2.2/bld> [525]$ ../configure --prefix=/dsw/sqlite-3.2.2 --with-tcl=/tps/lib

gmake

gmake test

then got error

gcc -v

Reading specs from

/dsw/gcc-3.4.0/lib/gcc/sparc-sun-solaris2.9/3.4.0/specs Configured with: ../configure --prefix=/dsw/gcc-3.4.0 --disable-nls --enable-languages=c,c++,f77,objc --disable-libgcj --srcdir=/export/build/gcc-3.4.0 --with-ld=/usr/ccs/bin/ld Thread model: posix gcc version 3.4.0

using ActiveTcl8.4.5.0

 
1327 active 2005 Jul ghaering VDBE Pending     2 3 UNION causes sqlite3_column_decltype to always return NULL edit
create table test(foo bar);

select foo from test union select foo from test;

Normally, sqlite3_column_decltype() will return 'bar', but with the UNION, it will always return NULL.

This is quite annoying for pysqlite users, because it renders the type detection useless for UNION queries (and EXCEPT and INTERSECT).

 
1342 active 2005 Jul anonymous   Pending     1 1 sqlite 3.2.2 will not load on Suse Linux 9.3 edit
when trying to load the sqlite 3.2.2 .so lib with tcl I get this problem:

  couldn't load file "/usr/lib/sqlite3/tclsqlite-3.2.2.so": /usr/lib/sqlite3/tclsqlite-3.2.2.so: undefined symbol: sqlite3_version

Sqlite 3.2.1 does not give an error, with the same script

 
1351 active 2005 Aug anonymous Shell Pending     1 1 Unable to parse UTF8 input edit
I'm in the process of writing a program which parses in UTF8 data, and then processes it and writes a UTF8 output into a text file. This textfile needs to be imported into SQLite.

However the commandline SQLite program doesnt support UTF8 input text files for its ".read" command. Considaring the database itself supports UTF8, would it be possible to allow UTF8 text file input.

I can't progress much further on my program if this can't be fixed.

2005-Aug-08 13:56:24 by drh:
Please attach an example UTF-8 script that ".read" is not reading correctly.


2005-Aug-08 17:58:44 by anonymous:
I can't seem to attach a text file, so instead i'll put it on my FTP, and it should be accessible from there. If you have trouble with that, i could email the text file. So if you have trouble, give me an email to send it to.

There will be a sample of a text file that won't ".read" available here: ftp://62.231.38.73/ in approximately 10 minutes.

Thanks for the fast reply. I'd be surprised if it was a problem with my text file generation, but stranger things have happened :p


2005-Aug-11 01:40:08 by drh:
Attach files using the [Attach] hyperlink at the top-right of this page.

Please do not send RAR files since that is an obscure archive format. If you want to use a compressed archive, make it either ZIP or GZIP.


2005-Aug-11 09:46:23 by anonymous:
I can only attach one because neither winzip or gzip are great at compressing text files. The files keep ending up over 100kb except for the artists.txt file. Thats the only one that went below 100kb.

I'm unwilling to try editing the file to remove lines of text from it because i want you to have the exact output that i'm getting from my program. Not the output that i'd get from notepad if i edited it. That will help identify whether it is a problem in my program, or a problem with SQLite.

Its always possible its a proglem with my source data, and its not actually proper UTF8, but i doubt that as SQLite doesn't seem to read the normal text correctly (such as the first line which is supposed to start a transaction, but SQLite instead ignores the line, and throws an error when it reachs the commit; at the end.

 
1365 active 2005 Aug anonymous   Pending     3 3 64 bit types not completely overridable edit
The current 64 bit types in sqlite3.h and sqliteInt.h do not allow the type to be overriden using a preprocessor definition, unlike all the other base types. The current 64 bit typedefs assume that a "long long" is 64 bits - this is not guaranteed (and on PS2 it is wrong, long long is 128 bits).

Here are some minor patches that should allow these types to be overriden, but keep the old behavior if they are not:

  ==== //sqlite-3.2.2/src/sqlite3.h#1 - sqlite-3.2.2\src\sqlite3.h ==== 81,83c81,83
  < #if defined(_MSC_VER) || defined(__BORLANDC__)
  <   typedef __int64 sqlite_int64;
  <   typedef unsigned __int64 sqlite_uint64;
  ---
  > #ifdef INT64_TYPE
  >   typedef INT64_TYPE sqlite_int64;
  >   typedef unsigned INT64_TYPE sqlite_uint64;
  85,86c85,93
  <   typedef long long int sqlite_int64;
  <   typedef unsigned long long int sqlite_uint64;
  ---
  > # if defined(_MSC_VER) || defined(__BORLANDC__)
  >     typedef __int64 sqlite_int64;
  >     typedef unsigned __int64 sqlite_uint64;
  > # else
  >     typedef long long int sqlite_int64;
  >     typedef unsigned long long int sqlite_uint64;
  > # endif
  > # define INT64_TYPE sqlite_int64
  > # define UINT64_TYPE sqlite_uint64

  ==== sqlite-3.2.2/src/sqliteInt.h#1 - sqlite-3.2.2\src\sqliteInt.h ==== 157,163d156
  < #ifndef UINT64_TYPE
  < # if defined(_MSC_VER) || defined(__BORLANDC__)
  < #   define UINT64_TYPE unsigned __int64
  < # else
  < #   define UINT64_TYPE unsigned long long int
  < # endif
  < #endif
  183c176
  < typedef UINT64_TYPE u64;           /* 8-byte unsigned integer */
  ---
  > typedef sqlite_uint64 u64;         /* 8-byte unsigned integer */
2005-Aug-27 16:45:09 by drh:
Can someone suggest a suitable #ifdef that will automatically identify a PS and do the right thing to provide a 64-bit integer type, similar to what is down for windows?
 
1382 active 2005 Aug anonymous   Pending   drh 1 1 Assert nErr==0 on corrupt db edit
I'm working on an embedded filesystem where files can be randomly altered. Sometimes my .db files get messed up. I've attached an example db.

I'd like to catch the asserts and return an error rather than crash.

$ sqlite corrupt-assert.db 'select count(*) from sensor'

sqlite: src/main.c:120: sqliteInitCallback: Assertion `nErr==0' failed.

Aborted

 
1397 new 2005 Aug anonymous Shell Pending     1 1 .mode csv creates ASCII output instead of UTF-8 edit
compiled from source

.mode csv mixes up the charset IMHO

Example "f&#252;r" becomes "f\37777777703\37777777674r"

which makes the output file not usable

2005-Aug-31 13:54:57 by anonymous:
it's better but not UTF-8 utrac -p says ASCII

G&#252;ssing now is G\303\274ssing

hope it's ok to reopen the bug


2005-Sep-03 15:29:37 by anonymous:
.mode tab csv list all destroy the output now


2005-Sep-20 08:10:34 by anonymous:
same problem in 3.2.6


2005-Oct-06 20:14:37 by anonymous:
csv of today:

.mode cvs still does not work,

.mode tab works fine now

 
1415 active 2005 Sep anonymous Unknown Pending   drh 1 1 Querying for BLOB type fields edit
How do I query for BLOB type fields?

I tried

  1. field LIKE 'abc'
  2. field LIKE quote('abc') and
  3. field LIKE X'616263'

but nothing seems to return back the record that I am interested in.

2005-Sep-13 08:00:28 by anonymous:
Using version 3.2.5, you might use the quote function to convert a blob into a string which you can filter using the like operator: select * from test where quote(text) like '%6263%'; is working and usable but may not work as expected because like '%26%' would find the same and this was not expected, isn't it? select * from test where like(quote(text),'%6263%'); doesn't work and select * from test where like(quote(text),'%6263%','%'); doesn't work either


2005-Oct-04 05:44:08 by anonymous:
This should really be taken to the mailing list, preferably with descriptions of how other DBMSs handle LIKE as applied to BLOB columns.
 
1428 active 2005 Sep anonymous TclLib Pending     3 3 tclinstaller.tcl script problems edit
The pkgIndex.tcl file generated by the tclinstaller.tcl script contains absolute pathnames to the TCL extension library. This causes problems if the extension subdirectory is moved. A more portable solution is shown below.

A second problem is that the shared library does not have executable permission. This is a problem on HPUX operating systems. Adding the 0755 permission mode to the open command solves the problem for HPPA and does not cause problems for the other platforms.

Here's a diff of the changes I made to address these two problems:

  cvs diff tclinstaller.tcl
  Index: tclinstaller.tcl
  ===================================================================
  RCS file: /sqlite/sqlite/tclinstaller.tcl,v
  retrieving revision 1.2
  diff -r1.2 tclinstaller.tcl
  17c17
  < puts $fd "package ifneeded sqlite3 $VERSION \[list load $LIB sqlite3\]"
  ---
  > puts $fd "package ifneeded sqlite3 $VERSION \[list load \[file join \$dir $LIBNAME\]\]"
  25c25,26
  < set out [open $LIB w]
  ---
  > #Some platforms such as the HP requre that libraries have the executable bit set
  > set out [open $LIB w 0755]
 
1445 active 2005 Sep anonymous   Pending     3 3 Errors testing sqlite 3.2.6 (& v3.3.7) edit
  $ make test
  [...]
  conflict-6.0... Ok
  conflict-6.1... Ok
  conflict-6.2...
  Expected: [0 {7 6 9} 1 1]
       Got: [0 {7 6 9} 1 0]
  conflict-6.3...
  Expected: [0 {6 7 3 9} 1 1]
       Got: [0 {6 7 3 9} 1 0]
  conflict-6.4... Ok
  conflict-6.5... Ok
  conflict-6.6... Ok
  conflict-6.7...
  Expected: [0 {6 7 3 9} 1 1]
       Got: [0 {6 7 3 9} 1 0]
  conflict-6.8...
  Expected: [0 {7 6 9} 1 1]
       Got: [0 {7 6 9} 1 0]
  conflict-6.9...
  Expected: [0 {6 7 3 9} 1 1]
       Got: [0 {6 7 3 9} 1 0]
  conflict-6.10...
  Expected: [0 {7 6 9} 1 1]
       Got: [0 {7 6 9} 1 0]
  conflict-6.11...
  Expected: [0 {6 7 3 9} 1 1]
       Got: [0 {6 7 3 9} 1 0]
  conflict-6.12...
  Expected: [0 {6 7 3 9} 1 1]
       Got: [0 {6 7 3 9} 1 0]
  conflict-6.13...
  Expected: [0 {7 6 9} 1 1]
       Got: [0 {7 6 9} 1 0]
  conflict-6.14... Ok
  conflict-6.15... Ok
  conflict-6.16... Ok
  [...]
  date-3.12... Ok
  date-3.13... Ok
  date-3.14... Ok
  date-3.15... Ok
  date-3.16... Ok
  date-3.17... Ok
  /tmp/sqlite-3.2.6/.libs/lt-testfixture: invalid command name "clock"
      while executing
  "clock seconds"
      invoked from within
  "clock format [clock seconds] -format "%Y-%m-%d" -gmt 1"
      invoked from within
  "set now [clock format [clock seconds] -format "%Y-%m-%d" -gmt 1]"
      (file "./test/date.test" line 142)
      invoked from within
  "source $testfile"
      ("foreach" body line 4)
      invoked from within
  "foreach testfile [lsort -dictionary [glob $testdir/*.test]] {
    set tail [file tail $testfile]
    if {[lsearch -exact $EXCLUDE $tail]>=0} continue
    so..."
      (file "./test/quick.test" line 45)
  make: *** [test] Error 1
2005-Sep-19 23:03:56 by drh:
The test scripts do not (yet) work with Tcl 8.5. Use Tcl 8.4.


2005-Sep-20 01:59:42 by anonymous:
FYI, The conflict failures occur even when using tcl-8.4. The problem was reported on the mailing list:

  http://www.mail-archive.com/sqlite-users%40sqlite.org/msg10203.html

Curiously, the failures correspond exactly to the test cases that were changed by the following patch:

  http://www.sqlite.org/cvstrac/filediff?f=sqlite/test/conflict.test&v1=1.24&v2=1.25


2006-Aug-31 23:49:40 by anonymous:
building v337 on OSX 10.4.7 w/ TCL8.5 installed as Framework, 'make test' still fails w/:

  date-3.16... Ok
  date-3.17... Ok
  /usr/ports/sqlite-3.3.7/build/.libs/testfixture: invalid command name "clock"
      while executing
  "clock seconds"
      invoked from within
  "clock format [clock seconds] -format "%Y-%m-%d" -gmt 1"
      invoked from within
  "set now [clock format [clock seconds] -format "%Y-%m-%d" -gmt 1]"
      (file "../test/date.test" line 142)
      invoked from within
  "source $testfile"
      ("foreach" body line 4)
      invoked from within
  "foreach testfile [lsort -dictionary [glob $testdir/*.test]] {
    set tail [file tail $testfile]
    if {[lsearch -exact $EXCLUDE $tail]>=0} continue
    so..."
      (file "../test/quick.test" line 66)
  make: *** [test] Error 1

any resolution for this, other than revert to TCL 8.4?


2006-Sep-01 01:26:37 by anonymous:
SQLite under Cygwin fails all tests that involve integers larger than 32 bits. Sqlite produces the correct 64 bit values, but Tcl as distributed with Cygwin cannot grok 64 bit ints, so the comparisons fail. Would it be possible to change Sqlite's test harness to compare SQL results as strings rather than as integers? Then it would not matter if Tcl worked in 64 bit or not.


2006-Sep-01 15:50:48 by drh:
The test suite has been revised so that it now works with Tcl8.5. But, no, it is not practical to rewrite the tests to compare the results using strings instead of integers in order to work with the (broken) tcl implementation that comes with cygwin.


2006-Sep-06 02:39:24 by anonymous:

  updating to latest cvs-checkout to get the aforementioned fix for:

      date-3.17... Ok
      /usr/ports/sqlite-3.3.7/build/.libs/testfixture: invalid command name "clock"
          while executing

  i can verify that _that_ is now ok:

      ...
      date-3.14... Ok
      date-3.15... Ok
      date-3.16... Ok
      date-3.17... Ok
      date-4.1...
      Expected: [2006-09-01]
           Got: [2006-09-06]
      date-5.1... Ok
      date-5.2... Ok
      date-5.3... Ok
      ...

  but now, 'make test' fails next @:

      delete-8.4... Ok
      delete-8.5... Ok
      delete-8.6... Ok
      delete-8.7... Ok
      /usr/ports/sqlite-cvs/build/.libs/testfixture: error deleting "test.db": not owner
          while executing
      "file delete -force test.db"
          (file "../test/tester.tcl" line 62)
          invoked from within
      "source $testdir/tester.tcl"
          (file "../test/delete2.test" line 36)
          invoked from within
      "source $testfile"
          ("foreach" body line 4)
          invoked from within
      "foreach testfile [lsort -dictionary [glob $testdir/*.test]] {
        set tail [file tail $testfile]
        if {[lsearch -exact $EXCLUDE $tail]>=0} continue
        so..."
          (file "../test/quick.test" line 66)
      make: *** [test] Error 1


2006-Sep-06 11:11:19 by drh:
Run the build starting from an empty directory as a non-root user.


2006-Sep-06 13:27:18 by anonymous:
per INSTALL instructions, i did:

  cvs -d :pserver:anonymous@www.sqlite.org:/sqlite checkout -d sqlite-cvs sqlite
  cd /usr/ports/sqlite-cvs

  mkdir build
  cd build

  ../configure \
  ...

  make

  chown -R myuser:wheel /usr/ports/sqlite-cvs
  sudo -u myuser make test

and, as reported, the error was the result.


2006-Sep-30 21:43:45 by anonymous:
bump.

anyone?


2006-Sep-30 22:19:24 by anonymous:
If you don't happen to be testing on Linux/gcc or Windows/VC++ I find that the Tcl test results have more than a few failures. It is not always easy to discern which failures are due to some odd quirk of Tcl or whether it is a legitimate SQLite issue on a given platform. Be prepared to change test scripts and tinker with the code.
 
1447 active 2005 Sep anonymous BTree Pending     1 1 Abnormal program termination in src/btree.c line 1339 edit
In some circumstances (after having used wxgrid ..) a call to sqlite gives a strange : Assertion Failed: pCur->idx>=0 && pCur->idx < pCur->pPage->nCell, file src/btree line 1339 abnormal program termination there seems to be non way of making a trace back ... any idea?

Thanx Doriaqn Tessore

2005-Sep-23 14:32:27 by drh:
Not much to go on. What version of SQLite is being used? ("SQLite 2" is kind of vague.)
 
1451 active 2005 Sep anonymous Shell Pending     4 3 .mode insert does not output BLOBs in an usable way edit
sqlite> CREATE TABLE a(b);
sqlite> INSERT INTO a VALUES (X'41424300500051');
sqlite> .dump
BEGIN TRANSACTION;
CREATE TABLE a(b);
INSERT INTO "a" VALUES(X'41424300500051');
COMMIT;
sqlite> .mode insert
sqlite> SELECT * FROM a;
INSERT INTO table VALUES('ABC');
It would be nice for ".mode insert" to print a command that would actually re-create the same data, the same as ".dump" (the obvious difference is that .dump can't filter data in any way, it just dumps it all) or, at least, it would be very nice if the already existing function that "prints binary data as X'-encoded-string" were reachable from SQL, so that one could use something like:
SELECT xencode(b) FROM a;

and obtain

X'41424300500051'
2005-Sep-25 17:03:40 by anonymous:
I am not the original ticket poster, but I noticed that this feature request is related to Monotone migrating away from Base64 encoding to using straight Sqlite blobs.


2005-Sep-26 01:49:33 by drh:
The built-in quote() function converts BLOBs into ascii BLOB literals. Will it not server for the requested xencode() function?

   SELECT quote(b) FROM a


2005-Sep-26 08:57:47 by anonymous:
Yes, I guess it can perfectly do. What about ".mode insert" output? Is it supposed to print raw data?
 
1455 active 2005 Sep anonymous Shell Pending     3 2 .import error: comma inside a string is read as field separator edit
In Sqlite version 3, when you need to import data into a table you use .import. (You cannot do it with COPY).

Well, if you need to import data in 'csv' format, and if there is a string in the input data that contains a <comma> inside itself, the reading is impossible since the comma is interpreted as a field separator. Example error message: "Error: There are 10 fields in file, and 8 fields were expected."

To me its pretty much a nuisance, since csv format is the most usual format for the data I use, and I've got to change the separating character, or else locate and eliminate those extra commas.

Hope this helps Thanks Antoni Francino afrancino@mesvilaweb.com

 
1457 active 2005 Sep anonymous Shell Pending     3 1 non-latin chars not recognized by CLI edit
Hello, If i am trying to execute an SQL statement via the CLI of a self-compiled (Win via MinGW) or the prebuilded exe all chars that are not iso are converted to something ugly.

For example, the German Umlaute or the &#232; are not entered correctly into a table name, field value and so on. If the same commands are executed from the sqlite>> promt everything works well.

As I need to compile my own sqlite.exe I need to be able to change that in code.

Thank you for this great Product and a solution to this problem.

btw: This is my first request here, so please be patient with me if I&#180;ve set the priority wrong... It&#180;s obviously highest priority to me ;)

2005-Sep-26 21:44:12 by anonymous:
&#232 should be the accented e in my post, sorry for that
 
1458 active 2005 Sep anonymous Shell Pending     2 2 Error at .importing in csv format (another) edit
Sqlite3 version 3.2.6 :

When importing data in csv format the programa adds commas when importing strings enclosed with commas in the source file.

In particular it shouldnt add commas when field data is already enclosed in comma, but it does. Curiously enough, it correctly imports numeric data.

Example: Take de demodatabase file, and take the table clients. If you try to add these records to the table with .import, it is impossible (the only workaround is deleting commas in the file and importing it in <list .mode>

504,"New Enterprise","Mr Smart","93-2275400"
505,"Another Enterprise","John Dongu","93-8765432"
506,"And here we are","Mr Strange","973-237131"

This file would be impossible to import correctly with .import If you set .mode list, it is imported incorrectly, since it keeps the commas around character fields in the table -which is what it should do anyway, since in this mode the program does not expect commas around field data. But when you set <.mode csv>, it imports them also incorrectly - it adds new commas around character data.

Souce data in csv format is important, -probably the most general data format available, and often a last resort format for difficult cases...

Thank in advance. I would like to help at fixing problems myself, but I do not understand a word of C.

Antoni Francino afrancino@mesvilaweb.com

2005-Sep-27 23:07:43 by anonymous:
You've got the names of your punctuation characters confused. What's happening is that the double quote marks around the text fields are gettting imported, whereas the usual understanding of "CSV" is that they should be stripped off.


2005-Sep-27 23:11:27 by anonymous:
On further inspection, this turns out to be the same problem as in ticket #1312.
 
1461 active 2005 Sep anonymous Unknown Pending   drh 1 2 3.2.7 DLL can not deal File paths with international characters edit
My platform:winxpsp2 chinese version,my database file under a path with chinese charater,with the dll(3.2.1) sqliteexplore works fine,if I change the dll to 3.2.7,it show sqlite error 14:can't open the file,and then I change the path fully english,it can work fine again,so I think mybe it relate to Check-in:2656
2005-Sep-28 15:31:34 by anonymous:
and more,the source version has no such problem.


2005-Sep-28 16:09:04 by anonymous:
I notice in os_win.c, function "sqlite3OsFileExists" use GetFileAttributesA and GetFileAttributesW,but I tried GetFileAttributes works ok.
 
1465 active 2005 Oct anonymous CodeGen Pending     1 1 fdatasync not available and not yet fixed edit
fdatasync is still there ... i downloaded the current configure files which should check for it. do i have to use something else too?

./libtool --mode=link cc -g -DOS_UNIX=1 -DHAVE_USLEEP=1 -I. -I./src -DNDEBUG -DTHREADSAFE=0 -DSQLITE_OMIT_CURSOR -DHAVE_READLINE=0 \ -o sqlite3 ./src/shell.c libsqlite3.la -lcurses cc -g -DOS_UNIX=1 -DHAVE_USLEEP=1 -I. -I./src -DNDEBUG -DTHREADSAFE=0 -DSQLITE_OMIT_CURSOR -DHAVE_READLINE=0 -o .libs/sqlite3 ./src/shell.c ./.libs/libsqlite3.so -lcurses "./src/shell.c", line 355: warning: argument #1 is incompatible with prototype: prototype: pointer to const unsigned char : "./src/shell.c", line 84 argument : pointer to const char "./src/shell.c", line 523: warning: argument #1 is incompatible with prototype: prototype: pointer to const unsigned char : "./src/shell.c", line 84 argument : pointer to char "./src/shell.c", line 694: warning: argument #2 is incompatible with prototype: prototype: pointer to const char : "./src/shell.c", line 583 argument : pointer to const unsigned char Undefined first referenced symbol in file fdatasync ./.libs/libsqlite3.so ld: fatal: Symbol referencing errors. No output written to .libs/sqlite3 make: *** [sqlite3] Error 1

2005-Oct-04 21:42:26 by drh:
Clear out your build directory (or start a new one) and rerun configure from scratch. Save the output of configure. Then rerun make. If you still have a problem, attach the output of configure to this ticket.
 
1485 active 2005 Oct anonymous Unknown Pending   jshen 3 1 cyrilic problem(suppose Unicode as a whole, for PPC) edit
I think there is a problem in the utf.c file.
2005-Oct-14 06:18:45 by anonymous:
Pocket PC support isn't provided in the default SQLite provider. The code necessary to support the Pocket PC is at http://sourceforge.net/projects/sqlite-wince and its there that a bug report should be filed.
 
1487 active 2005 Oct anonymous BTree Pending     1 3 Corrupt database causes indefinite loop in sqlite3_step() edit
I had a database become corrupt (no idea why, 57 other databases of similiar information are fine). When attempting to work with the database and execute a SELECT query the application froze in an endless loop inside sqlite3_step(). Upon further investigation (which is when I found the db was corrupt) it seems to be stuck inside the btree code (as reported by Sample). I tested the same query and alternates from sqlite3 CLI and got the same results. The exact query causes an infinite loop. Leaving off part of the WHERE statement (and making it broader) or removing one of the reporting columns simply causes a corruption error.

I have the original database as-is and the SQL query that can be run to cause the problem.

OS: Mac OS X 10.4.2

 
1488 active 2005 Oct anonymous   Pending     1 1 Collate Reverse does not exists? edit
when I execute below SQL
CREATE Unique INDEX index10 On Test2 (
F1 Collate BINARY ,
F2 Collate REVERSE DESC)

there is a error message:
no such collation sequence: REVERSE

but the latest document said that binary ,nocase , reverse is common collate function. what is wrong with my sql?

 
1489 active 2005 Oct anonymous   Pending     3 2 Bad permissions on install-sh prevent 'make install' from completing edit
It's a trivial problem -

install-sh has incorrect permissions via CVS, resulting in 'make install' failing.

Error: make installtclsh ../sqlite/tclinstaller.tcl 3.2../sqlite/install-sh -c -d /usr/local/libmake: execvp: ../sqlite/install-sh: Permission deniedmake: *** [install] Error 127

Permissions: -rw-r--r-- 1 cat other 5598 Sep 28 2001 ../sqlite/install-sh

Fix: chmod 755 ../sqlite/install-sh

 
1493 active 2005 Oct anonymous Parser Pending     3 3 lemon: pathsearch uses wrong directory separator under Win32 edit
The pathsearch function in lemon.c uses a semicolon (;) to separate the directories in the path. Under Win32 systems this should be a colon (:).
2005-Oct-18 09:37:10 by anonymous:
--- lemon.c.orig 2005-10-18 11:27:55.753467000 +0200 +++ lemon.c 2005-10-18 11:29:11.897825400 +0200 @@ -2791,13 +2791,16 @@ { char *pathlist; char *path,*cp; + char ds; char c; extern int access();

#ifdef __WIN32__ cp = strrchr(argv0,'\\'); + ds = ';'; #else cp = strrchr(argv0,'/'); + ds = ':'; #endif if( cp ){ c = *cp; @@ -2812,7 +2815,7 @@ path = (char *)malloc( strlen(pathlist)+strlen(name)+2 ); if( path!=0 ){ while( *pathlist ){ - cp = strchr(pathlist,':'); + cp = strchr(pathlist,ds); if( cp==0 ) cp = &pathlist[strlen(pathlist)]; c = *cp; *cp = 0;


2005-Oct-18 09:39:00 by anonymous:
More readable version of the patch:

  --- lemon.c.orig        2005-10-18 11:27:55.753467000 +0200
  +++ lemon.c     2005-10-18 11:29:11.897825400 +0200
  @@ -2791,13 +2791,16 @@
   {
     char *pathlist;
     char *path,*cp;
  +  char ds;
     char c;
     extern int access();

   #ifdef __WIN32__
     cp = strrchr(argv0,'\\');
  +  ds = ';';
   #else
     cp = strrchr(argv0,'/');
  +  ds = ':';
   #endif
     if( cp ){
       c = *cp;
  @@ -2812,7 +2815,7 @@
       path = (char *)malloc( strlen(pathlist)+strlen(name)+2 );
       if( path!=0 ){
         while( *pathlist ){
  -        cp = strchr(pathlist,':');
  +        cp = strchr(pathlist,ds);
           if( cp==0 ) cp = &pathlist[strlen(pathlist)];
           c = *cp;
           *cp = 0;


2005-Oct-19 14:05:27 by anonymous:
Cygwin and other Posix emulation layers on Windows require ':' for path separators, so you cannot blindly rely on __WIN32__ to make this determination.
 
1494 active 2005 Oct anonymous Unknown Pending     1 2 intermittent null reference exception in sqlite3_open edit
Doing a c# (.net 1.1 / visual studio 2003 / winxp) project with a (so far) small db (5 tables, max 1000 lines per table). About 10% of startups will fail with a NullReferenceException in sqlite3_open().

This is on a dell optiplex, win xp pro sp 2, 1G ram, P4@2.6GHz, using c# in visual studio 2003 version 7.1.3088 with .net framework 1.1.4322 sp 1.

Methods in Sqlite3.dll are imported with dllimport like this:

  [DllImport("Sqlite3.Dll", EntryPoint="sqlite3_open")]
  public static extern int sqlite3_open( string filename, out IntPtr dbhandle );

This would seem to be an error on my part or .net, except that this is the first sqlite call in the entire application and it fails only sometimes and the file name parameter is hard-coded. 90% of the time the application works fine. Of course, it could be .net messing things up. Anyway I can't find this on the web, sorry for taking up your time if it's not a bug. And no, I haven't got a program specifically for testing this thing.

Below is the stack trace for the System.NullReferenceException, the top line is the call into sqlite3.dll. The filename parameter is not null, it is "C:\DOCUMENTS AND SETTINGS\MARTIN.WANGEL\MY DOCUMENTS\VISUAL STUDIO PROJECTS\SOLUTION\BIN\DEBUG\SOLUTIONDATA.SQLITE" and it is thoroughly checked for null and for emptiness.

  at Solution.mwDatabaseSQLite.sqlite3_open(String filename, IntPtr& dbhandle)\r\n

  at Solution.mwDatabaseSQLite.Open() in c:\\documents and settings\\martin\\my documents\\visual studio projects\\solution\\mwdatabaseclasses.cs:line 149\r\n

  at Solution.DBUtils.VerifyDatabase(Type t, String connstr, dbtype dbtyp) in c:\\documents and settings\\martin\\my documents\\visual studio projects\\solution\\dbutils.cs:line 287\r\n

  at Solution.FrmMain..ctor() in c:\\documents and settings\\martin\\my documents\\visual studio projects\\solution\\form1.cs:line 1499\r\n

  at Solution.FrmMain.Main() in c:\\documents and settings\\martin\\my documents\\visual studio projects\\solution\\form1.cs:line 1285

/Martin

2005-Oct-27 21:12:22 by anonymous:
Maybe you should look at "Wiki", "SQLite wrappers", ".NET Framework" for an other way to do this.


2005-Oct-28 00:39:27 by anonymous:
the calling conventions of standard built DLL are __cdecl, not __stdcall (or WINAPI)... check if .NET Framework is able to call __cdecl import functions... in my mind it will call only STDCALL routines (which is the default on WIN32 API)
 
1502 active 2005 Oct anonymous Unknown Pending   anonymous 2 3 When selected in a union, view column names are incorrect. edit
When used in a union, a view transfers an underlying table's column names into the result set. The expected result is that column names in the second half (or further) of the union needn't match those in the first. Problem is also visible in TCL binding.
.mode columns
.headers on

select 'Create two tables, with nasty column names.' as remark;
create table t_a (c_a integer);
create table t_b (c_a integer);

select 'Create two views which each alias the column names of the above tables.' as remark;
create view v_a as select c_a as pretty from t_a;
create view v_b as select c_a as pretty from t_b;

select 'Insert some data' as remark;
insert into t_a values (1);
insert into t_b values (2);

select 'Notice that the views work fine by themselves.' as remark;
select 'The column names are both as we asked.' as remark;
select pretty from v_a;
select pretty from v_b;

select 'Notice that used in concert, with a join, the column name is now wrong.' as remark;
select pretty from v_a
 union select pretty from v_b;

select 'Aliasing the name of the column in the first half of the join is no help.' as remark;
select pretty as pretty from v_a
 union select pretty from v_b;

select 'Alias the name of the column in the second half of the join "fixes" the result.' as remark;
select pretty from v_a
 union select pretty as pretty from v_b;
2005-Oct-25 03:05:27 by anonymous:
same as ticket 1228


2005-Oct-26 07:59:17 by anonymous:
see also #1327
 
1504 active 2005 Nov anonymous Pager Pending     1 3 Multithreaded DB lock not released using Begin/Commit between threads edit
When using transaction-based insertion of rows in v3.2.7 in a multi-threaded environment, one thread appears not to release the database lock for any competing threads to be able to issue an "INSERT" statement (the thread issuing the "COMMIT" apparently). This problem does not appear in v2.8.16. The problem also appears in Linux (RH-9) as well.

I'm attaching "testsqlite.c", a test application (compiled in a WIN32 environment) that will duplicate the issue (Define SQLITE_2_8_16 or SQLITE_3_2_7 depending on the version of SQLite library to test against).

Direct any questions to Erik -> lonepenguin@hotmail.com

Thank You.

 
1508 active 2005 Nov anonymous BTree Pending     1 3 sqlite 2.8.16 crashes on 64-bit / strict memory alignment archs edit
A few months ago, sqlite3 was fixed on 64-bit / strict memory alignment architectures. Would it be possible for those fixes to be backported to the version_2 code? I have an OpenBSD/sparc64 machine which I can provide ssh access to (as I did before to drh). I know sqlite2 is mostly unsupported, but as php5 uses sqlite2, it would be nice to have these fixes backported.
2006-Jan-05 02:28:23 by anonymous:
with the attached patch, sqlite 2.8.17 passes all the regressions tests on openbsd/amd64 and openbsd/sparc64.
 
1515 active 2005 Nov anonymous Unknown Pending     4 3 Quotes incorrect included in colum names. edit
The names of columns returns by sqlite_exec has the quotes which has written in the query. I will try to show with examples in the sqlite command line.

  create table test("Full Name" varchar(30), "Login" varchar(15), Age integer);
  insert into test ("Full Name", "Login", Age) values ("Enrique Esquivel", "the_kique", 24);
  .headers on
  select * from test;

SQLite returns:

  Full Name|Login|Age
  Enrique Esquivel|the_kique|24

But when write:

  select "Full Name", "Login", Age from test;

returns:

  "Full Name"|"Login"|Age
  Enrique Esquivel|the_kique|24

Moreover when quote all fields:

  select "Full Name", "Login", "Age" from test;

returns:

  "Full Name"|"Login"|"Age"
  Enrique Esquivel|the_kique|24

Also:

  select [Full Name], [Login], [Age] from test;

SQLite returns wrong:

  [Full Name]|[Login]|[Age]
  Enrique Esquivel|the_kique|24

The quotes should be used for SQLite only for understand the identifiers, the fields in result must be unquoted. Try to test with other dbms and anyone has this behavior.

 
1521 active 2005 Nov anonymous Unknown Pending     3 2 ORDER BY sorts incorrectly with aliased fields edit
When executing a SELECT call and aliasing fields that already exist in the table, sorting does not work correctly on the aliased fields.

Here's a quick example:

  CREATE TABLE sort_table (name, name_alt);
  INSERT INTO sort_table (name, name_alt) VALUES ("a", "z");
  INSERT INTO sort_table (name, name_alt) VALUES ("b", "y");
  INSERT INTO sort_table (name, name_alt) VALUES ("c", "x");

This simple query works correctly:

  sqlite> SELECT name_alt FROM sort_table ORDER BY name_alt;
  name_alt
  x
  y
  z

Aliasing name_alt as name throws off the sorter:

  sqlite> SELECT name_alt AS name FROM sort_table ORDER BY name;
  name
  z
  y
  x

The results should be the same as in the first query and it works correctly in MySql. I'm trying to use this kind of query for a translation library. The only workaround I can think of is something like this:

  SELECT name_alt AS name, name_alt FROM sort_table ORDER BY name_alt;

This works, but is not ideal.

2005-Nov-12 17:28:20 by anonymous:
In cases like this you can use ordinal numbers as arguments to ORDER BY; replacing the last 'name' with '1' (no quotes) returns the correct results.


2005-Nov-12 19:20:24 by anonymous:
Yes, this does indeed work, and it makes for a simpler hack than my original one. This is still not the correct behavior though, and should be fixed so that a hack is not required at all.


2005-Nov-13 15:36:22 by anonymous:
Citing one database's behavior is interesting, but can you quote the paragraph in the SQL standard that shows the behavior you seek is correct?


2005-Nov-13 17:19:37 by anonymous:
Whether or not its in the SQL standard is somewhat irrelevent if you keep up to date with the mailing list. "Expected behavior", "how do all the other databases do it" and "makes sense" are often the governing factors in implementing changes to SQLite. In this case, proposing this change meets all 3 criteria.


2005-Nov-13 18:03:49 by anonymous:
SQLite does it one way. MySQL does it another way. Both behaviors could be argued to be correct. The SQL standard is certainly a good way to decide on a valid behavior. Besides, the way MySQL does it is not necessarily representative of "all other databases". At least list the output of a few major databases before making such an assumption.


2005-Nov-13 20:22:40 by anonymous:
I'm the guy who originally filed the ticket. I'm just an average guy, without the time or resources to run this on lots of different databases. The sql standards are not free, and I don't have a copy of any of them. I've also had very little luck finding information on the internet.

All that said, I believe it would be difficult to argue that the current behavior in SQLite is correct. In all other cases, the aliased column is accepted as an ORDER BY field. If I didn't want to override the original field for the purposes of the query, why would I explicitly do so? On the other hand, there are good reasons for wanting to explicitly override the column, for goals that cannot otherwise be achieved without hacks. If I didn't want to override the field, I could get the desired effect by doing nothing.


2005-Nov-14 00:08:30 by anonymous:
This should really be taken to the mailing list; there are people there with access to many different databases and some of them own copies of the SQL standard. This issue of name resolution also touches on the concerns expressed in #1111, #1213, and #1228.


2005-Nov-14 03:40:28 by anonymous:
What should the following query return?

  SELECT name_alt AS name, * FROM sort_table ORDER BY name;

Who knows - it is ambiguous.


2005-Nov-14 03:57:44 by anonymous:
I don't understand your point. Your example is ambiguous because there are two columns with the same name in the result set. The example in the original ticket is not ambiguous and should give the expected result.


2005-Nov-14 05:08:06 by anonymous:
The original query is ambiguous because you can refer to columns not explicitly mentioned in the SELECT to ORDER BY. No different from:

   select a, b, c from foo order by d;


2005-Nov-14 13:41:32 by anonymous:
Not true: SELECT name_alt AS name FROM sort_table ORDER BY name. The field referenced by ORDER BY does appear in the SELECT clause, after the AS. This is perfectly legitimate and the preferred way of sorting expressions.


2005-Nov-14 14:33:21 by anonymous:
"ORDER BY name" is abmiguous because it can refer to the name_alt column via the alias OR the original name column from sort_table. You can ORDER BY things not mentioned in the SELECT.

  SELECT name_alt AS name FROM sort_table ORDER BY name


2005-Nov-14 16:26:42 by anonymous:
Yeah, true -- but it's pretty obvious which one we're referring to, since we explicitly aliased the original field for the purposes of this query.


2005-Nov-14 16:48:43 by anonymous:
It's only obvious that you explicitly created an ambiguous column and that is undefined behaviour. Whether it happens to appear in the SELECT is not relevant. If you can demonstrate that the majority of databases support MySQL's behaviour, then that's another matter.


2005-Nov-14 19:35:25 by anonymous:
I just verified that this works as I expected on MySQL, PostgreSQL, and MS SQL Server. I don't have access to Oracle.
 
1522 active 2005 Nov anonymous Unknown Pending     1 3 Make test fails in manydb 1.82-3.299 mac os x 10.4.3 ppc edit
OS: Mac OS X 10.4.3 ppc

Compiler: powerpc-apple-darwin8-gcc-4.0.1 (GCC) 4.0.1 (Apple Computer, Inc. build 5247)

While running "make test" from a cvs checkout on Sun Nov 13 19:01:54 PST 2005, I get these errors:

653 errors out of 23390 tests

Failures on these tests: manydb-1.82 manydb-1.83 manydb-1.84 ................. manydb-3.296 manydb-3.297 manydb-3.298 manydb-3.299

2005-Nov-15 01:13:13 by drh:
These failures likely result from running out of file descriptors. The manydb tests need about 1000 file descriptors. Linux provides this many (on most distributions). But perhaps Mac OS X does not. Does anybody know?
 
1541 active 2005 Nov anonymous Unknown Pending     1 1 Ticket #924 not fixed in the 2.8 branch edit
Ticket 924: http://www.sqlite.org/cvstrac/tktview?tn=924

This problem seams to have been fixed on the 3.2 branch but is still in the 2.8.16

 
1546 active 2005 Nov anonymous   Pending     1 1 Creating unique index on non-unique column leads to corr. on SQLite2 edit
Like ticket #1115, SQLite version 2 suffer too from the bug where :

  BEGIN;
  CREATE TABLE t1(a);
  INSERT INTO t1 VALUES(1);
  INSERT INTO t1 VALUES(1);
  CREATE UNIQUE INDEX i1 ON t1(a);
  COMMIT;
  PRAGMA integrity_check

fails.

When "CREATE UNIQUE INDEX" fails within a transaction, (and within a transaction only), the index is still created.

 
1573 active 2005 Dec anonymous Shell Pending     3 3 Bad CSV output when data contains double quotes edit
SQLite 3.2.7 emits invalid CSV when a field value contains double quotes (or at least, it's CSV that Gnumeric cannot parse). Here's an example:

  sqlite> create table foo (a text, b text);
  sqlite> insert into foo values ("hello", "world");
  sqlite> insert into foo values ("mr.", "o'reilly");
  sqlite> insert into foo values ('12" EP', 'blah');
  sqlite> .mode csv
  sqlite> .output foo.csv
  sqlite> select * from foo;

Here's what foo.csv looks like:

  $ cat foo.csv
  "hello","world"
  "mr.","o'reilly"
  "12" EP","blah"

Note the ambiguous quoting on line 3. If I load this file into Gnumeric, it parses the first two lines just fine. But the last line confuses it.

It appears that doubling the quote works -- at least for Gnumeric's CSV parser. That is, if I edit foo.csv to

  "hello","world"
  "mr.","o'reilly"
  "12"" EP","blah"

then it's OK.

This is basically a duplicate of [1312] and related shell problem reports, though it provides a better test case than the previous reports.
 
1598 review 2006 Jan anonymous   Pending     3 2 Incorrect case-insensitive comparison of non-latin UTF-8 characters edit
Sqlite incorrectly compares case-insensitivly UTF-8 non-latin characters. I created a patch that fixes this problem and posted it to the mailing list. I wonder if someone could review my patch and eventually include it in the main project.

Regards Stanislav Nikolov

2006-Jan-10 22:49:08 by drh:
The sqlite3_create_collation() and sqlite3_create_function() APIs exists for the purpose of allowing users to define comparisons and any other operations in any way they see fit. There is no need to make changes to the SQLite core to accomodate cyrillic comparisons. Indeed, there are good reasons not to, namely if we correctly compare cyrillic, we should also need to correctly compare chinese, japanese, and korean to name but a few. Very quickly the comparison functions can grow to be many many times larger than the rest of SQLite. We conclude, therefore, that this is all best left to the discretion of the programmer who uses SQLite in their project. Hence we provide the afore mentioned sqlite3_create_collation() and sqlite3_create_function() APIs.


2006-Jan-11 00:24:50 by anonymous:
Okay, let me try that again. First, the patch I created does not correspond only to Cyrillic letters, but also to the Greek and the accented characters up to U+044F. According to UNICODE.ORG there are actually five alphabets in the world (of which one does not use cases anymore) that have different cases:

(from http://www.unicode.org/reports/tr21/tr21-5.html#Introduction) Case is a normative property of characters in specific alphabets (Latin, Greek, Cyrillic, Armenian, and archaic Georgian) whereby characters are considered to be variants of a single letter. These variants, which may differ markedly in shape and size, are called the uppercase letter (also known as capital or majuscule) and the lower&#173;case letter (also known as small or minuscule). The uppercase letter is generally larger than the lowercase letter. Alphabets with case differences are called bicameral; those without are called unicameral.

Therefore, I don't think someone will need support for case-insensitive comparison for Japanese, Chinese or Korean characters and I guess that adding support for the remaining Armenian alphabet is a matter of minutes and will not add up to the complexity of the code. Of course, perhaps it is possible for every project and/or developer to design their own "collation schemes" but I don't find it very practical. I can't really see the reason behind rejecting the patch. Perhaps you could actually look at it ?

Regards, Stanislav Nikolov


2006-Jan-11 00:33:19 by drh:
Please attach the patch to this ticket.


2006-Jan-11 00:52:38 by drh:
OK, I was able to reconstruct the patch from the mailing list.

I observer that as written it increases the size of the SQLite library by a little over 4KiB. That might not seem like much, but embedded device manufacturers (that is to say, most of my paying customers) are very sensitive to this kind of library size growth. I will look into reducing the size somewhat and getting it into a future release as a compile-time option.


2006-Jan-11 03:10:09 by drh:
Based on what I can glean from http://www.unicode.org/Public/UNIDATA/CaseFolding.txt, the case folding table in the patch seems to be incomplete. A full unicode case folding table would need to be much larger.

Perhaps somebody with more experience in unicode case folding can comment.


2006-Jan-11 09:52:38 by anonymous:
I think that the size of the library could be effectivly shrunk if we don't use an array, because over 50% of the information is redundant. I think that the same effect could be achieved by changing sqlite3UpperToLower[] to a function, and in there to check for the ranges of the capital letters (that is, we need to check for 4-5 different regions and return x+20 ort x+1 for capital letters for example). I can try do that?


2006-Jun-08 10:32:06 by anonymous:
Has anybody worked on this lately? This is quite an issue if you happen to use non-latin chars.

Keep up the good work.

Anze


2006-Oct-11 10:58:15 by anonymous:
could you tell me how to run the patch for WindowXP?

Thanks a lot.

 
1622 active 2006 Jan danielk1977   Pending     1 1 Compiling with OMIT_PRAGMA causes an error in the test suite edit
Compiling with OMIT_PRAGMA causes an error in the test suite. The error is a Tcl level error thrown by a [db eval] command when it encounters the unknown SQL keyword "PRAGMA".
 
1633 active 2006 Jan anonymous VDBE Pending     3 3 sqlite3_step returns SQLITE_ERROR when interrupted edit
When interrupting an execution of sqlite3_step, sqlite3_step will return a generic SQLITE_ERROR instead of the SQLITE_INTERRUPT error code I'd expect. This is because although in vdbe.c:4427 the SQLITE_INTERRUPT result code is set to the internal 'rc' field of the database connection, a plain SQLITE_ERROR is returned:

   vdbe_halt:
     if( rc ){
       p->rc = rc;
       rc = SQLITE_ERROR;
     }else{
       rc = SQLITE_DONE;
     }

The SQLITE_ERROR returned is then also stored as the errCode inside the db handle by the calling function, so there's no way to find out whether the error occured due to an interrupt or some other error (since sqlite3_error() will return SQLITE_ERROR as well).

I'd like to ignore interrupt errors where I call sqlite3_step, since those are not of importance to my users, but with the current scheme, I have no way of finding out whether an sqlite3_interrupt causes the error or whether it's a serious error...

You get a more specific error-code (for example SQLITE_INTERRUPT) when you call sqlite3_finalize() or sqlite3_reset() on the statement.

There shouldn't be any reason not to call one of these functions after sqlite3_step() returns SQLITE_ERROR, as you cannot do anything else with the statement handle at that point anyway.


2006-Apr-26 11:58:01 by anonymous:
Well, I get SQLITE_ERROR in sqlite3_finalize(), too after I interrupted the query, so I can't even find out at finalize time.

However, it might be interesting to see the real error when stepping: in our C++ wrapper, the finalize occurs very late and we usually raise database exceptions when errors occur while stepping. I would really like to raise the proper exception (some sqlite_execution_interrupted exception) when the query was interrupted instead of raising some generic "sql error" exception...


2006-Apr-26 12:24:02 by anonymous:
Sorry, my previous comment was a bit too quick: I have to admit that most of the time, I do in fact get SQLITE_INTERRUPTED as result from sqlite3_finalize, but every now and then, I do get SQL_ERROR. Maybe this happens when I try to interrupt just before the actual statement execution has started or when execution has just finished?


2006-Jul-26 13:26:19 by anonymous:
This might, of course, be related to my incorrect usage of sqlite3_interrupt from a different thread (Ticket #1897) ?
 
1700 active 2006 Mar anonymous Parser Pending     2 2 Handling column names for aliased queries is broken edit
The following query does not work,

SELECT DISTINCT * FROM (SELECT t1.ID FROM GR_ADDRESS t1 WHERE t1.ID > 1 UNION ALL SELECT t1.ID FROM PERSON t1) t1 ORDER BY t1.ID DESC

but this one does,

SELECT DISTINCT * FROM (SELECT t1.ID FROM GR_ADDRESS t1 WHERE t1.ID > 1 UNION ALL SELECT t1.ID FROM PERSON t1 ORDER BY t1.ID DESC)

Dennis Cote responded with:

I think you have found another example of the problems SQLite has handling columns names.

The following log first shows what SQLite thinks the column name is for the query without the order by clause (i.e. t1.ID). Then we try to order by that column name, with or without the table alias. Both cases result in an error. Finally there is a work around that you could use that applies an alias to the selected columns in the two tables that are combined by the union operation.

		SQLite version 3.3.2
		Enter ".help" for instructions
		sqlite> create table GR_ADDRESS(id, data);
		sqlite> create table PERSON(id, data);
		sqlite> .mode column
		sqlite> .header on
		sqlite> insert into gr_address values(1, 10);
		sqlite> insert into person values(2, 20);
		sqlite> insert into gr_address values(3, 30);
		sqlite> SELECT DISTINCT *
		   ...> FROM
		   ...>     (SELECT t1.ID
		   ...>     FROM GR_ADDRESS t1
		   ...>     WHERE t1.ID > 1
		   ...> UNION ALL
		   ...>     SELECT t1.ID
		   ...>     FROM PERSON t1)
		   ...> t1;
		t1.ID
		----------
		3
		2
		sqlite> SELECT DISTINCT *
		   ...> FROM
		   ...>     (SELECT t1.ID
		   ...>     FROM GR_ADDRESS t1
		   ...>     WHERE t1.ID > 1
		   ...> UNION ALL
		   ...>     SELECT t1.ID
		   ...>     FROM PERSON t1)
		   ...> t1 ORDER BY t1.ID DESC;
		SQL error: no such column: t1.ID
		sqlite> SELECT DISTINCT *
		   ...> FROM
		   ...>     (SELECT t1.ID
		   ...>     FROM GR_ADDRESS t1
		   ...>     WHERE t1.ID > 1
		   ...> UNION ALL
		   ...>     SELECT t1.ID
		   ...>     FROM PERSON t1)
		   ...> t1 ORDER BY ID DESC;
		SQL error: no such column: ID
		sqlite> SELECT DISTINCT *
		   ...> FROM
		   ...>     (SELECT t1.ID as ID
		   ...>     FROM GR_ADDRESS t1
		   ...>     WHERE t1.ID > 1
		   ...> UNION ALL
		   ...>     SELECT t1.ID as ID
		   ...>     FROM PERSON t1)
		   ...> t1 ORDER BY t1.ID DESC;
		ID
		----------
		3
		2

You may also be interested in the discussion of a similar problem under ticket #1688.

 
1733 active 2006 Mar anonymous VDBE Pending   drh 4 3 Unaligned Access on ia64: aggregate_context ptr isn't 16-bytes aligned edit
There is a problem on ia64 with pointer returned by sqlite3_aggregate_context function. If the size requested is less than NBFS bytes, then the pointer returned is 8 bytes aligned while every pointer returned by allocator function must be 16-bytes aligned (the specification requires that the pointer is aligned so that every basic typed can be stored there and long double is 16 bytes on Itanium).

So if a user allocates, say, 24 bytes for his context, and the first member in his context happens to be a long double, he will get unaligned access exception. This will lead to performance hit on Linux and to crash on HP-UX, since no default SIGBUS handler is present on HP-UX (IIRC).

2006-Mar-27 10:37:37 by anonymous:
Additional details can be found in this mailing list thread: http://thread.gmane.org/gmane.comp.db.sqlite.general/18144
 
1735 active 2006 Mar anonymous Unknown Pending     1 3 Encoding problem edit
I use latin2 (iso-8859-2) encoding in my system. When operating on sqlite 3 I can insert data that contains national characters into a database (for example using sqlite3 console) and then when I select them back, I am given the proper result. But when I use sqlite driver from Qt4, which uses sqlite3_column_text16() to fetch data from the database, I don't get the expected result (meaning the conversion to UTF-16 probably messed things up). Now the problem can be in one of two places -- either sqlite3 console application doesn't use a proper conversion to convert from my locale encoding into its internal encoding or the database internal mechanisms mess some things up.

In short: sqlite3(somelatin2string) ==> SQLITE DMBS ==> sqlite3_column_text16() ==> garbage != somelatin2string

At first I thought this was Qt problem as data stored through sqlite console and retrieved from it was correct and data stored by Qt and retrieved by Qt was also correct whereas data stored by Qt and retrieved by sqlite3 console or stored by the console and retrieved by Qt was not correct. I contacted Qt support guys @ trolltech and talked about it and it looks like Qt side if fine -- it expects a UTF-16 encoded data (because it uses the function mentioned earlier) and it converts from UTF-16 to whatever encoding it needs (and vice versa). So the error is probably somewhere in the line between the console and the database itself or in the database internally. It could be that sqlite3 expects UTF-8 (or UTF-16) encoded data on input but is given ISO-8859-2 data (entered manually by me at the console).

2006-Mar-27 16:36:26 by anonymous:
The console app doesn't convert from your local code page to UTF-8 (or UTF-16).


2006-Mar-27 22:45:21 by anonymous:
It probably should, in the documentation of sqlite a suggested method of converting databases between versions 2 and 3 is:

  sqlite OLD.DB .dump | sqlite3 NEW.DB

Now sqlite outputs the data in "local" format and if sqlite3 doesn't encode it properly, such a conversion will be invalid because the incoming data won't be utf encoded.

A solution could be to do:

  sqlite OLD.DB .dump | iconv -f <something> -t UTF-8 | sqlite3 NEW.DB

But it is the console which should be responsible for the conversion. Also because otherwise using sqlite3 console on a non-utf system with a perfectly well UTF-8 encoded database will result in improper output too.

 
1742 active 2006 Mar anonymous Unknown Pending   drh 2 3 ORDER BY on more than one column causes a big slowdown edit
Put simply, any query which contains an ORDER BY clause that sorts on more than one column incurs a strange slowdown.

Running SQLite 3.3.4 on WindowsXPSP2 and on OS X 1.4.5, the behavior is similar; if the ORDER BY clause contains one column, the query is very fast; on two or more columns, it is terribly slow.

2006-Mar-28 23:58:15 by anonymous:
Also worth noting that this behavior seems to start with SQLite 3.3.x; earlier versions of SQLite handle multiple ORDER BY columns much faster.


2006-Mar-29 01:22:11 by anonymous:
Note also that this behavior is being exhibited when sorting on indexed columns


2006-Mar-29 01:50:38 by drh:
Some examples would be helpful.


2006-Mar-29 18:11:43 by anonymous:
Most definitely! I will attach a sample 3.3.4 database dump, that displays this behavior.
 
1743 active 2006 Mar anonymous Parser Pending     3 3 A very very deep IN statement failure edit
Ok the problem is simple. I need to create a VERY VERY large IN statement. The problem is SQLite seems to have a limit on either query length or depth of an IN statement. Here is my example

See attached 1

That would be a 2 levels deep In statement. I can only get up to 9 with SQLite but I need to get to 20. Since it works for 9 I can only assume that my 10 is correct even though the error is a syntax error. Below is the code that creates the select statement.

See attached 4

Attachment 2 and 3 show a 9 and 10 level respectively.

Thanks for your help

2006-Mar-30 21:30:51 by anonymous:

  Select "Wow !!" from "Wow !!"
  :-)
  Maybe VIEWs could help ??


2006-Mar-30 21:37:50 by anonymous:
This may be a work around for your problem. From looking at your sample SQL:

    SELECT * FROM xs
    where classname like '%Bonus_Pay_Weight_Entry%'
    or classname in
        (
        select parentname from xs
        where classname in
            (
            select parentname from xs
            where classname in
                (
                select parentname from xs
                where classname like '%Bonus_Pay_Weight_Entry%'
                )
            )
        )
    or classname in
        (
        select parentname from xs
        where classname in
            (
            select parentname from xs
            where classname like '%Bonus_Pay_Weight_Entry%'
            )
        )
    or classname in
        (
        select parentname from xs
        where classname like '%Bonus_Pay_Weight_Entry%'
        )
    ;

It seems you are trying to find all the parent classes of all the classes with this magic string in their name. If so, I think there is another way to do this. Instead of using a C program to build a huge SQL statement and then collect the results, use a different C program to execute a series of small SQL commands that generate the same result set.

The following series of SQL statements should generate the same set of results.

    create temp table xt as
        select classname from xs
        where classname like '%Bonus_Pay_Weight_Entry%';
    insert into xt select parentname from xs where classname in xt and parentname not in xt;
    select changes();
    insert into xt select parentname from xs where classname in xt and parentname not in xt;
    select changes();
    insert into xt select parentname from xs where classname in xt and parentname not in xt;
    select changes();
    ... repeat until changes returns zero
    select * from xs where classname in xt;
    drop table xt;

This can be execute by code that looks something like the following pseudo-C code.

    string sql;
    sql = "create temp table xt as select classname from xs where classname like '%Bonus_Pay_Weight_Entry%'";
    sqlite3_exec(db, sql);

    sql = "insert into xt select parentname from xs where classname in xt and parentname not in xt";
    sqlite3_stmt* extend = sqlite3_prepare(db, sql);

    sql = "select changes()"
    sqlite3_stmt* check = sqlite3_prepare(db, sql);

    int changes = 0;
    do {
        sqlite3_step(extend);
        sqlite3_reset(extend);
        sqlite3_step(check);
        changes = sqlite3_column_int(check, 0);
        sqlite3_reset(check);
    } while (changes > 0);

    sqlite3_finalize(extend);
    sqlite3_finalize(check);

    sql = "select * from xs where classname in xt";
    sqlite3_stmt* get = sqlite3_prepare(db, sql);

    int rc;
    do {
        rc = sqlite3_step(get);
        if (rc == SQLITE_DONE) break;
        // process a result row
    } while (1);

    sqlite3_finalize(get);

    sql = "drop table xt";
    sqlite3_exec(db, sql);


2006-Apr-05 17:25:30 by anonymous:
Where did you find the select changes(); function? I would like to find all the functions that SQLite has and their uses.

(and no I dont want the C API. I found that)


2006-Apr-05 18:53:08 by anonymous:
There is no complete listing of the functions in the documentation that I am aware of. Most are documented on this page http://www.sqlite.org/lang_expr.html but some are missing.

The ultimate list of the predefined functions is the source file func.c which implements all the functions. You can view it here http://www.sqlite.org/cvstrac/rlog?f=sqlite/src/func.c

 
1754 active 2006 Apr anonymous Pager Pending   anonymous 2 2 Version 3.3.5 error if SQLITE_OMIT_MEMORYDB is defined edit
My solution is to move the following code to de line the the syncJournal is "forwarded" at line 1809.

#ifndef SQLITE_OMIT_MEMORYDB
/*
** Clear a PgHistory block
*/
static void clearHistory(PgHistory *pHist){
  sqliteFree(pHist->pOrig);
  sqliteFree(pHist->pStmt);
  pHist->pOrig = 0;
  pHist->pStmt = 0;
}
#else
#define clearHistory(x)
#endif
 
1775 active 2006 Apr anonymous Unknown Pending     1 1 strftime() not working in Windows Mobile 2005 edit
The strftime() is not working in windows mobile 2005 pocket pc. I am using the beta version of visual studio 2005.
2006-Apr-17 13:10:20 by anonymous:
Is it a compile/link problem or a runtime problem?


2006-Apr-18 07:26:08 by anonymous:
It's a runtime problem. I am not able to get the dates formated using strftime(). datetime(), date() and time() are working properly.
 
1782 active 2006 Apr anonymous Unknown Pending     1 1 journal file exclusion edit
I have a long running process which opens a connection to a sqlite3 database, let's called it a.rdb. The connection is never closed during the life time of the process, and it's set to auto commit mode. Now, if I delete a.rdb, and re-create it again(the long running process is still holding a fd to the deleted file at this point), the long running process is still creating a.rdb-journal from time to time. To make it worse, if at that time, I use the sqlite3 command line to modify the database when the file a.rdb-journal exists, the file in a.rdb-journal is also played back into the new a.rdb file, which doesn't seem to be the correct behavior.

Is it the intended design?

Thanks, John

 
1783 active 2006 Apr anonymous Unknown Pending     3 2 insert times increase with growing table size (when indexed) edit
The time needed to insert (or update) entries in a table with an index on one of the fields increases with the size of the table. For large databases inserts become very slow (which I suppose is likely the problem in ticket #1547). sqlite2 does not have this scaling problem on inserts. (Some of our queries do not scale on sqlite2 however, making its use also impossible.)

  ----- example code -----
  package require dbi
  package require dbi_sqlite3
  dbi_sqlite3 db
  db create /tmp/test.db
  db open /tmp/test.db
  db exec {create table "region" (
  	"id" integer not null primary key,
  	"start" integer,
  	"end" integer
  )}
  db exec {create index "region_index" on "region"("start")}
  set num 1
  for {set j 1} {$j < 20} {incr j} {
  	puts [lindex [time {
  		db begin
  		for {set i 1} {$i < 100000} {incr i} {
  			set s [expr {round(rand()*1000000)}]
  			set e [expr {round(rand()*1000000)}]
  			db exec {
  				insert into "region"("id","start","end")
  				values(?,?,?)
  			} $num $s $e
  			incr num
  		}
  		db commit
  	}] 0]
  }

----- timings -----

5712186 6621934 9492997 13234978 14881322 19119044 25296162 26670866 35378986 35877042 44383517 54576510 53317621 63516664 76587973 73791188 88460462 101650099

 
1790 active 2006 May anonymous Pager Pending     3 3 :memory: performance difference between v2 and v3 edit
Please see the following link for details: http://www.mail-archive.com/sqlite-users%40sqlite.org/msg14937.html

Possible fix?

  RCS file: /sqlite/sqlite/src/pager.c,v
  retrieving revision 1.266
  diff -u -r1.266 pager.c
  --- pager.c     7 Apr 2006 13:54:47 -0000       1.266
  +++ pager.c     3 May 2006 19:02:17 -0000
  @@ -1663,7 +1663,7 @@
     pPager->memDb = memDb;
     pPager->readOnly = readOnly;
     /* pPager->needSync = 0; */
  -  pPager->noSync = pPager->tempFile || !useJournal;
  +  pPager->noSync = pPager->tempFile || !pPager->useJournal;
     pPager->fullSync = (pPager->noSync?0:1);
     /* pPager->pFirst = 0; */
     /* pPager->pFirstSynced = 0; */
2006-May-03 19:32:12 by drh:
The suggested change makes no difference in performance when I try it.


2006-May-03 21:41:24 by anonymous:
If transactions are not used, 85% of the time of this memory database benchmark is spent in pager_get_all_dirty_pages().

  Each sample counts as 0.01 seconds.
    %   cumulative   self              self     total
   time   seconds   seconds    calls  ms/call  ms/call  name
   85.25     31.20    31.20   100002     0.31     0.31  pager_get_all_dirty_pages
    1.39     31.71     0.51   100011     0.01     0.20  sqlite3VdbeExec
    1.17     32.14     0.43 10487713     0.00     0.00  parseCellPtr
    0.63     32.37     0.23 12943618     0.00     0.00  sqlite3VdbeSerialGet
    0.61     32.59     0.23  3432951     0.00     0.00  pager_lookup
    0.52     32.78     0.19  4849544     0.00     0.00  sqlite3VdbeRecordCompare
    0.44     32.95     0.16   400006     0.00     0.00  sqlite3BtreeMoveto
    0.41     33.09     0.15  2064924     0.00     0.00  sqlite3pager_get
    0.40     33.24     0.14  6471807     0.00     0.00  sqlite3MemCompare

                  0.06   31.25  100002/100002      sqlite3BtreeCommit [4]
  [5]     85.6    0.06   31.25  100002         sqlite3pager_commit [5]
                 31.20    0.00  100002/100002      pager_get_all_dirty_pages [6]
                  0.05    0.00  389365/389365      clearHistory [65]
  -----------------------------------------------
                 31.20    0.00  100002/100002      sqlite3pager_commit [5]
  [6]     85.2   31.20    0.00  100002         pager_get_all_dirty_pages [6]


2006-May-03 21:51:30 by anonymous:
Stats with BEGIN/COMMIT enabled:

  Each sample counts as 0.01 seconds.
    %   cumulative   self              self     total
   time   seconds   seconds    calls  ms/call  ms/call  name
   11.88      0.34     0.34  4849544     0.00     0.00  sqlite3VdbeRecordCompare
    8.16      0.56     0.23 10487713     0.00     0.00  parseCellPtr
    7.80      0.79     0.22 12943618     0.00     0.00  sqlite3VdbeSerialGet
    6.38      0.96     0.18   100013     0.00     0.03  sqlite3VdbeExec
    4.26      1.08     0.12    29816     0.00     0.02  balance_nonroot
    3.90      1.20     0.11  6471807     0.00     0.00  sqlite3MemCompare
    3.19      1.28     0.09  1964925     0.00     0.00  sqlite3pager_get
    3.19      1.38     0.09   400006     0.00     0.00  sqlite3BtreeMoveto
    2.84      1.46     0.08 19170231     0.00     0.00  get2byte
    2.66      1.53     0.07   700015     0.00     0.00  sqlite3VdbeSerialPut
    2.13      1.59     0.06   600993     0.00     0.00  sqlite3Malloc
    1.77      1.64     0.05  4400155     0.00     0.00  sqlite3pager_unref
    1.77      1.69     0.05  3332952     0.00     0.00  pager_lookup
    1.77      1.74     0.05  1418379     0.00     0.00  decodeFlags
    1.77      1.79     0.05  1332302     0.00     0.00  initPage
    1.60      1.83     0.04  5270826     0.00     0.00  findOverflowCell
    1.42      1.88     0.04 12181181     0.00     0.00  findCell
    1.42      1.92     0.04  4849549     0.00     0.00  fetchPayload
    1.42      1.96     0.04   359548     0.00     0.00  insertCell
    1.24      1.99     0.04  4896877     0.00     0.00  parseCell
    1.06      2.02     0.03  5284245     0.00     0.00  cellSizePtr
    1.06      2.05     0.03  3227291     0.00     0.00  binCollFunc
    1.06      2.08     0.03  2616113     0.00     0.00  _page_ref
    1.06      2.11     0.03  1368027     0.00     0.00  reparentPage
    1.06      2.14     0.03   934205     0.00     0.00  sqlite3GenericMalloc
    1.06      2.17     0.03   300010     0.00     0.00  sqlite3BtreeCursor
    0.89      2.19     0.03  2536689     0.00     0.00  get4byte
    0.89      2.22     0.03  1864920     0.00     0.00  getPage
...
    0.00      2.82     0.00        3     0.00     0.00  pager_get_all_dirty_pages

                  0.00    0.00       3/3           sqlite3BtreeCommit [116]
  [119]    0.0    0.00    0.00       3         sqlite3pager_commit [119]
                  0.00    0.00    6551/6551        clearHistory [118]
                  0.00    0.00       3/3           pager_get_all_dirty_pages [370]
  -----------------------------------------------
                  0.00    0.00       3/3           sqlite3pager_commit [119]
  [370]    0.0    0.00    0.00       3         pager_get_all_dirty_pages [370]


2006-May-03 22:27:35 by anonymous:
with the outer BEGIN/COMMIT disabled, the memory database benchmark stats:

  static PgHdr *pager_get_all_dirty_pages(Pager *pPager){
    // this point is reached 100,002 times
    PgHdr *p, *pList;
    pList = 0;
    for(p=pPager->pAll; p; p=p->pNextAll){
      // this point is reached 322,956,271 times
      if( p->dirty ){
        // this point is reached 389,365 times
        p->pDirty = pList;
        pList = p;
      }
    }
    return pList;
  }


2006-May-04 05:23:08 by anonymous:
This patch makes the test (with transaction) run 7% faster for gcc 3.4.4 with -O2. At -O3, gcc performs the inlining of these functions even without the inline hint, so this patch has no effect.

  RCS file: /sqlite/sqlite/src/btree.c,v
  retrieving revision 1.324
  diff -u -3 -p -r1.324 btree.c
  --- btree.c     4 Apr 2006 01:54:55 -0000       1.324
  +++ btree.c     4 May 2006 05:12:35 -0000
  @@ -439,17 +439,17 @@ static int checkReadLocks(BtShared*,Pgno
   /*
   ** Read or write a two- and four-byte big-endian integer values.
   */
  -static u32 get2byte(unsigned char *p){
  +inline static u32 get2byte(unsigned char *p){
     return (p[0]<<8) | p[1];
   }
  -static u32 get4byte(unsigned char *p){
  +inline static u32 get4byte(unsigned char *p){
     return (p[0]<<24) | (p[1]<<16) | (p[2]<<8) | p[3];
   }
  -static void put2byte(unsigned char *p, u32 v){
  +inline static void put2byte(unsigned char *p, u32 v){
     p[0] = v>>8;
     p[1] = v;
   }
  -static void put4byte(unsigned char *p, u32 v){
  +inline static void put4byte(unsigned char *p, u32 v){
     p[0] = v>>24;
     p[1] = v>>16;
     p[2] = v>>8;


2006-May-04 19:44:57 by anonymous:
I just want to confirm that a file database is faster than a memory database for 3.3.5+. Are these numbers correct? 43,478 inserts/second best case for file for 3.3.5+ and 40,000 inserts/second best case for memory? Even with the OS caching the entire database file entirely in RAM, this finding is quite surprising.

  Test    DB      IDX TX  RC      3.3.5+  3.3.5   2.8.17
  1       mem     n   y   1000000 40000   33333   76923
  2       mem     y   y   1000000 27027   22727   58824
  3       mem     n   n   1000000 35714   5263    83333
  4       mem     y   n   1000000 24390   2778    62500
  5       file    n   y   1000000 43478   35714   40000
  6       file    y   y   1000000 28571   24390   23256
  7       file    n   n   1000    11      11      13
  8       file    y   n   1000    9       10      13

http://www.sqlite.org/cvstrac/attach_get/256/sqlite_speed.txt


2006-May-04 20:19:18 by anonymous:
I'm seeing slightly different results. The memory database using a transaction is (slightly) faster than the file-based database using a transaction.

Timings on 3.3.5+ on Windows XP, gcc 3.4.4 -O3 -fomit-frame-pointer

        IDX  TX   # inserts   wall time  inserts/sec
        ---  ---  ---------  ----------  -----------
  mem   no   no     100,000        4.8s       20,833
  mem   no   yes    100,000        4.3s       23,255
  file  no   yes    100,000        4.7s       21,276
  file  no   no       1,000       99.8s           10

...things get worse for :memory: as you increase the number of inserts, while the file database numbers remain constant:

        IDX  TX   # inserts   wall time  inserts/sec
        ---  ---  ---------  ----------  -----------
  mem   no   yes  1,000,000       48.5s       20,638
  mem   no   yes  2,000,000      118.6s       16,863
  mem   no   yes  4,000,000      364.7s       10,967
  file  no   yes  1,000,000       46.8s       21,354
  file  no   yes  2,000,000       93.8s       21,321
  file  no   yes  4,000,000      187.5s       21,333

Do Linux users get similar results?

Considering I have 512K CPU L2 cache, I wonder if there's some CPU cache effect going on here with the way the :memory: db is allocated.


2006-May-04 21:35:07 by anonymous:
It seems there is some quadratic behavior in pager_lookup (latest CVS). 52% of the time is spent in that function. Profile data from :memory: db, TX on, no IDX, 4 million inserts:

  /*
  ** Find a page in the hash table given its page number.  Return
  ** a pointer to the page or NULL if not found.
  */
  static PgHdr *pager_lookup(Pager *pPager, Pgno pgno){
    PgHdr *p = pPager->aHash[pager_hash(pgno)];
    while( p && p->pgno!=pgno ){
      p = p->pNextHash;
    }
    return p;
  }

    %   cumulative   self               self     total
   time   seconds   seconds     calls  ms/call  ms/call  name
   51.97    118.31   118.31 119658386     0.00     0.00  pager_lookup
    4.36    128.25     9.94   4000009     0.00     0.06  sqlite3VdbeExec
    3.06    135.21     6.96 315629923     0.00     0.00  parseCellPtr
    3.05    142.16     6.95 171797186     0.00     0.00  sqlite3VdbeRecordCompare
    2.67    148.22     6.07  12000005     0.00     0.01  sqlite3BtreeMoveto
    2.14    153.10     4.88 343594380     0.00     0.00  sqlite3VdbeSerialGet
    1.68    156.93     3.83 171797188     0.00     0.00  sqlite3MemCompare
    1.63    160.65     3.72  77995781     0.00     0.00  sqlite3pager_get
    1.60    164.29     3.65 169734946     0.00     0.00  sqlite3pager_unref
    1.58    167.88     3.59 654100795     0.00     0.00  get2byte
    1.30    170.84     2.95    973877     0.00     0.07  balance_nonroot
    1.27    173.74     2.90  56939555     0.00     0.00  initPage
    1.24    176.56     2.83 171797188     0.00     0.00  binCollFunc
    0.93    178.69     2.12 386371475     0.00     0.00  findCell
    0.86    180.65     1.96  96207437     0.00     0.00  pageDestructor
    0.83    182.53     1.89  95976540     0.00     0.00  _page_ref
    0.80    184.36     1.83   2708031     0.00     0.00  assemblePage
    0.80    186.19     1.82  41662605     0.00     0.00  reparentPage
    0.74    187.88     1.70 171797188     0.00     0.00  fetchPayload
    0.73    189.55     1.67  73995778     0.00     0.00  getPage
    0.67    191.07     1.52  59647596     0.00     0.00  decodeFlags
    0.63    192.51     1.44 132945443     0.00     0.00  findOverflowCell
    0.62    193.93     1.41  40148167     0.00     0.00  sqlite3PutVarint
    0.59    195.27     1.34 134687272     0.00     0.00  releasePage
    0.59    196.62     1.34  73764879     0.00     0.00  getAndInitPage
    0.54    197.84     1.22   8000003     0.00     0.02  sqlite3BtreeInsert
    0.52    199.01     1.18  60000030     0.00     0.00  sqlite3VdbeSerialType
    0.52    200.19     1.18  24000011     0.00     0.00  moveToRoot
    0.49    201.30     1.10 179797130     0.00     0.00  getCellInfo
    0.43    202.28     0.98   9882306     0.00     0.00  insertCell
    0.42    203.22     0.94  47288132     0.00     0.00  moveToChild
    0.40    204.15     0.92 173434760     0.00     0.00  parseCell
    0.40    205.06     0.91  95806930     0.00     0.00  get4byte
    0.34    205.83     0.78  41662605     0.00     0.00  sqlite3pager_lookup
    0.33    206.57     0.74 165099370     0.00     0.00  sqlite3MallocFailed
    0.32    207.31     0.73  20000010     0.00     0.00  sqlite3VdbeSerialPut
    0.31    208.02     0.71   8000015     0.00     0.00  sqlite3VdbeHalt
    0.30    208.70     0.68  27052986     0.00     0.00  sqlite3GetVarint
    0.28    209.33     0.63 174637767     0.00     0.00  put2byte
    0.27    209.96     0.62   8000006     0.00     0.00  sqlite3BtreeCursor
    0.26    210.54     0.59   8148152     0.00     0.00  fillInCell
    0.25    211.12     0.57   3385610     0.00     0.01  reparentChildPages
    0.25    211.69     0.57  16000006     0.00     0.00  checkReadLocks
    0.22    212.19     0.51  48000093     0.00     0.00  sqlite3VdbeFreeCursor
    0.22    212.69     0.50 133898861     0.00     0.00  cellSizePtr
    0.22    213.19     0.50  24000010     0.00     0.00  popStack
    0.20    213.65     0.46  50076560     0.00     0.00  sqlite3pager_ref
    0.20    214.10     0.45                              pager_reset
    0.19    214.54     0.44   8000024     0.00     0.00  closeAllCursors
    0.19    214.97     0.42  12000024     0.00     0.00  sqlite3VdbeMemMakeWriteable
    0.18    215.38     0.41  32000052     0.00     0.00  sqlite3VdbeMemSetStr
    0.18    215.78     0.40  11616158     0.00     0.00  allocateSpace
    0.17    216.16     0.39   8000000     0.00     0.00  bindText
    0.16    216.51     0.35  25098767     0.00     0.00  sqlite3MallocRaw
    0.16    216.87     0.35   8000005     0.00     0.00  sqlite3BtreeCloseCursor
    0.15    217.22     0.34  45560699     0.00     0.00  sqlite3FreeX
    0.15    217.56     0.34  36000014     0.00     0.00  sqlite3VarintLen
    0.15    217.90     0.34  36000009     0.00     0.00  sqlite3VdbeMemShallowCopy
    0.14    218.22     0.33  47999969     0.00     0.00  sqlite3VdbeSerialTypeLen
    0.14    218.54     0.33   4000008     0.00     0.00  sqlite3VdbeMakeReady

  -----------------------------------------------
                 41.19    0.00 41662605/119658386     sqlite3pager_lookup [15]
                 77.12    0.00 77995781/119658386     sqlite3pager_get [8]
  [5]     52.0  118.31    0.00 119658386         pager_lookup [5]
  -----------------------------------------------
                  0.19    4.02 4000003/77995781     sqlite3BtreeGetMeta [28]
                  3.53   74.30 73995778/77995781     getPage [9]
  [8]     36.0    3.72   78.31 77995781         sqlite3pager_get [8]
                 77.12    0.00 77995781/119658386     pager_lookup [5]
                  1.12    0.00 56939550/95976540     _page_ref [40]
                  0.03    0.00  230897/230897      page_remove_from_stmt_list [139]
                  0.03    0.00  230897/230897      makeClean [138]
                  0.01    0.00  230897/461804      sqlite3pager_pagecount [150]
                  0.00    0.00  230897/25098767     sqlite3MallocRaw [58]
  -----------------------------------------------
                  1.82   44.94 41662605/41662605     reparentChildPages [13]
  [14]    20.5    1.82   44.94 41662605         reparentPage [14]
                  0.78   41.96 41662605/41662605     sqlite3pager_lookup [15]
                  2.21    0.00 41672966/131189801     sqlite3pager_unref <cycle 2> [31]
                  0.00    0.00   93099/50076560     sqlite3pager_ref [75]

  -----------------------------------------------
                  0.78   41.96 41662605/41662605     reparentPage [14]
  [15]    18.8    0.78   41.96 41662605         sqlite3pager_lookup [15]
                 41.19    0.00 41662605/119658386     pager_lookup [5]
                  0.77    0.00 39036990/95976540     _page_ref [40]
  -----------------------------------------------
                  0.77    0.00 39036990/95976540     sqlite3pager_lookup [15]
                  1.12    0.00 56939550/95976540     sqlite3pager_get [8]
  [40]     0.8    1.89    0.00 95976540         _page_ref [40]


2006-May-04 21:41:37 by anonymous:
I guess increasing this array size is in order:

  PgHdr *aHash[N_PG_HASH];    /* Hash table to map page number to PgHdr */

Too many hash collisions leading to growing linked lists in buckets.

Or perhaps pager_hash has to be replaced with a better hash function.


2006-May-04 22:04:47 by anonymous:
Increasing the size of N_PG_HASH to 8192 seems to help the "4 million insert in a transaction into a memory database" benchmark. It now runs in 203.5 seconds (19656 inserts/sec), as opposed to 364.7 seconds (10967 inserts/sec) previously. This is closer to the 187.5 seconds for the file-based database timing.


2006-May-04 22:13:16 by anonymous:
Increasing N_PG_HASH to 16384 yields 21,052 inserts/second for a 4 million insert single-transaction :memory: database no-index run. This is very close to the file database figure of 21,333 inserts/second.


2006-May-04 22:23:19 by anonymous:
Setting N_PG_HASH to 32768 yields 21,621 inserts/second in the 4M insert s in a single-transaction in a memory db test. This is marginally faster than the file based database timing. Increasing N_PG_HASH has diminishing returns after 16384.


2006-May-05 15:33:31 by anonymous:
You should get the same effect if you increase the page size instead of increasing the size of the hash table. With a larger page size there will be fewer pages to be managed by the hash table. This might be a better solution for many applications. A hash table with 32K entries occupies 128K of RAM, whether it is used or not.


2006-May-05 19:37:51 by anonymous:
128K of RAM when dealing with a 230M :memory: database is not terribly significant.

Here's the timings for various N_PG_HASH and SQLITE_DEFAULT_PAGE_SIZE values for 4 million inserts into a :memory: database in a single transaction:

  N_PG_HASH  SQLITE_DEFAULT_PAGE_SIZE  inserts/sec
  ---------  ------------------------  -----------
      16384                      4096       21,622
      32768                      1024       21,621
       8192                      8192       20,513
       4096                      4096       20,101
       4096                      8192       19,417
       2048                      4096       16,878
       2048                      8192       16,598
       2048                     16384       15,038
       2048                     32768       13,937
       2048                      1024       10,782

So it seems the default values of N_PG_HASH and SQLITE_DEFAULT_PAGE_SIZE should be raised.


2006-May-05 21:34:01 by anonymous:
My point was that most users do not have 230 MB memory databases, so having a large hash table which is fixed at that size may be a burden. 128K for the hash table is a lot if you only have 128K in your memmory database.

I agree that increasing these values would seem to provide a substantial performance increase at little cost. I would suggest using the 4K hash table and the 4K page size. These values are close to the current values. Many users have reported a general speed improvement using a page size of 4K which matches the value used by WinXP (and think many other Os's as well) for disk I/O blocks. These values nearly double the insert rate over the current default values. The fixed size hash table only takes twice the space.


2006-May-06 14:54:32 by anonymous:
Memory page speed should be as fast as possible as it effects the general performance of SQLite. Perhaps a static hash table is not the best data structure here. Don't temp tables and intermediate select results on file-based tables use memory-based pages? Making memory page speed as fast as possible will improve overall SQLite performance whether you are using a file or memory based database. For example, when ordering result sets from a file-based database select this routine is used to generate the code:

  static void pushOntoSorter(
    Parse *pParse,         /* Parser context */
    ExprList *pOrderBy,    /* The ORDER BY clause */
    Select *pSelect        /* The whole SELECT statement */
  ){
    Vdbe *v = pParse->pVdbe;
    sqlite3ExprCodeExprList(pParse, pOrderBy);
    sqlite3VdbeAddOp(v, OP_Sequence, pOrderBy->iECursor, 0);
    sqlite3VdbeAddOp(v, OP_Pull, pOrderBy->nExpr + 1, 0);
    sqlite3VdbeAddOp(v, OP_MakeRecord, pOrderBy->nExpr + 2, 0);
    sqlite3VdbeAddOp(v, OP_IdxInsert, pOrderBy->iECursor, 0);

For those of us who have very complicated nested sub-selects of file-based tables in many queries or even ORDER BYs on huge result sets, speeding up the memory page performance should be a performance win for SQLite in general.


2006-May-06 17:37:32 by anonymous:
The following test demonstrates that this memory page issue can greatly effect the performance of queries against file-based tables if temp_store is set to MEMORY.

"big" is a file-based table in foo.db with 10 million rows. It was created with "create table big(x,y)".

  # unmodified stock SQLite built from May 5 2006 CVS (after check-in [3178])
  # compiled with default settings for SQLITE_DEFAULT_PAGE_SIZE and N_PG_HASH
  $ time ./may5-sqlite/sqlite3 foo.db "PRAGMA temp_store = MEMORY; select x, y from big order by y, x" >/dev/null
  real    13m23.828s
  user    13m18.452s
  sys     0m0.811s

  # SQLite built from May 5 2006 CVS, but compiled with proposed change of
  # SQLITE_DEFAULT_PAGE_SIZE set to 4096, and N_PG_HASH set to 16384
  $ time ./may5-sqlite-hash-opt/sqlite3 foo.db "PRAGMA temp_store = MEMORY; select x, y from big order by y, x" >/dev/null
  real    6m16.031s
  user    6m13.108s
  sys     0m0.811s

This is not even what I would consider to be a big table.

I should mention that compiling with SQLITE_DEFAULT_PAGE_SIZE = 1024, and N_PG_HASH = 32768 resulted in same timing as the may5-sqlite-hash-opt test run above. A pretty good return for an extra 126K.


2006-May-08 04:07:25 by anonymous:
You now get 20,725 inserts/second as of the latest check-in [3180] for 4 million inserts into a :memory: database in a single transaction (using the default SQLITE_DEFAULT_PAGE_SIZE of 1K). This is nearly twice as fast as SQLite prior to the check-in [3180] (10,782 inserts/second). However, it is 4% slower than the best timing prior to [3180] when compiled with N_PG_HASH=32768 and SQLITE_DEFAULT_PAGE_SIZE=1024 which got 21,622 inserts/second (see table above). Increasing the size of SQLITE_DEFAULT_PAGE_SIZE with the latest CVS either has no effect or makes the memory insert benchmark timings slightly worse.
 
1791 active 2006 May anonymous Unknown Pending     1 1 Native threads support for BeOS edit
BeOS ports lacks native thread support. BeOS has very powerful but lightweight threading system, being throughout multithreaded, but it differs from posix-thread ideology, thus our pthreads implementation atm looks more like flacky workaround. Ideally will be to have separate implementation for thread-support, like for Win16/32 versions.

At the moment this problem caused bustage of BeOS Mozilla port, https://bugzilla.mozilla.org/show_bug.cgi?id=330340 nearest workaround might be pthreads usage, inspite its flackyness, but it also causes mess for Mozilla build/configure system, because for other parts in Mozilla we use nspr-threads, which, for BeOS, use native version

2006-Oct-27 05:48:51 by anonymous:
BeOS locking extensions (using native bthreads) have been written and are included in the SQLite3 built into Mozilla Firefox. Is there some process wherein these changes might be incorporated into the SQLite tree?


2006-Oct-27 12:48:11 by anonymous:
Follow the example of OS/2 and propose a patch against the latest SQLite CVS that has proper #ifdef's around BeOS code so it won't break other platforms. Since you're probably the only one interested in this patch, you'll have to do the diffing/merging/testing work yourself.


2006-Nov-07 03:55:36 by anonymous:
Thanks for the advice. We've completed updates to code so it works with the sqlite 3.3.8 patches proposed for Firefox. Current implementation has a parallel os-specific file (os_beos.c). However, with the latest round of locking enhancements to os_unix.c, we're now wondering if it makes more sense to simply enhance this file to support BeOS locking. (yes, we. surprisingly, there is more than one BeOS user left on the planet.) :)
 
1797 active 2006 May anonymous TclLib Pending   drh 1 1 COPY command doesn't work in tclsqlite 3.3.5 edit
The COPY command doesn't seem to work in the tcl sqlite lib. This same script and datafile works in version 3.2.7.

load ./lib/libtclsqlite[info sharedlibextension] sqlite MEMORY_DB :memory: MEMORY_DB onecolumn "PRAGMA empty_result_callbacks=1" puts [MEMORY_DB version] MEMORY_DB eval "create table xyz (col1,col2)" MEMORY_DB copy ignore win_pol /home/centadm/win_pol4.csv \t MEMORY_DB eval "select * from xyz" sqlite_array { puts "Here in the callback" foreach sqlite_value $sqlite_array(*) { puts "$sqlite_value $sqlite_array($sqlite_value)" } }

The data file win_pol4.csv consists of two columns, tab seperated. DATA1 DATA2

And the output: -bash-3.00$ tclsh test_sqlite.tcl

3.3.5

    while executing
"MEMORY_DB copy ignore win_pol /home/centadm/win_pol4.csv \t"
    (file "test_sqlite.tcl" line 5)
-bash-3.00$ pwd
/home/centadm
-bash-3.00$ ls -l /home/centadm/win_pol4.csv
-rw-r--r--  1 centadm centadm 12 May  5 14:21 /home/centadm/win_pol4.csv
-bash-3.00$ more /home/centadm/win_pol4.csv
DATA1   DATA2

A TCL Error is returned from the copy command, no message tho. I have used catch to capture the command and verified that there is no data going into the table.

Also, PRAGMA empty_result_callbacks=1 still doesn't seem to work in the tcllib. If you catch the COPY command above, you still never see the "Here in the callback" message.

2006-May-05 17:57:42 by anonymous:
Clarification:

The line MEMORY_DB copy ignore win_pol /home/centadm/win_pol4.csv \t

should read

MEMORY_DB copy ignore xyz /home/centadm/win_pol4.csv \t

However the result is the same:

-bash-3.00$ tclsh test_sqlite.tcl 3.3.5

    while executing
"MEMORY_DB copy ignore xyz /home/centadm/win_pol4.csv \t"
    (file "test_sqlite.tcl" line 7)
-bash-3.00$


2006-May-05 19:46:56 by anonymous:
I have narrowed it down to the code here in tclsqlite.c:

    zSql = sqlite3_mprintf("SELECT * FROM '%q'", zTable);
    if( zSql==0 ){
      Tcl_AppendResult(interp, "Error: no such table: ", zTable, 0);
      return TCL_ERROR;
    }
    nByte = strlen(zSql);
    rc = sqlite3_prepare(pDb->db, zSql, 0, &pStmt, 0);
    sqlite3_free(zSql);
    if( rc ){
      Tcl_AppendResult(interp, "Error: ", sqlite3_errmsg(pDb->db), 0);
      nCol = 0;
    }else{
      nCol = sqlite3_column_count(pStmt); <--- RETURNING 0 FOR COLUMN COUNT, HAVE VERIFIED TABLE HAS TWO COLUMNS
    }
    sqlite3_finalize(pStmt);
    if( nCol==0 ) {
      return TCL_ERROR;       <--- NO ERROR MESSAGE RETURNED
    }


2006-May-16 17:51:28 by anonymous:
I found the problem. The first sqlite3_prepare under DB_COPY should have -1 as it's third argument. When this was change from a 0 to -1 the copy command works in tclsqlite.

rc = sqlite3_prepare(pDb->db, zSql,0, &pStmt, 0);

should be

rc = sqlite3_prepare(pDb->db, zSql,-1, &pStmt, 0);


2006-May-16 18:01:11 by anonymous:
There is also another reference (the insert statement) to the prepare statement under DB_COPY that needs to change it's third argument from 0 to -1.


2006-Sep-27 16:24:53 by anonymous:
The same problem is present with version 3.3.7 over here. However, the indicated patch seem to work.
 
1799 active 2006 May anonymous Pager Pending     2 3 temp_store=MEMORY slower than FILE for large intermediate result sets edit
(This ticket was split off from #1790 because that ticket was becoming too broad.)

When temp_store=MEMORY it can negatively effect the performance of queries with large intermediate result sets generated from SELECTs of either file-based tables or memory-based tables. This is true when sufficient RAM is available to the SQLite process to completely hold the intermediate results in memory without swapping to disk.

In the example below, "big" is a file-based table in foo.db with 10 million rows. It was created with "create table big(x,y)".

  # unmodified stock SQLite built from May 5 2006 CVS (after check-in [3178])
  # compiled with default settings for SQLITE_DEFAULT_PAGE_SIZE and N_PG_HASH
  $ time ./may5-sqlite/sqlite3 foo.db "PRAGMA temp_store = MEMORY; select x, y from big order by y, x" >/dev/null
  real    13m23.828s
  user    13m18.452s
  sys     0m0.811s

  # SQLite built from May 5 2006 CVS, but compiled with proposed change of
  # SQLITE_DEFAULT_PAGE_SIZE set to 4096, and N_PG_HASH set to 16384
  $ time ./may5-sqlite-hash-opt/sqlite3 foo.db "PRAGMA temp_store = MEMORY; select x, y from big order by y, x" >/dev/null
  real    6m16.031s
  user    6m13.108s
  sys     0m0.811s

Compiling with SQLITE_DEFAULT_PAGE_SIZE = 1024, and N_PG_HASH = 32768 resulted in the same timing as the may5-sqlite-hash-opt test run above.

If temp_store=FILE (with default SQLite values for SQLITE_DEFAULT_PAGE_SIZE and N_PG_HASH), the timings are comparable to temp_store=MEMORY with SQLITE_DEFAULT_PAGE_SIZE=4096, and N_PG_HASH=16384.

Large intermediate results sets can cause SQLite to spend more than half of its CPU time in the function pager_lookup(). By increasing the value of N_PG_HASH and SQLITE_DEFAULT_PAGE_SIZE, the time spent in pager_lookup can be reduced to near zero, thus doubling performance in such cases.

    %   cumulative   self               self     total
   time   seconds   seconds     calls  ms/call  ms/call  name
   51.97    118.31   118.31 119658386     0.00     0.00  pager_lookup
    4.36    128.25     9.94   4000009     0.00     0.06  sqlite3VdbeExec
    3.06    135.21     6.96 315629923     0.00     0.00  parseCellPtr
    3.05    142.16     6.95 171797186     0.00     0.00  sqlite3VdbeRecordCompare
    2.67    148.22     6.07  12000005     0.00     0.01  sqlite3BtreeMoveto
    2.14    153.10     4.88 343594380     0.00     0.00  sqlite3VdbeSerialGet
    1.68    156.93     3.83 171797188     0.00     0.00  sqlite3MemCompare
    1.63    160.65     3.72  77995781     0.00     0.00  sqlite3pager_get
    1.60    164.29     3.65 169734946     0.00     0.00  sqlite3pager_unref
    1.58    167.88     3.59 654100795     0.00     0.00  get2byte
2006-May-07 18:37:50 by anonymous:
Timings on same Windows machine with check-in [3180] applied:

  # FILE
  $ time ./may7-sqlite/sqlite3 foo.db "PRAGMA temp_store = FILE; select x, y from big order by y, x" >/dev/null
  real    5m7.157s
  user    4m19.905s
  sys     0m20.827s

  # MEMORY
  $ time ./may7-sqlite/sqlite3 foo.db "PRAGMA temp_store = MEMORY; select x, y from big order by y, x" >/dev/null
  real    5m12.328s
  user    5m9.781s
  sys     0m0.984s

Much better.

temp_store=MEMORY is now competitive with FILE, although temp_store=FILE (when the OS is able to cache the file entirely in memory) is marginally faster.

I still think the MEMORY time can be reduced further by another 20 seconds judging by the sys time of 20.827s in the FILE test. The MEMORY subsystem of SQLite ought to have an advantage over the FILE subsystem because it does not incur any system call overhead. I'll see if a profile turns up anything obvious.

 
1804 active 2006 May anonymous Unknown Pending     4 3 Inconsistent value type returned by SUM when using a GROUP BY edit
Using a schema with test table:

  CREATE TABLE Tbl1 (Key1 INTEGER, Num1 REAL)

And test data:

  INSERT INTO Tbl1 (Key1,Num1) VALUES (1,5.0)

The query:

  SELECT SUM(Tbl1.Num1) AS Num1Sum FROM Tbl1

Returns a column with the value type correctly reported as FLOAT (2).

However, the query:

  SELECT Tbl1.Key1, SUM(Tbl1.Num1) AS Num1Sum FROM Tbl1 GROUP BY Tbl1.Key1

Returns two columns with value types INT (1) and INT (1).

The SUM function is returning a different value type for these two queries when both should return FLOAT (2).

This problem does not occur when any SUMmed value is not a whole number in which case, both queries return a value type of FLOAT for the SUM column.

I have applied the patch from Check In 3169 (relating to #1726 and #1755) to select.c but this does not resolve the problem.

2006-May-10 09:34:11 by anonymous:
I should have added that this problem was seen in a Windows CE build running on Pocket PC 2003 and built using eMbedded Visual C++ 4.0.


2006-May-10 11:24:47 by anonymous:
I can confirm that exactly the same behaviour is exhibited when built under Windows XP (32-bit).


2006-May-10 12:40:21 by drh:
The answer you are getting back is exactly correct. Why do you care what its datatype is? If you don't like the datatype, cast it.


2006-May-10 13:42:14 by anonymous:
For maximum compatibility with other SQL databases, both

  SELECT SUM(field2) FROM table

and

  SELECT field1, SUM(field2) FROM table GROUP BY field1

should return the same data type for the SUM column. All other databases I have worked with do this. I understand that SQLite uses manifest typing but believe that it should be consistent.

The problem I have is that in my query function (which takes an SQL string and returns a page of results as a 2D array of objects), I don't know whether to use sqlite3_column_double or sqlite3_column_int because I don't know what the calling function requires this column to be returned as. I am currently using the sqlite3_column_decltype call to switch which sqlite3_column_* function I use (and falling back on sqlite3_column_type when the declared type is not known e.g. for aggregate functions like SUM). If the return type of SUM is unpredictable, my calling functions can't assume returned values will be of the same type as the field that is being SUMmed (as is the case with other SQL databases).

If you don't consider this to be a problem with SQLite, then I think my only option will be for calling functions to pass in an array of return types so that I always return objects of the correct type.

 
1809 active 2006 May anonymous CodeGen Pending     1 3 Huge slowdown/increased memory use when using GROUP BY on big dataset edit
This seemingly nonsensical query is a greatly reduced test case taken from several queries I use with SQLite 3.2.1. The real example joins various huge tables and much more complicated views. I'd like to upgrade beyond SQLite 3.2.1, but this is a showstopper.

It takes 13 seconds to run on SQLite 3.2.1 and uses just 1.2M of memory. With 3.3.5+ from CVS it takes 185 seconds and uses 230M of memory.

  PRAGMA temp_store=MEMORY;

  CREATE TABLE n1(a integer primary key);
  INSERT INTO "n1" VALUES(1);
  INSERT INTO "n1" VALUES(2);
  INSERT INTO "n1" VALUES(3);
  INSERT INTO "n1" VALUES(4);
  INSERT INTO "n1" VALUES(5);
  INSERT INTO "n1" VALUES(6);
  INSERT INTO "n1" VALUES(7);
  INSERT INTO "n1" VALUES(8);
  INSERT INTO "n1" VALUES(9);
  INSERT INTO "n1" VALUES(10);
  INSERT INTO "n1" VALUES(11);
  INSERT INTO "n1" VALUES(12);
  INSERT INTO "n1" VALUES(13);
  INSERT INTO "n1" VALUES(14);
  INSERT INTO "n1" VALUES(15);
  CREATE VIEW vu as select v3.a a, v5.a-v2.a*v7.a b
                      from n1 v1,n1 v2,n1 v3,n1 v4,n1 v5,n1 v6,n1 v7;

  select a a, sum(b) T from vu where a=7 group by a;

It seems that SQLite 3.2.1 had a much more efficient GROUP BY algorithm that discarded unnecessary data as the view was traversed.

2006-May-13 03:01:28 by anonymous:
Seeing as this ticket concerns the GROUP BY statement it would make more sense to have an example like this:

  select a a, sum(b) T from vu where a<4 group by a;

But both queries exhibit the same slowdown and memory increase, in any event.


2006-May-13 15:09:39 by anonymous:
This GROUP BY slowdown/memory increase is not specific to VIEWs. I repeated the test against a comparably sized table with the same results. You'll see this effect for any SELECT operating on a large number of rows using GROUP BY.


2006-May-13 16:44:04 by anonymous:
The slowdown first appears in SQLite 3.2.6 in check-in [2662] .


2006-May-24 13:19:29 by anonymous:
Here's an example to show an actual real-life use of GROUP BY in SQLite <= 3.2.5... Imagine performing mathematical operations on every combination of rows in several large tables for statistical analysis. The GROUP BY algorithm change in 3.2.6 now makes using GROUP BY on huge cross joins not usable for this purpose because it creates an intermediate result set of the product of all cross joins - several times larger than the size of the (already huge) database itself. Indexing is not useful in this case because there is nothing to index by design. All table rows must be traversed.

Older versions of SQLite performed this operation extremely efficiently because grouping took place in the main traversal loop. I would think that the old algorithm could be used, but instead of keeping the intermediate results in memory, an index and a table in temp store could be used.

 
1815 active 2006 May anonymous Parser Pending     3 3 Support of W3C-DTF(ISO8601 subset) is incomplete edit
"Z" of a time zone is ignored.

Reference: http://www.w3.org/TR/NOTE-datetime

  CREATE table test(dt);
  INSERT INTO "test" VALUES('2006-05-20T01:10:20+00:00');
  INSERT INTO "test" VALUES('2006-05-20T01:10:20Z');
  INSERT INTO "test" VALUES('2006-05-20T10:10:20+09:00');
  SELECT datetime(dt) from test;
  2006-05-20 01:10:20

  2006-05-20 01:10:20
 
1816 active 2006 May anonymous VDBE Pending     1 2 Database corruption with pragma auto_vacuum edit
We had a database created with PRAGMA auto_vacuum=1, that started returning the following message on a DELETE statement.

  SQL error: database disk image is malformed

Running the VACUUM command and running the same DELETE statement succeeds.

  Running PRAGMA integrity_check on the database (before the VACUUM command is issued) results in the following output:
sqlite> PRAGMA integrity_check;
*** in database main ***
Page 3393 is never used
Page 3398 is never used
Page 3400 is never used
Page 3401 is never used
Page 3402 is never used
Page 3405 is never used
Page 3406 is never used
sqlite> VACUUM;
sqlite> PRAGMA integrity_check;
ok

We tried as a temporary workaround, running PRAGMA integrity_check and, based on the result, deciding whether or not to run VACUUM, but this can consume too much time.

If needed, I can send a small database that exhibits this problem.

2006-May-22 21:45:47 by drh:
The database is probably not helpful. What I need to know is:

  • What sequence of SQL statements do you issue to cause this to occur?
  • What operating system you are using.
  • Is the application multi-threaded?
  • Is the problem reproducible?
  • Are you using a precompiled binary or did you compile it yourself?
  • Does the problem go away if you turn off autovacuum?


2006-May-22 22:11:09 by anonymous:
  • What sequence of SQL statements do you issue to cause this to occur? It is unknown exactly what all of the the statements are leading up to the corruption. I can send the possible statements via private e-mail.

  • What operating system you are using. Windows XP Professional w/ Service Pack 2.

  • Is the application multi-threaded? Yes.

  • Is the problem reproducible? The corruption happens on occasion -- so far it is not known to be easily reproducable in a finite number of steps.

  • Are you using a precompiled binary or did you compile it yourself? Self-compiled library. When we use the database in our application, it is contained in abstracted classes with concurrency control.

  • Does the problem go away if you turn off autovacuum? We have not seen database corruption if auto_vacuum is off when the database is initially created. Is it possible to turn off auto vacuum after the database tables have been created (no when using pragma auto_vacuum, according to the docs)?


2006-May-22 22:28:46 by anonymous:
Rather than relying on trial and error to reproduce the bug, one technique the bug reporter might try to reproduce the problem is to take a snapshot of the database when it is in a known good state and save it somewhere and then have every process that comes into contact with the database file log every SQLite command (and pragma) complete with millisecond-resolution timestamp and process/thread ID as follows:

  SELECT * FROM WHATEVER;         -- 2006-05-23 14:44:45.237 PID 345 Thread 0
  insert into blah values(3,4,5); -- 2006-05-23 14:50:15.345 PID 345 Thread 0
  update foo set v=5 where y>4;   -- 2006-05-23 15:05:12.930 PID 239 Thread 0

Should the problem happen again, each command could easily be replayed in an appropriate thread in the same order from the last known "good" state, greatly increasing the chances of repeating the bug. If repeating these commands does not lead to database corruption, it is fairly likely that the bug is in your multithreaded code, and not in SQLite.

Perhaps SQLite already has such a command tracing facility already. I don't know.


2006-May-22 22:42:04 by anonymous:

  sqlite3_trace();

It passes all the caller-generated SQL statements to a callback (although it doesn't fill in bindings).

It also outputs a lot of "internal" SQL statements (VACUUM, for example, is a collection of operations on a temp table), but you should be able to recognize that stuff as something your app would never generate.

 
1822 active 2006 May anonymous   Duplicate     3 3 Table Alias together with Subquery seems not to work proper edit
SELECT * FROM auth AS a LEFT JOIN (SELECT tm.team FROM teammbs AS tm) AS tr ON a.ateam=tr.team;

Error message: No such colum tr.team

But if I run the sub-query itself, it works fine. Of course, this example can be expressed different, so no subquery required. But the complete expression looks like this:

SELECT a.auth, a.avalue FROM auth a LEFT JOIN (SELECT tm.member, tm.team FROM teammbs tm, team t WHERE tm.team=t.teamid AND (t._state<64 or (t._state>120 AND t._state<192)) AND (tm._state<64 or (tm._state>120 AND tm._state<192))) AS tr ON a.ateam=tr.team WHERE (a._state<64 or (a._state>120 AND a._state<192)) AND (a.auser='test' OR tr.member='test') ORDER BY a.auth;

It works fine with MySQL 5, and brings the same error on SQLite 3: No such column tr.team.

Any idea?

 
1850 active 2006 Jun anonymous Unknown Pending     2 1 NUMERIC data type ERROE when read on uClinux edit
I have update some data of a tables's NUMERIC TYPE column on Windows or Linux,but when I use "select *......." to read on uClinux 2.4.24,I get the wrong value,example:the date I've written is 12.5,but readback is 2.3534826093695e -18.5(use the sqlite3_column_text API). I tried to get the value used "sqlite3_column_double" API,but the result is also wrong;

But when I update some data with this column on uClinux,I can read the data right!

2006-Jun-16 01:53:24 by drh:
What CPU is this happening on?

SQLite assumes that floating point values are stored as IEEE 64-bit floats in the same byte order as a 64-bit integer. If your chip does not match this expectation, then floating point won't work.

 
1851 active 2006 Jun anonymous Unknown Pending     2 1 USE "ORDER BY" error on uClinux edit
when I use "ORDER BY" function in a "select" on uClinux 2.4.24, I get a error:"SQL error or missing database" but the same program run on windows or Linux OK.
2006-Jun-16 11:20:15 by drh:
This is certainly a strange error. Combined with #1850, it suggests a problem with your build, not a problem in SQLite.

I have no ability to use or run uCLinux. So if the error cannot be reproduced on a desktop system, there is not much I can do to address the problem. I am afraid you are on your own on this one.


2006-Jun-19 03:12:31 by anonymous:
I'm just wondering that SQLite 3.2.8 runs on this uClinux system OK but SQLite 3.3.5 is error.
 
1856 active 2006 Jun anonymous   Pending     2 3 SQLITE_OMIT_UTF16 breaks 'make test' edit
When compiling sqlite 3.3.6 with -DSQLITE_OMIT_UTF16 and you say 'make test' it fails:

  make test
./libtool --mode=link gcc -g -O2 -DOS_UNIX=1 -DHAVE_USLEEP=1 -DHAVE_FDATASYNC=1 -I. -I./src -DSQLITE_DEBUG=2 -DSQLITE_MEMDEBUG=2 -DSQLITE_OMIT_UTF16  -I/usr/include -DTHREADSAFE=1 -DSQLITE_THREAD_OVERRIDE_LOCK=-1 -DSQLITE_OMIT_CURSOR -DTCLSH=1 -DSQLITE_TEST=1 -DSQLITE_CRASH_TEST=1 \
        -DTEMP_STORE=1 -o testfixture ./src/btree.c ./src/date.c ./src/func.c ./src/os.c ./src/os_unix.c ./src/os_win.c ./src/os_os2.c ./src/pager.c ./src/pragma.c ./src/printf.c ./src/test1.c ./src/test2.c ./src/test3.c ./src/test4.c ./src/test5.c ./src/test6.c ./src/test7.c ./src/test_async.c ./src/test_md5.c ./src/test_server.c ./src/utf.c ./src/util.c ./src/vdbe.c ./src/where.c ./src/tclsqlite.c \
        libsqlite3.la -L/usr/lib -ltcl8.4 -ldl  -lpthread -lieee -lm
gcc -g -O2 -DOS_UNIX=1 -DHAVE_USLEEP=1 -DHAVE_FDATASYNC=1 -I. -I./src -DSQLITE_DEBUG=2 -DSQLITE_MEMDEBUG=2 -DSQLITE_OMIT_UTF16 -I/usr/include -DTHREADSAFE=1 -DSQLITE_THREAD_OVERRIDE_LOCK=-1 -DSQLITE_OMIT_CURSOR -DTCLSH=1 -DSQLITE_TEST=1 -DSQLITE_CRASH_TEST=1 -DTEMP_STORE=1 -o .libs/testfixture ./src/btree.c ./src/date.c ./src/func.c ./src/os.c ./src/os_unix.c ./src/os_win.c ./src/os_os2.c ./src/pager.c ./src/pragma.c ./src/printf.c ./src/test1.c ./src/test2.c ./src/test3.c ./src/test4.c ./src/test5.c ./src/test6.c ./src/test7.c ./src/test_async.c ./src/test_md5.c ./src/test_server.c ./src/utf.c ./src/util.c ./src/vdbe.c ./src/where.c ./src/tclsqlite.c  ./.libs/libsqlite3.so -L/usr/lib -ltcl8.4 -ldl -lpthread -lieee -lm -Wl,--rpath -Wl,/home/cla/proj/caissadb/sqlite/sqlite/lib
./src/test1.c: In function 'Sqlitetest1_Init':
./src/test1.c:3742: error: 'unaligned_string_counter' undeclared (first use in this function)
./src/test1.c:3742: error: (Each undeclared identifier is reported only once
./src/test1.c:3742: error: for each function it appears in.)
make: *** [testfixture] Error 1

Maybe there is a '#ifndef SQLITE_OMIT_UTF16' / '#endif' needed around

  Tcl_LinkVar(interp, "unaligned_string_counter",
      (char*)&unaligned_string_counter, TCL_LINK_INT);

in Line 3742 in file src/test1.c?

Regards.

 
1861 active 2006 Jun anonymous Pager Pending     1 1 Problem in using Triggers and multithreading edit
I am using SQLite3 database with triggers . This database is used by my processing engine which is having 10 threads accessing the same database. Trigger is used to updata and insert records in a table and that very table is also updated by threads.

Processing engine crashes whenever a trigger updates or inserts a record in the table.

Can you tell me how to configure my existing engine to avoid crashing? Is it safe to use trigger?

 
1862 active 2006 Jun anonymous TclLib Pending   tclguy 1 1 SQLite cannot load/import data from file edit
I found the problem when I tried to load a data file into a table. To reproduce the problem, I got a mini testcase.

DATA FILE - test.dat


    1       0       0
    2       90000   0
    3       366000  0
---------------------------

Log from SQLite:


    khronos-yajun>sqlite3 test
    SQLite version 3.3.6
    Enter ".help" for instructions
    sqlite> create table test (id INT, x1 INT, x2 INT);
    sqlite> .import test.dat test
    test.dat line 1: expected 3 columns of data but found 1
    sqlite> .exit


The problem also exists when I use tcl wrapper (sql copy abort test test.dat).

I looked into the code in src/tclsqlite.c,

In Lines

   1045     nByte = strlen(zSql);
   1046     rc = sqlite3_prepare(pDb->db, zSql, 0, &pStmt, 0);
   1047     sqlite3_free(zSql);

Is the third argument of sqlite3_prepare supposed to be the length of zSql, hence nByte?

Also in lines

   1070     zSql[j++] = ')';
   1071     zSql[j] = 0;
   1072     rc = sqlite3_prepare(pDb->db, zSql, 0, &pStmt, 0);
   1073     free(zSql);

If I change these two places to reflect the length of zSql, I seem to succeed.

Yajun

2006-Sep-27 16:25:47 by anonymous:
This is a duplicate of #1797
 
1867 active 2006 Jun anonymous BTree Pending     1 3 Access Violation after set a new page_size edit
An access violation occured on W2K when I try to create a new table in the empty database. There was a following sequence of SQL commands

select count(*)==2 as cnt from sqlite_master where type='table' and tbl_name in ('tbl1', 'tbl2');

so if cnt is equal 0 then I execute command pragma page_size=4096; and then create a new table.

I gess that some of internal structures by this time have been initialized and so when I try to create new table the page_size is lower then needed.

we overwrite memory in the function zeroPage in instruction: memset(&data[hdr], 0, pBt->usableSize - hdr); Size of structure data less then pBt->usableSize

Below result after memset

0:000> dt MemPage 004c3cf0
+0x000 isInit : 0 ''
+0x001 idxShift : 0 ''
+0x002 nOverflow : 0 ''
+0x003 intKey : 0x1 ''
+0x004 leaf : 0x1 ''
+0x005 zeroData : 0 ''
+0x006 leafData : 0x1 ''
+0x007 hasData : 0 ''
+0x008 hdrOffset : 0 ''
+0x009 childPtrSize : 0 ''
+0x00a maxLocal : 0
+0x00c minLocal : 0
+0x00e cellOffset : 0
+0x010 idxParent : 0
+0x012 nFree : 0xf94
+0x014 nCell : 0
+0x018 aOvfl : [5] _OvflCell
+0x040 pBt : (null)
+0x044 aData : (null)
+0x048 pgno : 0
+0x04c pParent : (null)

0012ea50 10006861 004c3cf0 0000000d 00000064 dblited!decodeFlags+0x80 [D:\sqllite\sqlite-3.3.6\btree.c @ 1349]
0012ea70 10006710 004c3cf0 0000000d 004c3cf0 dblited!zeroPage+0xd0 [D:\sqllite\sqlite-3.3.6\btree.c @ 1466]
0012ea8c 10006215 002fd390 002fd390 00000000 dblited!newDatabase+0xf9 [D:\sqllite\sqlite-3.3.6\btree.c @ 2061]
0012eaa0 10052ba0 002f7c30 00000001 0012f0e4 dblited!sqlite3BtreeBeginTrans+0xd6 [D:\sqllite\sqlite-3.3.6\btree.c @ 2141]
0012f0a4 10057cf5 004c3d80 0012f13c 0012f478 dblited!sqlite3VdbeExec+0x2c6d [D:\sqllite\sqlite-3.3.6\vdbe.c @ 2386]
0012f0e4 00412801 004c3d80 0012f1d4 0012f478 dblited!sqlite3_step+0x1db [D:\sqllite\sqlite-3.3.6\vdbeapi.c @ 223]
 
1872 active 2006 Jun anonymous   Pending     4 3 sqlite3_open doesn't support RFC1738 format for filename edit
sqlite3_open only supports UTF-8 encoding as a format for its filename argument (http://www.sqlite.org/capi3ref.html#sqlite3_open).

If your application receives a RFC1738 encoded URL for filename, that has to be UTF-8-encoded for use in SQLite. It would be nice if that could be instead passed directly to sqlite3_open.

Is RFC1738 URL decoding support planned for SQLite?

(RFC1738 link: http://www.cse.ohio-state.edu/cgi-bin/rfc/rfc1738.html)

 
1878 active 2006 Jun anonymous CodeGen Pending     2 3 No index used when specifying alias name in ORDER BY clause edit
Using an alias name in the ORDER BY clause prevents indices from being used in the query for sorting purposes:

For this schema:

  CREATE TABLE t1 (c1, c2);
  CREATE TABLE t2 (c3, c4);
  CREATE INDEX t1_idx ON t1(c2);

the following select query:

  EXPLAIN QUERY PLAN
  SELECT t1.c2 AS col2, t2.c4 AS col4
    FROM t1 LEFT JOIN t2 ON t1.c1=t2.c3 ORDER BY t1.c2;

will indeed use index t1_idx:

  sqlite> EXPLAIN QUERY PLAN
          SELECT t1.c2 AS col2, t2.c4 AS col4
            FROM t1 LEFT JOIN t2 ON t1.c1=t2.c3
           ORDER BY t1.c2;
  0|0|TABLE t1 WITH INDEX t1_idx
  1|1|TABLE t2

However, when using the alias name col2 in the ORDER BY clause, the index won't be used:

  sqlite> EXPLAIN QUERY PLAN
          SELECT t1.c2 AS col2, t2.c4 AS col4
            FROM t1 LEFT JOIN t2 ON t1.c1=t2.c3
           ORDER BY col2;
  0|0|TABLE t1
  1|1|TABLE t2

IMHO, the same index should be used in both queries?

2006-Jun-30 13:54:10 by anonymous:
Not sure whether it's a different issue, but when using a second column in the ORDER BY clause, also no index will be used:

  sqlite> EXPLAIN QUERY PLAN
          SELECT t1.c2 AS col2, t2.c4 AS col4
            FROM t1 LEFT JOIN t2 ON t1.c1=t2.c3 ORDER BY t1.c2, t2.c4;
  0|0|TABLE t1
  1|1|TABLE t2

Personally, I'd expect sqlite to use the t1_idx index as well to fulfill the primary ordering?


2006-Jun-30 16:04:31 by anonymous:
As a workaround try "ORDER BY 1"


2006-Jul-03 08:41:01 by anonymous:
Sorry, I'm not sure how "ORDER BY 1" would be a workaround, when I really need the results to be sorted by table column data...

(I don't want to start a discussion in the bug tracker, so you're welcome to take any suggestions/answers to the sqlite-user mailing list, which I also monitor.)

2006-Jul-03 15:51:11 by anonymous:
I'm not the poster of previous comment, but ORDER BY (n) order by result column index. In your case, using ORDER BY 1, it will be ordered by the first column.


2006-Jul-04 07:33:30 by anonymous:
Thanks for the clarification. This would be a workaround for the first problem mentioned, but when sorting by two columns, still no index will be used, even if using ORDER BY 1,2


2006-Jul-04 21:34:24 by anonymous:
SQLite really needs a way to explicitly state which index(es) to use. Perhaps something similar to Oracle's comment hints.
 
1882 active 2006 Jul anonymous   Pending     1 1 Wrong algorithm of SQLITE_VERSION_NUMBER calculation edit
The sqlite3.h comment describing how numeric version number is calculated is as follows:

"The SQLITE_VERSION_NUMBER is an integer with the value (X*100000 + Y*1000 + Z). For example, for version "3.1.1beta", SQLITE_VERSION_NUMBER is set to 3001001."

But the value of SQLITE_VERSION_NUMBER is greater than the equation above suggests. The value X*100000 should be changed to X*1000000 (one milion).

 
1884 active 2006 Jul anonymous   Pending     3 2 pragma table_info caches results from previous query edit
this problem is observed with pysqlite's latest windows build 2.3.2 and others. it does not occur on unix-based builds, which is why I suspect the issue is in sqlite, since pysqlite's code is platform-neutral.

if you get a result from a "pragma table_info()" call, and do not consume all the results, then a subsequent call to the same statement does not return up-to-date results, i.e. if the table had been dropped in between. it behaves as though the results of "pragma table_info" are globally cached somewhere, ignoring the fact that is was executed again.

this test program illustrates the problem:

	from pysqlite2 import dbapi2 as sqlite

	connection = sqlite.connect(':memory:')

	# check for a nonexistent table
	c = connection.execute("pragma table_info(users)")
	row = c.fetchone()
	assert row is None   # its good.

	# now create the table
	connection.execute("""
	create table users (
		foo VARCHAR(10),
		name VARCHAR(40)
	)
	""")

	# do the table_info pragma.  returns two rows
	c = connection.execute("pragma table_info(users)")

	# get the first row
	row = c.fetchone()
	print row

	# but then dont get the second, close out the cursor instead.

	#row2 = c.fetchone()  # uncomment to fully consume both rows, then it works

	c.close()
	c = None

	# rollback too.
	connection.rollback()

	# now drop the table
	connection.execute("DROP TABLE users")

	print "dropped"

	# now it should be gone, right? well it is, but the pragma
	# call starts off with the former result set
	c = connection.execute("pragma table_info(users)")
	row = c.fetchone()
	print row
	assert row is None  # fails.
 
1885 active 2006 Jul anonymous Shell Pending     2 3 sqlite3 .mode insert and .dump do not list column names for selects edit
In sqlite3 .mode insert does not list column names for selects - it should. This makes dumping selected columns from tables when intending to add or delete columns problematic. .dump doesn't list column names either, IMHO it should. Consider

sqlite> .mode tabs
sqlite> select * from users;
ed 2006-07-05 52
sqlite> .mode insert
sqlite> select abs_tgt from users;
INSERT INTO table VALUES(52);
sqlite>

Obviously the workaround is to hand edit the output SQL

2006-Jul-11 10:20:08 by anonymous:
I've just noticed it doesn't include the table name in the INSERT statements either.
 
1893 active 2006 Jul anonymous   Pending     3 3 sqlite doesn't use indexes containing primary key in prim. key selects edit
I have table:

    CREATE TABLE IF NOT EXISTS 'customers' (
       'rowid'      INTEGER       PRIMARY KEY AUTOINCREMENT NOT NULL,
       'fname'      CHAR(40)      NOT NULL,
       'sname'      CHAR(40)      NOT NULL,
       'birthno'    CHAR(11)      NULL)

And index:

    CREATE UNIQUE INDEX IF NOT EXISTS 'idx_customers_sname'
       ON 'customers' (
       'sname'  ASC,
       'fname'  ASC,
       'rowid'  ASC
       );

Command

   SELECT * FROM customers ORDER BY sname ASC, fname ASC, rowid ASC;

doesn't use created index.

Command

   SELECT * FROM customers ORDER BY sname ASC, fname ASC;

uses index idx_customers_sname.

I think this is a bug, but maybe (i don't know), it is by desing. If I don't specify rowid in ORDER BY, is the resultset ordered by rowid anyway?

2006-Jul-24 16:02:29 by anonymous:
In SQL single quotes are used around string literals, and double quotes are used around identifiers where required to enclose keywords and/or embedded spaces. In your case no quotes are required at all because your table and column identifiers are continuos (i.e. do not contain embedded spaces) non-keyword names. If you are going to include unnecessary quotes then you should at least use the correct ones.

    CREATE TABLE IF NOT EXISTS "customers" (
       "rowid"      INTEGER       PRIMARY KEY AUTOINCREMENT NOT NULL,
       "fname"      CHAR(40)      NOT NULL,
       "sname"      CHAR(40)      NOT NULL,
       "birthno"    CHAR(11)      NULL);

    CREATE UNIQUE INDEX IF NOT EXISTS "idx_customers_sname" ON "customers" ( "sname" ASC, "fname" ASC, "rowid" ASC );

Aside from that, this does look like a bug. SQLite is doing an unnecessary sort for the first query, and correctly using the index for the second.

I suspected that it might be related to handling of the special column name rowid, but it does the same thing if rowid is replaced with a more generic name like id as shown below.

    SQLite version 3.3.6
    Enter ".help" for instructions
    sqlite> CREATE TABLE IF NOT EXISTS "customers" (
       ...>    "id"      INTEGER       PRIMARY KEY AUTOINCREMENT NOT NULL,
       ...>    "fname"      CHAR(40)      NOT NULL,
       ...>    "sname"      CHAR(40)      NOT NULL,
       ...>    "birthno"    CHAR(11)      NULL);
    sqlite>
    sqlite> CREATE UNIQUE INDEX IF NOT EXISTS "idx_customers_sname"
       ...>    ON "customers" ( "sname" ASC, "fname" ASC, "id" ASC );
    sqlite>
    sqlite>
    sqlite> explain query plan SELECT * FROM customers ORDER BY sname ASC, fname ASC
    , id ASC;
    0|0|TABLE customers
    sqlite> explain query plan SELECT * FROM customers ORDER BY sname ASC, fname ASC
    ;
    0|0|TABLE customers WITH INDEX idx_customers_sname ORDER BY
    sqlite>
    sqlite> .explain on
    sqlite>
    sqlite> explain SELECT * FROM customers ORDER BY sname ASC, fname ASC, id ASC;
    addr  opcode          p1          p2          p3
    ----  --------------  ----------  ----------  ---------------------------------
    0     OpenVirtual     1           5           keyinfo(3,BINARY,BINARY)
    1     Goto            0           34
    2     Integer         0           0
    3     OpenRead        0           2
    4     SetNumColumns   0           4
    5     Rewind          0           19
    6     Rowid           0           0
    7     Column          0           1
    8     Column          0           2
    9     Column          0           3
    10    MakeRecord      4           0
    11    Column          0           2
    12    Column          0           1
    13    Rowid           0           0
    14    Sequence        1           0
    15    Pull            4           0
    16    MakeRecord      5           0
    17    IdxInsert       1           0
    18    Next            0           6
    19    Close           0           0
    20    OpenPseudo      2           0
    21    SetNumColumns   2           4
    22    Sort            1           32
    23    Integer         1           0
    24    Column          1           4
    25    Insert          2           0
    26    Column          2           0
    27    Column          2           1
    28    Column          2           2
    29    Column          2           3
    30    Callback        4           0
    31    Next            1           23
    32    Close           2           0
    33    Halt            0           0
    34    Transaction     0           0
    35    VerifyCookie    0           2
    36    Goto            0           2
    37    Noop            0           0
    sqlite> explain SELECT * FROM customers ORDER BY sname ASC, fname ASC;
    addr  opcode          p1          p2          p3
    ----  --------------  ----------  ----------  ---------------------------------
    0     Noop            0           0
    1     Goto            0           21
    2     Integer         0           0
    3     OpenRead        0           2
    4     SetNumColumns   0           4
    5     Integer         0           0
    6     OpenRead        2           4           keyinfo(3,BINARY,BINARY)
    7     Rewind          2           18
    8     RowKey          2           0
    9     IdxIsNull       0           17
    10    IdxRowid        2           0
    11    MoveGe          0           0
    12    Rowid           0           0
    13    Column          0           1
    14    Column          0           2
    15    Column          0           3
    16    Callback        4           0
    17    Next            2           8
    18    Close           0           0
    19    Close           2           0
    20    Halt            0           0
    21    Transaction     0           0
    22    VerifyCookie    0           2
    23    Goto            0           2
    24    Noop            0           0
    sqlite>


2006-Aug-03 17:33:51 by anonymous:
Thank you for clarification of single and double quotes usage. I will drop the quotes completely since using double quotes is little bit annoying inside C string literals...

It seems that the problem with index and sorting on primary key is independent of primary key column name. In fact, previously I was using "id" :-) and the result was the same as you mentioned.

 
1900 active 2006 Jul anonymous Unknown Pending   a.rottmann 1 1 CURRENT_TIMESTAMP keyword not inserting UTC date in column edit
This is the schema for my table.

create table char (player varchar(64) NOT NULL default '~', name varchar(64) NOT NULL default '~', date timestamp NOT NULL default current_timestamp)

Whenever an insert is made to the table the column 'date' does get a UTC timestamp, it gets a string value 'current_timestamp'. Is my schema wrong?

2006-Jul-30 22:31:06 by anonymous:
*doesnt get a UTC timestap


2006-Jul-31 00:38:49 by anonymous:
Works fine for me. What's the exact syntax of your INSERT statement?
 
1901 active 2006 Jul anonymous Unknown Pending   adamd 2 2 problem in select request with a alias table edit
I have a table with 3 columns : c0, c1 and c2

My request is: select * from (select *, 'test' as new_col from table) as tmp inner join (select 'test' as new_col) as tmp1 on tmp.new_col = tmp1.new_col;

The column's name as a result of this request (sqlite 3-3.3.6) is: |tmp.table.c0|tmp.table.c1|tmp.table.c2|tmp.new_col|tmp1.new_col

In sqlite 3-3.2.7, the column's name is: |c0|c1|c2|collected|new_col|new_col

Before this version, my request ran on mysql, postgresql and sqlite. Now I don't have the possibility of using this request with the new sqlite version.

2006-Jul-31 10:11:59 by anonymous:
sorry in In sqlite 3-3.2.7, the column's name is: |c0|c1|c2|new_col|new_col


2007-Jan-08 14:52:43 by anonymous:
I had a similar problem with SQLite in PHP, see my bug report here: http://bugs.php.net/bug.php?id=40064
 
1941 active 2006 Aug anonymous   Pending     1 1 Unrevolved _sqlite3ExprCodeAndCache with SQLITE_OMIT_TRIGGER edit
If SQLITE_OMIT_TRIGGER is set, linker complains about an unresolved _sqlite3ExprCodeAndCache symbol.

sqlite3ExprCodeAndCache is defined in expr.c and wrapped with #ifndef SQLITE_OMIT_TRIGGER.

However, references in

  insert.c, line 536
  update.c, line 348 and 362

are not wrapped with #ifndef SQLITE_OMIT_TRIGGER.

I followed the suggestion quoted below (posted earlier to this list) without avail.

Is it safe (or even required?) to change sqliteInt.h to

  #ifndef SQLITE_OMIT_TRIGGER
  void sqlite3ExprCodeAndCache(Parse*, Expr*);
  #else
  # define sqlite3ExprCodeAndCache(A,B)
  #endif

In the mailing list, DRH argued that the above change will probably fail and suggested that a safer fix would be to remove the #ifndef SQLITE_OMIT_TRIGGER from around the sqlite3ExprCodeAndCache function.

2006-Oct-12 17:35:32 by anonymous:
The problem is still present in 3.3.8. Removing the

  #ifndef SQLITE_OMIT_TRIGGER

from around the

  sqlite3ExprCodeAndCache

function seems to fix it. Could you commit this?

 
1946 new 2006 Aug anonymous Unknown New     2 2 .read file fails on blob fields with end-of-file char edit
I've a table with a blob fields. I put there binary data that contains 0x1a (end of file) symbol. It's alright until i try to dump table to file and then trying to import that file.

sqlite3 my_db
>.output my_file
>.dump table_with_blob
>.exit
del my_db

sqlite3 my_db
>.read my_file

Fails with "Incomplete SQL: ..." SQL break before 0x1a char

I'm on windows. Possibly solving with opening file as binary file.

Sorry for my English

 
1947 active 2006 Aug anonymous Shell Pending     3 3 ".mode insert" works bad with BLOBs edit
.mode insert displays BLOBs as strings, which isn't very good for embedded NULs.

Having output more like the one from .dump would be better, IMO.

  sqlite> select * from t;
  INSERT INTO table VALUES('');
  sqlite> .dump
  BEGIN TRANSACTION;
  CREATE TABLE t(f BLOB);
  INSERT INTO "t" VALUES(X'0041');
  COMMIT;
 
1948 active 2006 Aug anonymous Shell Pending     2 3 Double quotes are not escaped in csv mode edit
If text is exported using "csv" mode, double quotes in strings are not escaped. Generally double-quotes in a quoted field in CSV should be escaped by repeate. I.e., 'This is a "test".' could be about as "This is a ""test""."

This doesn't appear to be the behavior SQLite uses, so, in the meantime, I'll have to export my data using another method and then transform that data into CSV for my import script.

 
1953 active 2006 Sep anonymous TclLib Pending     4 3 Fix for false 64-bit comparisons "make test" failures on Cygwin edit
The trivial patch below allows Cygwin to correctly pass all (two dozen or so) 64-bit integer-related tests in "make test". It does so by treating all 64-bit integer SQL results as strings. (Note: SQLite has always produced correct 64-bit integer results, it's just that the test harness on Cygwin produces false failures without this patch.) There is no impact to other platforms, and allows us unfortunate Windows users to be useful members of society.

  RCS file: /sqlite/sqlite/src/tclsqlite.c,v
  retrieving revision 1.172
  diff -u -r1.172 tclsqlite.c
  --- src/tclsqlite.c     31 Aug 2006 15:07:15 -0000      1.172
  +++ src/tclsqlite.c     1 Sep 2006 17:27:44 -0000
  @@ -432,7 +432,12 @@
             if( v>=-2147483647 && v<=2147483647 ){
               pVal = Tcl_NewIntObj(v);
             }else{
  +#ifndef __CYGWIN__
               pVal = Tcl_NewWideIntObj(v);
  +#else
  +            int bytes = sqlite3_value_bytes(pIn);
  +            pVal = Tcl_NewStringObj((char *)sqlite3_value_text(pIn), bytes);
  +#endif
             }
             break;
           }
  @@ -1420,7 +1425,11 @@
                 if( v>=-2147483647 && v<=2147483647 ){
                   pVal = Tcl_NewIntObj(v);
                 }else{
  +#ifndef __CYGWIN__
                   pVal = Tcl_NewWideIntObj(v);
  +#else
  +                pVal = dbTextToObj((char *)sqlite3_column_text(pStmt, i));
  +#endif
                 }
                 break;
               }

Example test failures before patch:

  $ ./testfixture.exe test/misc2.testmisc2-1.1... Ok
  misc2-1.2... Ok
  misc2-2.1... Ok
  misc2-2.2... Ok
  misc2-2.3... Ok
  misc2-3.1... Ok
  misc2-4.1...
  Expected: [4000000000]
       Got: [-294967296]
  misc2-4.2...
  Expected: [4000000000 2147483648]
       Got: [-294967296 -2147483648]
  misc2-4.3... Ok
  misc2-4.4...
  Expected: [1 2147483648 2147483647]
       Got: [1 -2147483648 2147483647]
  misc2-4.5...
  Expected: [1 4000000000 2147483648 2147483647]
       Got: [1 -294967296 -2147483648 2147483647]
  misc2-4.6...
  Expected: [1 2147483647 2147483648 4000000000]
       Got: [1 2147483647 -2147483648 -294967296]
  misc2-5.1... Ok
  misc2-6.1... Ok
  misc2-7.1... Ok
  misc2-7.2... Ok
  misc2-7.3... Ok
  misc2-7.4... Ok
  misc2-7.5... Ok
  misc2-7.6... Ok
  misc2-7.7... Ok
  misc2-7.8... Ok
  misc2-8.1... Ok
  misc2-9.1... Ok
  misc2-9.2... Ok
  misc2-9.3... Ok
  misc2-10.1... Ok
  Thread-specific data deallocated properly
  5 errors out of 28 tests
  Failures on these tests: misc2-4.1 misc2-4.2 misc2-4.4 misc2-4.5 misc2-4.6

After patch applied:

  $ ./testfixture.exe test/misc2.testmisc2-1.1... Ok
  misc2-1.2... Ok
  misc2-2.1... Ok
  misc2-2.2... Ok
  misc2-2.3... Ok
  misc2-3.1... Ok
  misc2-4.1... Ok
  misc2-4.2... Ok
  misc2-4.3... Ok
  misc2-4.4... Ok
  misc2-4.5... Ok
  misc2-4.6... Ok
  misc2-5.1... Ok
  misc2-6.1... Ok
  misc2-7.1... Ok
  misc2-7.2... Ok
  misc2-7.3... Ok
  misc2-7.4... Ok
  misc2-7.5... Ok
  misc2-7.6... Ok
  misc2-7.7... Ok
  misc2-7.8... Ok
  misc2-8.1... Ok
  misc2-9.1... Ok
  misc2-9.2... Ok
  misc2-9.3... Ok
  misc2-10.1... Ok
  Thread-specific data deallocated properly
  0 errors out of 28 tests
  Failures on these tests:

The only new regression on Cygwin is this test, which is expected:

  types3-2.3...
  Expected: [wideInt]
       Got: []
2006-Sep-01 18:55:25 by drh:
The TCL interface is more than just part of the test harness. A lot of people use the TCL interface as part of their applications.

I believe what this patch does is mask a real problem.

I would prefer to fix the underlying problem, not just treat the symptom.


2006-Sep-02 02:48:57 by anonymous:
I have no interest in fixing bugs in Tcl itself on Cygwin. I just want to reliably build and test SQLite. The proposed fix is purely pragmatic and is intended only for the test harness. Indeed, when dealing with testing only, the fix is not Cygwin-specific and would work on any platform. The test harness under stock Cygwin as it stands simply does not work for 64 bit values. When you see such a failure you assume that SQLite is in error. Perhaps a compromise can be made and the code fix in question can be wrapped in #ifdef SQLITE_TESTFIXTURE or equivalent instead of #ifdef __CYGWIN__. I would hate to see someone else waste any time on this trivially fixable issue.


2006-Sep-02 13:27:08 by drh:
Perhaps you could put an "if" statement in the test scripts that skipped over the tests that do not work if running under cygwin. You can probably figure out if you are running under cygwin by looking at elements of the tcl_platform array.


2006-Sep-02 13:48:56 by drh:
I retract my previous suggestion. I do not want such patches in the SQLite source tree.

I will resist any patches such as shown here because they are really hacks to work around a faulty Tcl build on Cygwin. The correct way to fix this is to fix the Tcl build for Cygwin. This is probably as simple as download a copy of Tcl and recompiling. I'm curious to know why the default Tcl build for Cygwin only supports 32-bit integers. Is there some problem with 64-bit integer support on Cygwin?

The patch shown in the Description section above is not good because it presumes that Cygwin will always be broken. I think a better assumption is that Cygwin will get fixed. And I do not want to cripple the TCL interface to work around a bug that is not related to SQLite and which might not exist on every system. That is so wrong.

I will be willing to put in a test that checks for the cygwin brokenness and prints a warning to the user. Perhaps something like this:

   if {"*[db eval {SELECT 10000000000}]*"!="*10000000000*"} {
      puts "*********** WARNING *************"
      puts "Your build of TCL only supports 32-bit integers."
      puts "This will cause many test failures.  To run these"
      puts "tests you must install a version of TCL that supports"
      puts "64-bit integers."
      exit
   }

The question is, does that test correctly detect the broken Cygwin? Since I have no ready access to a windows machine, I have no way of testing it.


2006-Sep-02 14:06:28 by anonymous:
Then you would have an on-going maintenance issue with future tests. If'ing out valid tests just masks the problem and defeats the purpose of having a test regression suite. If a test fails legitimately, it should be reported as such. But these particular 64-bit tests work correctly if the simple proposed patch to the test harness is checked in. There is nothing wrong with the tests themselves - just the test harness on certain platforms for which Tcl does support 64-bit integers for whatever reason. Is the purpose of the test suite to test SQLite or Tcl implementations?

I know that Cygwin is a considered a tier "C" platform for SQLite, but appreciate that from a Cygwin environment me and many others have reported at least couple of dozen non-platform-specific SQLite bugs over the past year. You probably have as many or more Cygwin users on the mailing list than Mac OSX users. Why put up artificial ideological roadblocks?


2006-Sep-02 14:10:35 by anonymous:
Please do not check the "Your build of TCL only supports 32-bit integers". It is couter-productive to exit when the great majority of tests will pass. Such a check will basically exclude stock Cygwin installs from testing SQLite. Given the choice between having a broken test harness and this TCL 32-bit check, it is more useful to have a broken test harness.


2006-Sep-04 01:25:00 by anonymous:
Cygwin Tcl 8.5 64-bit integer math bug report

Cygwin Tcl 8.5a5 64-bit integer math fix

 
1954 active 2006 Sep anonymous Unknown Pending     1 1 Dual Core Processor Lockup edit
I seem to be seeing a problem with dual core processors in the the Open call is locking and does not release or throw an exception. It does not occur every time, but occurs around 50% of the time. I have not seen the problem on non dual core processors.
2006-Sep-02 21:06:38 by anonymous:
This ticket is way too vague to be actionable. What operating system? AMD or Intel? What specific version of SQLite? Was the library precompiled or did you compile it yourself?

Personally, I can report no errors or problems with dual-core CPU's on Windows XP using an AMD X2 4400+ dual-core CPU. Tested with both a 32-bit build and a 64-bit build of SQLite on x64 Windows.

 
1960 active 2006 Sep anonymous   Pending     4 2 Issues with .import in sqlite.exe edit
I ran into two possible problems when using the .import operation in sqlite3:

- .import seems to be confused by NULLs; in the file NullTest.dat the null is at the end of the line

- .import chokes on empty field when importing to field of type: integer PRIMARY KEY AUTOINCREMENT

For example line like:

~2~3~4~5~6

Example:

Schema:

--Table with autoincrement

CREATE TABLE test1( id integer PRIMARY KEY AUTOINCREMENT, c1 integer NULL , c2 integer NULL , c3 text NULL, c4 text NULL, c5 text NULL );

-- Table with no autoincrement field

CREATE TABLE test2( id integer NULL, c1 integer NULL , c2 integer NULL , c3 text NULL, c4 text NULL, c5 text NULL );

.separator ~

.import NullTest.dat test1

.import NullTest.dat test2

.import NoNullTest.dat test2

I have short test files that I can email to the person who is looking at this.

 
1974 active 2006 Sep anonymous Unknown Pending     1 1 column type not consistent in views edit
package require sqlite3

sqlite3 db test.db

db eval {

create table one ( size FLOAT );

create view two as select size from one; }

db eval {insert into one values(50.0)}

puts [db eval {select size from one}]

puts [db eval {select size from two}]

outputs:

50.0

50

 
1980 active 2006 Sep drh   Pending     1 1 Initializing FTS1 twice causes it to fail. edit
If you try to load the shared module twice, it causes the module to no longer work.
 
1983 active 2006 Sep anonymous   Pending     2 2 I/O Error at a size of 4GB and auto_vacuum=1 edit
when i'm building a database with auto_vacuum=1 and page_size=8192, i get an I/O error at a size of about 4GB. All tables are still readable but then it isn't possible to insert any more data.

The table is filled with a column of BLOBs and some columns with numbers. I use the 3.3.7 binary with Windows 2000 Server.

 
1990 active 2006 Sep anonymous   Pending     1 1 sqlite3_close doesn't release always the file handle edit
I think that sqlite3_close behave strangly.

I use version 3.3.7 on Linux (Fedora Core 5).

What I do is to open a database, and start a transaction in it. Then, without ending the transaction, open again the database and simply close it. I found out, that the inner sqlite3_close return 0 (SQLITE_OK), but the file handle is not released. So if I do it too many times, I run out of file handles.

You are free to ask why I open and close that many times the same database while it is already in transaction. This is my mistake. Actually, it is already fixed. But I still wonder - shouldn't the sqlite3_close return other thing then just SQLITE_OK? Especially if the file handle is not released? If it did, I would find my mistake much earlier.

Here is my script that demonstrate it (you can use /usr/sbin/lsof in linux to see how many times the file is opened):

  #include <sqlite3.h>
  int main(int argc, char **argv) {
    sqlite3* db;
    sqlite3* db_inner;
    int rc;
    int i;
    system("rm -f open_many_test.db");

    rc = sqlite3_open("open_many_test.db", &db);
    sqlite3_exec(db, "begin", 0, 0, 0);
    sqlite3_stmt *pStmt;
    rc = sqlite3_prepare(db,
                         "create table a (id varchar)",
                         -1,
                         &pStmt,
                         0);
    rc = sqlite3_step(pStmt);
    sqlite3_finalize(pStmt);

    rc = sqlite3_prepare(db,
                         "insert into a values('bla')",
                         -1,
                         &pStmt,
                         0);
    rc = sqlite3_step(pStmt);
    sqlite3_finalize(pStmt);

    for (i = 0; i < 10000; i++) {
      rc = sqlite3_open("open_many_test.db", &db_inner);
      printf("sqlite3_open gives %d\n", rc);

      rc = sqlite3_close(db_inner);
      printf("sqlite3_close gives %d\n", rc);
    }

    sqlite3_exec(db, "commit", 0, 0, 0);
    rc = sqlite3_close(db);
  }
2006-Sep-23 15:29:46 by drh:
This behavior is intentional. It is there to work around bugs in the design of posix advistory locks. See ticket #561 and check-in [1171] .

Under posix, if you have the same file open multiple times and you close one of the file descriptors, all locks on that file for all file descriptors are cleared. To prevent this from occurring, SQLite defers closing file descriptors until all locks on the file have been released.

One possible work-around would be to reuse file descriptors that waiting to be closed for the next open, rather than creating a new file descriptor.


2006-Sep-23 15:35:21 by anonymous:
The inner call should to sqlite3_open() should simply fail in that case, rather than set up a condition where by a file descriptor is leaked (which no one wants). This is unfortunate because sqlite3_open()'s behavior would not be uniform across platforms.


2006-Sep-23 16:43:32 by anonymous:
SQLite should do a lookup via stat()'s st_dev/st_ino fields prior to open() and if found to be the same as an already opened database file, it should use the same (refcounted) file descriptor, eliminating the need for open() in this case.

...upon reflection, having two sqlite connections using the same file descriptor would be a bad thing. stat() could be used to decide if a fd pending close() is recyclable, though.


2006-Sep-23 18:17:34 by drh:
Two points:

  1. SQLite does not and has never leaked file descriptors. All file descriptors are eventually closed. The close is merely deferred until the pending transaction COMMITs.

  2. I will be taking a very caution and careful approach toward resolving this issue. The issue itself is minor (it has only just now been reported but the behavior has been there for 3 years) but the consequences of getting the fix wrong are severe (database corruption.) And there are abundant opportunities for getting the fix wrong.
 
1992 active 2006 Sep anonymous   Fixed   shess 1 1 FTS1: Problems after dropping utility tables edit
There are problems if FTS1 utilities tables are dropped from a database. See following SQL for details.

  drop table if exists x;

  -- Create a FTS1 table.

  create virtual table x using fts1 ('content');

  -- Drop table x_content: Works fine, but should this be allowed?
  -- The same errors below also show if table x_term is dropped.

  drop table x_content;

  -- All attempts to access table x now result in errors,
  -- including dropping table x. There seems to be no way out
  -- except of recreating the database. All three commands below
  -- cause the same error, regardless if executed in sequence
  -- or individually:

  insert into x (content) values ('one two three'); -- Error!

  delete from x; -- Error!

  drop table x; -- Error!
Added "not exists" to allow dropping an fts table with corrupted backing. Allowing updates to such tables is unlikely to happen (not even clear what it would mean, in most cases!).
 
1994 active 2006 Sep anonymous Parser Pending     1 3 Columns from nested joins aren't properly propagated edit
When using this query:

    SELECT * FROM ROLE_ATTRIBUTE INNER JOIN (ROLE INNER JOIN PERSON ON ROLE.PERSON_ID=PERSON.ID) ON ROLE_ATTRIBUTE.PERSON_ID=ROLE.PERSON_ID AND ROLE_ATTRIBUTE.PROJECT_ID=ROLE.PROJECT_ID WHERE ((PERSON.FIRSTNAME = "bob"));

the parser fails with an error "no such column: ROLE.PROJECT_ID". It seems that doing an inner join with more than one subexpression doesn't work.

2006-Sep-25 22:41:52 by anonymous:
Your query will run without the brackets.

  SELECT *
  FROM PERSON P INNER JOIN ROLE_ATTRIBUTE RA
       ON P.ID = RA.PERSON_ID
       INNER JOIN ROLE R
       ON RA.PROJECT_ID = R.PROJECT_ID AND
          P.ID          = R.PERSON_ID
  WHERE P.FIRSTNAME = 'bob';


2006-Sep-25 23:03:28 by navaraf:
Hm, you're right. So actually the thing SQLite chokes on is the parenthesis syntax as JOIN parameter. I can try to modify the generator to produce the expanded form, but since the same code is used for MSSQL, MySQL and Oracle I still think it would be handy to allow it in SQLite too. Also it's not my code that generates these horrible expressions and I'd rather try to avoid modifying it.


2006-Sep-26 09:59:13 by anonymous:
I changed the title to correctly describe the problem. Also I found another thread on the mailing list that describes exactly the same problem: http://marc.10east.com/?t=115378699000001


2006-Sep-26 11:42:38 by navaraf:
I believe the "lookupName" function in src/expr.c should do recursion for ephemeral tables found in the pSrcList (at least those that were created as subqueries in the FROM clause of the SELECT statement).
 
2010 active 2006 Oct anonymous   Fixed_in_3.0     3 3 Timeout ignored in Shared-Cache locking model edit
With shared cache enabled, the busy timeout seems to be ignored. SQLITE_BUSY comes immediately. This occurs at least for locking situations within one shared cache.

My server (if i may call the cache sharing thread that way) has its own timeout handling. But I thought that a small timeout in sqlite3 might help to distinguish locks from deadlocks.

This was reproduced with both Python wrappers. These just call sqlite3_enable_shared_cache and sqlite3_busy_timeout and then execute BEGIN IMMEDIATE from two connections.

2006-Oct-06 13:56:21 by anonymous:
Weird, I thought it's my fault, but I see exactly the same behaviour with the C# ADO.NET 2.0 wrapper w/ the shared cache patch.
 
2011 active 2006 Oct anonymous   New     3 2 Escaping Porblem with .mode insert (double apostrophe) edit
select * from messages where message_id="74B23AAF-5FFD6BF2";

74B23AAF-5FFD6BF2|75|0|0|0|0|Europe talks, acts tough on Iran||http://www.ncr-iran.org/index.php?option=com_content&task=view&id=1052&Itemid=71|1140529235.0|By Gareth HardingThe United Press International, BRUSSELS -- Europeans are supposed to prefer soft to hard power, jaw-jaw to war-war and appeasement to confrontation. In short, in the words of neo-conservative scholar Robert Kagan: \'Americans are from Mars; Europeans are from Venus.\'

The ".mode insert / .output" file looks like this.

INSERT INTO messages VALUES('74B23AAF-5FFD6BF2',75,0,0,0,0,'Europe talks, acts tough on Iran','','http://www.ncr-iran.org/index.php?option=com_content&task=view&id=1052&Itemid=71',1140529235.0,'By Gareth HardingThe United Press International, BRUSSELS -- Europeans are supposed to prefer soft to hard power, jaw-jaw to war-war and appeasement to confrontation. In short, in the words of neo-conservative scholar Robert Kagan: \''Americans are from Mars; Europeans are from Venus.\''');

Now there are two apostrophe and the Escaping is broken.

 
2012 active 2006 Oct anonymous   New     4 3 trigger4.test aborts "make test" on Windows edit
The failure to remove these files causes "make test" to abort without completing remaining tests:

  trigger4-99.9... Ok
  ./testfixture: error deleting "trigtest.db": permission denied
      while executing
  "file delete -force trigtest.db trigtest.db-journal"
      (file "test/trigger4.test" line 199)

fix:

  Index: test/trigger4.test
  ===================================================================
  RCS file: /sqlite/sqlite/test/trigger4.test,v
  retrieving revision 1.9
  diff -u -3 -p -r1.9 trigger4.test
  --- test/trigger4.test        4 Oct 2006 11:55:50 -0000       1.9
  +++ test/trigger4.test        9 Oct 2006 14:09:07 -0000
  @@ -195,6 +195,6 @@ do_test trigger4-7.2 {

   integrity_check trigger4-99.9

  -file delete -force trigtest.db trigtest.db-journal
  +catch {file delete -force trigtest.db trigtest.db-journal}

   finish_test
Not sure why this ticket was set to Fixed_in_3.0, but I can reproduce the "make test" abort on Windows.


2006-Oct-11 00:27:16 by drh:
I do not know why the resolution was set to "Fixed_In_3.0" either. It seems to have been set that why by the original submitter.

I will fix this eventually, but since it does not represent a real malfunction, it has a lower priority.

 
2013 active 2006 Oct anonymous   Pending   drh 4 3 Autoincrement increments on failing INSERT OR IGNORE edit
  % package require sqlite3
  3.3.8
  % sqlite3 db ""
  % db eval "CREATE TABLE test (counter INTEGER PRIMARY KEY AUTOINCREMENT, value text NOT NULL UNIQUE)"
  % db eval "INSERT INTO test VALUES(4, 'hallo')"
  % db eval "SELECT * FROM sqlite_sequence"
  test 4
  % db eval "INSERT OR IGNORE INTO test(value) VALUES('hallo')"
  % db eval "SELECT * FROM sqlite_sequence"
  test 5
---> there has no dataset been inserted but the AUTOINCREMENT-counter is incremented

  % db eval "INSERT OR IGNORE INTO test VALUES(4, 'hallo')"
  % db eval "SELECT * FROM sqlite_sequence"
  test 5
---> right behavior: no inserted dataset and no incrementation

This maybe could be a problem if the "INSERT OR IGNORE" happens very often.

 
2014 active 2006 Oct anonymous   Pending   anonymous 4 3 Enhancement Req: CREATE [TEMP | TEMPORARY] VIRTUAL TABLE edit
Regarding the experimental VIRTUAL TABLE implementation, I believe it would of benefit to provide a "temp", or volatile construct when working with them.

-- From a SQL syntax perspective, adding an optional keyword "TEMP" to the declaration: CREATE [TEMP | TEMPORARY] VIRTUAL TABLE.

-- From a code perspective, I would envision this to invoke xCreate as it does now, but when the database is closed, the table is automatically dropped like any temp table, and xDestroy invoked rather than xDisconnect.

One sticky point I can picture is behavior when multiple opens exist to a single database from the same process space. Since virtual tables are already reference counted (in SQLite 3.3.8), perhaps the reference count could be made to span database handles and be bubbled up to the process level instead. That would allow the table to be CREATEd on one handle, CONNECTed on a second handle, then DISCONNECTed/DESTROYed based on the process-wide reference count.

I feel that there are numerous implementation possibilities for this. Having no option to auto-drop a virtual table can lead to stray module references, creating SQLite database files that cannot be properly utilized if the vtable module is not available.

Of course this can be implemented by the application calling DROP TABLE on it's own, but an embedded solution that takes care of it seems more 'proper' given the thought that goes into SQLite as a whole.

 
2017 active 2006 Oct anonymous   Pending     1 1 DROP TABLE fails on FTS1 utility tables with certain OMIT_s defined edit
The following SQL fails when SQLite is compiled with the SQLITE_OMIT_ defines stated below:

  create virtual table foo using fts1 (content);
  drop table foo;
  create virtual table foo using fts1 (content);

Cause: The foo_content and foo_term tables are not deleted.

To verify, please define these SQLITE_OMIT_s:

  OPTS += -DSQLITE_OMIT_ALTERTABLE
  OPTS += -DSQLITE_OMIT_ANALYZE
  OPTS += -DSQLITE_OMIT_AUTHORIZATION
  OPTS += -DSQLITE_OMIT_AUTOINCREMENT
  OPTS += -DSQLITE_OMIT_AUTOVACUUM
  OPTS += -DSQLITE_OMIT_BETWEEN_OPTIMIZATION
  OPTS += -DSQLITE_OMIT_BLOB_LITERAL
  OPTS += -DSQLITE_OMIT_CAST
  OPTS += -DSQLITE_OMIT_CHECK
  OPTS += -DSQLITE_OMIT_COMPLETE
  OPTS += -DSQLITE_OMIT_COMPOUND_SELECT
  OPTS += -DSQLITE_OMIT_EXPLAIN
  OPTS += -DSQLITE_OMIT_FLAG_PRAGMAS
  OPTS += -DSQLITE_OMIT_FOREIGN_KEY
  OPTS += -DSQLITE_OMIT_GET_TABLE
  OPTS += -DSQLITE_OMIT_GLOBALRECOVER
  OPTS += -DSQLITE_OMIT_INTEGRITY_CHECK
  OPTS += -DSQLITE_OMIT_LIKE_OPTIMIZATION
  OPTS += -DSQLITE_OMIT_MEMORYDB
  OPTS += -DSQLITE_OMIT_OR_OPTIMIZATION
  OPTS += -DSQLITE_OMIT_ORIGIN_NAMES
  OPTS += -DSQLITE_OMIT_PAGER_PRAGMAS
  OPTS += -DSQLITE_OMIT_PROGRESS_CALLBACK
  OPTS += -DSQLITE_OMIT_QUICKBALANCE
  OPTS += -DSQLITE_OMIT_REINDEX
  OPTS += -DSQLITE_OMIT_SCHEMA_VERSION_PRAGMAS
  OPTS += -DSQLITE_OMIT_SHARED_CACHE
  OPTS += -DSQLITE_OMIT_SUBQUERY
  OPTS += -DSQLITE_OMIT_TCL_VARIABLE
  OPTS += -DSQLITE_OMIT_TEMPDB
  OPTS += -DSQLITE_OMIT_TRACE
  OPTS += -DSQLITE_OMIT_TRIGGER
  OPTS += -DSQLITE_OMIT_UTF16
  OPTS += -DSQLITE_OMIT_VACUUM
  OPTS += -DSQLITE_OMIT_VIEW

Without the SQLITE_OMIT_s, everything works just fine.

 
2019 active 2006 Oct anonymous   Pending     1 1 FTS1: Create table in transaction raises Out of Sequence error (21) edit
This error:

  SQL error: library routine called out of sequence

is caused if the following script is executed by the Windows version of the SQLite3 console application with .load fts1.dll extension. If it does not show immediately, it will eventually surface if the script is run multiple times.

The cause of the problem seems to be related to the transaction, the create virtual table as well as the amount of data inserted.

Finally, the script is attached.

 
2022 active 2006 Oct anonymous   Pending     1 1 .import command is not working edit
I have a windows system running version 3.3.6 and a linux system running 3.3.3 when I run .import catalog.csv TEMPDATA on the windows system, it works fine. On the linux system, no data gets imported. There are no error messages.

Is this a known issue in 3.3.3?

2006-Oct-14 01:15:07 by anonymous:
A sample SQL schema and a 3 line import file demonstrating the problem would be helpful.


2006-Nov-08 15:48:28 by anonymous:
Schema: CREATE TABLE Catalog ( UPC text , SKU text primary key , DESC text , PACK text , PRICE text , SIZE text );

test.csv contents 00000000103,103,EFFEM CHOCOLATE FUNSIZE 75PPK 1 X1EA,1,$155.94,1 EA 00000000152,414317,CLEARLIGHT SLUSH CUP 16OZ CDL16 1X50EA,1,$5.04,50 EA 00000000152,56880,CLEARLIGHT SLUSH CUP 16OZ CDL16 20X50EA,20,$96.31,50 EA

Command that does nothing: .import test.csv Catalog


2006-Nov-08 15:52:40 by anonymous:
Sorry, I'll try this again:

  Schema:
CREATE TABLE Catalog (
UPC text
, SKU text primary key
, DESC text
, PACK text
, PRICE text
, SIZE text );

  test.csv contents
00000000103,103,EFFEM CHOCOLATE FUNSIZE 75PPK 1 X1EA,1,$155.94,1 EA
00000000152,414317,CLEARLIGHT SLUSH CUP 16OZ CDL16 1X50EA,1,$5.04,50 EA
00000000152,56880,CLEARLIGHT SLUSH CUP 16OZ CDL16 20X50EA,20,$96.31,50 EA

  Command that does nothing:
.import test.csv Catalog
 
2027 active 2006 Oct anonymous   Pending     1 1 FTS: Phrase searches return Offsets for individual phrase words edit
With FTS (one as well as two), phrase searches return offsets for all individual words instead of the phrase as a whole, like in

  select name, ingredients from recipe where ingredients match '"broccoli cheese"';

Offsets() returns at least two matches for both individual words:

  • broccoli
  • cheese
 
2028 active 2006 Oct anonymous   Pending     4 2 FTS1: UNIQUE() expression and UPDATE command not working edit
I'm working with tables, containing around 1,4 million entries (1GB file size). To allow faster fulltext search I tried FTS1 now.

What I saw is: creating the virtual FTS1 table with one keyword "UNIQUE(code), reference, text, ..." I had the idea to have faster access to "code", because this entry is only one time existing in table. In my actual SQLITE table "UNIQUE" was good idea, because "UPDATE"ing of entries was much faster as without "UNIQUE" expression. Unfortunately, in that moemnt I use "UNIQUE" expression in fulltext table, the FTS1 table doesn't accept insertion of entries like "INSERT into receipe (code, reference, text) values ('4711', 'RefnotAvailable', 'Test');"

So I removed the "UNIQUE" keyword, knowing that later "UPDATE" command to modify entries will be slower.

So I built new table with additional FTS1 fulltext table. Then I tried to "UPDATE" one entry. In that moment the program stopped immediately working (WIN XP system), what means that the application stopped without comment and returned to desktop. I tried the same in SQLITE3.exe (command line program) but also that program suspended immediately after the UPDATE command (like "UPDATE Volltext SET code = '4710', reference = 'RefChanged', text = 'notext';"

That seems to me to be a bug.

By the way, creating fulltext table to search inside my whole database increased the filesize a lot (4 times). May be that is solved in FTS2? Last wish: Fulltext search like "foo*" to find "fool" and "foot" would be a really great improvement.

Best regards Ingo

2006-Oct-23 13:56:59 by anonymous:
Ooops, as I saw today, also "DELETE" statements are causing SQLITE to stop working (crash). Program returns to Desktop on WIN XP after DELETE command.
 
2032 active 2006 Oct anonymous   Pending     1 1 AV in btree.c running FTS2 compiled with SQLITE_OMIT_SHARED_CACHE edit
If compiled with FTS2 support as well as SQLITE_OMIT_SHARED_CACHE=1, the sqlite console application causes an

  Access Violation: btree.c, line 3538: Read of address x00000014
    if( pCur->idx>=pPage->nCell ){

if the SQL (attatched) is executed.

I believe that this is a bug in btree.c, for the following reasons:

  • The AV does not show if the #ifndef SQLITE_OMIT_SHARED_CACHE (lines 3514 and 3525) are commentet out.

  • From my reading, all virtual tables use the extension API only and do not access the btree directly.
2006-Oct-25 06:30:43 by shess:
Note that the attached SQL has exactly 273 INSERT statements. 273==256+16+1, so this is kicking in at a merge point. Don't know how that's relevant, but it seems suspicious.


2006-Oct-25 16:31:34 by anonymous:
Many thanks for looking into this - it was driving me mad until I came up with the rather simple SQL to reproduce it.

I am not sure if the number of INSERTS is 100% the number needed to cause the problem, but the crash always happens after the exact same number of inserts. I did not count them but added roughly enough of them to cause the error.

Sidenote: I can also make FTS2 to crash at another point, which I thought was related to the sizeof() bug I also reported. But apprarently it is not. Unfortunately I can not provide a test case for this since I can reproduce it only after adding some 3000 or so copyrighted documents to an empty database. At the time of the crash the DB is about 250 MB in size. However, I will run a test after the next commits to FTS2.


2006-Oct-26 08:57:41 by anonymous:
My previious comments from yesterday seem to be invalidated by the latest checkins [3486] , [3488] and [3489] . Many thanks for those!

However, the problem with SQLITE_OMIT_SHARED_CACHE still persists.

 
2037 active 2006 Oct anonymous   Pending     1 1 Sqlite3 can't use datafile in Chinese path with Win2000 and WindowsXP. edit
Sqlite3 can't use datafile in Chinese path with Win2000 and WindowsXP. This is a bug in os_win.c . My firend modify code to so , it work right.

/* ** Convert a UTF-8 string to UTF-32. Space to hold the returned string ** is obtained from sqliteMalloc. */ static WCHAR *utf8ToUnicode(const char *zFilename){ int nChar; WCHAR *zWideFilename;

  if( !isNT() ){
    return 0;
  }
  nChar = MultiByteToWideChar(CP_THREAD_ACP, MB_COMPOSITE, zFilename, -1, NULL, 0);
  zWideFilename = sqliteMalloc( nChar*sizeof(zWideFilename[0]) );
  if( zWideFilename==0 ){
    return 0;
  }
  nChar = MultiByteToWideChar(CP_THREAD_ACP, MB_COMPOSITE, zFilename, -1, zWideFilename, nChar);
  if( nChar==0 ){
    sqliteFree(zWideFilename);
    zWideFilename = 0;
  }
  return zWideFilename;
}

/* ** Convert UTF-32 to UTF-8. Space to hold the returned string is ** obtained from sqliteMalloc(). */ static char *unicodeToUtf8(const WCHAR *zWideFilename){ int nByte; char *zFilename;

  nByte = WideCharToMultiByte(CP_THREAD_ACP, WC_COMPOSITECHECK, zWideFilename, -1, 0, 0, 0, 0);
  zFilename = sqliteMalloc( nByte );
  if( zFilename==0 ){
    return 0;
  }
  nByte = WideCharToMultiByte(CP_THREAD_ACP, WC_COMPOSITECHECK, zWideFilename, -1, zFilename, nByte,
                              0, 0);
  if( nByte == 0 ){
    sqliteFree(zFilename);
    zFilename = 0;
  }
  return zFilename;
}
2006-Oct-20 10:26:46 by anonymous:
The proposed fix is completely wrong, but the bug exists nonetheless. The problem is that SQLite expects file names in UTF-8 encoding (and there is probably bug in your application too guessing from the proposed fix). While this works fine on NT systems where the UTF-8 encoding is converted to UTF-16 and passed to system wide-character APIs, the code path for non-NT systems (Win 9x) with ANSI-only APIs doesn't convert the UTF-8 file names into the ANSI code page which is expected by the system APIs.
 
2043 active 2006 Oct anonymous   Pending     1 1 Spaces in view statement edit
If you have a table defined with fields that contain spaces.

create table table1 ("field one", "field two", "field three");

Then you do a select

select "field one" from table1;

That works fine. However if you save it as a view

create view view_one as select "field one" from table1;

Then if you run a select on the view it fails.

select * from view_one;

 
2046 active 2006 Oct anonymous   Fixed   shess 1 1 FTS1 - Error closing database due to unfinished statements edit
The following script causes an error in SQLite3.exe with FTS1. The error will surface only AFTER the script has finished AND you have typed .exit at the sqlite> prompt to quit SQLite3.

The problem seems that the SELECT statement is not properly finalized due to an internal error.

  -- The next line is for Windows only, please adopt it
  -- if running Linux or use a FTS1-enabled SQLite3 binary.
  select load_extension ('fts1.dll');

  CREATE TABLE Snippets(
    SnippetID INTEGER PRIMARY KEY,
    SnippetTitle TEXT,
    FtsID INTEGER);

  CREATE VIRTUAL TABLE SnippetsFts USING FTS1 (SnippetTitle, SnippetText);

  INSERT INTO Snippets (SnippetTitle) VALUES ('one');
  INSERT INTO Snippets (SnippetTitle) VALUES ('two');

  SELECT SnippetID FROM Snippets JOIN SnippetsFts ON FtsID = +SnippetsFts.RowID
    WHERE SnippetsFts MATCH 'one';

  -- After the script is done, type .exit at the prompt to close the database.
  --
  -- SQLite3 will close, but report the following error before doing so:
  --
  --   "error closing database: Unable to close due to unfinalised statements"
  --
  --  Does this qualify for a bug?

The script is also attached to this ticket.

2006-Nov-27 22:58:49 by shess:
Attached tighter version of the replication script, generated in isolating what mattered to the bug.
 
2048 active 2006 Oct anonymous   Pending   drh 1 1 table_info on columns with no default value are returned as string edit
On line 486, noDflt is declared as
static const Token noDflt = { (unsigned char*)"", 0, 0 };

And on line 493:
if( pDflt->z ){
sqlite3VdbeOp3(v, OP_String8, 0, 0, (char*)pDflt->z, pDflt->n);
}else{
sqlite3VdbeAddOp(v, OP_Null, 0, 0);

So columns with no default value aren't being set to null because the (pDflt->z) condition is non-null.
 
2057 active 2006 Nov anonymous   Pending     3 1 full_column_names when 2 or more tables are joined is not working edit
Version 2.8 has the behavior described in the documentation in respect to full_column_names when 2 or more tables are present with (table/alias).*, but 3.3.8 doesn't, mixing the pragmas "full_column_names" and "short_column_names" can only force to have full_column_names allways or never, some programs expect the behavior described in the documentation to remain working.
2006-Nov-08 20:10:13 by anonymous:
Version 3.3.3 as well has the same problem.


2006-Nov-09 09:34:52 by anonymous:
Changing the line 977 of select.c (3.3.8) from:

if( pEList->a[i].zName){

to:

if( pEList->a[i].zName && pTabList->nSrc==1){

with pragma short_column_names = 0 behaves like 2.8 series.

 
2059 active 2006 Nov anonymous   Pending     1 1 Still missing .DEF file from Windows 3.3.8 source code distribution edit
The file sqlite3.def is missing from the zip archive of sources used to build sqlite3 on Windows. Ticket number 2031 was closed with a remark that this file is generated during the build process. That is true if one is building on Linux with MinGW32 configured as a cross-compiler. If one were building using that method then I assume one would not be downloading the src.zip archive anyway.

My impression is that the src.zip archive is prepared once the build has been performed on Linux so Windows developers can directly build sqlite (and the generated files) without need of the other tools that the build process depends on. If this is accurate, then it would be very helpful if the src.zip archive could also include the sqlite3.def file. Without this file it is not possible for Windows developers to create a DLL from the src.zip archive.

Thanks

2006-Nov-09 20:05:23 by anonymous:
Works fine as is with MinGW ./configure && make sqlite3.exe
 
2060 active 2006 Nov anonymous   Pending     1 1 Table references enclosed in parenthesis become "invisible" edit
Hi,

I'm developing an RDF-based system, which translates queries from SPARQL into SQL. While trying to add support for SQLite (MySQL is already supported) I came across the following problem: when table references in a FROM clause are enclosed in parenthesis, they cannot be referenced from outside the parenthesized expression. For example, given the table definitions

  CREATE TABLE IF NOT EXISTS t1 (a, b);
  CREATE TABLE IF NOT EXISTS t2 (c, d);
  CREATE TABLE IF NOT EXISTS t3 (e, f);

The following queries all fail with "no such column" errors:

  SELECT t1.a, t3.f
  FROM (t1 CROSS JOIN t2 ON t1.b = t2.c) LEFT JOIN t3 ON t2.d = t3.e;

  SELECT t1.a, t3.f
  FROM t1 CROSS JOIN (t2 LEFT JOIN t3 ON t2.d = t3.e) ON t1.b = t2.c;

  SELECT t1.a, t2.d
  FROM (t1), (t2)
  WHERE t1.b = t2.c;

I'm not sure if it is always possible to reformulate the queries in such a way that the extra parenthesis aren't necessary, but I suspect that complex expressions involving joins may require them to achieve the intended semantics. In any case, my system would require large changes to be able to get rid of the parenthesized subjoins, so it would be nice if this problem could be fixed. :-)

2006-Nov-10 03:56:46 by anonymous:
For what it's worth, here's the parse trees of two similar queries ("SELECT t1.a, t2.d FROM t1, t2 WHERE t1.b = t2.c" and "SELECT t1.a, t2.d FROM (t1), (t2) WHERE t1.b = t2.c"), as well as one of the other more complicated join queries previously listed.

  SELECT t1.a, t2.d FROM t1, t2 WHERE t1.b = t2.c;
  Select {
    op: TK_SELECT
    isResolved: 1
    pSrc: {
      a[0]: {
        zName: t1
        iCursor: 0
        colUsed: 0x00000003
        pTab: t1
        jointype: JT_INNER
      }
      a[1]: {
        zName: t2
        iCursor: 1
        colUsed: 0x00000003
        pTab: t2
      }
    }
    pEList: {
      a[0]: {
        pExpr: {
          op: TK_COLUMN
          span: {t1.a}
          affinity: SQLITE_AFF_NONE
          iTable: 0
          iColumn: 0
          pTab: t1
        }
      }
      a[1]: {
        pExpr: {
          op: TK_COLUMN
          span: {t2.d}
          affinity: SQLITE_AFF_NONE
          iTable: 1
          iColumn: 1
          pTab: t2
        }
      }
    }
    pWhere: {
      op: TK_EQ
      span: {t1.b = t2.c}
      pLeft: {
        op: TK_COLUMN
        span: {t1.b}
        affinity: SQLITE_AFF_NONE
        iTable: 0
        iColumn: 1
        pTab: t1
      }
      pRight: {
        op: TK_COLUMN
        span: {t2.c}
        affinity: SQLITE_AFF_NONE
        iTable: 1
        iColumn: 0
        pTab: t2
      }
    }
  }

  SELECT t1.a, t2.d FROM (t1), (t2) WHERE t1.b = t2.c;
  Select {
    op: TK_SELECT
    isResolved: 1
    pSrc: {
      a[0]: {
        zAlias: sqlite_subquery_5C0A10_
        iCursor: 0
        pTab: sqlite_subquery_5C0A10_
        pSelect: {
          op: TK_SELECT
          isResolved: 1
          pSrc: {
            a[0]: {
              zName: t1
              iCursor: 1
              colUsed: 0x00000003
              pTab: t1
            }
          }
          pEList: {
            a[0]: {
              zName: a
              pExpr: {
                op: TK_COLUMN
                token: {a}
                span: {a}
                affinity: SQLITE_AFF_NONE
                iTable: 1
                iColumn: 0
                pTab: t1
              }
            }
            a[1]: {
              zName: b
              pExpr: {
                op: TK_COLUMN
                token: {b}
                span: {b}
                affinity: SQLITE_AFF_NONE
                iTable: 1
                iColumn: 1
                pTab: t1
              }
            }
          }
        }
        jointype: JT_INNER
      }
      a[1]: {
        zAlias: sqlite_subquery_5BE4F0_
        iCursor: 2
        pTab: sqlite_subquery_5BE4F0_
        pSelect: {
          op: TK_SELECT
          isResolved: 1
          pSrc: {
            a[0]: {
              zName: t2
              iCursor: 3
              colUsed: 0x00000003
              pTab: t2
            }
          }
          pEList: {
            a[0]: {
              zName: c
              pExpr: {
                op: TK_COLUMN
                token: {c}
                span: {c}
                affinity: SQLITE_AFF_NONE
                iTable: 3
                iColumn: 0
                pTab: t2
              }
            }
            a[1]: {
              zName: d
              pExpr: {
                op: TK_COLUMN
                token: {d}
                span: {d}
                affinity: SQLITE_AFF_NONE
                iTable: 3
                iColumn: 1
                pTab: t2
              }
            }
          }
        }
      }
    }
    pEList: {
      a[0]: {
        pExpr: {
          op: TK_COLUMN
          span: {t1.a}
          flags: EP_Resolved EP_Error
          iTable: -1
          iColumn: 0
        }
      }
      a[1]: {
        pExpr: {
          op: TK_DOT
          span: {t2.d}
          pLeft: {
            op: TK_ID
            token: {t2}
            span: {t2}
          }
          pRight: {
            op: TK_ID
            token: {d}
            span: {d}
          }
        }
      }
    }
    pWhere: {
      op: TK_EQ
      span: {t1.b = t2.c}
      pLeft: {
        op: TK_DOT
        span: {t1.b}
        pLeft: {
          op: TK_ID
          token: {t1}
          span: {t1}
        }
        pRight: {
          op: TK_ID
          token: {b}
          span: {b}
        }
      }
      pRight: {
        op: TK_DOT
        span: {t2.c}
        pLeft: {
          op: TK_ID
          token: {t2}
          span: {t2}
        }
        pRight: {
          op: TK_ID
          token: {c}
          span: {c}
        }
      }
    }
  }
  SQL error: no such column: t1.a

  SELECT t1.a, t3.f FROM (t1 CROSS JOIN t2 ON t1.b = t2.c) LEFT JOIN t3 ON t2.d = t3.e;
  Select {
    op: TK_SELECT
    isResolved: 1
    pSrc: {
      a[0]: {
        zAlias: sqlite_subquery_5BFA30_
        iCursor: 0
        pTab: sqlite_subquery_5BFA30_
        pSelect: {
          op: TK_SELECT
          isResolved: 1
          pSrc: {
            a[0]: {
              zName: t1
              iCursor: 1
              colUsed: 0x00000003
              pTab: t1
              jointype: JT_INNER JT_CROSS
            }
            a[1]: {
              zName: t2
              iCursor: 2
              colUsed: 0x00000003
              pTab: t2
            }
          }
          pEList: {
            a[0]: {
              zName: a
              pExpr: {
                op: TK_COLUMN
                span: {t1.a}
                affinity: SQLITE_AFF_NONE
                iTable: 1
                iColumn: 0
                pTab: t1
              }
            }
            a[1]: {
              zName: b
              pExpr: {
                op: TK_COLUMN
                span: {t1.b}
                affinity: SQLITE_AFF_NONE
                iTable: 1
                iColumn: 1
                pTab: t1
              }
            }
            a[2]: {
              zName: c
              pExpr: {
                op: TK_COLUMN
                span: {t2.c}
                affinity: SQLITE_AFF_NONE
                iTable: 2
                iColumn: 0
                pTab: t2
              }
            }
            a[3]: {
              zName: d
              pExpr: {
                op: TK_COLUMN
                span: {t2.d}
                affinity: SQLITE_AFF_NONE
                iTable: 2
                iColumn: 1
                pTab: t2
              }
            }
          }
          pWhere: {
            op: TK_EQ
            span: {t1.b = t2.c}
            flags: EP_FromJoin EP_Resolved
            iRightJoinTable: 2
            pLeft: {
              op: TK_COLUMN
              span: {t1.b}
              affinity: SQLITE_AFF_NONE
              flags: EP_FromJoin EP_Resolved
              iTable: 1
              iColumn: 1
              iRightJoinTable: 2
              pTab: t1
            }
            pRight: {
              op: TK_COLUMN
              span: {t2.c}
              affinity: SQLITE_AFF_NONE
              flags: EP_FromJoin EP_Resolved
              iTable: 2
              iColumn: 0
              iRightJoinTable: 2
              pTab: t2
            }
          }
        }
        jointype: JT_LEFT JT_OUTER
      }
      a[1]: {
        zName: t3
        iCursor: 3
        pTab: t3
      }
    }
    pEList: {
      a[0]: {
        pExpr: {
          op: TK_COLUMN
          span: {t1.a}
          flags: EP_Resolved EP_Error
          iTable: -1
          iColumn: 0
        }
      }
      a[1]: {
        pExpr: {
          op: TK_DOT
          span: {t3.f}
          pLeft: {
            op: TK_ID
            token: {t3}
            span: {t3}
          }
          pRight: {
            op: TK_ID
            token: {f}
            span: {f}
          }
        }
      }
    }
    pWhere: {
      op: TK_EQ
      span: {t2.d = t3.e}
      flags: EP_FromJoin
      iRightJoinTable: 3
      pLeft: {
        op: TK_DOT
        span: {t2.d}
        flags: EP_FromJoin
        iRightJoinTable: 3
        pLeft: {
          op: TK_ID
          token: {t2}
          span: {t2}
          flags: EP_FromJoin
          iRightJoinTable: 3
        }
        pRight: {
          op: TK_ID
          token: {d}
          span: {d}
          flags: EP_FromJoin
          iRightJoinTable: 3
        }
      }
      pRight: {
        op: TK_DOT
        span: {t3.e}
        flags: EP_FromJoin
        iRightJoinTable: 3
        pLeft: {
          op: TK_ID
          token: {t3}
          span: {t3}
          flags: EP_FromJoin
          iRightJoinTable: 3
        }
        pRight: {
          op: TK_ID
          token: {e}
          span: {e}
          flags: EP_FromJoin
          iRightJoinTable: 3
        }
      }
    }
  }
  SQL error: no such column: t1.a


2006-Nov-11 18:29:33 by anonymous:
The resolving bug appears to be that unique column names or column aliases are searched across all subqueries, but table names and table aliases are only searched at their current SELECT level only.

With this in mind, here are mechanical workarounds without using column aliases (assumes the column names in all joined tables are unique):

  SELECT a, f FROM (t1 CROSS JOIN t2 ON t1.b = t2.c) LEFT JOIN t3 ON d = e;

  SELECT t1.a, f FROM t1 CROSS JOIN (t2 LEFT JOIN t3 ON t2.d = t3.e) ON t1.b = c;

  SELECT a, d FROM (t1), (t2) WHERE b = c;

And here are mechanical workarounds using column aliases (assumes the column names are not unique between tables):

  SELECT t1.a, t3f FROM t1 CROSS JOIN (select t3.f t3f, t2.c t2c from t2 LEFT JOIN t3 ON t2.d = t3.e) ON t1.b = t2c;

  SELECT t1a, t3.f FROM (select t1.a t1a, t2.d t2d from t1 CROSS JOIN t2 ON t1.b = t2.c) LEFT JOIN t3 ON t2d = t3.e;

  SELECT t1a, t2d FROM (select t1.a t1a, t1.b t1b from t1), (select t2.c t2c, t2.d t2d from t2) WHERE t1b = t2c;

Notice that t3.f in the second query did not require an alias because the table "t3" was part of its immediate SELECT. You could make an alias for every column just in case, I just wanted to highlight the difference.


2007-Feb-13 15:40:31 by anonymous:
Fixing this issue would slow down SELECT parsing and column resolution for all queries (more specifically all prepared statements) due to the recursion required for column resolution. It would be easier to change your SQL code generator to accomodate SQLite. Just make aliases for every table at every subselect level and have the SELECT at any given level only work with the table aliases at that level.
 
2066 active 2006 Nov anonymous   Pending     2 2 Incorrect error message in the case of ENOLCK edit
If you're trying to open a sqlite database that is stored on a filesystem that doesn't support locking, then you'll get the error when you try to execute any commands on it:

    Error: file is encrypted or is not a database

If you run sqlite under strace, you see:

   read(0, ".schema\n.quit\n", 4096)       = 14
   fcntl64(3, F_SETLK64, {type=F_RDLCK, whence=SEEK_SET, start=1073741824, len=1}, 0xafa5cd70) = 0
   fcntl64(3, F_SETLK64, {type=F_RDLCK, whence=SEEK_SET, start=1073741826, len=510}, 0xafa5cd70) = 0
   fcntl64(3, F_SETLK64, {type=F_UNLCK, whence=SEEK_SET, start=1073741824, len=1}, 0xafa5cd70) = 0
   access("/mnt/www/zzz_old_sites/trac.db-journal", F_OK) = -1 ENOENT (No such file or directory)
   fstat64(3, {st_mode=S_IFREG|0644, st_size=584704, ...}) = 0
   _llseek(3, 0, [0], SEEK_SET)            = 0
   read(3, "** This file contains an SQLite "..., 1024) = 1024
   fcntl64(3, F_SETLK64, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}, 0xafa5cdd0) = -1 ENOLCK (No locks available)
   write(2, "Error: file is encrypted or is n"..., 46Error: file is encrypted or is not a database

Sqlite should really check the exact error code, and give a more helpful error (eg "Locking not available on this filesystem. Databases may only be stored on filesystems that support locking")

 
2075 active 2006 Nov anonymous   Pending     3 3 Improve VACUUM speed and INDEX page locality edit
In testing several 100 Meg - 1 Gig databases (including the Monotone OpenEmbedded database) I found that changing the order of the SQL commands executed by VACUUM to create indexes after table inserts results in 15% faster VACUUM times, and up to 25% faster cold-file-cache queries when indexes are used. This patch effectively makes the pages of each index contiguous in the database file after a VACUUM, as opposed to being scattered throughout the pages of the table related to the index. Your results may vary, but I think this is a very safe change that can potentially boost average database performance.

  Index: src/vacuum.c
  ===================================================================
  RCS file: /sqlite/sqlite/src/vacuum.c,v
  retrieving revision 1.65
  diff -u -3 -p -r1.65 vacuum.c
  --- src/vacuum.c      18 Nov 2006 20:20:22 -0000      1.65
  +++ src/vacuum.c      20 Nov 2006 21:09:27 -0000
  @@ -143,14 +143,6 @@ int sqlite3RunVacuum(char **pzErrMsg, sq
         "   AND rootpage>0"
     );
     if( rc!=SQLITE_OK ) goto end_of_vacuum;
  -  rc = execExecSql(db,
  -      "SELECT 'CREATE INDEX vacuum_db.' || substr(sql,14,100000000)"
  -      "  FROM sqlite_master WHERE sql LIKE 'CREATE INDEX %' ");
  -  if( rc!=SQLITE_OK ) goto end_of_vacuum;
  -  rc = execExecSql(db,
  -      "SELECT 'CREATE UNIQUE INDEX vacuum_db.' || substr(sql,21,100000000) "
  -      "  FROM sqlite_master WHERE sql LIKE 'CREATE UNIQUE INDEX %'");
  -  if( rc!=SQLITE_OK ) goto end_of_vacuum;

     /* Loop through the tables in the main database. For each, do
     ** an "INSERT INTO vacuum_db.xxx SELECT * FROM xxx;" to copy
  @@ -162,10 +154,22 @@ int sqlite3RunVacuum(char **pzErrMsg, sq
         "FROM sqlite_master "
         "WHERE type = 'table' AND name!='sqlite_sequence' "
         "  AND rootpage>0"
  -
     );
     if( rc!=SQLITE_OK ) goto end_of_vacuum;

  +  /* Create indexes after the table inserts so that their pages
  +  ** will be contiguous resulting in (hopefully) fewer disk seeks.
  +  */
  +  rc = execExecSql(db,
  +      "SELECT 'CREATE UNIQUE INDEX vacuum_db.' || substr(sql,21,100000000) "
  +      "  FROM sqlite_master WHERE sql LIKE 'CREATE UNIQUE INDEX %'");
  +  if( rc!=SQLITE_OK ) goto end_of_vacuum;
  +
  +  rc = execExecSql(db,
  +      "SELECT 'CREATE INDEX vacuum_db.' || substr(sql,14,100000000)"
  +      "  FROM sqlite_master WHERE sql LIKE 'CREATE INDEX %' ");
  +  if( rc!=SQLITE_OK ) goto end_of_vacuum;
  +
     /* Copy over the sequence table
     */
     rc = execExecSql(db,
2007-Feb-11 00:49:50 by drh:
My alternative plan is to modify insert.c so that it recognizes the special case of

    INSERT INTO table1 SELECT * FROM table2;

when table1 and table2 have identical schemas, including all the same indices. When this special case is recognized, the generated bytecode will first transfer all table entries from table2 to table1, using a row by row transfer without decoding each row into its constituient columns. Then it will do the same for each index.

There will be two benefits here. First, when the above construct occurs during the course of a VACUUM, the table and each index, including intrisic indices associated with UNIQUE and PRIMARY KEY constraints, will be transferred separately so that all of there pages will be adjacent in the database file. The second benefit will occur when trying to load large quantities of data into an indexed table. Loading indexed data into a very large table is currently slow because the index entries are scattered haphazardly around in the file. But if data is first loaded into a smaller temporary table with the same schema, it can then be transferred to the main table using an INSERT statement such as the above in what amounts to a merge operation.


2007-Feb-11 06:58:36 by anonymous:
There's no question that your proposal will greatly improve VACUUM speed which relies on the "INSERT INTO table1 SELECT * from table2" construct. But would it be possible for you to relax the restriction on having identical indexes for table1 and table2? For that matter it would be nice if table2 could be any subselect or view. Then "REPLACE INTO table1 SELECT ...anything..." could also be optimized. Since you can detect that SQLite is doing a bulk insert anyway, it could generate code to make a temporary staging table with automatically generated identical indexes to table1 which could be periodically merged into table1 and truncated every X rows. X could be either set via pragma or be a function of the size of the page cache. The temporary staging table would be dropped after the bulk INSERT INTO ... SELECT. Every user inserting large volumes of data would have to perform this procedure anyway. Manually recreating all the indexes for a given temporary table to match the original table and performing the looping logic is cumbersome and error-prone. It would be very conveniant if SQLite were to do it on the user's behalf. This scheme could only work if there are no triggers on table1, of course.


2007-Feb-11 09:16:25 by drh:
My initial enhancement does nothing to preclude the more agressive enhancement described by anonymous. In order to avoid subtle bugs, and in view of my limited time available to work on this, I think it best to take the more conservative approach first and defer the more elaborate optimization suggested by anonymous until later.


2007-Feb-11 13:54:34 by anonymous:
It should be possible to identify contiguous blocks of individual "INSERT INTO table1 VALUES(...)" statements to the same table within a large transaction and perform the same proposed optimization as with "INSERT INTO table1 SELECT ...". This would require higher level coordination by the parser. Anytime a read operation (SELECT, UPDATE) occurs on such a table marked for bulk INSERT within the large transaction, its temp staging table could be merged into the INSERT destination table and the staging table truncated. The process could be repeated for the remainder of the transaction.

Such an optimization would be a huge benefit to SQLite users since they would need not know the idiosynchracies of the implementation of "INSERT INTO table1 SELECT ..." in order to have efficient table and index population.

Alternatively, if you wish to avoid the complexity of re-assembling and staging individual INSERT statements, it might be a good opportunity for SQLite to support the multi-row variant of the INSERT command:

  INSERT INTO table1 (a,b,c) VALUES(1,2,3), (4,5,6), (7,8,9);

Which is essentially a transform of:

  CREATE TEMP TABLE table1_staging (a,b,c);
  INSERT INTO table1_staging VALUES(1,2,3);
  INSERT INTO table1_staging VALUES(4,5,6);
  INSERT INTO table1_staging VALUES(7,8,9);
  INSERT INTO table1 SELECT * FROM table1_staging;
  -- TRUNCATE OR DROP table1_staging as necessary

which could use the same bulk staging optimization.


2007-Feb-13 02:42:41 by anonymous:
Any harm in checking in the simple patch above for the 3.3.13 release?


2007-Feb-13 12:51:47 by drh:
I have a much better fix standing by that I intend to check-in as soon as I get 3.3.13 out the door. I don't want this in 3.3.13 for stability reasons.


2007-Feb-18 23:07:08 by anonymous:
Some related analysis and an .import patch using a :memory: staging table with the "INSERT INTO table1 SELECT FROM table2" construct can be found here:

http://www.mail-archive.com/sqlite-users%40sqlite.org/msg22143.html

 
2076 active 2006 Nov anonymous   Pending   a.rottmann 1 1 % exists as value in varchar edit
abnormal abend of client application (C++) when sqlite returns stream of data containing "%" value. Is % a special character?
2006-Nov-21 14:14:25 by anonymous:
% is not a special character. Can you post a small C program demonstrating the problem?
 
2077 active 2006 Nov anonymous   Pending     2 1 Problems with using ASCII symbols 0x80 - 0xFF in database path edit
Platform: Windows.

The SQLite library and executable doesn't see database files that are placed into folders named using ASCII symbols with codes 0x80-0xFF. That symbols are used to represent language-specific symbols (for example, Russian). In result, database cannot be placed into folder with name in Russian language. This bug is "unstable": it doesn't appear in all cases. Below are logs from my experiments with this problem. In all cases the path I requested exists, and database file is placed there. I have noticed that problem depends on filename path and name lengths. =========================================================
// creating test database
E:\!DISTRIB\sqlite-3_3_7>sqlite3.exe test.sqb
SQLite version 3.3.7
Enter ".help" for instructions
sqlite> create table a(id int);
sqlite> insert into a values (1);
sqlite> ^C

E:\!DISTRIB\sqlite-3_3_7>copy test.sqb e:\test.sqb
'3'\'`'a'Z'b'`'S'Q'_'` 'f'Q'[']'`'S: 1. //This means that 1 file was copied

E:\!DISTRIB\sqlite-3_3_7>sqlite3 e:\test.sqb
SQLite version 3.3.7
Enter ".help" for instructions
sqlite> select * from a;
1
sqlite> ^C
// Works!

E:\!DISTRIB\sqlite-3_3_7>mkdir e:\'/

//Using ASCII symbol "'/" (0x8D) to represent cyrillic letter which can be entered in the command line by using Alt+(141) combination

E:\!DISTRIB\sqlite-3_3_7>copy test.sqb E:\'/\test.sqb
'3'\'`'a'Z'b'`'S'Q'_'` 'f'Q'[']'`'S: 1.

E:\!DISTRIB\sqlite-3_3_7>sqlite3 e:\'/\test.sqb
SQLite version 3.3.7
Enter ".help" for instructions
sqlite> select * from a;
1
sqlite> ^C
// That is works too!

E:\!DISTRIB\sqlite-3_3_7>mkdir E:\'/\1
E:\!DISTRIB\sqlite-3_3_7>copy test.sqb E:\'/\1\test.sqb
'3'\'`'a'Z'b'`'S'Q'_'` 'f'Q'[']'`'S: 1.
E:\!DISTRIB\sqlite-3_3_7>sqlite3 E:\'/\1\test.sqb
Unable to open database "E:\(T\1\test.sqb": unable to open database file
// Doesn't work, and writes the wrong symbol "(T" in place of "'/"! I've noticed that if we convert symbol "'/" from DOS encoding to Windows encoding and then write it in DOS encoding, then we'll get "(T".

E:\!DISTRIB\sqlite-3_3_7>copy test.sqb E:\'/\tst.sqb
'3'\'`'a'Z'b'`'S'Q'_'` 'f'Q'[']'`'S: 1.

E:\!DISTRIB\sqlite-3_3_7>sqlite3 E:\'/\tst.sqb
SQLite version 3.3.7
Enter ".help" for instructions
sqlite> select * from a;
SQL error: no such table: a
sqlite> ^C

// It seems to work, i don't get an error, but doesn't see the tables! =(
=================================
 
2081 active 2006 Nov anonymous   Pending   doughenry 1 1 sqlite3_column_decltype throws exception, if selection is grouped edit
If I "group by" a selection over several columns I can't find out the orgin type of these columns using sqlite3_column_decltype(..). An exception is thrown.
2006-Nov-23 18:37:47 by anonymous:
You also get no decl type from a subselect. This goes to the typeless nature of SQLite - I don't think a type can even be derived in this case.
 
2084 active 2006 Nov anonymous   Pending     4 3 Add API function mapping column decl string to SQLite type edit
This is an API feature request.

It would be nice to be able to obtain the SQLite type (e.g. SQLITE_INTEGER) from the declared column type string as returned by sqlite3_column_decltype.

This was discussed briefly on the mailing list here: http://marc.10east.com/?l=sqlite-users&m=116422872301957&w=2

The function I have in mind is:

int sqlite3_decltype_to_type(const char *decl) { Token decl_token; char aff_type; int col_type;

    decl_token.z = decl;
    if( decl_token.z ){
        decl_token.n = strlen(decl_token.z);
        aff_type = sqlite3AffinityType(&decl_token);
        switch( aff_type ){
        case SQLITE_AFF_INTEGER:
            col_type = SQLITE_INTEGER;
            break;
        case SQLITE_AFF_NUMERIC: /* falls through */
        case SQLITE_AFF_REAL:
            col_type = SQLITE_FLOAT;
            break;
        case SQLITE_AFF_TEXT:
            col_type = SQLITE_TEXT;
            break;
        case SQLITE_AFF_NONE:
            col_type = SQLITE_BLOB;
            break;
        default:
            col_type = 0; /* unknown */
            break;
        }
    }
    return col_type;
}

If this seems agreeable, I would be willing to put together a real patch. However, I would need some guidance on where it should go. I'm not sure what should happen when no type can be determined.

2006-Nov-26 22:32:45 by anonymous:
According to the comment above the function sqlite3AffinityType: "If none of the substrings in the above table are found, SQLITE_AFF_NUMERIC is returned". The default condition in sqlite3_decltype_to_type will not be reached.


2006-Nov-26 23:04:23 by anonymous:
Thanks for pointing to that comment. Looks like SQLITE_AFF_NUMERIC is, for these purposes, unknown. So the case statement could be:

        switch( aff_type ){
        case SQLITE_AFF_INTEGER:
            col_type = SQLITE_INTEGER;
            break;
        case SQLITE_AFF_REAL:
            col_type = SQLITE_FLOAT;
            break;
        case SQLITE_AFF_TEXT:
            col_type = SQLITE_TEXT;
            break;
        case SQLITE_AFF_NONE:
            col_type = SQLITE_BLOB;
            break;
        case SQLITE_AFF_NUMERIC: /* falls through */
        default:
            col_type = 0; /* unknown */
            break;
        }


2006-Nov-27 02:43:06 by anonymous:
Your first function was correct, it just had some unreachable code. There's no unknown affinity, in the absence of a match the affinity is assumed to be numeric:

  int sqlite3_decltype_to_type(const char *decl) {
    int type = SQLITE_FLOAT;
    if( decl ){
        Token token;
        token.z = decl;
        token.n = strlen(token.z);
        switch( sqlite3AffinityType(&token) ){
        case SQLITE_AFF_INTEGER:
            type = SQLITE_INTEGER;
            break;
        case SQLITE_AFF_TEXT:
            type = SQLITE_TEXT;
            break;
        case SQLITE_AFF_NONE:
            type = SQLITE_BLOB;
            break;
        default:
            break;
        }
    }
    return type;
  }
 
2089 active 2006 Nov anonymous   Pending     3 3 Decouple sqlite_int64 from other 64bit datatypes edit
Currently sqlite3 makes the (valid) assumption that sqlite_int64 (or i64, u64) is 64 bit wide, matches with Tcl_WideInt and has the same datasize (and byte order) than double. The following patch fixes this and allows sqlite_int64 to be any integral type, e.g. a 32bit int (with the limitations of the reduced datatype size).

The use case for this is for systems that do not support 64bit integers (e.g. lack of compiler feature, embedded system), db's of small data size, and systems without large file support. The patch allows compiling with -DSQLITE_INT64_TYPE=int -DSQLITE_32BIT_ROWID for such a system.

2006-Nov-29 01:13:07 by anonymous:
Hm, now I wanted to add the patch file but I don't get the formatting right without editing the file and removing empty lines. How am I supposed to add a patch file (created with diff -ru)?
 
2093 active 2006 Dec anonymous   Pending     2 3 sqlite3_vtab_cursor doesn't have errMsg edit
The sqlite3_vtab_cursor structure doesn't have a zErrMsg pointer. Only the containing vtable does. This means that operations on cursor objects that have an error have to set the error on the vtable not the cursor. Unfortunately this means that there are race conditions since two different cursors on the same vtable could have errors at the same time. If the cursors are in different threads then a crash or worse can happen.
 
2096 active 2006 Dec anonymous   Pending     3 3 ATTACH DATABASE returns SQLITE_ERROR when database is locked edit
From an email sent to DRH:

I am working on a problem surrounding the inability to ATTACH to a database file. The error text being returned is "database is locked", which should be SQLITE_BUSY, however, the error code being returned by sqlite3_exec is SQLITE_ERROR. Is sqlite3_exec wrong in returning SQLITE_ERROR rather than SQLITE_BUSY? I have some nagging feeling that I determined or read that the attachFunc function does not return a truly-relevant status code, but I can't see why offhand nor can I find any evidence to support that theory. If sqlite3_exec is doing the right thing, however, then the question becomes one of identifying when to retry the ATTACH statement; we're currently keying off SQLITE_BUSY or SQLITE_LOCKED, as appropriate, and I'd rather not be trying to trap errors based on error text.

 
2100 active 2006 Dec anonymous   Pending     1 1 Fixes for SQL lower() and upper() edit
As acknowledged in the documentation, the SQL lower() and upper() functions might not work correctly on UTF-8 characters. This bug might show if a country specific locale is used instead of the standard C locale. Under certain circumstances, SQL lower() or upper() can even corrupt the UTF-8 string into invalid UTF-8 if the tolower() and toupper() C functions convert character values starting from 0x80.

Below I propose implementations of lowerFunc() and upperFunc() which work correctly with UTF-8 characters, regardless of the implementation of the C library tolower() and toupper() functions. If these C functions are implemented to support high ASCII or even Unicode case conversion, the new SQL lower() and upper() will support them as well.

The proposed C implementation applies a technique also found in sqlite3VdbeMemTranslate() in utf.c and makes use of some macros contained in that unit. To avoid duplicating existing code, it could make sense to move lowerFunc() and lowerFunc() to utf.c, just as it has been done with sqlite3utf16Substr().

Finally, here is the code:

  /*
  ** Implementation of the upper() and lower() SQL functions.
  */
  static void upperFunc(sqlite3_context *context, int argc, sqlite3_value **argv){
    const unsigned char *zIn, *zInTerm;
    unsigned char *z, *zOut;
    int c, l;
    if( argc<1 || SQLITE_NULL==sqlite3_value_type(argv[0]) ) return;
    zIn = sqlite3_value_text(argv[0]);
    if( zIn==0 ) return;
    l = sqlite3_value_bytes(argv[0]);
    zInTerm = &zIn[l];
    /* When converting case, the maximum growth results from
    ** translating a 1-byte UTF-8 character to a 4-byte UTF-8 character.
    */
    zOut = sqliteMalloc( l * 4 );
    z = zOut;
     while( zIn<zInTerm ){
       READ_UTF8(zIn, c);
       c = toupper(c);
       WRITE_UTF8(z, c);
     }
    sqlite3_result_text(context, zOut, z-zOut, sqlite3FreeX);
  }

  static void lowerFunc(sqlite3_context *context, int argc, sqlite3_value **argv){
    const unsigned char *zIn, *zInTerm;
    unsigned char *z, *zOut;
    int c, l;
    if( argc<1 || SQLITE_NULL==sqlite3_value_type(argv[0]) ) return;
    zIn = sqlite3_value_text(argv[0]);
    if( zIn==0 ) return;
    l = sqlite3_value_bytes(argv[0]);
    zInTerm = &zIn[l];
    /* When converting case, the maximum growth results from
    ** translating a 1-byte UTF-8 character to a 4-byte UTF-8 character.
    */
    zOut = sqliteMalloc( l * 4 );
    z = zOut;
     while( zIn<zInTerm ){
       READ_UTF8(zIn, c);
       c = tolower(c);
       WRITE_UTF8(z, c);
     }
    sqlite3_result_text(context, zOut, z-zOut, sqlite3FreeX);
  }
2006-Dec-05 19:07:32 by anonymous:
I'm not sure this is an improvement. The argument to the ctype.h functions tolower(int C) and toupper(int C) is only defined when C is an integer in the range `EOF' to `255'. The compiler implementation may return garbage for values outside this range.


2006-Dec-06 16:17:23 by anonymous:
The integer in the range 'EOF' to '255' is exactly the problem with the existing implementation: This range will (and does, in some cases) modify UTF-8 bytes (not a character) starting from 128, which can lead to invalid UTF-8 sequences.

If the result of tolower() is indeed undefined for values greater than 255 in some C-implementations then I agree that adding this check to the new code will probably be a good thing:

  if( c < 256 ){
    c = tolower(c);
  }

If, on the other hand, tolower() just returns the character unchanged for values greater than 255, then this check is not necessary.

I want to emphasize once more that the proposed new SQL lower() and upper() implementations never return invalid UTF-8, whereas the current implementation sometimes does. This is, by itself, already a great improvement.


2006-Dec-07 15:29:07 by anonymous:
This modified patch does better than previous if SQLITE_UNICODE_UPPERLOWERFUNCS is defined at compile time, because it uses libc _wcsupr / _wcslwr routines.

  #ifdef SQLITE_UNICODE_UPPERLOWERFUNCS
    #define WCHAR_T_SIZE		sizeof(wchar_t)
    #if (WCHAR_T_SIZE == 2)
    #define MAXUPPERLOWERCHAR_AVAIL 	0x0000ffff
    #else // (WCHAR_T_SIZE == 4)
    #define MAXUPPERLOWERCHAR_AVAIL 	0x7fffffff
    #endif // (WCHAR_T_SIZE == 2)
    #define TOLOWERSQLFUNC(c) unicode_tolower
    #define TOUPPERSQLFUNC(c) unicode_toupper
    int unicode_tolower(const int c) {
      wchar_t buff [2];
      if (c > MAXUPPERLOWERCHAR_AVAIL)
        return c;
      buff[0] = (wchar_t) c;
      buff[1] = 0;
      _wcslwr(buff);
      return (int) buff[0];
    }
    int unicode_toupper(const int c) {
      wchar_t buff [2];
      if (c > MAXUPPERLOWERCHAR_AVAIL)
        return c;
      buff[0] = (wchar_t) c;
      buff[1] = 0;
      _wcsupr(buff);
      return (int) buff[0];
    }
  #else // SQLITE_UNICODE_UPPERLOWERFUNCS
    #define TOLOWERSQLFUNC(c) (c > 255 ? c : tolower(c))
    #define TOUPPERSQLFUNC(c) (c > 255 ? c : toupper(c))
  #endif // SQLITE_UNICODE_UPPERLOWERFUNCS

  /*
  ** Implementation of the upper() and lower() SQL functions.
  */
  static void upperFunc(sqlite3_context *context, int argc, sqlite3_value **argv){
    const unsigned char *zIn, *zInTerm;
    unsigned char *z, *zOut;
    int c, l;
    if( argc<1 || SQLITE_NULL==sqlite3_value_type(argv[0]) ) return;
    zIn = sqlite3_value_text(argv[0]);
    if( zIn==0 ) return;
    l = sqlite3_value_bytes(argv[0]);
    zInTerm = &zIn[l];
    /* When converting case, the maximum growth results from
    ** translating a 1-byte UTF-8 character to a 4-byte UTF-8 character.
    */
    zOut = sqliteMalloc( l * 4 );
    z = zOut;
     while( zIn<zInTerm ){
       READ_UTF8(zIn, c);
       c = TOUPPERSQLFUNC(c);
       WRITE_UTF8(z, c);
     }
    sqlite3_result_text(context, zOut, z-zOut, sqlite3FreeX);
  }

  static void lowerFunc(sqlite3_context *context, int argc, sqlite3_value **argv){
    const unsigned char *zIn, *zInTerm;
    unsigned char *z, *zOut;
    int c, l;
    if( argc<1 || SQLITE_NULL==sqlite3_value_type(argv[0]) ) return;
    zIn = sqlite3_value_text(argv[0]);
    if( zIn==0 ) return;
    l = sqlite3_value_bytes(argv[0]);
    zInTerm = &zIn[l];
    /* When converting case, the maximum growth results from
    ** translating a 1-byte UTF-8 character to a 4-byte UTF-8 character.
    */
    zOut = sqliteMalloc( l * 4 );
    z = zOut;
     while( zIn<zInTerm ){
       READ_UTF8(zIn, c);
       c = TOLOWERSQLFUNC(c);
       WRITE_UTF8(z, c);
     }
    sqlite3_result_text(context, zOut, z-zOut, sqlite3FreeX);
  }
 
2122 review 2006 Dec anonymous   Fixed     1 3 Potential wrong string could stay in with convertUtf8Filename edit
sqlite3WinOpenExclusive uses the following code to convert the utf8 string to a os string:

  void *zConverted = convertUtf8Filename(zFilename);
  if( zConverted==0 ){
    return SQLITE_NOMEM;
  }
and use another string converted with another function to save the filename that will be deleted in windows ce:

#if OS_WINCE
  f.zDeleteOnClose = delFlag ? utf8ToUnicode(zFilename) : 0;
  f.hMutex = NULL;
#endif
also, the variable fileflags should be a DWORD (unsigned integer 32-bit) instead of a native C compiler integer.

the following patch correct those things:


int sqlite3WinOpenExclusive(const char *zFilename, OsFile **pId, int delFlag){
  winFile f;
  HANDLE h;
  DWORD fileflags;
  void *zConverted = convertUtf8Filename(zFilename);
  if( zConverted==0 ){
    return SQLITE_NOMEM;
  }
  assert( *pId == 0 );
  fileflags = FILE_FLAG_RANDOM_ACCESS;
#if !OS_WINCE
  if( delFlag ){
    fileflags |= FILE_ATTRIBUTE_TEMPORARY | FILE_FLAG_DELETE_ON_CLOSE;
  }
#endif
  if( isNT() ){
    int cnt = 0;
    do{
      h = CreateFileW((WCHAR*)zConverted,
         GENERIC_READ | GENERIC_WRITE,
         0,
         NULL,
         CREATE_ALWAYS,
         fileflags,
         NULL
      );
    }while( h==INVALID_HANDLE_VALUE && cnt++ < 2 && (Sleep(100), 1) );
  }else{
#if OS_WINCE
    return SQLITE_NOMEM;
#else
    int cnt = 0;
    do{
      h = CreateFileA((char*)zConverted,
        GENERIC_READ | GENERIC_WRITE,
        0,
        NULL,
        CREATE_ALWAYS,
        fileflags,
        NULL
      );
    }while( h==INVALID_HANDLE_VALUE && cnt++ < 2 && (Sleep(100), 1) );
#endif /* OS_WINCE */
  }
  if( h==INVALID_HANDLE_VALUE ){
    sqliteFree(zConverted);
    return SQLITE_CANTOPEN;
  }
  f.h = h;
#if OS_WINCE
  f.zDeleteOnClose = delFlag ? (WCHAR*) zConverted : 0;
  f.hMutex = NULL;
  if (!delFlag)
#endif
    sqliteFree(zConverted);
  TRACE3("OPEN EX %d \"%s\"\n", h, zFilename);
  return allocateWinFile(&f, pId);
}
2006-Dec-21 00:48:28 by drh:
On wince, convertUtf8Filename() calls utf8ToUnicode() to do its work, so these routines end up computing exactly the same filename.


2006-Dec-21 01:30:33 by anonymous:
Yes, I figured out that they still calling the same function, but why converting / freeing then converting agains ? just save some CPU cycles (this could help a lot in Windows CE < 200 mhz, just like my app runs (windows ce 2.11, SH3 130MHz HP Jornada 680)!!! Please apply my patch to make it faster.
 
2126 active 2006 Dec anonymous   Pending     3 1 Update hook not invoked when deleteing all rows from table edit
I was testing the update hook feature of SQLite and incidentally I noticed that hook is not invoked for "DELETE FROM" statement with no WHERE clause. Hook works well for "DELETE FROM ... WHERE ..." statement.

Steps to reproduce:

  1. Open database, setup update hook
  2. Execute:
    CREATE TABLE Test(Test INTEGER);
  3. Insert some data:
    INSERT INTO TEST (Test) VALUES (1); -- update hook invoked for INSERT
    INSERT INTO TEST (Test) VALUES (2); -- update hook invoked for INSERT
    INSERT INTO TEST (Test) VALUES (3); -- update hook invoked for INSERT
  4. Execute:
    DELETE FROM TEST; -- update hook IS NOT INVOKED(!) for each row.
2006-Dec-22 15:38:44 by drh:
Triggers don't work either. This is a feature not a bug. When you do "DELETE FROM table" with no WHERE clause, SQLite drops and recreates the table. Doing it this way have a huge speed boost.

If you really need the update hook to work add a "WHERE 1" to the end of the query.


2006-Dec-22 16:11:11 by anonymous:
I don't really need it, maybe other users. You should update the documentation of sqlite3_update_hook() and triggers mentioning this behaviour.

Another solution is to perform "DELETE FROM" as if it was with WHERE clause, if there is an update hook regitered.

 
2127 active 2006 Dec anonymous   Pending     2 3 Virtual tables do not always free zErrmsg edit
The documentation for virtual tables and in particular the sqlite3_vtab structure says "The SQLite core will free and zero the content of zErrMsg when it delivers the error message text to the client application or when it destroys the virtual table."

The latter part does not appear to be true ("when it destroys the virtual table"). I can't find any code that does actually that. (eg vtab.c:496 definitely doesn't, nor does vtab.c:76)

Usually the former case happens. However some operations have their error codes ignored (eg xClose). This can result in the zErrMsg pointing to a message but no error code returned upstream (which would clear the message). Finally as far as I can tell the responsibility for freeing sqlite3_vtab is with the xDisconnect/xDestroy callbacks since the corresponding xCreate/xConnect callbacks allocated it. Consequently there is no way for SQLite to even access zErrmsg since it would be a member of a freed structure after xDisconnect/xDestroy returned.

 
2128 active 2006 Dec anonymous   Pending     4 3 virtual table code doesn't verify type of rowid (calling xUpdate) edit
The virtual tables code doesn't verify the type of rowid when calling update. For example I used the following query:

UPDATE foo SET rowid='a string' WHERE 1

This results in a call to xUpdate with argv[0] equal the current rowid but argv[1] is 'a string'. While I'd be quite happy for rowids to be any SQLite type, the xRowid call only allows 64 bit integers. I believe SQLite should check the new rowid in a case like this is an integer and reject it, rather than calling xUpdate with the bad value. (I also just checked with rowid=3.4 and rowid=NULL and they get passed through as is as well)

A workaround is to document that the xUpdate method must check the new rowid is an integer type.

 
2131 active 2006 Dec anonymous   Pending     2 3 Add substring() function (Part of SQL 99) edit
sqlite> SELECT substring('foobar.class',-6,6) = '.class'; SQL error: no such function: substring sqlite> SELECT SUBSTRING('foobar.class',-6,6) = '.class'; SQL error: no such function: SUBSTRING sqlite> SELECT SUBSTR('foobar.class',-6,6) = '.class'; 1 sqlite> SELECT substring('foobar.class' FROM -6 FOR 6) = '.class'; SQL error: near "FROM": syntax error

Looking at: http://www.oreilly.com/catalog/sqlnut/chapter/ch04.html

SQL99 Syntax

SUBSTRING(extraction_string FROM starting_position [FOR length] [COLLATE collation_name])

It would be useful for sqlite to support this syntax too to make the SQL more portable.

2006-Dec-28 16:03:03 by anonymous:
sqlite has the substr() routine (func.c code):

    { "substr",             3, 0, SQLITE_UTF8,    0, substrFunc },
#ifndef SQLITE_OMIT_UTF16
    { "substr",             3, 0, SQLITE_UTF16LE, 0, sqlite3utf16Substr },
#endif
this could be 'aliased' to help you using the substring() SQL99 std just doing:

    { "substr",             3, 0, SQLITE_UTF8,    0, substrFunc },
    { "substring",          3, 0, SQLITE_UTF8,    0, substrFunc },
#ifndef SQLITE_OMIT_UTF16
    { "substr",             3, 0, SQLITE_UTF16LE, 0, sqlite3utf16Substr },
    { "substring",          3, 0, SQLITE_UTF16LE, 0, sqlite3utf16Substr },
#endif
 
2140 active 2007 Jan anonymous   Pending     3 1 sqlite doesn't link to readline edit
sqlite relies on another library to link to libreadline, causing this error with LDFLAGS=-Wl,--as-needed:

gcc -O2 -march=i686 -pipe -DOS_UNIX=1 -DHAVE_USLEEP=1 -DHAVE_FDATASYNC=1 -I. -I./src -DNDEBUG -DTHREADSAFE=1 -DSQLITE_THREAD_OVERRIDE_LOCK=-1 -DSQLITE_OMIT_CURSOR -DHAVE_READLINE=1 -I/usr/include/readline -o .libs/sqlite3 ./src/shell.c ./.libs/libsqlite3.so -lpthread -lncurses /tmp/cclOD1M7.o: In function `process_input': shell.c:(.text+0x37a5): undefined reference to `readline' shell.c:(.text+0x37c0): undefined reference to `add_history' /tmp/cclOD1M7.o: In function `main': shell.c:(.text+0x3f01): undefined reference to `read_history' shell.c:(.text+0x3f1a): undefined reference to `stifle_history' shell.c:(.text+0x3f22): undefined reference to `write_history' collect2: ld returned 1 exit status

 
2165 active 2007 Jan anonymous   Pending   drh 4 3 pager performance and checksum edit
pager.c

Embedd the 2 byte pager_pagehash() result into the page, near the beginning of the page. Use the intire page to calculate the pager_pagehash exclusive of the two byte page_hash data embedded in the page. That way a simple xor as in CHECK_PAGE of the entire page including the 2 byte pager_pagehash is all that is needed to validate a page. Also you could include the "4 byte" random at the beginning and at the end... But that would be a bit of overkill. The sampling of only every 200 bytes is interesting.

Review change the SQLITE_CHECK_PAGES ifdef to a SQLITE_OMIT_PAGE_CHECK.. As on disk page validity is very important. Could this be integrated into a pragma setting?

2007-Jan-12 18:08:49 by anonymous:
uint16_t pg_chkval(Dpage *pg, uint32_t pg_size) { register int i; register uint16_t val = 0; register uint16_t *bw = (uint16_t *) pg;

  for(i= 0; i < pg_size;i=i+2  )
      val= val^ *bw++  ;

  return val;
}

uint16_t pg_calcval(Dpage *pg, uint32_t page_size) { int i; register uint16_t val = 0; register uint16_t *bw = (uint16_t *) pg;

  /* Scan up to location where chk val is stored */
  for(i= 0; i < 8;i=i+2  )
       val= val^ *bw++  ;

  val = val^ 0;
  bw ++ ;

  /* Now scan the tail of the block */
  for(i=10; i < page_size;i=i+2 )
      val= val^ *bw++  ;

  return val;
}
 
2171 review 2007 Jan anonymous   Fixed   drh 3 3 Fix a major leakage of token minors in lemon generated parsers edit
Fix a major leakage of token minors in lemon generated parsers due to the fact that lemon was not genmerating destructor code for elements in the RHS of rules without C-code.

This is a copy of a fix mafe to the lemon parser used by the wireshark project http://wireshark.org

http://anonsvn.wireshark.org/viewvc/viewvc.py/trunk/tools/lemon/lemon.c?r1=20442&r2=20441&pathrev=20442

--- trunk/tools/lemon/lemon.c	2007/01/15 19:11:29	20441
+++ trunk/tools/lemon/lemon.c	2007/01/15 19:48:06	20442
@@ -3245,6 +3245,8 @@
   for(i=0; i<rp->nrhs; i++) used[i] = 0;
   lhsused = 0;

+  if (! rp->code) rp->code = "\n";
+
   append_str(0,0,0,0);
   for(cp=rp->code; *cp; cp++){
     if( safe_isalpha(*cp) && (cp==rp->code || (!safe_isalnum(cp[-1]) && cp[-1]!='_')) ){
@@ -3875,7 +3877,7 @@

   /* Generate code which execution during each REDUCE action */
   for(rp=lemp->rule; rp; rp=rp->next){
-    if( rp->code ) translate_code(lemp, rp);
+		translate_code(lemp, rp);
   }
   for(rp=lemp->rule; rp; rp=rp->next){
     struct rule *rp2;
It appears that the change in [3593] causes yy_destructor() to be called more than once for tokens, while the one posted here (and applied to wireshark) does not. in the meanwhile I changed the wireshark copy back to have
if (! rp->code)  rp->code = "\n";

   append_str(0,0,0,0);
   for(cp=rp->code; *cp; cp++){

but there's an issue with this the, generated parser gets

#line 0 "./dtd_grammar.lemon"

instead of the number pointing to the line where the rule is.

I'll keep investigating and keep you POSTed (literally :-)

 
2183 active 2007 Jan anonymous   Pending   drh 1 1 OMIT_SHARED_CACHE: AV and crash with FTS2 INSERT edit
Given that SQLite is compiled with

  -DSQLITE_ENABLE_FTS2=1
  -DSQLITE_OMIT_SHARED_CACHE=1

the following code crashes after about 273 insertions with

  Access violation: Read of address 0x00000014

at

  btree.c, line 3451: if( pCur->idx>=pPage->nCell ){

Here is the code to reproduce:

  int main(int argc, char* argv[])
  {
     sqlite3_stmt *pStmt;
     int i;

     check( sqlite3_open( "test_fts2.db3", &db) );

     check( sqlite3_exec( db, "CREATE VIRTUAL TABLE FTS USING FTS2 (Content);", 0, 0, 0));
     check( sqlite3_exec( db, "BEGIN TRANSACTION;", 0, 0, 0));

     check( sqlite3_prepare( db, "INSERT INTO FTS (Content) VALUES ('Far out in the uncharted backwaters of the unfashionable end of the Western Spiral arm of the Galaxy lies a small unregarded yellow sun.');", -1, &pStmt, NULL));
     for( i = 1; i < 1000; i++) {
        printf( "%d\n", i);
        check( sqlite3_step( pStmt) );
        check( sqlite3_reset( pStmt) );
     }

     check( sqlite3_exec( db, "COMMIT;", 0, 0, 0));
     check( sqlite3_finalize( pStmt ));
     check( sqlite3_close( db ));

     printf ("Done");
     scanf ("*%s");

    return 0;
  }

Could this be related to ticket #2032?

 
2203 active 2007 Jan anonymous   Pending     2 3 table_info pragma "default" column format changed? edit
Beginning with SQLite3 3.3.8, it looks like the format of the 'default' value returned by the table_info pragma has changed. Before, it used to be a bare string:

  dev:~> sqlite3
  SQLite version 3.3.7
  Enter ".help" for instructions
  sqlite> create table testings (a integer primary key, b string default 'Tester', c string default NULL);
  sqlite> pragma table_info(testings);
  0|a|integer|0||1
  1|b|string|0|Tester|0
  2|c|string|0||0

After 3.3.8, the 'defaults' column is now a SQL-quoted string:

  dev:~> sqlite3
  SQLite version 3.3.11
  Enter ".help" for instructions
  sqlite> create table testings (a integer primary key, b string default 'Tester', c string default NULL);
  sqlite> pragma table_info(testings);
  0|a|integer|0||1
  1|b|string|0|'Tester'|0
  2|c|string|0|NULL|0

Now, I think I do prefer the latter, where the default is a SQL-quoted string. However, this seems a rather significant change to make mid-stream, in a minor point release. It broke all Ruby on Rails applications that use sqlite3, for instance, because Rails reads that default value to determine how to default the value of each new record.

Was this intentional? Or is this a bug? I'd love to see this behavior reverted and saved for a release with a more significant release number.

2007-Jan-29 22:01:54 by anonymous:
One of your fellow Railers requested this change: Ticket #2078


2007-Jan-29 22:10:55 by drh:
See also ticket #1919 which might also be an issue here.


2007-Jan-29 22:33:19 by anonymous:
Anonymous, you make it sound as if anyone associated with Rails can make a request of the sqlite3 team and have it be automatically assumed to be sanctioned by the Rails core team. Whoever did the original request did not do so under the umbrella of Rails core. If that change was the one that resulted in this behavior, it most definitely should not have been recommended, and would not have been blessed by any of the core team.

At this point, though, I'm not interested in blame. I just want to see what can be done to make sqlite3 work with Rails again, preferably in a way that is backwards compatible with previous sqlite3 releases.


2007-Jan-30 05:20:56 by anonymous:
I agree the feature should be fixed due to backwards compatability, but Rails should try to accomodate both pragma variants since they are both "in the wild". You could base your decision on the sqlite version string, for instance.


2007-Jan-30 18:39:44 by anonymous:
Just FYI, there's another related ticket at the Rails trac at http://dev.rubyonrails.org/ticket/6523, and a Debian bug report with a patch at http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=397531.
 
2215 active 2007 Feb anonymous   Pending     1 2 error messages in virtual table are not propagated edit
I'm trying to return a customized error message in xBestIndex in my virtual table implementation. Rather than copying my implementation here the problem can be reproduced by changing the fulltextBestIndex method from fts1.

For example:

    /* Decide how to handle an SQL query. */
    static int fulltextBestIndex(sqlite3_vtab *pVTab, sqlite3_index_info *pInfo){
      int i;
      TRACE(("FTS1 BestIndex\n"));

      pVTab->zErrMsg = sqlite3_mprintf ("THIS IS AN ERROR MESSAGE");
      return SQLITE_ERROR;

      for(i=0; i<pInfo->nConstraint; ++i){
        const struct sqlite3_index_constraint *pConstraint;

If I run the following I do not see the error message reported when using the shell.

    $ ./sqlite3
    SQLite version 3.3.12
    Enter ".help" for instructions
    sqlite> create virtual table foo using fts1(name, address);
    sqlite> insert into foo (name, address) values ('amanda', '43 elm avenue');
    sqlite> select * from foo;
    SQL error: SQL logic error or missing database
 
2219 active 2007 Feb shess   Fixed   shess 2 2 Creating an fts table in an attached database works wrong. edit
   ATTACH DATABASE 'test2.db' AS two;
   CREATE VIRTUAL TABLE two.t2 USING fts2(content);

will put t2_content, t2_segments, and t2_segdir in database 'main' rather than database 'two'. In some cases everything will appear to work, because the tables will be defaults for that name.

 
2223 active 2007 Feb scouten   Pending     3 3 pragma auto_vacuum doesn't survive .dump & reconstitute edit
When you run sqlite3 path/to/database .dump, it does not contain pragma auto_vacuum even if that option was chosen when creating the source database.
2007-Feb-08 18:13:27 by scouten:
We wonder if other pragmas are also not being propogated.


2007-Feb-08 18:53:42 by anonymous:
No pragmas are output from .dump. SQLite should have a .dump_with_pragmas command or equivalent.


2007-Nov-27 02:11:05 by anonymous:
auto_vacuum is especially important since you need to specify it before loading the tables in the dump; if you notice that it's missing after loading a dump, it's too late.

I'd also appreciate user_version surviving.

Would patches be welcome for these? I would be happy to contribute one: mail glasser@davidglasser.net.

 
2241 active 2007 Feb scouten   Pending     3 3 pragma integrity_check no longer affects command-line tool's exit code edit
With the 3.3.5 build, if you did the following:

   sqlite3 /path/to/db "PRAGMA integrity_check;"

You seemed get a 1 result back on the command line if the check failed in any way (and 0 if it was "ok").

With 3.3.13, this doesn't seem to be the case, even if you specify the -bail command line parameter. (I also tried "PRAGMA integrity_check=1;", which is all I care about in this case.) I have a database with errors that returns 0 to the calling process (that is, the sqlite command line's main function returns 0).

We would like to see the 3.3.5 behavior return.

2007-Mar-29 18:32:54 by drh:
I pulled versions of SQLite going back to 2006-01-01 and I could not find one where the shell returned a non-zero exit code when PRAGMA integrity_check failed. Are you sure that you do not have a customized version of the 3.3.5 shell?
 
2288 active 2007 Apr anonymous   Pending     4 2 FTS does not support REPLACE edit
Simple to replicate:

CREATE VIRTUAL TABLE fts_table USING fts2(text);

INSERT OR REPLACE INTO fts_table (rowid, text) VALUES (1, 'text1');

INSERT OR REPLACE INTO fts_table (rowid, text) VALUES (1, 'text2');

The first insert succeeds, the second fails. Also occurs with fts1.

2007-Apr-10 15:27:10 by anonymous:
http://www.mail-archive.com/sqlite-users%40sqlite.org/msg23865.html
 
2294 active 2007 Apr anonymous   Pending     2 1 segfault when destroying lock on WinCE with threads edit
DestroyLock emulation on WinCE platform releases the zDeleteOnClose file outside the mutex acquire section. This lead to frequent segfault when working with several databases concurrently.

Patch simply consists in moving code:

    if( pFile->zDeleteOnClose ){
      DeleteFileW(pFile->zDeleteOnClose);
      sqliteFree(pFile->zDeleteOnClose);
      pFile->zDeleteOnClose=NULL;
    }

from winClose() (os_win.c:980) to winceDestroyLock(), inside scope of winceMutexAcquire(pFile->hMutex):

    /* De-reference and close our copy of the shared memory handle */
    UnmapViewOfFile(pFile->shared);
    CloseHandle(pFile->hShared);

    + if( pFile->zDeleteOnClose ){
    +   DeleteFileW(pFile->zDeleteOnClose);
    +   sqliteFree(pFile->zDeleteOnClose);
    +   pFile->zDeleteOnClose=NULL;
    + }

    /* Done with the mutex */
    winceMutexRelease(pFile->hMutex);
    CloseHandle(pFile->hMutex);
    pFile->hMutex = NULL;
2007-Apr-13 00:21:33 by anonymous:
Check in [3836] fixes it for me. eTcl regression tests, related to running several sqlite database concurrently in several threads, are now passed while they were frequently segfaulting without it.

To help being confident with this blind commit, let's mention that exactly same patch has been introduced in eTcl built since a couple of monthes, to fix issue reported by WM2003 users, and all reported a correct fix. However, I did suggest the patch, so testing and feedback from others may help :-)

Also, note that [3836] has a typo, requesting feedback in ticket #2249 instead of #2294


2007-Sep-25 03:24:09 by anonymous:
The fix doesn't work and should be reverted. Temporary files do not create locks, so when they are closed and hMutex is null, the winceDestroyLock() file is never called and the temporary files are not cleaned up properly.
 
2297 active 2007 Apr anonymous   Pending   drh 3 3 uninitialized var (with patch) edit
Warnings with amalgamation and NDEBUG.
2007-Apr-12 21:21:29 by drh:
I looked at the suggested changes and I didn't find any cases where it really was possible to use an uninitialized variable, at least not in a harmful way. Did I overlook something, or is this ticket just a request to silence compiler warnings?


2007-Apr-13 00:08:36 by anonymous:
vdbe.c with n, n64, payloadSize and payloadSize64
sqlite3BtreeKeySize,sqlite3BtreeLast return are not checked. You can not be sure the pointer passed as second argument will be init depending on the return of restoreOrClearCursorPosition (btree.c).

page.c with ro
Compiled with -DNDEBUG, the return of sqlite3OsOpenReadWrite is not checked before making a move with 'ro'.

For sContext.zAuthContext in delete.c/update.c, you're the one.

gcc (compiler in general) warnings are quite usefull, i don't think it's a good idea to ignore them and accumulate danger. Perhaps one day, one line in a subroutine will modify some tricky behavior and (re)raise a previous checked warning, making it completely normal and 'under control'.

 
2303 active 2007 Apr anonymous   Pending     1 1 Encrypted databases: No page cache due to problem in pagerSharedLock edit
With codec encryption enabled, pagerSharedLock always invalidates the page cache, even if no changes have occured since the cache was last valid and it would be safe to retain the cached pages. This in fact disables the newly improved page cache for encrypted databases and slows down performance.

The problem occurs because pagerSharedLock reads the change counter directly from the database file without codec decryption. Since the codec always encrypts full pages, the 4 bytes at offset 24 are read as encrypted data and do not match Pager->iChangeCount.

To solve, codecs would be required to store the 4 bytes at offset 24 of page 1 unencrypted. This would, however, render those 4 bytes vulnerable to attacks. It would therefore be more secure if pagerSharedLock could decrypt page one prior to extracting the change counter.

Check-in [3844] does not fix the problem to reset the cache if the codec is changed but the database file is not. The following procedure for opening an encrypted database no longer works with the improved page cache:

  • Open an encrypted database. Do not set a key yet as we (pretend to) believe that the database is not encrypted.

  • Access the DB for reading. This returns SQLITE_NOTADB, so we conclude that the DB is encrypted.

  • Attach the proper codec using sqlite3CodecAttach.

  • Access the DB again. Problem: This still returns SQLITE_NOTADB because the old page cache is still in use and is not reloaded.

The codec change is not detected because the pager checks the unencrypted DB file instead of the decrypted page. The file of course did not change, but the decrypted page did because of the new codec. The cache should therefore be cleared.

A workaround would be possible if sqlite3CodecAttach could reset the page cache. Unfortunately, the method to do so (pager_reset) is static to pager.c. It seems that there once was an external function sqlite3PagerReset (it is still defined in pager.h), but its implementation is unfortunately no longer available.

Could this be fixed in a way that pagerSharedLock checks the decrypted page 1 to see if the database has been modified or, alternatively, by reverting the static pager_reset back to the external sqlite3PagerReset?

 
2320 active 2007 Apr anonymous   Pending   drh 1 1 sqlite3_open(sFN_with_umlaut) edit
Do it in a standard MS Visual Studio Project:

0. CString sFnWithUmlaut = "c:\\long path\\path with umlauts äÄ\\db";
1. call sqlite3_open(sFnWithUmlaut);
2. db cannot be opened, because the transformation functions utf8ToUnicode/unicodeToUtf8 work incorrect

Is there a way to correct this error on win32?
Is there a workaround?

For a solution thanks in advance...
2007-Apr-25 20:21:24 by anonymous:
The functions work correctly, but you are not using them in the correct way. The parameter to sqlite3_open function should be UTF8 string, but you are passing one that is specific to your code page.
 
2322 active 2007 Apr anonymous   Pending     1 1 Windows error: datetime('2000-10-29 06:00:00','localtime') edit
NY time zone.

Windows (from http://sqlite.org/sqlite-3_3_17.zip)

  SELECT coalesce(datetime('2000-10-29 06:00:00','localtime'),'NULL');
  2000-10-29 02:00:00

Linux (from latest CVS, same TZ)

  SELECT coalesce(datetime('2000-10-29 06:00:00','localtime'),'NULL');
  2000-10-29 01:00:00

make test errors on Windows only:

  date-6.2...
  Expected: [{2000-10-29 01:00:00}]
       Got: [{2000-10-29 02:00:00}]

  date-6.3...
  Expected: [{2000-04-02 01:59:00}]
       Got: [{2000-04-02 02:59:00}]

  date-6.6...
  Expected: [{2000-10-29 07:00:00}]
       Got: [{2000-10-29 06:00:00}]

  date-6.7...
  Expected: [{2000-04-02 06:59:00}]
       Got: [{2000-04-02 05:59:00}]
2007-Apr-26 23:09:12 by anonymous:
Confirmed Windows bug on Windows 2000 in NY time zone with Y2K7DST OS patch.


2007-Apr-27 22:03:25 by drh:
Do I correctly understand the previous remark to say that this is confirmed to be a bug in Windows, not a bug in SQLite? It is identical code in SQLite for both operating systems, so I would certainly suspect that the problem is in windows and not in SQLite. But it would be nice to have confirmation of this before closing the ticket.


2007-Apr-28 03:19:06 by anonymous:
I meant to write "This erroneous SQLite datetime() output can also be seen on my Windows 2000 machine."

Does it work correctly for you under Windows XP or 2000 with the DST patch?

OS bug or not, it would be strange to not have the datetime() function correctly on a primary platform. If that were the case, it would be better to #ifdef it out of the Windows compile altogether.


2007-Apr-28 03:59:27 by anonymous:
This is a bigger mess than I thought.

http://groups.google.com/group/perl.perl5.porters/msg/e632557614474014?hl=en&

Key phrase:

"This API only provides the transition times according to the current DST rules. There is no database of historical transition times. That means that localtime() applied to previous years will use the new transition times even for old timestamps."

Please don't close this bug. Perhaps some industrious Windows programmer will have a correct solution for it one day. But in the meantime, Windows SQLite users should be aware of it.

 
2328 active 2007 Apr anonymous   Pending     1 1 Makefile sqlite3.c target breakage for C++ edit
This is generated by "make sqlite3.c":

  #if 0
  extern "C" {
  #endif
2007-Apr-29 06:02:20 by anonymous:
If the entire sqlite3.c almalgomation is wrapped with:

  #ifdef __cplusplus
  extern "C" {
  #endif

  ...

  #ifdef __cplusplus
  }
  #endif

then it could be compiled with a C++ compiler.

Please close this ticket if you did not intend to have this capability.

 
2340 active 2007 May anonymous   Pending     3 3 "./configure && make testfixture" link problem edit
Until the (extremely welcome) sqlite3 blob I/O functionality was checked in, it was possible to run this command on a clean source tree:

  ./configure && make testfixture

But it now produces these unresolved externals:

  ./src/tclsqlite.c:297: warning: excess elements in struct initializer
  ./src/tclsqlite.c:297: warning: (near initialization for `IncrblobChannelType')
  /tmp/ccmyzR97.o: In function `test_blob_read':
  /sqlite/cvs/sqlite/./src/test1.c:1545: undefined reference to `_sqlite3_blob_read'
  /tmp/ccmyzR97.o: In function `test_blob_write':
  /sqlite/cvs/sqlite/./src/test1.c:1592: undefined reference to `_sqlite3_blob_write'
  /tmp/ccb5dcDK.o: In function `incrblobClose':
  /sqlite/cvs/sqlite/./src/tclsqlite.c:156: undefined reference to `_sqlite3_blob_close'
  /tmp/ccb5dcDK.o: In function `incrblobInput':
  /sqlite/cvs/sqlite/./src/tclsqlite.c:194: undefined reference to `_sqlite3_blob_bytes'
  /sqlite/cvs/sqlite/./src/tclsqlite.c:202: undefined reference to `_sqlite3_blob_read'
  /tmp/ccb5dcDK.o: In function `incrblobOutput':
  /sqlite/cvs/sqlite/./src/tclsqlite.c:226: undefined reference to `_sqlite3_blob_bytes'
  /sqlite/cvs/sqlite/./src/tclsqlite.c:235: undefined reference to `_sqlite3_blob_write'
  /tmp/ccb5dcDK.o: In function `incrblobSeek':
  /sqlite/cvs/sqlite/./src/tclsqlite.c:264: undefined reference to `_sqlite3_blob_bytes'
  /tmp/ccb5dcDK.o: In function `DbObjCmd':
  /sqlite/cvs/sqlite/./src/tclsqlite.c:322: undefined reference to `_sqlite3_blob_open'
  collect2: ld returned 1 exit status
  make: *** [testfixture.exe] Error 1

The 2 warnings related to src/tclsqlite.c:297 also seem to indicate a problem.

I'm using Tcl 8.4.

2007-May-06 21:16:01 by anonymous:
Workaround - hack the generated Makefile to use:

  # Compiler options needed for programs that use the TCL library.
  #
  TCC += -I/usr/local/include -DSQLITE_OMIT_INCRBLOB=1
 
2350 active 2007 May anonymous   Pending     1 1 Temp files not deleted on WinCE edit
When using SQLite 3.3.17 on WinCE device temp files end up being accumulated in Temp directory, eventually filling up device's open file quota (999?) and making further operation that requires creation of files impossible.

I have noticed that after a while all queries start failing on the device although if I copied the database to PC and tried the same query I had no problems with it. Started tracing through the code and noticed that sqlite3WinOpenExclusive() is consistently failing to create a file (CreateFileW() would always fail). It turned out that Temp directory was full of 0 length temp files -- after deleting all of them device started working again for a while.

Then started looking for the cause -- winceDestroyLock() never gets to execute DeleteFileW() and delete the file even though pFile->zDeleteOnClose is set because check on the top of the winceDestroyLock() function (if (pFile->hMutex)) always fails thanks to pFile->hMutex being 0.

sqlite3WinOpenReadWrite() calls winceCreateLock() on temporary file (zFilename) but sqlite3WinOpenExclusive() doesn't hence winceDestroyLock() cannot delete temporary file.

sqlite3PagerClose() contains this piece of commented out code:

  /* Temp files are automatically deleted by the OS
  ** if( pPager->tempFile ){
  **   sqlite3OsDelete(pPager->zFilename);
  ** }
  */

Comment is not true for WinCE as WinCE doesn't support FILE_ATTRIBUTE_TEMPORARY and FILE_FLAG_DELETE_ON_CLOSE flags. Perhaps it should be surrounded with #if OS_WINCE/#endif instead of being commented out?


2007-May-14 00:09:20 by anonymous:
Tried re-enabling that piece of code in sqlite3PagerClose() and it doesn't help.

Looks like even file handles for temporary files never get closed on WinCE so not only number of temporary file in Temp directory just keeps growing but also filed descriptor for each of them remains open, draining system resources (consistent with observed behavior of rapid and nasty system performance deterioration).

It appears that sqlite3VdbeFreeCursor() always exits immediately because pCx is always NULL and thus sqlite3BtreeClose() never gets called:

	void sqlite3VdbeFreeCursor(Vdbe *p, Cursor *pCx){
	  if( pCx==0 ){
	    return;
	  }
	  if( pCx->pCursor ){
	    sqlite3BtreeCloseCursor(pCx->pCursor);
	  }
	  if( pCx->pBt ){
	    sqlite3BtreeClose(pCx->pBt);
	  }

sqlite3BtreeClose() then calls sqlite3PagerClose() which probably explains why re-enabling that code doesn't help -- sqlite3PagerClose() itself never gets called.


2007-May-14 05:16:59 by anonymous:
Did some comparison between operation of the SQLite on WinXP and WinCE.

Comment in previous remark about sqlite3VdbeFreeCursor() always exiting immediately because pCx is always NULL is not correct. pCX is NULL most of the time but this is true for both WinXP and WinCE. WinXP has no problems getting rid of its temp files thanks to its support for FILE_ATTRIBUTE_TEMPORARY and FILE_FLAG_DELETE_ON_CLOSE flags. WinCE also does get its temp file descriptors closed (previous statement incorrect on that account too), it's just that its temp files do not get cleaned up automatically. Initial statement that this was caused because of winceDestroyLock() actually is correct.

Temp files are opened with sqlite3WinOpenExclusive() which DOESN'T use mutex (instead of with sqlite3WinOpenReadWrite() which DOES use mutex) so part of winClose() that was supposed to take care of deleting files after handle is closed (i.e. calling winceDestroyLock()) doesn't work:

	static int winClose(OsFile **pId){
	  winFile *pFile;
	  int rc = 1;
	  if( pId && (pFile = (winFile*)*pId)!=0 ){
	    int rc, cnt = 0;
	    OSTRACE2("CLOSE %d\n", pFile->h);
	    do{
	      rc = CloseHandle(pFile->h);
	    }while( rc==0 && cnt++ < MX_CLOSE_ATTEMPT && (Sleep(100), 1) );
	#if OS_WINCE
	    winceDestroyLock(pFile);
	#endif

OS_WINCE part should go something like this (although probably not exactly like this as this feels like an ugly hack):

	#if OS_WINCE
	    if (pFile->hMutex){
	      winceDestroyLock(pFile);
	    }else{
	      if( pFile->zDeleteOnClose ){
	        DeleteFileW(pFile->zDeleteOnClose);
	        sqliteFree(pFile->zDeleteOnClose);
	        pFile->zDeleteOnClose = 0;
	      }
	    }
	#endif


2007-May-14 05:20:33 by anonymous:
Tried the code from the previous remark and it does clean up Temp directory!

I also tried uncommenting that piece of code in sqlite3PagerClose() mentioned in the first remark and it looks like that temp files it tries to delete are never there. I'm guessing that that piece of code can remain commented out. Will keep an eye on it -- in case temp files still eventually start accumulating again even after the fix from previous remark is applied I'll reconsider.

 
2352 active 2007 May anonymous   Pending     5 3 timeout just 500 msec to soon edit
After upgrading from 3.3.12 to 3.3.17, the setting of a timeout behaves differently. It occurs exactly 500 msecs sooner. Of course assuming the database is locked and the timeout is set to a value larger than 500 msecs.
 
2363 active 2007 May anonymous   Pending     3 3 Couldn't build 3.3.17 on Cygwin/Vista edit
I didn't see anything quite resembling this on a cursory scan thru the buglist; pardon my goof if it's a duplicate.

When I tried to build on Cygwin on Vista, things got to the install phase, and then the Makefile tried to execute

  cc sqlite3.c -o sqlite3

which didn't appear to be a defined rule in the Makefile. This then failed because of a lack of a main program, and the install failed.

After a quick scan through the Makefile, I decided to change the target for the install rule to

    install:	sqlite3$(TEXE) libsqlite3.la sqlite3.h ${HAVE_TCL:1=tcl_install}

which seemed to remedy the problem.

Suggested fix: modify Makefile as shown above.

2007-May-23 14:29:20 by anonymous:
Another vote for this
 
2371 active 2007 May anonymous   Pending     2 2 sqlite3_errcode() and sqlite3_errmsg() return unexpected results edit
The manual says that sqlite3_step() directly returns explicit error code if the query has been prepared with the new API sqlite3_prepare_v2().

Subsequently, the functions sqlite3_errcode() and sqlite3_errmsg() should return the correct appropriate error values as well, which they don't - instead something not matching the error is returned. One has to call sqlite3_reset() to get the correct values which should be unnecessary.

See the example output below :

  sqlite3_step: result rc = 19, errcode = 1, errmsg = SQL logic error or missing database
  sqlite3_reset: result rc = 19, errcode = 19, errmsg = PRIMARY KEY must be unique

Code for this example:

  #include <stdio.h>
  #include <stdlib.h>
  #include <unistd.h>
  #include <string.h>
  #include <sqlite3.h>

  int
  main(int argc, char **argv)
  {
      sqlite3 *db;
      int rc, errcode;
      sqlite3_stmt *stmt;
      const char *pztail;
      char *query;
      const char *errmsg;

      rc = sqlite3_open("test.db", &db);
      rc = sqlite3_exec(db, "create table t (x integer not null primary key)", NULL, NULL, NULL);
      rc = sqlite3_exec(db, "insert into t values(1)", NULL, NULL, NULL);
      query = "insert into t values(?)";
      rc = sqlite3_prepare_v2(db, query, strlen(query), &stmt, &pztail);
      if( rc!=SQLITE_OK ){
          fprintf(stderr, "SQL error: %d\n", rc);
          exit(1);
      }
      rc = sqlite3_bind_int(stmt, 1, 1);
      if( rc!=SQLITE_OK ){
          fprintf(stderr, "SQL error: %d\n", rc);
          exit(1);
      }
      rc = sqlite3_step(stmt);
      errcode = sqlite3_errcode(db);
      errmsg = sqlite3_errmsg(db);
      printf("sqlite3_step: result rc = %d, errcode = %d, errmsg = %s\n",
          rc, errcode, errmsg);
      rc = sqlite3_reset(stmt);
      errcode = sqlite3_errcode(db);
      errmsg = sqlite3_errmsg(db);
      printf("sqlite3_reset: result rc = %d, errcode = %d, errmsg = %s\n",
          rc, errcode, errmsg);
  }
 
2376 active 2007 May anonymous   Pending     3 3 -batch doesn't work soon enough edit
on Microsoft XP, given any.db and a batch file bug.bat containing

sqlite3 -batch -init bug.rc any.db "select 1;"

and bug.rc containing

.mode column

Running bug.bat gives the message "Loading resources from bug.rc"

I put in "-batch" so that the "Loading..." message would be suppressed, but it printed anyway.

 
2378 active 2007 May anonymous   Pending     2 2 Quoted fields come back corrupted, using GROUP BY edit
Description:

When executing a query, where field names are quoted, and using GROUP BY, the field names are returned with quotes around.

SQLite version:

  SQLite support => enabled
  PECL Module version => 2.0-dev $Id: sqlite.c,v 1.166.2.13.2.7 2007/03/06 02:17:13 stas Exp $
  SQLite Library => 2.8.17
  SQLite Encoding => iso8859

Reproduce code

  <?php
  $db = sqlite_open(":memory:");
  sqlite_query($db, '
    CREATE TABLE test (
      id INTEGER PRIMARY KEY
    )
  ');
  sqlite_query($db, "INSERT INTO test (id) VALUES (1)");
  $result = sqlite_query($db, 'SELECT "id" FROM "test" GROUP BY "id"');
  var_dump(sqlite_fetch_array($result, SQLITE_ASSOC));

Expected result

  array(1) {
    ["id"]=>
    string(1) "1"
  }

Actual result

  array(1) {
    [""id""]=>
    string(1) "1"
  }
2007-May-21 18:14:27 by anonymous:
Corrupted is probably not the right term, but the fields are returned with quotes around them in sqlite3 as well when group by is used on a quoted column:

  SQLite version 3.3.17
  Enter ".help" for instructions
  sqlite> .header on
  sqlite> create table t1(a);
  sqlite> insert into t1 values(1);
  sqlite> select "a" from t1;
  a
  1
  sqlite> select "a" from t1 order by 1;
  a
  1
  sqlite> select "a" from t1 group by 1;
  "a"
  1
  sqlite> select "a" from t1 group by 1 order by 1;
  "a"
  1
  sqlite> select "a" from t1 group by a;
  "a"
  1
 
2383 active 2007 May anonymous   Pending     3 3 Inconsistent conversion of BLOB revisited edit
BLOBs of function results are converted as UTF-8 even if the encoding is UTF-16.

  $ ./sqlite3
  SQLite version 3.3.17
  Enter ".help" for instructions
  sqlite> pragma encoding = 'UTF-16BE';
  sqlite> select quote(cast(cast('ab' as blob) as text));             -- OK
  'ab'
  sqlite> select quote(cast(a as text)) from (select X'00610062' a);  -- OK
  'ab'
  sqlite> select quote(cast(X'00610062' as text));                    -- fixed by [3975]
  'ab'
  sqlite> select quote(cast(substr(X'110061006222', 2, 4) as text));  -- NG
  ''
  sqlite> select quote(cast(substr(X'11616222', 2, 2) as text));
  'ab'
  sqlite> select quote(upper(substr(X'11616222', 2, 2)));
  'AB'
 
2388 active 2007 May anonymous   Pending     1 3 ORDER BY fails on compound select edit
The query below was known to be working as late as 3.3.14, but in 3.3.17 it errors with "ORDER BY term number 1 does not match any result column".

Debugging with GDB shows that sqlite3NameFromToken(&pE->token) doesn't return anything, and pE->token looks pretty empty, so possibly this could be a parsing issue?

The query is:

  SELECT 	docs.guid, docs.info, conversations.subject, conversations.date
  FROM	conversations
  JOIN 	docs	ON 	docs.guid = conversations.guid
  WHERE 	conversations.guid IN (
  	SELECT	docs.guid
  	FROM 	conversations
  	JOIN 	docs 	ON 	docs.guid = conversations.guid
  	WHERE 	conversations.date >= (SELECT date FROM conversations WHERE guid = ?6)
  		AND 	docs.collection = ?1
  		AND 	((?2 == 0) OR (conversations.sources & ?2) !=   0)
  		AND 	((conversations.sources & ?3) == 0)
  		AND 	test_info_flags(docs.info, ?7, ?8)
  	LIMIT 	(?5 + 1)
  	)
  UNION
  SELECT 	docs.guid, docs.info, conversations.subject,   conversations.date
  FROM 	conversations
  JOIN 	docs   	ON 	docs.guid = conversations.guid
  WHERE 	conversations.guid IN (
  	SELECT 	docs.guid
  	FROM 	conversations
  	JOIN 	docs 	ON docs.guid = conversations.guid
  	WHERE 	conversations.date < (SELECT date FROM conversations WHERE guid = ?6)
  		AND 	docs.collection = ?1
  		AND 	((?2 == 0) OR (conversations.sources & ?2) != 0)
  		AND 	((conversations.sources & ?3) == 0)
  		AND 	test_info_flags(docs.info, ?7, ?8)
  	LIMIT ?4
  	)
  ORDER BY conversations.date DESC, docs.guid DESC;
2007-May-27 16:27:23 by anonymous:
2 workarounds:

  ORDER BY 4 DESC, 1 DESC;

or

  SELECT docs.guid             as "guid",
         docs.info             as "info",
         conversations.subject as "subject",
         conversations.date    as "date"
  ...
  ORDER BY date DESC, guid DESC;

Smaller test case:

  create table x1(a);

  select x1.a from x1 union select x1.a from x1 order by x1.a;
  -- SQL error: ORDER BY term number 1 does not match any result column
  -- error in latest 3.3.17+ CVS, but works in SQLite 3.2.2, 3.3.13

  select x1.a from x1 union select x1.a from x1 order by a;
  -- SQL error: ORDER BY term number 1 does not match any result column
  -- has never worked in previous versions

  select x1.a from x1 union select x1.a from x1 order by 1;
  -- ok in all versions

  select x1.a as a from x1 union select x1.a from x1 order by a;
  -- ok in all versions


2007-May-27 16:46:19 by anonymous:
Does this query work on other popular databases?

  create table x1(a integer);
  select x1.a from x1 union select x1.a from x1 order by x1.a;

I'd expect "order by a" to work on most databases, but not "order by x1.a" because the table name would have been lost by the UNION.

 
2396 active 2007 Jun anonymous   Pending     1 3 -column output truncates characters from varchar field. edit
  CREATE TABLE unix (unix_id integer primary key, ip_address varchar(16) unique, status enum, updated timestamp);

  INSERT INTO unix VALUES (NULL,'172.26.242.92','in use',timestamp('now'));
  INSERT INTO unix VALUES (NULL,'172.26.242.129','in use',timestamp('now'));
  INSERT INTO unix VALUES (NULL,'172.26.242.131','in use',timestamp('now'));
  INSERT INTO unix VALUES (NULL,'172.26.242.132','in use',timestamp('now'));
  INSERT INTO unix VALUES (NULL,'172.26.242.136','in use',timestamp('now'));
  INSERT INTO unix VALUES (NULL,'172.26.242.213','in use',timestamp('now'));
  INSERT INTO unix VALUES (NULL,'10.193.33.7','in use',timestamp('now'));
  INSERT INTO unix VALUES (NULL,'10.193.32.239','in use',timestamp('now'));

Now: sqlite3 -column test.db "SELECT ip_address, status, updated FROM unix";

  172.26.242.92  in use      2007-05-31 22:11:39
  172.26.242.12  in use      2007-05-31 22:11:47
  172.26.242.13  in use      2007-05-31 22:11:51
  172.26.242.13  in use      2007-05-31 22:11:59
  172.26.242.13  in use      2007-05-31 22:12:04
  172.26.242.21  in use      2007-05-31 22:12:11
  10.193.33.7    in use      2007-05-31 23:27:09
  10.193.32.239  in use      2007-05-31 23:27:17

The output of the varchar column is always truncated regardless of which column it is placed in. Even if it's the only column.

  sqlite3 -column test.db "SELECT ip_address FROM unix";

  172.26.242.92
  172.26.242.12
  172.26.242.13
  172.26.242.13
  172.26.242.13
  172.26.242.21
  10.193.33.7
  10.193.32.239
2007-Jun-01 18:54:42 by anonymous:
Smaller test case:

  sqlite> CREATE TABLE t9(a);
  sqlite> INSERT INTO "t9" VALUES('a234567890');
  sqlite> INSERT INTO "t9" VALUES('a23456789012345');
  sqlite> select * from t9;
  a234567890
  a23456789012345
  sqlite> .mode column
  sqlite> select * from t9;
  a234567890
  a234567890
  sqlite> select * from t9 order by a desc;
  a23456789012345
  a234567890

.mode column seems to be remembering the first row's column lengths and using them as the maximum column lengths for subsequent rows.


2007-Jun-01 19:11:11 by anonymous:
.mode column could only work correctly if the outputting of the rows took place only after the maximum column width of each column for each row was determined in advance. This would require buffering alls rows in memory before any data rows would be output.

I recommend to remove MODE_Column functionality from shell.c completely, as it never worked correctly.


2007-Jun-05 14:54:30 by anonymous:
Column mode works fine for most applications. It bases it column widths on the width of the title or a minimum width of 10. If this is not suitable, you can set the column widths manually using the .width command in the shell. If you set your IP address column width to 16 or more everything will be fine.
 
2398 active 2007 Jun anonymous   Pending     2 3 systemcalls should be restarted when they return EINTR edit
Introduction: I'm a developer for Alcatel-Lucent, where I program softswitches. I'm a bit of a code-correctness nazi, because I've programmed several systems that have to be very reliable (5 figures), or I've corrected them if they didn't. I was browsing thru the source code of SQLite, to become familiar with it, and to see how reliable it was. Not that we currently use it in a product, but you may never known.

The very first thing I noticed was that none of the systemcalls in os_unix.c was checked for EINTR. From experience, I know that a systemcall can be interrupted while inside the kernel, and return EINTR in errno. The correct reaction is to repeat the system call, just as if nothing happened. This is not an error, just a temporary situation which will be fixed the next time you call it again. One reason is that you call was interrupted by a higher priority interrupt (incoming IO for instance), or to avoid a deadlock in the kernel (returning your syscall will release any locks you might have). It's pretty rare in most usage, but I've seen it lots of times when testing a server under load.

Example of the solution for seekAndWrite (this only shows write, but pwrite is similar) :

   got = write(id->h, pBuf, cnt);

should be :

   do {
      got = write(id->h, pBuf, cnt);
   } while ( (got < 0) && (errno == EINTR) );

This has to be done for every systemcall that mentions it on its manpage (but please note, not every OS is the same). This includes many calls where everyone assumes that the call is safe to use, for instance close(). Almost nobody seems to realize that a close can fail, and often there's not even a check for the return code. But I can assure you, it can fail.

2007-Jun-05 13:39:08 by anonymous:
I prefer the current SQLite behavior NOT to retry in the event of EINTR. I've seen too many UNIX OSes with bugs related to EINTR for system calls within threads over the years, including the most popular ones. Most of these OS bugs have since been fixed, but it's still not worth the trouble in case you come across an unpatched platform. Getting an infinite loop of EINTR errnos in a tight loop due to an OS bug is no fun in a production application.

Technically, POSIX requires that you only read errno if -1 is returned by the system call. QNX is picky about this sort of thing.


2007-Jun-05 17:02:55 by anonymous:
Read my introduction again : I help make systems reliable in my daytime job. Ignoring a read- or write-error is not how you achieve 99.999% availability, especially not when it's easily correctable (a disk-full message or a I/O error due to a corrupt harddisk is another matter ofcourse).

>Technically, POSIX requires that you only read errno if -1 is returned by the system call. QNX is picky about this sort of thing.

well yes, that's what I mean.


2007-Jun-05 17:20:07 by anonymous:
All of SQLite's operations can also be retried if they fail.

SQLite has an extensive test suite for disk failures, malloc failures and many other types of failures that you've likely never considered. I tend to trust the judgement of the authors of SQLite more than an anonymous guy puffing his chest out in this ticket tracking system.


2007-Jun-05 22:24:09 by anonymous:
hi. instead of blaming in this ticket, the ticket author could place a patch like:
got = write(id->h, pBuf, cnt);
to do things like:
#ifdef SQLITE3_RETRY_EINTR_SYSCALLS
   do {
#endif // SQLITE3_RETRY_EINTR_SYSCALLS
      got = write(id->h, pBuf, cnt);
#ifdef SQLITE3_RETRY_EINTR_SYSCALLS
   } while ( (got < 0) && (errno == EINTR) );
#endif // SQLITE3_RETRY_EINTR_SYSCALLS
by leaving SQLITE3_RETRY_EINTR_SYSCALLS undefined, this should not affect the current code behaviour and should address the issues ticket author says and he can compile with those 'paranoic' checks. []'s


2007-Jun-05 22:25:06 by anonymous:
I did pass my name (visible for the developers), but I'm not logged in. That why it's shown as anonymous (just as you are).

Note that my co-workers have already decided it would be seen up as a negative point for this DBMS. At least for us, because it's a requirement for us to survive these errors. So if we ever are going to use SQLite, we're definitely going to use those changes.

Hey, I'm a code-review nazi, don't blame me. It's just my job :-)


2007-Jun-06 00:58:16 by anonymous:
I think that the retrying would destroy the function expected of sqlite3_interrupt.

Is there a problem in the following methods?

  • Re-execute the SQLite API.
  • Mask the signals while calling the API of SQLite.
  • Mask the signals in the threads that call the API of SQLite.


2007-Jun-06 01:04:47 by anonymous:
As long as #ifdef SQLITE3_RETRY_EINTR_SYSCALLS is not enabled by default, knock yourselves out.


2007-Jun-06 06:38:51 by anonymous:
sqlite3_interrupt() knows nothing about UNIX signals. It just sets a flag that sqlite's VDBE checks from time to time to see if it should gracefully abort an SQL operation.

shell.c happens to install an SIGINT signal handler that happens to call sqlite3_interrupt(), but that's unrelated. sqlite3_interrupt() could be called via any other means.


2007-Jun-06 10:47:27 by anonymous:
See also the SA_RESTART flag for sigaction(), which automatically will restart the systemcall instead of returning EINTR.

Unfortunately not available everywhere, and not very consistent either : see Linus reply at http://lkml.org/lkml/2005/7/23/119 . Applications should still be able to handle EINTR returns gracefully. They're not errors per se (that would be EIO or similar), you can safely restart the system call. I haven't seen endless loops yet, but it's true that on some OS you might want this solution. But you still have to deal with the error (which might cause a rollback in SQLite).


2007-Jun-06 14:51:14 by anonymous:
I would be in favor of optionally compiling SQLite with a #define SQLITE_USE_SA_RESTART or #define SQLITE_RETRY_ON_EINTR, where they are mutually exclusive and disabled by default. SQLITE_RETRY_ON_EINTR should have a maximum retry limit on buggy OSes, though.
 
2408 active 2007 Jun anonymous   Pending     2 1 BLOBs not output correctly in .mode insert (shell.c - isnumber) edit
The method
static int isNumber(const char *z, int *realnum)

from shell.c is wrong.
Steps to reproduce:
1. Get the file www.smatei.3x.ro/project1.zip
2. extract project1.db from the zip file
3. execute
sqlite3 project1.db
sqlite>.mode insert
sqlite> select ID, Name, Color, Active, Priority, PrioritySource, IndexOrder, Language, 0 from Keywords;
INSERT INTO Keywords VALUES(42,#####,0,1,0,0,42,'',0);
INSERT INTO Keywords VALUES(41,'######',0,1,0,0,43,'',0);

If you look at the 42 item, the string next to 42 is not enclosed by the string delimiter '.
This is because the method isnumber returns that the string is number. It is not a number, it is a string in Hebrew (I inserted ### instead of the real strings).
A fix for this might be to set the first parameter unsigned char
static int isNumber(unsigned const char *z, int *realnum)
but I am not sure, because I haven't written C code for a long time.
If you have any more questions, please ask.
Best Regards,
Stefan

2007-Jun-11 14:46:03 by drh:
Please show me what you get from the following query:

    SELECT ID, quote(cast(Name AS BLOB)) FROM Keywords
     WHERE ID IN (41,42);

I need this information in order to track down the problem.


2007-Jun-11 14:50:33 by anonymous:
The output is

41|X'D791D7A8D799D799D7A7D793D790D7A0D7A1'
42|X'D7A1D798D7A4D7A1'


2007-Jun-11 16:29:46 by drh:
When I do this:

  CREATE TABLE t1(x);
  INSERT INTO t1 VALUES(cast(X'D791D7A8D799D799D7A7D793D790D7A0D7A1' as text));
  INSERT INTO t1 VALUES(cast(X'D7A1D798D7A4D7A1' AS text));
  .mode insert xyz
  SELECT * FROM t1;

I see the Hebrew characters, properly quoted. Can you suggest another way to reproduce the problem?


2007-Jun-11 16:34:55 by anonymous:

  SQLite version 3.3.17
  Enter ".help" for instructions
  sqlite> create table t1(b blob);
  sqlite> insert into t1 values(X'0102030405060708090a0dABCD');
  sqlite> .dump
  BEGIN TRANSACTION;
  CREATE TABLE t1(b blob);
  INSERT INTO "t1" VALUES(X'0102030405060708090A0DABCD');
  COMMIT;
  sqlite> .mode insert
  sqlite> select * from t1;
  INSERT INTO table VALUES('
  &#171;&#205;');
  sqlite>


2007-Jun-11 20:07:09 by anonymous:
If I recall correctly, sqlite3.exe assumes all I/O to and from the console is UTF-8. Windows consoles are either MBCS or UNICODE depending on the build settings. Sqlite3.exe was not compiled as UNICODE so anything inserted into a sqlite3 database will be inserted as MBCS. So if you insert into a database using a 3rd party application that does proper UTF-8, and then query it from the command-line, it will look wrong. Likewise, anything you insert from the command-line will look wrong when queried from an application that uses UTF-8.


2007-Jun-11 20:42:18 by anonymous:
.mode insert treats BLOBs as strings, which is a problem since it stops outputting at the first nil character.

  CREATE TABLE t1(a blob);
  INSERT INTO t1 VALUES(X'00000008090A0D0D0A00');
  .mode insert whatever
  select * from t1;


2007-Jun-12 07:39:12 by anonymous:
Hi,

My problem is not the way the string is displayed in the console. The application that reads the database reads it correctly. The problem is when I try to dump a certain table in a file using the following sequence:

.output "file.txt"
.mode insert whatever
select f1, f2, f3 from t1;

The result is a file that some Hebrew strings are not enclosed in ' (apostrophe). After debugging the code, I saw that the method isnumber does not return 0 for that string. It considers all the characters digits.

This problem occurs only on Windows. We tried this on Mac and Linux, and it worked fine.

Best Regards,
Stefan

 
2409 active 2007 Jun anonymous   New   drh 1 1 Database malformed with SQLite3.3.17 on WindowsXP edit
I encountered a problem with SQLite3.3.17 on Windows XP. Under certain situation, database file got seriously corrupted.

SQLite version: 3.3.17 Windows Binary Platform:Windows XP SP2(Japanese) Code wrtten in: Visual C++ 6.0

Here are the procedures to reproduce the problem:

1) Run a program SQLiteCrush.exe. This program updates 'test.db' repeatedly. Insert data to work table, copy them into items table, then delete records from work.

2) Open 'test.db' from sqlite3.exe.

3) Do '.read check.sql' repeatedly. check.sql is made from many lines of 'pragma integrity_check;'.

4) Keep doing 1 -3 for several minuites, and 'pragma integrity_check' starts to report something like "rowid 91667 missing from index sqlite_autoindex_link_1".

So far, I didn't see the database corrupted with SQLite 3.3.7. Also, without 3), the database was not corrupted. Instead of 'pragma integrity_check', issueing many select statements also make it currupted.

2007-Jun-12 02:51:26 by anonymous:
I did more tests to make it clear from which version it happens.

With 3.3.8, I couldn't reproduce the problem. With 3.3.9, I can reproduce the problem.

It seems there's some change between these two that causes the problem...


2007-Jun-12 03:06:47 by anonymous:
Can you try it against the latest version in CVS, or 3.3.17? A lot of code has changed since 3.3.9.


2007-Jun-12 05:00:50 by anonymous:
I Already tried 3.3.17. It happenes.

Where can I get the latest CVS precompiled binary for windows platform ?

tamagawa


2007-Jun-13 14:25:47 by drh:
The problem was introduced in SQLite version 3.3.0, specifically check-in [2848] . There are related problems that go back even further in time, we believe. The root of the problem is a logic error in my design of the pager layer. We are working on a fix now, as well as a set of test cases that will ensure that similar errors do not reappear in the future. I will also soon publish instructions on how to work around the problem in effected versions of SQLite.

The problem can be easily reproduced by running the script below using the "testfixture" program that we use for testing SQLite.

# Prepare the database:
#
file delete -force test.db test.db-journal
sqlite3 db test.db
db eval {
  CREATE TABLE t1 (
    x TEXT UNIQUE NOT NULL,
    y BLOB
  );
}

# Open a second connection to the database that will be
# used to lock the database file.
#
set DB2 [sqlite3 db2 test.db]
if {$DB2==""} {
  set DB2 [sqlite3_connection_pointer db2]
}

# A small cache will cause an early cache spill.
#
db eval {PRAGMA cache_size=10}

# Acquire read lock on the database file using the second connection.
#
set STMT [sqlite3_prepare $DB2 {SELECT rowid FROM sqlite_master} -1 TAIL]
sqlite3_step $STMT

# Insert a short record into the index (10 bytes) and a large record
# into the table (15K).  The index record goes in Ok, but during the
# insert into the table, SQLite attempts to upgrade to an EXCLUSIVE
# lock to do a cache flush. When this happens, the cache is left in
# an inconsistent state.
#
set zShort [string repeat 0123456789 1]
set zLong  [string repeat 0123456789 1500]
db eval {BEGIN}
set rc [catch { db eval {INSERT INTO t1 VALUES($zShort, $zLong)} } msg]
puts "rc=$rc msg=$msg"
sqlite3_finalize $STMT
db eval {COMMIT}
db close
sqlite3 db test.db
puts [db eval {PRAGMA integrity_check}]
 
2413 active 2007 Jun anonymous   Pending   drh 1 1 1 bug and 2 suggestions in lemon edit
Hello, ...
Sorry for my english :-) and if i post this with Severity/Priority error.
I found some not serious bug and have some suggetions.
=============================================================================
BUG FIX:
lemon.c for Win32. It not found lempar.c - backslash-bug.
function:
PRIVATE char *pathsearch(argv0,name,modemask);

PATCH:

---- CUT --------------------------------------------------------------------

--- C:/lemon.c Wed Jun 13 15:02:37 2007
+++ D:/Den/Lemon/lemon.c Wed Jun 13 16:25:22 2007
@@ -2911,7 +2911,11 @@
c = *cp;
*cp = 0;
path = (char *)malloc( strlen(argv0) + strlen(name) + 2 );
- if( path ) sprintf(path,"%s/%s",argv0,name);
+ #ifdef __WIN32__
+ if( path ) sprintf(path,"%s\\%s",argv0,name);
+ #else
+ if( path ) sprintf(path,"%s/%s",argv0,name);
+ #endif
*cp = c;
}else{
extern char *getenv();
@@ -2920,11 +2924,19 @@
path = (char *)malloc( strlen(pathlist)+strlen(name)+2 );
if( path!=0 ){
while( *pathlist ){
- cp = strchr(pathlist,':');
+ #ifdef __WIN32__
+ cp = strchr(pathlist,';');
+ #else
+ cp = strchr(pathlist,':');
+ #endif
if( cp==0 ) cp = &pathlist[strlen(pathlist)];
c = *cp;
*cp = 0;
- sprintf(path,"%s/%s",pathlist,name);
+ #ifdef __WIN32__
+ sprintf(path,"%s\\%s",pathlist,name);
+ #else
+ sprintf(path,"%s/%s",pathlist,name);
+ #endif
*cp = c;
if( c==0 ) pathlist = "";
else pathlist = &cp[1];

---- CUT --------------------------------------------------------------------

=============================================================================
SUGGESTION 1:

Why we allocate parser with mallocProc parameter of ParseAlloc function
and free with freeProc of ParseFree function?

We do this because we want what parser is user-allocatable
with USER-DEFINED-MEMORY-ALOCATION-WAY but not with "malloc"/"free" from stdlib... am i right?

If so... why we still allocate memory for parser stack with "realloc" function?

It's bad for solutions where is no stdlib.

My suggestion is

FIRST WAY:

To add to yyParser struct 3 variables like

void *mem_alloc_fn;
void *mem_realloc_fn;
void *mem_free_fn;

and add 3 directives like

%memory_alloc
%memory_realloc
%memory_free

and if it declared - use it for allocating/free/reallocating memory in parser.

and
- void ParseAlloc(void *(*mallocProc)(size_t));
will now as void *ParseAlloc();
- void ParseFree(void *pParser, void (*freeProc)(void
));
will now as void ParseFree(void *pParser);

OR SECOND WAY (very simple):

To add to yyParser struct 1 variable like
void *mem_realloc_fn;

- void *ParseAlloc(void *(*mallocProc)(size_t));
will now as void *ParseAlloc(void *(*mallocProc)(size_t), void *(*reallocProc)(void *, size_t));

store reallocProc in mem_realloc_fn in yyParser

and in yyGrowStack something like this:
... yyGrowStack (...)
{
....
if(pParser->mem_realloc_fn != NULL)
{
pNew = pParser->mem_realloc_fn(p->yystack, newSize*sizeof(pNew[0]));
}
else
{
pNew = realloc(p->yystack, newSize*sizeof(pNew[0]));
}
....
}

and use it for reallocating memory in parser.

In this ways - memory allocating in parser is under FULL user control.

=============================================================================
SUGGESTION 2:

I build lemon with VC 8.0 with option /Wp64 (Detect 64-Bit Portability Issues)
and have warnings. Type int, size_t, pointer and unsigned long have diferent size on x32 and x64 platforms.

Can you fix type difference, please?

Only you can choice better way for this - type conversion OR change type of 'warning' variables.

WARNINGS:

d:\den\lemon\lemon.c(1331) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data
d:\den\lemon\lemon.c(1337) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data
d:\den\lemon\lemon.c(1455) : warning C4113: 'int (__cdecl )()' differs in parameter lists from 'int (__cdecl *)(const void *,const void *)'
d:\den\lemon\lemon.c(1578) : warning C4311: 'type cast' : pointer truncation from 'char *' to 'unsigned long'
d:\den\lemon\lemon.c(1578) : warning C4312: 'type cast' : conversion from 'unsigned long' to 'char **' of greater size
d:\den\lemon\lemon.c(1581) : warning C4311: 'type cast' : pointer truncation from 'char *' to 'unsigned long'
d:\den\lemon\lemon.c(1581) : warning C4312: 'type cast' : conversion from 'unsigned long' to 'char **' of greater size
d:\den\lemon\lemon.c(1586) : warning C4311: 'type cast' : pointer truncation from 'char *' to 'unsigned long'
d:\den\lemon\lemon.c(1586) : warning C4312: 'type cast' : conversion from 'unsigned long' to 'char **' of greater size
d:\den\lemon\lemon.c(1588) : warning C4311: 'type cast' : pointer truncation from 'char *' to 'unsigned long'
d:\den\lemon\lemon.c(1588) : warning C4312: 'type cast' : conversion from 'unsigned long' to 'char **' of greater size
d:\den\lemon\lemon.c(1590) : warning C4311: 'type cast' : pointer truncation from 'char *' to 'unsigned long'
d:\den\lemon\lemon.c(1590) : warning C4312: 'type cast' : conversion from 'unsigned long' to 'char **' of greater size
d:\den\lemon\lemon.c(1592) : warning C4311: 'type cast' : pointer truncation from 'char *' to 'unsigned long'
d:\den\lemon\lemon.c(1592) : warning C4312: 'type cast' : conversion from 'unsigned long' to 'char **' of greater size
d:\den\lemon\lemon.c(1595) : warning C4311: 'type cast' : pointer truncation from 'char *' to 'unsigned long'
d:\den\lemon\lemon.c(1595) : warning C4312: 'type cast' : conversion from 'unsigned long' to 'char **' of greater size
d:\den\lemon\lemon.c(1596) : warning C4311: 'type cast' : pointer truncation from 'char *' to 'unsigned long'
d:\den\lemon\lemon.c(1596) : warning C4312: 'type cast' : conversion from 'unsigned long' to 'char **' of greater size
d:\den\lemon\lemon.c(1624) : warning C4311: 'type cast' : pointer truncation from 'char **' to 'unsigned long'
d:\den\lemon\lemon.c(1624) : warning C4311: 'type cast' : pointer truncation from 'char *' to 'unsigned long'
d:\den\lemon\lemon.c(1628) : warning C4311: 'type cast' : pointer truncation from 'char *' to 'unsigned long'
d:\den\lemon\lemon.c(1628) : warning C4312: 'type cast' : conversion from 'unsigned long' to 'char **' of greater size
d:\den\lemon\lemon.c(1629) : warning C4311: 'type cast' : pointer truncation from 'char *' to 'unsigned long'
d:\den\lemon\lemon.c(1629) : warning C4312: 'type cast' : conversion from 'unsigned long' to 'char **' of greater size
d:\den\lemon\lemon.c(1658) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data
d:\den\lemon\lemon.c(1661) : warning C4267: '+=' : conversion from 'size_t' to 'int', possible loss of data
d:\den\lemon\lemon.c(1774) : warning C4311: 'type cast' : pointer truncation from 'char *' to 'unsigned long'
d:\den\lemon\lemon.c(1774) : warning C4311: 'type cast' : pointer truncation from 'char *' to 'unsigned long'
d:\den\lemon\lemon.c(1785) : warning C4311: 'type cast' : pointer truncation from 'char *' to 'unsigned long'
d:\den\lemon\lemon.c(1785) : warning C4311: 'type cast' : pointer truncation from 'char *' to 'unsigned long'
d:\den\lemon\lemon.c(1883) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data
d:\den\lemon\lemon.c(2722) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data
d:\den\lemon\lemon.c(3171) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data
d:\den\lemon\lemon.c(3173) : warning C4018: '>=' : signed/unsigned mismatch
d:\den\lemon\lemon.c(3184) : warning C4267: '+=' : conversion from 'size_t' to 'int', possible loss of data
d:\den\lemon\lemon.c(3340) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data
d:\den\lemon\lemon.c(3346) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data
d:\den\lemon\lemon.c(3542) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data

Ups ... drh, sorry - title change.
 
2414 active 2007 Jun anonymous   Unable_to_fix     1 1 Unable t edit
I designed a tool in C# using the Sqlite.Net.dll and sqlite3.dll v 3.2.5. The call to sqlite3_step after the sqlite3_prepare function, causes a huge delay to return (~7MINS), sometimes it doesn't return at all. This happens when I'm using v3.2.5 but when I replace it with v 3.3.7, everything works normal. I have tried so many combination of things to get it to work on v3.2.5 (this is the version of the libraries the hardware uses), which includes setting the PRAGMA legacy_file_format to 1, but after every save it reverts back to 0. i will appreciate if I can get a response with any suggestions. Thanks in advance.
2007-Jun-13 15:58:00 by anonymous:
Please post the schema and the SELECT command that is slow under 3.2.5 and is fast under 3.3.7 so it can be reproduced using the sqlite3 commandline shell.
 
2440 active 2007 Jun rse   Pending     2 2 pkg-config(1) script "sqlite.pc" does not provide Autoconf's LIBS edit
On some platforms it isn't sufficient to link a library just against "-lsqlite". For instance under Solaris one needs "-lsqlite -lrt" because of the use of "fdatasync()". The SQLite Autoconf glue already contains the necessary check for this in order to correctly build SQLite and especially link its sqlite(1). But this information is not passed through to the applications which use pkg-config(1) to build against SQLite. Possible fix from OpenPKG's "sqlite" package is following:

Index: sqlite3.pc.in
--- sqlite3.pc.in.orig  2004-07-19 06:25:47 +0200
+++ sqlite3.pc.in   2007-06-20 18:09:00 +0200
@@ -8,5 +8,5 @@
 Name: SQLite^M
 Description: SQL database engine^M
 Version: @VERSION@^M
-Libs: -L${libdir} -lsqlite3^M
+Libs: -L${libdir} -lsqlite3 @LIBS@^M
 Cflags: -I${includedir}^M
 
2479 active 2007 Jun anonymous   Pending     1 1 WinCE regression on some systems. Any db open fails. edit
Because Windows CE is a modular system, meaning many parts of it can be optionally ommited by the system builder, some don't include the CP_UTF8 conversion algorithms for MultiByteToWideChar and family. I believe Windows 95 and early 98 systems can also lack this encoding if not updated with a later Internet Explorer version.

Solution is to just use the sqlite internal functions that already know how to do the same thing.

Attached is an untested patch to os_win.c (I don't have a windows machine nor a cross-compiler set up) to show where the problem is and a possible (sub-optimal) solution.

I believe the right thing to do would be to just drop the utf8ToUnicode and unicodeToUtf8 functions, add the sqlite3Utf8to16 equivalent to utf.c and use them instead.

~Nuno Lucas

2007-Jun-29 14:54:11 by anonymous:
The title is wrong. It should say "Any db open using the UTF-8 API", as using the open16 API will work.
 
2487 active 2007 Jul anonymous   Pending     1 1 SQLite database locked error on NFS mounted home dir edit
I have a c program using the provided API. My home directory is NFS mounted, Im using SQLite 3.3.17. I open a new database using "sqlite3_open", then strcpy () a SQL command to create a table, and run "sqlite3_exec" with this string. I get a return code of 5=database locked. I then tried to manually (command line using sqlite3) create a table within a database in my home dir, that fails too.

   ===========
   x@y> sqlite3 db2
   SQLite version 3.3.17
   Enter ".help" for instructions
   sqlite> create table test (Lastname varchar);
   SQL error: database is locked
   sqlite>
   ==============

If I try this on my local machine (a Mac), it works fine, but I need it to work in my home directory mounted via NFS as that is where the output of our program goes

2007-Jul-06 19:04:15 by anonymous:
If you're using a Mac, compile sqlite with SQLITE_ENABLE_LOCKING_STYLE in os_unix.c


2007-Jul-07 11:51:10 by drh:
This is a problem with your NFS implementation - it does not appear to support posix advisory locking. There is nothing much that SQLite can do about this.

Anonymous above suggests making use of the dot-locking mechanism contributed by Apple. This might be an effective work-around. But remember that there is performance impact. Also remember that an SQLite database that uses dot-locking is subtly imcompatible with a standard SQLite database. The file format itself is the same, but if two processes try to access the database file at the same time and one uses dot-locks and the other uses posix advisory locks, you will end up with corruption.


2007-Jul-07 12:44:09 by anonymous:
It's very odd that Apple does not fix their Mac OSX POSIX locks for NFS given their resources.
 
2491 active 2007 Jul anonymous   Pending     1 1 Mingw Warnings w/ 3.4.0 Amalgamation edit
When compiling the 3.4.0 amalgamation sqlite3.c file w/ no defines, you get the following warnings:

sqlite3/sqlite3.c: In function `sqlite3BtreeFindCell':
sqlite3/sqlite3.c:23249: warning: unused variable `data'
sqlite3/sqlite3.c: In function `vxprintf':
sqlite3/sqlite3.c:8488: warning: 'xtype' might be used uninitialized in this function
sqlite3/sqlite3.c: In function `sqlite3BtreeOpen':
sqlite3/sqlite3.c:19488: warning: 'nameLen' might be used uninitialized in this function
sqlite3/sqlite3.c: In function `getOverflowPage':
sqlite3/sqlite3.c:25386: warning: 'rc' might be used uninitialized in this function
sqlite3/sqlite3.c: In function `sqlite3Select':
sqlite3/sqlite3.c:56300: warning: 'pEList' might be used uninitialized in this function
sqlite3/sqlite3.c:56301: warning: 'pTabList' might be used uninitialized in this function
sqlite3/sqlite3.c: At top level:
sqlite3/sqlite3.c:16020: warning: 'sqlite3GenericAllocationSize' defined but not used
sqlite3/sqlite3.c:6188: warning: 'sqlite3Utf16Substr' declared `static' but never defined
sqlite3/sqlite3.c:6307: warning: 'sqlite3Get2byte' declared `static' but never defined
sqlite3/sqlite3.c:6309: warning: 'sqlite3Put2byte' declared `static' but never defined
sqlite3/sqlite3.c:23248: warning: 'sqlite3BtreeFindCell' defined but not used
sqlite3/sqlite3.c:63547: warning: 'sqlite3ParserAlloc' defined but not used
sqlite3/sqlite3.c:63673: warning: 'sqlite3ParserFree' defined but not used
sqlite3/sqlite3.c:65286: warning: 'sqlite3Parser' defined but not used

I know the uninitialized warnings are false warnings but the defined functions that aren't used seem to be an error in building the amalgamation.

 
2498 active 2007 Jul anonymous   Pending     3 2 sqlite memory org on linux edit
(related ticket #2473)... he sample programme that I run(wrote) in tty1 and there I operate the command of ps at tty2, there seems two items from the programme of ps command. This error was not at the version 3.3.13 but now it is happening at sqlite versions although i change nothing from the programme, If I turn to old versions, there is seen only one item again. When I upgrade to version 3.3.13 or later, there is seen two items again Is it normal or there is any mistake? (excuse my poor english)
2007-Jul-11 16:44:22 by anonymous:
So you are seeing 2 processes instead of 1 on Linux? Linux 2.4 and earlier kernels show threads as seperate processes with unique process IDs. Is your program creating any threads?

The only place where SQLite creates threads is the function below - but it joins with the thread right away.

/*
** This procedure attempts to determine whether or not threads
** can override each others locks then sets the
** threadsOverrideEachOthersLocks variable appropriately.
*/
static void testThreadLockingBehavior(int fd_orig){
  int fd;
  struct threadTestData d[2];
  pthread_t t[2];

  fd = dup(fd_orig);
  if( fd<0 ) return;
  memset(d, 0, sizeof(d));
  d[0].fd = fd;
  d[0].lock.l_type = F_RDLCK;
  d[0].lock.l_len = 1;
  d[0].lock.l_start = 0;
  d[0].lock.l_whence = SEEK_SET;
  d[1] = d[0];
  d[1].lock.l_type = F_WRLCK;
  pthread_create(&t[0], 0, threadLockingTest, &d[0]);
  pthread_create(&t[1], 0, threadLockingTest, &d[1]);
  pthread_join(t[0], 0);
  pthread_join(t[1], 0);
  close(fd);
  threadsOverrideEachOthersLocks =  d[0].result==0 && d[1].result==0;
}
If you post a small C program demonstrating what you're seeing, someone may be able to offer a suggestion.


2007-Jul-11 16:47:10 by anonymous:
I suppose it's not inconceivable that the join failed. Perhaps these pthread_join calls' return codes should be examined for errors.


2007-Jul-11 18:53:36 by anonymous:
If you're playing games with tty's and you've got an early Linux 2.6 kernel, it's possible that processes are dying because of http://lkml.org/lkml/2004/10/21/119. It was, last I checked, fixed in 2.6.10.

The SIGHUP being generated might also interfer with a pthread_join(), although pthread_join() doesn't say anything about ever generating EINTR...

c.


2007-Jul-12 06:12:28 by anonymous:
my example program is very simple, i not use threading-multithreading structure... If I turn to old versions of sqlite, there is seen only one item again, when I upgrade to version 3.3.13 or later, there is seen two items again Is it.

note: /lib/libpthread.so.0 linked to /lib/libpthread-0.10.so (size 55468 byte)


2007-Jul-12 11:36:37 by anonymous:
Your description of the problem isn't clear enough, so the answers you're getting are just guesses.

You may have more luck by describing the problem (with as much detail as possible) in your native language and hoping someone in the SQLite community can add a translation. I know you're doing your best with the english you speak, but it's not working well enough for someone to help with your problem.

Adding code samples and command-line output would also help considerably, since that sort of this is mostly language independent.

 
2508 active 2007 Jul anonymous   Pending     1 1 utf8ToUnicode() does not work on some WinCE devices edit
On some WinCE devices first call to MultiByteToWideChar() in utf8ToUnicode() always fails. Tried calling GetLastError() after it fails and it returns error code 87 -- ERROR_INVALID_PARAMETER.

To fix this had to change code page from CP_UTF8 to CP_ACP -- no idea why this works.

Original utf8ToUnicode()


static WCHAR *utf8ToUnicode(const char *zFilename) { int nChar; WCHAR *zWideFilename;

  nChar = MultiByteToWideChar(CP_UTF8, 0, zFilename, -1, NULL, 0);
  zWideFilename = sqliteMalloc( nChar*sizeof(zWideFilename[0]) );
  if( zWideFilename==0 ){
    return 0;
  }
  nChar = MultiByteToWideChar(CP_UTF8, 0, zFilename, -1, zWideFilename, nChar);
  if( nChar==0 ){
    sqliteFree(zWideFilename);
    zWideFilename = 0;
  }
  return zWideFilename;
}


Fixed utf8ToUnicode()
static WCHAR *utf8ToUnicode(const char *zFilename) { int nChar; WCHAR *zWideFilename;

  nChar = MultiByteToWideChar(CP_ACP, 0, zFilename, -1, NULL, 0);
  if( nChar == 0 )
  {
    DWORD dwError = GetLastError();
    OSTRACE2("MultiByteToWideChar() failed, last error: %d\n", dwError);
    return 0;
  }
  zWideFilename = sqliteMalloc( nChar*sizeof(zWideFilename[0]) );
  if( zWideFilename==0 ){
    return 0;
  }
  nChar = MultiByteToWideChar(CP_ACP, 0, zFilename, -1, zWideFilename, nChar);
  if( nChar==0 ){
    sqliteFree(zWideFilename);
    zWideFilename = 0;
  }
  return zWideFilename;
}


2007-Jul-17 23:56:10 by anonymous:
unicodeToUtf8() needs to be fixed the same way.

Before:


static char *unicodeToUtf8(const WCHAR *zWideFilename){ int nByte; char *zFilename;

  nByte = WideCharToMultiByte(CP_UTF8, 0, zWideFilename, -1, 0, 0, 0, 0);
  zFilename = sqliteMalloc( nByte );
  if( zFilename==0 ){
    return 0;
  }
  nByte = WideCharToMultiByte(CP_UTF8, 0, zWideFilename, -1, zFilename, nByte,
                              0, 0);
  if( nByte == 0 ){
    sqliteFree(zFilename);
    zFilename = 0;
  }
  return zFilename;
}


After:
static char *unicodeToUtf8(const WCHAR *zWideFilename){ int nByte; char *zFilename;

  nByte = WideCharToMultiByte(CP_ACP, 0, zWideFilename, -1, NULL, 0, NULL, NULL);
  if ( nByte == 0 )
  {
    DWORD dwError = GetLastError();
    OSTRACE2("WideCharToMultiByte() failed, last error = %d\n", dwError);
    return 0;
  }
  zFilename = sqliteMalloc( nByte );
  if( zFilename==0 ){
    return 0;
  }
  nByte = WideCharToMultiByte(CP_ACP, 0, zWideFilename, -1, zFilename, nByte,
                              0, 0);
  if( nByte == 0 ){
    sqliteFree(zFilename);
    zFilename = 0;
  }
  return zFilename;
}


Note that while original code with CP_UTF8 works on Windows and SOME WinCE devices, this modified code works well and Windows and all WinCE devices I've tested so far.


2007-Jul-18 16:01:21 by anonymous:
Why not using the conversions from SQLite internals ? It can change a UTF-16 to UTF-8 and vice-versa.

Or using UTF-16 variants in windows ce should be the best case.


2007-Aug-09 20:47:04 by anonymous:

  Why not using the conversions from SQLite internals ? It can change a UTF-16 to UTF-8 and vice-versa.

  Or using UTF-16 variants in windows ce should be the best case.

Not so simple. unicodeToUtf8() is used a lot internally regardless of what whether you use UTF-16 or UTF-8 yourself. For example, unicodeToUtf8() is used by sqlite3WinTempFileName() which is in turn used by sqlite3PagerOpentemp() -- I think you get the idea.


2007-Dec-20 00:29:33 by anonymous:
We've found that using CP_UTF8 fails on WinCE kernels that don't include SYSGEN_CORELOC (http://msdn2.microsoft.com/en-us/library/ms903883.aspx). To make the code handle any device it should be changed to:

  static WCHAR *utf8ToUnicode(const char *zFilename)
  {
    int nChar;
    WCHAR *zWideFilename;

    nChar = MultiByteToWideChar(CP_UTF8, 0, zFilename, -1, NULL, 0);
    if( nChar == 0 )
    {
      nChar = MultiByteToWideChar(CP_ACP, 0, zFilename, -1, NULL, 0);
      if( nChar == 0 )
      {
        DWORD dwError = GetLastError();
        OSTRACE2("MultiByteToWideChar() failed, last error: %d\n", dwError);
        return 0;
      }
    }
    zWideFilename = sqliteMalloc( nChar*sizeof(zWideFilename[0]) );
    if( zWideFilename==0 )
    {
      return 0;
    }
    nChar = MultiByteToWideChar(CP_UTF8, 0, zFilename, -1, zWideFilename, nChar);
    if( nChar==0 )
    {
      nChar = MultiByteToWideChar(CP_ACP, 0, zFilename, -1, zWideFilename, nChar);
      if( nChar==0 )
      {
        sqliteFree(zWideFilename);
        zWideFilename = 0;
      }
    }
    return zWideFilename;
  }
 
2509 active 2007 Jul anonymous   New     1 1 SQLITE_DATE edit
SELECT CAST(MyDate AS DATE), CAST(MyTime AS TIME) FROM MyData

I hope, it will result/return DATE, TIME. Please support to SQLITE_DATE and SQLITE_TIME. Thanks.

 
2510 active 2007 Jul anonymous   Pending     1 1 Vacuum modified FTS2 rowids edit
VACUUM modifies FTS2 rowids. Here is the test:

  drop table if exists a;

  create virtual table a using fts2 (t);

  insert into a (t) values ('one');
  insert into a (t) values ('two');
  insert into a (t) values ('three');

  select rowid, * from a;

  delete from a where t = 'two';
  vacuum;

  select rowid, * from a;

Unfortunately there is no workaround since table a is auto-generated by the FTS2 module.

2007-Jul-17 14:05:58 by anonymous:
http://www.sqlite.org/cvstrac/chngview?cn=4157


2007-Jul-17 14:24:29 by anonymous:
Yes, this behavior has been recently documented, but there is no user workaround like PRIMARY KEY for FTS2 rowids. Therefore I consider this as a bug which should be fixed in fts2.c.


2007-Jul-17 14:55:57 by anonymous:
Should virtual tables be VACUUMable? What exactly is being vacuumed here - an internal table?


2007-Jul-17 16:34:55 by shess:
I agree, I think this is a bug. Rather severe, too, the entire fts system implicitely depends on rowids not changing, this means that vacuum will break fts tables (fts1 or fts2).

drop table if exists t;
create virtual table t using fts2;
insert into t (content) values ('This is a test');
insert into t (content) values ('This is a string');
insert into t (content) values ('That was a test');
insert into t (content) values ('A random string');
select content from t where t MATCH 'test';
delete from t where content = 'This is a string';
vacuum;
select content from t where t MATCH 'test';
The first select outputs 'This is a test' and 'That was a test'. The second select outputs 'This is a test', and 'A random string'.


2007-Jul-17 17:27:21 by anonymous:
This patch seems to address the FTS2 VACUUM problem and passes all fts2 tests.

It adds an INTEGER PRIMARY KEY docid column to the hidden %_content table.

Note: this new table format is not backwards compatible with existing FTS2 databases.

-Joe Wilson

Index: ext/fts2/fts2.c
===================================================================
RCS file: /sqlite/sqlite/ext/fts2/fts2.c,v
retrieving revision 1.40
diff -u -3 -p -r1.40 fts2.c
--- ext/fts2/fts2.c	2 Jul 2007 10:16:50 -0000	1.40
+++ ext/fts2/fts2.c	17 Jul 2007 17:19:49 -0000
@@ -1769,9 +1769,9 @@ typedef enum fulltext_statement {
 */
 static const char *const fulltext_zStatement[MAX_STMT] = {
   /* CONTENT_INSERT */ NULL,  /* generated in contentInsertStatement() */
-  /* CONTENT_SELECT */ "select * from %_content where rowid = ?",
+  /* CONTENT_SELECT */ "select * from %_content where docid = ?",
   /* CONTENT_UPDATE */ NULL,  /* generated in contentUpdateStatement() */
-  /* CONTENT_DELETE */ "delete from %_content where rowid = ?",
+  /* CONTENT_DELETE */ "delete from %_content where docid = ?",

   /* BLOCK_INSERT */ "insert into %_segments values (?)",
   /* BLOCK_SELECT */ "select block from %_segments where rowid = ?",
@@ -1860,14 +1860,14 @@ static struct fulltext_vtab *cursor_vtab
 static const sqlite3_module fts2Module;   /* forward declaration */

 /* Return a dynamically generated statement of the form
- *   insert into %_content (rowid, ...) values (?, ...)
+ *   insert into %_content (docid, ...) values (?, ...)
  */
 static const char *contentInsertStatement(fulltext_vtab *v){
   StringBuffer sb;
   int i;

   initStringBuffer(&sb);
-  append(&sb, "insert into %_content (rowid, ");
+  append(&sb, "insert into %_content (docid, ");
   appendList(&sb, v->nColumn, v->azContentColumn);
   append(&sb, ") values (?");
   for(i=0; i<v->nColumn; ++i)
@@ -1878,7 +1878,7 @@ static const char *contentInsertStatemen

 /* Return a dynamically generated statement of the form
  *   update %_content set [col_0] = ?, [col_1] = ?, ...
- *                    where rowid = ?
+ *                    where docid = ?
  */
 static const char *contentUpdateStatement(fulltext_vtab *v){
   StringBuffer sb;
@@ -1893,7 +1893,7 @@ static const char *contentUpdateStatemen
     append(&sb, v->azContentColumn[i]);
     append(&sb, " = ?");
   }
-  append(&sb, " where rowid = ?");
+  append(&sb, " where docid = ?");
   return stringBufferData(&sb);
 }

@@ -2027,15 +2027,15 @@ static int sql_step_leaf_statement(fullt
   return rc;
 }

-/* insert into %_content (rowid, ...) values ([rowid], [pValues]) */
-static int content_insert(fulltext_vtab *v, sqlite3_value *rowid,
+/* insert into %_content (docid, ...) values ([docid], [pValues]) */
+static int content_insert(fulltext_vtab *v, sqlite3_value *docid,
                           sqlite3_value **pValues){
   sqlite3_stmt *s;
   int i;
   int rc = sql_get_statement(v, CONTENT_INSERT_STMT, &s);
   if( rc!=SQLITE_OK ) return rc;

-  rc = sqlite3_bind_value(s, 1, rowid);
+  rc = sqlite3_bind_value(s, 1, docid);
   if( rc!=SQLITE_OK ) return rc;

   for(i=0; i<v->nColumn; ++i){
@@ -2047,7 +2047,7 @@ static int content_insert(fulltext_vtab
 }

 /* update %_content set col0 = pValues[0], col1 = pValues[1], ...
- *                  where rowid = [iRowid] */
+ *                  where docid = [iRowid] */
 static int content_update(fulltext_vtab *v, sqlite3_value **pValues,
                           sqlite_int64 iRowid){
   sqlite3_stmt *s;
@@ -2075,7 +2075,7 @@ static void freeStringArray(int nString,
   free((void *) pString);
 }

-/* select * from %_content where rowid = [iRow]
+/* select * from %_content where docid = [iRow]
  * The caller must delete the returned array and all strings in it.
  * null fields will be NULL in the returned array.
  *
@@ -2101,10 +2101,10 @@ static int content_select(fulltext_vtab

   values = (const char **) malloc(v->nColumn * sizeof(const char *));
   for(i=0; i<v->nColumn; ++i){
-    if( sqlite3_column_type(s, i)==SQLITE_NULL ){
+    if( sqlite3_column_type(s, i+1)==SQLITE_NULL ){
       values[i] = NULL;
     }else{
-      values[i] = string_dup((char*)sqlite3_column_text(s, i));
+      values[i] = string_dup((char*)sqlite3_column_text(s, i+1));
     }
   }

@@ -2120,7 +2120,7 @@ static int content_select(fulltext_vtab
   return rc;
 }

-/* delete from %_content where rowid = [iRow ] */
+/* delete from %_content where docid = [iRow ] */
 static int content_delete(fulltext_vtab *v, sqlite_int64 iRow){
   sqlite3_stmt *s;
   int rc = sql_get_statement(v, CONTENT_DELETE_STMT, &s);
@@ -2870,7 +2870,7 @@ static int fulltextCreate(sqlite3 *db, v
   if( rc!=SQLITE_OK ) return rc;

   initStringBuffer(&schema);
-  append(&schema, "CREATE TABLE %_content(");
+  append(&schema, "CREATE TABLE %_content(docid INTEGER PRIMARY KEY, ");
   appendList(&schema, spec.nColumn, spec.azContentColumn);
   append(&schema, ")");
   rc = sql_exec(db, spec.zDb, spec.zName, stringBufferData(&schema));
@@ -3731,8 +3731,8 @@ static int fulltextFilter(

   TRACE(("FTS2 Filter %p\n",pCursor));

-  zSql = sqlite3_mprintf("select rowid, * from %%_content %s",
-                          idxNum==QUERY_GENERIC ? "" : "where rowid=?");
+  zSql = sqlite3_mprintf("select * from %%_content %s",
+                          idxNum==QUERY_GENERIC ? "" : "where docid=?");
   sqlite3_finalize(c->pStmt);
   rc = sql_prepare(v->db, v->zDb, v->zName, &c->pStmt, zSql);
   sqlite3_free(zSql);

2007-Jul-18 00:13:56 by shess:
BTW, AFAICT this only happens for sqlite3.4. Older versions don't seem to have the problem.


2007-Jul-18 01:31:49 by anonymous:
The rowid changing after VACUUM predates 3.4.0...

SQLite version 3.3.7
Enter ".help" for instructions
sqlite> CREATE TABLE t(a);
sqlite> INSERT INTO "t" VALUES('one');
sqlite> INSERT INTO "t" VALUES('two');
sqlite> INSERT INTO "t" VALUES('three');
sqlite> select rowid, * from t;
1|one
2|two
3|three
sqlite> delete from t where a = 'one';
sqlite> select rowid, * from t;
2|two
3|three
sqlite> vacuum;
sqlite> select rowid, * from t;
1|two
2|three

SQLite version 3.2.0
Enter ".help" for instructions
sqlite> CREATE TABLE t(a);
sqlite> INSERT INTO "t" VALUES('one');
sqlite> INSERT INTO "t" VALUES('two');
sqlite> INSERT INTO "t" VALUES('three');
sqlite> select rowid, * from t;
1|one
2|two
3|three
sqlite> delete from t where a = 'one';
sqlite> select rowid, * from t;
2|two
3|three
sqlite> vacuum;
sqlite> select rowid, * from t;
1|two
2|three

2007-Jul-18 15:59:24 by anonymous:
As you may know, INTEGER PRIMARY KEY indexes are the ROWID, so I must supect they would change after a VACUUM.

The best workaround is to put docid as INTEGER, then adding a PRIMARY KEY index for the docid column.

 
2511 active 2007 Jul anonymous   Pending   drh 3 2 Inconsistent Pragma output edit
Pragma output is inconsistent when setting the value. Most do not generate any output and silently set the value, while others generate a singleton row with the set value. Here is a list of pragmas that generate output while setting the values:

sqlite> PRAGMA locking_mode = NORMAL; normal sqlite> PRAGMA max_page_count = 100000; 100000

The following do not generate any output upon query: PRAGMA case_sensitive_like; PRAGMA incremental_vacuum;

Sqlite was built from almagamation using the following configuration flags: --enable-threadsafe --disable-tcl --enable-tempstore

 
2512 active 2007 Jul shess   Pending     1 1 FTS virtual table name quoting problem edit
All table names should be quoted in the FTS module code.

with TRACE enabled in ext/fts2/fts2.c:

sqlite> create virtual table "a b c" using fts2 (t);
FTS2 Create
FTS2 sql: CREATE TABLE main.a b c_content(c0t)
SQL error: vtable constructor failed: a b c
2007-Jul-18 06:44:21 by anonymous:
A similar problem shows if a FTS column has the same name as the FTS table:

  CREATE VIRTUAL TABLE a USING fts2 (a);

Returns "vtable constructor failed: a.".

 
2517 active 2007 Jul anonymous   Pending   dflam 1 1 exception on reading text in vista but not xp edit
My companies sqlite 3.1 db works perfectly on Win XP but when we moved to Vista (I'm using Vista 64)it is trowing an exception when I access a text field that contains this data:

'A/C Pressure Sensor, raw1 = A/C on, 0 = A/C off (A/C status determines which IACTx cell is used)'

Interestingly when I view data I've inserted using sqliteman3 it has unprintable characters added to it.

(A/C status determines which IACTx cell is used)9

If I define the field as Char[512] this artifact goes away. But reading your literature this isn't supposed to make a difference because everything is char.

I've changed the values in the error column, but the error seems to be depending on length rather than value.

Any help appreciatied!

Jim

2007-Jul-19 15:13:41 by drh:
We will be better able to help you with your problem on the SQLite mailing list. See http://www.sqlite.org/support.html for instructions on joining the mailing list.
 
2530 active 2007 Jul anonymous   Pending     2 3 Unable to write to windows share, even with exclusive lock edit
It has been mentioned that the file locking does not work on windows shared network drives (Samba or SMB drives from Windows or Linux). It seems that an exclusive lock should be a workaround for this problem if you need to write to a shared drive. Currently a more complicated locking is being attempted and failing on network drives.

With an exclusive lock, SQLite could resort to simply holding a open write or append enabled file handle to the database as a more primitive locking system that is more likely to work on network drive. No other process could open the database but that would be expected with an exclusive lock. The following case should then function:

  grudy@gamma:~$ mount | grep Files
  //winserver/FileDump on /mnt/Files type cifs (rw,mand,noexec,nosuid,nodev)
  grudy@gamma:~$ touch /mnt/Files/i_have_write_permissions.txt; rm /mnt/Files/i_have_write_permissions.txt
  grudy@gamma:~$ sqlite3 /mnt/Files/foo.sqlite
  SQLite version 3.3.17
  Enter ".help" for instructions
  sqlite> PRAGMA locking_mode = EXCLUSIVE;
  exclusive
  sqlite> create table bar (foobar);
  SQL error: database is locked
  sqlite>
 
2539 active 2007 Jul anonymous   Pending     2 2 WinCE: Temporary etilqs_ files are not removed from temporary folder edit
Hi, when temporary etilqs_* files are created during SQLite work on Windows CE devices, they are not removed at all.

Temporary folder at CE devices: /Application Data/Volatile

I've research that it winClose(os_win.c) function has been changed at do not remove this file, assuming it to be removed at winceDestroyLock(os_win.c), so if no lock was happened then files will stay here forever.

Has fixed it in my local copy, with hope that it will be fixed when new cool versions of SQLite will be available.

My fix at os_win.c:

static int winClose(OsFile **pId){
  winFile *pFile;
  int rc = 1;
  if( pId && (pFile = (winFile*)*pId)!=0 ){
    int rc, cnt = 0;
    OSTRACE2("CLOSE %d\n", pFile->h);
    do{
      rc = CloseHandle(pFile->h);
    }while( rc==0 && cnt++ < MX_CLOSE_ATTEMPT && (Sleep(100), 1) );
#if OS_WINCE
    winceDestroyLock(pFile);
// fix begin
	if( pFile->zDeleteOnClose ){
		DeleteFileW(pFile->zDeleteOnClose);
		sqliteFree(pFile->zDeleteOnClose);
	}
// fix end
#endif
    OpenCounter(-1);
    sqliteFree(pFile);
    *pId = 0;
  }
  return rc ? SQLITE_OK : SQLITE_IOERR;
}
Thanks,

Fedor

2007-Jul-28 16:41:41 by anonymous:
The solution is to revert checkin 3836 and re-open ticket #2294.

Looking at the wince locking mechanism, the only time we ever use the zDeleteOnClose flag is when we've opened a database for exclusive access in sqlite3WinOpenExclusive. To save time and resources (and because its not necessary) we never bother creating a locking mechanism for exclusively-opened files. So pFile->hMutex is NULL when hitting winceDestroyLock(), and the file is never deleted.

Is it possible that the original poster of #2294 was trying to close the same connection on multiple threads at the same time?


2007-Jul-31 05:32:39 by anonymous:
This is actually a duplicate of #2533


2007-Sep-21 14:20:05 by anonymous:
So when the fix of [3836] was applied, the code to delete the file was only put in the section that is called when we have a mutex. I wonder, if the deletion of the file should also take place if there was no mutex. Works for me at least:

	static void winceDestroyLock(winFile *pFile){
	  if (pFile->hMutex){
	    /* Acquire the mutex */
	    winceMutexAcquire(pFile->hMutex);

	    /* The following blocks should probably assert in debug mode, but they
	       are to cleanup in case any locks remained open */
 	   if (pFile->local.nReaders){
 	     pFile->shared->nReaders --;
 	   }
 	   if (pFile->local.bReserved){
 	     pFile->shared->bReserved = FALSE;
 	   }
 	   if (pFile->local.bPending){
 	     pFile->shared->bPending = FALSE;
 	   }
 	   if (pFile->local.bExclusive){
 	     pFile->shared->bExclusive = FALSE;
 	   }

	    /* De-reference and close our copy of the shared memory handle */
	    UnmapViewOfFile(pFile->shared);
	    CloseHandle(pFile->hShared);

	*    if( pFile->zDeleteOnClose ){
 	*     DeleteFileW(pFile->zDeleteOnClose);
 	*     sqliteFree(pFile->zDeleteOnClose);
 	*     pFile->zDeleteOnClose = 0;
 	*   }

	    /* Done with the mutex */
	    winceMutexRelease(pFile->hMutex);
	    CloseHandle(pFile->hMutex);
	    pFile->hMutex = NULL;
	  }
	+ else
	+ {
	+   if( pFile->zDeleteOnClose ){
	+      DeleteFileW(pFile->zDeleteOnClose);
	+      sqliteFree(pFile->zDeleteOnClose);
	+      pFile->zDeleteOnClose = 0;
	+    }
	+  }
	}

The code marked with * was put there in

 
2543 active 2007 Jul anonymous   Pending     1 1 Chinese charset not support?? edit
when i create a table. the table name is " " (chinese) after this "alter table   add column aaa text null" error why??/ thank you
 
2547 active 2007 Aug danielk1977   Pending     5 3 Changing db encoding of an attached db can confuse shared cache mode. edit
This is quite obscure, but in shared-cache mode:

  1) Open db A, attach empty db B.
  2) Using another connection from the same thread, set the
     encoding of B to be different from that of A. Add some
     data to B.
  3) Using the original connection, access database B. It assumes
     the encoding of A (and therefore mangling any text data).

The correct response is to return an error - "attached databases must use the same text encoding as main database".

 
2558 active 2007 Aug anonymous   Pending     2 3 Multiple JOIN USING() gives incorrect results edit
I'm having a problem joining multiple tables with USING. It appears to work, but the results are incorrect. Here is an example to illustrate the problem. I believe the three SELECT statements should be equivalent, but they produce three different results.

  .header on
  .mode column

  CREATE TABLE Main (pk INTEGER PRIMARY KEY, name VARCHAR);
  CREATE TABLE OptA (pk INTEGER PRIMARY KEY, alpha VARCHAR);
  CREATE TABLE OptB (pk INTEGER PRIMARY KEY, beta VARCHAR);

  INSERT INTO Main VALUES (1, 'One');
  INSERT INTO Main VALUES (2, 'Two');
  INSERT INTO Main VALUES (3, 'Three');
  INSERT INTO Main VALUES (4, 'Four');

  INSERT INTO OptA VALUES (1, 'Alpha1');
  INSERT INTO OptA VALUES (4, 'Alpha4');

  INSERT INTO OptB VALUES (2, 'Beta2');
  INSERT INTO OptB VALUES (4, 'Beta4');

  SELECT * FROM Main LEFT JOIN OptA USING (pk) LEFT JOIN OptB USING (pk);

  SELECT * FROM Main LEFT JOIN OptB USING (pk) LEFT JOIN OptA USING (pk);

  SELECT Main.pk, name, alpha, beta
    FROM Main LEFT JOIN OptA ON Main.pk = OptA.pk
              LEFT JOIN OptB ON Main.pk = OptB.pk;

Joining Main, OptA, and OptB omits Beta2:

  pk          name        alpha       beta
  ----------  ----------  ----------  ----------
  1           One         Alpha1
  2           Two
  3           Three
  4           Four        Alpha4      Beta4

Joining Main, OptB, and OptA omits Alpha1:

  pk          name        beta        alpha
  ----------  ----------  ----------  ----------
  1           One
  2           Two         Beta2
  3           Three
  4           Four        Beta4       Alpha4

Only by using ON instead of USING do we get the correct results:

  pk          name        alpha       beta
  ----------  ----------  ----------  ----------
  1           One         Alpha1
  2           Two                     Beta2
  3           Three
  4           Four        Alpha4      Beta4

I think this is basically the same issue as ticket #1637, but it's a more serious example. In that one, the query simply failed to compile. In this case, it seems to work, but gives you the wrong results.

I've also tried this script in PostgreSQL 8.0.13. All three queries give (the same) correct results.

2007-Aug-08 17:34:27 by anonymous:
The problem is that SQLite is transforming

  SELECT * FROM Main LEFT JOIN OptA USING (pk) LEFT JOIN OptB USING (pk);

into

  SELECT Main.pk, name, alpha, beta
    FROM Main LEFT JOIN OptA ON Main.pk = OptA.pk
              LEFT JOIN OptB ON OptA.pk = OptB.pk;

Here is a workaround to this bug that makes use of a subquery:

  select * from (SELECT * FROM Main LEFT JOIN OptA USING (pk))
    LEFT JOIN OptB USING (pk);

Conceivably all LEFT JOIN chains could be transformed into the above form, but that would decrease performance due to the intermediate result set of the subquery. Having it work without the subquery is tricky since sqlite must deduce that the last USING (pk) is equivalent to the first pk in the chain of joined tables, namely Main.pk, and not OptA.pk.

Joe Wilson

 
2580 active 2007 Aug anonymous   Pending   anonymous 1 2 Can't open a query if text to search is Greek edit
for example:

SELECT * FROM mytable WHERE mycolumn LIKE '%some greek text%'

I get wrong results, using the 3.4.2 version. No problem instead using other earlier version. I tested only in Windows.

 
2613 active 2007 Sep anonymous   Pending   drh 3 3 replace doesn't work with blobs containing \x0, otherwise it does edit
The replace expression function does not work with blobs in case of contained zero terminator character; but it does if there is not this special character included. I expected the function to work similar like substr with blob-safety in case of type is blob only. X'nnnn' is of type blob, so following example should have returned a blob type result X'0102FF0405' in the 2nd and 3rd line. How to get to this result?
SQLite version 3.4.2
Enter ".help" for instructions
sqlite> select hex(replace(X'0102030405',X'03',X'FF'));
0102FF0405
sqlite> select hex(replace(X'0102000405',X'00',X'FF'));

sqlite> select typeof(replace(X'0102000405',X'00',X'FF'));
null
sqlite>
2007-Sep-03 04:21:12 by anonymous:
Replace was designed to work with strings. However, working with blobs would be an interesting extension.


2007-Oct-18 06:13:10 by anonymous:
I've seen a similar situation where I can't reliably store stings with nulls in the middle of them as TEXT.

I can convert them to blobs, in which case length(...) works correctly. I if convert them back to strings, length(...) treats them as C-strings.

Is this the expected behavior? I notice the entire column is preserved even when it's has TEXT affinity, I can append data to it as a string, cast back to a blob and see everything (am I explaining this poorly?)

This all seems a bit counter intuitive in some ways. Perhaps strings shouldn't treat NULL characters as special?


2007-Oct-27 16:45:41 by anonymous:
Treatment of length operator is - as fas as I know - dependent on type:
As text it is the length number of UTF-8 characters and as blob it is the number of bytes. As long as all the UTF-8 characters out of the lower half ASCII char-set (127 of them), this is identical beside the fact of different 0-terminator interpretation.
To append is something different than using the replace operator. My suggestion would be to make the replace operator work with bytes (not UTF-8) in case of all 3 parameters are of type blob.
Another suggestion: the UTF aware functions are Private declared and not usable from within a loadable extension dll/so. This should be changed.


2008-Jan-28 19:36:39 by anonymous:
Will there come a solution for this with the next release? It is really not fair to handle a blob only like text which cannot contain a zero terminator. With this unique useful function a zero-containing blob could be formed into a normal text string without loosing the part behind the zero terminator. It would be really a step forward without too much effort.
 
2627 active 2007 Sep anonymous   Pending     3 2 Improper parsing of nested JOIN edit
SQLite has a problem with multiple nested JOINs. The only way to get it workig is to remove the surrounding brackets. Removing the brackets unfortunately do not work in other DB systems such as MS SQL, mysql etc.

This does not work:

Select ContactPhone.* From (ContactPhone LEFT OUTER JOIN ContactLocation ON ContactPhone.PHNLCT_ID = ContactLocation.LCT_ID) LEFT OUTER JOIN ContactItem ON ContactLocation.LCTITM_ID = ContactItem.ITM_ID

(It complains about LCT_ID or similar)

This works after removing the brackets:

Select ContactPhone.* From ContactPhone LEFT OUTER JOIN ContactLocation ON ContactPhone.PHNLCT_ID = ContactLocation.LCT_ID LEFT OUTER JOIN ContactItem ON ContactLocation.LCTITM_ID = ContactItem.ITM_ID

All other major DB systems require the surrounding brackets. Do you think it is possible to fix it? Apart from this little little SQLite is an awesome project.

Thank you Jakub Klos

2007-Sep-06 13:07:32 by anonymous:
I don't have access to MS SQL Server, but MySQL and Oracle have no issue with the query without parentheses:

create table x1(a int, b int);
create table x2(c int, d int);
create table x3(e int, f int);

mysql> select x1.* from x1 left join x2 on x1.a=x2.c left join x3 on x2.d=x3.e;
Empty set (0.00 sec)

2007-Sep-06 19:03:33 by anonymous:
MSSQL also has no problems without the parens. As a matter of fact, the only DB that I know of that requires them is MS Access (JET).


2007-Sep-06 20:11:31 by anonymous:
I guess he had no luck filing a JET bug.


2007-Sep-11 17:22:28 by anonymous:
True, MS access requires the parens but all other major DBs support the query syntax with the parens. So why SQLite does not like it? It should simply ignore them if possible. Thank you
 
2632 review 2007 Sep anonymous   Works_As_Designed   danielk1977 5 2 sqlite3* not available within loadable extension <sqlite3ext.h> edit
It is not really a bug but it may be the right moment to introduce a valuable improvement for developers of loadable extensions, which are really great (may be for all and in general) with the migration to version 3.5 branch.
By standard usage of sqlite3ext.h within loadable extensions source code there is common to all functions the 1st parameter as sqlite3_context* context, as you know. Unfortunately there is no database context direct accessible from this. However, if you take the definitions of Mem and sqlite3_context out of vdbeInt.h and remove type by space holders, then
sqlite3* db = context->s.db;
is the required variable for use with sqlite3_prepare, sqlite3_step and sqlite3_finalize etc. (That's my work-around)
What is needed only for loadable extension developers to stay aligned with new versions is the re-arrangement of existing
struct sqlite3_context {Mem s} and struct Mem {sqlite3 *db}
to become able to simply typecast sqlite3_context* into a sqlite3*; all the other internals of Mem would stay private to vdbe even when version numbers will grow.
And now to answer the question "why the hell is he asking for this?" here my explanation by an example of what I made running:
select execSQL("select 'iniTable' as Title;select * from iniTable", '|', X'0D0A', X'0D0A'||'----'||X'0D0A' );

iniTable
----
Common|Integer|4711
Common|Float|3.1416927
Common|Double|3.1416927
Common|Text|Dies ist Text
Common|Blob|X'0D0A'

Think about an application which has to create a whole InMemory-Database table structure inclusive triggers and views:

select execSQL(stmt) from StatementTable where Usage='InMemory';
And all within one and the same transaction. It is really that easy! This could answer also some points in the wishlist of the SQLite3 web site; all to do this is reordering Mem s in sqlite3_context and sqlite3*db in struct Mem to the beginning of the structures.
Doing this with the release of version 3.5 is really a good moment, isn't it?
2007-Sep-07 11:54:38 by danielk1977:
The usual approach is to pass the sqlite3* handle to sqlite3_create_function() as the 5th argument and then retrieve it from within the user function implementation using sqlite3_user_data().

If you depend on the definitions of internal data structures, you're living dangerously. ;)


2007-Sep-07 12:54:56 by anonymous:
Is then the following correct and will work in the same expected way (same pointer within ld-ex-function)? Even with sqlite_auto_extension called at DLL load time? (Win32 with Amalgamation and BDS2006)
#include <sqlite3ext.h>
SQLITE_EXTENSION_INIT1

int sqlite3_extension_init( sqlite3 *db, char **pzErrMsg, const sqlite3_api_routines *pApi )
{
  SQLITE_EXTENSION_INIT2(pApi)
  sqlite3_create_function(db, "execSQL",  -1, SQLITE_UTF8, db, execSqlFunc, 0, 0);
...
}

static void execSqlFunc(
  sqlite3_context *context, int argc, sqlite3_value **argv )
{
  sqlite3 *db = sqlite3_user_data(context );
/*
  instead of using context's internals ( sqlite3 *db = context->s.db );
  or waiting for reordering of them and using typecase ( sqlite3 *db = (sqlite3*)context; )
  respective ( sqlite3 *db = *(sqlite3**)context; )?
*/
...
}

Thank's for the smilie and your hint. Of cause, I do not want to live dangerous. This was the purpose of my posting to get this fixed officially with the next release. But if you say it is not worth the "small change" of reordering and I'm doing it correct this way, then that's fine. I started similar to your proposal by using a static variable and assigned it from within sqlite3_extension_init but this would not work with more than one db-connection (the last one would win).

The other open point for me is the right place and methode to link the same source code as used for the Loadable Extension Functions to become a DLL (for SQlite3.exe .load) as auto-loaded extensions into a specialized made of SQLite3.DLL for my applications. Do you have another trick than modifying the sqlite3_open-functions with compiler switches? This would be great. Even if above statements work, do you think the reordering makes sense for an easier transfer for the sqlite3* into the loadable extension function? The 5th argument would be kept free for other purposes. What is your estimation of likelyhood for that proposed change? What's against it?


2007-Sep-07 14:11:38 by danielk1977:
The code above is correct as far as I can see.

As far as reordering structures to allow tricky casting, there's no particular argument against, other than it makes the code a bit tricky to follow. And limits the ways in which we can change the contents of the currently opaque sqlite3_context structure in the future. Something like:

  sqlite3 *sqlite3_get_connection(sqlite3_context *p){
    return p->s.db;
  }

might be a chance I guess.


2007-Sep-10 18:58:09 by anonymous:
That is a very good idea. Now I have a case, where I would like to make use of both, the db connection AND the user data:
I have two database selections (by a list of keys) and I would like to used one function to compare "equal" and another "unequal". In this case I could use the same function body and use the userdata to distinguish between the cases equal and unequal while using the db connection to query the selection with the given keys.
Using 2 functions which are sharing the same body of a 3rd function will be my workaround until this new API function becomes available with the next 3.5.x version. Many thanks.


2007-Sep-10 19:09:51 by anonymous:
I agree. It is very reasonable to want to use an sqlite3* connection from within a registered function as well. Scenario - perform an SQL query on same connection from within a user-defined function. Setting user data just for this common action is awkward.


2008-Jan-08 14:46:14 by anonymous:
when will this function become usable? will it come with 3.5.5 ?
sqlite3 *sqlite3_get_connection(sqlite3_context *p)
 
2634 active 2007 Sep anonymous   Pending     3 3 .schema uses incorrect ORDER BY giving wrong dependency order edit
When the schema is exported, views are sorted by name instead of by dependency. If there are nested views, the schema may be invalid when used to re-create the database.

  sqlite3
  create table t ( f text );
  create view v2 as select f from t;
  create view v1 as select f from v2;
  .output test.txt
  .schema
  .exit

  sqlite3
  .read test.txt
  SQL error near line 2: no such table: main.v2
2007-Sep-07 15:33:06 by anonymous:
Use .dump instead as a workaround. Unlike .schema, .dump does not use ORDER BY in its queries on sqlite_master and it outputs its rows in order of entry.

SQLite version 3.5.0
Enter ".help" for instructions
sqlite> create table t ( f text );
sqlite> create view v2 as select f from t;
sqlite> create view v1 as select f from v2;
sqlite>
sqlite> .schema
CREATE TABLE t ( f text );
CREATE VIEW v1 as select f from v2;
CREATE VIEW v2 as select f from t;
sqlite>
sqlite> .dump
BEGIN TRANSACTION;
CREATE TABLE t ( f text );
CREATE VIEW v2 as select f from t;
CREATE VIEW v1 as select f from v2;
COMMIT;
Suggested patch:

Index: src/shell.c
===================================================================
RCS file: /sqlite/sqlite/src/shell.c,v
retrieving revision 1.167
diff -u -3 -p -r1.167 shell.c
--- src/shell.c 7 Sep 2007 01:12:32 -0000       1.167
+++ src/shell.c 7 Sep 2007 15:28:24 -0000
@@ -1411,8 +1411,7 @@ static int do_meta_command(char *zLine,
           "SELECT sql FROM "
           "  (SELECT * FROM sqlite_master UNION ALL"
           "   SELECT * FROM sqlite_temp_master) "
-          "WHERE tbl_name LIKE shellstatic() AND type!='meta' AND sql NOTNULL "
-          "ORDER BY substr(type,2,1), name",
+          "WHERE tbl_name LIKE shellstatic() AND type!='meta' AND sql NOTNULL",
           callback, &data, &zErrMsg);
         zShellStatic = 0;
       }
@@ -1421,8 +1420,7 @@ static int do_meta_command(char *zLine,
          "SELECT sql FROM "
          "  (SELECT * FROM sqlite_master UNION ALL"
          "   SELECT * FROM sqlite_temp_master) "
-         "WHERE type!='meta' AND sql NOTNULL AND name NOT LIKE 'sqlite_%'"
-         "ORDER BY substr(type,2,1), name",
+         "WHERE type!='meta' AND sql NOTNULL AND name NOT LIKE 'sqlite_%'",
          callback, &data, &zErrMsg
       );
     }
after patch:

SQLite version 3.5.0
Enter ".help" for instructions
sqlite>  create table t ( f text );
sqlite>   create view v2 as select f from t;
sqlite>   create view v1 as select f from v2;
sqlite> .schema
CREATE TABLE t ( f text );
CREATE VIEW v2 as select f from t;
CREATE VIEW v1 as select f from v2;
sqlite> .q
 
2652 active 2007 Sep drh   Pending     1 1 Aggregate function cannot be used from within a subquery edit
The following SQL fails:

   CREATE TABLE t1(x,y);
   INSERT INTO t1 VALUES(1,2);
   CREATE TABLE t2(z);
   INSERT INTO t2 VALUES(1);
   SELECT (SELECT y FROM t1 WHERE x=min(z)) FROM t2;

Problem reported on the mailing list.

2007-Sep-23 16:01:09 by anonymous:
Your syntax appears to be incorrect.

  SQLite v3.4.2
  CREATE TABLE t1(x,y);
  CREATE TABLE t2(z);
  INSERT INTO t1 VALUES(1,21);
  INSERT INTO t1 VALUES(2,22);
  INSERT INTO t1 VALUES(3,23);
  INSERT INTO t2 VALUES(3);
  INSERT INTO t2 VALUES(2);
  INSERT INTO t2 VALUES(1);

What you wanted to do:

  SELECT y FROM t1 WHERE x=(SELECT min(z) FROM t2);
  21    -- works as expected

What you did:

  SELECT (SELECT y FROM t1 WHERE x=min(z)) FROM t2;
  SQL error near line []: misuse of aggregate function min()
 
2664 active 2007 Sep danielk1977   Pending     1 1 attaching the same db twice in shared-cache mode fails edit
The following SQL script can cause an assert() to fail in shared-cache mode.

  ATTACH 'db' AS aux1;
  ATTACH 'db' AS aux2;
  CREATE TABLE aux1.abc(a, b, c);
  CREATE TABLE aux2.abc(a, b, c);
See also #2653
 
2684 active 2007 Oct anonymous   Pending     1 1 Accessing sqlite from an NT service will lock the complete databse. edit
Accessing sqlite from a NT service (application 1) will lock the complete database. Any other process trying to open an sqlite db (application 2) will get error "80004005 unable to lock database"

If application 1 runs as normal application, started by local user, this problem doesnt occur and both applications can open the db.

2007-Oct-02 15:48:05 by anonymous:
SQLite has no knowledge of Windows services. How do you propose to work around this Windows anachronism?


2007-Oct-02 17:20:38 by anonymous:
Suggesion: Try running the service in the same account as the other program that needs to access the database.

Anachronism? Service is just another word for daemon.

-knu-


2007-Oct-02 17:33:56 by anonymous:
Re: Anachronism, the OP suggested there was something fundamentally different about file access using a service. You've pointed out that it's just a file permissions issue.


2007-Oct-05 14:45:07 by drh:
Two points:

  1. The error message "80004005 unable to lock database" is not generated by SQLite. There must be some middleware someplace that is producing this message. The problem might be in that middleware and not in SQLite.

  2. None of the SQLite developers run windows. Consequently any fixes for this problem will need to come from the community. Please append patches to this ticket if you find a fix. Or close the ticket if you discover that the problem is outside of SQLite.
 
2708 active 2007 Oct anonymous   Pending     4 2 SQL error:disk I/O error edit
I cross-compile sqlite to embedded Linux,but after I insert data to the table ,it failed.the warning is "SQL error:disk I/O error".
2007-Oct-09 05:12:28 by anonymous:

  Why do you think it is SQLite error ??


2007-Oct-09 05:46:06 by danielk1977:
We'll need a bit more data than that to figure this out. Did earlier SQLite versions work? Can you post the entire output of the compile process so that we can see if there are any clues there? Can you run strace so that we can see if there really is an IO error, or at least when SQLite believes there to be one?
 
2715 active 2007 Oct anonymous   Pending     1 1 no authorization needed to remove authorizer edit
there should be a new auth code created and the auth function should be consulted for permission for removal.
2007-Oct-10 01:08:48 by drh:
I'm assuming that this feature request comes from RockShox and that the development language is Tcl.

No. If your adversary has the ability to invoke the interface that removes an authorizer, then you system is already pwned.

What you really need is the ability to [interp alias] the eval method into a safe interpreter. That way you can:

  • Open the database in the main interpreter
  • Set up the authorizer in the main interpreter to invoke a script in the main interpreter
  • Set up the [interp alias] so that the safe interpreter can do [db eval ...] but not [db auth ...]

It seems like an "-interp" option on the "eval" method of the database connection object would likely be the right interface. Or perhaps there should be separate "safeeval" method. Either way, it has been years and years since I have done anything with safe interpreters so I will have to look into what needs to be done to make that happen.


2007-Oct-17 20:11:23 by anonymous:
ok i think i agree with that.

currently you cannot use an interp alias since the target command runs in the target interp and all your variables and commands are in the wrong scope. this means one needs to load sqlite again in the new interp, and sqlite will not load in a safe interp so a regular interp is required.

to be useful, a -interp flag would need to execute in the current scope of the interp and not the global scope.

 
2721 active 2007 Oct anonymous   Pending     2 1 if db file is in a folder with non-ansi character some functions fail edit
If database file is located in directory with some non-ANSI characters (in my case with a Russian subdirectory c:\&#1052;&#1086;&#1080; &#1076;&#1086;&#1082;&#1091;&#1084;&#1077;&#1085;&#1090;&#1099;\Data_Jobs), or it's name is non-ansi. Some functions fail to execute sql.

For example (with defined UNICODE):

TCHAR sql[512];
_stprintf(sql, _T("INSERT INTO tab_SurveyedPoints (name, comment, code,")
		_T("coordinatetype, b, l, h, solutiontype, sigmah, sigmav)")
		_T(" VALUES ('%s','%s','%s',0,%lf,%lf,%lf,0,%lf,%lf);"),
		point.m_name.c_str(), point.m_description.c_str(), point.m_code.c_str(),
		point.m_coordinates.b, point.m_coordinates.l, point.m_coordinates.h,
		point.m_sigmah, point.m_sigmav);
int rc1 = sqlite3_prepare16(m_db, sqlfmt, -1, &stmt, (const void**)&pszTail);
rc != SQLITE_OK

But if I move the file to c:\My documents\Data_Jobs this works ok.

It's improbable behaviour, but I can't work around yet. Although, prepare() functions work ok as well in both cases.

Yuri Noyanov.

2007-Oct-11 19:33:34 by drh:
All string arguments to SQLite, and especially filename arguments, must be UTF-8 or UTF-16 (depending on the function). If you use string parameters which are not UTF-8 or UTF-16 (as appropriate) then the behavior of SQLite is undefined and probably not what you want.


2007-Oct-12 04:25:56 by anonymous:
but ALL programs to handle SQLite DBs (SQLIteBrowser, SQLite Control) fail to handle the files as well. Till I move the file to different directory !!!


2007-Oct-12 04:27:54 by anonymous:
Also I must note, that I CAN open the database, I CAN execute some SQLs with sqlite_prepare function OK. But sqlite_prepare16 FAILS if I just rename my database !!!


2007-Oct-12 04:31:46 by anonymous:
Also note to make my issue clearer:

sqlite_prepare16() with the same code either works OK either doesn't work. depends on database filename or folder path. The database is opened OK in both cases (I used utf8 conversion). sql_prepare() works ok in both cases.


2007-Oct-13 06:37:43 by anonymous:
That appears to be only with INSERT sql statement. Both SELECT and UPDATE work fine with sqlite_prepare16.
 
2725 active 2007 Oct anonymous   Pending     1 1 memory leak in sqlite3_open_v2() when it fails edit
only happens with

flags = SQLITE_OPEN_READWRITE;

and when

  res = sqlite3_open_v2(sourcename, &conn, flags, NULL);

seems to leak 674 bytes per call

2007-Oct-15 07:07:07 by danielk1977:
Are you calling sqlite3_close(conn) after the error occurs?

All calls to sqlite3_open_v2() need to be matched by a call to sqlite3_close(), even if an error occurs.

 
2753 active 2007 Nov anonymous   New   drh 3 3 Master journal files sometimes not deleted edit
In the 3.4.1 amalgamation, in vdbeCommit, the master journal file is created, and deleted at the end or if there is an error. But it looks like there is one case where it gets closed but not deleted. The code is:

    for(i=0; rc==SQLITE_OK && i<db->nDb; i++){
      Btree *pBt = db->aDb[i].pBt;
      if( pBt && sqlite3BtreeIsInTrans(pBt) ){
        rc = sqlite3BtreeCommitPhaseOne(pBt, zMaster);
      }
    }
    sqlite3OsClose(&master);
    if( rc!=SQLITE_OK ){
      sqliteFree(zMaster);
      return rc;
    }

It seems like that last bit should be:

    if( rc!=SQLITE_OK ){
      sqlite3OsDelete(zMaster);
      sqliteFree(zMaster);
      return rc;
    }
2007-Nov-01 19:26:28 by anonymous:
Should have read the comment above that code segment closer. Looks like this was by design. Trying to figure out what to do with the client that has hundreds of orphaned master journal files...
 
2755 active 2007 Nov anonymous   Pending     3 3 trace interfere with transaction Tcl interface edit
When using the transaction method of the Tcl interface to the SQLite with a registered "trace" function, the stack trace is lost in case an error occurs inside the transaction.

As an example I provide two outputs, the first one without a registered trace function and the second one with one (in which it cannot be seen where the exception cames from):


First:

   > ./a.tcl
vorher
BUMMM
    while executing
"a"
    invoked from within
"db transaction {
        puts "vorher"
        a
        puts "nachher"
    }"
    ("uplevel" body line 1)
    invoked from within
"uplevel 1 [list db transaction {
        puts "vorher"
        a
        puts "nachher"
    }]"
    (procedure "b" line 2)
    invoked from within
"b"
    (file "./a.tcl" line 28)


Second:

   > ./a.tcl
BEGIN
vorher
ROLLBACK

    while executing
"db transaction {
        puts "vorher"
        a
        puts "nachher"
    }"
    ("uplevel" body line 1)
    invoked from within
"uplevel 1 [list db transaction {
        puts "vorher"
        a
        puts "nachher"
    }]"
    (procedure "b" line 2)
    invoked from within
"b"
    (file "./a.tcl" line 28)

********

A scritp that demostrates this behaviour is attached. The only workaround is not to trace.

Thanks

 
2761 active 2007 Nov anonymous   Pending     3 3 CLI (shell.c) should be bundled with amalgamation edit
The CLI (shell.c) should be bundled with the amalgamation for database administrative purposes without downloading the matching shell.c from the full source tree.
I second that! Qt ships with the amalgamated source files, but we also ship shell.c, whch we have to retrieve from the non-amalgamated source files.


2007-Dec-26 15:20:04 by anonymous:
I also agree. It is inconvenient to retrieve the matching shell.c from the source tree.
 
2766 active 2007 Nov drh   Pending     1 1 TCL transaction started from within a query does not commit edit
This is a problem with the TCL interface. Consider the following TCL script:

   file delete -force test.db test.db-journal
   sqlite3 db test.db
   db eval {
     CREATE TABLE t1(x,y);
     INSERT INTO t1 VALUES(1,2);
     CREATE TABLE t2(a,b);
     INSERT INTO t2 VALUES(8,9);
   }
   db eval {SELECT * FROM t1} {
     db transaction {
       db eval {UPDATE t2 SET a=a*2}
     }
   }

The [db transaction] statement starts a transaction and it is suppose to commit the tranaction at the end of the code block. But because the transaction started while a query was active, the tranaction is unable to commit. The TCL interface never commits the tranaction nor does it give any kind of error indication.

It is unclear if an error should be returned or if the commit should be deferred until outer query finishes.

If the code within the [db transaction] block throws an error, we really need the transaction to rollback right away. Perhaps there should be a new API that cancels all pending queries. Perhaps a call to sqlite3_interrupt() would suffice for this. Need to investigate further....

 
2770 active 2007 Nov anonymous   Pending     1 1 Problem with BLOB in 3.5.x ? edit
After I've switched from 3.3.18 to 3.5.2, selecting from table which contains BLOB LONGER THAN ABOUT 990 BYTES returns error "SQL logic error or missing database" after call to _sqlite3_step().

I'm using preprocessed sources downloaded from here. DEBUG build of preprocessed sources works correctly, problem is only in RELEASE build. I'm using VC6.0 to compile.

Any idea what could be wrong? Thank you!

Can you try to reproduce this with the sqlite shell tool? Thanks.

Large blobs work for me with both release and debug builds (not msvc though, gcc/linux).


2007-Nov-12 18:41:37 by anonymous:
sqlite3.exe provided here works with the database. Problem is only with release build (static library linked into test application). Here is test app which exits with "Error 1" in release build:

  int main(int argc, char* argv[])
  {
	int rc;
	sqlite3* db;
	sqlite3_stmt* stmt;

	rc = sqlite3_open("n2.db3", &db);

	rc = sqlite3_prepare(db, "CREATE TABLE [ttt] ([bbb] BLOB)", -1, &stmt, 0 );
	rc = sqlite3_step(stmt);
	rc = sqlite3_reset(stmt);

	char text[10000],query[20000];
	strnset(text,'a',sizeof(text)-1);
	sprintf(query,"insert into [ttt] values (?)");

	rc = sqlite3_prepare(db, query, -1, &stmt, 0 );

	rc = sqlite3_bind_blob(stmt,1,text,sizeof(text), SQLITE_TRANSIENT);

	rc = sqlite3_step(stmt);
	rc = sqlite3_reset(stmt);

	rc = sqlite3_prepare(db, "select * from ttt", -1, &stmt, 0 );

	rc = sqlite3_step(stmt);

	if (rc == SQLITE_ROW)
	{
		printf("%s: OK",sqlite3_column_text(stmt,1));
	}
	else
	if (rc == SQLITE_DONE)
	{
		printf("DONE");
	}
	else
	{
		printf("Error %d",rc);
	}

	return 0;
}


2007-Nov-12 18:56:24 by drh:
You should be using sqlite3_finalize() instead of sqlite3_reset(). You are leaking memory. Also, you should use sqlite3_prepare_v2() to avoid problems with changing schemas.

But even without those fixes, I cannot reproduce the problem on Linux.


2007-Nov-12 19:39:31 by anonymous:
Suggested fixes didn't help.

I've tried to debug it. It fails in btree.c, line 3056:

if( offset+amt > nKey+pCur->info.nData ){ /* Trying to read or write past the end of the data is an error */ return SQLITE_ERROR; }

there seems to be different values in release mode. My debugger does not show values of variables in release mode, so I can be wrong, but it seems in release offset is 5 and in debug it is 4. There can be something wrong with compilation, I'll try to figure this out tomorrow.

BTW compilation of static libraty in VC6.0 gives 185 warnings. I don't know if it is ok, it haven't caused problems in older sqlite


2007-Nov-13 08:47:28 by anonymous:
I've turned off "Maximize Speed" option - this is causing the problem. No optimizations and optimize for size seems to be working. But it still makes me nervous :(( I really don't need corrupted database and now I hope it won't slow down too much. Unfortunately old library does not implement replace function so I don't want to switch back. This could be warning to others, I'm using VC++ 6.0 SP 6.

Thank you for your time.


2007-Nov-22 17:20:31 by anonymous:
I have exactly the same problem here (win XP, vc6 SP2) when I link against my sqlite static or dynamic library in release.

I have also used boundschecker to check sqlite, and it detects many dangling pointers ! But the strange thing is that I cannot find why these pointers are dangling, here an example:

  In prepare.c@188
  pTab = sqlite3FindTable(db, zMasterName, db->aDb[iDb].zName);

Boundchecker say that zMasterName is a dangling pointer, previously released here:

  in build.c@711:
  void sqlite3StartTable(
    Parse *pParse,   /* Parser context */
    Token *pName1,   /* First part of the name of the table or view */
    Token *pName2,   /* Second part of the name of the table or view */
    int isTemp,      /* True if this is a TEMP table */
    int isView,      /* True if this is a VIEW */
    int isVirtual,   /* True if this is a VIRTUAL table */
    int noErr        /* Do nothing if table already exists */
  ){
  }

It does not make sens for me, maybe it a false positive from boundchecker, but it is weird.

I don't know if these "errors" are related to the "blob" bug in release mode. I will try to debug these error with some "printf" in release mode.

Note: The provided dll (the one from the sqlite site) does not have this "bug".


2007-Nov-22 18:27:35 by anonymous:
More info: It seems that there is a bug in the VC6 (SP6) compiler.

In btree.c, line 3056:

  if( offset+amt > nKey+pCur->info.nData ){
    /* Trying to read or write past the end of the data is an error */
    return SQLITE_ERROR;
  }

After adding some printf around, It seems that the "speed optimization" compilation flag of VC6 changes the code order in a way that the offset variable is miss incremented !!

Two remarks:

  • I've traced the calling function, sqlite3BtreeData, and the it call accessPayload with the good offset value
  • VC6 produces an internal error: "fatal error C1001: INTERNAL COMPILER ERROR" in the accessPayload function, if I try to access the offset value before this line:

  aPayload = pCur->info.pCell + pCur->info.nHeader;

A dirty workaround could be to change the code order or the local var usage. I'm trying ....

 
2791 active 2007 Nov anonymous   Pending     1 1 Allow building FTS[123] as part of sqlite library with configure edit
See attached patch.
 
2793 active 2007 Nov anonymous   Pending     3 3 fts3 lacks scoping edit
It would be nice if the fts3 symbols could optionally be made private/static as the rest of the sqlite3 library. Not sure why sqlite3_api becomes public when used with the amalgamation, for that matter.

  make TOP=`pwd` BCC=gcc TCC=gcc AR=ar RANLIB=echo NAWK=gawk -f \
    main.mk sqlite3.h sqlite3.c fts3amal.c
  cat fts3amal.c >> sqlite3.c
  gcc -DSQLITE_THREADSAFE -DSQLITE_API=static -DSQLITE_PRIVATE=static \
    -DSQLITE_EXTERN=static -DSQLITE_ENABLE_FTS3 -c sqlite3.c
  nm sqlite3.o | grep -v ' [trUbd] '

  00000004 C sqlite3_api
  00064da2 T sqlite3Fts3HashClear
  000652a4 T sqlite3Fts3HashFind
  00064d60 T sqlite3Fts3HashInit
  0006533b T sqlite3Fts3HashInsert
  00064b4c T sqlite3Fts3Init
  00066b34 T sqlite3Fts3InitHashTable
  000669bd T sqlite3Fts3PorterTokenizerModule
  0006702d T sqlite3Fts3SimpleTokenizerModule
 
2809 active 2007 Nov anonymous   Pending     1 1 PRAGMA collation_list shows unregistered collations edit
As presented on the mailing list:

Imagine that a SQLite3 database opened in a custom application with a registered a collation sequence named "unknown" has created the following table:

  CREATE TABLE a (b COLLATE unknown);

Now open this table in the default SQLite3 CLI. Up to here, everything works as expected.

Next issue "PRAGMA collation_list;" and notice that "unknown" lists next to the other registered collations, even though "unknown" is not registered with the default SQLite3 CLI:

  sqlite> PRAGMA collation_list;
  0|unknown
  1|NOCASE
  2|BINARY

Responses from the mailing list indicate that this is not the expected behaviour. "PRAGMA collation_list;" should list registered collations only.

2007-Nov-28 16:12:17 by anonymous:
I don't think this is a bug. If the CLI is not aware of the collation, it should not process the query that makes use of the collation because it would certainly be wrong if it simply ignored the collation. This is not unlike a user-registered SQL function that does not exist in the CLI. I would not expect or want the sqlite3 CLI to ignore the unknown function, nor would I want the CLI to process queries ignoring the custom collation.
 
2810 active 2007 Nov anonymous   Pending     1 1 Unregistered collation problems with simple subselects edit
As discussed on the mailing-list:

Imagine that a SQLite3 database opened in a custom application with a registered a collation sequence named "unknown" has created the following table:

  CREATE TABLE a (b COLLATE unknown);

Now open this table in the default SQLite3 CLI. Up to here, everything works as expected. Simple queries like "SELECT * FROM a;" work fine.

But subselects, in their most basic form and with no sorting or comparisons, result in an error:

  sqlite> INSERT INTO a VALUES ('one');

  sqlite> SELECT * FROM a, (SELECT * FROM a);
  SQL error: no such collation sequence: unknown

  sqlite> SELECT * FROM (SELECT * FROM a);
  SQL error: no such collation sequence: unknown

  sqlite> SELECT *, * FROM a;
  one|one

This is surprising because the collation sequence should not matter to the queries. In fact, the union without the subselect works just fine and without errors.

To demonstrate, here is the explain output of a table with a registered collation sequence. No mention of the collation name here:

  sqlite> CREATE TABLE b (b collate nocase);
  sqlite> EXPLAIN SELECT * FROM b, (SELECT * FROM b);
  0|Goto|0|17|
  1|Integer|0|0|
  2|OpenRead|0|3|
  3|SetNumColumns|0|1|
  4|Integer|0|0|
  5|OpenRead|2|3|
  6|SetNumColumns|2|1|
  7|Rewind|0|14|
  8|Rewind|2|13|
  9|Column|0|0|
  10|Column|2|0|
  11|Callback|2|0|
  12|Next|2|9|
  13|Next|0|8|
  14|Close|0|0|
  15|Close|2|0|
  16|Halt|0|0|
  17|Transaction|0|0|
  18|VerifyCookie|0|4|
  19|TableLock|0|3|b
  20|Goto|0|1|
  21|Noop|0|0|
 
2814 active 2007 Nov anonymous   Pending     3 3 _XOPEN_SOURCE again edit
Ideally setting _XOPEN_SOURCE should be an opt-in detected by configure, rather than a hardcoded opt-out as it is now. I find you create more problems in setting it than just leaving it out on modern platforms.

Can you please give users the option of not defining _XOPEN_SOURCE at all?

+#ifndef SQLITE_DONT_DEFINE_XOPEN_SOURCE
 #if !defined(_XOPEN_SOURCE) && !defined(__DARWIN__) && SQLITE_THREADSAFE
 #  define _XOPEN_SOURCE 500  /* Needed to enable pthread recursive mutexes */
 #endif
+#endif
2007-Dec-01 09:23:15 by anonymous:
Also when using Python, it sets _XOPEN_SOURCE to 600. No idea what the 500 vs 600 difference is about.


2007-Dec-01 15:58:28 by anonymous:
I've used a couple of different Linux OSes and _XOPEN_SOURCE is not needed. Maybe it's for OSes more than 5 years old. Recursive mutexes are pretty much standard these days since the popularity of Java which uses them extensively.


2007-Dec-01 17:21:05 by drh:
See also tickets #2673, #2681, and #2741.


2007-Dec-02 02:08:26 by anonymous:
On Linux, PTHREAD_MUTEX_RECURSIVE is the same as PTHREAD_MUTEX_RECURSIVE_NP:

    PTHREAD_MUTEX_RECURSIVE = PTHREAD_MUTEX_RECURSIVE_NP,

Since PTHREAD_MUTEX_RECURSIVE_NP is always available, you could avoid defining _XOPEN_SOURCE and use this code instead:

-        pthread_mutexattr_settype(&recursiveAttr, PTHREAD_MUTEX_RECURSIVE);
+        pthread_mutexattr_settype(&recursiveAttr,
+#ifdef linux
+          PTHREAD_MUTEX_RECURSIVE_NP
+#else
+          PTHREAD_MUTEX_RECURSIVE
+#endif
+        );

2007-Dec-02 02:17:22 by anonymous:
A quick google search reveals how various projects deal with this recursive mutex declaration problem (in no particular order):

  • #define _XOPEN_SOURCE 500 and use PTHREAD_MUTEX_RECURSIVE
  • #define _XOPEN_SOURCE 600 and use PTHREAD_MUTEX_RECURSIVE
  • #define _GNU_SOURCE and use PTHREAD_MUTEX_RECURSIVE
  • don't define anything and use PTHREAD_MUTEX_RECURSIVE_NP on linux, and PTHREAD_MUTEX_RECURSIVE elsewhere.

Unfortunately, since PTHREAD_MUTEX_RECURSIVE is an enum on Linux, so you can't use the #ifdef PTHREAD_MUTEX_RECURSIVE compile-time technique.

 
2825 active 2007 Dec anonymous   Pending     3 3 FormatMessage (win32) should use extra flag and convert from Unicode edit
The call to FormatMessageA in the win32 source code needs to have the flags changed from: FORMAT_MESSAGE_FROM_SYSTEM to FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS

This ensures that any system messages that expect arguments do not try to grab the argument from some random memory location.

ref: http://blogs.msdn.com/oldnewthing/archive/2007/11/28/6564257.aspx

2007-Dec-06 14:07:53 by anonymous:
I also noticed that the result is NOT converted to UTF-8.

FormatMessageA returns the text in the local ANSI codepage. FormatMessageW should be used on NT systems, and either result should be converted to the SQLite UTF-8 default.


2007-Dec-11 00:34:37 by anonymous:
to simplify what is meant even more...

http://www.sqlite.org/cvstrac/fileview?f=sqlite/src/os_win.c&v=1.118

Search for FormatMessageA (only 1 instance)

  -    FORMAT_MESSAGE_FROM_SYSTEM,
  +    FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS,

No breakage, ensures that no crashes with some messages (e.g. filesystem errors). The encoding issue should be addressed separately.


2007-Dec-11 01:27:07 by anonymous:
The function should be changed to the following to correctly handle the conversion from Unicode/MBCS.

static void winDlError(sqlite3_vfs *pVfs, int nBuf, char *zBufOut){
  int error = GetLastError();
#if OS_WINCE
  if( error>0x7FFFFFF ){
    sqlite3_snprintf(nBuf, zBufOut, "OsError 0x%x", error);
  }else{
    sqlite3_snprintf(nBuf, zBufOut, "OsError %d", error);
  }
#else
  if( isNT() ){
    LPWSTR zWinTemp = NULL;
    DWORD dwLen = FormatMessageW(
      FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS,
      NULL,
      error,
      0,
      (LPWSTR) &zWinTemp,
      0,
      0
    );
    if (dwLen > 0) {
      char * zOut = unicodeToUtf8(zWinTemp);
      LocalFree(zWinTemp);
      sqlite3_snprintf(nBuf, zBufOut, "%s", zOut);
      free(zOut);
    }
  }else{
    LPSTR zWinTemp = NULL;
    DWORD dwLen = FormatMessageA(
      FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS,
      NULL,
      error,
      0,
      (LPSTR) &zWinTemp,
      0,
      0
    );
    if (dwLen > 0) {
      char * zOut = mbcsToUtf8(zWinTemp);
      LocalFree(zWinTemp);
      sqlite3_snprintf(nBuf, zBufOut, "%s", zOut);
      free(zOut);
    }
  }
#endif
}
 
2842 active 2007 Dec anonymous   Pending     1 1 .import does not recongnise NULL values edit
.import function fails to see NULL values in csv files as NULL values...instead they are treated as the string "NULL".

This is with .mode list and separator , But behaves similarly for .mode csv

Also if one outputs a table with NULL values to a file, then re-imports that file, again .import does not recognise the values as NULL, but as "NULL".

Everything here also applies to empty strings in files, e.g. instead of "NULL" using nothing...

This is a showstopper for us since we want to import a large amount of data with many tables containing NULL values. I can't see any valid reason for .import not to recognise the same syntax as the command line.

Note that something like:

sqlite3 my.db insert into MY_TABLE values (1,"foo","bar",NULL)

..works fine. It is just .import that appears to be broken.

2007-Dec-14 16:39:51 by rdc:
.import only inserts string values into database tables. If your column has a declared type that changes the columns affinity to numeric or integer, then those strings will be converted to numeric values by the SQLIte library.

The workaround is to simply insert a unique string where ever you want a NULL value, and then run an update that replaces those strings with real NULL values. If you inserted the string 'NULL' then do this after the .import

update t set field = null where field = 'NULL';

You will have to repeat this for each field in your table that might contain the 'NULL' string.

 
2857 active 2007 Dec anonymous   Pending     2 2 GROUP BY cost estimate wrong with WHERE clause edit
There seems to be an issue with the sqlite cost heuristic with an INDEX present on GROUP BY with certain types of WHERE clauses.

Given the database formed by running these statements:

  create table stuff(a,b,c,d);
  insert into stuff values(1,2,3,4);
  create temp view v1 as select random()%100,
    random()%100, random()%1000, random()%10000
     from stuff x, stuff y;
  insert into stuff select * from v1;
  insert into stuff select * from v1;
  insert into stuff select * from v1;
  insert into stuff select * from v1;
  insert into stuff select * from v1;
  create index stuff_b on stuff(b);
  create index stuff_c on stuff(c);
  create index stuff_d on stuff(d);
  analyze;

Using sqlite.org's sqlite3-3.5.4.bin, this query takes 47 seconds:

  select c from stuff where a=23 group by c;

while this query takes just 2 seconds:

  select c from stuff where a=23 group by +c;

It is more efficient in this case to do a full table scan instead of using the INDEX on column c.

2007-Dec-23 23:14:06 by anonymous:
The queries above both run in a couple of seconds with this naive patch:

--- src/where.c 12 Dec 2007 17:42:53 -0000      1.266
+++ src/where.c 23 Dec 2007 22:48:37 -0000
@@ -1514,6 +1514,12 @@ static double bestIndex(
     flags = 0;
   }

+  if( pWC && pWC->nTerm>0 && pOrderBy ){
+    /* Reduce cost if both an ORDER/GROUP BY exists with a WHERE. */
+    cost /= 100; /* A very rough guess. */
+    WHERETRACE(("... WHERE + ORDER BY decreases cost to: %.9g\n", cost));
+  }
+
   /* If the table scan does not satisfy the ORDER BY clause, increase
   ** the cost by NlogN to cover the expense of sorting. */
   if( pOrderBy ){
But it has not been tested on queries with more than one table. Its logic could be flawed.


2007-Dec-24 00:09:00 by drh:
The complaint is centered around these two queries:

    /* 1 */  SELECT c FROM stuff WHERE a=23 GROUP BY c;
    /* 2 */  SELECT c FROM stuff WHERE a=23 GROUP BY +c;

Query 1 runs in about 40 seconds and query 2 in about 1.5 seconds on my macbook. But with the patch, both queries run in about 1.5 seconds. Fair enough. But now consider these two queries:

    /* 3 */  SELECT c FROM stuff WHERE a!=23 GROUP BY c;
    /* 4 */  SELECT c FROM stuff WHERE a!=23 GROUP BY +c;

In this case, query 3 runs in 42 seconds on an unpatched version of 3.5.4 and query 4 runs in about 109 seconds. So in cases where the WHERE clause is not particularly selective, the first version is faster than the second by a good margin. On a patched version of 3.5.4, both queries 3 and 4 run in about 110 seconds.

So it seems to me that the patch is robbing Peter to pay Paul. It makes ORDER BY queries with very selective WHERE clauses run faster but at the expense of making queries with unselective WHERE clauses running slower.

But notice this: in the current (unpatched) implementation, the programmer at least has the ability to select a different algorithm by the judicious placement of a "+" sign. After the patch, this is no longer possible. The patch forces the second algorithm to be used in all cases, even cases where it is slower.

It seems to me to be better to leave things as they are since the current approach at least allows the programmer to override SQLite's algorithm selection if SQLite chooses incorrectly.

The only way, it seems to me, to automatically choose the correct algorithm is to devise some test that will determine (in advance) whether or not the WHERE clause weeds out many or few rows from the result set. I'm thinking that determination is going to be very hard (or impossible) to do without first doing a full table scan.


2007-Dec-24 05:40:47 by anonymous:
It think it would be surprising to average users that adding an index (on column C in this case) may significantly decrease query performance for some queries. It was surprising to me, at least. In my opinion, a query being 20 times slower in a default bad guess situation is worse than a query only being 2.5 times slower with a default bad guess in a worst case scenario. It's a question of relative magnitude of the difference. This is why I think that the database should err on the side of the WHERE clause having a more selective bias.

(Side note: the query timings difference is less pronounced if you use PRAGMA temp_store=memory, in which case query 3 running on an unpatched 3.5.4 takes just 50% more time to run than query 4 on my machine.)

But you raise a good point in that if there's a wrong guess in the selectivity bias it would be nice to be able to manually override it.

How much do you hate this type of syntax that some other databases use?

  select c from stuff where a!=23 group by /*+stuff_c*/ c;

SQLite does not currently offer a way to pick a specific index. I think it would be quite useful.


2007-Dec-24 17:05:16 by anonymous:
Another option is to collect WHERE clause statistics in a table like

  create table sqlite_stat2(
    where_clause_md5 BLOB primary key,
    where_clause     TEXT,
    rows_examined    INT,
    rows_true        INT
  );

where the last 2 columns are cumulative for each query. The statistics option could be enabled/disabled via a PRAGMA sqlite_collect_statistics.

The where_clause column could be a string generated fairly easily from the walking the parse tree of the resolved Select statement's pWhere. This way the where_clause is normalized and a single query with many subselects could generate more than 1 where_clause, and different queries that happen to use the same normalized where clause would update the same entry in the stat2 table. where_clause normalization would strip off aliases and only refer to the original table and column names.

For example the 2 queries below:

  -- CREATE TABLE t1(a, b);
  -- CREATE TABLE t2(b, c);
  SELECT t1.a*c as AC, t2.b-a as BA FROM t1, t2 WHERE AC>BA;
  SELECT *, t1.a Foo FROM t2, t1 WHERE Foo*c > t2.b - t1.a;

would generate the same normalized where_clause string "(T1.A*T2.C)>(T2.B-T1.A)". The table information is already encoded within it.

The generated VDBE code would have to generate Mem counters that would be incremented by each WHERE test, and lazily updated at the end of transactions or periodically written to the stat2 table to minimize disk use, as this information is not critical.

One could also manually set the stat2 table with statistical values they would like their queries to use even if PRAGMA sqlite_collect_statistics=off;

Any time the schema is changed, the entire sqlite_stat2 table would be cleared.

 
2859 active 2007 Dec anonymous   Pending   drh 3 2 Inconsistent column names with DISTINCT edit
Given the following SQL:

  CREATE TABLE foo(a,b);
  INSERT INTO foo (a, b) VALUES (1,2);

SQLite returns inconsistent column names when using the DISTINCT clause:

  SELECT DISTINCT foo.A, foo.B FROM foo;
  foo.A|foo.B
  1|2

  SELECT DISTINCT a, b FROM foo;
  a|b
  1|2

  SELECT DISTINCT * FROM foo;
  a|b
  1|2

  SELECT DISTINCT foo.* FROM foo;
  a|b
  1|2

Compared with SELECT without DISTINCT:

  SELECT foo.A, foo.B FROM foo;
  a|b
  1|2

  SELECT a, b FROM foo;
  a|b
  1|2

  SELECT * FROM foo;
  a|b
  1|2

  SELECT foo.* FROM foo;
  a|b
  1|2
 
2863 active 2007 Dec anonymous   Pending     2 3 test cast-3.14, cast-3.18 and cast-3.24 fail edit
  test cast-3.{14,18,24} fail on freebsd-6.3-PRERELEASE2:
  cast-3.14...^M
  Expected: [9223372036854774784]^M
       Got: [9223372036854773760]^M
  cast-3.18...^M
  Expected: [-9223372036854774784]^M
       Got: [-9223372036854773760]^M
  cast-3.24...^M
  Expected: [9223372036854774784]^M
       Got: [9223372036854773760]^M
  I used tcl8.4 from ports with no threads and here was the config line:
  ../sqlite-3.5.4/configure --prefix=/home/marc/local --with-tcl=/usr/local/lib/tcl8.4/

This was built on an ibm t30 laptop

 
2865 active 2007 Dec anonymous   Pending     1 2 FTS3 does not build with amalgamation in CVS edit
Grab the latest CVS sources, then run:

  ./configure
  make sqlite3.c

  grep sqlite3Fts3Init sqlite3.c

    extern int sqlite3Fts3Init(sqlite3*);
    rc = sqlite3Fts3Init(db);

If you compile sqlite3.c with -DSQLITE_ENABLE_FTS3, then sqlite3Fts3Init is unresolved.

For some reason, sqlite3Fts3Init and fts3.c was not included in the sqlite3.c amalg.

It used to work correctly in 3.5.4.

2007-Dec-30 18:17:57 by anonymous:
Nevermind, "make sqlite3.c" has never built with the fts3 sources in 3.5.4 or before.

You have to run ext/fts3/mkfts3amal.tcl


2007-Dec-30 18:20:56 by anonymous:
It seems that the sqlite3+fts3 amalg can only be built from main.mk, not Makefile.
 
2867 active 2008 Jan anonymous   Pending     2 2 doesn't build on Cygwin - wrong sqlite3 exe suffix edit
The new Makefile used $(EXE), which doesn't seem to be defined (typo?)
2008-Jan-02 11:12:39 by anonymous:
Same on mingw:

Following patch fixes things:

--- sqlite-3.5.4/Makefile.in    Thu Dec 13 19:17:42 2007
+++ sqlite-3.5.4-mingw-fix/Makefile.in  Wed Jan  2 11:37:50 2008
@@ -322,7 +322,7 @@
                -o $@ $(TOP)/src/shell.c libsqlite3.la \
                $(LIBREADLINE) $(TLIBS)

-sqlite3$(EXE): $(TOP)/src/shell.c sqlite3.c sqlite3.h
+sqlite3$(TEXE):        $(TOP)/src/shell.c sqlite3.c sqlite3.h
        $(LTLINK) $(READLINE_FLAGS) -o $@                          \
                -DSQLITE_MAX_SQL_LENGTH=1000000000                  \
                -USQLITE_THREADSAFE -DSQLITE_THREADSAFE=0           \
@@ -577,7 +577,7 @@
          -e 's,$$,\\n",' \
          $(TOP)/tool/spaceanal.tcl >spaceanal_tcl.h
        $(LTLINK) -DTCLSH=2 -DSQLITE_TEST=1 $(TEMP_STORE)\
-                -o sqlite3_analyzer$(EXE) $(TESTSRC) $(TOP)/src/tclsqlite.c \
+                -o sqlite3_analyzer$(TEXE) $(TESTSRC) $(TOP)/src/tclsqlite.c \
                libtclsqlite3.la $(LIBTCL)
 
2873 active 2008 Jan anonymous   Pending     1 1 HAVE_USLEEP, HAVE_FDATASYNC=1 detected but not used by configure; make edit
I noticed that a couple of open source projects were not picking up usleep() for recent sqlite builds and used the coarser grained sleep() instead. Around 11 months ago something changed in sqlite's build process.

It seems that both -DHAVE_USLEEP=1, -DHAVE_FDATASYNC=1 and -DOS_UNIX=1 are detected correctly by configure but not used by the generated Makefile. As result, UNIX builds of sqlite3 via ./configure do not use usleep() and fdatasync() and do not define OS_UNIX.

I don't know whether the lack of fdatasync() versus the default fsync() affects anyone.

Please apply the following patch which corrects the problem with "./configure && make". Thank you.

Index: configure
===================================================================
RCS file: /sqlite/sqlite/configure,v
retrieving revision 1.45
diff -u -3 -p -r1.45 configure
--- configure	27 Nov 2007 14:50:07 -0000	1.45
+++ configure	5 Jan 2008 07:41:00 -0000
@@ -18520,9 +18520,9 @@ if test "$TARGET_EXEEXT" = ".exe"; then
     OS_UNIX=0
     OS_WIN=0
     OS_OS2=1
-    TARGET_CFLAGS="$TARGET_CFLAGS -DOS_OS2=1"
+    CFLAGS="$CFLAGS -DOS_OS2=1"
     if test "$ac_compiler_gnu" == "yes" ; then
-      TARGET_CFLAGS="$TARGET_CFLAGS -Zomf -Zexe -Zmap"
+      CFLAGS="$CFLAGS -Zomf -Zexe -Zmap"
       BUILD_CFLAGS="$BUILD_CFLAGS -Zomf -Zexe"
     fi
   else
@@ -18530,14 +18530,14 @@ if test "$TARGET_EXEEXT" = ".exe"; then
     OS_WIN=1
     OS_OS2=0
     tclsubdir=win
-    TARGET_CFLAGS="$TARGET_CFLAGS -DOS_WIN=1"
+    CFLAGS="$CFLAGS -DOS_WIN=1"
   fi
 else
   OS_UNIX=1
   OS_WIN=0
   OS_OS2=0
   tclsubdir=unix
-  TARGET_CFLAGS="$TARGET_CFLAGS -DOS_UNIX=1"
+  CFLAGS="$CFLAGS -DOS_UNIX=1"
 fi


@@ -19392,7 +19392,7 @@ fi
 echo "$as_me:$LINENO: result: $ac_cv_func_usleep" >&5
 echo "${ECHO_T}$ac_cv_func_usleep" >&6
 if test $ac_cv_func_usleep = yes; then
-  TARGET_CFLAGS="$TARGET_CFLAGS -DHAVE_USLEEP=1"
+  CFLAGS="$CFLAGS -DHAVE_USLEEP=1"
 fi


@@ -19491,7 +19491,7 @@ fi
 echo "$as_me:$LINENO: result: $ac_cv_func_fdatasync" >&5
 echo "${ECHO_T}$ac_cv_func_fdatasync" >&6
 if test $ac_cv_func_fdatasync = yes; then
-  TARGET_CFLAGS="$TARGET_CFLAGS -DHAVE_FDATASYNC=1"
+  CFLAGS="$CFLAGS -DHAVE_FDATASYNC=1"
 fi


Index: configure.ac
===================================================================
RCS file: /sqlite/sqlite/configure.ac,v
retrieving revision 1.31
diff -u -3 -p -r1.31 configure.ac
--- configure.ac	27 Nov 2007 14:50:07 -0000	1.31
+++ configure.ac	5 Jan 2008 07:41:00 -0000
@@ -310,9 +310,9 @@ if test "$TARGET_EXEEXT" = ".exe"; then
     OS_UNIX=0
     OS_WIN=0
     OS_OS2=1
-    TARGET_CFLAGS="$TARGET_CFLAGS -DOS_OS2=1"
+    CFLAGS="$CFLAGS -DOS_OS2=1"
     if test "$ac_compiler_gnu" == "yes" ; then
-      TARGET_CFLAGS="$TARGET_CFLAGS -Zomf -Zexe -Zmap"
+      CFLAGS="$CFLAGS -Zomf -Zexe -Zmap"
       BUILD_CFLAGS="$BUILD_CFLAGS -Zomf -Zexe"
     fi
   else
@@ -320,14 +320,14 @@ if test "$TARGET_EXEEXT" = ".exe"; then
     OS_WIN=1
     OS_OS2=0
     tclsubdir=win
-    TARGET_CFLAGS="$TARGET_CFLAGS -DOS_WIN=1"
+    CFLAGS="$CFLAGS -DOS_WIN=1"
   fi
 else
   OS_UNIX=1
   OS_WIN=0
   OS_OS2=0
   tclsubdir=unix
-  TARGET_CFLAGS="$TARGET_CFLAGS -DOS_UNIX=1"
+  CFLAGS="$CFLAGS -DOS_UNIX=1"
 fi

 AC_SUBST(BUILD_EXEEXT)
@@ -565,13 +565,13 @@ AC_SUBST(TARGET_DEBUG)
 #########
 # Figure out whether or not we have a "usleep()" function.
 #
-AC_CHECK_FUNC(usleep, [TARGET_CFLAGS="$TARGET_CFLAGS -DHAVE_USLEEP=1"])
+AC_CHECK_FUNC(usleep, [CFLAGS="$CFLAGS -DHAVE_USLEEP=1"])

 #--------------------------------------------------------------------
 # Redefine fdatasync as fsync on systems that lack fdatasync
 #--------------------------------------------------------------------

-AC_CHECK_FUNC(fdatasync, [TARGET_CFLAGS="$TARGET_CFLAGS -DHAVE_FDATASYNC=1"])
+AC_CHECK_FUNC(fdatasync, [CFLAGS="$CFLAGS -DHAVE_FDATASYNC=1"])

 #########
 # Generate the output files.
2008-Jan-05 08:24:29 by anonymous:
It appears that http://www.sqlite.org/sqlite3-3.5.4.bin.gz and http://www.sqlite.org/sqlite-3.5.4.so.gz use sleep and fsync even though usleep and fdatasync are available on Linux.

On the Linux man page, it claims that fdatasync is more efficient than fsync:

"Unfortunately, fsync() will always initiate two write operations: one for the newly written data and another one in order to update the modification time stored in the inode. If the modification time is not a part of the transaction concept fdatasync() can be used to avoid unnecessary inode disk write operations."

 
2874 active 2008 Jan anonymous   Pending     1 1 THREADSAFE #define HAVE_LOCALTIME_R, HAVE_GMTIME_R in os_unix.c edit
The precompiled shared sqlite3 library for Linux on sqlite.org which appears to be built with pthread support is using localtime and gmtime which are not threadsafe.

For THREADSAFE builds could either configure be changed to detect the functions gmtime_r and localtime_r or change os_unix.c to explicitly #define HAVE_LOCALTIME_R and HAVE_GMTIME_R?

 
2875 active 2008 Jan anonymous   Pending     3 2 LIKE does not work with lowercase swedish characters edit
Swedish letters &aring;,&auml;,&ouml; is not supported by the LIKE statement. When trying to perform a query like

SELECT * FROM table WHERE name LIKE "&aring;%"

we will not get a match for names starting on &Aring; (which is uppercase for &aring;).

To recreate:


SQLite version 3.5.4 Enter ".help" for instructions sqlite> CREATE TABLE TestingTable(Name varchar(20)); sqlite> INSERT INTO TestingTable values ('Sweden'); sqlite> INSERT INTO TestingTable values ('sweden'); sqlite> INSERT INTO TestingTable values ('&Aring;land'); sqlite> INSERT INTO TestingTable values ('&aring;land'); sqlite> SELECT * FROM TestingTable; Sweden sweden &Aring;land &aring;land sqlite> SELECT * FROM TestingTable WHERE Name LIKE "swe%"; Sweden sweden sqlite> SELECT * FROM TestingTable WHERE Name LIKE &aring;la%"; &aring;land
 
2878 active 2008 Jan anonymous   Fixed     1 1 Memory leaks with latest CVS [4693] edit
This SQL leaks memory with CVS [4693] :

  CREATE TABLE x(id integer primary key, a TEXT NULL);
  INSERT INTO x (a) VALUES ('first');
  CREATE TABLE tempx(id integer primary key, a TEXT NULL);
  INSERT INTO tempx (a) VALUES ('t-first');
  CREATE VIEW tv1 AS SELECT x.id, tx.id FROM x JOIN tempx tx ON tx.id=x.id;

One leak is caused by "CREATE TABLE tempx", a second one by "CREATE VIEW tv1".

The above SQL is a digest of select7.test, select7-2.1.

2008-Jan-14 17:51:11 by anonymous:
I have tested again with CVS [4711] and it does no longer show the original leaks.

I therefore consider this issue fixed and I will now close this ticket.


2008-Jan-14 23:56:34 by anonymous:
Doing a fulltest with -MSQLITE_MEMDEUG I see reports of memory leaks. I assume these are of little or minimal interest at present because of the amount of code flux.

If you do want details, please let me know (I'll recheck this ticket tomorrow I guess).


2008-Jan-15 08:23:56 by anonymous:
Thanks for the follow-up. I am not running the original test-suite but have have ported a great number of them to Delphi.

If you could just let me know which tests caused the leaks you fixed in [4712] I'd be more than glad the port these test as well and let you know my findings.


2008-Jan-19 00:56:17 by anonymous:
A test from 20 minutes ago passes cleanly. This could be closed.
 
2879 active 2008 Jan anonymous   Pending   anonymous 4 3 VACUUM enters temporary sequence numbers in sqlite_sequence edit
I have two TEMPORARY tables added to a database, that both contain an INTEGER PRIMARY KEY AUTOINCREMENT field. After issuing a VACUUM command, the maximum value of these fields is added to the sqlite_sequence table. And stay there after the connection closes.
 
2882 active 2008 Jan anonymous   Pending     3 3 fulltest failure: ./testfixture: wrong # args: should be "cksum db" edit
  exclusive-ioerr-2.2.1... Ok

  ./testfixture: wrong # args: should be "cksum db"
      while executing
  "ifcapable vacuum {
    do_ioerr_test ioerr-2 -cksum true -sqlprep {
      BEGIN;
      CREATE TABLE t1(a, b, c);
      INSERT INTO t1 VALUES(1, randstr(50,..."
      (file "../test/ioerr.test" line 58)
      invoked from within
  "source $testdir/ioerr.test"
      (file "../test/exclusive3.test" line 50)
      invoked from within
  "source $testfile"
      ("foreach" body line 5)
      invoked from within
  "foreach testfile [lsort -dictionary [glob $testdir/*.test]] {
      set tail [file tail $testfile]
      if {[lsearch -exact $EXCLUDE $tail]>=0} continue
  ..."
      ("for" body line 7)
      invoked from within
  "for {set Counter 0} {$Counter<$COUNT && $nErr==0} {incr Counter} {
    if {$Counter%2} {
      set ::SETUP_SQL {PRAGMA default_synchronous=off;}
    } else ..."
      (file "..//test/all.test" line 85)
2008-Jan-14 23:25:49 by anonymous:
The latest code seems to have fixed this. I would close this but I don't see how to do that.
 
2886 active 2008 Jan anonymous   Pending     3 3 testfixture: -fPIC needed when building extension(s) edit
(this fix/change is probably needed in older versions too, i meant to send this in earlier)

-fPIC is needed when building extensions (some platforms don't need this or don't care --- x86-64 does)

  diff --git a/test/loadext.test b/test/loadext.test
  index 81e152f..2a7fa2e 100644
  --- a/test/loadext.test
  +++ b/test/loadext.test
  @@ -64,7 +64,7 @@ if {![file exists $testextension]} {
     set srcdir [file dir $testdir]/src
     set testextsrc $srcdir/test_loadext.c
     if {[catch {
  -    exec gcc -Wall -I$srcdir -I. -g -shared $testextsrc -o $testextension
  +    exec gcc -Wall -fPIC -I$srcdir -I. -g -shared $testextsrc -o $testextension
     } msg]} {
       puts "Skipping loadext tests: Test extension not built..."
       puts $msg
 
2893 active 2008 Jan anonymous   Pending     1 1 incorrect integer range tests edit
recently a function that performs integer range tests was added to the cvs (check-in [4706] ), but if i am correct there is a problem in the return value of the function in the file vdbemem.c:

  static i64 doubleToInt64(double r){
    ...
    if( r<(double)minInt ){
      return minInt;
    }else if( r>(double)maxInt ){
      return minInt;     <-- is this correct, shouldn't it be maxInt?
    }else{
      return (i64)r;
    }
  }
2008-Jan-16 17:33:56 by drh:
See the remarks on ticket #2280. The code duplicates the behavior of the FPU on x86.


2008-Jan-16 18:21:28 by anonymous:
did you mean ticket #2880? didn't read that ticket before, but since there was no comment regarding that behavior in the function it seemed (to my eyes) that it was a mistake.

maybe adding a small comment in there would clarify this issue


2008-Jan-16 18:39:42 by anonymous:
Just because the double to int overflow behavior happens to be that way with GCC on x86, is it desirable?
 
2897 active 2008 Jan anonymous   Pending     1 1 String or BLOB exceed size limit edit
This error was shown after attemp to read script from SQLite 3.5.4 shell in order to recreate old DB. Details:

1. Database was created with SQLite 3.3.4. Around 20 standard fieds and one BLOB.

2. The only one existed table was dumped with shell of SQLite 3.5.4. SQL script seems to be coorrect.

3. Opened SQLite 3.5.4 and read script in new DB. The error "String or BLOB exceed size limit" are sown for several lines. Many records missing.

4. Attempted to dump table with shell of version 3.3.6 (have no more 3.3.4 shell) and read into new DB with 3.5.4 shell The same errors are shown.

The same steps was attempted with 3.3.6. shell only. All seems to be correct.

2008-Jan-17 20:23:25 by drh:
This size limit on BLOBs in SQLite version 3.5.4 is 1GB. How big is your blob, exactly?


2008-Jan-17 22:22:24 by anonymous:
BLOB in each record is no more than few MB. Mostly it is few KB (e-client and news application). Whole DB have around 200MB.


2008-Jan-18 02:28:11 by drh:
This issue is probably resolved by check-in [4636] , then.


2008-Jan-18 14:28:13 by anonymous:
If directive SQLITE_MAX_SQL_LENGTH is not defined it is set to 1,000,000 (10^6) in amalgamation code of 3.5.4.
 
2898 active 2008 Jan anonymous   Pending     1 1 Latest CVS for 3.5.4 fails to build test1.c edit
gcc -pipe -O3 -g -Wall -DSQLITE_DISABLE_DIRSYNC=1 -I. -I../src -DNDEBUG -I/usr/include -DSQLITE_THREADSAFE=1 -DSQLITE_THREAD_OVERRIDE_LOCK=1 -DSQLITE_OMIT_LOAD_EXTENSION=1 -DTCLSH=1 -DSQLITE_TEST=1 -DSQLITE_CRASH_TEST=1 -DSQLITE_NO_SYNC=1 -DTEMP_STORE=1 -o .libs/testfixture ../src/attach.c ../src/btree.c ../src/build.c ../src/date.c ../src/expr.c ../src/func.c ../src/insert.c ../src/malloc.c ../src/os.c ../src/os_os2.c ../src/os_unix.c ../src/os_win.c ../src/pager.c ../src/pragma.c ../src/prepare.c ../src/printf.c ../src/select.c ../src/test1.c ../src/test2.c ../src/test3.c ../src/test4.c ../src/test5.c ../src/test6.c ../src/test7.c ../src/test8.c ../src/test9.c ../src/test_autoext.c ../src/test_async.c ../src/test_btree.c ../src/test_config.c ../src/test_hexio.c ../src/test_malloc.c ../src/test_md5.c ../src/test_onefile.c ../src/test_schema.c ../src/test_server.c ../src/test_tclvar.c ../src/test_thread.c ../src/tokenize.c ../src/utf.c ../src/util.c ../src/vdbe.c ../src/vdbeapi.c ../src/vdbeaux.c ../src/vdbemem.c ../src/where.c parse.c ../src/tclsqlite.c ./.libs/libsqlite3.so -L/usr/lib64 -ltcl8.4 -ldl -lpthread -lieee -lm -Wl,--rpath -Wl,/common/pkgs/sqlite-3.5.4.3/lib ../src/build.c: In function 'sqlite3RefillIndex': ../src/build.c:2275: warning: cast to pointer from integer of different size ../src/func.c: In function 'trimFunc': ../src/func.c:919: warning: cast from pointer to integer of different size ../src/func.c: In function 'sqlite3RegisterBuiltinFunctions': ../src/func.c:1464: warning: cast to pointer from integer of different size ../src/func.c:1483: warning: cast to pointer from integer of different size ../src/insert.c: In function 'sqlite3GenerateConstraintChecks': ../src/insert.c:1200: warning: cast to pointer from integer of different size ../src/insert.c:1034: warning: 'j2' may be used uninitialized in this function ../src/insert.c: In function 'sqlite3Insert': ../src/insert.c:373: warning: 'regFromSelect' may be used uninitialized in this function ../src/test1.c: In function 'test_collate_func': ../src/test1.c:2085: warning: cast from pointer to integer of different size ../src/test1.c: In function 'test_collate_needed_cb': ../src/test1.c:2209: warning: cast to pointer from integer of different size ../src/test1.c: In function 'alignmentCollFunc': ../src/test1.c:2258: warning: cast from pointer to integer of different size ../src/test1.c:2259: warning: cast from pointer to integer of different size ../src/test8.c: In function 'echoBestIndex': ../src/test8.c:722: warning: 'nRow' may be used uninitialized in this function ../src/vdbe.c: In function 'sqlite3VdbeExec': ../src/vdbe.c:502: warning: 'pOut' may be used uninitialized in this function ../src/vdbe.c:501: warning: 'pIn3' may be used uninitialized in this function ../src/vdbe.c:501: warning: 'pIn2' may be used uninitialized in this function ../src/vdbe.c:501: warning: 'pIn1' may be used uninitialized in this function ../src/vdbeaux.c: In function 'sqlite3VdbeChangeP4': ../src/vdbeaux.c:529: warning: cast from pointer to integer of different size ../src/vdbemem.c: In function 'sqlite3ValueText': ../src/vdbemem.c:911: warning: cast from pointer to integer of different size /tmp/ccsuOeus.o: In function `reset_prng_state': /build/work/sqlite-3.5.4.3/bld/../src/test1.c:4280: undefined reference to `sqlite3ResetPrngState' /tmp/ccsuOeus.o: In function `restore_prng_state': /build/work/sqlite-3.5.4.3/bld/../src/test1.c:4267: undefined reference to `sqlite3RestorePrngState' /tmp/ccsuOeus.o: In function `save_prng_state': /build/work/sqlite-3.5.4.3/bld/../src/test1.c:4254: undefined reference to `sqlite3SavePrngState' collect2: ld returned 1 exit status make: *** [testfixture] Error 1
2008-Jan-17 23:54:58 by anonymous:
Problem appears to be here in libsqlite.3.so.0.8.6 as shown by:

   nm -A .libs/libsqlite3.so.0.8.6 | grep sqlite3ResetPrngState

which shows no entry point. And:

   nm -A .libs/random.o  | grep sqlite3ResetPrngState

which also shows no entry point.


2008-Jan-17 23:56:55 by anonymous:
Ah... It appears -DSQLITE_TEST should be passed when building test1.c and left off when building prior to install.


2008-Jan-21 20:16:00 by anonymous:
In the makefile the right flag appears to be set, it's just not making it through to the compile for some reason.


2008-Jan-21 20:16:24 by anonymous:
Still fails the same based on today's cvs update.


2008-Jan-23 03:14:49 by anonymous:
This bug fixed as of latest cvs pull
 
2901 active 2008 Jan anonymous   Pending     3 3 ROLLBACK and COMMIT statements should not expire edit
Currently, whenever a statement changes the schema of the database, all prepared statements will be expired, no matter whether they actually need to be prepared again or not. This is especially problematic for ROLLBACK statements in a multi-statement transaction. Currently there is no way to guaranty that a multi-statement transaction can at least be rolled back in case of an error, because one has to (re)prepare the ROLLBACK statement to roll back the transaction, which can fail because of OOM (in a multi-threaded application).
 
2902 active 2008 Jan anonymous   Pending   drh 3 3 Add watch support to SQLite edit
SQLite currently provides only TRIGGERs and the update_hook() as a way for applications to stay informed about changes to the database. But both of these alternatives do not provide enough details about the actual changes to the underlying database file(s).

We've prepared a patch for SQLite 3.5.x to allow applications to install a watch_hook into the database, that will be invoked everytime the database is changed with exact details about the change that was performed.

2008-Jan-22 16:06:59 by anonymous:
Great idea and nice job. This functionality is very useful.


2008-Jan-31 18:27:16 by anonymous:
Any chance to get this committed for the next release (i.e. 3.5.6)?


2008-Jan-31 19:38:35 by drh:
Unlikely, for two reasons:

  1. I am unconvinced that this patch solves a problem that needs solving. It is vitally important to a project like SQLite that we work to avoid clutter and cruft. That means that any change must have a compelling rational or else it is rejected.

  2. The patches are against version 3.5.4. There were many changes to the core for 3.5.5 and the patches no longer work.


2008-Jan-31 20:58:55 by anonymous:
We can (and will) port the changes to 3.5.5, so the second point will be done. First the first point, I'm not sure how many projects will actually need this functionality, but I guess there are quite a lot of projects that would benefit, and for the others, there's zero overhead due to this patch.


2008-Jan-31 21:21:30 by drh:
There is a lot of overhead for me because if I accept this patch, that means I have to maintain it forever. Most of the work is in maintenance, not coming up with the original patch.
 
2903 active 2008 Jan anonymous   Pending     2 2 tclinstaller.tcl fails on path and permissions issue edit
When compiling using custom PREFIX, pointing to private directory, tclinstaller.tcl fails, because it tries to remove contents from /usr/share/tcl8.4/sqlite3.

./configure --prefix=/my/private/sqlite/sqlite-3.5.4
... # success
make
... # success
make install
...
tclsh ./tclinstaller.tcl 3.5
error deleting "/usr/share/tcl8.4/sqlite3": not owner
    while executing
"file delete -force $LIBDIR/sqlite3"
    (file "./tclinstaller.tcl" line 17)
make: *** [tcl_install] Error 1
I've found two work-arounds:

  1. If you run make install as root.
  2. If you use ./configure --disable-tcl
2008-Jan-28 17:47:39 by anonymous:
I also ran into this problem.

make install as root will end up copying files into the system's library directory and is almost certainly not what you want if you specified your own --prefix.

 
2905 active 2008 Jan anonymous   Pending   pweilbacher 2 2 mutex_os2.c - incorrect mutex implementation edit
The OS/2 version of sqlite3_mutex_alloc() is badly broken.

It creates named mutexes which, by design, are global rather than process-specific as intended. This might be minimally acceptable except that the function reuses the same name every time it attempts to create a SQLITE_MUTEX_FAST or SQLITE_MUTEX_RECURSIVE.

The result is that every call returns the exact same semaphore to every thread in every process using sqlite3. Once this mutex is owned by one process, other processes calling sqlite3_mutex_enter() will be blocked. Much the same is true for the static mutexes. Every process ends up using the exact same SQLITE_MUTEX_STATIC_MASTER, SQLITE_MUTEX_STATIC_MEM, etc.

There's another flaw that is fairly minor compared with the above: in an attempt to avoid concurrency when creating the static mutexes, this function uses an API call that is thoroughly deprecated.

The attached patch remedies all of these issues. Since the logic that protects the creation of the static mutexes may not be self-evident, here's an explanation:

The existence (or non-existence) of a given named mutex is itself a semaphore. If the isInit flag is false, the code attempts to create a mutex whose name is unique to that process. If the attempt is successful, there are two possibilities: (1) either the current thread is the first to reach this code & may proceed; (2) or while the current thread was making its preparations, another thread created the mutex, did the init, then closed the mutex. Testing isInit immediately after creating the mutex determines which possibility is valid.

If mutex creation fails due to a duplicate name, then another thread is currently performing the init. In this case, the current thread simply has to wait a while until the other thread is done & isInit becomes true.

Submitted by Rich Walsh (richATe-vertise.com)

 
2907 active 2008 Jan anonymous   Pending     1 1 Issues of sqlite3 with Windows Mobile 5/6 edit
hi. we are currently using sqlite3 for our mobile application. it has been running without a hitch on pocket pc 2003 and previous versions. come windows mobile 5 and 6 we have been getting errors, although not consistent yet. one example is 'EXCEPTION_DATATYPE_MISALIGNMENT'. another is 'SELECT STATMENTS TO THE LEFT AND RIGHT OF UNION ARE NOT EQUAL'. i was wondering if you have any known compatibility issues of your product with this version of windows mobile.

thanks in advance.

2008-Jan-28 13:26:26 by anonymous:
EXCEPTION_DATATYPE_MISALIGNMENT is thrown when you try to use and Odd pointer address. I wrote a custom allocator for WinCE/ARM platform, and I have to take care about memory alignment (I used to align at 2 bytes, and at that time it solved the problem)
 
2908 active 2008 Jan anonymous   Pending     3 3 Add support to examine whether statements modify the database edit
Currently there is no way to check whether a compiled statement will modify the database when being executed. Of course, there is the work-around of misusing the authorizer callback for this purpose, but this is kinda error prone and causes quite some overhead for such a simple purpose.
 
2911 active 2008 Jan anonymous   Pending     2 2 Adding parentheses to a FROM clause edit
Hi,

Parentheses in a FROM statement seem to mess with the ability to use table aliases in the "what" part. Here is an example:

Start SQLite:

    $ sqlite3 employee.db
    SQLite version 3.5.4
    Enter ".help" for instructions

create a couple of tables and populate them with test data:

    sqlite> create table person (id integer, name text, employerid integer);
    sqlite> create table employer (id integer, name text);
    sqlite> insert into person (id, name, employerid) values (1, "Dave", 1);
    sqlite> insert into employer (id, name) values (1, "ACME");

Run a simple query with no parentheses in the FROM statement:

    sqlite> select b.id from person as a inner join employer as b on a.employerid = b.id;
    1

Everything works as expected. Now, repeat that query with parentheses:

    sqlite> select b.id from (person as a inner join employer as b on a.employerid = b.id);
    SQL error: no such column: b.id

There you have it.

This may be related to ticket #1822, although that ticket deals with aliases and subqueries. This problem seems to be more fundamental.

Many thanks,

-- Dave

 
2914 active 2008 Jan anonymous   Pending     3 3 ATTACH returns SQLITE_ERROR when it means SQLITE_BUSY edit
I'm seeing the same behavior as in #2096, with SQLite 3.5.4. ATTACH DATABASE fails with SQLITE_ERROR rather than SQLITE_BUSY when the database to be attached, or the main database of the connection being attached to, is EXCLUSIVE-locked by another database connection. For added confusion, sqlite3_errmsg() says "database is locked" when the ATTACH is done via sqlite3_exec(), but "SQL logic error or missing database" when the ATTACH is done via sqlite3_step().

As a result of this bug, it is difficult to distinguish between fatal and transient ATTACH errors, particularly when sqlite3_step() is used.

I am attaching a test program that demonstrates the problem.

 
2916 active 2008 Feb anonymous   Pending     1 1 sqlitedll-3_5_5.zip is older 3.5.4 binary edit
sqlitedll-3_5_5.zip in download section is same with old 3.5.4 binary.
2008-Feb-01 12:13:04 by anonymous:
Yes , I can confirm it