sqlobject-discuss Mailing List for SQLObject (Page 14)

On Thu, Oct 17, 2013 at 10:01:53PM +0100, "Maciej (Matchek) Blizi??ski" <ma...@op...> wrote:
> 2013/10/17 Oleg Broytman <ph...@ph...>
> import sqlobject
> 
> class Foo(sqlobject.SQLObject):
>   bar = sqlobject.UnicodeCol(length=250, unique=True)
> 
> db_uri = 'sqlite:/:memory:?cache=false'
> while True:
>   sqlobject.sqlhub.processConnection = sqlobject.connectionForURI(db_uri)
>   Foo.createTable()
>   sqlobject.sqlhub.processConnection.close()
>   sqlobject.sqlhub.processConnection = None
>   sqlobject.dbconnection.TheURIOpener.cachedURIs = {}
> 
> The process' memory kept growing. It started at around 30MB and was
> steadily raising up to 300MB

   It would be interesting to test if the problem lies in SQLite,
PySQLite or SQLObject.

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.

2013/10/17 Oleg Broytman <ph...@ph...>
>
> Hi!
>
> On Wed, Oct 16, 2013 at 10:58:24PM +0100, "Maciej (Matchek) Blizi??ski" <ma...@op...> wrote:
> > - there already is special handling for in-memory databases, so
> > maybe add a little more special handling for them; for example free
> > the memory on close().
>
>    The patch to test is attached.

Thanks!

I applied the patch and ran the the following test:

import sqlobject

class Foo(sqlobject.SQLObject):
  bar = sqlobject.UnicodeCol(length=250, unique=True)

db_uri = 'sqlite:/:memory:?cache=false'
while True:
  sqlobject.sqlhub.processConnection = sqlobject.connectionForURI(db_uri)
  Foo.createTable()
  sqlobject.sqlhub.processConnection.close()
  sqlobject.sqlhub.processConnection = None
  sqlobject.dbconnection.TheURIOpener.cachedURIs = {}

The process' memory kept growing. It started at around 30MB and was
steadily raising up to 300MB at which point I stopped the process. I
didn't do any more digging yet.

Maciej

Hi!

On Wed, Oct 16, 2013 at 10:58:24PM +0100, "Maciej (Matchek) Blizi??ski" <ma...@op...> wrote:
> - there already is special handling for in-memory databases, so
> maybe add a little more special handling for them; for example free
> the memory on close().

   The patch to test is attached.

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.

On Thu, Oct 17, 2013 at 06:24:02PM +0200, Petr Jake?? <pet...@tp...> wrote:
> On 17 October 2013 17:26, Oleg Broytman <ph...@ph...> wrote:
> > Do you close and reopen the memory database in you program(s)?
> 
> Nope. I am only creating table once at the start of the program:

   I see. Thank you!

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.

On 17 October 2013 17:26, Oleg Broytman <ph...@ph...> wrote:

> Do you close and reopen the memory database in you program(s)?



Nope. I am only creating table once at the start of the program:

connectionInMemory = connectionForURI('sqlite:/:memory:')

class TextyServisnihoDispleje(SQLObject):
    _connection = connectionInMemory
    level=StringCol(length=1, default=None)
    text=UnicodeCol(length=20, default=None)
    cislovatRadek=EnumCol(enumValues=('Y','N'), default='Y')
    akce=StringCol(length=100, default=None)
    predek=ForeignKey("TextyServisnihoDispleje", default=None)
    potomci=MultipleJoin("TextyServisnihoDispleje", joinColumn="predek_id")

TextyServisnihoDispleje.createTable()

On Thu, Oct 17, 2013 at 05:20:57PM +0200, Petr Jake?? <pet...@tp...> wrote:
> On 17 October 2013 00:25, Oleg Broytman <ph...@ph...> wrote:
> 
> >    Not much. I don't know anyone who uses the memory database for any
> > purpose other than testing.
> 
> Hi, maybe off topic, but I am using in memory database in production
> environment.
> The main purpose is to reduce writing to CF card.

   Do you close and reopen the memory database in you program(s)?

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.

On 17 October 2013 00:25, Oleg Broytman <ph...@ph...> wrote:

>    Not much. I don't know anyone who uses the memory database for any
> purpose other than testing.
>


Hi, maybe off topic, but I am using in memory database in production
environment.
The main purpose is to reduce writing to CF card.
Petr

On Wed, Oct 16, 2013 at 10:58:24PM +0100, "Maciej (Matchek) Blizi??ski" <ma...@op...> wrote:
> 2013/10/16 Oleg Broytman <ph...@ph...>
> > > It does what I need, but it doesn't seem like it's what SQLObject
> > > developers intended. Do you have any recommendations?
> >
> >    I don't. Do you want to propose an API? Or better a patch?
> 
> Hey, thanks for the reply. I'm looking at the code right now, I need
> to look more carefully at the current structure before I can suggest
> an API or a patch. In general, I can see 4 possibilities:
> 
> - when creating a connection, add an option to not cache that connection

   That would be easy, just one additional parameter in
connectionForURI: cache=True (or useCache or something like that).
   The approach could lead to problems, though. Every SQLObject's
connection maintain its own cache of retrieved rows, so different
connections for the same URI will have different caches, so the same row
could appear in a few caches; that would lead to unsynchronized
read/write operations.

> - add an option to clear / expire all the connections in the current cache

   What gives? Usually if there are a few connections with different
URIs the user needs all of them.

> - add an option to the close() method so it causes the connection (and
> the URI) to expire from the cache

   Not sure it could be implemented reliably. Connection doesn't know
its own DB URI, and reconstructing it could give an equivalent but
different URI.

> - there already is special handling for in-memory databases, so
> maybe add a little more special handling for them; for example free
> the memory on close().

   That's a good approach; requires testing.

> This could potentially break existing
> applications, does SQLObject make any guarantees about in-memory
> databases?

   Not much. I don't know anyone who uses the memory database for any
purpose other than testing.

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.

2013/10/16 Oleg Broytman <ph...@ph...>
> > It does what I need, but it doesn't seem like it's what SQLObject
> > developers intended. Do you have any recommendations?
>
>    I don't. Do you want to propose an API? Or better a patch?

Hey, thanks for the reply. I'm looking at the code right now, I need
to look more carefully at the current structure before I can suggest
an API or a patch. In general, I can see 4 possibilities:

- when creating a connection, add an option to not cache that connection
- add an option to clear / expire all the connections in the current cache
- add an option to the close() method so it causes the connection (and
the URI) to expire from the cache
- there already is special handling for in-memory databases, so
maybe add a little more special handling for them; for example free
the memory on close(). This could potentially break existing
applications, does SQLObject make any guarantees about in-memory
databases?

Maciej

Hi!

On Wed, Oct 16, 2013 at 09:41:54AM +0100, "Maciej (Matchek) Blizi??ski" <ma...@op...> wrote:
> Hi guys,
> 
> I'd like to use an in-memory sqlite database for testing, and I have a
> problem where I don't seem to be able to let go of a database which was
> once created. Here's a test that shows what I mean:
> 
> class Foo(sqlobject.SQLObject):
>   bar = sqlobject.UnicodeCol(length=250, unique=True)
> 
> db_uri = 'sqlite:/:memory:'
> conn = sqlobject.connectionForURI(db_uri)
> sqlobject.sqlhub.processConnection = conn
> Foo.createTable()
> # All is good so far. Now let's drop the database.
> conn.close()
> del conn
> # From http://www.sqlite.org/inmemorydb.html:
> # "The database is automatically deleted and memory is reclaimed when the
> last connection to the database closes."
> # Let's create a fresh in-memory database.
> conn = sqlobject.connectionForURI(db_uri)
> sqlobject.sqlhub.processConnection = conn
> Foo.createTable()
> # Here, an exception is thrown:
> sqlobject.dberrors.OperationalError: table foo already exists
> 
> I must have somehow gotten a connection to the same in-memory database that
> was created on the first call to connectionForURI(). After looking at

   Yes. Actually, even worse than that - connectionForURI caches
SQLObject's connections by URIs (you get the same SQLObject's connection
with the same URI) and SQLiteConnection caches a DBAPI connection to the
in-memory DB.

> dbconnection.py, I came up with an invasive method:
> 
> sqlobject.dbconnection.TheURIOpener.cachedURIs = {}
> conn = sqlobject.connectionForURI(db_uri)
> sqlobject.sqlhub.processConnection = conn
> Foo.createTable()
> 
> It does what I need, but it doesn't seem like it's what SQLObject
> developers intended. Do you have any recommendations?

   I don't. Do you want to propose an API? Or better a patch?

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.

Hi guys,

I'd like to use an in-memory sqlite database for testing, and I have a
problem where I don't seem to be able to let go of a database which was
once created. Here's a test that shows what I mean:

class Foo(sqlobject.SQLObject):
  bar = sqlobject.UnicodeCol(length=250, unique=True)

db_uri = 'sqlite:/:memory:'
conn = sqlobject.connectionForURI(db_uri)
sqlobject.sqlhub.processConnection = conn
Foo.createTable()
# All is good so far. Now let's drop the database.
conn.close()
del conn
# From http://www.sqlite.org/inmemorydb.html:
# "The database is automatically deleted and memory is reclaimed when the
last connection to the database closes."
# Let's create a fresh in-memory database.
conn = sqlobject.connectionForURI(db_uri)
sqlobject.sqlhub.processConnection = conn
Foo.createTable()
# Here, an exception is thrown:
sqlobject.dberrors.OperationalError: table foo already exists

I must have somehow gotten a connection to the same in-memory database that
was created on the first call to connectionForURI(). After looking at
dbconnection.py, I came up with an invasive method:

sqlobject.dbconnection.TheURIOpener.cachedURIs = {}
conn = sqlobject.connectionForURI(db_uri)
sqlobject.sqlhub.processConnection = conn
Foo.createTable()

It does what I need, but it doesn't seem like it's what SQLObject
developers intended. Do you have any recommendations?

Maciej

Thank you for your help, I am going to ask psycopg team about the problem.
If I have some news about how to fix it, i will tell you.

Cheers.


2013/9/30 Oleg Broytman <ph...@ph...>

> On Sat, Sep 28, 2013 at 01:22:26PM +0400, Oleg Broytman <ph...@ph...>
> wrote:
> >    I returned from the vacation. Will test if psycopg w/o SQLObject
> > returns the code/error.
>
>    The following program:
>
> from decimal import Decimal
> import psycopg2
>
> def report_error(e):
>     print e.__class__.__name__, e.pgcode, e.pgerror
>     raise SystemExit
>
> try:
>     con = psycopg2.connect(database="test")
> except psycopg2.Error, e:
>     report_error(e)
>
> cur = con.cursor()
>
> try:
>     cur.execute('SELECT * FROM test ORDER BY id')
> except psycopg2.Error, e:
>     report_error(e)
>
>    prints
>
> OperationalError None None
>
>    when it cannot connect to the database and prints
>
> ProgrammingError 42P01 ERROR:  relation "test" does not exist
>
>    when the database exists but there is no table "test". I.e., psycopg2
> doesn't return proper code/message for an OperationalError, so your
> original request cannot be fullfilled, alas. You can ask psycopg2
> authors if they are going to fix that.
>
>    I will do more tests and include the code into the next release. I
> hope to release SQLObject 1.5 RSN.
>
> Oleg.
> --
>      Oleg Broytman            http://phdru.name/            ph...@ph...
>            Programmers don't die, they just GOSUB without RETURN.
>
>
> ------------------------------------------------------------------------------
> October Webinars: Code for Performance
> Free Intel webinars can help you accelerate application performance.
> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
> from
> the latest Intel processors and coprocessors. See abstracts and register >
> http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk
> _______________________________________________
> sqlobject-discuss mailing list
> sql...@li...
> https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
>

Hello!

I'm pleased to announce version 1.5.0, the first stable release of branch
1.5 of SQLObject.


What's new in SQLObject
=======================

Features & Interface
--------------------

* Helpers for class Outer were changed to lookup columns in table's
  declarations.

* Support for Python 2.4 is declared obsolete and will be removed
  in the next release.

Minor features
--------------

* When a PostgresConnection raises an exception the instance has
  code/error attributes copied from psycopg2's pgcode/pgerror attributes.

* Encode unicode enum values to str.

* Removed setDeprecationLevel from the list of public functions.

* A number of fixes for tests.

Bugfixes
--------

* A bug was fixed in DBConnection.close(); close() doesn't raise
  an UnboundLocalError if connection pool is empty.

* Fixed parameters for pymssql.

Documentation
-------------

* GNU LGPL text was added as docs/LICENSE file.

* Old FSF address was changed to the new one.


Contributors for this release are Patrick Gendron, Rhubarb Sin,
Neil Muller, Robert Ayrapetyan, Gert Burger and Francisco Chiotta.


For a more complete list, please see the news:
http://sqlobject.org/News.html


What is SQLObject
=================

SQLObject is an object-relational mapper.  Your database tables are described
as classes, and rows are instances of those classes.  SQLObject is meant to be
easy to use and quick to get started with.

SQLObject supports a number of backends: MySQL, PostgreSQL, SQLite,
Firebird, Sybase, MSSQL and MaxDB (also known as SAPDB).


Where is SQLObject
==================

Site:
http://sqlobject.org

Development:
http://sqlobject.org/devel/

Mailing list:
https://lists.sourceforge.net/mailman/listinfo/sqlobject-discuss

Archives:
http://news.gmane.org/gmane.comp.python.sqlobject

Download:
https://pypi.python.org/pypi/SQLObject/1.5.0

News and changes:
http://sqlobject.org/News.html

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.

On Sat, Sep 28, 2013 at 01:22:26PM +0400, Oleg Broytman <ph...@ph...> wrote:
>    I returned from the vacation. Will test if psycopg w/o SQLObject
> returns the code/error.

   The following program:

from decimal import Decimal
import psycopg2

def report_error(e):
    print e.__class__.__name__, e.pgcode, e.pgerror
    raise SystemExit

try:
    con = psycopg2.connect(database="test")
except psycopg2.Error, e:
    report_error(e)

cur = con.cursor()

try:
    cur.execute('SELECT * FROM test ORDER BY id')
except psycopg2.Error, e:
    report_error(e)

   prints

OperationalError None None

   when it cannot connect to the database and prints

ProgrammingError 42P01 ERROR:  relation "test" does not exist

   when the database exists but there is no table "test". I.e., psycopg2
doesn't return proper code/message for an OperationalError, so your
original request cannot be fullfilled, alas. You can ask psycopg2
authors if they are going to fix that.

   I will do more tests and include the code into the next release. I
hope to release SQLObject 1.5 RSN.

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.

Hi!

On Thu, Sep 26, 2013 at 11:52:08AM -0300, Francisco Chiotta <fra...@gm...> wrote:
> It doesn't work for me either. I have installed the psycopg 2.5.1 (the
> latest), and I always get None if the authentication fails or the server is
> not working in the given IP. I don't know why.
> 
> 2013/9/15 Oleg Broytman <ph...@ph...>
> 
> > On Fri, Sep 13, 2013 at 11:40:09AM -0300, Francisco Chiotta <
> > fra...@gm...> wrote:
> > > It would be great.  Sure, I can test the results.
> >
> >    The patch is attached. The code and error string are available:
> >
> > try:
> >     do_something()
> > except OperationalError, e:
> >     print e.args[0].code
> >     print e.args[0].error
> >
> >    Doesn't work for me, though -- pgcode/pgerror are always None.
> > Perhaps I'm using old version of psycopg2 -- 2.4.5.

   I returned from the vacation. Will test if psycopg w/o SQLObject
returns the code/error.

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.

It doesn't work for me either. I have installed the psycopg 2.5.1 (the
latest), and I always get None if the authentication fails or the server is
not working in the given IP. I don't know why.


2013/9/15 Oleg Broytman <ph...@ph...>

> On Fri, Sep 13, 2013 at 11:40:09AM -0300, Francisco Chiotta <
> fra...@gm...> wrote:
> > It would be great.  Sure, I can test the results.
>
>    The patch is attached. The code and error string are available:
>
> try:
>     do_something()
> except OperationalError, e:
>     print e.args[0].code
>     print e.args[0].error
>
>    Doesn't work for me, though -- pgcode/pgerror are always None.
> Perhaps I'm using old version of psycopg2 -- 2.4.5.
>
> Oleg.
> --
>      Oleg Broytman            http://phdru.name/            ph...@ph...
>            Programmers don't die, they just GOSUB without RETURN.
>

Ten days vacation. Perhaps I'll be online from time to time...

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.

Hi!

On Tue, Sep 17, 2013 at 08:19:13PM +0200, to...@ce... wrote:
> To work with foo = RelatedJoin('Song') we use .foo, .addSong and .removeSong. But those names are configurable. How do I get them when all I know is "foo"?
> 
> One way seems to be sqlmeta.joins[0].performJoin, .add and .remove. Is that an internal API, and subject to change?

   I have less and less time to work on SQLObject so until someone
replaces me as the developer there will be no major changes for quite
some time.

> How do I get the list index?

   By looking up the join object you're interested in:

class Test1(SQLObject):
    test2 = RelatedJoin('Test2')

class Test2(SQLObject):
    test1 = RelatedJoin('Test1')

for j in Test2.sqlmeta.joins:
    print j.joinMethodName, j.otherClassName, j.addRemoveName

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.

To work with foo = RelatedJoin('Song') we use .foo, .addSong and .removeSong. But those names are configurable. How do I get them when all I know is "foo"?

One way seems to be sqlmeta.joins[0].performJoin, .add and .remove. Is that an internal API, and subject to change? How do I get the list index?

On Fri, Sep 13, 2013 at 11:40:09AM -0300, Francisco Chiotta <fra...@gm...> wrote:
> It would be great.  Sure, I can test the results.

   The patch is attached. The code and error string are available:

try:
    do_something()
except OperationalError, e:
    print e.args[0].code
    print e.args[0].error

   Doesn't work for me, though -- pgcode/pgerror are always None.
Perhaps I'm using old version of psycopg2 -- 2.4.5.

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.

Hi!

On Fri, Sep 13, 2013 at 07:17:35PM +0200, JM Germeshuysen <j.g...@gm...> wrote:
> Does anybody know if there are any support for the new FDB driver in 
> SQLObject, as far as I can tell it only supports the outdated kinterbasedb.

   The code that you see is the only code. There is no magic, no hidden
code. If you see only kinterbasedb -- that's all what's supported. But
you can send a patch to extend SQLObject. Do you want to help?

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.

Hi,

Does anybody know if there are any support for the new FDB driver in 
SQLObject, as far as I can tell it only supports the outdated kinterbasedb.

TIA

On Fri, Sep 13, 2013 at 11:40:09AM -0300, Francisco Chiotta <fra...@gm...> wrote:
> It would be great.  Sure, I can test the results.
> 
> 2013/9/12 Oleg Broytman <ph...@ph...>
> >    Do you want me to provide e.code for psycopg2? I'll ask you to test
> > the result.

   I will try to do that before taking a vacation next week. If not --
I'll be back at the 28th of September.

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.

It would be great.  Sure, I can test the results.

Cheers


2013/9/12 Oleg Broytman <ph...@ph...>

> On Thu, Sep 12, 2013 at 04:43:09PM -0300, Francisco Chiotta <
> fra...@gm...> wrote:
> > Ok, I thought that It was possible to recover an error code or something
> > like that.
>
>    psycopg2 provides some information:
> http://initd.org/psycopg/docs/module.html#exceptions
> http://initd.org/psycopg/docs/errorcodes.html#module-psycopg2.errorcodes
>    but SQLObject loose that information when it converts psycopg2's
> exceptions to SQLObject's exceptions. I can try to extract that
> information but only for SQLObject 1.5 (which is currently in
> pre-release state) and only for Postgres/psycopg2 (PySQLite doesn't
> return error codes, only strings; exceptions from MySQL already provide
> error code in e.code attribute).
>    Do you want me to provide e.code for psycopg2? I'll ask you to test
> the result.
>
> Oleg.
> --
>      Oleg Broytman            http://phdru.name/            ph...@ph...
>            Programmers don't die, they just GOSUB without RETURN.
>
>
> ------------------------------------------------------------------------------
> How ServiceNow helps IT people transform IT departments:
> 1. Consolidate legacy IT systems to a single system of record for IT
> 2. Standardize and globalize service processes across IT
> 3. Implement zero-touch automation to replace manual, redundant tasks
> http://pubads.g.doubleclick.net/gampad/clk?id=51271111&iu=/4140/ostg.clktrk
> _______________________________________________
> sqlobject-discuss mailing list
> sql...@li...
> https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
>

On Thu, Sep 12, 2013 at 04:43:09PM -0300, Francisco Chiotta <fra...@gm...> wrote:
> Ok, I thought that It was possible to recover an error code or something
> like that.

   psycopg2 provides some information:
http://initd.org/psycopg/docs/module.html#exceptions
http://initd.org/psycopg/docs/errorcodes.html#module-psycopg2.errorcodes
   but SQLObject loose that information when it converts psycopg2's
exceptions to SQLObject's exceptions. I can try to extract that
information but only for SQLObject 1.5 (which is currently in
pre-release state) and only for Postgres/psycopg2 (PySQLite doesn't
return error codes, only strings; exceptions from MySQL already provide
error code in e.code attribute).
   Do you want me to provide e.code for psycopg2? I'll ask you to test
the result.

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.