Attributes being deleted on .save()

[fiatjaf]

I have a piece of code that does the following:

        lsheet, rsheet = db.get(Sheet, {'_id': left['sheetId']}), \
                         db.get(Sheet, {'_id': right['sheetId']})
        print('saving %s with primary key %s.' % (lsheet.title, lsheet._id))
        db.save(lsheet)
        db.save(rsheet)
        print('saved.')

At the moment the first line is printed, the lsheet object has an _id (the pk is called _id in this class, that's working correctly in other parts of the script). But then the call fails with

Traceback (most recent call last):
  File "test.py", line 3, in <module>
    db = make_db('data/20160422.json')
  File "/home/fiatjaf/comp/hack-json/db.py", line 105, in make_db
    db.save(lsheet)
  File "/home/fiatjaf/comp/hack-json/venv/lib/python3.4/site-packages/blitzdb/backends/file/backend.py", line 459, in save
    serialized_attributes = self.serialize(obj.attributes)
  File "/home/fiatjaf/comp/hack-json/venv/lib/python3.4/site-packages/blitzdb/backends/base.py", line 159, in serialize
    output_obj[str(key) if convert_keys_to_str else key] = serialize_with_opts(value, embed_level=embed_level)
  File "/home/fiatjaf/comp/hack-json/venv/lib/python3.4/site-packages/blitzdb/backends/base.py", line 150, in <lambda>
    serialize_with_opts = lambda value,*args,**kwargs : self.serialize(value,*args,convert_keys_to_str = convert_keys_to_str,autosave = autosave,for_query = for_query, **kwargs)
  File "/home/fiatjaf/compk/hack-json/venv/lib/python3.4/site-packages/blitzdb/backends/base.py", line 161, in serialize
    output_obj = list(map(lambda x: serialize_with_opts(x, embed_level=embed_level), obj))
  File "/home/fiatjaf/comp/hack-json/venv/lib/python3.4/site-packages/blitzdb/backends/base.py", line 161, in <lambda>
    output_obj = list(map(lambda x: serialize_with_opts(x, embed_level=embed_level), obj))
  File "/home/fiatjaf/comp/hack-json/venv/lib/python3.4/site-packages/blitzdb/backends/base.py", line 150, in <lambda>
    serialize_with_opts = lambda value,*args,**kwargs : self.serialize(value,*args,convert_keys_to_str = convert_keys_to_str,autosave = autosave,for_query = for_query, **kwargs)
  File "/home/fiatjaf/comp/hack-json/venv/lib/python3.4/site-packages/blitzdb/backends/base.py", line 175, in serialize
    obj.save(self)
  File "/home/fiatjaf/comp/hack-json/venv/lib/python3.4/site-packages/blitzdb/document.py", line 390, in save
    return backend.save(self)
  File "/home/fiatjaf/comp/hack-json/venv/lib/python3.4/site-packages/blitzdb/backends/file/backend.py", line 456, in save
    if hasattr(obj, 'pre_save') and callable(obj.pre_save):
  File "/home/fiatjaf/comp/hack-json/venv/lib/python3.4/site-packages/blitzdb/document.py", line 175, in __getattribute__
    self.revert()
  File "/home/fiatjaf/comp/hack-json/venv/lib/python3.4/site-packages/blitzdb/document.py", line 427, in revert
    raise self.DoesNotExist("No primary key given!")
blitzdb.document.DoesNotExist: DoesNotExist(Record)

The error happens at https://github.com/adewes/blitzdb/blob/master/blitzdb/document.py#L490. I put some print() calls there and inside the def pk(self) method and discovered that the object was empty. A call to self.keys() returns an empty list, that's why self.pk returns None. I don't know at which time the object becomes empty and couldn't find it myself.

I must say that basic saving, fetching and filtering is working well in many other parts of the script.

[fiatjaf]

In fact, no, there are other bizarre problems happening in other parts of the application.

For example:

>>> r
Record({})
>>> r.__dict__                                                                            
{'_autoload': True, 'pk': '564a7c5dac00f40300a324ab', '_default_backend': <blitzdb.backends.file.backend.Backend object at 0x7f109d5a9320>, '_attributes': {}, '_lazy': True, 'embed': False}
>>> r.pk
>>> r.attributes
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/fiatjaf/comp/hack-json/venv/lib/python3.4/site-packages/blitzdb/document.py", line 164, in __getattribute__
    self.revert()
  File "/home/fiatjaf/comp/hack-json/venv/lib/python3.4/site-packages/blitzdb/document.py", line 424, in revert
    raise self.DoesNotExist("No primary key given!")
blitzdb.document.DoesNotExist: DoesNotExist(Record)

[adewes]

Thanks for reporting this! Can you send me a full code example of your Record class? When I do the following:

import blitzdb
doc = blitzdb.Document()
doc.attributes

it works as expected.

In your example I see that you set the pk value of your record(through initialize?), which tells BlitzDB that the object should exist in the database, so when you call attributes it tries to fetch it from there, which fails as there is no object with that given primary key.

It's hard to tell if it's a BlitzDB bug or some problem with your Record class, but if you can provide a more complete example I think I'd be better able to help you.

[fiatjaf]

The issue has probably something to do with the size of the documents embedded in others. See this code:

from blitzdb import Document
from blitzdb import FileBackend

class Sheet(Document):
    class Meta(Document.Meta):
        primary_key = '_id'

class Record(Document):
    class Meta(Document.Meta):
        primary_key = '_id'

db = FileBackend('/tmp/t.db')

sheet = Sheet({
    '_id': '3i24i3l',
    'something': 'papapa'
})

db.save(sheet)
db.commit()

sheet = db.get(Sheet, {'_id': '3i24i3l'})

sheet.records = []
for i in range(18):
    record = Record({
        '_id': '%s' % i,
        'prop': 'parara'
    })
    sheet.records.append(record)
    record.sheets = [sheet]

db.save(sheet)
db.save(record)
db.commit()

# ~

for s in db.filter(Sheet, {}):
    print('sheet: ', s)
    print('sheet: ', s.__dict__)
    print('sheet: ', s.pk)
    print('sheet: ', s.attributes)
    for r in s.records:
        print('record: ', r)
        print('record: ', r.__dict__)
        print('record: ', r.pk)
        print('record: ', r.attributes)

        r.pou = 'pôu'
        db.save(r)

This is the full runnable code.

This code fails almost always in my machine. That number 18 there, if we change it to 300 or something bigger, then the code always fails, but if we change it to 1 or 2 then it never fails.

It fails when printing r.attributes, which means a Record r actually is saved, but it has a blank .pk, and so its .attributes cannot be fetched, then the program errors

Traceback (most recent call last):
  File "iso.py", line 48, in <module>
    print('record: ', r.attributes)
  File "/home/fiatjaf/comp/hack-json/venv/lib/python3.4/site-packages/blitzdb/document.py", line 164, in __getattribute__
    self.revert()
  File "/home/fiatjaf/comp/hack-json/venv/lib/python3.4/site-packages/blitzdb/document.py", line 425, in revert
    obj = self._default_backend.get(self.__class__, {self.get_pk_name(): self.pk})
  File "/home/fiatjaf/comp/hack-json/venv/lib/python3.4/site-packages/blitzdb/backends/file/backend.py", line 508, in get
    raise cls.DoesNotExist
blitzdb.document.DoesNotExist: DoesNotExist(Record)

This is the exact same situation that is happening in the first message of this issue thread.

[adewes]

Thanks for reporting this @fiatjaf , I'll have a look at it this week!

[adewes]

Hey @fiatjaf , I can reproduce the issue and I'm working on a fix. Currently arbitrary id names are not yet fully supported, I'm working on it though.

[fiatjaf]

Oh, should I be using integers? This would make it work? That would be ok, if I could know about it before, now I kind of gave up this project, so please don't do any unplanned hotfixes specific to this issue.

[adewes]

No I think the problem is that the FileBackend does not yet support having a different name (_id) for the primary key field. I will write some unit tests for this and implement the behavior as intended in the coming days.

[nickjpg]

Any progress on this?

Just ran into this bug on our installation of gitlab 8.10.4. Populate a "/var/opt/gitlab/git-data/repository//.git/info/attributes" file and after checking in an update the 'attributes' file is deleted and upload changes specified by 'attributes' not completed.