Bug #64

subfolders in folders bug

Added by Wim Hoekman 177 days ago. Updated 69 days ago.

Status :Closed Start :04/17/2008
Priority :Low Due date :
Assigned to :- % Done :

100%
Category :-
Target version :-
Resolution :

Description

offlineimap fails with the following error:

Thread 'Account sync Test' terminated with exception:
Traceback (most recent call last):
  File "/var/tmp/offlineimap/offlineimap/threadutil.py", line 153, in run
    Thread.run(self)
  File "/usr/local/lib/python2.4/threading.py", line 422, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/var/tmp/offlineimap/offlineimap/accounts.py", line 114, in syncrunner
    self.sync()
  File "/var/tmp/offlineimap/offlineimap/accounts.py", line 148, in sync
    remoterepos.syncfoldersto(localrepos, [statusrepos])
  File "/var/tmp/offlineimap/offlineimap/repository/Base.py", line 157, in syncfoldersto
    copyfolder.makefolder(key)
  File "/var/tmp/offlineimap/offlineimap/repository/LocalStatus.py", line 45, in makefolder
    os.makedirs(os.path.dirname(filename), 0700)
  File "/usr/local/lib/python2.4/os.py", line 159, in makedirs
    mkdir(name, mode)
OSError: [Errno 17] File exists: '/var/tmp/wimh/.offlineimap/Account-Test/LocalStatus/Telefonkonferenz/Finance'

As far as I could see, the problem is that the statusflags of a folder are stored in a file with the same name as the folder itself.

If this statusfile is already created, it is no longer possible to create subdirectories in that folder.

I guess the statusfile has to be named something else, like: foldername + ".stat".
version: 5.99.11

Associated revisions

Revision bd1d1010e77fb10f243fea8be1bbaf2e0d2c8b65
Added by John Goerzen 70 days ago

Now create cur,new,tmp in '.' even if existsok is false

Otherwise, there was a race where sometimes subfolders would get created
first in the sep = '.' scneario

fixes #64

History

04/17/2008 11:16 AM - Wim Hoekman

The above error message is a result of having the following subfolders inside 'Finance':

Bestelling
Finance
PurchaseOrders

04/17/2008 01:12 PM - John Goerzen

  • Priority changed from High to Low
  • Status changed from New to Assigned

No matter what name I pick, I suppose someone could have files with that name.

I am unsure how to fix this in a backwards-compatible way, either. I'll have to think about it.

OfflineIMAP has had the same LocalStatus storage method for years, and you're the first one to ever have this problem. I'm somewhat inclined to just document this bug in the manual...

04/18/2008 12:19 AM - Wim Hoekman

As far as I can see, current method does not allow a folder to have messages, AND subfolders at the same time.

Or am I wrong?

04/18/2008 12:55 AM - Wim Hoekman

Simple test:

create an imap folder named: bla1
copy some messages in it.
create a imap folder inside bla1, name it bla2.
copy some messages in it.

run imapoffline.
the moment it encounters the messages in folder bla2, it will create a directory named bla1, containing a status file for folder bla2, named bla2.
Then it tries to sync folder bla1. it will try to create a status file with the name 'bla1'. However, there's already a folder created with that same name.

As far as I can see it now, it will never run if you have a folder, which contains both messages and subfolders.

04/18/2008 04:36 AM - Wim Hoekman

ok, I think I've found the real problem, but am not sure how/why.

I've created the folders 'bla1' and 'bla2' as per previous message.
First, I've changed LocalStatus.py in 'folder' and 'repository', so it will create a status file with the name of the folder, with '.stat' appended.

Now offlineimap completes without errors.
Contents of LocalStatus folder (stripped down):

./bla1
./bla1/bla2.stat
./bla1.stat
./bla1.bla2.stat

Note the ./bla1/bla2.stat is located in a subfolder!
File ./bla1/bla2.stat is exactly the same as ./bla1.bla2.stat

By the way: I'm trying to sync from IMAP to IMAP (so no local maildir sync)

After cleaning up, and trying to sync from IMAP to local Maildir:

Contents of LocalStatus folder:
./bla1.stat
./bla1.bla2.stat

There's not bla1 subfolder anymore!
I guess this is the reason other people have not encountered this problem before.

After adding debug statements, I noticed that the difference is in repository/LocalStatus.py:getfoldername
for syncing subfolders IMAP <--> local Maildir, it is called with 'bla1.bla2'
for syncing subfolders IMAP <--> IMAP, it is called with 'bla1/bla2'.

So I've edited repository/LocalStatus.py, function getfolderfilename:
after the 2 re.sub lines, I've added another one:
    # replace all '/' with '.'
    foldername = re.sub( '/', '.', foldername)

right before returning it.

Right now, it seems to run fine (even with the '.stat' removed from the status filename)

Is this the correct fix? Or am I likely to encounter problems elsewhere?

04/18/2008 06:04 AM - Wim Hoekman

As a bonus, the above fix also fixed another issue I had:

when migrating a mailbox from IMAP <--> IMAP, messages that were deleted (but not purged) in the source IMAP server were copied to the target IMAP server, but lost it's 'deleted' status.

With above fix the message retains it's deleted status.

04/18/2008 03:22 PM - John Goerzen

I don't think that's the right fix for several reasons:

1) If you have a server folder named INBOX/foo/bar and another named INBOX/foo.bar, you will likely encounter corruption

2) Now we have a problem if there's a folder named INBOX/foo/bar and another named INBOX/foo/bar.stat

so I think this just moves the problem, not fixing it.

04/21/2008 01:14 AM - Wim Hoekman

Answers to your remarks:
2) As mentioned in my previous messages, I already removed the '.stat' suffix, as it was only necessary to let offlineimap finish.

1) I agree. However, I don't see that this is caused by my fix. I think that without my fix, you'll encounter the same problem.

In my situation, every folder which has subfolders AND messages would cause offlineimap to fail, not just the ones I mentioned.

In my very simple testcase, offlineimap resulted in the following contents of the LocalStatus folder:
(IMAP <--> IMAP)
./bla1
./bla1/bla2.stat
./bla1.stat
./bla1.bla2.stat

(my test appended the '.stat' suffix. Try to read it without '.stat' suffix. However, without '.stat' suffix offlineimap stops with an error, as you can't create folder 'bla1' if there's already a file 'bla1')

It seems that offlineimap tries to keep status of folder bla1/bla2 in bla1/bla2.stat, and bla1.bla2.stat.
Both files contain the same content. To me, this seems to be a bug.

With exact same test situation, but now IMAP <--> Maildir sync results in:
./bla1.stat
./bla1.bla2.stat

No subfolders anymore.
To me, it seems that the status of /bla1/bla2 folder is stored in a file bla1.bla2.
So in my fix I tried to replicate that same situation.

Can you tell me how the LocalStatus folder should look? Should it be a flat folder, without subfolders?
Or should it contain subfolders?
Which of the two LocalStatus contents is the correct one?

Can you confirm the above experience? Or is it still due to my unique situation?
Or am I doing something wrong?

Maybe you'll want to take this discussion offlist? In that case, contact me directly via email...

05/28/2008 06:44 AM - Wim Hoekman

I just used 5.99.15, the issue is fixed in this version.

Status can be set to fixed.

08/02/2008 09:26 PM - John Goerzen

  • % Done changed from 0 to 100
  • Status changed from Assigned to Closed

Also available in: Atom PDF