mbsync vs offlineimap for gmail

[Philip DeBoer]

I am considering the use of an offline mail store for GMail, which I have not done before. I am looking at mbsync and offlineimap. Reports suggest that mbsync is faster and uses less memory than offlineimap.

Does mbsync support GMail's labels?

I have a hierarchy of gmail labels (eg a/b), with some mail tagged at interior nodes ("label:a") and some at the leaf nodes ("label:a/b"). Some messages have multiple labels. Will mbsync pull down this information (possibly as an X-Label header)? For multiply labeled messages, can it store a single copy (eg, maybe I can just sync [Gmail]\All Mail?). If it matters, I plan to try it with mu4e in emacs.

Part of the reason I'm asking is that offlineimap does not seem to support this out of the box, although there do seem to be patches that will make it work. It is not clear to me what support mbsync provides for gmail's IMAP extensions or non-standard implementation.

Thanks!

[Oswald Buddenhagen]

Post by Philip DeBoer
I am considering the use of an offline mail store for GMail, which I have not done before. I am looking at mbsync and offlineimap. Reports suggest that mbsync is faster and uses less memory than offlineimap.
Does mbsync support GMail's labels?

not really; mbsync has no support for custom flags.

gmail uses the labels to create virtual folders, and this is exposed to
imap. as mbsync supports hierarchical mailboxes, it will simply sync the
view, which may be close enough to what you need. but overlapping labels
definitely won't work.

[Philip DeBoer]

OK. Thanks for the quick response.

If I also have some unlabeled mail, I would need to download All Mail to see it, right? And then have duplicate copies of most mail?

What would be the main challenges in adding support for custom flags? Would they be difficult to fetch? Has anyone tried to add them?

Post by Oswald Buddenhagen

Post by Philip DeBoer
I am considering the use of an offline mail store for GMail, which I have not done before. I am looking at mbsync and offlineimap. Reports suggest that mbsync is faster and uses less memory than offlineimap.
Does mbsync support GMail's labels?

not really; mbsync has no support for custom flags.
gmail uses the labels to create virtual folders, and this is exposed to
imap. as mbsync supports hierarchical mailboxes, it will simply sync the
view, which may be close enough to what you need. but overlapping labels
definitely won't work.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
isync-devel mailing list
https://lists.sourceforge.net/lists/listinfo/isync-devel

[Oswald Buddenhagen]

Post by Philip DeBoer
If I also have some unlabeled mail, I would need to download All Mail
to see it, right? And then have duplicate copies of most mail?

yeah. a better approach would be defining a low-priority fallback label.
i presume gmail supports setting this via a filter rule.

Post by Philip DeBoer
What would be the main challenges in adding support for custom flags?
Would they be difficult to fetch? Has anyone tried to add them?

well, it's work, and there are interoperability constraints.
there was a cross-project thread back in december, but i dropped the
ball.

[Philip DeBoer]

Post by Oswald Buddenhagen

Post by Philip DeBoer
What would be the main challenges in adding support for custom flags?
Would they be difficult to fetch? Has anyone tried to add them?

well, it's work, and there are interoperability constraints.
there was a cross-project thread back in december, but i dropped the
ball.

OK, those aren't fundamental architecture issues then. So what would be the steps? We can use the offlineimap approach as an example.

Let's start simple: assume static labels.
* Ask google for the labels of a message. This is documented at https://developers.google.com/gmail/imap/imap-extensions. It should be possible to download labels for multiple messages (eg, whole mailbox) at once.
* Define a new internal header, like X-Keywords (or X-GM-LABELS). Populate and save this when writing messages to disk.
* Only download the All Mail folder, avoiding duplicates. Let an auxiliary program (eg, mu) construct the mailboxes by searching on our new X-Keywords.

Then allow for labels that change:
* On syncing, get all labels for all messages. Compare to the X-Keywords on disk and see if any have changed on the remote end.
This step requires matching up the messages, presumably uid. Will this be fast? Does mbsync have a database or hashtable?
* When changing the label locally, submit this information to gmail using the STORE command.

Do you have a good strategy for testing during development? Are there other IMAP configurations besides google that we need to avoid mangling as we add new functionality for this specific use case?

Does this seem like a feasible approach to you? You mentioned that this came up in December --- is there a thread that might have additional ideas to consider?

[Philip DeBoer]

Hi Oswald, I wanted to follow up on this. I've sketched some code on a new branch in my forked repo. It compiles, but I haven't tested it yet. Does it look reasonable to you, with the right data structures and imap interface?

The next step for me is probably to set up a test GMail account and play with it. Are there any gotchas to be concerned about?

Philip

Post by Philip DeBoer

Post by Oswald Buddenhagen

Post by Philip DeBoer
What would be the main challenges in adding support for custom flags?
Would they be difficult to fetch? Has anyone tried to add them?

well, it's work, and there are interoperability constraints.
there was a cross-project thread back in december, but i dropped the
ball.

OK, those aren't fundamental architecture issues then. So what would be the steps? We can use the offlineimap approach as an example.
Let's start simple: assume static labels.
* Ask google for the labels of a message. This is documented at https://developers.google.com/gmail/imap/imap-extensions. It should be possible to download labels for multiple messages (eg, whole mailbox) at once.
* Define a new internal header, like X-Keywords (or X-GM-LABELS). Populate and save this when writing messages to disk.
* Only download the All Mail folder, avoiding duplicates. Let an auxiliary program (eg, mu) construct the mailboxes by searching on our new X-Keywords.
* On syncing, get all labels for all messages. Compare to the X-Keywords on disk and see if any have changed on the remote end.
This step requires matching up the messages, presumably uid. Will this be fast? Does mbsync have a database or hashtable?
* When changing the label locally, submit this information to gmail using the STORE command.
Do you have a good strategy for testing during development? Are there other IMAP configurations besides google that we need to avoid mangling as we add new functionality for this specific use case?
Does this seem like a feasible approach to you? You mentioned that this came up in December --- is there a thread that might have additional ideas to consider?

[Oswald Buddenhagen]

Post by Philip DeBoer
Hi Oswald, I wanted to follow up on this.

well, i obviously can't be bothered to. sorry, just no motivation to
deal with this stuff currently. you can just do whatever, and i may pick
it up in a few months.

here's a braindump of stuff that seems relevant:
- it's not clear to me why google made an own imap extension instead of
just using regular imap message flags. the X-Keywords you're talking
of are actually regular (custom) flags, at least by the precedent of
libc-client.
- using a header for something flag-like is an utter performance
nightmare with maildir. flags should be encoded in the file name as
per the maildir spec, and there needs to be an additional file which
defines the mapping (not in the maildir spec, which is obviously a
major interoperability issue).
- i'm not that interested in virtual folders, but into easy flag
visibility and toggling in mutt. this is at odds with the previous
point, as mutt uses a custom X-Labels thingie for that.

i find the fact that there is no standard and that mbsync won't be
realistically able to set one rather demotivating.

did you read that thread i pointed out? did you do followup research to
see what they came up with?

Post by Philip DeBoer
I've sketched some code on a new branch in my forked repo. It compiles, but I haven't tested it yet. Does it look reasonable to you, with the right data structures and imap interface?
The next step for me is probably to set up a test GMail account and play with it. Are there any gotchas to be concerned about?
Philip

Post by Philip DeBoer

Post by Oswald Buddenhagen

Post by Philip DeBoer
What would be the main challenges in adding support for custom flags?
Would they be difficult to fetch? Has anyone tried to add them?

well, it's work, and there are interoperability constraints.
there was a cross-project thread back in december, but i dropped the
ball.

OK, those aren't fundamental architecture issues then. So what would be the steps? We can use the offlineimap approach as an example.
Let's start simple: assume static labels.
* Ask google for the labels of a message. This is documented at https://developers.google.com/gmail/imap/imap-extensions. It should be possible to download labels for multiple messages (eg, whole mailbox) at once.
* Define a new internal header, like X-Keywords (or X-GM-LABELS). Populate and save this when writing messages to disk.
* Only download the All Mail folder, avoiding duplicates. Let an auxiliary program (eg, mu) construct the mailboxes by searching on our new X-Keywords.
* On syncing, get all labels for all messages. Compare to the X-Keywords on disk and see if any have changed on the remote end.
This step requires matching up the messages, presumably uid. Will this be fast? Does mbsync have a database or hashtable?
* When changing the label locally, submit this information to gmail using the STORE command.
Do you have a good strategy for testing during development? Are there other IMAP configurations besides google that we need to avoid mangling as we add new functionality for this specific use case?
Does this seem like a feasible approach to you? You mentioned that this came up in December --- is there a thread that might have additional ideas to consider?