feat: Add local gocryptfs support

[daviewales]

Blocked by

• refactor: Separate code about Manage profile dialog #1865
• Test Mount API #2039

This is a rebase of Germar Reitze's gocrypt branch on the latest dev branch.

It seems to work, so I thought we should try to merge it as a base for future gocryptfs work.

I've tested backing up a test file, deleting it, then restoring it.

Related to #1734.

[buhtz]

Wow that is a big step. Awesome!

Do we need the latest GoCryptFS when running TavisCI? Are there known Issues with gocryptfs from the Ubuntu repos?

[daviewales]

I suspect that we don't need the latest version. I tested with the version in the latest Linux Mint, and it seemed to work.

[daviewales]

The code fetching the latest version was already in Germar's original branch. I haven't made many changes here, other than fixing merge conflicts.

[buhtz]

Please let me know if you prefer me to solve this merge conflicts.

[daviewales]

I'll do another rebase, and switch Travis to fetch the packaged version of gocryptfs rather than the latest release while I'm at it.

[daviewales]

I've rebased on the latest dev.
I also added a quick patch to the qt/configure script to include the new qt/manageprofiles directory when installing.

I confirmed that I can still backup a file, delete it, then restore it.

[buhtz]

When my refactoring work on the manage profiles dialog is finished I think about a release candidate and release 1.5.3.
After this we can merge your PR, after intense testing of course, and later release it as 1.6.0 somewhere around Christmas or New Year.

[buhtz]

FYI: I do plan a 1.5.3 release in the near future. After that release I would target your PR for a 1.6.0 release.

[daviewales]

I'll apply the suggestions above when I get a chance. There are also a few lint errors, so I'll fix those up too.

[daviewales]

The remaining lint errors are because there is similarity between the encfs and gocryptfs modules... Perhaps this can be ignored?

[buhtz]

The remaining lint errors are because there is similarity between the encfs and gocryptfs modules... Perhaps this can be ignored?

Usually I would say "no" and recommend to create a base class to avoid code duplication. But we will remove encfs, so I see no problem to add an ignore clause for pylint.

# pylint: disable=duplicate-code

Not sure if this need to be place in both files.

[daviewales]

Note to self: I need to test setting up a new gocryptfs backup from scratch. I've been tested this branch with one which is already configured.

EDIT: I've tested setting up a new backup profile from scratch, and the dialog needs some tweaking as it's missing a few inputs. I'll update it to match the EncFS local dialog.

[daviewales]

No guarantees for the April deadline, but I'll see what I can do. I'm trying a more cautious incremental approach to avoid the issues above. My plan is to submit the following changes as three separate PRs:

• Create tests for encfstools (WIP: Test Mount API #2039)
• Update mount API with explicit init_backend method (and ensure it passes the encfstools tests)
• Add gocryptfs support

[buhtz]

Hello David,
the 1.5.4 release is published. Therefore, the road is free now.

I don't have the overview currently. To my understanding everything related to gocrypt is work in progress. Am I right?

Let me know if I should take a look on something or test something. I usually read all notifications from Microsoft GitHub. But feel free to contact me via email or mailing list. Your task has priority now.

Regards,
Christian

[daviewales]

Thanks @buhtz. Everything is WIP currently. The current priority is to get some good tests for the existing EncFS and MountControl APIs. This is important, as this gocryptfs branch, which was rebased from Germar's old branch, changes the MountControl API to add an explicit init_backend method. However, this broke the existing EncFS module, and made me feel that we need to strongly test the current working state before changing the MountControl API.

I haven't had much bandwidth lately to look at this, but I do hope to get back into it when I get a chance.
If you have a chance to add some tests to #2039, it would be very helpful.

[buhtz]

Some more details in this comment: #2039 (comment)

[buhtz]

Hello David,
I just made some minor adjustments and fixes in the GUI code but not to your gocryptfs.

I still lack of deeper understanding but after a first test, the local gocryptfs-encrypted backup seems to work.

Looks better than I thought. Did you remember any other important problems with the current state, we need to tackle before merging?

Of course, I'll review the modifications line by line before merging anything.

Regards,
Christian

[daviewales]

@buhtz, the main problem with this branch is that it breaks EncFS, because it changes the MountControl API to require an init_backend() method, which is explicitly called when needed. This conflicts with the existing MountControl API, which handles initialisation inline.

Some details above:
#1897 (comment)

Previously, I was trying to update the EncFS code to use this new API.
However, I now feel that it's probably better to leave the EncFS code unchanged, and update this branch to work with the existing MountControl api. (See the current EncFS code for an example of how this would work.)

[buhtz]

it breaks EncFS

Ah, stupid me. 😄 Have not tested this case. ;)
Thanks for pointing me to it. ❤️

[buhtz]

Mhm.... OK, do I understand that right? The init_backend() is called only when creating a profile to initialize/setup the encrypted directory?

I checked the EncFS related code. In the current version there is an init_backend() added in this PR. This method does not exist in the latest dev version, but EncFS profiles do work in dev. Why? I am assuming that also an encfs profile somehow need to initialize its destination folder. But where in the dev code (without having init_backend()) does this happen?

---

EDIT: Not sure about it, but I am assuming it happens in Encfs_mount._mount() kind of implicit with this two lines:

        if not self.isConfigured():
            encfs += ['--standard']

The --standard switch makes EncFS initializing the directory before doing something else.

The Encfs_mount.init_backend() in this PR looks like a near copy of Encfs_mount._mount().

Why not just leave Encfs_mount.init_backend() empty?

At the current point I am not interested in fancy and high level super duper mega developer designs. I just want to make it work. 🤣 And EncFS is on the way to be removed soon. So I have no problems keeping some awkward workarounds until its final removal.

[daviewales]

Yes, I think most (all?) of the EncFS backend init logic is in these lines:

backintime/common/encfstools.py

Lines 77 to 78 in a8dcbff

	if not self.isConfigured():
	encfs += ['--standard']

Note that the isConfigured method does more than just check if it's configured. It also confirms through the user interface whether the user wants to create a new EncFS config, and if so, asks them to confirm the password.
It can also raise a few exceptions for various reasons.

I have a feeling that I tried just leaving init_backend() empty, but I can't remember whether it worked, or if there was some problem. Might be worth trying that again.

[buhtz]

Thank you for your feedback. OK, I'll try to leave it empty.