Add ability to Squash Changes when merging

[arthanson]

Fixes: #198

This squashes (collapses) all Changelogs (for a given object) in a branch to a single Change:

• CREATE + (any updates) + DELETE = skip entirely
• (anything other than CREATE) + DELETE = DELETE
• CREATE + UPDATEs = CREATE
• multiple UPDATEs = UPDATE

The old (iterative) and new (collapse) merge strategy are both supported (the default is the old iterative approach). There is a select-box on merge to enable the squash merge:

Because ObjectChanges can reference other objects (FK) it has to do dependency ordering: If an object references something that is created, the create has to come first, etc..

This will not fix 100% of the issues, but the goal is that the end user can fairly easily fix the branch (by either deleting a problem Object or modifying it's data) and be able to get the branch to merge.

The biggest thing this fixes is Where you have an Object that causes a Constraint error. (Site with a slug that conflicts with one in main). Currently using the replay of changelogs you cannot fix this as you will always try to replay the ObjectcChange with the conflicting state. With collapse migrations, the intermediate state of that caused the conflict is skipped.

The major thing this can't fix is the swap case: You have a Module Bay with two modules at position 1 and 2 and you swap these (move 1 to 3, 2 to 1, 3 to 2) as with collapse you miss the move to position 3 and no matter which one you try to merge you will run into a constraint error. But, you can easily fix this by temporarily modifying the data before merge then merging.

Note: I looked at doing partial collapse (i.e. only collapsing some of the changes) but quickly ran into issues because of the constraint issues mentioned above and that Objects can reference other objects which means you have to do a dependency change and at that point it is just cleaner to collapse all the migrations as you have to do all the work anyways and it gets messy figuring out what you can partially collapse.

Step1: For collapse you want to take the initial objects prechange_data, and then you want to collapse all the objects postchange_data down into one so later field modifications overwrite older ones.

Note: because you are collapsing you sort of loose the time sequence (do you take it from the first change or the last - which actually depends). So barring dependencies we generally do DELETEs, UPDATEs then CREATEs (within those in time order)

Step 2: For dependency ordering it matters what type of operation you are doing. For a create you care about if postchange_data from another object refers to it, but for a delete you care if the prechange_data for another object refers to it. Note: we only care if these references are to other objects being modified in this branch. If it is a reference to something in main we don't need to track the dependency.

Step 3: For actually doing the dependency ordering a given Change can reference more then one FK so we use an algorithm that handles this and have to have cycle detection, which hopefully shouldn't happen (A references B, references C, references A) but need to guard against it.

Note: There is an edge case with nullable FK fields and having a created item refer to each other (Circuit / CircuitTermiantions apparently do this) If we detect this we split the create back into a create (with the nulled fields) and an update setting those fields.

Note: All of these new ObjectChange are in-memory, we don't actually modify the existing ObjectChange in the database. Because of this we can't use the existing apply / revert code as those work on actual ObjectChange records, so we have new functions similar to those.

_build_fk_dependency_graph has very similar code for each type of Change, but it is just dissimilar enough I didn't see a good way of refactoring it to be more DRY without increasing the complexity. I also created some sub-functions which are only called once (like _build_fk_dependency_graph) but it encapsulates the given functionality and makes it far easier to understand IMHO.

Right now there is a lot of logging that will appear in the Merge job log making it fairly big. This info is useful for debugging and it was decided to error on the side of too-much info rather than paring it down. We can do that later after it has been out and proven itself.

Note: From review changes - updated so squash revert / merge use the requesting user as the user for the ChangeLogs.

[jnovinger]

@arthanson, looks like you have a CI failure due to some lingering merge conflict markers.

[jeremystretch]

Some quick testing notes...

Test 1

• Create branch
• Create Site 1 in main
• Create Site 1 in branch
• Attempt to merge branch (iterative)
  • ❌ Fails on duplicate name & slug for Site 1
• Rename Site 1 to Site 2 in branch
• Attempt to merge branch (iterative)
  • ❌ Still fails on duplicate name & slug for Site 1
• Attempt to merge branch (squash)
  • ✔️ Merge succeeds

Test 2

• Create Site 1 and Site 2 in main
• Create branch
• Delete Site 1 in main
• Create Location 1 in branch
  • Assign to Site 1
• Attempt to merge branch (iterative)
  • ❌ Fails on invalid site ID
• Change parent site of Location 1 to Site 2
• Attempt to merge branch (iterative)
  • ❌ Still fails on invalid site ID
• Attempt to merge branch (squash)
  • ✔️ Merge succeeds

Test 3

• Create Device 1 and Device 2 in main
• Create Virtual Chassis 1 in main
  • Assign Device 1 as member in position 1
  • Assign Device 2 as member in position 2
• Create branch
• Swap VC member positions in branch
  • Assign Device 1 to VC position 3
  • Assign Device 2 to VC position 1
  • Assign Device 1 to VC position 2
• Attempt to merge branch (squash)
  • ❌ Fails on duplicate VC position
• Attempt to merge branch (iterative)
  • ✔️ Merge succeeds

[jnovinger]

FWIW, this looks good. Just holding off on approval/merging until I have feature cleaned up and can re-point this PR at it.