Dealing with long running feature branches and the resulting merge-hell

I want to get my thoughts down on lessons learned from a crazy merge of a long running feature branch that I recently had to do while they are fresh in my head.  There are a few points I wanted to make and I wanted to invite you all to add on or tweak things.

Background: I tasked two developers with merging in 3 months of code from two branches.  Each branch had little to do with each other and diverged significantly.  The developers approached this by branching Master at the tip and merging in the feature branch.  There was also a discovery of a mistaken merge into master from the source branch that further complicated things and pulled technical leadership/experts into help fix it which further limited the amount of oversight on the merge process.  Given the crazy amount of work, the devs relied on discarding changes from particular places in the master branch since there was no domain expert on hand from that branch and there was limited technical oversight throughout the process.  In addition, the standup meetings had moved away from status updates to pure roadblock resolutions so there was no insight into the details a particular dev was struggling with.

Result: When the merge was complete we ended up with code where many areas of the domain additions from the master branch would not work as well as massive incongruities on the front end code.

So here are some lessons learned.  Granted, most of these have been worked out by the industry by some sort of process and I know these aren’t my original thoughts but I figured it would help your team out when they make decisions around these issues the next time.

  • If we’re ever in a situation where we end up having a long running feature branch that doesn’t get merged into master, it would be good for us to give the following direction to the devs.
    • Attempt to merge in smaller sets of code from each branch rather than bite off the whole thing.  This means take 1 week or 1 month increments from each branch if a clear delineation can be made.  The time lines of the commits don’t necessarily have to add up.  This allows for smaller bites to be taken between the two commits.  Encouraging the developers to take this approach would give them a strong sense of agency in the code that they are merging rather than having to fall back on faith.
    • Use the power of git.  In git the saying is “merges are easy”.  Using the rebase feature of git might have been a more effective mechanism than the cvs approach that I suggested to the developers.
      • The rebase in a long running merge combined with divide and conquer approach looks like the following.
        • Take a branch of the last known merge point on the source branch and check out to that branch.
        • Define two commits somewhere a couple of weeks ahead (Again taking domain situations into account)  in the source and target branch.
        • Interactively weave the commits together with manual conflict resolution
      • This results in a clean time line on the merge branch as well as merged code and greater developer confidence on the code, it’s effectively the same as the merge but forcing the developer to consider the histories of the code in small commit chunks one by one.
      • This also gives us much more power to backout change
    • If at all possible, consider building a dev team that consists of 1 developer from the source branch and 1 developer from the target branch.  Combining knowledge in this way gives the devs the ability to work out issues rather than hope things will just work out after the merge.    The scope of the merge in this long running case tends to require two developers.
    • Since these merges take days, a daily 15 minute standup with technical leadership should be setup and the developers should answer the following questions
      • Daily Standup Questions
        • What was completed yesterday?
        • What was completed today?
        • What issues came up in the merge today that resulted in confusion and require roadblock assistance.
      • This can give the technical leadership insight onto problems ahead of time rather than the traditional wait till it’s all merged.
    • Capture all of your merge domain issues that come up and are not immediately resolvable in your bug/issue tracking system.  You’re guaranteed to forget about the details of each thing so make sure you do yourself a favor and get them into your system.
    • If we can forsee a long running feature branch ahead of time then it would be good to favor some of the following processes.
      • Creation of a _merge branch and weekly, if not daily merges between the branches.
        • Identification of one or two developers (cross domain if possible) to perform this merge and provide status in the standups so that technical leaders can resolve any domain issues that pop up immediately in the source or target branches.
      • Creation of a –merge environment that QA can use to test code and look for domain issues.
      • Daily building of the _merge branch via CI.
    • Start to  encourage the developers to rebase and craft their code locally using git so that commits reference features.
      • There are a few occasions where commits in one branch need to be excluded or reconsidered because of domain conflicts in another branch.  Having resolution on these features will allow us to quickly execute on excluding or including features, or setting them to the side for a later release.
        • Currently a feature is crafted with multiple commits over time and pushed out to origin.  This makes it hard for developers and leaders to cherry pick features using “the git”.
      • Developers who are transitioning from the CVS or SVN way of doing things aren’t going to be aware on how to use git to make this work for them so this is going to come with time and experience.