Dealing with long running feature branches and the resulting merge-hell

I want to get my thoughts down on lessons learned from a crazy merge of a long running feature branch that I recently had to do while they are fresh in my head.  There are a few points I wanted to make and I wanted to invite you all to add on or tweak things.

 
Background: I tasked two developers with merging in 3 months of code from two branches.  Each branch had little to do with each other and diverged significantly.  The developers approached this by branching Master at the tip and merging in the feature branch.  There was also a discovery of a mistaken merge into master from the source branch that further complicated things and pulled technical leadership/experts into help fix it which further limited the amount of oversight on the merge process.  Given the crazy amount of work, the devs relied on discarding changes from particular places in the master branch since there was no domain expert on hand from that branch and there was limited technical oversight throughout the process.  In addition, the standup meetings had moved away from status updates to pure roadblock resolutions so there was no insight into the details a particular dev was struggling with.

 
Result: When the merge was complete we ended up with code where many areas of the domain additions from the master branch would not work as well as massive incongruities on the front end code.

 
So here are some lessons learned.  Granted, most of these have been worked out by the industry by some sort of process and I know these aren’t my original thoughts but I figured it would help your team out when they make decisions around these issues the next time.

  • If we’re ever in a situation where we end up having a long running feature branch that doesn’t get merged into master, it would be good for us to give the following direction to the devs.
    • Attempt to merge in smaller sets of code from each branch rather than bite off the whole thing.  This means take 1 week or 1 month increments from each branch if a clear delineation can be made.  The time lines of the commits don’t necessarily have to add up.  This allows for smaller bites to be taken between the two commits.  Encouraging the developers to take this approach would give them a strong sense of agency in the code that they are merging rather than having to fall back on faith.
    • Use the power of git.  In git the saying is “merges are easy”.  Using the rebase feature of git might have been a more effective mechanism than the cvs approach that I suggested to the developers.
      • The rebase in a long running merge combined with divide and conquer approach looks like the following.
        • Take a branch of the last known merge point on the source branch and check out to that branch.
        • Define two commits somewhere a couple of weeks ahead (Again taking domain situations into account)  in the source and target branch.
        • Interactively weave the commits together with manual conflict resolution
      • This results in a clean time line on the merge branch as well as merged code and greater developer confidence on the code, it’s effectively the same as the merge but forcing the developer to consider the histories of the code in small commit chunks one by one.
      • This also gives us much more power to backout change
    • If at all possible, consider building a dev team that consists of 1 developer from the source branch and 1 developer from the target branch.  Combining knowledge in this way gives the devs the ability to work out issues rather than hope things will just work out after the merge.    The scope of the merge in this long running case tends to require two developers.
    • Since these merges take days, a daily 15 minute standup with technical leadership should be setup and the developers should answer the following questions
      • Daily Standup Questions
        • What was completed yesterday?
        • What was completed today?
        • What issues came up in the merge today that resulted in confusion and require roadblock assistance.
      • This can give the technical leadership insight onto problems ahead of time rather than the traditional wait till it’s all merged.
    • Capture all of your merge domain issues that come up and are not immediately resolvable in your bug/issue tracking system.  You’re guaranteed to forget about the details of each thing so make sure you do yourself a favor and get them into your system.
    • If we can forsee a long running feature branch ahead of time then it would be good to favor some of the following processes.
      • Creation of a _merge branch and weekly, if not daily merges between the branches.
        • Identification of one or two developers (cross domain if possible) to perform this merge and provide status in the standups so that technical leaders can resolve any domain issues that pop up immediately in the source or target branches.
      • Creation of a –merge environment that QA can use to test code and look for domain issues.
      • Daily building of the _merge branch via CI.
    • Start to  encourage the developers to rebase and craft their code locally using git so that commits reference features.
      • There are a few occasions where commits in one branch need to be excluded or reconsidered because of domain conflicts in another branch.  Having resolution on these features will allow us to quickly execute on excluding or including features, or setting them to the side for a later release.
        • Currently a feature is crafted with multiple commits over time and pushed out to origin.  This makes it hard for developers and leaders to cherry pick features using “the git”.
      • Developers who are transitioning from the CVS or SVN way of doing things aren’t going to be aware on how to use git to make this work for them so this is going to come with time and experience.

    git gc issues and running out of memory

    I was recently running into some issues where I had a massive git repo (12G).   It was originally 2G.  Since this cloned repo is used by the continuous integration system, I end up doing a lot of git checkout — . on the repo in order to get it back to a state where any of the changes I made to build it are removed.  However, this constant head switching caused some cruft to accumulate over the past 2 months and now we’re in a spot where the repo is just way to massive.

     

    So I attempted to run git gc on the repository but ended up with the following issue.

     

    [tomcat@aa-cruisecontrol starter_relate_int]$ git gc
    Counting objects: 160591, done.
    Delta compression using up to 4 threads.
    fatal: Out of memory? mmap failed: Cannot allocate memoryerror: failed to run repack
    [tomcat@aa-cruisecontrol starter_relate_int]$

    After attempting a couple more attempts I still could not repack it.   So I ended up blowing it away and recloning it.  This reduced the size of the git repository to 2G.  Draconinan yes, but I am glad to have the space.

     

    Lost love for Apple, Google, Mozilla

    Two to three years ago it seemed that our heros in the tech world were Apple, Google, Facebook and Mozilla.   Apple was riding a streak of product wins and scored massive multi-ball points with the iPhone and iPod Touch.  The app store was an example of industry and made money for lots of people.   Google was showing the world that the Do no Evil mantra was still alive and showed us the PaaS model was alive.  Facebook started to become the destination for social conversation and calendar and Mozilla had a solid browser that gave it’s users access to plug-ins that worked and a platform that didn’t cause issues.

    Now in 2010 it’s a different story.

    Apple is being called arrogant from some of it’s most ardent supporters and important tech-cheerleaders.  The iPad is not being cheered on and hailed – it’s Apples version of the pinball TILT at this point (granted it’s only been a day since it’s been announced).   Google is trying to recapture some of it’s hacker and painter magic by standing up for human rights, throwing a fit about China’s cyber attack.  Facebook puts you on a roller-coaster of privacy concerns and Mozilla’s browser as of 3.5 has introduced a number of issues with Flash that cause stability issues and are leaving the tech adopters with Google Chrome as the choice for solid web browser.

    What is it that turned the fortunes?    Was it that American people are super distrustful of any large corporation after the latest recession?  Is it that these companies got enough mind share and are running into classic PR and image issues?   Is it arrogance and greed that is seeping in?

    Installing Ruby On Rails and the dreaded Error Message

    If you attempt to install rails via the standard

    yo@yo:/ sudo gem install rails

    And you get the dreaded message:

    ERROR:  could not find rails locally or in a repository

    There is a simple cure.  Simply install the new gem (1.3.1) package from source and try again.  Apparently 1.1.1 and 1.2.1 get the connection to the mother ship snipped some how.   I tried and failed and tried and failed to install Rails via gem until i reinstalled the new gem package.

    Keep in mind this is a full install.  You’ll have to get the package as follows

    wget http://rubyforge.org/frs/download.php/45905/rubygems-1.3.1.tgz

    unzip and untar and build the sources via

    sudo ruby setup.rb

    HTH,

    Shaun twitter.com/h0h0h0

    Working Capital and Your Small Business

    Catherine Juon of Pure Visibility sat with MichiganInnovators.org to discuss her journey with a new business and some of the ways she financed her business while bootstrapping.  I was really drawn to this particular set of interviews because she is one of the local entrepreneurs I followed on Twitter and because I am very interested in the different ways a business is financed.

    One thing that you should know is that I am pretty much dead set against all forms of debt financing.  I am more interested in bootstrapping with personal savings than with a debt instrument like a personal credit card.  This mostly has to do with my own poor behavior using credit cards and the fact that being out of debt is just so freeing.  There really was no underlying education on how to properly use a debt instrument for business financing.

    I’m a little smarter after watching these interviews.  Pure Visibility helps with marketing website and is a pure serviced based business.  Catherine used her personal credit card as the needed working capital to help finance her ongoing daily operations.  Working capital was a new term for me:  this is capital (cash) you will need to finance your business while you wait for recievables to post.

    Working capital, regardless of how it is financed, pays your employees and the electric company.  When you’re in bootstrapping mode, working capital is what you’re drawing on to keep the business going.  For many people that means financing with a debt instrument.

    Her needs for working capital grew from a personal credit card to a corporate credit card and now Catherine is exploring a Line Of Credit as a stable and more appropriate source for working capital. Her experience has taught her a lot about how credit can best be found.   She began to pre-qualify the bank sources by explaining exactly what Pure Visibility was and making sure that the relationship was going to be the right fit.  Specifically she was wondering if the bank even “got it”.   She also began to look at her business in a component based viewpoint.  This leads her to ask what parts of her business are consulting, and which parts of her business become trade secrets.  Knowing the answer to these questions let her explain to prospective bankers in terms they understood.

    Catherine was also very clear on her funding sources.  She acknowledges that venture capitalist funding is quite the cool thing to do but is seeking bank sources of credit to finance her business.   V.C. funding requires you to ask yourself what parts of the business will you give up to grow.  This is an question that I knew existed but had not really admitted to myself.

    These were very educational videos for me.  While I am not convinced about debt financing, I did learn a couple things on the operational aspects of running an enterprise.  I’m curious about the application to a business plan.  Should working capital be laid out in the business plan as part of the operations?   I’m also wondering if you have to burn through all the cash or if some can be saved to self-fund growth opportunities in the future?  A trip to Entreleadership might answer that question for me.

    Shaun Farrugia

    Angel Investors Pocket Knife – Terry Cross

    I recently watched both of the Terry Cross interviews on MichiganInnovators.org and am better for taking the time to do this.  Terry Cross has been angel-investing for 46 years and has “learned a few things” in his time.   I learned more about how angel investing works from watching these interviews than I knew before.  It’s true that I was pretty limited in what I knew about this topic.  The interview questions and answers were precise and informative as well as clear and understandable.  I felt so educated by these interviews that I wanted even more detail on how Terry thought.   

    Terry talked about how an entrepreneur needs to perform risk mitigation before he really begins to take them seriously.  I have never written any sort of business plan yet but I am glad I listened to this interview before doing this.  Terry made clear exactly how he expects the funded entrepreneur to think through his plan.  I plan on taking my plan through the same rigor that Terry laid out in the interview.  The litany of “things that can go wrong” was not supposed to be taken as a fear-inducer.  The point was to give your business plan more direction.  It can also allow the business plan to remind you of those pitfalls as they occur.  Clear thinking early in the process might be better than panic thinking when the problem arises.  Maybe..

     I found it to be a very cogent and clear way to think through risk in a way that will benefit your plan and the investor.  Then I got the feeling that risk becomes the guide at this point.  Is this a true statement?  I’m wondering if risk should be the right guidance factor for any business?  I am naturally risk-averse, but born and raised a Roman Catholic, so there is a natural avoidance of any sort of “control”.  It’s more palatable to me to see risk as a strong guidance mechanism, but I cannot accept that risk is the “point” of the whole enterprise.  

    Overall, the interviews gave me a number of great ideas and was HIGHLY educational for me.  Glad to have the opportunity to have viewed them.

    Thanks,

    Shaun Farrugia

     

    Michigan Pre-Seed Capital Fund

    The Michigan pre-seed capital fund allows technology companies that are just beginning to receive funds to help assist their growth and development.  The fund allows entrepreneurs come to the table with funds and have those funds matched by the Pre-Seed Capital Fund. Skip Simms, fund manager, tells us that this is a very unique capital fund.   Many areas of the country do not have such a fund and have approached the Michigan Pre-Seed Capital Fund for guidance on setting up their own funds.

    Many companies got out of investing in early-stage companies after the tech bubble burst in 2001.  Early-stage companies are those companies that have not had a round of funding and are seeking funding to help bring a product to market.  These rounds of funding are typically small.   These investors who started to avoid early stage rounds of funding wanted to concentrate their funds on companies that came out of the embryonic stage and had a solid foundation.   Skip refers to these types of investments as up-stream investments.

    This was my first time reviewing anything of this nature.  I’ve always wondered how funding was sought after and awarded.  I have become more aware of the different types of funding that are out there.   This is the first glimpse I’ve seen of a fund being a collaborative effort between government, industry and academics.  Then again, I haven’t seen many funds.  

    I am excited to see this type of opportunity given to entrepreneurs in the State of Michigan.   I was a resident of the State of Michigan for 29 years I relocated to New York City.  I know that I will champion the State of Michigan in my new home, both in spirit and in deed.  I can let people know about these types of opportunities in a city that is known for opportunities.

     

    One thing that concerns me is the low amount of business clusters this can be available to.  I am concerned about the lack of pure information technology categories that are qualified for funds.  Many creative and young people that are leaving the State are pure programmers or visual and creative designers.  These types of people do not want to be in an unsexy automotive or homeland security type operation.  

     

    The over designation of “Advanced automotive” is unsettling.  Most of the news in the state is focused on the automotive industry crumbling before our eyes.  Many young people believe that these companies are reaping the product of a corporate culture that protects “position” over “innovation” and talent.  There is no creative or fiduciary incentive for these individuals to contribute to this industry as a supplier or an employee.  Doing so would be almost un-tasteful.

     

    Furthermore, nanotechnology makes a brief bullet point appearance under “Advanced Automotive”.  This is the over-designation that is hiding some of the more interesting aspects of this pre-seed fund.  Nanotechnology is a much more sexier technology for creatives to follow.  Why is it hidden under automotive and manufacturing?  Nanotechnology should stand out on it’s own.  

     

    Lastly, why do we not hear about this?  It’s possible that commercials for this fund have been on WWJ but much of the energy that SPARK and SmartZONE is seeking is possibly listening to other forms of media.  Not to be trite but advertising these types of things on modern rock stations might enlist a younger more creative energy to help out in Michigan.  

     

    Overall I’m glad to see something like this happen in Michigan.  I’ll be exploring the other structures of support for new business in the coming future!

     

    Shaun Farrugia

    9-22-2008

    New Class and Final Class

    Hi All,

    I am pretty excited to be learning about entrepreneurship at Eastern Michigan University.  The class is 388 – Introduction to Entrepreneurship.  I will be using this blog to post comments and responses to blog posts as I read them.

    Thanks!

    Shaun Farrugia

    Java, where art thou? aka .isAvailable()?

    So for the past two years I’ve really wanted to jump on the Ruby On Rails train (is that what it’s called?).  It seems that PHP has had a similar MVC pattern happening with PEAR.  Rails is under the gun from this “hackers framework” called MERB.   Python has Django (pownce.com is using this) and Pylon (nothing like maintaining alliteration).

    As a web developer I am always looking for ways to maintain some semblance of automation amongst my coding activities.  There’s only so many view patterns one can write before you say “Really?  I need to do this again?”  I’m wondering why a pure POJO framework hasn’t caught fire amongst us yet.   Is RUBY that much of a step up that it allows such a strong framework to exist.  Are we doomed to have  Groovy fill in the cracks with scripting putty and Grails?