DVCS problem solving

How enterprises can scale Mercurial and Git to large organizations

Marcin Kuminski
Construction image via Shutterstock

Moving to a distributed version control system (DVCS) like Git or Mercurial can mean facing a number of problems. Here’s a look at the best ways to solve those problems while saving time when switching.

Half a decade ago, I moved my development team at a European telecom company from Subversion to Mercurial.  Although there were many immediate benefits, we also overcame many problems.  I’ll explain my personal experience of how I overcame the problems so that you can move to a Distributed Version Control System (DVCS) like Git or Mercurial with fewer problems and in less time.

The problems I faced were in three groups:

  1. Our developers became overwhelmed by the number of commits.
  2. Repository administrators needed to address enterprise compliance and scalability issues such as user security, connection into the integration process, and bulk management of routine tasks.
  3. Both Git and Mercurial needed to integrate into existing enterprise tools, many of which were custom-built and specific to each enterprise.

The first problem I solved was the display and creation of repositories.  I created a web interface that showed our projects and allowed the administrator to add a new project. I deployed it live in the evening to production servers used by developers.  People immediately started using it.  Through trial and error, I found out what worked to make the lives our developers easier.  After I was able to display the repositories with basic management, I added a login for basic security.  I then added permission settings.  After getting role-based permissions set up, I realized that I could make the history of the files easier to read.  I then added fine-grained permission control for read, write, and admin access.

Another of the common tasks I continually encountered when I moved from Subversion to Mercurial was to connect our development repositories to our continuous integration servers. The task was so common, that it honestly became a bit annoying.  I thought hard about what I could do to fix this.  Initially, we created custom tools for Subversion that automated processes for our enterprise development team.  I decided to create tools for Mercurial to mimic this functionality.

I got positive feedback.  I started to get passionate about the project.  I thought to myself, “Oh my gosh, this is so cool. I created something new.”  Of course, I kept working on it in my free time.  In the evenings after work, I spent all of my time on the project.  I kept getting amazing feedback and people contributed new ideas about features that would be cool to add.  I had a perfect scenario where I had feedback from my colleagues and a live audience.  I was making my life and the lives of my colleagues easier. By that time, I was responsible for managing all the administration of the repositories.

Going viral on BitBucket

After one year of working internally, I decided to open source the project on BitBucket.  Then, it went viral.  It rose up to the top 10 most popular projects on BitBucket.  I was surprised.  I started to see the code get downloaded and adopted by really large companies. Some of the larger companies started to offer to pay for features.

For example, one request from a company with several thousand people was to extend the user control security settings with more features such as permission delegation to enable the administrator to allocate certain resources to specific users.  These are features that are really important for mid-level managers in a tiered organization.

Another example of requirements that I learned enterprises needed were for security controls based on network IP address restrictions.  The enterprises also asked for bulk commands to manage thousands of users at once.  For example, they wanted the ability to add, enable or delete access for thousands of users or repositories at once.  They wanted to apply a group of the same settings to ten or a hundred users or repositories.

Up until now, I’ve described problems and solutions for the developers that use the DVCS and the administrators that have to main the systems.  I’ve explained how developers needed to adapt to a larger number of commits when they moved from Subversion to a DVCS like Git or Mercurial.  I’ve also described how administrators needed to adapt their security and bulk management work processes.  The last area I’ll cover relates to the people responsible for implementing Git or Mercurial into their existing enterprise application infrastructure.

Implementing Git or Mercurial

Take for example, a large company like Nintendo that contacted me when they were moving to DVCS.  They had a long history with software development.  Over a period of time, they developed a huge amount of internal code and system tools.  For example, if a user is created in Mercurial, go to LDAP and create the same user or ping the manager that oversees user controls.

To connect the different systems and tools within large enterprises, DVCS administrators and I developed a type of “hook” system.  For most of the actions, you can have it trigger another action.  For example, a large company like Nintendo will have a set of actions that automatically occur whenever a repository is created,  If a repository is forked or a user is deleted, another set of actions automatically happens.

By following a strategy of “hooking” into other systems, the DVCS administrator can automate any system.  The downside is that it takes longer to initially configure and set up.  We often see this conflict between flexibility and ease.  Of course, the trick is to combine both flexibility and ease of use.  However, for large enterprises, we often saw that the flexibility to integrate into existing systems was a must have requirement.  In my experience, we used a JSON RPC API to provide the flexibility to hook into Mercurial or Git tools.

I also saw that big companies sometimes had unusual requirements and decided to store the configuration hook settings in a separate file that was almost infinitely flexible.

As DVCS systems like Git and Mercurial become more widespread, either for cloud-based or on-premise use, we need to share best practices and exchange tips that will help other people.  The migration to DVCS will profoundly impact developers, administrators, and system integrators.  Each group will need to overcome problems in order to get the full benefits of DVCS.  As the benefits of DVCS are enormous, I’m confident that most organizations will solve the problems.  I’d love to hear your migration stories and what worked best for you.


Marcin Kuminski

Created and co-founded RhodeCode, used in many of the world largest organizations. Lead teams of developers in Warsaw, Tel-Aviv, New York. Designed and coded many large web platforms. Passionate about version control systems, and the way people collaborate on code and documents.

Inline Feedbacks
View all comments