Jamsync

Open-source version control for game development

It's time for a free and open source collaboration system for game developers. Current closed-source systems, like Perforce and Plastic/Unity DevOps, are expensive, complex, and limit their users. Current open-source solutions, like Git or SVN, are difficult to use and do not scale with large projects containing large files. Game developers want to focus on their game, not their version control.

Jamsync is a free and open source version control system that uses modern Content Defined Chunking strategies to efficiently track changes to large files and large projects. Currently it's around 2x-10x faster than Git. You can try out a hosted alpha version on this site by logging in or check out the AGPL-licensed source on Github.

You can join the Discord for updates or to get support. If you have any questions or a problem I can help solve, please email me at [email protected].

Problem

Third generation version control systems have enabled code collaboration and sharing on a massive scale. Most notably Git in combination with hosting platforms like Github have improved our ability to reuse and share code. However, game developers have mostly been removed from this movement due to Git's poor support for large files and large projects. There are several problems that will also continue to get worse.

Third generation version control systems, almost all of which were created in the mid-2000's, were created in a widely different storage, network and build environment than what we have today. These changes, as they continue to evolve over the next decade, will create a need for a "fourth generation" version control system.

Fourth Generation Version Control

Jamsync

Jamsync is an in-development fourth generation version control system that is being built for game developers. You can currently do things most that you would expect from a current version control system, like pulling, pushing, branching, and merging, but it's not quite ready for production use -- expect some bugs. The core algorithm has been implemented but there's more work remaining to build out features that developers expect from a full collaboration platform. Please join the Discord and star the repo in Github to follow development.

Terminology

Since Jamsync works a little differently than most version control systems, it's necessary to define some words since they may have slightly different meaning than other systems.

Benchmarks

This section compares Jamsync upload and download speed for a directory to Git. Note that these numbers are not final and future features will give Jamsync ways to make typical workflows faster, like directory mounting over NFS. Also, these are raw file measurements, meaning no previous versions are uploaded or downloaded (which is to Git's advantage).

Measurements from Jamsync hosted in us-west-2 from Seattle and measurements from Git repositories hosted by Github. Obviously, there are more operations that matter in version control than just uploading and downloading an entire repository, but this is just a demonstration to show what improvements can be made in our current systems.

Algorithm

The idea behind Jamsync based off of the rsync algorithm and Content Defined Chunking (CDC). If you haven't read these, I would highly recommend them!

How Jamsync uses Rsync and CDC

The main idea behind Jamsync is that we can store the operations sent by the sender in an rsync-like stream to track changes to a file. This means we treat rsync operations like a delta chain that we can use later to regenerate the file. The storage of deltas and their usage to regenerate a file is similar to the Mercurial concept of a Revlog. However, the advantage of using rsync blocks is that we can efficiently store changes to, and regenerate, arbitrarily large files since these blocks can be streamed and regenerated independently.

Data pointers

In each block, we can store the location of the last data block to regenerate the file efficiently. By using blocks instead of an xdelta approach, we can store pointers in each block find the last actual data block to use in the file, rather than regenerating the file through a delta chain which Mercurial does. Mercurial essentially caches the entire file at certain points and uses this later to have a smaller regeneration length.

Branches

A chain of changes, formed by the process above, can be used to regenerate every file in a project. Branches can be automatically rebased on top of the mainline. This means that every branch will always be up-to-date. If conflicts occur during the rebase, a branch will need manual merging.

Limitations

The goal is to be able to handle over 100M files and over 1TB-sized files in a single repository. We're not there yet in the current implementation (~1M files with 16GB-sized files) but should be there in the next couple months.

Implementation

Jamsync is being written from scratch in Golang and uses mattn/go-sqlite3 to store projects and change information. gRPC and Protocol buffers are used for service definitions and data serialization.

Acknowledgements

This awesome site theme is made by @panr and adapted to this site.