---
jupytext:
  formats: ipynb,md:myst
  text_representation:
    extension: .md
    format_name: myst
    format_version: 0.13
    jupytext_version: 1.11.5
kernelspec:
  display_name: Python 3
  language: python
  name: python3
---

```{code-cell} ipython3
:tags: [remove-cell]
from ndslib.config import jupyter_startup
jupyter_startup()
```

(git)=
# Version control

```{eval-rst}
.. index::
   single: git
```

```{eval-rst}
.. index::
   single: Version control
```

Another set of tools that give us more control over our computational
environment are tools that track changes in our software and analysis. If this
seems like a boring or trivial task to you, consider the changes that you need
to make to a program that you create to analyze your data, throughout a long
project. Also, consider what happens when you work with others on the same
project, and multiple different people can introduce changes to the code that
you are using. If you have ever tried to do either of these things, you may have
already invented your way to track changes to your programs. For example, naming
the files with the date you made the most recent changes or adding the name of
the author who made the most recent change to the filename. This is a form of
version control --  but it's often idiosyncratic and error-prone. Instead,
software tools for version control allow us to explicitly add a particular set
of changes to the code that we are writing as we go along, keeping track for us
of who made which changes and when. They also allow us to merge changes that
other people are doing. If needed, it marks where changes we have made conflict
with changes that others have made, and allows us to resolve these conflicts,
before moving on

## Getting started with git

One of the most widely-used software for version control is called "git".
Without going too much into the difference between git and other alternatives,
we'll just mention here that one of the reasons that git is very widely-used is
because of the availability of services that allow you to use your version
control to share your code with others through the web and to collaborate with
them on the same code-base fairly seamlessly. A couple of the most popular
services are [GitHub](https://github.com) and [GitLab](https://gitlab.com).
After we introduce git and go over some of the basic mechanics of using git, we
will demonstrate how you can use one of these services to collaborate with
others and to share your code openly with anyone. This is an increasingly common
practice and one that we will talk about more in {numref}`sharing`.

Git is a command-line application. Like the `ls`, `cd`, and `pwd` unix commands
that you saw in {numref}`unix`, we use it in a shell application.

```{admonition} Installing git

First of all, verify whether you need to install git. If you already have a unix terminal to work with, you might already have git installed. In particular, if you installed git for windows as a way to get the unix shell in {numref}`unix`, you're already set. Type `git` at the shell to check. If you get a long help document, this means that you have git installed. Otherwise, follow the instructions in the [git online documentation](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) to install the right version for your computer's operating system.

```


In contrast to the other shell commands you saw before, git can do many
different things. To do that, git has sub-commands. For example, before
you start using git, you need to configure it to recognize who you are using the
`git config` sub-command. We need to tell git both our name and our email
address.

```{eval-rst}

.. code-block:: console

   $ git config --global user.name "Ariel Rokem"
   $ git config --global user.email arokem@gmail.com
```

This configuration step only needs to be done once on every computer on which
you use git, and is stored in your home directory (in a file called
`~/.gitconfig`).

####  Exercise

Download and install git and configure it with your name and email address.

The next sub-command that we will usually use with git is the
one that you will start your work with from now on -- the sub-command to
initialize a *repository*. A repository is a folder within your filesystem that
will be tracked as one unit. One way to think about that is that your different
projects can be organized into different folders, and each one of them should
probably be tracked separately. So, one project folder becomes one git
repository (also often referred to as a "repo" for short). Before starting to
use git, we then make a directory in which our project code will go.

```{eval-rst}

.. code-block:: console

   $ mkdir my_project
   $ cd my_project
```

The unix `mkdir` command creates a new directory called (in this case)
`my_project` and the `cd` command changes our working directory so that we are
now working within the new directory that we have created. And here comes the
first git sub-command, to initialize a repository:

```{eval-rst}

.. code-block:: console

   $ git init
   Initialized empty Git repository in /Users/arokem/projects/my_project/.git/
```

```{eval-rst}
.. index::
   single: git init
```

As you can see, when this sub-command is issued, git reports back that the empty
repository has been created. We can start adding files to the repository by
creating new files and telling git that these are files that we would like to
track.

```{eval-rst}

.. code-block:: console

   $ touch my_file.txt
   $ git add my_file.txt
```

```{eval-rst}
.. index::
   single: git add
```


We've already seen the `touch` bash command before -- it creates a new empty
file. Here, we've created a text file, but this can also be (more typically) a
file that contains a program written in some programming language. The `git add`
command tells git that this is a file that we would like to track -- we would
like git to record any changes that we make in this file. A useful command that
we will issue early and often is the `git status` command. This command reports
to you about the current state of the repository that you are working in,
without making any changes to its content or state.

```{eval-rst}

.. code-block:: console

   $ git status
   On branch master

   No commits yet

   Changes to be committed:
   (use "git rm --cached <file>..." to unstage)
      new file:   my_file.txt
```

```{eval-rst}
.. index::
   single: git status
```

Here it is telling us a few things about the repository and our work within it.
Let's break this message down line by line. First of all, it tells us that we
are working on a *branch* called `master`. Branches are a very useful feature of
git. They can be thought of as different versions of the same repository that
you are storing side by side. The default when a repository is initialized is to
have only one branch called `master`. However, in recent years, git users have
raised objections to the use of this word to designate the default branch,
together with objections to other uses of "master"/"slave" designations in a
variety of computer science and engineering (e.g., in distributed computing, in
databases, etc.; to read more about this, see "Additional resources" at the end
of this section). Therefore, we will immediately change the name of the branch
to "`main`".

```{eval-rst}

.. code-block:: console

   $ git branch -M main
```

```{eval-rst}
.. index::
   single: git branch
```

In general, it is a good practice to use not the default branch (now called
`main`), but a different branch for all of your new development work. The
benefit of this approach is that as you are working on your project, you can
always come back to a "clean" version of your work that is stored on your `main`
branch. It is only when you are sure that you are ready to incorporate some new
development work that you *merge* a new development branch into your main
branch. The next line in the status message tells us that we have not made any
commits to the history of this repository yet. Finally, the last three lines
tell us that there are changes that we could commit and make part of the history
of the repository. This is the new file that we just added, and is currently
stored in something called the "staging area". This is a part of the repository
into which we have put files and changes that we might put into the history of
the repository next, but we can also "unstage" them so that they are not
included in the history. Either because we don't want to include them yet, or
because we don't want to include them at all. In parentheses git also gives us a
hint about how we might remove the file from the staging area -- that is, how we
could "unstage" it. But for now, let's assume that we do want to store the
addition of this file in the history of the repository and track additional
changes. Storing changes in the git repository is called *making a commit*, and
the next thing we do here is to issue the `git commit` command.

```{eval-rst}

.. code-block:: console

   $ git commit -m "Adds a new file to the repository"
   [main (root-commit) 1808a80] Adds a new file to the repository
   1 file changed, 0 insertions(+), 0 deletions(-)
   create mode 100644 my_file.txt
```

```{eval-rst}
.. index::
   single: git commit
```

As you can see, the `commit` command uses the `-m` flag. The text in quotes
after this flag is the *commit message*. We are recording a history of changes
in the files that we are storing in our repository, and alongside the changes,
we are storing a log of these messages that give us a sense of who made these
changes, when they made them, and -- stored in these commit messages -- what the
intention of these changes was. Git also provides some additional information: a
new file was created; the mode of the file is 664, which is the mode for files
that can be read by anyone and written by the user (for more about file modes
and permissions, you can refer to the man page of the `chmod` unix command).
There were 0 lines inserted and 0 lines deleted because the file is empty and
doesn't have any files yet.

We can revisit the history of the repository using the `git log` command.

```{eval-rst}

.. code-block:: console

   $ git log
```

```{eval-rst}
.. index::
   single: git log
```

This should drop you into a new buffer within the shell that looks something like this:

```{eval-rst}

.. code-block:: console

   commit 1808a8053803150c8a022b844a3257ae192f413c (HEAD -> main)
   Author: Ariel Rokem <arokem@gmail.com>
   Date:   Fri Dec 31 11:29:31 2021 -0800

      Adds a new file to the repository
```



```{eval-rst}
.. index::
   single: Simple Hashing Algorithm (SHA)
```

It tells you who made the change (Ariel, in this case), how you might reach that
person (yes, that's Ariel's email address; git knows it because of the
configuration step we did when we first started using git) and when the change
was made (Almost noon on the last day of 2021, as it so happens). In the body of
that entry, you can see the message that we entered when we issued the `git
commit` command. At the top of the entry, we also have a line that tells us that
this is the commit identified through a string of letters and numbers that
starts with "1808a". This string is called the "SHA" of the commit, which stands
for "Simple Hashing Algorithm". Git uses this clever algorithm to encode all of
the information that was included in the commit (in this case, the name of the
new file, and the fact that it is empty) and to encrypt this information into
the string of letters and numbers that you see. The advantage of this approach
is that this is an identifier that is unique within this repository because this
specific change can only be made once [^*]. This will allow us to look up
specific changes that were made within particular commits, and also allows us to
restore the repository, or parts of the repository, back to the state they had
when this commit was made. This is a particularly compelling aspect of version
control: you can effectively travel back in time to the day and time that you
made a particular change in your project and visit the code as it was on that
day and time. This also relates to the one last detail in this log message: next
to the SHA identifier, in parentheses is the message "`(HEAD -> main)`". The
word "`HEAD`" always refers to the state of the repository in the commit that
you are currently viewing. Currently, HEAD -- the current state of the
repository -- is aligned with the state of the `main` branch, which is why it is
pointing to `main`. As we continue to work, we'll be moving `HEAD` around to
different states of the repository. We'll see the mechanics of that in a little
bit. But first, let's leave this log buffer -- by typing the `q` key.

[^*]: It is *theoretically* possible to create another change that would have the same commit, but figuring out how to do that would be so computationally demanding that, while theoretically possible, it is not something we need to worry about in any practical sense.

Let's check what git thinks of all this.

```{eval-rst}

.. code-block:: console

   $ git status
   On branch main
   nothing to commit, working tree clean
```

We're still on the main branch. But now git tells us that there is nothing to
commit and that our working tree is clean. This is a way of saying that nothing
has changed since the last time we made a commit. When using git for your own
work, this is probably the state in which you would like to leave it at the end
of a working session. To summarize so far: we haven't done much yet, but we've
initialized a repository, and added a file to it. Along the way, you've seen how
to check the status of the repository, how to commit changes to the history of
the repository, and how to examine the log that records the history of the
repository. As you will see next, at the most basic level of operating with git,
we go through cycles of changes and additions to the files in the repository and
commits that record these changes, both operations that you have already seen.
Overall, we will discuss three different levels of intricacy. The first level,
which we will start discussing next, is about tracking changes that you make to
a repository and using the history of the repository to recover different states
of your project. At the second level, we will dive a little bit more into the
use of branches to make changes and incorporate them into your project's main
branch. The third level is about collaborating with others on a project.

## Working with git at the first level: tracking changes that you make

Continuing the project we started above, we might make some edits to the file
that we added. The file that we created,`my_file.txt`, is a text file and we can
open it for editing in many different applications (we discuss text editors --
used to edit code -- in {numref}`python-env`). We can also use the unix `echo`
command to add some text to the file from the command line.

```{eval-rst}

.. code-block:: console

   $ echo "a first line of text" >> my_file.txt
```

Here, the `>>` operator is used to redirect the output of the `echo` command --
you can check what it does by reading the man page -- into our text file. Note
that there are two `>` symbols in that command, which indicates that we would
like to concatenate the string of characters to the end of the file. If we used
only one, we would overwrite the contents of the file entirely. For now, that's
not too important because there wasn't anything in the file, to begin with, but
it will be important in what follows.

We've made some changes to the file. Let's see what git says.

```{eval-rst}

.. code-block:: console

   $ git status
   On branch main
   Changes not staged for commit:
   (use "git add <file>..." to update what will be committed)
   (use "git restore <file>..." to discard changes in working directory)
      modified:   my_file.txt

   no changes added to commit (use "git add" and/or "git commit -a")
```

Let's break that down again. We're still on the `main` branch. But we now have
some changes in the file, which are indicated by the fact that there are changes
not staged for a commit in our text file. Git provides two hints, in
parentheses, as to different things we might do next, either to move these
changes into the staging area or to discard these changes and return to the
state of affairs as it was when we made our previous commit.

Let's assume that we'd like to retain these changes. That means that we'd like
to first move these changes into the staging area. This is done using the
`git add` sub-command.

```{eval-rst}

.. code-block:: console

   $ git add my_file.txt
```

Let's see how this changes the message that `git status` produces.

```{eval-rst}

.. code-block:: console

   $ git status
   On branch main
   Changes to be committed:
   (use "git restore --staged <file>..." to unstage)
      modified:   my_file.txt
```

The file in question has moved from "changes not staged for commit" to "changes
to be committed". That is, it has been moved into the staging area. As suggested
in the parentheses, it can be unstaged, or we can continue as we did before and
commit these changes.

```{eval-rst}

.. code-block:: console

   $ git commit -m"Adds text to the file"
   [main 42bab79] Adds text to the file
   1 file changed, 1 insertion(+)
```

Again, git provides some information: the branch into which this commit was
added (`main`) and the fact that one file changed, with one line of insertion.
We can look at the log again to see what has been recorded:

```{eval-rst}

.. code-block:: console

   $ git log
```

Which should look something like the following, with the most recent commit at
the top.

```{eval-rst}

.. code-block:: console

   commit 42bab7959c3d3c0bce9f753abf76e097bab0d4a8 (HEAD -> main)
   Author: Ariel Rokem <arokem@gmail.com>
   Date:   Fri Dec 31 20:00:05 2021 -0800

      Adds text to the file

   commit 1808a8053803150c8a022b844a3257ae192f413c
   Author: Ariel Rokem <arokem@gmail.com>
   Date:   Fri Dec 31 11:29:31 2021 -0800

      Adds a new file to the repository
```

Again, we can move out of this buffer by pressing the `q` key. We can also check
the status of things again, confirming that there are no more changes to be
recorded.

```{eval-rst}

.. code-block:: console

   $ git status
   On branch main
   nothing to commit, working tree clean
```

This is the basic cycle of using git: make changes to the files, use `git add`
to add them to the staging area, and then `git` commit` to add the staged changes
to the history of the project, with a record made in the log. Let's do this one
more time, and see a couple more things that you can do along the way, as you
are making changes. For example, let's add another line of text to that file.

```{eval-rst}

.. code-block:: console

   $ echo "another line of text" >> my_file.txt
```

We can continue in the cycle, with a `git add` and then `git commit` right here,
but let's pause to introduce one more thing that git can do, which is to tell
you exactly what has changed in your repository since your last commit. This is
done using the `git diff` command. If you issue that command:

```{eval-rst}

.. code-block:: console


   $ git diff
```

you should be again dropped into a different buffer that looks something like this:

```{eval-rst}

.. code-block:: console

   diff --git a/my_file.txt b/my_file.txt
   index 2de149c..a344dc0 100644
   --- a/my_file.txt
   +++ b/my_file.txt
   @@ -1 +1,2 @@
   a first line of text
   +another line of text
```

This buffer contains the information that tells you about the changes in the
repository. The first few lines of this information tell you a bit about where
these changes occurred and the nature of these changes. The last two lines tell
you exactly the content of these changes: the addition of one line of text
(indicated by a "+" sign). If you had already added other files to the
repository and made changes across multiple different files (this is pretty
typical as your project grows and the changes that you introduce during your
work become more complex), different "chunks" that indicate changes in a
different file, would be delineated by headers that look like the fifth line in
this output, starting with the "@@" symbols. It is not necessary to run `git
diff` if you know exactly what changes you are staging and committing, but it
provides a beneficial look into these changes in cases where the changes are
complex. In particular, examining the output of `git diff` can give you a hint
about what you might write in your commit message. This would be a good point to
make a slight aside about these messages and to try to impress upon you that it
is a great idea to write commit messages that are clear and informative. This is
because as your project changes and the log of your repository grows longer,
these messages will serve you (and your collaborators, if more than one person
is working with you on the same repository, we'll get to that in a little bit)
as guide-posts to find particular sets of changes that you made along the way.
Think of these as messages that you are sending right now, with your memory of
the changes you made to the project fresh in your mind, to yourself six months
from now, when you will no longer remember exactly what changes you made and
why.

At any rate, committing these changes can now look as simple as:

```{eval-rst}

.. code-block:: console

   $ git add my_file.txt
   $ git commit -m"Adds a second line of text"
```

Which would now increase your log to three different entries:

```{eval-rst}

.. code-block:: console

   commit f1befd701b2fc09a52005156b333eac79a826a07 (HEAD -> main)
   Author: Ariel Rokem <arokem@gmail.com>
   Date:   Fri Dec 31 20:26:24 2021 -0800

      Adds a second line of text

   commit 42bab7959c3d3c0bce9f753abf76e097bab0d4a8
   Author: Ariel Rokem <arokem@gmail.com>
   Date:   Fri Dec 31 20:00:05 2021 -0800

      Adds text to the file

   commit 1808a8053803150c8a022b844a3257ae192f413c
   Author: Ariel Rokem <arokem@gmail.com>
   Date:   Fri Dec 31 11:29:31 2021 -0800

      Adds a new file to the repository
```

As you have already seen, this log serves as a record of the changes you made to
files in your project. In addition, it can also serve as an entry point to
undoing these changes or to see the state of the files as they were at a
particular point in time. For example, let's say that you would like to go back
to the state that the file had before the addition of the second line of text to
that file. To do so, you would use the SHA identifier of the commit that
precedes that change and the `git checkout` command.

```{eval-rst}

.. code-block:: console

   $ git checkout 42bab my_file.txt
   Updated 1 path from f9c56a9
```

Now, you can open the my_file.txt file and see that the contents of this file
have been changed to remove the second line of text. Notice also that we didn't
have to type the entire 40-character SHA identifier. That is because the five
first characters uniquely identify this commit within this repository and git
can find that commit with just that information. To re-apply the change, you can
use the same `checkout` command, but use the SHA of the later commit instead.

```{eval-rst}

.. code-block:: console

   $ git checkout f1bef my_file.txt
   Updated 1 path from e798023
```

One way to think of the `checkout` command is that as you are working on your
project git is creating a big library that contains all of the different
possible states of your filesystem. When you want to change your filesystem to
match one of these other states, you can go to this library and check it out
from there. Another way to think about this is that we are pointing `HEAD` to
different commits and thereby changing the state of the filesystem. This
capability becomes increasingly useful when we combine it with the use of
branches, which we will see next.

## Working with git at the second level: branching and merging

```{eval-rst}
.. index::
   single: git branch
```

```{eval-rst}
.. index::
   single: git merge
```

As we mentioned before, branches are different states of your project that can
exist side by side. One of the dilemmas that we face when we are working on data
analysis projects for a duration of time is that we would like to be able to
rapidly make experiments and changes to the code, but we often also want to be
able to switch over to a state of the code that "just works". This is exactly
the way that branches work: Using branches allows you to make rapid changes to
your project without having to worry that you will not be able to easily recover
a more stable state of the project. Let's see this in action in the minimal
project that we started working with. To create a new branch, we use the `git
branch` command. To start working on this branch we check it out from the
repository (the library) using the `git checkout` command.

```{eval-rst}

.. code-block:: console

   $ git branch feature_x
   $ git checkout feature_x
```

```{eval-rst}

.. code-block:: console

   $ echo "this is a line with feature x" >> my_file.txt
   $ git add my_file.txt
   $ git commit -m"Adds feature x to the file"
```

Examining the `git log` you will see that there is an additional entry in the
history. For brevity, we show here just the two top (most recent) entries, but
the other entries will also be in the log (remember that you can leave the log
buffer by pressing the `q` key).

```{eval-rst}

.. code-block:: console

   commit ab2c28e5c08ca80c9d9fa2abab5d7501147851e1 (HEAD -> feature_x)
   Author: Ariel Rokem <arokem@gmail.com>
   Date:   Sat Jan 1 13:27:48 2022 -0800

      Adds feature x to the file

   commit f1befd701b2fc09a52005156b333eac79a826a07 (main)
   Author: Ariel Rokem <arokem@gmail.com>
   Date:   Fri Dec 31 20:26:24 2021 -0800

      Adds a second line of text
```

You can also see that `HEAD` -- the current state of the repository -- is
pointing to the `feature_x` branch. We also include the second entry in the log,
so that you can see that the `main` branch is still in the state that it was
before. We can further verify that by checking out the `main` branch.

```{eval-rst}

.. code-block:: console

   $ git checkout main
```

```{eval-rst}
.. index::
   single: git checkout
```

If you open the file now, you will see that the feature x line is nowhere to be
seen and if you look at the log, you will see that this entry is not in the log
for this branch and that `HEAD` is now pointing to `main`. But -- and this is
part of what makes branches so useful -- it is very easy to switch back and
forth between any two branches. Another `git checkout feature_x` would bring
`HEAD` back to the `feature_x` branch and adds back that additional line to the
file. In each of these branches, you can continue to make changes and commit
these changes, without affecting the other branch. The history of the two
branches has forked in two different directions. Eventually, if the changes that
you made on the `feature_x` branch are useful, you will want to combine the
history of these two branches. This is done using the `git merge` command, which
asks git to bring all of the commits from one branch into the history of another
branch. The syntax of this command assumes that `HEAD` is pointing to the
branch into which you would like to merge the changes, so if we are merging
`feature_x` into `main` we would issue one more `git checkout main` before
issuing the merge command.

```{eval-rst}

.. code-block:: console

   $ git checkout main
   $ git merge feature_x
   Updating f1befd7..ab2c28e
   Fast-forward
   my_file.txt | 1 +
   1 file changed, 1 insertion(+)
```

The message from git indicates to us that the `main` branch has been updated
-- it has been "fast-forwarded". It also tells us that this update pulled in
changes in one file, with the addition of one line of insertion.
A call to `git log` will show us that the most recent commit (the one
introducing the "feature x" line) is now in the history of the `main` branch
and `HEAD` is pointing to both `main` and `feature_x`.

```{eval-rst}

.. code-block:: console

   commit ab2c28e5c08ca80c9d9fa2abab5d7501147851e1 (HEAD -> main, feature_x)
   Author: Ariel Rokem <arokem@gmail.com>
   Date:   Sat Jan 1 13:27:48 2022 -0800

      Adds feature x to the file
```

This is possible because both of these branches now have the same history. If
we'd like to continue working, we can remove the `feature_x` branch and keep
going.

```{eval-rst}

.. code-block:: console

   $ git branch -d feature_x
```

Using branches in this way allows you to rapidly make changes to your projects,
while always being sure that you can switch the state of the repository back to
a known working state. We recommend that you keep your `main` branch in this
known working state and only merge changes from other branches when these
changes have matured sufficiently. One way that is commonly used to determine
whether a branch is ready to be merged into `main` is by asking a collaborator
to look over these changes and review them in detail. This brings us to the most
elaborate level of using git -- in collaboration with others.

## Working with git at the third level: collaborating with others

So far, we've seen how git can be used to track your work on a project. This is
certainly useful when you do it on your own, but git really starts to shine when
you put it to work to track the work of more than one person on the same
project. For you to be able to collaborate with others on a git-tracked project,
you will need to put the git repository somewhere where different computers can
access it. This is called a "remote", because it is a copy of the repository
that is on another computer, usually located somewhere remote from your
computer. Git supports many different modes of setting up remotes, but we will
only show one here, using the [GitHub website](https://github.com) as the
remote.

```{eval-rst}
.. index::
   single: GitHub
```

To get started working with GitHub, you will need to set up a (free) account. If
you are a student or instructor at an educational institution, we recommend that
you look into the various educational benefits that are available to you through
[GitHub Education](https://education.github.com/).

Once you have set up your user account on GitHub, you should be able to create a
new repository by clicking on the "+" sign on the top right of the web page and
selecting "new repository" (see {numref}`Figure 1`).


```{figure} ./figures/gh1.png
:name: Figure 1
:class: bg-primary
:width: 4.56in
:align: center

Creating a new repository through the GitHub web interface.
```

On the following page ({numref}`Figure 2`), you will be asked to name your
repository. GitHub is a publicly available website, but you can choose whether
you would like to make your code publicly viewable (by anyone!) or only viewable
by yourself and by collaborators that you will designate. You will also be given
a few other options, such as an option to add a README file and to add a
license. We'll come back to these options and their implications when we talk about
sharing data analysis code in {numref}`sharing`).

```{figure} ./figures/gh2.png
:name: Figure 2
:class: bg-primary
:width: 4.56in
:align: center

Initializing a GitHub repository by giving it a name.
```

The newly-created web page will be the landing page for your repository on the
internet. It will have a URL that will look something like:
`https://github.com/<user name>/<project name>`. For example, since we created
this repository under Ariel's GitHub account, it now has the URL
`https://github.com/arokem/my_project` (feel free to visit that URL. It has been
designated as publicly viewable). But there is nothing there until we tell git
to transfer over the files from our local copy of the repository on our machine
into a copy on the remote -- the GitHub webpage. When a GitHub repository is
empty, the front page of the repository contains instructions for adding code to
it, either by creating a new repository on the command line or from an existing
repository. We will do the latter here because we already have a repository
that we have started working on. The instructions for this case are entered in
the command line with the shell set to have its working directory in the
directory that stores our repository.

```{eval-rst}

.. code-block:: console

   $ git remote add origin https://github.com/arokem/my_project.git
   $ git branch -M main
   $ git push -u origin main
```

The first line of these uses the `git remote` sub-command, which manages
remotes, and the sub-sub-command `git remote add` to add a new remote called
`origin`. We could name it anything we want (for example `git remote add github`
or `git remote add arokem`), but it is a convention to use `origin` (or
sometimes also `upstream`) as the name of the remote that serves as the central
node for collaboration. It also tells git that this remote exists at that URL
(notice the addition of `.git` to the URL). Addmitedly, it's a long command with
many parts.

The next line is something that we already did when we first initialized this
repository -- changing the name of the default branch to `main`. So, we can skip
that now.

The third line uses the `git push` sub-command to copy the repository -- the
files stored in it and its entire history -- to the remote. We are telling git
that we would like to copy the `main` branch from our computer to the remote
that we have named `origin`. The `-u` flag tells git that this is going to be
our default for `git push` from now on, so whenever we'd like to push again from
`main` to `origin`, we will only have to type `git push` from now on, and not
the entire long version of this incantation (`git push origin main`).

Importantly, though you should have created a password when you created your
GitHub account, GitHub does not support authentication with a password from the
command line. This makes GitHub repositories very secure, but it also means that
when you are asked to authenticate to push your repository to GitHub, you will
need to take a different route. There are two ways to go about this. The first
is to create a personal access token (PAT). The GitHub documentation has a
[webpage that describes this process](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token).
Instead of typing the GitHub password that you created when you created your
account, copy this token (typically, a long string of letters, numbers, and
other characters) into the shell when prompted for a password. This should look
as follows.

```{eval-rst}

.. code-block:: console

   $ git push -u origin main
   Username for 'https://github.com': arokem
   Password for 'https://arokem@github.com':<insert your token here>
   Enumerating objects: 12, done.
   Counting objects: 100% (12/12), done.
   Delta compression using up to 8 threads
   Compressing objects: 100% (5/5), done.
   Writing objects: 100% (12/12), 995 bytes | 497.00 KiB/s, done.
   Total 12 (delta 0), reused 0 (delta 0), pack-reused 0
   To https://github.com/arokem/my_project.git
   * [new branch]      main -> main
   Branch 'main' set up to track remote branch 'main' from 'origin'.
```

Another option is to use SSH keys for authentication. This is also described
in the GitHub documentation. The first step is to create the key and store
it on your computer, described [here](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent). The second step is to
let GitHub know about this SSH key, as described [here](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account). The
benefit of using an SSH key is that we do not need to generate it anew every
time that we want to authenticate from this computer. Importantly, if we
choose the SSH key route for authenticating, we need to change the URL that we
use to refer to the remote. This is done through another sub-command of the
`git remote` command:


```{eval-rst}

.. code-block:: console

   $ git remote set-url origin git@github.com:arokem/my_project.git
```

Once we have done that, we can visit the webpage of the repository again and see
that the files in the project and the history of the project have now been
copied into that website ({numref}`Figure 3`). Even if you are working on a
project on your own this serves as a remote backup, and that is already quite
useful to have, but we are now also ready to start collaborating. This is done
by clicking on the "settings" tab at the top right part of the repository page,
selecting the "manage access" menu item, and then clicking on the "add people"
button ({numref}`Figure 4`). This will let you select other GitHub users, or
enter the email address for your collaborator ({numref}`Figure 5`). As another
security measure, after they are added as a collaborator, this other person
will have to receive an email from GitHub and approve that they would like to be
designated as a collaborator on this repository, by clicking on a link in the
email that they received.

```{figure} ./figures/gh3.png
:name: Figure 3
:class: bg-primary
:width: 4.56in
:align: center

A copy of the files and history of the project in the GitHub repo.
```

```{figure} ./figures/gh4.png
:name: Figure 4
:class: bg-primary
:width: 4.56in
:align: center

Click the green "Add people" button to add collaborators.
```

```{figure} ./figures/gh5.png
:name: Figure 5
:class: bg-primary
:width: 4.56in
:align: center

Entering the GitHub user-name of a collaborator will let you select them from a drop-down.
```

Once they approve the request to join this GitHub repository as a collaborator,
this other person can then both `git push` to your repository, just as we
demonstrated above, as well as `git pull` from this repository, which we will
elaborate on below. The first step they will need to take, however, is to `git
clone` the repository. This would look something like this:

```{eval-rst}

.. code-block:: console

   $ git clone https://github.com/arokem/my_project.git
```

When this is executed inside of a particular working directory (for example,
`~/projects/`), this will create a sub-directory called `my_project`
(`~/projects/my_project`; note that you can't do that in the same directory that
already has a `my_project` folder in it, so if you are following along, issue
this `git clone` command in another part of your file-system) that contains all
of the files that were pushed to GitHub. In addition, this directory is a full
copy of the git repository, meaning that it also contains the entire history of
the project. All of the operations that we did with the local copy on our own
repository (for example, checking out a file using a particular commit's SHA
identifier) are now possible on this new copy.

The simplest mode of collaboration is one where a single person makes all of the
changes to the files. This person is the only person who issues `git push`
commands, which update the copy of the repository on GitHub and then all the
other collaborators issue `git pull origin main` within their local copies, to
update the files that they have on their machines. The `git pull` command is the
opposite of the `git push` command that you saw before, syncing the local copy
of the repository to the copy that is stored in a particular remote (in this
case, the GitHub copy is automatically designated as `origin` when `git clone`
is issued).

A more typical mode of collaboration is one where different collaborators are
all both pulling changes that other collaborators made, as well as pushing their
changes. This could work, but if you are working on the same repository at the
same time, you might run into some issues. The first arises when someone else
pushed to the repository during the time that you were working and the history
of the repository on GitHub is out of sync with the history of your local copy
of the repository. In that case, you might see a message such as this one:

```{eval-rst}

.. code-block:: console

   $ git push origin main
   To https://github.com/arokem/my_project.git
   ! [rejected]        main -> main (fetch first)
   error: failed to push some refs to 'https://github.com/arokem/my_project.git'
   hint: Updates were rejected because the remote contains work that you do
   hint: not have locally. This is usually caused by another repository pushing
   hint: to the same ref. You may want to first integrate the remote changes
   hint: (e.g., 'git pull ...') before pushing again.
   hint: See the 'Note about fast-forwards' in 'git push --help' for details.```
```

Often, it would be enough to issue a `git pull origin main`, followed by a `git
push origin main` (as suggested in the hint) and this issue would be resolved --
your collaborator's changes would be integrated into the history of the
repository together with your changes, and you would happily continue working.
But in some cases, if you and your collaborator introduced changes to the same
lines in the same files, git would not know how to resolve this conflict, and
you would then see a message that looks like this:

```{eval-rst}

.. code-block:: console

   $ git pull origin main
   remote: Enumerating objects: 5, done.
   remote: Counting objects: 100% (5/5), done.
   remote: Compressing objects: 100% (2/2), done.
   remote: Total 3 (delta 0), reused 3 (delta 0), pack-reused 0
   Unpacking objects: 100% (3/3), 312 bytes | 78.00 KiB/s, done.
   From https://github.com/arokem/my_project
   * branch            main       -> FETCH_HEAD
      ab2c28e..9670d72  main       -> origin/main
   Auto-merging my_file.txt
   CONFLICT (content): Merge conflict in my_file.txt
   Automatic merge failed; fix conflicts and then commit the result.
```

This means that there is a conflict between the changes that were introduced by
your collaborator and the changes that you introduced. There is no way for git
to automatically resolve this kind of situation because it can't tell which of
the two conflicting versions should be selected. Instead, git edits the file to
add conflict markers. That is, if you open the text file in a text editor, it
would now look something like this:

```{eval-rst}

.. code-block:: console

   a first line of text
   another line of text
   this is a line with feature x
   <<<<<<< HEAD
   an addition you made
   =======
   addition made by a collaborator
   >>>>>>> 9670d72897a5e15defb257010f928bd22f54c929
```

Everything up to the third line of the file is changes that both of you had in
your history. After that, your changes to the file diverge. At the same time
that you added the line of text "an addition you made", the collaborator added
the line "addition made by a collaborator". To highlight the source of the
conflict, git introduced the markers you see in the text snippet above. The
"`<<<<<<< HEAD`" marker starts the part of the file that was in `HEAD` on your
local copy. Everything below the "`=======`" marker, until the "`>>>>>>>
9670d72`..." comes from the collaborator. The SHA identifier at the end of this
marker is also the identifier of the commit that the collaborator made to the
repository while you were making your changes.

In addition, if you issue a `git status` command at this point, you will see
the following:

```{eval-rst}

.. code-block:: console

   $ git status
   On branch main
   Your branch and 'origin/main' have diverged,
   and have 1 and 1 different commits each, respectively.
   (use "git pull" to merge the remote branch into yours)

   You have unmerged paths.
   (fix conflicts and run "git commit")
   (use "git merge --abort" to abort the merge)

   Unmerged paths:
   (use "git add <file>..." to mark resolution)
      both modified:   my_file.txt

   no changes added to commit (use "git add" and/or "git commit -a")
```

Regardless of whether you would like to include your changes, as well as the
changes from your collaborator, or just one of these change sets, your next step
is to edit the files where conflicts exist to end up with the content that you
would want them to have going forward. This would include removing the conflict
markers that git introduced along the way. To resolve the conflict, you might
also need to talk to your collaborator, to understand why you both made
conflicting changes to the same lines. At the end of this process, you would
make one more `git commit`, followed by a `git push`, and keep working from this
new state. Your collaborators should be able to issue a `git pull` in their copy
of the repository to get the conflict resolved on their end as well.

Another slightly more elaborate pattern of collaboration would involve the use
of branches. Remember that we mentioned that maybe it would be a good idea to
work only on branches other than `main`? One reason for that is that this allows
you to go through a process of review of new changes before they get integrated
into the `main` branch. GitHub facilitates this process through an interface
called a "pull request". The process works as follows: whenever you would like
to make a change to the project, you create a new branch from `main` and implement
the change on this branch. For example:

```{eval-rst}

.. code-block:: console

   $ git branch feature_y
   $ git checkout feature_y
   $ echo "this is a line with feature y" >> my_file.txt
```

Then, we add the file to the staging area, commit this change in this branch,
and push the branch to GitHub:

```{eval-rst}

.. code-block:: console

   $ git add my_file.txt
   $ git commit -m"Implements feature y"
   [feature_y 948ea1d] Implements feature y
   1 file changed, 1 insertion(+)
   $ git push origin feature_y
   Enumerating objects: 5, done.
   Counting objects: 100% (5/5), done.
   Delta compression using up to 8 threads
   Compressing objects: 100% (2/2), done.
   Writing objects: 100% (3/3), 283 bytes | 283.00 KiB/s, done.
   Total 3 (delta 1), reused 0 (delta 0), pack-reused 0
   remote: Resolving deltas: 100% (1/1), completed with 1 local object.
   remote:
   remote: Create a pull request for 'feature_y' on GitHub by visiting:
   remote:      https://github.com/arokem/my_project/pull/new/feature_y
   remote:
   To https://github.com/arokem/my_project.git
   * [new branch]      feature_y -> feature_y
```

As you can see in the message emitted by git, a new branch has been created and
we can visit the GitHub repository at the URL provided to create a pull request.
The page that opens when you visit this URL already includes the commit message
that we used to describe. It also has a text box into which we can type more
details about the change ({numref}`Figure 6`). Once we hit the "Create pull
request" button, we are taken to a separate page that serves as a forum for
discussion of these changes ({numref}`Figure 7`). At the same time, all of the
collaborators on the repository receive an email that alerts them that a pull
request has been created. By clicking on the "Files changed" tab on this page,
collaborators can see a detailed view of all of the changes introduced to the
code ({numref}`Figure 8`). They can comment on the changes through the web
interface and discuss any further changes that need to be made for this
contribution to be merged into `main`. The original author of the branch that
led to this pull request can continue working on the branch, based on these
comments, making further commits. When consensus is reached, one of the
collaborators on the repository can click the "Merge pull request" button
({numref}`Figure 6`). This triggers the sending of another email to all of the
collaborators on this repository, alerting them that the `main` branch has
changed and that they should issue a `git pull origin main` on their local
copies of the repository to sync their `main` branch with the GitHub repository.

```{figure} ./figures/gh6.png
:name: Figure 6
:class: bg-primary
:width: 4.56in
:align: center

When opening a Pull Request, you can add some information about the proposed changes.
```

```{figure} ./figures/gh7.png
:name: Figure 7
:class: bg-primary
:width: 4.56in
:align: center

The Pull Request interface is a place to discuss proposed changes and eventually incorporate them into the project, by clicking the green "Merge pull request" button.
```

```{figure} ./figures/gh8.png
:name: Figure 8
:class: bg-primary
:width: 4.56in
:align: center

You can view a detailed line-by-line "diff" of the changes in the "Files changed" tab of a Pull Request.
```

While this model of collaboration might seem a bit odd at first, it has a lot of
advantages: individual contributions to the project are divided into branches
and pull requests. The pull request interface allows for a detailed line-by-line
review of changes to the project and discussion. This is particularly important
for projects where changes to complex code are required. In tandem with
automated code testing (which we will discuss in {numref}`python-env`), thorough
code review mitigates merging changes to the code that introduce new bugs to the
code base. In addition to these advantages, the pull request is retained after
the change is merged, as a permanent record of these discussions. This allows all
of the collaborators on a project to revisit how decisions were made to
introduce particular changes. This is a great way to retain collective memory
about the way that a project evolved, even after collaborators on the project
graduate and/or move on to do other things.

#### More complex collaboration patterns

Here, we demonstrated a pattern that works well for a relatively small
collaboration, with a few collaborators who all know each other. But, git and
GitHub also enable much more complex patterns, that support large and
distributed collaborations between people who don't even know each other but
would like to contribute to the same project. These patterns enable the large
open-source software projects that we will start discussing in {numref}`numpy`.


```{admonition} Datalad is version control for data

As you read the section about git you might be wondering whether it is a good
idea to also store the data that you are working with within your project git
repository. The answer is that your probably shouldn't. Git was designed to
track changes to text-based files, and it would not operate very well if you
started adding data files to your repositories. This is particularly true for
very large files, like the ones that we routinely deal with in the analysis of
neuroimaging data. Fortunately, there is a git-like solution for tracking data.
This is an open-source software project called
["Datalad"](https://www.datalad.org/). Similar to git, it operates as a
command-line program that can be called with sub-commands to add data to a
repository and to track changes to the data. One of the most compelling
use-cases for datalad is in tandem with git to track changes to code and the
changes to derivatives of the code at the same time. We will not go into depth
on how to use datalad here, but to learn more about datalad and to get started
using it, you can read an openly available
[book](http://handbook.datalad.org/en/latest/) that introduces datalad and walks
you through its usage.

```

## Additional resources

To learn more about the history and more recent objections to the
"master"/"slave" terminology in computer science and engineering, you can refer
to {cite}`eglash2007broken`.

To explain the use of git and GitHub, we went through a whole sequence of
operations using the command line. However, there are several graphical user
interfaces (GUIs) for git and GitHub that are worth checking out. First of all,
GitHub has a [GUI that they developed](https://desktop.github.com/). In
addition, there is an entire page of options to choose from [in the git documentation](https://git-scm.com/downloads/guis).

To learn about more elaborate collaboration patterns for version control and
collaboration, we would recommend studying the [git workflow](https://nipy.org/nibabel/gitwash/git_development.html) used by the [nibabel](https://nipy.org/nibabel/) project. You will
learn more about what nibabel does in {numref}`nibabel`, but for now, you can
use their documentation to learn more about how you might use git.