How does github figure out a project's language?

Question

I was recently working on a github project in both JavaScript and C++, and noticed that github tagged the project as C++. If you have to pick a single language, this is probably the correct designation since the C++ code is compiled as a JavaScript library, but this made me wonder... how does github figure out what language to tag each project?

Answer 1

Update April 2013, by nuclearsandwich (GitHub support team or "supportocat"):

• the help page "My repository is marked as the wrong language" mentions using now the linguist library to determine file language for syntax highlighting and repo statistics. Linguist will exclude certain file names and paths from statistic, excluding certain vendor files and directories.

• the help page "Why isn't my favorite language recognized?" adds:

If your desired language is not receiving syntax highlighting you can contribute to the Linguist library to add it.

---

(Original answer, Oct. 2012)

This thread on GitHub support explains it:

It just sums up file sizes for each extension. Largest one "wins".

We'd like to avoid opening files up and parsing their content, as both would slow down the process... but that might be the only method of resolving conflicts like this one.

Since this is not 100% accurate, that had lead some to add:

I, too, would vote for a simple manual-override switch for the cases where the guess is wrong.

---

Note: as Mark Rushakoff mentions in his answer (upvoted), the guessing got better since then with the linguist project (open-sourced from June 2011).
You can see there are still issues though: GitHub Linguist Issues.
See here for more details:

Once the language has been detected, it is passed to Albino, a Pygments wrapper, which does the actual syntax highlighting.

And you can add linguist directives in a .gitattributes file.

Answer 2

Currently, Github's linguist project is what is used to determine language statistics, as described in this Github blog post (which came out a few months after this question was originally asked).

Answer 3

First, know that you can override the language detected for files in your repository using Linguist overrides.

Now, in a nutshell,

0. Each repository is tagged with the first language from language statistics.
1. Language statistics count the total size of files for each detected programming or markup language. Vendored, documentation, and generated files are not counted.
2. The language of each file is detected by the open source project Linguist.

---

How does Linguist detect languages?

Linguist relies on the following strategies, in order, and returns the language as soon as it found a perfect match (strategy with a single language returned).

0. Look for Emacs and Vim modelines.
1. Known filename. Some filenames are associated to specific languages (think Makefile).
2. Look for a shebang. A file with a #!/bin/bash shebang will be classified as Shell.
3. Known file extension. Languages have a set of extensions associated to them. There are, however, lots of conflicts with this strategy. The conflicting results (think C++, C and Objective-C for .h) are refined by the subsequent strategies.
4. A set of heuristic rules. They usually rely on regular expressions over the content of files to try and identify the language (e.g., ^[^#]+:- for Prolog).
5. A naive Bayesian classifier trained on sample files. Last strategy, lowest accuracy. The Bayesian classifier always takes a subset of languages as input; it is not meant to classify among all languages. The best match found by the classifier is returned.

What are unvendored and documentation files?

Linguist considers some files as vendored, meaning they are not included in language statistics. These include third-party libraries such as jQuery and are defined in the vendor.yml configuration file. You can also vendor or unvendor files in your repository using Linguist overrides.

Similarly, documentation files are defined in documentation.yml and can be changed using Linguist overrides.

How are generated files detected?

Linguist relies on simple rules to detect generated files, using both the paths and the content of files. Generated files are not counted in language statistics and are not displayed in diffs on github.com.

What about programming and markup languages?

In Linguist, each language is given a type. These types can be found in the main configuration file, languages.yml. Only the programming and markup languages are counted in statistics.

Answer 4

After some tinkering with linguist I have noticed this.

For files with a Shebang, the Shebang is considered when determining the language but seems to be evenly weighted against other tokens. This seems to be a big error because the Shebang should definitively define the language of the file.

This can cause issues with highlighting.

Answer 5

File extensions is the first thing that comes to my mind.