21

I want to count the lines of code in a multi-file Python project as accurately as possible, but without including comments, docstrings or blank lines in the total.

I first tried using cloc, which is available as a Debian package. But cloc treats most docstrings as code - even though they are comments. (Update: no longer - recent versions of cloc now treat Python docstrings as comments.)

I notice some comments below saying that docstrings should be included in the total because they might be used by the code to influence behaviour at runtime and hence count as part of the programs code/data/config. A prominent example of this is 'ply', which asks you to write functions with docstrings which, as I recall, contain grammar and regular expressions which are central to the program's operation. However, this seems to me to be very much a rare exception. Most of the time docstrings act just like comments. Specifically, I know for a fact that is true for all the code I want to measure. So I want to exclude them as such from my line counts.

Jonathan Hartley
  • 15,462
  • 9
  • 79
  • 80
Wolkenarchitekt
  • 20,170
  • 29
  • 111
  • 174
  • 20
    I'd say counting comments _is the right way_, because in general __the comments are just as valuable as the actual code lines__ – Kimvais Jan 31 '12 at 08:45
  • 4
    @Kimvais I must say I've had the opposite experience in 20 years of programming - comments are generally worthless because the compiler never checks them :-) – Adrian Cornish Jan 31 '12 at 08:47
  • 8
    Python docstrings *are* code - they become the `__doc__` attribute of the function and can contain tests. Maybe you need to define what you mean by 'lines of code' – Hamish Jan 31 '12 at 08:48
  • 9
    @AdrianCornish: LOC count is pretty worthless, too, so that works out just fine then. – Cat Plus Plus Jan 31 '12 at 08:49
  • 4
    @AdrianCornish WTF are you talking about .. python compiler? and in your 20 years of programming you learned that "comments are generally worthless"? – wim Jan 31 '12 at 09:10
  • 1
    Docstrings are code but not all multiline strings are docstrings, they may be used as a multiline comment without generating code. – Rik Poggi Jan 31 '12 at 09:27
  • 1
    @wim I can't count the number of times I'v read completely outdated and misleading comments. I believe that was Adrian Cornish' point. – ychaouche Feb 04 '14 at 12:20
  • 1
    At one place I worked (FWIW, the best software engineering team I've ever worked on) we called comments "lies". As in "This code has no tests, but he did write lots of lies about it." – Jonathan Hartley Sep 24 '15 at 09:09
  • The correct way to answer this question is to examine the parsed Python bytecode, or maybe the AST. Any other approaches are fraught with peril and will fail to work properly in many different circumstances. I don't have a full working solution however - just that vague hunch. – Jonathan Hartley Sep 24 '15 at 09:13
  • @Kimvais Say you are counting lines to get a measure on feature creep. You want to keep the number of code lines limited, but it would be counter productive to punish comments. – Thomas Ahle Aug 29 '16 at 10:35
  • @ThomasAhle that's a perfect example of __you get what you measure__ - counting lines is quite stupid way to measure feature creep... – Kimvais Aug 31 '16 at 12:13

4 Answers4

8

Tahar doesn't count the docstrings. Here's its count_loc function : 

def count_loc(lines):
    nb_lines  = 0
    docstring = False
    for line in lines:
        line = line.strip()

        if line == "" \
           or line.startswith("#") \
           or docstring and not (line.startswith('"""') or line.startswith("'''"))\
           or (line.startswith("'''") and line.endswith("'''") and len(line) >3)  \
           or (line.startswith('"""') and line.endswith('"""') and len(line) >3) :
            continue

        # this is either a starting or ending docstring
        elif line.startswith('"""') or line.startswith("'''"):
            docstring = not docstring
            continue

        else:
            nb_lines += 1

    return nb_lines
ychaouche
  • 4,922
  • 2
  • 44
  • 52
  • 12
    Thank you for the reasonable recommendation, and for not making preposterous and pontificating claims, like your fellow responders, about docstrings being code. Lines of code _is_ a valid (and in fact the best: http://herraiz.org/blog/2010/11/22/making-software-is-out/) measure of code complexity and when I need that complexity to reflect the raw source code (rather than my copious amount of _math_ notes in docstrings), I need to omit docstrings! – Ahmed Fasih Feb 19 '13 at 14:18
  • 3
    I beleive that the doc in docstrings is for documentation – ychaouche Jun 23 '13 at 20:42
  • The above code would fail on docstrings which use single quotes, or on some regular strings which use triple quotes. The right way to solve this problem is to look at the AST. – Jonathan Hartley Feb 03 '14 at 10:09
  • @JonathanHartley Can you provide an example where the code would possibly fail ? – ychaouche Feb 03 '14 at 17:42
  • @ychaouche. Hey. Any docstring which doesn't use triple quotes will be counted as code line. Conversely, any regular code which uses triple-quotes will be counted as a docstring (see the example in wim's answer.) – Jonathan Hartley Feb 04 '14 at 11:51
  • ...and, on reflection, it fails if the docstring ends with a line containing text then the closing triple quoets. And in lines consisting of just four or five consecutive quote chars. This simply isn't a suitable way to try and detect docstrings. – Jonathan Hartley Feb 04 '14 at 11:55
  • @JonathanHartley I tried the function on OP's code and correctly outputs 2, discarding the docstring. – ychaouche Feb 04 '14 at 12:14
  • But you are correct it won't work if the docstring doesn't use triple (simple or double) quotes, it only knows about """ docstring """ and ''' docstring '''. – ychaouche Feb 04 '14 at 12:17
  • 1
    @ychaouche The OP's code doesn't fall into any of the categories I enumerated. – Jonathan Hartley Feb 04 '14 at 13:37
7

Comment lines can be lines of code in python. See doctest for example.

Moreover, you will have trouble to find a sensible/reliable way to consider a case like this as being a comment or code:

foo = ('spam', 
       '''eggs
          eggs
          eggs'''
       '''more spam''',
       'spam')

Just count the comment lines as well, I think most programmers will agree it is as good a measure for whatever you are actually trying to measure.

wim
  • 338,267
  • 99
  • 616
  • 750
  • 2
    Disagree. While technically docstrings are compiled and accessible from code, the vast predominance of their usage and semantic is a comments. They should be excluded from line counts. The way to detect ambiguous looking cases like the one in this answer is to do the line count using the AST. – Jonathan Hartley Feb 03 '14 at 10:06
7

It is probably correct to include Python docstrings in a "lines of code" count. Normally a comment would be discarded by the compiler, but docstrings are parsed:

See PEP 257 - Docstring Conventions:

A docstring is a string literal that occurs as the first statement in a module, function, class, or method definition. Such a docstring becomes the __doc__ special attribute of that object.

...

String literals occurring elsewhere in Python code may also act as documentation. They are not recognized by the Python bytecode compiler and are not accessible as runtime object attributes..

In other words, docstrings are compiled and constitute, in a very real way, the code of the program. Additionally, they're commonly used by the doctest module for unit testing, as usage strings for command line utilities, and so on.

Hamish
  • 22,860
  • 8
  • 53
  • 67
  • 4
    Disagree. While docstrings are compiled and can be used by the code, their use and semantic is as comments. They should be excluded from any meaningful line count. – Jonathan Hartley Feb 03 '14 at 10:07
  • 3
    @JonathanHartley personally I think that "compiled and can be used by the code" is a good argument *for* it being counted. – Hamish Feb 09 '14 at 20:07
  • 1
    Hey. I guess I feel the opposite because even though they can be used by the code, they almost never are. By which I mean, yes they are used by 'pydoc' et al, but I think the only program I've seen that stores data in docstrings and then examines that data is David Beazley's 'Ply'. So it's very rare. If you're comparing two modules to see which contains more code, and one has docstrings but the other does not, it seems most useful to me to exclude the docstrings and get the result 'they are about the same'. – Jonathan Hartley Feb 10 '14 at 10:31
  • No part of my project relies on the docstrings in any way whatsoever. I just want to get the number of Python instructions in my program, without my huge docstrings included, so I can check my ratio of production code to test code. For my purposes it makes zero sense to include them in a "lines of code" count. – Dan Passaro Oct 29 '14 at 16:29
  • @leo-the-manic ok, well do whatever you need to do for "your purposes". Note that "number of Python instructions" is totally different to LOC. "number of instructions" might be a better metric, for some definitions of "better". – Hamish Nov 02 '14 at 22:18
  • I'm not interested in counting "instructions," which I assume implies some kind of inspection of the bytecode. I want lines of non-blank, non-comment source code. – Dan Passaro Nov 03 '14 at 05:06
  • @leo-the-manic You're the one that said "I just want to get the number of Python instructions in my program" :). If you know what you want to count, count that. Not sure what the problem is. – Hamish Nov 03 '14 at 21:46
  • My problem is the question asks how to get a line count without docstrings, and the first answer is, "Well you probably want the docstrings." No, I don't. On a side note, I don't see how docstrings could reasonably constitute 'the code of the program' when they are stripped away if you run Python with the `-OO` flag. :) A more direct answer to the question is located further down the page. – Dan Passaro Nov 03 '14 at 22:36
3

Have you looked at http://www.ohloh.net/p/ohcount - always been pretty on the money for me - although I do not use python

Adrian Cornish
  • 23,227
  • 13
  • 61
  • 77
  • 2
    Thanks, but like cloc this tool also counts docstrings with triple apostrophes as code, so it's also not really Python-aware. – Wolkenarchitekt Jan 31 '12 at 08:51