5

After some research and a few questions, I ended up exploring libclang library in order to parse C++ source files in Python.

Given a C++ source 

int fac(int n) {
    return (n>1) ? n∗fac(n−1) : 1;
}

for (int i = 0; i < linecount; i++) {
   sum += array[i];
}

double mean = sum/linecount;

I am trying to identify the tokens fac as a function name, n as variable name, i as a variable name, mean as variable name, along with each ones position. I interested in eventually tokenizing them.

I have read some very useful articles (eli's, Gaetan's) as well as some stack overflow questions 35113197, 13236500.

However, given I am new in Python and struggling to understand the basics of libclang, I would very much appreciate some example chunk of code which implements the above for me to pick up and understand from.

Community
  • 1
  • 1
  • 1
    You don't tokenize language elements by yourself if you have Clang or another tool; parsers have that ability built in. There's the question of how you get at result. You can see the output of a C lexer at http://stackoverflow.com/a/36681568/120163 The issue is, what is the API Clang offers to provide access to such tokens? – Ira Baxter Apr 23 '16 at 08:52
  • If libclang offers a tool of tokenizing that would even better. However, I am struggling to, firstly, get an output similar to the one you mention and, secondly, to use that output in order to handle my needs as those stated in the example in my description. Any guidelines for that? –  Apr 23 '16 at 08:57
  • My point was more about who was doing the tokenizing. If *you* are going to do it, it is unclear how clang is going to help or why you want it. ( personally doubt if you are going to do much interesting with C++ code if all you have are tokens, regardless of how you get them). I don't know a lot about Clang APIs. Maybe this answer will help you: http://stackoverflow.com/a/25371656/120163 – Ira Baxter Apr 23 '16 at 09:14
  • 1
    You should at least try write something, you have enough hints in links that you provided, and then if you have some problems post your script into question. – fghj Apr 23 '16 at 10:33

1 Answers1

12

It's not immediately obvious from the libclang API what an appropriate approach to extracting token is. However, it's rare that you would ever need (or want) to drop down to this level - the cursor layer is typically much more useful.

However, if this is what you need - a minimal example might look something like:

import clang.cindex

s = '''
int fac(int n) {
    return (n>1) ? n*fac(n-1) : 1;
}
'''

idx = clang.cindex.Index.create()
tu = idx.parse('tmp.cpp', args=['-std=c++11'],  
                unsaved_files=[('tmp.cpp', s)],  options=0)
for t in tu.get_tokens(extent=tu.cursor.extent):
    print t.kind

Which (for my version of clang) produces

TokenKind.KEYWORD
TokenKind.IDENTIFIER
TokenKind.PUNCTUATION
TokenKind.KEYWORD
TokenKind.IDENTIFIER
TokenKind.PUNCTUATION
TokenKind.PUNCTUATION
TokenKind.KEYWORD
TokenKind.PUNCTUATION
TokenKind.IDENTIFIER
TokenKind.PUNCTUATION
TokenKind.LITERAL
TokenKind.PUNCTUATION
TokenKind.PUNCTUATION
TokenKind.IDENTIFIER
TokenKind.PUNCTUATION
TokenKind.IDENTIFIER
TokenKind.PUNCTUATION
TokenKind.IDENTIFIER
TokenKind.PUNCTUATION
TokenKind.LITERAL
TokenKind.PUNCTUATION
TokenKind.PUNCTUATION
TokenKind.LITERAL
TokenKind.PUNCTUATION
TokenKind.PUNCTUATION
Andrew Walker
  • 40,984
  • 8
  • 62
  • 84
  • Looks great! However, as can be seen [here](http://stackoverflow.com/questions/36821176/libclang-error-specified-module-could-not-be-found) I am struggling to make libclang run. Any ideas to overcome the error so I can test your answer, please? –  Apr 24 '16 at 10:34
  • 1
    Could you please provide a link of where I can read about libclang methods like `parse()`, `get_tokens()` etc.? Having a hard time locating a reference source and/or docs. –  Apr 25 '16 at 15:44