2

I am feeling really dumb right now but I am not understanding the concept of tokens. I am currently reading Dr. Stroustrup's textbook, Programming Principals and Practices, am and in Chapter 6 learning about tokens and writing programs in C++. I have gotten every concept in the book thus far, but this has really got me stuck. Can anyone please dumb it down for me? Thank you!

I have honestly tried to watch YouTube videos that have not been of any help along with reading the textbook that hasn't helped either.

273K
  • 29,503
  • 10
  • 41
  • 64
Alicia
  • 59
  • 4
  • [A token is the smallest element of a C++ program that is meaningful to the compiler.](https://learn.microsoft.com/en-us/cpp/cpp/character-sets?view=msvc-170) – Jason Nov 11 '22 at 04:04
  • Also see [**What exactly is a token**, in relation to parsing](https://stackoverflow.com/questions/5639177/what-exactly-is-a-token-in-relation-to-parsing) and [a token is a lexical element,and exactly what it is depends on the problem you are addressing.](https://stackoverflow.com/questions/45394225/tokens-in-c-and-how-they-are-used). – Jason Nov 11 '22 at 04:07
  • Tokens are comparable to words in English. You read words, not letters. – chris Nov 11 '22 at 04:07

1 Answers1

4

A token corresponds roughly to a word of source-code. For example, in a line of source code like this:

int a = 10;

The tokens would be "int", "a", "=", "10", and ";".

In a compiler, it's the job of the tokenizer to interpret the source code files (which are really just plain-text files, i.e. a series of characters) into a series of tokens (which have some sort of specific meaning in the context of the language). The series of tokens will then be fed into the next stage of the compiler for further processing.

Jeremy Friesner
  • 70,199
  • 15
  • 131
  • 234