Vector losing contents between files in C++

Question

I have some methods in lexer.h which make use of a Vector made of Tokens.

In this method void getNextToken() I am making use of the said vector where I am adding new tokens to it.

The problem is, that when I go to a different file, I am trying to access ANOTHER method which makes use of this vector, but it is crashing with an out of bounds error (most probably it's being deferenced or something)

Is there a way how I can fix this?

The methods in concern are:

    Token* nextToken()
    {
        if (it!= tokensUsed.end())
        {
            // we Assigned what is found in the iterator it (of the vector)
            // so we get the data found in that pointer
            itrToken = &*it;
            //Move Iterator forward
            it ++;
            return itrToken;
        }
    }

    /*
        Used in Parser to go get the PREVIOUS Tokens
    */
    Token* prevToken()
    {
        itrToken --;

        if (it!= tokensUsed.begin())
        {
            itrToken = &*this->it;
            return itrToken;
        }
    }

void getNextToken()
{
  //CODE  ADDING TOKENS
//EXAMPLE
if (ch == '"')
        {
            addAndGetNext(ch);
            cout << "STRING:  " << strBuffer << endl; //TEST
            //create new token and push it into the vector

            tk = new Token (Token::ttString, strBuffer, row, col);
            tokensUsed.push_back(*tk); //Add the new token to the Vector
            startNewString(); //Clear the string
        }
        tokenMatch = true;
}

The above is just partial code, to show an example.

Now in Parser.h I am using this method to call the lexer.h:

   void relOpP()
    {
        Token* tk = nextToken();
        if (tk -> getType() == Token::ttString)
        {
            cout << "Ture";
        }
    }

which calls the Lexer's nextToken() it crashes, and when I tried checking it's contents it goes outofBounds error (and CodeBlocks giving me a SIGSEGV error)

I know it's something from the pointers that it's going awry, but how can I fix it?

Edit:

These are the global variables I have declared:

vector<Token>::iterator it;
vector<Token> tokensUsed;

 Token* itrToken; // used for iterator

 bool checkQuote = false;
 Token* tk = new Token (syNewToken, "", 1,0);

 Token token; // Creates an instance of the class Token found in the file token.h
                Token* t;

SAMPLE CODE:

main.cpp

#include <iostream>

#include "lexer.h"
#include "parser.h"
using namespace std;

int main()
{
    Lexer* l;

    l -> getNextToken();

    Parser p(l);
    p.relOpP();
}


    Token (int type, string sBuffer, int rRow, int cCol)
    {
        this->tType = type;
        this->strBuffer = sBuffer;
        this->row = rRow;
        this->col = cCol;
}

parser.h

#ifndef PARSER_H_INCLUDED
#define PARSER_H_INCLUDED


#include <string>
#include <vector>
#include "lexer.h"
#include "token.h"

using namespace std;

class Parser{
private:

    Lexer* lexer;
    string tree = "";

public:

    Parser (Lexer* l)
    {
        this -> lexer = l;
    }

    Token nextToken()
    {
        Token tk = lexer -> nextToken();
        return tk;
    }



    void relOpP()
    {
        Token tk = nextToken();
        if (tk.getType() == 1)
        {
            cout << "Ture";
        }
    }



#endif // PARSER_H_INCLUDED
};

token.h

#ifndef TOKEN_H_INCLUDED
#define TOKEN_H_INCLUDED


#include <iostream>

using namespace std;


class Token
{
private:

    int tType; //identifier or reserved by compiler?
    string strBuffer; //string found in buffer at that moment
    int row;
    int col;

public:


    enum tokenType
    {
        tkString
    };

    Token()
    {

    }

    // The instance of a token with 4 parameters resulting the type, the contents of the string that represents that type
    // the row it is found in and the column.
    Token (int type, string sBuffer, int rRow, int cCol)
    {
        this->tType = type;
        this->strBuffer = sBuffer;
        this->row = rRow;
        this->col = cCol;
    }

       Token (Token* getT)
    {
        this-> tType = getT -> tType;
        this->strBuffer = getT -> strBuffer;
        this->row = getT -> row;
        this->col = getT -> col;
    }


    int getType ()
    {
        return this->tType;
    }

    //return the string contents
    string getBuffer()
    {
        return this->strBuffer;
    }

    //return row
    int getRow()
    {
        return row;
    }

    //return col
    int getCol ()
    {
        return col;
    }
};
#endif // TOKEN_H_INCLUDED

Lexer.h

    #ifndef LEXER_H_INCLUDED
    #define LEXER_H_INCLUDED
    
    #include "token.h"
    #include <vector>
    using namespace std;
    
    class Lexer
    {
    
    private:
    
         Token tk = new Token (1, "", 1,0);
         vector<Token>::iterator it;
         vector<Token> tokensUsed;
        Token itrToken; // used for iterator
    
    public:
    
          Token nextToken()
            {
                if (it!= tokensUsed.end())
                {
                    // we Assigned what is found in the iterator it (of the vector)
                    // so we get the data found in that pointer
                    itrToken = &*it;
                    //Move Iterator forward
                    it ++;
                    return &itrToken;
                }
                else
                {
                    cout << "ERROR" << endl;
    
                }
                return nullptr;
    
            }
    
             void getNextToken()
             {
                 cout << "Test" << endl;
                 string strBuffer = "test";
                 int row = 0;
                 int col = 0;
                 tk = new Token (1,strBuffer,row,col);
             }
    
    
    };
    
    
    #endif // LEXER_H_INCLUDED

There is no reason to use pointers (and certainly not `new`) in any of the places you are using them. Cut it out. — Benjamin Lindley, May 29 '14 at 16:40
What do you mean? My Token Vector is mostly based as `Vector tokensUsed` ... is that what you are saying? remove all `*`? — DodoSombrero, May 29 '14 at 16:43
Yes, I am saying that. Although what you say in that comment does not match your code. You are dereferencing the pointer when pushing it back onto the vector, indicating that the vector holds `Token` objects (as it should), not `Token` pointers. — Benjamin Lindley, May 29 '14 at 16:46
In `getNextToken`, you allocate a new `Token`, but then dereference the pointer before passing it to `push_back`, so you leak the memory. If you have `std::vector tokensUsed`, then this won't even compile. If you're using a custom `Vector` rather than `std::vector`, then that is probably your problem. — Chris Dodd, May 29 '14 at 16:48
I have edited the code just to show you what I have as Global Variables maybe they would help understand my situtation better — DodoSombrero, May 29 '14 at 16:50
What is the definition of `Token`? You appear to be passing `char *`s to temporaries in the constructor, so if it does not not copy the those to `std::string`s (or equivalent), you likely have dangling pointers there. — Chris Dodd, May 29 '14 at 16:52
Token, is found in another class, called Token, which has 4 parameters in it, am adding it also to the code check 2nd EDIT — DodoSombrero, May 29 '14 at 16:55
Please present a minimal but complete, compileable example (main function and all) which demonstrates the problem. http://sscce.org/ — Benjamin Lindley, May 29 '14 at 16:57
I'm still writing the minimal, code... when it is ready.. where will I upload it? (as I've never done this before) — DodoSombrero, May 29 '14 at 17:04
-1 for uploading fake code. Why didn't you upload actual code which you compiled? Because this code definitely did not compile, and your problem description indicates you have code which you actually executed, but got a crash. (i.e. you already compiled it) — Benjamin Lindley, May 29 '14 at 18:04
I only changed a bit of it as otherwise I have to include like 5 other classes... On mine it worked.. I'm using codeblocks.. I compiled it and executed.. This was the most minimal I could do — DodoSombrero, May 29 '14 at 18:15
You have, what appears to be a constructor for the `Token` class, after your `main` function. But it's not contained in your `Token` class definition, and it has no qualifier indicating that it is part of the `Token` class (i.e. `Token::Token(...)`). You're telling me that compiles? — Benjamin Lindley, May 29 '14 at 18:28
Believe it or not it does.. it display "TEST" and then crashes... — DodoSombrero, May 29 '14 at 18:33

PeterSW · Answer 1 · 2014-05-29T17:12:14.873

2

In nextToken() and prevToken() there is no return for the case the if evaluates to false. The return value in that case is not very likely to be something (it could be anything...) that you can then dereference.

If you want to keep the current design you should return nullptr or (NULL if you don't have C++11 support) in that case. Then you need to change any code that uses the result of those functions to check if the pointer is valid before dereferencing it.

You would probably be better changing your design to not involve so much manual pointer manipulation. But to fix up your current version you should change your prevToken and nextToken to be something along the lines of:

Token* nextToken()
{
    if (it!= tokensUsed.end())
    {
       ...
        return itrToken;
    }
    else
    {
       return nullptr;
    }
}

Then if tk is the result of calling one of these functions you must not use tk-> or *tk if it is nullptr. Any code wanting to work with the result will need to check first.

So for example you could change you if statement to be:

if (tk && // Make sure tk is not nullptr
    tk -> getType() == Token::ttString)
{
    ...

edited May 29 '14 at 17:12

answered May 29 '14 at 16:43

PeterSW

4,921
1
24
35

I added a `cout` to that `else` stating ERROR.. Yeah it's outputting error, but even with the `nullptr` it is still crasing – DodoSombrero May 29 '14 at 16:58
@DodoSerebro Is it crashing when you do: `tk -> getType()` by any chance? – PeterSW May 29 '14 at 17:00
in my original coding, Yes, it is.. as it trying to access the data inside – DodoSombrero May 29 '14 at 17:02
Any Idea, why it is crashing there?? there is still a nullptr – DodoSombrero May 29 '14 at 17:56
@DodoSerebro Are you using `->` or `*` on a pointer holding `nullptr`? Oh I see you've posted more code... – PeterSW May 29 '14 at 17:59
Only what you see... but it's that like of code `tk -> getType()` that is crashing all, as even if I'm not returning anything, just `cout` it will still crash – DodoSombrero May 29 '14 at 18:17
I see that you are dereferencing `nullptr`s specifically in your very unusual constructor: `Token (Token* getT)` – PeterSW May 29 '14 at 18:35
Without that, `Tokens` do not work... as in I would need to add an `*` after each one – DodoSombrero May 29 '14 at 18:42

score 1 · Answer 2 · edited May 23 '17 at 12:31

There are too many problems with your code for me to address them all in this post. The first, and most obvious one is this.

In the main function:

Lexer* l;
l -> getNextToken();

Here, you did not create a Lexer object. You just created an uninitialized pointer to one. Then you called a member function as if it pointed to an object. This is undefined behavior. You then pass this pointer to your Parser class, which continues to treat it as a valid object, resulting in more undefined behavior.

There are many other problems with your code, but most of them have to do with your mishandling of pointers, indicating a lack of understanding of how they work. The best suggestion for you is to stop using them entirely. There is no reason you need to use any pointers whatsoever to do what you are doing. If you can't figure out how to do what you are trying to do without pointers, it is because of a lack of fundamental understanding of the language. You need to read a C++ book, to completion. Here's a list of some good ones.

The Definitive C++ Book Guide and List

Vector losing contents between files in C++

main.cpp

parser.h

token.h

Lexer.h

2 Answers2