26

I'm trying to use this function in a C program that needs to be able to compile in Linux and Windows. At first I tried using strtok_r, but then when I compiled on windows, it complained about the function not existing and said it would assume it's an extern function, but then failed. I then used strtok_s and it compiled! Then I tried on Linux but now it's complaining that there is an "undefined reference to 'strtok_s'".

Is one a windows only function and the other a linux function??? What can I do to make it compile on both?

petranaya
  • 769
  • 1
  • 6
  • 23
  • 2
    [`strtok_r()`](http://pubs.opengroup.org/onlinepubs/9699919799/functions/strtok_r.html) is POSIX; [`strtok_s()`](http://msdn.microsoft.com/en-us/library/ftsafwz3.aspx) is Windows. Use plain `strtok()` which is available in all implementations *(supporting C89)*. – pmg Jan 26 '12 at 16:37
  • 4
    The problem is I need to strtok a string while strtok-ing another string, which is why I need this one since it saves state. – petranaya Jan 26 '12 at 16:39
  • 3
    @pmg - unless you need thread or buffer overrun safety – Martin Beckett Jan 26 '12 at 16:40
  • Well ... then I refer to [R..'s answer](http://stackoverflow.com/a/9021604/25324). – pmg Jan 26 '12 at 16:41
  • 1
    @MartinBeckett Wait what? Where does it save the state? There are only two arguments, one is NULL (on the second token), the other is what character you want to find. I want to be able to strtok a string, find something in that string and strtok that one, then go back to the original string and continue to strtok it. – petranaya Jan 26 '12 at 16:46
  • @pmg - sorry was thinking of something else, my mistake! – Martin Beckett Jan 26 '12 at 16:47
  • 1
    @user390608: The problem you're encountering is exactly why `strtok` is such a horrible interface. It is *stateful* and *non-reentrant*. – R.. GitHub STOP HELPING ICE Jan 26 '12 at 16:49
  • 7
    This is an ancient question by now, but I felt the need to point out that strtok_s is C11, not just Windows. – remmy Jan 25 '15 at 15:33
  • @kyrias close - it's C11 Annex K, which Ulrich Drepper has fought tooth-and-nail to keep from libc users. No, really, that's why linux devs can't use it. One guy. – Alexander Riccio Dec 20 '17 at 06:12

6 Answers6

43

strtok_s is simply the Windows version of strtok_r which is standard everywhere else.

One (common I would think) way to make a program portable when it comes to functions like strtok_s/strtok_r is to use the preprocessor:

#if defined(_WIN32) || defined(_WIN64)
/* We are on Windows */
# define strtok_r strtok_s
#endif

As the prototypes and functionality is the same, you can now use only strtok_r.

Some programmer dude
  • 400,186
  • 35
  • 402
  • 621
  • This would probably break if compiling on cygwin which reports itself as windows but has posix interfaces like `strtok_r` already defined. Using something like `#ifndef HAVE_STRTOK_R` (and detecting it in the build scripts) would be better. – R.. GitHub STOP HELPING ICE Jan 26 '12 at 16:38
  • 6
    @R..: cygwin does not in fact define `_WIN32` (or `_WIN64`) for precisely this reason -- its commonly used to identify code that is non-unix like. Cygwin defines `__CYGWIN__` and `__unix__` instead – Chris Dodd Apr 23 '13 at 22:17
  • 3
    It may be good to know that there is an other `strtok_s` in C11 with a completely different signature: http://en.cppreference.com/w/c/string/byte/strtok – Tor Klingberg Jan 06 '17 at 14:37
  • 2
    @ChrisDodd There are still problems on MinGW. MinGW predefines `_WIN32` but it supports `strtok_r`. I think it is better to check the `_MSC_VER` macro. – Victor Feb 15 '19 at 03:01
11

I don't have enough reputation to comment on other answers, so I'll have to provide my own.

1) To address this statement:

"strtok_s is a buffer overrun safe version of strtok on Windows. The standard strtok on windows is thread safe..."

This is not true. strtok_s is the thread safe version for the MSVC compiler. strtok is not thread safe!

2) To address this statement:

"This would probably break if compiling on Cygwin which reports itself as windows but has POSIX interfaces like strtok_r already defined."

Again, not true. The difference is which compiler you use. When using Microsoft's Visual C++ compiler, MSVC, the function is strtok_s. Another compiler, such as the GNU Compiler Collection, GCC, may use a different standard library implementation such as strtok_r. Think compiler, not target platform, when identifying which function to use.

In my opinion, Joachim Pileborg's answer is the best one on this page. However, it needs a small edit:

#if defined(_WIN32) /* || defined(_WIN64) */
#define strtok_r strtok_s
#endif

Both _WIN32 and _WIN64 are predefined macros provided by the MSVC compiler. _WIN64 is defined when compiling a 64 bit target. _WIN32 is defined for both 32 and 64 bit targets. This is a compromise that Microsoft made for backwards compatibility. _WIN32 was created to specify the Win32 API. Now you should consider _WIN32 to specify Windows API -- it is not specific to a 32 bit target.

informatik01
  • 16,038
  • 10
  • 74
  • 104
brad-tot
  • 783
  • 7
  • 9
  • 3
    You are wrong about the first point: strtok **is** thread-safe on Windows, since two distinct threads can safely use that function within the same process, even if strtok have other problems as explained in [another answer](http://stackoverflow.com/a/16821026/656988) by @JameyKirby. – AntoineL Feb 24 '14 at 20:10
  • MinGW predefines `_WIN32` but it supports `strtok_r`, so it is better to check the `_MSC_VER` macro. – Victor Feb 15 '19 at 03:02
11

Both of these functions are really ugly, unintuitive idioms for parsing strings, and usually fail to meet your particular application's requirements in subtle ways. Even moreso for the plain strtok in standard C. Just throw them out and write your own code to iterate over the char array and break it up as needed. strchr, strspn, and strcspn can be helpful in doing this or you can just work from scratch on the array.

R.. GitHub STOP HELPING ICE
  • 208,859
  • 35
  • 376
  • 711
  • 2
    Can you explain a little more about why these functions are bad? It seems more intuitive to me than going through character by character and searching for things. – petranaya Jan 26 '12 at 16:52
  • 4
    (1) the separator char is overwritten/lost, so unless you only have one possible separator, you're losing information. (2) multiple consecutive separators are treated as one, which makes sense if your separator is whitespace but not if it's a comma, for instance.. – R.. GitHub STOP HELPING ICE Jan 26 '12 at 17:01
  • 5
    Please note that surely in *some* situations, the `strtok` semantics are just what you want and the only issue is the statefulness (which `strtok_r` and `strtok_s` fix). I just find that every time I tried to use them, there's at least one aspect of my parsing needs they don't support, and it ends up being easier to just write the code directly. – R.. GitHub STOP HELPING ICE Jan 26 '12 at 17:03
  • 9
    -1: For parsing a string that's been read into a temporary buffer, where you want to throw away the delimeters and you only want the strings valid until you're done parsing, strtok is precisely what you want and strtok_r/strtok_s is much better as it is reentrant. – Chris Dodd Apr 23 '13 at 22:19
7

Just to clarify. strtok is thread safe in Windows. strtok uses a TLS variable to maintain the last pointer for each thread. However, you can not use strtok to interleave access to more than one token string per thread. strtok_r and strtok_s both address this interleaving problem by allowing the user to maintain the context via the third parameter. Hope this helps.

Md. Sabbir Ahmed
  • 850
  • 8
  • 22
Jamey Kirby
  • 143
  • 2
  • 6
  • Although not helpful for the OP, who is looking for a portable solution, this was very helpful for me to find out why my legacy code crashes on Linux but not on Windows. As I thought I knew how strtok behaves, I didn't care to read the Windows strtok documentation thoroughly... – Matthias Jul 06 '21 at 05:10
5

strtok_r is a thread safe version of strtok on POSIX systems

strtok_s is a buffer overrun safe version of strtok on Windows. The standard strtok on windows is thread safe, so strtok_s should be.

Martin Beckett
  • 94,801
  • 28
  • 188
  • 263
  • `strtok_r` is part of POSIX and has been for years (decades?) – R.. GitHub STOP HELPING ICE Jan 26 '12 at 16:36
  • I still think the answer isn't terribly useful since it just covers what OP (mostly) already knew, not how to solve the problem. – R.. GitHub STOP HELPING ICE Jan 26 '12 at 16:45
  • @R - the solution is you can use strok_r on Posix and strtok_s on Windows, Joachim's solution describes how. OR you can program windows using only the Posix subsystem – Martin Beckett Jan 26 '12 at 16:46
  • @MartinBeckett it's C11 Annex K standard, not just Windows :) – Alexander Riccio Dec 20 '17 at 06:13
  • 2
    @AlexanderRiccio Microsoft `strtok_s` and C11 `strtok_s` are completely different! Microsoft `strtok_s` has only 3 parameters, while C11 `strtok_s` has 4 parameters. The prototype of Microsoft `strtok_s` is `char* strtok_s(char* str, const char* delimiters, char** context);` while the prototype of C11 `strtok_s` is `char *strtok_s(char *restrict str, rsize_t *restrict strmax, const char *restrict delim, char **restrict ptr);`. – Victor Feb 15 '19 at 03:11
1

MinGW also predefines _WIN32 but it supports strtok_r, so I don't think it is a good idea to check the _WIN32 macro. It is better to check the _MSC_VER macro, which is the macro for Microsoft Visual Studio.

#ifdef _MSC_VER
#define strtok_r strtok_s
#endif

WARNING: Microsoft strtok_s and C11 strtok_s are completely different! Microsoft strtok_s has only 3 parameters, while C11 strtok_s has 4 parameters, so it can be a compatible issue in the future.

The prototype of Microsoft strtok_s is

char* strtok_s(char* str, const char* delimiters, char** context);

The prototype of C11 strtok_s is

char *strtok_s(char *restrict str, rsize_t *restrict strmax, const char *restrict delim, char **restrict ptr);
Victor
  • 1,303
  • 1
  • 10
  • 28