8

Given a string (str), how can one convert that into a TokenStream in Rust?

I've tried using the quote! macro.

let str = "4";
let tokens = quote! { let num = #str; }; // #str is a str not i32

The goal here is to generate tokens for some unknown string of code.

let thing = "4";
let tokens = quote! { let thing = #thing }; // i32

or

let thing = ""4"";
let tokens = quote! { let thing = #thing }; // str
Peter Hall
  • 53,120
  • 14
  • 139
  • 204
lovelikelando
  • 7,593
  • 6
  • 32
  • 50
  • 1
    Your question doesn't make sense. In the title you want to know, how to convert a String into a TokenStream, but in your text, you are complaining, that `str` is not an `i32`, which is a complet different question. – hellow Jan 13 '19 at 00:32
  • Not just that, what else would it be? You define it as a string literal by quoting it. try `let str = 4 as i32` and watch it turn into an `i32` literal in `quote!`. – Sébastien Renauld Jan 13 '19 at 00:34
  • This certainly might not be possible. I'll update the question with more details to clarify. – lovelikelando Jan 13 '19 at 00:36
  • Your title still does not match your question. – hellow Jan 13 '19 at 00:42
  • 1
    I honestly feel like you're both using the wrong tool, and trying to solve a problem different to the one you have. Why are you trying to get a `TokenStream` from code you already know the structure of, and where does the additional data on types come from (even in your example, the type of `thing` could very well be an `u8`, or an `i16`, or anything else that `4` conforms to). What are you *actually* trying to parse? – Sébastien Renauld Jan 13 '19 at 00:57
  • @hellow I disagree. I’d like to know how to convert strings such as “(1,2)”, “func(1)”, “4” to TokenStreams. – lovelikelando Jan 13 '19 at 00:59
  • I do think I’m going about this the wrong way though. I’m writing a macro and I’ve been managing the attributes to the macro by using regexes. I then want to go back to code during the generation step. It’s seeming like I should just work with the attributes I’m the TokenSteam that is provided to me. – lovelikelando Jan 13 '19 at 01:00
  • What is the actual purpose of the macro? From this and your other question, it feels like the examples you give have very little to do with what you're actually attempting to do. – Sébastien Renauld Jan 13 '19 at 01:43
  • You haven't even specified where this `TokenStream` comes from. Do you mean [`proc_macro::TokenStream`](https://doc.rust-lang.org/proc_macro/struct.TokenStream.html) or [`proc_macro2::TokenStream`](https://docs.rs/proc-macro2/0.4.24/proc_macro2/struct.TokenStream.html), or some other one? – Shepmaster Jan 13 '19 at 21:18

1 Answers1

20

how can one convert [a string] into a TokenStream

Rust has a common trait for converting strings into values when that conversion might fail: FromStr. This is usually accessed via the parse method on &str.

proc_macro2::TokenStream

use proc_macro2; // 0.4.24

fn example(s: &str) {
    let stream: proc_macro2::TokenStream = s.parse().unwrap();
}

proc_macro::TokenStream

extern crate proc_macro;

fn example(s: &str) {
    let stream: proc_macro::TokenStream = s.parse().unwrap();
}

You should be aware that this code cannot be run outside of the invocation of an actual procedural macro.

Shepmaster
  • 388,571
  • 95
  • 1,107
  • 1,366