Shared GAE datastore, Go <-> Java, regexp.FindStringIndex index shifting (byte-index vs utf-8-char-index)

Question

Short version: This prints 3, which makes sense because in Go strings are basically a slice of bytes, and it takes three bytes to represent this character. How can I get len, and regexp functions to work in terms of characters, not bytes.

package main
import "fmt"
func main() {
    fmt.Println(len("ウ"))//returns 3
    fmt.Println(utf8.RuneCountInString("ウ"))//returns 1
}

Background:

I'm saving text into the GAE datastore using JDO (Java).

Then I'm processing the text using Go, specifically I'm using regexp.FindStringIndex and saving the index to the datastore.

Then back in Java land I send the unmodified text, and index to the GWT client via json.

Somewhere along the way the indexes are 'shifting', so by the time its on the client, they are off.

It seems the issue has to do with character encoding, I'm assuming Java/Go are interpreting the text (indexes) differently utf-8 char/byte?. I see references to Runes in the regexp package.

I think I can either make regexp.FindStringIndex return byte indexes in go, or make GWT client understand the utf-8 indexes.

Any suggestions? I should be using UTF-8 incase I need to internationalize the app in the future, right?

Thanks

EDIT:

Also when I was finding the index using Java on the server things just worked.

On the client (GWT) I'm using text.substring(start,end)

TEST:

package main

import "regexp"
import "fmt"

func main() {
    fmt.Print(regexp.MustCompile(`a`).FindStringIndex("ウィキa")[1])
}

The code outputs 10, not 4.

The plan is to get FindStringIndex to return 4, any ideas?

Update 2: Position Conversion

func main() {
    s:="ab日aba本語ba";
    byteIndex:=regexp.MustCompile(`a`).FindAllStringIndex(s,-1)
    fmt.Println(byteIndex)//[[0 1] [5 6] [7 8] [15 16]]

    offset :=0
    posMap := make([]int,len(s))//maps byte-positions to char-positions
    for pos, char := range s {
        fmt.Printf("character %c starts at byte position %d, has an offset of %d, and a char position of %d.\n", char, pos,offset,pos-offset)
        posMap[pos]=offset
        offset += utf8.RuneLen(char)-1
    }
    fmt.Println("posMap =",posMap)
    for pos ,value:= range byteIndex{
        fmt.Printf("pos:%d value:%d subtract %d\n",pos,value,posMap[value[0]])
        value[1]-=posMap[value[0]]
        value[0]-=posMap[value[0]]
    }
    fmt.Println(byteIndex)//[[0 1] [3 4] [5 6] [9 10]]

}

* Update 2 *

    lastPos:=-1
    for pos, char := range s {
        offset +=pos-lastPos-1
        fmt.Printf("character %c starts at byte position %d, has an offset of %d, and a char position of %d.\n", char, pos,offset,pos-offset)
        posMap[pos]=offset
        lastPos=pos
    }

Should I be thinking of it as byte-index vs utf-8-char(Rune)-index? — Nick Siderakis, Apr 12 '12 at 23:57
Looks like I need to use a RuneReader, and use FindReaderIndex. — Nick Siderakis, Apr 13 '12 at 01:00
Related reading: http://stackoverflow.com/questions/1752414/how-to-reverse-a-string-in-go — Nick Siderakis, Apr 13 '12 at 02:54

score 4 · Accepted Answer · answered Apr 13 '12 at 01:14

4

As you've probably gathered, Go and Java treat strings differently. In Java, a string is a series of codepoints (characters); in Go, a string is a series of bytes. Text manipulation functions in Go understand UTF-8 codepoints when necessary, but since the string is represented as bytes, the indices they return and work with are byte indexes, not character indexes.

As you observe in the comments, you can use a RuneReader and FindReaderIndex to get indexes in characters rather than bytes. strings.Reader provides an implementation of RuneReader, so you can use strings.NewReader to wrap a string in a RuneReader.

Another option is to take the substring you want the length of in characters and pass it to utf8.RuneLen, which returns the number of characters in the UTF-8 string. Using a RuneReader is probably more efficient, however.

answered Apr 13 '12 at 01:14

Nick Johnson

100,655
16
128
198

fmt.Println(regexp.MustCompile(`a`).FindReaderIndex(strings.NewReader("ウィキa"))[1])//outputs 10; what am i doing wrong? – Nick Siderakis Apr 13 '12 at 02:51
The return value of FindReaderIndex is unclear (char-index or byte-index) to me based on the doc. Specifically, I'm not sure what 's' is referring to. "The match itself is at s[loc[0]:loc[1]]." http://golang.org/pkg/regexp/#Regexp.FindReaderIndex – Nick Siderakis Apr 13 '12 at 03:13
@NickSiderakis You're right, it would appear that, ambiguous as it is, it still returns byte indices. In which case, your best option is to use `utf8.RuneLen` to count how many characters occur before the match. – Nick Johnson Apr 13 '12 at 04:30

Shared GAE datastore, Go <-> Java, regexp.FindStringIndex index shifting (byte-index vs utf-8-char-index)

1 Answers1