1

I have a text file with 8000+ lines of information. I want to parse it to JSON for using it in iOS application. Because parsing that kind of txt file takes about 35 seconds, which is not good for user experience.

Text file goes like this:

TURKIYE;ADANA;ADANA;36,9914194;35,3308285;03:00:00
TURKIYE;ADANA;ALADAĞ;37,545348;35,394608;03:00:00
TURKIYE;ADANA;CEYHAN;37,028765;35,8124309;03:00:00
TURKIYE;ADANA;FEKE;37,814467;35,910391;03:00:00
TURKIYE;ADANA;İMAMOĞLU;37,2984443;35,6095474;03:00:00
TURKIYE;ADANA;KARAISALI;37,2758825;35,1268781;03:00:00
TURKIYE;ADANA;KARATAŞ;36,6649776;35,2587964;03:00:00

I just want the "TURKIYE", "ADANA", "ALADAĞ", "Latitude" and "Longitude" parts and don't need the last "03:00:00" part.

Edit: Forget the mention about something. Text files some lines doesn't includes third column. For example;

AVUSTURYA;HORBRANZ;47,5557073;9,7525947;01:00:00
AVUSTURYA;HORN;48,66607;15,65716;01:00:00
AVUSTURYA;IMST;47,24013;10,73954;01:00:00
AVUSTURYA;IMSTERAU;47,21018;10,70901;01:00:00

I want to parse them like "if third column doesn't exist print nil value there."

My txt parsing code is:

let textFilePath = Bundle.main.path(forResource: "LatLongData", ofType: "txt")
let fileManager = FileManager.default

if fileManager.fileExists(atPath: textFilePath!) {

    do {
        let fullText = try String(contentsOfFile: textFilePath!, encoding: String.Encoding.utf8)
        let lines = fullText.components(separatedBy: "\n") as [String]
            for line in lines {

                let data = line.components(separatedBy: ";")
                let locationData = LocationData()

                if data.first == "TURKIYE" || data.first == "ABD" {
                    locationData.country = data[0]
                    locationData.city = data[1]
                    locationData.district = data[2]
                    locationData.latitude = data[3]
                    locationData.longitude = data[4]
                    locationData.compoundKey = "\(data[0])-\(data[1])-\(data[3])"
                 } else {
                    locationData.country = data[0]
                    locationData.city = data[1]
                    locationData.latitude = data[2]
                    locationData.longitude = data[3]
                    locationData.compoundKey = "\(data[0])-\(data[1])-\(data[3])"
                }
                locationData.writeToRealmDB()
            }
    } catch let error as NSError {
        print(error.localizedDescription)
    }

How can I convert this text to JSON for using in iOS app?

Thanks in advance.

onurgenes
  • 127
  • 4
  • 18
  • I can't see how parsing a JSON file is going to be any faster than parsing the original text file. JSON will increase the size of the file. Perhaps it is the method by which you are trying to parse the text file; show your code. – Paulw11 Nov 29 '17 at 07:21
  • @Paulw11 added the code snippet and forget to mention about a special case. Added that as well. – onurgenes Nov 29 '17 at 07:32
  • How long does it take if you comment out the `locationData.writeToRealmDB()`? Have you used the time profiler instrument to see where your time is being spent? You could also look at [this question](https://stackoverflow.com/questions/25846459/simple-way-to-read-local-file-using-swift) for an answer that parses as the file is read – Paulw11 Nov 29 '17 at 07:51
  • @Paulw11 you were right, writing to Realm Database takes too much time. – onurgenes Nov 29 '17 at 08:06
  • So you probably need to see if you can push that to a background thread or something to improve the user experience; perhaps you can dispatch this whole block of code to a background thread. I haven't used Realm. Is there some way you can build an array and write the data all at once? Did you write the `writeToRealmDB` function? If so, take a look at that and see what it is doing and if it could be more efficient; perhaps it is opening a database connection each time, when you could keep a connection open – Paulw11 Nov 29 '17 at 08:06
  • Thanks for helping out. Will use background thread as well. @Paulw11 – onurgenes Nov 29 '17 at 08:12

1 Answers1

1
import pandas as pd

with open('filename.txt', 'r') as f:
lines = f.readlines()
for line in lines:
    if line.count(';') == 4:
        index = line.index(';', 2)
        line = line[:index] + ';' + line[index:]
    with open('filename2.txt', 'a') as f2:
        f2.write(line)

df = pd.read_csv('filename2.txt', header=None, sep=';')
df = df.iloc[:, :-1]
df.columns = ['col1', 'col2', 'col3', 'Latitude', 'Longitude']
df.to_json('filename.json')

http://pandas.pydata.org/pandas-docs/stable/io.html

Akichan
  • 26
  • 3