0

I'm trying to take a long list of objects (in this case, applications from the iTunes App Store) and classify them more specifically. For instance, there are a bunch of applications currently classified as "Education," but I'd like to label them as Biology, English, Math, etc.

Is this an AI/Machine Learning problem? I have no background in that area whatsoever but would like some resources or ideas on where to start for this sort of thing.

bmay2
  • 382
  • 2
  • 4
  • 17
  • try to search for fuzzy logic, neural network – mrok Jul 29 '12 at 19:40
  • 1
    http://en.wikipedia.org/wiki/Statistical_classification#Algorithms – Karoly Horvath Jul 29 '12 at 19:45
  • Google has a Prediction API that you might be interested in: https://developers.google.com/prediction/ – Casey Chu Jul 29 '12 at 19:47
  • http://stackoverflow.com/questions/2079471/web-mining-classification-algorithms – Karoly Horvath Jul 29 '12 at 19:48
  • You might want to check out the book _Programming Collective Intelligence_ .It's examples are in python and it's all about machine learning and the type of things you're asking about. Additionally, it's fairly approachable for someone with a programming background. – Paul Rubel Jul 31 '12 at 21:47

2 Answers2

3

Yes, you are correct. Classification is a machine learning problem, and classifying stuff based on text data involves natural language processing.

The canonical classification problem is spam detection using a Naive Bayes classifier, which is very simple. The idea is as follows:

  1. Gather a bunch of data (emails), and label them by class (spam, or not spam)
  2. For each email, remove stopwords, and get a list of the unique words in that email
  3. Now, for each word, calculate the probability it appears in a spam email, vs a non-spam email (ie count occurrences in spam, vs non spam)
  4. Now you have a model- the probability of a email being spam, given it contains a word. However, an email contains many words. In Naive Bayes, you assume the words occur independently of each other (which turns out to to be an ok assumption), and multiply the probabilities of all words in the email against each other.
  5. You usually divide data into training and testing, so you'll have a set of emails you train your model on, and then a set of labeled stuff you test against where you calculate precision and recall.

I'd highly recommend playing around with NLTK, a python machine learning and nlp library. It's very user friendly and has good docs and tutorials, and is a good way to get acquainted with the field.

EDIT: Here's an explanation of how to build a simple NB classifier with code.

nflacco
  • 4,972
  • 8
  • 45
  • 78
2

Probably not. You'd need to do a fair bit of work to extract data in some usable form (such as names), and at the end of the day, there are probably few enough categories that it would simply be easier to manually identify a list of keywords for each category and set a parser loose on titles/descriptions.

For example, you could look through half a dozen biology apps, and realize that in the names/descriptions/whatever you have access to, the words "cell," "life," and "grow" appear fairly often - not as a result of some machine learning, but as a result of your own human intuition. So build a parser to classify everything with those words as biology apps, and do similar things for other categories.

Unless you're trying to classify the entire iTunes app store, that should be sufficient, and it would be a relatively small task for you to manually check any apps with multiple classifications or no classifications. The labor involved with using a simple parser + checking anomalies manually is probably far less than the labor involved with building a more complex parser to aid machine learning, setting up machine learning, and then checking everything again, because machine learning is not 100% accurate.

algorowara
  • 1,700
  • 1
  • 15
  • 15