Java - How To Take Text From Web Browser?

Question

I am wondering if anyone knows of a good technique to get all text on a current webpage from a Java application.

I have tried two methods:

1. OCR: this wasn't accurate enough for me to use as the text was roughly only 60% correct. Also it only got the text that the screen shot could see, I need all text on the page

2. Robot class: the method I have got now is using the robot class to us the Control-A, Control-C method and then taking the text from the clipboard. In terms of getting the text this method has proved useful. The only problem I have with it is the user sees the highlighted text for a split second, something I don't want them to see.

This might sound to some as some form of spyware, though this is a final year university project and its an anti cyber bulling/child grooming program, and will only store info when it detects foul play.

Can anyone think of a better way to get the text off the browser?

Many thanks

Answer 1

Get the URL and read the page with an HTTP client class. i.e. Apache Commons HTTPGet.

For more information, read here: http://hc.apache.org/httpclient-3.x/tutorial.html

Answer 2

You can simply get all HTML from the website using URLConnection or Apache's HTTPClient Here's the question explaining how to do that: Get html file Java

Ofcourse it will not give You text in binaries (ie flash files) images etc. For those only OCR will work.

Answer 3

You can try something like this

GetMethod get = new GetMethod("http://ThePage.com");
InputStream in = get.getResponseBodyAsStream();
String htmlText = readString(in);

static String readString(InputStream is) throws IOException {
char[] buf = new char[2048];
Reader r = new InputStreamReader(is, "UTF-8");
StringBuilder s = new StringBuilder();
while (true) {
   int n = r.read(buf);
    if (n < 0)
      break;
    s.append(buf, 0, n);
  }
  return s.toString();
}

Answer 4

the most generic solution would be traffic sniffer.

Answer 5

This is the utility class I created for this purpose. It has runtime and non-runtime versions, and also provides for verifying the tail end of the retrieved source.

   import  java.io.BufferedInputStream;
   import  java.io.IOException;
   import  java.io.InputStream;
   import  java.net.MalformedURLException;
   import  java.io.EOFException;
   import  java.net.URL;

/**
   <P>Append the source-code from a web-page into a <CODE>java.lang.Appendable</CODE>.</P>

   <P>Demo: {@code java AppendWebPageSource}</P>
 **/
public class AppendWebPageSource  {
   public static final void main(String[] igno_red)  {
      String sHtml = AppendWebPageSource.get("http://usatoday.com", null);
      System.out.println(sHtml);   

      //Alternative:
      AppendWebPageSource.append(System.out, "http://usatoday.com", null);
   }
   /**
      <P>Get the source-code from a web page, with runtime-errors only.</P>

      @return  {@link #append(Appendable, String, String) append}{@code ((new StringBuilder()), s_httpUrl, s_endingString)}
    **/
   public static final String get(String s_httpUrl, String s_endingString)  {
      return  append((new StringBuilder()), s_httpUrl, s_endingString).toString();
   }
   /**
      <P>Append the source-code from a web page, with runtime-errors only.</P>

      @return  {@link #appendX(Appendable, String, String) appendX}{@code (ap_bl, s_httpUrl, s_endingString)}
      @exception  RuntimeException  Whose {@link getCause()} contains the original {@link java.io.IOException} or {@code java.net.MalformedURLException}.
    **/
   public static final Appendable append(Appendable ap_bl, String s_httpUrl, String s_endingString)  {
      try  {
         return  appendX(ap_bl, s_httpUrl, s_endingString);
      }  catch(IOException iox)  {
         throw  new RuntimeException(iox);
      }
   }
   /**
      <P>Append the source-code from a web-page into a <CODE>java.lang.Appendable</CODE>.</P>

      <P><I>I got this from {@code <A HREF="http://www.davidreilly.com/java/java_network_programming/">http://www.davidreilly.com/java/java_network_programming/</A>}, item 2.3.</I></P>

      @param  ap_bl  May not be {@code null}.
      @param  s_httpUrl  May not be {@code null}, and must be a valid url.
      @param  s_endingString  If non-{@code null}, the web-page's source-code must end with this. May not be empty.
      @see  #get(Appendable, String, String)
      @see  #append(Appendable, String, String)
    **/
   public static final Appendable appendX(Appendable ap_bl, String s_httpUrl, String s_endingString)  throws MalformedURLException, IOException  {
      if(s_httpUrl == null  ||  s_httpUrl.length() == 0)  {
         throw  new IllegalArgumentException("s_httpUrl (\"" + s_httpUrl + "\") is null or empty.");
      }
      if(s_endingString != null  &&  s_endingString.length() == 0)  {
         throw  new IllegalArgumentException("s_endingString is non-null and empty.");
      }

      // Create an URL instance
      URL url = new URL(s_httpUrl);

      // Get an input stream for reading
      InputStream is = url.openStream();

      // Create a buffered input stream for efficency
      BufferedInputStream bis = new BufferedInputStream(is);

      int ixEndStr = 0;

      // Repeat until end of file
      while(true)  {
         int iChar = bis.read();

         // Check for EOF
         if (iChar == -1)  {
            break;
         }

         char c = (char)iChar;

         try  {
            ap_bl.append(c);
         }  catch(NullPointerException npx)  {
            throw  new NullPointerException("ap_bl");
         }

         if(s_endingString != null)  {
            //There is an ending string;
            char[] ac = s_endingString.toCharArray();

            if(c == ac[ixEndStr])  {
               //The character just retrieved is equal to the
               //next character in the ending string.

               if(ixEndStr == (ac.length - 1))  {
                  //The entire string has been found. Done.
                  return ap_bl;
               }

               ixEndStr++;
            }  else  {
               ixEndStr = 0;
            }
         }
      }

      if(s_endingString != null)  {
         //Should have exited at the "return" above.
         throw  new EOFException("s_endingString " + (new String(s_endingString)) + " (is non-null, and was not found at the end of the web-page's source-code.");
      }
      return  ap_bl;
   }
}