0

To be more precise, can my application take data from user and show a loading box to them and then open a website (non-interactable) and enter that data on behalf of the user on the website, also extract captcha and ask user to fill it in their UI and once that is filled by the user in the application's UI submit the captcha value to the website. An android application like a medium of interaction between the user and the website?

I am a beginner level android developer and want to achieve this kind of functionality, any help would be great!

Coder Marine
  • 73
  • 10
  • yes, you can. just need to search for ```Web scraping``` & ```web crawling``` . – javadroid Jan 13 '20 at 05:24
  • Thanks @javaddroid I did and found about scrapy framework and all but sir, please forgive my ignorance I am unable to just make a connection between these frameworks and java for Android. Any code or help, even a little push of concept will be of great help. Regards – Coder Marine Jan 13 '20 at 05:35

1 Answers1

1

You need to create your own web service for accessing from mobile to that data on your server.

on your own server create a wrapper to access the primary data, when need to authorize or entering captcha open that page as a iframe on your server or direct link on a webview in your app to pass it.

also take look at jsoup library for accessing web element and parse that on mobile app.

hope to help you. :)

javadroid
  • 1,421
  • 2
  • 19
  • 43
  • Thanks for the answer and mentioning about jsoup let me try and I will update accordingly. Also the webserver is not mine, if it would have been the case I would have exposed an API end point for the same but the website I am referring to is a remote website which doesn't provide any API. How to proceed in that case. Please see that it also uses captcha religiously !! I need to acquire that captcha image in my UI and once user fills it in my UI I shall proceed submitting the captcha. That was my motive. Best regards – Coder Marine Jan 13 '20 at 06:06
  • you can write a wrapper to main server on your own server. for parsing captcha see this link: https://github.com/ismailkocacan/Android-Captcha-Handler and this: https://stackoverflow.com/questions/28619161/getting-captcha-image-using-jsoup if answer was helpful you can use upvote button or mark as solved question. – javadroid Jan 13 '20 at 06:43
  • Thanks for guiding me through, this is exactly what I was looking for, and sorry for being late in reply, now I am stuck only in one problem please guide me through. The website I am trying to scrap, goes like this, once I fill up the form, it presents me with a captcha that is also being captured by me through my users on the android application, if the data is proper the website coughs out a "zip" file that can be downloaded. How can I catch that and store it for local processing on android storage, accessing and manipulation I know. Regards – Coder Marine Jan 25 '20 at 09:02
  • your welcome. for downloading file look at : https://stackoverflow.com/questions/1714761/download-a-file-programmatically-on-android and for unzipping file see this: https://stackoverflow.com/questions/16142311/unzipping-files-programmatically-in-android – javadroid Jan 25 '20 at 11:16
  • Thank you very much sir, please if you could provide me a little code or please edit the answer to elaborate, it would be very very helpful. Actually the resource that is prompted to download after form submission, it is not in a unique URL, it outputs the different version of the "zip" file for each input a user gives to it. That is if User A fills out the form, A's version of the zip is provided and if User B fills out the form B's version of the zip is provided. The zip however is everytime located at, "server.com/download/compiledReport", In scraping, sessions or cookies ? Regards – Coder Marine Jan 25 '20 at 11:22
  • your welcome. this is a huge topic, cant explain that here, there are many useful tutorial for that, the download link not matter, you need to write a download manager or use download manager library. if you strange with this topics just go to youtube and search for tutorials. there are many useful video. have good coding :) – javadroid Jan 25 '20 at 17:05
  • I agree sir, It would be but I would appreciate if you could guide me better, as you know for a noob searching for something like this could take so much time exploring rather than coding. Regards – Coder Marine Jan 27 '20 at 17:55