Goal:
Be able to use AI for ui elements selection in combination with selenium
Description and results:
The classical elements selection is so brittle and needed an alternative to that, therefore one can use the work that has been accomplished here already: https://github.com/testdotai/appium-classifier-plugin
Work has been completed for mobile using appium. Refer to this stackoverflow link How to properly install test-ai-classifier plugin using appium for ui selectors?
The problem that has to be solved now is to make sure or possible that one can use it with selenium for web client automation.
What has been tried?:
Please check the following links(only appium): https://github.com/ericnana/appium_classifier_ai_python How to properly install test-ai-classifier plugin using appium for ui selectors?
I have tried here to solve it in python.
For java I did not try. For those who are interested please check this: https://appiumpro.com/editions/101-ai-for-appium--and-selenium
Environment:
Ubuntu 20.04,
python=2.7.18/3.8.5,
node = 12.14.0,
npm=6.13.4,
test-ai-classifier
selenium
pip
pytest
I have installed test-ai-classifier globally(npm install -g test-ai-classifier)
Erorr pointing to object detection problem
npm install -g test-ai-classifier
npm WARN deprecated request-promise@4.2.6: request-promise has been deprecated because it extends the now deprecated request package, see https://github.com/request/request/issues/3142
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated har-validator@5.1.5: this library is no longer supported
/home/ecnw/.npm-global/bin/test-ai-classifier -> /home/ecnw/.npm-global/lib/node_modules/test-ai-classifier/build-js/lib/rpc.js
> @tensorflow/tfjs-node@1.7.4 install /home/ecnw/.npm-global/lib/node_modules/test-ai-classifier/node_modules/@tensorflow/tfjs-node
> node scripts/install.js
CPU-linux-1.7.4.tar.gz
* Downloading libtensorflow
[==============================] 1728105/bps 100% 0.0s
* Building TensorFlow Node.js bindings
> canvas@2.6.1 install /home/ecnw/.npm-global/lib/node_modules/test-ai-classifier/node_modules/canvas
> node-pre-gyp install --fallback-to-build
node-pre-gyp WARN Using request for node-pre-gyp https download
[canvas] Success: "/home/ecnw/.npm-global/lib/node_modules/test-ai-classifier/node_modules/canvas/build/Release/canvas.node" is installed via remote
> grpc@1.24.4 install /home/ecnw/.npm-global/lib/node_modules/test-ai-classifier/node_modules/grpc
> node-pre-gyp install --fallback-to-build --library=static_library
node-pre-gyp WARN Using request for node-pre-gyp https download
[grpc] Success: "/home/ecnw/.npm-global/lib/node_modules/test-ai-classifier/node_modules/grpc/src/node/extension_binary/node-v72-linux-x64-glibc/grpc_node.node" is installed via remote
> test-ai-classifier@4.0.2 install /home/ecnw/.npm-global/lib/node_modules/test-ai-classifier
> node-gyp rebuild
make: Entering directory '/home/ecnw/.npm-global/lib/node_modules/test-ai-classifier/build'
CXX(target) Release/obj.target/test-ai-classifier/cc/main.o
CXX(target) Release/obj.target/test-ai-classifier/cc/detection.o
SOLINK_MODULE(target) Release/obj.target/test-ai-classifier.node
COPY Release/test-ai-classifier.node
make: Leaving directory '/home/ecnw/.npm-global/lib/node_modules/test-ai-classifier/build'
> protobufjs@6.10.2 postinstall /home/ecnw/.npm-global/lib/node_modules/test-ai-classifier/node_modules/protobufjs
> node scripts/postinstall
> test-ai-classifier@4.0.2 postinstall /home/ecnw/.npm-global/lib/node_modules/test-ai-classifier
> node ./postinstall.js
Will download object detection model from remote host
Checking whether /home/ecnw/.npm-global/lib/node_modules/test-ai-classifier/model/obj_detection_model exists...
Downloading model from https://data.test.ai/appium-plugin/object-detection-model.pb...
RequestError: Error: certificate has expired
at new RequestError (/home/ecnw/.npm-global/lib/node_modules/test-ai-classifier/node_modules/request-promise-core/lib/errors.js:14:15)
at Request.plumbing.callback (/home/ecnw/.npm-global/lib/node_modules/test-ai-classifier/node_modules/request-promise-core/lib/plumbing.js:87:29)
at Request.RP$callback [as _callback] (/home/ecnw/.npm-global/lib/node_modules/test-ai-classifier/node_modules/request-promise-core/lib/plumbing.js:46:31)
at self.callback (/home/ecnw/.npm-global/lib/node_modules/test-ai-classifier/node_modules/request/request.js:185:22)
at Request.emit (events.js:210:5)
at Request.onRequestError (/home/ecnw/.npm-global/lib/node_modules/test-ai-classifier/node_modules/request/request.js:877:8)
at ClientRequest.emit (events.js:210:5)
at TLSSocket.socketErrorListener (_http_client.js:406:9)
at TLSSocket.emit (events.js:210:5)
at emitErrorNT (internal/streams/destroy.js:92:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)
at processTicksAndRejections (internal/process/task_queues.js:80:21)
WARN: Download of object detection model failed. Object detection mode will not work.
+ test-ai-classifier@4.0.2
My code:
import unittest
#import pytest
from os import path
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from testai_classifier import ClassifierClient
HOST = 'localhost'
PORT = 50051
FX_DIR = path.join(path.dirname(path.realpath(__file__)), 'fixtures')
SEL_CE = 'http://localhost:4444/wd/hub'
import time
class TestWebAI_ClassifierDemoTest(unittest.TestCase):
def setUp(self):
self.driver = webdriver.Remote(command_executor=SEL_CE,
desired_capabilities=DesiredCapabilities.CHROME)
#self.driver.implicitly_wait(5)
def tearDown(self):
self.driver.quit()
def test_ai(self):
self.driver.implicitly_wait(20)
self.classifier = ClassifierClient(HOST, PORT)
#time.sleep(20)
self.driver.get("https://test.ai")
els = self.classifier.find_elements_matching_label(self.driver, "twitter")
els[0].click()
assert self.classifier.current_url == "https://twitter.com/testdotai"
time.sleep(10)
if __name__== '__main__':
suite = unittest.TestLoader().loadTestsFromTestCase(TestWebAI_ClassifierDemoTest)
unittest.TextTestRunner(verbosity=2).run(suite)
Error code:
Ran python3 webaiclassifier.py from the location where my script is located
Traceback (most recent call last):
File "webaiclassifier.py", line 7, in <module>
from testai_classifier import ClassifierClient
File "/home/ecnw/.local/share/virtualenvs/web_test_classifier_ai-r9wXOiFG/lib/python3.8/site-packages/testai_classifier/__init__.py", line 3, in <module>
from client import ClassifierClient
File "/home/ecnw/.local/share/virtualenvs/web_test_classifier_ai-r9wXOiFG/lib/python3.8/site-packages/testai_classifier/client.py", line 2, in <module>
from classifier_pb2 import ElementClassificationRequest
File "/home/ecnw/.local/share/virtualenvs/web_test_classifier_ai-r9wXOiFG/lib/python3.8/site-packages/testai_classifier/classifier_pb2.py", line 7, in <module>
from google.protobuf import descriptor as _descriptor
ModuleNotFoundError: No module named 'google'
What is expected?:
Being able to use test-ai-classifier plugin in combination with selenium for web client automation. Clear and well explained steps used to solve this problem.
