3

I have this GUI on our project (please bear with my illustration, I'm not allowed to take a screenshot at work)

----------------------------------------------
    (1)     Header                            |
----------------------------------------------|
             (2)      Files             | (3) | Drop-down button (Options)
----------------------------------------|-----|          
                                        |     |
----------------------------------------|-----|
                                        |     |
----------------------------------------|-----|
                                        |     |
-----------------------------------------------

I need to test one file in each file type our product can support. The ultimate goal is to use that (3) drop down button. Here is the summary of my code when I'm in this part

find_filetype = find.imagedict(filetypeicon.png) #(2) filetype

freg = find_filetype.right()

if freg.exists(drop-down.png):                   #(3) drop-down icon
    freg.click(drop-down.png)

Now after clicking (3), a drop-down menu with 4 action items will appear. So for example if I click (3a): it will appear below from (3b - 3d) showing at least 2-3 drop-down items.

----------------------------------------------
            Header                            |           
----------------------------------------------|
                                        | (3a)|
----------------------------------------------|
                                |  (a1)   |   |   #a1 = action 1
--------------------------------|---------|---|   
                                |  (a2)   |   |   #a2 = action 2
--------------------------------|---------|---|
                                |__(a3)___|   |
-----------------------------------------------

If I click (3c): The drop-down menu will choose to appear above (f.a.d.) Showing at least 2 drop-down items.

----------------------------------------------
            Header                            |           
----------------------------------------------|
                                |  (a1)   |   |
--------------------------------|---------|---|         
                                |  (a2)   |   |
--------------------------------|_________|---|
                                        | (3c)|           
----------------------------------------|-----|
                                        |     |
----------------------------------------------

My workaround for this are :

  1. screenshot all action items and

  2. if/elifs + dragDrop()

which makes my scripts slow (I perform hundred of iterations everyday)

Is there a way I can find (a1) (because its always present after clicking (3)), save its coordinates and use it as a param for scrolldown_codes like this

   #after clicking drop-down
   #find and get (action1) coordinates
   def scroll_down(coordinates):              #pass it here
       while not exist(action(n).png):        #n = action choice 
          end_Y = coordinates
          start_Y = coodinates - few_Y
          dragdrop(start_Y, end_Y)
          wait(1)
       click(action(n).png)

Please help me! Feel free to suggest anything helpful!

Note: Im only allowed to click and drag while on that GUI because its a touchscreen product we're developing.

vpibano
  • 2,415
  • 1
  • 13
  • 26
  • What is your ultimate goal here? Do you need to see these menu items? Interact with them? Ensure they exist? – Eugene S Dec 19 '16 at 15:04
  • BTW, good job on describing the issue here with no use of screenshots! – Eugene S Dec 19 '16 at 15:05
  • Thank you! Id like to interact with the drop-down menu thru scrolling! There are 4 menu items after clicking that (4) button and (because we have a 4.3in ui panel) most of the time, only 2-3 menu items appear. I want to pick the 4th menu item but Im having trouble on how to scroll down thru hard coded region because it either pops out above/below (4) button. – vpibano Dec 20 '16 at 00:53
  • @EugeneS I found my comment confusing too. (a1) is always present after clicking (4) so I want to save (a1)'s coordinates and use it as a reference to point the cursor few pixels below (a1). So the code would look like what Ive editted above. *scroll_down() – vpibano Dec 20 '16 at 03:20
  • Your description is fine. Please have a look at my answer and let me know if that helps. – Eugene S Dec 20 '16 at 04:06
  • @EugeneS Thank you! From your answer and reference, Ive learned to set a region splitting the screen beforehand. That really helped me do a better script! On the other hand, I forgot to mention that the actual selection menu doesnt have any down arrow button in its design; the ONLY way to pick the 4th option is thru scrolling and then clicking the image. – vpibano Dec 20 '16 at 07:58
  • What I meant by the down button is the actual keyboard button that you can emulate using Sikuli built-in Java Robot framework. This is to avoid complex mouse functions. So once you click on the menu, an item usually gets highlighted. Then you can use your keyboard to move between the items. Does that make more sense? – Eugene S Dec 20 '16 at 08:02
  • Yes it does, but we are required to use only clicks for our test since its a touchscreen product. – vpibano Dec 20 '16 at 08:07
  • @EugeneS BTW, I'm still working on the code. I found many useful functions from your reference and blog! I will get back to you with results! – vpibano Dec 20 '16 at 08:09

1 Answers1

2

Actual scrolling, as you would do with the scrollbar, is probably not the best solution as it requires many additional actions that all prone to failure due to various reasons. What I usually suggest in such cases is using keyboard to control the selection. There are few options really and that depends on how your application behaves. So I will list few tricks that might be used in your scenario and you decide what suits you best.

First of all, keep in mind that you can speed up the search time by limiting the area you are looking in. There are many ways to do that. For example:

  1. Split the screen if you know that the patters you are looking for are expected to appear only in certain area.

  2. Relative search / Region extension. You can find the relevant (3) item and then expand the region to its right to capture the area where the (4) selector (3 dots) is located. This will speed up the search significantly as it will only look in a small area rather than the whole screen and it will ensure you are clicking the correct (4) item. More info and examples about region expansion here. More info on optimizing search time here.

Now to the actual item selection in a menu. So, as I mentioned, I reckon you better off using your keyboard to select the item. That means selecting/clicking on the first item in the list and then pressing the down arrow button x times until you reach the menu item you are after. There are few ways to hit the down arrow button correct number of times:

  1. Hard code it. If you know that the specific item is fourth in list just click 4 times.

  2. Press arrow down button until you reach the desired item. That might be a bit slower and might require taking additional screenshots but might prove useful if you do not know what order the items are in or if that is prone to change.

Eugene S
  • 6,709
  • 8
  • 57
  • 91
  • 1
    Thank you so much for the reference! I found useful tools. I used find(img) and then getTarget(); Declared those Location as a param to perform a dragDrop() while using offset() to drop elsewhere. – vpibano Dec 21 '16 at 06:16
  • @PetIbaño No worries at all. Glad that helped. There are many ways to achieve good results with Sikuli, feel free to post more questions here (or leave comments in the blog) if you stuck or looking for more info/examples. – Eugene S Dec 21 '16 at 06:36