Bot.find_all shuffle the elements after convert py to .exe

Frameworks BotCity Framework - Python
1

Hello everyone, botcity started showing a strange problem after compiling with auto-py-to-exe. I always used the same machine and always compiled it that way. However, since yesterday, when I use the self.find_all function and create a list, when running Pycharm it finds the items in the stipulated order, but after compiling in .exe the order of the objects simply changes. I’ve been using it for 2 years and have never had any problems. Code follows:

    while self.find(label="arena_150", matching=0.7, waiting_time=3000, grayscale=True):
        print('LH energy found')
        elements = self.find_all(label="arena_150", matching=0.7, waiting_time=3000, grayscale=True)
        elements_list = list(elements)
        number_of_elements = len(elements_list)
        #if number_of_elements > 5:
        #    number_of_elements = 5
        print('Energy available: ' + str(number_of_elements))
        for _ in range(int(number_of_elements)):
                #x = elements_list[number_of_elements - plays].left
                #y = elements_list[number_of_elements - plays].top
                x = elements_list[-1].left
                y = elements_list[-1].top
                a = elements_list[-2].left
                b = elements_list[-2].top
                self.mouse_move(x, y)
                pyautogui.click()
                print('Battle X')
                plays +=1
                print(number_of_elements - plays)
                time.sleep(3)
                self.mouse_move(a, b)
                pyautogui.click()
                print('Battle X')
                time.sleep(3)

Objects [-1] and [-2] are correctly identified in pycharm, but after compiling in .exe the order changes completely.

I know that computer vision looks for objects from left to right and top to bottom, but this apparently changes after compiling

expected order of objects:

[Box(left=1413, top=654, width=66, height=38)
Box(left=981, top=753, width=66, height=38)
Box(left=1154, top=753, width=66, height=38)
Box(left=1326, top=753, width=66, height=38)
Box(left=1499, top=753, width=66, height=38)]

after compiled:

[Box(left=1154, top=753, width=66, height=38)
Box(left=1499, top=753, width=66, height=38)
Box(left=981, top=753, width=66, height=38)
Box(left=1413, top=654, width=66, height=38)
Box(left=1326, top=753, width=66, height=38)]

2

Can someone help me?

3

If it can help, I edited botcity/core/cv2 find.py and added the lines:

ordered = sorted(zip(matchx, matchy), key=lambda p: result[p[1]][p[0]], reverse=True)
for x, y in ordered:
print(f"Box: x={x}, y={y}, w={needle_width}, h={needle_height}, score={result[y]}")
yield Box(x, y, needle_width, needle_height)

the problem seems to be that botcity recognizes some duplicate objects, and thus messes up the order

The sort order seems to be determined by the template-match score, and not in the order in which the objects are on the screen

Box: x=1536, y=739, w=52, h=29, score=0.9999576210975647
Box: x=1191, y=739, w=52, h=29, score=0.967707633972168
Box: x=1709, y=739, w=52, h=29, score=0.9400887489318848
Box: x=1364, y=739, w=52, h=29, score=0.8545419573783875
Box: x=1363, y=739, w=52, h=29, score=0.8460010886192322
Box: x=1622, y=640, w=52, h=29, score=0.82472825050354

objects 4 and 5 are the same

objects are shown in order of best score
4

@dimitriusss take a look at the find_all where we deduplicate the entries (line 361 at bot.py).

We could add an optional flag to the cv2 locate_all method to switch the ordering based on the flag with the default being to order by best matching to keep it backwards compatible.

I believe your sorting solution alongside the deduplication should fix it.

A Pull Request adding this change would be very welcome. Otherwise our team can look into it soon.

Could you please open an issue for us to keep track of this?

Thank you

1 Like
5

actually the problem occurred after botcity and opencv updates.

The correct thing to do is to list the objects in the order they are found on the screen. What is no longer happening. Is being ordered by matchscore.
Therefore, duplicates end up being a big problem.

Base: v043
Core: v114
Opencv: 49080

Could you please open an issue for us to keep track of this?

Where do you want me to open the ticket? Github?

Tks

6

The ordering was changed 2 years ago in version 0.4.0 and is the correct approach for the most common use case which is to find the best image matching the template.

Please open the issue here: GitHub - botcity-dev/botcity-framework-core-python: BotCity Framework - Python

We will fix the find_all to take care of this issue and order by the order in which images appear without duplicates.

Could you please attach to the issue sample needle and haystack images?

That will speed up a lot the development and testing by our team.

1 Like
7

How can I continue listing objects in the order they appear on the screen (left-right-top-bottom)? Just downgrading? Which version do you suggest?

It would be great if I could choose (on find.all) between listing objects by their score or by the order they appear on the screen. In fact, in my application, it is only practical if it is listed in the order it appears

8

Thank you for opening the issue. We will get back to you as soon as the new version is available.

If you need an immediate solution, we recommend downgrading or implementing the sort/deduplication in a separate function for now.

1 Like