I need to find an image (detect text in game) using transparency (the background is changing every few seconds). My script:
#include <ImageSearch.au3>
HotKeySet("s", "Start")
$x = 0
$y = 0
Func Start()
$Search = _ImageSearchArea("*Trans0x40FF00 " & "trans_test.png",1,90,90,#DesktopWidth,#DesktopHeight,$x,$y,50)
If $Search = 1 Then
MouseMove($x, $y,10)
EndIf
EndFunc
While 1
Sleep(100)
WEnd
It doesn't work. My image to search has a bright green colour 0x40FF00 all around, representing transparency. How to make my image match?
First of all, if your game runs in DirectX mode or similar (as most fullscreen games do). This won't work at all. (A DirectX hook or something like that would be required and I don't think someone has been going through the troubles of writing the code necessary to do in from AutoIt/AHK.)
Otherwise, in Autohotkey there is the *n option which may do the job for you when adjusted correctly. It's possible this exists in AutoIt too.
(*TransN does exist in AutoHotkey too btw.)
Related
So I am using OpenCV (in Go with OpenCV) to attempt to extract the pieces from a boardgame. Originally I was approaching this problem with somewhat success by manually finding the HSV values for each player piece colour and the board positions. I managed to get this working, and a programmatic representation of every piece and its position on the board. The downside being that this requires quite serious human interaction if using a different board - "finding" all the correct HSV values. I asked here and got a suggestion to start by ignoring the colour, find all the pieces and then using a clustering algorithm on colour to work out which player it is. I might have to do something for the positions as well but thats stage two.
So now I am attempting to just extract all pieces regardless of colour.
I started out trying to use the NewSimpleBlobDetectorWithParams - however made little progress it seems to struggle alot on false negatives/positives.
I tried HoughCirclesWithParams but again this seems very dependant on the parameters and I wasn't making much progress in the actual pieces being detected. Currently I am using FindContours and that seems to be giving me some reasonable accuracy. Lets look at the picures.
The original image looks like this:
I have built a "dashboard" of controls and the thing that seems to be most "useful" is erosion, dilation and threshold.
My current setup is a load of trackerbars/sliders to adjust the values and then
gocv.CvtColor(clone, &clone, gocv.ColorRGBToGray)
erodeKernel := gocv.GetStructuringElement(gocv.MorphRect, image.Pt(trackers.erosionValue, trackers.erosionValue))
gocv.Erode(clone, &clone, erodeKernel)
dilateKernel := gocv.GetStructuringElement(gocv.MorphRect, image.Pt(trackers.dilateValue, trackers.dilateValue))
gocv.Dilate(clone, &clone, dilateKernel)
gocv.Threshold(clone, &clone, float32(trackers.thresTruncValue), 255, gocv.ThresholdTrunc)
gocv.Threshold(clone, &clone, float32(trackers.threshBinaryValue), 255, gocv.ThresholdBinary)
cannies := gocv.NewMat()
gocv.Canny(clone, &cannies, float32(trackers.cannyMin), float32(trackers.cannyMax))
cnts := gocv.FindContours(cannies, gocv.RetrievalTree, gocv.ChainApproxSimple)
followed by
for i := 0; i < cnts.Size(); i++ {
cnt := cnts.At(i)
if len(cnt.ToPoints()) < 5 {
continue
}
rect := gocv.FitEllipse(cnt)
gocv.Circle(&colorImage, image.Pt(rect.Center.X, rect.Center.Y), (rect.Height + rect.Width)/4, cntColor, 3)
if gocv.ContourArea(cnt) < gocv.ArcLength(cnt, false) {
continue
}
gocv.Rectangle(&colorImage, rect.BoundingRect, rectColor, 2)
psVector := gocv.NewPointsVector()
psVector.Append(cnt)
gocv.DrawContours(&clone, psVector, 0, rectColor, 3)
if rect.Center.X == (rect.BoundingRect.Max.X + rect.BoundingRect.Min.X) / 2 && rect.Center.Y == (rect.BoundingRect.Min.Y + rect.BoundingRect.Max.Y) / 2 {
//Does the circle fit inside the square?
if float64(rect.Width * rect.Height) > math.Pi * math.Pow(float64((rect.Height+rect.Width)/4), 2) {
gocv.Circle(&colorImage, image.Pt(rect.Center.X, rect.Center.Y), 2, matchColor, 3)
pieces = append(pieces, image.Pt(rect.Center.X, rect.Center.Y))
}
}
}
The idea being if the contour has 5 points then you can find the bounding bounding rectangle, then if the contour is closed, draw a circle at the center of the contour and if it fits inside the bounding rectangle, and they share the same center, its probably a playing piece. Note - I came up with this principle based on seeing where the circles and bounding rectangles were lying and when they matched up it more often than not seemed to be a playing piece.
So I am making some nice progress. However my questions are help with approaches to dig out the other colour pieces and perhaps more "robustly" dig out the white pieces. I feel that I don't quite have enough tools at my disposal as if i increase one thing i have to decrease another and I for some reason feel finding 30 round chequers on a board should be reasonably robust.
When I adjust the values looking for the maroon pieces I can get a few of them
but as you can see the diference when playing with threshold/erosion/dilation is not doing a wonderful job of finding them.
EDIT:
I have added the hough circle algorithm back in to sort of show that it hits on false negatives alot - in this case it got 1.
gocv.HoughCirclesWithParams(
clone,
&circles,
gocv.HoughGradient,
1, // dp
15, // minDist
75, // param1
20, // param2
20, // minRadius
45, // maxRadius
)
blue := color.RGBA{0, 0, 255, 0}
for i := 0; i < circles.Cols(); i++ {
v := circles.GetVecfAt(0, i)
// if circles are found
if len(v) > 2 {
x := int(v[0])
y := int(v[1])
r := int(v[2])
gocv.Circle(&colorImage, image.Pt(x, y), r, blue, 2)
}
}
Here is the threshold I was using.
So I realise I have said a lot here. I am looking for some help to detect all the playing pieces on the board.
I am doing this in go with gocv, but I can use python/convert python code if anyone has a good reference or something.
The original image without any ammendments is here. As I say my goal is to automatically detect the 30 pieces on the board and then i can use a clustering algo to work out which group they are in (I think...) I want to do it with the least amount of human interaction dragging sliders as that is not a fun/nice user experience....
Thoughts I had
the user could drag bounding boxes around groups and then that would make the computers job easier knowing it had to find pieces in there.
the user could select a colour of the page and that would tell the computer what roughly HSV values it should be looking in
the user could calibrate against a known start position of the pieces so the computer knew where to look.
Not exactly an answer to your questions, but this would be so much easier if you used object detection instead. Same way in my tutorials I find different objects. In this case, I would have 2 or possibly 3 classes: light pieces, dark pieces, and possibly another class for the empty spaces.
I usually use OpenCV and Darknet/YOLO to solve these kinds of things. I have many tutorials on my youtube channel. Here is a simple one to detect a few shapes: https://www.youtube.com/watch?v=yOJIRArZeig Here is another that shows OpenCV and Darknet/YOLO used to solve Sudoku: https://www.youtube.com/watch?v=BUG7HlhuArw
Your case would be similar to that last one. You'd get back a vector of objects detected, with the bounding box coordinates of each one within the image or video frame. If interested, this is the tutorial video I recommend to start: https://www.youtube.com/watch?v=pJ2iyf_E9PM
I'm using Paintcode V3 and last Xcode for my project.
In Paintcode I draw a red path, kind of "clock hand" with a radius variable to be able to rotate it.
In my Xcode project, this custom UIView is added to a main view.
When the red path is rotating, I need to know when it intersects with another element of the view.
I tried:
if self.redPathView.frame.intersects(self.anotherView.frame) {
//do something...
}
but it can't work because redPathView is the entire Paintcode canvas (the square frame) and not only the red path.
I'm searching a way to access to the red path of my view.
Interesting problem.
I don't think there is a way to do exactly what you are looking for...
Like you mentioned, when we use PaintCode, our custom drawings become an entire UIView.
We can't access its specific elements (like your redPathView).
(I'm favoriting this question in case anyone knows if we actually can. I may be wrong.)
Anyway, there is a workaround. You could use PaintCode itself to tell you when "collisions" happen. Of course, then you would need to draw everything there... And implement some sort of internal control that would make sense to you.
For example, in the picture below I created this "clock hand", with functions to detect when the red thing touches other UI elements (key point there is the touchedElementID expression -- observe how it updates itself with the "ID" of the rectangles):
There are some issues you may find with this approach, for example the need to know the exact position of other elements on the screen and possibly complex ternary constructions like:
(rotationSanitized <= -352 || rotationSanitized >= -8) ?
1 :
(rotationSanitized <= -82 && rotationSanitized >= -98) ?
2 :
(rotationSanitized <= -172 && rotationSanitized >= -188) ?
3 :
(rotationSanitized <= -262 && rotationSanitized >= -278) ?
4 : 0
Also, how to "export" and consume this information along with the StyleKit in Xcode?
The trick I did was to add an empty (no Fill, no Stroke) UI element in PaintCode and associate the touchedElementID expression result to the alpha value of the element.
This way I could access the expression in my custom view, for example via StyleKit.touchedElementID. But that introduces a new problem since now I have to change the generated StyleKit every time it is exported (two lines of code, but still). That is kind of a fragile solution.
However, if you are ok with that, then it's just a matter of consuming this new information...
For that I created a protocol, so my ViewController can act as the delegate of my custom view, and perform whatever it needs to do when the most internal UI elements in PaintCode are "touched".
Here is the final result:
Finally, check this project in GitHub with all the code and samples.
I am trying to capture the pixel color of a specific letter in a font within an iOS app. I'd then use that color in an if/then statement to determine the next action of my script. I'm finding no easy way to determine if the color/colors I'm finding are the right ones. Here's my current process: I start recording my inputs and continuously click around the letter. I end up with something like touchdown(123,456). I change that to alert(getColor(123,456)) and run, which produces a popup that tells me the color such as 3094841 (not sure why the colors are in numeric format, but they are). I do this for each touchdown line that is captured. The problem is, I don't know an easy way to determine which color is the small letter I'm trying to tap.
Is there a lua function that will capture and display a range of colors between 2 points? If there were, I could see the commonality of all of the colors within the 2 points and make at least an educated guess as to which is the color in the font. Or even more useful - is there a tool I can use to type in the color I'm getting back and have it display the corresponding color, so I can compare them. That'd be the easiest. Hope this makes sense. Any help would be awesome. In case it matters, I'm using Autotouch 8 on an iPhone 5.
TIA
I use this function often in my games.
I find the easiest way to get a color you want to execute every single time is to take a snap of the screen you're checking and then use the helper on getColor(x, y)
And you can use this to alert your color.
local color = getColor(x, y)
alert(color)
-- You can also use:
log(color)--this one keeps it in your log in case you can write it down immediately.
To use this in an if/then statement
function = namedfunction()
local color = getColor(x, y)
if color == YOURCOLOR then
else
end
end
namedfunction();
Note that I have the iPhone 5 iOS 8.3 and I have the most recent AutoTouch from Cydia. If you don't have the helper when you're scripting it might be worth it to check if Cydia offers a capatable version.
Edit: I am running version 3.5.3-8 of Autotouch.
I've got a TSpeedButton, and a TImageList that holds various glyphs. One of them has an alpha channel, and it looks very nice when drawn to certain parts of the UI... but when that glyph is drawn on a TSpeedButton, it doesn't take the alpha channel into effect.
Looking at the relevant code in TButtonGlyph.DrawButtonGlyph, it's getting passed true for the Transparent parameter, but Transparent isn't taken into account at all in the if FThemesEnabled code path; it only gets referenced in the else section. Since this program has themes enabled, does that mean it's impossible to draw an alpha-blended image to a TSpeedButton?
Is there any way to work around this?
EDIT: Looking at this a bit more closely, it appears that it takes transparency into account... sort of. Pixels that are fully transparent don't get drawn at all, which is correct behavior. But the antialiasing (partial transparency) around the edges gets drawn as fully opaque.
From what I can see, TSpeedButton takes your bitmap and adds it to an image list. That's FGlyphList in the private section of TButtonGlyph. That image list has ColorDepth of cdDeviceDependent. You'd need that image list to have ColorDepth of cd32Bit if you wanted alpha transparency to be respected. So that right there explains the artifacts.
I'm not sure how you go about working around this. I think that I'd personally subclass TButtonGlyph and connect it directly to an image list of your selection. Then replace TButtonGlyph.DrawButtonGlyph with code that draws from your image list. I'm sure you'll be able to work out the coding.
But there'll be no simple workaround. Without major modifications to the VCL, that image list is going to use cdDeviceDependent and there's no simple way for you to change that. There's no obvious quick fix that I can see.
Actually, what I would do would be to use TButton, but that's just me.
Regarding the Transparent property of the speed button the documentation says:
Specifies whether the background of the button is transparent.
Use Transparent to specify whether the background of the button is transparent.
This property works only if the Flat property of the TSpeedButton is set to True.
When Windows themes are enabled, the Transparent property cannot make the button background transparent.
That is, the property influences the button background, for flat buttons, when unthemed, and has no relevance to glyph drawing.
In the interest of full disclosure: I ran into a similar problem and used this Q&A to base my initial solution on. Since I needed a quick fix for my app and compiler (I use C++ Builder 2009) I decided to 'hack' the VCL's private members and set Glyphlist's ColorDepth to cd32Bit, until I had time to work on something more permanent. This hack worked, with a few more changes, but was far from ideal, for all the obvious reasons, but also functionality wise, there were issues to consider.
However, while doing so I experimented further and finally arrived at below, working solution, without any dirty tricks needed. Not being that familiar with the internal workings of VCL and the rather 'it's not possible as is' feedback from people who are far more in depth in VCL I'm a bit unsure on what I'm missing right now !? Because below solution works for me. Confirmed on XP, XP-High-contrast, W7 and W10. The Glyphs don't look good on W2K but then that is true for Main Menu and Popup icons as well, so that is that.
I code in C++ but use the same underlying VCL, so I will post my C++ solution here, as answer to a Delphi question. I hope that is OK, in the interest of offering a path to a solution rather than keeping it to myself. A Delphi guru can convert and post as answer as well then, unless I'm missing something and my solution isn't what was needed all along.
Since I started with the 'hack' approach first I subclassed TSpeedButton and put the code in a MyTSpeedButton function, but that is not needed to make this work. However, it is the code that I'm posting:
header
class MyVCLTSpeedButton : public TSpeedButton
{
public:
__fastcall MyVCLTSpeedButton(Classes::TComponent* AOwner) ;
void __fastcall AddGlyphWithAlphaChannel(Imglist::TCustomImageList* ImageList, int UpIndex, int DisabledIndex = -1, int ClickedIndex = -1, int DownIndex = -1) ;
};
cpp
void __fastcall MyVCLTSpeedButton::AddGlyphWithAlphaChannel(Imglist::TCustomImageList* ImageList, int UpIndex, int DisabledIndex, int ClickedIndex, int DownIndex)
{
Glyph->Assign(NULL) ; // Clear the current bitmap
NumGlyphs = (DisabledIndex > -1)?((ClickedIndex > -1)?((DownIndex > -1)?(4):(3)):(2)):(1) ;
Graphics::TBitmap *BitMap = new Graphics::TBitmap ;
//BitMap->AlphaFormat = afDefined ; // Don't Change AlphaFormat, it will negatively affect rendering !
BitMap->PixelFormat = pf32bit ; // Doesn't seem to be that important
BitMap->Height = ImageList->Height ;
BitMap->Width = (ImageList->Width * NumGlyphs) ;
BitMap->Canvas->Brush->Color = Parent->Brush->Color ;
BitMap->Canvas->Brush->Style = bsSolid ;
BitMap->Canvas->FillRect(BitMap->Canvas->ClipRect) ;
TIcon *Icon = new TIcon() ;
for (int x = 0 ; x < NumGlyphs ; x++)
{
ImageList->GetIcon((x == 0)?(UpIndex):((x == 1)?(DisabledIndex):((x == 2)?(ClickedIndex):(DownIndex))), Icon) ;
BitMap->Canvas->Draw((ImageList->Width * x), 0, Icon) ;
}
Glyph->Assign(BitMap) ;
delete Icon ;
delete BitMap ;
}
I want to create a script to automatically click on a moving target on a game.
To do this I want to check colours on the screen to determine where my target is and then click him, repeating this if he moves.
I'm not skilled at programming and there might be an easier solution to what I have proposed below:
1/Split the screen into equal tiles - size of tile should represent the in game object.
2/Loop through each pixel of each tile and create a histogram of the pixel colours.
3/If the most common recorded colour matches what we need, we MIGHT have the correct tile. Save the coords and click the object to complete task
4/Every 0.5 seconds check colour to determine if the object has moved, if it hasnt, keep clicking, if it has repeat steps 1, 2 and 3.
The step I am unsure of how to do technically is step 1. What data structure would I need for a tile? Would a 2D array suffice? Store the value of each colour in this array and then determine if it is the object. Also in pseudo how would I split the screen up into tiles to be searched? The tile issue is my main problem.
EDIT for rayryeng 2:
I will be using Python for this task. This is not my game, I just want to create a macro to automatically perform a task for me in the game. I have no code yet, I am looking more for the ideas behind making this work than actual code.
3rd edit and final code:
#!/usr/bin/python
import win32gui, win32api
#function to take in coords and return colour
def colour_return(x,y):
colours = win32gui.GetPixel(win32gui.GetDC(win32gui.GetActiveWindow()), x,y)
return colours
def click(x,y):
win32api.SetCursorPos((x,y))
win32api.mouse_event(win32con.MOUSEEVENTF_LEFTDOWN,x,y,0,0)
win32api.mouse_event(win32con.MOUSEEVENTF_LEFTUP,x,y,0,0)
#variable declaration
x = 1
y = 1
pixel_value = []
colour_found = 0
while x < 1600:
pixel_value = colour_return(x,y)
if pixel_value == 1844766:
click(x,y)
x=x+1
#print x
print y
if x == 1600:
y=y+1
x=1
#print tile
pixel_value = 0
This is the final code that I have produced. It works but it is incredibly slow. It takes 30 seconds seconds to search all 1600 pixels of y=1. I guess it is my method that is not working. Instead of using histograms and tiles I am now just searching for a colour and clicking the coordinates when it matches. What is the fastest method to use when searching an entire screen for a certain colour? I've seen colour detection bots that manage to keep up every second with a moving character.