I'm using the following sample app that Apple provides to do some object detection.
https://developer.apple.com/documentation/vision/tracking_multiple_objects_or_rectangles_in_video
I'm trying to paste an image of a face on top of the green rectangle in the video. (Video Download Link: https://drive.google.com/file/d/1aw5L-6uBMTxeuq378Y98dZcTh6N_Y2Pf/view?usp=sharing)
So far, I'm able to detect the green rectangle from the video very consistently, but whenever I try to overlay an image, the frame just does not appear in the view.
Here's what I've tried so far:
In TrackingImageView.swift, I've added an instance variable called faceImage and I've tried adding it to the screen by adding the following code to the bottom of the draw function.
UIGraphicsBeginImageContextWithOptions(self.imageAreaRect.size, false, 0.0)
// self.faceImage.draw(in: CGRect(origin: CGPoint.init(x: rect.minX, y: rect.minY), size: rect.size))
self.faceImage.draw(in: CGRect(x: previous.x, y: previous.y, width: polyRect.boundingBox.width, height: polyRect.boundingBox.height))
// self.faceImage.draw(in: rect)
let newImage = UIGraphicsGetImageFromCurrentImageContext()
UIGraphicsEndImageContext()
self.image = newImage
Then in TrackingViewController, in the function called func displayFrame(_ frame: CVPixelBuffer?, withAffineTransform transform: CGAffineTransform, rects: [TrackedPolyRect]?), I've added the following lines.
self.trackingView.faceImage = UIImage(named: "dwight1")
self.trackingView.displayImage(rect: self.trackingView.polyRects[0].boundingBox)
UPDATE, Here's another approach I tried:
This is what it says in the documentation: Use the observation’s boundingBox to determine its location, so you can update your app or UI with the tracked object’s new location. Also use it to seed the next round of tracking.
So in the function func performTracking(type: TrackedObjectType) in VisionTrackerProcessor, I added this:
delegate?.updateImage(observation.boundingBox)
And in TrackingViewController I added this:
func updateImage(_ rect: CGRect) {
print(rect)
self.faceImage.frame = rect
}
And faceImage is this:
#IBOutlet weak var faceImage: UIImageView!
When I print out the CGPoints of the rectangle where I want to place the image, I get the following output:
(0.45066666666666666, 0.5595238095238095, 0.09599999999999997, 0.16666666666666663)
(0.4521519184112549, 0.5643428802490235, 0.09600000381469731, 0.16666666666666663)
(0.4546553611755371, 0.5875609927707248, 0.09555779099464418, 0.16589893764919705)
(0.4543778896331787, 0.5984047359890408, 0.09505770206451414, 0.1650307231479221)
(0.454343843460083, 0.6052030351426866, 0.09476101398468023, 0.16451564364963112)
(0.45296874046325686, 0.6065650092230903, 0.09457258582115169, 0.16418851216634112)
(0.4510493755340576, 0.6057157728407118, 0.09507998228073117, 0.1650694105360243)
(0.4481017589569092, 0.5987161000569662, 0.09499880075454714, 0.16492846806844075)
(0.44568862915039065, 0.5735456678602431, 0.09511266946792607, 0.16512615415785048)
(0.4434205532073975, 0.5485235426161025, 0.09506692290306096, 0.16504673428005645)
(0.4413131237030029, 0.5238201141357421, 0.09566491246223452, 0.1660849147372776)
(0.4388014316558838, 0.5072469923231336, 0.09601176977157588, 0.1666870964898003)
(0.4374812602996826, 0.4967741224500868, 0.09586981534957884, 0.16644064585367835)
(0.43827009201049805, 0.48819330003526473, 0.09551617503166199, 0.1658266809251574)
(0.44115781784057617, 0.4852377573649089, 0.09499365091323853, 0.1649195247226291)
(0.4417849540710449, 0.4845396253797743, 0.0949023962020874, 0.1647610982259115)
(0.4476351737976074, 0.49016346401638455, 0.09391363859176638, 0.16304450564914275)
(0.4497058391571045, 0.49209620157877604, 0.09434010386466984, 0.16378489600287544)
(0.4514862060546875, 0.49223976135253905, 0.09459822773933413, 0.16423302756415475)
(0.454580020904541, 0.4904879252115885, 0.0949873864650726, 0.16490865283542205)
(0.4566154479980469, 0.48613760206434464, 0.09480695724487309, 0.16459540261162653)
(0.45992450714111327, 0.47563196818033854, 0.09525291323661805, 0.1653696378072103)
(0.464534330368042, 0.46896955702039933, 0.09566755294799806, 0.1660895029703776)
(0.4682444095611572, 0.4513437059190538, 0.09700422883033755, 0.16841011047363275)
(0.4709425926208496, 0.438845952351888, 0.09843692183494568, 0.17089743084377712)
(0.47597203254699705, 0.4264893849690755, 0.10058027505874634, 0.17461851967705622)
(0.48175721168518065, 0.42467672559950087, 0.10141149759292606, 0.1760616196526421)
(0.483599328994751, 0.44046991136338975, 0.10279589891433716, 0.17846510145399308)
(0.4847916603088379, 0.44517923990885416, 0.10338790416717525, 0.17949288686116532)
(0.4889643669128418, 0.45437651740180124, 0.09983686804771424, 0.17332788043551978)
(0.49118928909301757, 0.4580091264512804, 0.09644789695739747, 0.16744425031873916)
(0.4905869483947754, 0.45951224433051213, 0.09397981166839603, 0.16315938101874455)
(0.4874621868133545, 0.45792486402723526, 0.09055853486061094, 0.15721967485215932)
(0.48279714584350586, 0.4531046549479167, 0.08872739672660823, 0.1540406121148004)
(0.4783169269561768, 0.4456812964545356, 0.0860174298286438, 0.1493358188205295)
(0.4728221893310547, 0.44693773057725694, 0.084199583530426, 0.14617982440524635)
(0.471103572845459, 0.4579927232530382, 0.08219499588012691, 0.14269964430067272)
(0.4676462173461914, 0.47325596279568144, 0.08054903745651243, 0.1398420651753744)
(0.463164234161377, 0.4803483327229818, 0.07916470766067507, 0.13743872112698025)
(0.4597337245941162, 0.4865601857503255, 0.07723031044006345, 0.1340803888108995)
(0.4575923442840576, 0.4861404842800564, 0.07577759623527525, 0.13155832290649416)
(0.456453275680542, 0.48211678398980035, 0.0741972386837006, 0.12881464428371853)
(0.45630569458007814, 0.47852266099717883, 0.0741972386837006, 0.12881464428371853)
(0.45930023193359376, 0.4749870724148221, 0.0741972386837006, 0.12881464428371847)
(0.4619853973388672, 0.460075675116645, 0.0741972386837006, 0.12881464428371853)
(0.4647641658782959, 0.44653006659613714, 0.0741972386837006, 0.12881464428371858)
(0.46242194175720214, 0.43739403618706596, 0.07220322489738468, 0.1253528171115451)
(0.4625579357147217, 0.41982913547092016, 0.07062785029411311, 0.12261778513590493)
(0.46608676910400393, 0.4134985182020399, 0.06866733431816097, 0.11921412150065108)
(0.46996197700500486, 0.41352043151855467, 0.0672459602355957, 0.11674645741780598)
(0.4733128547668457, 0.42267172071668835, 0.06592562794685364, 0.11445420583089194)
(0.4805797576904297, 0.4420909881591797, 0.06590123176574703, 0.11441185209486215)
(0.48854408264160154, 0.46238810221354165, 0.06529000997543333, 0.11335069868299696)
(0.4921866416931152, 0.47235264248318143, 0.06412824392318728, 0.11133375167846682)
(0.4948731899261475, 0.481452645195855, 0.06294543147087095, 0.10928025775485567)
(0.49323139190673826, 0.48434698316786023, 0.06219365000724797, 0.10797508027818464)
(0.4935962200164795, 0.47917471991644967, 0.061773008108139016, 0.10724479887220595)
(0.49112601280212403, 0.4626174502902561, 0.06177300810813907, 0.107244798872206)
(0.48893303871154786, 0.4498925950792101, 0.06069326996803287, 0.10537025663587785)
(0.4902684688568115, 0.45128373040093317, 0.06060827970504756, 0.10522270202636719)
(0.4870577812194824, 0.45470954047309026, 0.06060827970504756, 0.10522270202636724)
(0.45066666666666666, 0.5595238095238095, 0.09599999999999997, 0.16666666666666663)
(0.45066666666666666, 0.5595238095238095, 0.09599999999999997, 0.16666666666666663)
Any help with overlaying the image on top of my detected object would be amazing. Thanks!
Are you realising that the coordinates you get from the Vision framework are normalised ones(between 0 and 1)?. You will have to transform those to fit the size of your view.
In addition, as far as I remember, Vision coordinates start from the bottom left corner (contrary to UIKit, starting from the top- left), so you might have to flip them vertically as well(not 100% sure here).
Edit:
I see you have available videoReader.affineTransform, you can give it a try modifying your CGRects using that transform.
I am currently trying to develop a drawing app. The problem that I've come across is optimising the app's performance.
I have subclassed an UIView, in which I am detecting user's inputs.
At first I tried to draw all the lines with CoreGraphics in draw(_ rect: CGRect) method but when the number of lines was 10 000+, the lag was very noticeable. So then I thought of creating a CGImage that I would make after every draw(...) cycle and then when the next draw cycle came, I would put just the new lines on top of that CGImage, therefore not drawing ALL previous lines again and saving precious time.
I am registering new lines in touchesBegan() and touchesMoved() methods and then I am calling self.needsDisplay().
This is the code that I am currently using:
var mainImage: CGImage?
var newLines: [Line]
override func draw(_ rect: CGRect) {
let context = UIGraphicsGetCurrentContext()
//context?.translateBy(x: 0, y: bounds.height)
//context?.scaleBy(x: 1, y: -1)
if let image = mainImage {
context?.draw(image, in: bounds)
}
for line in newLines {
context?.setStrokeColor(UIColor.black.cgColor)
context?.setLineWidth(1)
context?.move(to: line.previousLocation)
context?.addLine(to: line.location)
context?.strokePath()
}
newLines = []
mainImage = context?.makeImage()
}
Now the problem with it is that the lines I draw are divided in two parts and are mirrored vertically. (Image split vertically). I have drawn here two continuous lines - one straight line and one curved line. The weird thing is that if I flipped half of the UIView, the lines would be aligned continuously with no space between them.
If I uncommented the two lines from my previously mentioned code (context?.translateBy(x: 0, y: bounds.height) and context?.scaleBy(x: 1, y: -1)), the image would look like this.
The only wrong thing now is that I was drawing on the lower side of the screen and getting results on the upper one.
How do I correct it? What is the proper solution for my problem?
Thank you very much!
When you comment out those two lines, then you are NOT making any changes to the graphics context. If you un-comment them, then you ARE changing the context at each call and not saving it. So I think you're essentially flipping it each time. Not sure if it's that but you should definitely be pushing/saving your context before drawing (and obviously restoring/popping it).
Check out this post: using UIImage drawInRect still renders it upside down.
I'm currently working on a SpriteKit project and need to create a comet with a fading tail that animates across the screen. I am having serious issues with SpriteKit in this regards.
Attempt 1. It:
Draws a CGPath and creates an SKShapeNode from the path
Creates a square SKShapeNode with gradient
Creates an SKCropNode and assigns its maskNode as line, and adds square as a child
Animates the square across the screen, while being clipped by the line/SKCropNode
func makeCometInPosition(from: CGPoint, to: CGPoint, color: UIColor, timeInterval: NSTimeInterval) {
... (...s are (definitely) irrelevant lines of code)
let path = CGPathCreateMutable()
...
let line = SKShapeNode(path:path)
line.lineWidth = 1.0
line.glowWidth = 1.0
var squareFrame = line.frame
...
let square = SKShapeNode(rect: squareFrame)
//Custom SKTexture Extension. I've tried adding a normal image and the leak happens either way. The extension is not the problem
square.fillTexture = SKTexture(color1: UIColor.clearColor(), color2: color, from: from, to: to, frame: line.frame)
square.fillColor = color
square.strokeColor = UIColor.clearColor()
square.zPosition = 1.0
let maskNode = SKCropNode()
maskNode.zPosition = 1.0
maskNode.maskNode = line
maskNode.addChild(square)
//self is an SKScene, background is an SKSpriteNode
self.background?.addChild(maskNode)
let lineSequence = SKAction.sequence([SKAction.waitForDuration(timeInterval), SKAction.removeFromParent()])
let squareSequence = SKAction.sequence([SKAction.waitForDuration(1), SKAction.moveBy(CoreGraphics.CGVectorMake(deltaX * 2, deltaY * 2), duration: timeInterval), SKAction.removeFromParent()])
square.runAction(SKAction.repeatActionForever(squareSequence))
maskNode.runAction(lineSequence)
line.runAction(lineSequence)
}
This works, as shown below.
The problem is that after 20-40 other nodes come on the screen, weird things happen. Some of the nodes on the screen disappear, some stay. Also, the fps and node count (toggled in the SKView and never changed)
self.showsFPS = true
self.showsNodeCount = true
disappear from the screen. This makes me assume it's a bug with SpriteKit. SKShapeNode has been known to cause issues.
Attempt 2. I tried changing square from an SKShapeNode to an SKSpriteNode (Adding and removing lines related to the two as necessary)
let tex = SKTexture(color1: UIColor.clearColor(), color2: color, from: from, to: to, frame: line.frame)
let square = SKSpriteNode(texture: tex)
the rest of the code is basically identical. This produces a similar effect with no bugs performance/memory wise. However, something odd happens with SKCropNode and it looks like this
It has no antialiasing, and the line is thicker. I have tried changing anti-aliasing, glow width, and line width. There is a minimum width that can not change for some reason, and setting the glow width larger does this
. According to other stackoverflow questions maskNodes are either 1 or 0 in alpha. This is confusing since the SKShapeNode can have different line/glow widths.
Attempt 3. After some research, I discovered I might be able to use the clipping effect and preserve line width/glow using an SKEffectNode instead of SKCropNode.
//Not the exact code to what I tried, but very similar
let maskNode = SKEffectNode()
maskNode.filter = customLinearImageFilter
maskNode.addChild(line)
This produced the (literally) exact same effect as attempt 1. It created the same lines and animation, but the same bugs with other nodes/fps/nodeCount occured. So it seems to be a bug with SKEffectNode, and not SKShapeNode.
I do not know how to bypass the bugs with attempt 1/3 or 2.
Does anybody know if there is something I am doing wrong, if there is a bypass around this, or a different solution altogether for my problem?
Edit: I considered emitters, but there could potentially be hundreds of comets/other nodes coming in within a few seconds and didn't think they would be feasible performance-wise. I have not used SpriteKit before this project so correct me if I am wrong.
This looks like a problem for a custom shader attached to the comet path. If you are not familiar with OpenGL Shading Language (GLSL) in SpriteKit it lets you jump right into the GPU fragment shader specifically to control the drawing behavior of the nodes it is attached to via SKShader.
Conveniently the SKShapeNode has a strokeShader property for hooking up an SKShader to draw the path. When connected to this property the shader gets passed the length of the path and the point on the path currently being drawn in addition to the color value at that point.*
controlFadePath.fsh
void main() {
//uniforms and varyings
vec4 inColor = v_color_mix;
float length = u_path_length;
float distance = v_path_distance;
float start = u_start;
float end = u_end;
float mult;
mult = smoothstep(end,start,distance/length);
if(distance/length > start) {discard;}
gl_FragColor = vec4(inColor.r, inColor.g, inColor.b, inColor.a) * mult;
}
To control the fade along the path pass a start and end point into the custom shader using two SKUniform objects named u_start and u_end These get added to the custom shader during initialization of a custom SKShapeNode class CometPathShape and animated via a custom Action.
class CometPathShape:SKShapeNode
class CometPathShape:SKShapeNode {
//custom shader for fading
let pathShader:SKShader
let fadeStartU = SKUniform(name: "u_start",float:0.0)
let fadeEndU = SKUniform(name: "u_end",float: 0.0)
let fadeAction:SKAction
override init() {
pathShader = SKShader(fileNamed: "controlFadePath.fsh")
let fadeDuration:NSTimeInterval = 1.52
fadeAction = SKAction.customActionWithDuration(fadeDuration, actionBlock:
{ (node:SKNode, time:CGFloat)->Void in
let D = CGFloat(fadeDuration)
let t = time/D
var Ps:CGFloat = 0.0
var Pe:CGFloat = 0.0
Ps = 0.25 + (t*1.55)
Pe = (t*1.5)-0.25
let comet:CometPathShape = node as! CometPathShape
comet.fadeRange(Ps,to: Pe) })
super.init()
path = makeComet...(...) //custom method that creates path for comet shape
strokeShader = pathShader
pathShader.addUniform(fadeStartU)
pathShader.addUniform(fadeEndU)
hidden = true
//set up for path shape, eg. strokeColor, strokeWidth...
...
}
func fadeRange(from:CGFloat, to:CGFloat) {
fadeStartU.floatValue = Float(from)
fadeEndU.floatValue = Float(to)
}
func launch() {
hidden = false
runAction(fadeAction, completion: { ()->Void in self.hidden = true;})
}
...
The SKScene initializes the CometPathShape objects, caches and adds them to the scene. During update: the scene simply calls .launch() on the chosen CometPathShapes.
class GameScene:SKScene
...
override func didMoveToView(view: SKView) {
/* Setup your scene here */
self.name = "theScene"
...
//create a big bunch of paths with custom shaders
print("making cache of path shape nodes")
for i in 0...shapeCount {
let shape = CometPathShape()
let ext = String(i)
shape.name = "comet_".stringByAppendingString(ext)
comets.append(shape)
shape.position.y = CGFloat(i * 3)
print(shape.name)
self.addChild(shape)
}
override func update(currentTime: CFTimeInterval) {
//pull from cache and launch comets, skip busy ones
for _ in 1...launchCount {
let shape = self.comets[Int(arc4random_uniform(UInt32(shapeCount)))]
if shape.hasActions() { continue }
shape.launch()
}
}
This cuts the number of SKNodes per comet from 3 to 1 simplifying your code and the runtime environment and it opens the door for much more complex effects via the shader. The only drawback I can see is having to learn some GLSL.**
*not always correctly in the device simulator. Simulator not passing distance and length values to custom shader.
**that and some idiosyncrasies in CGPath glsl behavior. Path construction is affecting the way the fade performs. Looks like v_path_distance is not blending smoothly across curve segments. Still, with care constructing the curve this should work.