Make view draggable but not tappable - ios

Question

I have a label in my view controller that I set like this in viewDidLoad:
var label: UILabel!
override func viewDidLoad() {
super.viewDidLoad()
label = UILabel()
view.add(label)
label.frame = CGRect(x: 0, y, 0, width: 20, height: 20)
label.text = "A"
let recognizer = UIPanGestureRecognizer(target: self, action: #selector(didDrag(_:))
label.addGestureRecognizer(recognizer)
label.isUserInteractionEnabled = true
}
I want the label to be draggable, so I implemented didDrag like this:
#objc func didDrag(_ gesture: UIPanGestureRecognizer) {
let center = label.center
let translation = gesture.translation(in: view)
label.center = CGPoint(x: center.x + translation.x, y: center.y + translation.y)
gesture.setTranslation(.zero, in: view)
}
It works perfectly and I can drag my label around.
However, if the label is over a button in my view and I try to tap the button, the button does not get the touch. I have tried:
label.isUserInteractionEnabled = false // then I can't also drag around
recognizer.cancelsTouchesInView = false // nothing changes
label.isMultipleTouchEnabled = false // nothing changes
Any idea how I can let single or double taps be passed/be ignored by the label's pan recognisers, but still be able to drag it?
I can delete this question if it's a duplicate, but I have not found any other that asks exactly the same.
Edit
Following Pass taps through a UIPanGestureRecognizer, the closest question I've found so far, I also tried:
recognizer.delaysTouchesBegan = true
recognizer.maximumNumberOfTouches = 1
recognizer.cancelsTouchesInView = true
But it, unfortunately, did not work.
Update
The view has all sorts of button and textfields, so comparing with each of them is not really possible, mostly because there are also stack views that contain buttons and sometimes they are there and somethings they are not.

Answer 1

After thinking I came up with the solution that might work. The idea is playing around with touch events. Every view has a set of methods like touchesBegan(), touchesMoved() etc. Now what you can do is when touchesBegan is invoked you can check the location of the touch that is happening and if this location contained in the frame of the button, call the function manually.
Here's the pseudocode so you get the concept. Not tested but from my experience something like that should work
override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
touches.forEach { touch in
let touchLocation = touch.location(in: myButton) // that may need to be self instead of myButton
if myButton.frame.contains($0.location) {
//invoke button method here
}
}
}