Remove delay for some emails using SideKiq - ruby-on-rails

Question

I'm using SideKiq and Redis to deliver emails asynchronously on my Rails (4.2) site but some emails take too long to send. Most emails take between 5 and 15 minutes to to be delivered (I'm not sure if this is normal and may be part of the problem). For most emails I send this wait is fine, but for confirmation emails, I need them to be sent more or less immediately.
Normally when I send an email, I send it like this:
UserMailer.delay.my_email(args)
To send an email immediately, I have tried all of the following:
UserMailer.my_email(args).deliver
UserMailer.my_email(args).deliver!
UserMailer.my_email(args).deliver_now
UserMailer.my_email(args).deliver_now!
However all of these result in a 504 error. When I try to send an email, the page waits for several seconds and then I get in my unicorn.log is:
Started POST "/my-path/" for 0.0.0.0 at 0000-00-00 00:00:00 +0000
Processing by MyController#my_path as HTML
Parameters: {"utf8"=>"✓", "authenticity_token"=>"...", ...}
Rendered user_mailer/my_email.html.haml within layouts/user_mailer (0.0ms)
E, [0000-00-00T00:00:00.000000 #00000] ERROR -- : worker=1 PID:00000 timeout (61s > 60s), killing
E, [0000-00-00T00:00:00.000000 #00000] ERROR -- : reaped #<Process::Status: pid 00000 SIGKILL (signal 9)> worker=1
I, [0000-00-00T00:00:00.000000 #00000] INFO -- : worker=1 spawning...
I, [0000-00-00T00:00:00.000000 #00001] INFO -- : worker=1 spawned pid=00001
I, [0000-00-00T00:00:00.000000 #00001] INFO -- : Refreshing Gem list
I, [0000-00-00T00:00:00.000000 #00001] INFO -- : worker=1 ready
This is my sidekiq.yml (the both production and preview are taking a very long time):
---
:concurrency: 5
:logfile: log/sidekiq.log
preview:
:pidfile: /home/unicorn/sidekiq_preview.pid
production:
:pidfile: /home/unicorn/sidekiq_production.pid
:concurrency: 15
I'm not getting anything in my sidekiq.log or mail.log.