Rails: link_to delete from different controller - ruby-on-rails

Question

Right now I have a like button that allows you to like foods. When you try to unlike the food, I get this error:
The action 'destroy' could not be found for UsersController
I'm not sure why it is looking for the destroy action in the users controller. My only guess is because the button is on the user show page, so I assume it defaults to that controller, but how would I access the delete method from my votes controller?
Shared like form
<% unless current_user.votes.empty? || current_user.votes.pluck(:food_id).include?(food.id) %>
<%= form_for #vote do |f| %>
<%= f.hidden_field 'food_id', food.id %>
<%= f.hidden_field 'user_id', food.user.id %>
<%= f.submit "Vote", :class => "like_button" %>
<% end %>
<% else %>
<% vote = food.votes.where(user_id: current_user.id).first %>
<div class="unlike_button">
<%= button_to "Unlike", vote, method: :delete %>
</div>
<% end %>
class VotesController < ApplicationController
def index
#votes = Vote.all
end
def new
#vote = Vote.new
end
def create
#vote = Vote.new(vote_params)
if #vote.save
puts #vote
flash[:notice] = "Thanks for voting!"
redirect_back(fallback_location: root_path)
else
puts "No"
flash[:notice] = "Something went wrong"
redirect_back(fallback_location: root_path)
end
end
def destroy
#vote = Vote.find(params[:id])
if #vote.destroy!
flash[:notice] = "Unvoted!"
redirect_back(fallback_location: root_path)
end
end
private
def vote_params
params.require(:vote).permit(:food_id, :user_id)
end
end
class Vote < ApplicationRecord
belongs_to :user
belongs_to :food
end