Extra object in Google Cloud Storage when using Fog - ruby-on-rails

I am following the fog.io/storage example for creating a directory and then uploading a file. Everything works great when I push my file into Google Cloud Storage except there is always a "binary/octet-stream" file named exactly as the deepest file I create.
My code is very similar to the AWS example in that I create a directory and from that new directory, I create a file. The directory structure is created properly and the file is uploaded properly but there is always an extra, 0-byte file. My code looks like:
job_number = 100
connection = Fog::Storage.new({
:provider => 'Google',
:google_storage_access_key_id => YOUR_GCE_ACCESS_KEY_ID,
:google_storage_secret_access_key => YOUR_GCE_SECRET_ACCESS_KEY
})
directory = connection.directories.create(
:key => "test-project/uploads/#{job_number}",
:public => false
)
file = directory.files.create(
:key => 'file.pdf',
:content_type => 'application/pdf',
:body => File.open("/path/to/my/file.pdf"),
:public => false
)
The directory structure is perfect (gs://test-project/uploads/100 folder exists) and the file.pdf file exists in that directory as well (gs://test-project/uploads/100/file.pdf).
The problem is that after the:
directory = connection.directories.create(
:key => "test-project/uploads/#{job_number}",
:public => false
)
command runs, there is a file at gs://test-project/uploads/100 as well as a directory gs://test-project/uploads/100/. When I walk through the code, the connection.directories.create(...) command is definitely creating the extra file but I cannot figure out why.
I have also tried to add a trailing slash to the key value for the connection.directories.create(...) command but that actually creates a different directory structure problem that is worse than this (this isn't bad, just annoying).
Has anyone seen this or know how to correctly have the directory structure created through Fog?

Instead of creating the directory right up to the file, just create/get the base directory/bucket and then save the file with the rest of the directory structure. So it would look like this:
job_number = 100
connection = Fog::Storage.new({
:provider => 'Google',
:google_storage_access_key_id => YOUR_GCE_ACCESS_KEY_ID,
:google_storage_secret_access_key => YOUR_GCE_SECRET_ACCESS_KEY
})
directory = connection.directories.create(
:key => "test-project",
:public => false
)
file = directory.files.create(
:key => 'uploads/#{job_number}/file.pdf',
:content_type => 'application/pdf',
:body => File.open("/path/to/my/file.pdf"),
:public => false
)

Related

Rails Fog create a file based on a remote link

I have Rails application and use fog gem to upload files to cloud storage (Rackspace). As of now I have been successfully uploading local files to cloud storage.
#service = Fog::Storage.new(options)
directory = #service.directories.new :key => 'test'
directory.files.create :key => path, :body => file, :content_type => content_type
I have a new requirement now. I want to be able to use remote link (public-url) and have it uploaded to cloud storage. Is there a way to achieve this without downloading it locally or loading the whole thing in memory?
I'm looking for something like this:
directory.files.create :key => path, :body => 'url-to-remote-file', :content_type => content_type
A stream-based approach would also be quite helpful.
Thanks.

Paperclip validation issue on production

I have a Problem when I deploy my application on google cloud I get this error
has contents that are not what they are reported to be
Locally it works fine! I already tried to using the command_path. So I really don't know what I have to do next...
This is my model
has_mongoid_attached_file :image,
:styles => { :large => "380x380!" , :medium => "240x240", :small => "120x120!" },
:storage => :fog,
:fog_public => true,
:fog_directory => 'XXXX',
:path => "images/:id/:style/:basename.:extension",
:fog_credentials => { :provider => 'Google',
:google_storage_access_key_id => 'XXXXX',
:google_storage_secret_access_key => 'XXXXX'}
validates_attachment_content_type :image, :content_type => ["image/jpg", "image/jpeg", "image/png", "image/gif"]
Thank you for your efforts. I hope you guys can help me
Okay I found a result. I just created a initializers/paperclip.rb file
require 'paperclip/media_type_spoof_detector'
module Paperclip
class MediaTypeSpoofDetector
def spoofed?
false
end
end
end
Right now it work's perfectly for me.
If you have problems with ImageMagick on App Engine using Rails see this link
That issue occurs because the content-type discovered from file command returns empty string.
Actually system is not able to find the file executable so a exception is raised and empty string is returned back.
Check the code below
begin
Paperclip.run("file", "-b --mime :file", :file => '/tmp/RackMultipart20160826-15649-kwvnq2.png').split(/[:;]\s+/).first
rescue Cocaine::CommandLineError
""
end
Solution:-
Add below line in you initializer file.
Paperclip.options[:command_path] = '/usr/bin'
Looks like Google Cloud can't determine MIME type of uploaded files.
You can map file extensions to types in you initializer (application.rb, production.rb or create initializers/paperclip.rb)
Paperclip.options[:content_type_mappings] = {
:jpg => "image/jpeg",
:png => "image/png",
:gif => "image/gif"
}
But this way spoofing check won't be performed for image files.
I know I am late to the party, but in working with a legacy RoR system, I ran into this issue. The problem arose in setting the app up in Docker. Ultimately paperclip calling imagemagick was attempting to use file to identify mime-type and the minimal Docker did not have it installed. apt-get install file fixed it.

Using Paperclip, Fog, and Ceph

I'm writing a Rails 3 app that uses Paperclip to transcode a video file attachment into a bunch of other formats, and then to store the resulting files. It all works fine for local storage, but I am trying to make it work using Paperclip's Fog support to store files in a bucket on our own Ceph cluster. However, I can't seem to find the right configuration options to make Fog talk to my Ceph server.
Here is a snippet from my Rails class:
has_attached_file :videofile,
:storage => :fog,
:fog_credentials => { :aws_access_key_id => 'xxx', :aws_secret_access_key => 'xxx', :provider => 'AWS'},
:fog_public => true,
:url => ":id/:filename",
:fog_directory => 'replay',
:fog_host => 'my-hostname',
Writes using this setup fail because Paperclip attempts to save to Amazon S3 rather than the host I've provided. I have a non-Rails / non-Paperclip toy script working just fine:
conn = Fog::Storage.new({
:aws_access_key_id => 'xxx',
:aws_secret_access_key => 'xxx',
:host => 'my-hostname',
:path_style => true,
:provider => "AWS",
})
This correctly connects to my local Ceph server. So I suspect there is something I'm not configuring in Paperclip properly - but what?
Here's the relevant hunk from fog.rb that I think is causing the connection to only go to AWS:
def host_name_for_directory
if #options[:fog_directory].to_s =~ Fog::AWS_BUCKET_SUBDOMAIN_RESTRICTON_REGEX
"#{#options[:fog_directory]}.s3.amazonaws.com"
else
"s3.amazonaws.com/#{#options[:fog_directory]}"
end
end
the error was just from an improperly configured Ceph cluster. For anyone who finds this thread, as long as you:
Have your wildcard DNS set up properly for your Ceph frontend;
Ceph configured to recognize as such
Pass in :host in :fog_credentials, which would be the FQDN of the Ceph frontend
:fog_host, which apparently needs to be the URL for your bucket, e.g. https://bucket.ceph-server.foobar.com.
Paperclip will work out of the box. I don't think that it is documented anywhere that you can use :host but it works.

Paperclip saving the image outside the :rails_root

Hi am using paperclip for image upload, problem is i want to access the image for two application deployed in the same server, for that i have to save the image in common folder outside of rails root. how can i do this?
Help me to resolve this problem.
https://github.com/thoughtbot/paperclip under this got to Understanding Storage. You can specify path to whatever folder you want.
You can change it by two ways:
1) config/application.rb or in any of the config/environments/*.rb files
module YourApp
class Application < Rails::Application
# Other code...
config.paperclip_defaults = {:storage => :fog, :fog_credentials => {:provider => "Local", :local_root => "#{Rails.root}/public"}, :fog_directory => "", :fog_host => "localhost"}
end
end
2) Rails initializer:
Paperclip::Attachment.default_options[:storage] = :fog
Paperclip::Attachment.default_options[:fog_credentials] = {:provider => "Local", :local_root => "#{Rails.root}/public"}
Paperclip::Attachment.default_options[:fog_directory] = ""
Paperclip::Attachment.default_options[:fog_host] = "http://localhost:3000"

Rails MIME::Types.type_for(photoshop_1354320001.psd) - MIME Type not found?

I'm using Rails 3, Paperclip(3.3.0), aws-sdk (1.7.1).
My paperclip attachments are being stored securely on S3.
attachment.rb
has_attached_file :attachment,
:storage => :s3,
:s3_credentials => "#{Rails.root}/config/s3.yml",
:s3_protocol => 'https',
:s3_permissions => :private, # Sets the file, not the folder as private in S3
:use_timestamp => false,
:default_style => :original, # NEEDS to be original or download_url method below wont work
:default_url => '/images/:attachment/default_:style.png',
:path => "/:rails_env/private/s/:s_id/uuploaded_files/:basename.:extension"
In order to download the files I generate a secure URL like so:
def authenticated_url(style = nil, expires_in = 1.hour)
mime_type = MIME::Types.type_for(self.attachment_file_name)[0]
attachment.s3_object(style).url_for(:read, :secure => true, :response_content_type => mime_type.to_s, :expires => expires_in).to_s
end
The problem is for PSDs: This is return empty:
Rails MIME::Types.type_for('photoshop_1354320001.psd')
In the code it looks like:
mime_type = MIME::Types.type_for(self.attachment_file_name)[0]
It works for other files fine but not PSDs. Any idea why and how to resolve?
Thanks
Sure. MIME::Types lets you specify custom types.
Stick this into an initializer
# Not quite sure what the appropriate MIMEtype for PSDs are,
# but this is the gist of it.
# .PSB is a larger version of .PSD supporting up to 300000x300000 px
psd_mime_type = MIME::Type.new('image/x-photoshop') do |t|
t.extensions = %w(psd psb)
t.encoding = '8bit'
end
MIME::Types.add psd_mime_type
Now MIME::Types.type_for "test.psd" should give you "image/x-photoshop".

Resources