I want to read a CSV file produced by an export from one of my controllers. Here is the simple code:
def export_houses
houses_file = open("http://127.0.0.1:3000/houses/export.csv")
houses = CSV.open(houses_file, 'r:bom|utf-8', { headers: true })
...
end
The problem occurs on the CSV.open line, where I get the following error message:
TypeError: no implicit conversion of StringIO into String
from /Users/htaidirt/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/csv.rb:1256:in `initialize'
houses_file is correctly found. It's a StringIO class, but I wanted a File class to use with CSV.open.
Do you have an idea how to proceed? Thanks.
You can read a csv file from http server like this :
require 'open-uri'
require 'csv'
url = 'http://127.0.0.1:3000/houses/export.csv'
houses = CSV.new(open(url), :headers => :first_row)
or you can parse it with parse method
require 'open-uri'
require 'csv'
url = 'http://127.0.0.1:3000/houses/export.csv'
houses = CSV.parse(open(url).read, :headers => true)
Hope this helps
Try adding .string at the end of open(uri)
StringIO to String Conversion
def export_houses
houses_file = open("http://127.0.0.1:3000/houses/export.csv").string
houses = CSV.open(houses_file, 'r:bom|utf-8', { headers: true })
...
end
Related
I've been playing around with API calls for a project I'm working on and I'm getting a problem when I try to pass some JSON to a POST request. The call works in Postman, but I just can't figure out how to format it in Ruby. Here's my code:
require 'httparty'
require 'json'
require 'pp'
#use the HTTParty gem
include HTTParty
#base_uri 'https://app.api.com'
#set some basic things to make the call,
#apiUrl = "https://app.api.com/"
#apiUrlEnd = 'apikey=dontStealMePls'
#apiAll = "#{#apiUrl}#{#apiUrlEnd}"
#apiTest = "https://example.com"
def cc_query
HTTParty.post(#apiAll.to_s, :body => {
"header": {"ver": 1,"src_sys_type": 2,"src_sys_name": "Test","api_version": "V999"},
"command1": {"cmd": "cc_query","ref": "test123","uid": "abc01", "dsn": "abcdb612","acct_id": 7777}
})
end
def api_test
HTTParty.post(#apiTest.to_s)
end
#pp api_test()
pp cc_query()
This code gives me this error:
{"fault"=>
{"faultstring"=>"Failed to execute the ExtractVariables: Extract-Variables",
"detail"=>{"errorcode"=>"steps.extractvariables.ExecutionFailed"}}}
I know that error because I would get it if I tried to make a call without any JSON in the body of the call (through Postman). So from that I assume that my code above is not passing any JSON when making an API call. Am I formatting my JSON incorrectly? Am I even formatting the .post call correctly? Any help is appreciated! :)
the api_test() method just makes a POSt call to example.com and it works (saving my sanity).
Just use HTTParty as a mixin instead in a class instead:
require 'httparty'
class MyApiClient
include HTTParty
base_uri 'https://app.api.com'
format :json
attr_accessor :api_key
def initalize(api_key:, **options)
#api_key = api_key
#options = options
end
def cc_query
self.class.post('/',
body: {
header: {
ver: 1,
src_sys_type: 2,
src_sys_name: 'Test',
api_version: 'V999'
},
command1: {
cmd: 'cc_query',
ref: 'test123',
uid: 'abc01',
dsn: 'abcdb612',
acct_id: 7777
}
},
query: {
api_key: api_key
}
)
end
end
Example usage:
MyApiClient.new(api_key: 'xxxxxxxx').cc_query
When you use format :json HTTParty will automatically set the content type and handle JSON encoding and decoding. I'm guessing thats where you failed.
I'm trying to import a CSV file from a url but i'm getting Errno::ENAMETOOLONG: file name too long. I process the file as follow:
require 'open-uri'
url = "http://de65.grepolis.com/data/csv.txt"
url_data = open(url).read()
SmarterCSV.process(url_data, {
...
})
What am i missing ?
You have to pass a filename which should be on server. rightnow you are passing all data . Do something like this
require 'open-uri'
url = "http://de65.grepolis.com/data/csv.txt"
url_data = open(url).read()
File.open('/tmp/file_name', 'w') { |file| file.write(url_data) }
SmarterCSV.process('/tmp/file_name',{ })
I had the same problem using the standard CSV library to pull in a CSV file via an http url. I was able to solve the issue without needing to write to a temporary server file with code like this:
require 'open-uri'
require 'csv'
url = "http://de65.grepolis.com/data/csv.txt"
url_data = open(url).read()
CSV.parse(url_data, headers: true).each do |row|
# per row processing code ...
end
Hope this helps you.
# models/concerns/import.rb
require 'open-uri'
require 'import_error'
module Import
extend ActiveSupport::Concern
class_methods do
def import_remote(url)
csv = CSV.parse(open(url), headers: true)
begin
ActiveRecord::Base.transaction do
counter = 0
csv.each do |row|
row_hash = row.to_hash
begin
instance = self.name.constantize.create!(row_hash)
rescue => e
raise ImportError.new("#{e.message}. at row: #{row_hash}")
end
counter += 1 if instance.persisted?
end
end
rescue => e
return puts e.message
end
puts "Imported #{counter} records"
end
end
end
# lib/tasks/import.rake
namespace :remote_import do
desc "Import companies from CSV"
task :your_model, [:url] do |t, args|
YourModel.import_remote(args.url)
end
end
# lib/import_error.rb
class ImportError < StandardError
end
# models/your_model.rb
class YourModel < ActiveRecord::Base
include Import
end
Gist: https://gist.github.com/victorhazbun87/9ac786961bbf7c235f76
My rb file reads:
require "csv"
puts "Program1 initialized."
contents = CSV.open "data.csv", headers: true
contents.each do |row|
name = row[4]
puts name
end
...but when i run it in ruby it wont load the program. it gives me the error message about the headers:
syntax error, unexpected ':', expecting $end
contents = CSV.open "data.csv", headers: true
so I'm trying to figure out, why won't ruby let me parse this file? I've tried using other csv files I have and it won't load, and gives me an error message. I'm trying just to get the beginning of the program going! I feel like it has to do with the headers. I've updated as much as I can, mind you I'm using ruby 1.8.7. I read somewhere else that I could try to run the program in irb but it didn't seem like it needed it. so yeah... thank you in advance!!!!
Since you are using this with Ruby 1.8.7, :headers => true won't work in this way.
The simplest way to ignore the headers and get your data is to shift the first row in the data, which would be the headers:
require 'csv'
contents = CSV.open("data.csv", 'r')
contents.shift
contents.each do |row|
name = row[4]
puts name
end
If you do want to use the syntax with headers in ruby 1.8, you would need to use FasterCSV, something similar to this:
require 'fastercsv'
FasterCSV.foreach("data.csv", :headers => true) do |fcsv_obj|
puts fcsv_obj['name']
end
(Refer this question for further read: Parse CSV file with header fields as attributes for each row)
I meet an encoding problem... No errors in the console, but the output is not well encoded.
I must use Digest::SHA1.hexdigest on a string and then must pack the result.
The below example should outputs '{´p)ODýGΗ£Iô8ü:iÀ' but it outputs '{?p)OD?GΗ?I?8?:i?' in the console and '{�p)OD�G^BΗ�I�8^D�:i�' in the log file.
So, my variable called pack equals '{?p)OD?GΗ?I?8?:i?' and not '{´p)ODýGΗ£Iô8ü:iÀ'. That's a big problem... I'm doing it in a Rails task.
Any idea guys?
Thanks
# encoding: utf-8
require 'digest/sha1'
namespace :my_app do
namespace :check do
desc "Description"
task :weather => :environment do
hexdigest = Digest::SHA1.hexdigest('29d185d98c984a359e6e6f26a0474269partner=100043982026&code=34154&profile=large&filter=movie&striptags=synopsis%2Csynopsisshort&format=json&sed=20130527')
pack = [hexdigest].pack("H*")
puts pack # => {?p)OD?GΗ?I?8?:i?
puts '{´p)ODýGΗ£Iô8ü:iÀ' # => {´p)ODýGΗ£Iô8ü:iÀ
end
end
end
This is what I did (my conversion from PHP to Ruby)
# encoding: utf-8
require 'open-uri'
require 'base64'
require 'digest/sha1'
class Allocine
$_api_url = 'http://api.allocine.fr/rest/v3'
$_partner_key
$_secret_key
$_user_agent = 'Dalvik/1.6.0 (Linux; U; Android 4.2.2; Nexus 4 Build/JDQ39E)'
def initialize (partner_key, secret_key)
$_partner_key = partner_key
$_secret_key = secret_key
end
def get(id)
# build the params
params = { 'partner' => $_partner_key,
'code' => id,
'profile' => 'large',
'filter' => 'movie',
'striptags' => 'synopsis,synopsisshort',
'format' => 'json' }
# do the request
response = _do_request('movie', params)
return response
end
private
def _do_request(method, params)
# build the URL
query_url = $_api_url + '/' + method
# new algo to build the query
http_build_query = Rack::Utils.build_query(params)
sed = DateTime.now.strftime('%Y%m%d')
sig = URI::encode(Base64.encode64(Digest::SHA1.digest($_secret_key + http_build_query + '&sed=' + sed)))
return sig
end
end
Then call
allocine = Allocine.new(ALLOCINE_PARTNER_KEY, ALLOCINE_SECRET_KEY)
puts allocine.get('any ID')
get method return 'e7RwKU9E%2FUcCzpejSfQ4BPw6acA%3D' in PHP and 'cPf6I4ZP0qHQTSVgdKTbSspivzg=%0A' in Ruby...
thanks again
I think this "encoding" issue has turned up due to debugging other parts of a conversion from PHP to Ruby. The target API that will consume a digest of params looks like it will accept a signature variable constructed in Ruby as follows (edit: well this is guess, there may also be relevant differences between Ruby and PHP in URI encoding and base64 defaults):
require 'digest/sha1'
require 'base64'
require 'uri'
sig_data = 'edhefhekjfhejk8edfefefefwjw69partne...'
sig = URI.encode( Base64.encode64( Digest::SHA1.digest( sig_data ) ) )
=> "+ZabHg22Wyf7keVGNWTc4sK1ez4=%0A"
The exact construction of sig_data from the parameters that are being signed is also important. That is generated by the PHP method http_build_query, and I do not know what order or escaping that will apply to input params. If your Ruby version gets them in a different order, or escapes differently to PHP, the signature will be wrong (edit: Actually it is possible we are looking here for a signature on the exact query string sent the API - I don't know). It is possibly an issue of that sort that has led you down the rabbit hole of how the signature is constructed?
Thank you guys for your help.
Problem is solved. With the following code I obtain exactly the same string as with PHP:
http_build_query = Rack::Utils.build_query(params)
sed = DateTime.now.strftime('%Y%m%d')
sig = CGI::escape(Base64.strict_encode64(Digest::SHA1.digest($_secret_key + http_build_query + '&sed=' + sed)))
Now I've another problem for which I opened a new question here.
thanks you very much.
I can't seem to get this to work. I want to pull a CSV file from a different webserver to read in my application. This is how I'd like to call it:
url = 'http://www.testing.com/test.csv'
records = FasterCSV.read(url, :headers => true, :header_converters => :symbol)
But that doesn't work. I tried Googling, and all I came up with was this excerpt: Practical Ruby Gems
So, I tried modifying it as follows:
require 'open-uri'
url = 'http://www.testing.com/test.csv'
csv_url = open(url)
records = FasterCSV.read(csv_url, :headers => true, :header_converters => :symbol)
... and I get a can't convert Tempfile into String error (coming from the FasterCSV gem).
Can anyone tell me how to make this work?
require 'open-uri'
url = 'http://www.testing.com/test.csv'
open(url) do |f|
f.each_line do |line|
FasterCSV.parse(line) do |row|
# Your code here
end
end
end
http://www.ruby-doc.org/core/classes/OpenURI.html
http://fastercsv.rubyforge.org/
I would retrieve the file with Net::HTTP for example and feed that to FasterCSV
Extracted from ri Net::HTTP
require 'net/http'
require 'uri'
url = URI.parse('http://www.example.com/index.html')
res = Net::HTTP.start(url.host, url.port) {|http|
http.get('/index.html')
}
puts res.body
You just had a small typo. You should have used FasterCSV.parse instead of FasterCSV.read:
data = open('http://www.testing.com/test.csv')
records = FasterCSV.parse(data)
I would download it with rio - as easy as:
require 'rio'
require 'fastercsv'
array_of_arrays = FasterCSV.parse(rio('http://www.example.com/index.html').read)
I upload CSV file with Paperclip and save it to Cloudfiles and then start file processing with Delayed_job.
This worked for me:
require 'open-uri'
url = 'http://www.testing.com/test.csv'
open(url) do |file|
FasterCSV.parse(file.read) do |row|
# Your code here
end
end