Clojure building of URL from constituent parts - url

In Python I would do the following:
>>> q = urllib.urlencode({"q": "clojure url"})
>>> q
'q=clojure+url'
>>> url = "http://stackoverflow.com/search?" + q
>>> url
'http://stackoverflow.com/search?q=clojure+url'
How do I do all the encoding that's done for me above in Clojure? In other words, how do I do something akin to the following:
=> (build-url "http://stackoverflow.com/search" {"q" "clojure url"})
"http://stackoverflow.com/search?q=clojure+url"

There is a url-encode function in Ring's ring.util.codec namespace:
(ring.util.codec/url-encode "clojure url")
; => "clojure+url"
I'm not sure if there's a prepackaged function to turn a map into a query string, but perhaps this could do the job:
(use '[ring.util.codec :only [url-encode]])
(defn make-query-string [m]
(->> (for [[k v] m]
(str (url-encode k) "=" (url-encode v)))
(interpose "&")
(apply str)))
An example:
user> (make-query-string {"q" "clojure url" "foo" "bar"})
"q=clojure+url&foo=bar"
All that remains is concatenating the result onto the end of a URL:
(defn build-url [url-base query-map]
(str url-base "?" (make-query-string query-map)))
Seems to work:
user> (build-url "http://stackoverflow.com/search" {"q" "clojure url"})
"http://stackoverflow.com/search?q=clojure+url"
Update:
Perhaps a modified version might make for a more Clojure-friendly experience. Also handles encoding via a Ring-style optional argument with utf-8 as default.
(defn make-query-string [m & [encoding]]
(let [s #(if (instance? clojure.lang.Named %) (name %) %)
enc (or encoding "UTF-8")]
(->> (for [[k v] m]
(str (url-encode (s k) enc)
"="
(url-encode (str v) enc)))
(interpose "&")
(apply str))))
(defn build-url [url-base query-map & [encoding]]
(str url-base "?" (make-query-string query-map encoding)))
So now we can do
user> (build-url "http://example.com/q" {:foo 1})
"http://example.com/q?foo=1"

Here's one way:
user=> (import [java.net URLEncoder])
java.net.URLEncoder
user=> (str "http://stackoverflow.com/search?q=" (URLEncoder/encode "clojure url" "UTF-8"))
"http://stackoverflow.com/search?q=clojure+url"
I know this is not exactly the same as your Python snippet though. Please see the following post from the Clojure mailing list for a more complete answer:
http://www.mail-archive.com/clojure#googlegroups.com/msg29338.html
The code from there will allow you to do this:
user=> (encode-params {"q" "clojure url"})
"q=clojure+url"

This is the exact REPL equivalent of your python session, using clj-http.
user=> (require ['clj-http.client :as 'client])
nil
user=> (str "http://stackoverflow.com/search?"
user=* (client/generate-query-string {"q" "clojure url"}))
"http://stackoverflow.com/search?q=clojure+url"
but clj-http makes it even easier:
user=> (client/get "http://stackoverflow.com/search?"
user=* {:query-params {"q" "clojure url"}})
... <a lot of output, omitted to protect the innocent>...
assuming that you want to perform a GET request, that is.

clj-apache-http is pretty useful. With it you can do the following:
user=> (require ['com.twinql.clojure.http :as 'http])
nil
user=> (def q (http/encode-query {"q" "clojure url"}))
#'user/q
user=> (def url (str "http://stackoverflow.com/search?" q))
#'user/url
user=> url
"http://stackoverflow.com/search?q=clojure+url"

For anyone still looking for this in 2021 - ring.util.codec/form-encode does this perfectly:
http://ring-clojure.github.io/ring-codec/ring.util.codec.html#var-form-encode

A simple solution using str(ings):
(def q (str "clojure+url"))
(def url (str "http://stackoverflow.com/search?" q))

Related

Which approach is better in terms of performance to modify a hash, is it `Hash#merge` or double splat operator?

From the below example, which approach is better in terms of performance?
h = {a: 1, b: 2}
{**h, c: 3} => {:a=>1, :b=>2, :c=>3}
# or
h.merge(c: 3) => {:a=>1, :b=>2, :c=>3}
Basic benchmarking
require 'benchmark/ips'
Benchmark.ips do |x|
x.config(:time => 10, :warmup => 2)
h = {a: 1, b: 2}
x.report("splat") {{**h, c: 3}}
x.report("merge") {h.merge(c: 3)}
x.compare!
end
suggests that merge is faster, for example
Warming up --------------------------------------
splat 243.017k i/100ms
merge 315.349k i/100ms
Calculating -------------------------------------
splat 3.388M (±11.8%) i/s - 33.293M in 10.005951s
merge 4.721M (±12.5%) i/s - 46.356M in 10.037133s
Comparison:
merge: 4720869.7 i/s
splat: 3388413.3 i/s - 1.39x (± 0.00) slower

How to Parse with Commas in CSV file in Ruby

I am parsing the CSV file with Ruby and am having trouble in that the delimiter is a comma my data contains commas.
In portions of the data that contain commas the data is surrounded by "" but I am not sure how to make CSV ignore commas that are contained within Quotations.
Example CSV Data (File.csv)
NCB 14591 BLK 13 LOT W IRR," 84.07 FT OF 25, ALL OF 26,",TWENTY-THREE SAC HOLDING COR
Example Code:
require 'csv'
CSV.foreach("File.csv", encoding:'iso-8859-1:utf-8', :quote_char => "\x00").each do |x|
puts x[1]
end
Current Output: " 84.07 FT OF 25
Expected Output: 84.07 FT OF 25, ALL OF 26,
Link to the gist to view the example file and code.
https://gist.github.com/markscoin/0d6c2d346d70fd627203317c5fe3097c
Try with force_quotes option:
require 'csv'
CSV.foreach("data.csv", encoding:'iso-8859-1:utf-8', quote_char: '"', force_quotes: true).each do |x|
puts x[1]
end
Result:
84.07 FT OF 25, ALL OF 26,
The illegal quoting error is when a line has quotes, but they don't wrap the entire column, so for instance if you had a CSV that looks like:
NCB 14591 BLK 13 LOT W IRR," 84.07 FT OF 25, ALL OF 26,",TWENTY-THREE SAC HOLDING COR
NCB 14592 BLK 14 LOT W IRR,84.07 FT OF "25",TWENTY-FOUR SAC HOLDING COR
You could parse each line individually and change the quote character only for the lines that use bad quoting:
require 'csv'
def parse_file(file_name)
File.foreach(file_name) do |line|
parse_line(line) do |x|
puts x.inspect
end
end
end
def parse_line(line)
options = { encoding:'iso-8859-1:utf-8' }
begin
yield CSV.parse_line(line, options)
rescue CSV::MalformedCSVError
# this line is misusing quotes, change the quote character and try again
options.merge! quote_char: "\x00"
retry
end
end
parse_file('./File.csv')
and running this gives you:
["NCB 14591 BLK 13 LOT W IRR", " 84.07 FT OF 25, ALL OF 26,", "TWENTY-THREE SAC HOLDING COR"]
["NCB 14592 BLK 14 LOT W IRR", "84.07 FT OF \"25\"", "TWENTY-FOUR SAC HOLDING COR"]
but then if you have a mix of bad quoting and good quoting in a single row this falls apart again. Ideally you just want to clean up the CSV to be valid.

How to quickly join two strings in Ruby

It's common to need to join strings, and in Ruby we have common ways of doing it: appending, concatenating and interpolating one into the other, or using the built-in concat method in String. (We have multiple ways of doing it for flexibility and to ease the transition from other languages to Ruby.)
Starting with:
'a ' << 'z' # => "a z"
'a '.concat('z') # => "a z"
'a ' + 'z' # => "a z"
"a #{'z'}" # => "a z"
Assuming we don't want to change either string and that the strings won't be assigned to variables, what is the fastest way to join them, and does the fastest way change as the size of the "left" string grows?
For those who can't figure out why we'd post questions like this, it's to help educate and show the most efficient way to do a particular task. Newcomers to a language, Ruby in this case, often drag old ways of doing something with them, and inadvertently write code that runs more slowly than necessary. A simple change to their coding style can result in faster code.
Starting with short strings:
z = 'z'
'a ' << z # => "a z"
'a '.concat(z) # => "a z"
'a ' + z # => "a z"
"a #{z}" # => "a z"
require 'fruity'
compare do
append { 'a ' << z}
concat { 'a '.concat(z)}
plus { 'a ' + z}
interpolate { "a #{z}" }
end
# >> Running each test 65536 times. Test will take about 2 seconds.
# >> interpolate is similar to append
# >> append is similar to plus
# >> plus is faster than concat by 2x ± 0.1
Increasing the "left" string to 11 characters:
require 'fruity'
compare do
append { 'abcdefghij ' << z}
concat { 'abcdefghij '.concat(z)}
plus { 'abcdefghij ' + z}
interpolate { "abcdefghij #{z}" }
end
# >> Running each test 65536 times. Test will take about 2 seconds.
# >> interpolate is similar to append
# >> append is similar to plus
# >> plus is faster than concat by 2x ± 1.0
51 characters:
compare do
append { 'abcdefghijabcdefghijabcdefghijabcdefghijabcdefghij ' << z}
concat { 'abcdefghijabcdefghijabcdefghijabcdefghijabcdefghij '.concat(z)}
plus { 'abcdefghijabcdefghijabcdefghijabcdefghijabcdefghij ' + z}
interpolate { "abcdefghijabcdefghijabcdefghijabcdefghijabcdefghij #{z}" }
end
# >> Running each test 32768 times. Test will take about 2 seconds.
# >> plus is faster than append by 2x ± 1.0
# >> append is similar to interpolate
# >> interpolate is similar to concat
101:
compare do
append { 'abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij ' << z}
concat { 'abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij '.concat(z)}
plus { 'abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij ' + z}
interpolate { "abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij #{z}" }
end
# >> Running each test 32768 times. Test will take about 2 seconds.
# >> plus is faster than interpolate by 2x ± 0.1
# >> interpolate is similar to append
# >> append is similar to concat
501:
compare do
append { 'abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij ' << z}
concat { 'abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij '.concat(z)}
plus { 'abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij ' + z}
interpolate { "abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij #{z}" }
end
# >> Running each test 16384 times. Test will take about 1 second.
# >> plus is faster than append by 2x ± 0.1
# >> append is similar to interpolate
# >> interpolate is similar to concat
Once the strings got past 50 characters + consistently outperformed the others.
In the comments there are mention of some of these mutating the string on the left. Here's what would happen if it was a variable on the left, not a literal string:
a = 'a'
z = 'z'
a << z # => "az"
a # => "az"
a = 'a'
a.concat(z) # => "az"
a # => "az"
compared to:
a + z # => "az"
a # => "a"
"#{a} #{z}" # => "a z"
a # => "a"
Note: The initial version of answer had a bad test using:
"a #{'z'}"
The problem with that is Ruby is smart enough to recognize that 'z' is another literal and converts the string into:
"a z"
with the end result that the test would be unfairly faster than the others.
It's been a while and Ruby's smarter and faster. I added a couple additional tests:
puts "Running Ruby v%s" % RUBY_VERSION
require 'fruity'
z_ = 'z'
compare do
append { 'abcdefghij ' << 'z' }
concat { 'abcdefghij '.concat('z') }
plus { 'abcdefghij ' + 'z' }
interpolate1 { "abcdefghij #{'z'}" }
interpolate2 { "abcdefghij #{z_}" }
adjacent { 'abcdefghij' ' z' }
end
# >> Running Ruby v2.7.0
# >> Running each test 65536 times. Test will take about 3 seconds.
# >> adjacent is similar to interpolate1
# >> interpolate1 is faster than interpolate2 by 2x ± 1.0
# >> interpolate2 is similar to append
# >> append is similar to concat
# >> concat is similar to plus
interpolate1 and adjacent are basically the same as far as the interpreter is concerned and will be concatenated prior to running. interpolate2 forces Ruby to do it at run-time so it's a little slower.

How do I sign a POST request with a body using Clojure and clj-oauth?

I'm trying to communicate with the quickbooks online REST API, which uses OAuth 1:
I can construct arbitrary GET requests like this:
(require '[oauth.client :as oauth])
(require '[cheshire.core :refer :all])
(require '[clj-http.client :as client])
(def OAuth-Consumer-Key "qyprdtoUhI8AjLxEQ9tucTJkSiklKn")
(def OAuth-Consumer-Secret "*********************")
(def OAuth-Access-Token "qyprdaXOWBph7MUnkqzRVruovEMlvgUH52Gup8kfinSgbnJL")
(def OAuth-Access-Token-Secret "********************************" )
(def consumer (oauth/make-consumer OAuth-Consumer-Key
OAuth-Consumer-Secret
""
""
""
:hmac-sha1))
(def get-url "https://sandbox-quickbooks.api.intuit.com/v3/company/123145835981692/account/4")
(def get-user-params {})
(def get-credentials (oauth/credentials consumer
OAuth-Access-Token
OAuth-Access-Token-Secret
:GET
get-url
get-user-params))
(client/get get-url {:query-params (merge get-credentials get-user-params) })
But I can't construct the POST requests. If I put the body in the user-params, then it seems that the authentication works, but the API refuses to accept the request, but if I don't, it doesn't authenticate in the first place.
(def post-body "<Customer xmlns=\"http://schema.intuit.com/finance/v3\" domain=\"QBO\" sparse=\"false\">\n<CompanyName>Best Company</CompanyName>\n<DisplayName>Sir Jonhn Doe</DisplayName>\n<BillAddr>\n<Line1>123 Main Street</Line1>\n<City>Mountain View</City>\n<Country>USA</Country>\n<CountrySubDivisionCode>CA</CountrySubDivisionCode>\n<PostalCode>94042</PostalCode>\n</BillAddr>\n</Customer>")
(def post-user-params {:content-type "application/xml" :body post-body})
(def post-credentials (oauth/credentials consumer
OAuth-Access-Token
OAuth-Access-Token-Secret
:POST
"https://sandbox-quickbooks.api.intuit.com/v3/company/123145835981692/customer"
post-user-params))
(client/post "https://sandbox-quickbooks.api.intuit.com/v3/company/123145835981692/customer"
{:query-params (merge post-user-params post-credentials)
:body post-body})
This hand constructed version works fine though. (done with the Postman extension for Chrome).
(client/post "https://sandbox-quickbooks.api.intuit.com/v3/company/123145835981692/customer"
{:headers {:authorization "OAuth oauth_consumer_key=\"qyprdtoUhI8AjLxEQ9tucTJkSiklKn\",oauth_token=\"qyprdaXOWBph7MUnkqzRVruovEMlvgUH52Gup8kfinSgbnJL\",oauth_signature_method=\"HMAC-SHA1\",oauth_timestamp=\"1470828646\",oauth_nonce=\"WXZu67\",oauth_version=\"1.0\",oauth_signature=\"a66RFI8clxNIhv8M1YzzijDgE1A%3D\""
:content-type "application/xml",
:cache-control "no-cache",
:postman-token "0effd852-8271-7f66-f43d-6710443c5107"}
:body post-body})
Can anyone see what I should be doing to sign the request properly?
Eventually I found that this incantation works. It appears that you don't actually need to sign the body. And the crucial change was to move the content-type into the post request and out of the signed bit.
I have no idea what is going on here!
(def post-url "https://sandbox-quickbooks.api.intuit.com/v3/company/123145835981692/customer")
(def post-body (str "<Customer xmlns=\"http://schema.intuit.com/finance/v3\" domain=\"QBO\" sparse=\"false\">\n"
"<CompanyName>Company</CompanyName>\n"
"<DisplayName>Sir Jeaweaaan Doe</DisplayName>\n"
"<BillAddr>\n"
"<Line1>123 Main Street</Line1>\n"
"<City>Mountain View</City>\n"
"<Country>USA</Country>\n"
"<CountrySubDivisionCode>CA</CountrySubDivisionCode>\n"
"<PostalCode>94042</PostalCode>\n"
"</BillAddr>\n"
"</Customer>"))
(def post-user-params {
:cache-control "no-cache"
})
(def post-credentials (oauth/credentials consumer
OAuth-Access-Token
OAuth-Access-Token-Secret
:POST
post-url
post-user-params))
(que? (client/post post-url
{:query-params (merge post-user-params post-credentials)
:content-type "application/xml"
:body post-body
}))

URL substring matching regular expression?

Suppose I only want user to type in url starts with http://www.google.com
What is the regular expression for this?
Thanks!
Just get the substring from 0 to the length of http://www.google.com and you're done.
Rather than use a regex, you might want to consider using the URI library that comes with Ruby. It's made to take apart and build URLs, is well tested, and less error-prone than trying to reinvent the same functionality.
require 'uri'
url = URI.parse('http://www.google.com/path/to/page.html?a=1&b=2')
url.scheme # => "http"
url.host # => "www.google.com"
url.path # => "/path/to/page.html"
url.query # => "a=1&b=2"
If that's not good enough, the Addressable::URI gem is even more capable.
Try this:
/\Ahttp:\/\/www\.google\.com(.*)?\Z/
ruby-1.9.2-p0 > "http://www.google.com" =~ /\Ahttp:\/\/www\.google\.com(.*)?\Z/
=> 0
ruby-1.9.2-p0 > "http://www.google.com/foobar" =~ /\Ahttp:\/\/www\.google\.com(.*)?\Z/
=> 0
ruby-1.9.2-p0 > $1
=> "/foobar"
Rails has a convenient start_with? method for this. If it's just a static string, no regular expression is needed.
url.start_with?("http://www.google.com")

Resources