Different output from encodeURIComponent vs URLSearchParams - url

I built an oauth2 url with query params using URLSearchParmas API. However, the output URL didn't return an expected url. Can anyone help me understand the difference between those two APIs and how can I get the same result as the output of encodeURIComponent, using URLSearchParams? Thanks!
const expected = encodeURIComponent('code id_token'); // code%20id_token
const search = new URLSearchParams();
search.set('response_type', 'code id_token');
search.toString(); // code+id_token

According to WHATWG, URLSearchParams uses application/x-www-form-urlencoded format. While it's suitable for decoding URL queries, for encoding it can lead to unexpected results such as spaces being encoded as + and extra characters such as ~ being percent-encoded. It's better to use encodeURIComponent instead:
Having an object:
const params = {
text1: 'aaa bbb',
text2: '-._*~()'
}
Instead of:
url.search = (new URLSearchParams(params)).toString()
Use:
url.search = Object.entries(params)
.map(([key, value]) => `${encodeURIComponent(key)}=${encodeURIComponent(value)}`)
.join('&')
Also, according to MDN even encodeURIComponent doesn't conform to newer RFC 3986 which defines more characters to escape, for example *. While it's probably safe not to escape these additional characters if you aren't using them as field separators, if you want to be strictly conformant to latest RFC, use this updated implementation from MDN:
function fixedEncodeURIComponent(str) {
return encodeURIComponent(str).replace(/[!'()*]/g, function(c) {
return '%' + c.charCodeAt(0).toString(16).toUpperCase();
});
}
A playground for experimenting:
const params = {
text1: 'aaa bbb',
text2: '-._*~()'
}
const url1 = new URL('http://example.com')
const search1 = new URLSearchParams(params)
url1.search = search1 // Incorrect
console.log('URLSearchParams', url1.toString())
function fixedEncodeURIComponent(str) {
return encodeURIComponent(str).replace(/[!'()*]/g, function(c) {
return '%' + c.charCodeAt(0).toString(16).toUpperCase()
})
}
const url2 = new URL('http://example.com')
const search2 = Object
.entries(params)
.map(([key, value]) => `${fixedEncodeURIComponent(key)}=${fixedEncodeURIComponent(value)}`)
.join('&')
url2.search = search2 // Correct
console.log('fixedEncodeURIComponent', url2.toString())

Related

Yahoo Finance URL not working

I have been using the following URL to fetch historical data from yahoo finance for quite some time now but it stopped working as of yesterday.
https://ichart.finance.yahoo.com/table.csv?s=SPY
When browsing to this site it says:
Will be right back...
Thank you for your patience.
Our engineers are working quickly to resolve the issue.
However, since this issue is still existing since yesterday I am starting to think that they discontinued this service?
My SO search only pointed me to this topic, which was related to https though...
Is anyone else experiencing this issue?
How can I resolve this problem? Do they offer a different access to their historical data?
Yahoo has gone to a Reactjs front end which means if you analyze the request headers from the client to the backend you can get the actual JSON they use to populate the client side stores.
Hosts:
query1.finance.yahoo.com HTTP/1.0
query2.finance.yahoo.com HTTP/1.1
(difference between HTTP/1.0 & HTTP/1.1)
If you plan to use a proxy or persistent connections use query2.finance.yahoo.com. But for the purposes of this post, the host used for the example URLs is not meant to imply anything about the path it's being used with.
Fundamental Data
(substitute your symbol for: AAPL)
/v10/finance/quoteSummary/AAPL?modules=
Inputs for the ?modules= query:
[
'assetProfile',
'summaryProfile',
'summaryDetail',
'esgScores',
'price',
'incomeStatementHistory',
'incomeStatementHistoryQuarterly',
'balanceSheetHistory',
'balanceSheetHistoryQuarterly',
'cashflowStatementHistory',
'cashflowStatementHistoryQuarterly',
'defaultKeyStatistics',
'financialData',
'calendarEvents',
'secFilings',
'recommendationTrend',
'upgradeDowngradeHistory',
'institutionOwnership',
'fundOwnership',
'majorDirectHolders',
'majorHoldersBreakdown',
'insiderTransactions',
'insiderHolders',
'netSharePurchaseActivity',
'earnings',
'earningsHistory',
'earningsTrend',
'industryTrend',
'indexTrend',
'sectorTrend']
Example URL: querying for all of the above modules
https://query2.finance.yahoo.com/v10/finance/quoteSummary/AAPL?modules=assetProfile%2CsummaryProfile%2CsummaryDetail%2CesgScores%2Cprice%2CincomeStatementHistory%2CincomeStatementHistoryQuarterly%2CbalanceSheetHistory%2CbalanceSheetHistoryQuarterly%2CcashflowStatementHistory%2CcashflowStatementHistoryQuarterly%2CdefaultKeyStatistics%2CfinancialData%2CcalendarEvents%2CsecFilings%2CrecommendationTrend%2CupgradeDowngradeHistory%2CinstitutionOwnership%2CfundOwnership%2CmajorDirectHolders%2CmajorHoldersBreakdown%2CinsiderTransactions%2CinsiderHolders%2CnetSharePurchaseActivity%2Cearnings%2CearningsHistory%2CearningsTrend%2CindustryTrend%2CindexTrend%2CsectorTrend
The %2C is the Hex representation of , and needs to be inserted between each module you request. details about the hex encoding bit(if you care)
Options contracts
/v7/finance/options/AAPL (current expiration)
/v7/finance/options/AAPL?date=1679011200 (March 17, 2023 expiration)
Example URL:
https://query2.finance.yahoo.com/v7/finance/options/AAPL (current expiration)
https://query2.finance.yahoo.com/v7/finance/options/AAPL?date=1679011200 (Match 17, 2023 expiration)
Any valid future expiration represented as a UNIX timestamp can be used in the ?date= query. If you query for the current expiration the JSON response will contain a list of all the valid expirations that can be used in the ?date= query. (here is a post explaining converting human-readable dates to UNIX timestamp in Python)
Price
/v8/finance/chart/AAPL?symbol=AAPL&period1=0&period2=9999999999&interval=3mo
Possible inputs for &interval=: 1m, 5m, 15m, 30m, 90m, 1h, 1d, 5d, 1wk, 1mo, 3mo
m (minute) intervals are limited to 30days with period1 and period2 spaning a maximum of 7 days per/request. Exceeding either of these limits will result in an error and will not round
h (hour) interval is limited to 730days with no limit to span. Exceeding this will result in an error and will not round
period1=: UNIX timestamp representation of the date you wish to start at.
d (day), wk (week), mo (month) intervals with values less than the initial trading date will be rounded up to the initial trading date.
period2=: UNIX timestamp representation of the date you wish to end at.
For all intervals: values greater than the last trading date will be rounded down to the most recent timestamp available.
Add pre & post market data
&includePrePost=true
Add dividends & splits
&events=div%7Csplit
%7C is hex for |. , will work but internally yahoo uses pipe
Example URL:
https://query1.finance.yahoo.com/v8/finance/chart/AAPL?symbol=AAPL&period1=0&period2=9999999999&interval=1d&includePrePost=true&events=div%7Csplit
The above request will return all price data for ticker AAPL on a 1-day interval including pre and post-market data as well as dividends and splits.
Note: the values used in the price example URL for period1= & period2= are to demonstrate the respective rounding behavior of each input.`
It looks like they have started adding a required cookie, but you can retrieve this fairly easily, for example:
GET https://uk.finance.yahoo.com/quote/AAPL/history
Responds with the header in the form:
set-cookie:B=xxxxxxxx&b=3&s=qf; expires=Fri, 18-May-2018 00:00:00 GMT; path=/; domain=.yahoo.com
You should be able to read this and attach it to your .csv request:
GET https://query1.finance.yahoo.com/v7/finance/download/AAPL?period1=1492524105&period2=1495116105&interval=1d&events=history&crumb=tO1hNZoUQeQ
cookie: B=xxxxxxxx&b=3&s=qf;
Note the crumb query parameter, this seems to correspond to your cookie in some way. Your best bet is to scrape this from the HTML response to your initial GET request. Within that response, you can do a regex search for: "CrumbStore":\{"crumb":"(?<crumb>[^"]+)"\} and extract the crumb matched group.
It looks like once you have that crumb value though you can use it with the same cookie on any symbol/ticker for the next year meaning you shouldn't have to do the scrape too frequently.
To get current quotes just load:
https://query1.finance.yahoo.com/v8/finance/chart/AAPL?interval=2m
With:
AAPL substituted with your stock ticker
interval one of [1m, 2m, 5m, 15m, 30m, 60m, 90m, 1h, 1d, 5d, 1wk, 1mo, 3mo]
optional period1 query param with your epoch range start date e.g. period1=1510340760
optional period2 query param with your epoch range end date e.g. period2=1510663712
I managed to work out a .NET class to obtain valid token (cookie and crumb) from Yahoo Finance
For complete API library in fetching historical data from new Yahoo Finance, you may visit YahooFinanceAPI in Github
Here is the class to grab the cookie and crumb
Token.cs
using System;
using System.Diagnostics;
using System.Net;
using System.IO;
using System.Text.RegularExpressions;
namespace YahooFinanceAPI
{
/// <summary>
/// Class for fetching token (cookie and crumb) from Yahoo Finance
/// Copyright Dennis Lee
/// 19 May 2017
///
/// </summary>
public class Token
{
public static string Cookie { get; set; }
public static string Crumb { get; set; }
private static Regex regex_crumb;
/// <summary>
/// Refresh cookie and crumb value Yahoo Fianance
/// </summary>
/// <param name="symbol">Stock ticker symbol</param>
/// <returns></returns>
public static bool Refresh(string symbol = "SPY")
{
try
{
Token.Cookie = "";
Token.Crumb = "";
string url_scrape = "https://finance.yahoo.com/quote/{0}?p={0}";
//url_scrape = "https://finance.yahoo.com/quote/{0}/history"
string url = string.Format(url_scrape, symbol);
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
request.CookieContainer = new CookieContainer();
request.Method = "GET";
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
string cookie = response.GetResponseHeader("Set-Cookie").Split(';')[0];
string html = "";
using (Stream stream = response.GetResponseStream())
{
html = new StreamReader(stream).ReadToEnd();
}
if (html.Length < 5000)
return false;
string crumb = getCrumb(html);
html = "";
if (crumb != null)
{
Token.Cookie = cookie;
Token.Crumb = crumb;
Debug.Print("Crumb: '{0}', Cookie: '{1}'", crumb, cookie);
return true;
}
}
}
catch (Exception ex)
{
Debug.Print(ex.Message);
}
return false;
}
/// <summary>
/// Get crumb value from HTML
/// </summary>
/// <param name="html">HTML code</param>
/// <returns></returns>
private static string getCrumb(string html)
{
string crumb = null;
try
{
//initialize on first time use
if (regex_crumb == null)
regex_crumb = new Regex("CrumbStore\":{\"crumb\":\"(?<crumb>.+?)\"}",
RegexOptions.CultureInvariant | RegexOptions.Compiled, TimeSpan.FromSeconds(5));
MatchCollection matches = regex_crumb.Matches(html);
if (matches.Count > 0)
{
crumb = matches[0].Groups["crumb"].Value;
}
else
{
Debug.Print("Regex no match");
}
//prevent regex memory leak
matches = null;
}
catch (Exception ex)
{
Debug.Print(ex.Message);
}
GC.Collect();
return crumb;
}
}
}
Updated 1 Jun 17
credits to #Ed0906
modify crumb regex pattern to Regex("CrumbStore\":{\"crumb\":\"(?<crumb>.+?)\"}"
For the python lovers out there, I've updated the yahooFinance.py in tradingWithPython library.
There is also an example notebook based on the tips by Ed0906, demonstrating how to get the data step by step. See it on
In this forum: https://forums.yahoo.net/t5/Yahoo-Finance-help/Is-Yahoo-Finance-API-broken/td-p/250503/page/3
Nixon said:
Hi All - This feature was discontinued by the Finance team and they will not be reintroducing that functionality.
The URL for downloading historical data is now something like this:
https://query1.finance.yahoo.com/v7/finance/download/SPY?period1=1492449771&period2=1495041771&interval=1d&events=history&crumb=9GaimFhz.WU
Note the above URL will not work for you or anyone else. You'll get something like this:
{
"finance": {
"error": {
"code": "Unauthorized",
"description": "Invalid cookie"
}
}
}
It seems that Yahoo is now using some hashing to prevent people from accessing the data like you did. The URL varies with each session so it's very likely that you can't do this with a fixed URL anymore.
You'll need to do some scrapping to get the correct URL from the main page, for example:
https://finance.yahoo.com/quote/SPY/history?p=SPY
I had found another yahoo site that does not require cookies, but generates jason output: https://query1.finance.yahoo.com/v7/finance/chart/YHOO?range=2y&interval=1d&indicators=quote&includeTimestamps=true
it was pointed out from here: https://www.stock-data-solutions.com/kb/how-to-load-historical-prices-from-yahoo-finance-to-excel.htm
As it turned out they seem to support 'perod1' and 'period2' (in unix time) parameters which could be used instead of the 'interval'.
String quoteSite = "https://query1.finance.yahoo.com/v7/finance/chart/"
+ symbolName + "?"
+ "period1=" + period1
+ "&period2=" + period2
+ "&interval=1d&indicators=quote&includeTimestamps=true";
And the following parses Jason for me:
JSONObject topObj = new JSONObject(inp);
Object error = topObj.getJSONObject("chart").get("error");
if (!error.toString().equals("null")) {
System.err.prinltn(error.toString());
return null;
}
JSONArray results = topObj.getJSONObject("chart").getJSONArray("result");
if (results == null || results.length() != 1) {
return null;
}
JSONObject result = results.getJSONObject(0);
JSONArray timestamps = result.getJSONArray("timestamp");
JSONObject indicators = result.getJSONObject("indicators");
JSONArray quotes = indicators.getJSONArray("quote");
if (quotes == null || quotes.length() != 1) {
return null;
}
JSONObject quote = quotes.getJSONObject(0);
JSONArray adjcloses = indicators.getJSONArray("adjclose");
if (adjcloses == null || adjcloses.length() != 1) {
return null;
}
JSONArray adjclose = adjcloses.getJSONObject(0).getJSONArray("adjclose");
JSONArray open = quote.getJSONArray("open");
JSONArray close = quote.getJSONArray("close");
JSONArray high = quote.getJSONArray("high");
JSONArray low = quote.getJSONArray("low");
JSONArray volume = quote.getJSONArray("volume");
I'm in the same boat. Getting there slowly. The download link on the historical prices page still works. So I added the export cookies extension to firefox, logged in to yahoo, dumped the cookies. Used the crumb value from interactive session and I was able to retrieve values. Here's part of a test perl script that worked.
use Time::Local;
# create unix time variables for start and end date values: 1/1/2014 thru 12/31/2017
$p1= timelocal(0,0,0,1,0,114);
$p2= timelocal(0,0,0,31,11,117);
$symbol = 'AAPL';
# create variable for string to be executed as a system command
# cookies.txt exported from firefox
# crumb variable retrieved from yahoo download data link
$task = "wget --load-cookies cookies.txt --no-check-certificate -T 30 -O $symbol.csv \"https://query1.finance.yahoo.com/v7/finance/download/$symbol?period1=$p1&period2=$p2&interval=1d&events=history&crumb=7WhHVu5N4e3\" ";
#show what we're executing
print $task;
# execute system command using backticks
`$task`;
#output is AAPL.csv
It'll take a while to automate what I do. Hopefully yahoo will simplify or give some guidance on it if they really intend for people to use it.
Fully working PHP example, based on this post and related sources:
function readYahoo($symbol, $tsStart, $tsEnd) {
preg_match('"CrumbStore\":{\"crumb\":\"(?<crumb>.+?)\"}"',
file_get_contents('https://uk.finance.yahoo.com/quote/' . $symbol),
$crumb); // can contain \uXXXX chars
if (!isset($crumb['crumb'])) return 'Crumb not found.';
$crumb = json_decode('"' . $crumb['crumb'] . '"'); // \uXXXX to UTF-8
foreach ($http_response_header as $header) {
if (0 !== stripos($header, 'Set-Cookie: ')) continue;
$cookie = substr($header, 14, strpos($header, ';') - 14); // after 'B='
} // cookie looks like "fkjfom9cj65jo&b=3&s=sg"
if (!isset($cookie)) return 'Cookie not found.';
$fp = fopen('https://query1.finance.yahoo.com/v7/finance/download/' . $symbol
. '?period1=' . $tsStart . '&period2=' . $tsEnd . '&interval=1d'
. '&events=history&crumb=' . $crumb, 'rb', FALSE,
stream_context_create(array('http' => array('method' => 'GET',
'header' => 'Cookie: B=' . $cookie))));
if (FALSE === $fp) return 'Can not open data.';
$buffer = '';
while (!feof($fp)) $buffer .= implode(',', fgetcsv($fp, 5000)) . PHP_EOL;
fclose($fp);
return $buffer;
}
Usage:
$csv = readYahoo('AAPL', mktime(0, 0, 0, 6, 2, 2017), mktime(0, 0, 0, 6, 3, 2017));
Python
I used this code to get cookie (copied from fix-yahoo-finance):
def get_yahoo_crumb_cookie():
"""Get Yahoo crumb cookie value."""
res = requests.get('https://finance.yahoo.com/quote/SPY/history')
yahoo_cookie = res.cookies['B']
yahoo_crumb = None
pattern = re.compile('.*"CrumbStore":\{"crumb":"(?P<crumb>[^"]+)"\}')
for line in res.text.splitlines():
m = pattern.match(line)
if m is not None:
yahoo_crumb = m.groupdict()['crumb']
return yahoo_cookie, yahoo_crumb
then this code to get response:
cookie, crumb = get_yahoo_crumb_cookie()
params = {
'symbol': stock.symbol,
'period1': 0,
'period2': int(time.time()),
'interval': '1d',
'crumb': crumb,
}
url_price = 'https://query1.finance.yahoo.com/v7/finance/download/{symbol}'
response = requests.get(url_price, params=params, cookies={'B': cookie})
This looks nice as well http://blog.bradlucas.com/posts/2017-06-03-yahoo-finance-quote-download-python/
For java lovers.
You can access your cookies from a URLConnection this way.
// "https://finance.yahoo.com/quote/SPY";
URLConnection con = url.openConnection();
...
for (Map.Entry<String, List<String>> entry : con.getHeaderFields().entrySet()) {
if (entry.getKey() == null
|| !entry.getKey().equals("Set-Cookie"))
continue;
for (String s : entry.getValue()) {
// store your cookie
...
}
}
now you can search for the crumb in the yahoo site:
String crumb = null;
InputStream inStream = con.getInputStream();
InputStreamReader irdr = new InputStreamReader(inStream);
BufferedReader rsv = new BufferedReader(irdr);
Pattern crumbPattern = Pattern.compile(".*\"CrumbStore\":\\{\"crumb\":\"([^\"]+)\"\\}.*");
String line = null;
while (crumb == null && (line = rsv.readLine()) != null) {
Matcher matcher = crumbPattern.matcher(line);
if (matcher.matches())
crumb = matcher.group(1);
}
rsv.close();
and finally, setting the cookie
String quoteUrl = "https://query1.finance.yahoo.com/v7/finance/download/IBM?period1=1493425217&period2=1496017217&interval=1d&events=history&crumb="
+ crumb
...
List<String> cookies = cookieStore.get(key);
if (cookies != null) {
for (String c: cookies)
con.setRequestProperty("Cookie", c);
}
...
con.connect();
I used a php script using fopen() to access the financial data, here are the snippets that I modified to get it back to work:
Creating the timestamps for start date and end date:
$timestampStart = mktime(0,0,0,$startMonth,$startDay,$startYear);
$timestampEnd = mktime(0,0,0,$endMonth,$endDay,$endYear);
Force fopen() to send the required cookie with hard coded values:
$cookie="YourCookieTakenFromYahoo";
$opts = array(
'http'=>array(
'method'=>"GET",
'header'=>"Accept-language: en\r\n" .
"Cookie: B=".$cookie."\r\n"
)
);
$context = stream_context_create($opts);
Use fopen() to get the csv file:
$ticker="TickerSymbol";
$crumb="CrumbValueThatMatchesYourCookieFromYahoo";
$handle = fopen("https://query1.finance.yahoo.com/v7/finance/download/".$ticker."?period1=".$timestampStart."&period2=".$timestampEnd."&interval=1d&events=history&crumb=".$crumb."", "r", false, $context);
Now you can do all the magic you did before inside this while loop:
while (!feof($handle) ) {
$line_of_text = fgetcsv($handle, 5000);
}
Make sure to set your own values for $ticker, $crumb and $cookie in the snippets above.
Follow Ed0906's approach on how to retrieve $crumb and $cookie.
I am the author of this service
Basic info here
Daily prices
You need to be familiar with RESTFUL services.
https://quantprice.herokuapp.com/api/v1.1/scoop/day?tickers=MSFT&date=2017-06-09
Historical prices
You have to provide a date range :
https://quantprice.herokuapp.com/api/v1.1/scoop/period?tickers=MSFT&begin=2012-02-19&end=2012-02-20
If you don't provide begin or end it will use the earliest or current date:
https://quantprice.herokuapp.com/api/v1.1/scoop/period?tickers=MSFT&begin=2012-02-19
Multiple tickers
You can just comma separate tickers:
https://quantprice.herokuapp.com/api/v1.1/scoop/period?tickers=IBM,MSFT&begin=2012-02-19
Rate limit
All requests are rate limited to 10 requests per hour. If you want to register for a full access API send me DM on twitter. You will receive an API key to add to the URL.
We are setting up a paypal account for paid subscription without rates.
List of tickers available
https://github.com/robomotic/valueviz/blob/master/scoop_tickers.csv
I am working also to provide fundamental data and company data from EDGAR.
Cheers.
VBA
Here are some VBA functions that download and extract the cookie / crumb pair and return these in a Collection, and then use these to download the csv file contents for a particular code.
The containing project should have a reference to the 'Microsoft XML, v6.0' library added (other version might be fine too with some minor changes to the code).
Sub Test()
Dim X As Collection
Set X = FindCookieAndCrumb()
Debug.Print X!cookie
Debug.Print X!crumb
Debug.Print YahooRequest("AAPL", DateValue("31 Dec 2016"), DateValue("30 May 2017"), X)
End Sub
Function FindCookieAndCrumb() As Collection
' Tools - Reference : Microsoft XML, v6.0
Dim http As MSXML2.XMLHTTP60
Dim cookie As String
Dim crumb As String
Dim url As String
Dim Pos1 As Long
Dim X As String
Set FindCookieAndCrumb = New Collection
Set http = New MSXML2.ServerXMLHTTP60
url = "https://finance.yahoo.com/quote/MSFT/history"
http.Open "GET", url, False
' http.setProxy 2, "https=127.0.0.1:8888", ""
' http.setRequestHeader "Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
' http.setRequestHeader "Accept-Encoding", "gzip, deflate, sdch, br"
' http.setRequestHeader "Accept-Language", "en-ZA,en-GB;q=0.8,en-US;q=0.6,en;q=0.4"
http.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"
http.send
X = http.responseText
Pos1 = InStr(X, "CrumbStore")
X = Mid(X, Pos1, 44)
X = Mid(X, 23, 44)
Pos1 = InStr(X, """")
X = Left(X, Pos1 - 1)
FindCookieAndCrumb.Add X, "Crumb"
'======================================
X = http.getResponseHeader("set-cookie")
Pos1 = InStr(X, ";")
X = Left(X, Pos1 - 1)
FindCookieAndCrumb.Add X, "Cookie"
End Function
Function YahooRequest(ShareCode As String, StartDate As Date, EndDate As Date, CookieAndCrumb As Collection) As String
' Tools - Reference : Microsoft XML, v6.0
Dim http As MSXML2.XMLHTTP60
Dim cookie As String
Dim crumb As String
Dim url As String
Dim UnixStartDate As Long
Dim UnixEndDate As Long
Dim BaseDate As Date
Set http = New MSXML2.ServerXMLHTTP60
cookie = CookieAndCrumb!cookie
crumb = CookieAndCrumb!crumb
BaseDate = DateValue("1 Jan 1970")
If StartDate = 0 Then StartDate = BaseDate
UnixStartDate = (StartDate - BaseDate) * 86400
UnixEndDate = (EndDate - BaseDate) * 86400
url = "https://query1.finance.yahoo.com/v7/finance/download/" & ShareCode & "?period1=" & UnixStartDate & "&period2=" & UnixEndDate & "&interval=1d&events=history&crumb=" & crumb
http.Open "GET", url, False
http.setRequestHeader "Cookie", cookie
http.send
YahooRequest = http.responseText
End Function
For those Excel/VBA users I have used the suggestions above to develop a VBA method to extract historical prices from the updated Yahoo website. The key code snippets are listed below and I have also provided my testing workbook.
First a request to get the Crumb and Cookie values set before attempting to extract the data from Yahoo for the prices..
Dim strUrl As String: strUrl = "https://finance.yahoo.com/lookup?s=%7B0%7D" 'Symbol lookup used to set the values
Dim objRequest As WinHTTP.WinHttpRequest
Set objRequest = New WinHttp.WinHttpRequest
With objRequest
.Open "GET", strUrl, True
.setRequestHeader "Content-Type", "application/x-www-form-urlencoded; charset=UTF-8"
.send
.waitForResponse
strCrumb = strExtractCrumb(.responseText)
strCookie = Split(.getResponseHeader("Set-Cookie"), ";")(0)
End With
See the following Yahoo Historical Price Extract link to my website for a sample file and more details on the method I have used to extract historical security prices from the Yahoo website
I was on the same boat. I managed to get the CSV downloaded from Yahoo with some vb.net frankencode I made from bits and pieces off Google, SOF and some head-scratching.
However, I discovered Intrinio (look it up), signed up, and my free account gets me 500 historic data api calls a day, with much more data and much more accurate than Yahoo. I rewrote my code for the Intrinio API, and I'm happy as a clam.
BTW, I don't work or have anything to do with Intrinio, but they saved my butt big time...
You actually don't need to do 2 requests to get Yahoo data. I use this link https://ca.finance.yahoo.com/quote/AAAP/history?period1=1474000669&period2=1505536669&interval=1d&filter=history&frequency=1d
You could grab the cookie from the this but instead it includes that data for you historical quote in Json format. After I download the page I scarpe the Json data out of it. Saves a url request.
Javascript
Find cookie;
match = document.cookie.match(new RegExp('B=([^;]+)'));
alert (match[1]);
Find crumb;
i=document.body.innerHTML.search("CrumbStore")
if (i>=0) alert (document.body.innerHTML.substr(i+22,11))
Find crumb for mobile;
i=document.body.innerHTML.search('USER={\"crumb\":');
if (i>=0) alert(document.body.innerHTML.substr(i+15,11));
and it's probably best to wait for the page (e.g https://finance.yahoo.com/quote/goog) to load up first, you can
check it with;
document.readyState
An alternative approach to those mentioned so far (Yahoo, Google and Intrinio) is to get the historical data from Alpha Vantage for free. Their web service delivers intra-day, daily, adjusted stock prices and 50+ technical indicators. They even deliver straight to Excel - also for free - through Deriscope. (I am the author of the latter.)
If you are trying to connect yahooFinance api with java. just add the following dependency.
<dependency>
<groupId>com.yahoofinance-api</groupId>
<artifactId>YahooFinanceAPI</artifactId>
<version>3.13.0</version>
</dependency>
For Python 3 users change to
url='https://query1.finance.yahoo.com/v7/finance/download/AAAP?period1=1494605670&period2=1495815270&interval=1d&events=history&crumb=IJ.ilcJlkrZ'
from
url='https://chartapi.finance.yahoo.com/instrument/1.0/AAAP/chartdata;type=quote;range=10d/csv/'
and
response = request.urlopen(url)
to
response = requests.get(url,cookies={'B':cookie})
data in response.text
the data format is totally different but at least its working fine for now
There is a fix that I have found to work well. Please see my post:
Yahoo Finance API / URL not working: Python fix for Pandas DataReader where I followed the steps in https://pypi.python.org/pypi/fix-yahoo-finance to: $ pip install fix_yahoo_finance --upgrade --no-cache-dir (and also upgraded pandas_datareader to be sure) and tested ok:
from pandas_datareader import data as pdr
import fix_yahoo_finance
data = pdr.get_data_yahoo('BHP.AX', start='2017-04-23', end='2017-05-24')
Also note that the order of the last 2 data columns are 'Adj Close' and 'Volume' so for my purpose, I have reset the columns to the original order:
cols = ['Date', 'Open', 'High', 'Low', 'Close', 'Volume', 'Adj Close']
data = data.reindex(columns=cols)
I've combined some of the above ideas that handles the crumb / cookie refresh, specifically from #Dennis, and created a vb.net class that can be called like this:
Dim f = Await YahooFinanceFactory.CreateAsync
Dim items1 = Await f.GetHistoricalDataAsync("SPY", #1/1/2018#)
Dim items2 = Await f.GetHistoricalDataAsync("^FTSE", #1/1/2018#)
The class itself is here:
Imports System.Net
Imports System.Net.Http
Imports System.Text.RegularExpressions
Namespace YahooFinance
Public Class YahooHistoryPrice
Public Property [Date] As DateTime
Public Property Open As Double
Public Property High As Double
Public Property Low As Double
Public Property Close As Double
Public Property Volume As Double
Public Property AdjClose As Double
End Class
Public Class YahooFinanceFactory
Public Property Cookie As String
Public Property Crumb As String
Public Property CrumbUrl As String = "https://finance.yahoo.com/quote/{0}?p={0}"
Public Property DownloadUrl As String = "https://query1.finance.yahoo.com/v7/finance/download/{0}?period1={1}&period2={2}&interval=1d&events={3}&crumb={4}"
Public Property Timeout As Integer = 5
Public Property NoRefreshRetries As Integer = 10
Public Property NoDownloadRetries As Integer = 10
Private Property Regex_crumb As Regex
Public Shared Async Function CreateAsync(Optional noRefreshRetries As Integer = 10, Optional noDownloadRetries As Integer = 10, Optional timeout As Integer = 5, Optional crumbUrl As String = "https://finance.yahoo.com/quote/{0}?p={0}", Optional downloadUrl As String = "https://query1.finance.yahoo.com/v7/finance/download/{0}?period1={1}&period2={2}&interval=1d&events={3}&crumb={4}") As Task(Of YahooFinanceFactory)
Return Await (New YahooFinanceFactory With {
.NoRefreshRetries = noRefreshRetries,
.NoDownloadRetries = noDownloadRetries,
.Timeout = timeout,
.CrumbUrl = crumbUrl,
.DownloadUrl = downloadUrl
}).RefreshAsync()
End Function
Public Async Function GetHistoricalDataAsync(symbol As String, dateFrom As Date) As Task(Of IEnumerable(Of YahooHistoryPrice))
Dim count As Integer = 0
If Not IsValid Then
Throw New Exception("Invalid YahooFinanceFactory instance")
End If
Dim csvData = Await GetRawAsync(symbol, dateFrom, Now).ConfigureAwait(False)
If csvData IsNot Nothing Then
Return ParsePrice(csvData)
End If
Return Array.Empty(Of YahooHistoryPrice)
End Function
Public Async Function GetRawAsync(symbol As String, start As DateTime, [end] As DateTime) As Task(Of String)
Dim count = 0
While count < NoDownloadRetries
Try
Dim cookies = New CookieContainer
cookies.Add(New Cookie("B", If(Cookie.StartsWith("B="), Cookie.Substring(2), Cookie), "/", ".yahoo.com"))
Using handler = New HttpClientHandler With {.CookieContainer = cookies}
Using client = New HttpClient(handler) With {.Timeout = TimeSpan.FromSeconds(Timeout)}
Dim httpResponse = Await client.GetAsync(GetDownloadUrl(symbol, start)).ConfigureAwait(False)
Return Await httpResponse.Content.ReadAsStringAsync
End Using
End Using
Catch ex As Exception
If count >= NoDownloadRetries - 1 Then
Throw
End If
End Try
count += 1
End While
Throw New Exception("Retries exhausted")
End Function
Private Function ParsePrice(ByVal csvData As String) As IEnumerable(Of YahooHistoryPrice)
Dim lst = New List(Of YahooHistoryPrice)
Dim rows = csvData.Split(Convert.ToChar(10))
For i = 1 To rows.Length - 1
Dim row = rows(i)
If String.IsNullOrEmpty(row) Then
Continue For
End If
Dim cols = row.Split(","c)
If cols(1) = "null" Then
Continue For
End If
Dim itm = New YahooHistoryPrice With {.Date = DateTime.Parse(cols(0)), .Open = Convert.ToDouble(cols(1)), .High = Convert.ToDouble(cols(2)), .Low = Convert.ToDouble(cols(3)), .Close = Convert.ToDouble(cols(4)), .AdjClose = Convert.ToDouble(cols(5))}
If cols(6) <> "null" Then
itm.Volume = Convert.ToDouble(cols(6))
End If
lst.Add(itm)
Next
Return lst
End Function
Public ReadOnly Property IsValid() As Boolean
Get
Return Not String.IsNullOrWhiteSpace(Cookie) And Not String.IsNullOrWhiteSpace(Crumb)
End Get
End Property
Public Function GetDownloadUrl(symbol As String, dateFrom As Date, Optional eventType As String = "history") As String
Return String.Format(DownloadUrl, symbol, Math.Round(DateTimeToUnixTimestamp(dateFrom), 0), Math.Round(DateTimeToUnixTimestamp(Now.AddDays(-1)), 0), eventType, Crumb)
End Function
Public Function GetCrumbUrl(symbol As String) As String
Return String.Format(Me.CrumbUrl, symbol)
End Function
Public Function DateTimeToUnixTimestamp(dateTime As DateTime) As Double
Return (dateTime.ToUniversalTime() - New DateTime(1970, 1, 1, 0, 0, 0, 0, DateTimeKind.Utc)).TotalSeconds
End Function
Private Async Function RefreshAsync(Optional symbol As String = "SPY") As Task(Of YahooFinanceFactory)
Dim count = 0
While count < NoRefreshRetries And Not IsValid
Try
Using client = New HttpClient With {.Timeout = TimeSpan.FromSeconds(Timeout)}
Dim httpResponse = Await client.GetAsync(GetCrumbUrl(symbol)).ConfigureAwait(False)
Me.Cookie = httpResponse.Headers.First(Function(f) f.Key = "Set-Cookie").Value.FirstOrDefault?.Split(";"c)(0)
Dim html = Await httpResponse.Content.ReadAsStringAsync
Me.Crumb = GetCrumb(html)
If Crumb IsNot Nothing Then
Return Me
End If
End Using
Catch ex As Exception
If count >= NoRefreshRetries - 1 Then
Cookie = ""
Crumb = ""
Throw
End If
End Try
count += 1
End While
Cookie = ""
Crumb = ""
Throw New Exception("Could not refresh YahooFinanceFactory")
End Function
Private Function GetCrumb(html As String) As String
Dim crumb As String = Nothing
If Regex_crumb Is Nothing Then
Regex_crumb = New Regex("CrumbStore"":{""crumb"":""(?<crumb>.+?)""}", RegexOptions.CultureInvariant Or RegexOptions.Compiled, TimeSpan.FromSeconds(5))
End If
Dim matches As MatchCollection = Regex_crumb.Matches(html)
If matches.Count > 0 Then
crumb = matches(0).Groups("crumb").Value
crumb = System.Text.RegularExpressions.Regex.Unescape(crumb)
Else
Throw New Exception("Regex no match")
End If
Return crumb
End Function
End Class
End Namespace
Why not using the ready one which provides full access. without malfunctioning:
tickers='AAPL'
from pandas_datareader import data as wb
new_data = pd.DataFrame()
for t in tickers :
new_data[t] = wb.DataReader(t, data_source ='yahoo', start = '2004-1-1')['Adj Close']
a = new_data[t]
It's possible to get current and historical data from google finance api. Works very good for me.

Is it possible to access the matrix parameters (name-value pair separated by semicolon) in ColdFusion?

I'm new to matrix parameter and I know CF10 can access them through their new RESTful API support.
However, is there a way to access these parameters without using RESTful API support?
E.g. http://moremaps.com/map/color.cfm;lat=50;long=20;scale=32000
You can use:
color.cfm;lat=50;long=20;scale=32000
Then get the param string with:
ListRest(getPageContext().getRequest().getRequestUri(),';')
This worked back in CFMX - it's not specific to CF10 or part of the RESTful API, and is available due to the servlet container (Tomcat/Jrun/etc) following the servlet spec with the ability to get the original URL.
(And you can of course use URL rewriting to hide the .cfm from the user.)
There isn't a matrix scope because CF hasn't implemented it fully - it is done as part of REST webservices, (where it's as an argument with the appropriate RestArgSource attribute). Only the CF team can say why they designed it that way.
However, you can easily create your own scope/struct like so:
var MatrixString = ListRest(getPageContext().getRequest().getRequestUri(),';');
var Matrix = {};
for ( var CurParam in ListToArray(MatrixString,';') )
Matrix[ UrlDecode( ListFirst(CurParam,'=') ) ] = UrlDecode( ListRest(CurParam,'=') );
(Obviously, remove the var scoping if not using inside a function.)
That works both directly and via IIS, and should work fine on other servers too, even where path_info may have been modified.
Update: This answer is incomplete/inaccurate.
Reading up further on matrix params, they can actually appear at any point in the request_uri (the entire part after the host_name, before the query_string) - that is, both script_name and path_info can contain parameters, without affecting their final value.
To clarify this, both these URLs:
htp://domain.com/a/b.cfm/c/d
http://domain.com/a;alpha=1/b.cfm;beta=2/c;gamma=3/d;delta=4
Result in these CGI vars:
script_name = /a/b.cfm
path_info = /c/d
(Except in IIS, where path_info is incorrectly implemented.)
Obviously extracting and acting upon these properties is more complex than the code above - I'll update this answer again once I've made sure I understand them more fully.
In the meantime, here's a couple of potential options - the first returns a struct of params if a path element has one, the second returns an array containing every path element - whether either of these are suitable will depend on how the matrix params are to be used:
<cffunction name="getMatrixStruct" returntype="Struct" output=false
hint="Returns struct with item for each path element with params"
>
<cfargument name="RequestUri" type="String" required hint="The non-host and non-querystring part of a URL." />
<cfscript>
var Result = {};
for ( var CurSegment in ListToArray(RequestUri,'/') )
{
var SegName = UrlDecode( ListFirst(CurSegment,';') );
for ( var CurParam in ListToArray(ListRest(CurSegment,';')) )
Result[SegName][UrlDecode( ListFirst(CurParam,'=') ) ] = UrlDecode( ListRest(CurParam,'=') );
}
return Result;
</cfscript>
</cffunction>
<cffunction name="getMatrixArray" returntype="Array" output=false
hint="Returns array of structs for all path element, with any parameters included."
>
<cfargument name="RequestUri" type="String" required hint="The non-host and non-querystring part of a URL." />
<cfscript>
var Result = [];
var Pos = 0;
for ( var CurSegment in ListToArray(RequestUri,'/') )
{
Result[++Pos] = { 'Name' = UrlDecode( ListFirst(CurSegment,';') ) };
for ( var CurParam in ListToArray(ListRest(CurSegment,';')) )
Result[Pos][UrlDecode( ListFirst(CurParam,'=') ) ] = UrlDecode( ListRest(CurParam,'=') );
}
return Result;
</cfscript>
</cffunction>
You can access it via cgi.path_info. For example:
http://localhost/myApp/index.cfm;this=that;pet=cat
Becomes
cgi.PATH_INFO=/MyApp/index.cfm;this=that;pet=cat
Then you can
struct function getMatrix() output="false" {
var arURLData = ListToArray(cgi.path_Info, ";");
var stData = {};
if (arrayLen(arData) <= 1) {
return stData;
}
for(var i = 2; i <= ArrayLen(arData); i++) {
// setVariable("stData.#listfirst(arURLData[i], "=")#", listlast(arURLData[i]);
stData[listfirst(arURLData[i], "=")] = getToken(arURLData[i], 2, "=");
}
return stData;
}
You will probably want to add some code to protect against URL injection attacks.

JSON2 Error / Conflict with another script

I am using the JSON2 library in order to use JSON.stringify to send some JSON data to my MVC controller.
When I include another script in my view (Telerik MVC) I start to get script conflicts when using IE7.
When I click the refresh button in the grid, I get the following error:
Line: 191
Error: Object doesn't support this property or method
String.prototype.toJSON =
Number.prototype.toJSON =
Boolean.prototype.toJSON = function (key) {
return this.valueOf();
};
The error occurs on the following line specifically:
return this.valueOf();
Does anyone have any insight into why this conflict is occurring and how to resolve it? Specifically, why would this work in IE8/Chrome but fail in IE7. What would cause the error? Are both scripts trying to define the same method and that's why it is failing or is it impossible to tell without digging through tons of code?
Edit:
This is the json2.js library I am speaking of: https://github.com/douglascrockford/JSON-js
Probably the reply is too late, but I thought it's worth replying as this might save some valuable lives ;)
The JSON2 script won't initialize/extend the JSON object if there is an existing implementation(Native or Included). However if the JSON object does not exist, the script will create that object and attach few methods to it (JSON.stringify and JSON.parse to be precise). However in order to make those methods to work, there are other objects (like Date, String, Number and Boolean objects) which need to be extended to support certain methods (like toJSON method). The JSON2 script takes care of extending the required objects as well.
Now coming to the specific issue here (Telerik MVC). I faced the same problem while working with Telerik for one of the Projects. However I was able to trace it. The probable cause is the conflict between Telerik scripts and the current JSON2 script. The Date and Boolean Objects' toJSON method somehow conflicts with Telerik's implmentation of the same method for those two objects which breaks the Telerik script at some places. I have modified the JSON2 library for a more robust check which doesn't fail in any scenario (even on use of Telerik MVC on the page). I have tested the script and it works fine for me, however in case someone finds any further conflicts, please reply back.
var JSON;
if (!JSON) {
JSON = {};
}
(function () {
'use strict';
function f(n) {
// Format integers to have at least two digits.
return n < 10 ? '0' + n : n;
}
if (typeof Date.prototype.toJSON !== 'function') {
Date.prototype.toJSON = function (key) {
return isFinite(this.valueOf())
? this.getUTCFullYear() + '-' +
f(this.getUTCMonth() + 1) + '-' +
f(this.getUTCDate()) + 'T' +
f(this.getUTCHours()) + ':' +
f(this.getUTCMinutes()) + ':' +
f(this.getUTCSeconds()) + /*added - start*/ '.'+
f(this.getUTCMilliseconds()) + /*added - end*/ 'Z'
: null;
};
//pushed the below code outside current if block
// String.prototype.toJSON =
// Number.prototype.toJSON =
// Boolean.prototype.toJSON = function (key) {
// return this.valueOf();
// };
}
/*added - start*/
if (typeof String.prototype.toJSON !== 'function') {
String.prototype.toJSON = function (key) {
return ((typeof this.valueOf === 'function') ? this.valueOf(): this.toString());
};
}
if (typeof Number.prototype.toJSON !== 'function') {
Number.prototype.toJSON = function (key) {
return ((typeof this.valueOf === 'function') ? this.valueOf(): this.toString());
};
}
if (typeof Boolean.prototype.toJSON !== 'function') {
Boolean.prototype.toJSON = function (key) {
return ((typeof this.valueOf === 'function') ? this.valueOf(): this.toString());
};
}
/*added - end*/
var cx = /[\u0000\u00ad\u0600-\u0604\u070f\u17b4\u17b5\u200c-\u200f\u2028-\u202f\u2060-\u206f\ufeff\ufff0-\uffff]/g,
escapable = /[\\\"\x00-\x1f\x7f-\x9f\u00ad\u0600-\u0604\u070f\u17b4\u17b5\u200c-\u200f\u2028-\u202f\u2060-\u206f\ufeff\ufff0-\uffff]/g,
gap,
indent,
meta = { // table of character substitutions
'\b': '\\b',
'\t': '\\t',
'\n': '\\n',
'\f': '\\f',
'\r': '\\r',
'"' : '\\"',
'\\': '\\\\'
},
rep;
function quote(string) {
// If the string contains no control characters, no quote characters, and no
// backslash characters, then we can safely slap some quotes around it.
// Otherwise we must also replace the offending characters with safe escape
// sequences.
escapable.lastIndex = 0;
return escapable.test(string) ? '"' + string.replace(escapable, function (a) {
var c = meta[a];
return typeof c === 'string'
? c
: '\\u' + ('0000' + a.charCodeAt(0).toString(16)).slice(-4);
}) + '"' : '"' + string + '"';
}
function str(key, holder) {
// Produce a string from holder[key].
var i, // The loop counter.
k, // The member key.
v, // The member value.
length,
mind = gap,
partial,
value = holder[key];
// If the value has a toJSON method, call it to obtain a replacement value.
if (value && typeof value === 'object' &&
typeof value.toJSON === 'function') {
value = value.toJSON(key);
}
// If we were called with a replacer function, then call the replacer to
// obtain a replacement value.
if (typeof rep === 'function') {
value = rep.call(holder, key, value);
}
// What happens next depends on the value's type.
switch (typeof value) {
case 'string':
return quote(value);
case 'number':
// JSON numbers must be finite. Encode non-finite numbers as null.
return isFinite(value) ? String(value) : 'null';
case 'boolean':
case 'null':
// If the value is a boolean or null, convert it to a string. Note:
// typeof null does not produce 'null'. The case is included here in
// the remote chance that this gets fixed someday.
return String(value);
// If the type is 'object', we might be dealing with an object or an array or
// null.
case 'object':
// Due to a specification blunder in ECMAScript, typeof null is 'object',
// so watch out for that case.
if (!value) {
return 'null';
}
// Make an array to hold the partial results of stringifying this object value.
gap += indent;
partial = [];
// Is the value an array?
if (Object.prototype.toString.apply(value) === '[object Array]') {
// The value is an array. Stringify every element. Use null as a placeholder
// for non-JSON values.
length = value.length;
for (i = 0; i < length; i += 1) {
partial[i] = str(i, value) || 'null';
}
// Join all of the elements together, separated with commas, and wrap them in
// brackets.
v = partial.length === 0
? '[]'
: gap
? '[\n' + gap + partial.join(',\n' + gap) + '\n' + mind + ']'
: '[' + partial.join(',') + ']';
gap = mind;
return v;
}
// If the replacer is an array, use it to select the members to be stringified.
if (rep && typeof rep === 'object') {
length = rep.length;
for (i = 0; i < length; i += 1) {
if (typeof rep[i] === 'string') {
k = rep[i];
v = str(k, value);
if (v) {
partial.push(quote(k) + (gap ? ': ' : ':') + v);
}
}
}
} else {
// Otherwise, iterate through all of the keys in the object.
for (k in value) {
if (Object.prototype.hasOwnProperty.call(value, k)) {
v = str(k, value);
if (v) {
partial.push(quote(k) + (gap ? ': ' : ':') + v);
}
}
}
}
// Join all of the member texts together, separated with commas,
// and wrap them in braces.
v = partial.length === 0
? '{}'
: gap
? '{\n' + gap + partial.join(',\n' + gap) + '\n' + mind + '}'
: '{' + partial.join(',') + '}';
gap = mind;
return v;
}
}
// If the JSON object does not yet have a stringify method, give it one.
if (typeof JSON.stringify !== 'function') {
JSON.stringify = function (value, replacer, space) {
// The stringify method takes a value and an optional replacer, and an optional
// space parameter, and returns a JSON text. The replacer can be a function
// that can replace values, or an array of strings that will select the keys.
// A default replacer method can be provided. Use of the space parameter can
// produce text that is more easily readable.
var i;
gap = '';
indent = '';
// If the space parameter is a number, make an indent string containing that
// many spaces.
if (typeof space === 'number') {
for (i = 0; i < space; i += 1) {
indent += ' ';
}
// If the space parameter is a string, it will be used as the indent string.
} else if (typeof space === 'string') {
indent = space;
}
// If there is a replacer, it must be a function or an array.
// Otherwise, throw an error.
rep = replacer;
if (replacer && typeof replacer !== 'function' &&
(typeof replacer !== 'object' ||
typeof replacer.length !== 'number')) {
throw new Error('JSON.stringify');
}
// Make a fake root object containing our value under the key of ''.
// Return the result of stringifying the value.
return str('', {'': value});
};
}
// If the JSON object does not yet have a parse method, give it one.
if (typeof JSON.parse !== 'function') {
JSON.parse = function (text, reviver) {
// The parse method takes a text and an optional reviver function, and returns
// a JavaScript value if the text is a valid JSON text.
var j;
function walk(holder, key) {
// The walk method is used to recursively walk the resulting structure so
// that modifications can be made.
var k, v, value = holder[key];
if (value && typeof value === 'object') {
for (k in value) {
if (Object.prototype.hasOwnProperty.call(value, k)) {
v = walk(value, k);
if (v !== undefined) {
value[k] = v;
} else {
delete value[k];
}
}
}
}
return reviver.call(holder, key, value);
}
// Parsing happens in four stages. In the first stage, we replace certain
// Unicode characters with escape sequences. JavaScript handles many characters
// incorrectly, either silently deleting them, or treating them as line endings.
text = String(text);
cx.lastIndex = 0;
if (cx.test(text)) {
text = text.replace(cx, function (a) {
return '\\u' +
('0000' + a.charCodeAt(0).toString(16)).slice(-4);
});
}
// In the second stage, we run the text against regular expressions that look
// for non-JSON patterns. We are especially concerned with '()' and 'new'
// because they can cause invocation, and '=' because it can cause mutation.
// But just to be safe, we want to reject all unexpected forms.
// We split the second stage into 4 regexp operations in order to work around
// crippling inefficiencies in IE's and Safari's regexp engines. First we
// replace the JSON backslash pairs with '#' (a non-JSON character). Second, we
// replace all simple value tokens with ']' characters. Third, we delete all
// open brackets that follow a colon or comma or that begin the text. Finally,
// we look to see that the remaining characters are only whitespace or ']' or
// ',' or ':' or '{' or '}'. If that is so, then the text is safe for eval.
if (/^[\],:{}\s]*$/
.test(text.replace(/\\(?:["\\\/bfnrt]|u[0-9a-fA-F]{4})/g, '#')
.replace(/"[^"\\\n\r]*"|true|false|null|-?\d+(?:\.\d*)?(?:[eE][+\-]?\d+)?/g, ']')
.replace(/(?:^|:|,)(?:\s*\[)+/g, ''))) {
// In the third stage we use the eval function to compile the text into a
// JavaScript structure. The '{' operator is subject to a syntactic ambiguity
// in JavaScript: it can begin a block or an object literal. We wrap the text
// in parens to eliminate the ambiguity.
j = eval('(' + text + ')');
// In the optional fourth stage, we recursively walk the new structure, passing
// each name/value pair to a reviver function for possible transformation.
return typeof reviver === 'function'
? walk({'': j}, '')
: j;
}
// If the text is not JSON parseable, then a SyntaxError is thrown.
throw new SyntaxError('JSON.parse');
};
}
}());
Note: The above code is not my implementation. It is from the source https://github.com/douglascrockford/JSON-js I have just modified it a little to avoid any conflicts with Telerik or otherwise.
I have had exactly the same problem.
I couldn't find any other solution than to edit the json2.js file like you suggested, thanks for that.
However, I found that this would fix the issue for IE7 and still work in IE8/9 as well as firefox, but it now stopped working in Chrome ("this" for [this.valueOf === 'function'] is undefined).
Have you run into that issue, too, or did yours work in Chrome? I'm trying to figure out if this is related to my data or telerik-internal.
Thanks for your post!
Edit:
For now I have just returned null if "this" is undefined/null (in all three functions). Seems to work in all browsers and allows the Telerik grid to rebind without problems.
I don't know how correct this is in the global context of json2.js .toJSON method, though.

Node.js url.parse result back to string

I am trying to do some simple pagination.
To that end, I'm trying to parse the current URL, then produce links to the same query, but with incremented and decremented page parameters.
I've tried doing the following, but it produces the same link, without the new page parameter.
var parts = url.parse(req.url, true);
parts.query['page'] = 25;
console.log("Link: ", url.format(parts));
The documentation for the URL module seems to suggest that format is what I need but I'm doing something wrong.
I know I could iterate and build up the string manually, but I was hoping there's an existing method for this.
If you look at the latest documentation, you can see that url.format behaves in the following way:
search will be used in place of query
query (object; see querystring) will only be used if search is absent.
And when you modify query, search remains unchanged and it uses it. So to force it to use query, simply remove search from the object:
var url = require("url");
var parts = url.parse("http://test.com?page=25&foo=bar", true);
parts.query.page++;
delete parts.search;
console.log(url.format(parts)); //http://test.com/?page=26&foo=bar
Make sure you're always reading the latest version of the documentation, this will save you a lot of trouble.
Seems to me like it's a bug in node. You might try
// in requires
var url = require('url');
var qs = require('querystring');
// later
var parts = url.parse(req.url, true);
parts.query['page'] = 25;
parts.query = qs.stringify(parts.query);
console.log("Link: ", url.format(parts));
The other answer is good, but you could also do something like this. The querystring module is used to work with query strings.
var querystring = require('querystring');
var qs = querystring.parse(parts.query);
qs.page = 25;
parts.search = '?' + querystring.stringify(qs);
var newUrl = url.format(parts);
To dry out code and get at URL variables without needing to require('url') I used:
/*
Used the url module to parse and place the parameters into req.urlparams.
Follows the same pattern used for swagger API path variables that load
into the req.params scope.
*/
app.use(function(req, res, next) {
var url = require('url');
var queryURL = url.parse(req.url, true);
req.urlparams = queryURL.query;
next();
});
var myID = req.urlparams.myID;
This will parse and move the url variables into the req.urlparams variable. It runs early in the request workflow so is available for all expressjs paths.

What grammar is this?

I have to parse a document containing groups of variable-value-pairs which is serialized to a string e.g. like this:
4^26^VAR1^6^VALUE1^VAR2^4^VAL2^^1^14^VAR1^6^VALUE1^^
Here are the different elements:
Group IDs:
4^26^VAR1^6^VALUE1^VAR2^4^VAL2^^1^14^VAR1^6^VALUE1^^
Length of string representation of each group:
4^26^VAR1^6^VALUE1^VAR2^4^VAL2^^1^14^VAR1^6^VALUE1^^
One of the groups:
4^26^VAR1^6^VALUE1^VAR2^4^VAL2^^1^14 ^VAR1^6^VALUE1^^
Variables:
4^26^VAR1^6^VALUE1^VAR2^4^VAL2^^1^14^VAR1^6^VALUE1^^
Length of string representation of the values:
4^26^VAR1^6^VALUE1^VAR2^4^VAL2^^1^14^VAR1^6^VALUE1^^
The values themselves:
4^26^VAR1^6^VALUE1^VAR2^4^VAL2^^1^14^VAR1^6^VALUE1^^
Variables consist only of alphanumeric characters.
No assumption is made about the values, i.e. they may contain any character, including ^.
Is there a name for this kind of grammar? Is there a parsing library that can handle this mess?
So far I am using my own parser, but due to the fact that I need to detect and handle corrupt serializations the code looks rather messy, thus my question for a parser library that could lift the burden.
The simplest way to approach it is to note that there are two nested levels that work the same way. The pattern is extremely simple:
id^length^content^
At the outer level, this produces a set of groups. Within each group, the content follows exactly the same pattern, only here the id is the variable name, and the content is the variable value.
So you only need to write that logic once and you can use it to parse both levels. Just write a function that breaks a string up into a list of id/content pairs. Call it once to get the groups, and then loop through them calling it again for each content to get the variables in that group.
Breaking it down into these steps, first we need a way to get "tokens" from the string. This function returns an object with three methods, to find out if we're at "end of file", and to grab the next delimited or counted substring:
var tokens = function(str) {
var pos = 0;
return {
eof: function() {
return pos == str.length;
},
delimited: function(d) {
var end = str.indexOf(d, pos);
if (end == -1) {
throw new Error('Expected delimiter');
}
var result = str.substr(pos, end - pos);
pos = end + d.length;
return result;
},
counted: function(c) {
var result = str.substr(pos, c);
pos += c;
return result;
}
};
};
Now we can conveniently write the reusable parse function:
var parse = function(str) {
var parts = {};
var t = tokens(str);
while (!t.eof()) {
var id = t.delimited('^');
var len = t.delimited('^');
var content = t.counted(parseInt(len, 10));
var end = t.counted(1);
if (end !== '^') {
throw new Error('Expected ^ after counted string, instead found: ' + end);
}
parts[id] = content;
}
return parts;
};
It builds an object where the keys are the IDs (or variable names). I'm asuming as they have names that the order isn't significant.
Then we can use that at both levels to create the function to do the whole job:
var parseGroups = function(str) {
var groups = parse(str);
Object.keys(groups).forEach(function(id) {
groups[id] = parse(groups[id]);
});
return groups;
}
For your example, it produces this object:
{
'1': {
VAR1: 'VALUE1'
},
'4': {
VAR1: 'VALUE1',
VAR2: 'VAL2'
}
}
I don't think it's a trivial task to create a grammar for this. But on the other hand, a simple straight forward approach is not that hard. You know the corresponding string length for every critical string. So you just chop your string according to those lengths apart..
where do you see problems?

Resources