Using Gambas, is it possible to download a webpage to a string, and then parse that string.
I know I can parse the data in the string once I have the data, I'm struggling with getting the data from the webpage into a string.
You can use the HttpClient class from the gb.net.curl component
There you can also find an example how to read the data either synchronous or asynchronous.
To get the data from the web in a string you could write following function (it would be synchronous in this case)
Public Function GetTextFromUrl(url As String) As String
Dim client As New HttpClient As "client"
client.URL = url
client.async = False
client.Get()
' an error occured
If client.Status < 0 Then
Return ""
Endif
' no data available
If Not Lof(client) Then
Return ""
Endif
' Reads the data from the server and returns it as a String
Return Read #client, Lof(client)
End
And you could call the function like this:
Public Sub Main()
Print GetTextFromUrl("http://stackoverflow.com")
End
Related
I use gatling to send data to an ActiveMQ. The payload is generated in a separate method. The response should also be validated. However, how can I access the session data within the checks
check(bodyString.is()) or simpleCheck(...)? I have also thought about storing the current payload in a separate global variable, but I don't know if this is the right approach. My code's setup looks like this at the moment:
val scn = scenario("Example ActiveMQ Scenario")
.exec(jms("Test").requestReply
.queue(...)
.textMessage{ session => val message = createPayload(); session.set("payload", payload); message}
.check(simpleCheck{message => customCheck(message, ?????? )})) //access stored payload value, alternative: check(bodystring.is(?????)
def customCheck(m: Message, string: String) = {
// check logic goes here
}
Disclaimer: providing example in Java as you don't seem to be a Scala developper, so Java would be a better fit for you (supported since Gatling 3.7).
The way you want to do things can't possibly work.
.textMessage(session -> {
String message = createPayload();
session.set("payload", payload);
return message;
}
)
As explained in the documentation, Session is immutable, so in a function that's supposed to return the payload, you can't also return a new Session.
What you would have to do it first store the payload in the session, then fetch it:
.exec(session -> session.set("payload", createPayload()))
...
.textMessage("#{payload}")
Regarding writing your check, simpleCheck doesn't have access to the Session. You have to use check(bodyString.is()) and pass a function to is, again as explained in the documentation.
I use Rest Assured framework (Java).
I need to send integer array as http-param in get request: http://example.com:8080/myservice?data_ids=11,22,33
Integer[] ids = new Integer[] {11, 22, 33};
...
RequestSpecificationImpl request = (RequestSpecificationImpl)RestAssured.given();
request.baseUri("http://example.com");
request.port(8080);
request.basePath("/myservice");
...
String ids_as_string = Arrays.toString(ids).replaceAll("\\s|[\\[]|[]]", "");
request.params("data_ids", ids_as_string);
System.out.println("Params: " + request.getRequestParams().toString());
System.out.println("URI" + request.getURI());
What I see in the console:
Params: {data_ids=11,22,33}
URI: http://example.com:8080/myservice?data_ids=11%2C22%2C33
Why do my commas transform into '%2C'?
What needs to be done to ensure that commas are passed as they should?
Disable URL encoding, simple as that
given().urlEncodingEnabled(false);
Official documentation
Verified locally,
I'm using Citrus to automate our RESTful services where service1's input param is passed onto service2' input after URLEncoding.
Below is the code snippet, to explain the ways I tried but somehow unable to URLEncode, on top of designer.traceVariables().
Owing to the limitation (possible known!) that the response params can't be passed outside of context, I had to call multiple send(), receive() calls, defined in the same Gherkin method as below.
I'm able to print the variable myVar (extracted from payLoad) using designer.traceVariables() but unable to process & persist it for further use, to pass onto the next service call.
#When("^I call CPAuth service for ChasePay$")
public void i_call_CPAuth_service_for_ChasePay() throws Throwable {
designer.send("CitrusContext")
.messageType(MessageType.JSON)
.header(X_JPMC_CSRF_TOKEN_HEADER, csrfToken)
.header(HTTP.CONTENT_TYPE, APPLICATION_JSON)
.header(COOKIE, ConnRoutePNames.DEFAULT_PROXY + "=" + proxy)
.header("citrus_endpoint_uri", authUrl +"/auth/fcc/login")
.message(new HttpMessage("<large_string=no_issues_here>").method(HttpMethod.POST));
designer.receive("CitrusContext").messageType(MessageType.JSON)
.validate("$.response", "secauth")
.validate("$.newstoken", false)
.validate("$.smtoken", "void")
.extractFromPayload("$.spid", "myVar");
String request = URLEncoder.encode("appStoreBundleId=com.testmerchant.sampleapp&version=APP%2BBUILD%2BVERSION&channelId=MPD&spid=**${myVar}**&type=json&chasePayMerchantId=1390xxx", CharEncoding.UTF_8);
System.out.println("request: "+request);
designer.send("CitrusContext")
.messageType(MessageType.JSON)
.header(X_JPMC_CSRF_TOKEN_HEADER, csrfToken)
.header(HTTP.CONTENT_TYPE, APPLICATION_JSON)
.header(COOKIE, ConnRoutePNames.DEFAULT_PROXY + "=" + proxy)
.header("citrus_endpoint_uri", hostUrl +"/pwc/provisioning/pos/otp/contact/v20160313/list.action")
.message(new HttpMessage(request).method(HttpMethod.POST));
Result:
Before encoding: MSZp2V/czcKsxej+Q04Da/QeVlo=MCwCFAqrBN/6/J8WZENecE7JQlEODnecAhQCcnXGBjMoiQ7deunlyXKacFo/lQ== (This should be the string, to be encoded.)
After encoding: %3D%24%7BmyVar%7D%26
Any help would be greatly appreciated, Thanks.
The encode operation destroys the "${myVar}" expression before Citrus is able to parse and interpret the expression as part of the message payload. You need to execute the encode operation as a Citrus function. How to write this function is described here: http://citrusframework.org/docs/custom-functions/
You end up having something like:
designer.send("CitrusContext")
.messageType(MessageType.JSON)
.message(new HttpMessage("custom:urlEncode('appStoreBundleId=com.testmerchant.sampleapp&version=APP%2BBUILD%2BVERSION&channelId=MPD&spid=**${myVar}**&type=json&chasePayMerchantId=1390xxx')").method(HttpMethod.POST));
As part of our project we need to call Twilio to get E164 format of the Phone Number. In order to do that I followed the below steps to achieve. If the phone number is valid then the below step is working but if the phone number is not valid and we are passing the same to Twillo then it is not returning anything back and halt the process.
I know that the below mention number is not valid but that doesn't mean when I am trying to lookup with invalid number it should not return any response. It should return at least an error but actually it is not which is making me more confused. I am trying with IBM websphere server.
Can some one please help me to fix this issue or is there any way where I can set request time out for twilio lookup so that it will allow other process to continue if any failures in twilio lookup. Also, correct me if I did any mistake in my coding for twilio lookup.
Sample URL "https://lookups.twilio.com/v1/PhoneNumbers/phoneNumber"
Steps Followed:
LookupsClient client = new LookupsClient(ACCOUNT_SID, AUTH_TOKEN);
client.setHttpClient(getProxyClient());
// Twilio Lookup with country code
String twilioURL = createURL(phoneNumber, countryCd);
String twilioE164Number = getTwilioNumber(client, twilioURL);
//Twilio Lookup without country code
StringBuilder appender = new StringBuilder();
appender.append(TWILIO_URL);
appender.append(PLUS);
appender.append(phoneNumber);
String twilioE164Number = getTwilioNumber(client, appender.toString());
//Create lookup URL
private String createURL(String phoneNumber, String countryCode){
StringBuilder appender = new StringBuilder();
appender.append(phoneNumber);
appender.append(QUERY_PARAM);
appender.append(countryCode);
return TWILIO_URL+appender.toString();
}
//Get Twilio E164 Number
private String getTwilioNumber(LookupsClient client,String url){
TwilioRestResponse response;
String twilioResult = "";
try {
response = client.get(url);
String str = response.getResponseText();
//Code for parsing the JSON response and set the twilioResult.
} catch (TwilioRestException e) {
logger.error("TwilioRestException while calling twilio "+e);
}catch(Exception ex){
logger.error("Error in twilio Calling "+ex);
}
return twilioResult;
}
//Get proxy client to set.
private HttpClient getProxyClient() {
HttpHost proxy = new HttpHost(PROXY_HOST, PROXY_PORT);
//new DefaultHttpClient().getParams().setParameter(arg0, arg1)
DefaultProxyRoutePlanner routePlanner = new DefaultProxyRoutePlanner(proxy);
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(new AuthScope(AuthScope.ANY_HOST, AuthScope.ANY_PORT),new UsernamePasswordCredentials(ACCOUNT_SID, AUTH_TOKEN));
CloseableHttpClient httpClient = HttpClients.custom()
.setRoutePlanner(routePlanner)
.setDefaultCredentialsProvider(credentialsProvider)
.build();
return httpClient;
}
v6.3.0 of twilio-java has a LookupsClient has a .getPhoneNumber() method that allows you to pass a phone number instead of requiring that you construct the URL from scratch. The whole point of using LookupsClient instead of a vanilla TwilioClient is that the class does the bulk of the work for you.
Have you tried client.getPhoneNumber(phoneNumber) in this case? This page also demonstrates the structure of the response that you should receive when using that particular endpoint. The E164-formatted phone number should be the phone_number property in the response, although I use twilio-python and I don't know how the Java library exposes response objects for your consumption.
If you aren't getting a response at all, you might want to try making the same request without your proxy configuration to make sure the proxy isn't causing the problem. As for your timeout question, twilio-java's requests are processed synchronously, one way to get around that would be to wrap the request in an asynchronous method call.
Hope some of this helps you diagnose your issue!
I'm working on twitter client for win8, using RestSharp ( http://restsharp.org/ ) and I have such problem:
When I'm posting new tweet with RestClient
var timeLine = new RestRequest("/1/statuses/update.json", Method.POST);
var txt = "Hello world";
timeLine.AddParameter("status", txt);
everything works excelent, but if I add more complex status like:
var txt = "Hello, World! What a nice day! #1May";
timeLine.AddParameter("status", txt);
I recieve 401 error. In debugger I saw, that status parameter in Signature Base String is incorrect. I have:
status%3DHello%2C%2520World%21%2520What%2520a%2520nice%2520day%21%2520%231May
and right string (from dev.twitter.com):
status%3DHello%252C%2520World%2521%2520What%2520a%2520nice%2520day%2521%2520%25231May
You can see, that punctuation marks ,!# and other encodes incorrect. How can I fix it?
Signature base generation and Encoding are in /Authenticators/OAuth/OAuthTools.cs
I have the same problem when I display twitter feed in website. Hence, I used this code to convert the text.
Regex.Replace(str, "#(.*?):", #"#http://twitter.com/#!/$1>$1:");