How to implement Web Crawling technology in iOS.
Any reference/ sample program will help.
Thanks in advance.
Web Crawling
A web crawler (also known as a web spider or web robot) is a program or automated script which browses the World Wide Web in a methodical, automated manner. This process is called Web crawling or spidering. Many legitimate sites, in particular search engines, use spidering as a means of providing up-to-date data.
Generally, iOS is not suitable for Web Crawling, because it is not easily flexible for programmers.
But if you want, it is possible. Basically you will use AFNetworking (or Alamofire, or the System default method) to send web requests. Once you get the response, analyze the text, mainly by using Regular Expression. For example, you can write a category for NSString:
#implementation NSString(StringRegular)
-(NSMutableArray *)substringByRegular:(NSString *)regular{
NSString *reg=regular;
NSRange r = [self rangeOfString:reg options:NSRegularExpressionSearch];
NSMutableArray *arr=[NSMutableArray array];
if (r.length != NSNotFound &&r.length != 0) {
int i=0;
while (r.length != NSNotFound &&r.length != 0) {
NSString* substr = [self substringWithRange:r];
[arr addObject:substr];
NSRange startr=NSMakeRange(r.location+r.length, [self length]-r.location-r.length);
r=[self rangeOfString:reg options:NSRegularExpressionSearch range:startr];
}
}
return arr;
}
#end
And then, you will need to store your data. I recommend you to use an online database. If not, you can store the data to your iOS device using FMDatabase or simply SQLite
I have used DIFFBOT for web crawling in iOS. The site gives API for different purposes like product, analysing pages or articles. It comes with a free trail of 14 days too. Here is the code for a product web crawling:
let url = "https://api.diffbot.com/v3/product?token=YOURTOKEN&url=TheUrlWhichYouwantToSearchinURLENCODEDFORMAT"
let requestUrl = RequestHandlerToken(url: url, withPostMethod: false)
requestUrl.startRequest { (response, error) in
print(response)
self.parseData(resp: response)
}
The response will be coming in JSON format. Make sure that the URL is in URLencoded format. Here I am using custom class for hitting the request. You can do this by using nsurl session or connection.
Related
I have an ESP32 Battery monitor system (BMS) whose status I want to view on an iPhone, there are about 100 values to be monitored with a maximum update rate of once per second. I also needs to be able to send settings to the BMS occasionally.
Blynk seemed to be the perfect app to do this but the new version 2.0 doesn't support BLE! Does anyone know a similar app that could do this?
Have you tried setting the ESP32 up as a web server and creating an API based on the values? Using this instead of bluetooth allows you to view the values from anywhere with internet connection, and is easier to implement and use across different platforms
You can do this by implementing code into the ESP32, so that when you make http requests, it returns with a value, just like any other API would. Then in Xcode where you are building your IOS app you can make http requests to the web server and retrieve values
for example, in the ESP32 you can do something like this (assuming you are using the Arduino IDE to program it):
server.on("/value", [](AsyncWebServerRequest * request))
{
String value = "value"
request->send(200,"text/plain", value)
}
This YouTube video shows you this in more detail: https://www.youtube.com/watch?v=cWZP7Y8qP6E
Then, in Xcode you can implement something along the lines of:
func loadData() async {
guard let url = URL(string: "http://IP-address-of-esp32/path-of-value", ) else {
print("Invalid URL")
return
}
do {
let (data, _) = try await URLSession.shared.data(from: url)
// process data here with the 'data' variable
} catch {
print("Invalid Data")
}
}
//Be aware that this is asynchronous when calling it
This person also explains it in more detail in a YouTube video: https://www.youtube.com/watch?v=MBCX1atOvdA
Now, since you have a web server and API set up on your ESP32, and a method of retrieving the values in your IOS app, you can go on and use the values to build the app to your liking.
Good Luck!
I'm working on an iOS application using Xamarin.Forms. This application is using UIWebView controller that shows a web application that is hosting on my server. Each time that I make a request I have to send a custom header in order to identify that this request comes to the mobile application and not from a browser, to do this I'm using an NSUrlProtocol object that overrides the method Request that inserts the custom header on each request.This is my code:
public override NSUrlRequest Request {
get {
NSMutableDictionary headers = null;
if (null == base.Request.Headers) {
headers = new NSMutableDictionary ();
} else {
headers = new NSMutableDictionary (base.Request.Headers);
}
headers.Add(NSObject.FromObject(AppVariables.headerVariable), NSObject.FromObject (AppVariables.appVersion));
NSMutableUrlRequest newRequest = (NSMutableUrlRequest)base.Request.MutableCopy ();
newRequest.Headers = headers;
return newRequest;
}
}
The problem that I have right now is that I noticed since I started using the NSUrlProtocol the loading time of the pages is increasing a lot. Right now the loading is taking 10 seconds, before this implementation the page took 3 seconds approximately.
Can anyone please point out some helpful direction to overcome this??
I don't see any reasons for the delay in response time when you're using custom headers. Like Andreas mentioned in the comments, I believe it has to do with your server code. I would recommend profiling your server code.
Do you see similar results when you send the requests (with custom headers) from Fiddler or cURL?
Just like #AndreasPaulsson and #prashant had mentioned, server might be the culprit. I would recommend testing the API with tools like Postman and check the response speed. I would also recommend you to check ModernHttpClient by Paul C Betts. In iOS the library uses NSUrlSession.
Parse's documentation states that a cloud function would look like so:
[PFCloud callFunctionInBackground:#"averageStars"
withParameters:#{#"movie": #"The Matrix"}
block:^(NSNumber *ratings, NSError *error) {
if (!error) {
// ratings is 4.5
}
}];
However, I can't seem to figure out where the API request would be placed if I wanted to ping eBay's item database. The format of the request I'm referring to is below:
var url = "http://svcs.ebay.com/services/search/FindingService/v1";
url += "?OPERATION-NAME=findItemsByKeywords";
url += "&SERVICE-VERSION=1.0.0";
url += "&SECURITY-APPNAME=MyAppID";
url += "&GLOBAL-ID=EBAY-US";
url += "&RESPONSE-DATA-FORMAT=JSON";
url += "&callback=_cb_findItemsByKeywords";
url += "&REST-PAYLOAD";
url += "&keywords=harry%20potter";
url += "&paginationInput.entriesPerPage=3";
The request to PFCloud class is used to interact with parse's servers, it can't connect directly with third party services.
You can however, interact with third party services via http requests. You can perform an httpRequest via your cloud code function. Http requests are pretty standard JavaScript, you can read about them here!
If eBay has a javascript library, you might be able to wrap it as a custom module and use it that way. You can read about that here.
And, just a reminder, Make sure you're not doing anything too heavy, as timeout is I think, 15 seconds.
Suppose, the webservice provider's server exposing the webservice in the form http://example.com:8000/api
What is best framework or library to access the webservice in ios 7 project
There are a few marshaling frameworks that support generating an object-graph from XML, however I would simply go for the following:
Invoke the service endpoint. My favorite library is BBHTTP, however you could use AFNetworking, NSURLConnection with gcd or whatever you prefer for asynch network calls.
Extract the relevant contents of the XML payload onto your use-case specific payload object using RaptureXML
I recommend having use-case specific payload objects because they model exactly what is needed for a given service invocation - supporting the notion of contract-first development. This allows you to change you internal model without effecting the integration to external systems. Similarly the external API can change without effecting your model.
You can create a category method on RXMLElement to return the element mapped to a use-case-specific object. A typical mapping usually takes just a handful of lines of code to marshal from wire-format to your payload object for the service invocation.
Here's an example (the code that I took it from wanted the payload wrapped in a SOAP envelope - just ignore that bit).
- (void)request:(MyUseCaseRequstPayload*)request onComplete:(void (^)(MyResponsePayload*))onSuccess
onError:(void (^)(NSError*))onError;
{
//Even more XML! You can stick your payload inside an envelope if you want
SoapEnvelope* envelope = [SoapEnvelope envelopeWithContent:[request xmlString]];
[[BBHTTPRequest postToURL:_serviceUrl data:[envelope data] contentType:#"text/xml"] execute:^(BBHTTPResponse* response)
{
RXMLElement* element = [RXMLElement elementFromXMLData:[response content]];
MyResponsePayload* response = [[element child:#"elementToBeMapped"] asMyObjectType];
if (onSuccess)
{
onSuccess(response);
}
} error:^(NSError* error)
{
LogDebug(#"Got error: %#", error);
if (onError)
{
onError(error);
}
}];
}
I am trying to build a Survey application where the surveys will be taken offline on multiple ipads and when these ipads are online they are going to upload the data(survey answers) to our servers? I am really struggling how to send the survey to multiple ipads and more importantly to capture from different ipads to one source ?
I need help to clear my architecture part and I need some examples to do the coding part. Do you know anything similar?
What are you ideas?
Many Thanks in advance,
Arda
Create a web server to accept and send survey questions and answers.
I would envision an app that goes like this:
1) Slave iPads makes a HTTP POST request to server asking for the survey
This is usually done using a networking library for iOS like MKNetworkKit or AFNetworking. The general process is to:
create a NSDictionary of key-value pairs to form the HTTP POST request
submit the data through a block construct with completion handler
So something like:
MKNetworkOperation *op = [engine operationWithURLString:#"http://www.mywebserver.com/api/fetchQuestions"
params:nil
httpMethod:#"POST"];
2) Server receives request, grabs all survey questions in database and return JSON encoded questions to slave ipads.
I'm not sure what platform your web server is on but in the past, I used Symfony 2.0 which is a PHP web framework.
It provides very helpful tools like Doctrine (an Object Relational Mapper or ORM) to let me work with my MySQL data as if they're programming objects.
So my general process for fetching data would be something like:
// pseudo php function codes
public function sendSurveyQuestionAction()
{
$repository = $this->getDoctrine()->getRepository('MyAppBundle:Survey');
$query = $repository->createQueryBuilder('query')->getQuery();
$arrObjs = $query->getResult();
$arrObjDatas = NULL;
foreach($arrObjs as $obj)
{
$arrObjDatas[] = $obj->toArray();
}
$response = new Response(json_encode(array('data' => $arrObjDatas)));
$response->headers->set('Content-Type', 'application/json');
$return $response;
}
This would return all survey in JSON format, ready to be parsed by your master iPad app.
3) Users on slave iPads fill in the questions through the app UI and submits. The app saves
the data to disk, checks for a working internet connection before sending data back to server.
Submitting the answer is very similar grabbing the questions, so your iOS code should be something like:
// ------------------------------------------------------------------------------------
// store all question-answers into a dictionary to be submitted as HTTP POST variables
// obviously, you wouldn't create it here, this is just example code, you would likely
// have stored your questions and answers when user presses 'finish' button
// ------------------------------------------------------------------------------------
NSMutableDictionary *paramDictionary = [[NSMutableDictionary alloc] init];
[paramDictionary setObject:#"5" forKey:#"q1"];
[paramDictionary setObject:#"10" forKey:#"q2"];
[paramDictionary setObject:#"15" forKey:#"q3"];
// this helps your web server know how many question-answers to expect, or you could hard code it into your business logic
[paramDictionary setObject:[NSNumber numberWithInteger:3] forKey:#"numberOfQA"];
MKNetworkOperation *op = [engine operationWithURLString:#"http://www.mywebserver.com/api/submitAnswers"
params:paramDictionary
httpMethod:#"POST"];
This will submit your answers for each of your question. You may have noticed I used q1, q2, q3.
These are for your web server code to identify each questions and extract the respective answers from them.
4) Server receives finished answers and commit them to database
So if you were using Symfony 2.0 PHP code, then something like:
// pseudo php function
public function saveAnswersAction()
{
$numOfQA = $_REQUEST['numberOfQA'];
for($i = 0; $i < $numOfQA; $i++)
{
// ----------------------------------------------------------------------
// looping through all the questions such as q1, q2, q3, q4, q5....
// by appending the counter variable to the question identifier
// ----------------------------------------------------------------------
$currentAnswer = $_REQUEST['q'.$i];
// use Doctrine to create new answer entities, and fill in their data
$answerEntity.answer = $currentAnswer;
$surveyEntity->addAnswerEntity($answerEntity);
// mark survey as complete so we can fetch all 'completed' surveys later
$surveyEntity.complete = true;
}
// tell Doctrine to commit changes to MySQL Database
// return HTTP OK status message
}
5) Now all that's left is for your master iPad app to make a HTTP POST request to get all surveys.
The process is the same with your iOS code making a HTTP POST requesting for all 'completed' survey entities from your web server.
The web server grabs them and return them as JSON encoded data.
Your app then receives the completed surveys with question answer like this:
surveys
{
{
questionNumber: 1,
questionAnswer: "5"
},
{
questionNumber: 2,
questionAnswer: "10"
},
{
questionNumber: 3,
questionAnswer: "15"
}
}
Now you use JSONKit to parse this JSON data. You should end up with a NSDictionary from JSONKit.
You can then go something like:
// pseudo code
-(void)displayCompletedSurveys
{
[MKNetworkOperationEngine doRequest:
...
^completionBlock {
// parse JSON data
NSDictionary *surveyData = [JSONKit dictionaryFromJSONData:data)
NSEnumerator *enumerator = [surveyData enumerator];
NSDictionary *currentQuestion = nil;
while([enumerator nextObject] != nil)
{
// do something with each of your question-answer e.g. show it on screen
}
}];
}
Points To Consider
Most of the code above are pseudo-codes. Your final real code would probably be much more in depth.
You'll need to build some master login into your app to prevent everyone from seeing the completed surveys.
Some Extra Information You Should Know
Here are some extra information to help you
JSONKit for fast JSON data decoding from your web server
MKNetworking or AFNetworking to submit your data to your web server
You need to know how to write web services to handle accepting the survey answers. I recommend learning a web framework like Symfony 2.0
Hope that helps.