converting string to map in dart - dart

I wanted to convert a string to map.
String value = "{first_name : fname,last_name : lname,gender : male, location : { state : state, country : country, place : place} }"
into
Map = {
first_name : fname,
last_name : lname,
gender : male,
location = {
state : state,
country : country,
place : place
}
}
How do I convert the string into a map<String, dynamic> where the value consists of string, int, object, and boolean?
I wanted to save the string to a file and obtain the data from the file.

That's not possible.
If you can change the string to valid JSON, you can use
import 'dart:convert';
...
Map valueMap = json.decode(value);
// or
Map valueMap = jsonDecode(value);
The string would need to look like
{"first_name" : "fname","last_name" : "lname","gender" : "male", "location" : { "state" : "state", "country" : "country", "place" : "place"} }

You would have to change the way you create the string.
I'm guessing you are creating the string using the yourMap.toString() method. You should rather use json.encode(yourMap), which converts your map to valid JSON, which you can the parse with json.decode(yourString).

create two objects
class User {
final String firstName;
final String lastName;
final String gender;
final location;
User({
this.firstName,
this.lastName,
this.gender,
this.location,
});
User.fromJson(Map json)
: firstName = json['firstName'],
lastName = json['lastName'],
gender = json['gender'],
location = Location.fromJson(json['location']);
}
class Location {
final String state;
final String country;
final String place;
Location({
this.state,
this.country,
this.place,
});
Location.fromJson(Map json)
: state = json['state'],
country = json['country'],
place = json['place'];
}
then use it like this
var user = User.fromJson(value);
print(user.firstName);
or convert it to list like this
var user = User.fromJson(value).toList();

you can do like this ->
import 'dart:convert';
...
if your data like this **
{'bus1':'100Tk','bus2':'150TK','bus3':'200TK'}
**;
then you can do like this ->
Map valueMap = json.decode(value);
// or
Map valueMap = jsonDecode(value);
or if like this ->var data = {'1':'100TK','2':'200TK','3':'300TK'};
var dataSp = data.split(',');
Map<String,String> mapData = Map();
dataSp.forEach((element) => mapData[element.split(':')[0]] = element.split(':')[1]);
Note: Map first value was Int that's why I did that.

Make a wrapper class for the location where you define the methods fromMap, toMap

Yeah, that's not possible.
But i have workaround to fix that.
Remove space in ur invalid json
Fix ur invalid string json to valid string json
Convert valid string json to map
Here's the full code for above process:
import 'dart:convert';
void main() {
String value = "{first_name : fname,last_name : lname,gender : male, location : { state : state, country : country, place : place} }";
String jsonString = _convertToJsonStringQuotes(raw: value);
print("Test 1: $jsonString");
final Map<dynamic, dynamic> result = json.decode(jsonString);
print('Test 2: $result');
}
String _convertToJsonStringQuotes({required String raw}) {
/// remove space
String jsonString = raw.replaceAll(" ", "");
/// add quotes to json string
jsonString = jsonString.replaceAll('{', '{"');
jsonString = jsonString.replaceAll(':', '": "');
jsonString = jsonString.replaceAll(',', '", "');
jsonString = jsonString.replaceAll('}', '"}');
/// remove quotes on object json string
jsonString = jsonString.replaceAll('"{"', '{"');
jsonString = jsonString.replaceAll('"}"', '"}');
/// remove quotes on array json string
jsonString = jsonString.replaceAll('"[{', '[{');
jsonString = jsonString.replaceAll('}]"', '}]');
return jsonString;
}

To convert a string into a map<String, dynamic>, you can use the
following code:
String value = "{first_name : fname,last_name : lname,gender : male, location : { state : state, country : country, place : place} }";
String result = value
.replaceAll("{","{\"")
.replaceAll("}","\"}")
.replaceAll(":","\":\"")
.replaceAll(",","\",\"");
print(result);
Here, we first replace the opening and closing curly braces with double quotes, and then replace the colons and commas with quotes to create a valid JSON string. Then, we use the jsonDecode method to convert the JSON string into a map.

I found a way to cast that string
Ok, lets use a complex model to cast:
final testMap = {
'userName': 'Igor',
'age': 22,
'totalCash': 138.57,
'isMale:': true,
'userStatus': {
'isUserActive': true,
'isAPremiumUser': false,
},
'userTags': ['Flutter Developer', 'Proactive', 'Clean code'],
'userCourses': [
{
'title': 'How to use TDD in flutter',
'finished': false,
'coursePercentage': 47.4,
'buyDate': '1969-07-20T20:18:04.000Z',
'courseTag': ['New', 'Popular'],
'courseDetails': null,
},
{
'title': 'Clean arquiteture in flutter',
'finished': false,
'coursePercentage': 20.8,
'buyDate': '1969-07-20T20:18:04.000Z',
'courseTag': ['New'],
'courseDetails': {
'teacherName': 'Tayler Mostoult',
'totalSubscribers': 5402,
},
},
{
'title': 'Micro-frontends in flutter',
'finished': true,
'coursePercentage': 100.0,
'buyDate': '1969-07-20T20:18:04.000Z',
'courseTag': [],
'courseDetails': {},
},
]
};
Know, cast it to string:
final testMapInStringFormat = testMap.toString();
To convert this String to map, we can use:
final String response = _getJsonFromString(testMap.toString());
final Map jsonConvertido = jsonDecode(response); // Decoded, back to map format
The function that will effectively do the casting:
String _getJsonFromString(String rawText) {
// Will find, for exemple, the text: "{isUserActive:"
final regexMapKeyWithOpenBracket = RegExp('(?<={)(.*?):+');
// Will find, for exemple, the text: ", userCourses:"
final regexMapKeyWithCommaAndSpace = RegExp(r'(?<=, )([^\]]*?):');
final regexOnlyKeyInLine = RegExp(r'^.+:$');
final splitedSentences = rawText
.replaceAllMapped(regexMapKeyWithCommaAndSpace,
(Match match) => '\n${match.text.trim()}\n')
.replaceAllMapped(regexMapKeyWithOpenBracket,
(Match match) => '\n${match.text.trim()}\n')
.replaceAll(RegExp(r'}(?=,|]|}|$|\s+)'), '\n}\n')
.replaceAll(RegExp(r'(?<=(,|:|^|\[)\s?){'), '\n{\n')
.replaceAll(RegExp('\\[\\s?\\]'), '\n[\n]\n')
.replaceAll(RegExp('\\{\\s?\\}'), '\n{\n}\n')
.split('\n')
..removeWhere((element) => element.replaceAll(' ', '').isEmpty);
final List<String> correctLines = [];
for (String line in splitedSentences) {
final isMapKey = regexOnlyKeyInLine.hasMatch(line);
if (isMapKey) {
final lineWithoutFinalTwoDots = line.substring(0, line.length - 1);
final lineWithQuaot = _putQuotationMarks(lineWithoutFinalTwoDots);
correctLines.add('$lineWithQuaot:');
} else {
String l = line.trim();
// If it falls in this else, it is a value of a key or a map structure
final isNumber = double.tryParse(l) != null || int.tryParse(l) != null;
final isBolean = l == 'false' || l == 'true';
final isStructureCaracter = ['{', '}', '[', ']', ','].any((e) => e == l);
final isNull = l == 'null';
if (isStructureCaracter || isNumber || isBolean || isNull) {
correctLines.add(l);
continue;
}
final hasCommaInFinal = l.endsWith(',');
if (hasCommaInFinal) {
l = l.substring(0, l.length - 1);
}
// If you got to this point, i'm sure it's a value string, so lets add a double quote
final lineWithQuaot = _putQuotationMarks(l);
if (hasCommaInFinal) {
correctLines.add('$lineWithQuaot,');
} else {
correctLines.add(lineWithQuaot);
}
}
}
return correctLines.join('');
}
extension MatchExtension on Match {
String get text => input.substring(start, end);
}
String _putQuotationMarks(String findedText) {
if (!findedText.startsWith('\'') && !findedText.startsWith('"')) {
findedText = findedText[0] + findedText.substring(1);
}
if (!findedText.endsWith('\'')) {
final lastIndex = findedText.length - 1;
findedText = findedText.substring(0, lastIndex) + findedText[lastIndex];
}
return '"$findedText"';
}

Use below method
just pass String json data it will give Map data
jsonStringToMap(String data){
List<String> str = data.replaceAll("{","").replaceAll("}","").replaceAll("\"","").replaceAll("'","").split(",");
Map<String,dynamic> result = {};
for(int i=0;i<str.length;i++){
List<String> s = str[i].split(":");
result.putIfAbsent(s[0].trim(), () => s[1].trim());
}
return result;
}

Related

How to send a GET request with an array as a parameter?

I was trying to create a function to make a GET with query parameters. I was dealing with the Mangadex API and was to send a parameter called 'manga' as an array. I created the code as follows:
Future<http.Response> getCoverArtResponse(String mangaID) async {
var queryParameters = {
'limit': '10',
'manga': [mangaID] //Here
};
var unencodedPath = '/cover';
var response = await http.get(
Uri.https(authority, unencodedPath, queryParameters),
headers: {HttpHeaders.contentTypeHeader: 'application/json'});
return response;
}
However, the response was the following error:
{"result":"error","errors":[{"id":"9c346772-7b14-5982-b4b6-7b5888522762","status":400,"title":"validation_exception","detail":"Error validating \/manga: String value found, but an array is required","context":null}]}
How am I supposed to send the parameters? So far I have tried -
'manga': [mangaID]
'manga': '[$mangaID]'
None of them seem to work.
import 'dart:async';
import 'package:wnetworking/wnetworking.dart';
class MangaDex {
static const _base = 'https://api.mangadex.org';
static FutureOr<void> _getter({required String url, required Function(JMap item, int idx) onItem}) async {
await HttpReqService.getJson<JMap>(url)
.then((response) {
var results = response?['results'];
if (results != null) {
if (results is List) {
var i = 0;
results.forEach((manga) => onItem(manga, ++i));
} else {
print(response);
}
}
});
}
static FutureOr<void> cover({int limit = 10, int offset=0, String? mangaId, String? coverId}) async {
final mangas = mangaId != null ? '&manga[]=$mangaId' : '';
final covers = coverId != null ? '&ids[]=$coverId' : '';
final url = '$_base/cover?limit=$limit&offset=$offset$mangas$covers';
await _getter(
url: url,
onItem: (item, idx) {
print('$idx) "${item['data']?['attributes']?['fileName']}"');
print(' id: ${item['data']?['id']}\n');
},
);
}
}
void main(List<String> args) async {
await MangaDex.cover(mangaId: '32d76d19-8a05-4db0-9fc2-e0b0648fe9d0', limit: 2);
print('\nJob done');
}
Result:
1) "f5873770-80a4-470e-a11c-63b709d87eb3.jpg"
id: b6c7ce9c-e671-4f26-90b0-e592188e9cd6
2) "e9f926db-b469-48c4-8cc4-a8e523ad75ca.jpg"
id: 00aae6e0-46bb-4f92-a82a-1c740789b704
Job done
Replace wnetworking package with http package, and JMap with Map<String, dynamic>
NOTE: MangaDex Documentation is lacking and misleading about how to correctly use its endpoints.

How to get first character from words in flutter dart?

Let's say we have a name set to "Ben Bright". I want to output to the user "BB", with the first characters of each word. I tried with the split() method, but I failed to do it with dart.
String getInitials(bank_account_name) {
List<String> names = bank_account_name.split(" ");
String initials;
for (var i = 0; i < names.length; i++) {
initials = '${names[i]}';
}
return initials;
}
Allow me to give a shorter solution than the other mentioned:
void main() {
print(getInitials('')); //
print(getInitials('Ben')); // B
print(getInitials('Ben ')); // B
print(getInitials('Ben Bright')); // BB
print(getInitials('Ben Bright Big')); // BB
}
String getInitials(String bank_account_name) => bank_account_name.isNotEmpty
? bank_account_name.trim().split(' ').map((l) => l[0]).take(2).join()
: '';
The take(2) part ensures we only take up to two letters.
EDIT (7th October 2021):
Or if we must be able to handle multiple spaces between the words we can do (thanks #StackUnderflow for notice):
void main() {
print(getInitials('')); //
print(getInitials('Ben')); // B
print(getInitials('Ben ')); // B
print(getInitials('Ben Bright')); // BB
print(getInitials('Ben Bright Big')); // BB
print(getInitials('Ben Bright Big')); // BB
}
String getInitials(String bankAccountName) => bankAccountName.isNotEmpty
? bankAccountName.trim().split(RegExp(' +')).map((s) => s[0]).take(2).join()
: '';
Notice that split takes a RegExp(' +') compared to the original solution.
Just a slight modification since you only need the first letters
String getInitials(bank_account_name) {
List<String> names = bank_account_name.split(" ");
String initials = "";
int numWords = 2;
if(numWords < names.length) {
numWords = names.length;
}
for(var i = 0; i < numWords; i++){
initials += '${names[i][0]}';
}
return initials;
}
Edit:
You can set the value of num_words to print the intials of those many words.
If the bank_account_name is a 0 letter word, then return an empty string
If the bank_account_name contains lesser words than num_words, print the initials of all the words in bank_account_name.
var string = 'William Henry Gates';
var output = getInitials(string: string, limitTo: 1); // W
var output = getInitials(string: string, limitTo: 2); // WH
var output = getInitials(string: string); // WHG
String getInitials({String string, int limitTo}) {
var buffer = StringBuffer();
var split = string.split(' ');
for (var i = 0 ; i < (limitTo ?? split.length); i ++) {
buffer.write(split[i][0]);
}
return buffer.toString();
}
A more general solution can be found below. It takes care of empty strings, single word strings and situations where anticipated word count is less than actual word count:
static String getInitials(String string, {int limitTo}) {
var buffer = StringBuffer();
var wordList = string.trim().split(' ');
if (string.isEmpty)
return string;
// Take first character if string is a single word
if (wordList.length <= 1)
return string.characters.first;
/// Fallback to actual word count if
/// expected word count is greater
if (limitTo != null && limitTo > wordList.length) {
for (var i = 0; i < wordList.length; i++) {
buffer.write(wordList[i][0]);
}
return buffer.toString();
}
// Handle all other cases
for (var i = 0; i < (limitTo ?? wordList.length); i++) {
buffer.write(wordList[i][0]);
}
return buffer.toString();
}
Edit:
I actually use this for CircleAvatars with no images in my projects.
I used CopsOnRoad solution but I was getting the following error.
RangeError (index): Invalid value: Only valid value is 0: 1
So I modified it to
String getInitials(String string, [int limitTo = 2]) {
if (string == null || string.isEmpty) {
return '';
}
var buffer = StringBuffer();
var split = string.split(' ');
//For one word
if (split.length == 1) {
return string.substring(0, 1);
}
for (var i = 0; i < (limitTo ?? split.length); i++) {
buffer.write(split[i][0]);
}
return buffer.toString();
}
Here are some tests in case you are interested
void main() {
group('getInitials', () {
test('should process one later word name correctly', () {
final result = getInitials('J');
expect(result, 'J');
});
test('should process one word name correctly', () {
final result = getInitials('John');
expect(result, 'J');
});
test('should process two word name correctly', () {
final result = getInitials('John Mamba');
expect(result, 'JM');
});
test('should process more than two word name correctly', () {
final result = getInitials('John Mamba Kanzu');
expect(result, 'JM');
});
test('should return empty string when name is null', () {
final result = getInitials(null);
expect(result, '');
});
test('should return empty string when name is empty', () {
final result = getInitials('');
expect(result, '');
});
});
}
String getInitials(full_name) {
List<String> names = full_name.split(" ");
print("org::: $full_name");
print("list ::: $names");
print("Substring ::: ${names[0].substring(0,1)}");
String initials = "";
int numWords = 2;
numWords = names.length;
for(var i = 0; i < numWords; i++)
{
initials += '${names[i].substring(0,1)}';
print("the initials are $initials");
}
return initials;
}
On Nov, 2022
Working solution using Regex:
String getInitials(String string) => string.isNotEmpty
? string.trim().split(RegExp(' +')).map((s) => s[0]).join()
: '' ;

Xamarin.ios: Detail command is null because it shows before function

I'm building an iOS app with Xamarin.ios MvvmCross. And I have a function that puts a random id in a text file every day. So I get a recipe of the day.
The problem is that the code for the Detail command function (for the button) runs before the function that stores everything in the text file. So the detail command returns null and nothing happens when I push the button. The second time I run the code it does what it should do because there's already an id stored in the text file.
The view:
public override void ViewDidLoad()
{
base.ViewDidLoad();
MvxFluentBindingDescriptionSet<TabHomeView, TabHomeViewModel> set = new MvxFluentBindingDescriptionSet<TabHomeView, TabHomeViewModel>(this);
set.Bind(MorningImage).For(img => img.Image).To(res => res.MorningContent.picture).WithConversion<StringToImageConverter>();
set.Bind(MorningJuiceName).To(vm => vm.MorningContent.name);
set.Bind(MorningBtn)
.To(vm => vm.NavigateToMorningJuice);
set.Apply();
}
The function to put a random id in the text file:
public async void GetAfternoonJuice()
{
Recipes = await _recipeService.GetRecipes();
int counter = Recipes.Count;
Random rnd = new Random();
int RandomNumber = rnd.Next(1, counter);
string rndNumToStr = RandomNumber.ToString();
DateTime dateAndTime = DateTime.Now;
string day = dateAndTime.ToString("dd/MM/yyyy");
string folderValue = (day + "," + rndNumToStr);
var _folderName = "TextFilesFolder2";
var _fileName = "AfternoonJuice";
if (!_fileStore.FolderExists(_folderName))
_fileStore.EnsureFolderExists(_folderName);
//Content van de file uitlezen
string value = string.Empty;
_fileStore.TryReadTextFile(_folderName + "/" + _fileName, out (value));
string CheckFileContent = value;
string[] TextFileList;
//Als er niets in zit, default data in steken
if (CheckFileContent == null)
{
_fileStore.WriteFile(_folderName + "/" + _fileName, "00/00/00,0");
string d = "00/00/00,0";
TextFileList = d.Split(',');
}
else
{
TextFileList = CheckFileContent.Split(',');
}
if (TextFileList[0] != day)
{
//File verwijderen om overbodige data te verwijderen.
_fileStore.DeleteFile(_folderName + "/" + _fileName);
//File aanmaken.
if (!_fileStore.FolderExists(_folderName))
_fileStore.EnsureFolderExists(_folderName);
_fileStore.WriteFile(_folderName + "/" + _fileName, folderValue);
string NewValue = string.Empty;
_fileStore.TryReadTextFile(_folderName + "/" + _fileName, out (NewValue));
string NValue = NewValue;
List<string> NewTextFileList = new List<string>(
NValue.Split(new string[] { "," }, StringSplitOptions.None));
int numVall = Int32.Parse(NewTextFileList[1]);
int NewRandomValue = numVall;
AfternoonContent = await _recipeService.GetRecipeById(NewRandomValue);
RaisePropertyChanged(() => AfternoonContent);
}
else
{
int numVall = Int32.Parse(TextFileList[1]);
int NewRandomValue = numVall;
AfternoonContent = await _recipeService.GetRecipeById(NewRandomValue);
RaisePropertyChanged(() => AfternoonContent);
}
}
The detail command:
public MvxCommand<Recipe> NavigateToAfternoonJuice
{
get
{
var _folderName = "TextFilesFolder2";
var _fileName = "AfternoonJuice";
string value = string.Empty;
_fileStore.TryReadTextFile(_folderName + "/" + _fileName, out (value));
string fV = value;
List<string> TextFileList = new List<string>(
fV.Split(new string[] { "," }, StringSplitOptions.None));
int numVall = Int32.Parse(TextFileList[1]);
int NewRandomValue = numVall;
return new MvxCommand<Recipe>(SelectedRecipe =>
{
ShowViewModel<DetailJuiceListViewModel>(new { RecipeId = NewRandomValue });
});
}
}
Some of code in your public property NavigateToAfternoonJuice runs before your command is executed. It will be run, when the binding occurs and not when the command actually executes the body.
You probably want to modify your command to something as follows instead.
private MvxCommand<Recipe> _navigateToAfternoonJuice;
public MvxCommand<Recipe> NavigateToAfternoonJuice
{
get
{
if (_navigateToAfternoonJuice == null)
_navigateToAfternoonJuice = new MvxCommand<Recipe>(DoNavigateToAfternoonJuice);
return _navigateToAfternoonJuice;
}
}
private void DoNavigateToAfternoonJuice(Reciepe selectedRecipe)
{
var _folderName = "TextFilesFolder2";
var _fileName = "AfternoonJuice";
string value = string.Empty;
_fileStore.TryReadTextFile(_folderName + "/" + _fileName, out (value));
string fV = value;
List<string> TextFileList = new List<string>(
fV.Split(new string[] { "," }, StringSplitOptions.None));
int numVall = Int32.Parse(TextFileList[1]);
int NewRandomValue = numVall;
ShowViewModel<DetailJuiceListViewModel>(new { RecipeId = NewRandomValue });
}
This will make the text file to be read when the command executes.

Dart json.encode returns json string with key values without quotes

I am trying to convert a dictionary to json string. However I am not getting quotes around any of the strings. I am using dart 2 . Here is what I have
var resBody = {};
resBody["email"] = "employerA#gmail.com";
resBody["password"] = "admin123";
var user = {};
user["user"] = resBody;
String str = json.encode(user);
Output is:
{user: {email: employerA#gmail.com, password: admin123}}
I would like this to be like an actual json object
{"user": {"email": "employerA#gmail.com", "password: admin123"}}
How can I tell dart to put quotes around it ?
I looked at this thread and am doing exactly what works for the user
Am I doing something wrong ?
This is working as expected
import 'dart:convert';
void main() {
var resBody = {};
resBody["email"] = "employerA#gmail.com";
resBody["password"] = "admin123";
var user = {};
user["user"] = resBody;
String str = json.encode(user);
print(str);
}
prints
{"user":{"email":"employerA#gmail.com","password":"admin123"}}
DartPad example
[or]
import 'dart:convert';
void main() {
const JsonEncoder encoder = JsonEncoder.withIndent(' ');
try {
var resBody = {};
resBody["email"] = "employerA#gmail.com";
resBody["password"] = "admin123";
var user = {};
user["user"] = resBody;
String str = encoder.convert(user);
print(str);
} catch(e) {
print(e);
}
}
which gives you the beautified output
{
"user": {
"email": "employerA#gmail.com",
"password": "admin123"
}
}

Crawler4j With Grails App

I am making a crawler application in Groovy on Grails. I am using Crawler4j and following this tutorial.
I created a new grails project
Put the BasicCrawlController.groovy file in controllers->package
Did not create any view because I expected on doing run-app, my crawled data would appear in my crawlStorageFolder (please correct me if my understanding is flawed)
After that I just ran the application by doing run-app but I didn't see any crawling data anywhere.
Am I right in expecting some file to be created at the crawlStorageFolder location that I have given as C:/crawl/crawler4jStorage?
Do I need to create any view for this?
If I want to invoke this crawler controller from some other view on click of a submit button of a form, can I just write <g:form name="submitWebsite" url="[controller:'BasicCrawlController ']">?
I asked this because I do not have any method in this controller, so is it the right way to invoke this controller?
My code is as follows:
//All necessary imports
public class BasicCrawlController {
static main(args) throws Exception {
String crawlStorageFolder = "C:/crawl/crawler4jStorage";
int numberOfCrawlers = 1;
//int maxDepthOfCrawling = -1; default
CrawlConfig config = new CrawlConfig();
config.setCrawlStorageFolder(crawlStorageFolder);
config.setPolitenessDelay(1000);
config.setMaxPagesToFetch(100);
config.setResumableCrawling(false);
PageFetcher pageFetcher = new PageFetcher(config);
RobotstxtConfig robotstxtConfig = new RobotstxtConfig();
RobotstxtServer robotstxtServer = new RobotstxtServer(robotstxtConfig, pageFetcher);
CrawlController controller = new CrawlController(config, pageFetcher, robotstxtServer);
controller.addSeed("http://en.wikipedia.org/wiki/Web_crawler")
controller.start(BasicCrawler.class, 1);
}
}
class BasicCrawler extends WebCrawler {
final static Pattern FILTERS = Pattern
.compile(".*(\\.(css|js|bmp|gif|jpe?g"+ "|png|tiff?|mid|mp2|mp3|mp4" +
"|wav|avi|mov|mpeg|ram|m4v|pdf" +"|rm|smil|wmv|swf|wma|zip|rar|gz))\$")
/**
* You should implement this function to specify whether the given url
* should be crawled or not (based on your crawling logic).
*/
#Override
boolean shouldVisit(WebURL url) {
String href = url.getURL().toLowerCase()
!FILTERS.matcher(href).matches() && href.startsWith("http://en.wikipedia.org/wiki/Web_crawler/")
}
/**
* This function is called when a page is fetched and ready to be processed
* by your program.
*/
#Override
void visit(Page page) {
int docid = page.getWebURL().getDocid()
String url = page.getWebURL().getURL()
String domain = page.getWebURL().getDomain()
String path = page.getWebURL().getPath()
String subDomain = page.getWebURL().getSubDomain()
String parentUrl = page.getWebURL().getParentUrl()
String anchor = page.getWebURL().getAnchor()
println("Docid: ${docid} ")
println("URL: ${url} ")
println("Domain: '${domain}'")
println("Sub-domain: ' ${subDomain}'")
println("Path: '${path}'")
println("Parent page:${parentUrl} ")
println("Anchor text: ${anchor} " )
if (page.getParseData() instanceof HtmlParseData) {
HtmlParseData htmlParseData = (HtmlParseData) page.getParseData()
String text = htmlParseData.getText()
String html = htmlParseData.getHtml()
List<WebURL> links = htmlParseData.getOutgoingUrls()
println("Text length: " + text.length())
println("Html length: " + html.length())
println("Number of outgoing links: " + links.size())
}
Header[] responseHeaders = page.getFetchResponseHeaders()
if (responseHeaders != null) {
println("Response headers:")
for (Header header : responseHeaders) {
println("\t ${header.getName()} : ${header.getValue()}")
}
}
println("=============")
}
}
I'll try to translate your code into a Grails standard.
Use this under grails-app/controller
class BasicCrawlController {
def index() {
String crawlStorageFolder = "C:/crawl/crawler4jStorage";
int numberOfCrawlers = 1;
//int maxDepthOfCrawling = -1; default
CrawlConfig crawlConfig = new CrawlConfig();
crawlConfig.setCrawlStorageFolder(crawlStorageFolder);
crawlConfig.setPolitenessDelay(1000);
crawlConfig.setMaxPagesToFetch(100);
crawlConfig.setResumableCrawling(false);
PageFetcher pageFetcher = new PageFetcher(crawlConfig);
RobotstxtConfig robotstxtConfig = new RobotstxtConfig();
RobotstxtServer robotstxtServer = new RobotstxtServer(robotstxtConfig, pageFetcher);
CrawlController controller = new CrawlController(crawlConfig, pageFetcher, robotstxtServer);
controller.addSeed("http://en.wikipedia.org/wiki/Web_crawler")
controller.start(BasicCrawler.class, 1);
render "done crawling"
}
}
Use this under src/groovy
class BasicCrawler extends WebCrawler {
final static Pattern FILTERS = Pattern
.compile(".*(\\.(css|js|bmp|gif|jpe?g"+ "|png|tiff?|mid|mp2|mp3|mp4" +
"|wav|avi|mov|mpeg|ram|m4v|pdf" +"|rm|smil|wmv|swf|wma|zip|rar|gz))\$")
/**
* You should implement this function to specify whether the given url
* should be crawled or not (based on your crawling logic).
*/
#Override
boolean shouldVisit(WebURL url) {
String href = url.getURL().toLowerCase()
!FILTERS.matcher(href).matches() && href.startsWith("http://en.wikipedia.org/wiki/Web_crawler/")
}
/**
* This function is called when a page is fetched and ready to be processed
* by your program.
*/
#Override
void visit(Page page) {
int docid = page.getWebURL().getDocid()
String url = page.getWebURL().getURL()
String domain = page.getWebURL().getDomain()
String path = page.getWebURL().getPath()
String subDomain = page.getWebURL().getSubDomain()
String parentUrl = page.getWebURL().getParentUrl()
String anchor = page.getWebURL().getAnchor()
println("Docid: ${docid} ")
println("URL: ${url} ")
println("Domain: '${domain}'")
println("Sub-domain: ' ${subDomain}'")
println("Path: '${path}'")
println("Parent page:${parentUrl} ")
println("Anchor text: ${anchor} " )
if (page.getParseData() instanceof HtmlParseData) {
HtmlParseData htmlParseData = (HtmlParseData) page.getParseData()
String text = htmlParseData.getText()
String html = htmlParseData.getHtml()
List<WebURL> links = htmlParseData.getOutgoingUrls()
println("Text length: " + text.length())
println("Html length: " + html.length())
println("Number of outgoing links: " + links.size())
}
Header[] responseHeaders = page.getFetchResponseHeaders()
if (responseHeaders != null) {
println("Response headers:")
for (Header header : responseHeaders) {
println("\t ${header.getName()} : ${header.getValue()}")
}
}
println("=============")
}
}

Resources