I am using nutch 1.15 to crawl the link that contains the zip files which comprises of file1.txt, file2.txt and file3.txt.
I used the parse-zip, parse-tika plugin in nutch "plugin.includes" but it is not able to crawl the content of the text files and index it.
Parsed content returns like this,
"content" : "file1.txt\nfile2.txt\nfile3.txt\n"
Why it is not able to get the contents of file1.txt, etc?
removed zip from regex-urlfilter.txt,
#-(?i)\.(gif|jpg|png|ico|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|exe|jpeg|bmp|js)$
-(?i)\.(gif|jpg|png|ico|css|sit|eps|wmf|ppt|mpg|xls|gz|rpm|tgz|mov|exe|jpeg|bmp|js)$
plugin.includes in nutch-site.xml:
<property>
<name>plugin.includes</name>
<value>protocol-http|protocol-httpclient|urlfilter-regex|parse-(html|text|tika|zip|metatags)|index-(basic|anchor|metadata)|query-(basic|site|url)|response-(json|xml)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)|indexer-elastic</value>
</property>
<property>
<name>http.content.limit</name>
<value>-1</value>
</property>
Parse-plugins.xml file:
<parse-plugins>
<!-- by default if the mimeType is set to *, or
if it can't be determined, use parse-tika -->
<mimeType name="*">
<plugin id="parse-tika" />
</mimeType>
<mimeType name="application/rss+xml">
<plugin id="parse-tika" />
<plugin id="feed" />
</mimeType>
<mimeType name="application/x-bzip2">
<!-- try and parse it with the zip parser -->
<plugin id="parse-zip" />
</mimeType>
<mimeType name="application/x-gzip">
<!-- try and parse it with the zip parser -->
<plugin id="parse-zip" />
</mimeType>
<mimeType name="application/x-javascript">
<plugin id="parse-js" />
</mimeType>
<mimeType name="application/x-shockwave-flash">
<plugin id="parse-swf" />
</mimeType>
<mimeType name="application/zip">
<plugin id="parse-zip" />
</mimeType>
<mimeType name="text/html">
<plugin id="parse-html" />
</mimeType>
<mimeType name="application/xhtml+xml">
<plugin id="parse-html" />
</mimeType>
<mimeType name="text/xml">
<plugin id="parse-tika" />
<plugin id="feed" />
</mimeType>
<!-- Types for parse-ext plugin: required for unit tests to pass. -->
<mimeType name="application/vnd.nutch.example.cat">
<plugin id="parse-ext" />
</mimeType>
<mimeType name="application/vnd.nutch.example.md5sum">
<plugin id="parse-ext" />
</mimeType>
<!-- alias mappings for parse-xxx names to the actual extension implementation
ids described in each plugin's plugin.xml file -->
<aliases>
<alias name="parse-tika"
extension-id="org.apache.nutch.parse.tika.TikaParser" />
<alias name="parse-ext" extension-id="ExtParser" />
<alias name="parse-html"
extension-id="org.apache.nutch.parse.html.HtmlParser" />
<alias name="parse-js" extension-id="JSParser" />
<alias name="feed"
extension-id="org.apache.nutch.parse.feed.FeedParser" />
<alias name="parse-swf"
extension-id="org.apache.nutch.parse.swf.SWFParser" />
<alias name="parse-zip"
extension-id="org.apache.nutch.parse.zip.ZipParser" />
</aliases>
</parse-plugins>
Am I missing any configuration in nutch side?
Related
Initially we were using angular cordova-ios version as 4.5.4 and while uploading the IPA to Apple Store Connect it gave UIWebView error hence, upgraded the cordova-ios platform to 5.1.1 and added cordova-plugin-wkwebview-engine plugin. Now on generating IPA , it shows blank screen and doesn't route to the login screen.
Version details are as below:
angular-devkit/architect 0.10.7
#angular-devkit/build-angular 0.10.7
#angular-devkit/build-optimizer 0.10.7
#angular-devkit/build-webpack 0.10.7
#angular-devkit/core 7.0.7
#angular-devkit/schematics 0.8.9
#angular/cli 6.2.9
#ngtools/webpack 7.0.7
#schematics/angular 0.8.9
#schematics/update 0.8.9
rxjs 5.5.12
typescript 2.9.2
webpack 4.19.1
cordova-ios 5.1.1
We used the below command to build:
ng build --prod --source-map=false && cordova build ios
Our config.xml file
<?xml version='1.0' encoding='utf-8'?>
<widget android-activityName="appname" android-versionCode="10002" id="appname" version="1.0.2" xmlns="http://www.w3.org/ns/widgets" xmlns:android="http://schemas.android.com/apk/res/android" xmlns:cdv="http://cordova.apache.org/ns/1.0">
<name>appname</name>
<description>
Description
</description>
<author email="Support#appname.com" href="http://appname.com">
appname
</author>
<content src="index.html" />
<allow-navigation href="*" />
<allow-intent href="*" />
<access origin="*" />
<edit-config file="app/src/main/AndroidManifest.xml" mode="merge" target="/manifest/application">
<application android:usesCleartextTraffic="true" />
</edit-config>
<allow-intent href="http://*/*" />
<allow-intent href="https://*/*" />
<allow-intent href="tel:*" />
<allow-intent href="sms:*" />
<allow-intent href="mailto:*" />
<allow-intent href="geo:*" />
<platform name="android">
<allow-intent href="market:*" />
<icon density="ldpi" src="res/icon/android/ldpi.png" />
<preference name="CustomURLSchemePluginClearsAndroidIntent" value="true" />
</platform>
<platform name="ios">
<allow-intent href="itms:*" />
<allow-intent href="itms-apps:*" />
<icon density="ldpi" src="res/icon/android/ldpi.png" />
</platform>
<hook src="hooks/appBeforeBuild.js" type="before_build" />
<hook src="hooks/appBeforeBuild.js" type="before_run" />
<engine name="browser" spec="^5.0.3" />
<plugin name="cordova-plugin-dbcopy" spec="^2.1.2" />
<plugin name="cordova-plugin-whitelist" spec="^1.3.3" />
<plugin name="cordova-plugin-network-information" spec="^2.0.1" />
<plugin name="uk.co.workingedge.cordova.plugin.sqliteporter" spec="^1.1.0" />
<plugin name="cordova-plugin-device" spec="^2.0.2" />
<plugin name="cordova-plugin-printer" spec="^0.7.3" />
<plugin name="cordova-plugin-file-transfer" spec="^1.7.1" />
<plugin name="cordova-plugin-customurlscheme" spec="^4.4.0">
<variable name="URL_SCHEME" value="URL_SCHEME_NAME" />
<variable name="ANDROID_SCHEME" value=" " />
<variable name="ANDROID_HOST" value=" " />
<variable name="ANDROID_PATHPREFIX" value="/" />
</plugin>
<plugin name="cordova-plugin-dialogs" spec="^1.3.3" />
<plugin name="cordova-plugin-bluetoothle" spec="^4.5.10" />
<plugin name="cordova-plugin-bluetooth-peripheral-usage-description" spec="^1.0.1">
<variable name="TEXT" value="Some string" />
</plugin>
<plugin name="cordova-plugin-geolocation" spec="^4.0.2" />
<plugin name="cordova-plugin-android-permissions" spec="^1.0.2" />
<plugin name="cordova-plugin-ble-central" spec="^1.2.5" />
<plugin name="cordova-plugin-inappbrowser" spec="^3.2.0" />
<plugin name="cordova-plugin-permission" spec="^0.1.0" />
<plugin name="cordova-plugin-vibration" spec="^3.1.1" />
</widget>
Unfortunately, white screen errors are pretty generic, as the error condition isn't listed in the console (see Cordova - white screen after splash, no exceptions in console).
You can, however, launch the app in the iOS simulator and hook it up to Safari's dev tools, and then hit refresh. It should kick out the exception that iOS is running into.
Hi i'm trying to compile my Ionic 3 app to iOS and it does not seams to be working.
The compilation works fine, however, when i'm trying to test the app using an iOS emulator, I got a white screen.
There is no error in the console, nor in the XCode error log.
I know the code is Ok since it's working perfeclty on android and browser.
I have tried to remove and reinstall the iOS platform, to remove and reisntall the node_module folder with no success.
My guess is, that the code isn't even loaded since when i'm trying to inspect the emulator using Safari Developer tool, the title of the page is index.html rather than the title i've set on the login page.
Here are the relevents files :
My config.xml
<?xml version='1.0' encoding='utf-8'?>
<widget id="com.mycompany.myAwesomeApp" version="3.0.2" xmlns="http://www.w3.org/ns/widgets" xmlns:cdv="http://cordova.apache.org/ns/1.0">
<name>my awesome app</name>
<description>My awesome app description</description>
<author email="support#myCompany.com" href="https://myCompany.com">My company dev team</author>
<content src="index.html" />
<allow-intent href="http://*/*" />
<allow-intent href="https://*/*" />
<allow-intent href="tel:*" />
<allow-intent href="sms:*" />
<allow-intent href="mailto:*" />
<allow-intent href="geo:*" />
<preference name="ScrollEnabled" value="false" />
<preference name="android-minSdkVersion" value="19" />
<preference name="BackupWebStorage" value="none" />
<preference name="SplashMaintainAspectRatio" value="true" />
<preference name="FadeSplashScreenDuration" value="300" />
<preference name="SplashShowOnlyFirstTime" value="false" />
<preference name="SplashScreen" value="screen" />
<preference name="SplashScreenDelay" value="3000" />
<preference name="loadUrlTimeoutValue" value="60000" />
<preference name="CordovaWebViewEngine" value="CDVWKWebViewEngine" />
<platform name="android">
<!-- irrelevent android icons -->
</platform>
<platform name="ios">
<allow-intent href="itms:*" />
<allow-intent href="itms-apps:*" />
<!-- irrelevent icon files -- >
</platform>
<allow-navigation href="http://*/*" />
<allow-navigation href="https://*/*" />
<plugin name="phonegap-plugin-push" spec="~2.2.2" />
<plugin name="cordova-plugin-console" spec="^1.1.0" />
<plugin name="cordova-plugin-device" spec="^1.1.7" />
<plugin name="cordova-plugin-splashscreen" spec="^4.1.0" />
<plugin name="cordova-plugin-statusbar" spec="^2.4.1" />
<plugin name="cordova-plugin-camera" spec="^4.0.2" />
<plugin name="cordova-plugin-geolocation" spec="^4.0.1" />
<plugin name="cordova-plugin-inappbrowser" spec="^2.0.2" />
<plugin name="ionic-plugin-keyboard" spec="^2.2.1" />
<plugin name="cordova-plugin-x-socialsharing" spec="^5.3.2" />
<plugin name="cordova-plugin-headercolor" spec="^1.0.0" />
<plugin name="cordova-plugin-network-information" spec="^2.0.1" />
<plugin name="cordova-plugin-file" spec="^6.0.1" />
<plugin name="cordova-plugin-advanced-http" spec="2.0.4" />
<plugin name="cordova-plugin-ionic-webview" spec="^4.0.1">
<variable name="ANDROID_SUPPORT_ANNOTATIONS_VERSION" value="27.+" />
</plugin>
<plugin name="cordova-plugin-whitelist" spec="^1.3.3" />
<engine name="android" spec="7.1.4" />
<engine name="browser" spec="5.0.1" />
<engine name="ios" spec="4.5.5" />
Ionic Info :
Ionic:
ionic (Ionic CLI) : 4.12.0 (/Users/nic/.nvm/versions/node/v10.15.3/lib/node_modules/ionic)
Ionic Framework : ionic-angular 3.9.0
#ionic/app-scripts : 3.1.4
Cordova:
cordova (Cordova CLI) : 9.0.0 (cordova-lib#9.0.1)
Cordova Platforms : android 7.1.4, ios 4.5.5
Cordova Plugins : cordova-plugin-ionic-webview 1.2.1, (and 16 other plugins)
System:
NodeJS : v10.15.3 (/Users/nic/.nvm/versions/node/v10.15.3/bin/node)
npm : 6.4.1
OS : macOS High Sierra
Xcode : Xcode 10.0 Build version 10A255
I ended up finding my problem. I was loading code in the constructor rather than in an ionViewDidLoad function. I have no idea why this works on Android, but not on iOS.
I just cant consistently get a network connection for my ionic app on my ios device. It works sometimes with the reset, remove&add platforms but recently it just doesnt work. Im trying it over 10 times already. Its crazy.
How do i get network working consistently on ionic ? (cordova-plugin-network-information)
Im using ionic version 1.7.14
I'ved tried several ways like
ionic state reset
or
ionic platform remove ios
ionic platform add ios
I'ved got this error:
Error: undefined is not an object (evaluating 'navigator.connection.type')
in my config.xml, here are my plugins.
<plugin name="cordova-plugin-device" spec="~1.1.1" />
<plugin name="cordova-plugin-network-information" spec="~1.2.0" />
<plugin name="cordova-plugin-image-picker" spec="https://github.com/wymsee/cordova-imagePicker.git" />
<plugin name="cordova-plugin-whitelist" spec="~1.2.1" />
<plugin name="cordova-plugin-console" spec="~1.0.2" />
<plugin name="cordova-plugin-file-transfer" spec="~1.5.0" />
<plugin name="com.raananw.imageResizer" spec="https://github.com/RaananW/PhoneGap-Image-Resizer" />
<plugin name="cordova-plugin-inappbrowser" spec="~1.2.0" />
<plugin name="ionic-plugin-keyboard" spec="~2.0.1" />
<plugin name="cordova-plugin-splashscreen" spec="~3.1.0" />
<plugin name="cordova-plugin-camera" spec="~2.1.1" />
<plugin name="cordova-plugin-actionsheet" spec="https://github.com/EddyVerbruggen/cordova-plugin-actionsheet.git" />
<plugin name="cordova-plugin-app-version" spec="https://github.com/whiteoctober/cordova-plugin-app-version.git" />
<plugin name="onesignal-cordova-plugin" spec="~1.12.5" />
I've got the following entries in my Web.config file in an asp.net mvc application:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<configSections>
<section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler,log4net" />
...
</configSections>
<log4net>
<appender name="PublicAccessAppender" type="log4net.Appender.RollingFileAppender">
<datePattern value="'C:\Users\my_user_name\Documents\Temp\logs\public-access.'yyyy-MM-dd'.log'" />
<staticLogFileName value="false" />
<appendToFile value="true" />
<rollingStyle value="Date" />
<maxSizeRollBackups value="5" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%date [%thread] %-5level %logger - %message%newline" />
</layout>
</appender>
<root>
<level value="DEBUG" />
<appender-ref ref="PublicAccessAppender" />
</root>
</log4net>
...
</configuration>
I hope it's pretty self-explanatory what I'm trying to achieve, but when I run the application (hosted in IIS), I get no log file. FWIW, the directory hierarchy exists up to Temp folder, and I'd like log4net to generate the rest of the directories/files in the path.
I've added the log4net nuget package to my application, and I'm logging with the INFO level.
What am I missing here?
I think you can't put full path into datePattern, there must be just YYYYmmdd and things like that. Put the file path into file element:
<file value="C:\Users\my_user_name\Documents\Temp\logs\public-access.log" />
<datePattern value="yyyyMMdd" />
<preserveLogFileNameExtension value="true" />
The last element forces to put datePattern before the .log extension which was probably your original goal..
Here's the working appender configuration:
<appender name="PublicAccessAppender" type="log4net.Appender.RollingFileAppender">
<file value="C:\temp\my_user_name\Documents\Temp\logs\app.log" />
<datePattern value=".yyyy-MM-dd" /><!-- can be omitted as it's the default datePattern value -->
<rollingStyle value="Date" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%date [%thread] %-5level %logger - %message%newline" />
</layout>
</appender>
Note to readers struggling with log4net configuration: asp.net hosted in IIS pretty much has 0 write permssions, so the issue you're most likely struggling with is that your web app simply doesn't have permission to write to the log file.
That's what was happening for me, and I noticed it by inspecting the contents of log4net.LogManager.GetRepository().ConfigurationMessages in the debugger after calling .Configure().
I followed this post to give my web app the necessary persmissions to write to the log file (namely Read/Write/Modify).
Having trouble getting the database target for NLog to work. I have 2 targets: FILE and DB.. here is my NLog.config:
<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
autoReload="true"
throwExceptions="false">
<variable name="appName" value="KoreCMS" />
<targets async="true">
<target xsi:type="File"
name="default"
layout="${longdate} - ${level:uppercase=true}: ${message}${onexception:${newline}EXCEPTION\: ${exception:format=ToString}}"
fileName="${basedir}\App_Data\Log\Debug.log"
keepFileOpen="false"
archiveFileName="${basedir}\App_Data\Log\Debug_${shortdate}.{##}.log"
archiveNumbering="Sequence"
archiveEvery="Day"
maxArchiveFiles="30"/>
<target xsi:type="Database"
name="database"
keepConnection="true"
useTransactions="true"
dbProvider="System.Data.SqlClient"
connectionStringName="DefaultConnection"
commandText="INSERT INTO Kore_Log(EventDateTime, EventLevel, UserName, MachineName, EventMessage, ErrorSource, ErrorClass, ErrorMethod, ErrorMessage, InnerErrorMessage) VALUES (#EventDateTime, #EventLevel, #UserName, #MachineName, #EventMessage, #ErrorSource, #ErrorClass, #ErrorMethod, #ErrorMessage, #InnerErrorMessage)">
<parameter name="#EventDateTime" layout="${date:s}" />
<parameter name="#EventLevel" layout="${level}" />
<parameter name="#UserName" layout="${aspnet-user-identity}" />
<parameter name="#MachineName" layout="${machinename}" />
<parameter name="#EventMessage" layout="${message}" />
<parameter name="#ErrorSource" layout="${event-context:item=error-source}" />
<parameter name="#ErrorClass" layout="${event-context:item=error-class}" />
<parameter name="#ErrorMethod" layout="${event-context:item=error-method}" />
<parameter name="#ErrorMessage" layout="${event-context:item=error-message}" />
<parameter name="#InnerErrorMessage" layout="${event-context:item=inner-error-message}" />
</target>
</targets>
<rules>
<logger name="*" writeTo="default" minlevel="Info" />
<logger name="*" writeTo="database" minlevel="Info" />
</rules>
</nlog>
And here is the DB design:
And here is the connection string:
<connectionStrings>
<add name="DefaultConnection" connectionString="Server=.\SQLExpress2012;Initial Catalog=MyDatabaseName;Integrated Security=True;Persist Security Info=True;MultipleActiveResultSets=True"
providerName="System.Data.SqlClient" />
</connectionStrings>
What am I missing?
Alright, I figured it out... When I opened up SQL Profiler, I could see the INSERT command was indeed being sent, so I attempted to run the same command myself manually and discovered that I was being an idiot; basically, my Id column is setup as a uniqueidentifier and so of course it could not be auto incremented. The solution was to change the commandText in the Nlog.config to the following:
commandText="INSERT INTO Kore_Log(Id, EventDateTime, EventLevel, UserName, MachineName, EventMessage, ErrorSource, ErrorClass, ErrorMethod, ErrorMessage, InnerErrorMessage) VALUES (NEWID(), #EventDateTime, #EventLevel, #UserName, #MachineName, #EventMessage, #ErrorSource, #ErrorClass, #ErrorMethod, #ErrorMessage, #InnerErrorMessage)">
I got into same problem, reason was dbProvider value and connectionStringName key in nlog.config. I got it from here
<target name="database" xsi:type="Database" connectionString="${configsetting:name=sqlconnection.connectionString}"
dbProvider="Microsoft.Data.SqlClient.SqlConnection, Microsoft.Data.SqlClient">
Also,
connectionString should be used in place of connectionStringName
if Microsoft.Data.SqlClient is not added in dependency, it need to be added either using nuget package manager or manually.