I am new to the social network analysis and twitter API.I wanted to collect tweets on certain topic .So i have written the following code
package com;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.util.List;
import twitter4j.Query;
import twitter4j.QueryResult;
import twitter4j.Status;
import twitter4j.Tweet;
import twitter4j.Twitter;
import twitter4j.TwitterException;
import twitter4j.TwitterFactory;
public class TwitterSearchAdvance {
public static void main(String[] vishal) throws TwitterException,
IOException {
// List<Tweet> Data = new ArrayList<Tweet>();
StringBuffer stringbuffer = new StringBuffer();
Twitter twitter = new TwitterFactory().getInstance();
for (int page = 0; page <= 4; page++) {
Query query = new Query("Airtel");
// query.setRpp(100); // 100 results per page
QueryResult qr = twitter.search(query);
List<Status> qrTweets = qr.getTweets();
System.out.println("-------------------" + qrTweets.size());
// break out if there are no more tweets
for (Status t : qrTweets) {
System.out.println(t.getCreatedAt() + ": " + t.getText());
stringbuffer.append(t);
stringbuffer.append("\n");
BufferedWriter bw = new BufferedWriter(new FileWriter(new File(
"/home/vishal/FirstDocu.txt"), true));
bw.write(stringbuffer.toString());
bw.newLine();
// bw.write(t.getCreatedAt() + ": " + t.getText());
bw.close();
}
}
}
}
But when i run the following program the following error starts coming
Exception in thread "main" java.lang.IllegalStateException: Authentication credentials are missing. See http://twitter4j.org/configuration.html for the detail.
at twitter4j.TwitterBaseImpl.ensureAuthorizationEnabled(TwitterBaseImpl.java:200)
at twitter4j.TwitterImpl.get(TwitterImpl.java:1833)
at twitter4j.TwitterImpl.search(TwitterImpl.java:282)
at com.TwitterSearchAdvance.main(TwitterSearchAdvance.java:28)
Where do i need to provide credentials in my program
Thanks
Have a look at the options here http://twitter4j.org/en/configuration.html
There are a number of ways to provide credentials to your program:
Properties File
ConfigurationBuilder class
System properties
Environment Variables
All details and instructions can be found in the link
Related
But where do you change the font size of the method
HtmlConverter.convertToElements?
Why do I get this result with this code result code after execute ?
because the text between the tags does not change size ?
I also tried these solutions using the various methods (but the size does not change) :
https://stackoverflow.com/a/59044415/18323778
itext7 set font and size of HtmlConverter elements (In this question it is explained how to use the css too I am interested in changing only 2 properties of the element)
Run this code:
import com.itextpdf.html2pdf.ConverterProperties;
import com.itextpdf.html2pdf.HtmlConverter;
import com.itextpdf.html2pdf.resolver.font.DefaultFontProvider;
import com.itextpdf.io.font.constants.StandardFonts;
import com.itextpdf.kernel.font.PdfFont;
import com.itextpdf.kernel.font.PdfFontFactory;
import com.itextpdf.kernel.pdf.PdfDictionary;
import com.itextpdf.kernel.pdf.PdfDocument;
import com.itextpdf.kernel.pdf.PdfName;
import com.itextpdf.kernel.pdf.PdfWriter;
import com.itextpdf.layout.Document;
import com.itextpdf.layout.Style;
import com.itextpdf.layout.element.IElement;
import com.itextpdf.layout.element.Paragraph;
import com.itextpdf.layout.font.FontProvider;
import com.itextpdf.layout.properties.Property;
import com.itextpdf.layout.properties.UnitValue;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.text.SimpleDateFormat;
import java.util.List;
/**
*
* #author UC9001309
*/
public class TestConvertHtml2pdf {
/**
* #param args the command line arguments
*/
public static void main(String[] args) throws FileNotFoundException, IOException {
PdfWriter pdfWriter = new PdfWriter("C:\\Temp\\" + new SimpleDateFormat("yyyyMMddHHmmss").format(new java.util.Date()) + ".pdf");
PdfDocument pdfDocument = new PdfDocument(pdfWriter);
Document document = new Document(pdfDocument);
FontProvider provider = new DefaultFontProvider(true, false, false);
ConverterProperties cvProp = new ConverterProperties();
cvProp.setFontProvider(provider);
cvProp.setImmediateFlush(true);
List<IElement> lst = HtmlConverter.convertToElements("Testo <b><u><i>prova</i></u></b> con tag",cvProp);
lst.get(0).setProperty(Property.FONT_SIZE,UnitValue.createPointValue(Float.parseFloat("20")));
Paragraph p = (Paragraph) lst.get(0);
p.setProperty(Property.FONT_SIZE,UnitValue.createPointValue(Float.parseFloat("20")));
document.add(p);
document.close();
}
}
i tried various methods but still don't get results, does anyone know how to fix ?
Thanks for your help .
The goal is to create a method in Itext 7 to pass as parameters a string with html tag and font and font size, get it translated by the html parser to apply bold, italics, uderline without setting them with Itex commands as paragraph. selBold ec ..
Because my colleagues get tired of programming.
After several attempts I solved this (I create a main to try):
package testconverthtml2pdf;
import com.itextpdf.html2pdf.ConverterProperties;
import com.itextpdf.html2pdf.HtmlConverter;
import com.itextpdf.html2pdf.resolver.font.DefaultFontProvider;
import com.itextpdf.io.font.constants.StandardFonts;
import com.itextpdf.kernel.pdf.PdfDocument;
import com.itextpdf.kernel.pdf.PdfWriter;
import com.itextpdf.layout.Document;
import com.itextpdf.layout.element.IElement;
import com.itextpdf.layout.element.Paragraph;
import com.itextpdf.layout.font.FontProvider;
import com.itextpdf.layout.properties.UnitValue;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.text.SimpleDateFormat;
import java.util.List;
/**
*
* #author UC9001309
*/
public class TestConvertHtml2pdf {
/**
* #param args the command line arguments
*/
public static void main(String[] args) throws FileNotFoundException, IOException {
// Future parameters of the method change it to test !!!
float vardim = 10.5f;
String tipofont =StandardFonts.COURIER.toString().toLowerCase();
String strInput = "Testo <b><u><i>prova</i></u></b> con tag";
// -------------------------------------------------------------
PdfWriter pdfWriter = new PdfWriter("C:\\Temp\\" + new SimpleDateFormat("yyyyMMddHHmmss").format(new java.util.Date()) + ".pdf");
PdfDocument pdfDocument = new PdfDocument(pdfWriter);
Document document = new Document(pdfDocument);
FontProvider provider = new DefaultFontProvider(true, false, false);
ConverterProperties cvProp = new ConverterProperties();
cvProp.setFontProvider(provider);
cvProp.setImmediateFlush(true);
StringBuilder stringBuilder = new StringBuilder();
stringBuilder.append("<p style='font-family:"+tipofont +";font-size:"+UnitValue.createPointValue(vardim)+"'>") ;
stringBuilder.append(strInput) ;
stringBuilder.append("</p>") ;
System.out.println(stringBuilder);
List<IElement> lst = HtmlConverter.convertToElements(stringBuilder.toString(),cvProp);
Paragraph p = (Paragraph) lst.get(0);
document.add(p);
document.close();
}
}
To set Font and Font-Size to the input string I add a with a Style attribute inside for Font and Font-Size taken from parameter.
stringBuilder.append("<p style='font-family:"+tipofont +";font-size:"+UnitValue.createPointValue(vardim)+"'>") ;
tipofont -----> Font parameter
vardim -----> Font size parameter
I have to thank André Lemos who in his comment (lost because instead of modifying the question I deleted and recreated) gave me a solution because the bold was not created because the font type that had bold as its attribute was missing. I loaded Currier but not Currier-Bold.
If this solution can help anyone, it is available to the community.
I have a program that connects to a user defined URL from a TextField and scrapes the images on that web page. The user defined URL is gotten from the textfield via .getText() and assigned to a String. The String is then used to connect to the Web page with JSoup and puts the webpage into a document.
String address = labelforAddress.getText();
try {
document = Jsoup.connect(address).get();
}catch(IOException ex){
ex.printStackTrace();
}
I've tried differently formatted URLS: "https://www.", "www.", "https://" but everything I use throws the malformed URL error.
Someone please show me how to get the text from the TextField the correct way.
Below is the entire code.
package sample;
import javafx.application.Application;
import javafx.fxml.FXMLLoader;
import javafx.scene.Parent;
import javafx.scene.Scene;
import javafx.scene.control.Button;
import javafx.scene.control.Label;
import javafx.scene.control.TextField;
import javafx.scene.layout.GridPane;
import javafx.stage.FileChooser;
import javafx.stage.Stage;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.*;
import java.net.URL;
import java.util.ArrayList;
import java.util.LinkedList;
import java.util.List;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
public class Main extends Application {
Document document;
LinkedList<String> imageURLList = new LinkedList<String>();
ArrayList<File> fileList = new ArrayList<File>();
int fileCount = 1;
#Override
public void start(Stage primaryStage) throws Exception{
Parent root = FXMLLoader.load(getClass().getResource("sample.fxml"));
primaryStage.setTitle("Webpage Photo Scraper");
GridPane gp = new GridPane();
Label labelforAddress = new Label("URL");
GridPane.setConstraints(labelforAddress, 0,0);
TextField URLAddress = new TextField();
GridPane.setConstraints(URLAddress, 1,0);
Button scrape = new Button("Scrape for Photos");
GridPane.setConstraints(scrape, 0,1);
scrape.setOnAction(event->{
String address = labelforAddress.getText();
try {
document = Jsoup.connect(address).get();
}catch(IOException ex){
ex.printStackTrace();
}
Elements imgTags = document.getElementsByAttributeValueContaining("src", "/CharacterImages");
for(Element imgTag: imgTags){
imageURLList.add(imgTag.absUrl("src"));
}
for(String url: imageURLList){
File file = new File("C:\\Users\\Andrei\\Documents\\file" + fileCount + ".txt");
downloadFromURL(url, file);
fileList.add(file);
fileCount++;
}
});
Button exportToZipFile = new Button("Export to Zip File");
GridPane.setConstraints(exportToZipFile, 0,2);
exportToZipFile.setOnAction(event -> {
FileChooser fileChooser = new FileChooser();
FileChooser.ExtensionFilter exfilt = new FileChooser.ExtensionFilter("Zip Files", ".zip");
fileChooser.getExtensionFilters().add(exfilt);
try{
FileOutputStream fos = new FileOutputStream(fileChooser.showSaveDialog(primaryStage));
ZipOutputStream zipOut = new ZipOutputStream(fos);
for(int count = 0; count<=fileList.size()-1; count++){
File fileToZip = new File(String.valueOf(fileList.get(count)));
FileInputStream fis = new FileInputStream(fileToZip);
ZipEntry zipEntry = new ZipEntry(fileToZip.getName());
zipOut.putNextEntry(zipEntry);
byte[] bytes = new byte[1024];
int length;
while((length = fis.read(bytes)) >= 0) {
zipOut.write(bytes, 0, length);
}
fis.close();
}
zipOut.close();
fos.close();
}catch(IOException e1){
e1.printStackTrace();
}
});
primaryStage.setScene(new Scene(gp, 300, 275));
primaryStage.show();
gp.getChildren().addAll(exportToZipFile, labelforAddress, scrape, URLAddress);
}
public static void downloadFromURL(String url, File file){
try {
URL Url = new URL(url);
BufferedInputStream bis = new BufferedInputStream(Url.openStream());
FileOutputStream fis = new FileOutputStream(file);
byte[] buffer = new byte[1024];
int count = 0;
while((count = bis.read(buffer, 0,1024)) !=-1){
fis.write(buffer, 0, count);
}
fis.close();
bis.close();
}catch(IOException e){
e.printStackTrace();
}
}
public static void main(String[] args) {
launch(args);
}
}
Your text field containing the value entered by user is stored in URLAddress object but you always try to get the url from labelforAddress object which is a label always containing "URL" text.
So the solution is to use:
String address = URLAddress.getText();
If you read carefully error message it would help you to find the cause, because it always displays the value it considers wrong. In this case I see:
Caused by: java.net.MalformedURLException: no protocol: URL
and it shows the unrecognized address is: URL.
If you encounter this kind of error next time try:
debugging the aplication in runtime to see values of each variable
logging variable values in the console to see if variables contain values you expect
I've been working on Kafka twitter streaming feed data.
I'm following the sample from below link:
http://www.hahaskills.com/tutorials/kafka/Twitter_doc.html
I'm able to use Producer code and it is working fine. Able to get twitter feed and send to Kafka Producer.
I'm not able to use Consumer code, since it has been throwing as deprecated error for many APIs.
Here is the Consumer code:
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import kafka.consumer.Consumer;
//import kafka.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import kafka.consumer.ConsumerIterator;
//import kafka.consumer.KafkaStream;
//import kafka.consumer.KafkaStream;
import kafka.javaapi.consumer.ConsumerConnector;
//import org.apache.kafka.clients.producer.KafkaProducer;
public class KafkaConsumer {
private final ConsumerConnector consumer;
private final String topic;
public KafkaConsumer(String zookeeper, String groupId, String topic) {
Properties props = new Properties();
props.put("zookeeper.connect", zookeeper);
props.put("group.id", groupId);
props.put("zookeeper.session.timeout.ms", "500");
props.put("zookeeper.sync.time.ms", "250");
props.put("auto.commit.interval.ms", "1000");
consumer = Consumer.createJavaConsumerConnector(new ConsumerConfig(props));
this.topic = topic;
}
public void testConsumer() {
System.out.println("Test Con called");
Map<String, Integer> topicCount = new HashMap<>();
topicCount.put(topic, 1);
Map<String, List<KafkaStream<byte[], byte[]>>> consumerStreams = consumer.createMessageStreams(topicCount);
List<KafkaStream<byte[], byte[]>> streams = consumerStreams.get(topic);
System.out.println("For");
for (final KafkaStream stream : streams) {
ConsumerIterator<byte[], byte[]> it = stream.iterator();
System.out.println("Size"+it.length());
while (it.hasNext()) {
System.out.println("Stream");
System.out.println("Message from Single Topic: " + new String(it.next().message()));
}
}
if (consumer != null) {
consumer.shutdown();
}
}
public static void main(String[] args) {
System.out.println("Started");
String topic="twittertopic";
KafkaConsumer simpleTWConsumer = new KafkaConsumer("localhost:XXXX", "testgroup", topic);
simpleTWConsumer.testConsumer();
System.out.println("End");
}
}
It throws error : ConsumerConnector, ConsumerIterator, KafkaStream are deprecated.
ConsumerConfig is not visible.
Is there fixed version of this sample code (Kafka consumer for twitter)?
The tutorial you are following is very old and it's using the old Scala Kafka clients that have been deprecated, see http://kafka.apache.org/documentation/#legacyapis
The classes that have been deprecated are:
kafka.consumer.* and kafka.javaapi.consumer instead use the newer Java Consumer under org.apache.kafka.clients.consumer.*
kafka.producer.* and kafka.javaapi.producer instead use the newer Java Producer under org.apache.kafka.clients.producer.*
Apart from using deprecated classes, your code was mostly correct, I only had to fix a few imports. See below a fixed version. Using it I was able to consume messages I was producing to a topic called twittertopic.
package example;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import kafka.consumer.Consumer;
import kafka.consumer.ConsumerConfig;
import kafka.consumer.ConsumerIterator;
import kafka.consumer.KafkaStream;
import kafka.javaapi.consumer.ConsumerConnector;
public class MyConsumer {
private final ConsumerConnector consumer;
private final String topic;
public MyConsumer(String zookeeper, String groupId, String topic) {
Properties props = new Properties();
props.put("zookeeper.connect", zookeeper);
props.put("group.id", groupId);
props.put("zookeeper.session.timeout.ms", "500");
props.put("zookeeper.sync.time.ms", "250");
props.put("auto.commit.interval.ms", "1000");
consumer = Consumer.createJavaConsumerConnector(new ConsumerConfig(props));
this.topic = topic;
}
public void testConsumer() {
Map<String, Integer> topicCount = new HashMap<>();
topicCount.put(topic, 1);
Map<String, List<KafkaStream<byte[], byte[]>>> consumerStreams = consumer.createMessageStreams(topicCount);
List<KafkaStream<byte[], byte[]>> streams = consumerStreams.get(topic);
for (final KafkaStream stream : streams) {
ConsumerIterator<byte[], byte[]> it = stream.iterator();
while (it.hasNext()) {
System.out.println("Message from Single Topic: " + new String(it.next().message()));
}
}
if (consumer != null) {
consumer.shutdown();
}
}
public static void main(String[] args) {
System.out.println("Started");
String topic = "twittertopic";
MyConsumer simpleTWConsumer = new MyConsumer("localhost:2181", "testgroup", topic);
simpleTWConsumer.testConsumer();
System.out.println("End");
}
}
While the code above can be used, the next major Kafka release is likely to remove classes that are currently deprecated, so you should not write new logic using these.
Instead you should get started with the Java clients, you can use the examples provided on Github: https://github.com/apache/kafka/tree/trunk/examples/src/main/java/kafka/examples
Using the new Java Consumer, your logic would look like:
import java.util.Arrays;
import java.util.Properties;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
public class MyConsumer {
static final String TOPIC = "twittertopic";
static final String GROUP = "testgroup";
public static void main(String[] args) {
System.out.println("Started");
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", GROUP);
props.put("auto.commit.interval.ms", "1000");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);) {
consumer.subscribe(Arrays.asList(TOPIC));
for (int i = 0; i < 1000; i++) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(1L));
System.out.println("Size: " + records.count());
for (ConsumerRecord<String, String> record : records) {
System.out.println("Received a message: " + record.key() + " " + record.value());
}
}
}
System.out.println("End");
}
}
I am using Lucene 6.6 and I am facing difficulty in importing lucene.queryparser and I did check the lucene documentations and it doesn't exist now.
I am using below code. Is there any alternative for queryparser in lucene6.
import java.io.IOException;
import java.text.ParseException;
import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.StringField;
import org.apache.lucene.document.TextField;
import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.queryparser.classic.QueryParser;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.ScoreDoc;
import org.apache.lucene.search.TopScoreDocCollector;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.RAMDirectory;
import org.apache.lucene.util.Version;
public class HelloLucene {
public static void main(String[] args) throws IOException, ParseException {
// 0. Specify the analyzer for tokenizing text.
// The same analyzer should be used for indexing and searching
StandardAnalyzer analyzer = new StandardAnalyzer();
// 1. create the index
Directory index = new RAMDirectory();
IndexWriterConfig config = new IndexWriterConfig(analyzer);
IndexWriter w = new IndexWriter(index, config);
addDoc(w, "Lucene in Action", "193398817");
addDoc(w, "Lucene for Dummies", "55320055Z");
addDoc(w, "Managing Gigabytes", "55063554A");
addDoc(w, "The Art of Computer Science", "9900333X");
w.close();
// 2. query
String querystr = args.length > 0 ? args[0] : "lucene";
// the "title" arg specifies the default field to use
// when no field is explicitly specified in the query.
Query q = null;
try {
q = new QueryParser(Version.LUCENE_6_6_0, "title", analyzer).parse(querystr);
} catch (org.apache.lucene.queryparser.classic.ParseException e) {
e.printStackTrace();
}
// 3. search
int hitsPerPage = 10;
IndexReader reader = DirectoryReader.open(index);
IndexSearcher searcher = new IndexSearcher(reader);
TopScoreDocCollector collector = TopScoreDocCollector.create(hitsPerPage, true);
searcher.search(q, collector);
ScoreDoc[] hits = collector.topDocs().scoreDocs;
// 4. display results
System.out.println("Found " + hits.length + " hits.");
for (int i = 0; i < hits.length; ++i) {
int docId = hits[i].doc;
Document d = searcher.doc(docId);
System.out.println((i + 1) + ". " + d.get("isbn") + "\t" + d.get("title"));
}
// reader can only be closed when there
// is no need to access the documents any more.
reader.close();
}
private static void addDoc(IndexWriter w, String title, String isbn) throws IOException {
Document doc = new Document();
doc.add(new TextField("title", title, Field.Store.YES));
// use a string field for isbn because we don't want it tokenized
doc.add(new StringField("isbn", isbn, Field.Store.YES));
w.addDocument(doc);
}
}
Thanks!
The problem got solved.
Initially, in the build path, only Lucene-core-6.6.0 was added but lucene-queryparser-6.6.0 is a separate jar file that needs to be added separately.
I am successfully retrieve the data of response using xpath expression /abcde/response from the xml ,
<abcde>
<response>000</response>
</abcde>
But couldnt retrieve the data of response from the same xml but with some additional data
<abcde version="8.1" xmlns="http://www.litle.com/schema"
response="0" message="Valid Format">
<response>000</response>
</abcde>
What am i doing wrong ?
package stackoverflow;
import java.io.ByteArrayInputStream;
import java.util.HashMap;
import java.util.Map;
import org.dom4j.Document;
import org.dom4j.DocumentException;
import org.dom4j.DocumentFactory;
import org.dom4j.DocumentHelper;
import org.dom4j.XPath;
import org.dom4j.io.SAXReader;
import org.dom4j.xpath.DefaultXPath;
import org.jaxen.VariableContext;
public class MakejdomWork {
public static void main(String[] args) {
new MakejdomWork().run();
}
public void run() {
ByteArrayInputStream bis = new ByteArrayInputStream("<abcde version=\"8.1\" xmlns=\"http://www.litle.com/schema\" response=\"0\" message=\"Valid Format\"> <response>000</response></abcde>".getBytes());
//ByteArrayInputStream bis = new ByteArrayInputStream("<abcde><response>000</response></abcde>".getBytes());
Map nsPrefixes = new HashMap();
nsPrefixes.put( "x", "http://www.litle.com/schema" );
DocumentFactory factory = new DocumentFactory();
factory.setXPathNamespaceURIs( nsPrefixes );
SAXReader reader = new SAXReader();
reader.setDocumentFactory( factory );
Document doc;
try {
doc = reader.read( bis );
Object value = doc.valueOf("/abcde/x:response");
System.out.println(value);
} catch (DocumentException e) {
e.printStackTrace();
}
}
}
Short answer: you need to use namespace prefixes if your parser is namespace aware (which dom4j is)