Trying to create a test app that imports another test package.
The test apps are listed, as is the compile cmd and the err..
The files are all in the same dir. I was able to compile the MyIncorrectnessListener.java, and the MycssErrorHandler.java with no warnings/errs.
I'm trying to figure out what's the cause of the err >>> sjsu_classes.java:33: '.' expected <<<<<
Any pointers/comments are appreciated...
thanks
the compile cmd/err
[root#toshiba parseapp2]# javac -Xlint -classpath '/opt/htmlunit/lib/
*:/parseapp2/' sjsu_classes.java sjsu_classes.java:33: '.' expected
import MyIncorrectnessListener;
^
sjsu_classes.java:33: ';' expected
import MyIncorrectnessListener;
^
sjsu_classes.java:34: class, interface, or enum expected
import MycssErrorHandler;
^
3 errors
3 warnings
/*
// sjsu_classes.java
//
// function handles getting the data from the sjsu course page, and returns the course page/list
// for the given Alphabet letter. - handles ane letter at a time..
// -the page is then returned to the calling routinne
//
// input:
// args[0] - start url
// args[1] - term
// args[2] - dept
//
// return:
// course page for the term/dept
//
*/
import java.io.IOException;
import java.net.URL;
import java.util.List;
import java.util.Set;
import org.apache.commons.httpclient.NameValuePair;
import com.gargoylesoftware.htmlunit.*;
import com.gargoylesoftware.htmlunit.html.*;
import org.w3c.css.sac.ErrorHandler;
import org.xml.sax.SAXParseException;
//==my test packages
import MyIncorrectnessListener;
import MycssErrorHandler;
public class sjsu_classes {
//public static void main(String[] args) throws Exception {
public void main(String[] args) throws Exception {
// Create and initialize WebClient object
WebClient webClient = new WebClient(BrowserVersion.FIREFOX_3);
IncorrectnessListener ilisten = new MyIncorrectnessListener();
ErrorHandler ierr = new MycssErrorHandler();
webClient.setIncorrectnessListener(ilisten);
webClient.setCssErrorHandler(ierr);
webClient.setThrowExceptionOnScriptError(false);
webClient.setRefreshHandler(new RefreshHandler() {
public void handleRefresh(Page page, URL url, int arg) throws IOException {
//System.out.println("handleRefresh");
}
});
// visit the page, get the form, get the btn
String url_tmp;
String url;
String Term;
String Dept;
String tmp1;
String tmp2;
url_tmp="?FolderPath=PORTAL_ROOT_OBJECT.PA_HC_CLASS_SEARCH&PortalActualURL=https%3a%2f%2fcmshr.sjsu.edu%2fpsc%2fHSJPRDF%2fEMPLOYEE%2fHSJPRD%2fc%2fCOMMUNITY_ACCESS.CLASS_SEARCH.GBL&PortalRegistryName=EMPLOYEE&PortalServletURI=https%3a%2f%2fcmshr.sjsu.edu%2fpsp%2fHSJPRDF%2f&PortalURI=https%3a%2f%2fcmshr.sjsu.edu%2fpsc%2fHSJPRDF%2f&PortalHostNode=HRMS&NoCrumbs=yes";
System.exit(0);
}
}
=======================
MyIncorrectnessListener test package
--------------------------------------------
/*
// MyIncorrectnessListener.java
//
// function handles getting the data from the sjsu course page, and returns the course page/list
// for the given Alphabet letter. - handles ane letter at a time..
// -the page is then returned to the calling routinne
//
// input:
// args[0] - start url
// args[1] - term
// args[2] - dept
//
// return:
// course page for the term/dept
//
*/
//==handle the warnings thrown from the js functions..
package MyIncorrectnessListener;
import com.gargoylesoftware.htmlunit.*;
import com.gargoylesoftware.htmlunit.html.*;
//public static class MyIncorrectnessListener implements IncorrectnessListener
public class MyIncorrectnessListener implements IncorrectnessListener
{
#Override
public void notify(String arg0, Object arg1)
{
//System.err.println("Argument : " + arg0.toString() + ", Object :");
}
}
MycssErrorHandler test package
--------------------------------
/*
// MycssErrorHandler.java
//
// function handles getting the data from the sjsu course page, and returns the course page/list
// for the given Alphabet letter. - handles ane letter at a time..
// -the page is then returned to the calling routinne
//
// input:
// args[0] - start url
// args[1] - term
// args[2] - dept
//
// return:
// course page for the term/dept
//
*/
package MycssErrorHandler;
import org.w3c.css.sac.ErrorHandler;
import org.xml.sax.SAXParseException;
import com.gargoylesoftware.htmlunit.*;
import com.gargoylesoftware.htmlunit.html.*;
//==handle the warnings thrown from the js functions..
//==handle the warnings thrown from the css functions..
// public static class MycssErrorHandler implements DefaultCssErrorHandler
// public static class MycssErrorHandler implements ErrorHandler
// public class MycssErrorHandler implements ErrorHandler
public abstract class MycssErrorHandler implements ErrorHandler
// protected class MycssErrorHandler implements ErrorHandler
{
//#Override
public void notify(String arg0, Object arg1)
{
//System.err.println("Argument : " + arg0.toString() + ",Object : ");
}
//#Override
public void fatalError(SAXParseException ex)
{
//fatals.add(ex);
}
}
Your java files have to be in a directory that matches their package names, so you have to place
MyIncorrectnessListener.java in a directory named MyIncorrectnessListener, MycssErrorHandler.java in a directory named MycssErrorHandler and so on.
You could place them all in the same directory, but you have to remove the 'package' statement in the files.
The error you're getting is because you're only giving a package name in the import statement. You should import something specific, or everything, from a package. Change it to
import MyIncorrectnessListener.MyIncorrectnessListener;
or
import MyIncorrectnessListener.*;
Or if you don't place your files in any packages, you don't need any import statements.
Related
In Xtext, how does one auto-generate a single file containing information from multiple model files.
Consider the following simple Xtext grammar.
grammar org.example.people.People with org.eclipse.xtext.common.Terminals
generate people "http://www.example.org/people/People"
People:
people+=Person*;
Person:
'person' name=ID ';';
In the launched workspace I create a project with two files, friends.people
// friends
person Alice;
person Bob;
and enemies.people
// enemies
person Malice;
person Rob;
How do I auto-generate a single file listing everyone when the global index changes?
Alice
Bob
Malice
Rob
For ease of future reference, here is the solution obtained by combining the various references given by Christian Dietrich. Note that the solution is Eclipse dependent.
Anyone who finds themselves with this requirement should perhaps try to find a better way of modelling the problem. For example a singleton model element All that generates the required list by finding everyone in the model using the standard API. This is independent of Eclipse, and requires non of the following complexity.
In the generator package of the grammar project, create an Java interface IPeopleGenerator extending IGenerator2.
package org.example.people.generator;
import org.eclipse.emf.ecore.resource.ResourceSet;
import org.eclipse.xtext.generator.IFileSystemAccess2;
import org.eclipse.xtext.generator.IGenerator2;
import org.eclipse.xtext.generator.IGeneratorContext;
public interface IPeopleGenerator extends IGenerator2{
public void doGenerate(ResourceSet input, IFileSystemAccess2 fsa, IGeneratorContext context);
}
and edit the existing generator PeopleGenerator as follows.
/*
* generated by Xtext 2.14.0
*/
package org.example.people.generator
import org.eclipse.emf.ecore.resource.Resource
import org.eclipse.emf.ecore.resource.ResourceSet
import org.eclipse.xtext.generator.IFileSystemAccess2
import org.eclipse.xtext.generator.IGeneratorContext
import org.example.people.people.Person
/**
* Generates code from your model files on save.
*
* See https://www.eclipse.org/Xtext/documentation/303_runtime_concepts.html#code-generation
*/
class PeopleGenerator implements IPeopleGenerator {
override doGenerate(ResourceSet rs, IFileSystemAccess2 fsa, IGeneratorContext context) {
val people = rs.resources.map(r|r.allContents.toIterable.filter(Person)).flatten
fsa.generateFile("all.txt", people.compile)
}
override afterGenerate(Resource input, IFileSystemAccess2 fsa, IGeneratorContext context) {
}
override beforeGenerate(Resource input, IFileSystemAccess2 fsa, IGeneratorContext context) {
}
override doGenerate(Resource input, IFileSystemAccess2 fsa, IGeneratorContext context) {
}
def compile (Iterable<Person> entities) '''
«FOR e : entities»
«e.name»
«ENDFOR»
'''
}
and add the method
def Class<? extends IPeopleGenerator> bindIPeopleGenerator () {
return PeopleGenerator
}
to the existing runtime module PeopleRuntimeModule in the grammar project.
Work needs to be done in the UI project org.example.people.ui. Consequently this solution is Eclipse dependent.
Create a Java class org.example.people.ui.PeopleBuilderParticipant as follows (the complexity being the need to ensure that the global generated file is only created once).
package org.example.people.ui;
import java.util.List;
import org.eclipse.core.runtime.CoreException;
import org.eclipse.core.runtime.IProgressMonitor;
import org.eclipse.core.runtime.NullProgressMonitor;
import org.eclipse.emf.ecore.resource.Resource;
import org.eclipse.xtext.builder.BuilderParticipant;
import org.eclipse.xtext.builder.EclipseResourceFileSystemAccess2;
import org.eclipse.xtext.builder.MonitorBasedCancelIndicator;
import org.eclipse.xtext.generator.GeneratorContext;
import org.eclipse.xtext.resource.IContainer;
import org.eclipse.xtext.resource.IResourceDescription;
import org.eclipse.xtext.resource.IResourceDescription.Delta;
import org.eclipse.xtext.resource.IResourceDescriptions;
import org.eclipse.xtext.resource.impl.ResourceDescriptionsProvider;
import org.example.people.generator.IPeopleGenerator;
import com.google.inject.Inject;
public class PeopleBuilderParticipant extends BuilderParticipant {
#Inject
private ResourceDescriptionsProvider resourceDescriptionsProvider;
#Inject
private IContainer.Manager containerManager;
#Inject(optional = true)
private IPeopleGenerator generator;
protected ThreadLocal<Boolean> buildSemaphor = new ThreadLocal<Boolean>();
#Override
public void build(IBuildContext context, IProgressMonitor monitor) throws CoreException {
buildSemaphor.set(false);
super.build(context, monitor);
}
#Override
protected void handleChangedContents(Delta delta, IBuildContext context,
EclipseResourceFileSystemAccess2 fileSystemAccess) throws CoreException {
super.handleChangedContents(delta, context, fileSystemAccess);
if (!buildSemaphor.get() && generator != null) {
invokeGenerator(delta, context, fileSystemAccess);
}
}
private void invokeGenerator(Delta delta, IBuildContext context, EclipseResourceFileSystemAccess2 access) {
buildSemaphor.set(true);
Resource resource = context.getResourceSet().getResource(delta.getUri(), true);
if (shouldGenerate(resource, context)) {
IResourceDescriptions index = resourceDescriptionsProvider.createResourceDescriptions();
IResourceDescription resDesc = index.getResourceDescription(resource.getURI());
List<IContainer> visibleContainers = containerManager.getVisibleContainers(resDesc, index);
for (IContainer c : visibleContainers) {
for (IResourceDescription rd : c.getResourceDescriptions()) {
context.getResourceSet().getResource(rd.getURI(), true);
}
}
MonitorBasedCancelIndicator cancelIndicator = new MonitorBasedCancelIndicator(
new NullProgressMonitor()); //maybe use reflection to read from fsa
GeneratorContext generatorContext = new GeneratorContext();
generatorContext.setCancelIndicator(cancelIndicator);
generator.doGenerate(context.getResourceSet(), access, generatorContext);
}
}
}
and bind this build participant by adding
override Class<? extends IXtextBuilderParticipant> bindIXtextBuilderParticipant() {
return PeopleBuilderParticipant;
}
to the existing UI module org.example.people.ui.PeopleUiModule.
I added the validation code to the answer of fundagain to eliminate invalid resources. However, this will not work when last modified resource is invalid because doGenerate is not invoked when invalid. When any valid resource is saved invalid resources will be discarded from all.txt .
override doGenerate(ResourceSet rs, IFileSystemAccess2 fsa, IGeneratorContext context) {
var valid_rs = new ArrayList<Resource>
for(r : rs.resources)
if (( r as XtextResource)
.getResourceServiceProvider()
.getResourceValidator()
.validate(r,CheckMode.ALL, null)
.map(issue | issue.severity)
.filter[it === Severity.ERROR]
.size == 0)
valid_rs.add(r)
val types = valid_rs.map(r|r.allContents.toIterable.filter(Person)).flatten
fsa.generateFile("all.txt", people.compile)
}
Following
https://christiandietrich.wordpress.com/2011/10/15/xtext-calling-the-generator-from-a-context-menu/
and using EclipseResourceFileSystemAccess2 instead of EclipseResourceFileSystemAccess when the line
final EclipseResourceFileSystemAccess2 fsa = fileAccessProvider.get();
give an exception. The only information I have is
// Compiled from InvocationTargetException.java (version 1.8 : 52.0, super bit)
public class java.lang.reflect.InvocationTargetException extends java.lang.ReflectiveOperationException {
I don't know how to get the stack trace in Eclipse.
does the code in the blog still function in the most recent release of Xtext?
Update 1
Snippets from plugin.xml
Handler:
<extension
point="org.eclipse.ui.handlers">
<handler
class="tuks.mcrl2.dsl.ui.handlers.Mcrl22Lps"
commandId="tuks.mcrl2.dsl.ui.commands.mcrl2lps">
</handler>
</extension>
Commands:
<extension
point="org.eclipse.ui.commands">
<command
categoryId="tuks.mcrl2.dsl.ui.category.processalgebra"
defaultHandler="tuks.mcrl2.dsl.ui.handlers.Mcrl22Lps"
description="Conver a mclr2 file to lps"
id="tuks.mcrl2.dsl.ui.commands.mcrl2lps"
name="mcrl22lps">
</command>
<category
id="tuks.mcrl2.dsl.ui.category.processalgebra"
name="Process Algebra">
</category>
</extension>
it basically works, if you do the update from EclipseResourceFileSystemAccess and Stuff and (maybe) IGenerator.
I assume in your case you dont set the Accesses ProgressMonitor and other props.
package org.xtext.example.mydsl.ui.handler;
import org.eclipse.core.commands.AbstractHandler;
import org.eclipse.core.commands.ExecutionEvent;
import org.eclipse.core.commands.ExecutionException;
import org.eclipse.core.commands.IHandler;
import org.eclipse.core.resources.IFile;
import org.eclipse.core.resources.IFolder;
import org.eclipse.core.resources.IProject;
import org.eclipse.core.runtime.CoreException;
import org.eclipse.core.runtime.NullProgressMonitor;
import org.eclipse.emf.common.util.URI;
import org.eclipse.emf.ecore.resource.Resource;
import org.eclipse.emf.ecore.resource.ResourceSet;
import org.eclipse.jface.viewers.ISelection;
import org.eclipse.jface.viewers.IStructuredSelection;
import org.eclipse.ui.handlers.HandlerUtil;
import org.eclipse.xtext.builder.EclipseResourceFileSystemAccess2;
import org.eclipse.xtext.generator.GeneratorContext;
import org.eclipse.xtext.generator.IGenerator2;
import org.eclipse.xtext.resource.IResourceDescriptions;
import org.eclipse.xtext.ui.resource.IResourceSetProvider;
import com.google.inject.Inject;
import com.google.inject.Provider;
public class GenerationHandler extends AbstractHandler implements IHandler {
#Inject
private IGenerator2 generator;
#Inject
private Provider<EclipseResourceFileSystemAccess2> fileAccessProvider;
#Inject
IResourceDescriptions resourceDescriptions;
#Inject
IResourceSetProvider resourceSetProvider;
#Override
public Object execute(ExecutionEvent event) throws ExecutionException {
ISelection selection = HandlerUtil.getCurrentSelection(event);
if (selection instanceof IStructuredSelection) {
IStructuredSelection structuredSelection = (IStructuredSelection) selection;
Object firstElement = structuredSelection.getFirstElement();
if (firstElement instanceof IFile) {
IFile file = (IFile) firstElement;
IProject project = file.getProject();
IFolder srcGenFolder = project.getFolder("src-gen");
if (!srcGenFolder.exists()) {
try {
srcGenFolder.create(true, true,
new NullProgressMonitor());
} catch (CoreException e) {
return null;
}
}
final EclipseResourceFileSystemAccess2 fsa = fileAccessProvider.get();
fsa.setProject(project);
fsa.setOutputPath("src-gen");
fsa.setMonitor(new NullProgressMonitor());
URI uri = URI.createPlatformResourceURI(file.getFullPath().toString(), true);
ResourceSet rs = resourceSetProvider.get(project);
Resource r = rs.getResource(uri, true);
generator.doGenerate(r, fsa, new GeneratorContext());
}
}
return null;
}
#Override
public boolean isEnabled() {
return true;
}
}
and make sure you register the handler properly.
the
class="org.xtext.example.mydsl.ui.MyDslExecutableExtensionFactory:org.xtext.example.mydsl.ui.handler.GenerationHandler"
is crucial, especially that it consists of 2 parts, the ExtensionFactory followed by a : followed by the actual class name
How can Univocity Parsers read a .csv file when the headers are not on the first line?
There are errors if the first line in the .csv file is not the headers.
The code and stack trace are below.
Any help would be greatly appreciated.
import com.univocity.parsers.csv.CsvParserSettings;
import com.univocity.parsers.common.processor.*;
import com.univocity.parsers.csv.*;
import java.io.InputStreamReader;
import java.io.Reader;
import java.io.UnsupportedEncodingException;
import java.lang.IllegalStateException;
import java.lang.String;
import java.util.List;
public class UnivocityParsers {
public Reader getReader(String relativePath) {
try {
return new InputStreamReader(this.getClass().getResourceAsStream(relativePath), "Windows-1252");
} catch (UnsupportedEncodingException e) {
throw new IllegalStateException("Unable to read input", e);
}
}
public void columnSelection() {
RowListProcessor rowProcessor = new RowListProcessor();
CsvParserSettings parserSettings = new CsvParserSettings();
parserSettings.setRowProcessor(rowProcessor);
parserSettings.setHeaderExtractionEnabled(true);
parserSettings.setLineSeparatorDetectionEnabled(true);
parserSettings.setSkipEmptyLines(true);
// Here we select only the columns "Price", "Year" and "Make".
// The parser just skips the other fields
parserSettings.selectFields("AUTHOR", "ISBN");
CsvParser parser = new CsvParser(parserSettings);
parser.parse(getReader("list2.csv"));
List<String[]> rows = rowProcessor.getRows();
String[] strings = rows.get(0);
System.out.print(strings[0]);
}
public static void main(String arg[]) {
UnivocityParsers univocityParsers = new UnivocityParsers();
univocityParsers.columnSelection();
}
}
Stack trace:
Exception in thread "main" com.univocity.parsers.common.TextParsingException: Error processing input: java.lang.IllegalStateException - Unknown field names: [author, isbn]. Available fields are: [list of books by author - created today]
Here is the file being parsed:
List of books by Author - Created today
"REVIEW_DATE","AUTHOR","ISBN","DISCOUNTED_PRICE"
"1985/01/21","Douglas Adams",0345391802,5.95
"1990/01/12","Douglas Hofstadter",0465026567,9.95
"1998/07/15","Timothy ""The Parser"" Campbell",0968411304,18.99
"1999/12/03","Richard Friedman",0060630353,5.95
"2001/09/19","Karen Armstrong",0345384563,9.95
"2002/06/23","David Jones",0198504691,9.95
"2002/06/23","Julian Jaynes",0618057072,12.50
"2003/09/30","Scott Adams",0740721909,4.95
"2004/10/04","Benjamin Radcliff",0804818088,4.95
"2004/10/04","Randel Helms",0879755725,4.50
As of today, on 2.0.0-SNAPSHOT you can do this:
settings.setNumberOfRowsToSkip(1);
On version 1.5.6 you can do this to skip the first line and correctly grab the headers:
RowListProcessor rowProcessor = new RowListProcessor(){
#Override
public void processStarted(ParsingContext context) {
super.processStarted(context);
context.skipLines(1);
}
};
An alternative is to comment the first line if your input file (if you have control over how the file is generated) by adding a # at the beginning of the line you want to discard:
#List of books by Author - Created today
I am writing a class to recursively extract files from inside a zip file and produce them to a Kafka queue for further processing. My intent is to be able to extract files from multiple levels of zip. The code below is my implementation of the tika ContainerExtractor to do this.
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import java.util.Stack;
import org.apache.commons.lang.StringUtils;
import org.apache.tika.config.TikaConfig;
import org.apache.tika.detect.DefaultDetector;
import org.apache.tika.detect.Detector;
import org.apache.tika.exception.TikaException;
import org.apache.tika.extractor.ContainerExtractor;
import org.apache.tika.extractor.EmbeddedResourceHandler;
import org.apache.tika.io.TemporaryResources;
import org.apache.tika.io.TikaInputStream;
import org.apache.tika.metadata.Metadata;
import org.apache.tika.mime.MediaType;
import org.apache.tika.parser.AbstractParser;
import org.apache.tika.parser.AutoDetectParser;
import org.apache.tika.parser.ParseContext;
import org.apache.tika.parser.Parser;
import org.apache.tika.parser.pkg.PackageParser;
import org.xml.sax.ContentHandler;
import org.xml.sax.SAXException;
import org.xml.sax.helpers.DefaultHandler;
public class UberContainerExtractor implements ContainerExtractor {
/**
*
*/
private static final long serialVersionUID = -6636138154366178135L;
// statically populate SUPPORTED_TYPES
static {
Set<MediaType> supportedTypes = new HashSet<MediaType>();
ParseContext context = new ParseContext();
supportedTypes.addAll(new PackageParser().getSupportedTypes(context));
SUPPORTED_TYPES = Collections.unmodifiableSet(supportedTypes);
}
/**
* A stack that maintains the parent filenames for the recursion
*/
Stack<String> parentFileNames = new Stack<String>();
/**
* The default tika parser
*/
private final Parser parser;
/**
* Default tika detector
*/
private final Detector detector;
/**
* The supported container types into which we can recurse
*/
public final static Set<MediaType> SUPPORTED_TYPES;
/**
* The number of documents recursively extracted from the container and its
* children containers if present
*/
int extracted;
public UberContainerExtractor() {
this(TikaConfig.getDefaultConfig());
}
public UberContainerExtractor(TikaConfig config) {
this(new DefaultDetector(config.getMimeRepository()));
}
public UberContainerExtractor(Detector detector) {
this.parser = new AutoDetectParser(new PackageParser());
this.detector = detector;
}
public boolean isSupported(TikaInputStream input) throws IOException {
MediaType type = detector.detect(input, new Metadata());
return SUPPORTED_TYPES.contains(type);
}
#Override
public void extract(TikaInputStream stream, ContainerExtractor recurseExtractor, EmbeddedResourceHandler handler)
throws IOException, TikaException {
ParseContext context = new ParseContext();
context.set(Parser.class, new RecursiveParser(recurseExtractor, handler));
try {
Metadata metadata = new Metadata();
parser.parse(stream, new DefaultHandler(), metadata, context);
} catch (SAXException e) {
throw new TikaException("Unexpected SAX exception", e);
}
}
private class RecursiveParser extends AbstractParser {
/**
*
*/
private static final long serialVersionUID = -7260171956667273262L;
private final ContainerExtractor extractor;
private final EmbeddedResourceHandler handler;
private RecursiveParser(ContainerExtractor extractor, EmbeddedResourceHandler handler) {
this.extractor = extractor;
this.handler = handler;
}
public Set<MediaType> getSupportedTypes(ParseContext context) {
return parser.getSupportedTypes(context);
}
public void parse(InputStream stream, ContentHandler ignored, Metadata metadata, ParseContext context)
throws IOException, SAXException, TikaException {
TemporaryResources tmp = new TemporaryResources();
try {
TikaInputStream tis = TikaInputStream.get(stream, tmp);
// Figure out what we have to process
String filename = metadata.get(Metadata.RESOURCE_NAME_KEY);
MediaType type = detector.detect(tis, metadata);
if (extractor == null) {
// do nothing
} else {
// Use a temporary file to process the stream
File file = tis.getFile();
System.out.println("file is directory = " + file.isDirectory());
// Recurse and extract if the filetype is supported
if (SUPPORTED_TYPES.contains(type)) {
System.out.println("encountered a supported file:" + filename);
parentFileNames.push(filename);
extractor.extract(tis, extractor, handler);
parentFileNames.pop();
} else { // produce the file
List<String> parentFilenamesList = new ArrayList<String>(parentFileNames);
parentFilenamesList.add(filename);
String originalFilepath = StringUtils.join(parentFilenamesList, "/");
System.out.println("producing " + filename + " with originalFilepath:" + originalFilepath
+ " to kafka queue");
++extracted;
}
}
} finally {
tmp.dispose();
}
}
}
public int getExtracted() {
return extracted;
}
public static void main(String[] args) throws IOException, TikaException {
String filename = "/Users/rohit/Data/cd.zip";
File file = new File(filename);
TikaInputStream stream = TikaInputStream.get(file);
ContainerExtractor recursiveExtractor = new UberContainerExtractor();
EmbeddedResourceHandler resourceHandler = new EmbeddedResourceHandler() {
#Override
public void handle(String filename, MediaType mediaType, InputStream stream) {
// do nothing
}
};
recursiveExtractor.extract(stream, recursiveExtractor, resourceHandler);
stream.close();
System.out.println("extracted " + ((UberContainerExtractor) recursiveExtractor).getExtracted() + " files");
}
}
It works on multiple levels of zip as long as the files inside the zips are in a flat structure. for ex.
cd.zip
- c.txt
- d.txt
The code does not work if there the files in the zip are present inside a directory. for ex.
ab.zip
- ab/
- a.txt
- b.txt
While debugging I came across the following code snippet in the PackageParser
try {
ArchiveEntry entry = ais.getNextEntry();
while (entry != null) {
if (!entry.isDirectory()) {
parseEntry(ais, entry, extractor, xhtml);
}
entry = ais.getNextEntry();
}
} finally {
ais.close();
}
I tried to comment out the if condition but it did not work. Is there a reason why this is commented? Is there any way of getting around this?
I am using tika version 1.6
Tackling your question in reverse order:
Is there a reason why this is commented?
Entries in zip files are either directories or files. If files, they include the name of the directory they come from. As such, Tika doesn't need to do anything with the directories, all it needs to do is process the embedded files as and when they come up
The code does not work if there the files in the zip are present inside a directory. for ex. ab.zip - ab/ - a.txt - b.txt
You seem to be doing something wrong then. Tika's recursion and package parser handle zips with folders in them just fine!
To prove this, start with a zip file like this:
$ unzip -l ../tt.zip
Archive: ../tt.zip
Length Date Time Name
--------- ---------- ----- ----
0 2015-02-03 16:42 t/
0 2015-02-03 16:42 t/t2/
0 2015-02-03 16:42 t/t2/t3/
164404 2015-02-03 16:42 t/t2/t3/test.jpg
--------- -------
164404 4 files
Now, make us of the -z extraction flag of the Tika App, which causes Tika to extract out all of the embedded contents of a file. Run like that, and we get
$ java -jar tika-app-1.7.jar -z ../tt.zip
Extracting 't/t2/t3/test.jpg' (image/jpeg) to ./t/t2/t3/test.jpg
Then list the resulting directory, and we see
$ find . -type f
./t/t2/t3/Test.jpg
I can't see what's wrong with your code, but sadly for you we've shown that the problem is there, and not with Tika... You'd be best off reviewing the various examples of recursion that Tika provides, such as the Tika App tool and the Recursing Parser Wrapper, then re-write your code to be something simple based from those
I am trying to build a simple component to understand how and why JSF 2.X works the way it does. I have been using the newer annotations and have been trying to piece together a clear example.
So I have built my component and deployed it in a xhtml file as follows:
<kal:day value="K.Day" title="Kalendar" model="#{kalendarDay}"/>
The within the UIComponent I do the following:
ValueExpression ve = getValueExpression("model");
if (ve != null)
{
System.out.println("model expression "+ve.getExpressionString());
model = (KalendarDay) ve.getValue(getFacesContext().getELContext());
System.out.println("model "+model);
}
The expression "#{kalendarDay}" is correctly displayed indicating that the value has been successfully transmitted between the page and the component. However the evaluation of the expression results in "null".
This seems to indicate that the backing bean is unavailable at this point, although the page correctly validates and deploys. So I am 95% certain that the bean is there at run time.
So perhaps this is a phase thing? Should I be evaluating this in the decode of the renderer and setting the value in the attributes map? I am still a little confused about the combination of actual values and value expressions.
So my question is where should I fetch and evaluate the valueExpression for model and should I store the result of the evaluation in the UIComponent or should I simply evaluate it every time?
SSCCE files below I think these are the only required files to demonstrate the problem
Bean Interface -----
/**
*
*/
package com.istana.kalendar.fixture;
import java.util.Date;
/**
* #author User
*
*/
public interface KalendarDay
{
public Date getDate();
}
Bean Implementation ---
/**
*
*/
package com.istana.kalendar.session.wui;
import java.util.Calendar;
import java.util.Date;
import javax.ejb.Stateful;
import javax.inject.Named;
import com.istana.kalendar.fixture.KalendarDay;
/**
* #author User
*
*/
#Named ("kalendarDay")
#Stateful
public class KalKalendarDay
implements KalendarDay
{
private Calendar m_date = Calendar.getInstance();
/* (non-Javadoc)
* #see com.istana.kalendar.fixture.KalendarDay#getDate()
*/
#Override
public Date getDate()
{
return m_date.getTime();
}
}
UIComponent ---
/**
*
*/
package com.istana.kalendar.fixture.jsf;
import javax.el.ValueExpression;
import javax.faces.component.FacesComponent;
import javax.faces.component.UIOutput;
import com.istana.kalendar.fixture.KalendarDay;
/**
* #author User
*
*/
#FacesComponent (value=UIDay.COMPONENT_TYPE)
public class UIDay extends UIOutput
{
static final
public String COMPONENT_TYPE = "com.istana.kalendar.fixture.jsf.Day";
static final
public String COMPONENT_FAMILY = "com.istana.kalendar.fixture.jsf.Day";
private KalendarDay m_model;
private String m_title;
#Override
public String getRendererType()
{
return UIDayRenderer.RENDERER_TYPE;
}
#Override
public String getFamily()
{
return COMPONENT_FAMILY;
}
public KalendarDay getModel()
{
KalendarDay model = (KalendarDay) getStateHelper().eval("model");
System.out.println("model "+model);
return model;
}
public void setModel(KalendarDay model)
{
getStateHelper().put("model",model);
}
public String getTitle()
{
return (String) getStateHelper().eval("title");
}
public void setTitle(String title)
{
getStateHelper().put("title",title);
}
}
UIComponentRenderer ---
/**
*
*/
package com.istana.kalendar.fixture.jsf;
import java.io.IOException;
import javax.el.ValueExpression;
import javax.faces.component.UIComponent;
import javax.faces.context.FacesContext;
import javax.faces.context.ResponseWriter;
import javax.faces.render.FacesRenderer;
import javax.faces.render.Renderer;
import com.istana.kalendar.fixture.KalendarDay;
/**
* #author User
*
*/
#FacesRenderer (componentFamily = UIDay.COMPONENT_FAMILY
,rendererType = UIDayRenderer.RENDERER_TYPE
)
public class UIDayRenderer extends Renderer
{
static final
public String RENDERER_TYPE = "com.istana.kalendar.fixture.jsf.DayRenderer";
#Override
public void encodeBegin (FacesContext context,UIComponent component)
throws IOException
{
UIDay uic = (UIDay) component;
ResponseWriter writer = context.getResponseWriter();
writer.startElement("p", uic);
/*
* This is the call that triggers the println
*/
writer.write("Day - "+uic.getModel().getDate());
}
#Override
public void encodeEnd (FacesContext context,UIComponent component)
throws IOException
{
ResponseWriter writer = context.getResponseWriter();
writer.endElement("p");
writer.flush();
}
}
kalendar.taglib.xml ---
<facelet-taglib
id="kalendar"
xmlns="http://java.sun.com/xml/ns/javaee"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-facelettaglibrary_2_0.xsd"
version="2.0"
>
<namespace>http://istana.com/kalendar</namespace>
<tag>
<tag-name>day</tag-name>
<component>
<component-type>com.istana.kalendar.fixture.jsf.Day</component-type>
</component>
</tag>
</facelet-taglib>
I'm not sure why it's null, but the symptoms indicate that the #{kalendarDay} is been specified during view render time while you're trying to evaluate it during the view build time.
So perhaps this is a phase thing? Should I be evaluating this in the decode of the renderer and setting the value in the attributes map? I am still a little confused about the combination of actual values and value expressions.
You should use the encodeXxx() methods of the component or the associated renderer (if any) to generate HTML based on the component's attributes/properties.
You should use the decode() method of the component or the associated renderer (if any) to set component's attributes/properties based on HTTP request parameters which are been sent along with a HTML form submit.
So my question is where should I fetch and evaluate the valueExpression for model and should I store the result of the evaluation in the UIComponent or should I simply evaluate it every time?
Since JSF 2.x it's recommended to explicitly specify a getter and setter for component attributes which in turn delegates to UIComponent#getStateHelper().
public String getValue() {
return (String) getStateHelper().eval("value");
}
public void setValue(String value) {
getStateHelper().put("value", value);
}
public String getTitle() {
return (String) getStateHelper().eval("title");
}
public void setTitle(String title) {
getStateHelper().put("title", title);
}
public Object getModel() {
return getStateHelper().eval("model");
}
public void setModel(Object model) {
getStateHelper().put("model", model);
}
That's all you basically need (note that the getter and setter must exactly match the attribute name as per Javabeans specification). Then in the encodeXxx() method(s) just call getModel() to get (and evaluate) the value of the model attribute.