I'm pretty new using Flume, just started testing it using a CDH 4.7.0 distributuion.
I'm configuring Flume through Cloudera Manager.
I've set up an agent using a sequence generator as source and everything went fine, but I've got an error when configured the source as a spooling directory:
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /var/log/flume-ng/flume-cmf-flume1-AGENT-xxxxx.log (Permission denied)
Didn't change anything else, any ideas?
My .conf:
SOURCE
agent_pbe2.sources.spldir-src1.type = spooldir
agent_pbe2.sources.spldir-src1.spoolDir = /tmp/Flume-PoC
CHANNEL
agent_pbe2.channels.mem-chn1.type = memory
agent_pbe2.channels.mem-chn1.capacity = 1000
agent_pbe2.channels.mem-chn1.transactionCapacity = 100
SINK
agent_pbe2.sinks.hdfs-snk1.type = hdfs
agent_pbe2.sinks.hdfs-snk1.hdfs.path = hdfs://martehadoop/user/hduser/Flume-PoC/pbe2
BIND SOURCES
agent_pbe2.sources.spldir-src1.channels = mem-chn1
BIND SINKS
agent_pbe2.sinks.hdfs-snk1.channel = mem-chn1
Thanks!
I answer my own question.
The problem was related to user rights. The user used by Flume service must be able to access all resources involved in the agent(s).
Cheers!
Related
I'm hoping some of you may be able to help me with setting up SSL on my fuseki server. I've been battling with it for a few days now and am running out of possible solutions!
OS: RHEL 8.5 (Ootpa)
Fuseki: Version 4.2.0
Currently running as system service with:
ExecStart=/home/fuseki/apache-jena-fuseki-4.2.0/fuseki-server -v -tdb2 -update -config=/home/fuseki/fuseki_data/config.ttl
This is the manual I've been working with- https://jena.apache.org/documentation/fuseki2/fuseki-data-access-control.html
The following are the provided arguments to add in the startup sequence of fuseki-server.
–https=SETUP [Name of file for certificate details.]
& –httpsPort=PORT [The port for https. Default: 3043]
The --https argument names a file in JSON which includes the name of
the certificate file and password for the certificate.
The issue is that no matter how I phrase the arguments the process returns "fuseki-server[9469]: Unknown argument: https" in the journalctl logs.
I have tried -https=dir, --https=dir, & -httpsConf=dir (where dir is the directory to my cert_details.json file).
Based on the docs https should have native support but when I check fuseki-server -help there is no mention of an https argument. I have created a .jks from my cert, have set the correct file permissions, and have allowed 3043.
I have also located the block of code in fuseki that resolves the keystore and passwd from the .json file (which led me to try using -httpsConf=)
private void setHttpsCert(String filename) {
try {
JsonObject httpsConf = JSON.read(filename);
Path path = Path.of(filename).toAbsolutePath();
String keystore = httpsConf.get("keystore").getAsString().value();
// Resolve relative to the https setup file.
this.httpsKeystore = path.getParent().resolve(keystore).toString();
this.httpsKeystorePasswd = httpsConf.get("passwd").getAsString().value();
Not sure what I'm missing here. For what it's worth I'm a chemist and I definitely don't know java all that well so it very well could be me being stupid. Any suggestions/knowledge would be greatly appreciated.
there are two ways to get HTTPS+Fuseki:
The document referred to is for the jar file here: https://repo1.maven.org/maven2/org/apache/jena/jena-fuseki-server/4.2.0/jena-fuseki-server-4.2.0.jar, not the jar in the apache-jena-fuseki download. (BTW It does not have the UI.)
The one in the apache-jena-fuseki can be use HTTPS by using a Jetty configuration using --jetty=jetty.xml (https://www.eclipse.org/jetty/documentation/current/jetty-xml-config.html) -- example: https://github.com/apache/jena/blob/main/jena-fuseki2/examples/fuseki-jetty-https.xml (which will need modifying).
I have changed our Jenkins setup from everything running on one machine to a master-agent/slave setup. Before that everything worked fine, now I am facing issues that some programs I am calling that access files can't find them.
Case 1:
(Pls don't ask why it is so complicated, but the file structure is given and I can't change it)
I am calling a python script, that iself calls a batch file
filepath= os.path.abspath(os.path.join(pamFolder, "run.bat"))
p = subprocess.Popen(filepath, cwd=pamFolder, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
... and the batch file again a jar file with the actual program
java -XX:-UseGCOverheadLimit -cp "../..;../../libs/*" -jar ..\..\myjarfile.jar
Within the jar file there is an access to a file on the disk failing with an error message that the file can't be found:
ERR : The file was not found in the specified path 'U:\somefile.txt'. Please check this path for access and your configuration!
Case 2:
I am calling a batch file from Jenkins that is calling some other exe and in the end trying to open a file in Excel via the COM interface. Here I am getting the following exception (Excel can't access the file):
Unhandled Exception: System.Runtime.InteropServices.COMException: Microsoft Excel kann auf die Datei 'D:\Jenkins\workspace\myJob\someDir\someFile.xlsm' nicht zugreifen.
Question
As previously mentioned, both jobs were working in the previous setup. Both files DO exist.
I suspect that Jenkins / the programs are trying to find the files on the master where they are not available.
Is there any way to tell Jenkins that the called tools are fully executed on the slave node or in some other way tell them where to find these files?
EDIT
The job is already running on the slave. The console shows Running on [slave name] in D:/Jenkins/workspace/xxxxx.
The master is configured in a way that only jobs assigned to it run on the master. So pretty much all jobs should run on the slave.
EDIT2 / SOLUTION
It turned out that the 2 issues are caused by different things.
Case1: Solved this by using the UNC path
Case2: Solved by a mixture of giving the necessary permissions as described here and starting the slave service with a user with admin rights.
From my experience with this issue, usually it has to do with your SCM setup.
But as you stated that the files DO exist I think there might be the possibility that U:\ is a network share? Then consider changing your path to use a UNC path.
If that's not the case check if your jenkins slave as sufficient user rights to access said file.
You can tell Jenkins to run the job on the designated slave as follows:
Under Nodes > [SLAVE] > Configure, specify a label for the slave.
Under [Job] > Configure > Restrict where this project can be run, enter the label.
Now when you build, the console output of the job should read correctly along the lines Running on [SLAVE] (build_agent_01) in C:/jenkins and the files must be accessible.
In the project installer i am creating a custom event log. But when my service starts my all logs are going to the "Application" instead of my cuustom log. Below is the code which i have added to installer.
// Create Event Source and Event Log
EventLogInstaller logInstaller = new EventLogInstaller();
logInstaller.Source = "MyServices";
logInstaller.Log = "MyService Events";
Installers.Add(logInstaller);
Furthurmore the name of service is MyService.exe.
When I uninstall and re-install the service, i installation fail with the following install log;
Running a transacted installation.
Beginning the Install phase of the installation. See the contents of
the log file for the D:\MyService\MyService\bin\Release\MyService.exe
assembly's progress. The file is located at
D:\MyService\MyService\bin\Release\MyService.InstallLog.
An exception occurred during the Install phase.
System.ArgumentException: Source MyServices already exists on the
local computer.
The Rollback phase of the installation is beginning. See the contents
of the log file for the
D:\MyService\MyService\bin\Release\MyService.exe assembly's progress.
The file is located at
D:\MyService\MyService\bin\Release\MyService.InstallLog.
The Rollback phase completed successfully.
The transacted install has completed.
And this is how i write log entry;
EventLog.WriteEntry("MyServices", logMessage, logType);
Can someone please help me what wrong i am doing.
I had a similar problem and solved it by:
Stop current service.
Close task manager, service manage tool and windows event viewer.
Use intallutil.exe /u to uninstall current service.
delete event log registry key of current service.(This is important!)
(\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\EventLog\yourCurrentLogName)
You can find your current log and you can confirm your deletion under windows event viewer.
Use installutil.exe to install service again.
Restart Windows.(This is important, too!)
I'm not sure whether it is necessary or not. But I kept getting errors even I logged out.
Everything got fine since I restart windows.
(Maybe AWS cached something?)
Need to create a EventSource in this way:
if (!EventLog.SourceExists(source))
{
var eventSourceData = new EventSourceCreationData(source, logName);
EventLog.CreateEventSource(eventSourceData);
}
source - "MyServices"
logName - some of your program name.
Insert the data need in this way:
log = new EventLog();
log.Source = "MyServices";
log.WriteEntry(message, entryType);
We recently were able to get a small redhat server to experiment with shiny-server. Our IT department got shiny-server running and installed the Oracle client but I can't get ROracle to work in shiny-server. They (IT) have decided that it is an application issue and are starting to give up...
Initially ROracle didn't work on the server at all but we got it working from my user account by setting the LD_LIBRARY_PATH in my .bashrc file. With that done I can log into the server, and query the database from R. I can even use runApp() to run my shiny app from R.
When I try to access that same app through shiny-server I get the following error:
Listening on port 40679
Loading required package: DBI
Error in dyn.load(file, DLLpath = DLLpath, ...) :
unable to load shared object '/usr/lib64/R/library/ROracle/libs/ROracle.so':
libclntsh.so.11.1: cannot open shared object file: No such file or directory
Error : package or namespace load failed for 'ROracle'
Error : package or namespace load failed for 'ROracle'
which is the same error I was getting on my account before I set the LD_LIBRARY_PATH variable. The server is running as user shiny but apparently won't run any startup scripts so what fixed it for my user won't fix it for the shiny user. This is all very far outside my area of knowledge and as I said, our IT department says they are out of ideas.
I don't have sudo access to the server so the things I can try are limited. I tried setting the LD_LIBRARY_PATH from my server.R script before loading ROracle with Sys.setenv() and by using system() but those didn't work. Our DBA that has been trying to help me tried setting the LD_LIBRARY_PATH in /etc/init/shiny-server.conf but that doesn't seem to work either.
I am really hoping that someone here has some ideas.
Thanks
After a couple of frustrating days I found the solution. You need to set the LD_LIBRARY_PATH variable in the upstart script located at /etc/init/shiny-server.conf, but per the upstart documentation, you need to define it with the env keyword. So adding:
env LD_LIBRARY_PATH=/usr/lib/oracle/11.2/client64/lib:$LD_LIBRARY_PATH
to beginning of the shiny-server.conf script seems to have fixed the problem.
Here's my blog post (about a year old) with a detailed description on how to get R Studio Server going with Oracle. LD_LIBRARY_PATH, OCI Lib and all the stuff is provided. Maybe this helps someone else: http://learnfrominfo.tumblr.com/post/38382388429/connect-r-studio-server-to-an-oracle-database-with
I had the same problem with PHP and Apache.
Please refer to the Setting the Oracle Environment in the PHP documentation.
Also, see comment - which syntax did you use in /etc/init/shiny-server.conf?
SERVER.R
library
library(RMySQL)
library(caTools)
library(rpart)
library(RJDBC)
shinyServer(
function(input ,output)
{
dvr =JDBC("oracle.jdbc.OracleDriver",classPath="D:/ojdbc6.jar")
url = ""
user = ""
password = ""
jd =dbConnect(dvr,url, user, password)
a1 <- eventReactive(input$predict,
{
a<-input$ref
table2<-data.frame(dbGetQuery(jd,paste0("
select colnames from Tablename where REFNO=",a," and ROWNUM<15
")))
print(table2)
print(bs<-table2)
}
)
output$dis<-renderTable({
a1()
})
}
)
I am quite new to erlang and yaws. I after playing around with the language and easy yaws features like ehtml and normal appmods I tried to write a yapp. I followed http://yaws.hyber.org/yapp_intro.yaws and https://github.com/davide/yawn/wiki to get it more or less running. It simply consists of an .app file and a simple test.erl/beam that just has a out/1.
Yaws starts without a problem and loads the yapp. When I run yaws in the interactive mode and call test:out(somehting) it works fine. When I surf the url (with browser or curl) I get an
=ERROR REPORT==== Date ===
Yaws process died:...
and nothing returned to the client. I have been trying around a couple of things, but I haven't found the problem yet.
I believe the problem is in my .app file...
I hope you can help my. I added the content of the .app and my localhost-ssl.conf below.
marc#server:~$ cat /usr/lib/yaws/lib/api/ebin/api.app
{application, api,
[{description,"cloudia api as yapp"},
{vsn,"0.1"},
{modules,[]},
{registered, []},
{env, [
{yapp_appmods,[{"/",test}]},
]}]}.
marc#server:~$ sudo cat /etc/yaws/conf.d/localhost-ssl.conf
<server localhost>
port = 443
listen = 0.0.0.0
docroot = /usr/share/yaws
arg_rewrite_mod = api
#dir_listings = true
<ssl>
keyfile = /etc/yaws/yaws-key.pem
certfile = /etc/yaws/yaws-cert.pem
</ssl>
<opaque>
yapp_server_id = edo
</opaque>
</server>
Thanks
If you carefully follow the yapp instructions provided at the Yaws website everything should work fine. Be sure to remember to:
set the global runmod configuration variable to yapp
set your server configuration arg_rewrite_mod to yapp
within your server configuration opaque block, set bootstrap_yapps to yapp
To avoid the need to create an mnesia database for your yapp you can either try setting your yapp as an additional bootstrap_yapp in your server configuration, or just use the yapp_ets_server module instead. You can find instructions for using yapp_ets_server at the top of its source file.