I am trying to run Twitter Streaming Example in Zeppelin. After I searched around, I added "org.apache.bahir:spark-streaming-twitter_2.11:2.0.0" into Spark Interpreter. So I can make the first part work, as in:
Apache Zeppelin 0.6.1: Run Spark 2.0 Twitter Stream App
Now I am trying to add the second half as:
case class Tweet(createdAt:Long, text:String, screenName:String)
twt.map(status=>
Tweet(status.getCreatedAt().getTime()/1000, status.getText(), status.getUser().getScreenName())
).foreachRDD(rdd=>
rdd.toDF().registerTempTable("tweets")
)
Now I got the error:
<console>:56: error: not found: type StreamingContext
val ssc = new StreamingContext(sc, Seconds(2))
^
<console>:56: error: not found: value Seconds
val ssc = new StreamingContext(sc, Seconds(2))
^
<console>:61: error: not found: value Seconds
val twt = tweets.window(Seconds(60))
Actually I added the case line, I got the above error. I really had no idea what happened here.
Any one has any clue here?
Here are details
Spark: 2.0.0
Zeppelin: 0.6.2
Thanks a lot.
=====================================================================
// All codes for your reference:
import org.apache.spark.streaming.twitter
import org.apache.spark.streaming._
import org.apache.spark.storage.StorageLevel
import scala.io.Source
import scala.collection.mutable.HashMap
import java.io.File
import org.apache.log4j.Logger
import org.apache.log4j.Level
import sys.process.stringSeqToProcess
import org.apache.spark.SparkConf
// ********************************* Configures the Oauth Credentials for accessing Twitter ****************************
def configureTwitterCredentials(apiKey: String, apiSecret: String, accessToken: String, accessTokenSecret: String) {...}
// ***************************************** Configure Twitter credentials ********************************************
val apiKey = ...
val apiSecret = ...
val accessToken = ...
val accessTokenSecret = ...
configureTwitterCredentials(apiKey, apiSecret, accessToken, accessTokenSecret)
// ************************************************* The logic itself *************************************************
val ssc = new StreamingContext(sc, Seconds(2))
val tweets = TwitterUtils.createStream(ssc, None)
val twt = tweets.window(Seconds(60))
twt.print
// above codes work correctly
// If added the following line, it failed with the above error
case class Tweet(createdAt:Long, text:String, screenName:String)
I had the same problem, and I have no idea why moving the import statements from the top to right before the new StreamingContext fixed it, but it did.
import org.apache.spark.streaming._ //moved here from top
import org.apache.spark.streaming.twitter._ //moved here from top
val ssc = new StreamingContext(sc, Seconds(2)) //existing
I had a similar issue. Using the FQCNs worked ok, so I ended up using that as a workaround.
Related
I am using a condition in my script in the REST endpoint and I am receiving the following errors:
Anyone knows what the problem is and what needs to be fixed?
Here is the code:
package CreateMultipleSubtasks
import org.apache.log4j.Logger
import com.atlassian.jira.plugin.webfragment.model.JiraHelper
import com.atlassian.jira.component.ComponentAccessor
def log1 = Logger.getLogger("atlassian-jira.log")
log1.warn("MOUNA BEFORE")
Configuration_CreateMultipleSubtasks conf = new Configuration_CreateMultipleSubtasks()
def subTaskCreatorHashMap= conf.getSubTaskCreatorHashMap()
String projectKey = subTaskCreatorHashMap["projectKey"];
String issueTypeName = subTaskCreatorHashMap["issueTypeName"];
log.warn("MOUNA PROJECT KEY "+ projectKey+" issuetypename "+issueTypeName)
def val=jiraHelper.project?.key==projectKey && issue.issueType.name==issueTypeName
return val
I have working Durable Function based on following tutorial. I have not modified code yet.
https://learn.microsoft.com/en-us/azure/azure-functions/durable/quickstart-python-vscode
How to send json file to Activity Function.
For debugging purposes how to do logging.info for json in Activity Function.
This function an HTTP starter function for Durable Functions.
import logging
import azure.functions as func
import azure.durable_functions as df
async def main(req: func.HttpRequest, starter: str) -> func.HttpResponse:
client = df.DurableOrchestrationClient(starter)
instance_id = await client.start_new(req.route_params["functionName"], None, None)
logging.info(f"Started orchestration (Ken) with ID = '{instance_id}'.")
return client.create_check_status_response(req, instance_id)
This function is an Orchestration Function
import logging
import json
import azure.functions as func
import azure.durable_functions as df
def orchestrator_function(context: df.DurableOrchestrationContext):
logging.info(f"CalcOrc")
result1 = yield context.call_activity('CalculateActivity', "Tokyo")
result2 = yield context.call_activity('CalculateActivity', "Seattle")
result3 = yield context.call_activity('CalculateActivity', "London")
return [result1, result2, result3]
main = df.Orchestrator.create(orchestrator_function)
This function is an Activity function
import logging
def main(name: str) -> str:
logging.info(f"CalcAct") # Could log contents of json sent by HTTP Post
return f"Hello {name}!"
This modified version an HTTP starter function. It works with Get but not with Post.
import logging
import json
import azure.functions as func
import azure.durable_functions as df
async def main(req: func.HttpRequest, starter: str) ->
func.HttpResponse:
#Added for testing
jsoninput = req.params.get('jsoninput')
client = df.DurableOrchestrationClient(starter)
#instance_id = await
client.start_new(req.route_params["functionName"], None, None)
instance_id = await
client.start_new(req.route_params["functionName"], jsoninput, None)
logging.info(f"Started orchestration with ID = '{instance_id}'.")
logging.info(f"jsonInput = '{jsoninput}'.")
return client.create_check_status_response(req, instance_id)
Below code will helps you to pass the json object data to the activity function:
Activity function:
def main(req: func.HttpRequest) -> func.HttpResponse:
req_body = req.get_json()
return func.HttpResponse(f"description is {req_body.get('description')}")
I am looking for a Groovy script that can create a masked password pair in Jenkins.
I've tried the following but it doesn't work.
import java.util.logging.Logger
import jenkins.model.*
import hudson.logging.*
//import com.michelin.cio.hudson.plugins.maskpasswords.*
import com.michelin.cio.hudson.plugins.maskpasswords.MaskPasswordsConfig
import com.michelin.cio.hudson.plugins.maskpasswords.MaskPasswordsBuildWrapper.VarPasswordPair
maskPasswordsConfig = MaskPasswordsConfig.getInstance()
varPasswordPairs = maskPasswordsConfig.getGlobalVarPasswordPairs()
MaskPasswordsConfig plugin = new MaskPasswordsConfig()
VarPasswordPair pwdPair = new VarPasswordPair("PWD", "myPassword")
plugin.addGlobalVarPasswordPair(pwdPair)
plugin.save()
I'm getting no results in the script console. Any help is appreciated.
#mweish, Just come across your post as I was trying to setup something similar, this should do what you are looking for:
import java.util.logging.Logger
import com.michelin.cio.hudson.plugins.maskpasswords.MaskPasswordsConfig
import com.michelin.cio.hudson.plugins.maskpasswords.MaskPasswordsBuildWrapper.VarPasswordPair
import jenkins.model.*
import hudson.logging.*
Logger logger = Logger.getLogger("")
Jenkins jenkins = Jenkins.getInstance()
MaskPasswordsConfig plugin = new MaskPasswordsConfig()
MASKED_PARAMETER_CLASSES = [
'com.michelin.cio.hudson.plugins.passwordparam.PasswordParameterDefinition',
'hudson.model.PasswordParameterDefinition'
]
def NAME_PASSWORD_PAIRS = [
[NAME:'PWD', PASSWORD:'myPassword']
]
// Add classes that should automatically be masked
MASKED_PARAMETER_CLASSES.each { maskedClass ->
plugin.addMaskedPasswordParameterDefinition(maskedClass)
}
// Add Global name/password pairs
NAME_PASSWORD_PAIRS.each { namePassPair ->
VarPasswordPair passwordPair = new VarPasswordPair(namePassPair.NAME, namePassPair.PASSWORD)
plugin.addGlobalVarPasswordPair(passwordPair)
}
plugin.save(plugin)
logger.info('Successfully Configured the Mask Passwords plugin')
I am new to the gatling load test. And i want to load test my simple project.But I got the error on response(406 not acceptable) and my gatling code is below
import io.gatling.core.Predef._
import io.gatling.http.Predef._
class simu extends Simulation {
val httpConf = http
.baseURL("http://172.24.15.225:10050/sample")
.header(HttpHeaderNames.Accept, HttpHeaderValues.ApplicationJson)
.acceptHeader("application/json, text/plain, */*")
.acceptEncodingHeader("gzip, deflate")
.acceptLanguageHeader("en-US,en;q=0.5")
val scn = scenario("Scenario Name")
.exec(
http("request_1")
.post("http://172.24.15.225:10050/sample")
.header(HttpHeaderNames.Accept, HttpHeaderValues.ApplicationJson)
//.check(status.is(406))
.body(StringBody("""{ "inputData": "Wonderful" }""")).asJSON
)
setUp(scn.inject(atOnceUsers(30)).protocols(httpConf))
}
And the response for the above is
failed in Response
Errors ------------------------------------------------------------
status.find.in(200,304,201,202,203,204,205,206,207,208,209),
but actually found 406
someone please correct my code.
But the RestAPI(postman) returns response correctly.
finally i found an answer. There is no error in the gatling load test. But the problem is my backend coding. I have changed my response from String type into the JSON format like below
object ServiceJsonProtocol extends DefaultJsonProtocol {
implicit val RequestProtocol : RootJsonFormat[Text] = jsonFormat1(Text)//request json format
implicit val ResponseProtocol : RootJsonFormat[SampleText] = jsonFormat1(SampleText) // response json format
}
It works fine
I have socket server in Java and now I need simple TCP socket in my iOS app that will:
Open socket connection
Be able to send and recieve messages
Close connection
So, only basic stuff... I need simplest solution possible.
I found few solutions, but none seem to work for me. If you could point me in right direction I will be very grateful. I need code with instructions.
Thanks!
package jcolibri.examples.ABXRecommender;
import java.io.BufferedInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.net.ServerSocket;
import java.net.Socket;
public class FileServer {
public static void main (String [] args ) throws IOException {
// create socket
#SuppressWarnings("resource")
ServerSocket servsock = new ServerSocket(13267);
while (true) {
String str = "temp.png";
File myFile = new File(str);
String absolutePathOfFirstFile = myFile.getAbsolutePath();
FileInputStream fis = new FileInputStream(absolutePathOfFirstFile);
//File myFile = new File ("temp.png");
//FileInputStream fis = new FileInputStream(myFile);
System.out.println("Waiting...");
Socket sock = servsock.accept();
System.out.println("Accepted connection : " + sock);
// sendfile
byte [] mybytearray = new byte [(int)myFile.length()];
#SuppressWarnings("resource")
BufferedInputStream bis = new BufferedInputStream(fis);
bis.read(mybytearray,0,mybytearray.length);
OutputStream os = sock.getOutputStream();
System.out.println("Sending...");
os.write(mybytearray,0,mybytearray.length);
os.flush();
sock.close();
}
}
}
This code is lightly adapted from somewhere. can't remember where though and it works for me. Your iOS code should know the IP address of your computer.
Hope that helps.