How to parse the avro schema encoded as string from kafka in apache spark ?
I am working with apache spark streaming. I have stored my clickstreams in avro format, I used divolte collector to get the clicks. Further i am using kafka to get the clickstreams in real time into the spark streaming. Now i want to deserialise this avro string into a schema for further use by spark.
I used scala case class and genseler.scalavro library for achieving it but wasnt successful. Below is the code
case class Change(detectedDuplicate : Boolean,
detectedCorruption: Boolean,
firstInSession: Boolean,
timestamp: Long,
remoteHost: String,
referer: String,
location: String,
viewportPixelWidth: Int,
viewportPixelHeight: Int,
screenPixelWidth: Int,
screenPixelHeight: Int,
partyId: String,
sessionId: String,
pageViewId: String,
eventType: String,
userAgentString: String,
userAgentName: String,
userAgentFamily: String,
userAgentVendor: String,
userAgentType: String,
userAgentVersion: String,
userAgentDeviceCategory: String,
userAgentOsFamily: String,
userAgentOsVersion: String,
userAgentOsVendor: String)
object kafkaParser{
def parse(event: String): Change = {
val m = AvroType[Change]
return new Change(m.schema)// gives me a error at this point unspecified value parameters
}
}
m.schema gives me the schema for the avro file.
http://genslerappspod.github.io/scalavro/
Please help me how to go forward with this.
Related
func generateDescription(_ prefix: String) {
return (prefix + " Some Text Here")
}
let str: String = generateDescription("Some prefix text here")
How do I cast or generate a StaticString from this runtime generated string? I need to pass str to a method from a library that has a StaticString parameter (I have no control over the library).
I am thinking of something like this:
let staticStr = StaticString(str)
But this is not the correct way.
Thanks!
(Btw this is not a duplicate to this question: Convert String to StaticString)
I'm afraid it is a duplicate of Convert String to StaticString - there isn't a mechanism to convert to a static string during the execution of your program, as the whole purpose of static string is to have a string that's fully defined at compile time.
If you want to use a static string, you need to define it entirely up front in your code as a StaticString:
let staticString = "Hello, World!"
I try to use Swift 4.1's new feature to convert snake-case to camelCase during JSON decoding.
Here is the example:
struct StudentInfo: Decodable {
internal let studentID: String
internal let name: String
internal let testScore: String
private enum CodingKeys: String, CodingKey {
case studentID = "student_id"
case name
case testScore
}
}
let jsonString = """
{"student_id":"123","name":"Apple Bay Street","test_score":"94608"}
"""
do {
let decoder = JSONDecoder()
decoder.keyDecodingStrategy = .convertFromSnakeCase
let decoded = try decoder.decode(StudentInfo.self, from: Data(jsonString.utf8))
print(decoded)
} catch {
print(error)
}
I need provide custom CodingKeys since the convertFromSnakeCase strategy can't infer capitalization for acronyms or initialisms (such as studentID) but I expect the convertFromSnakeCase strategy will still work for testScore. However, the decoder throws error ("No value associated with key CodingKeys") and it seems that I can't use convertFromSnakeCase strategy and custom CodingKeys at the same time. Am I missing something?
The key strategies for JSONDecoder (and JSONEncoder) are applied to all keys in the payload – including those that you provide a custom coding key for. When decoding, the JSON key will first be mapped using the given key strategy, and then the decoder will consult the CodingKeys for the given type being decoded.
In your case, the student_id key in your JSON will be mapped to studentId by .convertFromSnakeCase. The exact algorithm for the transformation is given in the documentation:
Capitalize each word that follows an underscore.
Remove all underscores that aren't at the very start or end of the string.
Combine the words into a single string.
The following examples show the result of applying this strategy:
fee_fi_fo_fum
Converts to: feeFiFoFum
feeFiFoFum
Converts to: feeFiFoFum
base_uri
Converts to: baseUri
You therefore need to update your CodingKeys to match this:
internal struct StudentInfo: Decodable, Equatable {
internal let studentID: String
internal let name: String
internal let testScore: String
private enum CodingKeys: String, CodingKey {
case studentID = "studentId"
case name
case testScore
}
}
I try to use Swift 4.1's new feature to convert snake-case to camelCase during JSON decoding.
Here is the example:
struct StudentInfo: Decodable {
internal let studentID: String
internal let name: String
internal let testScore: String
private enum CodingKeys: String, CodingKey {
case studentID = "student_id"
case name
case testScore
}
}
let jsonString = """
{"student_id":"123","name":"Apple Bay Street","test_score":"94608"}
"""
do {
let decoder = JSONDecoder()
decoder.keyDecodingStrategy = .convertFromSnakeCase
let decoded = try decoder.decode(StudentInfo.self, from: Data(jsonString.utf8))
print(decoded)
} catch {
print(error)
}
I need provide custom CodingKeys since the convertFromSnakeCase strategy can't infer capitalization for acronyms or initialisms (such as studentID) but I expect the convertFromSnakeCase strategy will still work for testScore. However, the decoder throws error ("No value associated with key CodingKeys") and it seems that I can't use convertFromSnakeCase strategy and custom CodingKeys at the same time. Am I missing something?
The key strategies for JSONDecoder (and JSONEncoder) are applied to all keys in the payload – including those that you provide a custom coding key for. When decoding, the JSON key will first be mapped using the given key strategy, and then the decoder will consult the CodingKeys for the given type being decoded.
In your case, the student_id key in your JSON will be mapped to studentId by .convertFromSnakeCase. The exact algorithm for the transformation is given in the documentation:
Capitalize each word that follows an underscore.
Remove all underscores that aren't at the very start or end of the string.
Combine the words into a single string.
The following examples show the result of applying this strategy:
fee_fi_fo_fum
Converts to: feeFiFoFum
feeFiFoFum
Converts to: feeFiFoFum
base_uri
Converts to: baseUri
You therefore need to update your CodingKeys to match this:
internal struct StudentInfo: Decodable, Equatable {
internal let studentID: String
internal let name: String
internal let testScore: String
private enum CodingKeys: String, CodingKey {
case studentID = "studentId"
case name
case testScore
}
}
I'm trying to initialise a value in dictionary as follow,
var og:image: String
But after og: it tries to assign the type considering og as the variable to site_name, which is obviously wrong.
Is there a way to assign og:image as variable to String type using
special or escape characters?
In reference to this, apple does not provide any meaningful explanations to variable naming conventions in such a situation.
Edit-1:
Here is code snippet to clarify dictionary usage is in JSON parsing data structure,
struct Metadata: Decodable{
var metatags : [enclosedTags]
}
struct enclosedTags: Decodable{
var image: String
var title: String
var description: String
var og:site_name: String
}
You cannot use : (colon). But if you really want:
var ogCOLONimage: String
Joking apart. You could use a Dictionary or something like that:
var images: [String: String] = ["og:image" : "your string"]
Now you can access your og:image data with images["og:image"].
Swift allow you to use almost any character when naming variables. You can even use Unicode characters.
However, there are a few restrictions:
Constant and variable names can’t contain whitespace characters, mathematical symbols, arrows, private-use (or invalid) Unicode code points, or line- and box-drawing characters. Nor can they begin with a number, although numbers may be included elsewhere within the name.
Said that, it is not possible to use : in the name of a variable. However, you can use a Unicode character similar to that symbol. Depending on what you want it for, this may be a valid solution.
Here you have a list of the Unicode characters similar to : that can be used in the name of a variable:
︓
﹕
：
(https://www.compart.com/en/unicode/based/U+003A)
Based on the example you provided it would be this:
struct Metadata: Decodable{
var metatags : [enclosedTags]
}
struct enclosedTags: Decodable{
var image: String
var title: String
var description: String
var og：site_name: String
}
Turns out swift has it's own feature in terms of naming specificity of variables in structs i.e. CodingKeys, so in terms for me the below naming convention worked,
struct Metadata: Decodable{
var metatags: [enclosedTags]
}
struct enclosedTags: Decodable{
let image: String
let title: String
let description: String
let siteName: String
private enum CodingKeys : String, CodingKey{
case image = "og:image", title = "og:title", description = "og:description", siteName = "og:site_name"
}
This was rightfully pointed out by #hamish in comments (Thanks mate!)
In swift why must you pass in the first argument value without writing its name, and every subsequent value with its name when calling a function like so:
func greet(name: String, day: String) -> String {
return "Hello \(name), today is \(day)."
}
greet("Anna", day: "Tuesday")
Because the naming convention for Objective-C (see here) and Swift is to end your method name by the name of your first argument:
func greetPersonNamed(name: String, onDay day: String) -> String {
return "Hello \(name), today is \(day)."
}
greetPersonNamed("Anna", onDay: "Tuesday")
If you prefer to write the name of the first argument, you can do it like so:
func greet(name name: String, day: String) -> String { /* ... */ }
greet(name: "Anna", day: "Tuesday")
The first name refers to the external name, the second one is the one used inside your method.
EDIT
The naming guidelines for Swift 3 have been released (see here) and they differ from the ones used with Objective-C.
The name of the first argument should not be included in the method name. The external name of the first parameter can be omitted if the function intent is clear. Otherwise you should name it.
Let's say you can greet persons and pets. In that case, you should add an external name for the first argument:
func greet(person name: String, day: String)
func greet(pet name: String, day: String)
greet(person: "Anna", day: "Tuesday")
If you can only greet persons, then you can ommit it:
func greet(name: String, day: String)
greet("Anna", day: "Tuesday")