Grok extracting data from matched pattern - parsing

I have this message as input:
Feb 18 04:35:46 xxxx zzzz-nginx_error 2016/02/18 04:35:39 [error] 28585#0: *3120 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: xx.xx.xx.xx, server: xxxxxx, request: "HEAD / HTTP/1.1", upstream: "fastcgi://unix:/var/run/default.sock:", host: "xxxxxx"
And I am parsing it with:
grok {
match => {
"message" => [
"(?<logstamp>\h{3} \d{2} \d{2}:\d{2}:\d{2}) (?<hostname>[^\s]+) (?<source>[^\s]+) (?<ngxstamp>[^\s]+ [^\s]+) %{GREEDYDATA:log}"
}
}
Which is fine, but I also want to extract client: xx.xx.xx.xx while keeping it inside %{GREEDYDATA:log}.
I've tried
"(?<logstamp>\h{3} \d{2} \d{2}:\d{2}:\d{2}) (?<hostname>[^\s]+) (?<source>[^\s]+) (?<ngxstamp>[^\s]+ [^\s]+) %{DATA:log} (?<client>%{IP})%{GREEDYDATA:log}"
but this just breaks the output as:
log: [error] 28585#0: *3120 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client:, , server: xxxxxx, request: "HEAD / HTTP/1.1", upstream: "fastcgi://unix:/var/run/default.sock:", host: "xxxxxx"
client: xx.xx.xx.xx
(notice the IP is truncated from log)
Can I just extract the data I need or should I join them with something like:
mutate {
replace => {
"log" => "%{DATA:log} (?<client>%{IP})%{GREEDYDATA:log}"
}
}
?

I just realized the answer was staring me in the face. This is the pattern:
"(?<logstamp>\h{3} \d{2} \d{2}:\d{2}:\d{2}) (?<hostname>[^\s]+) (?<source>[^\s]+) (?<ngxstamp>[^\s]+ [^\s]+) %{DATA:log} (?<client>%{IP})%{GREEDYDATA:log2}"
And this is the join:
mutate {
replace => {
"log" => "%{log} %{client}%{log2}"
}
}

Related

SAP CAP using cds.User attributes

I authenticate on the server by feeding req.user with an instance of cds.User, and I add some attributes:
User {
id: '110226363079182595683',
attr: { name: 'depth1', email: 'depth1#protonmail.com' },
_roles: { 'identified-user': true, 'authenticated-user': true }
}
This allows me to call my CDS services with an authenticated user.
It works well.
Then, I have in my CDS schema an entity :
entity Comments {
key ID : Integer;
project : Association to Projects;
title : String;
text : String;
CreatedBy : String #cds.on.insert : $user;
CreatedByName : String #cds.on.insert : $user.name;
}
My schema on an SQLite database.
I start the server, I launch a UI5 application which allows to insert comments in OData V4 and here is what happens:
HTTP request:
--batch_id-1642708209182-45
Content-Type:application/http
Content-Transfer-Encoding:binary
POST Projects(1)/comments HTTP/1.1
Accept:application/json;odata.metadata=minimal;IEEE754Compatible=true
Accept-Language:fr-FR
Content-Type:application/json;charset=UTF-8;IEEE754Compatible=true
{"ID":0,"text":"test comment"}
--batch_id-1642708209182-45--
Group ID: $auto
Server Log:
[cds] - > CREATE Projects(1)/comments
HTTP Response:
--batch_id-1642708209182-45
content-type: application/http
content-transfer-encoding: binary
HTTP/1.1 201 Created
odata-version: 4.0
content-type: application/json;odata.metadata=minimal;IEEE754Compatible=true
location: Comments(101)
{"#odata.context":"../$metadata#Comments/$entity","ID":101,"project_ID":1,"title":null,"text":"test comment","CreatedBy":"110226363079182595683","CreatedByName":"depth1"}
--batch_id-1642708209182-45--
HTTP Request:
--batch_id-1642708209355-46
Content-Type:application/http
Content-Transfer-Encoding:binary
GET Projects(1)/comments(101)?$select=CreatedByName,ID,text HTTP/1.1
Accept:application/json;odata.metadata=minimal;IEEE754Compatible=true
Accept-Language:fr-FR
Content-Type:application/json;charset=UTF-8;IEEE754Compatible=true
--batch_id-1642708209355-46--
Group ID: $auto
Server log
[cds] - > READ Projects(1)/comments(101) { '$select': 'CreatedByName,ID,text' }
HTTP Response
--batch_id-1642708209355-46
content-type: application/http
content-transfer-encoding: binary
HTTP/1.1 200 OK
odata-version: 4.0
content-type: application/json;odata.metadata=minimal;IEEE754Compatible=true
{"#odata.context":"../$metadata#Comments(CreatedByName,ID,text)/$entity","CreatedByName":null,"ID":101,"text":"aaa"}
--batch_id-1642708209355-46--
In my Database, CreatedBy is filled but not CreatedByName.
Also in the Create request i got the CreatedByName filled by the server and returned it's really strange.
How can i insert some cds.User attributes to the database ?!
Thank you !
Oww
I found a solution ! By adding a behavior in service.js
const cds = require('#sap/cds')
module.exports = cds.service.impl(function() {
this.before('CREATE', 'Comments', fillData)
})
async function fillData(req) {
req.data.CreatedByName = req.user.attr.name;
}

Missing access token for authorization on Ebay Browse API

I try to search items from eBay API. Within the server.js file at my Apollo Server 2, I pass the token string by context property while instantiation (s. Doku: Apollo context argument). So every request contains the authentication HTTP header property. As a tryout, for now, I just use the fixed token string. This will be changed later if I work for the client.
server.js
import { ApolloServer } from 'apollo-server'
import schema from './schema'
const server = new ApolloServer({
schema,
context: ({ req }) => {
const token = 'Bearer v^1.1#i^1#I^3#f^0#p^1#r^0#t^H4sIAAA...' // my token
return {
...req,
headers: {
...req.headers,
// enrich the header with oauth token
authorization: token,
},
}
},
})
server.listen().then(({ url }) => console.log(`🚀 Server ready at ${url}`))
resolver method
// A map of functions which return data for the schema.
const resolvers = {
Query: {
books(root, { keyword = '' }, context) {
console.log(context.headers)
fetch(`https://api.ebay.com/buy/browse/v1/item_summary/?q=${keyword}`)
.then(response => response.json())
.then(json => console.log(json))
return []
}
}
}
The context.header contains the authorization property:
{ host: 'localhost:4000',
connection: 'keep-alive',
'content-length': '108',
accept: '*/*',
origin: 'http://localhost:4000',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
dnt: '1',
'content-type': 'application/json',
referer: 'http://localhost:4000/',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'de,en;q=0.9',
authorization: 'Bearer v^1.1#i^1#f^0#p^1#r^0#I^3#t^H4sIAAAAAAAAAOV...'
}
The JSON response contains the error with errorId 1002. It says Access token is missing in the Authorization HTTP request header.:
{ errors:
[ { errorId: 1002,
domain: 'OAuth',
category: 'REQUEST',
message: 'Missing access token',
longMessage: 'Access token is missing in the Authorization HTTP request header.' } ] }
Additionally, I use a new browser tab, enter the URL https://api.ebay.com/buy/browse/v1/item_summary/search?q=test and add the same authorization header property (I use the ModHeader chrome extension). I hit enter, the request works and I get the expected JSON.
It is confusing and I don't know what I'm doing wrong while passing the token. Does somebody have an idea?
The headers you see are the ones being sent in the request to your GraphQL server. All you've done is modified them to include the Authorization header and then included your entire request object as your context -- you're not passing any header information to the fetch call actually getting the data from eBay. Minimally, you want to do something like this:
fetch(`https://api.ebay.com/buy/browse/v1/item_summary/?q=${keyword}`, {
headers: {
Authorization: context.headers.authorization,
},
})
Also bear in mind that the fetch call should be returned inside your resolver, otherwise it won't be awaited.

Rails xapi call to scorm cloud fails

I'm trying to send data from Rails to an LRS. At the moment, testing is happening using SCORM cloud. I have a valid endpoint and authentication values. The data is being sent from the controller, but the server is refusing it. I have tested the xapi statement in http://tincanapi.com/statement-generator - it validates and sends. I have added statements/ to the end of the endpoint which has stopped it moaning about malformed URL.
Full error result
Started GET "/sendx" for 127.0.0.1 at 2017-03-01 18:20:28 +0000
Started GET "/sendx" for 127.0.0.1 at 2017-03-01 18:20:28 +0000
Processing by SendxapisController#send as HTML
Processing by SendxapisController#send as HTML
Lrsconfig Load (0.0ms) SELECT "lrsconfigs".* FROM "lrsconfigs" WHERE "lrscon
figs"."id" = $1 LIMIT 1 [["id", 1]]
Lrsconfig Load (0.0ms) SELECT "lrsconfigs".* FROM "lrsconfigs" WHERE "lrscon
figs"."id" = $1 LIMIT 1 [["id", 1]]
opening connection to cloud.scorm.com:443...
opened
starting SSL for cloud.scorm.com:443...
SSL established
<- "POST /tc/E9D3QJZJST/sandbox/statements/?actor[mbox]=mailto%3AKarl%40example.
com&actor[name]=Karl&verb[id]=http%3A%2F%2Fadlnet.gov%2Fexpapi%2Fverbs%2Fanswere
d&verb[display][en-US]=answered&object[id]=http%3A%2F%2Fadlnet.gov%2Fexpapi%2Fac
tivities%2Fexample&object[definition][name][en-US]=Karl%20Activity&object[defini
tion][description][en-US]=Karl%20activity%20description HTTP/1.1\r\nContent-Type
: application/json\r\nX-Experience-Api-Version: 1.0.0\r\nAuthorization: Basic Zl
dIQjFWSGtid3lVLWFCUTNkVTpzSVpKQnRwT1hWS2lPc2QzbTZB\r\nConnection: close\r\nHost:
cloud.scorm.com\r\nContent-Length: 0\r\n\r\n"
<- ""
-> "HTTP/1.1 400 Bad Request\r\n"
-> "Content-Length: 37\r\n"
-> "Connection: close\r\n"
-> "Access-Control-Allow-Headers: Content-Type,Content-Length,Authorization,If-M
atch,If-None-Match,X-Experience-API-Version,X-Experience-API-Consistent-Through\
r\n"
-> "Access-Control-Allow-Methods: HEAD,GET,POST,PUT,DELETE\r\n"
-> "Access-Control-Allow-Origin: *\r\n"
-> "Access-Control-Expose-Headers: ETag,Last-Modified,Cache-Control,Content-Type
,Content-Length,WWW-Authenticate,X-Experience-API-Version,X-Experience-API-Consi
stent-Through\r\n"
-> "Date: Wed, 01 Mar 2017 18:20:28 GMT\r\n"
-> "Server: Apache\r\n"
-> "X-Experience-API-Version: 1.0.0\r\n"
-> "X-XSS-Protection: 1; mode=block\r\n"
-> "X-Cache: Error from cloudfront\r\n"
-> "Via: 1.1 f5d27f80802e2b6e66ec3970da5568b8.cloudfront.net (CloudFront)\r\n"
-> "X-Amz-Cf-Id: I4IiYfXehWEqBN04LhHpCdhivUq8_6xrBCztJQemPHg8cV7vjhElEg==\r\n"
-> "\r\n"
reading 37 bytes...
-> "Missing required argument: statements"
read 37 bytes
Conn close
Code that generates this
#lrsconfig = Lrsconfig.find(1)
#auth = { username: #lrsconfig.lrsusername, password: #lrsconfig.lrspassword }
#header = {
'Content-Type' => 'application/json',
'X-Experience-API-Version' => #lrsconfig.XAPIversion
}
#xapi = {
"actor":
{
"mbox": "mailto:Karl#example.com",
"name": "Karl"
},
"verb": {
"id": "http://adlnet.gov/expapi/verbs/answered",
"display": {
"en-US": "answered"
}
},
"object": {
"id": "http://adlnet.gov/expapi/activities/example",
"definition": {
"name": {
"en-US": "Karl Activity"
},
"description": {
"en-US": "Karl activity description"
}
}
}
}
# Make and send an xAPI call
response = HTTParty.post(#lrsconfig.lrsendpoint,
basic_auth: #auth,
headers: #header,
query: #xapi,
#body: #xapi,
debug_output: $stdout
)
It looks from the error message as though SCORM cloud can't read the statement from the HTTP request. There could be a couple of reasons for this.
Firstly, you have the #xapi object being sent in the query rather than the body (it looks like you've tried a few things here, but body is where it should live). Secondly, your #xapi object is a Ruby object when you send it, rather than a JSON object. You need to assign #xapi.to_json to the body, rather than just #xapi.

google auth1 to oauth2 migration: Invalid authorization header

I've a script to migrate from google oauth1 to oauth2 using oauth gem
everything seems right and i can't detect any problem with the headers but it still responding with: Invalid authorization header.
Here're the headers:
opening connection to accounts.google.com:443...
opened
starting SSL for accounts.google.com:443...
SSL established
<- "POST /o/oauth2/token HTTP/1.1\r\nContent-Type: application/x-www-form-urlencoded\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nAccept: /\r\nUser-Agent: OAuth gem v0.4.7\r\nContent-Length: 193\r\nAuthorization: OAuth oauth_consumer_key=\"mykey.com\", oauth_nonce=\"LdBeaxxxxxxxxxxxxxxIxgd03U1DHYbs\", oauth_signature=\"wXIuxxxxxxxyxxxxj0%3D\", oauth_signature_method=\"HMAC-SHA1\", oauth_timestamp=\"1416497830\", oauth_token=\"1%2F9G2sKmQxxxxxxxxxxmDXbqqifoRBGUAii-D5sw2o\", oauth_version=\"1.0\"\r\nConnection: close\r\nHost: accounts.google.com\r\n\r\n"
<- "client_id=the_id&client_secret=C-7xxxxxxxxxboMcinh3ofV&grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Amigration%3Aoauth1"> "HTTP/1.1 400 Bad Request\r\n"
-> "Content-Type: application/json\r\n"
-> "Cache-Control: no-cache, no-store, max-age=0, must-revalidate\r\n"
-> "Pragma: no-cache\r\n"
-> "Expires: Fri, 01 Jan 1990 00:00:00 GMT\r\n"
-> "Date: Thu, 20 Nov 2014 12:37:03 GMT\r\n"
-> "X-Content-Type-Options: nosniff\r\n"
-> "X-Frame-Options: SAMEORIGIN\r\n"
-> "X-XSS-Protection: 1; mode=block\r\n"
-> "Server: GSE\r\n"
-> "Alternate-Protocol: 443:quic,p=0.01\r\n"
-> "Connection: close\r\n"
-> "\r\n"
reading all...
-> "{\n \"error\" : \"invalid_request\",\n \"error_description\" : \"Invalid authorization
header.\"\n}"
read 90 bytes
Conn close
and here's the code:
oauth1_consumer_key = "mykey"
oauth1_consumer_secret = "Gxxxxxxxxxxxxxxxxqb8"
# OAuth 1 - User Token / Secret.
oauth1_token = '1/9G2xxxxxxxxxxxxxxsw2o'
oauth1_secret = 'a-xxxxxxxxxxxxxxxmqG'
# OAuth 2 - Application ID / Secret
oauth2_client_id = "the_id"
oauth2_client_secret = "C-xxxxxxxxxxxxxxxxV"
# Migration Parameters.
params = {
"grant_type" => "urn:ietf:params:oauth:grant-type:migration:oauth1",
"client_id" => oauth2_client_id,
"client_secret" => oauth2_client_secret,
# "oauth_signature_method" => "HMAC-SHA1"
}
# Create the consumer object.
consumer = OAuth::Consumer.new(
oauth1_consumer_key,
oauth1_consumer_secret,
:site => 'https://accounts.google.com',
:scheme => :header
)
# Create the access token object.
access_token = OAuth::AccessToken.new(consumer, oauth1_token, oauth1_secret)
resp = access_token.post(
"/o/oauth2/token",
params,
{ 'Content-Type' => 'application/x-www-form-urlencoded' })
if resp.code.to_s != "200"
# Raise an error.
raise "#{resp.code} - #{resp.body}"
end
Any ideas?
After updating the oauth gem, I tried it again and it worked.

Groovy Httpbuilder authentication with Realm + RABBIT MQ

I am trying to to make a rabbitmq http api call to know how queues are there and other infos...
I need 3 variables to pass on to the api
1) url: (http://localhost:55672/api) 2) username/password: guest/guest
3) realm: "RabbitMQ Management" //i am not sure if this is important
4) path: "/queues"
when i make curl statement it gives a positive response
sudo curl -i -u guest:guest (http://localhost:55672)/api/queues
HTTP/1.1 200 OK
Server: MochiWeb/1.1 WebMachine/1.7 (participate in the frantic)
Date: Tue, 03 Jul 2012 01:39:05 GMT
Content-Type: application/json
Content-Length: 6176
Cache-Control: no-cache
but using httpbuilder from groovy. here is the code
def http = new HTTPBuilder("(http://localhost:55672/api)")
http.auth.basic 'guest','guest'
http.request(GET) { req ->
uri.path = '/queues'
response.success = { resp, reader ->
assert resp.statusLine.statusCode == 200
println "Got response: ${resp.statusLine}"
println "Content-Type: ${resp.headers.'Content-Type'}"
println reader.json
}
response.'404' = { println 'Not found' }
}
I am getting "not found" as the result. I am not including realm because I am unable to if i can insert "realm" in httpbuilder. it only comes with OAuth however I need to use basic auth for rabbit mq http api calls.
Does anyone knows how to include realm name in httpbuilder groovy for basic authentication? is there any other way. Kindly let me know! thanks!
Does this work?
def http = new HTTPBuilder( 'http://localhost:55672' )
http.auth.basic 'guest','guest'
http.request(GET) { req ->
uri.path = '/api/queues'
response.success = { resp, reader ->
assert resp.statusLine.statusCode == 200
println "Got response: ${resp.statusLine}"
println "Content-Type: ${resp.headers.'Content-Type'}"
println reader.json
}
response.'404' = { println 'Not found' }
}
Took the braces and the path out of your base url, added /api to the path

Resources