I have a streaming job in which I'm listening to message from PubSub and after that reading the data from BigQuery. Data has is queried using the data received from PubSUb. This means I need to form the query dynamically and then pass it to the BigQueryIO.Read.fromQuery() function. Below is the code which is going to read data from the BigQuery and return a TableRow, but it is giving me NullPointerException where my code is executing data to read.
public class RequestDailyUsageTransform extends PTransform<PCollection<DailyUsageJob>, PCollection<TableRow>> {
private String mQuery;
private String mForDate;
private LocalDateTime billingDateTime;
#Override
public PCollection<TableRow> apply(PCollection<DailyUsageJob> input) {
TableReference tableReference = getRequestTableReference();
return input
.apply(ParDo.named("RequestUsageQuery")
.of(new RequestUsageQueryStringDoFn()))
.apply(BigQueryIO.Read.named("RequestUsageReader")
.fromQuery(mQuery)
.from(tableReference).withoutValidation())
.apply(ParDo.named("DailyRequestMapper").of(new DailyRequestMapperDoFn()))
.apply(ParDo.named("BillDailyRequestUsage")
.of(new DailyRequestsBillDoFn(mForDate, billingDateTime)));
}}
I also wanted to know how to pass the string which was generated in a DoFn in BigQueryIO.Read.fromQuery() function.
I think in the case the best thing to do would be to run a daily batch job that queries all the data, and is keyed by the userid. This will pull in more data than you would like, but allow you to locate the per user information. Unfortuately there is not currently a way do perform data dependent reads
Related
My problem is I want to change the document/table unique ID from string to int in firebase realtime database.
This is how it looks in my database:
.
I want to look like this:
.
This is my code in inserting data to firebase:
public async Task<bool> Save(CUSTOMER customer)
{
//var token = await authProvider.CreateUserWithEmailAndPasswordAsync(customer.CusEmail,customer.CusPassword);&& !string.IsNullOrEmpty(token.FirebaseToken);
var token = await authProvider.CreateUserWithEmailAndPasswordAsync(customer.CusEmail, customer.CusPassword);
var data = await firebaseClient.Child(nameof(CUSTOMER)).PostAsync(JsonConvert.SerializeObject(customer));
if (!string.IsNullOrEmpty(data.Key) && !string.IsNullOrEmpty(token.FirebaseToken))
{
return true;
}
return false;
}
When you call PostAsync, Firebase creates a new child node with its own unique ID for that data. The IDs always have the form that you see in your first screenshot, and there's no way to change that.
To specify your own ID, generate that ID in your client-side application code, pass it to the API as an additional Child() call, and use Put instead of Post. For example:
firebaseClient.Child(nameof(CUSTOMER)).Child("4815").PutAsync(JsonConvert.SerializeObject(customer));
If the number you want to use is based on the existing keys in the database, you'll need to use a transaction to perform the necessary read-then-write sequence.
Since you're considering using numeric keys, I recommend checking out Best Practices: Arrays in Firebase.
We've been using Spring Cloud Gateway for about 2 years in production. We're in the process of adding rate limiting support but we've run into a road block. One of the key endpoints we want to rate limit is our /oauth2/token endpoint. The trick here is this endpoint has credentials contained in the body that we need for our KeyResolver (to determine alleged identity). So we implemented the following KeyResolver:
public class OAuth2KeyResolver implements KeyResolver {
#Override
public Mono<String> resolve(ServerWebExchange exchange) {
String ipAddress = exchange.getRequest().getRemoteAddress().getHostName();
ServerRequest serverRequest = ServerRequest.create(exchange,
HandlerStrategies.withDefaults().messageReaders());
return serverRequest.bodyToMono(String.class).flatMap(requestBody -> {
JsonNode json = jackson.readTree(requestBody);
String clientId = json.path("clientId").asText(null);
return clientId != null ?
Mono.just(String.format("%s/%s", ipAddress, clientId) :
Mono.empty();
});
}
}
This works great to read the body and produce the desired key. However we quickly discovered that doing this consumes the body from the DataBuffer contained within the ServerHttpRequest. This seems to leave Spring Cloud Gateway unable to send the request to the downstream auth service.
Given the interface of this class, I don't think I can mutate the exchange instance to return the contents to the buffer.
How can I safely parse the body here?
Hi i am new to fix protocols, i was wondering how to process a NewOrderList from an ExecutionReport. As per a fix message document on the internet https://gainfutures.com/api/html/133e2886-5411-44b7-b781-2311a45ab540.htm i came to know that every order in the NewOrderList will be processed one at a time in the ExecutionReport.
From the above doc:
order statuses and trades are reported for individual orders, not for the list as whole.
Is there a way i can get the list of order details back from the ExecutionReport event listener using below
public void onMessage(ExecutionReport report, SessionID sessionID) {
// handler implementation
}
I'm building a plugin for Jenkins and I'm trying to validate my form (connection test method). This worked fine when all #QueryParameter were Strings.
Now I'm trying to send my form validation method an Object like this:
public FormValidation doTestConnection(
#QueryParameter("url") final String url,
#QueryParameter("timeout") final String timeout,
#QueryParameter("bypassProxy") final boolean bypassProxy,
#QueryParameter("deployerCredentialsConfig") final CredentialsConfig deployerCredentialsConfig,
#QueryParameter("resolverCredentialsConfig") final CredentialsConfig resolverCredentialsConfig
) throws ServletException {
In my global.jelly file I have this:
<f:validateButton
title="${%Test Connection}" progress="${%Testing...}"
method="testConnection"
with="url,timeout,bypassProxy,deployerCredentialsConfig,resolverCredentialsConfig"/>
My CredentialConfig class implements Serializable but I guess that is not enough becuase I'm getting this when clicking the "Test Connection" button:
java.lang.IllegalArgumentException: Failed to invoke public hudson.util.FormValidation
org.jfrog.hudson.MyBuilder$DescriptorImpl.doTestConnection(java.lang.String,java.lang.String,boolean,org.jfrog.hudson.CredentialsConfig,org.jfrog.hudson.CredentialsConfig) throws javax.servlet.ServletException
Jenkins has no good documentation for using objects inside of FormValidation calls.
Looking at the Jenkins documentation and the code behind <f:validateButton/>, I believe it's impossible have objects bind in validation logic.
The docs say (https://wiki.jenkins-ci.org/display/JENKINS/Jelly+form+controls):
The 'with' attribute specifies the input fields sent to the server for
the validation. They are matched against the field attribute or the
name attribute of other input controls. The values of the nearest
input fields above the are sent to the server, so
this means the button has to come after the input fields. Multiple
fields can be specified by using ','.
The code simply gets fields by the names - there is no "object assembly" (I believe it's only done during actual config submission).
https://github.com/jenkinsci/jenkins/blob/96ec7a267e0efba2ec99590c871db0940e35920f/war/src/main/webapp/scripts/hudson-behavior.js#L2856
I bumped into a similar problem. Looking at the code, it seems stapler tries to convert your parameter to the type you provided in the doCheck function declaration.
class HandlerImpl extends AnnotationHandler<QueryParameter> {
public Object parse(StaplerRequest request, QueryParameter a, Class type, String parameterName) throws ServletException {
String name = a.value();
if(name.length()==0) name=parameterName;
if(name==null)
throw new IllegalArgumentException("Parameter name unavailable neither in the code nor in annotation");
String value = request.getParameter(name);
if(a.required() && value==null)
throw new ServletException("Required Query parameter "+name+" is missing");
if(a.fixEmpty() && value!=null && value.length()==0)
value = null;
return convert(type,value); // <--- HERE
}
}
As a workaround, I changed the type to boolean, like so:
public FormValidation doTestConnection(
#QueryParameter("url") final String url,
#QueryParameter("timeout") final String timeout,
#QueryParameter("bypassProxy") final boolean bypassProxy,
#QueryParameter("deployerCredentialsConfig") final boolean deployerCredentialsConfig,
#QueryParameter("resolverCredentialsConfig") final boolean resolverCredentialsConfig
) throws ServletException {
This allows me to at least check if the variable is set. It might not be enough for your use case, though.
I have setup basic Request/Reply Messaging via spring amqp using POJOs and these are exchanged over RabbitMQ via a JSON message converter.
My response pojo's payload so far has been simple strings but now I need to reply back with an image at times as well.
Q1) What would be the best way to handle this? Should the image be base64 encoded as the payload string for the JSON message?
Q2) Is it possible to simply return the image as a BytesMessage instead?
Q2.a) Would spring-amqp be able to handle two separate listeners, one that returns POJOs and another that returns BytesMessage? I would think not but here's some pseudo-code around what I'm asking:
<listener-container>
<listener ref="requestHandlerA" method="handleMessage" queue-names="#{requestQueue.name}" />
<listener ref="requestHandlerB" method="handleMessage" queue-names="#{requestQueue.name}" />
</listener-container>
public class RequestHandlerA {
public ResponseDelegatePojo handleMessage(RequestDelegatePojo requestDelegate) {...}
}
public class RequestHandlerB {
public BytesMessage handleMessage(RequestDelegatePojo requestDelegate) {...}
}
Q2.b) OR ... If the MessageListener returns an Object (may be a POJO sometimes, may be a BytesMessage at other times) ... would the spring-amqp framework be able to adjust accordingly at runtime to send back POJO serialized as json sometimes and the BytesMessage as-is at other times?
Two listeners like that won't work - it will simply create two competing consumers on the same queue. One solution would be to use different queues (and different listeners).
If you want to use the same queue for both request types, probably the simplest solution would be to have your listener POJO return Object (POJO or byte[]) (Q2b) and use a custom MessageConverter.
The custom messageConverter would be pretty simple; assuming you always get JSON on the inbound, it would always delegate to a JsonMessageConverter in fromMessage() and on the outbound side (toMessage) should delegate to either a JsonMessageConverter or a SimpleMessageConverter:
if (object instanceof byte[]) {
return this.simpleMessageConverter.toMessage(object, messageProperties);
else {
return this.jsonMessageConverter.toMessage(object, messageProperties);
}
The SimpleMessageConverter will set the contentType property to application/octet-stream but you can change that if you want after calling its toMessage method (message.getMessageProperties().setContentType(...);.