How to guess Checksum Algorithm Type? - checksum

I am trying to figure out if what I have so far is correct. Just feeling that I am chasing my own tail here, but I do feel that maybe I understand it correctly.
I have this file, where it includes a checksum that is oddly short. This is a set of two files, this one and the one I copied at pastbin under. I noticed that the first file includes the checksum, however it seems that if I change any parameter or the serial number the file will than fail to load.
I am correct to assume that the software looks for some of the xml values add in a string and create the checksum, and than compares it to the checksum in the file? I am not sure how they would have been able to add the checksum in the file itself otherwise, so I guess maybe it just takes some of the values.
Second, am I correct to assume that they just truncated the checksum? I never seen one that small.
In the end I am trying to figure out how to create my own checksum calibration but at the moment I am trying to at minimum understand how it works. Far from braking it :)
<?xml version="1.0" encoding="utf-8"?>
<InstrumentData xmlns:i="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://schemas.datacontract.org/2004/07/AcquisitionEngine.Types">
<CalibrationMode>parabolic</CalibrationMode>
<CalibrationTemperatures
xmlns:d2p1="http://schemas.microsoft.com/2003/10/Serialization/Arrays">
<d2p1:double>28</d2p1:double>
<d2p1:double>52</d2p1:double>
<d2p1:double>76</d2p1:double>
<d2p1:double>90</d2p1:double>
</CalibrationTemperatures>
<Checksum>6E23D45E</Checksum>
<ClusterSize>1</ClusterSize>
<HardwareVariant>Hardware_Legacy</HardwareVariant>
<InstrumentID>undefined</InstrumentID>
<LineFrequency>LineFrequency_50Hz</LineFrequency>
<MaxCalibrationDeviation>0.152</MaxCalibrationDeviation>
<SerialNo>6328ZG200015</SerialNo>
<Version>3.2</Version>
</InstrumentData>
The second file is included here https://pastebin.com/YQ1qKZ2v
Update: I have been able to find the code that generates this but I am still not getting the same hash.
private string GenerateChecksum(string serialno)
{
StringBuilder stringBuilder = new StringBuilder();
stringBuilder.Append(this.HardwareVariant);
stringBuilder.Append(serialno);
return HashGenerator.GetHash32AsHex(stringBuilder.ToString());
}
From this line it seems to take the HadwareVariant and the serial number.
Than it seems to take this and generate a hash32 and than add an X2 at the end of it? My background is php, but am I understand this correct?
internal static class HashGenerator
{
private static HashAlgorithm CryptographicHasher;
static HashGenerator()
{
HashGenerator.CryptographicHasher = MD5.Create();
}
public static int GetHash32(string value)
{
byte[] bytes = Encoding.UTF8.GetBytes(value);
return
BitConverter.ToInt32(HashGenerator.CryptographicHasher.ComputeHash(bytes), 0);
}
public static string GetHash32AsHex(string value)
{
return HashGenerator.GetHash32(value).ToString("X2");
}
}

Related

Dataflow output parameterized type to avro file

I have a pipeline that successfully outputs an Avro file as follows:
#DefaultCoder(AvroCoder.class)
class MyOutput_T_S {
T foo;
S bar;
Boolean baz;
public MyOutput_T_S() {}
}
#DefaultCoder(AvroCoder.class)
class T {
String id;
public T() {}
}
#DefaultCoder(AvroCoder.class)
class S {
String id;
public S() {}
}
...
PCollection<MyOutput_T_S> output = input.apply(myTransform);
output.apply(AvroIO.Write.to("/out").withSchema(MyOutput_T_S.class));
How can I reproduce this exact behavior except with a parameterized output MyOutput<T, S> (where T and S are both Avro code-able using reflection).
The main issue is that Avro reflection doesn't work for parameterized types. So based on these responses:
Setting Custom Coders & Handling Parameterized types
Using Avrocoder for Custom Types with Generics
1) I think I need to write a custom CoderFactory but, I am having difficulty figuring out exactly how this works (I'm having trouble finding examples). Oddly enough, a completely naive coder factory appears to let me run the pipeline and inspect proper output using DataflowAssert:
cr.RegisterCoder(MyOutput.class, new CoderFactory() {
#Override
public Coder<?> create(List<? excents Coder<?>> componentCoders) {
Schema schema = new Schema.Parser().parse("{\"type\":\"record\,"
+ "\"name\":\"MyOutput\","
+ "\"namespace\":\"mypackage"\","
+ "\"fields\":[]}"
return AvroCoder.of(MyOutput.class, schema);
}
#Override
public List<Object> getInstanceComponents(Object value) {
MyOutput<Object, Object> myOutput = (MyOutput<Object, Object>) value;
List components = new ArrayList();
return components;
}
While I can successfully assert against the output now, I expect this will not cut it for writing to a file. I haven't figured out how I'm supposed to use the provided componentCoders to generate the correct schema and if I try to just shove the schema of T or S into fields I get:
java.lang.IllegalArgumentException: Unable to get field id from class null
2) Assuming I figure out how to encode MyOutput. What do I pass to AvroIO.Write.withSchema? If I pass either MyOutput.class or the schema I get type mismatch errors.
I think there are two questions (correct me if I am wrong):
How do I enable the coder registry to provide coders for various parameterizations of MyOutput<T, S>?
How do I values of MyOutput<T, S> to a file using AvroIO.Write.
The first question is to be solved by registering a CoderFactory as in the linked question you found.
Your naive coder is probably allowing you to run the pipeline without issues because serialization is being optimized away. Certainly an Avro schema with no fields will result in those fields being dropped in a serialization+deserialization round trip.
But assuming you fill in the schema with the fields, your approach to CoderFactory#create looks right. I don't know the exact cause of the message java.lang.IllegalArgumentException: Unable to get field id from class null, but the call to AvroCoder.of(MyOutput.class, schema) should work, for an appropriately assembled schema. If there is an issue with this, more details (such as the rest of the stack track) would be helpful.
However, your override of CoderFactory#getInstanceComponents should return a list of values, one per type parameter of MyOutput. Like so:
#Override
public List<Object> getInstanceComponents(Object value) {
MyOutput<Object, Object> myOutput = (MyOutput<Object, Object>) value;
return ImmutableList.of(myOutput.foo, myOutput.bar);
}
The second question can be answered using some of the same support code as the first, but otherwise is independent. AvroIO.Write.withSchema always explicitly uses the provided schema. It does use AvroCoder under the hood, but this is actually an implementation detail. Providing a compatible schema is all that is necessary - such a schema will have to be composed for each value of T and S for which you want to output MyOutput<T, S>.

How to add column name as header when using dataflow to export data to csv

I am exporting some data to csv by Dataflow, but beyond data I want to add each column names as the first line on the output file such as
col_name1, col_name2, col_name3, col_name4 ...
data1.1, data1.2, data1.3, data1.4 ...
data2.1 ...
Is there anyway to do with current API?(searched around TextIO.Write but didn't find anything seems relevant...) or is there anyway I could sort of "insert" column name at the head of to-be-exported PCollection and enforce the the data to be written in order...?
There is no built-in way to do that using TextIO.Write. PCollections are unordered so it isn't possible ot add an eleemnt to the front. You could write a custom BoundedSink which does this.
Custom sink APIs are now available if you want to be the brave one to craft a CSV sink. Current workaround which builds up the output as a single string and outputs it all at finish bundle:
PCollection<String> output = data.apply(ParDo.of(new DoFn<String, String>() {
private static final long serialVersionUID = 0;
String new_line = System.getProperty("line.separator");
String csv_header = "id, stuff1, stuff2, stuff3" + new_line;
StringBuilder csv_body = new StringBuilder().append(csv_header);
#Override
public void processElement(ProcessContext c) {
csv_body.append(c.element()).append(newline);
}
#Override
public void finishBundle(Context c) throws Exception {
c.output(csv_body);
}
})).apply(TextIO.Write.named("WriteData").to(options.getOutput()));
This will only work if your BIG output string fits in memory
As of Dataflow SDK version 1.7.0, you have withHeader function in TextIO.Write .
So you can do this:
TextIO.Write.named("WriteToText")
.to("/path/to/the/file")
.withHeader("col_name1,col_name2,col_name3,col_name4")
.withSuffix(".csv"));
A new line character is automatically added to the end of the header.

Is there anything like "CheckSum" in Dart (on Objects)?

For Testing purposes I'm trying to design a way to verify that the results of statistical tests are identical across versions, platforms and such. There are a lot things that go on that include ints, nums, dates, Strings and more inside our collections of Objects.
In the end I want to 'know' that the whole set of instantiated objects sum to the same value (by just doing something like adding the checkSum of all internal properties).
I can write low level code for each internal value to return a checkSum but I was thinking that perhaps something like this already exists.
Thanks!
_swarmii
This sounds like you should be using the serialization library (install via Pub).
Here's a simple example to get you started:
import 'dart:io';
import 'package:serialization/serialization.dart';
class Address {
String street;
int number;
}
main() {
var address = new Address()
..number = 5
..street = 'Luumut';
var serialization = new Serialization()
..addRuleFor(address);
Map output = serialization.write(address, new SimpleJsonFormat());
print(output);
}
Then depending on what you want to do exactly, I'm sure you can fine tune the code for your purpose.

Biztalk mapping Date to String

I'm working on a biztalk project and use a map to create the new message.
Now i want to map a datefield to a string.
I thought i can do it on this way with an Function Script with inline C#
public string convertDateTime(DateTime param)
{
return System.Xml.XmlConvert.ToString(param,ÿyyyMMdd");
}
But this doesn't work and i receive an error. How can i do the convert in the map?
It's a Biztalk 2006 project.
Without the details of the error you are seeing it is hard to be sure but I'm quite sure that your map is failing because all the parameters within the BizTalk XSLT engine are passed as strings1.
When I try to run something like the function you provided as inline C# I get the following error:
Object of type 'System.String' cannot be converted to type 'System.DateTime'
Replace your inline C# with something like the following:
public string ConvertDateTime(string param1)
{
DateTime inputDate = DateTime.Parse(param1);
return inputDate.ToString("yyyyMMdd");
}
Note that the parameter type is now string, and you can then convert that to a DateTime and perform your string format.
As other answers have suggested, it may be better to put this helper method into an external class - that way you can get your code under test to deal with edge cases, and you also get some reuse.
1 The fact that all parameters in the BizTalk XSLT are strings can be the source of a lot of gotchas - one other common one is with math calculations. If you return numeric values from your scripting functoids BizTalk will helpfully convert them to strings to map them to the outbound schema but will not so helpfully perform some very random rounding on the resulting values. Converting the return values to strings yourself within the C# will remove this risk and give you the expected results.
If you're using the mapper, you just need a Scripting Functiod (yes, using inline C#) and you should be able to do:
public string convertDateTime(DateTime param)
{
return(param.ToString("YYYYMMdd");
}
As far as I know, you don't need to call the System.Xml namespace in anyway.
I'd suggest
public static string DateToString(DateTime dateValue)
{
return String.Format("{0:yyyyMMdd}", dateValue);
}
You could also create a external Lib which would provide more flexibility and reusability:
public static string DateToString(DateTime dateValue, string formatPicture)
{
string format = formatPicture;
if (IsNullOrEmptyString(formatPicture)
{
format = "{0:yyyyMMdd}";
}
return String.Format(format, dateValue);
}
public static string DateToString(DateTime dateValue)
{
return DateToString(dateValue, null);
}
I tend to move every function I use twice inside an inline script into an external lib. Iit will give you well tested code for all edge cases your data may provide because it's eays to create tests for these external lib functions whereas it's hard to do good testing on inline scripts in maps.
This blog will solve your problem.
http://biztalkorchestration.blogspot.in/2014/07/convert-datetime-format-to-string-in.html?view=sidebar
Regards,
AboorvaRaja
Bangalore
+918123339872
Given that maps in BizTalk are implemented as XSL stylesheets, when passing data into a msxsl scripting function, note that the data will be one of types in the Equivalent .NET Framework Class (Types) from this table here. You'll note that System.DateTime isn't on the list.
For parsing of xs:dateTimes, I've generally obtained the /text() node and then parse the parameter from System.String:
<CreateDate>
<xsl:value-of select="userCSharp:GetDateyyyyMMdd(string(s0:StatusIdChangeDate/text()))" />
</CreateDate>
And then the C# script
<msxsl:script language="C#" implements-prefix="userCSharp">
<![CDATA[
public System.String GetDateyyyyMMdd(System.String p_DateTime)
{
return System.DateTime.Parse(p_DateTime).ToString("yyyyMMdd");
}
]]>

How to autogenerate random password in asp.net mvc application?

No need to write it again... the question says it all.
You can use the built-in function included in the namespace System.Web.Security.
Membership.GeneratePassword Method
Generates a random password of the specified length.
Here's a nice article that might help you.
In the past I've done it once by using a piece of a Guid. I just created a new guid, converted it to a string and took the piece I wanted, I think I used the characters in the back, or the other way around.
Tested it with 100 loops and every time the string was different.
Doesn't has anything to do with MVC though...
public string CreatePassword(int length)
{
const string valid = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890";
StringBuilder res = new StringBuilder();
Random rnd = new Random();
while (0 < length--)
{
res.Append(valid[rnd.Next(valid.Length)]);
}
return res.ToString();
}

Resources