I am working on a project that receives data (json) from a number of different sources. Each source returns json in a different format, however all services fall into the same category i.e. Issues from Jira and Stories from PivotalTracker each have the same core information.
I am looking for a way to normalize this as much as possible so that I can add other services and formats in the future. Right now I am handling each response type (Jira, PivotalTracker) separately and taking action on each response independently.
So far I am thinking that I'll need a parser for each service, i.e. JiraIssueParser, PivotalTrackerStoryParser etc which transforms the response into a common format that can be used by one method to post onwards, rather than having methods for each to do the receive/parse/post.
Something like this format:
{
issue: {
title: ,
description: ,
assignee: ,
comments: {
1: {
id: ,
title: ,
body:
}
time_entries: {
1: {
id: ,
time: ,
date:
}
}
}
}
I would like to define the common schema somewhere so that each parser's output is always identical. I'm thinking this could be done with a YAML file but I'm not sure how to go about it, and how to use that in the parser.
I would greatly appreciate some suggestions on how to do this. Maybe this is a really stupid question and I should just be outputting the above format from each parser, but I think it would make sense to have some kind of format that is enforced/validated.
Suggestions are appreciated and I'm open to taking a new direction with this if anyone has any ideas. Thanks in advance.
If you're using Rails, I assume then that you are going to have a relational database at some point.
What I would suggest is to define ActiveRecord models that express your "normalized" format: Issue, Comment, TimeEntry, etc.
The job of your parsers, then, is to coerce the JSON data into the appropriate model objects and attributes and save them. Your models thus enforce the canonical data structure (i.e. the schema), and you can even use validators to do further sanity checks.
Finally, I would also save the raw JSON somewhere alongside your model, preferably also in the database. Even though you have already parsed the JSON, keeping it around will come in handy for troubleshooting. For example, if you find and fix a parsing bug, you can re-run the parser on the saved JSON without having to re-download everything the original external sources.
Related
I'm using Opeonlayers 3.18 + GeoServer. I can make a ogc filter for comparing a field and a value. How can i compare two fields?
Code below shows what i'm looking for:
var f = ol.format.ogc.filter.greaterThan('Field1', 100); // this works nicely
var f = ol.format.ogc.filter.greaterThan('Field1', 'Field2'); // this doesn't work
Equivalent working CQL filter is: 'Field1 > Field2'
Regards
This sort of filtering should work. I'd note that many databases may not have a way to perform that filter in an optimal way, and as such, there's a chance that the handling for such a filter is not correct.
The first thing to check out is if the GeoServer logs have any additional information. For most requests, there are log messages which will provide all the details (including the filter) for the request.
If OL3 is making a non-sense request, there should be some kind of parse error first. If the datastore is having trouble with the request, you might see an exception.
To help further, which version of GeoServer and which datastore are you using? Also, is there additional information in the logs?
Update: Based on the comment below, I checked out the Open Layers 3 source. If you look here, it looks like OL3 is treating the first argument as a properyName and the second as a literal. It is probably worth filling a bug/feature request on the project's GitHub page.
I need some architectural advice. I'm more into java, but trying to get up to speed with Ruby-on-rails. In the app I am building I need a convenient place to store some dictionary values that will be later used in various places of the application. These will be usually key value pairs - e.g. list of values to be used in select list.
The main objective is to keep this logic in one place of the application.
I am considering following options:
Store values in the database - i'm kind of reluctant from that, as values won't change very often.
Put all of the values in one class. In JAVA I'd have some static properties in one class holding this values (e.g. call Utils.getStates() will return list of states). How to do it ruby way?
Have some .yml file with values - read from the values. How to do it? I guess I have to parse the file in the initializer, but is there any tutorial how to do it?
Precise example? Let's say that have a model that have a field called "Type". Type can be: ['Type A', 'Type B', 'Type C'...]. And of course, for each type I want to have key and value.
I'd appreciate some suggestions about how you solve this problem in your apps.
Thanks,
Maciek
How often does the list change? Is it acceptable to have developers involved each time a value changes (updating code, re-deploying the app)? If the answer is no then store the values in a database.
Is the list of values reuseable? Then a gem or a yaml file with an initializer might be a good choice.
Is it just a small list and does not change often? Then you might want to consider a constant.
I think in Rails any data that would change at runtime and needs to be persisted, would normally be stored in the database. I think that would be the "rails way". You could save the data to yaml or json file, but that would not follow the normal flow of the MVC pattern that is so common in rails
Hey guys this has been tripping me up quite a bit. So here is the general problem:
I am writing an application that requires users to enter their Summoner Names from league of legends. I do a pretty simple data scrape of a match and enter the data into my database. Unfortunately I am having some errors registering users with "special characters".
For this example I will use one problem user: RIÇK
As you can see RICK != RIÇK. So when I do the data scrub from the site I get the correct value which I push onto an array for later use.
Once I need the player names I pull from the array as follows: (player_names is the array)
#temp_player = User.find_by_username(player_names[i].to_s)
The problem is the users with any special characters are not being pulled. Should I not be using find_by? Is to_s changing my original values? I am really quite lost on what to do and would greatly appreciate any help / advice.
Thanks in advance,
Dan
I would like to thank Brian Kung for the link to the following: joelonsoftware.com/articles/Unicode.html It does a great job giving the bare minimum a programmer truly needs to understand.
For my particular issue I had used a HTML scraper to get the contents but which kept HTML entries throughout. When using these with my SQL lookups it was obvious that things were not being found. In order to fix this I used the HTMLEntities Gem to decode the text as follows (as soon as I put the into the array originally):
requires 'RubyGems' #without this cannot include htmlentries as a gem
requires 'HTMLEntries'
coder = HTMLEntries.new
line = '<'
player_names.push(coder.decode(line))
The Takeaway
When working with text and if running into errors I would strongly recommend tracing the strings you are working with to the origin and truly understanding what encoding is being used in each process. By doing this you can easily find where things are going wrong.
Using NSXMLDocument, without using XPath, is there an easy way to parse an XML document and deserialize it into custom objects to create an object tree hierarchy?
For example, if I have the xml shown below, is it possible to put the details into a Restaurant object and a content object within it?
<restaurants>
<content>spanish name</content>
<content>english name</content>
</restaurant>
<spa>
<content>spa spanish name</content>
<content>spa english name</content>
</spa>
I will be using your answer above to extend it for programming in kissxml in iOS. As the kissXML document mentions that the XML parser behaves in the same way as NSXMLDocument, so I've asked the question using NSXMLDocument.
If you know your expected content structure, the easiest is to just use NXMLParser and loop through it looking for the bits you need and keeping track of the previous bit, building an object as you find them.
If you want tree-based approach, consider learning XQuery and XPath, they are not all that bad. Without them, the only thing NSXMLDocument really gives you is Cocoa bindings.
At the end of the day you must transform your data somehow.
With NSXMLDocument you will still do well to validate against an XML DTD if possible, to ensure you have good data.
With NSXMLParser, you are able to handle things without a formal DTD.
You only need worry about how big the data is, an how you want to parse it, then do some trial and error with test data to ensure it's grabbing what you want or need.
I've been asked to prototype a replacement "file transformation process" (that currently is a mess of SQL) using Altova's MapForce. My input is a CSV file with headers. My problem is that I need to capture both the data AND the column name to use in downstream processing.
I need to have MapForce feed a C# method (imported as that takes two parameters: fieldName and value. I can access the value trivially, but after hours pouring over the manual (1000 pages!) I haven't found any examples of how to access the field name as an output.
The reason each output needs the field name and the value has to do with how all our mappings/transformations are currently managed - on a database. The .NET code jumps in at this point and does any necessary database lookups.
For example, if I had the following file:
"Symbol", "Account", "Price", ...
"FOO", "10101", "1.23", ...
"BAR", "10201, "13.56", ...
And a static method string TransformField( string fieldName, string value ),
I'd like to map the CSV file's Symbol data output to the method's value parameter and the Field Name "Symbol" to the method's fieldName parameter.
Some limitations:
I need to keep the "wiring" visible in the MapForce GUI. I'll have non-programmers maintaining the mappings in the future. So doing all this in code is not an option.
MapForce is the tool of choice by the company. Part of the reason our original process is such a mess is because the original programmer rolled his own mapping/transformation tool (out of TSQL no less - ouch).
We can treat all inputs/outputs to the method call as strings. Conversions will happen later.
I would like to avoid using scalar literals as inputs. I already have the column names from the file - I do not want to re-type each one and feed it to my method.
I'm not sure how many users out there have experience with this tool, but after 3 days of tinkering with it, I see much potential. If only I can get past this current sticking point, I think the company will have a solid alternative to their current mess.
Thanks for any/all suggestions.
I solved my issue and, for future reference, want to post a solution. I handled my problem by using MapForce's FlexText. This allowed me to extract the header from the CSV file and "invert" the column names as data inputs to the transformation process. Once I knew the approach to take, I was able to find more information directly from Altova.
I found a couple helpful tutorials while digging through their website:
Altova Online Videos
Web Tutorial
Hope this can help someone else in the future!