I'm building a linter program for a templating (like jinja, Twig) language that has a structure called "block".
I'm sure many of you are familiar with the concept of a block in templating languages. Now, a template that contains these blocks can be overridden by another template, the child template, having the same names for the blocks, and defining their own content and functionality for it.
So when evaluating a given source code (for scope checking) the requirement is that I first evaluate all the blocks in the child template independently. Then, I have to place the blocks in the places where they were defined in the root template, and evaluation should take place again so as to check for any violation of root template's template scope attributes (variables, macros etc).
I started out by using ANTLR4 to visit the child template. So in ANTLR, I hit the block, I evaluate it there, and then call on another visitor on the root template's parse tree.
But how do I place the child block's content in root template for evaluation? There is no tree modification / mutation in ANTLR4. I'm new to this whole compiler and parser paradigm. Can anyone suggest what is the best approach to take from here, considering my requirements?
Also I'm ready to answer any other question you have just in case you feel the question is too vague, or I haven't provided enough information.
This problem -- versioning or inheritance -- can be easily handled through the use of a symbol table.
Visit the contexts corresponding to the root template blocks, saving corresponding symbol/value pairs for each block. Then visit the child block contexts to update and/or add to the symbol/value pairs for each block. Finally, evaluate the symbol table to read out the merged results.
This SO answer provides a simple symbol table.
The Antlr git repo provides an exemplary fully-typed symbol table.
Related
I am attempting to build a dataflow pipeline to process a text file which contains events that span multiple lines. The dataflow SDK TextIO class assumes each line is a new event.
My plan is to create a new TextReader and register it with the DataPipelineRunner. This new reader will know how to aggregate the multiple lines into a single line.
I am pretty sure that this approach will work but I am wondering if this is the right way to do it or if there is a simpler solution?
The text I am trying to parse is:
==============> len:45 pktype:4 mtype:2
SYMBOL: USOCSTIA151632.00
OPEN_INT: 212
PR_OPEN_INTEREST: 212
TIME_STAMP: 04/10/2015 06:30:17:420 val:1428661817
The result should be the last 4 lines concatenated together and the first line dropped.
Best regards,
Peter
Note that TextReader is an internal implementation detail class, so subclassing it would be highly discouraged and challenging to do properly.
The recommended way to define a new file-based format like yours is to subclass FileBasedSource using the user-defined source API.
In your case, I would recommend to base your class on the LineIO example from documentation, and wrap the LineReader defined there into your own class which would use LineReader as a helper for reading individual lines, but:
In startReading() it would skip until the line starting with "====>"
In readNextRecord() it would read lines until the next "====>" and bundle them into a single record.
Please make sure to carefully read the documentation to FileBasedSource and FileBasedReader: the parallelization mechanism relies on the consistency properties described there, which your format has to satisfy, for ensuring that records are not duplicated or omitted on the boundaries between adjacent processing shards. XmlSource tests are a good example of how to unit-test these properties.
Please tell us how it goes and report back with any problems or questions - we are very interested in feedback on this API.
Martin Fowler's PoEAA catalog is like a repository for Ruby gems and Rails modules, for example the ActiveRecord ORM from Rails is based on Fowler's ActiveRecord, and the DataMaper gem is based on the Data Mapper pattern. Are there any useful implementations of Martin Fowler's two-step view pattern in Ruby, e.g. in combination with a template engine?
The pattern turns domain data into HTML in two steps. It is particularly interesting if you want to compose your views into decoupled, reusable view components.
One possible solution to implement the two step view seems to be an XSLT transformation, for example with XML and Nokogiri. This means to create an intermediate xml representation of the page:
XML == (XSLT) ==> XML
XML == (XSLT) ==> HTML
A second possible solution is to use a JS template engine like vue.js, KnockoutJS, Ractive.js or React. Rails does the first step and creates an intermediate view, the JS template engine the second:
Rails Template == (Rails) == > View-Template
View-Template + JSON-Data == (vue.js/KnockoutJS/Ractive.js/React) ==> HTML
This is a new pattern to me, but I can two ways to conceptualize it. The core of the pattern seems to be building an intermediate representation first, then run it through a formatting step. In each case, the outcome is a view that looks identical regardless of which class of ActiveRecord model is being displayed.
Option 1: Intermediate Ruby Object
Using a Presenter library (Draper, ActiveDecorator, roll-your-own), you can normalize multiple ActiveRecord classes to a single public API. Then you write a single view template that can render objects with that API.
In this case you create a single view template plus one Presenter object for each ActiveRecord class you need to render. If you need to add new data to the page, you have to touch the template and all of the Presenter classes.
Option 2: HTML + CSS
It's odd, but I think HTML is a valid format to represent data, and could be considered as an intermediate, unformatted representation.
In this case you create a view template (probably a partial, possibly polymorphic) for each ActiveRecord class that produces (nearly) identical HTML. Then you use a CSS component framework to "format" the HTML into identical renderings. The HTML doesn't need to be strictly identical, as long as it all conforms to what your component framework is expecting. Adding data here means changing each view template (the CSS usually won't need modification).
I think both of these approaches are valid. The second feels more "rails-y" to me, but I think it's a departure from the spirit of the pattern, even if it technically conforms (which might be debatable).
When doing J2EE development, I find it handy for debugging to view the Java classes that are generated by the JSP compiler.
How can I do the equivalent in Ruby? Since it is all in memory, it won't generate a file that I can view. I believe it's the ERB module that generates the corresponding object for a template, so how can I actually view the object? Can I drop a debugger statement somewhere and use rdb? Is there some configuration value I can tell it to dump the object definition? I'm using rails, in case that makes a difference.
I don't think rails generates a class for your view. It basically calls eval after processing the file. Or do you mean inspecting the erb object while it's parsing your template?
If it's the latter you can find erb.rb in lib\ruby\1.9.1 I'd imagine you could just drop a debugger statement throughout that file.
I always make a habit of adding the following to my views (layout) which allows me to inspect or debug the parameters being used by the view in question.
<%= debug(params) %>
This will format all the parameters in yaml and display them in a Hash format.
Have a look at the method in the source code to get a better understanding. SOURCE
There are some differences compared with the Java way due to language differences.
Most template libraries for Ruby follow these steps when compiling/optimizing:
The template is compiled into Ruby source code -- not a class but a long procedure that appends to a string buffer while traversing the logic of the original template.
This ruby code is evaluated in order to be bound for later reference, preferably within a method body. This way, it is only parsed once by the interpreter.
The method (or other context) containing the logic of the parsed template is invoked to render it.
Anyway, the compiled template code therefore looks a lot like a much noisier version of your original template, and will generally not help you debugging, unless you're debugging the template language itself.
Anyone interested in template language implementation might enjoy a look around the Tilt code (use different template languages with the same rendering interface and optimization), and Temple (a great template language meta-implementation).
Recently I created a spike of a view engine, in which views are plain classes, and the content is created by using funny using-scope blocks.
The code together with a simple sample site is available at http://code.google.com/p/sharp-view-engine/
Here I'd like to hear your opinions regarding such an idea. Is it completely weird or maybe someone likes it?
I would actually not like that.
I can agree with DSLs (such as a Parser-Combinator or for generating XML Nodes in a data-context), but in this case I think that too much is being put in code that. And, in the end, this just complicates boundaries and leads to hard-to-maintain code. (You can already do the same, but with more verbosity just using the "standard" Web Controls. You can always use {subblock} in C# to limit a variables scope.)
The approach I prefer to use is templates with bindings (but no "code in templates"). That makes it easy for the "designer" (hopefully not me, or the next person to come along and) edit the layout of the view how they see fit. However, the core logic (the available controls and bindings) are kept in the code -- uncluttered. (Another advantage with the templates is that if they externally housed they do not require a recompile for every little change.)
Simplicity and maintainability are like ... zen.
This is an interesting idea taken to the extreme I'd say. At my shop we're using html conventions for pretty much everything except our layout. The only real html we have in the project is our Spark master page. For generating the content itself we use a convention engine that that spits out a semantic html model. (We're using the HtmlTags library from FubuMVC to build the semantic model.)
An example convention for rendering a multiline text box looks like:
public static HtmlTag Build(ElementRequest req)
{
return Tags.TextArea
.Rows(6)
.Id(req.ElementId)
.Attr("name", req.ElementId)
.Text(req.StringValue());
}
These conventions get triggered from reflecting on the view model (or we can manually call them from a helper method). The output is rendered (via ToString()) into the content section of our master page. We're joking that pretty soon we won't even need a view engine.
ps here's how we handle nesting. (Your using blocks look cluttered!)
return Tags.Div.Nest(
Tags.Button("save").AddClass("positive"),
Tags.Span.Text(" or "),
Tags.Anchor.Text("cancel").AddClass("negative")
);
Nest() is an extension method that simply takes a params array of HtmlTag and appends them to the parent's children collection.
I'm using a closed-source application that loads Lua scripts and allows some customization through modifying these scripts. Unfortunately that application is not very good at generating useful log output (all I get is 'script failed') if something goes wrong in one of the Lua scripts.
I realize that dynamic languages are pretty much resistant to static code analysis in the way C++ code can be analyzed for example.
I was hoping though, there would be a tool that runs through a Lua script and e.g. warns about variables that have not been defined in the context of a particular script.
Essentially what I'm looking for is a tool that for a script:
local a
print b
would output:
warning: script.lua(1): local 'a' is not used'
warning: script.lua(2): 'b' may not be defined'
It can only really be warnings for most things but that would still be useful! Does such a tool exist? Or maybe a Lua IDE with a feature like that build in?
Thanks, Chris
Automated static code analysis for Lua is not an easy task in general. However, for a limited set of practical problems it is quite doable.
Quick googling for "lua lint" yields these two tools: lua-checker and Lua lint.
You may want to roll your own tool for your specific needs however.
Metalua is one of the most powerful tools for static Lua code analysis. For example, please see metalint, the tool for global variable usage analysis.
Please do not hesitate to post your question on Metalua mailing list. People there are usually very helpful.
There is also lua-inspect, which is based on metalua that was already mentioned. I've integrated it into ZeroBrane Studio IDE, which generates an output very similar to what you'd expect. See this SO answer for details: https://stackoverflow.com/a/11789348/1442917.
For checking globals, see this lua-l posting. Checking locals is harder.
You need to find a parser for lua (should be available as open source) and use it to parse the script into a proper AST tree. Use that tree and a simple variable visibility tracker to find out when a variable is or isn't defined.
Usually the scoping rules are simple:
start with the top AST node and an empty scope
item look at the child statements for that node. Every variable declaration should be added in the current scope.
if a new scope is starting (for example via a { operator) create a new variable scope inheriting the variables in the current scope).
when a scope is ending (for example via } ) remove the current child variable scope and return to the parent.
Iterate carefully.
This will provide you with what variables are visible where inside the AST. You can use this information and if you also inspect the expressions AST nodes (read/write of variables) you can find out your information.
I just started using luacheck and it is excellent!
The first release was from 2015.