Using libclang, I have a cursor into a AST, which corresponds to the statement resulting from a macro expansion. I want to retrieve the original, unexpanded macro text.
I've looked for a libclang API to do this, and can't find one. Am I missing something?
Assuming such an API doesn't exist, I see a couple of ways to go about doing this, both based on using clang_getCursorExtent() to obtain the source range of the cursor - which is, presumably, the range of the original text.
The first idea is to use clang_getFileLocation() to obtain the filename and position od the range start and end, and to read the text directly from the file. If I've compiled from unsaved files then i need to deal with that, but my main concern with this approach is that it just doesn't seem right to be going outside to the filesystem when I'm sure clang holds all this information internally. There also would be implications if the AST has been loaded rather than generated, or if the source files have been modified since they were parsed.
The second approach is to call clang_tokenize() on the cursor extent. I tried doing this, and found that it fails to produce a token list for most of the cursors in the AST. Tracing into the code, it turns out that internally clang_tokenize() manipulates the supplied range and ends up concluding that it spans multiple files (presumably due to some effect of the macro expansion), and aborts. This seems incorrect to me, but I do feel that in any case I'm abusing clang_tokenize() trying to do this.
So, what's the best approach?
This is the only way I've found.
So you get the top level cursor with clang_getTranslationUnitCursor(). Then, you do clang_visitChildren(), with the visitor function passed into this returning CXChildVisit_Continue so that only the immediate children are returned. Among the children, you see the usual cursor types for top level declarations (like CXCursor_TypedefDecl, CXCursor_EnumDecl) but among them there's also CXCursor_MacroExpansion. Every single macro expansion appears to show up in a cursor with this type. You can then call clang_tokenize() on any of these cursors and it gives you the unexpanded macro text.
I have no idea why macro expansions get stuck near the top of the AST instead of within elements where they get used, it makes things pretty awkward. Example:
enum someEnum{
one = SOMEMACRO,
two,
three
}
It'd be nice if the macro expansion cursor for SOMEMACRO were within the enum declaration instead of being a sibling to it.
(I realize this is ridiculously late but I'm hoping this gets libclang more exposure, maybe someone more experienced with it can provide more insight).
Related
I'm working with Delphi 10.4.
At some places of my code I have to write long identifiers on long statements (e.g. ProductsForSale.fieldByName('quantity').asInteger * ..... / .... etc.). Is there a way to assign a friendly alias (e.g. TheCost) to this piece of code to substitute it everywhere in the application?
I don't want to use a variable or a function to do this. Only text substitution.
No, Delphi (and Pascal) knows
neither aliases (using a different name for a type/variable/function still needs you to define it anew),
nor macros (executing code in the compiler's context is very limited, mostly definings and conditions).
Even if, chances are you'd have to define it bound to a given context/scope: i.e. your favorite alias TooMuchToType might access three variables named one, two and three, but as per scope those variable's types can vary drastically. Its usage would be prone to obfuscated code and a lot of hassle the compiler has to go thru when trying to give you an error message he wants you to understand.
Why not doing exactly that at the end, but in the opposite way? First using a placeholder and when you're done you replace all of them with the actual code. Otherwise this is bascially what functions are there for, if you want it or not.
I currently work on a personal writing project which has ended up with me maintaining a few different versions due to the differences of the relevant platforms and output formats I want to support that are not trivially solved. After several instances of me glancing at pandoc and the sheer forest that it represents, I have concluded mere templates don't do what I need, and worse, that I seem to need a combination of a custom filter and writer... suffice to say: messing with the AST is where I feel way out of my depth. Enough so that, rather than asking specific questions of 'how do I do X' here, this is a question of 'is X the right way to go about it, or what is the proper way to do it, and can you give an example of how it ties together?'... so if this question is rather lengthy: my apologies.
My current goal is to have custom markup like the following which is supposed to 'track' which character says something:
<paul|"Hi there">
If I convert to HTML, I'd want something similar to:
<span class="speech paul">"Hi there"</span>
to pop out (and perhaps the <p> tags), whereas if it is just pure markdown / plain text, I'd want it to silently disappear:
"Hi there"
Looking at the JSON AST structures I've studied, it would make sense that I'd want a new structure type similar to the 'Emph' tag called 'Speech' which allows whole blobs of text to be put inside of it with a bit of extra information attached (the person speaking). So something like this:
{"t":"Speech","speaker":"paul","c":[ ... ] }
Problem #1: At the point a lua-filter sees the document, it is obviously already distilled to an AST. This means replacing the items in a manner similar to what most macro expander samples do cannot really work since it would require reading forward. With this method, I just replace bits and pieces in place (<NAME| becomes a StartSpeech and the first solitary > that follows becomes an EndSpeech, but that would make malformed input a bigger potential problem because of silent-ish failures. Additionally, these tags would be completely out of sorts with how an AST is supposed to look.
To complicate matters even further, some of my characters end up learning a secondary language throughout the story, for which I apply a different format that contains a simplified understanding of the spoken text with perspective-characters understanding of what was said. Example:
<paul|"Heb je goed geslapen?"|"Did you ?????">
I could probably add a third 'UnderstoodSpeech' group to my filter, but (problem #2) at this point, the relationship between the speaker, the original speech, and the understood translation is completely gone. As long as the final documents need these values in these respective orders and only in these orders, it is fine... but what if I want my HTML version to look like
"Did you?????"
with a tool-tip / hover-over effect containing the original speech? That would be near impossible to achieve because the AST does not contain that kind of relational detail.
Whatever kind of AST I create in the filter is what I need to understand in my custom writer. Ideally, I want to re-use as much stock functionality of pandoc as possible for the writer, but I don't even know if that is feasible at this point.
So now my question: could someone with great pandoc understanding please give me an example on how to keep relevant data-bits together and apply them in the correct manner? By this I mean show a basic example of what needs to be put in the lua-filter and lua-writer scripts in the following toolchain
[CUSTOMIZED MARKDOWN INPUT] -> lua-filter -> lua-writer -> [CUSTOMIZED HTML5 OUTPUT]
I'm re-building a Lua to ES3 transpiler (a tool for converting Lua to cross-browser JavaScript). Before I start to spend my ideas on this transpiler, I want to ask if it's possible to convert Lua labels to ECMAScript 3. For example:
goto label;
:: label ::
print "skipped";
My first idea was to separate each body of statements in parts, e.g, when there's a label, its next statements must be stored as a entire next part:
some body
label (& statements)
other label (& statements)
and so on. Every statement that has a body (or the program chunk) gets a list of parts like this. Each part of a label should have its name stored in somewhere (e.g, in its own part object, inside a property).
Each part would be a function or would store a function on itself to be executed sequentially in relation to the others.
A goto statement would lookup its specific label to run its statement and invoke a ES return statement to stop the current statements execution.
The limitations of separating the body statements in this way is to access the variables and functions defined in different parts... So, is there a idea or answer for this? Is it impossible to have stable labels if converting them to ECMAScript?
I can't quite follow your idea, but it seems someone already solved the problem: JavaScript allows labelled continues, which, combined with dummy while loops, permit emulating goto within a function. (And unless I forgot something, that should be all you need for Lua.)
Compare pages 72-74 of the ECMAScript spec ed. #3 of 2000-03-24 to see that it should work in ES3, or just look at e.g. this answer to a question about goto in JS. As usual on the 'net, the URLs referenced there are dead but you can get summerofgoto.com [archived] at the awesome Internet Archive. (Outgoing GitHub link is also dead, but the scripts are also archived: parseScripts.js, goto.min.js or goto.js.)
I hope that's enough to get things running, good luck!
I was wondering if there is a component like the 'edit', but just for numbers so I can use the .value function in my code.
My textbook says I must make a program, that when the user enters a number and clicks the execute button, the results of the functions must be determined.
The functions are: Trunc, round, frac, sqr and sqrt.
I have to enter the value into, what looks like an 'edit', but whenever I use the .value in my code, it gives me an error saying :Undeclared identifier: 'value'. Although it works when I use a 'SpinEdit'.
Forgive me for being really thick, I do have a severe chest and sinus infection with a fairly bad fever, so my mind is somewhere else at the moment.
Thanks!
Oh, and by the way, I have also used the 'MaskEdit' component but it still gives me the same error
For an edit control there is no property named Value, which is what the compiler is telling your. For an edit control the property you need is Text. That's a string containing the contents of the edit control. You'll need to use StrToFloat or TryStrToFloat to convert to a real type.
You can use a masked edit if you like, and validate the input on entry. The TMaskEdit control derives from TCustomEdit, and again the property used for accessing its content is Text and of type string.
Personally I don't like that because I don't think it gives the clearest feedback to users. It's also hard to write a mask for a general floating point value. Myself, I would validate at the point where the program needs to convert from string to real.
Well, since you asked if there is an edit like component for that, I use TMS AdvEdit. It does a very decent job handling integers and floats. If you can afford it, it's really useful.
It has .FloatValue and .IntValue properties for reading and writing the value, and an EditType that specifies what kind of input is accepted.
I'm developing an application that parses Cobol programs. In these programs some respect the traditional coding style (programm text from column 8 to 72), and some are newer and don't follow this style.
In my application I need to determine the coding style in order to know if I should parse content after column 72.
I've been able to determine if the program start at column 1 or 8, but prog that start at column 1 can also follow the rule of comments after column 72.
So I'm trying to find rules that will allow me to determine if texts after column 72 are comments or valid code.
I've find some but it's hard to tell if it will work everytime :
dot after column 72, determine the end of sentence but I fear that dot can be in comments too
find the close character of a statement after column 72 : " ' ) }
look for char at columns 71 - 72 - 73, if there is not space then find the whole word, and check if it's a key word or a var. Problem, it can be a var from a COPY or a replacement etc...
I'd like to know what do you think of these rules and if you have any ideas to help me determine the coding style of a Cobol program.
I don't need an API or something just solid rules that I will be able to rely on.
I think you need to know the COBOL compiler for each program. Its documentation should tell you what conventions/configurations/switches it uses to decide if the source code ends at column 72 or not.
So.... which compiler(s)?
And if you think the column 72 issue is a pain, wait till you get around to actually parsing the COBOL itself. If you are not well prepared to handle the lexical issues of the language, you are probably very badly prepared to handle the syntactic ones.
There is no absolutely reliable way to determine if a COBOL program
is in fixed or free format based only on the source code. Heck it is sometimes difficult to identify
the programming language based only on source code. Check out
this classic polyglot - it is valid under 8 different language compilers. That
said, you could try a few heuristics that might yield
the correct answer more often than not.
Compiler directives imbedded in source code
Watch for certain compiler directives that determine code format.
Unfortunately, every compiler vendor uses their own flavour of directive.
For example, Microfocus COBOL uses the
SOURCEFORMAT directive. This directive will appear near the top of the program so a short pre-scan
could be used to find it. On the other hand, OpenCobol uses >>SOURCE FORMAT IS FREE and
>>SOURCE FORMAT IS FIXED to toggle between free and fixed format, different parts of the same program
could be formatted differently!
The bottom line here is that you will have to support the conventions of multiple COBOL compilers.
Compiler switches
Source code format can be also be specified using a compiler switch. In this case, there are no concrete
clues to go on. However, you can be reasonably sure that the entire source program will be either
fixed or free. All you can do here is guess. Unless the programmer is out to "mess with
your head" (and some will), a program in free format will have the keywords IDENTIFICATION DIVISION or ID DIVISION, starting before column 8.
Every COBOL program will begin with these keywords so you can use them as the anchor point for determining code format in the
absence of imbedded compiler directives.
Warning - this is far from fool proof, but might be a good start.
There won't be an algorithm to do this with 100% certainty, because if comments can be anything, they can also be compilable COBOL code. So you could theoretically write a program that means one thing if the comments are ignored, and something else entirely if the comments are treated as part of the COBOL.
But that's extremely unlikely. What's most likely to happen is that if you try to compile the code under the wrong convention, it will simply fail. So the only accurate way to do this is to try compiling/parsing the program one way, and if you come to a line that can't make sense, switch to the other style. You could also support passing an argument to the compiler when the style is already known.
You can try using heuristics like what you've described, but that will never be totally accurate. The most they can give you is a probability that the code is one or the other style, which will increase as they examine more and more lines of code. They could be useful for helping you guess the style before you start compiling, or for figuring out when the problem is really just a typo in the code.
EDIT:
Regarding ideas for heuristics, it's hard to say. If there were a standard comment sigil like // or # in other languages, this would be a lot easier (actually, there is, but it sounds like your code doesn't follow this convention). The only thing I can think of would be to check whether every line (or maybe 99% of lines, and not counting empty lines or lines commented with *) has a period somewhere before position 72.
One thing you DON'T want to do is apply any heuristics to the part after position 72. That is, you don't want to be checking the comments to see if they're valid COBOL. You want to check what you know is COBOL first, and see if that works by itself. There are several reasons for this:
Comments written in English are likely to have periods and quotes in them, so your first and second bullet points are out.
Natural languages are WAY harder to parse than something like COBOL.
The comments could easily have COBOL in them (maybe someone commented out the previous version of the line).
An important rule for comments is that they should never affect what the program does. If changing the comments can change how the program is compiled, you violate that.
All that in mind, my opinion is that you shouldn't use heuristics at all. You should always try to compile the program under both conventions unless one is explicitly specified. There's a chance that code will compile successfully under both conventions, and then you'll have two different programs and no way to tell which one is correct.
If that happens, you need to compare the two results (perhaps with a hash or something) to see if they're the same program. If they're the same, great, but if not, you'll need to force the user to explicitly choose a convention.
Most COBOL compilers will allow you to generate and analyze the post text manipulation phase.
The text preprocessor output can be seen (using OpenCOBOL for the example)
cobc -E program.cob
The text manipulation processor deals with any COPY ... REPLACING compiler directives, as well as converting SOURCE FORMAT IS FIXED (with line continuations, string literal concatenations, comment line removal, among other things) to the actual free format that the compiler lexical analyzer needs. A lot of the OpenCOBOL toolkits (Cross referencer and Animator, to name two) use source code AFTER the preprocessor pass. I don't think you'll lose any street cred if your parser program relies on post processed source code files.