I want to dynamically load code by traversing a directory structure and dynamically load whatever modules I find there.
The purpose for doing so is to run a series of validations. If a top-level validation fails, any child validations will not be run.
My thinking was that a controller object could scan the directories, build up a hierarchy of modules and then make the decisions on whether or not to traverse a particular part of the tree based on the success/failure of higher-level validations.
For example, I might have a series of validations I want to run against a regex, however, none of the validations should be run if the regex doesn't exist or is empty. In this case, the top level directory would contain just the exists validation, and a child directory would contain all the other validations to be run if the regex exists.
Being able to define these validations in separate files and create the needed hierarchy would be extremely useful for ease of adding additional validations later, rather than having to crack open an existing class and add methods.
Is there a way an application can dynamically scan a directory, save the filenames in a collection and then use the elements of that collection in a require? I don't think so. What about a load?
Is there any way to achieve such a design? Or am I thinking about it all wrong and should think of some other methodology instead?
Your request is very doable, but no language will do it for you automatically. You have to write the code to dive into the directories, determine the existence of the tests and then decide whether you should drill down further.
Ruby will help you though. There is the Find module, which is included in the standard library. This is from its docs:
The Find module supports the top-down traversal of a set of file paths.
For example, to total the size of all files under your home directory,
ignoring anything in a "dot" directory (e.g. $HOME/.ssh):
require 'find'
total_size = 0
Find.find(ENV["HOME"]) do |path|
if FileTest.directory?(path)
if File.basename(path)[0] == ?.
Find.prune # Don't look any further into this directory.
else
next
end
else
total_size += FileTest.size(path)
end
end
From that code you would look for the signatures of the files and embedded folders, to decide if you should drill down further. For each file found that is one you want, use require to load it.
You can find other examples out on the "internets" showing how people use Find. Also the Dir module has similar functionality using glob, only you have to tell it where to descend, and then can iterate over the returned results.
Related
I have rule A implemented with a macro that uses declare_directory to produce a set of files:
output = ctx.actions.declare_directory("selected")
Names of those files are not known in advance. The implementation returns the directory created by declare_directory with the following:
return DefaultInfo(
files = depset([output]),
)
Rule A is included in "srcs" attribute of rule B. Rule B is also implemented with a macro. Unfortunately the list of files passed to B implementation through "srcs" attribute only contains the "selected" directory created by rule A instead of files residing in that directory.
I know that Args class supports expansion of directories so I could pass names of all files in "selected" directory to a single action. What I need, however, is a separate action for every individual file for parallelism and caching. What is the best way to achieve that?
This is one of the intended use cases of directory outputs (called TreeArtifacts in the implementation), and it's implemented using ActionTemplate:
https://github.com/bazelbuild/bazel/blob/c2100ad420618bb53754508da806b5624209d9be/src/main/java/com/google/devtools/build/lib/actions/ActionTemplate.java#L24-L57
However, this is not exposed to Starlark, and has only a couple usages currently, in the Android rules AndroidBinary.java and C++ rules CcCompilationHelper.java. The Android rules and C++ rules are going to be migrated over to Starlark, so this functionality might eventually be made available in Starlark, but I'm not sure of any concrete timelines. It would probably be good to file a feature request on Github.
Upon running rails g graphql:install, a set of useful base type files are created in /app/graphql/types for us to extend when defining our own types. When running any of the generators, it creates these files in the same folder also. I set about creating sub directories thinking I could add some sense to this giant catch-all directory, but couldn't get things to load properly.
Since there is a base file for each type (base_enum.rb, base_object.rb, etc.), I created a folder for extensions of each of these types (enum_types, object_types, etc.). This broke auto loading though and I had to explicitly import these files to be able to use these custom types. So, at the top of query_type.rb, mutation_type.rb and app/graphql/mutations/base_mutation.rb I added the following:
['enum_typs', 'input_object_types', 'interface_types', 'object_types', 'scalar_types', 'union_types'].each do |dir|
Dir[File.dirname(__FILE__) + "/#{dir}/*.rb"].each {|file| require file }
end
This allowed things to run, but any change would break auto loading so I would have to restart the server on each change. I started reading through this article about auto loading on the rails site, but it was quite honestly a little over my head. Though it led me to believe I had to either find the correct names for my folders or namespace the objects defined in my type definition files properly to be able to do this.
Is there a sane way to organize these files in sub-directories which doesn't break auto loading? Do most projects just have a flat folder structure for these files?
Thank you!
I have been given the task of creating a DXL script. First problem is that I have never used DXL before, even though I have many years experience with DOORS itself. I have been surfing the Net to seek guidance on my particular problem. I also have a few specimen DXL scripts for reference.
My new client requires that for each View of a given Module, of which there are many Views, new "reduced" Modules are to be produced reflecting each View.
By "reduced", I mean that these new Modules are to contain nothing that isn't actually needed for that View., i.e. Columns, Attributes etc. These new Modules will only have the single View.
So, the way forward as I see it, is to take copies of the single master Module, one for each View, rename those copies to reflect a given Master Module/Required View, select that required View in the given copy Module and then delete everything that is not needed by that View, i.e. available Columns, Attributes etc.
This would be simple if I had the required DXL knowledge, which I am endeavouring to pick up as fast as I can.
If at all possible, this script has to be generic and be able to work upon any of the master Module copies to produce the associated "reduced" Module reflecting a particular View.
The client aims to use the script periodically for View archiving (I know, that's the way they want it).
Clarification
Some clarification of what I believe is required, given the following text from my original question:
If at all possible, this script has to be generic and be able to work upon any of the master Module copies to produce the associated "reduced" Module reflecting a particular View.
So, say there are ten views of the master Module, outside of the DXL script, I would copy the master Module ten times, renaming each copy to reflect each of the ten views. Unless you know different, each of those ten copies will reflect the same “Absolute Number”s as are in the master Module, so no problem there?
So, starting with the first of the copied Modules, each named to reflect the View it will eventually represent, its View would be set from the ten Views available to it, that which matches its title.
The single generic DXL script would then be run against that first copy Module, the aim being to delete everything not actually needed for that view, i.e. Attributes, Columns etc. Would some kind of purging command be required in the script for any aforementioned deleted items?
The single generic DXL script would then delete ALL views from that copy Module. The log that is produced when running the script also needs capturing, but I’m not sure whether this should be done from within the script, if possible or as a separate manual task outside of the script.
The aforementioned (indented) process would then be repeated, using the same generic script, against the remaining nine copied Modules. The intension is to leave us with ten copy Modules, each one reflecting one of the ten possible Views, with each one containing only the Attributes, Columns etc. required for that View.
Creating a mirror of a module with this approach is not so easy IMO. Think e.g. about "Absolute Number". If the original module contains the numbers 15 (level 1), 2000 (level 2), 1 (level 1), you will have to create 2000 objects, purge 1997 of them and move them to the correct place.
There is a "duplicate" tool at https://www.ibm.com/developerworks/community/forums/html/topic?id=43862118-113d-4eac-b3f1-21d3b73959d1 which tries to do this, but as stated there, this script is said not to work correctly in all situations.
So, I would rather use the approach "string clipCopy (Item i); string clipPaste(Folder folderRef)". Should be faster and less error prone. But: all Out-Links will also be copied with this method, you will probably have to delete these after the copy or else the link target module(s) will have lots of In-Links.
The problem is still not so easy to solve, as every view might have DXL columns that rely on some or other attribute, and it might contain DXL attributes which again might rely on sth else. I doubt that there is a way to analyze DXL code "on the fly" and find out which columns may be deleted.
Perhaps a totally different approach would be feasible: open each view and create an export to Excel, this way you will get rid of any dynamic dependencies. Then re-import the excel sheet to a new DOORS module. You will still have the "Absolute Number" problem, but perhaps you can make a deal that you will have a pseudo attribute "Original Absolute Number" and disregard the "new" "Absolute Number"'
Quite a big task for a DXL beginner....
Update: On second thought, perhaps you might want to combine these approaches
agree with your employer that you will use an alternative attribute for Absolute Number
use a loop like Russel suggested, when creating objects remember that objects might have to be created "below" or "after" its predecessor or sibling
for DXL attributes do not copy the DXL code but the actual current value of the object
for DXL columns create pseudo attributes _ and create a new view that uses these pseudo attributes instead of the original value
Copying the entire module, then deleting everything not in that view, seems worse than just copying the things you need from each particular view.
I would take the following as the outline of your program:
for view in main module do {
for column in view do {
Find attribute for each column and store (possibly in a skip list?)
Store name of column
}
create new module
create needed types / attributes in new module
create new view in new module
for object in main module {
create object in new module
for attribute in main module {
check if attribute is in new module {
copy info from old object to new
}
}
}
}
Each of these for X in y loops should be in the DXL reference manual in some for or another.
If you need more help, let me know!
I'm currently writing some functions that are related to lists that I could possibly be reused.
My question is:
Are there any conventions or best practices for organizing such functions?
To frame this question, I would ideally like to "extend" the existing lists module such that I'm calling my new function the following way: lists:my_funcion(). At the moment I have lists_extensions:my_function(). Is there anyway to do this?
I read about erlang packages and that they are essentially namespaces in Erlang. Is it possible to define a new namespace for Lists with new Lists functions?
Note that I'm not looking to fork and change the standard lists module, but to find a way to define new functions in a new module also called Lists, but avoid the consequent naming collisions by using some kind namespacing scheme.
Any advice or references would be appreciated.
Cheers.
To frame this question, I would ideally like to "extend" the existing lists module such that I'm calling my new function the following way: lists:my_funcion(). At the moment I have lists_extensions:my_function(). Is there anyway to do this?
No, so far as I know.
I read about erlang packages and that they are essentially namespaces in Erlang. Is it possible to define a new namespace for Lists with new Lists functions?
They are experimental and not generally used. You could have a module called lists in a different namespace, but you would have trouble calling functions from the standard module in this namespace.
I give you reasons why not to use lists:your_function() and instead use lists_extension:your_function():
Generally, the Erlang/OTP Design Guidelines state that each "Application" -- libraries are also an application -- contains modules. Now you can ask the system what application did introduce a specific module? This system would break when modules are fragmented.
However, I do understand why you would want a lists:your_function/N:
It's easier to use for the author of your_function, because he needs the your_function(...) a lot when working with []. When another Erlang programmer -- who knows the stdlb -- reads this code, he will not know what it does. This is confusing.
It looks more concise than lists_extension:your_function/N. That's a matter of taste.
I think this method would work on any distro:
You can make an application that automatically rewrites the core erlang modules of whichever distribution is running. Append your custom functions to the core modules and recompile them before compiling and running your own application that calls the custom functions. This doesn't require a custom distribution. Just some careful planning and use of the file tools and BIFs for compiling and loading.
* You want to make sure you don't append your functions every time. Once you rewrite the file, it will be permanent unless the user replaces the file later. Could use a check with module_info to confirm of your custom functions exist to decide if you need to run the extension writer.
Pseudo Example:
lists_funs() -> ["myFun() -> <<"things to do">>."].
extend_lists() ->
{ok, Io} = file:open(?LISTS_MODULE_PATH, [append]),
lists:foreach(fun(Fun) -> io:format(Io,"~s~n",[Fun]) end, lists_funs()),
file:close(Io),
c(?LISTS_MODULE_PATH).
* You may want to keep copies of the original modules to restore if the compiler fails that way you don't have to do anything heavy if you make a mistake in your list of functions and also use as source anytime you want to rewrite the module to extend it with more functions.
* You could use a list_extension module to keep all of the logic for your functions and just pass the functions to list in this function using funName(Args) -> lists_extension:funName(Args).
* You could also make an override system that searches for existing functions and rewrites them in a similar way but it is more complicated.
I'm sure there are plenty of ways to improve and optimize this method. I use something similar to update some of my own modules at runtime, so I don't see any reason it wouldn't work on core modules also.
i guess what you want to do is to have some of your functions accessible from the lists module. It is good that you would want to convert commonly used code into a library.
one way to do this is to test your functions well, and if their are fine, you copy the functions, paste them in the lists.erl module (WARNING: Ensure you do not overwrite existing functions, just paste at the end of the file). this file can be found in the path $ERLANG_INSTALLATION_FOLDER/lib/stdlib-{$VERSION}/src/lists.erl. Make sure that you add your functions among those exported in the lists module (in the -export([your_function/1,.....])), to make them accessible from other modules. Save the file.
Once you have done this, we need to recompile the lists module. You could use an EmakeFile. The contents of this file would be as follows:
{"src/*", [verbose,report,strict_record_tests,warn_obsolete_guard,{outdir, "ebin"}]}.
Copy that text into a file called EmakeFile. Put this file in the path: $ERLANG_INSTALLATION_FOLDER/lib/stdlib-{$VERSION}/EmakeFile.
Once this is done, go and open an erlang shell and let its pwd(), the current working directory be the path in which the EmakeFile is, i.e. $ERLANG_INSTALLATION_FOLDER/lib/stdlib-{$VERSION}/.
Call the function: make:all() in the shell and you will see that the module lists is recompiled. Close the shell.
Once you open a new erlang shell, and assuming you exported you functions in the lists module, they will be running the way you want, right in the lists module.
Erlang being open source allows us to add functionality, recompile and reload the libraries. This should do what you want, success.
I wanted to make a generator that created files (and directories, etc...) based on already existing files in the app (for instance, the views or controllers). So if we had views set up like this
-app
-views
- layouts
- application.html.erb
- users
- index.html.erb
- show.html.erb
- etc ...
and I wanted to create files based on them I can do (with just ruby)
directories = Dir.entries("#{Rails.root}/app/views")
directories.each do |directory|
unless directory == "." or directory == ".."
files = Dir.entries("#{Rails.root}/app/views/#{directory}")
files.each do |file|
unless file == "." or file == ".."
text = File.read("#{Rails.root}/app/views/#{directory}/#{file}")
something #=> whatever else needs to go here to edit the file
something else #=> output_file.puts whatever
end
end
end
end
so this is basically what I would like to do with a generator so I can roll my code into a plugin and use it for other apps.
First question, how can I generate arbitrary files (with filenames based on existing filenames using the generator. Is it appropriate to cycle through the directories like I did above, grab the directory/file and generate files? Is there a way to do what I did using a simpler method (mine seems easily breakable).
Also, should I put all that read/format/write code inside the generator itself and just pass a string into the "initialize content" section of create_file or should I put it somewhere else. Or should I use the generator to create the bare files and populate it with an init script?
Is there a more rails type of way of populating generated files, or should I just shove all my formatting code inside the generator. If so, what is the appropriate way to approach this.
I am not sure if you want to know how generators are built in rails3 or not. The code you are showing is not very generator-like. In generators you can use all commands from Thor, which offers you a very powerful toolset of manipulating files, and injecting code (strings) into classes or files.
So I would most definitely fill your files inside a generator, because then it happens on user request, and the user can choose whether or not certain files need or can be overwritten or not.
Inside your gem, you will have a lib/generators folder, containing a templates folder, containing all files you might want to place inside the rails application.
From the Thor documentation, here is a nice example to construct files in a generator.
Hope this helps.
there's a simple API to use generators in Rails. here you can find a good guide:
http://guides.rubyonrails.org/generators.html
if you want to check some code:
https://github.com/coderloop/tamed_beast (I'm the author of its generators)
https://github.com/pilu/web-app-theme (another clean example)