I have TF modules on bitbucket in the terraform-modules project.
In my main module (in the azure/main_module directory) on bitbucket I have:
azure
|
+- sub_module
main.tf
main_module
main.tf
In main_module/main.tf I have the following file structure:
azure
|
+-general
|
+-main.tf
module "my_sub_module" {
source = "../azure/sub_module/"
...
}
Now, in another project, in another module I have:
module "credit" {
source = "git::bitbucket.org:myorg/terraform-modules.git//main_module"
...
}
I get the following error when running tf plan in the other project:
Error: Unreadable module directory
Unable to evaluate directory symlink: lstat
.terraform/modules/main_module/azure/azure: no such file or directory
Everything works when the local references are replaced with bitbucket references.
So it seems as though the main_module is imported, but then the local references in that project is are interpreted as local references in the importing project (the one with the general module) and then it can't find additional modules because it is searching in the wrong project?
Any ideas how this can be fixed, while still using local references in the imported project?
Related
I've been working on a small project trying to teach myself how to use Bazel. The goal is to download a http_archive that contains a python interpreter. once the python interpreter has been added to the environment I want to run the command `python.exe --version and write the output of that into a file
The issue I have the most difficulty with at the moment are the following:
I am not confident that I am correctly injecting the hermetic python BUILD file into the hermetic python package (I keep getting the message "BUILD file not found in any of the following directories. Add a BUILD file to a directory to mark it as a package").
I'm pretty sure that when I pass in python_compiler = ["#hermetic_python"] in the BUILD file I'm just getting a string and not a reference to the files in the package
Here is an overview of my project and the code files. Any help would be appreciated! :D
Project structure:
|-- WORKSPACE
|-- BUILD
|-- custom_rules.bzl
|-- main.py
|-- custom-rules/
|-- BUILD.custom_python
|-- custom_python_rules.bzl
WORKSPACE
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "hermetic_python",
urls = ["https://www.python.org/ftp/python/3.9.10/python-3.9.10-embed-amd64.zip"],
sha256 = "67161cab713a52f6658b76274f8dbe0cd2f256aab1e84f94cd557d4a906fa5d2",
build_file = "#//:custom-rules/BUILD.custom_python"
)
BUILD File:
load("//:custom_rules.bzl","build_with_custom_python")
build_with_custom_python(
name = "write-to-file",
python_compiler = ["#hermetic_python"]
)
custom_rules.bzl
def _build_with_custom_python_impl(ctx):
out_file = ctx.actions.declare_file("file_with_python_version.txt")
ctx.actions.run(
outputs = [out_file]
executable = ctx.attr.python_compiler,
arguments = [--version],
)
return DefaultInfo(files=[out_file])
build_with_custom_python = rule(
implementation = _build_with_custom_python_impl,
attrs = {
"python_compiler": attr.label_list(allow_files=True)
}
)
BUILD.custom_python
load("//:custom_python_rules.bzl","run_me")
run_me(
name="my_py_run",
python_files = glob(["**"]),
visibility = ["//visibility:public"],
)
custom_python_rules.bzl
def _run_me_impl(ctx):
pass
run_me = rule (
implementation = _run_me_impl,
attrs = {
"python_files" : attr.label_list(allow_files=True),
}
)
I've spent some more time on this I've managed to mostly do what I intended to do. Here is an overview of what I've learned that fixes the problems in the original question
1 - Mark the custom build file / custom rule folder as a package
Just by adding an empty BUILD file into the custom-rules folder I had market it as a bazel package. That allowed me to reference the files
|-- WORKSPACE
|-- BUILD
|-- custom_rules.bzl
|-- main.py
|-- custom-rules/
|-- BUILD # NEW
|-- BUILD.custom_python
|-- custom_python_rules.bzl
2 - The workspace file references a BUILD file. That build file needs to reference a custom rules file
The workspace should just reference the build file directly // means workspace root and then its just the path to the BUILD file
http_archive(
# build_file = "#//:custom-rules/BUILD.custom_python" // WRONG
build_file = "//custom-rules/BUILD.custom_python" // RIGHT
)
3 - BUILD file can not reference the custom rules (bzl file)
This is the one that took the longest time for me to figure out! The BUILD file that is loaded into the package (hermetic_python from the workspace file) NEEDS TO reference the current workspace in order to
# BUILD.custom_python
load("#root_workspace//custom-rules:custom-python-rules.bzl","execute_python_file")
Notice that this starts with #root_workspace which tells the BUILD file to look in that workspace. the WORKSPACE file in the project now contains this line
# WORKSPACE
workspace(name = "root_workspace")
4 - Passing in a build step as the dependancy
build_with_custom_python(
name = "write-to-file",
# python_compiler = ["#hermetic_python"] WRONG
python_compiler = ["#hermetic_python:my_py_run"] RIGHT
)
The key here is that I need to reference a build action which then makes the files in that build action available through the dependency. In this case #hermetic_python is the package and :my_py_run is the build action.
I still need to figure out how to properly use the files in the dependency but that's outside of the scope of this question
In my bazel BUILD file, I have a line:
srcs = glob([<pattern1>, <pattern2>, ...])
I tried to have one of my patterns be "../dir/*.cc" but I get an error that I'm not allowed to use the .. sequence here.
Checking the documentation, I have found that it's not permitted, but I'm not sure what the expected substitute is.
Similarly, up-level references (..) and current-directory references (./) are forbidden.
How can I include these other source files in my srcs list given my current file structure? If I can't reference the up-level directory, is there a way to use the package name of the other directory instead?
Going "up" from your BUILD file would violate the package boundaries. If you really need that structure and cannot or don't want to change it, you have to make files from one package available to the other package by declaring the corresponding target(s) or at least export the files and making those visible. For instance assuming the following structure:
.
├── BUILD
├── WORKSPACE
├── hello.c
└── tgt
└── BUILD
It the // (top-level) package BUILD I could say:
filegroup(
name = "hello",
srcs = ["hello.c"],
visibility = ["//tgt:__pkg__"],
)
(Could also be: exports_files(["hello.c"], ["//tgt:__pkg__"]) instead in which case I would refer to the file by its name //:hello.c from tgt.)
And inside //tgt (tgt/BUILD) it can then read:
cc_binary(
name="tgt",
srcs=["//:hello"],
)
Which would give me:
$ bazel run //tgt
WARNING: /tmp/bzl1/tgt/BUILD:3:10: in srcs attribute of cc_binary rule //tgt:tgt: please do not import '//:hello.c' directly. You should either move the file to this package or depend on an appropriate rule there
INFO: Analyzed target //tgt:tgt (11 packages loaded, 68 targets configured).
INFO: Found 1 target...
Target //tgt:tgt up-to-date:
bazel-bin/tgt/tgt
INFO: Elapsed time: 0.247s, Critical Path: 0.09s
INFO: 2 processes: 2 linux-sandbox.
INFO: Build completed successfully, 6 total actions
INFO: Build completed successfully, 6 total actions
Hello World!
Note: bazel still flags this as something weird and noteworthy going on. I have to say I do not disagree with it. The tree structure does not seem to correspond to the content very well.
Perhaps in this example the tgt package boundary is artificial and not actually useful? Or hello.c is in the wrong place.
I'm studying Bazel building system at present. I always see the # symbol in Bazel script, but I cannot find any documentation about it. I searched it on the website of Bazel but the result seems useless.
# in Bazel.
For example:
filegroup(
name = "toolchain_fg",
srcs = [
":cc-compiler-amd64",
"#x86_64_unknown_linux_gnu_gcc_730//:compiler_components",
],
)
Could anybody explain the # symbol here for me?
This is to reference a remote repository.
From the doc, depending on other Bazel projects
local_repository(
name = "coworkers_project",
path = "/path/to/coworkers-project",
)
If your coworker has a target //foo:bar, your project can refer to it
as #coworkers_project//foo:bar.
See also the design doc of remote repository and bind example in workspace rules.
In Bazel, targets are referred by labels.
Bazel labels have the following format:
#repoName//packageName:target
For example, in the following packages found in myRepo:
myRepo
├── WORKSPACE
├── package1
│ └── BUILD
│ └── src
└── package2
├── BUILD
└── src
a target called myTarget in package1/BUILD can be labeled as as #myRepo//package1:myTarget globally.
If referenced from the same repo, for example from package2/BUILD, then the #myRepo prefix can be omitted and you can use //package1:myTarget.
If referenced from the same package, for example in another target from package1/BUILD, then the package name can be omitted and you can use :myTarget. The colon can also be omitted if it does not create confusion with a name. Such short form labels should not be confused with the names. Labels start with '//' or ':'. But names never do. For example, the name of the first package is package1 but its label is //package1.
Reference: https://docs.bazel.build/versions/master/build-ref.html
we are trying to create a Scala project which uses Spark also but we are facing issue Encountered error while reading extension file 'intellij_info_bundled.bzl': no such package '#intellij_aspect//': No WORKSPACE file found in C:/users//_bazel_user/i45wuf6d/external/intellij_aspect. Is it has something missing in Intellij?
Scala file
package src.main.scala
object HelloWorld extends App {
def main(args: Array[String]) {
println("Hello, world!")
}
}
Build file
package(default_visibility = ["//visibility:public"])
load("#io_bazel_rules_scala//scala:scala.bzl", "scala_library", "scala_test")
scala_library(
name = "hello-world",
srcs = glob(["src/main/scala/*.scala"]),
)
scala_test(
name = "Hello_test",
srcs = glob(["src/main/scala/*.scala"]),
size = "small", # Expect this test to run quickly
)
Work Space
workspace(name = "scala_example")
rules_scala_version="7522c866450cf7810eda443e91ff44d2a2286ba1" # update this as needed
http_archive(
name = "io_bazel_rules_scala",
url = "https://github.com/bazelbuild/rules_scala/archive/%s.zip"%rules_scala_version,
type = "zip",
strip_prefix= "rules_scala-%s" % rules_scala_version
)
load("#io_bazel_rules_scala//scala:scala.bzl", "scala_repositories")
scala_repositories()`enter code here`
# register default scala toolchain
load("#io_bazel_rules_scala//scala:toolchains.bzl", "scala_register_toolchains")
scala_register_toolchains()
Command and Error from console
Command: C:\ProgramData\chocolatey\bin\bazel.exe build --tool_tag=ijwb:IDEA:community --keep_going --curses=no --color=yes --experimental_ui=no --progress_in_terminal_title=no --aspects=#intellij_aspect//:intellij_info_bundled.bzl%intellij_info_aspect --override_repository=intellij_aspect=C:\Users\ADMIN.IdeaIC2017.3\config\plugins\ijwb\aspect --output_groups=intellij-compile-java,intellij-compile-py -- //...:all
INFO: Loading complete. Analyzing...
ERROR: Encountered error while reading extension file 'intellij_info_bundled.bzl': no such package '#intellij_aspect//': No WORKSPACE file found in C:/users/admin/appdata/local/temp/_bazel_sandhya/criyrv6d/external/intellij_aspect.
INFO: Found 3 targets...
WARNING: failed to create one or more convenience symlinks for prefix 'bazel-':
cannot create symbolic link bazel-out -> C:/users/admin/appdata/local/temp/_bazel_sandhya/criyrv6d/execroot/scala_example/bazel-out: Cannot create junction (name=C:\users\admin\scalaprojects\example1\bazel-out, target=C:\users\admin\appdata\local\temp_bazel_sandhya\criyrv6d\execroot\scala_example\bazel-out): ERROR: src/main/native/windows/file-jni.cc(86): nativeCreateJunction(C:\users\admin\scalaprojects\example1\bazel-out, C:\users\admin\appdata\local\temp_bazel_sandhya\criyrv6d\execroot\scala_example\bazel-out): ERROR: src/main/native/windows/file.cc(128): CreateJunction(\?\C:\users\admin\scalaprojects\example1\bazel-out): Cannot create a file when that file already exists.
cannot create symbolic link bazel-out -> C:/users/admin/appdata/local/temp/_bazel_sandhya/criyrv6d/execroot/scala_example/bazel-out: Cannot create junction (name=C:\users\admin\scalaprojects\example1\bazel-out, target=C:\users\admin\appdata\local\temp_bazel_sandhya\criyrv6d\execroot\scala_example\bazel-out): ERROR: src/main/native/windows/file-jni.cc(86): nativeCreateJunction(C:\users\admin\scalaprojects\example1\bazel-out, C:\users\admin\appdata\local\temp_bazel_sandhya\criyrv6d\execroot\scala_example\bazel-out): ERROR: src/main/native/windows/file.cc(128): CreateJunction(\?\C:\users\admin\scalaprojects\example1\bazel-out): Cannot create a file when that file already exists.
.
INFO: Building...
ERROR: command succeeded, but not all targets were analyzed.
INFO: Elapsed time: 18.108s, Critical Path: 0.05s
Make failed
This is a sample Helloworld program only
In general, like #Ittai, I would suggest you open an issue in the intellij plugin github repo.
Unfortunately, your version of the plugin is no longer supported. I, too, previously ran into an issue with an older version of the plugin and was recommended to upgrade to the latest version. Which resolved the specific issue I was facing.
When reporting the issue make sure to include the following bits of information:
intellij build number
plugin version number
rules_scala version
operating system (it seems your using Windows, while most users use unix based systems)
bazel release number
how you have opened the intellij project (BUILD file, WORKSPACE, .blazeproject)
Additionally, to verify this is in fact an issue with the plugin, I would also suggest you try to reproduce this issue on a Unix based system. It seems you are using Intellij
compile on Windows. This may be Windows specific issue with aspects not being recognized.
When attempting to reproduce, make sure to clone your repository in a separate directory, close the intellij project, and reopen the project
I would like to:
use the Frege programming language to write a simple "Hello World" piece of code,
then using the Frege compiler generating the equivalent Java source code,
then building an executable Jar file to run from the command line,
all the previous steps should be "controlled" by Gradle.
I am able to generate the source code (items 1. and 2. from the previous list), but I am not able to specify a "package" structure of the Java source code in output, i.e. I can not see the package Java statement as the first line of code in the generate Java source code. I can specify to the Frege compiler where to put the generated code though (via the -d argument).
I think this is the reason why when building an executable Jar, then launching it, I am seeing similar errors (according to different attempts on Gradle tasks) e.g.: no main manifest attribute.
The Frege source code is saved in a file named HelloFrege.fr, the generated Java source code is in a file named HelloFrege.java (I verified the file contains the expected main method).
Here there's a version of the Gradle "Jar task":
//create a single Jar with all dependencies
task fatJar(type: Jar) {
manifest {
attributes 'Implementation-Title': 'Hello Frege Jar Example',
'Implementation-Version': version,
'Main-Class': 'HelloFrege'
}
baseName = project.name + '-all'
from { configurations.compile.collect { it.isDirectory() ? it : zipTree(it) } }
with jar
}
Here there's another version of the Gradle "Jar" task:
jar {
manifest {
attributes 'Main-Class': 'HelloFrege'
}
}
How can I solve this problem? I would like to avoid to manually add the package reference to the automatically generated Java source code file.
If your module name in Frege is unqualified such as HelloWorld, you will not see the package statement generated in Java. The module name will become the class name and the package will be empty or default package.
If your module name is qualified such as foo.bar.HelloWorld, then foo.bar will be the package name and HelloWorld will be the class name in the generated Java source.
The rule is that the last part of the module name becomes the class name and the qualifiers form the package name in the generated Java source.
I am not sure what gradle can do for you in this regard, but without gradle, the following should at least be possible:
... build your jar, as before ...
jar -uvfe project-all.jar HelloFrege
java -jar project-all.jar # run it
This, of course, is just another way to create a manifest. If this works, then it would be time to investigate why gradle refuses to do it.
Postscriptum: After thinking another minute about what the problem might be, it occurs to me that you may think that the source file name/path has anything to do with the java package name. This is not so in Frege, though it is good practice to have the file path match the package name, and the file base name match the class name (just like in Java). In addition, to remove some confusion, use the module keyword in frege. As explained by Marimuthu, the Java package and class name is derived from the frege module name.
Example:
$ cat Foo.fr
module my.org.Baz where
...
$ java -jar fregec.jar -d bin Foo.fr
This generates the Baz class in package my.org, and creates the class file in bin/my/org/Baz.class
I am posting here my findings so far. The combination of Gradle commands that works for me is the following one (calling it from the command line typing gradle clean generateJavaSrcFromFregeSrc fatJar):
task generateJavaSrcFromFregeSrc {
ant.java(jar:"lib/frege3.21.586-g026e8d7.jar",fork:true) {
arg(value: "-j") // do not run the java compiler
arg(value: "-d")
arg(value: "src/main/java") // the place where to put the generated source code (paired to the -d argument)
arg(value: "src/main/frege/HelloFrege.fr")
}
}
jar {
manifest {
attributes 'Main-Class': 'org.wathever.HelloFrege'
}
}
task fatJar(type: Jar) {
from files(sourceSets.main.output.classesDir)
from files(sourceSets.main.output.resourcesDir)
//from {configurations.compile.collect {zipTree(it)}} // this does not include the autogenerated source code
baseName = project.name + '-fatJar'
from { configurations.compile.collect { it.isDirectory() ? it : zipTree(it) } }
with jar
}
The manifest details need to be specified in the jar block of code, if I specify them in the task fatJar then when running the jar I get no main manifest attribute, in [...].
If I use just the block of code jar with the property from("$projectDir") { include 'lib/**'} to include the Frege jar, then I get errors like java.lang.ClassNotFoundException (I think because the Jar is included as it is and not as a set of .class files).
The folder src/main/java/org/wathever needs to be there before running Gradle (additional info: the Maven convention prefix src/main/java with as a suffix the "Java package" as specified inside the HelloFrege.fr source code: module org.wathever.HelloFrege where)
Some useful details I found:
How to build a fat Jar
Another how to build a fat Jar
An "Hello Frege" example without the Gradle management
The Gradle documentation on how to use the Jar task