I create my own repository to fetch some git source.
# packages.nix
with (import <nixpkgs> {});
rec {
rustcSource = fetchgit {
url = https://github.com/rust-lang/rust;
rev = "3191fbae9da539442351f883bdabcad0d72efcb6";
sha256 = "0w1l14kz9kxyj5dw3w9xxk1fzww5xqs3sf8saay0mh7nkmvrdb59";
};
}
Then I build rustcSource,
sudo nix-env -f package.nix -A rustcSource
It reveals a store path of /nix/store/096fpy9qjbz5r14aadjnq9d2md9ql9cg-rust-3191fba. The problem is, I forgot to download it's submodules, so I change my expression to include submodules,
with (import <nixpkgs> {});
rec {
rustcSource = fetchgit {
url = https://github.com/rust-lang/rust;
rev = "3191fbae9da539442351f883bdabcad0d72efcb6";
sha256 = "0w1l14kz9kxyj5dw3w9xxk1fzww5xqs3sf8saay0mh7nkmvrdb59";
leaveDotGit = true;
fetchSubmodules = true;
};
}
But, I discovered that nix-build doesn't recalculate the hash and see that the path has been built. So, it ends up not downloading the submodules.
Q: Is it nix bug? How can I "rebuild" or redownloading the repository?
P.S. Actually I create a derivation with fetchgit in it, but it fails because the submodules doesn't being downloaded. So, the above case simplify my problem.
Not a bug, this is by design.
Derivations that specify a hash are called fixed-output derivations and they only check if hash matches any existing derivations in store and use that. So your fetchSubmodules change is ignored.
For more discussion see https://github.com/NixOS/nix/issues/969
To fix the issue, you need to change the hash to some value that isn't already a valid hash of any path in your nix store.
For fixed-output derivations (those are the ones that have an explicit hash specified and only those get network access), if the hash already matches a path in the nix store, then nix will skip the download and just use the existing path. So slightly changing the hash (so that it no longer matches) should be enough to force a rebuild.
Fetching submodules will result in a package with a different hash. The easiest way to fix this is to change the hash to an invalid value and rebuild the package. The error message will include the correct hash. Use that and rebuild.
Related
From the manual:
Components are installed from a set of Nix expressions that tell Nix
how to build those packages, including, if necessary, their
dependencies. There is a collection of Nix expressions called the
Nixpkgs package collection that contains packages ranging from basic
development stuff such as GCC and Glibc, to end-user applications like
Mozilla Firefox.
Lets assume I want to search for the nix expression of the package Go for example. Where should I look in the repository to find the right file?
My tool of choice is nix repl. Often its tab completion is sufficient to find attribute names. Sometimes you might want to use https://search.nixos.org/packages.
For a convenient workflow, you could set EDITOR and use the :edit command in nix repl. Usually I open nixpkgs in VSCode and then run nix repl . in a VSCode terminal so I can Ctrl+click file locations as well.
I never use the directory structure, because search is so much more convenient.
[~/nixpkgs]$ export EDITOR=... # nano doesn't seem to work for this
[~/nixpkgs]$ nix repl .
nix-repl> :edit go
or
nix-repl> go.meta.position
"~/nixpkgs/pkgs/development/compilers/go/1.17.nix:278"
This generally gives you the location of a mkDerivation call, or a call to a similar function.
To get the location where an attribute is defined, use
nix-repl> builtins.unsafeGetAttrPos "go" pkgs
{ column = 3; file = "~/nixpkgs/pkgs/top-level/all-packages.nix"; line = 12753; }
And then there's the recursive directory search option (like grep -R, IDE-integrated search, etc). This generally works really well as package names tend to be specific. Too bad go isn't. We generally don't do crazy code formatting in Nix, a leading space and an equals sign do a pretty good job at finding definitions, even for go, if you ignore the ones in lib/.
[~/nixpkgs]$ git grep -n ' go ='
lib/attrsets.nix:125: go = prefixLength: hasValue: value: updates:
lib/debug.nix:234: go = x: generators.toPretty
lib/deprecated.nix:92: let go = xs: acc:
lib/filesystem.nix:29: let go = path:
lib/generators.nix:237: go = indent: v: with builtins;
lib/trivial.nix:500: go = i:
nixos/modules/security/apparmor/includes.nix:9: let go = { path ? null, mode ? "r", trail ? "" }:
pkgs/stdenv/booter.nix:63: go = pred: n:
pkgs/top-level/all-packages.nix:12753: go = go_1_17;
pkgs/top-level/all-packages.nix:21043: go = buildPackages.go_1_16;
pkgs/top-level/all-packages.nix:21046: go = buildPackages.go_1_17;
pkgs/top-level/all-packages.nix:21049: go = buildPackages.go_1_18;
pkgs/top-level/all-packages.nix:21055: go = buildPackages.go_1_16;
pkgs/top-level/all-packages.nix:21058: go = buildPackages.go_1_17;
pkgs/top-level/all-packages.nix:21061: go = buildPackages.go_1_18;
pkgs/top-level/all-packages.nix:26493: go = go_1_16;
The Bazel Starlark API does strange things with files in external repositories. I have the following Starlark snippet:
print(ctx.genfiles_dir)
print(ctx.genfiles_dir.path)
print(output_filename)
ret = ctx.new_file(ctx.genfiles_dir, output_filename)
print(ret.path)
It is creating the following output:
DEBUG: build_defs.bzl:292:5: <derived root>
DEBUG: build_defs.bzl:293:5: bazel-out/k8-fastbuild/genfiles
DEBUG: build_defs.bzl:294:5: google/protobuf/descriptor.upb.c
DEBUG: build_defs.bzl:296:5: bazel-out/k8-fastbuild/genfiles/external/com_google_protobuf/google/protobuf/descriptor.upb.c
That extra external/com_google_protobuf comes seemingly out of nowhere, and it makes my rule fail:
I tell protoc to generate into ctx.genfiles_dir.path (which is bazel-out/k8-fastbuild/genfiles).
So protoc generates bazel-out/k8-fastbuild/genfiles/google/protobuf/descriptor.upb.c
Bazel fails because I didn't generate bazel-out/k8-fastbuild/genfiles/external/com_google_protobuf/google/protobuf/descriptor.upb.c
Likewise, when I try to call file.short_path on a source file from an external repository, I get a result like ../com_google_protobuf/google/protobuf/descriptor.proto. This seems quite unhelpful, so I just wrote some manual code to strip off the leading ../com_google_protobuf/.
Am I missing something? How can I write this rule in a way that doesn't feel like I'm fighting Bazel the whole time?
Am I missing something?
The basic problem, as you already realized, is that you have two path "namespaces" the one that protoc sees (i.e. import paths) and the one that bazel sees (i.e. the path you pass to declare_file().
2 things to note:
1) All paths declared with declare_file() get the path <bin dir>/<package path incl. workspace>/<path you passed to declare_file()>
2) All actions are executed from <bin dir> (unless output_to_genfils=True in which case this switches to <gen dir> as in your example.
Trying to solve the exact same problem you encountered, I resorted to stripping the known path from the output_file's path to determine which directory to pass as p:
# This code is run from the context of the external protobuf dependency
proto_path = "google/a/b.proto"
output_file = ctx.actions.declare_file(proto_path)
# output_file.path would be `<gen_dir>/external/protobuf/google/a/b.proto`
# Strip the known proto_path from output_file.path
protoc_prefix = output_file.path[:-len(proto_path)]
print(protoc_prefix) # Prints: <gen_dir>/external/protobuf
command = "{protoc} {proto_paths} {cpp_out} {plugin} {plugin_options} {proto_file}".format(
...
cpp_out = "--cpp_out=" + protoc_prefix,
...
)
Alternatives
You may also be able to construct the same path with ctx.bin_dir, ctx.label.workspace_name, ctx.label.package, and ctx.label.name.
Misc.
proto_library recently gained an attribute strip_import_prefix. When used, the above is not correct, as all dependent files are symlinked into a new directory from which they have the relative paths declared with strip_import_prefix.
The path format is:
<bin dir>/<repo>/<package>/_virtual_base/<label name>/<path `import`ed in .proto files>
i.e.
<bin dir>/external/protobuf/_virtual_base/b_proto/google/a/b.proto
Assuming you are building an external repo called protobuf, which contains a BUILD file at its root with a target named b_proto, which in turn, relies on a proto_library wrapping google/a/b.proto AND uses the strip_import_prefix attribute.
Is there an easy way to get hold of a path object so I can check if a given label path exists. Say for example if path.exists("#external_project_name//:filethatmightexist.txt"):. I can see that the repository context has this. But I need to have a wrapping repository rule. Is it possible to do this in a macro or Skylark native call instead?
Even with a repository_rule, I had a lot of trouble with this due to what you already pointed out:
if you create a Label with a path that doesn't exist, it will cause the build to fail
But if you're willing to do a repository rule, here's a possible solution...
In this example, my rule allows specification of a default configuration if a config file is not present. The configuration can be checked into .gitignore and overridden for individual developers, but work out of the box for most cases.
I think I understand why the ctx.actions have sibling arguments now, same idea here. The trick is config_file_location is a true label, and then config_file is a string attribute. I chose BUILD arbitrarily, but since all workspaces have a top level BUILD that's public seemed legit-ish.
WORKSPACE Definition
...
workspace(name="E02_mysql_database")
json_datasource_configuration(name="E02_datasources",
config_file_location="#E02_mysql_database//:BUILD",
config_file="database.json")
The definition for json_datasource_configuration looks like this:
json_datasource_configuration = repository_rule(
attrs = {
"config_file_location": attr.label(
doc="""
Path relative to the repository root for a datasource config file.
"""),
"config_file": attr.string(
doc="""
Config file, maybe absent
"""),
"default_config": attr.string(
# better way to do this?
default="None",
doc = """
If no config is at the path, then this will be the default config.
Should look something like:
{
"datasource_name": {
"host": "<host>"
"port": <port>
"password": "<password>"
"username": "<username>"
"jdbc_connection_string": "<optional>"
}
}
There can be more than datasource configured... maybe, eventually.
""",
),
},
local = True,
implementation = _json_config_impl,
)
Then in the rule I can test for the file existence, and if not present, do other logic.
def _json_config_impl(ctx):
"""
Allows you to specify a file on disk to use for data connection.
If you pass a default
"""
config_path = ctx.path(ctx.attr.config_file_location).dirname.get_child(ctx.attr.config_file)
config = ""
if config_path.exists:
config = ctx.read(config_path)
elif ctx.attr.default_config == "None":
fail("Could not find config at %s, you must supply a default_config if this is intentional" % ctx.attr.config_file)
else:
config = ctx.attr.default_config
...
probably too late to help, but your question is the only thing I found referencing this goal. If someone knows a better way I am looking for other options. It's complicated to explain to other developers why the rule has to work the way it does.
Also note, if you change the config file, you have to clean to get the workspace to re-read the config. I haven't been able to figure out any way to fix that. glob() does not work in the workspace.
I want to edit project.pbxproj straight up using command line (for CI server script)
what tools can allow me to do this?
I used to use PlistBuddy to edit the output Info.plist; however, what i really want to do is to edit this user defined field, which is used in multiple places, and i really don't want to have to hunt that down in every plist location
project.pbxproj is an old-style ASCII property list file, too. So you can use /usr/libexec/PlistBuddy to edit it.
Print some User-Defined key's value like this,
# Get the key A83311AA20DA4A80004B8C0E in your project.pbxproj
# LZD_NOTIFICATION_SERVICE_BUNDLE_ID is defined by me,
# Replace key paths with your own.
/usr/libexec/PlistBuddy -c 'print :objects:A83311AA20DA4A80004B8C0E:buildSettings:LZD_NOTIFICATION_SERVICE_BUNDLE_ID' LAAppAdapter.xcodeproj/project.pbxproj
Set its value like this,
/usr/libexec/PlistBuddy -c 'set :objects:A83311AA20DA4A80004B8C0E:buildSettings:LZD_NOTIFICATION_SERVICE_BUNDLE_ID com.dawnsong.notification-service' LAAppAdapter.xcodeproj/project.pbxproj
UPDATE
PlistBuddy will automatically convert project.pbxproj into a xml-format plist file since macOS Catalina (or some earlier version). It's better to move the setting item into xcconfig file instead since xcconfig is much smaller and simpler than project.pbxproj and not easy to make mistakes when editing with perl script.
I know this has been answered for a while, but since the original question is about tools supporting the manipulation of .pbxproj files, and many other people may be looking for the same information, here's how I do it. It took me quite a while to figure this out because I was very unfamiliar with Xcode when I started attempting this, so I hope this saves others the hours of grief I had to put in.
You can use the plutil command to transform the .pbxproj file from the legacy .plist format into an XML or JSON format you will be able to manipulate more easily. I'm using JSON. To do so, just run:
plutil -convert json project.pbxproj
This will convert the format of project.pbxproj, but be aware that -contrary to common sense- the output won't be another file with a JSON extention such as project.json. What will happen is that project.pbxproj will be converted to JSON format, but retain it's cryptic .pbxproj extension. So even though the file's format has been changed, Xcode will still pick it up and use it in its new JSON format.
Then you can change project.pbxproj with ease using any JSON manipulation tool of your choosing. I'm using Groovy's JsonSlurper class in a Groovy script.
Note I also explored the XML option, but I found the project.pbxproj file in XML format to be cumbersome to parse. The elements are not properly nested to allow for traversing the tree with ease. It's plagued with:
<key>someKey</key>
<dict>
<!--More elements which provide configuration for the key above-->
</dict>
So it's positional in nature. You have to look for the key element corresponding to the setting you want to manipulate and then jump to the dict element just after it. Which means you have to mount the children of each XML element into an array, in order to index them.
Here are 3 open-source tools which implement .pbxproj file editing:
https://github.com/CocoaPods/Xcodeproj (Ruby based)
https://github.com/apache/cordova-node-xcode (NodeJS based)
https://github.com/kronenthaler/mod-pbxproj (Python based)
Personally, I made the best experience with the NodeJS based tool. So far it has covered all our needs reliably.
In the following is listed an example javascript file update-project.js which sets the developer team ID, app entitlements, adds a GoogleService-Info.plist file to the project and checks it as part of the build target. Take it as an inspiration and adapt the scripts and its paths to your needs:
const fs = require('fs')
const xcode = require('xcode')
if (process.argv.length !== 3) {
console.error("Please pass the development team ID as the first argument")
process.exit(1)
}
const developmentTeamId = process.argv[2]
const path = 'ios/App/App.xcodeproj/project.pbxproj'
const project = xcode.project(path)
project.parse(error => {
const targetKey = project.findTargetKey('App')
const appGroupKey = project.findPBXGroupKey({path: 'App'})
project.addBuildProperty('CODE_SIGN_ENTITLEMENTS', 'App/App.entitlements')
project.addBuildProperty('DEVELOPMENT_TEAM', developmentTeamId)
project.addFile('App.entitlements', appGroupKey)
project.removeFile('GoogleService-Info.plist', appGroupKey)
const f = project.addFile('GoogleService-Info.plist', appGroupKey, {target: targetKey})
f.uuid = project.generateUuid()
project.addToPbxBuildFileSection(f)
project.addToPbxResourcesBuildPhase(f)
fs.writeFileSync(path, project.writeSync())
})
Above script can be executed with
yarn run update-project <arguments...>
given that update-project is registered in package.json:
{
...,
"scripts": {
...
"update-project": "node update-project.js"
},
...
}
I'm using aapt tool to remove some files from different folders of my apk. This works fine.
But when I want to add files to the apk, the aapt tool add command doesn't let me specify the path to where I want the file to be added, therefore I can add files only to the root folder of the apk.
This is strange because I don't think that developers would never want to add files to a subfolder of the apk (res folder for example). Is this possible with aapt or any other method? Cause removing files from any folder works fine, and adding file works only for the root folder of the apk. Can't use it for any other folder.
Thanks
The aapt tool retains the directory structure specified in the add command, if you want to add something to an existing folder in an apk you simply must have a similar folder on your system and must specify each file to add fully listing the directory. Example
$ aapt list test.apk
res/drawable-hdpi/pic1.png
res/drawable-hdpi/pic2.png
AndroidManifest.xml
$ aapt remove test.apk res/drawable-hdpi/pic1.png
$ aapt add test.apk res/drawable-hdpi/pic1.png
The pic1.png that will is added resides in a folder in the current working directory of the terminal res/drawable-hdpi/ , hope this answered your question
There is actually a bug in aapt that will make this randomly impossible. The way it is supposed to work is as the other answer claims: paths are kept, unless you pass -k. Let's see how this is implemented:
The flag that controls whether the path is ignored is mJunkPath:
bool mJunkPath;
This variable is in a class called Bundle, and is controlled by two accessors:
bool getJunkPath(void) const { return mJunkPath; }
void setJunkPath(bool val) { mJunkPath = val; }
If the user specified -k at the command line, it is set to true:
case 'k':
bundle.setJunkPath(true);
break;
And, when the data is being added to the file, it is checked:
if (bundle->getJunkPath()) {
String8 storageName = String8(fileName).getPathLeaf();
printf(" '%s' as '%s'...\n", fileName, storageName.string());
result = zip->add(fileName, storageName.string(),
bundle->getCompressionMethod(), NULL);
} else {
printf(" '%s'...\n", fileName);
result = zip->add(fileName, bundle->getCompressionMethod(), NULL);
}
Unfortunately, the one instance of Bundle used by the application is allocated in main on the stack, and there is no initialization of mJunkPath in the constructor, so the value of the variable is random; without a way to explicitly set it to false, on my system I (seemingly deterministically) am unable to add files at specified paths.
However, you can also just use zip, as an APK is simply a Zip file, and the zip tool works fine.
(For the record, I have not submitted the trivial fix for this as a patch to Android yet, if someone else wants to the world would likely be a better place. My experience with the Android code submission process was having to put up with an incredibly complex submission mechanism that in the end took six months for someone to get back to me, in some cases with minor modifications that could have just been made on their end were their submission process not so horribly complex. Given that there is a really easy workaround to this problem, I do not consider it important enough to bother with all of that again.)