For existing flow, there would be a whole bunch of namespaces loaded when running some script job.
However, if I want to check & trace the usage of some command in some namespace, I need to find the script path of the certain namespace.
Is there some way to get that? Particularly, I'm talking about Primetime scripts.
Technically, namespaces don't have script paths. But we can do something close enough:
proc report_current_file {call code result op} {
if {$code == 0} {
# If the [proc] call was successful...
set cmd [lindex $call 1]
set qualified_cmd [uplevel 1 [list namespace which $cmd]]
set file [file normalize [info script]]
puts "Defined $qualified_cmd in $file"
}
}
trace add execution proc leave report_current_file
It's not perfect if you've got procedures creating procedures dynamically — the current file might be wrong — but that's fortunately not what most code does.
Another option that might work for you is to use tcl::unsupported::getbytecode, which produces a lot of information in machine-readable format (a dictionary). One of the pieces of information is the sourcefile key. Here's an example running interactively on my machine:
% parray tcl_platform
tcl_platform(byteOrder) = littleEndian
tcl_platform(engine) = Tcl
tcl_platform(machine) = x86_64
tcl_platform(os) = Darwin
tcl_platform(osVersion) = 20.2.0
tcl_platform(pathSeparator) = :
tcl_platform(platform) = unix
tcl_platform(pointerSize) = 8
tcl_platform(threaded) = 1
tcl_platform(user) = dkf
tcl_platform(wordSize) = 8
% dict get [tcl::unsupported::getbytecode proc parray] sourcefile
/opt/local/lib/tcl8.6/parray.tcl
Note that the procedure has to be already defined for this to work. And if Tcl's become confused about what file the code was in (because of dynamic programming trickery) then that key is absent.
Related
I have the following three files as below:
main.tf, variables.tf and dev.auto.tfvars
Snippet from main.tf
module "sql_vms" {
source = "git::git#github.com:xxxxxxxxxxxx/terraform-modules//azure/"
rg_name = var.resource_group_name
location = module.resource_group.external_rg_location
vnet_name = var.virtual_network_name
subnet_name = var.sql_subnet_name
app_nsg = var.application_nsg
vm_count = var.count_vm
base_hostname = var.sql_host_basename
sto_acc_suffix = var.storage_account_suffix
vm_size = var.virtual_machine_size
vm_publisher = var.virtual_machine_image_publisher
vm_offer = var.virtual_machine_image_offer
vm_sku = var.virtual_machine_image_sku
vm_img_version = var.virtual_machine_image_version
username = var.username
password = var.password
}
Snippet from variables.tf
variable "app_subnet_name" {
type = string
}
variable "sql_subnet_name" {
type = string
}
Snippet from dev.auto.tfvars
app_subnet_name = "subnet_1"
sql_subnet_name = "subnet_2"
application_nsg = "test_nsg"
However, I'm getting error like below
Error: Unsupported argument
on main.tf line 7, in module "sql_vms":
7: subnet_name = var.sql_subnet_name
An argument named "subnet_name" is not expected here.
Error: Unsupported argument
on main.tf line 8, in module "sql_vms":
8: app_nsg = var.application_nsg
An argument named "app_nsg" is not expected here.
My modules directory structure looks like below
$ ls -R terraform-modules/
terraform-modules/:
aws azure gcp
terraform-modules/aws:
alb ec2-instance-rhel
terraform-modules/aws/alb:
terraform-modules/aws/ec2-instance-rhel:
main.tf
terraform-modules/azure:
compute resourcegroup sqlserver
terraform-modules/azure/compute:
main.tf README.md variable.tf
terraform-modules/azure/resourcegroup:
data.tf outputs.tf variables.tf
terraform-modules/azure/sqlserver:
main.tf README.md variables.tf
terraform-modules/gcp:
compute
terraform-modules/gcp/compute:
main.tf
Any idea what is going wrong here?
If you are starting out with Terraform, you will get that error message ("An argument named "example" is not expected here") if your module arguments refer to the resource properties and not to variable names, see below for an example:
Example of a Terraform module "example_mod.tf" you want to call from your module:
variable "sg_name" { } # Usually in a separate file
variable "sg_desc" { } # called variables.tf
resource "example_resource" "example_name" {
name = var.sg_name
description = var.sg_desc
...
}
CORRECT WAY:
module "my_module" {
source = "./modules/example_mod.tf"
sg_name = "whatever" # NOTE the left hand side "sg_name" is the variable name
sg_desc = "whatever"
...
}
INCORRECT WAY: (Gives the error "An argument named "name" is not expected here" )
module "my_module" {
source = "./modules/example_mod.tf"
name = "whatever" # WRONG because the left hand side "name" is a resource property
description = "whatever" # WRONG for the same reason
...
}
I think the issue is that you do not refer to the exact module with the source. I see you have three modules in the source:
source = "git::git#github.com:xxxxxxxxxxxx/terraform-modules//azure/"
They are compute, resourcegroup and sqlserver. But you want to load them in one module. So it cannot find the related variables for the modules. I also don't think it's the right way to load all the modules like that. I would recommend you load the modules one by one like below:
module "compute" {
source = "git::git#github.com:xxxxxxxxxxxx/terraform-modules//azure/compute"
...
}
module "resourcegroup" {
source = "git::git#github.com:xxxxxxxxxxxx/terraform-modules//azure/resourcegroup"
...
}
module "sqlserver" {
source = "git::git#github.com:xxxxxxxxxxxx/terraform-modules//azure/sqlserver"
...
}
Without knowing the details about the module it is usually hard to say what's the reason for an error, but in this particular case it seems that there isn't a requirement in the module you're importing to use those two arguments (subnet_name and app_nsg), or rather that you are using a version of the module that doesn't require them to be present. What helps with that type of error is to check if there is a version of the module that does have such a requirement. The syntax for using a particular module version from Github is explained in Terraform Module Sources documentation, Selecting a Revision section:
module "vpc" {
source = "git::https://example.com/vpc.git?ref=v1.2.0"
}
You are probably using SSH to fetch the module, so the recommended way to do that is:
When using Git over SSH, we recommend using the ssh://-prefixed URL form for consistency with all of the other URL-like git address forms.
In your example, this translates to:
module "sql_vms" {
source = "git::ssh://git#github.com/org/terraform-modules-repo.git//azure/module-name?ref=v1.2.0"
where org is your organisation's (or your private) Github account, terraform-modules-repo is the repo where modules reside, module-name is the module you are using and ref=v1.2.0 represents the module revision number.
The error An argument named "example" is not expected here. means that the module doesn't expect to see an input argument with that name. Think about Terraform modules as functions in a programming language: in order to have a function provide a result, you pass the function a set of required arguments. If you provide more (or less) input arguments than required by that function call, you will get an error. (There are special cases but it is out of the scope of this question.)
Another similarity between modules and functions is that Terraform modules can also provide output values, besides creating resources that are specified. That can be handy in cases where output can be used as input in other modules or resources. The line module.resource_group.external_rg_location is doing exactly that: getting the output value from another module and using it to assign a value to an argument location.
I had a similar issue when working with AWS Eventbridge and Terraform.
When I run terraform plan I get the error below:
Error: Unsupported argument
│
│ on ../../modules/aws/eventbridge/main.tf line 37, in resource "aws_cloudwatch_event_target" "ecs_cloudwatch_event_target":
│ 37: maximum_age_in_seconds = var.maximum_age_in_seconds
│
│ An argument named "maximum_age_in_seconds" is not expected here.
Here's how I solved it:
The issue was that I was not using the correct attribute for the AWS Eventbridge resource block.
The attribute should have been maximum_event_age_in_seconds and not maximum_age_in_seconds.
Another issue that could this is not defining a variable in your terraform script that is already defined in a module.
That's all
It could be happening due to plenty of reasons.
I'd suggest some verification:
Check if you are using the correct source URL, path or revision branch/tag.
I'm not sure about your implementation, but you probably want to double check the revision you are referencing contains theses variable declarations.
GitHub Modules addressing allows ref argument.
Refer to the GitHub Module Addressing for Terraform Documentation and how to specify a revision.
Check if all necessary variables are declared on every module, including the root module.
Did you declare those variables both in a variables.tf file on your root directory and on the module context/path?
I know that's exhausting and repetitive, but every module should be designed as an "independent project". Each module **MUST have its own declared variables.tf**, which work as inputs for that module, and it is also desirable that it has its own mapped outputs.tf, provider.tf, backend.tf, etc., though these last ones are not required.
FYI: Doing so you guarantee scalability, reusability, as well as reliability to work with different tfstate files and even different repositories for each module in order to guarantee atomicity and minimum permissions, hence preventing your infrastructure from being destroyed by undesired code changes.
I highly recommend this read to understand the importance of independent modularization design.
Furthermore, tools like Terragrunt, Terratest can make this job less painful by keeping your code DRY ( Don't Repeat Yourself ).
Check if the **type constraints of the related variables match.**
If that's not your case, try looking if the type constraints match between all declarations of the variables used both as arguments ( on your root variables.tf ) and inputs ( on your module level variables.tf ).
I'll share my pain as well.
Writing block configuration like this
vpc_config = {
subnet_ids = [aws_subnet.example1.id, aws_subnet.example2.id]
}
Instead of (Notice the = Equal Sign):
vpc_config {
subnet_ids = [aws_subnet.example1.id, aws_subnet.example2.id]
}
Will give an error of An argument named "vpc_config" is not expected here and will waste you a few good hours.
In pyspark reading csv files from different paths gets failed if even one path does not exist.
Logs = spark.read.load(Logpaths, format="csv", schema=logsSchema, header="true", mode="DROPMALFORMED");
Here Logpaths is an array that contain multiple paths. And these paths are created dynamically depending upon given startDate and endDate range. If Logpaths contain 5 paths and first 3 exists but 4th does not exist. Then whole extraction gets failed. How can I avoid this in pyspark or how can I check there existance before reading?
In scala I did this by checking file existance and filter out non-existed records by using hadoop hdfs filesystem globStatus function.
Path = '/bilal/2018.12.16/logs.csv'
val hadoopConf = new org.apache.hadoop.conf.Configuration()
val fs = org.apache.hadoop.fs.FileSystem.get(hadoopConf)
val fileStatus = fs.globStatus(new org.apache.hadoop.fs.Path(Path));
So I got what I was looking for. Like the code I posted in the question which can be used in scala for file existance check. We can use below code in case of PySpark.
fs = sc._jvm.org.apache.hadoop.fs.FileSystem.get(sc._jsc.hadoopConfiguration())
fs.exists(sc._jvm.org.apache.hadoop.fs.Path("bilal/logs/log.csv"))
This is exactly the same code also used in scala, so in this case we are using java library for hadoop and java code runs on JVM on which spark is running.
In my Informix 4GL program, I have an input field where the user can insert a URL and the feed is later being sent over to the web via a script.
How can I validate the URL at the time of input, to ensure that it's a live link? Can I make a call and see if I get back any errors?
I4GL checking the URL
There is no built-in function to do that (URLs didn't exist when I4GL was invented, amongst other things).
If you can devise a C method to do that, you can arrange to call that method through the C interface. You'll write the method in native C, and then write an I4GL-callable C interface function using the normal rules. When you build the program with I4GL c-code, you'll link the extra C functions too. If you build the program with I4GL-RDS (p-code), you'll need to build a custom runner with the extra function(s) exposed. All of this is standard technique for I4GL.
In general terms, the C interface code you'll need will look vaguely like this:
#include <fglsys.h>
// Standard interface for I4GL-callable C functions
extern int i4gl_validate_url(int nargs);
// Using obsolescent interface functions
int i4gl_validate_url(int nargs)
{
if (nargs != 1)
fgl_fatal(__FILE__, __LINE__, -1318);
char url[4096];
popstring(url, sizeof(url));
int r = validate_url(url); // Your C function
retint(r);
return 1;
}
You can and should check the manuals but that code, using the 'old style' function names, should compile correctly. The code can be called in I4GL like this:
DEFINE url CHAR(256)
DEFINE rc INTEGER
LET url = "http://www.google.com/"
LET rc = i4gl_validate_url(url)
IF rc != 0 THEN
ERROR "Invalid URL"
ELSE
MESSAGE "URL is OK"
END IF
Or along those general lines. Exactly what values you return depends on your decisions about how to return a status from validate_url(). If need so be, you can return multiple values from the interface function (e.g. error number and text of error message). Etc. This is about the simplest possible design for calling some C code to validate a URL from within an I4GL program.
Modern C interface functions
The function names in the interface library were all changed in the mid-00's, though the old names still exist as macros. The old names were:
popstring(char *buffer, int buflen)
retint(int retval)
fgl_fatal(const char *file, int line, int errnum)
You can find the revised documentation at IBM Informix 4GL v7.50.xC3: Publication library in PDF in the 4GL Reference Manual, and you need Appendix C "Using C with IBM Informix 4GL".
The new names start ibm_lib4gl_:
ibm_libi4gl_popMInt()
ibm_libi4gl_popString()
As to the error reporting function, there is one — it exists — but I don't have access to documentation for it any more. It'll be in the fglsys.h header. It takes an error number as one argument; there's the file name and a line number as the other arguments. And it will, presumably, be ibm_lib4gl_… and there'll be probably be Fatal or perhaps fatal (or maybe Err or err) in the rest of the name.
I4GL running a script that checks the URL
Wouldn't it be easier to write a shell script to get the status code? That might work if I can return the status code or any existing results back to the program into a variable? Can I do that?
Quite possibly. If you want the contents of the URL as a string, though, you'll might end up wanting to call C. It is certainly worth thinking about whether calling a shell script from within I4GL is doable. If so, it will be a lot simpler (RUN "script", IIRC, where the literal string would probably be replaced by a built-up string containing the command and the URL). I believe there are file I/O functions in I4GL now, too, so if you can get the script to write a file (trivial), you can read the data from the file without needing custom C. For a long time, you needed custom C to do that.
I just need to validate the URL before storing it into the database. I was thinking about:
#!/bin/bash
read -p "URL to check: " url
if curl --output /dev/null --silent --head --fail "$url"; then
printf '%s\n' "$url exist"
else
printf '%s\n' "$url does not exist"
fi
but I just need the output instead of /dev/null to be into a variable. I believe the only option is to dump the output into a temp file and read from there.
Instead of having I4GL run the code to validate the URL, have I4GL run a script to validate the URL. Use the exit status of the script and dump the output of curl into /dev/null.
FUNCTION check_url(url)
DEFINE url VARCHAR(255)
DEFINE command_line VARCHAR(255)
DEFINE exit_status INTEGER
LET command_line = "check_url ", url
RUN command_line RETURNING exit_status
RETURN exit_status
END FUNCTION {check_url}
Your calling code can analyze exit_status to see whether it worked. A value of 0 indicates success; non-zero indicates a problem of some sort, which can be deemed 'URL does not work'.
Make sure the check_url script (a) exits with status zero on success and non-zero on any sort of failure, and (b) doesn't write anything to standard output (or standard error) by default. The writing to standard error or output will screw up screen layouts, etc, and you do not want that. (You can obviously have options to the script that enable standard output, or you can invoke the script with options to suppress standard output and standard error, or redirect the outputs to /dev/null; however, when used by the I4GL program, it should be silent.)
Your 'script' (check_url) could be as simple as:
#!/bin/bash
exec curl --output /dev/null --silent --head --fail "${1:-http://www.example.com/"
This passes the first argument to curl, or the non-existent example.com URL if no argument is given, and replaces itself with curl, which generates a zero/non-zero exit status as required. You might add 2>/dev/null to the end of the command line to ensure that error messages are not seen. (Note that it will be hell debugging this if anything goes wrong; make sure you've got provision for debugging.)
The exec is a minor optimization; you could omit it with almost no difference in result. (I could devise a scheme that would probably spot the difference; it involves signalling the curl process, though — kill -9 9999 or similar, where the 9999 is the PID of the curl process — and isn't of practical significance.)
Given that the script is just one line of code that invokes another program, it would be possible to embed all that in the I4GL program. However, having an external shell script (or Perl script, or …) has merits of flexibility; you can edit it to log attempts, for example, without changing the I4GL code at all. One more file to distribute, but better flexibility — keep a separate script, even though it could all be embedded in the I4GL.
As Jonathan said "URLs didn't exist when I4GL was invented, amongst other things". What you will find is that the products that have grown to superceed Informix-4gl such as FourJs Genero will cater for new technologies and other things invented after I4GL.
Using FourJs Genero, the code below will do what you are after using the Informix 4gl syntax you are familiar with
IMPORT com
MAIN
-- Should succeed and display 1
DISPLAY validate_url("http://www.google.com")
DISPLAY validate_url("http://www.4js.com/online_documentation/fjs-fgl-manual-html/index.html#c_fgl_nf.html") -- link to some of the features added to I4GL by Genero
-- Should fail and display 0
DISPLAY validate_url("http://www.google.com/testing")
DISPLAY validate_url("http://www.google2.com")
END MAIN
FUNCTION validate_url(url)
DEFINE url STRING
DEFINE req com.HttpRequest
DEFINE resp com.HttpResponse
-- Returns TRUE if http request to a URL returns 200
TRY
LET req = com.HttpRequest.create(url)
CALL req.doRequest()
LET resp = req.getResponse()
IF resp.getStatusCode() = 200 THEN
RETURN TRUE
END IF
-- May want to handle other HTTP status codes
CATCH
-- May want to capture case if not connected to internet etc
END TRY
RETURN FALSE
END FUNCTION
I need to use Lua to run a binary program that may write something in its stdout and also returns a status code (also known as "exit status").
I searched the web and couldn't find something that does what I need. However I found out that in Lua:
os.execute() returns the status code
io.popen() returns a file handler that can be used to read process output
However I need both. Writing a wrapper function that runs both functions behind the scene is not an option because of process overhead and possibly changes in result on consecutive runs. I need to write a function like this:
function run(binpath)
...
return output,exitcode
end
Does anyone has an idea how this problem can be solved?
PS. the target system rung Linux.
With Lua 5.2 I can do the following and it works
-- This will open the file
local file = io.popen('dmesg')
-- This will read all of the output, as always
local output = file:read('*all')
-- This will get a table with some return stuff
-- rc[1] will be true, false or nil
-- rc[3] will be the signal
local rc = {file:close()}
I hope this helps!
I can't use Lua 5.2, I use this helper function.
function execute_command(command)
local tmpfile = '/tmp/lua_execute_tmp_file'
local exit = os.execute(command .. ' > ' .. tmpfile .. ' 2> ' .. tmpfile .. '.err')
local stdout_file = io.open(tmpfile)
local stdout = stdout_file:read("*all")
local stderr_file = io.open(tmpfile .. '.err')
local stderr = stderr_file:read("*all")
stdout_file:close()
stderr_file:close()
return exit, stdout, stderr
end
This is how I do it.
local process = io.popen('command; echo $?') -- echo return code of last run command
local lastline
for line in process:lines() do
lastline = line
end
print(lastline) -- the return code is the last line of output
If the last line has fixed length you can read it directly using file:seek("end", -offset), offset should be the length of the last line in bytes.
This functionality is provided in C by pclose.
Upon successful return, pclose() shall return the termination status
of the command language interpreter.
The interpreter returns the termination status of its child.
But Lua doesn't do this right (io.close always returns true). I haven't dug into these threads but some people are complaining about this brain damage.
http://lua-users.org/lists/lua-l/2004-05/msg00005.html
http://lua-users.org/lists/lua-l/2011-02/msg00387.html
If you're running this code on Win32 or in a POSIX environment, you could try this Lua extension: http://code.google.com/p/lua-ex-api/
Alternatively, you could write a small shell script (assuming bash or similar is available) that:
executes the correct executable, capturing the exit code into a shell variable,
prints a newline and terminal character/string onto standard out
prints the shell variables value (the exit code) onto standard out
Then, capture all the output of io.popen and parse backward.
Full disclosure: I'm not a Lua developer.
yes , your are right that os.execute() has returns and it's very simple if you understand how to run your command with and with out lua
you also may want to know how many variables it returns , and it might take a while , but i think you can try
local a, b, c, d, e=os.execute(-what ever your command is-)
for my example a is an first returned argument , b is the second returned argument , and etc.. i think i answered your question right, based off of what you are asking.
I need to compare 2 executables and/or shared objects, compiled using the same compiler/flags and verify that they have not changed. We work in a regulated environment, so it would be really useful for testing purposes to isolate exactly what parts of the executable has changed.
Using MD5Sums/Hashes doesn't work due to the headers containing information about the file.
Does anyone know of a program or way to verify that 2 files are executionally the same even if they were built at a different time?
An interesting question. I have a similar problem on linux. Intrusion detection systems like OSSEC or tripwire may generate false positives if the hashsum of an executable changes all of a sudden. This may be nothing worse than the Linux "prelink" program patching the executable file for faster startups.
In order to compare two binaries (in the ELF format), one can use the "readelf" executable and then "diff" to compare outputs. I'm sure there are refined solutions, but without further ado, a poor man's comparator in Perl:
#!/usr/bin/perl -w
$exe = $ARGV[0];
if (!$exe) {
die "Please give name of executable\n"
}
if (! -f $exe) {
die "Executable $exe not found or not a file\n";
}
if (! (`file '$exe'` =~ /\bELF\b.*?\bexecutable\b/)) {
die "file command says '$exe' is not an ELF executable\n";
}
# Identify sections in ELF
#lines = pipeIt("readelf --wide --section-headers '$exe'");
#sections = ();
for my $line (#lines) {
if ($line =~ /^\s*\[\s*(\d+)\s*\]\s+(\S+)/) {
my $secnum = $1;
my $secnam = $2;
print "Found section $1 named $2\n";
push #sections, $secnam;
}
}
# Dump file header
#lines = pipeIt("readelf --file-header --wide '$exe'");
print #lines;
# Dump all interesting section headers
#lines = pipeIt("readelf --all --wide '$exe'");
print #lines;
# Dump individual sections as hexdump
for my $section (#sections) {
#lines = pipeIt("readelf --hex-dump='$section' --wide '$exe'");
print #lines;
}
sub pipeIt {
my($cmd) = #_;
my $fh;
open ($fh,"$cmd |") or die "Could not open pipe from command '$cmd': $!\n";
my #lines = <$fh>;
close $fh or die "Could not close pipe to command '$cmd': $!\n";
return #lines;
}
Now you can run for example, on machine 1:
./checkexe.pl /usr/bin/curl > curl_machine1
And on machine 2:
./checkexe.pl /usr/bin/curl > curl_machine2
After having copypasted, SFTP-ed or NSF-ed (you don't use FTP, do you?) the files into the same filetree, compare the files:
diff --side-by-side --width=200 curl_machine1 curl_machine2 | less
In my case, differences exist in section ".gnu.conflict", ".gnu.liblist", ".got.plt" and ".dynbss", which might be ok for a "prelink" intervention, but in the code section, ".text", which would be a Bad Sign.
To follow up, here is what I came up with finally:
Instead of comparing the final executables & shared objects, we compared the .o files output before linking. We assumed that the linking process was sufficiently reproducible that this would be fine.
It works in some of our cases, where we have two builds were we've made some small change that shouldn't effect the final code (Code pretty-printer) but doesn't help us if we do not have the build intermediary output.
You can compare the contents of RO and RW initialized sections by generating a binary file from the ELF file.
objcopy <elf_file> -O binary <binary_file>
Use the generated binary files to compare if they are identical, using diff, for example.
In my opinion, this is enough to grantee you are generating the same executable.
A few years back I had to do the same thing. We had to prove that we could rebuild the executable from source when given only a revision number, revision control repository, build tools, and build configuration. Note: If any of these change you may see a difference.
I remember there is some timestamps in the executable. The trick is to realize that the file is not just a bunch of bytes, that can not be interpreted. The file has sections, most will not change, but there will be a section for time of build (or some such thing).
I don't remember all the details, but the commands you will need are { objcopy, objdump, nm }, I think objdump would be the first to try.
Hope this helps.