unable to define ssh key when using terraform to create linux vm - terraform-provider-azure

I'm trying to use terraform to create linux vm. what I see online is pretty straight forward
resource "tls_private_key" "this" {
for_each = local.worker_env_map
algorithm = "RSA"
rsa_bits = 4096
}
resource "azurerm_linux_virtual_machine" "example" {
name = "worker-machine"
resource_group_name = "rogertest"
location = "australiaeast"
size = "Standard_D2_v4"
admin_username = data.azurerm_key_vault_secret.kafkausername.value
network_interface_ids = [
azurerm_network_interface.example.id,
]
admin_ssh_key {
username = "adminuser"
public_key = tls_private_key.this["env1"].public_key_openssh
}
os_disk {
caching = "ReadWrite"
storage_account_type = "Standard_LRS"
}
source_image_reference {
publisher = "Canonical"
offer = "UbuntuServer"
sku = "18_04-lts-gen2"
version = "latest"
}
}
but i'm keep getting this error
Code="InvalidParameter" Message="Destination path for SSH public keys is currently limited to its default value /home/kafkaadmin/.ssh/authorized_keys due to a known issue in Linux provisioning agent."
Target="linuxConfiguration.ssh.publicKeys.path"
but I'm following as exactly outline on this page?
https://learn.microsoft.com/en-us/azure/virtual-machines/linux/quick-create-terraform

I tired to reproduce the same issue in my environment and got the below results
This is the error I am getting for destination path for SSH public keys are currently limited to its default value, destination path on the VM for the SSH keys if the file is already exist the specific keys are appended to the file
If we need a non-default location for public keys then at the moment, the only way is to create our own custom solution.
I have used the below command to create own path for keys
az vm create --resource-group rg_name --name myVM --image UbuntuLTS --admin-username user_name --generate-ssh-keys --ssh-dest-key-path './'
I have the Linux-vm terraform code using this Document
I have followed the below steps to execute the file
terraform init
Using the above command it will initialize the file
terraform plan
This will creates an execution plan and it will preview the changes that terraform plans to make the infrastructure it will show the monitoring and email notification rules
terraform apply
This will creates or updates the infrastructure depending on the configuration and also creates the metric rules for the flexible server
I am able to see the created Linux-virtual machine
NOTE: For creating Linux-vm we can use this terraform Document also for reference

Related

Terraform: "Error: Reference to undeclared resource when calling modules from terragrunt" in Azure

I'm trying to use terragrunt for the first time. I have followed the directory structure referred to https://terratest.gruntwork.io/docs/getting-started/quick-start/. I wanted to ret gid of duplicate main.tf, outputs.tf, and vars.tf that I have been using inside my environment folders. Below are the version and error that I'm facing. Any help would be greatly appreciated. Thanks in advance.
Terragrunt version
terragrunt version v0.23.10
Terraform version
Terraform v0.12.24
Directory Structure
terraform-live/
├── prod
│ └── resource_group
│ ├── main.tf
│ └── terragrunt.hcl
└── terragrunt.hcl
contents of terraform-live/terragrunt.hcl
backend = "azurerm"
config = {
key = "${path_relative_to_include()}/terraform.tfstate"
resource_group_name = "common-rg"
storage_account_name = "testsa01"
container_name = "tfstate"
}
}
contents of terraform-live/prod/resource_group/main.tf
backend "azurerm" {}
}
contents of terraform-live/prod/resource_group/terragrunt.hcl
terraform {
source = "git::git#github.com:adi4dpeople/terraform_modules.git//resource_group?ref=v0.0.1"
}
# Include all settings from the root terragrunt.hcl file
include {
path = find_in_parent_folders()
}
# These are the variables we have to pass in to use the module specified in the terragrunt configuration above
inputs = {
location = "westus"
rg_name = "testrg01"
}
When i run terragrunt plan, i get the following error:
[terragrunt] 2020/04/24 22:24:39 Reading Terragrunt config file at /home/aditya/terraform-live/prod/resource_group/terragrunt.hcl
[terragrunt] [/home/aditya/terraform-live/prod/resource_group] 2020/04/24 22:24:39 Running command: terraform --version
[terragrunt] 2020/04/24 22:24:44 Terraform files in /home/aditya/terraform-live/prod/resource_group/.terragrunt-cache/Hovi5Z9TKrGgHU_Lf1P2xFmhkm0/4M87gZKvnrwMknqj9CwuSBSfiHk/resource_group are up to date. Will not download again.
[terragrunt] 2020/04/24 22:24:44 Copying files from /home/aditya/terraform-live/prod/resource_group into /home/aditya/terraform-live/prod/resource_group/.terragrunt-cache/Hovi5Z9TKrGgHU_Lf1P2xFmhkm0/4M87gZKvnrwMknqj9CwuSBSfiHk/resource_group
[terragrunt] 2020/04/24 22:24:44 Setting working directory to /home/aditya/terraform-live/prod/resource_group/.terragrunt-cache/Hovi5Z9TKrGgHU_Lf1P2xFmhkm0/4M87gZKvnrwMknqj9CwuSBSfiHk/resource_group
[terragrunt] [/home/aditya/terraform-live/prod/resource_group] 2020/04/24 22:24:44 Backend azurerm has not changed.
[terragrunt] [/home/aditya/terraform-live/prod/resource_group] 2020/04/24 22:24:44 Running command: terraform init -backend-config=access_key=xxxxxxxxxxxx -backend-config=container_name=tfstate -backend-config=key=prod/resource_group/terraform.tfstate -backend-config=resource_group_name=testrg01 -backend-config=storage_account_name=testsa01
Initializing the backend...
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
[terragrunt] 2020/04/24 22:24:52 Running command: terraform plan
Acquiring state lock. This may take a few moments...
Error: Reference to undeclared resource
on outputs.tf line 2, in output "id":
2: value = azurerm_resource_group.rg.id
A managed resource "azurerm_resource_group" "rg" has not been declared in the
root module.
Error: Reference to undeclared resource
on outputs.tf line 6, in output "name":
6: value = azurerm_resource_group.rg.name
A managed resource "azurerm_resource_group" "rg" has not been declared in the
root module.
Releasing state lock. This may take a few moments...
[terragrunt] 2020/04/24 22:25:01 Hit multiple errors:
exit status 1
aditya#LAPTOP-6C2MPJDV:~/terraform-live/prod/resource_group$
I have solved my problem with this GitHub issue on terragrunt
https://github.com/gruntwork-io/terragrunt/issues/1151
I faced a similar issue when trying to create assign a pull role to an Azure Kubernetes cluster to pull images from an Azure container registry using a Managed system identity
Azure role assignment (outputs.tf file)
output "acr_id" {
value = azure_container_registry.acr.id
}
This was put in a module directory called azure-role-assignment
However, when I call the module output file in my Test environment (main.tf file):
# Create azure container registry
module "azure_container_registry" {
source = "../modules/azure-container-registry"
container_registry_name = var.container_registry_name
resource_group_name = var.resource_group_name
location = var.location
sku = var.sku
admin_enabled = var.admin_enabled
}
# Create azure role assignment
module "azure_role_assignment" {
source = "../modules/azure-role-assignment"
scope = module.azure_container_registry.acr_id
role_definition_name = var.role_definition_name
principal_id = module.azure_kubernetes_cluster.principal_id
}
However, when I run terraform apply, I get the error:
Error: Reference to undeclared resource
on ../modules/azure-container-registry/outputs.tf line 2, in output "acr_id":
2: value = azure_container_registry.acr.id
A managed resource "azure_container_registry" "acr" has not been declared in
module.azure_container_registry.
Here's how I solved it:
The issue was from how I defined the value of the arc_id in the outputs.tf file. Instead of this:
Azure role assignment (outputs.tf file)
output "acr_id" {
value = azure_container_registry.acr.id
}
It should be this:
Azure role assignment (`outputs.tf` file)
output "acr_id" {
value = azurerm_container_registry.acr.id
}
That is azurerm_container_registry.acr.id and not azure_container_registry.acr.id
That's all.
I hope this helps

pyhdfs.HdfsIOException: Failed to find datanode, suggest to check cluster health. excludeDatanodes=null

I am trying to run hadoop using docker provided here:
https://github.com/big-data-europe/docker-hadoop
I use the following command:
docker-compose up -d
to up the service and am able to access it and browse file system using: localhost:9870. Problem rises whenever I try to use pyhdfs to put file on HDFS. Here is my sample code:
hdfs_client = HdfsClient(hosts = 'localhost:9870')
# Determine the output_hdfs_path
output_hdfs_path = 'path/to/test/dir'
# Does the output path exist? If not then create it
if not hdfs_client.exists(output_hdfs_path):
hdfs_client.mkdirs(output_hdfs_path)
hdfs_client.create(output_hdfs_path + 'data.json', data = 'This is test.', overwrite = True)
If test directory does not exist on HDFS, the code is able to successfully create it but when it gets to the .create part it throws the following exception:
pyhdfs.HdfsIOException: Failed to find datanode, suggest to check cluster health. excludeDatanodes=null
What surprises me is that my code is able to create the empty directory but fails to put the file on HDFS. My docker-compose.yml file is exactly the same as the one provided in the github repo. The only change I've made is in the hadoop.env file where I change:
CORE_CONF_fs_defaultFS=hdfs://namenode:9000
to
CORE_CONF_fs_defaultFS=hdfs://localhost:9000
I have seen this other post on sof and tried the following command:
hdfs dfs -mkdir hdfs:///demofolder
which works fine in my case. Any help is much appreciated.
I would keep the default CORE_CONF_fs_defaultFS=hdfs://namenode:9000 setting.
Works fine for me after adding a forward slash to the paths
import pyhdfs
fs = pyhdfs.HdfsClient(hosts="namenode")
output_hdfs_path = '/path/to/test/dir'
if not fs.exists(output_hdfs_path):
fs.mkdirs(output_hdfs_path)
fs.create(output_hdfs_path + '/data.json', data = 'This is test.')
# check that it's present
list(fs.walk(output_hdfs_path))
[('/path/to/test/dir', [], ['data.json'])]

docker containers - missing attribute

I needed a custom centos image with docker installed. So I built it using centos image and tagged it custom (shown below).
$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
centos custom 84766562f881 4 hours ago 664MB
centos/systemd latest 05d3c1e2d0c1 7 weeks ago 202MB
I am trying to deploy couple of containers using Terraform on my local machine, each with a unique name that comes from another file. The docker images are on local machine. Here's the TF code.
$ cat main.tf
provider "docker" {
}
resource "docker_image" "centos" {
name = "centos:custom"
}
resource "docker_container" "app_swarm" {
image = "${docker_image.centos.custom}"
count = "${length(var.docker_cont)}"
name = "${element(var.docker_cont, count.index)}"
}
When I run terraform apply, I get this error which I am not sure how to fix. Can someone point me in the right direction please?
Error: Error running plan: 1 error(s) occurred:
* docker_container.app_swarm: 3 error(s) occurred:
* docker_container.app_swarm[0]: Resource 'docker_image.centos' does not have attribute 'custom' for variable 'docker_image.centos.custom'
* docker_container.app_swarm[1]: Resource 'docker_image.centos' does not have attribute 'custom' for variable 'docker_image.centos.custom'
* docker_container.app_swarm[2]: Resource 'docker_image.centos' does not have attribute 'custom' for variable 'docker_image.centos.custom'
Yes, the other file exists with names, its a simple list.
EDIT:
Thanks David, tried your suggestion and amended the code to look like-
provider "docker" {
}
resource "docker_image" "centos" {
name = "centos:custom"
}
resource "docker_container" "app_swarm" {
image = "${docker_image.centos.latest}"
count = "${length(var.docker_cont)}"
name = "${element(var.docker_cont, count.index)}"
}
But now I get this error.
Error: Error applying plan:
1 error(s) occurred:
* docker_image.centos: 1 error(s) occurred:
* docker_image.centos: Unable to read Docker image into resource: Unable to pull image centos:custom: error pulling image centos:custom: Error response from daemon: manifest for centos:custom not found
I guess I will have to setup a local Docker repository to get this working but I am not sure?
You can only use the specific fields listed in the docker_image resource documentation in the ${docker_image.centos...} interpolation. In particular, even though you don't use the tag :latest, you need a .latest property reference:
image = "${docker_image.centos.latest}"
(If the image actually is one you've built locally, you may also want to specify the keep_locally option on your docker_image resource so that terraform destroy won't delete it.)

Erlang :ssh authentication error. How to connect to ssh using identity file

I'm getting an authentication error when trying to connect ssh host.
The goal is to connect to the host using local forwarding. The command below is an example using drop bear ssh client to connect to host with local forwarding.
dbclient -N -i /opt/private-key-rsa.dropbear -L 2002:1.2.3.4:2006 -p 2002 -l
test_user 11.22.33.44
I have this code so far which returns empty connection
ip = "11.22.33.44"
user = "test_user"
port = 2002
ssh_config = [
user_interaction: false,
silently_accept_hosts: true,
user: String.to_charlist(user),
user_dir: String.to_charlist("/opt/")
]
# returns aunthentication error
{:ok, conn} = :ssh.connect(String.to_charlist(ip), port, ssh_config)
This is the error Im seeing
Server: 'SSH-2.0-OpenSSH_5.2'
Disconnects with code = 14 [RFC4253 11.1]: Unable to connect using the available authentication methods
State = {userauth,client}
Module = ssh_connection_handler, Line = 893.
Details:
User auth failed for: "test_user"
I'm a newbie to elixir and have been reading this erlang ssh document for 2 days. I did not find any examples in the documentation which makes it difficult to understand.
You are using non-default key name, private-key-rsa.dropbear. Erlang by default looks for this set of names:
From ssh module docs:
Optional: one or more User's private key(s) in case of publickey authorization. The default files are
id_dsa and id_dsa.pub
id_rsa and id_rsa.pub
id_ecdsa and id_ecdsa.pub`
To verify this is a reason, try renaming private-key-rsa.dropbear to id_rsa. If this works, the next step would be to add a key_cb callback to the ssh_config which should return the correct key file name.
One example implementation of a similar feature is labzero/ssh_client_key_api.
The solution was to convert dropbear key to ssh key. I have used this link as reference.
Here is the command to convert dropbear key to ssh key
/usr/lib/dropbear/dropbearconvert dropbear openssh /opt/private-key-rsa.dropbear /opt/id_rsa

How to build a NixOps deployment on hydra

Deploying my NixOps machines takes alot of time, as packages need to build. I want to do the building regularly on my trusted private Hydra instance.
My current approach involves this release.nix file, but it doesn't work out so well.
{ nixpkgs ? <nixpkgs>, onlySystem ? true, extraModules ? [] }:
let
nixos = import "${nixpkgs}/nixos";
buildEnv = conf: (nixos {
configuration = conf;
});
buildTarget = m: let build = buildEnv (buildConf m); in
if onlySystem then build.system else build.vm;
buildConf = module: { ... }:
{
imports = [ module ] ++ extraModules;
};
in
{
machine1 = buildTarget ./machine1/configuration.nix;
machine2 = buildTarget ./machine2/configuration.nix
machine3 = buildTarget ./machine3/configuration.nix
machine4 = buildTarget ./machine4/configuration.nix
}
I don't really understand this code, as I copied it from here.
This builds fine if I run nix-build release.nix locally, but on hydra I never get a full build. Sometimes builds don't dequeue (they just don't get build), sometimes they fail with various error messages. As nothing of the hydra problems are reproducible (beside the fact, that I never get a full build), I wonder what the best practice for building a NixOps deployment is.
Please note, that I have unfree packages in my deployment. The option nixpkgs.config.allowUnfree = true; is set on the hydra server.
This is not a question about my hydra failures, but about what would be a good way to build a NixOps deployment with Hydra CI.
As far as I know, there is no way to make this super easy. Your code looks ok, but my method is a slightly different. I only build the toplevel attribute and I construct the NixOS configuration differently.
I build NixOS 'installations' from inside Nix using something like:
let
modules = [ ./configuration.nix ];
nixosSystem = import (pkgs.path + "/nixos/lib/eval-config.nix") {
inherit (pkgs) system;
inherit modules;
};
in
nixosSystem.config.system.build.toplevel

Resources