GitLab CE docker container keeps crashing at startup - docker

Question

I'm trying to run GitLab CE edition as a docker container (gitlab/gitlab-ce) via docker compose, following the instructions at http://doc.gitlab.com/omnibus/docker.
The problem is that every time I start with docker-compose up -d, the container crashes/exits after about a minute. I collected all information that could be useful, there are some chef-related error message that I'm not able to decrypt. The environment runs insides an Ubuntu Vagrant virtual machine.
I tried to use a different tagged version of the image instead of the :latest, but getting similar results.
docker-compose.yml relevant snippet:
gitlab:
image: gitlab/gitlab-ce
container_name: my_gitlab
volumes:
- ./runtime/gitlab/config:/etc/gitlab
- ./runtime/gitlab/data:/var/opt/gitlab
- ./runtime/gitlab/logs:/var/log/gitlab
ports:
- 443:443
- 22:22
- 8082:80
following is the log file saved in ./runtime/gitlab/logs (volume for /var/log/gitlab)
# Logfile created on 2016-04-28 08:07:43 +0000 by logger.rb/44203
[2016-04-28T08:07:44+00:00] INFO: Started chef-zero at chefzero://localhost:8889 with repository at /opt/gitlab/embedded
One version per cookbook
[2016-04-28T08:07:44+00:00] INFO: Forking chef instance to converge...
[2016-04-28T08:07:44+00:00] INFO: *** Chef 12.6.0 ***
[2016-04-28T08:07:44+00:00] INFO: Chef-client pid: 36
[2016-04-28T08:07:47+00:00] INFO: HTTP Request Returned 404 Not Found: Object not found: chefzero://localhost:8889/nodes/bcfc5b569532
[2016-04-28T08:07:48+00:00] INFO: Setting the run_list to ["recipe[gitlab]"] from CLI options
[2016-04-28T08:07:48+00:00] INFO: Run List is [recipe[gitlab]]
[2016-04-28T08:07:48+00:00] INFO: Run List expands to [gitlab]
[2016-04-28T08:07:48+00:00] INFO: Starting Chef Run for bcfc5b569532
[2016-04-28T08:07:48+00:00] INFO: Running start handlers
[2016-04-28T08:07:48+00:00] INFO: Start handlers complete.
[2016-04-28T08:07:48+00:00] INFO: HTTP Request Returned 404 Not Found: Object not found:
[2016-04-28T08:07:52+00:00] INFO: Loading cookbooks [gitlab#0.0.1, runit#0.14.2, package#0.0.0]
[2016-04-28T08:07:54+00:00] INFO: directory[/etc/gitlab] owner changed to 0
[2016-04-28T08:07:54+00:00] INFO: directory[/etc/gitlab] group changed to 0
[2016-04-28T08:07:54+00:00] INFO: directory[/etc/gitlab] mode changed to 775
[2016-04-28T08:07:54+00:00] WARN: Cloning resource attributes for directory[/var/opt/gitlab] from prior resource (CHEF-3694)
[2016-04-28T08:07:54+00:00] WARN: Previous directory[/var/opt/gitlab]: /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb:43:in `from_file'
[2016-04-28T08:07:54+00:00] WARN: Current directory[/var/opt/gitlab]: /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/users.rb:24:in `from_file'
[2016-04-28T08:07:54+00:00] WARN: Selected upstart because /sbin/init --version is showing upstart.
[2016-04-28T08:07:54+00:00] WARN: Cloning resource attributes for directory[/etc/sysctl.d] from prior resource (CHEF-3694)
[2016-04-28T08:07:54+00:00] WARN: Previous directory[/etc/sysctl.d]: /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/definitions/sysctl.rb:22:in `block in from_file'
[2016-04-28T08:07:54+00:00] WARN: Current directory[/etc/sysctl.d]: /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/definitions/sysctl.rb:22:in `block in from_file'
[2016-04-28T08:07:54+00:00] WARN: Cloning resource attributes for file[/etc/sysctl.d/90-postgresql.conf] from prior resource (CHEF-3694)
.
. several similar WARN: log entries
.
[2016-04-28T08:07:55+00:00] INFO: directory[/var/opt/gitlab] owner changed to 0
[2016-04-28T08:07:55+00:00] INFO: directory[/var/opt/gitlab] group changed to 0
[2016-04-28T08:07:55+00:00] INFO: directory[/var/opt/gitlab] mode changed to 755
.
.
.
[2016-04-28T08:07:57+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/rack_attack.rb] owner changed to 0
[2016-04-28T08:07:57+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/rack_attack.rb] group changed to 0
[2016-04-28T08:07:57+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/rack_attack.rb] mode changed to 644
[2016-04-28T08:07:58+00:00] INFO: Running queued delayed notifications before re-raising exception
[2016-04-28T08:07:58+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab.yml] sending run action to execute[clear the gitlab-rails cache] (delayed)
[2016-04-28T08:09:02+00:00] ERROR: Running exception handlers
[2016-04-28T08:09:02+00:00] ERROR: Exception handlers complete
[2016-04-28T08:09:02+00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/chef-stacktrace.out
[2016-04-28T08:09:02+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2016-04-28T08:09:02+00:00] ERROR: Chef::Exceptions::MultipleFailures
[2016-04-28T08:09:02+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
/opt/gitlab/embedded/bin/chef-client:23:in `<main>'root#bcfc5b569532:/# tail -f /opt/gitlab/embedded/cookbooks/cache/chef-stacktrace.out
/opt/gitlab/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/local_mode.rb:44:in `with_server_connectivity'
/opt/gitlab/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/application.rb:203:in `run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/application/client.rb:413:in `block in interval_run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/application/client.rb:403:in `loop'
/opt/gitlab/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/application/client.rb:403:in `interval_run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/application/client.rb:393:in `run_application'
/opt/gitlab/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/application.rb:58:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.1.0/gems/chef-12.6.0/bin/chef-client:26:in `<top (required)>'
/opt/gitlab/embedded/bin/chef-client:23:in `load'
/opt/gitlab/embedded/bin/chef-client:23:in `<main>'
<...here the container terminates and my exec bash shell returns...>
Below the output from docker logs -f for the container. The log is very long (>12K lines), so I tried to look for lines containing useful info but am not sure I found them all:
Thank you for using GitLab Docker Image!
Current version: gitlab-ce=8.7.0-ce.0
Configure GitLab for your system by editing /etc/gitlab/gitlab.rb file
And restart this container to reload settings.
To do it use docker exec:
docker exec -it gitlab vim /etc/gitlab/gitlab.rb
docker restart gitlab
For a comprehensive list of configuration options please see the Omnibus GitLab readme
https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md
If this container fails to start due to permission problems try to fix it by executing:
docker exec -it gitlab update-permissions
docker restart gitlab
Preparing services...
Starting services...
Configuring GitLab package...
Configuring GitLab...
[2016-04-28T08:02:39+00:00] INFO: GET /organizations/chef/nodes/bcfc5b569532
[2016-04-28T08:02:39+00:00] INFO: #<ChefZero::RestErrorResponse: 404: Object not found: chefzero://localhost:8889/nodes/bcfc5b569532>
.
.
.
/opt/gitlab/embedded/bin/chef-client:23:in `load'
/opt/gitlab/embedded/bin/chef-client:23:in `<main>'
[2016-04-28T08:02:39+00:00] INFO:
--- RESPONSE (404) ---
{
"error": [
"Object not found: chefzero://localhost:8889/nodes/bcfc5b569532"
]
}
--- END RESPONSE ---
.
.
.
...a lot of logs (~12K lines), including some errors like the following one:
.
.
.
--- END RESPONSE ---
init (upstart 1.12.1)
[0m
================================================================================[0m
[31mError executing action `create` on resource 'link[/var/log/gitlab/gitlab-rails/sidekiq.log]'[0m
================================================================================[0m
[0mErrno::EPROTO[0m
-------------[0m
Protocol error # sys_fail2 - (/var/log/gitlab/sidekiq/current, /var/log/gitlab/gitlab-rails/sidekiq.log)[0m
.
.
.
================================================================================
Error executing action `create` on resource 'link[/var/log/gitlab/gitlab-rails/sidekiq.log]'
================================================================================
Errno::EPROTO
-------------
Protocol error # sys_fail2 - (/var/log/gitlab/sidekiq/current, /var/log/gitlab/gitlab-rails/sidekiq.log)
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-rails.rb
281: link legacy_sidekiq_log_file do
282: to File.join(node['gitlab']['sidekiq']['log_directory'], 'current')
283: not_if { File.exists?(legacy_sidekiq_log_file) }
284: end
285:
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-rails.rb:281:in `from_file'
link("/var/log/gitlab/gitlab-rails/sidekiq.log") do
action [:create]
retries 0
retry_delay 2
default_guard_interpreter :default
to "/var/log/gitlab/sidekiq/current"
link_type :symbolic
target_file "/var/log/gitlab/gitlab-rails/sidekiq.log"
declared_type :link
cookbook_name "gitlab"
recipe_name "gitlab-rails"
not_if { #code block }
end
<output ends>

Answer 1

My Gitlab container was crashing on run too, until I noticed that there was a rights issue (Gitlab not having rights on its own files because there were externally replaced, especially the config file gitlab.rb).
This fixed my problem:
docker exec -it my-gitlab-container update-permissions
docker exec -it my-gitlab-container gitlab-ctl reconfigure
docker restart my-gitlab-container

Answer 2

I'm not sure my issue is related to yours but in my case, I wanted to migrate the gitlab volumes to others directory because of space availability. There was a permission issue because I ran :
cp -R /my/old/gitlab /my/new/gitlab
insteda of :
cp -a /my/old/gitlab /my/new/gitlab
The -a preserve the attributes including permissions which were problematic for our container.
cheers

Answer 3

sudo chmod g+s /opt/gitlab/data/git-data/repositories/
Where /opt/gitlab/ is the linked docker share