Hi I have an docker image and I noticed after yesterday I cannot run it as it run the error that some repository url is invalid. And when I check I saw that it is true the debian repository structure has already been change https://packages.microsoft.com/debian/10/ on 16/6/2021. Does any one know the solution?
FROM python:3.7-buster
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
RUN curl https://packages.microsoft.com/config/debian/10/prod.list > /etc/apt/sources.list.d/mssql-release.list
RUN apt-get update && ACCEPT_EULA=Y apt-get install -y \
msodbcsql17 \
unixodbc-dev \
openssh-server \
nginx-full \
--no-install-recommends \
&& rm -rf /var/lib/apt/lists/*
error
It is a general issue, the issue is partially solved but the download speed is very slow. Here is a detailed report from bleepingcomputer blog post :Microsoft Linux repos suffer day-long outage, still recovering
Although Microsoft's initial ETA to resolve the issue was "two hours or so," the problem spanned well over 14 hours, with users continuing to experience degraded performance.
Microsoft engineer Rahul Bhandari stepped in on the same GitHub thread to confirm:
"Our infra team is working on this. There is an issue with some of the mirrors on packages.microsoft.com so as per them, the current ETA to resolve this issue is in next two hours or so," said Bhandari.
Bhandari later confirmed that some storage issues were the root cause of these problems.
Microsoft's principal engineering manager, Ravindra Bhartiya said:
"We had an incident with packages.microsoft.com that resulted in packages being unavailable."
"Our engineering team has mitigated the issue and our internal data shows improvement in the availability"
"If you still have problems, please provide us more information (output of "apt-get update|install") and we can investigate it further," said Bhartiya.
But even into today, at the time of writing, users are complaining about slow download speeds when retrieving packages from Microsoft's repo.
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 1 year ago.
Improve this question
Recently, my legacy Docker image stopped building because certain files refuse to download while building the image even though they download fine on my host system (and worked fine in the build before). This Dockerfile reproduces the problem:
FROM ubuntu:14.04
RUN apt-get update
RUN apt-get install -y ca-certificates
RUN update-ca-certificates
RUN apt-get update
RUN apt-get -y upgrade
#RUN apt-get install -y curl
#RUN curl -O https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/SphinxSearch/+archive/refs/heads/REL1_24.tar.gz
RUN apt-get install -y wget
RUN wget https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/SphinxSearch/+archive/refs/heads/REL1_24.tar.gz
Then, attempt to build the above Dockerfile with docker build .
When the wget approach (bottom) is used, I get the error:
ERROR: cannot verify gerrit.wikimedia.org's certificate, issued by '/C=US/O=Let\'s Encrypt/CN=R3':
Issued certificate has expired.
When I use the curl approach (top, commented out currently), I get the error:
curl: (60) SSL certificate problem: certificate has expired
I could bypass these issues by instructing wget and/or curl to ignore certificates, but I would prefer not to expose that security hole if at all possible to avoid. The top section is me flailing around trying to make sure the system's CA certificates are all up to date, but apparently what I'm doing isn't effective.
There are couple of ways I would do this without upgrading:
Before you try this make sure your ca-certificates.conf is in /etc/ca-certificates.conf.
In my Ubuntu 16.04 the ca-certificates.conf is in /etc/.
Add the next line BEFORE your "RUN update-ca-certificates" line.
For this to work, you MUST keep "RUN update-ca-certificates" line.
RUN sed '/DST_Root_CA_X3.crt/d' /etc/ca-certificates.conf > /tmp/cacerts.conf && mv /tmp/cacerts.conf /etc/ca-certificates.conf
RUN update-ca-certificates
This will remove the DST_Root_CA_X3.crt that expired on Sep 30 14:01:15 2021 GMT, assuming the expired DST Root CA certificate is the cause of your issue.
I would troubleshoot this manually as per this detailed guide: https://stackoverflow.com/a/69411107/1549092
If in your case step (1) doesn't work and that's not the issue, I would follow the guide in step (2) to identify the root cause of your issue. There could be another Root CA cert that has expired at different time.
NOTE: I can't be 100% sure what the root cause is in your case, unless you share that ca-cert bundle so I can test it. Or you can do it following step (2) above.
Meta: it's not clear this is a programming or development issue; you've already gotten close voters who think it isn't.
library/ubuntu:14.04 already contains ca-certificates 20170717~14.04.2 which is the last update issued for Trusty, so no, trying to update it doesn't help. That version DOES contain the ISRG root cert.
However, when accessing a host that uses the LetsEncrypt 'compatibility' chain from software based on OpenSSL, as both curl and wget in Ubuntu14.04 are, you not only need the ISRG root in the truststore but you also need either a recent version of OpenSSL code (at least 1.1.x and I believe specifically 1.1.1(I was wrong about the latter)) OR you need the now-obsolete DST root removed.
You could download a near-current OpenSSL (3.0.0 was just released and you don't want to mess with that) and build it yourself, then download curl and/or wget and build it/them to use that new OpenSSL. That's a good deal of work.
https://serverfault.com/questions/1079199/client-on-debian-9-erroneously-reports-expired-certificate-for-letsencrypt-issue has ways to remove DST root for Debian, which also applies to Ubuntu, and the sed method works for me in a docker build.
Alternatively, if you only need one file that doesn't change (and I'm guessing REL with a number shouldn't), why not just download on the host (where you apparently have modern code running) and copy into the container (or mount, if you care about the space)?
I hope you're all well!
This is my first question ever on stack overflow, so if i'm missing anything, just tell me and it'll be my pleasure to update my question.
So, to summarize my problem, I am trying to install Docker-ce on an Ubuntu 18.04 VM by following the official steps at Docker-Engine Installation and I am getting the error "Hash Sum Mismatch" when running the command "sudo apt-get install docker-ce docker-ce-cli containerd.io".
I went through the official steps a few times and I've tried the following things so far :
1) The following commands, followed by the official installation steps
apt-get clean
rm -rf /var/lib/apt/lists/*
apt-get clean
apt-get update
apt-get upgrade
2) Creation of a file with the following lines, placed at "/etc/apt/apt.conf.d/99fixbadproxy", followed by the official installation steps
Acquire::http::Pipeline-Depth 0;
Acquire::http::No-Cache true;
Acquire::BrokenProxy true;
3) Installation of other versions of docker-ce with the following commands
sudo apt-get install docker-ce 5:19.03.8~3-0~ubuntu-bionic
sudo apt-get install docker-ce 5:19.03.6~3-0~ubuntu-bionic
sudo apt-get install docker-ce 5:18.09.9~3-0~ubuntu-bionic
4) I tried installing different repositories (i think they're called repositories) by adding "nightly" or "test" or both after the word "stable" of the following command
sudo add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
And I think that is pretty much it. I did tried other things but unfortunately i don't remember them because I've been working on this issue for quite some time now.
I did not try on a fresh new installation of Ubuntu 18.04 though.
If I'm missing anything in my question, please let me know!
Take care everybody!
EXPLANATION and Solution: Quick Fix
This issue is caused by the Windows Hypervisor Platform. This issue cannot be resolved for now (asfar as I know).
A partial fix is at hand though. And I say"partial" because it involves disabling the platform (also known as"Hyper-V") which will probably break other virtualization solutions you have installed since this is enabled manually. Anyway, here's how to disable it and get your VM running again,
Shut down the Virtual Machine.
Press Windows logo key + X, then hit A to run Command Prompt(powershell) as administrator.
Type
bcdedit /set hypervisorlaunchtype off
When you see"The operation completed succesfully", reboot your windows. After reboot, boot your VM and update/upgrade.
For anyone still having issues with this, I managed to get it working by disabling HyperV (as most people have already suggested). However, on my setup, disabling HyperV using the mentioned approaches didnt actually disable HyperV since my windows machine used HyperV for both CredentialGuard and Secure Boot, both of which needed to be disabled before HyperV was actually fully disabled
If running systeminfo.exe in an elevated terminal shows A hypervisor has been detected. Features required for Hyper-V will not be displayed, then HyperV is not fully disabled. Disabling credential guard and secure boot coupled with a few restarts (after running all the commands to disable HyperV that have already been mentioned) solved all issues for me.
Just for reference, if HyperV is fully disabled, you should see
Hyper-V Requirements: VM Monitor Mode Extensions: Yes
Virtualization Enabled In Firmware: Yes
Second Level Address Translation: Yes
Data Execution Prevention Available: Yes
when running systeminfo.exe
I have my ASP.NET Core app running beautifully (more or less) on microsoft/aspnetcore:2.0-jessie. Now I want to try to get it to deploy to amazonlinux:2.
So far, the biggest hurdle has been libicu. I tried setting Globalization to Invariant, but this caused weird failures in, e.g., mySQL database calls.
Here's the relevant step from my Dockerfile:
RUN curl -L --http1.1 http://download.icu-project.org/files/icu4c/57.1/icu4c-57_1-RHEL6-x64.tgz --output icu.tgz \
&& tar -xf icu.tgz -C / \
&& export LD_LIBRARY_PATH=/usr/local/lib \
&& rm icu.tgz
(SourceForge was down while I was trying to work on this yesterday, which didn't improve matters.)
In any case, I still get the message of doom from .NET Core:
FailFast: Couldn't find a valid ICU package installed on the system. Set the configuration flag System.Globalization.Invariant to true if you want to run with no globalization support.
Any suggestions how to proceed?
Well, I revisited this yesterday. I don't know if it's because the base .tar of the Amazon Linux image has been updated, or because I was doing something wrong last time, but I installed the following packages using yum and all was well:
libunwind
libicu
dotnet-hosting-2.0.5
Note that for the dotnet package I needed first to set up Microsoft's package repository for yum, i.e.
rpm --import https://packages.microsoft.com/keys/microsoft.asc
and copying the following file to /etc/yum.repos.d/dotnetdev.repo :
[packages-microsoft-com-prod]
name=packages-microsoft-com-prod
baseurl=https://packages.microsoft.com/yumrepos/microsoft-rhel7.3-prod
enabled=1
gpgcheck=1
gpgkey=https://packages.microsoft.com/keys/microsoft.asc
(see Microsoft's instructions for CentOS and other Linux distros)
I've created a Dockerfile for an application I'm building that has a lot of large apt-get package dependencies. It looks something like this:
FROM ubuntu:15.10
RUN apt-get update && apt-get install -y \
lots-of-big-packages
RUN install_my_code.sh
As I develop my application, I keep coming up with unanticipated package dependencies. However, since all the packages are in one Dockerfile instruction, even adding one more breaks the cache and requires the whole lot to be downloaded and installed, which takes forever. I'm wondering if there's a better way to structure my Dockerfile?
One thought would be to put a separate RUN apt-get update && apt-get install -y command for each package, but running apt-get update lots of times probably eats up any savings.
The simplest solution would be to just add a second RUN apt-get update && apt-get install -y right after the first as a catchall for all of the unanticipated packages, but that divides the packages in an unintuitive way. (ie, "when I realized I needed it") I suppose I could combine them when dependencies are more stable, but I find I'm always overly optimistic about when that is.
Anyway, if anyone has a better way to structure it I'd love to hear it. (all of my other ideas run against the Docker principles of reproducibility)
I think you need to run apt-get update only once within the Dockerfile, typically before any other apt-get commands.
You could just first have the large list of known programs to install, and if you come up with a new one then just add a new RUN apt-get install -y abc to you Dockerfile and let docker continue form the previously cached command. Periodically (once a week, one a month?) you could re-organize them as you see fit or just run everything in a single command.
I suppose I could combine them when dependencies are more stable, but
I find I'm always overly optimistic about when that is.
Oh you actually mentioned this solution already, anyway there is no harm doing these "tweaks" every now and then. Just run apt-get update only once.
It would seem that apt-get is having issues connecting with the repository servers. I suppose it is likely compatibility issues, as mentioned here, however the proposed solution of apt-get clean does not work for me. Also I am surprised, if this is the case, that there are not more people having my issue.
MWE
Dockerfile
FROM debian:jessie
RUN apt-get clean && apt-get update && apt-get install -y --no-install-recommends \
git
$ docker build .
docker build .
Sending build context to Docker daemon 2.048 kB
Step 0 : FROM debian:jessie
---> 4a5e6db8c069
Step 1 : RUN apt-get clean && apt-get update && apt-get install -y --no-install-recommends git
---> Running in 43b93e93feab
Get:1 http://security.debian.org jessie/updates InRelease [63.1 kB]
... some omitted ...
Get:6 http://httpredir.debian.org jessie-updates/main amd64 Packages [3614 B]
Fetched 9552 kB in 7s (1346 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following extra packages will be installed:
... some omitted ...
0 upgraded, 26 newly installed, 0 to remove and 0 not upgraded.
Need to get 13.2 MB of archives.
After this operation, 64.0 MB of additional disk space will be used.
Get:1 http://security.debian.org/ jessie/updates/main libgnutls-deb0-28 amd64 3.3.8-6+deb8u2 [694 kB]
... some omitted ...
Get:5 http://httpredir.debian.org/debian/ jessie/main libnettle4 amd64 2.7.1-5 [176 kB]
Err http://httpredir.debian.org/debian/ jessie/main libffi6 amd64 3.1-2+b2
Error reading from server. Remote end closed connection [IP: 176.9.184.93 80]
... some omitted ...
Get:25 http://httpredir.debian.org/debian/ jessie/main git amd64 1:2.1.4-2.1 [3624 kB]
Fetched 13.2 MB in 10s (1307 kB/s)
E: Failed to fetch http://httpredir.debian.org/debian/pool/main/libf/libffi/libffi6_3.1-2+b2_amd64.deb Error reading from server. Remote end closed connection [IP: 176.9.184.93 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
The command '/bin/sh -c apt-get clean && apt-get update && apt-get install -y --no-install-recommends git' returned a non-zero code: 100
Please note that I also posted here with a different issue. I believe it to be unrelated, but it may well actually be.
For whoever is having an issue with this, this is my attempt to "fix" the issue by swapping out httpredir with a single working domain whenever the Dockerfile is being built:
FROM debian:je...
# Insert this line before "RUN apt-get update" to dynamically
# replace httpredir.debian.org with a single working domain
# in attempt to "prevent" the "Error reading from server" error.
RUN sed -i "s/httpredir.debian.org/`curl -s -D - http://httpredir.debian.org/demo/debian/ | awk '/^Link:/ { print $2 }' | sed -e 's#<http://\(.*\)/debian/>;#\1#g'`/" /etc/apt/sources.list
# Continue with your apt-get update...
RUN apt-get update...
What this command does is:
Curl the http://httpredir.debian.org/demo/debian/ from the building machine to get the headers from debian demo page (-s is silent, don't output. -D is to dump headers)
Extract the headers, find the Link header fragment. This contains the best route as recommended by httpredir.
The last sed -e ... is to extract out the domain name of the link in step 2.
Then finally, the domain found in step 3 is being feed into the global sed command, and replace the domain httpredir.debian.org found in /etc/apt/sources.list.
This is not a fix, but rather a simple hack to (greatly) reduce the chances of failed build. And... pardon me if it looks weird, as it's my virgin sed & piping attempt.
Edit
On a side note, if the domain that it picks simply too slow or not responding as it should, you may want to do it manually by
Visit http://httpredir.debian.org/demo.html, and you should see a link there like http://......./debian/. For example, at the point of writing, I saw http://mirrors.tuna.tsinghua.edu.cn/debian/
Instead of the long RUN sed -i.... command, use this instead:
RUN sed -i "s/httpredir.debian.org/mirrors.tuna.tsinghua.edu.cn/" /etc/apt/sources.list
I added apt-get clean to my dockerfile before the apt-get update line, it seems to have done the trick.
I guess I have no way of knowing whether or not it was the extra command or if it was luck that fixed my build, but I took the advice from https://github.com/CGAL/cgal-testsuite-dockerfiles/issues/19
The httpredir.debian.org mirror is "magic" in that it will load-balance and geo-ip you to transparent increase performance and availability. I would therefore immediately suspect it of causing your problem, or at least be the first thing to rule out.
I would check if you could:
Still reproduce the problem; httpredir.debian.org will throw out "bad" mirrors from its internal lists so your issue may have been temporary.
Reproduce the problem with a different, non-httpredir.debian.org mirror. Try something like ftp.de.debian.org. If it then works with this mirror, do please contact the httpredir.debian.org maintainer and report the issue to them. They are quite responsive and open to bug reports.
For those visiting with similar issues, using the --no-cache flag in the docker build may help. Similar issues (though not this exact one) can occur if the apt-get update is old and not being recalled sue to caching.
Not enough reputation to comment on previous answers, so I will (confusingly) add a new answer:
I don't think hardcoding a single mirror is really a viable solution, since as for example seen here, there's a reason debian implemented the whole httpredir thing -- mirrors go down or out of date.
I've dealt with this issue a lot of times, and the logs always indicate that docker's actually running the apt-get command, which means --no-cache is unlikely to be fixing it -- it's just that if you rebuild, httpredir is likely to pick a different mirror, even if you don't change anything in your docker file, and the build will work.
I was able to solve this problem by adding -o 'Acquire::Retries=3' to the apt-get install command.
RUN apt-get update && apt-get install -o 'Acquire::Retries=3' -y git
Seems that it automatically retries to acquire the package from another mirror.
apt-get acquire documentation
EDIT: After some investigation I found that my problem was related with the proxy server I was using. I'll leave the answer here anyway in case it helps anyone.