Is there a way to expand shortened URLs in Go [duplicate] - url

So, I'm using the net/http package. I'm GETting a URL that I know for certain is redirecting. It may even redirect a couple of times before landing on the final URL. Redirection is handled automatically behind the scenes.
Is there an easy way to figure out what the final URL was without a hackish workaround that involves setting the CheckRedirect field on a http.Client object?
I guess I should mention that I think I came up with a workaround, but it's kind of hackish, as it involves using a global variable and setting the CheckRedirect field on a custom http.Client.
There's got to be a cleaner way to do it. I'm hoping for something like this:
package main
import (
"fmt"
"log"
"net/http"
)
func main() {
// Try to GET some URL that redirects. Could be 5 or 6 unseen redirections here.
resp, err := http.Get("http://some-server.com/a/url/that/redirects.html")
if err != nil {
log.Fatalf("http.Get => %v", err.Error())
}
// Find out what URL we ended up at
finalURL := magicFunctionThatTellsMeTheFinalURL(resp)
fmt.Printf("The URL you ended up at is: %v", finalURL)
}

package main
import (
"fmt"
"log"
"net/http"
)
func main() {
resp, err := http.Get("http://stackoverflow.com/q/16784419/727643")
if err != nil {
log.Fatalf("http.Get => %v", err.Error())
}
// Your magic function. The Request in the Response is the last URL the
// client tried to access.
finalURL := resp.Request.URL.String()
fmt.Printf("The URL you ended up at is: %v\n", finalURL)
}
Output:
The URL you ended up at is: http://stackoverflow.com/questions/16784419/in-golang-how-to-determine-the-final-url-after-a-series-of-redirects

I would add a note that http.Head method should be enough to retrieve the final URL. Theoretically it should be faster comparing to http.Get as a server is expected to send back just a header:
resp, err := http.Head("http://stackoverflow.com/q/16784419/727643")
...
finalURL := resp.Request.URL.String()
...

Related

How to extract list of docker images inside GCP artifact registry

I want to list all the repositories inside GCP artifact registry in golang.
Current code : (https://pkg.go.dev/cloud.google.com/go/artifactregistry/apiv1beta2)
c, err := artifactregistry.NewClient(ctx, option.WithCredentialsFile("<service account json>"))
if err != nil {
// no error here
}
defer c.Close()
req := &artifactregistrypb.ListRepositoriesRequest{
Parent: "<project-id>",
}
it := c.ListRepositories(ctx, req)
for {
resp, err := it.Next()
if err == nil {
fmt.Println("resp", resp)
} else {
fmt.Println("err ==>", err)
}
}
The error prints: Invalid field value in the request. OR sometimes I get Request contains an invalid argument
What am I doing wrong here ? and What does the "Parent" mean ? (in ListRepositoriesRequest)
On further digging, I found that the value passed in the Parent goes to : "x-goog-request-params", what should be the correct format for this ?
Sometime the libraries/api are well documented, sometime not...
Here the REST API that you can test in the API explorer (right hand side bar). After some tests, the parent must have that format
projects/<PROJECT_ID>/locations/<REGION>
Try with that to solve your issue

http.NewRequest in golang converting some charterers to % values

go version go1.8.1 windows/amd64
"net/http" package used to build http request.
req, err := http.NewRequest("GET",`http://domain/_api/Web/GetFolderByServerRelativeUrl('` + root_folder_url + `')?$expand=Folders,Files`, nil)
Here if I print url it shows
http://domain/_api/Web/GetFolderByServerRelativeUrl%28%27rooturl%27%29?$expand=Folders,Files
Not understanding why url parser is replacing ' to %27 here.
Whereas I need ' to be sent as is while requesting.
The http.NewRequest function calls url.Parse to set the Request.URL. The URL.RequestURI method is called to get the request URI written to the network.
An application can override any transformation made by Parse/RequestURI by setting the request URL Opaque field:
req, err := http.NewRequest("GET", "http://domain/", nil)
if err != nil {
// handle error
}
req.URL.Opaque = `/_api/Web/GetFolderByServerRelativeUrl('` + root_folder_url + `')?$expand=Folders,Files`
In this snippet, the argument to NewRequest specifies the protocol and host for the request. The opaque value specifies the request URI written to the network. The request URI does not include host or protocol.

Go url.Parse(string) fails with certain user names or passwords

Using a URL that has worked in the past, I know receive a parsing error from net/url. What's wrong with it?
parse postgres://user:abc{DEf1=ghi#example.com:5432/db?sslmode=require: net/url: invalid userinfo
Sample application
See https://play.golang.com/p/mQZaN5JN3_q to run.
package main
import (
"fmt"
"net/url"
)
func main() {
dsn := "postgres://user:abc{DEf1=ghi#example.com:5432/db?sslmode=require"
u, err := url.Parse(dsn)
fmt.Println(u, err)
}
Well, you can just
url.QueryEscape("your#$%^&*(proper$#$%%^(password")
and use this one to parse your url.
It turns out up until Go v1.9.3 net/url didn't validate the user info when parsing a url. This may break existing applications when compiled using v1.9.4 if the username or password contain special characters.
It now expects the user info to be percent encoded string in order to handle special characters. The new behaviour got introduced in ba1018b.
Fixed sample application
package main
import (
"fmt"
"net/url"
)
func main() {
dsn1 := "postgres://user:abc{DEf1=ghi#example.com:5432/db?sslmode=require" // this works up until 1.9.3 but no longer in 1.9.4
dsn2 := "postgres://user:abc%7BDEf1=ghi#example.com:5432/db?sslmode=require" // this works everywhere, note { is now %7B
u, err := url.Parse(dsn1)
fmt.Println("1st url:\t", u, err)
u, err = url.Parse(dsn2)
fmt.Println("2nd url:\t", u, err)
}
Run the code on https://play.golang.com/p/jGIQgbiKZwz.
Use url.UserPassword func :
package main
import (
"fmt"
"net/url"
)
func main() {
dsn := "postgres://example.com:5432/db?sslmode=require"
u, err := url.Parse(dsn)
if err != nil {
fmt.Printf("ERROR: %v\n", err)
return
}
u.User = url.UserPassword("user", "abc{DEf1=ghi")
fmt.Println("url:\t", u)
}

How to wrap exec.Command inside an io.Writer

I'm trying to compress a JPEG image in go using mozjpeg. Since it doesn't have official go binding, I think I'll just invoke its CLI to do the compression.
I try to model the usage after compress/gzip:
c := jpeg.NewCompresser(destFile)
_, err := io.Copy(c, srcFile)
Now the question is, how do I wrap the CLI inside Compresser so it can support this usage?
I tried something like this:
type Compresser struct {
cmd exec.Command
}
func NewCompressor(w io.Writer) *Compresser {
cmd := exec.Command("jpegtran", "-copy", "none")
cmd.Stdout = w
c := &Compresser{cmd}
return c
}
func (c *Compresser) Write(p []byte) (n int, err error) {
if c.cmd.Process == nil {
err = c.cmd.Start()
if err != nil {
return
}
}
// How do I write p into c.cmd.Stdin?
}
But couldn't finish it.
Also, a second question is, when do I shut down the command? How to shut down the command?
You should take a look at the Cmd.StdinPipe. There is an example in the documentation, which suits your case:
package main
import (
"fmt"
"io"
"log"
"os/exec"
)
func main() {
cmd := exec.Command("cat")
stdin, err := cmd.StdinPipe()
if err != nil {
log.Fatal(err)
}
go func() {
defer stdin.Close()
io.WriteString(stdin, "values written to stdin are passed to cmd's standard input")
}()
out, err := cmd.CombinedOutput()
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s\n", out)
}
In this case, CombinedOutput() executes your command, and the execution is finished, when there are no more bytes to read from out.
As per Kiril's answer, use the cmd.StdInPipe to pass on the data you receive to Write.
However, in terms of closing, I'd be tempted to implement io.Closer. This would make *Compresser automatically implement the io.WriteCloser interface.
I would use Close() as the notification that there is no more data to be sent and that the command should be terminated. Any non-zero exit code returned from the command that indicates failure could be caught and returned as an error.
I would be wary of using CombinedOutput() inside Write() in case you have a slow input stream. The utility could finish processing the input stream and be waiting for more data. This would be incorrectly detected as command completion and would result in an invalid output.
Remember, the Write method can be called an indeterminate number of times during IO operations.

Golang HTTPS/TLS POST client/server

I have written a simple client/server in Go that will do an HTTP GET over TLS, but I'm trying to also make it capable of doing an HTTP POST over TLS.
In the example below index.html just contains the text hello, and the HTTP GET is working fine. I want the client to get the HTTP GET and write back, hello world to the server.
client
package main
import (
"crypto/tls"
"fmt"
"io/ioutil"
"net/http"
"strings"
)
func main() {
link := "https://10.0.0.1/static/index.html"
tr := &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}
client := &http.Client{Transport: tr}
response, err := client.Get(link)
if err != nil {
fmt.Println(err)
}
defer response.Body.Close()
content, _ := ioutil.ReadAll(response.Body)
s := strings.TrimSpace(string(content))
fmt.Println(s)
// out := s + " world"
// Not working POST...
// resp, err := client.Post(link, "text/plain", &out)
}
server
package main
import (
"fmt"
"log"
"net/http"
)
func main() {
http.HandleFunc("/static/", func (w http.ResponseWriter, r *http.Request) {
fmt.Println("Got connection!")
http.ServeFile(w, r, r.URL.Path[1:])
})
log.Fatal(http.ListenAndServeTLS(":443", "server.crt", "server.key", nil))
}
I also currently have nothing to handle the POST on the server side, but I just want it to print it out to the screen so when I run the client I will see the server print hello world.
How should I fix my client code to do a proper POST? And what should the corresponding server code look like to accept the POST? Any help would be appreciated, I'm having trouble finding HTTPS/TLS POST examples.
You didn't share the error message, but I assume the client.Post call wasn't allowing a string as its third parameter, because it requires an io.Reader. Try this instead:
out := s + " world"
resp, err := client.Post(link, "text/plain", bytes.NewBufferString(out))
On the server side, you already have the right code set up to handle the POST request. Just check the method:
http.HandleFunc("/static/", func (w http.ResponseWriter, r *http.Request) {
if r.Method == "POST" {
// handle POST requests
} else {
// handle all other requests
}
})
I noticed one other issue. Using index.html probably won't work here. http.ServeFile will redirect that path. See https://golang.org/pkg/net/http/#ServeFile:
As a special case, ServeFile redirects any request where r.URL.Path
ends in "/index.html" to the same path, without the final
"index.html". To avoid such redirects either modify the path or use
ServeContent.
I'd suggest just using a different file name to avoid that issue.

Resources