Cloud Function robots.txt 404 page not found issue

I have deployed one simple app via go language in Cloud Function.

*A robots.txt file is also included when distributing the app.

enter image description here

In this regard, a simple app normally shows the image below. enter image description here

But it shows 404 page not found even though the robots.txt file is normal. enter image description here

Does anyone know if the robots.txt file itself can't be served by Cloud Function?

##Function.go

// Package p contains an HTTP Cloud Function.
package p

import (
    "encoding/json"
    "fmt"
    "html"
    "io"
    "log"
    "net/http"
)

// HelloWorld prints the JSON encoded "message" field in the body
// of the request or "Hello, World!" if there isn't one.
func HelloWorld(w http.ResponseWriter, r *http.Request) {
    var d struct {
        Message string `json:"message"`
    }

    if err := json.NewDecoder(r.Body).Decode(&d); err != nil {
        switch err {
        case io.EOF:
            fmt.Fprint(w, "Hello World!")
            return
        default:
            log.Printf("json.NewDecoder: %v", err)
            http.Error(w, http.StatusText(http.StatusBadRequest), http.StatusBadRequest)
            return
        }
    }

    if d.Message == "" {
        fmt.Fprint(w, "Hello World!")
        return
    }
    fmt.Fprint(w, html.EscapeString(d.Message))
}

##go.mod

module example.com/cloudfunction

##robots.txt

User-agent: *
Disallow: /

User-agent: Mediapartners-Google
Allow: /

Thank you in advance to those who have responded.

Comments


  • Outlier

    Cloud Functions is not a web server.

    You cannot just add files and expect those will be served as you would you do it with an NGINX or Apache web server.

    When you reach your CF endpoint the code you added will be executed and you'll get a result from that and in any case the files you add will be used by the code but not served.

    I'd suggest to first understand what Cloud Functions is intended for and as an alternative you may want to go with App Engine Standard.

    Another way to go is to use a workaround to handle the routes and that all stuff as guillaume shows in this article for Python + Flask:

    from flask import Flask, request
    
    #Define an internal Flask app
    app = Flask("internal")
    
    #Define the internal path, idiomatic Flask definition
    @app.route('/user/<string:id>', methods=['GET', 'POST'])
    def users(id):
        
        print(id)
        return id, 200
    
    #Comply with Cloud Functions code structure for entry point
    def my_function(request):
        
        #Create a new app context for the internal app
        internal_ctx = app.test_request_context(path=request.full_path,
                                                method=request.method)
        
        #Copy main request data from original request
        #According to your context, parts can be missing. Adapt here!
        internal_ctx.request.data = request.data
        internal_ctx.request.headers = request.headers
        
        #Activate the context
        internal_ctx.push()
        
        #Dispatch the request to the internal app and get the result 
        return_value = app.full_dispatch_request()
        
        #Offload the context
        internal_ctx.pop()
        
        #Return the result of the internal app routing and processing      
        return return_value
    

    And it is important what he mentions at the end:

    However, keep in mind that’s a workaround, even a hack, and Cloud Functions aren’t designed for this purpose.

    Here's a sample for Go:

    package function
    
    import (
        "net/http"
    )
    
    var mux = newMux()
    
    //F represents cloud function entry point
    func F(w http.ResponseWriter, r *http.Request) {
        mux.ServeHTTP(w, r)
    }
    
    func newMux() *http.ServeMux {
        mux := http.NewServeMux()
        mux.HandleFunc("/one", one)
        mux.HandleFunc("/two", two)
        mux.HandleFunc("/subroute/three", three)
    
        return mux
    }
    
    func one(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("hello from one"))
    }
    
    func two(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("hello from two"))
    }
    
    func three(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("hello from three"))
    }
    
  • Dirge

    Cloud Functions isn't a webserver and you can't serve file directly like that. You need to process them in Go.

    For that you need to know the code structure of the functions. All the original files are stored in /workspace/serverless_function_source_code directory. So, you can simply serve them using the URL path like that

    
    var functionSourceCodeDir = "/workspace/serverless_function_source_code"
    
    func ServeFile(w http.ResponseWriter, r *http.Request) {
        file := r.URL.Path
        if file == "/" {
            w.WriteHeader(http.StatusBadRequest)
            fmt.Fprintf(w, "you must provide a file pathname")
            return
        }
        http.ServeFile(w, r, functionSourceCodeDir+file)
    }
    
    

Add Comment