Consider a small Go application that reads a large JSON file 2GB , marshals the JSON data into a struct, and POSTs the JSON data to a web service endpoint.

The web service receiving the payload changed its functionality, and now has a limit of 25MB per payload. What would be the best approach to overcome this issue using Go? I've thought of the following, however I'm not sure it is the best approach:

1. Creating a function to split the large JSON file into multiple smaller ones (up to 20MB), and then iterate over the files sending multiple smaller requests.

Similar function to the one being used to currently send the entire JSON payload:

func sendDataToService(data StructData) {
    payload, err := json.Marshal(data)
    if err != nil {
        log.Println("ERROR:", err)
    }

    request, err := http.NewRequest("POST", endpoint, bytes.NewBuffer(payload))
    if err != nil {
        log.Println("ERROR:", err)
    }

    client := &http.Client{}
    response, err := client.Do(request)
    log.Println("INFORMATIONAL:", request)
    if err != nil {
        log.Println("ERROR:", err)
    }
    defer response.Body.Close()
}

CodePudding user response：

As the server-side can process data progressively, I assume that the large JSON object can be split into smaller pieces. From this point, I can propose several options.

1. Use HTTP requests

   Pros: Pretty simple to implement on the client-side.

   Cons: Making hundreds of HTTP requests might be slow. You will also need to handle timeouts - this is additional complexity.

2. Use WebSocket messages

   If the receiving side supports WebSockets, a step-by-step flow will look like this:

   • Split the input data into smaller pieces.
   • Connect to the WebSocket server.
   • Start sending messages with the smaller pieces till the end of the file.
   • Close connection to the server.

   This solution might be more performant as you won't need to connect and disconnect from the server each time you send a message, as you'd do with HTTP.

However, both solutions suppose that you need to assemble all pieces on the server-side. For example, you would probably need to send along with the data a correlation ID to let the server know what file you are sending right now and a specific end-of-file message to let the server know when the file ends. In the case of the WebSocket server, you could assume that the entire file is sent during a single connection session if it is relevant.

CodePudding user response：

You can break the input into chunks and send each piece individually:

dec := json.NewDecoder(inputStream)
tok, err := dec.Token()
if err != nil {
    return err
}
if tok == json.Delim('[') {
    for {
        var obj json.RawMessage
        if err := dec.Decode(&obj); err != nil {
            return err
        }
        // Here, obj contains one element of the array. You can send this
        // to the server.
        if !dec.More() {
            break
        }
    }
}