httpclient

httpclient

A simple, "anti-detect" HTTP client for Go. It seamlessly combines TLS fingerprint evasion, dynamic user-agent rotation, and a highly intelligent retry mechanism that automatically handles request body buffering.

Built on top of tls-client, it is designed for scraping, automation, and interacting with strict APIs (like Cloudflare-protected endpoints) without getting blocked.

✨ Features

• 🛡️ TLS Evasion: Automatically fetches and impersonates the latest Google Chrome TLS fingerprint.
• 🎭 Dynamic User-Agents: Generates and injects modern, realistic User-Agent headers (with a 7-day disk cache).
• 🔄 Smart Retries with Exponential Backoff: Automatically retries requests on timeouts, network errors, or specific HTTP status codes (e.g., 429, 500, 502).
• 🧠 Auto Body-Replay (Magic Buffering): Automatically buffers non-seekable POST/PUT bodies (up to 4MB by default) to make them replayable during retries. Safely detects huge streams (e.g., 4GB files) via Content-Length or lazy-reading to prevent OOM panics.
• 🌐 Network Control: Force IPv4, IPv6, or let it auto-resolve.
• 🔌 Proxy Support: Ready-to-use HTTP and SOCKS5 proxy integration.

📦 Installation

go get github.com/imbecility/httpclient

🚀 Quick Start

Here is a basic example of how to initialize the client and make a request.

package main

import (
	"context"
	"fmt"
	"io"
	"net/http"
	"time"

	"github.com/imbecility/httpclient"
)

func main() {
	// 1. Initialize the client
	client, err := httpclient.NewHttpClient(httpclient.ClientSettings{
		MaxTimeoutSeconds: 30,
		TargetURL:         "https://yandex.ru/pogoda/ru",
		StrictSSL:         true,
		NetMode:           httpclient.NetworkModeIPv4,
		// ProxyURI:       "socks5://127.0.0.1:2080", // Optional proxy
		Retry:             httpclient.DefaultRetryConfig,
	})
	if err != nil {
		panic(err)
	}

	// 2. Create a request
	ctx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
	defer cancel()

	req, err := http.NewRequestWithContext(ctx, http.MethodGet, "https://yandex.ru/pogoda/ru", nil)
	if err != nil {
		panic(err)
	}

	// 3. Execute (TLS evasion, UA injection, and retries happen automatically)
	fmt.Println("Sending request...")
	resp, err := client.Do(req)
	if err != nil {
		panic(fmt.Sprintf("Request failed: %v", err))
	}
	defer resp.Body.Close()

	body, _ := io.ReadAll(resp.Body)
	fmt.Printf("Status: %s\n", resp.Status)
	fmt.Printf("Downloaded %d bytes\n", len(body))
}

🧠 Smart Body Replay (How it works)

In standard Go, if an HTTP request with an io.Reader body fails, retrying it is impossible because the stream has already been consumed. You normally have to provide a req.GetBody factory manually.

httpclient brings "syntactic sugar" to this problem. It automatically makes your bodies replayable:

1. Fast Path: If you provide a small JSON/String via strings.NewReader or bytes.Reader, Go handles it.
2. Smart Buffering: If you pass a raw stream (e.g., an API response or a pipe), the client will lazily read up to MaxBodyBuffer (default: 4 MiB) into memory. If the request fails, it can safely retry it.
3. OOM Protection: If you upload a 4GB file, the client checks the Content-Length. It realizes the file is too large for memory, skips buffering, reconstructs the stream seamlessly, and attempts the upload exactly once without crashing your app.

Customizing Retry & Buffer limits

You can tweak the retry logic via RetryConfig:

config := httpclient.ClientSettings{
	Retry: httpclient.RetryConfig{
		MaxAttempts:   5,
		BaseDelay:     500 * time.Millisecond,
		MaxDelay:      10 * time.Second,
		RetryOnCodes:  []int{429, 500, 502, 503, 504},
		MaxBodyBuffer: 10 * 1024 * 1024, // Buffer up to 10 MB for retries
	},
}

• Set MaxAttempts: 0 to disable retries.
• Set MaxBodyBuffer: -1 to disable body buffering entirely.

⚙️ Configuration Options (ClientSettings)