Distributed Residential Edge Workers: Zero-Proxy Anti-Bot Scraping Mesh

Connect local PCs, servers, and Android smartphones (via Termux) to the Apps Manager edge mesh. Jobs are executed on genuine residential IPs over outbound WebSockets, bypassing datacenter IP bans, Cloudflare, and Akamai without costly proxy subscriptions.

Jump to:

Overview & Architecture

Cloud datacenter IP addresses (AWS, Hetzner, Render, DigitalOcean) are permanently flagged and blocked by anti-bot systems like Cloudflare, PerimeterX, and Akamai. Commercial residential proxy networks charge expensive bandwidth fees ($10–$25 per GB). Distributed Edge Workers solve this by turning your everyday devices into an autonomous, encrypted scraping cluster.

Core Capabilities

Outbound WebSocket Tunneling: Zero firewall configuration, router port forwarding, or public IPs needed.

Genuine Residential IP Routing: Target websites see real residential browser fingerprints, bypassing Cloudflare and WAF screens.

Android Phone Support: Run continuous scraping nodes 24/7 on spare Android smartphones using Termux.

Chromium Auto-Provisioning: Automatically detects or installs the appropriate Chromium binary for your host OS.

Single-Flight Deduplication: Eliminates redundant concurrent page loads across your worker cluster.

Automated Server Keep-Alive: Sends anti-sleep pings to keep cloud instances warm and prevent cold-start delays.

Detailed Implementation Guides

1How Outbound WebSocket Tunneling Works

Traditional proxies require inbound open ports, public static IP addresses, or dynamic DNS. The app-manager-edge-worker package reverses this model.

Persistent Outbound Connection: The worker dials out to wss://apps-manager.in/workers using your Worker Key.

NAT & Firewall Traversal: Works effortlessly behind home Wi-Fi routers, corporate firewalls, and carrier-grade NAT (CGNAT).

Job Dispatch & Result Streaming: When an API request arrives, the central gateway routes the job to an idle worker node, executes the scrape locally, and streams back the JSON result.

Zero Port Forwarding

You do not need to open any router ports or configure dynamic DNS. If your device can browse the web, it can run as an Edge Worker node.

2Turn Any Android Phone into a 24/7 Scraping Node (Termux)

Older Android smartphones consume only 2W–4W of power and have native Wi-Fi/4G/5G residential IP connections. You can turn any Android phone into a 24/7 scraping node using Termux.

Download Termux from F-Droid (do not use outdated Google Play Store build).

Install Node.js LTS, Chromium, and x11-repo packages.

Acquire Android wake-lock to prevent CPU sleep when screen locks.

Set Termux Battery settings to "Unrestricted" in Android Settings.

Why x11-repo is Mandatory on Termux

On Android ARM64, the native Chromium browser package is hosted inside Termux x11-repo. You must run `pkg install -y x11-repo` before installing `chromium` to prevent package not found errors.

CLI Flags & Options

Flag	Type	Default	Description
--key <WORKER_KEY>	string	—	Worker authentication key from dashboard (or set APP_MANAGER_WORKER_KEY env var)
--server <URL>	string	https://apps-manager.onrender.com (use --server https://apps-manager.in)	Central Apps Manager server gateway URL
--concurrency <NUM>	number	3	Maximum concurrent browser tabs (use 1 or 2 on low-RAM phones)
--headless <BOOL>	boolean	true	Run Chromium in headless mode without graphical window
--anti-sleep <MINUTES>	number	7	Interval in minutes to ping central server and prevent Render spindown
--chrome-path <PATH>	string	—	Explicit path to local Chrome or Chromium executable

Step-by-Step Quickstart

1Option 1: Quick Run via NPX (Zero-Install)

Run the worker directly from your terminal. On first run, it prompts for your Worker Key and saves credentials for subsequent runs.

bash

npx app-manager-edge-worker

2Option 2: Run with Flags Directly

Pass your worker key directly via CLI flag or environment variable.

bash

npx app-manager-edge-worker \
  --server="https://apps-manager.in" \
  --key="wkey_live_your_secret_key_here"

3Option 3: Android Phone 1-Liner Setup (Termux)

Copy and paste this single command into Termux to update packages, install Chromium, acquire wake lock, and launch worker.

bash

pkg update -y && pkg install -y x11-repo && pkg update -y && pkg install -y nodejs-lts chromium termux-tools && termux-wake-lock && npx app-manager-edge-worker --server https://apps-manager.in --key <YOUR_WORKER_KEY>

Production Code Integration

Running 24/7 as Background Daemon with PM2

# Install PM2 process manager
npm install -g pm2

# Start edge worker with automatic reboot recovery
pm2 start "npx app-manager-edge-worker --key wkey_live_xxx --server https://apps-manager.in" --name edge-worker

# Save PM2 process list
pm2 save
pm2 startup

Docker Deployment (Dockerfile)

FROM node:20-slim

# Install Chromium dependencies
RUN apt-get update && apt-get install -y \
    chromium \
    fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst fonts-freefont-ttf \
    --no-install-recommends \
    && rm -rf /var/lib/apt/lists/*

ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true \
    PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium

WORKDIR /app
RUN npm install -g app-manager-edge-worker

CMD ["app-manager-edge-worker", "--server", "https://apps-manager.in"]

Frequently Asked Questions

Does running an edge worker consume heavy bandwidth on my home network?

No. Workers only process scraping jobs assigned to your account, and by default block heavy media assets (video, heavy fonts) during scraping unless screenshots are explicitly requested.

What happens if my laptop goes to sleep or Wi-Fi disconnects?

The central gateway automatically detects disconnected workers and immediately re-routes pending scraping jobs to other available workers or falls back to the cloud engine.

← Previous GuideApps Manager SDK Next Guide →Store Studio & Portal

Start Building with Apps Manager Today

Build databases, create scraping APIs, convert websites to JSON, and issue Ed25519 cryptographic software licenses in minutes. Zero credit card required.

Get Started Free