Build an AI Data Annotation Pipeline with Label Studio and Make.com: The Complete Step-by-Step Guide

This is the execution guide for building the AI Data Annotation and Training Service we outlined in our opportunity deep‑dive. Follow the steps below and you will have a production‑grade annotation pipeline with AI pre‑labeling, human review, and automated quality assurance that delivers datasets at 95%+ accuracy.

Ready to understand the full business opportunity? Read our opportunity deep‑dive.

Prerequisites

Total upfront cost per month: Approximately $82–$112 before annotator costs.

Estimated initial setup time: 3 to 4 hours for the complete pipeline.

Step 1: Install and Configure Label Studio

1.1 Deploy Label Studio on Hostinger VPS

Label Studio is the backbone of your entire annotation operation. It manages projects, annotator accounts, labeling interfaces, quality workflows, and data export.

1. SSH into your Hostinger VPS: ssh root@your-server-ip
2. Install Docker:

curl -fsSL https://get.docker.com -o get-docker.sh
sh get-docker.sh

3. Deploy Label Studio:

docker run -d \
  --name label-studio \
  -p 8080:8080 \
  -v /root/label-studio/data:/label-studio/data \
  heartexlabs/label-studio:latest

4. Open http://your-server-ip:8080 in your browser and create an admin account.
5. Configure the server with SSL using Let’s Encrypt:

sudo apt install certbot python3-certbot-nginx -y
sudo certbot --nginx -d annotation.yourdomain.com

Check‑in: Do you see the Label Studio login screen at https://annotation.yourdomain.com? If you get a connection refused error, check that Docker is running (docker ps) and that port 8080 is open in your firewall.

1.2 Configure Annotator Roles and Permissions

1. In Label Studio, go to Organization then Members and click Add Member.
2. Create two roles:
   • Annotator: Can view and label assigned tasks. Cannot modify project settings or export data.
   • Reviewer: Can view, label, and review annotations. Can override annotator labels and flag tasks for revision.
3. For each annotator you recruit, create an account with the Annotator role. For senior annotators and quality leads, use the Reviewer role.
4. Enable Task Distribution under Settings so that tasks are automatically assigned to available annotators on a round‑robin basis.

1.3 Set Up Project Templates

Create template projects for each annotation type you plan to offer. The most common templates:

Image Classification:

<View>
  <Image name="image" value="$image"/>
  <Choices name="choice" toName="image"
    choice="single-radio" showInline="true">
    <Choice value="Normal"/>
    <Choice value="Abnormal"/>
    <Choice value="Uncertain"/>
  </Choices>
</View>

Bounding Box Annotation:

<View>
  <Image name="image" value="$image"/>
  <RectangleLabels name="label" toName="image"
    strokeWidth="3" opacity="0.5">
    <Label value="Tumor" background="#FF0000"/>
    <Label value="Lesion" background="#00FF00"/>
    <Label value="Artifact" background="#0000FF"/>
  </RectangleLabels>
</View>

Text Classification:

<View>
  <Text name="text" value="$text"/>
  <Choices name="sentiment" toName="text"
    choice="single-radio">
    <Choice value="Liability Clause"/>
    <Choice value="Indemnification"/>
    <Choice value="Confidentiality"/>
    <Choice value="Termination"/>
    <Choice value="Other"/>
  </Choices>
</View>

Named Entity Recognition:

<View>
  <Text name="text" value="$text"/>
  <Labels name="label" toName="text">
    <Label value="Person" background="#FF0000"/>
    <Label value="Organization" background="#00FF00"/>
    <Label value="Date" background="#0000FF"/>
    <Label value="Amount" background="#FFFF00"/>
  </Labels>
</View>

Save these templates in a Notion database so you can quickly create new projects by copying the template and importing the client’s data.

Step 2: Build the AI Pre‑Annotation Pipeline

AI pre‑annotation is the secret weapon that makes your business profitable. By generating first‑pass labels automatically, you reduce human annotation time by 70 to 80%, which means you can deliver projects faster and with higher margins than competitors who rely on manual annotation alone.

2.1 Set Up the Pre‑Annotation Server

Create a new Replit project called annotation-prelabeler:

// src/index.js
require('dotenv').config();
const express = require('express');
const axios = require('axios');
const fs = require('fs');

const app = express();
app.use(express.json());

const OPENAI_API_KEY = process.env.OPENAI_API_KEY;
const LABEL_STUDIO_URL = process.env.LABEL_STUDIO_URL;
const LABEL_STUDIO_API_KEY = process.env.LABEL_STUDIO_API_KEY;

// Pre-annotate text classification tasks
app.post('/prelabel/text', async (req, res) => {
  const { tasks, categories, instructions } = req.body;

  const prompt = `You are a data annotation assistant. Classify each text into one of these categories: ${categories.join(', ')}.

Instructions: ${instructions}

Texts to classify:
${tasks.map((t, i) => `${i + 1}. "${t.text}"`).join('\n')}

Respond with a JSON array of classifications, one per text.`;

  try {
    const response = await axios.post(
      'https://api.openai.com/v1/chat/completions',
      {
        model: 'gpt-4o-mini',
        messages: [
          { role: 'system', content: 'You are a precise data annotation assistant. Respond only with valid JSON arrays.' },
          { role: 'user', content: prompt },
        ],
        temperature: 0.1,
        max_tokens: 2000,
      },
      { headers: { 'Authorization': `Bearer ${OPENAI_API_KEY}` } }
    );

    const classifications = JSON.parse(response.data.choices[0].message.content);

    // Push pre-annotations back to Label Studio
    for (let i = 0; i < tasks.length; i++) {
      await axios.post(
        `${LABEL_STUDIO_URL}/api/annotations/${tasks[i].id}`,
        {
          result: [{
            from_name: 'choice',
            to_name: 'text',
            type: 'choices',
            value: { choices: [classifications[i]] },
          }],
          was_pre_annotated: true,
        },
        { headers: { 'Authorization': `Token ${LABEL_STUDIO_API_KEY}` } }
      );
    }

    res.json({ success: true, annotated: tasks.length });
  } catch (error) {
    console.error('Pre-annotation error:', error.message);
    res.status(500).json({ error: error.message });
  }
});

// Pre-annotate image tasks using a vision model
app.post('/prelabel/image', async (req, res) => {
  const { tasks, categories } = req.body;

  // For image pre-annotation, use a Hugging Face model or custom endpoint
  // This is a simplified example
  res.json({ message: 'Image pre-annotation endpoint — configure with your vision model' });
});

app.get('/health', (req, res) => {
  res.json({ status: 'healthy', timestamp: new Date().toISOString() });
});

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
  console.log(`Pre-labeler running on port ${PORT}`);
});

2.2 Install Dependencies and Deploy

npm init -y
npm install express axios dotenv
npm run start

Test the pre‑annotation endpoint:

curl -X POST http://localhost:3000/prelabel/text \
  -H "Content-Type: application/json" \
  -d '{
    "tasks": [{"id": 1, "text": "The defendant shall indemnify the plaintiff for all damages"}],
    "categories": ["Liability Clause", "Indemnification", "Confidentiality", "Termination", "Other"],
    "instructions": "Classify legal contract clauses by type."
  }'

2.3 Configure Image Pre‑Annotation

For image annotation projects, use a Hugging Face model endpoint. Create a script that:

1. Downloads each image from the Label Studio task
2. Sends it to a pre‑trained model (e.g., DETR for object detection, SAM for segmentation)
3. Converts the model output to Label Studio annotation format
4. Pushes the pre‑annotations to Label Studio

This pipeline reduces human annotation time for bounding box tasks by approximately 75% and for segmentation tasks by approximately 80%.

Step 3: Build the Make.com Automation Scenarios

3.1 Scenario 1: Project Intake and Task Distribution

This scenario automates the process of receiving client data and creating Label Studio projects:

1. Trigger: Webhook (client uploads data via a form on your site)
2. Module 1: HTTP — Create a new Label Studio project with the appropriate template
3. Module 2: HTTP — Import the uploaded data into the project
4. Module 3: HTTP — Trigger the pre‑annotation pipeline
5. Module 4: Notion — Create a project tracking page with client details, deadline, and quality targets
6. Module 5: ActiveCampaign — Send the client a confirmation email with project details and estimated delivery date

3.2 Scenario 2: Quality Assurance Pipeline

This scenario runs quality checks on completed annotations:

1. Trigger: Scheduler (every 6 hours)
2. Module 1: HTTP — Fetch all annotations completed in the last 6 hours from Label Studio
3. Module 2: Filter — Pass only annotations that have not been reviewed
4. Module 3: Iterator — Loop through each unreviewed annotation
5. Module 4: HTTP — Assign the annotation to a Reviewer in Label Studio
6. Module 5: Notion — Update the quality tracking database with review assignments

3.3 Scenario 3: Delivery and Invoicing

This scenario handles project completion and billing:

1. Trigger: Webhook (Label Studio project marked as complete)
2. Module 1: HTTP — Export all annotations in the client’s preferred format
3. Module 2: Notion — Update project status to “Delivered”
4. Module 3: ActiveCampaign — Send the client a delivery email with the dataset attached and an invoice
5. Module 4: HTTP — Generate a quality report with accuracy metrics and inter‑annotator agreement scores

Step 4: Build the Annotator Onboarding System

4.1 Qualification Test Pipeline

Every annotator must pass a qualification test before they can work on client data. Build this pipeline in Notion and Make .com:

1. Notion Database: “Annotator Pipeline” with the following fields:

   • Name, Email, Domain, Status (Applied → Testing → Qualified → Active → Inactive)
   • Qualification Score, Projects Completed, Average Accuracy
   • Hourly Rate, Availability (hours/week)

2. Qualification Test Workflow:

   • When an annotator submits an application, Make.com sends them a link to a Label Studio qualification project
   • The project contains 20 to 30 pre‑labeled examples that the annotator must reproduce
   • After submission, a script calculates accuracy against the known correct labels
   • Annotators scoring above 85% are automatically moved to “Qualified” status
   • Annotators scoring above 95% are flagged as “Expert” and prioritized for high‑value projects

4.2 Training Materials Production

Create professional training materials for each annotation domain:

1. Video walkthrough: Record a 10‑minute Loom video showing how to use Label Studio’s interface for the specific annotation task. Use ElevenLabs to add a professional voiceover.

2. Written guidelines: Use ChatGPT to draft annotation guidelines, then edit them for domain specificity. Include 20+ examples covering common cases, edge cases, and ambiguous cases with the correct annotation for each.

3. Practice dataset: Create a small practice dataset (50 to 100 items) with known correct labels. New annotators complete this dataset before working on any client project. Their accuracy on the practice set predicts their accuracy on real data with 80%+ correlation.

Step 5: Deploy the Hostinger Production Server

5.1 Server Architecture

annotation.yourdomain.com
├── Label Studio (Docker, port 8080)
├── Pre-annotation API (Node.js, port 3000)
├── Nginx reverse proxy (port 443)
├── PostgreSQL (Docker, port 5432)
└── Redis (Docker, port 6379)

5.2 Docker Compose Configuration

Create docker-compose.yml:

version: '3.8'
services:
  label-studio:
    image: heartexlabs/label-studio:latest
    ports:
      - "8080:8080"
    volumes:
      - ./label-studio-data:/label-studio/data
    environment:
      - DJANGO_DB=default
      - POSTGRE_HOST=postgres
      - POSTGRE_PORT=5432
      - POSTGRE_USER=labelstudio
      - POSTGRE_PASSWORD=${DB_PASSWORD}
      - POSTGRE_NAME=labelstudio
    depends_on:
      - postgres

  postgres:
    image: postgres:15
    environment:
      POSTGRES_USER: labelstudio
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_DB: labelstudio
    volumes:
      - ./postgres-data:/var/lib/postgresql/data

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"

  prelabeler:
    build: ./prelabeler
    ports:
      - "3000:3000"
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - LABEL_STUDIO_URL=http://label-studio:8080
      - LABEL_STUDIO_API_KEY=${LABEL_STUDIO_API_KEY}
    depends_on:
      - label-studio

Start everything:

docker-compose up -d

5.3 Nginx Configuration

server {
    listen 443 ssl;
    server_name annotation.yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/annotation.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/annotation.yourdomain.com/privkey.pem;

    client_max_body_size 100M;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location /api/prelabel/ {
        proxy_pass http://127.0.0.1:3000/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

Step 6: Quality Assurance Methodology

6.1 Multi‑Pass Review System

Implement a three‑pass quality system:

6.2 Inter‑Annotator Agreement Metrics

For every project, calculate agreement metrics:

• Cohen’s Kappa: Measures agreement between two annotators beyond chance. Target: kappa > 0.75 (substantial agreement).
• Fleiss’ Kappa: Measures agreement among three or more annotators. Target: kappa > 0.70.
• IoU (Intersection over Union): For bounding box and segmentation tasks. Target: IoU > 0.80.

Build a simple Python script that calculates these metrics from Label Studio export data:

import json
from sklearn.metrics import cohen_kappa_score

def calculate_kappa(annotator_1_labels, annotator_2_labels):
    kappa = cohen_kappa_score(annotator_1_labels, annotator_2_labels)
    return kappa

# Load annotations from Label Studio export
with open('annotations.json') as f:
    data = json.load(f)

# Extract paired labels
a1_labels = []
a2_labels = []
for task in data:
    if len(task['annotations']) >= 2:
        a1_labels.append(task['annotations'][0]['result'][0]['value']['choices'][0])
        a2_labels.append(task['annotations'][1]['result'][0]['value']['choices'][0])

kappa = calculate_kappa(a1_labels, a2_labels)
print(f"Cohen's Kappa: {kappa:.3f}")
print(f"Agreement level: {'Substantial' if kappa > 0.75 else 'Moderate' if kappa > 0.60 else 'Fair'}")

6.3 Annotator Performance Tracking

Maintain a performance dashboard in Notion that tracks:

• Per‑annotator accuracy: Average accuracy across all projects
• Per‑annotator speed: Average time per task compared to project benchmarks
• Flag rate: How often this annotator’s work is flagged for revision
• Domain expertise score: Accuracy broken down by annotation type

Annotators who consistently score above 95% accuracy get rate increases and priority on high‑value projects. Annotators who drop below 80% accuracy on two consecutive projects are moved to a remediation program or released from the pool.

Step 7: Client Reporting and Retention

7.1 Automated Quality Reports

Generate a client report for every project that includes:

• Total annotations delivered
• Average accuracy (from the quality review sample)
• Inter‑annotator agreement scores
• Distribution of labels across categories
• Edge cases discovered and how they were resolved
• AI pre‑annotation accuracy (how much the AI got right before human review)
• Time to delivery versus estimated timeline

7.2 Retention Strategies

The Data Quality Guarantee: Offer a “quality guarantee” in your contract — if the client’s model performance does not improve after training on your annotated data, you will re‑annotate the disputed labels at no charge. This sounds risky, but in practice it almost never triggers because your quality processes are rigorous. It is the ultimate trust builder and deal closer.

The Iterative Improvement Loop: After the client trains their model, offer to analyze the model’s failure cases and produce a targeted supplementary dataset that addresses the specific patterns the model gets wrong. This creates a recurring revenue cycle: initial annotation, model training, error analysis, supplementary annotation, model retraining.

The Annotator Knowledge Moat: Over time, your annotators develop deep familiarity with each client’s domain, data patterns, and labeling preferences. This institutional knowledge is incredibly valuable and difficult for the client to replicate internally. Frame your retainer as maintaining this expertise, not just producing labels.

Troubleshooting Common Issues

Recommended Tools

These are the tools we recommend for building and scaling AI automation businesses:

• Label Studio — Open‑source data labeling platform — annotate images, text, audio, and video
• Make.com — Visual automation platform — connect any app without code
• ElevenLabs — AI voice synthesis — create training materials and onboarding content
• Semrush — SEO and content marketing — outrank your competitors
• Replit — Cloud IDE — build and deploy without infrastructure