We Built an AI Video Aggregator — Here Are the Hard Parts Nobody Talks About

Last year we set out to build something that seemed simple: a unified interface for AI video generation.

One prompt. Multiple models. Pick the best result.

Six months later, I can tell you: the concept was simple. The execution was not.

Here's what we learned building Reelive.ai — the technical challenges nobody warned us about.

The Promise vs. The Reality

On paper, AI video aggregation is straightforward:

1. User submits a prompt
2. Send it to OpenAI Sora, Google Veo, Kling, Wan
3. Return the videos
4. Charge credits

In practice, every single step has edge cases that will make you question your career choices.

Become a member

Challenge #1: Every API Speaks a Different Language

You'd think AI video APIs would have some standardization. You'd be wrong.

Here's what we deal with:

Sora returns a task ID immediately, then you poll for completion. Progress updates are sporadic. Sometimes it jumps from 10% to 100% with no warning.

Veo uses a different authentication flow entirely. Response payloads are nested three levels deep. Error codes are... creative.

Kling has rate limits that change based on time of day (we think). Documentation exists, but it's in Chinese and machine-translated English simultaneously.

Wan sometimes returns video URLs that expire in 15 minutes. Sometimes 24 hours. We've never figured out the pattern.

Our solution? A normalization layer that translates every provider's quirks into a consistent internal format:

interface NormalizedTask {
  id: string;
  provider: 'sora' | 'veo' | 'kling' | 'wan';
  status: 'pending' | 'processing' | 'completed' | 'failed';
  progress: number; // 0-100, even if the provider doesn't support it
  videoUrl?: string;
  expiresAt?: Date;
  error?: NormalizedError;
}

Simple? Yes. Getting there? 47 if-else branches and counting.

Challenge #2: When Do You Charge Credits?

This sounds trivial. It's not.

Consider the failure modes:

• User submits prompt → generation starts → provider crashes → do they get charged?
• User submits prompt → generation completes → video URL expires before download → refund?
• User submits prompt → generation "completes" → video is 3 seconds of black screen → what now?

We went through four different credit deduction strategies before landing on the current one:

Strategy 1: Charge upfront Problem: Users get charged for failed generations. Support tickets explode.

Strategy 2: Charge on completion Problem: Malicious users spam generations, only "accept" the good ones.

Strategy 3: Charge on task creation, refund on failure Problem: Refund timing is tricky. Some providers take 5 minutes to fail. Some take 5 hours.

Strategy 4 (current): Reserve credits, then deduct We now reserve credits when a task starts, but only finalize the deduction when we've verified the output is valid. If the task fails or produces garbage, the reservation is released.

This required building a creditTransaction table with expiration handling:

CREATE TABLE credit_transaction (
  id UUID PRIMARY KEY,
  user_id UUID REFERENCES users(id),
  amount INTEGER NOT NULL,
  type VARCHAR(20) NOT NULL, -- 'deduct', 'reserve', 'release', 'refund'
  expires_at TIMESTAMP,
  finalized_at TIMESTAMP,
  task_id UUID REFERENCES generate_task(id)
);

The finalized_at column was a late addition after we realized reservations were leaking credits into limbo.

Challenge #3: Long-Running Tasks in a Serverless World

AI video generation takes 30 seconds to 5 minutes. Sometimes longer.

We're running on Vercel and Cloudflare Workers. Function timeout: 30 seconds max.

The obvious solution is polling. But polling has its own problems:

• How often do you poll? Too frequent = rate limited. Too slow = bad UX.
• What if the client disconnects? The task is still running on the provider's side.
• What if your server restarts mid-poll?

Our architecture:

1. Task initiation: Serverless function sends request to provider, saves task to database, returns immediately
2. Status tracking: Separate serverless function polls provider APIs on a schedule (Vercel Cron)
3. Client updates: Real-time status via polling from frontend (we tried WebSockets, but Cloudflare Workers made it painful)
4. Completion handling: Another serverless function processes completed tasks, downloads videos, uploads to our S3, updates credits

This is more complex than a monolith. But it scales to zero when nobody's using it, which matters when you're bootstrapped.

The downside: debugging distributed systems is hell. We've had tasks stuck in "processing" for days because a cron job silently failed.

Challenge #4: Video Storage Economics

AI-generated videos are big. A 10-second 1080p video is 20-50MB.

Users generate a lot of videos. Many of them are garbage (see: the randomness problem from our previous post).

Storing everything forever is expensive. Deleting too aggressively makes users angry.

Our current policy:

• Videos are stored for 7 days after generation
• Users can "save" videos to extend retention to 30 days
• Paid users get permanent storage (with fair use limits)

But here's the thing nobody tells you: video transcoding is a hidden cost.

Providers return videos in different formats. Sora gives you MP4 with H.264. Veo sometimes returns WebM. Kling has its own codec preferences.

For consistent playback across browsers, we transcode everything. That's CPU time. CPU time on serverless platforms is expensive.

We ended up offloading transcoding to a dedicated worker on a cheap VPS. It's not elegant, but it's 10x cheaper than doing it on Lambda.

Challenge #5: Provider Outages and Degradation

AI providers go down. A lot.

In the past 6 months:

• Sora had 3 major outages (hours, not minutes)
• Veo had silent degradation where videos would generate but be corrupted
• Kling's API returned 500 errors for an entire weekend
• Wan had a period where every video included a watermark they'd removed months ago

When you're a single-provider wrapper, an outage is your outage.

When you're an aggregator, you have options:

async function generateWithFallback(prompt: string, preferredProvider: Provider) {
  const providers = [preferredProvider, ...getFallbackProviders(preferredProvider)];

  for (const provider of providers) {
    try {
      const health = await checkProviderHealth(provider);
      if (!health.available) continue;

      return await generateVideo(provider, prompt);
    } catch (error) {
      logger.warn(`Provider ${provider} failed, trying next`, { error });
      continue;
    }
  }

  throw new AllProvidersFailedError();
}

This sounds great in theory. In practice:

• Different providers have different strengths. Falling back from Sora to Kling might give a completely different result.
• Users get confused when they select one model and get output from another.
• Credit costs vary by provider. Do you charge the original price or the fallback price?

We ended up making fallbacks opt-in and very explicit in the UI. Automatic magic causes more problems than it solves.

Challenge #6: The Prompt Translation Problem

Here's something we didn't anticipate: prompts that work great on one model fail miserably on another.

"Cinematic drone shot over mountains, golden hour lighting, 8K"

• Sora: Understands this perfectly
• Veo: Interprets "8K" literally and tries to generate insane resolution
• Kling: Ignores "drone shot" and gives you a static camera
• Wan: Great with the lighting, weird with camera movements

We experimented with automatic prompt adaptation — rewriting prompts to match each provider's strengths.

It didn't work. The transformations were too unpredictable, and users felt like they'd lost control.

Our current approach: documentation. Lots of it. We show users what each model is good at and let them decide.

Less magical. More transparent. Fewer support tickets.

What We'd Do Differently

If we started over:

1. Build the normalization layer first. We hacked it together incrementally and now it's spaghetti.

2. Invest in observability earlier. We added proper logging and tracing at month 4. Should have been day 1.

3. Don't fight the platform. We spent weeks trying to make WebSockets work on Cloudflare. Polling is fine.

4. Talk to users before building fallback logic. We assumed they'd want automatic failover. They didn't.

5. Plan for provider changes. APIs change. Rate limits change. Pricing changes. Build for flexibility.

The State of AI Video Aggregation

Six months in, here's my honest assessment:

The tech is hard but solvable. The real challenge is trust.

Users are giving you money to access AI that they could theoretically access directly. Why use an aggregator?

• Convenience (one account vs. five)
• Cost (bulk pricing on credits)
• Comparison (A/B test models easily)
• Reliability (we handle outages so you don't have to)

But all of that falls apart if your platform is buggy, slow, or eats credits unfairly.

We've spent more time on edge cases and error handling than on features. That's probably the right trade-off for a platform built on trust.

Try It Yourself

If you're building something similar, I hope this post saves you some pain.

If you just want to generate AI videos without dealing with any of this, that's why we built Reelive.ai.