Seoul Labs

One line in a generated Postgres policy, USING (true), is enough to make every row in a table readable by any logged in user. CodeRabbit found AI written code carries a 2.74x multiplier on exactly this kind of security vulnerability across 470 pull requests. The demo works flawlessly, the walkthrough video looks great, and nothing in the developer experience flags a permissive policy as wrong because nothing technically failed. Lovable alone has 170 of 1,645 scanned apps exposed with row level security flaws, shipped mostly by the non developers these platforms were built for. So if the demos work and the code compiles, what's actually failing? And why does it stay invisible until real users show up? The rest of this post breaks down how workspace architecture, not model quality, determines whether that gap gets caught before real users show up.

None of this matches the marketing. I've spent the better part of this year running Claude Code across real client projects, watching MCP servers connect and disconnect, and comparing notes with other people doing the same thing in Slack channels and half finished Notion docs. The gap between what these tools promise and what the data says is measurable, and the measurements are starting to look consistent across completely different research groups. The first measurement is about speed, not security, and it's the one most teams haven't internalized yet.

The METR Study Found AI Assistance Slowed Developers Down by 19 Percent

The METR study keeps coming up in conversations with senior engineers, and for good reason. Sixteen developers is a small sample, but the methodology was tight: real repositories, real tasks, before and after timing with AI tool access. The result was a 19 percent slowdown, not a speedup. That number contradicts basically every vendor claim published this year. It also contradicted my own gut sense from the first few weeks of using Claude Code, when everything felt fast because the small stuff got fast.

Here's what seems to actually be happening. Autocomplete style suggestions and short function generation feel instant and satisfying, so the brain logs a win. The slowdown shows up later, in review, in the fifteen minutes spent figuring out why a generated function handles a null case in a way that made sense to the model and no sense to the actual data. I hit this exact wall with a FastAPI project this spring, where a Claude Code generated dependency injection pattern worked in isolation and broke silently in a route I hadn't touched in weeks.

Generated by Claude Code, looked fine in isolation
Lovable Apps Scanned: Exposed vs Secure Row Level Security

  
Out of 1,645 apps scanned, 170 had row level security flaws (10.3%)
  

    
10.3%
    
89.7%
  
  

    
Exposed apps: 170
    
Secure apps: 1,475
  
  

    Each exposed app allows any authenticated user to read or write rows they should never touch, mostly shipped by non developers.
  
Source: CodeRabbit scan of 1,645 Lovable generated apps

def get_db_session():
    session = SessionLocal()
    try:
        yield session
    finally:
        session.close()

Problem: another route was already holding a session
Why the Security Gap Stays Invisible Until Real Users Arrive

  

    

      
Step 1: Generate
      
AI writes code such as USING (true) policy or a DB session dependency
    
    
↓
    

      
Step 2: Demo Passes
      
Code compiles, walkthrough looks flawless, nothing technically fails
    
    
↓
    

      
Step 3: Ship
      
Non developers deploy without a dedicated security review step
    
    
↓
    

      
Step 4: Failure Surfaces
      
Silent data loss or exposed rows appear days later as a support ticket, not a build error
    
  
Source: Author analysis based on CodeRabbit and METR findings



open via a cached dependency, closing it out from under


a background task. No exception. Just silent data loss


on the second write.

That kind of bug doesn't show up in a benchmark. It shows up three days later as a support ticket. METR is the first study I've seen that tries to account for that tail cost instead of measuring only the moment of code generation. Nineteen percent is the figure teams should be quoting instead of the vendor demo reel. A silent session bug is one kind of tail cost. A security flaw that never throws an exception is another, and it's the one that scales worst once non developers are the ones shipping.

CodeRabbit's Analysis Reveals How Fast Security Debt Accumulates

CodeRabbit's analysis of 470 pull requests found a 1.7x multiplier on major issues in AI written code and a 2.74x multiplier specifically on security vulnerabilities. That second figure should worry any team shipping vibe coded features into production without a dedicated security review step, and most teams don't have one.

The Lovable numbers make this concrete instead of abstract. Out of 1,645 apps scanned, 170 had row level security flaws, a rate of 10.3 percent, meaning any authenticated user could read or write rows they should never have touched. That's not a theoretical CVE sitting in a disclosure database somewhere. That's a Postgres table with permissive policies, because the platform generated a working demo instead of a secured backend, and the person building it didn't know enough Postgres to notice the difference.

-- What a generated RLS policy often looks like
-- when the model prioritizes "it works" over correct scope
CREATE POLICY "Users can view their own data"
ON public.orders
FOR SELECT
USING (true);  -- should be: auth.uid() = user_id

That USING (true) clause is the whole problem in one line. It compiles, it passes the demo, the app works flawlessly in the walkthrough video. It also means every row in the orders table is readable by every logged in user, and nothing in the developer experience flags that as wrong, because nothing failed. I caught this exact pattern twice this year in Supabase backed apps built by non developers who used Lovable or Bolt.new to ship something fast, then asked me to look at it once they had actual paying users.

The uncomfortable part is that fixing this requires knowing SQL well enough to recognize what a correct RLS policy looks like, which is precisely the skill the tool was sold as replacing. Lovable's usage data specifically shows that 63 percent of vibe coding platform users identify as non-developers, though this figure originates from broader industry analysis rather than a statement made directly by Lovable. That's the target market working as intended, and it's also the group least equipped to catch a USING (true) clause before it ships. 2.74x is the number that turns that gap into a liability, and no better model fixes it. It's a problem of what environment the model is generating code inside of, which is where the workspace itself starts to matter more than which LLM is doing the writing.

Workspace Architecture Now Determines Outcomes More Than Model Quality Does

The tools pulling ahead in 2026 aren't winning on raw model capability. They're winning on what parts of the industry call workspace DNA: memory plus intelligence plus execution, bundled into one environment instead of scattered across a chat window, a terminal, and a browser tab. Claude Code's MCP server ecosystem illustrates this shift clearly, because the value isn't the model itself. It's the persistent context and tool access wrapped around it.

I run a handful of MCP servers locally for client work: one for Postgres introspection, one for filesystem access scoped to a specific repo, one that hits an internal ticketing API. The setup works cleanly on my machine and has failed in CI twice this year for reasons that had nothing to do with Claude Code itself.

.github/workflows/ci.yml , the MCP server needs a


local socket that doesn't exist in the runner
services:
  mcp-postgres:
    image: internal/mcp-pg-server:latest
    ports:
     , "5433:5433"
    # missing: the auth token the server expects
    # as an env var, which lived only in my shell profile

That's the pattern with MCP right now. Genuinely useful for local development, because it turns Claude Code from a text predictor into something closer to an agent with real access to an actual environment. But every MCP server is its own small integration project, with its own auth model, its own failure modes, its own assumptions about what environment it's running in. Nobody documents the CI case because most people aren't running these servers in CI. They're running them on a laptop, and that laptop has months of accumulated shell configuration that CI doesn't have.

Platforms built specifically around this memory, intelligence, execution triad from day one, rather than bolted together after the fact, behave differently. Replit's agent workflows and Cursor's project level context features are both explicitly designed so the execution environment and the memory layer are the same system, not two systems talking over a socket. That design choice is likely why non developer friendly platforms report the usage numbers they do: the friction of gluing tools together disappears when the platform owns the whole stack. The tradeoff is control, less visibility into what's actually happening under the hood, which is exactly how a USING (true) clause slips through unnoticed.

What current benchmarks capture poorly is the long tail cost of that tradeoff, the maintenance burden six months after the demo works. CodeRabbit's 2.74x vulnerability multiplier and Lovable's 10.3 percent RLS flaw rate are the closest things available to a number on that cost, and neither one measures a hypothetical. They measure apps that already shipped. The question this post opened with was why a flawless demo and a compromised backend can be the same app. The answer: demos test whether code runs, not whether a workspace was built to catch what happens when it runs wrong. Until that architecture gets fixed, 10.3 percent is the number worth remembering the next time a demo looks done.