Pattern 1: Static website vs. database-backed website

Start with the most common question a public web team faces, even if nobody says it out loud: when a resident visits a page on your website, where does that page come from?

There are two common answers, and the difference between them once decided which parts of a famous federal website stayed standing.

The database-backed way

Most content management systems — WordPress and Drupal are the big names — work like a short-order kitchen. The page does not exist until someone orders it.

When a visitor asks for a page, a web server receives the request, asks a database for the content, waits for the answer, assembles the content into a page, and sends the finished page back.

sequenceDiagram
    participant V as Visitor's browser
    participant W as Web server
    participant D as Database
    V->>W: Ask for a page
    W->>D: Look up this page's content
    D-->>W: The content, as raw data
    Note over W: Assemble the content<br/>into a finished page
    W-->>V: The finished page

This happens for every visit. A thousand visitors means a thousand round trips to the database, each one assembled to order.

There are good reasons this pattern became the default. Editors get a friendly editing screen. Content lives in one organized place. The same database can also power search, logins, and forms.

But look at the diagram again. The visitor is waiting on every arrow. If the database is slow, every page is slow. If the database is down, every page is down — including pages whose content has not changed in three years. The database is a single point that everything else stands on.

The static way

A static website works like a printed handout instead of a short-order kitchen. The pages are built ahead of time — once, when the content changes — and stored as complete, finished files in a simple storage service, such as a cloud bucket.

When a visitor asks for a page, the storage service hands over the file. That is the whole transaction.

sequenceDiagram
    participant V as Visitor's browser
    participant S as File storage (cloud bucket)
    V->>S: Ask for a page
    S-->>V: The page, already finished
    Note over S: Nothing is computed.<br/>The page existed before<br/>anyone asked for it.

No query, no waiting on a database, no assembly. The work of building the page happened earlier, exactly once, out of the visitor’s way.

Serving a file is about the cheapest, most reliable thing a computer can do. Storage services do it for pennies, at almost any scale, and there is very little to break and very little for an attacker to attack — there is no live software on the page’s path to compromise.

What HealthCare.gov taught everyone

On October 1, 2013, HealthCare.gov launched, and its collapse became national news.

The detail that matters here is which part collapsed.

The informational side of the site — the pages explaining the law, the plans, the deadlines — had been built as a static site: finished pages, built ahead of time, served as files. Those pages held up.

The application side — creating an account, checking eligibility, enrolling — was dynamic. Every click depended on a chain of live systems and databases answering in real time, and when several times the expected crowd arrived at once, that chain jammed. Visitors were left staring at error screens, not because the content failed, but because the machinery assembling responses on the fly could not keep up.

Federal web teams took the lesson. In the years since, much of the government web has moved to the static pattern for anything that is fundamentally content — hundreds of federal and agency sites now publish this way, and platforms like cloud.gov Pages exist specifically to do it. The rule of thumb that emerged: never make a resident’s request wait on a database unless the page genuinely depends on who is asking.

Side by side

	Database-backed	Static
When the page is built	At the moment of each visit	Ahead of time, when content changes
What a visit touches	Web server + database, live	A stored file
When the database has a bad day	The whole site does too	Nothing happens; visitors never touch it
Hosting cost	A server (or several) running around the clock	Pennies for file storage
Security surface	Live software to patch and defend on every page	Almost nothing on the page’s path
Editing	Friendly editing screens, instant publish	Requires a publishing step after edits

Where a database still belongs

Static is not a verdict against databases. It is a verdict about where they belong: behind the pages that genuinely need them, and off the path of the pages that don’t.

Some things only a live system can do. Searching thousands of records that change daily. Logging in and seeing your own case, your own bill, your own application status. Accepting a form submission. Anything where two visitors should rightly see two different pages.

The pattern that works is separation. Publish everything that is the same for every visitor — which for most agencies is the large majority of the website — as static files. Keep the live, database-backed pieces small, separate, and behind their own doors, so that when one of them has a bad day, the phone number, the office hours, and the “how do I apply” page are still standing.

And honestly: a well-run database-backed site can be tuned toward this too, by putting caches in front of it so most visitors receive a saved copy instead of a fresh assembly. That works — but notice what it is. It is an imitation of the static pattern, bolted onto the dynamic one, with more moving parts than either.

The question to ask your web team or vendor: does every visitor see the same page here? If yes, that page should be built ahead of time and served as a file. If no, that page has earned its database — keep it small and keep it separate.

Pattern 2: Forms and intake

The first pattern was about information flowing out — pages going to visitors. This one is about information flowing in. A resident needs to tell the agency something: an application, a renewal, a complaint, a request.

The most common flow, still, is the fillable PDF.

The PDF way

The website offers a form to download. The resident prints it, fills it in by hand, and sends it back — by mail, by email, or across a counter. Then a staff member reads the handwriting and types the answers into the actual system.

sequenceDiagram
    participant R as Resident
    participant Q as Mailroom / shared inbox
    participant S as Staff member
    participant Sys as Agency system
    R->>Q: Filled-out PDF, sent in
    Note over Q: Sits in a queue<br/>with everything else
    S->>Q: Pick up the submission
    S->>Sys: Re-type the answers by hand
    Sys-->>S: Rejected — a required field is blank
    S->>R: Letter or call: please resubmit
    Note over R: Back to the start,<br/>weeks after submitting

Look at where the work lands. The resident wrote the information down once, on paper. Staff write it down again, into the system. The same information is entered twice, with handwriting in between — and every re-entry is a chance for the copies to disagree.

Now look at where errors surface. The blank field is discovered at re-typing time, days or weeks after submission — when fixing it means reaching back to the resident and starting the queue over. The resident could have fixed it in ten seconds on the day they applied. Nobody asked them then.

The structured way

A web form checks the answers at the door.

sequenceDiagram
    participant R as Resident
    participant F as Web form
    participant Sys as Agency system
    R->>F: Fill out the form
    F-->>R: A required field is blank — flagged immediately
    R->>F: Fix it and submit
    F->>Sys: A complete, structured record — entered once
    Sys-->>R: Confirmation, with a reference number

The blank field costs ten seconds instead of three weeks. The information is entered once, by the person who knows it best, and arrives as a structured record staff can work with instead of handwriting staff must decode. And the resident leaves holding a reference number, which is the difference between “I sent it in” and “it exists, and I can point to it.”

A form is one of the things that genuinely needs a live piece — this is exactly the separation rule from Pattern 1. A mostly-static website with one small form service behind it is a common and healthy shape: the live part stays tiny, and the rest of the site cannot be taken down by it.

Paper should remain an option — not everyone is online. The pattern is not about closing the paper door. It is about what happens to the information once it arrives: whether it flows onward as structured data, or waits for a person to re-type it.

The question to ask: when a resident submits this form, does a person re-type it into something else? Every re-typing is a delay, an error rate, and a cost — hiding in plain sight.

Pattern 3: The nightly batch file

Inside an agency, systems need to share information with each other. The permit system needs to tell the finance system what was paid. The finance system needs to tell the reporting system what came in.

The most common way they do it is the nightly batch file. Once a day, usually in the small hours, one system exports its new records into a file. The file is dropped on a shared server. Later that morning, the other system picks the file up and imports it.

sequenceDiagram
    participant A as Permit system
    participant F as File server
    participant B as Finance system
    Note over A: 2:00 a.m.
    A->>F: Export yesterday's records as a file
    Note over F: 5:00 a.m.
    B->>F: Pick up the file
    B->>B: Import the records
    Note over B: Staff arrive to<br/>yesterday's data — every day

This pattern is everywhere, it is decades old, and it mostly works. It is the static-website idea applied between systems: prepare the data ahead of time, hand over a finished file, keep the systems out of each other’s way. Nothing about “batch” deserves scorn.

Two things about it are worth seeing clearly, though.

The data is always yesterday’s. One nightly file means a one-day lag. A chain of systems, each passing files nightly, means the lags add up. When a resident hears “please allow two to three business days for your payment to show,” that is usually not a person working for three days. It is a chain of nightly files, each waiting for its scheduled hour. The delay is cadence, not effort — and knowing that changes what “faster” would take.

It fails silently. A jammed export looks exactly like a quiet night.

sequenceDiagram
    participant A as Permit system
    participant F as File server
    participant B as Finance system
    Note over A: 2:00 a.m. — export fails quietly
    B->>F: Pick up the file
    F-->>B: Only the old file is there
    B->>B: Imports it again, reports success
    Note over B: Nobody is alerted.<br/>Discovered weeks later,<br/>during reconciliation.

The fix is not expensive. The receiving side checks that a file arrived, that it is dated today, and that the record count looks sane — and raises a loud alarm when any of that fails. A transfer that cannot announce its own failure will eventually fail without announcement.

The question to ask: for each nightly file, what happens on the morning it doesn’t arrive — and who finds out?

Pattern 4: Open data feeds

The first pattern was pages flowing out to people. This one is data flowing out to programs — other agencies’ systems, researchers, journalists, civic apps, and increasingly, software assistants acting on a resident’s behalf.

The by-request way

The default, in most agencies, is that data leaves by request. Someone emails, or files a public records request. A staff member queries the system, exports a spreadsheet, and emails it back.

sequenceDiagram
    participant P as Person or program that needs the data
    participant S as Staff member
    participant Sys as Agency system
    P->>S: Request the data (email, records request)
    Note over S: Days or weeks in a queue
    S->>Sys: Query and export
    S-->>P: A spreadsheet, by email
    Note over P: A one-time copy, aging already.<br/>Next month: ask again.

Every request costs staff time. Every requester gets a slightly different copy. And the copy starts going stale the moment it is sent, so the same question comes back next month.

The published way

The alternative is to publish the data as files — comma-separated, JSON, XML — on a schedule, at a stable web address, in the same simple file storage that serves the website.

sequenceDiagram
    participant Sys as Agency system
    participant B as File storage (same bucket as the website)
    participant C as Anyone's program
    Note over Sys: Nightly, on a schedule
    Sys->>B: Publish the data as files (.csv / .json)
    C->>B: Fetch the file at its stable address
    B-->>C: The data, current as of last night
    Note over C: No queue, no staff time,<br/>the same answer for everyone

Here is the idea underneath, and it connects everything on this page. A webpage is the agency answering a question for a person — “when does the council meet,” “what does a permit cost.” A data file at a stable address is the same answer, in a form a program can read. Same publishing motion as Pattern 1, same storage, same cost — pennies — and nothing on the request path that can fall over.

It often costs almost nothing to start, because Pattern 3 already built it. The nightly export that feeds the finance system can drop one more copy — the public fields only, chosen deliberately — into the public bucket. The habit the agency already has becomes the interface the public gets.

Software teams call the formal version of this an API — a defined doorway where one program asks and another answers, live. A published file is the smallest, sturdiest first step down that road. It is not real-time; the data is as fresh as the last publish. But it has the three things a consumer of data actually needs first: a stable address, a known shape, and a schedule that gets kept. A program can build on that. Nobody can build on a records-request queue.

The path onward runs in small steps, each useful on its own: publish the files, document what each field means, keep the schedule religiously, and add a live doorway later — only where freshness genuinely earns its cost.

A yardstick that already exists

None of this is new thinking. In December 2007, thirty open government advocates gathered in Sebastopol, California and wrote down eight principles of open government data that still hold up. Open government data, they said, is:

Complete — all public data is made available, in bulk, not just slices of it.
Primary — as collected at the source, not aggregated or reworked.
Timely — released quickly enough to preserve its value.
Accessible — on the open web, retrievable by people and by automated tools alike.
Machine processable — structured data, not free-form text or pictures of text.
Non-discriminatory — available to anyone, with no registration required.
Non-proprietary — in formats no single company controls.
License-free — public data clearly marked as free to use.

They added one more condition: compliance must be reviewable — a named contact who answers, and real recourse when the principles are not met.

Worth noticing: the group’s own annotations say bulk data should be published before fancier interfaces are built, because even the simplest analysis needs the whole dataset, and a live doorway typically returns only slices. So the published-file pattern above is not a compromise on the way to open data. It is where open data starts — and the principles are a checklist an agency can hold its feeds against, one file at a time.

The question to ask: what data do staff repeatedly export by hand for someone else — and what would it take to publish it on a schedule at a stable address instead?

Data Flow Patterns in Public Agencies