Using Generative LLMs to Explore the DOGE Layoffs

What can a generative LLM tell us about what factors are predictive of the DOGE layoffs?

Adam Bonica recently posted an analysis on BlueSky and Substack where he found that the DOGE purges are not about efficiency or government waste; instead, agencies perceived to be liberal are more likely to face staffing cuts. But what other factors, besides ideology, could be predictive of the cuts?

Here, I explore another dimension: whether the agency is perceived to be a knowledge institution, or an agency that is perceived to produce, distribute, and/or legitimize knowledge. I use a generative LLM to estimate this measure. Examining this measure using a graph similar to Bonica’s suggests that agencies more likely to be perceived as knowledge institutions are more likely to face staffing cuts, and is perhaps an additional factor beyond ideology driving the DOGE layoffs.

Estimating the Knowledge Institution Measure

Some writers have identified Curtis Yarvin’s (aka Mencius Moldbug) ideas as an influential source shaping the federal government's current actions. Vice President JD Vance, for example, has directly referenced Yarvin’s ideas: in a 2021 podcast, Vance said, “There’s this guy Curtis Yarvin who’s written about some of these things. One has to basically accept that the whole thing is going to fall in on itself.”

One of Yarvin’s core ideas is that knowledge institutions—those institutions he believes create, distribute, and legitimize knowledge—have been dominated by progressive messaging and need to be dissolved. These ideas are also echoed in Project 2025, which calls for dismantling the Department of Education and significant changes across science agencies.

To estimate a knowledge institution measure, I used the approach I proposed in one of my previous papers: I made pairwise comparisons between agencies using a generative LLM using the following prompt:

Knowledge institutions create, distribute, and/or legitimize knowledge. Which agency is more likely to be perceived to be producing knowledge, distributing knowledge, and/or supporting knowledge institutions such as academic and educational institutions, the media, and civil society organizations: [Agency A] or [Agency B]?

Specifically, I used Meta’s Llama 3.3 70B Instruct (the version from this commit). I didn’t want to use a closed source and proprietary LLM such as GPT because it likely has been updated with recent news. This version of Llama 3.3 was last updated December 6, 2024.

After collecting the outcomes of all possible pairwise comparisons between the federal agencies from Bonica’s data, I used the Bradley-Terry model to estimate latent knowledge institution scores for all agencies. The Bradley-Terry model estimates how likely one agency is to be perceived to be more of a knowledge institution than another agency based on their relative latent measure on this dimension. The Bradley-Terry model is similar to the more well-known Elo rating system, which is often used in competitive contexts such as chess. In short, these scores are a continuous measure of the perception of an agency as a knowledge institution. Some of the top scoring agencies along this dimension include the NSF, Department of Education, NIH, USAID, NASA, and NIST.

One of the benefits of using an LLM to estimate scales is that we are not limited to measures that require an extensive survey or data collection effort. We can estimate scales for many latent concepts of potential interest. It’s important to note that these LLM-driven measures are not validated, and the reasoning behind pairwise comparisons is not transparent because of the black box problem of LLMs. Nonetheless, this approach allows us to leverage the massive quantity of internet and digital media data on which these models have been trained, enabling us to quickly estimate exploratory measures that help identify what currently correlates with the actions of DOGE.

Running a logistic regression using the knowledge institution measure, I find that this variable is predictive of DOGE layoffs.

The positive coefficient indicates that agencies more likely to be perceived as knowledge institutions are more likely to face DOGE layoffs. The LLM knowledge institution measure still remains predictive of these layoffs even when we control for an LLM-derived ideology measure, estimated using the same approach (more on this in the next section).

The graph and logistic regressions indicate that an agency’s perception as a knowledge institution is one predictive dimension of the DOGE layoffs, but it doesn’t tell the whole story. And, at least with the current situation, ideology still seems more predictive of whether an agency will face layoffs.

The findings suggest that the DOGE layoffs go beyond the perceived ideological bent of the agency: they appear to be a fight over who gets to wield and distribute knowledge. For example, 18F, a cost-recoverable division of the General Services Administration, was recently eliminated. Clearly, it was not for budgetary reasons. The open source solutions—the knowledge—that the division created to make not only the federal government but state and local governments more efficient, however, was clearly a threat to DOGE’s central mission.

The eliminations and layoffs seem a little mindless. As Dominic Packer and Jay Van Bavel note in a recent post, “Defunding the world’s leading scientific community is akin to performing a prefrontal lobotomy on the nation.” But this analysis suggests it is not mindless—the lobotomy is going exactly as planned.

Replicating Bonica’s Analysis Using a Generative LLM

Bonica used the perceived ideology measure from Richardson, Clinton, and Lewis (2018), which used a survey of federal executives to measure the ideological leanings of federal agencies. To replicate his analysis with another measure of ideology, I made pairwise comparisons between agencies using Llama 3.3 70B Instruct using the following prompt:

Which agency is perceived to be more liberal: [Agency 1] or [Agency 2]?

Using a logistic regression, I found that the LLM-based ideology score replicated Bonica’s findings:

Because the pairwise comparisons were made asking which agency is more liberal, a higher ideology score implied a more perceived liberal agency. A positive coefficient indicates that agencies perceived as more liberal are more likely to face DOGE layoffs. Like Bonica, I also found that agency size had no significant effect, and annual budget had a modest effect on the odds of facing layoffs.