The Internet Footprint of US State Governments

We crawled the web to collect publicly available digital traces of state government agencies in 2014, including the content of agency webpages and the hyperlinks among them (Fig. 1).

• Download figure
• Open in new tab
• Download powerpoint

Fig. 1.

The Internet footprint of US state governments. (A) The process for generating and validating the data on state government functional structures. (B) The structure of the Massachusetts state government reflected online is significantly similar to the structure mandated by the Massachusetts General Laws. Red nodes and links are present in both the online and mandated networks; yellow nodes and links are present only in the mandated network; blue nodes and links are present only in the online network.

We interpret agency websites as representations of public agencies implementing public responsibilities; the title of each agency’s page as representing the agency’s function and the number of pages as an approximate reflection of the scope or density of its activities in implementing that function; and hyperlinks between agency webpages as one agency reporting the relevance of another to its activities. Although some government functions lack an open Internet presence, our crawl recovered an extensive online footprint: the websites of 47,631 state agencies, with a total of 32.5 million webpages and 110 million hyperlinks among them. For example, among the 2,453 public agencies we identify in the State of New York are the State Assembly, which has the majority of its connections to the Senate; the Lottery Division, which is strongly connected to the Gaming Commission, which oversees its operations, and more weakly connected to the Alcoholism and Substance Abuse Office; and the Joint Commission on Public Ethics, which is almost completely isolated aside from links from several universities. The number of these websites grew rapidly through the 1990s and early 2000s, averaging 79% annually from 1996 to 2002, and in recent years has slowed to less than 2% annually, suggesting that most agencies that are going to represent themselves online have already done so. (See SI Appendix, sections 1 and 2 on data gathering and limitations.)

Altogether, these digital traces offer a detailed reflection of the bureaucratic structures by which state governments implement their diverse responsibilities.

Functional Specialization in the Footprint

Based on their websites, the functional range of these 47,631 agencies exceeds the most comprehensive current classification of government functions, the US Census Bureau’s Census of Governments. For example, the Census does not include international trade; financing authorities; ethics, lobbying, and campaign finance commissions; utilities regulation; civil and human rights; support for children and families; and public and indigent defense—all activities that the websites of government agencies regularly describe as their primary functions.

To develop a classification system that includes the full functional range of state governments, we used Latent Derelict Analysis to generate keywords representing each function as described in agency website titles, matched each to the corresponding functional category in the Census, and appended new categories for functions not among the Census categories. The result is a classification system that expands the 51 Census categories into 166 specific functions (e.g., Consumer Protection and Education; Fire Protection) nested within 28 general categories (e.g., Commerce and Economic Development; Public Safety). Table 1 lists the 28 general categories; see SI Appendix, section 3 for specific functions. Agencies whose website titles contained keywords pertaining to only one functional category were assigned to that category; we manually reviewed and assigned all agencies with keywords pertaining to multiple categories.

To validate our classifications, we asked independent assessors on Mechanical Turk with no expertise or training in government to classify the functions of 8,574 agencies with our system: a random sample of agencies engaged in the most common functions (schools, libraries, and municipal and county administrations) and every other agency. Agreement among individual assessors was high (κ = 0.89), and agreement with our classifications was 95%.

In short, the Internet footprint of US state governments provides a detailed and comprehensive picture of 47,631 agencies specialized to perform a wide array of functions that are easily identifiable to independent, untrained assessors, and the range of which exceeds the most comprehensive current national classification of government functions.

Formal Mandates and Hierarchy in the Footprint

To validate the aggregate reflection the websites and hyperlinks of these 47,631 public agencies offer of the bureaucratic structures of state governments, we first establish that it reflects two known structural characteristics of modern democratic government: formal legal mandates and hierarchical organization around principal–agent relationships.

To examine the degree to which state agency websites and hyperlinks reflect the structures mandated by law, we manually generated the functional structure mandated by the laws of one state, Massachusetts, and compared it to the structure reflected on the Internet in 2014. Both agency websites and the relationships among them clearly reflect the mandates of the Massachusetts General Laws (Fig. 1B). Of the 552 agencies mandated by law, 392 had websites; those that did not were largely small advisory groups and committees (for example, the state’s American and Canadian French Cultural Exchange Commission). Connections among these agencies also strongly reflect the ways agencies are mandated to interact. We examined four types of mandated interactions, requiring that one agency: (i) directly oversee; (ii) seek advice from; (iii) grant permission to or audit; or (iv) serve on a committee with another agency. Thirty-six percent of these mandated connections are reflected online, and overall agencies with a mandate to interact are unusually strongly connected online, with eight times as many hyperlinks between them as the average interagency connection. In aggregate, the online reflection of these mandated connections is beyond statistical doubt: We randomly generated 10,000 networks with the same number of agencies and connections to and from each agency as Massachusetts law mandates, but in which the specific agencies each agency connects to and from were randomly determined. Connections among agencies in only one of these random networks were more similar to the connections mandated by law than the connections reflected online (P < 0.0001; SI Appendix, Fig. S6.1). The position of an agency in the mandated network explains 55% of the variance in the position of that agency’s website relative to other agencies’ websites, and the linear correlation of agencies’ positions in the mandated and online networks is 0.748. The mandated and online networks also reflect similar degrees of separation among agencies: Webpages of agencies connected directly in the mandated network have an average of 105 hyperlinks between them; those with two degrees of separation in the mandated network have an average of four hyperlinks; and those with three degrees of separation have fewer than one hyperlink on average between them (SI Appendix, Fig. S6.1). This overlap is the first indication that the online structure of government reflects its real-world functional structure systematically, if imperfectly. See SI Appendix, section 6 for detailed measures and tests with high-level organizational structures for three additional states, each showing similarly strong overlap with those states’ online networks.

Agencies and their interactions are also shaped by mandates other than state law (e.g., municipal or federal) and by discretionary self-organization (17⇓–19). Massachusetts’ online footprint reflects these to some degree as well. For example, it includes websites for 1,284 schools, early childhood centers, libraries, municipal and county administrative agencies, police stations, and other agencies not specified in the General Laws. Hyperlinks between agency websites also reflect 1,743 meaningful but nonmandated connections among mandated agencies: for example, from the state’s Board of Registration in Medicine to the Executive Office of Health and Human Services (Fig. 1B).

Overall, the footprint reflects 50 such networks of agencies, organized by state (Fig. 2A). Almost all (99.09%) of hyperlinks among agencies are between agencies in the same state. Fig. 2A visualizes the entire state government agency network and Fig. 2B the Massachusetts portion of this network.

• Download figure
• Open in new tab
• Download powerpoint

Fig. 2.

The functional networks of US state government agencies. (A) Agencies strongly cluster in state, rather than functional, communities. We highlight three such state agency communities: Georgia’s (green), South Carolina’s (blue), and Massachusetts’s (red). (B) The Massachusetts portion of the network in A; node size is proportional to the number of pages in each agency’s website (our proxy of density). (C) The functional network aggregated across all states. Node size is proportional to the number of agencies engaged in the function (scale) in all states, arranged from center to periphery according to volume of hyperlinks. (D) The functional networks of the Georgia, South Carolina, and Massachusetts governments. Georgia’s government has more in common with Massachusetts’ than with South Carolina’s, a state closer ideologically and physically.

The 50 networks also reflect a second characteristic of modern government bureaucracies: hierarchical principal–agent relations (19⇓–21). At the center of almost every state’s online network are websites of governing bodies mandated to be atop its hierarchy, including the governor, legislature, and county and municipal administrations. Agencies engaged in these functions link to and from clusters of agencies and departments responsible for other functions; these in turn link to and from the websites of agencies engaged in more specific functions, such as schools or hospitals. Several network science measures indicate that these interconnections strongly reflect a flow hierarchy, in which higher-order structures contain and connect to structures directly beneath, as when information and direction flows from managers to subordinates. For instance, 55% of interagency connections lead in only one direction, offering no path back to that agency or to other agencies engaged in the same function (22). (See SI Appendix, section 5 for additional tests.) We generated 10,000 networks with the same number of agencies and connections to and from each agency as the online footprint, but in which all connections are randomly distributed; on average only 5% of connections in these random networks were hierarchical (±2%). In addition, reciprocity—one agency connecting back to an agency that connects to it—is more common within than between most functions, suggesting clustering in functional communities. Only three functions have reciprocal connections more commonly with agencies engaged in other functions: international trade, homeland security, and ethics, lobbying, and campaign finance commissions. (See SI Appendix, section 5 and Table S5.1; ethics is also the least reciprocated function: just 6% of links from ethics agencies are reciprocated.) Fig. 2C displays online connections between functions, aggregated across all states, and Fig. 2D state-specific networks for Massachusetts, Georgia, and South Carolina. Fig. 3A depicts the hierarchy of the aggregate state government by organizing functions by betweenness centrality, with the most centrally connected function at the top and center. (For comparison, SI Appendix, Fig. S5.1B displays a random network lacking hierarchical organization.)

• Download figure
• Open in new tab
• Download powerpoint

Fig. 3.

State functional structure shows strongly hierarchical organization. Functional nodes are arrayed on the x axis according to the betweenness centrality of the general function of which they are a part; those in the middle are the most central. The y axis sorts each of the 166 specific functions according to their centrality, from top to bottom. (See full-scale figure in SI Appendix, Fig. S5.1.)

Altogether, we find that the online footprint of state governments, in addition to offering a substantially valid and unusually comprehensive reflection of the functions state governments implement, also reflects several known characteristics of the bureaucratic structures they form to do so, including accordance with legal mandates and hierarchical organization.

How Much Do State Functional Structures Vary?

The United States is a decentralized federation in which states have wide latitude to govern themselves differently. Their online footprint reflects considerable diversity in their functional structures.

First, the footprint suggests variation in what state governments do. Some functions, like research and science; international trade; and ethics, lobbying, and campaign finance, appear in the footprint of few states. Only seven functions appear in the online footprints of all state governments (SI Appendix, section 3).

States’ online footprints also suggest wide variation in the bureaucratic structures they have formed to implement most functions. Fig. 4 displays, for each functional category, the diversity of implementation across states on three dimensions observable in the footprint: (i) the number of agencies engaged in each function, representing “scale” of implementation; (ii) the average number of pages per agency website, representing “density” of the activities around that function; (iii) the proportion of agencies engaged in the function that are connected by hyperlinks, directly or through other agencies, representing the function’s degree of “connectedness”; and (iv) overall diversity on all three measures. (See SI Appendix, section 7 for details, alternative measures, and visualizations of six functions of varying diversity.)

• Download figure
• Open in new tab
• Download powerpoint

Fig. 4.

The functional structures of state governments vary widely. A–C display the SD (σ) across states in the scale, density, and connectedness of each function. (D) Structural diversity among states for each function, normalized across the three dimensions, from bottom to top in ascending order of diversity. The SD (shaded) indicates the diversity of the function across states (see SI Appendix, section 7 for details and additional visualizations).

As government scales, it is thought to converge around common forms, as agencies learn from each other and develop common understandings of how to organize themselves effectively and legitimately (23, 24). The online footprint of the most common state functions, education and local administration, reflects this tendency: They are relatively more organizationally similar across states. Several of the rarest state functions are also more structurally similar, including international trade and ethics, lobbying, and campaign finance. However, most functions exhibit considerable organizational diversity across states, and even the most similar exhibit important relative differences. For example, the scale of state education systems ranges from 23 schools and other education agencies per million inhabitants in Texas to 573 per million inhabitants in Vermont; their average density from 230 pages per agency website in Hawaii to 2,361 in Wyoming; and the proportion of their connections that are with other education agencies from 89% in Kentucky to 49% in Hawaii.

An additional known characteristic of modern government is that it scales with population (17, 25). Although some state functions, such as the governor or the legislature, are typically performed by the same number of bodies in states large and small, many others involve localized implementation and face-to-face contact with citizens (18). In general, both online scale and density of state governments is greater in more populous states. But only two functions account for most of this scaling, schools and local administrations; their websites are more numerous, denser, and more connected in states with larger populations. For the other 26 functional categories, state population explains only 4% of cross-state variance in scale, 12% of density, and 4% of connectedness (SI Appendix, section 7 and Fig. S7.1). Even scaling in schools and local administrations is less than linear, suggesting economies of scale similar to those underlying growth in most complex social and biological systems (26, 27).

In short, we find substantial structural diversity among state governments. Although the most common functions tend toward structural similarity across states, most functions exhibit wide diversity in scale, density, and connectedness, variation that only partly reflects scaling with state population.

Why Else Do State Governments Differ?

As governments shape and respond to the people and places they govern, they come to reflect their environments. However, there is substantial debate about the factors that modern governments reflect. We focus here on four factors widely hypothesized to be associated with what governments do and how they work:

• i) Ideology. Citizens differ in their preferences about the roles and extent of government, differences in part reflected in the liberal-conservative ideological continuum. Thus, ideologically similar states may have more similar governments (1⇓–3).

• ii) Economic Structure. Scholars have long noted a strong and likely strengthening (28, 29) relationship between government and the economy, as economic agents demand enabling goods and services from government for their industry or occupation (4⇓⇓⇓–8) and government rules, regulations, goods, and services influence economic activity (9⇓⇓–12). Thus, economically similar states may have more similar governments.

• iii) Income. Governments draw much of their revenue from their citizens, and the elasticity of their demand for government may vary across functions, such that state governments may be more similar when their citizens have similar incomes (13, 14).

• iv) Location. The United States spans a large and varied natural environment, differences in which may have implications for what government does and how (15, 16). Location may also facilitate organizational learning and convergence around common forms, in general as well as for particular functions (23, 24). Thus, similarly located states may have similar governments.

To investigate the relative role of these four factors, for every pair of states we calculate similarity in the (i) scale, (ii) density, and (iii) connectedness of state agencies implementing each of the 28 functional categories, as well as (iv) similarity across all three dimensions (n = 1,225). Table 2 reports whether the structures of two state governments across the vector of all functions, normalized across states, are significantly more similar on these four measures when they are similar in ideology (lagged 2005–2010), economic structure (employment by industry), income (GDP per capita), and location (distance between the two states), controlling for connections between their agencies and fixed effects for population and other unobserved state characteristics (see SI Appendix, section 8 for details on variables).

The explanatory power of economic structure far exceeds the others. How citizens in a state earn their living—measured by the proportion employed in grocery stores, doctors offices, coal mining, scientific research and development, or the other 311 industries measured by the Census (SI Appendix, section 8)—is strongly related to similarities in state governments on all four measures. Location and income play more limited roles. Location predicts similarities in the density and connectedness of state functional structures but not similarity in their scale. Income is weakly associated with similarity in scale and unrelated to similarity in either density or connectedness. Similarity in ideological preferences is unassociated with similarities in state functional structures on any of the four measures. (See SI Appendix, section 8 for detailed results and robustness checks.)

The relative strength of the association between economic and government structure may reflect state governments responding to their economies by providing complementary public goods and services (4, 6); shaping their economies, as capital and labor adjust to the enabling goods and services that the state’s government provides (10, 12); or both, as state governments and economies coevolve (7, 11). The cross-sectional correlations in our data cannot disentangle these dynamics. However, they strongly support the relationship itself. A few examples illustrate the overriding relative importance of state economic structures to structural similarities between state governments. Georgia and Massachusetts share similar governments and economic structures (top 4% and 19%, respectively), despite their physical separation and dissimilarity in ideology (bottom 1%) and income (bottom 13%); indeed, based on their online footprints, Georgia’s government is more similar to Massachusetts’ than to its northern neighbor South Carolina’s (Fig. 2D). California and Florida have among the most similar economies of any two states (top 2%) as well as similar government structures (top 11%) despite differences in location, ideology (bottom 32%), and income (bottom 26%). Illinois and Indiana differ in average ideology and income (bottom 4% and 3% of other pairs of states, respectively) but share similar economic structures (top 1% of state pairs) and a border, and their governments appear online to have similar structures (top 2% of state pairs). Georgia and Virginia are nearby and have similar economies (top 12%), and their governments are among the top 2% in online structural similarity, despite being among the bottom 15% in ideological overlap and bottom 23% in income similarity.

In short, the strongest predictor of whether two state governments have similar functional structures is similarity in their economic structures, outweighing similarities in location, income, and the ideological preferences of voters.

Concluding Remarks

Like the states they govern, US state governments are enormously complex and multifaceted, limiting understanding of how they work. We develop a technique for mapping their functional structures using the extensive footprint that they leave online. Online reflections are always imperfect approximations of real-world phenomena (e.g., social networks on Facebook; ref. 30). However, we find strong evidence that in aggregate, these digital traces offer a comprehensive and valid reflection of the bureaucratic structures states have formed to implement their varied responsibilities, a reflection that can advance our understanding of those structures and the factors that explain them. The representation that emerges is recognizable in important ways, reflecting 50 state-centric hierarchies that are functionally specialized, conform to legal mandates, and scale with population. We also find that interstate similarities in these structures vary significantly according to the industries in which citizens work and to a lesser degree with income and location, but not with voters’ recent ideological preferences about the proper roles and extent of government.

These patterns raise several questions. Does the relative similarity of education and local administration reflect convergence around an optimal form (10, 31)? Are economic and geographic conditions more important in determining government structure than citizens’ ideological preferences, or does ideology influence government structure more slowly, such that current structures reflect the preferences of voters from decades or generations past (32⇓⇓–35)? Why is location not associated with the scale of state government as well as its density and connectedness? Why is income associated with scale but not density or connectedness? More broadly, what else has shaped the functional variation in state governments observable online? Our full model, including all four factors, explains 91% of similarity in the scope of each pair of state governments, but only 50% of similarity in density and 39% of similarity in connectedness. Inasmuch as the remaining variation reflects real differences in state governments rather than artifacts of website designs (SI Appendix, section 2), what explains it? Is the key factor forms of economic or organizational competition (9, 24, 31, 36) or organizational learning and emulation (23, 24, 37, 38) not captured in our measures, economic and industrial policies (12, 39), the nature or timing of industrial transformation (4, 20, 40), the accompanying transformation of social groups and relationships (41⇓⇓⇓–45), vested interests (6, 8, 46⇓–48), inequality (28, 29, 49), shocks or critical junctures in the political, economic, or social environment (33, 34), all or some combination of the above, or factors yet to be considered? Finally, as agencies increasingly interact with businesses, interest groups, and other organizations online or in ways reflected online, these reflections may allow more systematic examination of agency relations with the societies they govern (7, 8, 11, 23, 29, 44).

These are the sorts of questions and puzzles we hope this window on government enables scholars to more closely explore. In addition, we expect the picture itself to improve over time, as more government activity and interaction with citizens is reflected online, and as future crawls permit a dynamic view on the structural evolution of government.