COVID-19 response
COVID-19 was a major focus of ODI in FY 2020-21 and a significant part of ODI’s work in FY 2021-22. Highlights include:
- Made 24,699 updates to covid19.ca.gov in FY 2020-21, the state’s most visited website with 754,461 average weekly users. ODI made another 3,106 updates in FY 2021-22.
- Gathered 144,024 survey responses about people’s attitudes and concerns toward the COVID-19 vaccine.
- Built a vaccine sentiment scoring model that allowed decision makers to find locations where vaccination uptake was low but willingness was high.
- Used natural language processing, sentiment analysis, and topic modeling to analyze parts of speech, sentiment, allowing us to help inform mobile and pop-up vaccine deployment at local health jurisdictions, inform ad targeting and message development, and inform direct engagement.
- Built a dashboard that allowed analysts to explore and quantify the open-ended data in relation to key factors like geography, race/ethnicity, and age. It was used by Third- Party Administrator analysts and ODI to provide deeper insight into the most common concerns and barriers to getting vaccinated.
COVID-19 website (covid19.ca.gov)
ODI continued maintenance of covid19.ca.gov in 2020-21, playing a central role in the coordination and distribution of COVID-19-related information and guidance. ODI collaborated with multiple agencies to make updates to the site on a daily basis. From January through December 2021, ODI made 38,882 GitHub commits to covid19.ca.gov. Each commit represents an update to the website. These changes ranged from routine daily data updates to the launch of new pages or charts in response to changing circumstances. Three examples reflect the complexity and variety of the covid19.ca.gov site changes:
- Based on feedback from Californians, ODI developed, tested, and iterated on the extremely popular “What’s Open” search tool. Before it was retired, it covered 125 types of businesses or activities, with four statuses each, depending on local conditions, so Californians could learn what was happening in their communities.
- ODI updated the entire COVID-19 site to support the state’s reopening on June 15, 2021, including removing the Blueprint for a Safer Economy. This major overhaul was delivered on time and seamlessly from the user’s perspective.
In FY 2021-22, ODI continued to maintain the COVID-19 site. Major milestones included:
- Launching a new page detailing the state’s reopening under the SMARTER Plan
- Redesigning vaccination data visualizations to make it easier for Californians to see key information
- Creating innovative new ways to make data visualizations accessible, including:
- Developing and launching the state dashboard
- Creating the homepage trackerbox with mini, lightweight charts that load quickly and are easily scanned
- Creating charts comparing COVID-19 cases, tests, and deaths between vaccinated and unvaccinated Californians
- Moving publishing to Amazon Web Services, which sped it up and improved reliability
- Adding a new chart to track variants, a growing public concern during the Delta and Omicron waves
- Adding a new chart to compare cases, hospitalization, and deaths between vaccinated and unvaccinated people to communicate the relative riskiness of remaining unvaccinated
- Adding date-range options to most charts so Californians could adjust time frames when charts were visually crowded with more than two years’ worth of data
- Making improvements to health equity charts to make it easier to understand the disparate impact of COVID-19 on communities and the state’s response to vaccinate and meet the needs of state residents equitably
In June 2022, ODI transitioned the COVID-19 site to the California Department of Public Health (CDPH). ODI advised CDPH on procuring the necessary talent. The new CDPH engineer and content designer embedded with ODI from April through June 2022 for training. The plan included continued significant post-handoff engineering and content design consulting through the second half of 2022 to assist CDPH in getting up to speed. Though ODI continues to host and maintain underlying services that support the COVID-19 site, all day-to-day activities are now maintained by CDPH.
State website COVID-19 survey
In 2020-21, in partnership with the California Office of Emergency Services (CalOES), ODI designed, fielded, and managed a voluntary survey to gather data directly from Californians about their attitudes, needs, concerns, actions and inactions around COVID-19 non- pharmaceutical interventions and vaccines.
The survey was placed on 12 of the most visited state websites and ran continuously from January 28 through September 8, 2021. The survey gathered 144,024 structured responses and 124,269 open-ended responses. ODI applied standard survey methodology to the respondent sample to make it reflect the state’s demographic distribution. This approach of tapping into state website traffic to gather voluntary data from Californians is an innovative, cost-effective, easily replicable way to gather real-time data, at scale, about any issue.
Vaccine response research, data, and analytics
At the request of CalOES, ODI leveraged the data collected by the state website COVID-19 survey in 2020-21 to inform the state’s equity-based, geo-targeted vaccine distribution. As part of this effort, ODI developed and delivered multiple products to help identify and address friction points in vaccine uptake, including:
Vaccine hesitancy ZIP code scoring model
There were many unvaccinated Californians in underserved communities that, through our research, we found had a barrier to access. We used this research and data to improve their access dramatically.
ODI built a vaccine sentiment scoring model that allowed decision makers to find locations where vaccination uptake was low but willingness was high. This model leveraged multiple data sets in addition to the state website COVID-19 survey data, including Healthy Places Index, Census, digital engagement data from different ca.gov websites including myturn.ca.gov, and vaccination rates data. The final product provided an understanding of the hesitancy and willingness statewide at the county, city and zip code levels. The data was regularly validated against the Carnegie Mellon/Facebook COVIDCast data set. ODI delivered bi-weekly data updates with the newest scores to stakeholders.
Qualitative analysis dashboard
ODI built out an analyst-level dashboard that allowed analysts to explore and quantify the open-ended data in relation to key factors like geography, race/ethnicity, and age. It was used by Third-Party Administrator (TPA) analysts and ODI to provide deeper insight into the most common concerns and barriers to getting vaccinated.
Topic modeling dashboard
ODI used natural language processing, sentiment analysis, and topic modeling to analyze parts of speech, sentiment, and “toxicity” of these data. This allowed us to identify the dominant topics, the sentiment around those topics, and changes as they occurred.
These products were used by multiple stakeholders:
- TPA analysts and local health jurisdictions used the qualitative and scoring data to help inform mobile and pop-up vaccine deployment.
- Duncan Channon, the Vaccinate All 58 ad agency, used the scoring data to help inform ad targeting and message development.
- The COVID Outreach Rapid Deployment tool, used by state-contracted community- based organizations, used the scoring data to help inform direct engagement.
California Design System
There are over 300 active California state government websites. While there is some standardization and consistency, the user experience varies significantly from site to site.
To help departments move toward user-focused website and service design and improve the user experience, ODI led a collaborative effort with the Department of Technology (CDT) to launch the first version of the California Design System in February 2022.
A design system is a set of tools and resources established to facilitate the creation of modern, accessible, and fast websites. The Design System builds on ODI’s alpha.ca.gov project and CDT’s Website Standards and State Template. It offers scalable, reusable tools that departments can use to improve access to services. The Design System is the base for new websites ODI builds.
The Design System both informs and is informed by the work ODI does to develop new websites. For example, features and improvements from drought.ca.gov were fed back into the Design System. As a result, drought.ca.gov made the Design System a better service for all state departments.
In addition to technical assets like a library of performant components, ODI also created and shared content design and user research principles and guidelines.
ODI transitioned ongoing maintenance of the Design System to CDT in June 2022. ODI continues to support CDT’s rollout of the Design System to replace the State Template. In October 2022, ODI and CDT won Best IT Collaboration at the Digital Government Summit in Sacramento for the Design System.
Cannabis website (cannabis.ca.gov)
While continuing to collaborate with the Department of Cannabis Control (DCC) on improving their digital services, ODI is simultaneously building their capacity in a way that accelerates ODI’s mission.
Building on the California Design System, as well as leveraging reusable, scalable tools and assets that ODI developed for other projects like covid19.ca.gov, and working in partnership with Department of Technology (CDT) and DCC, ODI built and launched cannabis.ca.gov in just two months (May through June 2021).
Prior to the creation of DCC and the new website, people had to visit multiple websites to find information on cannabis licensing or regulations. Information was often duplicative and confusing and finding definitive answers was difficult. The process and technology was complicated and not serving end user needs.
The new site provides a new, one-stop location for all California-specific cannabis information by bringing together and distilling information from the Department of Public Health, the Department of Food and Agriculture, the Bureau of Cannabis Control, under the banner of the newly-created DCC.
In April 2022, ODI collaborated with DCC to launch an interactive map. This map clarifies what types of cannabis businesses are allowed in cities, counties, and unincorporated areas within California. The visualization loads quickly, is easy-to-use, and is mobile-friendly.
In October 2022, ODI migrated cannabis.ca.gov’s backend system from CDT’s standard publishing stack to ODI’s publishing stack. This change resulted in a 75% reduction in time to interaction on low-end mobile devices. This technology pattern change improves experiences for website users in low-bandwidth circumstances.
The upgrade:
- Included system improvements from covid19.ca.gov, drought.ca.gov, and abortion.ca.gov
- Enhanced security
- Sped up user experience design, flexibility, and responsiveness to the needs of the cannabis industry
Drought website (drought.ca.gov)
ODI built and launched drought.ca.gov in less than a week in July 2021 by leveraging the Design System and lessons learned from alpha.ca.gov and covid.ca.gov.
By deploying best in class software engineering practice, ODI created a faster, more accessible, and flexible site than it could have using traditional tools. This resulted in the site ranking third in a speed test of 270 state websites.
After launch, ODI spent months training web teams from the Department of Water Resources and the California Environmental Protection Agency in human-centered content design. This led to a successful transfer of site ownership in June 2022. With this training and support, ODI ensured that Californians can continue to easily find current drought facts and know what they can do to save water.
Abortion access website (abortion.ca.gov)
abortion.ca.gov was developed in just over a month (August – September 2022) at the request of the Office of the Governor. It provides anonymous, secure abortion information. ODI accomplished this by building the site without integrating any third-party applications such as maps or analytics.
The development also included a public workshop to get user feedback. This grounded the site in demonstrated user needs. As a result, abortion.ca.gov provides useful guidance and assistance that people can understand and use to take action.
ODI is currently working with the Department of Technology (CDT) and the California Department of Public Health (CDPH) on new releases to continue to improve the site. This project is in the process of being handed off to CDPH for ongoing maintenance and improvements.