Legislative Report: July 1, 2023-July 1, 2024

Overview

The Office of Data and Innovation (ODI) was established on July 1, 2019, as an office within the Government Operations Agency (GovOps) and became a standalone department on July 1, 2023. This year’s legislative report showcases the progress made during the 2023-24 fiscal year (FY).

ODI’s mission and vision statement are: 

• MISSION: “We drive innovation across California state government, using data, technology, and human-centered design to achieve exceptional, equitable outcomes for all Californians. We collaborate with leaders and communities throughout California and the world. We partner to build empathy for the people we serve and create a government that is easy to interact with and solves big problems.”

• VISION: “A state government that is modern, effective, and provides exceptional service to all Californians.” 

From its creation, the department prioritized building strong relationships across state government and leveraged these partnerships to improve services to Californians. In FY 2023-24, ODI launched initiatives that allowed for transformational work through training, human-centered design, service innovation, and data analysis with the purpose of improving service delivery for all Californians. The department houses a multi-disciplinary staff with expertise in engineering, user research, analytics, content design, training, talent, engagement, and communications. 

To fulfill the department’s mission, ODI established a set of strategic goals:

• Develop a statewide culture of innovation that drives efficiency and improves customer experience.
• Establish statewide standards to transform service delivery, increase government efficiency, and improve outcomes.
• Enable departments to adopt the tools and practices that support continuous improvement and performance measurement.

ODI is positioned to lead statewide initiatives in partnership with GovOps, the Governor’s Office, and state departments to collaborate on scalable solutions to improve delivery of vital state services. ODI partners with departments to improve their service delivery by introducing modern tools and training them in the skills necessary for innovation. This innovation leads to streamlined processes and drives government efficiency, resulting in a better customer experience for those the state serves. ODI sets statewide standards for data use and service delivery to minimize risk and increase product success to ensure solutions effectively meet Californian’s needs.

Activities of the Office Fiscal Year 2023-24

ODI’s activities during this FY fall into two main categories: statewide strategic initiatives and projects with partner departments. 

Our statewide strategic initiatives are cross-departmental efforts that require collaboration between multiple state entities. This work is ODI’s opportunity to drive government transformation at scale through our work helping state government entities improve their service delivery by embedding a human-centered approach in statewide standards, policy, programs, and digital services. 

We also partner directly with departments to perform problem definition, discovery, user research, prototyping, and piloting of innovative solutions to address customer issues in their service delivery process. These projects are funded by the Data and Innovation Services Revolving Fund (DISRF), known as the Data and Innovation Fund (DIF).

Statewide Strategic Initiatives

Implementing the Generative Artificial Intelligence (GENAI) Executive Order 

In September 2023, Governor Gavin Newsom issued Executive Order EO N-12-23 (GenAI EO) requesting multiple departments across the state to author a first-of-its-kind benefits and risks analysis on GenAI use in state government. 

ODI was a key contributor to the Benefits and Risks of the Generative Artificial Intelligence Report published November 2023, alongside GovOps, the California Department of Technology (CDT), the Department of General Services (DGS), and the Governor’s Office of Business and Economic Development (GO-Biz). This team of departments researched and reported on the beneficial use cases and potential risks of GenAI. ODI expertise played a key role in both the community engagement efforts, as well as the research and drafting of this historic report. The GenAI EO and its directives showcased California’s leadership in technological advances, positioned the state as pioneers in innovation, and established guardrails and engagement needed before any consideration or use of GenAI by state government entities.

Additionally, as part of the GenAI EO, ODI was individually responsible for making a human-centered toolkit to enable state teams to operationalize the new processes to procure GenAI. ODI developed and implemented product management principles in the creation of GenAI Proofs of Concept (POC), and to ensure business and user value is both delivered and considered in the overall evaluation prior to a potential procurement.

Developing Data and Service Delivery Standards

Due to ODI’s expertise in human-centered design and data, our office was specifically tasked in Governor Newsom’s Executive Order N-16-22 (EO) with developing statewide standards for collecting race, ethnicity, and other demographic data and for developing a set of service delivery standards to identify gaps and improve reach of programs and services to eligible Californians, among other actions.  

Delivering these data standards and service delivery recommendations for the EO will help improve equity across the state. Adopting these best practices enables improved access for Californians to state services and promotes consistent data practices across state agencies and departments for greater efficiency and collaboration.

Demographic Data Standards

ODI’s work in this critical area began in FY 2022-23 and continued in FY 2023-24, with a focus on alignment with the federal data standard, user research, community engagement, and user testing efforts to refine the standards to better serve California’s diverse population.

Race and Ethnicity Data Standard

ODI developed recommendations for a data standard on race and ethnicity data collection and analysis. The standard aims to ensure consistent practices across state departments and support the development of metrics and programs. ODI’s recommendations incorporate best practices and critical stakeholder engagement, while addressing California’s unique diversity and privacy considerations. To gather feedback, ODI partnered with GovOps to conduct community roundtables, consultations, and interviews with academic experts.

Plain Language Standard

In FY 2022-23, ODI recommended use of plain language as the foundation for successful communication and service delivery to Californians, and recommended that state entities use an 8th grade reading level or below as a base for the written communications. ODI identified that plain language: 

• Increases access and usability for all Californians, across literacy levels and for different languages.
• Works better with screen readers.
• Respects the mental and emotional state of those seeking services during a time of crisis.

In FY 2022-23 ODI’s work on the plain language standard included:

• Creating an Innovation Hub with resources on human-centered design and guidance for state entities to implement plain language in their communications  to support broader state adoption of this plain language standard. 
• Launching a Plain Language course as a first initiative for this effort. This course is offered free of charge to all state employees on the California Department of Human Resources (CalHR’s) Learning Management System (CalLearns) training platform. The course provides hands-on practice in applying the principles of ODI’s plain language standard.

In FY 2023-24 ODI’s work included:

• Hosting the Introduction to Plain Language for the Public Sector on CalLearns. This interactive training program provides a guide to plain language writing for the public sector, prioritizing readers’ needs and Californians’ access to information. It also allows time to practice writing in plain language to apply learnings quickly, and supports state entities meeting the requirements of EO N-16-22.
  • This course began with a live course to allow ODI to test different training methods and refine the content. 
  • By the end of FY 2023-24, the training had a completion rate of more than 91% , and 486 state staff and leadership had taken the Introduction to plain language for the public sector training.
  • Training participants have shared strong positive feedback in relation to the value of the information learned through the training.
    • Hosting 4 executive level webinars on “plain language” inviting all directors and deputies in state service to participate. 
  • Executive webinars had a total attendance of 732 state leaders.  
  • Executive webinars were hosted by ODI’s Director Jeffery Marino and the State Chief Equity Officer Pam Chueh.

Data and Innovation Fund (DIF) Partner Projects

ODI is legislatively mandated to lead innovation and statewide transformation in service delivery. The DIF was funded to foster collaboration between ODI and state agencies, allowing for innovative solutions that improve services to Californians. DIF-funded engagements are unique because of their people, process, innovation, and data-focused problem statements. ODI builds pilots, tools, and resources for state agencies and departments partnering with ODI through the DIF

ODI launched the first DIF project solicitation in June 2022. These projects have improved state government service delivery for Californians’ through service innovation, increased social safety net program uptake, and improved access to historically underrepresented Californians. Additionally, these projects have also reduced data errors, increased state departments ability to mitigate potential service delivery challenges, reduced fraud, and expanded access to social benefits. By leveraging technology, data analytics, user research and human center design, ODI is transforming how the state government delivers services to the people of California.

How the Data and Innovation Fund Works

Applications to the DIF require a competitive process reviewed by an impartial Advisory Committee, and final selection by a Selection Committee. The DIF Selection Committee prioritizes projects with the greatest potential return on investment, and the potential to scale across departments to accelerate innovation in the state.

Services Offered through the Data and Innovation Fund

ODI’s program staff are experts in many disciplines. Through a series of pre-application webinars and office hours, ODI educates potential DIF partner applicants about the services ODI offers and which may be best suited to their unique problem(s). ODI services include:

CalData

• The Data Science Accelerator (DSA) helps departments apply traditional artificial intelligence (e.g., machine learning, predictive algorithms) to resolve problems proactively, quickly, or at a reduced cost.
• The Modern Data Stack Accelerator (MDSA) helps departments rapidly adopt modern cloud-based data tools while working on a data challenge they want to solve.

CalInnovate

• Problem definition: ODI helps departments identify solutions and mitigate risk by providing support in accurately defining problems and identifying solutions before technical development or procurement begins. 
• Service design: ODI helps departments understand their customer’s needs in order to help them improve their services through research, data, technology, and design.

DIF Completed and Ongoing Projects, FY 2023-24

CalData Completed Projects

California Department of Housing and Community Development (HCD)

• Project Name: Automated Mining and Analysis of Housing Data
• What we did: Built a tool to mine, analyze, and visualize housing progress as submitted by 539 jurisdictions on an annual basis.

• Outcome:
  • Helps HCD meet legislative requirement AB 2653 which requires HCD to request corrections to reports from jurisdictions within 90 days.
  • Saves HCD nearly 2,000 manual review hours every year. 
  • Provides new visibility into housing costs by county, income level, and unit size.
  • Calculates the timeline for each housing project’s time-to-build. 
  • Tags housing projects as eligible or ineligible for SB 423 streamlining.

State Water Resources Control Boards (SWRCB)

• Project Name: Forecasting Community Water System Outages
• What we did: Built a machine learning model to forecast which community water systems are at risk of running out of water so that preventative measures can be taken.
• Outcome:
  • A forecasting model that is easy to use, flexible and interpretable.
  • This tool empowers staff to take prevention and intervention measures faster for impacted communities.

California Air Resources Board (CARB)

• Project Name: Streamlined Data Collection for Enforcement Annual Report
• What we did:  CARB’s Enforcement Division is tasked with enforcing air quality regulations. ODI helped CARB develop a cloud data warehouse and analytics engineering practices to support reporting and operations.
• Outcome:
  • Previously, compiling and analyzing data for reporting requirements took dozens of staff and months of time. Now data integration, transformation, and reporting is automated, and occurs continuously.

California Bureau for Private and Postsecondary Education (BPPE)

• Project Name: Prioritizing Inspections & Investigations for Private Postsecondary Institutions
• What we did: Developed a prioritization tool that allows the Bureau to monitor institutional risk factors so BPPE can quickly respond to emerging issues. 
• Outcome:
  • This tool Improves educational quality and student outcomes by ensuring institutions meet and maintain high standards of performance

California Department of Human Resources (CalHR)

• Project Name: Data and Analytics Enablement
• What we did: ODI worked with CalHR to build an automated data warehouse allowing CalHR to surface up-to-date data to the staff that analyze data from CalCareers to answer strategic questions about demographics, hiring timelines, positions, and exams. Previously, this process required going to an information technology (IT) staff member with highly specialized knowledge of the system. Reports would take days or weeks to complete, leading to analyses based on incomplete or out-of-date information. 
• Outcome:
  • Analyzing CalCareers data is no longer bottlenecked on a single person.
  •  CalHR data analysts can self-serve questions that they need to answer for strategic or statutory reasons.
  • Reports that took weeks can now be run ongoing. 
  • Soon, departments will be able to access their own hiring data to self-serve analyses.

Department of Finance (DOF)

• Project Name: Building footprints data product
• What we did: ODI worked with the Demographics Research Unit at DOF to develop a derived data product based on the Microsoft product that is more cloud-native and has useful census data already attached for further demographic modeling.
  • DOF’s Demographics Research Unit provides demographic estimates for small areas statewide to support resource allocation, research, and analysis. 
  • A primary input of this process is a statewide geospatial building footprints dataset published by Microsoft. 
  • Historically, the dataset has been difficult to work with. 
• Outcome:
  • The process which traditionally took days to complete, can now be completed in minutes with this new product. 
  • This faster turnaround time on analysis allows more small area estimates to be conducted, and provides staff more time to conduct new analyses.

CalData Ongoing Projects

California Department of Tax and Fee Administration (CDTFA)

• Project Name: Improving Voluntary Tax Compliance using Data Science
• What we did: Built a machine learning model for identifying irregularities on sales and use tax returns. 
• Outcome:
  • The team is experimenting with using the model for their voluntary compliance letter selection, an important first step in modernizing and centralizing CDTFA’s current process.

California Department of Transportation (Caltrans)

• Project Name: Performance Measurement System (PeMS) Modernization
• What we did: Modernize the data infrastructure for Traffic Operations’ PeMS system, which measures traffic flow on all California state highways at 30-second resolution.

CalInnovate Completed Projects

California Department of Human Resources (CalHR)

• Project Name: CalHR Modernization 
• What we did: This project is part of a multi-phased, ongoing collaboration with CalHR in support of its Modernization project, using research and human-centered design to understand and improve the candidate journey and staff experience by enabling user experience (UX) improvements to CalCareers job filter and search, job postings, content design and technical support for enabling social media communications.
• Outcome:
  • ODI supported CalHR’s research, development, procurement, and implementation of a new tool to help meet the requirement for an anonymous hiring system in the Examination and Certification Online System (ECOS) for job applicants, per EO N-16-22.
  • These efforts substantially contributed to an improved candidate experience with a goal to increase the size of the candidate pool, the quality of candidates, and make it easier for candidates to find and apply for state jobs. 
  • This ODI engagement with CalHR was in support of Governor Newsom’s EO N-16-22.

California Department of Human Resources (CalHR)

• Project Name: CalHR Anonymous Hiring
• What we did: Collaborated with CalHR to conduct discovery, establish problem definition, and implement a solution to meet EO N-16-22’s requirement to implement a process for anonymizing candidate data. This work included:
  • Conducting market research to understand existing tools.
  • Conducting user research to understand user needs among both staff and candidates.
  • Developing requirements for the anonymous hiring tool. 
  • Conducting a vendor demo day to test and evaluate vendors’ ability to meet user needs and technical requirements.
  • Developing a procurement scope.
  • Assisting with implementation of the new tool. 
  • Building an API to enable Integration of the third-party Anonymous Hiring tool allowing for data redaction.

Department of Toxic Substances Control (DTSC)

• Project Name: Public Notice
• What we did: Helped DTSC understand how communities get information through research on relevant and helpful information in different California communities.
• Outcome:
  • This engagement resulted in a 633 percent increase in public comment participation and identified that 81 percent of comments were made on a mobile device.
  • As a direct result from this partnership, DTSC is now able to target specific geographic and demographic areas through the use of a social media outreach approach.
  • Plain language and content design refresh helps people better understand the work being done in their communities.

California Department of Emergency Services (Cal OES)

• Project Name: Cal OES Human-centered Design 
• What we did: ODI worked with Cal OES to develop human-centered design capability and capacity in their processes by focusing on business analysis and improvements. This work included making essential information collection and reporting more effective and efficient for the department to achieve their strategic goals.

California Department of Technology (CDT)

• Project Name: Digital ID
• What we did: ODI supported CDT’s Digital ID gateway team through user research to assist in research-informed opportunities and pilot testing of UX design.

California Department of Technology (CDT)

• Project Name: Project Approval Lifecycle
• What we did: Conducted user research in support of CDT to track the Project Approval Lifecycle (PAL) and provide return on investment data as part of CDT’s broader Project Services Modernization (PSM) effort.

CalInnovate Ongoing Projects

Benefits Recommender

• What we are doing: This is a unique collaboration across state departments that is designed to close the gap between Californians eligible for benefits programs and those actually receiving benefits. California is a key state leading the nation in social benefit program updates and best-in-class experience.
  • The Recommender widget is an ODI-hosted, open-source, low-code Application Programming Interface (API) placed on the confirmation web pages for a select group of benefit programs. It drives successful benefit applicants to additional resources and benefits within the state that they may be eligible for, with direct links to other relevant online applications. 
  • ODI’s main partner in this project is the Employment Development Department (EDD). ODI, in collaboration with EDD, has conducted stakeholder outreach with various other departments administering California benefit programs. The initial rollout was in late 2023 with an aim to scale the Recommender and release additional functionality in 2025. 
  • The pilot phase resulted in 2.1 million Recommender views, with an average click-through rate of 8.5%, resulting in over 3,200 submitted applications.

California Department of Public Health (CDPH)

• What we are doing: ODI created a website for California abortion access. This website will be taken over by CDPH in the near future. Currently, ODI completes monthly analytics reports, security updates and updates the lists of clinics available.

California Employment Development Department (EDD)

• What we are doing: ODI is helping EDD make immediate improvement possible by enabling EDD to measure the impact improvements to the Unemployment Insurance Online (UIO) customer experience through the application of user journey analytics. This work includes:
  • Supporting the EDDNext initiative by partnering with the UIO team to develop and apply user journey analytics (UJA) to inform improvements to the customer experience materially and measurably. 
  • Conducted a series of six problem definition and scoping workshops.
  • Developed the underlying tools needed to access user journey analytics and connect them to the UIO program.
  • Developed dashboards to allow the UIO team to see, access, measure, and better understand the data.
  • Assisted to improve the internal UJA “Report Request Form” to ensure that UJA is accessible to staff department-wide.

DIF Expenditures

DIF Fiscal Year 2023-24 Expenditures

The following expenditures were made from the DIF in FY 2023-24:

DIF Fiscal Year 2023-24 Encumbrances

To accomplish current and anticipated DIF projects, ODI entered into the following contracts using DIF funds. The following encumbrances reflect outstanding commitments of DIF funds by ODI in FY 2023-24:

DIF Fiscal Year 2023-24 Revenues

The following are FY 2023-24 DIF revenues:

$28 Million at beginning in FY 2023-24 (Includes amount above).

Appendix A: List of Past and Current Engagements Organized by State Entity

Pursuant to California Government Code section 12815(o)(2)(A), below is, “A list of past and current engagements organized by state entity.” ODI has partnered with 32 state departments and agencies on more than 60 projects.

• California Air Resources Board
  • Streamlined Data Collection for Enforcement Annual Report
• California Bureau for Private and Postsecondary Education (BPPE)
  • Prioritizing Inspections & Investigations for Private Postsecondary Institutions
• California Business, Consumer Services, and Housing Agency
  • Homelessness data story
  • Identifying core dataset needs
• California Department of Housing and Community Development
  • Automated Mining and Analysis of Housing Data
• California Department of Public Health
  • Abortion.ca.gov
  • Building human-centered design capacity across the state
  • Cannabis.ca.gov
  • Covid19.ca.gov
  • Human-centered procurement consulting
• California Department of Social Services
  • Benefits Recommender identifier 
  • CalWORKs discovery sprint
• California Department of Tax and Fee Administration (CDTFA)
  • Improving Voluntary Tax Compliance using Data Science
• California Department of Technology
  • Abortion.ca.gov
  • Ethical AI Framework and Toolkit
  • California Design System
  • Cannabis.ca.gov
  • Content design training
  • Master state entity list
  • Open data
  • Project Approval Life Cycle
  • Shared dataset development
  • ODI publishing system
  • Technology, Digital and Data Consulting Master Services Agreement
• California Department of Transportation
  • Credit card acceptance and fee reduction
  • PeMS Modernization
• California Environmental Protection Agency
  • Drought.ca.gov
  • Identifying core dataset needs
• California Health and Human Services Agency
  • Data career fair and teams microsite
  • Interagency Data Exchange Agreement
  • Benefits Recommender (CalWORKs, WIC referrals)
• California Public Utilities Commission
  • Emergency Broadband Benefit user research
  • Human-centered procurement consulting
  • Lifeline website redesign
• California State Transportation Agency
  • Identifying core dataset needs
• California Victims Compensation Board
  • Reaching and helping victims of crime
  • Website analytics training
• Department of Cannabis Control
  • Building human-centered design capacity across the state
  • cannabis.ca.gov
  • Website analytics training
• Department of Finance
  • Master state entity list
• Department of Fish and Wildlife
  • Shared dataset development
  • Building footprints data product
• Department of Food and Agriculture
  • cannabis.ca.gov
• Department of General Services (DGS)
  • Master state entity list
  • Small business procurement user research
  • Technology, Digital and Data Consulting Master Services Agreement
  • DGS Office of Public School Construction – evaluation research and training
• Department of Human Resources
  • Master state entity list
  • Modernizing data jobs classifications
  • PowerBI training
  • State human resources modernization initiative
  • Anonymous Hiring
  • Data and Analytics Enablement
• Department of Parks and Recreation
  • Shared dataset development
• Department of Toxic Substances Control
  • Content design training
  • Public comment pilot project
• Department of Water Resources
  • Drought.ca.gov
• Employment Development Department
  • Benefits recommender
  • EDDNext
• Government Operations Agency
  • Credit card acceptance and fee reduction
  • Executive Order N-16-22
  • Recruitment innovations
  • Small business procurement user research
  • Technology, Digital and Data Consulting Master Services Agreement
• Governor’s Office of Business and Economic Development
  • Credit card acceptance and fee reduction
• Governor’s Office of Emergency Services
  • Human-centered design
  • State website COVID-19 survey
  • Vaccine response research, data, and analytics
• Labor & Workforce Development Agency
  • Evaluation research and training
• Office of the Governor
  • Abortion.ca.gov
  • Benefits Recommender
  • Covid19.ca.gov
  • Drought.ca.gov
  • Homelessness data story
  • State human resources modernization initiative
• Secretary of State
  • Master state entity list
• State Controller’s Office
  • Master state entity list
• State Personnel Board
  • State human resources modernization initiative

• State Water Resources Control Board
  • Forecasting Community Water System Outages
  • Ethical AI Framework and Toolkit
  • Shared dataset development