Standards for public agencies

2020 Edition

Performance and Quality Improvement (PA-PQI) 3: Measures and Indicators

The agency identifies and utilizes measures and indicators for evaluating the following within the agency and with any contracted providers: 
  1. the impact of services on individuals and families; 
  2. the quality of service delivery; and
  3. management and operational performance.
2020 Edition

Currently viewing: PERFORMANCE AND QUALITY IMPROVEMENT (PA-PQI)

VIEW THE STANDARDS

Purpose

 An agency-wide performance and quality improvement system effectively engages staff, persons served, and other stakeholders in advancing the agency’s mission and achieving strategic goals through continuous, integrated, data-driven efforts to improve service delivery and administrative practice. 
Examples: Measures, indicators, and tools required by regulation can be utilized to go beyond measuring compliance by engaging staff and other stakeholders to: 
  1. review data that is important for their work or interest;
  2. use data to benchmark results with other agencies providing the same funded services; or
  3. compare data with other data collected by the agency that is not covered by contractual requirements in order to improve services.
Examples: Agencies providing child and family services are encouraged to integrate the Child and Family Services Review (CFSR) Outcome Measures and Systemic Factors, particularly those identified in Performance Improvement Plans, into their overall PQI system. 
 
1
Full Implementation, Outstanding Performance
A rating of (1) indicates that the agency's practices fully meet the standard and reflect a high level of capacity.  
  • All elements or requirements outlined in the standard are evident in practice, with rare or no exceptions: exceptions do not impact service quality or agency performance. 
2
Substantial Implementation, Good Performance
A rating of (2) indicates that an agency's infrastructure and practices are basically sound but there is room for improvement.
  • The majority of the standards requirements have been met and the basic framework required by the standard has been implemented. 
  • Minor inconsistencies and not yet fully developed practices are noted; however, these do not significantly impact service quality or agency performance.
3

Partial Implementation, Concerning Performance
A rating of (3) indicates that the agency's observed infrastructure and/or practices require significant improvement.  

  • The agency has not implemented the basic framework of the standard but instead has in place only part of this framework.  
  • Omissions or exceptions to the practices outlined in the standard occur regularly, or practices are implemented in a cursory or haphazard manner.  
  • Service quality or agency functioning may be compromised.  
  • Capacity is at a basic level.
4
Unsatisfactory Implementation or Performance
A rating of (4) indicates that implementation of the standard is minimal or there is no evidence of implementation at all.  
  • The agency’s observed administration and management infrastructure and practices are weak or non-existent; or show signs of neglect, stagnation, or deterioration.
Self-Study EvidenceOn-Site EvidenceOn-Site Activities
County/Municipality Administered Agency, State Administered Agency (Central Office), or other Public Entity
  • See PQI plan (PA-PQI 1) for a description of what is being measured, including:
    1. outcomes measures
    2. outputs
    3. data sources 
    4. performance indicators
    5. performance targets 
  • See outcomes information provided in the Client-Centered Logic Model Core Concept in each Service Standard
  • Agencies seeking re-accreditation only
State Administered Agency (Regional Office)
  • See Regional PQI plan for region-specific measures or indicators, if appropriate 
  • See outcomes information provided in the Client-Centered Logic Model Core Concept in each Service Standard
  • Agencies seeking re-accreditation only
All Agencies
  • Documentation of staff/stakeholder involvement in ongoing review of measures, indicators, data sources, and performance targets
  • Regulatory/licensing or other external reviews/reports 
  • Documentation that COA Stakeholder Surveys were distributed (e.g. email chains, Stakeholder Survey Recipient Reporting Form, etc.)
All Agencies
  • Interviews may include:
    1. PQI staff
    2. Relevant staff
    3. Other relevant stakeholders

 

PA-PQI 3.01

Staff throughout the agency and stakeholders, including contracted providers, participate in the ongoing review of outputs and outcomes, and related:
  1. quantitative and qualitative indicators;
  2. data sources including measurement tools and instruments for each identified measure; and
  3. performance targets.

Interpretation

Agencies are encouraged to use standardized or recognized outcomes evaluation tools when such tools are available and appropriate. Functional assessments permit the analysis of an individual or family’s status over time and, in the aggregate, this case-level data can inform the analysis of trends and relationships to correlating service delivery components.

Interpretation

Program outputs and client outcomes must be identified in the logic model submitted in the Client-Centered Logic Model Core Concept in each assigned Service Standard. 

Interpretation

Agencies should assess variation in service population, service area, staffing and other factors in order to develop baselines, performance targets, and benchmarks that are tailored to the local area or program.
Examples:
Outputs are what the program delivers. Examples of program outputs include:
  1. number of educational or clinical sessions provided;
  2. total number of clients served over a specified period of time; and
  3. number of housing placements made.
Outcomes are the observable and measureable effects of a program's activities on its service recipients. Examples include:
  1. improved functioning as measured by the Children's Functional Assessment Rating Scale (CFARS); 
  2. number/percent of homeless and runaway youth that are reunited with family during past quarter;
  3. reduction in criminal justice system involvement; and
  4. improved family/community involvement.

 

PA-PQI 3.02

To evaluate the quality of its service delivery practices, the agency identifies and uses outcome measures related to the following:
  1. training and supervision of program staff; and
  2. consumer satisfaction.

 

PA-PQI 3.03

To evaluate management and operational performance, the agency identifies and uses outcome measures across the agency, and with contracted providers when applicable, to:
  1. measure progress toward achieving its strategic goals and objectives;
  2. evaluate operational functions that influence service delivery; and
  3. identify and mitigate risk.
CFS

Interpretation

 For child and family services agencies, implementation of this standard includes an examination of relevant systemic factors assessed by the Child and Family Services Reviews (CFSRs).
Examples: Examples of outcome measures related to operations and management can include:
  1. efficiency in the allocation and utilization of its human and financial resources to further the achievement of agency objectives;
  2. effectiveness of risk prevention measures;
  3. staff retention/turnover and satisfaction;
  4. service delivery costs versus benefits derived by persons served;
  5. achievement of budgetary objectives;
  6. effectiveness of public education and outreach; and
  7. staff fidelity to the process and quality standards set by the agency.
Examples: Agencies that use contracted providers may also measure important contract oversight and system integration processes, such as: 
  1. the proportion of services that are meeting defined outcomes for persons served;
  2. the proportion of services that are evidence-based or meet nationally recognized treatment guidelines developed by consensus groups; 
  3. the integration of performance and outcomes data across the system;
  4. the integration and coordination of service provision processes across the system including ease of access to services;
  5. the effectiveness of contractor training and technical assistance efforts; 
  6. the satisfaction of stakeholders, such as high volume referral agents (e.g., judges, court workers, schools, and law enforcement); and
  7. results of case reviews, including the percentage of charts in which a placement decision includes an appropriate application of clinical criteria.

 

PA-PQI 3.04

Findings and recommendations from external review and monitoring processes are integrated into the organization’s PQI system.

Interpretation

When agencies are involved in litigated third-party oversight, such as consent decrees, strategic plans and PQI plans (agency-wide plans or jurisdiction-specific plans) should indicate how the overall PQI system balances pursuit of compliance with the larger quality improvement agenda.

CFS

Interpretation

For child and family services agencies, the PQI system must incorporate the findings of the Child and Family Services Review and support implementation of the strategies outlined in its Program Improvement Plan.
 
Examples: External review and monitoring processes can include:
  1. reviews related to federal, state, and local requirements;
  2. litigated third party oversight, including consent decrees;
  3. government audits;
  4. accreditation reviews; and 
  5. other reviews, where appropriate.