top of page
Search

IAPP AIGP, study guide domain iv

  • Writer: Ley Muller
    Ley Muller
  • 7 days ago
  • 17 min read

Domain iv: how to govern AI deployment & use

Domain IV.a Deployment decisions- factors and risks

Domain IV.b - key activities to assess the AI model


  

  

Domain IV.c govern the deployment and USE of the AI model

 

  • Apply the policies, procedures, best practices, and ethical considerations to the deployment of the AI model

    •  Data governance

    •  Risk management

    •  Issue management

    •  User training

  •  Continuous monitoring.

    •  Model Performance - whichever metrics are applicable

      •  Accuracy, precision, recall, F1 score

      •  Prediction confidence scores

      •  Error rates and types

      •  Performance across different subgroups (demographic parity)

      •  Model Drift from baseline metrics. Performance in real-world, production settings.

        •  Data draft = input data distribution is changing

        •  Prediction drift // model inference = changes in model output distribution

        •  Concept drift = changes in the underlying data distribution that degrade model performance

        •  Data drift = changes in the input data characteristics that the model wasn’t trained on → model’s predictive capabilities get worse

          •  Could be due to new data sources, new data collection methods, or actual population changes

        •  Emerging biases = new unfair or discriminatory behavior

    •  Fairness and bias

      •  Demographic parity, disparate impact metrics

      •  Subgroup performance analysis

      •  Intersectional fairness

      •  Feedback loops and bias amplification detection

    •  Data Quality:

      •  Input data distribution shifts

      •  Missing or corrupted data

      •  Data anomalies

      •  Feature importance changes

    •  System Health:

      •  Response times and latency

      •  Throughput and capacity utilization

      •  Error rates and system failures

      •  Resource consumption

    •  Security monitoring

      •  Adversarial attack detection

      •  Data poisoning indicators

      •  Model extraction attempts

      •  Prompt injection and jailbreaking (for LLMs, agents)

      •  Unauthorized access and use

      •  Anomalous behavior patterns

    •  Monitoring Frequency:

      •  Real-time monitoring for critical applications

      •  Daily or weekly for moderate-risk systems

      •  Monthly for lower-risk applications

      •  Event-triggered monitoring for specific scenarios

  •  And make a regular schedule for maintenance, updates, and retraining.

    •  Model Updates:

      •  Scheduled retraining frequency

      •  Trigger thresholds for retraining (e.g., performance degradation)

      •  Version control and rollback procedures

      •  A/B testing new model versions

    •  Software Updates:

      •  Security patches

      •  Dependency updates

      •  Infrastructure upgrades

      •  Compatibility testing

    •  Retraining Considerations:

      •  New training data availability

      •  Concept drift detection

      •  Regulatory requirement changes

      •  Business requirement evolution

  •  Periodic assessments - performance, reliability, safety

    •  Audits:

      •  Internal audits of AI systems

      •  External third-party audits

      •  Compliance audits (regulatory requirements)

      •  Technical audits (model performance)

      •  Process audits (governance adherence)

    •  Audit Frequency:

      •  Risk-based approach (high-risk: quarterly; low-risk: annually)

      •  Regulatory requirements

      •  After significant incidents

      •  Before major deployments or changes

    •  Red Teaming:

      •  Adversarial testing to find vulnerabilities

      •  Attempt to trigger harmful outputs

      •  Test edge cases and failure modes

      •  Simulate attack scenarios

      •  Cross-functional red team composition

    •  Threat Modeling:

      •  Identify potential threat actors

      •  Map attack vectors and vulnerabilities

      •  Assess likelihood and impact

      •  Prioritize mitigation efforts

      •  Update threat landscape regularly

    •  Security Testing:

      •  Penetration testing

      •  Vulnerability assessments

      •  Input validation testing

      •  Data poisoning resilience

      •  Model extraction attacks

      •  Prompt injection testing (for generative AI)

  •  Documentation Requirements and post-market monitoring

    •  Incident Documentation:

      •  Description of the incident

      •  Classification: severity & type

      •  Affected systems and users

      •  Root cause analysis

      •  Impact assessment

      •  Response actions taken

      •  Lessons learned

    •  Issue Tracking:

      •  Known limitations and bugs

      •  Feature requests

      •  Performance degradations

      •  User feedback

      •  Resolution status and timeline

    •  Risk Documentation:

      •  Risk register maintenance

      •  Risk assessments and updates

      •  Control effectiveness

      •  Residual risk tracking

      •  Risk reporting to governance bodies

    •  Post-Market Monitoring Plans:

      •  Ongoing surveillance activities

      •  Performance metrics to track

      •  Reporting requirements

      •  Stakeholder communication

      •  Continuous improvement process

  •  Forecasting and reducing risks of secondary and intended uses

    •  Forecasting potential misuses

      •  Consult stakeholders

      •  Scenario planning

    •  Preventing secondary use

      •  Model cards! Define clear use cases

      •  Access controls and authentication

      •  Usage monitoring and logging

      •  Terms of service enforcement

      •  Technical controls (rate limiting, watermarking)

      •  User education on appropriate use

    •  Downstream Harms:

      •  Impact of AI outputs on downstream decisions

      •  Cascading failures across systems

      •  Misuse of AI-generated content

      •  Reputation damage

      •  Societal impact

      •  Preventing downstream harms:

        •  Supply chain assessment, customer due diligence, contractual restrictions on use, audit rights and monitoring, termination provisions for mis-use

    •  Risk reduction Strategies:

      •  Output validation and verification

      •  Human-in-the-loop review

      •  Confidence thresholds

      •  Clear disclaimers and limitations

      •  Sandboxing and testing environments

      •  User educational and awareness

      •  Monitoring for weird patterns

      •  Ability to report misuse

  •  External Communication Plans

    •  Stakeholder Communication:

      •  Users: how AI affects them, their rights

      •  Regulators: show compliance

      •  Public: transparency and accountability

      •  Media: proactive communication strategy

    •  Transparency Reports:

      •  AI system capabilities and limitations

      •  Training data sources

      •  Performance metrics

      •  Known issues and risks

      •  Update history

    •  Incident Communication:

      •  Prompt notification to affected parties

      •  Clear explanation of what happened

      •  Steps being taken to address issue

      •  Timeline for resolution

      •  Contact information for questions

  •  Deactivation and Localization Controls

    •  Deactivation Policy:

      •  When to Deactivate:

        •  Performance falls below acceptable thresholds

        •  Security vulnerabilities discovered

        •  Regulatory requirements mandate cessation

        •  Unacceptable bias or discrimination detected

        •  Business case no longer justified

    •  Deactivation Process:

      •  Clear authority and decision-making

      •  Graceful shutdown procedures

      •  User notification

      •  Alternative solution provision

      •  Data retention and archival

      •  Post-deactivation monitoring

    •  Localization Controls:

      •  Geographic Restrictions:

        •  Compliance with local regulations

        •  Cultural appropriateness

        •  Language requirements

        •  Data residency requirements

    •  Access Controls:

      •  User group restrictions

        •  Feature gating by user type

        •  Environment-specific deployments (production vs. test)

        •  task-based access controls

    •  Rollback Procedures:

      •  Version control systems

      •  Automated rollback triggers

      •  Testing rollback procedures

      •  Communication during rollbacks

      •  Root cause analysis for failures

  •  Ensure interoperability of AI risk management with other operational risk strategies

    • Ex. security risk, privacy risk, business risk.

  •  “The AI models constitute valuable intellectual assets, demanding features that prevent unauthorized access or tampering.” 

  •  “Depending on the sector—such as healthcare or finance—the stack must be compliant with industry-specific regulations like HIPAA or PCI-DSS” 

  •  Integrate AI governance principles into the company

    • Adopt a pro-innovation mindset.

    • Ensure governance is risk-centric.

    • Ensure planning and design is consensus-driven.

    • Ensure the team is outcome-focused.

    • Adopt a non-prescriptive approach to allow for intelligent self-management

    • Ensure the framework is law-, industry-, and technology-agnostic.

  •  Establish an AI governance infrastructure

    • Determine if you are a developer, deployer (those that make an AI system available to third parties) or user; understand how responsibilities among companies that develop AI systems and those that use or deploy them differ; establish governance processes for all parties; establish framework for procuring and assessing AI software solutions.

    • Establish and understand the roles and responsibilities of AI governance people and groups including, but not limited to, the chief privacy officer, the chief ethics officer, the office for responsible AI, the AI governance committee, the ethics board, architecture steering groups, AI project managers, etc.

    • Advocate for AI governance support from senior leadership and tech teams by:

      •  Understanding pressures on tech teams to build AI solutions quickly and efficiently.

      •  Understanding how data science and model operations teams work.

      •  Being able to influence behavioral and cultural change.

    • Establish organizational risk strategy and tolerance.

    • Develop central inventory of AI and ML applications and repository of algorithms.

    • Develop responsible AI accountability policies and incentive structures.

    • Understand AI regulatory requirements.

    • Set common AI terms and taxonomy for the organization.

    • Provide knowledge resources and training to the enterprise to foster a culture that continuously promotes ethical behavior.

    • Determine AI maturity levels of business functions and address insufficiencies.

    • Use and adapt existing privacy and data governance practices for AI management.

    • Create policies to manage third party risk, to ensure end-to-end accountability.

    •  Understand differences in norms/expectations across countries.

  •  Map, plan and scope the AI project

    • Define the business case and perform cost/benefit analysis where trade-offs are considered in the design of AI systems. Why AI/ML?

    • Identify and classify internal/external risks and contributing factors (prohibitive, major, moderate).

    • Construct a probability/severity harms matrix and a risk mitigation hierarchy.

    • Perform an algorithmic impact assessment leveraging PIAs as a starting point and tailoring to AI process. Know when to perform and who to involve.

    • Establish level of human involvement/oversight in AI decision making.

    • Conduct a stakeholder engagement process that includes the following steps:

      •   Evaluate stakeholder salience.

      •  Include diversity of demographics, disciplines, experience, expertise and backgrounds.

      •  Perform positionality exercise.

      •  Determine level of engagement.

      •  Establish engagement methods.

      •  Identify AI actors during design, development, and deployment phases.

      •  Create communication plans for regulators and consumers that reflect compliance/disclosure obligations for transparency and explainability (UI copy, FAQs, online documentation, model or system cards).

    • Determine feasibility of optionality and redress.

    • Chart data lineage and provenance, ensuring data is representative, accurate and unbiased.

    • Use statistical sampling to identify data gaps.

    • Solicit early and continuous feedback from those who may be most impacted by AI systems.

    • Use test, evaluation, verification, validation (TEVV) process.

    • Create preliminary analysis report on risk factor and proportionate management.

    •  Test and validate the AI system during development

    • Evaluate the trustworthiness, validity, safety, security, privacy and fairness of the AI system using the following methods:

      •  Use edge cases, unseen data, or potential malicious input to test the AI models.

    •  Conduct repeatability assessments.

      •  Complete model cards/fact sheets.

      •  Create counterfactual explanations (CFEs).

      •  Conduct adversarial testing and threat modeling to identify security threats.

      •  Refer to OECD catalog of tools and metrics for trustworthy AI.

      •  Establish multiple layers of mitigation to stop system errors or failures at different levels or modules of the AI system.

      •  Understand trade-offs among mitigation strategies.

    • Apply key concepts of privacy-preserving machine learning and use privacy-enhancing technologies and privacy-preserving machine learning techniques to help with privacy protection in AI/ML systems.

    • Understand why AI systems fail. Examples include: brittleness; hallucinations; embedded bias; catastrophic forgetting; uncertainty; false positives.

    • Determine degree of remediability of adverse impacts.

    • Conduct risk tracking to document how risks may change over time.

    • Consider, and select among different deployment strategies.

  •  Manage and monitor AI systems after deployment

    • Perform post-hoc testing to determine if AI system goals were achieved, while being aware of ″automation bias.″

    • Prioritize, triage and respond to internal and external risks.

    • Ensure processes are in place to deactivate or localize AI systems as necessary (e.g., due to regulatory requirements or performance issues).

    • Continuously improve and maintain deployed systems by tuning and retraining with new data, human feedback, etc.

    • Determine the need for challenger models to supplant the champion model.

    • Version each model and connect them to the data sets they were trained with.

    • Continuously monitor risks from third parties, including bad actors.

    • Maintain and monitor communication plans and inform user when AI system updates its capabilities.

    • Assess potential harms of publishing research derived from AI models.Conduct bug bashing and red teaming exercises.

    • Forecast and reduce risks of secondary/unintended uses and downstream harm of AI models.

Contemplating Ongoing Issues and Concerns

Awareness of legal issues

  • How will a coherent tort liability framework be created to adapt to the unique circumstances of AI and allocate responsibility among developers, deployers and users?

  •   Web-scraping to train AI

    •  In the US

      • Computer Fraud and Abuse Act (1986), web data scraping expanded by U.S. Patriot Act in 2001 and, most recently, through passage of the Identity Theft Enforcement and Restitution Act in 2008.

      • Within the CCPA Final Regulations, approved by the California  Office of Administrative Law in March, Section 7012(h) further clarifies that: "A business that neither collects nor controls the collection of personal information directly from the consumer does not need to provide Notice at Collection to the consumer if it neither sells nor shares the consumer's personal information."

      • Thus, according to a blog by Nate Garhart, special counsel at Farella, Braun, and Martel, no notice needs to be provided for:

        •  A data scraper that does not sell the scraped personal information.

        •  A data scraper that uses the scraped information for their own purposes, even for marketing to identified customers.

        •  A data scraper that collects data, deidentifies it, and then sells the deidentified collection of data.

      • On the other hand, according to Garhart, a scraper selling collections of scraped data that include personal information would be subject to the requirement to provide notice at collection. This may apply to AI products trained on personal data scraped fromthe web.

    •  In the EU

      • The GDPR outlines six lawful bases that can justify data collection and processing: consent, contract, legal obligation, vital interest, public task and legitimate interest. Under most interpretations of the GDPR, these Article 6 requirements apply whether the information is obtained from a publicly accessible source or collected directly from the data subject.

    • What are the challenges surrounding AI model and data licensing? 

  •  Intellectual Property (IP)

    •  What intellectual property (IP) rights exist in the output of an AI model?

    •  Can AI models create trade secrets, copyrights, or even inventive subject matter? If so, who owns the associated IP rights? The owner could be the licensor or licensee (or maybe the model itself?)

  •  Performance and Reliability

    •  Licensees should insist on minimum performance metrics to ensure that the licensed model provides adequate accuracy, reliability, and robustness. If the model does not perform as expected, the business consequences may be severe, and could lead to litigation. Licensees should craft warranties and indemnities to ensure that they do not unreasonably bear the risk of underperformance.

  •  Data Protection and Confidentiality

    •  While data transfers are a standard component of software licenses, using licensee data to improve AI products presents additional privacy and cybersecurity concerns beyond those of a typical software agreement.

    •  Licensees should ensure that any uses of data are consistent with applicable privacy laws and consistent with the privacy notices provided to their users. Numerous states are adopting laws relating to AI and automated decision-making, so licensees will need to stay updated on developments in this area to ensure ongoing compliance.

  • Can we develop systems that respect IP rights?

    •  “The resulting generative AI models need not be trained from scratch but can build upon open-source generative AI that has used lawfully sourced content. This would enable content creators to produce content in the same style as their own work with an audit trail to their own data lake, or to license the use of such tools to interested parties with cleared title in both the AI’s training data and its outputs” 

    •  Tangent argument: “AI systems can also generate new works protectable by copyright, such as creating new artwork or music. However, most copyright statutes do not yet not clearly define who owns machine-generated works.“ 

    •  A deployed genAI system should respect IPR. this means adhering to licensing terms and ensuring that generated content complies with these terms. A system that categorizes and applies filters based on licensing terms ensures that content is used legally and ethically, respecting the rights of content creators.

Awareness of user concerns

  • How do we properly educate users about the functions and limitations of AI systems?

    •  AI Literacy - “equipping individuals with the knowledge and skills to understand, use, and interact with AI responsibly and effectively. It's about enabling people to make informed decisions about AI technologies, understand their implications, and navigate the ethical considerations they present.” 

  • How do we upskill and reskill the workforce to take full advantage of AI benefits? 

    •  By nurturing complementary skills, it is possible to create a workforce that seamlessly integrates with AI, leveraging its capabilities to amplify human potential.

      •  Creativity and Innovation through Critical Thinking and Problem-Solving by leveraging AI's insights to develop groundbreaking solutions

      •  Communication and Collaboration by translating AI's findings into actionable strategies

      •  Emotional Intelligence and Social Skills to foster trust and collaboration in AI-driven environments

      •  Ethical Decision-Making and Bias Awareness to be mindful of ethical implications and social impact, to ensure that AI is used responsibly and ethically

    •  While reskilling is the process of learning new skills to adapt to new job requirements, upskilling helps enhance existing skills to improve one's performance in their current job. There are several reasons why reskilling and upskilling are important in the AI-driven workplace, such as:

      •  Staying competitive: Workers with the latest skills are more likely to be hired and promoted.

      •  Increasing job satisfaction: Workers who are challenged and engaged in their work are more likely to be satisfied with their jobs.

      •  Improving productivity: Workers with the right skills are more productive and efficient.

      •  Reducing the risk of job displacement: Workers who are adaptable and can learn new skills are less likely to be displaced by automation.

    •  Here are a few ways in which people leaders can leverage AI / ML technology for skilling:

      •  Assessing skill gaps: Identify the skills that are needed for future jobs in the organization.

      •  Developing training programs: Create or partner with training providers to offer training programs that teach the skills that are needed.

      •  Providing financial assistance: Offer financial assistance to employees who are participating in training programs.

      •  Promoting a culture of lifelong learning: Encourage employees to take advantage of learning opportunities.

  • Can there be an opt-out for a non-AI alternative?

    •  “You should be able to opt out, where appropriate, and have access to a person who can quickly consider and remedy problems you encounter. You should be able to opt out from automated systems in favor of a human alternative, where appropriate.” 

    •  When assessing whether users should be given the right to opt out from an AI system, the primary considerations are feasibility, risk to users, and industry practice. Feasibility addresses whether the opt-out mechanism can be practically implemented. Risk to users assesses the potential harm or benefits users might face if they cannot opt out. Industry practice considers the norms and standards within the industry.

Awareness of AI auditing and accountability issues

  • How can we build a profession of certified third-party auditors globally – and consistent frameworks and standards for them?

  • What are the markers/indicators that determine when an AI system should be subject to enhanced accountability, such as third-party audits (e.g., automated decision-making, sensitive data, others)?

  • How do we enable companies to remain productive using automated checks for AI governance and associated ethical issues, while adapting this automation quickly to the evolving standards and technology?

  •  Other Notes:

    • One of the findings from the IAPP Privacy and Consumer Trust Report concerned a set of behaviors referred to as privacy self-defense. These include deciding against an online purchase, deleting a smartphone app or avoiding a particular website due to privacy concerns. When consumers lose trust in how their data is being collected and used, they are more likely to engage in these self-defensive behaviors to protect their privacy.

    • The European Commission signed off on its Ethical Guidelines for Trustworthy AI which outlines “four principles or “ethical imperatives” call for AI systems to respect human autonomy, prevent harm, incorporate fairness and enable explicability. Another layer of guidance advises that AI respect human dignity, individual freedom, democracy, justice, the rule of law, equality, non-discrimination, solidarity and citizens’ rights.”

  •   

 

 


 
 
 

Comments

bottom of page