---
title: "Fragile Foundations: Hidden Risks of Generative AI"
authors:
  - Anne L. Washington
year: 2025
type: journalArticle
doi: 10.11586/2025078
url: "https://www.bertelsmann-stiftung.de/doi/10.11586/2025078"
tags:
  - paper
llm_decision: Include
llm_confidence: 0.85
llm_categories:
  - AI_Literacies
  - Generative_KI
  - Bias_Ungleichheit
---

# Fragile Foundations: Hidden Risks of Generative AI

## Transformation Trail

### Stufe 1: Extraktion & Klassifikation (LLM)

**Extrahierte Kategorien:** AI_Literacies, Generative_KI, KI_Sonstige, Soziale_Arbeit, Bias_Ungleichheit, Gender, Diversitaet, Fairness
**Argumente:** 3 extrahiert

### Stufe 3: Verifikation (LLM)

| Metrik | Score |
|--------|-------|
| Completeness | 92 |
| Correctness | 98 |
| Category Validation | 96 |
| **Overall Confidence** | **95** |

### Stufe 4: Assessment

**LLM:** Include (Confidence: 0.85)

## Key Concepts

- [[AI Governance and Accountability]]
- [[Algorithmic Bias]]
- [[Gender Bias in AI]]

## Wissensdokument

# Fragile Foundations: Hidden Risks of Generative AI

## Kernbefund

Foundation models are trained on uncurated, biased internet data and sustained by business models that prioritize profit over safety, resulting in systems that reinforce historical inequalities and fail vulnerable populations. Alternative approaches—computational, participatory, data-sourced, and collaborative—demonstrate that responsible foundation models aligned with public interest are possible.

## Forschungsfrage

What structural weaknesses in foundation models impede innovation, fairness, and accountability, and how can alternative approaches serve the public interest?

## Methodik

Mixed Methods: Literature review of academic and corporate research papers, expert interviews with diverse perspectives (skeptics to practitioners to legal scholars), and comparative analysis of foundation models organized around expert-identified challenges.
**Datenbasis:** nicht angegeben (expert interviews and comparative analysis of foundation models as of early 2025)

## Hauptargumente

- Foundation models trained on unverified, internet-scraped datasets reflect and amplify dominant power structures (Western, patriarchal, discriminatory), causing documented harms in real-world applications serving vulnerable populations (eating disorder chatbots providing dieting tips, employment chatbots steering women toward stereotypical roles, mental health chatbots reinforcing suicidality).
- Current business models and governance structures prioritize profit maximization over safety and transparency, with closed proprietary systems preventing external evaluation and accountability, creating systematic risks particularly for mission-driven organizations serving marginalized communities.
- Computational, participatory, data-sourcing, and collaborative alternatives (constitutional AI, alignment assemblies, curated datasets like BLOOM and ROOTS, community-driven models like EthioLLM and UCCIX) demonstrate that foundation models can be designed to serve public good and represent diverse populations without sacrificing performance.

## Kategorie-Evidenz

### Evidenz 1

Report addresses critical understanding of foundation models for practitioners and decision-makers: 'This report is directed primarily at practitioners and decision-makers in mission-driven organizations, but also at anyone committed to a responsible digital future.'

### Evidenz 2

Central focus on foundation models like GPT, ChatGPT, Claude, Gemini, DALL-E, and CoPilot: 'Systems such as ChatGPT, Gemini, Dall-E, or CoPilot now shape search results, education, and workplace routines.'

### Evidenz 3

Discusses large language models, machine learning, NLP, and computer vision: 'Large language models (LLMs) are foundation models that capture patterns in human language' and extensive appendix on ML concepts.

### Evidenz 4

Explicit reference to mission-driven social organizations and vulnerable populations served: 'For mission-driven organizations, these risks are not abstract technicalities but immediate practical challenges. Precisely in contexts where vulnerable groups depend on accurate information and fair treatment, biased or erroneous AI outputs can cause real harm. A chatbot used in migration counseling...An automated tool in social work...'

### Evidenz 5

Core argument about bias perpetuation: 'These datasets reflect dominant power structures that are often Western, patriarchal, and discriminatory. Rather than reflecting diversity, the models reproduce and amplify existing biases.' Documents concrete harms: 'A chatbot from Austria's employment agency gave stereotypical career advice, steering women toward cooking or nursing professions and men toward IT jobs.'

### Evidenz 6

Analysis of gender-specific biases: 'A chatbot from Austria's employment agency gave stereotypical career advice, steering women toward cooking or nursing professions and men toward IT jobs.' Discussion of WEIRD (Western, Educated, Industrialized, Rich, Democratic) value embedding reflecting gendered perspectives.

### Evidenz 7

Extensive focus on representation and inclusion of marginalized groups and languages: 'Smaller language groups, often overlooked in large general-purpose models, now have alternative foundation models that represent their populations, languages, and cultures, making them visible in artificial intelligence.' Discusses low-resource languages, endangered language preservation (Gaelic, Kalaallisut, isiZulu), and African language models (EthioLLM).

### Evidenz 8

Explicit focus on algorithmic fairness and fair AI design: 'With deliberate curation, continuous improvement, and a service orientation, foundation models could evolve into infrastructures that are more firmly aligned with the public interest.' Discusses fairness through diverse participation, source transparency, and constitutional AI approaches.

## Assessment-Relevanz

**Domain Fit:** Highly relevant at the intersection of AI governance, social work practice, and equity concerns. Directly addresses how generative AI systems harm vulnerable populations served by mission-driven organizations, making it essential reading for social workers deploying AI tools.

**Unique Contribution:** Bridges gap between technical AI critique and practical social sector implementation by documenting real harms to vulnerable groups and proposing concrete alternatives (computational, participatory, data-sourced, collaborative) that align with public interest values.

**Limitations:** Limited to English-language academic literature and expert interviews; does not include systematic empirical evaluation of alternative models' effectiveness in social work contexts; specific recommendations lack detailed implementation guidance for resource-constrained organizations.

**Target Group:** Practitioners and decision-makers in mission-driven organizations (social services, NGOs, non-profits), social workers and counselors implementing AI tools, policymakers, AI developers committed to public interest technology, and anyone working at the intersection of AI governance and social equity

## Schlüsselreferenzen

- [[Bommasani_et_al_2022]] - Foundation models definition and taxonomy
- [[Atari_2023]] - ChatGPT reflects WEIRD values vs global perspectives
- [[BigScience_Workshop_et_al_2023]] - BLOOM: Open multilingual language model
- [[Anthropic_2023]] - Constitutional AI approach
- [[Laurençon_et_al_2022]] - ROOTS: Responsible open-source text sources
- [[Birhane_et_al_2022]] - Participatory methods for supervised ML and AI
- [[Tran_OSullivan_Nguyen_2024]] - UCCIX: Irish language LLM
- [[Tonja_et_al_2024]] - EthioLLM: Multilingual models for Ethiopian languages
- [[DeepSeekAI_et_al_2025]] - DeepSeek efficient foundation model
- [[Washington_2023]] - Ethical Data Science: Prediction in the Public Interest
