Project Category: Unleash Data / Community
Proposal in one sentence:
Phase II of the DAO Health Initiative will scientifically validate the DAO Health survey to ensure it measures what it intends to measure.
Project Description:
In phase one of the DAO Health initiative, we used the scientific literature to bridge the gap between previous conceptualizations of organizational health in traditional organizations and DAOs. We developed a psychometric instrument to assess DAO Health, open-sourced it, and wrote about what we discovered.
Now for phase two, as part of instrument development, we want to gather validity evidence that the survey measures what it says it measures. One strategy for gathering this evidence is called content validation, or rationally showing a connection between the content of the survey and the content of the construct we intend to measure. In a subsequent grant (Phase 3), we plan to gather criterion-related validity evidence. This is evidence showing an empirical connection between survey feedback and intended target outcome metrics.
The first validation study is the content validation study. This ensures our instrument accurately and holistically measures the concept of DAO health. To do this we will conduct a survey to assess the validity of the DAO Health Survey from the lens of DAO founders and contributors. Participants will receive POAPs for survey completion to make them eligible for future airdrops.
What’s the final product:
This project will deliver:
- A content validated DAO Health Survey V2.0
- Mirror write-up on procedures and learnings
And will ultimately contribute to the following after the completion of this and future phases:
- An open-source scientifically validated DAO Health Survey
- An analytical procedure for assessing DAO Health
- A research study published in the Journal of Decentralized Work
Project Details:
In phase one of the DAO Health initiative, we used the scientific literature to bridge the gap between previous conceptualizations of organizational health in traditional organizations and DAOs. We developed a psychometric instrument to assess DAO Health by measuring a variety of psychological factors related to contributors’ experience in DAOs. This included concepts like engagement, psychological safety, trust, shared leadership, health and well-being, culture, and more. The full write-up of our approach to developing the instrument and how it should be used can be found on Mirror: Developing the DAO Health survey: an open source tool for web3 organizational effectiveness.
We defined DAO health as a DAO’s ability to coordinate teams toward a shared vision and objectives as a function of the contributor socio-psychological factors that drive collective productivity, performance, leadership, and individual experience and well-being. Our research also provided evidence for organizational health as a predictor of organizational effectiveness and confirmation that organizational health conceptually applies to DAOs.
In the first development phase, we hypothesized that our measurement instrument should demonstrate both content and criterion validity evidence. Content validity is the evidence that survey items accurately and holistically measure the conceptual definition of DAO Health provided above. Criterion validity is the evidence that survey item scores correlate with or predict organizational outcome metrics such as performance, effectiveness, leadership, productivity, churn, and decision making.
The next two phases of this initiative consist of validating those claims. The proposed phase two constitutes the content validation study, which establishes the degree to which the constructs we measure are accurately captured by the item(s) included in the DAO Health Survey and the degree to which our items holistically measure the concept of DAO Health.
We will gather content validity evidence by following Colquitt et al.’s (2019) process for examining definitional correspondence, or “the degree to which a scale’s items correspond to the construct’s definition.”
In practice, experienced DAO contributors will be asked to evaluate how well a given item “matched” the construct’s conceptual definition (i.e., definitional correspondence) based on a 5-point scale (1=Not at all to 5=To a very great extent). For a given construct, respondents will rate the proposed single-item measure.
Readers of other literature may also recognize this process as an examination of face validity. As discussed by Allen et al. (2022, p. 1) though, “just as for multi-item measures, it is critically important for single-item measures to demonstrate face validity”, wherein face validity can be defined as the “clarity or relevance” of an item or measure.
Although content validity shows a rational linkage between survey content and conceptual definitions, it does not show a relationship between survey feedback and outcomes. For this reason, a criterion-related validity study is planned for phase three of this work where we outline an analytical procedure for assessing results, bringing in on-chain data, and utilizing network approaches to assess the robustness of the organizational network. Ultimately, phase three will validate whether this tool predicts the outcomes we believe it will.
If successful, this tool and procedure will help alleviate a key challenge currently facing DAOs: assess the effectiveness of their operation using a decentralized operating structure and procedure.
After each phase of this research initiative is completed, talentDAO will publish a research paper in the Journal of Decentralized Work. Going forward, we will conduct yearly surveys and encourage DAO contributors from across the ecosystem to participate. Data from each year’s survey will be listed on the Ocean marketplace. talentDAO will publish a yearly report to keep the ecosystem informed on the psychological factors contributing to the health of a wide array of DAOs.
Hosting survey results on a data marketplace is not something traditional organizations are willing or capable of doing because of privacy concerns and regulations such as GDPR. However, this problem should be subverted using Ocean protocol’s fine-grained permissions and compute-to-data. Survey data that was once a private commodity to research organizations becomes a publicly listed asset on the Ocean marketplace.
Project Deliverables:
Phase 2 of the DAO Health initiative will deliver:
- A content validated DAO Health Survey V2.0
- Mirror write-up on procedures and learnings
Roadmap
This roadmap has been revised since the completion of Phase 1.
-
Phase 1: round 15
- Mirror write-up
- Empirically driven DAO Health survey developed and open sourced on Github
- Phase 2: Content Validation Study
- Write-up on procedures and findings
- V2.0 content validated survey
- Phase 3: Criterion Validation Study & Analytical Framework
- Mirror write-up on procedure and findings
- Analytical procedure / jupyter notebooks released
- V3.0 criterion validated survey (if warranted)
- Shortened ‘pulse’ version of the survey
- Phase 4: Research Publication
- Research paper write-up
- Community peer-review
- Publication in the Journal of Decentralized Work
- Phase 5: Ongoing DAO Health Work and Support
- Yearly report
- Continuous work on survey development
Value add:
Usage of Ocean: Survey data on the OCEAN marketplace is precious to researchers and other DAOs who usually cannot get access to these data, attracting them to the marketplace. In addition to listing survey response data from study participants, survey data will be published on OCEAN each year, making this a recurring event that will ideally grow in participation over time. Recurring participation will create positive feedback loops for adoption and future data access demand.
Viability: High chance of success. We are a team of organizational scientists and researchers with experience working with psychometrics. However, technical details of the survey system and its interface with OCEAN are still being worked out and may require additional development resources.
Adding value to the OCEAN ecosystem: We invite OceanDAO to be a participant in this research. As a result, talentDAO will provide tailored insights into OceanDAO’s organizational health. We believe participation in the next phase [Phase 3: Criterion Validation Study & Analytical Framework] will be the most valuable to OceanDAO, as well as to the research itself.
Funding Requested: $OCEAN equivalent of 10,000 USD
Wallet Address: 0x0406Bf2dAE6A42d567b4e3DbA6ADA99069622fF1
Previous OceanDAO Grant?
Yes. Round 15, $OCEAN equivalent of $ 3000 USD / 7064 OCEAN
About talentDAO:
Core Team
- Kenneth Francis
- Role: Principal Investigator
- Creds:
- GitHub
- Website
- K3nn.eth
- Discord: @k3nn.eth#0270
- Email: k3nnethfrancis@gmail.com
- Exp:
- People Analytics @ SpaceX [current]
- Professor of Industrial/Organizational Psychology @ Brooklyn College [adjunct]
- 4 years of designing, implementing, and maintaining people technology tools for organizations
- 3 years of designing, implementing, and analyzing psychometric surveys for organizations
- Sherif al-Qallawi
- Role: Researcher, Co-author
- Creds:
- Exp:
- Talent Management Analytics @ Mercedes-Benz [current]
- Ph.D. candidate, Industrial/Organizational Psychology
- Post-graduate diploma, Human Resource Management
- 4 years of consulting experience focused on survey projects
- 6 years of experience in international/cross-cultural people operations
- Nemo
- Role: Researcher, Co-author
- Exp:
- People Scientist / Researcher
- Ph.D., Industrial/Organizational Psychology
- Professor of Industrial/Organizational Psychology
- 10 years of consulting and applied research experience related to organizational science and talent measurement
- Mr. Nobody
- Role: Researcher, Co-author
- Exp:
- People Analytics @ Deloitte Consulting [former]
- MSc, Industrial/Organizational Psychology
- Fellow, Center for Evidence-Based Management
- 10 years of consulting experience focused on complex human systems
Future Intentions:
- talentDAO intends to keep the final product open source with a GitHub repo for accessing the survey
- A yearly report to be produced using this survey with participating DAOs
- Data will be listed on the Ocean marketplace with profit-sharing
- talentDAO intends to continue to utilize the research POAP framework in our survey research to compensate participants in $OCEAN
- Future opportunity to build upon compute-to-data functionality
- Future opportunities for Data-NFT integrations
- Future opportunity to integrate Ocean into the Decentralized Science (DeSci) ecosystem infrastructure