Project Name
DAO Contributor Health Assessment Development
Project Category
Unleash Data
Proposal Earmark
2nd/3rd Grant
Proposal Description
Project Description:
In phase one of the DAO Health initiative, we used the scientific literature to bridge the gap between previous conceptualizations of organizational health in traditional organizations and DAOs. We developed a psychometric instrument to assess DAO Health, open-sourced it, and wrote about what we discovered.
Now for phase two, as part of instrument development, we want to gather validity evidence that the survey measures what it says it measures. One strategy for gathering this evidence is called content validation, or rationally showing a connection between the content of the survey and the content of the construct we intend to measure. In a subsequent grant (Phase 3), we plan to gather criterion-related validity evidence. This is evidence showing an empirical connection between survey feedback and intended target outcome metrics.
The first validation study is the content validation study. This ensures our instrument accurately and holistically measures the concept of DAO health. To do this we will conduct a survey to assess the validity of the DAO Health Survey from the lens of DAO founders and contributors. Participants will receive POAPs for survey completion to make them eligible for future airdrops.
Full Project Details:
In phase one of the DAO Health Assessment development, talentDAO used the scientific literature to bridge the gap between previous conceptualizations of organizational health in traditional organizations and DAOs. We developed a psychometric instrument to assess DAO Health by measuring a variety of psychological factors related to contributors’ experience in DAOs. This included concepts like engagement, psychological safety, trust, shared leadership, health and well-being, culture, and more. The full write-up of our approach to developing the instrument and how it should be used can be found on Mirror: Developing the DAO Health survey: an open source tool for web3 organizational effectiveness.
We defined DAO health as a DAO’s ability to coordinate teams toward a shared vision and objectives as a function of the contributor socio-psychological factors that drive collective productivity, performance, leadership, and individual experience and well-being. Our research also provided evidence for organizational health as a predictor of organizational effectiveness and confirmation that organizational health conceptually applies to DAOs.
In the first development phase, we hypothesized that our measurement instrument should demonstrate both content and criterion validity evidence. Content validity is the evidence that survey items accurately and holistically measure the conceptual definition of DAO Health provided above. Criterion validity is the evidence that survey item scores correlate with or predict organizational outcome metrics such as performance, effectiveness, leadership, productivity, churn, and decision making.
The next two phases of this initiative consist of validating those claims. The proposed phase two constitutes the content validation study, which establishes the degree to which the constructs we measure are accurately captured by the item(s) included in the DAO Health Survey and the degree to which our items holistically measure the concept of DAO Health.
We will gather content validity evidence by following Colquitt et al.’s (2019) process for examining definitional correspondence, or “the degree to which a scale’s items correspond to the construct’s definition.”
In practice, experienced DAO contributors will be asked to evaluate how well a given item “matched” the construct’s conceptual definition (i.e., definitional correspondence) based on a 5-point scale (1=Not at all to 5=To a very great extent). For a given construct, respondents will rate the proposed single-item measure.
Readers of other literature may also recognize this process as an examination of face validity. As discussed by Allen et al. (2022, p. 1) though, “just as for multi-item measures, it is critically important for single-item measures to demonstrate face validity”, wherein face validity can be defined as the “clarity or relevance” of an item or measure.
Although content validity shows a rational linkage between survey content and conceptual definitions, it does not show a relationship between survey feedback and outcomes. For this reason, a criterion-related validity study is planned for phase three of this work where we outline an analytical procedure for assessing results, bringing in on-chain data, and utilizing network approaches to assess the robustness of the organizational network. Ultimately, phase three will validate whether this tool predicts the outcomes we believe it will.
If successful, this tool and procedure will help alleviate a key challenge currently facing DAOs: assess the effectiveness of their operation using a decentralized operating structure and procedure.
After each phase of this research initiative is completed, talentDAO will publish a research paper in the Journal of Decentralized Work. Going forward, we will conduct yearly surveys and encourage DAO contributors from across the ecosystem to participate. Data from each year’s survey will be listed on the Ocean marketplace. talentDAO will publish a yearly report to keep the ecosystem informed on the psychological factors contributing to the health of a wide array of DAOs.
Hosting survey results on a data marketplace is not something traditional organizations are willing or capable of doing because of privacy concerns and regulations such as GDPR. However, this problem should be subverted using Ocean protocol’s fine-grained permissions and compute-to-data. Survey data that was once a private commodity to research organizations becomes a publicly listed asset on the Ocean marketplace.
Roadmap
- Phase 1: round 15
- Mirror write-up
- Empirically driven DAO Health survey developed and open-sourced on Github
- Phase 2: Content Validation Study [Current Phase]
- Write-up on procedures and findings
- V2.0 content validated survey
- Phase 3: Criterion Validation Study & Analytical Framework
- Mirror write-up on procedure and findings
- Analytical procedure / jupyter notebooks released
- V3.0 criterion validated survey (if warranted)
- Shortened ‘pulse’ version of the survey
- Phase 4: Research Publication
- Research paper write-up
- Community peer-review
- Publication in the Journal of Decentralized Work
- Phase 5: Ongoing DAO Health Work and Support
- Yearly report
- Continuous work on survey development
Grant Deliverables
This project will deliver:
- A content validated DAO Health Survey V2.0
- Mirror write-up on procedures and learnings
And will ultimately contribute to the following after the completion of this and future phases:
- An open-source scientifically validated DAO Health Survey
- An analytical procedure for assessing DAO Health
- A research study published in the Journal of Decentralized Work
Project Description
TalentDAO will develop a scientifically validated survey tool for measuring the psychological factors of DAO contributors that lead to a healthy organization while enabling the sharing of this data in a secure way via the OCEAN protocol.
Final Product
not set
Value Add Criteria
Value add:
Usage of Ocean: Survey data on the OCEAN marketplace is precious to researchers and other DAOs who normally cannot get access to these data, attracting them to the marketplace. In addition to listing survey response data from study participants, survey data will be published on OCEAN each year, making this a recurring event that will ideally grow in participation over time. Recurring participation will create positive feedback loops for adoption and future data access demand.
Viability: High chance of success. We are a team of organizational scientists and researchers with experience working with psychometrics. However, technical details of the survey system and its interface with OCEAN are still being worked out and may require additional development resources.
Adding value to the OCEAN ecosystem: We invite OceanDAO to be a participant in this research. As a result, talentDAO will provide tailored insights into OceanDAO’s organizational health. We believe participation in the next phase [Phase 3: Criterion Validation Study & Analytical Framework] will be the most valuable to OceanDAO, as well as to the research itself.
Funding Requested
10000
Minimum Funding Requested
0
Wallet Address
0xd25b8a4B94FbD494414ab209E97acDdd9625Aed0