https://blue.feedland.org/?river=http://data.feedland.org/blue/feeds/yamilvelez.xml It's just a feed for now Thu, 14 Dec 2023 19:45:42 GMT feedland v0.6.43 https://cyber.harvard.edu/rss/rss.html Thu, 14 Dec 2023 19:45:42 GMT http://rpc.rsscloud.io:5337/pleaseNotify Thu, December 14, 2023 2:45 PM EST <p>How should we measure opinion in rapidly changing information environments? In a working paper, I develop an approach - the crowdsourced adaptive survey (CSAS) - that unites LLMs and adaptive algorithms to create surveys that evolve with user input (<a href="https://bit.ly/3TqPofZ">https://bit.ly/3TqPofZ</a>)</p> <p>In my research on Latino misinformation, it has often been challenging to develop survey items due to a lack of fact-checking organizations focused on the community, and the use of private, encrypted platforms like WhatsApp that complicate the tracking of false claims.</p> <p>With those barriers in mind, learning from participants is one way forward. However, carrying out representative in-depth interviews or multi-wave studies to develop survey items can be expensive. How can we identify survey questions that spring from the bottom up?</p> <p>With the CSAS method, open-ended responses from participants are converted into survey items, uploaded to a question bank, and rated by other participants, with these ratings being used to prioritize which items are featured in future surveys using Gaussian Thompson sampling.</p> <p>I apply this method to the setting of Latino information environments, finding that beliefs in out-right misinformation are rare. Instead, party stereotypes and widely reported allegations rise to the top of accuracy ratings.</p> <p>I then use this method to examine beliefs among those who trust WhatsApp, a popular messaging platform among Latinos, versus those who do not. I find that bigger accuracy gaps are observed for newsworthy scandals than blatant misinformation.</p> <p>I also apply CSAS to the question of measuring issue importance in the aggregate. Though economic issues are highly rated, the method recovers less salient topics such as privacy protections and political transparency that would likely not appear on a traditional survey.</p> <p>The use of an adaptive algorithm to explore different issues yields a much larger set of questions than would otherwise be possible. This can be useful in identifying partisan gaps or other subgroup differences across a variety of items.</p> <p>This is still a work-in-progress, and I&#39;m open to feedback. So far, I&#39;ve found this to be a useful exploratory tool in settings where it&#39;s difficult to have strong priors about the best-performing survey items. I&#39;m hoping to make it more accessible in the coming months.</p> Thu, 14 Dec 2023 19:45:42 GMT https://blue.feedland.org/?item=225947 https://blue.feedland.org/?item=225947 How should we measure opinion in rapidly changing information environments? In a working paper, I develop an approach - the crowdsourced adaptive survey (CSAS) - that unites LLMs and adaptive algorithms to create surveys that evolve with user input (https://bit.ly/3TqPofZ) In my research on Latino misinformation, it has often been challenging to develop survey items due to a lack of fact-checking organizations focused on the community, and the use of private, encrypted platforms like WhatsApp that complicate the tracking of false claims. With those barriers in mind, learning from participants is one way forward. However, carrying out representative in-depth interviews or multi-wave studies to develop survey items can be expensive. How can we identify survey questions that spring from the bottom up? With the CSAS method, open-ended responses from participants are converted into survey items, uploaded to a question bank, and rated by other participants, with these ratings being used to prioritize which items are featured in future surveys using Gaussian Thompson sampling. I apply this method to the setting of Latino information environments, finding that beliefs in out-right misinformation are rare. Instead, party stereotypes and widely reported allegations rise to the top of accuracy ratings. I then use this method to examine beliefs among those who trust WhatsApp, a popular messaging platform among Latinos, versus those who do not. I find that bigger accuracy gaps are observed for newsworthy scandals than blatant misinformation. I also apply CSAS to the question of measuring issue importance in the aggregate. Though economic issues are highly rated, the method recovers less salient topics such as privacy protections and political transparency that would likely not appear on a traditional survey. The use of an adaptive algorithm to explore different issues yields a much larger set of questions than would otherwise be possible. This can be useful in identifying partisan gaps or other subgroup differences across a variety of items. This is still a work-in-progress, and I'm open to feedback. So far, I've found this to be a useful exploratory tool in settings where it's difficult to have strong priors about the best-performing survey items. I'm hoping to make it more accessible in the coming months. <p>It&#39;s that time of the year. We&#39;re just about done collecting data on 2,000+ participants for my Experimental Research class. This semester, we settled on AI image/video discernment as our key outcome, and 11 students submitted their own interventions.</p> <p>The interventions were modeled after popular methods in the misinformation literature such as flagging and digital literacy skills. Some students created fantastic infographics and videos. Others suggested improvements to social media UIs.</p> <p>I wanted to test out variations of social media UIs in a more ecologically valid environment, so I created Future Feed, a faux social media app that can be embedded in Qualtrics, allows you to vary UI features, and measures user behavior.</p> <p>With Future Feed, we could test interventions such as flagging and even experiment with newer methods such as &quot;provenance-enabled&quot; media. I was also able to measure user actions in the form of liking and sharing.</p> <p>Participants were first asked to interact with the feed as they normally would. This was followed by a rating task involving a balanced assortment of AI-generated and non-AI content they encountered.</p> <p>Given the large number of interventions, we used an adaptive experimental design that devotes more N to promising interventions. I thought flagging content was going to be the clear winner, but a version disclosing that the content was &quot;automatically flagged&quot; performed best.</p> <p>Most interventions have positive ATE estimates for AI accuracy, but as is common in the misinformation literature, some interventions simultaneously increased errors for non-AI content. The automatic flagging condition moved both outcomes in a positive direction.</p> <p>Our study is yet another reminder to assess how misinformation-reducing interventions impact perceptions of both genuine and false content.</p> <p>(thanks to @BrianMGuay who Zoomed into our class on measurement and emphasized this point)</p> <p>It is also a reminder that we shouldn&#39;t assume that what works for misinformation will also work for generative AI, where differences between non-AI and AI content are growing smaller with every leap in technology.</p> <p>I promised the students they&#39;d be surprised, and as is always the case when I teach experiments, I delivered on that promise.</p> <p>This class is modeled after @BrendanNyhan&#39;s wonderful project-based learning course on experiments.</p> Tue, 21 Nov 2023 03:52:38 GMT https://blue.feedland.org/?item=199600 https://blue.feedland.org/?item=199600 It's that time of the year. We're just about done collecting data on 2,000+ participants for my Experimental Research class. This semester, we settled on AI image/video discernment as our key outcome, and 11 students submitted their own interventions. The interventions were modeled after popular methods in the misinformation literature such as flagging and digital literacy skills. Some students created fantastic infographics and videos. Others suggested improvements to social media UIs. I wanted to test out variations of social media UIs in a more ecologically valid environment, so I created Future Feed, a faux social media app that can be embedded in Qualtrics, allows you to vary UI features, and measures user behavior. With Future Feed, we could test interventions such as flagging and even experiment with newer methods such as "provenance-enabled" media. I was also able to measure user actions in the form of liking and sharing. Participants were first asked to interact with the feed as they normally would. This was followed by a rating task involving a balanced assortment of AI-generated and non-AI content they encountered. Given the large number of interventions, we used an adaptive experimental design that devotes more N to promising interventions. I thought flagging content was going to be the clear winner, but a version disclosing that the content was "automatically flagged" performed best. Most interventions have positive ATE estimates for AI accuracy, but as is common in the misinformation literature, some interventions simultaneously increased errors for non-AI content. The automatic flagging condition moved both outcomes in a positive direction. Our study is yet another reminder to assess how misinformation-reducing interventions impact perceptions of both genuine and false content. (thanks to @BrianMGuay who Zoomed into our class on measurement and emphasized this point) It is also a reminder that we shouldn't assume that what works for misinformation will also work for generative AI, where differences between non-AI and AI content are growing smaller with every leap in technology. I promised the students they'd be surprised, and as is always the case when I teach experiments, I delivered on that promise. This class is modeled after @BrendanNyhan's wonderful project-based learning course on experiments. <p>It&#39;s that time of the year. We&#39;re just about done collecting data on 2,000+ participants for my Experimental Research class. This semester, we settled on AI image/video discernment as our key outcome, and 11 students submitted their own interventions.</p> <p>The interventions were modeled after popular methods in the misinformation literature such as flagging and digital literacy skills. Some students created fantastic infographics and videos. Others suggested improvements to social media UIs.</p> <p>I wanted to test out variations of social media UIs in a more ecologically valid environment, so I created Future Feed, a faux social media app that can be embedded in Qualtrics, allows you to vary UI features, and measures user behavior.</p> <p>With Future Feed, we could test interventions such as flagging and even experiment with newer methods such as &quot;provenance-enabled&quot; media. I was also able to measure user actions in the form of liking and sharing.</p> <p>Participants were first asked to interact with the feed as they normally would. This was followed by a rating task involving a balanced assortment of AI-generated and non-AI content they encountered.</p> <p>Given the large number of interventions, we used an adaptive experimental design that devotes more N to promising interventions. I thought flagging content was going to be the clear winner, but a version disclosing that the content was &quot;automatically flagged&quot; performed best.</p> <p>Most interventions have positive ATE estimates for AI accuracy, but as is common in the misinformation literature, some interventions simultaneously increased errors for non-AI content. The automatic flagging condition moved both outcomes in a positive direction.</p> <p>Our study is yet another reminder to assess how misinformation-reducing interventions impact perceptions of both genuine and false content.</p> <p>(thanks to @BrianMGuay who Zoomed into our class on measurement and emphasized this point)</p> <p>It is also a reminder that we shouldn&#39;t assume that what works for misinformation will also work for generative AI, where differences between non-AI and AI content are growing smaller with every leap in technology.</p> <p>I promised the students they&#39;d be surprised, and as is always the case when I teach experiments, I delivered on that promise.</p> <p>This class is modeled after @BrendanNyhan&#39;s wonderful project-based learning course on experiments.</p> Tue, 21 Nov 2023 03:51:29 GMT https://blue.feedland.org/?item=199599 https://blue.feedland.org/?item=199599 It's that time of the year. We're just about done collecting data on 2,000+ participants for my Experimental Research class. This semester, we settled on AI image/video discernment as our key outcome, and 11 students submitted their own interventions. The interventions were modeled after popular methods in the misinformation literature such as flagging and digital literacy skills. Some students created fantastic infographics and videos. Others suggested improvements to social media UIs. I wanted to test out variations of social media UIs in a more ecologically valid environment, so I created Future Feed, a faux social media app that can be embedded in Qualtrics, allows you to vary UI features, and measures user behavior. With Future Feed, we could test interventions such as flagging and even experiment with newer methods such as "provenance-enabled" media. I was also able to measure user actions in the form of liking and sharing. Participants were first asked to interact with the feed as they normally would. This was followed by a rating task involving a balanced assortment of AI-generated and non-AI content they encountered. Given the large number of interventions, we used an adaptive experimental design that devotes more N to promising interventions. I thought flagging content was going to be the clear winner, but a version disclosing that the content was "automatically flagged" performed best. Most interventions have positive ATE estimates for AI accuracy, but as is common in the misinformation literature, some interventions simultaneously increased errors for non-AI content. The automatic flagging condition moved both outcomes in a positive direction. Our study is yet another reminder to assess how misinformation-reducing interventions impact perceptions of both genuine and false content. (thanks to @BrianMGuay who Zoomed into our class on measurement and emphasized this point) It is also a reminder that we shouldn't assume that what works for misinformation will also work for generative AI, where differences between non-AI and AI content are growing smaller with every leap in technology. I promised the students they'd be surprised, and as is always the case when I teach experiments, I delivered on that promise. This class is modeled after @BrendanNyhan's wonderful project-based learning course on experiments. <p>It&#39;s that time of the year. We&#39;re just about done collecting data on 2,000+ participants for my Experimental Research class. This semester, we settled on AI image/video discernment as our key outcome, and 11 students submitted their own interventions.</p> <p>The interventions were modeled after popular methods in the misinformation literature such as flagging and digital literacy skills. Some students created fantastic infographics and videos. Others suggested improvements to social media UIs.</p> <p>I wanted to test out variations of social media UIs in a more ecologically valid environment, so I created Future Feed, a faux social media app that can be embedded in Qualtrics, allows you to vary UI features, and measures user behavior.</p> <p>With Future Feed, we could test interventions such as flagging and even experiment with newer methods such as &quot;provenance-enabled&quot; media. I was also able to measure user actions in the form of liking and sharing.</p> <p>Participants were first asked to interact with the feed as they normally would. This was followed by a rating task involving a balanced assortment of AI-generated and non-AI content they encountered.</p> <p>Given the large number of interventions, we used an adaptive experimental design that devotes more N to promising interventions. I thought flagging content was going to be the clear winner, but a version disclosing that the content was &quot;automatically flagged&quot; performed best.</p> <p>Most interventions have positive ATE estimates for AI accuracy, but as is common in the misinformation literature, some interventions simultaneously increased errors for non-AI content. The automatic flagging condition moved both outcomes in a positive direction.</p> <p>Our study is yet another reminder to assess how misinformation-reducing interventions impact perceptions of both genuine and false content.</p> <p>(thanks to @BrianMGuay who Zoomed into our class on measurement and emphasized this point)</p> <p>It is also a reminder that we shouldn&#39;t assume that what works for misinformation will also work for generative AI, where differences between non-AI and AI content are growing smaller with every leap in technology.</p> <p>I promised the students they&#39;d be surprised, and as is always the case when I teach experiments, I delivered on that promise.</p> <p>This class is modeled after @BrendanNyhan&#39;s wonderful project-based learning course on experiments.</p> Tue, 21 Nov 2023 03:50:37 GMT https://blue.feedland.org/?item=199597 https://blue.feedland.org/?item=199597 It's that time of the year. We're just about done collecting data on 2,000+ participants for my Experimental Research class. This semester, we settled on AI image/video discernment as our key outcome, and 11 students submitted their own interventions. The interventions were modeled after popular methods in the misinformation literature such as flagging and digital literacy skills. Some students created fantastic infographics and videos. Others suggested improvements to social media UIs. I wanted to test out variations of social media UIs in a more ecologically valid environment, so I created Future Feed, a faux social media app that can be embedded in Qualtrics, allows you to vary UI features, and measures user behavior. With Future Feed, we could test interventions such as flagging and even experiment with newer methods such as "provenance-enabled" media. I was also able to measure user actions in the form of liking and sharing. Participants were first asked to interact with the feed as they normally would. This was followed by a rating task involving a balanced assortment of AI-generated and non-AI content they encountered. Given the large number of interventions, we used an adaptive experimental design that devotes more N to promising interventions. I thought flagging content was going to be the clear winner, but a version disclosing that the content was "automatically flagged" performed best. Most interventions have positive ATE estimates for AI accuracy, but as is common in the misinformation literature, some interventions simultaneously increased errors for non-AI content. The automatic flagging condition moved both outcomes in a positive direction. Our study is yet another reminder to assess how misinformation-reducing interventions impact perceptions of both genuine and false content. (thanks to @BrianMGuay who Zoomed into our class on measurement and emphasized this point) It is also a reminder that we shouldn't assume that what works for misinformation will also work for generative AI, where differences between non-AI and AI content are growing smaller with every leap in technology. I promised the students they'd be surprised, and as is always the case when I teach experiments, I delivered on that promise. This class is modeled after @BrendanNyhan's wonderful project-based learning course on experiments. <p>It&#39;s that time of the year. We&#39;re just about done collecting data on 2,000+ participants for my Experimental Research class. This semester, we settled on AI image/video discernment as our key outcome, and 11 students submitted their own interventions.</p> <p>The interventions were modeled after popular methods in the misinformation literature such as flagging and digital literacy skills. Some students created fantastic infographics and videos. Others suggested improvements to social media UIs.</p> <p>I wanted to test out variations of social media UIs in a more ecologically valid environment, so I created Future Feed, a faux social media app that can be embedded in Qualtrics, allows you to vary UI features, and measures user behavior.</p> <p>With Future Feed, we could test interventions such as flagging and even experiment with newer methods such as &quot;provenance-enabled&quot; media. I was also able to measure user actions in the form of liking and sharing.</p> <p>Participants were first asked to interact with the feed as they normally would. This was followed by a rating task involving a balanced assortment of AI-generated and non-AI content they encountered.</p> <p>Given the large number of interventions, we used an adaptive experimental design that devotes more N to promising interventions. I thought flagging content was going to be the clear winner, but a version disclosing that the content was &quot;automatically flagged&quot; performed best.</p> <p>Most interventions have positive ATE estimates for AI accuracy, but as is common in the misinformation literature, some interventions simultaneously increased errors for non-AI content. The automatic flagging condition moved both outcomes in a positive direction.</p> <p>Our study is yet another reminder to assess how misinformation-reducing interventions impact perceptions of both genuine and false content.</p> <p>(thanks to @BrianMGuay who Zoomed into our class on measurement and emphasized this point)</p> <p>It is also a reminder that we shouldn&#39;t assume that what works for misinformation will also work for generative AI, where differences between non-AI and AI content are growing smaller with every leap in technology.</p> <p>I promised the students they&#39;d be surprised, and as is always the case when I teach experiments, I delivered on that promise.</p> <p>This class is modeled after @BrendanNyhan&#39;s wonderful project-based learning course on experiments.</p> Tue, 21 Nov 2023 03:48:29 GMT https://blue.feedland.org/?item=199595 https://blue.feedland.org/?item=199595 It's that time of the year. We're just about done collecting data on 2,000+ participants for my Experimental Research class. This semester, we settled on AI image/video discernment as our key outcome, and 11 students submitted their own interventions. The interventions were modeled after popular methods in the misinformation literature such as flagging and digital literacy skills. Some students created fantastic infographics and videos. Others suggested improvements to social media UIs. I wanted to test out variations of social media UIs in a more ecologically valid environment, so I created Future Feed, a faux social media app that can be embedded in Qualtrics, allows you to vary UI features, and measures user behavior. With Future Feed, we could test interventions such as flagging and even experiment with newer methods such as "provenance-enabled" media. I was also able to measure user actions in the form of liking and sharing. Participants were first asked to interact with the feed as they normally would. This was followed by a rating task involving a balanced assortment of AI-generated and non-AI content they encountered. Given the large number of interventions, we used an adaptive experimental design that devotes more N to promising interventions. I thought flagging content was going to be the clear winner, but a version disclosing that the content was "automatically flagged" performed best. Most interventions have positive ATE estimates for AI accuracy, but as is common in the misinformation literature, some interventions simultaneously increased errors for non-AI content. The automatic flagging condition moved both outcomes in a positive direction. Our study is yet another reminder to assess how misinformation-reducing interventions impact perceptions of both genuine and false content. (thanks to @BrianMGuay who Zoomed into our class on measurement and emphasized this point) It is also a reminder that we shouldn't assume that what works for misinformation will also work for generative AI, where differences between non-AI and AI content are growing smaller with every leap in technology. I promised the students they'd be surprised, and as is always the case when I teach experiments, I delivered on that promise. This class is modeled after @BrendanNyhan's wonderful project-based learning course on experiments.