My goodness, week 10! I can’t believe that it’s already August! :)

It makes me sad to think that I’ll be leaving Austin next week… time has flied by and I feel that I have really grown as a researcher by going to the office, being in the lab environment, and seeing my PI daily and meeting in person.

This week, we held an annotator meeting of minds discussion; where we discussed the annotation task and updates.

Open questions to clarify are:

• Two part claim: Intervention nuance of some abstracts need to verify both drugs if claim has 2 drugs if one of the drugs modulates other, otherwise not useful for verifying claim => Defer to medical input and also rank relevant as useful for claim verification
• Beatriz: If some abstracts don’t have a results section but everything else is relevant, then put N/A as overall relevance
• Confusion about so many different cases for irrelevant or N/A => need to put more examples in doc
• Some studies in abstracts are observational studies or cohort studies, do we just mark them down through comment and do N/A?
• Sometimes use medical expert to infer the population, like MS medication intervention mentioned, use medical expertise to infer population is MS patient
• Inconclusive? Instead of conditional supports?
• Dealing with low quality abstracts, rely on tiering for quality of evidence ranking?
• Also medical expertise: What happens when the abstracts don’t answer the claim but the medical experts know the answer? Ozempic example about weight loss. Or if the experts disagree with the abstracts presented, because there’s better evidence that they’re familiar with?

We are working to clarify these questions and annotation scheme with Barry, and also confirm with the professors.

Concurrently, I’m also working with Gustavo on claim filtering to curate the dataset of high quality claims. He proposed an idea which I think could be really helpful to filtering, which is framing the claims as scientific questions. I am also trying some automatic methods with prompting. I’m really happy that Gustavo is so passionate about the project and really likes helping with claim filtering.

To more experiments, filtering, and annotation updates!