Quantifying Uncertainty in Clinical Trials (LA Tech Week) Dr. Michael Fischer https://desci.nyc/transcripts/quantifying-uncertainty-in-clinical-trials-la-tech-week Automatically generated transcript. May contain errors. [0:04] Cool. All right. Well, thank you guys for coming. Um appreciate you everyone for coming out. Hope it's been a good LA Tech Week. So, I started this I think everyone here is sort of interested in like clinical trials. And um before I did a PhD, I was studying computer science and like natural language processing. And uh I I think that like I'm from New York. It was started this group called Decentralized Science New York and we've had like a bunch of speakers and we've had like Brian Johnson, we've had uh these people who do um the enhanced games, we've had people do talk about gene editing. So, if you guys are ever in New York, um drop your email onto this website called dsci.nyc [0:46] and um add your email address and you'll be on our our email list. We are also starting one in um San Francisco as well. And it's just kind of like a a speaker series where people who are interested in science come together. There's like a speaker and then people chat afterwards. So, normally I don't speak at the event, but I organize it, but uh today I'm I'm going to speak on this project that I've been working on called um Endpoint Arena. [1:10] And um I was just very interested in clinical trials as I mentioned before that I think that there's going to be a lot more clinical trials in the future. Um it's becoming very easy to um vibe to discover to create new drugs and to develop them, but it's still rather tricky to be able to like test them out. And the way like even like it used to be crazy to think that like oh, you're going to be able to vibe code an app, right? Like that seems ridiculous that you could be like build an app to um tell me when the subway is going to be here and like 5 minutes later like it would just like do that and you'd be like build for New York or do it and then it would be like build it for DC and it would do it. Um I think in the [1:51] future that we're going to have the ability to vibe code um drugs. So, you're going to be able to just say like, build this drug. And let me just make sure this is it's recording, right? Yeah, yeah, we're recording. Okay, perfect. And and you're going to say like, okay, have the ability to bio code this drug for me. And it'll it'll build the drug for you, but it's still going to be rather tricky to be able to test it out. And, um, so it's going to be a you're going to need a lot more, um, you're going to be need to do more trials, you're going to need to have a lot more coordination, you're going to have to change, um, how drugs are incentivized, how people are incentivized to take different drugs. [2:27] And, um, part of the one of the harder parts of that is trying to figure out the uncertainty if a if a drug is going to work or not. Okay. Okay, so in this talk I'm going to sort of talk about a couple different ways to assess, um, if a clinical trial is working. So, um, the first one is we are going to talk about how to use like a prediction market. So, this is the first one is through like a financial lens. Of, um, uh, if a if a clinical trial is So, how many people If you guys are familiar with like Kalshi or Polymarket, um, that you can like put on like, oh, is so-and-so going to win the presidential election? And it'll say like, yeah, it's [3:08] going to win it. And you can be like, okay, this is cool. Or if if the team if the Yankees or the Giants are going to win a sports game, you can say like, who's going to win? And you can have like a very accurate. But this isn't very interesting because it's just for people that are like betting on sports. There's no like There's no you you you create a lot of volume and you create a lot of uh, with people trading, but you don't actually create any like useful information. So, if we create a prediction market around clinical trials, you can actually like generate useful data that has like actual that it will that will be useful for a society as opposed to just like, um, generating volume on if a if a trial is going to work or not. [3:41] Um, the second way is, um, you know, how we make the prediction visible. Um, let me see. And, um, Okay. So, you know, what is a clinical trial? So, um you know, for people that don't know, it's just it's a way of testing to see if a drug is working in people. And you know, you need to weigh the benefits of testing the trial if you think it's going to work in people or you think it's going to cause a harm in them. And you need to sort of compare it to like the standard of care, which is hard part. Um so, during like phase one, um you're testing basically like the safety of it, if it has any bad side effects, and how you dose it. Um during the second phase is sort of the harder [4:22] phase. It's like um if it actually works and if it does the thing. And then in the third phase, you sort of compare it to like how it compares to things that are on the market. And then in the fourth phase, it's like, you know, the long-term after 5 years, is there any is there any sort of bad side effects on this? Um So, um why I really like clinical trials as a as a method for testing things is because I there's this concept called decentralized science, which is where people were trying to build out um all these like new-age protocols on the internet using crypto, and I was part of this, too. [4:58] And um it was a really like people had these pie-in-the-sky, like, oh, we're going to test like quantum sensing, we're going to test like longevity, we're going to test all these types of things. But none of them really worked because it was they were trying to like rebuild a lot of the infrastructure of science. And it so, they didn't really like none of them like had a lot of stickiness to them. What I really like about clinical trials is like they're sort of backed by the government, they're backed by these legitimate companies like Pfizer. Um you have to like pre-register you know, the FDA oversees them, you have CROs that are that are that are uh participating in them. And so, you know, um people enrolling in a trial, you have to pre-register your hypotheses, so you can't be changing things. And you you know, you have clear end points of of [5:39] what defines a success. So, if you're testing a a cancer drug, you have to see like, is the tumor going to is the patient's going to live longer after a certain amount of time? Is the tumor going to shrink down after a certain amount of And these are like pre-registered hypotheses. So, we're fitting very much into the um existing infrastructure of science. So, um Um So, why why is it important to know if a clinical trial is working? Um because clinical trials are very very very expensive. And there has not really been a great way of determining if a clinical trial is going to succeed or fail. Um it [6:21] was sort of like pre-moneyball. So, people would just say like, "Oh, I'm sensing that this drug is this is going to work well." There was not There wasn't any sort of like metrics that could be added to it. Also, it's very helpful um for clinical um decision-making. If you're a doctor, you want to be able to know if you can like uh recommend this drug to a patient or you're sort of sending them into a meat grinder. [6:44] Um you also want to know if you're a patient you know, I think a lot of patients go into a clinical trial and they don't really understand the risks that are associated or their like likelihood of success. So, some patients might want to um just continue with their with their disease and not participate cuz there might could be some really bad side effects or they they look at the cost-benefit of it and they don't think it's worth it. But, I think a lot of times um trials are shown to are projected to be like um you know, this is going to work and they think it's too They think if there's the the likelihood of success is uh very bad is very is much better than it is. [7:17] Um trials are very expensive. So, if you're not know if you don't have an accurate measure of if a trial is going to succeed or fail, um you can be spending a lot of money. So, you like how private equity does a lot to allocate capital. If you can um better allocate capital towards trials that are actually going to succeed, you can get more You can get a lot more trials. Um another one that is you can hedge risk um if you were a better understanding of clinical trials, so if one drug succeeds, that might cause another drug to fail, but if you sort that you may want to head create a hedge on that. Or if you bundle a lot of um, drugs together, you can actually create like a a basket of um, trials that have a uh, that can be much more investable [7:58] than each individual trial. So, if you guys read Andrew Lo's paper, he talks about biotech as a sector being a very good sector, but you sort of have to like um, bundle these things together to be make it investable. Um, so it's really important that we have a better way of um, uh, tracking which of these drugs is going to be successful. So, a really good way of tracking if a drug is going to work or not, I thought would be to make a prediction market. Um, it has it's over it's uh, has oversight by the FDA, so it has a and the FDA can act as the oracle, which is an important part of a prediction market, which means that you need someone to decide at the end of the day if a if the thing happened. So, [8:40] did it rain today or not? And you have the the weather channel being the oracle. Or who won the election? You have the um, Federal Election Commission as the oracle. Or you have like spot price on Bitcoin, and you have like um, finance as the oracle. So, using the FDA as an oracle is a great idea. Um, and basically the way it works is you have people who are betting based on information that they have, based on papers that they've read, based on um, uh, information that they know and stuff that they've injected in themselves. And the as as information changes, um, the price of the of the stock of if a clinical trial is going to succeed or not, changes also. [9:22] And I'll show you how it works. Um, so we created this thing called Endpoint Arena. And um, basically the way it works is all the data we look for around um, 30 different clinical trials at a time, and we do a lot of work to predict which trials are going to be coming in. So, even knowing when some of these trials are going to be announced is a little tricky. So, we scrape all the different public all the different publicly traded pharma companies like news and pipeline section to understand which drugs are going to have an announcement in phase one, phase two, or [10:02] phase three. And we train within the next 60 days and then we create a market for those types of drugs for those things. And then people can like buy and sell if the drug is going to work. And for each clinical trial, what happens is there's like a primary endpoint. So, a clinical trial will have like a primary [clears throat] endpoint or one or two primary endpoints and then a couple different second you can jump in. [10:30] >> Sure. >> Um >> We have a chair here. >> Um we have a a primary endpoint as well as there's a chair here also if you want or you can stand. We have a primary endpoint and then we have there's like a bunch of secondary endpoints and we do a lot of work to figure out which of if there's multiple endpoints, which is the one or which is the most important one so that will that will be predictive of if this trial is going to be successful. [10:54] Um you guys can check it out at endpointarena.com. We have some info about each disease so like um uh you know, how big of a problem it is >> uh and stuff like that. >> Um so people are betting on this quite a bit um during the trial run that we had but we also allowed agents to bet on it. So, like um AIs to be able to bet yes or no. And I think this is actually like a very interesting thing for for a couple reasons. So, firstly um people are these these types of questions that in science are very very hard. Like you [11:35] could be you could do a full 6-year PhD on one of these clinical trials and still not develop a definitive answer. And the reason is is because the questions that are being asked are sort of outside of the realm of what is answerable by a human. And but for and so AI agents have been done a very good job of like um summarizing emails, writing emails, writing code, but I think like the next level of um questions that will be answered by these some of these science agents are going to be in the realm of like science. [12:13] So, they haven't yet saturated this. So, what happens is is each agent reviews um the position of looks at the trial, has access to the internet, um has an understanding of um like how likely a trial how likely a trial is to succeed it can Google it, it can look at clinicaltrials.gov, it can look at Twitter, it can look at archive, it can look at different papers, and it develops a um hypothesis if it's going to work. So, you can see for one of the models, for one of the trials, um Claude bet yes. Um Claude thought it was going to succeed, um but ended up not betting. Um and some of the other ones bet uh no and [12:55] you know, put a certain position on it. So, um you know, each model has like a very different understanding of has a different viewpoint on if things are going to work or not. So, to do this, we basically um had a common prompt for all of these different ones. So, um you know, it was just like uh you are a with the prompt for each of these was like you are a AI agent. Here's um your you're an AI agent that is like a biotech investor. Um do the best you can and say yes, no, or hold, uh, if you think this trial is going to succeed. [13:31] And then also give us your, um, position size. So, like, how much do Here's how much you have in your wallet and here's how much and how much you want to bet. And so that they could all do that. Um, we also gave it access to like, um, news and contacts. So, um, this is the sort of if a trial like the different, um, trials that are upcoming or different news announcements, so it have access to all that as well. [13:59] And, um, GPT-5.5, uh, did the best here. Um, Lama did second best. Each had a very different, um, profile, but we're going to be putting a lot more effort into, um, uh, benchmarking these different AI agents cuz I think it's very valuable, um, to understand which AI agents are better or worse at scientific tasks around clinical trials. So, um, benchmarking is like a very big field in AI because, um, you guys Have you guys Have you guys heard of like terminal bench? It's a It's a It's a benchmark for, um, uh, predict for for like task computer tasks. So, like, um, coding or, um, [14:43] like, uh, setting up your Unix environment in a certain way. And these these these benchmarks are so important because, um, the the AI labs use these benchmarks to be able to, um, iteratively test to see if their model is getting better or worse. So, for each training run that they do, they'll say like they'll have like 10,000 different benchmarks and they'll say like, how did we do on Let's Let's do a run and we'll say like, okay, we did better on this benchmark, but worse on this benchmark. [15:12] And then they'll backprop sort of all their all their all the changes so that they can try and get as much as as good as possible on across multiple different benchmarks. But, there has not been a great There are some benchmarks for science, but I think adding more into there into them will be important. Um So, um but I'll take a step and say like, okay, who are the types of people who are betting on this these types of markets? [15:41] Um the first is um biotech people who who want to hedge. There's sort of two main companies. Two two main sides for any market. You need someone who's willing to buy and you need someone who's willing to sell. So, um one of the areas that we've been exploring is working with biotechs to hedge in case a clinical trial fails. So, they're sort of betting no on the trial. [16:01] And uh why is this interesting is because it allows for them to like allocate their capital in a more efficient way. They don't need to save like they can if they have if they're running maybe five trials, they would have to save some of their money in case one of them succeeds uh in case one of them succeeds so that they can, you know, have money left over. [16:21] Um but, if you if you're able to have a better expected value on if your trial is if your bundle of assets is going to work or not, you can actually maybe run instead of five, you can run six or seven. So, biotech companies are sort of betting on um uh this this working and then on their drug not working. And then, who's betting on it on the other side is like um people who have to pay. People who There's some people who don't want a trial to succeed and those are like health insurances and um big sort of um big uh employers or, you know, health health employers, health insurance companies who are on the hook if a trial succeeds because, um you know, Ozempic now [17:01] everyone who's on that plan has to take like a $2,000 uh extra drug year drug. And so, those people There's those are sort of like the structurally the the two sides of the market here. Um we do a lot on choosing the market. So, I talked about a little bit this earlier, like um there's multiple sides, there's multiple things that you could be the the endpoint, the the oracle for a market, and we did a lot of work to try and figure out which is the primary, which is the actual endpoint that is important for determining if this drug is going to be a clinical success. [17:39] And here's how well each of the models did in benchmarking. So, you know, Claude did very well, GPT-2 less well. We're going to do more with this in the future, but these are sort of some preliminary results. [18:04] Oh, to the next slide. So, one of the things that's interesting is like um you know, what made certain models uh succeed or fail was uh oh, there we go. Whether a model didn't succeed or fail was interesting, so when a model got it right, like why did they get it right? And when a model got it wrong, why did they get it wrong? So, they got it right when they did a did a good job looking at the science. [18:32] So, when they're like, "Okay, we're going to look at what the endpoint that we're trying to predict is." And his Oh, so an important part is like, how do we calculate how do we figure this out? Is we when when a model decides whether it should bet yes, no, or hold, we also ask it to give us a reason for it. So, it said like, "Okay, this reason for this model that I'm I'm betting this amount for this reason." [18:57] And so, we have like a we have a reason we have a reasoning trace. So, when we were reviewing the data and and looking to see why things went well, models did well when they were able to predict when they had looked at the actual endpoint and then determined like, "Oh, this is going to work for this specific reason." and they sort of worked backward from a specific endpoint. [19:17] Um and they also did well when they balanced risk in their entire portfolio. So, some things didn't do very good portfolio management and went were too confident in one thing working and not hedging risk around everything else. Um when models did badly was when they sort of didn't look closely at the specifics of a trial. So, um they said like, "Oh, uh in general phase three things work uh 70% of the time. I'm going to bet that this trial is going to work 70% of the time. So, uh I'm going to go with that." [19:51] Or they sort of looked too carefully at um too closely at uh when they were using the web at what news articles or people on Twitter were saying and they didn't do enough of their own reasoning but relied on on other people. Um and they didn't sort of like look at the specifics. So, um one of the next things that we're trying to look at is um getting people to fill out their cuz at the end of the day all these different models are sort of beating each other over the head and have more or less the same information. [20:22] And uh we we just came out with this and and it's still very early but I I wanted to show you guys is that it's a way for patients to upload um data on how well um certain drugs are working for them. So, in particular around like um GLPs, we were talking a little bit about like label expansion on GLPs for like alcohol abuse. So, um people who aren't enrolled in a trial might want to get an edge on this betting market by uh enrolling in a what we're calling like a shadow trial and being like, "Okay, um like I noticed that they're on clinicaltrials.gov there is a uh trial [21:04] on on some acne or sleep or GLP-1s for addiction or sleep or alcohol. And I want to like I know I I'm taking that GLP-1 and I've had an issue with alcohol and I can see that you guys have a market around that and I think that it's mispriced. And I'm going to upload my data into this like app and then have the app calculate how likely the the trial is to succeed or fail. [21:33] And then bet based on if there's like a mispricing of the clinical trial on the website compared to my own local estimation of how likely that trial is to succeed or not. And what I really like about this is that it's incentive it creates an incentive mechanism for people to take a lot of N-of-1 data and to like start to aggregate it together into a larger thing. So I think a lot of big problem with a lot of this the biohacker community these days is that it's a lot of people testing out individual things but not aggregating it together to be able to like make bolder bigger bigger claims on if they're working or not. Like there's a lot of [22:13] anecdotal evidence where like I did XYZ and I got this but we don't know if that's like a placebo effect or if there's anything but what's cool about this is it's sort of using the infrastructure from clinicaltrials.gov and creating an incentive mechanism for people to then calculate if something is working and then share their data. So you could be like what I've been taking this peptide it helped me a lot. Here's what and then they publish their data cuz now you want to get other people involved. [22:43] Like you want to like pump the price of the stock. You're going to be like oh I have this insight around like Chipotle or whatever like it's going to it's going to moon. And so you people can do that here. You can like say like you can show your primary you can show your data. You can develop a reputation and you can like get people to be into share their data. [22:58] So I'm very excited about this. Um and then we're also working on a a benchmark called Endpoint Bench, which is basically a new will be an addition to Terminal Bench in their science um science environment to allow for um these new models to be able to train and learn on new clinical trial data. And you know, it's really important and I just wanted to stress quickly that it's really important we develop these types of new benchmarks for AIs because the most a lot of the um [23:38] benchmarks that are being used to test these are becoming saturated. So, a lot of the coding challenges, a lot of like humanities last exam, agents last exam, GDP bench are becoming sort of like saturated because these models are improving so quickly and recursive self-improvement is is right on the horizon that they can learn pretty much anything that they want, but we have to be giving them questions that are outside of it. And I think that in general the realm of science is is going to be a very fruitful place for that to happen. [24:10] So, um we're going to say we're developing Endpoint Bench, which is like an open benchmark for clinical trial forecasting. There's there's two components to it. The first will be like a historical archive, which will allow back-testing on clinical trials, so you can do online testing on if if if your model is training, but then there'll also be like a new live benchmark for for all the models to compete with each other on new fresh data. So, a lot of a lot of a big problem with a lot of these benchmarks is you put your benchmark on the internet, the model trains on the internet, and it then realizes [clears throat] that oh it already like learned the answers just by [24:50] nature of like scraping all of it. So, people have done a lot to like try and keep these benchmarks off off certain parts of the internet or for them not to or for these models to not train on it. It does it sometimes works, sometimes doesn't work. But what's really cool about um clinical trials is there's 7,000 new ones a year. So, there's going to be like a an evergreen ability for uh clinical trials to to to test on new trials, which is very cool. [25:15] So, um in the benchmark, we have um we have a number of tasks that we've triaged. Um the first is getting models to uh predict a clinical trial. Um the second is to be able to um uh check around like uh if a patient is eligible for a trial or if a trial what if a trial fails, why it would be a why it would fail, um how long a trial is going to last, if it uh why someone drops out, if it will reach enrollment enrollment, um if it will reach its enrollment, um if there will be any sort of uh anyone dies or anything, or figuring out dosing. So, we're not going to do all these at once, but um you know, we're starting with with the [25:55] first one on on like trial outcome and getting it predicted, and then we'll add these other things in over time. And just to summarize here, so um at first we sort of talked about like um the you know, understanding risk associated with a clinical trial um from a financial perspective and understanding, you know, how we can understand the risks associated with a clinical trial and how likely it is to succeed. And that's interesting, but um it doesn't bring any new information or data into the into the play into into the fold here. [26:28] Um so, to fix that, we're working on like this anecdote app where people are able to like upload data and and earn money or or trade based on on new data that they bring into the system. So, it's it's adding value um through data. And then lastly, uh I think a very important thing is this endpoint bench, which will allow uh AI agents AI new new sort of new labs and AIs to like learn and train on scientific data uh specifically around clinical trials. [26:57] Um If you were hiring with a team of three people now, so we're very interested in clinical trials. If you if you have a background or you know someone that is interested in it, give us a follow on Twitter if you want to keep up. And of course or if you want to DM me any questions. And then if you want to check out the site, of course endpoint.ai or you can you can scan here. [27:19] Thank [applause] you. ANYONE HAVE I'M SURE SOMEONE HAS A QUESTION. >> YEAH, GO AHEAD. >> YEAH, can you go back to that historical archive part when you're maybe a few slides more back? Keep going. It's essentially what you were saying how you're going to test the agents versus like you know, streams of new data versus like a historical drug that or historical trial that's already been completed. [27:50] So I guess my yeah, so my question is, you know, all of these agents, you know, whatever model you pick are trained on the whole corpus of the internet. Clinicaltrials.gov is on the internet. So how can you actually have a clinical trial that's like siloed from these agents? Like just >> cuz you have new you have new stuff that comes out. So they're trained up until January 1st, 2026. [28:11] >> Okay. >> And so then you can actually that's one way to test if a um basically test on new data. So they have like a they they'll publish like oh this model finished training on this data or this model stopped collecting data on this date. And then sometimes people test it on stuff that happened after that date. But it's that's why yeah, it's like a a mechanism by which you can like test if things are actually working out. That it's called like memorization or benchmark maxing when some of these data leaks into the actual model. Some of the benchmark actually leaks into the model and then it learns it, which is like a a big issue. And there there are ways that are pretty [28:52] complicated. Like they will um try and uh train it so that it doesn't pick it up, but it's like basically impossible to do it because um I know a guy who's training a model on um clinical trials. So there So the people who are betting on our plat this platform are just standard models, but we're starting to get some people who are developing specific models around to to trade. [29:19] And they they need to do a lot of backtesting. So they need to be able to like make it so that to be able to test it they want to be able to make it so that the model doesn't have the answer. But it's like basically impossible because there's like So they'll strip out the name of the drug, or they'll strip out like the name of the sponsor or something like that, but there's so much related info around the timing or the NCT number or whatever that you can basically with just a few bits of information like get what happened back. So it's really really tricky. So that's why having these like And it's also very like for a lot of these RL environments it's tricky because you have to basically pay people who know what they're doing to come up with these new types of tests. Um and [30:00] it's expensive. And once you they once the model learns it then you can't really use them anymore. But what's cool about with clinical trials is there's new ones coming out each every day. There's 7,000 divided by 365. So there's many new ones each day. And um uh so there's a lot of uh ability for them to be able to predict um there's a lot of new test cases that come online sort of naturally. [30:24] >> Except thank you. >> Yeah. >> Yeah, could you explain a little bit more about how the anecdotal evidence you have informs your model and like how that helps into the prediction and what kind of challenges you have in collecting that and securing that? >> So for the models to do it? >> Correct. >> I don't have a ton of information cuz we've been focusing on creating like the infrastructure, so like the markets or the data or the RL environments. We don't have a ton. I haven't really taken like that big of a deep dive into what causes a model to work well or not cuz our goal isn't necessarily to build a model at this moment. [31:02] Yeah. I think that but there are a lot of companies out there that are creating models to predict clinical trial outcomes because it's a very If you can do it well, you can use it to like sell it to a hedge fund or trade on the stock market. Um but we haven't done that yet. >> The goal here is then >> The goal here is to create the markets to allow people to trade to allow people that have developed those types of things to trade or to create the RL environments to allow people to um to build those models or to create data new types of data sets to allow people to trade. [31:39] >> To allow people to build those types of models. With the data you have, what's the biggest challenge in creating a model? >> Oh, the challenge in creating a model? I mean, you got to need a I think it's becoming easier. So, you know, there's you can fine-tune an open open weights model like GLM or uh any of these new models out there or develop like a new test harness. Um I think it's compute. You have to know what you're doing. You just have to It's hard to train a model. It's not It's not like a It's a whole It's a whole project, right? Like it could take a million dollars, but I think that cost is >> slowing down significantly. [32:14] >> Yeah. >> Just a follow-up question. So, if if you don't you haven't defined the scope of information, so how do you build the benchmark for that? >> Benchmark for >> Just your motion model benchmark, but >> Oh, we're we're benchmarking existing models. So, we want to We've benchmarked, you know, all the top like off-the-shelf models using a common prompt, but we haven't built our And we have benchmarks to allow people to test their model. So we we've developed two benchmarks. One is like how well do you do on new trials and um well and and on existing trials and then how well do you do on new trials. But we [32:54] haven't developed our own model and benchmarked it. We but we have benchmarked the top mo- top um closed source and open source open weight closed weights and open weight models. >> Uh because I saw you were building kind of an environment which allowed a lot of people to play with. So that means you need to define the the the timeline of the information to which to which date by which date >> Oh, yeah, yeah. So we have different cutoffs on we have like a backward testing one that goes up until the beginning of this year and then an online one which is more like a leaderboard of how well things that have already been trained on are doing. [33:36] >> Okay. >> Thank you. Uh yeah. >> Go ahead. Um I'm originally from pharmaceutical business and I started my own company. So we are focused on development platform and 3D models particularly from oncology area. And we see for example the main issues not only in clinical trials but in preclinical trials because the industry still used very old models and animal models or something like this. That is we create biological models and 3D models on quality patient population together. [34:08] Uh is potential is there opportunity to implement some um uh biological data in your platform? I mean like existing experiments and then put this in your on your platform. >> Um I don't think we want to do that. So we want to create the market that will allow people to develop an AI model to use bio- to use a harness that will include that. Our goal there's so many different ways that you could do it that if there's no way for us to like track all of them. [34:40] But if you were to do it, then you could like then trade on this platform. Like that's the problem we don't know yet. It's still an open question like does putting say you have like a million dollar budget. Does spending 900 Does spending $900,000 on data and 100,000 on training the optimal thing? Or is the optimal thing to spend it all on data and or mostly on training and on data? [35:01] Which is the right and maybe you want to allocate your your data budget differently. So you might want to spend all of it on collecting a patient one-on-one or you might want to spend it all on it on a database. We don't have the answers to all those questions, but the goal of this project is to um create a marketplace to allow people to figure out those types of questions. So the model that is best at that will be able to trade. [35:24] Yeah. >> All right. Yeah. All right. Yeah. You're everything. >> Um for the for the anecdotal things from theoretical patients coming in. I guess like in theory if everyone is honest, that would be great. And but I don't know. Like people could just lie about things. It's like one one part. And the second part is trials are very well controlled. Like you can't be taking other drugs or things and people out there maybe the different age or taking something else or you know, have all these other factors that >> Yeah. [35:58] >> could be different than the trial population. >> Yeah. So I think what's cool about this platform is that it's a prediction market. So you're actually incentivized to tell the truth. So at the We're We're not the one who's like conducting the trial. At the end of the day, the FDA is still the one that will do the readout on the on the controlled study. But here um you're basically going to be left like holding holding the bag if you publish bad data. So you could say like this drug worked for me. You can see that I bet $10 on it. And then um if it's [clears throat] incorrect, you're going to like if the if that doesn't actually like come to fruition at the end when the FDA announces the result, like you'll the [36:39] that person will be left holding the bag of like this incorrect data that they published. >> They're going to be a lot like conspiracy theoretically there's a lot of money on the line. Like somebody puts a lot of money they get a bunch of people to post. >> Yeah, so you could try and do some like wash trading or lying. But I think the I I guess the idea here would be that like transparency and open markets would ultimately beat that type of thing would be the hope. [37:11] Because at the end of the day there is like a there is a truthful ground point. So you could try and I guess someone could like try and rug someone by like creating a lot of like hype. Oh, this is definitely working and then creating a bunch of things and then run it up. Um But I think other people would be able to call them out. It's not just like a like a scam where there's nothing to like oh this doge dogecoin is going to be like the next greatest thing in the entire world. Like someone could come in and be like hey look this data did not result in me getting better and you could try and run it down equally. I mean you have to rely it's a different way of thinking about doing science for sure. It's based on like a market approach as opposed to like like a a gold standard. And that the hypothesis is here is that that that's will be a better approach but I think we still [37:52] have to like prove that. Yeah. >> Well yeah, I was just curious. I'm just going off of some models I'm not familiar with all of them but you know, one of the most popular or I guess the most visible models right now are Palantir. >> Mhm. >> Um so I was curious if like you were to prediction markets you're primarily focused on using like Palantir for like the benchmarking and or if there's other types of models that you use. [38:17] >> I see. Like a random forest or >> a mathematical reasoning models. >> Um we It's a great It's a great idea. It's a great idea. We didn't do it, but I think we should. Um this was I think more of a um something that was relatively easy to do. Um our goal was just to more benchmark the top 10. Um but yeah, it's a good idea to try maybe maybe a regression model or other things that could poss- that would possibly be interesting. I think the goal here was to just get something on the board and then have people come and say like, "Oh, we want to try a random forest." and have them create an account and then be like, "Our [38:58] username is random forest 1 2 3." and then show that it works well or not well on the leaderboard. >> Yeah. Yeah. Yeah, so you're not working on developing models, but you are working on a market, right? And um when you're benchmarking, there's a lot of like reasoning traces. Great. Um do you do anything with the reasoning traces? Because I mean, that's a very easy way to improve models if a company was working on primarily um improving models for uh predictions, right? [39:30] >> Yeah. >> Um what do you guys do with the reasoning traces? Do you >> We don't do anything with them besides this um thing here that we just was just for this slide. But yeah, we we we could be doing more stuff with that for sure. And I need to do a better job of looking at how Terminal Bench um if they do any stuff with the reasoning models or with the reasoning um chain of thought sort of thing because right now they just kind of look at the final goal and score that. I don't think that they do too much with like how it got to that goal. Um but maybe there's like an interesting thing to do [clears throat] to do with that. [40:11] Yeah. >> I I have a question about the market. So, clinical trials can last multiple years and new information might come in every few months. So, how do you keep people engaged? Because in sports betting, 2 hours the game is over or even if you bet on something like a season outcome, every week the price changes because a team loses or wins. So, how do you keep people engaged? [40:35] >> I think it's a great point. Um we we create the market like 60 days before based on first scraping the data from the RSS feeds of the sites. Um but it would be nice if we could create more momentum and movement and velocity in the trials. So, I think maybe with the anecdote thing, there is room to um do more stuff where people could be like, "Oh, this is really working well for me." and like publish a live stream of new data and being like, "Okay, this is cool." But generating velocity in these types of things is is very important. Otherwise, it just stagnates just from a product point of view. [41:15] Um because otherwise people lose interest. But I think this is one way. I think another way is to create um ETFs around bundles of clinical trials, um which is something I can talk more about to people who are interested, but basically I'll give you a hint is like instead of just doing um trading on clinical trials, you could trade on technologies, targets, um phase one, phase two, phase three, and you could bundle those into ETFs style things and allow people to do that so that there could be a lot more like movement. [41:45] Um yeah, but it is it's a big problem. And we try and create it 60 days before, but yeah, keeping people's attention I think we were also talking about before um the FDA is trying to make trials shorter and hopefully they do do that and that would help that as well or smaller. Um but some things are just like take time, right? Like if you have like uh Alzheimer's thing, it's it's just not going to be like a 5-day thing. It's going to take longer. [42:11] >> I guess it could to on the bailout, too. If it's like a 2-year horizon, but you 10x your money in 2 years, it's still a better return than, you know. >> Yeah. There's all sorts of fun things you could do. You could do like a parlay. If this one fails, but this one doesn't, or Uh there's all sorts of random things you could you could try and precisely do so. [42:32] Anybody else? We'll just go back to this slide and um Yeah, I think that's it. But thank you guys for coming. Really appreciate it, and I'll BE AROUND AFTER. >> [applause] >> OH WAIT, BEFORE EVERYONE GOES, LET ME JUST GET EVERYONE TO SMILE. SMILE. 3 2 1. Oh, in there. All right, this way. 3 2 1. Amazing. Well, thank you. Thanks for coming. What kind of doctor are you? [43:37] Yeah, man. I'm visiting, so Oh, you're a resident? Really? Oh, so you are you Yeah.