Talk & transcript
DeSciBoston: Prediction Markets for Clinical Trials
Dr. Michael
Transcript
Download textAutomatically generated transcript, lightly edited for names, scientific terms, and obvious transcription errors. Some errors may remain. Select a timestamp to watch that moment on YouTube.
Okay, so everyone can find a chair. Find a seat, and a seat here, here, come on. Anyone else have a seat? You guys see there's a chair here if you're daring. Here if you don't block the camera. Also someone can move this one if they want it. Does everyone want to sit here? The captain's seat. There we go. Otherwise you're gonna have to stand which is like fine but this is better than deciding where people are placed on the floor. So just to give you guys some intro, my name is Michael. I normally host these in New York.
And they're called the DeSciNYC. And I see a couple familiar faces here who people who have attended those. So it's good to see it. But this is the first time that we're doing one in Boston as part of Boston Tech Week. So that's very exciting. And I'm just going to move this over. So this is very exciting to be the first one in Boston. So everyone's very excited. Okay, so...
I'm curious how many people are part of like, if I'm putting like pharma, just to get a sense. Pharma people? Okay, cool. And how many people are part of like finance? Okay, finance. What are other people? Just you can tell me. Still in school. Still in school. Okay, good. Learning. Anyone else? Engineering. Engineering. Engineering. You have a cool startup. So, you're seeing a cool startup. Okay, so does anyone have any questions before I get started?
Good, we'll do a group picture just for two seconds. Three, one, smile. So, I was very interested in prediction markets and specifically, you know, a lot of times there's like prediction markets and they're being used for like sports betting and how many times Elon Musk is going to tweet in a day. And this didn't seem like it was going to like make the world a better place at all. So,
I thought, what if we could instead of using prediction markets for that type of thing, what if we could instead use them for science and engineering and things like that? So I've had this idea to do prediction markets and science for a very long time, but I didn't yet have the idea on what to do it. Because the original idea was, okay, what if you had an archive paper, and instead of having someone do a peer review on it, You could instead bet on it. That would be really cool. And the issue with Archive today is people, you post up Archive, you post a paper, or any sort of journal article. You post a paper and the reviewer isn't incentivized to do a good job. They don't make any more or less money if they do a good or bad job. What would be cool, I thought, was what if you could like...
incentivize the person who's reviewing the paper so that if they do a good job of reviewing and say like oh this this paper is going to have a really big impact in the future you'll earn more money or and if you're correct on that you know so you're sort of like trying to forecast out which types of scientific papers are going to be interesting or not or accurate the problem with this though is that like um academic papers are so crazy long and unverifiable So they'll have like three or ten or ten pages, three claims and ten page PDF. They don't publish their data, nothing in it. So there's no oracle that can be used as the source of truth in an academic paper. Where there is a source of truth is in clinical trials because they have very clear endpoints. And so I was able to put these, an endpoint is basically sort of like what they're measuring for in a clinical trial. And we'll talk more about that in a second.
But with this idea, now I could sort of piece together the idea of like prediction market together with a scientific outcome. And the other thing that's cool about this is you're sort of, I'll just give another high level thing that was sort of the motivation behind this was I wanted to tie together financial incentives with scientific excellence. And because today when you get a grant, you know, there's no accountability and you have no...
Besides your honor, you have no incentive to do a good job. Whereas in the financial market, you're incentivized to really research things very carefully and make sure that your predictions are accurate. And so Gordon Gekko has this thing that says greed is good and it makes the world a better place. And I wanted to create a system that would encourage greediness and that would encourage people to be very, very accurate with their prediction. So now we'll get into...
What it is. So, you know, as many of you know, clinical trials are really, really, really expensive. You know, you could be spending anywhere from $10 million to $100 million. These CROs take a lot of money, and it's a binary outcome. So you have a lot of unhedged risk. You know, 90% fail at some point if you start from the beginning, and, you know, you have a zero or a one outcome. It's not like...
Costco, say, where you have like, oh, we sold like 100 million gallons of milk, but we were expecting to sell 120 million gallons of milk, like where it sort of degrades slowly. You either have a zero or a one. But the problem with this is that this is a lot of risk for someone to take on. And so if it works, it works. If it doesn't work, it doesn't work. So, you know, how can we sort of solve this problem of riskiness in this business?
What you want to do is you want to be able to hedge these bets and make this a more investable product. So a lot of industries will hedge their risk in order to make a trial more palatable. So airlines will buy fuel swaps to flatten out the risk profile of how much fuel costs. Or the oil industry buys futures, agriculture sells futures, credit. Has a clinical credit default swap, where basically people pool their risk together and then sell that off as an asset. And this makes it more investable. But the problem is that clinical trials have no such product yet to be able to group together risk for a clinical trial because you can't price it. Because it's very hard to figure out how likely a clinical trial is to either succeed or fail. So how can we fix this problem?
And there does exist solutions, obviously. Like, people are developing AIs to do it, and people have expert opinions. But I thought the best way to do it would be to... Oh, there's another, like, another slight issue is that you might say, like, oh, Michael, like, why don't we just use a stock market to do this? And, you know, you can already bet on if... You can already do a very similar thing with Pfizer or with AstraZeneca.
But I would say that this is a very messy way of doing that. So if you invest in one of these companies, they might have 100 plus assets that you're investing in, all of which have a varying level of market cap that is like how much that drug is worth. So how likely is it to succeed? What type of thing is it trying to solve? Is it trying to solve diabetes or is it trying to solve some very niche rare disease? So when you're buying Pfizer, when you're buying AstraZeneca, you're actually really investing in something like an index.
And it's very hard to get the granularity and the specificity that you would actually want to be able to make these very precise predictions on which types of scientific outcome are going to work. So how do we go about pricing some of this risk is we use prediction markets. So prediction markets are really, you know, as many of you are here know about Polymarket. We know about Kalshi.
And you know that these things are excellent at predicting the future outcome of things. So whether it be like a presidential election or the weather or things like that because they basically pool together many experts and they let people bet sort of based on how confident they are. So you can not only have to have a... There's a couple interesting things about prediction markets. One is you bet based on how confident you are.
Two, you have skin in the game. So if you're right, you get more payout. You're not just a news expert or a pundit spouting off on the news. If you're a pundit, you can say as much as you want. With a prediction market, it's your own money in the game. So this is why prediction markets are very good at pricing risk around scientific outcomes. So what we're going to do is we're going to use this prediction market to build a science pricing engine.
This is a live product, but we'll do a demo in a second and I'll talk about just to give you a demo how it works, I'll tell you about some of the outcomes from the AI and what they generated and how the lecture goes. So this is Endpoint Arena. We went viral on Twitter a couple times. Who's seen us on Twitter? A couple people? Okay, cool. So the way it works is that we put up these prediction markets around specific endpoints on this website. So it's like Polymarket or CalSheet, but with various well-chosen endpoints that we look at. And so we focus on sort of phase one, phase two, and phase three outcomes, and each market is given a certain probability of success. So I can give you a quick demo. So here's how it works.
So on the website here, we have all these different endpoints for different types of clinical trials that we're looking at. So we'll look at this one on macular degeneration. This is for a specific on related to prostate cancer. So the way it works is, so this is the probability of that specific trial for a specific endpoint working out over time. Here is the stock of the company, of the sponsor. So in this case, it's Telex. And so this is its stock price over time. And so one of the things we want to do is you want to see how the divergence of the actual trials, a specific asset, changes over time and how that changes with the price. So you might see some divergence there, which is interesting. And then here you can sort of learn more about the disease. You can see how big of a burden it is.
The survivorship, you know, how big of a sort of a market this might be for it. You can also see we collate all the news related for this clinical trial. So every time like pharma has so many different sources of data, it's actually very hard to collate it all together, knowing when a trial is going to happen, when they're going to release the data, when a presentation is happening, when a conference is happening. So we collate all that together here so people can see it.
We have various information about it, you know, who's the sponsor, what's the ticker, how big is the trial. I'll link to the trial on ClinicalTrial.gov here so you can get more information about it. We'll go into this more detail, but we have a bunch of AI models that are betting on it, so you can see which ones believe that this, which positions different models and how much stock of it each one has. And then if you click into it, You can see how much it bet and why it bet that. So it tries to give a summary of its thinking around why it's holding a certain position. And then finally, the oracle. So we're always monitoring to see if there's any results to see if we should close the market. So if they release that an endpoint, if they have published some sort of thing that, you know, or clinicaltrial.gov updates, we update and close the market and pay the winners.
So here you can actually bet money. So the current likelihood of success on it is based at 37% and 63% it won't work. And we'll talk more about this in a second, but we're actually looking at overall survivorship here. And what's interesting about this is, outcome measure, so overall survivorship is not the primary endpoint here. So we did a lot of research to sort of look to see which of the endpoints we actually want. You might say, oh, like the thing you want is the primary endpoint, which is this thing listed above here, which is PFS, which is progression-free survival. But that's actually not the thing that the stock market cares about. This is something that was chosen by the sponsor to, you know, say that this is the thing they care about. But if you look at for a lot of these things, and specifically around cancer things,
There's not as much movement on progression-free survival, but the market actually cares about is overall survivorship. And this is very interesting because it shows how financial incentives are actually probably the correct thing that people should care about in choosing it. So we chose overall survivorship here as the thing which is a secondary endpoint and not a primary endpoint. And then what we can do... is we can buy one dollar of Tether USDC testnet on base Sepolia and it's making that transaction and it confirmed and so now if we go into our account here we can see you know we bought a share of this and it's here and so this is like how
how the site works in terms of, you know, as I get down. Yeah, go ahead. Can you talk a little bit more about how you pick what the acceptance criteria is? And does that ever drift? Is there any judgment kind of as the trial goes along? Yeah, it's a great question. So we put a lot of thought into what, so there's two sort of like, you could in theory create a market For each one of these endpoints. So for this specific trial, there are four different endpoints. So in theory, what we could do is we try and look for things that's some, so there's this tension between if we create a market for each of these, you sort of spread out your liquidity. So one of the things you sort of have to balance is how much
Each time you create a liquidity pool or an order book or something like that, it costs maybe $10,000. So if we're spending $10,000 on one thing and then all of a sudden you have four and then you have 70 trials, all of a sudden your liquid costs go way up. So we try and choose only one per market. This is more like an engineering problem. In theory, maybe when we're big enough, we have one for each one.
But we basically did a regression on what changes stock prices with some certain features and then tried to see which things the market cares about and then try to piece those out. Now, we also want to have a market at some point if the protocol changes. So sometimes halfway through a thing, they'll change the protocol. and we'll have a market for that eventually that'll be like, oh, they changed the protocol, and if they do, you'll win, and maybe that will invalidate these things. But it is a tricky problem to figure out what is the correct endpoint to choose. Is that a great question, or what was your question? If the protocol changes, how do you handle that today? Does everyone get their money back, or do you... Right now, it invalidates it, and these people get their money back. That might change down the road.
Got it. We're still in like testnet version of this. So we haven't made a decision, but that will be in the resolution criteria so that it's deterministic as to what happens. And there's no Oracle. It's just today. It's just your team that decides. Oh, well, yeah. So that's a very good question. The Oracle is basically a predefined AI prompt that scans the web and makes the decision.
So, you know, on Kalshi or on Polymarket, they use like UMA and things like that. But these things also are hackable. I think actually the correct thing to do is for everyone to agree on a prompt and a model and then have that be the final adjudicator. Because a lot of times like these things, let me show you here. So the way we resolve these markets is we have this Oracle and then it basically, one of the models, We use like GPT-5. Scrapes the web and looks for any sort of publication around it. So this was looking to see a positive result for PFS in patients with this. And so it found this and rated it 93% that based on this press release from the sponsor that it was above 93% for PFS. And so it gives its whole reasoning
In its summary. So you could use Uma or you could use whatever Cal she uses, but I think this is actually like probably like the correct way to do it. But there's many different ways that I think it's just like a design thing and I think for this type of market it's the best. Because it is relatively, we also try and choose markets that have clear, that are pretty clear and can be determined without too much wiggle room.
What was it? The other AI bits on the endpoint. There was a tab. Oh, models? Yeah. Model positions? Yeah, those. Oh, yeah. So we have these 10 models. So these are 10. I'll talk about it in the second one. I'll talk about it now, too. These are 10 models that are trading on the market currently. So these are all out of the box. The ones like CLAW, GBT.
GROC, Gemini, Geeks, and then so there's a couple here that are interesting. I'll mention a couple things. A couple of these have access to the internet and a couple don't. So Claw, GPT, GROC, Gemini all have access to the web and the other ones don't. So they're obviously at a huge disadvantage because of that. But we try to basically take all these models as out of the box as possible and match these are your bets using that model. This is our play money.
given a prompt to each model, and then it determines how to trade based on that. Do you update the prompt a little bit for every single model? No, it's all the same. So every model gets the same prompt. Otherwise, it's really hard to benchmark these types of things. And we also have other people that are betting, given their own models, and they have totally different prompts, they have totally different tool sets, different MCPs, depending on... This is just more to...
Benchmark these as a sort of proof of concept. The longer term goal is for individual people to build their own agents to trade. Yeah, go ahead. People trading on... It's on the blockchain, so not... No. It's not a... It's, you know, no. People that do this job are incentivized to do it really well. So like, was insider trading kind of the goal with this? The goal is for people to understand the science and to look at the mechanism of action around certain types of drugs, to look at related work, to look at published literature, to figure out what types of science is going to work. Science is deterministic, right? Like there's nothing, it's like chess. Everyone has equal information to the pieces on the board. Everyone has equal information to the different molecules that exist. And we'll talk...
The goal of this is to replicate science. So I encourage people to replicate the experience and find out through their own hard work and knowledge which drug is going to work. Maybe you're AlphaFold and you can determine it. Maybe you run a parallel clinical trial using a different CRO and you create replicatable science is the hope here. Anything else? When you're peer reviewing, you have to sign conflict of interest forms. You can't be motivated.
How do you pay the reviewer? Well, this is different from peer review. The peer review for archive is just sort of the genesis that's got this product idea started. Now it's different and has taken on a different life of its own. Here the goal is to incentivize people to do rigorous scientific validation of different drugs. Okay so...
So we have the trials here. So we have these different agents that are betting currently. This is the public ones. We also have people that have developed their own private bots that are betting and are doing very well, which we'll talk about in a little bit. But basically each of them is betting. Some have access to the web. Some have access to thinking. Some have access to... And we try and max out the tokens for each one. And they all have access...
So you can give a token allocation of anywhere from 1 to 6,000 tokens, say. So we try and give maximal, we try and make it so that each model out of the box is maximally supercharged so that it can do its best. And so you can see the sort of leaderboard, yeah, so for each model, it predicts, for each clinical trial, it predicts, say, for each endpoint, it predicts how much yes it wants to have and how much no it wants to have.
And then, what was I going to say? It can also choose not to bet. So with each prediction, it can say, it gives a confidence score between 50 and 100. And if it's not very confident, it's 50% confident in its decision, it doesn't place the bet. But if it chooses when it wants to place the bet, though. And they all have a common prompt. In the prompt, I'll tell you a little bit about it.
It basically comes up with a yes or no or hold decision, and it has to explain its decision briefly with a couple sentences. And then it also has access to its portfolio, so it knows which other stocks it has. Now, we'll talk about it in a second, but one of the weaknesses is it's really hard for it to make decisions across multiple endpoints. So it needs to know how confident it is in one thing relative to it.
If you have $1,000 and you have access to every ticker symbol on the stock market, how do you want to allocate it? How do you want to divide it up? It's not a tricky problem. And that is certainly a weakness of the current method since we've given access to its other portfolio holdings, but it doesn't do a good job of allocating between different markets. We also have the news here. So we actually spend a lot of time gathering news.
Scraping across every sort of clinical, any sort of pharma company's website and then parsing it for, is it an outline? Are they going to release results? Are they going to have a conference? Is there a change in leadership? We hide all that, but you can switch between different tags to be able to see what type of information is going to be published. And then we also have different pipelines, so this is the point I was making earlier. Oh yeah, go ahead.
Is there a section where you're allowing them to add their own piece to this? Yes, we'll get to that at the end where you can upload your own data. Okay, so since the going standard, and I'll just say, you know, my background is as a Gen AI specialist for Google, left, you know, a year ago to do my own thing. But the standard there is that you're going to use an ontology and feed that in, and that's the one thing that...
You see a lot of different startups doing to fill in an ontology in a particular domain. Each one of these trials essentially creates its own ontology. Are you going to be providing that in your agentic framework, or are you going to be thinking that other people are adding that? Yeah, so the goal here is this is just the table stakes that are needed. People... Are developing their own agents that have access to many different types of data sources. This is just to get people started. There's a lot of people who are using private databases of drug trials, of EHR records, of all types of things to make their own trading decisions. This is literally just the table stakes that I would assume as a strong baseline.
But some people are scraping Twitter. Some people are trying to look at the mechanism of action. This is just table stakes here. Yeah. Sorry, just the last follow-on then is I kind of go back to what the guy said about what's insider trading because when somebody was just picked up today for insider trading on Google's search results when the outcome was who was going to be the best.
you know, entity search guy for Google, right? And so they had inside information. If you take special databases of, you know, health record information or, you know, this is like, this is like certain kinds of drug information, like how interactions are working or the pharmacology of something or. For instance, you could be at CVS and you can have access to point of sale data for. all of the information for an antibiotic. And so because it's a clinical trial, you'll be able to see, you know, is that person still prescribing anymore? You'll know, like, is the trial going or not going? You can use that kind of thing to predict how well the drug is uptaking. And I was just wondering, like, how do you make sure we're all on the same plant fields? That's kind of the question. Right? Because if I want to play, I want to know somebody is able to kind of sit on a piece of information and...
kind of outdo me simply because they're allowed to use that in this. Yeah. I mean, you have to use the type of information that people would use to bet on this. This is for scientific betting based on science. And how you want to bet on this market is based on that. The goal here is for people to develop AI models that are very good at predicting outcomes. So you imagine people buying tool molecules in the lab. Correctly. That's pretty cool. Thank you, yes. It is for people to train AI agents that are good at predicting this so that you can have less patients who need to go through a clinical trial because otherwise you're just churning them through this giant machine. Okay, go ahead. An outcome of five years is a really long time for a consumer to wait to get a payout.
Are you worried about user adoption? People might just lose interest. If you make a bad, you won't know if it's good for five years. Yeah, so that's part of the reason we spend a lot of time looking at these endpoints. And so you're correct that these trials do take five years. But the good thing, though, is that there's 7,000 trials a year, and they've been going on for more than five years. So at any given time, there's multiple closing per day. And so we try and scrape very actively.
to figure out which ones are going to be closing in the next, like, six months, and then only post those ones. I think I agree with you. If you talk to other experts in the field in prediction markets, like Robin Hanson, he has this idea that you should have, like, thousand-year prediction markets. I don't agree with that, but I think you want a lot of, like, fast-moving action so that people don't lose interest, and we try and post things, like, a couple months ahead of time before there's a decision.
Are your database costs and token costs pretty high? Our token costs are fairly expensive, just because we're sorting so much different types of information. Is it the ballpark of what that is? No. And the server costs are pretty cheap because the compute is pretty free. The token costs are kind of expensive, especially if you want good ones. Anyone else? Yeah. How exactly do you price the outcomes? Oh, yes. So good question.
So, we don't price the outcomes. We create the market that prices the outcomes. So, there's a couple different ways you could do this. You could use like a, you create a market. So, you could use like a central limit order book and have a market maker create these pricing of it. So, basically he's asking is like, you know, how do you create a market for something? It's not, it's not, you go in the stock market and it seems obvious like, oh, Tesla is.
$600 a share. That seems like a good price. I'll buy it. But actually when you do that, there's a lot that has to happen behind it. There's always a buyer and there's a seller. It works fairly well for things that have a lot of liquidity, like Tesla. But say you're on a smaller market. Like how many people think that a prostate drug is going to work for a specific endpoint. There's not that many people who are going to be doing it. So we used something called an automated market maker, which basically allocates a certain number of tokens in a pool.
And then it holds those. And then as people buy the yes tokens in that pool, the price of the no tokens sort of goes up exponentially along something called the bonding curve. So if it's 50-50 and you buy one token of yes, now all of a sudden no becomes more expensive. And so you're sort of constantly balancing this. But it's a good system for these low liquidity markets where you don't have...
something like a market maker to be able to take on the risk on both sides of these you have something oh okay okay okay so like for some of the single models that you have and you're as you were saying like some of them have web search something have you tried like adversarial methods where you like pit some of them against each other and have them run longer to make a decision and what do you mean just work together or oh yeah basically like have to debate the outcome No. I think there's a lot of different ways that we can optimize the out of the box things. At the end of the day, that's not our goal. Our goal is to create the market with deep liquidity so that other people can create agents that will figure this out. So the creativity is on other people now. Exactly. I mean, these are just out of the box benchmarks to get the market started.
But if anyone here has an agent that they're looking, we have an MCP or API that you can integrate your agent with and start betting on it. I have one more question. What's the latency between when an outcome is decided in the real world and when it's decided on the whole world? Yeah. It's like an hour or so. It's just rate limited by how quickly that Oracle is scraping every single...
So there is a chance that people will bet on it and it will resolve de facto more quickly than it resolves in practice, but that's fine because that's how it works. It should gradually go to 100% anyway. That's not like a super rate limit. That's not an issue. Even for things like did... Elon tweet X number of times, it just exponentially gets closer and closer to something. they're still bidding on tokens that they already know the answer to yeah but the market just self-closes that way like it prices itself correctly
Meaning if the outcome is known, it's pricing itself correct? Yeah. People will start buying it. Which is, it does point to a marginal inefficiency between buying, actually. But you can do the same thing for the Navy guy running into the middle of the Super Bowl. If you have that faster network than anyone else, then you can make a guaranteed money, a guaranteed outcome. But you have to be faster than people.
Yeah, we can, yeah, so it's not something I worry about too much. Did you have something over here? Do you have a question? No. Okay, we'll move on. So we have a leaderboard of which ones, which people are being, are the top of the leaderboard. This is kind of cool. So, okay, let's just talk about this briefly. At a high level, who are the types of people that want to take either side of this market? So, obviously, if you're a biotech company, you want to hedge your risk that your trial is going to fail.
that your trial is going to fail or succeed. So you might have five different drugs that are going through a pipeline. You want to take out insurance that maybe one to five of them is going to fail to alleviate, to free up more money so that you can maybe pursue a sixth one without having to tie up money. And so you'll take out an insurance. You'll bet that your clinical trial is going to fail.
And what this does is it actually makes you more investable because now an investor knows that they have downside protection. Now what wasn't immediately obvious is who takes the other side of the bet. Who's hedging on success here? And it's actually like health insurance companies and hospitals that are exposed to upside risk. So imagine you are a hospital or imagine you're an insurer and all of a sudden there's a drug that's a new drug.
with a very large reimbursement obligation, say that's like an Ozempic or something around sleep or something that costs a lot of money per patient, you're sort of hoping that these types of drugs are not approved. So you're willing to take the other side of that bet. And then this is sort of the structure of how these markets will work. Wouldn't competing biotechs also want to take the same side? Totally, yeah. There's many different types.
I was just trying to lay this out as because people were oftentimes very unclear who would take this other side of this risk. I think this side is pretty clear. But I imagine in the same way that credit default swaps created a whole new market around credit, around people paying or not paying their debt. So another really interesting one is like service providers. Say you're a CRO and the clinical trial fails. Oftentimes those people might not get paid because that company is like now out of money.
So they also want to use this as a type of insurance, as a way to ensure that they get paid. So there's all different types of parties, but this is sort of the, I think this is sort of the broad structure of how it would work. Your market are going to be in the area of $100 million? I don't know. I mean, it takes a while to get that level of liquidity, but that would be the goal. You know, because the amount of money that a pharmaceutical company is putting in is obviously, you know, hundreds of millions of dollars.
Yeah, I mean, you might hedge part of it, you might not hedge the whole thing. You can't get an insurance policy for everything, but that would be the hope. Okay, so this is how we sort of look to find the endpoints that are the most interesting, is we look, let's see, oh, so yeah, so we looked at the stock price around different clinical trials, around different pharma companies, and around different endpoints.
To understand which endpoints we should be choosing. Basically, you know, phase two and phase three have the largest price movement around stocks. Smaller biotech companies are much more swinging, swing much more per endpoint than larger ones. The market doesn't care about primary endpoints. They care about, you know, this exact endpoint or how much an endpoint changes. And just as a general sense, here's about... you know, for phase one, phase two, phase three, knowing absolutely nothing about a clinical trial, about, you know, how much it affects the stock price for each of those. So like a phase one readout is three. So the other thing that's interesting is positive readouts bump your stock a little bit, but like negative readouts bump your stock, like double negative. So people, positive in news is like marginally good, but negative news like sinks you. So we did this though, so that we could choose, choose the correct end points. And we talked about this a little bit before.
But the market doesn't care about what the primary endpoint is for the most part. The market cares about if the thing is going to be a successful drug, which is oftentimes how it competes with other drugs on the market, overall survivability, the things that patients care about, basically. And here's some of the baselines for a clinical trial being accepted or denied. Knowing nothing about it, but basically if it's phase one it's about 50%, if it's phase two it's like 30%, and if it's phase three it's around 50%, and then once you've done all those, once you sort of get your PDUFA, then it's like 90%. The reason this is important is because this is a baseline, so now I want to talk about how well these models did. So this is for season five. Claude was very very good for n equals seven.
Predicting these outcomes. GPT OSS sucked. So this is sort of the overall here. But then this is only, this is if you force each one to bet. Another really important thing is, oh, why these models succeeded and failed. So a model for the most, so because we forced reasoning, a model will oftentimes be very successful if it looks at the sort of mechanism of action and tries to identify a rationale between, if it knows the endpoint it's going after and it tries to look at the science for that specific endpoint and figure that out. A model, it also looks to see, it also tries to weigh the risks on these things.
Some of the best models basically looked at the risks for different types of things occurring and was able to sort of internally weigh, at least acknowledge them and then compare them. When things didn't work is basically when they went back to the overall reports for how likely a clinical trial is to succeed or fail and then just sort of went with that. That was an issue. Another one is when things tried to overfit based on what I read online.
When it tries to say, like, oh, so-and-so said such-and-such, therefore this is going to succeed, then it sort of oftentimes failed. And then another one that was sort of interesting is, like, if it was expecting a readout at a certain time, but it was taking longer to basically come up with the actual data, some of the models started freaking out, and they looked too much into, like, a delay in readout as being a negative signal.
So these are sort of some interesting success and failure mechanisms for the different AIs. And the other thing is like sizing. So you have to, based on how you, you have to know how competent you are in your predictions. So you could get all the things right, but if you don't make your allocation correctly, it doesn't really matter. And this is actually why I think prediction markets are a really, really good way of benchmarking AI models, because it forces both you to know what you're doing but also to accurately assess your confidence so if you're good at something you can't actually be confident in it you know you your sensitivity and specificity will sort of all get messed up but this is like a good a good middle ground of like a way of benchmarking things uh oh so uh another another thing that's interesting about the sizing is that um how how it manages its portfolio i think there was a pretty big failure in getting
Models to look at the risk profile of all, look at it to like look at every single market that exists and properly allocating between the different markets that it could. So that's definitely something that all of these and I'd imagine some of the people are going to sort of like do a lot of innovation there. Yeah. Why was GPT so bad? That seems surprising. I don't know for that specific one, but yeah, it's a good question.
The issue when I was making this presentation is that I had so many questions. I had to sort of just like scope it down at some point. Like, why did XYZ, why did I make this decision on this decision? I kind of, it became too long. But yeah, you can look on the website and look to see. If I had to make a guess, I would say it's probably because it's sized into one too much and it ended up being wrong and then it lost all its money.
But I don't know, I have to look. Why haven't traditional insurance frameworks work their way into these kind of markets? I think because it's really hard to price the risk associated with science. Fifth Tribe is just hard to do. Were you reading the thinking traces? Yeah, yeah. I mean, I was using meta-level, so I fed them all in and then tried to get it all from that.
Okay, meta level. Like, I can't read through 10,000 things, so I looked at the sentiment of everything, yeah. But I didn't, yeah, that's how I did, that's how I did that part, yeah. Because in a way, like, sometimes I read the traces when I'm coding and stuff, and it's like, it'll say things that it doesn't actually act on. Yeah, yeah. So then what about that? So in this case, what do you mean? What I mean is like, I don't know, like sometimes the traces are actually misleading. Oh, so this, this, this, for, do you, do you, I don't think I follow. Like, you were saying that you read the thinking traces. Yeah. And you, like, are looking at them at a meta level to figure out what they're doing. But what I'm saying is, like, sometimes the traces don't actually correspond with the outcome. Oh, so we looked at it after the fact. So when it got it correctly, what was its thinking? And when it got it incorrectly, what was its thinking? And then looked at the failure mode that way. Okay. I guess the only question is, like, sometimes, like, it might be, like...
I don't know, right? Like you saying that you believe the alignment or misalignment between the trace and the outcome. And I'm just saying sometimes it's not actually- Yeah, yeah, totally. That could be an issue. I mean, that's what we forced it to make, to print out its reasoning. Yeah. It could be, yeah. It's hard to debug these things. Yeah, it's definitely hard to debug. Yeah, I've definitely seen that as well. Like, you read the trace and you feel like it's going to answer a certain way, but then it doesn't answer. Yeah, yeah, yeah, totally. Yeah, that's a good point.
That's why you have money, because money equalizes all of that. So this is another thing that I thought was interesting. I asked it to sort of look at its personality types of each of its investing strategies. One of the things that I thought was interesting is Grock did not look at evidence a lot. I thought that having access to Twitter would give it a huge leg up in its ability to make assessments, but that didn't seem to be the case.
Claude was very good, I think partially because it had the most scientific reasoning. Same with Gemini. Gemini and Claude were pretty good. And they looked at these specific endpoints and they actually tried to do the science. You know, Groth wanted to try to read what other people were thinking and kind of lost its way. So this speaks to the importance of actually understanding the endpoint and understanding the science and doing your own.
First Order Calculations as opposed to sort of what listening to other people say. So I just want to talk a little bit about prediction markets as something that can do more than forecasting. So, you know, people think, oh, you're just looking, you're just looking at something and making, and you're looking at, you're an object and forecasting about it. But prediction markets actually have, you know, a lot of other interesting things. So people can intentionally bet no that disease won't be cured and drive down the stock price of that market. But then what that does is it encourages researchers to buy yes at a very low price and then make a lot of money if they themselves come up with the discovery that will cure this disease. So it's not just acting as like a measurement tool, it's acting as like an incentivization layer.
Another sort of fun output of this is like decision making. Say you are having to enroll in a clinical trial. You might not know any of the options between trial A and trial B. You don't know if trial A and you don't currently, they both look about the same. You don't know which one's more likely to succeed. If you go to this website in the future, you could say like, oh, oh my gosh, trial A has an 80% chance of success and trial B has a 10% trial of success.
Probably a success. I'm going to go with Trial A. Yeah, go ahead. To expand your horizon, critical problems are weird that you mentioned. Yeah. We just need to build up more volume and liquidity to make this work. But it's just that's sort of like an engineering issue. It's not necessarily like a... Yeah, but this is how it could... These are things that are interesting about the future of how this could work, not necessarily as it stands today.
Ideally, I like to have every endpoint and every trial on the market. Okay, help find a picture of the Prime Losing. We'll talk about the end of one in a second. Okay, so those are sort of the interesting things about that. So this is interesting. This is what we're working on now is doing this thing called the Shadow Trial. And what this is cool is basically you install the app on your phone and you get...
You collect data around how much you're sleeping and if you're on a GLP-1. Now, it uses that data that's on your phone to then make a bet for you. So it's using your own proprietary data, and there's a small open source model that lives on the phone also. And then you will deposit in a little bit of money, and then it would then make bets based on your own personal data that lives on your phone.
So this incentivizes like end of one style data collection for it to be pooled together to create like a replicatable shadow trial that you sort of do based on your own personal data. So I think this is like a very, very cool concept. How can people omit data basically, like if they chose to? Like suppose there's like random construction going on outside my bedroom. Yeah. So you can delete it from your health kit database. So there's a, and it gives you access to This basically collects the data and then proposes the trade. And it gives you a reason for that, why it came up with that trade for yes or no based on the endpoint on Clinical Trials. Yeah? Okay, cool. So basically every time that the prediction is being made on top of your data, you decide how real your own data is. Exactly. Okay, cool, thanks. So you're incentivizing people to accurately upload correct data. This isn't about uploading fake data because then you're shooting yourself in the foot.
But it's incentivizing accurate, robust, scientific data that you then make your bet on. So I think this is very cool. When did you first come up with this idea? Just very recently, actually. And the data never leaves your phone. It's running a small open source model on the device, like an OLAW 0.3b version. um and then it's making the trade and so you click that you find it the trials when rolling that's sort of related to one of the ones existing you try and collate your data around that and then you uh you trade so could you group together with a group of people also enrolled in a similar chat totally yeah totally yeah and so you can have a small data cabal but you can also sell it to someone right you can sell your data
like how you'd sell anything and um uh uh yeah so yeah but i'm very excited about this because you can you basically replicate science you've got an incentive mechanism to replicate science you might need to have like bedroom controls or something like that yeah well you can do it so we've been looking into doing this for like acne sleep psychiatry glp1s uh you know whatever whatever types of things that are sort of easily measurable Yeah, so the vision here is just to build the financial infrastructure to enable scientific truth, because I don't think that science is greedy or correctly motivated enough. And, you know, I think we can actually, through this new incentivization layer, you can accelerate science and make it more replicatable. And...
A couple things here. So we're hiring. So if you're interested in working together, I would encourage you to scan this QR code. It'll be specifically around like data and like partnerships with other companies and social media. We're looking to expand in that area. If you have a question, either if you want to reach out either in person or the people online who are watching, DM us on Twitter is the best way. I can also give you my email if you want it.
For the most part, DMing us on Twitter at Endpoint Arena on Twitter is good, or whatever you call it. And then, of course, the website is EndpointArena.com, and that's that third QR code there. And thank you, guys. So if you have any questions, let me know. But thank you. All right. Anyone have a question? We did a lot throughout a little bit. Yes, go ahead. What's going to surprise you most during this whole process of building Endpoint Arena? The uptake...
In agentic trading compared to human trading. So we're seeing a lot of trading based on agents. And because I think this is probably one of the first prediction markets that is really built towards AI as opposed to being built towards humans. So every human has an opinion if Biden or Trump is going to win the election. The types of questions that we're asking here around science are really actually better suited towards AIs trading on it. And so it's not something that's relied on human thought, but more on AI trading. That's been very surprising. There's a lot of that going on. Anyone else? Yeah, go ahead.
Do you think you'll get competition from Polymarket and Halshi? Because obviously you could create similar markets there. Yeah. I mean, I think we're going into more of the deeper science and trying to get the data firsthand. And, you know, you have to do a lot of these markets. Figuring out which one is correct is tricky. But we're going deeper and deeper into, like, eventually integrating with the lab equipment, right? And the outcome measurements will be different.
And the Oracles will be different. So it could happen, but I'm not too... And it's also a very different market. So the people... Kalshi for the most part is going after, like, you know, CME style stuff. And, you know, Polymarket is going after, like, Teenage Boys and DGEN stuff like that. I don't think it's... The type of markets that exist for these... I think there's just like a big customer difference also. Yeah, you were just talking about, like, generating data. I mean, would you ever consider having, like, sort of...
a data set like constantly generating like data yourself as a company to give out for other people to analyze and then potentially having like lab automation generate that so there's like you know some sort of verification on every data yeah absolutely yeah i mean i think uh i think at some point this could be like a very cool cro where basically uh cro is a clinical research organization where basically they are the ones that are conducting this clinical trial but if we're collecting the data firsthand and you know There's also going to be a very big rise in vibe drugging, I'm calling it. So it's like vibe coding, but for drug development. And where people are going to start developing their own drugs. You've seen this already today. You've seen little bits of this today with people trying to analyze their own genome on Twitter. Someone's trying to develop drugs for cancer for their dog on Twitter. And so people are trying to basically come up with new ways to develop drugs.
So I think you're going to see an explosion in the types of drugs, in the number of drugs that are going to be developed, but there's still going to be a bottleneck in how you can test those. So testing a program is very easy because you can just run it on silicon. Testing a drug is a resource allocation problem, so there's still only going to be 8 billion people. So basically, I think we could eventually become like a CRO or something. Yeah.
Yeah, go ahead. If there's a retraction in scientific data, or if there was like a poisoning of data in a trial result, and like two years down the line they retract, would you recuperate people that were on the wrong side of the bat? Hmm. I think there would probably be some language in there that says the thing that's posted on ClinicalTrial.gov is the de facto thing at a certain time. I think one of the important parts of all about law is like finality. And so eventually, like...
You have to just agree to it. Anyone else? Yeah, go ahead. What's the toughest thing you're going to need to do over the next 18 to 24 months to, like, make this, like, go to the next step, next stage? You know, build out which endpoints we're choosing, build out the liquidity, build out the agents that are betting on it, you know, bringing in new people, bringing in new data.
it's mostly around like choosing the markets correctly i think and getting the word out marketing it cool we very much follow our nose um we do things and try things out and see what works and then try again uh when you say we how many others are there uh like two others amazing well thank you guys all for coming and i'm around after if you guys have questions thank you Thank you.
Thank you.