Are you really drilling at that point, limiting your defined population by an attribute, or just changing the time-series grain at the top of the card?  In other words, Change from "This Year", to "This Year by Month", or "Previous Quarter by Week"?  

i was going to say the same. you should just load in the data at the daily granularity and then use the time grain at the top to group by year/month/week/day. or if you really want to have drill throughs, you can have the top level chart grouped yearly, then the next drill through level grouped, monthly and and the next drill through weekly and so on. Basically with 4 levels of the same chart just grouped differently. This way you can use the same exact field for the value in all the drill throughs and you won't have to worry about the daily not mattching weekly or anything like that.