Allow for different metrics in NPV calculation

[Aurashk]

Description

If annual fixed costs (AFC) are zero, this currently makes the metric in appraisal comparisons NaN. This PR modifies the behaviour so that we use the total annualised surplus to appraise assets in this case instead. Since there are two different possible metrics in this case, with one strictly better (AFC == 0 always better than AFC > 0). I've added a metric_precedence to AppraisalOutput which ranks the metrics by the order which they can be used. Then, select_best_assets will disregard all appraisals which have a precedence higher that the minimum. Another way of doing this may be to make the metric itself a struct and implement more sophisticated comparison logic, but since lcox uses the same struct it might end up getting too involved

Fixes #1012

Type of change

• Bug fix (non-breaking change to fix an issue)
• New feature (non-breaking change to add functionality)
• Refactoring (non-breaking, non-functional change to improve maintainability)
• Optimization (non-breaking change to speed up the code)
• Breaking change (whatever its nature)
• Documentation (improve or add documentation)

Key checklist

• All tests pass: $ cargo test
• The documentation builds and looks OK: $ cargo doc

Further checks

• Code is commented, particularly in hard-to-understand areas
• Tests added that prove fix is effective or that feature works

[codecov]

Codecov Report

❌ Patch coverage is 81.63265% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.46%. Comparing base (a148d69) to head (e205cc9).

Files with missing lines	Patch %	Lines
src/simulation/investment/appraisal.rs	76.92%	9 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1027   +/-   ##
=======================================
  Coverage   81.46%   81.46%           
=======================================
  Files          52       52           
  Lines        6500     6545   +45     
  Branches     6500     6545   +45     
=======================================
+ Hits         5295     5332   +37     
- Misses        949      957    +8     
  Partials      256      256           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[dalonsoa]

I'm not fully convinced by this. the profitability index is dimensionless, but the annualised surplus is Money. Even though it does not matter from the typing perspective since you are getting the underlying value in both cases, which is float, I wonder if this choice makes sense from a logic perspective.

Not that I've a better suggestion.

[dalonsoa]

Thinking again on this, I think this logic (whatever it becomes, see my other comment) should be within the ProfitabilityIndex.value, also adding a ProfitabilityIndex.precedence method that returns 0 or 1 depending on the value of AFC.

[tsmbland]

I think this is ok, I have an alternative suggestion though.

The idea would be introduce a new trait for appraisal metrics:

pub trait MetricTrait {
    fn value(&self) -> f64; 
    fn compare(&self, other: &Self) -> Ordering;
}

pub struct AppraisalOutput {
    pub metric: Box<dyn MetricTrait>,
    // ...
}

You could add this trait to your ProfitabilityIndex struct, and add a custom compare method here. You'd also have to make an equivalent struct for LCOX - it would probably be very simple, although there may be some edge cases we haven't thought of yet. I think this would help to contain the comparison logic and make the code cleaner. We'd also no longer have to make the profitability index negative as the custom compare method could be written to look for the maximum - I always found this a bit hacky and it makes the output files confusing

[tsmbland]

pub struct AppraisalOutput {
    pub metric: Box<dyn MetricTrait>,
    // ...
}

Or this:

pub struct AppraisalOutput<M: MetricTrait> {
    pub metric: M,
    // ...
}

I think there are various pros and cons of each option which I don't fully understand. I think possibly the latter is better if you don't need to store AppraisalOutputs with mixed metrics in the same Vec, which I don't think we need to/should do

[alexdewar]

I've taken the liberty of jumping in for a review here @Aurashk 🙃. Hope that's ok!

I agree with @tsmbland's suggestion that it would be better to use traits for this instead though -- I just think it will make it a bit clearer and more maintainable.

I'm wondering if it might be best to define a supertrait instead (that's just an alias for a combination of traits). In our case, we just need things which can be compared (Ord) and written to file (Serialize). We did talk about having an Ord implementation for unit types (#717) and I think I've actually done that somewhere, but didn't open a PR as we didn't need it, but I can do if that would be useful! That unit types would automatically define the supertrait.

I think the problem with having a value() method returning f64, as @tsmbland suggested, is that it wouldn't be obvious which value was being returned for the NPV case.

E.g.:

trait ComparisonMetric: Ord + Serialize {}

pub struct AppraisalOutput {
    pub metric: Box<dyn ComparisonMetric>,
    // ...
}

What do people think?

[alexdewar]

I'd probably make this a u32 instead. I know we won't ever need more than 256 different values here (if we did, that would be horrible!), but I think it's best to use 32-bit integers as a default, unless there's a good reason not to.

[alexdewar]

Can this actually happen? I'm struggling to think... @tsmbland?

If it's more of a sanity check instead (still worth doing!) then I'd change this to an assert! instead.

[tsmbland]

If it is then something has gone badly wrong! Agree, better to change to assert!

[alexdewar]

PS -- this isn't super important, but for the NPV metric, I'd include the numerator and denominator in the output CSV file (e.g. "1.0/2.0" or something) rather than just saving one of the two values. We want users to be able to see why the algo has made whatever decisions it has

[alexdewar]

Actually, on reflection, my suggestion won't work after all 😞. I still think we should use traits, but it's not as simple as adding a single supertrait.

The problem is that Ord is a trait for comparing a struct with another struct of the same type, but we need to be able to compare a generic AppraisalMetric (or whatever we call it) with another. We know that we will only be comparing things of the same type, but the compiler doesn't!

I think what you want is the Any type, which lets you downcast to a specific type. Then you can do something like this:

pub trait AppraisalMetric {
    fn compare(&self, other: &dyn Any) -> Ordering;
    // ...
}

impl AppraisalMetric for LCOXMetric {
    fn compare(&self, other: &dyn Any) -> Ordering {
        let other = other.downcast_ref::<Self>().expect("Cannot compare metrics of different types");
        // ...
    }
}

(This assumes you have an LCOXMetric struct defined.) Does this make sense?

I still think it might make sense to have a supertrait which is AppraisalMetric + Serialize, as we'll need to be able to do both of these things with our tool outputs. You'll need to define Serialize for the structs for it to work.

It's a little convoluted but that's the downside of using a statically typed language 😛

[alexdewar]

Btw, I know I'm a bit late to the party on this one, but I'm not sure about the compare_with_equal_metrics stuff (and apologies for the many messages...).

I'm not sure why we need to go ensure consistent ordering for assets with identical appraisal outputs. If they're identical -- or approximately identical -- then, by definition, we should just return cmp::Equal, right? And in that case, it doesn't matter what order we choose, as long as we don't pick a different one every time MUSE2 runs. Bear in mind that this is an unlikely situation to happen in the first place and users shouldn't rely on us appraising assets in a particular order anyway!

I'm only raising it because I think the added complexity will make @Aurashk's life harder here and I can't really see what benefit it brings. Is there something I'm missing?

[tsmbland]

Btw, I know I'm a bit late to the party on this one, but I'm not sure about the compare_with_equal_metrics stuff (and apologies for the many messages...).

I'm not sure why we need to go ensure consistent ordering for assets with identical appraisal outputs. If they're identical -- or approximately identical -- then, by definition, we should just return cmp::Equal, right? And in that case, it doesn't matter what order we choose, as long as we don't pick a different one every time MUSE2 runs. Bear in mind that this is an unlikely situation to happen in the first place and users shouldn't rely on us appraising assets in a particular order anyway!

I'm only raising it because I think the added complexity will make @Aurashk's life harder here and I can't really see what benefit it brings. Is there something I'm missing?

It's not actually that unlikely:

• You could have multiple existing assets with identical metrics. For example, an asset from 2025 and an asset from 2030 which have equal metrics because none of the technology parameters have changed. In this case I think we do need a consistent rule (e.g. favouring the newer one).
• You could have a candidate asset equal to an existing asset because capital costs are zero. Not likely for "real" tangible processes, but people can use processes to represent all sorts of conversions which will not always be associated with physical infrastructure. Again, we need a consistent rule here such as favouring the existing asset.
• Identical assets due to parent asset division (i.e. Make assets divisible #1030). Not something that users should have to be concerned with, but these warnings are a useful reminder to us that our investment algorithm is wasteful and needlessly appraising multiple identical assets.
• The case of two different processes with identical parameters. In this case, I think it's fair to raise a warning as we currently do. Users may very well wonder why one particular process was selected/retained over the other, and I think it's helpful to clarify that it's an arbitrary choice

I agree with you that it feels a little hacky for identical metrics not to return cmp::Equal. I think we did it this way because it was easiest, but by all means suggest a better approach. I'm pretty adamant that we do need something though!

[alexdewar]

It's not actually that unlikely:

• You could have multiple existing assets with identical metrics. For example, an asset from 2025 and an asset from 2030 which have equal metrics because none of the technology parameters have changed. In this case I think we do need a consistent rule (e.g. favouring the newer one).
• You could have a candidate asset equal to an existing asset because capital costs are zero. Not likely for "real" tangible processes, but people can use processes to represent all sorts of conversions which will not always be associated with physical infrastructure. Again, we need a consistent rule here such as favouring the existing asset.
• Identical assets due to parent asset division (i.e. Make assets divisible #1030). Not something that users should have to be concerned with, but these warnings are a useful reminder to us that our investment algorithm is wasteful and needlessly appraising multiple identical assets.
• The case of two different processes with identical parameters. In this case, I think it's fair to raise a warning as we currently do. Users may very well wonder why one particular process was selected/retained over the other, and I think it's helpful to clarify that it's an arbitrary choice

Ok, good point.

I agree with you that it feels a little hacky for identical metrics not to return cmp::Equal. I think we did it this way because it was easiest, but by all means suggest a better approach. I'm pretty adamant that we do need something though!

Can I just check what the motivation for this is? On reflection, I'm guessing that the idea was that it would make it easier to figure out why a particular asset was chosen over another. Is that right? If so, that seems reasonable.

Initially I was thinking that it was to make choosing between two assets with identical metrics less arbitrary which I'm less convinced about. A lot of things in MUSE2 are arbitrary, e.g. how HiGHS distributes activity across time slices, and the results fluctuate as we change the code anyway, so it seemed overkill to try to make guarantees to users about this when we can't do the same for so much of the rest of the model.

Anyway, in terms of the code, I think the problem is that it's not a good separation of concerns. It would be better if the compare_metric method just compared the metrics and we did the fallback check for asset properties somewhere else, e.g. in a compare_assets_fallback function in investment.rs. Then where you sort the appraisal outputs, you could try these one at a time, which would make it clearer what's going on. If we do this, it'll make the refactoring for this PR easier.

[tsmbland]

It's not actually that unlikely:

• You could have multiple existing assets with identical metrics. For example, an asset from 2025 and an asset from 2030 which have equal metrics because none of the technology parameters have changed. In this case I think we do need a consistent rule (e.g. favouring the newer one).
• You could have a candidate asset equal to an existing asset because capital costs are zero. Not likely for "real" tangible processes, but people can use processes to represent all sorts of conversions which will not always be associated with physical infrastructure. Again, we need a consistent rule here such as favouring the existing asset.
• Identical assets due to parent asset division (i.e. Make assets divisible #1030). Not something that users should have to be concerned with, but these warnings are a useful reminder to us that our investment algorithm is wasteful and needlessly appraising multiple identical assets.
• The case of two different processes with identical parameters. In this case, I think it's fair to raise a warning as we currently do. Users may very well wonder why one particular process was selected/retained over the other, and I think it's helpful to clarify that it's an arbitrary choice

Ok, good point.

I agree with you that it feels a little hacky for identical metrics not to return cmp::Equal. I think we did it this way because it was easiest, but by all means suggest a better approach. I'm pretty adamant that we do need something though!

Can I just check what the motivation for this is? On reflection, I'm guessing that the idea was that it would make it easier to figure out why a particular asset was chosen over another. Is that right? If so, that seems reasonable.

That's part of it at least. E.g. Before working on this I didn't previously consider that multiple existing assets from different commission years might have the same metric. If it's going to have to pick one over the other, I'd at least like some consistency so that decision is explainable.

Initially I was thinking that it was to make choosing between two assets with identical metrics less arbitrary which I'm less convinced about. A lot of things in MUSE2 are arbitrary, e.g. how HiGHS distributes activity across time slices, and the results fluctuate as we change the code anyway, so it seemed overkill to try to make guarantees to users about this when we can't do the same for so much of the rest of the model.

I don't think we can/should try guarantee to users that the model is completely unarbitrary, but we can still do our best and if there's an easy way to make certain behaviours just a little bit more predictable/explainable then I don't think there's an excuse not to.

Anyway, in terms of the code, I think the problem is that it's not a good separation of concerns. It would be better if the compare_metric method just compared the metrics and we did the fallback check for asset properties somewhere else, e.g. in a compare_assets_fallback function in investment.rs. Then where you sort the appraisal outputs, you could try these one at a time, which would make it clearer what's going on. If we do this, it'll make the refactoring for this PR easier.

I think that's a good idea, do you want to give it a try?

[tsmbland]

I think your other point is that the warning that we're currently raising if it ultimately does have to make an arbitrary decision isn't really worthy of a warning that users should have to be concerned about. I think that's fair, so happy if you'd rather change that to a debug message

[alexdewar]

@tsmbland Ok cool. Seems like we're on the same page now.

I'll have a go at the refactoring.

[alexdewar]

I've had a go at the refactoring in #1039. @Aurashk it probably makes sense to merge that before you have another go at this.