Models and the scientific method

Saul Griffith's Make column, Model It, or Make It Modelable, is a really smart look at the way that models fit into the scientific method, and how computer models allow for rigorous peer-review of theories about how the world and the machines we build might work:

But here's the beautiful thing about modeling. Computational models are digital, and that makes them inherently shareable, independently verifiable, and easy to collaborate on and improve.

Whereas my inclination was to immediately start to build something physical, Geoff's approach -- the approach of a new generation of engineers and scientists -- was to begin with a model. Start with bits. Make them perfect, beautiful, defendable, sharable bits, then render them physical once you've reached an optimum. Sure, someone might figure out a better optimum one day, but because they can start with working, executable code, they'll get to it faster.

There's an even more important reason to encourage this culture of shared models. The more people who have experience simulating the world with success, and making things from those models, the more people will trust in the models of our physical world that will guide how we shape our future.

Model It, or Make It Modelable

(Image: Printable Clock, Parametric Proof of Concept by syvwlch, Syvwich/Thingiverse )

• [randwolf]

  “In the good old days physicists repeated each other’s experiments, just to be sure. Today they stick to FORTRAN, so that they can share each other’s programs, bugs included.”–Edsger Dijskstra

  I do research which includes a lot of modeling. But sometimes I wonder…

  • [SamSam]

    I’d love to know how much actual code-sharing there is.

    When scientists say “We’ve been using this well-respected model that correctly models the temperature variation of the last hundred years,” do they mean a logical model, or actual code. Well, of course they mean a logical model, but the question is, which do they use?

    Don’t these models get published in articles? If so, what they’re publishing is the logical model (“Using a alpha factor of 0.05 and weighing the solar flare variation as 0.03% of…”). Does the code also get distributed online with the article, so that other researchers can take it apart and verify it?

    All code used in published research should be open source…

• [Syvwlch]

  As the maker of the printable clock PoC Cory chose to illustrate this post, let me say he got to the crux of why I share my work iteratively on Thingiverse.

  The end goal may be a 3D print in the physical world, but the design work is more collaborative, robust, flexible and protean when done in the digital world. In particular, representing the object as code allows me to share source code, methods, parameters, insights and design trade-offs for my peers to review, refine and re-publish.

  If some of them choose to print out some of the steps along the way… http://www.thingiverse.com/derivative:8594 we all learn from them, but the collaboration is enabled by the digital model.

  All printable objects should be treated as code, and there is no reason it can’t be source-code… and open-source code at that.

• [Kosmoid]

  What I think is more interesting is where visualizable scientific models fall short of describing scientific phenomena. This happens a lot in biological systems (evolved components not designed by single functions), and of course quantum mechanics makes its own kind of mess of the sub-atomic world.

  There’s a famous anecdote of August Kekulé coming up with the ring structure of benzene by visualizing a snake biting its own tail during a day dream. He had to further come up with the idea of the six carbons atoms joined by bonds that “oscillated” between being single and then double bonds. Linus Pauling came along to suggest the concept of resonance between quantum-mechanical structures to explain this.

• [Anonymous]

  There’s a acclaimed chestnut of August Kekulé advancing up with the arena anatomy of benzene by visualizing a snake bitter its own appendage during a day dream. He had to added appear up with the abstraction of the six carbons atoms abutting by bonds that “oscillated” amid actuality distinct and again bifold bonds. Linus Pauling came forth to advance the abstraction of resonance amid quantum-mechanical structures to explain this.

• [wgmleslie]

  Make them perfect, beautiful, defendable, sharable bits, then render them physical once you’ve reached an optimum. Sure, someone might figure out a better optimum one day, but because they can start with working, executable code, they’ll get to it faster.

  I think the term optimum is an absolute.

  It’s interesting that the general rule in software is to hold off optimizing until the very end of the development cycle whereas Saul Griffith recommends optimization right up front.

  The disadvantage of early optimizing approach is that time is lost if the optimized component is discarded or radically modified.

  • [SamSam]

    I think the term optimum is an absolute

    Actually, when you’re dealing with these kinds of models, it’s not.

    Unless you’re in an extremely limited system where you can enumerate every possible solution, then it can be very hard to know whether your optimal solution is a global optimum or not. It could be that you have a solution that works very well, and if you tweak with your solution at all the model behaves worse. This is an optimum solution, a local optimum.

    The notion is frequently compared to climbing in a foggy landscape: you keep following the mountain up and up and up, and at a certain point, you’re as high as you can go: in every direction, the ground slopes downwards. As far as you’re able to tell, you’re at the top of the mountain. However, unbeknownst to you, you may simply be at the peak of a smaller peak, and the real mountain peak is elsewhere. But it may be very difficult to reach that part of the mountain if your only algorithm is “look around me, and if I see the ground sloping upwards, go that way.” You’re stuck at a local optimum.

    For instance, suppose you’re designing an engine, and you tweak the pistons every way you know how — you vary the timing, the pressure of the gas, the springiness, etc. At some point, you may reach a solution which is a good as it appears to be possible — if you change any of your optimal variables, the engine doesn’t work as well. As far as you can see, it’s an optimal solution.

    Then someone comes along and says “hey, did you ever thing of putting your pistons in a V?” All of a sudden, you’re in a whole different part of the solution landscape, and you’ll be able to go much higher.

    • [sabik]

      The notion is frequently compared to climbing in a foggy landscape: you keep following the mountain up and up and up, and at a certain point, you’re as high as you can go: in every direction, the ground slopes downwards.

      Note that this is a somewhat misleading analogy, because it only involves two variables (north-south and east-west). It’s relatively easy to check every direction. With more variables, it’s disproportionately harder.

      • [SamSam]

        Reducing the number of variables doesn’t make it a misleading analogy, because the algorithm is identical whether you are in a two-dimensional landscape or a 12-dimensional landscape. Yes, one will take a lot longer than the other, but the situation is precisely analogous. Indeed, you could use the exact same code in either situation.

        Do you think it would have made a better analogy if I had said “imagine yourself in a 12-dimensional hilly landscape, and when I say “hilly” here, picture that as a 13th value-dimension, not one of the 12 variable-dimension.”

        That’s not an analogy, that’s a description of actual code. And it wouldn’t help anyone understand the problem. The purpose of an analogy is to explain a concept by transferring from an easily-understandable situation to an analogous but less-easily-understood situation.

        Also, in a simple hill-climbing algorithm you generally do check every direction, whether there are two variables or 20. In both cases, though, it’s the entire landscape that you can’t see at once, not the immediate surroundings.

        Notice, finally, that the very language of this branch of heuristics is littered with examples from this analogy — hill-climbing algorithms, lanscapes, local peaks, etc. This hilly-landscape image is what is in the minds of most every researcher on the topic, so why not provide the analogy that every other researcher uses? That way, when someone comes across something like the Moving Peaks Benchmark, the reader is working from the same common analogy and understands the concept?

        • [sabik]

          Yeah, the analogy is ubiquitous. It’s still a good idea to be aware of its limitations.

          In particular, in a physical landscape with only two variables being optimised, “every direction” is a one-dimensional space; pretty easy to check.

          In a 12-dimensional landscape, “every direction” is an 11-dimensional space; almost as hard to check as the original problem.

          Normally, in a 12-dimensional landscape, you don’t check every direction, you just check 24 more-or-less arbitrarily chosen ones (one along each axis) and call it done. In the analogy, you’re at a place in the mountain that slopes down in the four cardinal directions, but — unnoticed — there may be a ridge rising NNW.

• [penguinchris]

  Just as an anecdote, most scientists I know (and I am one) make fun of (in a nice way) those scientists who mainly do computer models. They even regularly make fun of themselves.

  It’s clear to all “in the know” that there are major issues with relying on models, and that some people take them way too seriously. All scientists prefer doing physical experiments, or observing physical phenomena, and most only very begrudgingly turn to models.

  But while they’re easy to make fun of, no one doubts that ultimately they *are* very useful, at least much of the time and for certain fields. We just have to keep their limitations in mind, and not take their results as “truth”.

• [Jewels Vern]

  The disadvantage is that all models are based on assumptions, and the assumptions may not be obvious. Once the model is accepted, there is zero chance that the assumption will ever be examined. For example there are thousands of models assuming that stars run on fusion. Our own star obviously does not, but the assumption has been accepted and likely will never be reconsidered.

  • [Anonymous]

    The disadvantage is that all models are based on assumptions, and the assumptions may not be obvious. Once the model is accepted, there is zero chance that the assumption will ever be examined.

    Which is why we still rely entirely on Newtonian mechanics, phlogiston, Neptunism, the stead state theory, and of course explain stars through gravitational contraction and electricity.

  • [JonStewartMill]

    This is news to me. What *does* power the Sun, if not fusion?

    • [Jewels Vern]

      “This is news to me. What *does* power the Sun, if not fusion?”

      http://thunderbolts.info

      This is a very unpopular theory. There’s nothing wrong with it, just unpopular.

      • [Anonymous]

        When somebody starts off by using ancient myths as evidence over repeatable spectroscopy, there’s something wrong with their model.

  • [liamo]

    “Essentially, all models are wrong, but some are useful” – George Box

  • [Pantograph]

    The assumptions will be examined when observations of reality differ from the models. So when solar measurements start to differ from those predicted by the fusion model, science will build a new model.
    Don’t worry, the unicorn tear theory will win out in the end.

• [JonStewartMill]

  For example there are thousands of models assuming that stars run on fusion. Our own star obviously does not…

  For this statement to be true, or even defensible, the “thunderbolt hypothesis” (for lack of a better term) would have to explain and predict observed phenomena better than the fusion theory. Does it?

  • [SamSam]

    The idea that our sun runs on fusion is obviously a conspiracy created by eco-nuts spreading their radical alternative-fuel theories! The sun is run on 100% American oil. Wake up, sheeple!