Elon Musk may be gone from the Trump administration — and his friendship status with President Donald Trump may be at best uncertain — but his whirlwind stint in government certainly left its imprint. The Department of Government Efficiency (DOGE), his pet government-slashing project, remains entrenched in Washington. During his 130-day tenure, Musk led DOGE in eliminating about 260,000 federal employee jobs and gutting agencies supporting scientific research and humanitarian aid. But to date, DOGE claims to have saved the government $180 billion — well short of its ambitious (and frankly never realistic) target of cutting at least $2 trillion from the federal budget. And with Musk’s departure still fresh, there are reports that the federal government is trying to rehire federal workers who quit or were let go. For Elaine Kamarck, senior fellow at the Brookings Institution, DOGE’s tactics will likely end up being disastrous in the long run. “DOGE came in with these huge cuts, which were not attached to a plan,” she told Today, Explained co-host Sean Rameswaram. Kamarck knows all about making government more efficient. In the 1990s, she ran the Clinton administration’s Reinventing Government program. “I was Elon Musk,” she told Today, Explained. With the benefit of that experience, she assesses Musk’s record at DOGE, and what, if anything, the billionaire’s loud efforts at cutting government spending added up to. Below is an excerpt of the conversation, edited for length and clarity. There’s much more in the full podcast, so listen to Today, Explained wherever you get podcasts, including Apple Podcasts, Pandora, and Spotify. What do you think Elon Musk’s legacy is? Well, he will not have totally, radically reshaped the federal government. Absolutely not. In fact, there’s a high probability that on January 20, 2029, when the next president takes over, the federal government is about the same size as it is now, and is probably doing the same stuff that it’s doing now. What he did manage to do was insert chaos, fear, and loathing into the federal workforce. There was reporting in the Washington Post late last week that these cuts were so ineffective that the White House is actually reaching out to various federal employees who were laid off and asking them to come back, from the FDA to the IRS to even USAID. Which cuts are sticking at this point and which ones aren’t?First of all, in a lot of cases, people went to court and the courts have reversed those earlier decisions. So the first thing that happened is, courts said, “No, no, no, you can’t do it this way. You have to bring them back.” The second thing that happened is that Cabinet officers started to get confirmed by the Senate. And remember that a lot of the most spectacular DOGE stuff was happening in February. In February, these Cabinet secretaries were preparing for their Senate hearings. They weren’t on the job. Now that their Cabinet secretary’s home, what’s happening is they’re looking at these cuts and they’re saying, “No, no, no! We can’t live with these cuts because we have a mission to do.”As the government tries to hire back the people they fired, they’re going to have a tough time, and they’re going to have a tough time for two reasons. First of all, they treated them like dirt, and they’ve said a lot of insulting things. Second, most of the people who work for the federal government are highly skilled. They’re not paper pushers. We have computers to push our paper, right? They’re scientists. They’re engineers. They’re people with high skills, and guess what? They can get jobs outside the government. So there’s going to be real lasting damage to the government from the way they did this. And it’s analogous to the lasting damage that they’re causing at universities, where we now have top scientists who used to invent great cures for cancer and things like that, deciding to go find jobs in Europe because this culture has gotten so bad.What happens to this agency now? Who’s in charge of it?Well, what they’ve done is DOGE employees have been embedded in each of the organizations in the government, okay? And they basically — and the president himself has said this — they basically report to the Cabinet secretaries. So if you are in the Transportation Department, you have to make sure that Sean Duffy, who’s the secretary of transportation, agrees with you on what you want to do. And Sean Duffy has already had a fight during a Cabinet meeting with Elon Musk. You know that he has not been thrilled with the advice he’s gotten from DOGE. So from now on, DOGE is going to have to work hand in hand with Donald Trump’s appointed leaders.And just to bring this around to what we’re here talking about now, they’re in this huge fight over wasteful spending with the so-called big, beautiful bill. Does this just look like the government as usual, ultimately?It’s actually worse than normal. Because the deficit impacts are bigger than normal. It’s adding more to the deficit than previous bills have done. And the second reason it’s worse than normal is that everybody is still living in a fantasy world. And the fantasy world says that somehow we can deal with our deficits by cutting waste, fraud, and abuse. That is pure nonsense. Let me say it: pure nonsense.Where does most of the government money go? Does it go to some bureaucrats sitting on Pennsylvania Avenue? It goes to us. It goes to your grandmother and her Social Security and her Medicare. It goes to veterans in veterans benefits. It goes to Americans. That’s why it’s so hard to cut it. It’s so hard to cut it because it’s us. And people are living on it. Now, there’s a whole other topic that nobody talks about, and it’s called entitlement reform, right? Could we reform Social Security? Could we make the retirement age go from 67 to 68? That would save a lot of money. Could we change the cost of living? Nobody, nobody, nobody is talking about that. And that’s because we are in this crazy, polarized environment where we can no longer have serious conversations about serious issues. See More:

Extra-sensory perception The nine-armed octopus and the oddities of the cephalopod nervous system A mix of autonomous and top-down control manage the octopus's limbs. Kenna Hughes-Castleberry – Jun 7, 2025 8:00 am | 19 Credit: Nikos Stavrinidis / 500px Credit: Nikos Stavrinidis / 500px Story text Size Small Standard Large Width * Standard Wide Links Standard Orange * Subscribers only   Learn more With their quick-change camouflage and high level of intelligence, it’s not surprising that the public and scientific experts alike are fascinated by octopuses. Their abilities to recognize faces, solve puzzles, and learn behaviors from other octopuses make these animals a captivating study. To perform these processes and others, like crawling or exploring, octopuses rely on their complex nervous system, one that has become a focus for neuroscientists. With about 500 million neurons—around the same number as dogs—octopuses’ nervous systems are the most complex of any invertebrate. But, unlike vertebrate organisms, the octopus’s nervous system is also decentralized, with around 350 million neurons, or 66 percent of it, located in its eight arms. “This means each arm is capable of independently processing sensory input, initiating movement, and even executing complex behaviors—without direct instructions from the brain,” explains Galit Pelled, a professor of Mechanical Engineering, Radiology, and Neuroscience at Michigan State University who studies octopus neuroscience. “In essence, the arms have their own ‘mini-brains.’” A decentralized nervous system is one factor that helps octopuses adapt to changes, such as injury or predation, as seen in the case of an Octopus vulgaris, or common octopus, that was observed with nine arms by researchers at the ECOBAR lab at the Institute of Marine Research in Spain between 2021 and 2022. By studying outliers like this cephalopod, researchers can gain insight into how the animal’s detailed scaffolding of nerves changes and regrows over time, uncovering more about how octopuses have evolved over millennia in our oceans. Brains, brains, and more brains Because each arm of an octopus contains its own bundle of neurons, the limbs can operate semi-independently from the central brain, enabling faster responses since signals don’t always need to travel back and forth between the brain and the arms. In fact, Pelled and her team recently discovered that “neural signals recorded in the octopus arm can predict movement type within 100 milliseconds of stimulation, without central brain involvement.” She notes that “that level of localized autonomy is unprecedented in vertebrate systems.” Though each limb moves on its own, the movements of the octopus’s body are smooth and conducted with a coordinated elegance that allows the animal to exhibit some of the broadest range of behaviors, adapting on the fly to changes in its surroundings. “That means the octopus can react quickly to its environment, especially when exploring, hunting, or defending itself,” Pelled says. “For example, one arm can grab food while another is feeling around a rock, without needing permission from the brain. This setup also makes the octopus more resilient. If one arm is injured, the others still work just fine. And because so much decision-making happens at the arms, the central brain is freed up to focus on the bigger picture—like navigating or learning new tasks.” As if each limb weren’t already buzzing with neural activity, things get even more intricate when researchers zoom in further—to the nerves within each individual sucker, a ring of muscular tissue, which octopuses use to sense and taste their surroundings. “There is a sucker ganglion, or nerve center, located in the stalk of every sucker. For some species of octopuses, that’s over a thousand ganglia,” says Cassady Olson, a graduate student at the University of Chicago who works with Cliff Ragsdale, a leading expert in octopus neuroscience. Given that each sucker has its own nerve centers—connected by a long axial nerve cord running down the limb—and each arm has hundreds of suckers, things get complicated very quickly, as researchers have historically struggled to study this peripheral nervous system, as it’s called, within the octopus’s body. “The large size of the brain makes it both really exciting to study and really challenging,” says Z. Yan Wang, an assistant professor of biology and psychology at the University of Washington. “Many of the tools available for neuroscience have to be adjusted or customized specifically for octopuses and other cephalopods because of their unique body plans.” While each limb acts independently, signals are transmitted back to the octopus’s central nervous system. The octopus’ brain sits between its eyes at the front of its mantle, or head, couched between its two optic lobes, large bean-shaped neural organs that help octopuses see the world around them. These optic lobes are just two of the over 30 lobes experts study within the animal’s centralized brain, as each lobe helps the octopus process its environment. This elaborate neural architecture is critical given the octopus’s dual role in the ecosystem as both predator and prey. Without natural defenses like a hard shell, octopuses have evolved a highly adaptable nervous system that allows them to rapidly process information and adjust as needed, helping their chances of survival. Some similarities remain While the octopus’s decentralized nervous system makes it a unique evolutionary example, it does have some structures similar to or analogous to the human nervous system. “The octopus has a central brain mass located between its eyes, and an axial nerve cord running down each arm (similar to a spinal cord),” says Wang. “The octopus has many sensory systems that we are familiar with, such as vision, touch (somatosensation), chemosensation, and gravity sensing.” Neuroscientists have homed in on these similarities to understand how these structures may have evolved across the different branches in the tree of life. As the most recent common ancestor for humans and octopuses lived around 750 million years ago, experts believe that many similarities, from similar camera-like eyes to maps of neural activities, evolved separately in a process known as convergent evolution. While these similarities shed light on evolution's independent paths, they also offer valuable insights for fields like soft robotics and regenerative medicine. Occasionally, unique individuals—like an octopus with an unexpected number of limbs—can provide even deeper clues into how this remarkable nervous system functions and adapts. Nine arms, no problem In 2021, researchers from the Institute of Marine Research in Spain used an underwater camera to follow a male Octopus vulgaris, or common octopus. On its left side, three arms were intact, while the others were reduced to uneven, stumpy lengths, sharply bitten off at varying points. Although the researchers didn’t witness the injury itself, they observed that the front right arm—known as R1—was regenerating unusually, splitting into two separate limbs and giving the octopus a total of nine arms. “In this individual, we believe this condition was a result of abnormal regeneration [a genetic mutation] after an encounter with a predator,” explains Sam Soule, one of the researchers and the first author on the corresponding paper recently published in Animals. The researchers named the octopus Salvador due to its bifurcated arm coiling up on itself like the two upturned ends of Salvador Dali’s moustache. For two years, the team studied the cephalopod’s behavior and found that it used its bifurcated arm less when doing “riskier” movements such as exploring or grabbing food, which would force the animal to stretch its arm out and expose it to further injury. “One of the conclusions of our research is that the octopus likely retains a long-term memory of the original injury, as it tends to use the bifurcated arms for less risky tasks compared to the others,” elaborates Jorge Hernández Urcera, a lead author of the study. “This idea of lasting memory brought to mind Dalí’s famous painting The Persistence of Memory, which ultimately became the title of the paper we published on monitoring this particular octopus.” While the octopus acted more protective of its extra limb, its nervous system had adapted to using the extra appendage, as the octopus was observed, after some time recovering from its injuries, using its ninth arm for probing its environment. “That nine-armed octopus is a perfect example of just how adaptable these animals are,” Pelled adds. “Most animals would struggle with an unusual body part, but not the octopus. In this case, the octopus had a bifurcated (split) arm and still used it effectively, just like any other arm. That tells us the nervous system didn’t treat it as a mistake—it figured out how to make it work.” Kenna Hughes-Castleberry is the science communicator at JILA (a joint physics research institute between the National Institute of Standards and Technology and the University of Colorado Boulder) and a freelance science journalist. Her main writing focuses are quantum physics, quantum technology, deep technology, social media, and the diversity of people in these fields, particularly women and people from minority ethnic and racial groups. Follow her on LinkedIn or visit her website. 19 Comments

Analogous color schemes, composed of colors adjacent to each other on the color wheel, naturally produce harmonious and visually appealing images. These palettes offer photographers a gentle yet compelling approach to color, creating soothing, cohesive photographs that effortlessly draw viewers into a serene visual experience. Understanding Analogous Colors Analogous colors share common undertones and naturally complement each other, creating visual comfort and unity. Common examples include: Warm Analogous Schemes: Red, orange, and yellow hues evoke warmth, energy, and positivity. Cool Analogous Schemes: Green, blue, and violet hues convey calmness, tranquility, and introspection. Emotional Impact of Analogous Colors Each analogous color palette carries unique emotional associations. Selecting colors intentionally allows photographers to reinforce specific moods: Warm palettes promote excitement, vitality, and optimism, ideal for lifestyle or dynamic portrait photography. Cool palettes emphasize relaxation, peace, and contemplation, perfect for landscapes, seascapes, or reflective narratives. Techniques for Successful Analogous Compositions Creating powerful analogous compositions requires mindful planning and execution: Dominant and Supporting Colors: Choose a primary color to dominate your composition, using adjacent hues to provide depth and visual interest. Balancing Tonal Values: Incorporate a variety of tones within your analogous palette—from dark to light—to maintain visual balance and ensure clear differentiation between elements. Lighting to Enhance Color Harmony Lighting profoundly affects the appearance and interaction of colors within analogous schemes: Natural Light: Soft, natural daylight enhances color cohesion, bringing out subtle tonal variations and supporting visual harmony. Studio Lighting: Controlled lighting setups enable photographers to precisely manage color intensity, highlighting specific hues to strengthen composition and mood. Composition Tips for Analogous Schemes Intentional composition enhances the natural beauty and harmony of analogous color photography: Simplify Your Scene: Removing unnecessary elements ensures the analogous palette remains clear, focused, and impactful. Layering and Depth: Thoughtfully layering colors within your composition adds depth and encourages viewer engagement, inviting exploration of visual details. Refine Through Subtle Post-Processing Analogous color photography often benefits from nuanced post-processing: Adjust subtle differences in hue, saturation, and luminance to fine-tune harmony and enhance emotional resonance. Consider gentle contrast adjustments to highlight variations without disrupting overall visual tranquility. Analogous color schemes offer photographers a unique opportunity to create naturally harmonious and emotionally resonant images. By thoughtfully selecting your palette, carefully managing composition and lighting, and subtly refining in post-processing, you can produce photographs characterized by visual ease, depth, and emotional clarity. Embrace the subtle power of analogous colors and elevate your photographic storytelling with visually cohesive, harmonious compositions. Extended reading: Using Color to Strengthen Your Photographic Narratives The post Creating Harmony with Analogous Color Schemes appeared first on 500px.

The Monty Hall Problem is a well-known brain teaser from which we can learn important lessons in Decision Making that are useful in general and in particular for data scientists. If you are not familiar with this problem, prepare to be perplexed . If you are, I hope to shine light on aspects that you might not have considered . I introduce the problem and solve with three types of intuitions: Common — The heart of this post focuses on applying our common sense to solve this problem. We’ll explore why it fails us and what we can do to intuitively overcome this to make the solution crystal clear . We’ll do this by using visuals , qualitative arguments and some basic probabilities (not too deep, I promise). Bayesian — We will briefly discuss the importance of belief propagation. Causal — We will use a Graph Model to visualise conditions required to use the Monty Hall problem in real world settings.Spoiler alert I haven’t been convinced that there are any, but the thought process is very useful. I summarise by discussing lessons learnt for better data decision making. In regards to the Bayesian and Causal intuitions, these will be presented in a gentle form. For the mathematically inclined I also provide supplementary sections with short Deep Dives into each approach after the summary. (Note: These are not required to appreciate the main points of the article.) By examining different aspects of this puzzle in probability you will hopefully be able to improve your data decision making . Credit: Wikipedia First, some history. Let’s Make a Deal is a USA television game show that originated in 1963. As its premise, audience participants were considered traders making deals with the host, Monty Hall . At the heart of the matter is an apparently simple scenario: A trader is posed with the question of choosing one of three doors for the opportunity to win a luxurious prize, e.g, a car . Behind the other two were goats . The trader is shown three closed doors. The trader chooses one of the doors. Let’s call this (without loss of generalisability) door A and mark it with a . Keeping the chosen door closed, the host reveals one of the remaining doors showing a goat (let’s call this door C). The trader chooses door and the the host reveals door C showing a goat. The host then asks the trader if they would like to stick with their first choice or switch to the other remaining one (which we’ll call door B). If the trader guesses correct they win the prize . If not they’ll be shown another goat (also referred to as a zonk). What is the probability of being Zonked? Credit: Wikipedia Should the trader stick with their original choice of door A or switch to B? Before reading further, give it a go. What would you do? Most people are likely to have a gut intuition that “it doesn’t matter” arguing that in the first instance each door had a ⅓ chance of hiding the prize, and that after the host intervention , when only two doors remain closed, the winning of the prize is 50:50. There are various ways of explaining why the coin toss intuition is incorrect. Most of these involve maths equations, or simulations. Whereas we will address these later, we’ll attempt to solve by applying Occam’s razor: A principle that states that simpler explanations are preferable to more complex ones — William of Ockham (1287–1347) To do this it is instructive to slightly redefine the problem to a large N doors instead of the original three. The Large N-Door Problem Similar to before: you have to choose one of many doors. For illustration let’s say N=100. Behind one of the doors there is the prize and behind 99 (N-1) of the rest are goats . The 100 Door Monty Hall problem before the host intervention. You choose one door and the host reveals 98 (N-2) of the other doors that have goats leaving yours and one more closed . The 100 Door Monty Hall Problem after the host intervention. Should you stick with your door or make the switch? Should you stick with your original choice or make the switch? I think you’ll agree with me that the remaining door, not chosen by you, is much more likely to conceal the prize … so you should definitely make the switch! It’s illustrative to compare both scenarios discussed so far. In the next figure we compare the post host intervention for the N=3 setup (top panel) and that of N=100 (bottom): Post intervention settings for the N=3 setup (top) and N=100 (bottom). In both cases we see two shut doors, one of which we’ve chosen. The main difference between these scenarios is that in the first we see one goat and in the second there are more than the eye would care to see (unless you shepherd for a living). Why do most people consider the first case as a “50:50” toss up and in the second it’s obvious to make the switch? We’ll soon address this question of why. First let’s put probabilities of success behind the different scenarios. What’s The Frequency, Kenneth? So far we learnt from the N=100 scenario that switching doors is obviously beneficial. Inferring for the N=3 may be a leap of faith for most. Using some basic probability arguments here we’ll quantify why it is favourable to make the switch for any number door scenario N. We start with the standard Monty Hall problem (N=3). When it starts the probability of the prize being behind each of the doors A, B and C is p=⅓. To be explicit let’s define the Y parameter to be the door with the prize , i.e, p(Y=A)= p(Y=B)=p(Y=C)=⅓. The trick to solving this problem is that once the trader’s door A has been chosen , we should pay close attention to the set of the other doors {B,C}, which has the probability of p(Y∈{B,C})=p(Y=B)+p(Y=C)=⅔. This visual may help make sense of this: By being attentive to the {B,C} the rest should follow. When the goat is revealed it is apparent that the probabilities post intervention change. Note that for ease of reading I’ll drop the Y notation, where p(Y=A) will read p(A) and p(Y∈{B,C}) will read p({B,C}). Also for completeness the full terms after the intervention should be even longer due to it being conditional, e.g, p(Y=A|Z=C), p(Y∈{B,C}|Z=C), where Z is a parameter representing the choice of the host . (In the Bayesian supplement section below I use proper notation without this shortening.) p(A) remains ⅓ p({B,C})=p(B)+p(C) remains ⅔, p(C)=0; we just learnt that the goat is behind door C, not the prize. p(B)= p({B,C})-p(C) = ⅔ For anyone with the information provided by the host (meaning the trader and the audience) this means that it isn’t a toss of a fair coin! For them the fact that p(C) became zero does not “raise all other boats” (probabilities of doors A and B), but rather p(A) remains the same and p(B) gets doubled. The bottom line is that the trader should consider p(A) = ⅓ and p(B)=⅔, hence by switching they are doubling the odds at winning! Let’s generalise to N (to make the visual simpler we’ll use N=100 again as an analogy). When we start all doors have odds of winning the prize p=1/N. After the trader chooses one door which we’ll call D₁, meaning p(Y=D₁)=1/N, we should now pay attention to the remaining set of doors {D₂, …, Dₙ} will have a chance of p(Y∈{D₂, …, Dₙ})=(N-1)/N. When the host reveals (N-2) doors {D₃, …, Dₙ} with goats (back to short notation): p(D₁) remains 1/N p({D₂, …, Dₙ})=p(D₂)+p(D₃)+… + p(Dₙ) remains (N-1)/N p(D₃)=p(D₄)= …=p(Dₙ₋₁) =p(Dₙ) = 0; we just learnt that they have goats, not the prize. p(D₂)=p({D₂, …, Dₙ}) — p(D₃) — … — p(Dₙ)=(N-1)/N The trader should now consider two door values p(D₁)=1/N and p(D₂)=(N-1)/N. Hence the odds of winning improved by a factor of N-1! In the case of N=100, this means by an odds ratio of 99! (i.e, 99% likely to win a prize when switching vs. 1% if not). The improvement of odds ratios in all scenarios between N=3 to 100 may be seen in the following graph. The thin line is the probability of winning by choosing any door prior to the intervention p(Y)=1/N. Note that it also represents the chance of winning after the intervention, if they decide to stick to their guns and not switch p(Y=D₁|Z={D₃…Dₙ}). (Here I reintroduce the more rigorous conditional form mentioned earlier.) The thick line is the probability of winning the prize after the intervention if the door is switched p(Y=D₂|Z={D₃…Dₙ})=(N-1)/N: Probability of winning as a function of N. p(Y)=p(Y=no switch|Z)=1/N is the thin line; p(Y=switch|Z)=N/(N-1) is the thick one. (By definition the sum of both lines is 1 for each N.) Perhaps the most interesting aspect of this graph (albeit also by definition) is that the N=3 case has the highest probability before the host intervention , but the lowest probability after and vice versa for N=100. Another interesting feature is the quick climb in the probability of winning for the switchers: N=3: p=67% N=4: p=75% N=5=80% The switchers curve gradually reaches an asymptote approaching at 100% whereas at N=99 it is 98.99% and at N=100 is equal to 99%. This starts to address an interesting question: Why Is Switching Obvious For Large N But Not N=3? The answer is the fact that this puzzle is slightly ambiguous. Only the highly attentive realise that by revealing the goat (and never the prize!) the host is actually conveying a lot of information that should be incorporated into one’s calculation. Later we discuss the difference of doing this calculation in one’s mind based on intuition and slowing down by putting pen to paper or coding up the problem. How much information is conveyed by the host by intervening? A hand wavy explanation is that this information may be visualised as the gap between the lines in the graph above. For N=3 we saw that the odds of winning doubled (nothing to sneeze at!), but that doesn’t register as strongly to our common sense intuition as the 99 factor as in the N=100. I have also considered describing stronger arguments from Information Theory that provide useful vocabulary to express communication of information. However, I feel that this fascinating field deserves a post of its own, which I’ve published. The main takeaway for the Monty Hall problem is that I have calculated the information gain to be a logarithmic function of the number of doors c using this formula: Information Gain due to the intervention of the host for a setup with c doors. Full details in my upcoming article. For c=3 door case, e.g, the information gain is ⅔ bits (of a maximum possible 1.58 bits). Full details are in this article on entropy. To summarise this section, we use basic probability arguments to quantify the probabilities of winning the prize showing the benefit of switching for all N door scenarios. For those interested in more formal solutions using Bayesian and Causality on the bottom I provide supplement sections. In the next three final sections we’ll discuss how this problem was accepted in the general public back in the 1990s, discuss lessons learnt and then summarise how we can apply them in real-world settings. Being Confused Is OK “No, that is impossible, it should make no difference.” — Paul Erdős If you still don’t feel comfortable with the solution of the N=3 Monty Hall problem, don’t worry you are in good company! According to Vazsonyi (1999)¹ even Paul Erdős who is considered “of the greatest experts in probability theory” was confounded until computer simulations were demonstrated to him. When the original solution by Steve Selvin (1975)² was popularised by Marilyn vos Savant in her column “Ask Marilyn” in Parade magazine in 1990 many readers wrote that Selvin and Savant were wrong³. According to Tierney’s 1991 article in the New York Times, this included about 10,000 readers, including nearly 1,000 with Ph.D degrees⁴. On a personal note, over a decade ago I was exposed to the standard N=3 problem and since then managed to forget the solution numerous times. When I learnt about the large N approach I was quite excited about how intuitive it was. I then failed to explain it to my technical manager over lunch, so this is an attempt to compensate. I still have the same day job . While researching this piece I realised that there is a lot to learn in terms of decision making in general and in particular useful for data science. Lessons Learnt From Monty Hall Problem In his book Thinking Fast and Slow, the late Daniel Kahneman, the co-creator of Behaviour Economics, suggested that we have two types of thought processes: System 1 — fast thinking : based on intuition. This helps us react fast with confidence to familiar situations. System 2 – slow thinking : based on deep thought. This helps figure out new complex situations that life throws at us. Assuming this premise, you might have noticed that in the above you were applying both. By examining the visual of N=100 doors your System 1 kicked in and you immediately knew the answer. I’m guessing that in the N=3 you were straddling between System 1 and 2. Considering that you had to stop and think a bit when going throughout the probabilities exercise it was definitely System 2 . The decision maker’s struggle between System 1 and System 2 . Generated using Gemini Imagen 3 Beyond the fast and slow thinking I feel that there are a lot of data decision making lessons that may be learnt. (1) Assessing probabilities can be counter-intuitive … or Be comfortable with shifting to deep thought We’ve clearly shown that in the N=3 case. As previously mentioned it confounded many people including prominent statisticians. Another classic example is The Birthday Paradox , which shows how we underestimate the likelihood of coincidences. In this problem most people would think that one needs a large group of people until they find a pair sharing the same birthday. It turns out that all you need is 23 to have a 50% chance. And 70 for a 99.9% chance. One of the most confusing paradoxes in the realm of data analysis is Simpson’s, which I detailed in a previous article. This is a situation where trends of a population may be reversed in its subpopulations. The common with all these paradoxes is them requiring us to get comfortable to shifting gears from System 1 fast thinking to System 2 slow . This is also the common theme for the lessons outlined below. A few more classical examples are: The Gambler’s Fallacy , Base Rate Fallacy and the The Linda [bank teller] Problem . These are beyond the scope of this article, but I highly recommend looking them up to further sharpen ways of thinking about data. (2) … especially when dealing with ambiguity or Search for clarity in ambiguity Let’s reread the problem, this time as stated in “Ask Marilyn” Suppose you’re on a game show, and you’re given the choice of three doors: Behind one door is a car; behind the others, goats. You pick a door, say №1, and the host, who knows what’s behind the doors, opens another door, say №3, which has a goat. He then says to you, “Do you want to pick door №2?” Is it to your advantage to switch your choice? We discussed that the most important piece of information is not made explicit. It says that the host “knows what’s behind the doors”, but not that they open a door at random, although it’s implicitly understood that the host will never open the door with the car. Many real life problems in data science involve dealing with ambiguous demands as well as in data provided by stakeholders. It is crucial for the researcher to track down any relevant piece of information that is likely to have an impact and update that into the solution. Statisticians refer to this as “belief update”. (3) With new information we should update our beliefs This is the main aspect separating the Bayesian stream of thought to the Frequentist. The Frequentist approach takes data at face value (referred to as flat priors). The Bayesian approach incorporates prior beliefs and updates it when new findings are introduced. This is especially useful when dealing with ambiguous situations. To drive this point home, let’s re-examine this figure comparing between the post intervention N=3 setups (top panel) and the N=100 one (bottom panel). Copied from above. Post intervention settings for the N=3 setup (top) and N=100 (bottom). In both cases we had a prior belief that all doors had an equal chance of winning the prize p=1/N. Once the host opened one door (N=3; or 98 doors when N=100) a lot of valuable information was revealed whereas in the case of N=100 it was much more apparent than N=3. In the Frequentist approach, however, most of this information would be ignored, as it only focuses on the two closed doors. The Frequentist conclusion, hence is a 50% chance to win the prize regardless of what else is known about the situation. Hence the Frequentist takes Paul Erdős’ “no difference” point of view, which we now know to be incorrect. This would be reasonable if all that was presented were the two doors and not the intervention and the goats. However, if that information is presented, one should shift gears into System 2 thinking and update their beliefs in the system. This is what we have done by focusing not only on the shut door, but rather consider what was learnt about the system at large. For the brave hearted , in a supplementary section below called The Bayesian Point of View I solve for the Monty Hall problem using the Bayesian formalism. (4) Be one with subjectivity The Frequentist main reservation about “going Bayes” is that — “Statistics should be objective”. The Bayesian response is — the Frequentist’s also apply a prior without realising it — a flat one. Regardless of the Bayesian/Frequentist debate, as researchers we try our best to be as objective as possible in every step of the analysis. That said, it is inevitable that subjective decisions are made throughout. E.g, in a skewed distribution should one quote the mean or median? It highly depends on the context and hence a subjective decision needs to be made. The responsibility of the analyst is to provide justification for their choices first to convince themselves and then their stakeholders. (5) When confused — look for a useful analogy … but tread with caution We saw that by going from the N=3 setup to the N=100 the solution was apparent. This is a trick scientists frequently use — if the problem appears at first a bit too confusing/overwhelming, break it down and try to find a useful analogy. It is probably not a perfect comparison, but going from the N=3 setup to N=100 is like examining a picture from up close and zooming out to see the big picture. Think of having only a puzzle piece and then glancing at the jigsaw photo on the box. Monty Hall in 1976. Credit: Wikipedia and using Visual Paradigm Online for the puzzle effect Note: whereas analogies may be powerful, one should do so with caution, not to oversimplify. Physicists refer to this situation as the spherical cow method, where models may oversimplify complex phenomena. I admit that even with years of experience in applied statistics at times I still get confused at which method to apply. A large part of my thought process is identifying analogies to known solved problems. Sometimes after making progress in a direction I will realise that my assumptions were wrong and seek a new direction. I used to quip with colleagues that they shouldn’t trust me before my third attempt … (6) Simulations are powerful but not always necessary It’s interesting to learn that Paul Erdős and other mathematicians were convinced only after seeing simulations of the problem. I am two-minded about usage of simulations when it comes to problem solving. On the one hand simulations are powerful tools to analyse complex and intractable problems. Especially in real life data in which one wants a grasp not only of the underlying formulation, but also stochasticity. And here is the big BUT — if a problem can be analytically solved like the Monty Hall one, simulations as fun as they may be (such as the MythBusters have done⁶), may not be necessary. According to Occam’s razor, all that is required is a brief intuition to explain the phenomena. This is what I attempted to do here by applying common sense and some basic probability reasoning. For those who enjoy deep dives I provide below supplementary sections with two methods for analytical solutions — one using Bayesian statistics and another using Causality. [Update] After publishing the first version of this article there was a comment that Savant’s solution³ may be simpler than those presented here. I revisited her communications and agreed that it should be added. In the process I realised three more lessons may be learnt. (7) A well designed visual goes a long way Continuing the principle of Occam’s razor, Savant explained³ quite convincingly in my opinion: You should switch. The first door has a 1/3 chance of winning, but the second door has a 2/3 chance. Here’s a good way to visualize what happened. Suppose there are a million doors, and you pick door #1. Then the host, who knows what’s behind the doors and will always avoid the one with the prize, opens them all except door #777,777. You’d switch to that door pretty fast, wouldn’t you? Hence she provided an abstract visual for the readers. I attempted to do the same with the 100 doors figures. Marilyn vos Savant who popularised the Monty Hall Problem. Credit: Ben David on Flickr under license As mentioned many readers, and especially with backgrounds in maths and statistics, still weren’t convinced. She revised³ with another mental image: The benefits of switching are readily proven by playing through the six games that exhaust all the possibilities. For the first three games, you choose #1 and “switch” each time, for the second three games, you choose #1 and “stay” each time, and the host always opens a loser. Here are the results. She added a table with all the scenarios. I took some artistic liberty and created the following figure. As indicated, the top batch are the scenarios in which the trader switches and the bottom when they switch. Lines in green are games which the trader wins, and in red when they get zonked. The symbolised the door chosen by the trader and Monte Hall then chooses a different door that has a goat behind it. Adaptation of Savant’s table³ of six scenarios that shows the solution to the Monty Hall Problem We clearly see from this diagram that the switcher has a ⅔ chance of winning and those that stay only ⅓. This is yet another elegant visualisation that clearly explains the non intuitive. It strengthens the claim that there is no real need for simulations in this case because all they would be doing is rerunning these six scenarios. One more popular solution is decision tree illustrations. You can find these in the Wikipedia page, but I find it’s a bit redundant to Savant’s table. The fact that we can solve this problem in so many ways yields another lesson: (8) There are many ways to skin a … problem Of the many lessons that I have learnt from the writings of late Richard Feynman, one of the best physics and ideas communicators, is that a problem can be solved many ways. Mathematicians and Physicists do this all the time. A relevant quote that paraphrases Occam’s razor: If you can’t explain it simply, you don’t understand it well enough — attributed to Albert Einstein And finally (9) Embrace ignorance and be humble ‍ “You are utterly incorrect … How many irate mathematicians are needed to get you to change your mind?” — Ph.D from Georgetown University “May I suggest that you obtain and refer to a standard textbook on probability before you try to answer a question of this type again?” — Ph.D from University of Florida “You’re in error, but Albert Einstein earned a dearer place in the hearts of people after he admitted his errors.” — Ph.D. from University of Michigan Ouch! These are some of the said responses from mathematicians to the Parade article. Such unnecessary viciousness. You can check the reference³ to see the writer’s names and other like it. To whet your appetite: “You blew it, and you blew it big!”, , “You made a mistake, but look at the positive side. If all those Ph.D.’s were wrong, the country would be in some very serious trouble.”, “I am in shock that after being corrected by at least three mathematicians, you still do not see your mistake.”. And as expected from the 1990s perhaps the most embarrassing one was from a resident of Oregon: “Maybe women look at math problems differently than men.” These make me cringe and be embarrassed to be associated by gender and Ph.D. title with these graduates and professors. Hopefully in the 2020s most people are more humble about their ignorance. Yuval Noah Harari discusses the fact that the Scientific Revolution of Galileo Galilei et al. was not due to knowledge but rather admittance of ignorance. “The great discovery that launched the Scientific Revolution was the discovery that humans do not know the answers to their most important questions” — Yuval Noah Harari Fortunately for mathematicians’ image, there were also quiet a lot of more enlightened comments. I like this one from one Seth Kalson, Ph.D. of MIT: You are indeed correct. My colleagues at work had a ball with this problem, and I dare say that most of them, including me at first, thought you were wrong! We’ll summarise by examining how, and if, the Monty Hall problem may be applied in real-world settings, so you can try to relate to projects that you are working on. Application in Real World Settings Researching for this article I found that beyond artificial setups for entertainment⁶ ⁷ there aren’t practical settings for this problem to use as an analogy. Of course, I may be wrong⁸ and would be glad to hear if you know of one. One way of assessing the viability of an analogy is using arguments from causality which provides vocabulary that cannot be expressed with standard statistics. In a previous post I discussed the fact that the story behind the data is as important as the data itself. In particular Causal Graph Models visualise the story behind the data, which we will use as a framework for a reasonable analogy. For the Monty Hall problem we can build a Causal Graph Model like this: Reading: The door chosen by the trader is independent from that with the prize and vice versa. As important, there is no common cause between them that might generate a spurious correlation. The host’s choice depends on both and . By comparing causal graphs of two systems one can get a sense for how analogous both are. A perfect analogy would require more details, but this is beyond the scope of this article. Briefly, one would want to ensure similar functions between the parameters (referred to as the Structural Causal Model; for details see in the supplementary section below called The Causal Point of View). Those interested in learning further details about using Causal Graphs Models to assess causality in real world problems may be interested in this article. Anecdotally it is also worth mentioning that on Let’s Make a Deal, Monty himself has admitted years later to be playing mind games with the contestants and did not always follow the rules, e.g, not always doing the intervention as “it all depends on his mood”⁴. In our setup we assumed perfect conditions, i.e., a host that does not skew from the script and/or play on the trader’s emotions. Taking this into consideration would require updating the Graphical Model above, which is beyond the scope of this article. Some might be disheartened to realise at this stage of the post that there might not be real world applications for this problem. I argue that lessons learnt from the Monty Hall problem definitely are. Just to summarise them again: (1) Assessing probabilities can be counter intuitive …(Be comfortable with shifting to deep thought ) (2) … especially when dealing with ambiguity(Search for clarity ) (3) With new information we should update our beliefs (4) Be one with subjectivity (5) When confused — look for a useful analogy … but tread with caution (6) Simulations are powerful but not always necessary (7) A well designed visual goes a long way (8) There are many ways to skin a … problem (9) Embrace ignorance and be humble ‍ While the Monty Hall Problem might seem like a simple puzzle, it offers valuable insights into decision-making, particularly for data scientists. The problem highlights the importance of going beyond intuition and embracing a more analytical, data-driven approach. By understanding the principles of Bayesian thinking and updating our beliefs based on new information, we can make more informed decisions in many aspects of our lives, including data science. The Monty Hall Problem serves as a reminder that even seemingly straightforward scenarios can contain hidden complexities and that by carefully examining available information, we can uncover hidden truths and make better decisions. At the bottom of the article I provide a list of resources that I found useful to learn about this topic. Credit: Wikipedia Loved this post? Join me on LinkedIn or Buy me a coffee! Credits Unless otherwise noted, all images were created by the author. Many thanks to Jim Parr, Will Reynolds, and Betty Kazin for their useful comments. In the following supplementary sections I derive solutions to the Monty Hall’s problem from two perspectives: Bayesian Causal Both are motivated by questions in textbook: Causal Inference in Statistics A Primer by Judea Pearl, Madelyn Glymour, and Nicholas P. Jewell (2016). Supplement 1: The Bayesian Point of View This section assumes a basic understanding of Bayes’ Theorem, in particular being comfortable conditional probabilities. In other words if this makes sense: We set out to use Bayes’ theorem to prove that switching doors improves chances in the N=3 Monty Hall Problem. (Problem 1.3.3 of the Primer textbook.) We define X — the chosen door Y— the door with the prize Z — the door opened by the host Labelling the doors as A, B and C, without loss of generality, we need to solve for: Using Bayes’ theorem we equate the left side as and the right one as: Most components are equal (remember that P(Y=A)=P(Y=B)=⅓ so we are left to prove: In the case where Y=B (the prize is behind door B ), the host has only one choice (can only select door C ), making P(X=A, Z=C|Y=B)= 1. In the case where Y=A (the prize is behind door A ), the host has two choices (doors B and C ) , making P(X=A, Z=C|Y=A)= 1/2. From here: Quod erat demonstrandum. Note: if the “host choices” arguments didn’t make sense look at the table below showing this explicitly. You will want to compare entries {X=A, Y=B, Z=C} and {X=A, Y=A, Z=C}. Supplement 2: The Causal Point of View The section assumes a basic understanding of Directed Acyclic Graphs (DAGs) and Structural Causal Models (SCMs) is useful, but not required. In brief: DAGs qualitatively visualise the causal relationships between the parameter nodes. SCMs quantitatively express the formula relationships between the parameters. Given the DAG we are going to define the SCM that corresponds to the classic N=3 Monty Hall problem and use it to describe the joint distribution of all variables. We later will generically expand to N. (Inspired by problem 1.5.4 of the Primer textbook as well as its brief mention of the N door problem.) We define X — the chosen door Y — the door with the prize Z — the door opened by the host According to the DAG we see that according to the chain rule: The SCM is defined by exogenous variables U , endogenous variables V, and the functions between them F: U = {X,Y}, V={Z}, F= {f(Z)} where X, Y and Z have door values: D = {A, B, C} The host choice is f(Z) defined as: In order to generalise to N doors, the DAG remains the same, but the SCM requires to update D to be a set of N doors Dᵢ: {D₁, D₂, … Dₙ}. Exploring Example Scenarios To gain an intuition for this SCM, let’s examine 6 examples of 27 (=3³) : When X=Y (i.e., the prize is behind the chosen door ) P(Z=A|X=A, Y=A) = 0; cannot choose the participant’s door P(Z=B|X=A, Y=A) = 1/2; is behind → chooses B at 50% P(Z=C|X=A, Y=A) = 1/2; is behind → chooses C at 50%(complementary to the above) When X≠Y (i.e., the prize is not behind the chosen door ) P(Z=A|X=A, Y=B) = 0; cannot choose the participant’s door P(Z=B|X=A, Y=B) = 0; cannot choose prize door P(Z=C|X=A, Y=B) = 1; has not choice in the matter(complementary to the above) Calculating Joint Probabilities Using logic let’s code up all 27 possibilities in python df = pd.DataFrame({"X": (["A"] * 9) + (["B"] * 9) + (["C"] * 9), "Y": ((["A"] * 3) + (["B"] * 3) + (["C"] * 3) )* 3, "Z": ["A", "B", "C"] * 9}) df["P(Z|X,Y)"] = None p_x = 1./3 p_y = 1./3 df.loc[df.query("X == Y == Z").index, "P(Z|X,Y)"] = 0 df.loc[df.query("X == Y != Z").index, "P(Z|X,Y)"] = 0.5 df.loc[df.query("X != Y == Z").index, "P(Z|X,Y)"] = 0 df.loc[df.query("Z == X != Y").index, "P(Z|X,Y)"] = 0 df.loc[df.query("X != Y").query("Z != Y").query("Z != X").index, "P(Z|X,Y)"] = 1 df["P(X, Y, Z)"] = df["P(Z|X,Y)"] * p_x * p_y print(f"Testing normalisation of P(X,Y,Z) {df['P(X, Y, Z)'].sum()}") df yields Resources This Quora discussion by Joshua Engel helped me shape a few aspects of this article. Causal Inference in Statistics A Primer / Pearl, Glymour & Jewell (2016) — excellent short text book (site) I also very much enjoy Tim Harford’s podcast Cautionary Tales. He wrote about this topic on November 3rd 2017 for the Financial Times: Monty Hall and the game show stick-or-switch conundrum Footnotes ¹ Vazsonyi, Andrew (December 1998 — January 1999). “Which Door Has the Cadillac?” (PDF). Decision Line: 17–19. Archived from the original (PDF) on 13 April 2014. Retrieved 16 October 2012. ² Steve Selvin to the American Statistician in 1975.[1][2] ³Game Show Problem by Marilyn vos Savant’s “Ask Marilyn” in marilynvossavant.com (web archive): “This material in this article was originally published in PARADE magazine in 1990 and 1991” ⁴Tierney, John (21 July 1991). “Behind Monty Hall’s Doors: Puzzle, Debate and Answer?”. The New York Times. Retrieved 18 January 2008. ⁵ Kahneman, D. (2011). Thinking, fast and slow. Farrar, Straus and Giroux. ⁶ MythBusters Episode 177 “Pick a Door” (Wikipedia) Watch Mythbuster’s approach ⁶Monty Hall Problem on Survivor Season 41 (LinkedIn, YouTube) Watch Survivor’s take on the problem ⁷ Jingyi Jessica Li (2024) How the Monty Hall problem is similar to the false discovery rate in high-throughput data analysis.Whereas the author points about “similarities” between hypothesis testing and the Monty Hall problem, I think that this is a bit misleading. The author is correct that both problems change by the order in which processes are done, but that is part of Bayesian statistics in general, not limited to the Monty Hall problem. The post 🚪🚪🐐 Lessons in Decision Making from the Monty Hall Problem appeared first on Towards Data Science.

Cool Finds Archaeologists Say They’ve Discovered a Hidden Chamber Where Elites Used Hallucinogens 2,500 Years Ago Unearthed in Peru, the small underground room may have been used for rituals involving psychoactive drugs. New research suggests these “exclusive” events were reserved for the elite An artistic rendering of the stone chamber where the tubes were discovered Daniel Contrera Long before the rise of the Inca empire, a group called the Chavín people thrived in ancient Peru. They’re known for their elaborate stone structures, which were located at an archaeological site now known as Chavín de Huántar in the Andean highlands. During recent excavations at the site, researchers found dozens of hollow bones packed with sediment. They think the artifacts could be ancient drug paraphernalia. “The tubes are analogous to the rolled-up bills that high-rollers snort cocaine through in the movies,” Daniel Contreras, an archaeologist at the University of Florida and a co-author of the study, tells Live Science’s Kristina Killgrove. The site of Chavín de Huántar, located in Peru at an elevation of 10,000 feet, holds the ruins of several monumental buildings. Daniel Contreras The tubes were found in small underground chambers, where experts think they may have been used by Chavín elites, according to a recent study published in the journal PNAS. Chemical and microscopic analyses of the tubes revealed traces of nicotine and vilca bean, a hallucinogen related to the drug DMT. Vilca was commonly used among pre-Columbian populations of the Andes. When smoked or consumed, vilca would bring about an altered mental state. Many ancient cultures used hallucinogens communally, but the Chavín rituals appear to have been “exclusive,” according to a statement from the university. These rituals may have taken place in rooms that held “only a handful of participants at a time, creating an air of mystique and control.” “This is compelling evidence that psychoactive plants were part of formalized and tightly-controlled rituals rather than individual vision-quests or shamanic healing practices,” Contreras tells CBS News’ Emily Mae Czachor. “As such, they seem to have been an important element in the long-term transition from small egalitarian societies to large stratified ones, where social, political and economic inequality were thought of as normal and to be expected rather than unusual.” Snuff tubes made from hollowed bones may have been used to inhale hallucinogenic drugs. Daniel Contreras The Chavín society flourished between roughly 900 and 200 B.C.E. Chavín de Huántar is known for its intricate stone carvings, which often depicted “animal-human hybrids or transformations of human into beast,” and its large network of tunnels, writes Science’s Lizzie Wade. The area is full of imported seashells and obsidian, and Chavín art has been discovered throughout the Andes and on the Peruvian coast, suggesting a “broad cultural reach.” “Chavín was part of the first big moment in Andean prehistory when people, ideas and goods were circulating quite extensively,” Contreras tells Science. Depictions of psychoactive effects in Chavín iconography PNAS For many years, historians assumed the Chavín people conducted rituals using drugs. “What’s exciting about this paper is that, for the first time, we have actual evidence,” José Capriles, an archaeologist at Pennsylvania State University who studies ancient psychoactive drug use but wasn’t involved in the study, tells Science. The researchers think the drugs may have influenced the Chavín society’s class system. By restricting access to these “profound, even terrifying” experiences, Chavín rulers could have convinced their people that “leadership was intertwined with mystical power and part of the natural order,” per the statement. “Taking psychoactives was not just about seeing visions,” says Contreras in the statement. “It was part of a tightly controlled ritual, likely reserved for a select few, reinforcing the social hierarchy.” Get the latest stories in your inbox every weekday.