You are right but also missing something : let's say your father gives you blue gene, and your mother brown gene. Then you will have brown eyes with blue recessive gene.
Now you marry a girl with brown eyes as well and two brown genes. The gene pool for your future children will look as following:
Parent 1: Rr
Parent 2: RR
Since your children receive half of your genes randomly, they have only 50% of getting your recessive blue gene - if they don't, it disappears. They also have 0% chance of having blue eyes, because they can't get two blue genes in this pool.
Recessive genes can go extinct very quickly, either because of interbreeding or people not having enough kids (thus lowering the chance of passing recessive genes).